GNU bug report logs - #78067
Conversion to unprivileged guix-daemon breaks ssh

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 78067 in the body.
You can then email your comments to 78067 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: "Zack Weinberg" <zack <at> owlfolio.org>
To: bug-guix <at> gnu.org
Subject: Conversion to unprivileged guix-daemon breaks ssh
Date: Fri, 25 Apr 2025 16:34:45 -0400

I just switched my Guix System-based server over to unprivileged guix-
daemon, after which I was unable to ssh into it.  From the client, the
syndrome looks like this (shell variables indicate redactions):

$ ssh $my_server
kex_exchange_identification: read: Connection reset by peer
Connection reset by $ip_address port 22

or with -v:

$ ssh -v $my_server
OpenSSH_9.9p2, OpenSSL 3.3.3 11 Feb 2025
debug1: Reading configuration data $HOME/.ssh/config
debug1: $HOME/.ssh/config line 31: Applying options for tinka
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Connecting to $my_server [$ip_address] port 22.
debug1: Connection established.
debug1: identity file $HOME/.ssh/$private_key type 0
[more lines about identity files omitted...]
debug1: Local version string SSH-2.0-OpenSSH_9.9
kex_exchange_identification: read: Connection reset by peer
Connection reset by $ip_address port 22

Fortunately, I can get into the server using a serial console,
and the problem is quite clear from server-side logs:

2025-04-25 20:27:40 localhost shepherd[1]: Accepted connection on 0.0.0.0:22 from $client_ip:51626.
2025-04-25 20:27:40 localhost shepherd[1]: Starting service sshd-69...
2025-04-25 20:27:40 localhost shepherd[1]: Service sshd-69 has been started.
2025-04-25 20:27:40 localhost shepherd[1]: Service sshd-69 started.
2025-04-25 20:27:40 localhost shepherd[1]: Service sshd-69 running with value #<<process> id: 234 command: ("/gnu/store/8kman284vvlzk2hgy1bv1xzys3rfdzlr-openssh-10.0p1/sbin/sshd" "-D" "-f" "/gnu/store/vwy5d5mj35rh147iwzkzxijld0gx06mb-sshd_config" "-i")>.
2025-04-25 20:27:40 localhost sshd[234]: fatal: /gnu/store/8kman284vvlzk2hgy1bv1xzys3rfdzlr-openssh-10.0p1/var/empty must be owned by root and not group or world-writable.
2025-04-25 20:27:40 localhost shepherd[1]: 0 connections still in use after sshd-69 termination.
2025-04-25 20:27:40 localhost shepherd[1]: Service sshd-69 (PID 234) exited with 255.
2025-04-25 20:27:40 localhost shepherd[1]: Service sshd-69 has been disabled.
2025-04-25 20:27:40 localhost shepherd[1]: Transient service sshd-69 terminated, now unregistered.

# ls -l /gnu/store/8kman284vvlzk2hgy1bv1xzys3rfdzlr-openssh-10.0p1
total 24
dr-xr-xr-x 2 guix-daemon guix-daemon 4096 Jan  1  1970 bin/
dr-xr-xr-x 2 guix-daemon guix-daemon 4096 Jan  1  1970 etc/
dr-xr-xr-x 2 guix-daemon guix-daemon 4096 Jan  1  1970 libexec/
dr-xr-xr-x 2 guix-daemon guix-daemon 4096 Jan  1  1970 sbin/
dr-xr-xr-x 4 guix-daemon guix-daemon 4096 Jan  1  1970 share/
dr-xr-xr-x 3 guix-daemon guix-daemon 4096 Jan  1  1970 var/

# ls -l /gnu/store/8kman284vvlzk2hgy1bv1xzys3rfdzlr-openssh-10.0p1/var
total 4
dr-xr-xr-x 2 guix-daemon guix-daemon 4096 Jan  1  1970 empty/

`chown root:root /gnu/store/8kman284vvlzk2hgy1bv1xzys3rfdzlr-openssh-10.0p1/var/empty`
is sufficient to fix ssh, but I wonder if store contents in general should maybe remain
owned by root regardless of whether the daemon is running as an unprivileged user.
It seems likely to me that this will not be the only such problem.

zw

Message #8 received at 78067 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: "Zack Weinberg" <zack <at> owlfolio.org>
Cc: 78067 <at> debbugs.gnu.org
Subject: Re: bug#78067: Conversion to unprivileged guix-daemon breaks ssh
Date: Mon, 05 May 2025 14:58:28 +0200

Hi Zack,

"Zack Weinberg" <zack <at> owlfolio.org> writes:

> I just switched my Guix System-based server over to unprivileged guix-
> daemon, after which I was unable to ssh into it.  From the client, the
> syndrome looks like this (shell variables indicate redactions):

[...]

> 2025-04-25 20:27:40 localhost shepherd[1]: Service sshd-69 running
> with value #<<process> id: 234 command:
> ("/gnu/store/8kman284vvlzk2hgy1bv1xzys3rfdzlr-openssh-10.0p1/sbin/sshd"
> "-D" "-f" "/gnu/store/vwy5d5mj35rh147iwzkzxijld0gx06mb-sshd_config"
> "-i")>.
> 2025-04-25 20:27:40 localhost sshd[234]: fatal:
> /gnu/store/8kman284vvlzk2hgy1bv1xzys3rfdzlr-openssh-10.0p1/var/empty
> must be owned by root and not group or world-writable.

D’oh.  The fix here is to tell OpenSSH to use /var/empty instead.  Do
you know how to do that via sshd_config?

> `chown root:root /gnu/store/8kman284vvlzk2hgy1bv1xzys3rfdzlr-openssh-10.0p1/var/empty`
> is sufficient to fix ssh, but I wonder if store contents in general should maybe remain
> owned by root regardless of whether the daemon is running as an unprivileged user.
> It seems likely to me that this will not be the only such problem.

You should never manually modify files in the store or change their
ownership.  In the case above, the daemon will now be unable to delete
this store item when you run ‘guix gc’.

Thanks for the bug report,
Ludo’.

Message #11 received at 78067 <at> debbugs.gnu.org (full text, mbox):

From: "Zack Weinberg" <zack <at> owlfolio.org>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 78067 <at> debbugs.gnu.org
Subject: Re: bug#78067: Conversion to unprivileged guix-daemon breaks ssh
Date: Mon, 05 May 2025 11:59:54 -0400

On Mon, May 5, 2025, at 8:58 AM, Ludovic Courtès wrote:
>> 2025-04-25 20:27:40 localhost sshd[234]: fatal:
>> /gnu/store/8kman284vvlzk2hgy1bv1xzys3rfdzlr-openssh-10.0p1/var/empty
>> must be owned by root and not group or world-writable.
>
> D’oh.  The fix here is to tell OpenSSH to use /var/empty instead.  Do
> you know how to do that via sshd_config?

I don't see any way to do that in `man sshd_config`, but there is a
relevant AC_ARG_WITH option:

> PRIVSEP_PATH=/var/empty
> AC_ARG_WITH([privsep-path],
> 	[  --with-privsep-path=xxx Path for privilege separation chroot (default=/var/empty)],
> 	[
>		if test -n "$withval"  &&  test "x$withval" != "xno"  &&  \
> 	 	    test "x${withval}" != "xyes"; then
> 			PRIVSEP_PATH=$withval
> 		fi
> 	]
> )
> AC_SUBST([PRIVSEP_PATH])

https://github.com/openssh/openssh-portable/blob/61525ba967ac1bb7394ea0792aa6030bcbbad049/configure.ac#L4984-L4994

... But maybe all we need to do is *remove* the 'reset-/var/empty stanza from here?
https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/ssh.scm#n277

> You should never manually modify files in the store or change their
> ownership.  In the case above, the daemon will now be unable to delete
> this store item when you run ‘guix gc’.

Good to know, thanks.

zw

Message #16 received at 78067-done <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Sergey Trofimov <sarg <at> sarg.org.ru>
Cc: 78067-done <at> debbugs.gnu.org, 77968-done <at> debbugs.gnu.org
Subject: Re: bug#77968: [PATCH 0/1] gnu: openssh: Trust store items owned by
 guix-daemon.
Date: Tue, 06 May 2025 00:22:29 +0200

Hi Sergey,

Sergey Trofimov <sarg <at> sarg.org.ru> writes:

> * gnu/packages/patches/openssh-trust-guix-store-directory.patch
> (openssh): Adjust to trust files in guix store owned by guix-daemon.
> * gnu/packages/ssh.scm (openssh): [phases]: Append ending slash when
> substituting STORE_DIRECTORY. Use default PRIVSEP_PATH (/var/empty).
>
> Change-Id: I3bd01f8b9d6406e3b886eea8f4b8c265a51cc72f

I adjusted the commit log to refer to the bug and committed it as
eab097c682ed31efd8668f46fce8de8f73b92849.

Thanks!

Ludo’.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.