GNU logs - #78067, boring messages

Message sent to bug-guix@HIDDEN:

Subject: bug#78067: Conversion to unprivileged guix-daemon breaks ssh
Resent-From: "Zack Weinberg" <zack@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Fri, 25 Apr 2025 20:36:02 +0000
Resent-Message-ID: <handler.78067.B.174561332226325 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: 78067 <at> debbugs.gnu.org
Feedback-ID: i876146a2:Fastmail
MIME-Version: 1.0
Date: Fri, 25 Apr 2025 16:34:45 -0400
From: "Zack Weinberg" <zack@HIDDEN>
Message-Id: <8d70405b-7f96-43a2-90de-8b5adde8873d@HIDDEN>
Content-Type: text/plain
Content-Transfer-Encoding: 7bit
Received-SPF: pass client-ip=103.168.172.159; envelope-from=zack@HIDDEN;
 helo=fhigh-a8-smtp.messagingengine.com
X-Spam_score_int: -27
X-Spam_score: -2.8
X-Spam_bar: --
X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,
 DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1,
 RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001,
 RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_PASS=-0.001,
 SPF_PASS=-0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-Spam-Score: 0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.3 (/)

I just switched my Guix System-based server over to unprivileged guix-
daemon, after which I was unable to ssh into it.  From the client, the
syndrome looks like this (shell variables indicate redactions):

$ ssh $my_server
kex_exchange_identification: read: Connection reset by peer
Connection reset by $ip_address port 22

or with -v:

$ ssh -v $my_server
OpenSSH_9.9p2, OpenSSL 3.3.3 11 Feb 2025
debug1: Reading configuration data $HOME/.ssh/config
debug1: $HOME/.ssh/config line 31: Applying options for tinka
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Connecting to $my_server [$ip_address] port 22.
debug1: Connection established.
debug1: identity file $HOME/.ssh/$private_key type 0
[more lines about identity files omitted...]
debug1: Local version string SSH-2.0-OpenSSH_9.9
kex_exchange_identification: read: Connection reset by peer
Connection reset by $ip_address port 22

Fortunately, I can get into the server using a serial console,
and the problem is quite clear from server-side logs:

2025-04-25 20:27:40 localhost shepherd[1]: Accepted connection on 0.0.0.0:22 from $client_ip:51626.
2025-04-25 20:27:40 localhost shepherd[1]: Starting service sshd-69...
2025-04-25 20:27:40 localhost shepherd[1]: Service sshd-69 has been started.
2025-04-25 20:27:40 localhost shepherd[1]: Service sshd-69 started.
2025-04-25 20:27:40 localhost shepherd[1]: Service sshd-69 running with value #<<process> id: 234 command: ("/gnu/store/8kman284vvlzk2hgy1bv1xzys3rfdzlr-openssh-10.0p1/sbin/sshd" "-D" "-f" "/gnu/store/vwy5d5mj35rh147iwzkzxijld0gx06mb-sshd_config" "-i")>.
2025-04-25 20:27:40 localhost sshd[234]: fatal: /gnu/store/8kman284vvlzk2hgy1bv1xzys3rfdzlr-openssh-10.0p1/var/empty must be owned by root and not group or world-writable.
2025-04-25 20:27:40 localhost shepherd[1]: 0 connections still in use after sshd-69 termination.
2025-04-25 20:27:40 localhost shepherd[1]: Service sshd-69 (PID 234) exited with 255.
2025-04-25 20:27:40 localhost shepherd[1]: Service sshd-69 has been disabled.
2025-04-25 20:27:40 localhost shepherd[1]: Transient service sshd-69 terminated, now unregistered.

# ls -l /gnu/store/8kman284vvlzk2hgy1bv1xzys3rfdzlr-openssh-10.0p1
total 24
dr-xr-xr-x 2 guix-daemon guix-daemon 4096 Jan  1  1970 bin/
dr-xr-xr-x 2 guix-daemon guix-daemon 4096 Jan  1  1970 etc/
dr-xr-xr-x 2 guix-daemon guix-daemon 4096 Jan  1  1970 libexec/
dr-xr-xr-x 2 guix-daemon guix-daemon 4096 Jan  1  1970 sbin/
dr-xr-xr-x 4 guix-daemon guix-daemon 4096 Jan  1  1970 share/
dr-xr-xr-x 3 guix-daemon guix-daemon 4096 Jan  1  1970 var/

# ls -l /gnu/store/8kman284vvlzk2hgy1bv1xzys3rfdzlr-openssh-10.0p1/var
total 4
dr-xr-xr-x 2 guix-daemon guix-daemon 4096 Jan  1  1970 empty/

`chown root:root /gnu/store/8kman284vvlzk2hgy1bv1xzys3rfdzlr-openssh-10.0p1/var/empty`
is sufficient to fix ssh, but I wonder if store contents in general should maybe remain
owned by root regardless of whether the daemon is running as an unprivileged user.
It seems likely to me that this will not be the only such problem.

zw

Message sent:

Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Mailer: MIME-tools 5.505 (Entity 5.505)
Content-Type: text/plain; charset=utf-8
X-Loop: help-debbugs@HIDDEN
From: help-debbugs@HIDDEN (GNU bug Tracking System)
To: "Zack Weinberg" <zack@HIDDEN>
Subject: bug#78067: Acknowledgement (Conversion to unprivileged
 guix-daemon breaks ssh)
Message-ID: <handler.78067.B.174561332226325.ack <at> debbugs.gnu.org>
References: <8d70405b-7f96-43a2-90de-8b5adde8873d@HIDDEN>
X-Gnu-PR-Message: ack 78067
X-Gnu-PR-Package: guix
Reply-To: 78067 <at> debbugs.gnu.org
Date: Fri, 25 Apr 2025 20:36:02 +0000

Thank you for filing a new bug report with debbugs.gnu.org.

This is an automatically generated reply to let you know your message
has been received.

Your message is being forwarded to the package maintainers and other
interested parties for their attention; they will reply in due course.

Your message has been sent to the package maintainer(s):
 bug-guix@HIDDEN

If you wish to submit further information on this problem, please
send it to 78067 <at> debbugs.gnu.org.

Please do not send mail to help-debbugs@HIDDEN unless you wish
to report a problem with the Bug-tracking system.

--=20
78067: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D78067
GNU Bug Tracking System
Contact help-debbugs@HIDDEN with problems

Message sent to bug-guix@HIDDEN:

Subject: bug#78067: Conversion to unprivileged guix-daemon breaks ssh
Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Mon, 05 May 2025 15:36:04 +0000
Resent-Message-ID: <handler.78067.B78067.174645931722582 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: "Zack Weinberg" <zack@HIDDEN>
Cc: 78067 <at> debbugs.gnu.org
From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
In-Reply-To: <8d70405b-7f96-43a2-90de-8b5adde8873d@HIDDEN> (Zack
 Weinberg's message of "Fri, 25 Apr 2025 16:34:45 -0400")
References: <8d70405b-7f96-43a2-90de-8b5adde8873d@HIDDEN>
Date: Mon, 05 May 2025 14:58:28 +0200
Message-ID: <871pt35i4r.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

Hi Zack,

"Zack Weinberg" <zack@HIDDEN> writes:

> I just switched my Guix System-based server over to unprivileged guix-
> daemon, after which I was unable to ssh into it.  From the client, the
> syndrome looks like this (shell variables indicate redactions):

[...]

> 2025-04-25 20:27:40 localhost shepherd[1]: Service sshd-69 running
> with value #<<process> id: 234 command:
> ("/gnu/store/8kman284vvlzk2hgy1bv1xzys3rfdzlr-openssh-10.0p1/sbin/sshd"
> "-D" "-f" "/gnu/store/vwy5d5mj35rh147iwzkzxijld0gx06mb-sshd_config"
> "-i")>.
> 2025-04-25 20:27:40 localhost sshd[234]: fatal:
> /gnu/store/8kman284vvlzk2hgy1bv1xzys3rfdzlr-openssh-10.0p1/var/empty
> must be owned by root and not group or world-writable.

D=E2=80=99oh.  The fix here is to tell OpenSSH to use /var/empty instead.  =
Do
you know how to do that via sshd_config?

> `chown root:root /gnu/store/8kman284vvlzk2hgy1bv1xzys3rfdzlr-openssh-10.0=
p1/var/empty`
> is sufficient to fix ssh, but I wonder if store contents in general shoul=
d maybe remain
> owned by root regardless of whether the daemon is running as an unprivile=
ged user.
> It seems likely to me that this will not be the only such problem.

You should never manually modify files in the store or change their
ownership.  In the case above, the daemon will now be unable to delete
this store item when you run =E2=80=98guix gc=E2=80=99.

Thanks for the bug report,
Ludo=E2=80=99.

Last modified: Mon, 5 May 2025 15:45:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.