GNU logs - #70761, boring messages

Message sent to bug-guix@HIDDEN:

Subject: bug#70761: Guix deploy cannot reboot remote machines without an error
Resent-From: Richard Sent <richard@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Fri, 03 May 2024 21:59:02 +0000
Resent-Message-ID: <handler.70761.B.171477351320753 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: 70761 <at> debbugs.gnu.org
From: Richard Sent <richard@HIDDEN>
Date: Fri, 03 May 2024 17:57:38 -0400
Message-ID: <87wmoapnhp.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Received-SPF: pass client-ip=136.175.108.194;
 envelope-from=richard@HIDDEN; helo=mail-108-mta194.mxroute.com
X-Spam_score_int: -16
X-Spam_score: -1.7
X-Spam_bar: -
X-Spam_report: (-1.7 / 5.0 requ) BAYES_00=-1.9, DKIM_INVALID=0.1,
 DKIM_SIGNED=0.1, SPF_HELO_NONE=0.001,
 SPF_PASS=-0.001 autolearn=no autolearn_force=no
X-Spam_action: no action
X-Spam-Score: 0.9 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.1 (/)

Hi Guix,

One neat feature of guix deploy is the ability to run a command on a
list of remote machines. One command that would commonly be run is
reboot, so that upgrades to, say, the Linux kernel take effect.

While the command itself /does/ run and the system /does/ restart, guix
deploy doesn't gracefully handle the connection loss. The first machine
rebooted will throw an error, halting the reboot of the rest of the
machines in the list.

tmux and screen aren't really compatible with '-x -- <command>' since
you can't start and detach from sessions and the & in '-x -- nohup
<command> &' gets swallowed by the host shell. I haven't found a
workaround. Even if they did work, I suspect there is a race condition
between "host closes session" and "remote restarts".

We shouldn't assume that any command may potentially close the SSH
session and catch errors by default.

One solution could be adding an alternative to -x that nohup's the
command and attempts to cleanly close the SSH session. If the session
errors out after the command is nohup'd (e.g. reboot race condition),
catch the SSH error and exit.

Alternatively we could add a --reboot flag, although I prefer the more
general solution.

Perhaps this can be the impetus for implementing the "deploy-hook"
functionality described at https://issues.guix.gnu.org/53486. In a
particularly fancy world, we could combine rebooting with pre and post
reboot command execution, but now I'm thinking of pies =F0=9F=A5=A7 in skie=
s =E2=9B=85.

Or maybe I'm completely wrong and this is possible (sorry!), in which
case we probably could add a quick mention of it in the manual.

--8<---------------cut here---------------start------------->8---
gibraltar :( rsent$ guix deploy rsent/machines/lan.scm --no-grafts -x -- re=
boot
guix deploy: warning: <machine-ssh-configuration> without a 'host-key' is d=
eprecated
guix deploy: sending 1 store item (0 MiB) to 'horizon.local'...
;;; [2024/05/03 17:16:45.032361, 0] [GSSH ERROR] Parent session is not conn=
ected: #<unknown channel (freed) 7fbe66aa71a0>
Backtrace:
          16 (primitive-load "/home/richard/.config/guix/current/bin/guix")
In guix/ui.scm:
   2312:7 15 (run-guix . _)
  2275:10 14 (run-guix-command _ . _)
In ice-9/boot-9.scm:
  1752:10 13 (with-exception-handler _ _ #:unwind? _ #:unwind-for-type _)
In guix/status.scm:
    839:4 12 (call-with-status-report _ _)
In ice-9/boot-9.scm:
  1752:10 11 (with-exception-handler _ _ #:unwind? _ #:unwind-for-type _)
In guix/store.scm:
   666:37 10 (thunk)
   1302:8  9 (call-with-build-handler _ _)
   1302:8  8 (call-with-build-handler #<procedure 7fbe69be4690 at guix/ui.s=
cm:1222:2 (continue store things mode)> _)
In guix/scripts/deploy.scm:
   274:23  7 (_)
In srfi/srfi-1.scm:
   460:18  6 (fold #<procedure 7fbe78c25540 at guix/scripts/deploy.scm:274:=
28 (machine result)> #t (#<<machine> operating-system: #<<operating-system>=
 ke=E2=80=A6>))
In guix/scripts/deploy.scm:
    214:2  5 (_ #<<machine> operating-system: #<<operating-system> kernel: =
#<package linux@HIDDEN nongnu/packages/linux.scm:118 7fbe6343c0b0> kernel-lo=
ada=E2=80=A6> =E2=80=A6)
In guix/store.scm:
  2182:25  4 (run-with-store #<store-connection 256.100 7fbe78cf8960> #<pro=
cedure 7fbe66b423c0 at guix/remote.scm:119:2 (state)> #:guile-for-build _ #=
 _ # _)
In guix/remote.scm:
    72:20  3 (_ _)
In unknown file:
           2 (channel-get-exit-status #<unknown channel (freed) 7fbe66aa71a=
0>)
In ice-9/boot-9.scm:
  1685:16  1 (raise-exception _ #:continuable? _)
  1685:16  0 (raise-exception _ #:continuable? _)

ice-9/boot-9.scm:1685:16: In procedure raise-exception:
Throw to key `guile-ssh-error' with args `("channel-get-exit-status" "Paren=
t session is not connected" #<unknown channel (freed) 7fbe66aa71a0> #f)'.
--8<---------------cut here---------------end--------------->8---


--=20
Take it easy,
Richard Sent
Making my computer weirder one commit at a time.

Message sent:

Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Mailer: MIME-tools 5.505 (Entity 5.505)
Content-Type: text/plain; charset=utf-8
X-Loop: help-debbugs@HIDDEN
From: help-debbugs@HIDDEN (GNU bug Tracking System)
To: Richard Sent <richard@HIDDEN>
Subject: bug#70761: Acknowledgement (Guix deploy cannot reboot remote
 machines without an error)
Message-ID: <handler.70761.B.171477351320753.ack <at> debbugs.gnu.org>
References: <87wmoapnhp.fsf@HIDDEN>
X-Gnu-PR-Message: ack 70761
X-Gnu-PR-Package: guix
Reply-To: 70761 <at> debbugs.gnu.org
Date: Fri, 03 May 2024 21:59:02 +0000

Thank you for filing a new bug report with debbugs.gnu.org.

This is an automatically generated reply to let you know your message
has been received.

Your message is being forwarded to the package maintainers and other
interested parties for their attention; they will reply in due course.

Your message has been sent to the package maintainer(s):
 bug-guix@HIDDEN

If you wish to submit further information on this problem, please
send it to 70761 <at> debbugs.gnu.org.

Please do not send mail to help-debbugs@HIDDEN unless you wish
to report a problem with the Bug-tracking system.

--=20
70761: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D70761
GNU Bug Tracking System
Contact help-debbugs@HIDDEN with problems

Message sent to bug-guix@HIDDEN:

Subject: bug#70761: deploy and reboot
References: <87wmoapnhp.fsf@HIDDEN>
In-Reply-To: <87wmoapnhp.fsf@HIDDEN>
Resent-From: Gabriel Wicki <gabriel@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Tue, 04 Feb 2025 16:44:02 +0000
Resent-Message-ID: <handler.70761.B70761.17386874056812 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: Richard Sent <richard@HIDDEN>
Cc: 70761 <at> debbugs.gnu.org
From: Gabriel Wicki <gabriel@HIDDEN>
Date: Tue, 04 Feb 2025 17:43:14 +0100
Message-ID: <871pwdu0d9.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

Hi Richard!

I just stumbled over the same issue and while i am not really sure what
to think about the screen/tmux/SIGHUP proposal (does this apply to all
deploy commands or just a more fancy/sophisticated usage scenario?) i'd
go with the special --reboot flag to the deploy command that

 1. causes a reboot and then

 2. waits for the machine(s) to come back up.

I am not sure whether this is possible already, but will happily dive in
a little further (and prepare a patch if circumstances allow).

Have a nice week,
gabber

Last modified: Tue, 4 Feb 2025 16:45:01 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.