Package: guix-patches;
Reported by: Richard Sent <richard <at> freakingpenguin.com>
Date: Tue, 18 Jun 2024 22:08:02 UTC
Severity: normal
Tags: patch
To reply to this bug, email your comments to 71639 AT debbugs.gnu.org.
Toggle the display of automated, internal messages from the tracker.
View this report as an mbox folder, status mbox, maintainer mbox
guix-patches <at> gnu.org
:bug#71639
; Package guix-patches
.
(Tue, 18 Jun 2024 22:08:02 GMT) Full text and rfc822 format available.Richard Sent <richard <at> freakingpenguin.com>
:guix-patches <at> gnu.org
.
(Tue, 18 Jun 2024 22:08:02 GMT) Full text and rfc822 format available.Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):
From: Richard Sent <richard <at> freakingpenguin.com> To: guix-patches <at> gnu.org Cc: ludo <at> gnu.org, Richard Sent <richard <at> freakingpenguin.com>, goodoldpaul <at> autistici.org Subject: [PATCH WIP 0/5] Improve on restic-backup-service Date: Tue, 18 Jun 2024 18:06:39 -0400
Hi all! This is the previously promised [1] patch series relating to restic-backup-service. The highlights: 1. init? was added that bootstraps a new backup. 2. RESTIC_PASSWORD_COMMAND support. 3. Add extra-packages field for jobs. 4. Move restic from restic-backup-job to restic-backup-configuration. 5. Add a basic system-level test suite. Regarding 4, the previous way felt a bit orthogonal to how services of a similar nature work. (For example, you don't specify an mcron package for every job.) I can't think of a use case for mixing-and-matching restic packages with different jobs. If there is a reason, I think a good compromise would be a "restic-override" job field. I'm marking this as WIP because there are a couple more things I want to investigate, but it feels complete enough to submit for feedback. My main point of interest is what can be done to improve support for Restic's other backends, see [2]. I imagine 0.9.6 only supports a subset of those, but still. In my dream world we could combine this patch with an upgrade to the restic package itself (which, among other things, supports compression). I asked the developers about (re)including a vendor directory in their tarball but haven't heard back [3]. Perhaps this service can spark some action on an old Guix tracking issue [4]. [1]: https://issues.guix.gnu.org/69513#7 [2]: https://restic.readthedocs.io/en/stable/030_preparing_a_new_repo.html [3]: https://github.com/restic/restic/issues/3945#issuecomment-2141090355 [4]: https://issues.guix.gnu.org/63019 Richard Sent (5): services: backup: Support bootstrapping an initial restic backup services: backup: Add password-command support to restic-service services: backup: Add extra-packages field to restic-backup-job services: backup: Move restic package to restic-configuration tests: Add restic system test. doc/guix.texi | 21 ++++++- gnu/services/backup.scm | 131 +++++++++++++++++++++++++++++----------- gnu/tests/restic.scm | 124 +++++++++++++++++++++++++++++++++++++ 3 files changed, 238 insertions(+), 38 deletions(-) create mode 100644 gnu/tests/restic.scm base-commit: a575d0f5d5322bac977423b6bd2742c8dc5a14a6 -- 2.45.1
pelzflorian <at> pelzflorian.de, ludo <at> gnu.org, matt <at> excalamus.com, maxim.cournoyer <at> gmail.com, guix-patches <at> gnu.org
:bug#71639
; Package guix-patches
.
(Tue, 18 Jun 2024 22:10:02 GMT) Full text and rfc822 format available.Message #8 received at 71639 <at> debbugs.gnu.org (full text, mbox):
From: Richard Sent <richard <at> freakingpenguin.com> To: 71639 <at> debbugs.gnu.org Cc: ludo <at> gnu.org, Richard Sent <richard <at> freakingpenguin.com>, goodoldpaul <at> autistici.org Subject: [PATCH WIP 1/5] services: backup: Support bootstrapping an initial restic backup Date: Tue, 18 Jun 2024 18:08:48 -0400
* gnu/services/backup.scm: (restic-backup-job): Add init? field. (restic-backup-job-program): Initialize repository if init? is set and repository does not already exist. * doc/guix.texi (Miscellaneous Services): Document it. Change-Id: I71d0cbaac646b9d160e662b69286f229b9a9f64d --- doc/guix.texi | 4 ++++ gnu/services/backup.scm | 20 ++++++++++++++++++++ 2 files changed, 24 insertions(+) diff --git a/doc/guix.texi b/doc/guix.texi index 0102fd0fad..63c9cbd1a7 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -41353,6 +41353,10 @@ Miscellaneous Services The list of files or directories to be backed up. It must be a list of values that can be lowered to strings. +@item @code{init?} (default: @code{#f}) (type: boolean) +Whether restic-backup-service should check and (if it does not exist) +initialize the repository before running the backup. + @item @code{verbose?} (default: @code{#f}) (type: boolean) Whether to enable verbose output for the current backup job. diff --git a/gnu/services/backup.scm b/gnu/services/backup.scm index 555e9fc959..eeef11eae7 100644 --- a/gnu/services/backup.scm +++ b/gnu/services/backup.scm @@ -1,5 +1,6 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2024 Giacomo Leidi <goodoldpaul <at> autistici.org> +;;; Copyright © 2024 Richard Sent <richard <at> freakingpenguin.com> ;;; ;;; This file is part of GNU Guix. ;;; @@ -38,6 +39,7 @@ (define-module (gnu services backup) restic-backup-job-password-file restic-backup-job-schedule restic-backup-job-files + restic-backup-job-init? restic-backup-job-verbose? restic-backup-job-extra-flags @@ -94,6 +96,9 @@ (define-configuration/no-serialization restic-backup-job (verbose? (boolean #f) "Whether to enable verbose output for the current backup job.") + (init? + (boolean #f) + "Whether to attempt to initialize a new repository for automated bootstrap purposes.") (extra-flags (list-of-lowerables '()) "A list of values that are lowered to strings. These will be passed as @@ -118,6 +123,8 @@ (define (restic-backup-job-program config) (restic-backup-job-files config)) (extra-flags (restic-backup-job-extra-flags config)) + (init? + (restic-backup-job-init? config)) (verbose (if (restic-backup-job-verbose? config) '("--verbose") @@ -130,6 +137,19 @@ (define (restic-backup-job-program config) (setenv "RESTIC_PASSWORD" (with-input-from-file #$password-file read-line)) + (when #$init? + ;; Check if the repository exists. See + ;; https://github.com/restic/restic/issues/1690 and + ;; https://github.com/NixOS/nixpkgs/pull/307962. + ;; + ;; XXX: restic returns values other than 1 on failure. Use + ;; unless EXIT_SUCCESS instead of when EXIT_FAILURE. + (unless (equal? EXIT_SUCCESS (system* #$restic "cat" "config" + "-r" #$repository)) + ;; Initialize it. + (unless (equal? EXIT_SUCCESS (system* #$restic "init" + "-r" #$repository)) + (error "Failed to initialize restic repository: " #$repository)))) (execlp #$restic #$restic #$@verbose "-r" #$repository #$@extra-flags -- 2.45.1
pelzflorian <at> pelzflorian.de, ludo <at> gnu.org, matt <at> excalamus.com, maxim.cournoyer <at> gmail.com, guix-patches <at> gnu.org
:bug#71639
; Package guix-patches
.
(Tue, 18 Jun 2024 22:10:02 GMT) Full text and rfc822 format available.Message #11 received at 71639 <at> debbugs.gnu.org (full text, mbox):
From: Richard Sent <richard <at> freakingpenguin.com> To: 71639 <at> debbugs.gnu.org Cc: ludo <at> gnu.org, Richard Sent <richard <at> freakingpenguin.com>, goodoldpaul <at> autistici.org Subject: [PATCH WIP 3/5] services: backup: Add extra-packages field to restic-backup-job Date: Tue, 18 Jun 2024 18:08:50 -0400
* gnu/services/backup.scm (restic-backup-job): Create extra-packages. (restic-guix-wrapper-package): Add the extra-packages field of every job as inputs. * doc/guix.texi (Miscellaneous Services): Document it. Change-Id: I4f0b070bc6dc895553ba69256d14e45898291c02 --- doc/guix.texi | 4 ++++ gnu/services/backup.scm | 33 ++++++++++++++++++++------------- 2 files changed, 24 insertions(+), 13 deletions(-) diff --git a/doc/guix.texi b/doc/guix.texi index f22d679023..32ce0c86b9 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -41367,6 +41367,10 @@ Miscellaneous Services @item @code{verbose?} (default: @code{#f}) (type: boolean) Whether to enable verbose output for the current backup job. +@item @code{extra-packages} (default: @code{'()} (type: list-of-packages) +The list of extra packages needed for restic to run the current backup +job. + @item @code{extra-flags} (default: @code{'()}) (type: list-of-lowerables) A list of values that are lowered to strings. These will be passed as command-line arguments to the current job @command{restic backup} diff --git a/gnu/services/backup.scm b/gnu/services/backup.scm index 2471d0ea7b..a6d8404a5a 100644 --- a/gnu/services/backup.scm +++ b/gnu/services/backup.scm @@ -41,6 +41,7 @@ (define-module (gnu services backup) restic-backup-job-files restic-backup-job-init? restic-backup-job-verbose? + restic-backup-job-extra-packages restic-backup-job-extra-flags restic-backup-configuration @@ -102,6 +103,9 @@ (define-configuration/no-serialization restic-backup-job (list-of-lowerables '()) "The list of files or directories to be backed up. It must be a list of values that can be lowered to strings.") + (extra-packages + (list-of-packages '()) + "The list of extra packages needed for restic to run this backup job.") (verbose? (boolean #f) "Whether to enable verbose output for the current backup job.") @@ -239,22 +243,25 @@ (define (restic-backup-job->mcron-job config) #:user #$user))) (define (restic-guix-wrapper-package jobs) - (package - (name "restic-backup-service-wrapper") - (version "0.0.0") - (source (restic-guix jobs)) - (build-system copy-build-system) - (arguments - (list #:install-plan #~'(("./" "/bin")))) - (home-page "https://restic.net") - (synopsis - "Easily interact from the CLI with Guix configured backups") - (description - "This package provides a simple wrapper around @code{restic}, handled + (let ((extra-packages (append-map restic-backup-job-extra-packages + jobs))) + (package + (name "restic-backup-service-wrapper") + (version "0.0.0") + (source (restic-guix restic-package jobs)) + (build-system copy-build-system) + (arguments + (list #:install-plan #~'(("./" "/bin")))) + (home-page "https://restic.net") + (synopsis + "Easily interact from the CLI with Guix configured backups") + (description + "This package provides a simple wrapper around @code{restic}, handled by the @code{restic-backup-service-type}. It allows for easily interacting with Guix configured backup jobs, for example for manually triggering a backup without waiting for the scheduled job to run.") - (license license:gpl3+))) + (inputs extra-packages) + (license license:gpl3+)))) (define restic-backup-service-profile (lambda (config) -- 2.45.1
pelzflorian <at> pelzflorian.de, ludo <at> gnu.org, matt <at> excalamus.com, maxim.cournoyer <at> gmail.com, guix-patches <at> gnu.org
:bug#71639
; Package guix-patches
.
(Tue, 18 Jun 2024 22:10:03 GMT) Full text and rfc822 format available.Message #14 received at 71639 <at> debbugs.gnu.org (full text, mbox):
From: Richard Sent <richard <at> freakingpenguin.com> To: 71639 <at> debbugs.gnu.org Cc: ludo <at> gnu.org, Richard Sent <richard <at> freakingpenguin.com>, goodoldpaul <at> autistici.org Subject: [PATCH WIP 2/5] services: backup: Add password-command support to restic-service Date: Tue, 18 Jun 2024 18:08:49 -0400
* gnu/services/backup.scm (restic-backup-job): Add password-command. (verify-restic-backup-job-configuration): Create. (restic-backup-job-program): Set either RESTIC_PASSWORD or RESTIC_PASSWORD_COMMAND depending on what is configured. * doc/guix.texi (Miscellaneous Services): Document it. Change-Id: Ice9cf85d1ee4485a2737f515c63c969918219df0 --- doc/guix.texi | 7 +++++++ gnu/services/backup.scm | 42 ++++++++++++++++++++++++++++++++++++----- 2 files changed, 44 insertions(+), 5 deletions(-) diff --git a/doc/guix.texi b/doc/guix.texi index 63c9cbd1a7..f22d679023 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -41344,6 +41344,13 @@ Miscellaneous Services that will be used to set the @env{RESTIC_PASSWORD} environment variable for the current job. +@item @code{password-command} (type: file-like) +String path or file-like object representing the executable file that +prints password to stdout. If a file-like object is used, it is placed +in the store globally executable and in plain text. The executable +should be designed such that it does not compromise the password if an +unauthorized user runs it. + @item @code{schedule} (type: gexp-or-string) A string or a gexp that will be passed as time specification in the mcron job specification (@pxref{Syntax, mcron job specifications,, diff --git a/gnu/services/backup.scm b/gnu/services/backup.scm index eeef11eae7..2471d0ea7b 100644 --- a/gnu/services/backup.scm +++ b/gnu/services/backup.scm @@ -66,6 +66,9 @@ (define (lowerable? value) (define list-of-lowerables? (list-of lowerable?)) +(define-maybe/no-serialization string) +(define-maybe/no-serialization file-like) + (define-configuration/no-serialization restic-backup-job (restic (package restic) @@ -80,10 +83,16 @@ (define-configuration/no-serialization restic-backup-job (string) "The restic repository target of this job.") (password-file - (string) + (maybe-string) "Name of the password file, readable by the configured @code{user}, that will be used to set the @code{RESTIC_PASSWORD} environment variable for the current job.") + (password-command + (maybe-file-like) + "Name of the password command that, when run, returns the password over +stdin. Due to the nature of the store this command will be globally executable +and should have external protections to ensure other users cannot retrieve the +password. This overrides password-file.") (schedule (gexp-or-string) "A string or a gexp that will be passed as time specification in the mcron @@ -104,6 +113,14 @@ (define-configuration/no-serialization restic-backup-job "A list of values that are lowered to strings. These will be passed as command-line arguments to the current job @command{restic backup} invokation.")) +(define (verify-restic-backup-job-configuration config) + (unless (or (maybe-value-set? (restic-backup-job-password-file config)) + (maybe-value-set? (restic-backup-job-password-command config))) + (error "either password-file or password-command must be configured.")) + (when (and (maybe-value-set? (restic-backup-job-password-file config)) + (maybe-value-set? (restic-backup-job-password-command config))) + (error "password-file and password-command can not be configured simultaneously."))) + (define list-of-restic-backup-jobs? (list-of restic-backup-job?)) @@ -113,12 +130,22 @@ (define-configuration/no-serialization restic-backup-configuration "The list of backup jobs for the current system.")) (define (restic-backup-job-program config) + (define (maybe-value-or-false maybe) + (if (maybe-value-set? maybe) + maybe + #f)) + + ;; TODO: Find a place to also verify restic-backup-configuration. Mainly that jobs >=1 + (verify-restic-backup-job-configuration config) + (let ((restic (file-append (restic-backup-job-restic config) "/bin/restic")) (repository (restic-backup-job-repository config)) (password-file - (restic-backup-job-password-file config)) + (maybe-value-or-false (restic-backup-job-password-file config))) + (password-command + (maybe-value-or-false (restic-backup-job-password-command config))) (files (restic-backup-job-files config)) (extra-flags @@ -134,9 +161,14 @@ (define (restic-backup-job-program config) #~(begin (use-modules (ice-9 popen) (ice-9 rdelim)) - (setenv "RESTIC_PASSWORD" - (with-input-from-file #$password-file read-line)) - + (or (and=> #$password-file (lambda (x) + (setenv "RESTIC_PASSWORD" + (with-input-from-file x read-line)))) + (and=> #$password-command (lambda (x) + (setenv "RESTIC_PASSWORD_COMMAND" x))) + ;; Have a backup error message in case + ;; verify-restic-backup-job-configuration is messed with + (error "Neither password-file or password-command set")) (when #$init? ;; Check if the repository exists. See ;; https://github.com/restic/restic/issues/1690 and -- 2.45.1
pelzflorian <at> pelzflorian.de, ludo <at> gnu.org, matt <at> excalamus.com, maxim.cournoyer <at> gmail.com, guix-patches <at> gnu.org
:bug#71639
; Package guix-patches
.
(Tue, 18 Jun 2024 22:10:04 GMT) Full text and rfc822 format available.Message #17 received at 71639 <at> debbugs.gnu.org (full text, mbox):
From: Richard Sent <richard <at> freakingpenguin.com> To: 71639 <at> debbugs.gnu.org Cc: ludo <at> gnu.org, Richard Sent <richard <at> freakingpenguin.com>, goodoldpaul <at> autistici.org Subject: [PATCH WIP 4/5] services: backup: Move restic package to restic-configuration Date: Tue, 18 Jun 2024 18:08:51 -0400
* gnu/services/backup.scm (restic-backup-configuration): Add restic field. (restic-backup-job-program): Add restic package to function signature. (restic-guix): Ditto. (restic-guix-wrapper-package): Ditto. Add restic package as package input. (restic-backup-service-profile): Remove excess lamba. * doc/guix.texi (Miscellaneous Services): Document it. Change-Id: I1e5f63c21cd072354225afe0ee270dca8d9d840b --- doc/guix.texi | 6 +++--- gnu/services/backup.scm | 38 ++++++++++++++++++++------------------ 2 files changed, 23 insertions(+), 21 deletions(-) diff --git a/doc/guix.texi b/doc/guix.texi index 32ce0c86b9..e4379c5e1c 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -41311,6 +41311,9 @@ Miscellaneous Services Available @code{restic-backup-configuration} fields are: @table @asis +@item @code{restic} (default: @code{restic}) (type: package) +The restic package to use for all jobs. + @item @code{jobs} (default: @code{'()}) (type: list-of-restic-backup-jobs) The list of backup jobs for the current system. @@ -41327,9 +41330,6 @@ Miscellaneous Services Available @code{restic-backup-job} fields are: @table @asis -@item @code{restic} (default: @code{restic}) (type: package) -The restic package to be used for the current job. - @item @code{user} (default: @code{"root"}) (type: string) The user used for running the current job. diff --git a/gnu/services/backup.scm b/gnu/services/backup.scm index a6d8404a5a..a59cf08c71 100644 --- a/gnu/services/backup.scm +++ b/gnu/services/backup.scm @@ -29,10 +29,10 @@ (define-module (gnu services backup) #:use-module (guix modules) #:use-module (guix packages) #:use-module (srfi srfi-1) + #:use-module (srfi srfi-26) #:export (restic-backup-job restic-backup-job? restic-backup-job-fields - restic-backup-job-restic restic-backup-job-user restic-backup-job-name restic-backup-job-repository @@ -46,6 +46,7 @@ (define-module (gnu services backup) restic-backup-configuration restic-backup-configuration? + restic-backup-configuration-restic restic-backup-configuration-fields restic-backup-configuration-jobs @@ -71,9 +72,6 @@ (define-maybe/no-serialization string) (define-maybe/no-serialization file-like) (define-configuration/no-serialization restic-backup-job - (restic - (package restic) - "The restic package to be used for the current job.") (user (string "root") "The user used for running the current job.") @@ -129,11 +127,14 @@ (define list-of-restic-backup-jobs? (list-of restic-backup-job?)) (define-configuration/no-serialization restic-backup-configuration + (restic + (package restic) + "The restic package to be used.") (jobs (list-of-restic-backup-jobs '()) "The list of backup jobs for the current system.")) -(define (restic-backup-job-program config) +(define (restic-backup-job-program restic-package config) (define (maybe-value-or-false maybe) (if (maybe-value-set? maybe) maybe @@ -143,7 +144,7 @@ (define (restic-backup-job-program config) (verify-restic-backup-job-configuration config) (let ((restic - (file-append (restic-backup-job-restic config) "/bin/restic")) + (file-append restic-package "/bin/restic")) (repository (restic-backup-job-repository config)) (password-file @@ -191,7 +192,7 @@ (define (restic-backup-job-program config) #$@extra-flags "backup" #$@files))))) -(define (restic-guix jobs) +(define (restic-guix restic-package jobs) (program-file "restic-guix" #~(begin @@ -199,7 +200,7 @@ (define (restic-guix jobs) (srfi srfi-1)) (define names '#$(map restic-backup-job-name jobs)) - (define programs '#$(map restic-backup-job-program jobs)) + (define programs '#$(map (cut restic-backup-job-program restic-package <>) jobs)) (define (get-program name) (define idx @@ -242,13 +243,13 @@ (define (restic-backup-job->mcron-job config) #$(string-append "restic-guix backup " name) #:user #$user))) -(define (restic-guix-wrapper-package jobs) +(define (restic-guix-wrapper-package restic jobs) (let ((extra-packages (append-map restic-backup-job-extra-packages jobs))) (package (name "restic-backup-service-wrapper") (version "0.0.0") - (source (restic-guix restic-package jobs)) + (source (restic-guix restic jobs)) (build-system copy-build-system) (arguments (list #:install-plan #~'(("./" "/bin")))) @@ -260,16 +261,17 @@ (define (restic-guix-wrapper-package jobs) by the @code{restic-backup-service-type}. It allows for easily interacting with Guix configured backup jobs, for example for manually triggering a backup without waiting for the scheduled job to run.") - (inputs extra-packages) + (inputs (cons restic extra-packages)) (license license:gpl3+)))) -(define restic-backup-service-profile - (lambda (config) - (define jobs (restic-backup-configuration-jobs config)) - (if (> (length jobs) 0) - (list - (restic-guix-wrapper-package jobs)) - '()))) +(define (restic-backup-service-profile config) + (let ((jobs (restic-backup-configuration-jobs config)) + (restic (restic-backup-configuration-restic config))) + ;; Even if we provide a wrapper we need to add restic to the profile so + ;; the service works in containers and VMs which only see a subset of + ;; /gnu/store. https://issues.guix.gnu.org/70553. Ergo pass restic to + ;; the wrapper package so it is added as an input. + (list (restic-guix-wrapper-package restic jobs)))) (define restic-backup-service-type (service-type (name 'restic-backup) -- 2.45.1
guix-patches <at> gnu.org
:bug#71639
; Package guix-patches
.
(Tue, 18 Jun 2024 22:10:04 GMT) Full text and rfc822 format available.Message #20 received at 71639 <at> debbugs.gnu.org (full text, mbox):
From: Richard Sent <richard <at> freakingpenguin.com> To: 71639 <at> debbugs.gnu.org Cc: ludo <at> gnu.org, Richard Sent <richard <at> freakingpenguin.com>, goodoldpaul <at> autistici.org Subject: [PATCH WIP 5/5] tests: Add restic system test. Date: Tue, 18 Jun 2024 18:08:52 -0400
* gnu/tests/restic.scm: Create. Change-Id: Iad5472414c140b133d9b402855bb2f01e96bb0cc --- gnu/tests/restic.scm | 124 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 124 insertions(+) create mode 100644 gnu/tests/restic.scm diff --git a/gnu/tests/restic.scm b/gnu/tests/restic.scm new file mode 100644 index 0000000000..f8ea76538e --- /dev/null +++ b/gnu/tests/restic.scm @@ -0,0 +1,124 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright © 2024 Richard Sent <richard <at> freakingpenguin.com>. +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>. + +(define-module (gnu tests restic) + ;; TODO: Remove what I can from here. + #:use-module (gnu bootloader) + #:use-module (gnu bootloader grub) + #:use-module (gnu packages) + #:use-module (gnu packages sync) ;rclone + #:use-module (gnu packages xorg) + #:use-module (gnu services) + #:use-module (gnu services desktop) + #:use-module (gnu services backup) ;restic + #:use-module (gnu services xorg) + #:use-module (gnu system) + #:use-module (gnu system vm) + #:use-module (gnu tests) + #:use-module (guix gexp) + #:use-module (guix modules) + #:use-module (srfi srfi-1) + #:export (%test-restic)) + +(define password "password") + +(define password-file + (plain-file "password-file" password)) + +(define password-command + (program-file "password-command" #~(display #$password))) + +(define (run-restic-test) + "Run tests in %restic-os." + + (define os + (marionette-operating-system + (simple-operating-system (extra-special-file "/root/.restic-test" + (plain-file "restic-test" + "Hello world!")) + ;; restic-backup-service only takes a string to avoid putting + ;; plaintext entries in the store. Ergo, symlink it. + (extra-special-file "/root/password-file" + password-file) + (service restic-backup-service-type + (restic-backup-configuration + (jobs + (list (restic-backup-job + (name "password-file-backup") + (repository "/root/restic-password-file-repo") + (schedule #~'(next-second '(0 15 30 45))) + (password-file "/root/password-file") + (files '("/root/.restic-test")) + (init? #t)) + (restic-backup-job + (name "password-command-backup") + (repository "/root/restic-password-command-repo") + (schedule #~'(next-second '(0 15 30 45))) + (password-command password-command) + (files '("/root/.restic-test")) + (init? #t))))))) + #:imported-modules '((gnu services herd) + (guix combinators)))) + + (define vm (virtual-machine + (operating-system os) + ;; TODO: We might be able to lower this. + (memory-size 1024))) + + (define test + (with-imported-modules (source-module-closure + '((gnu build marionette))) + #~(begin + (use-modules (gnu build marionette) + (srfi srfi-26) + (srfi srfi-64)) + + (let ((marionette (make-marionette (list #$vm)))) + + (test-runner-current (system-test-runner #$output)) + (test-begin "restic") + + (test-assert "backup-file-created" + (wait-for-file "/root/.restic-test" marionette)) + + (test-assert "mcron running" + (marionette-eval + '(begin + (use-modules (gnu services herd)) + (start-service 'mcron)) + marionette)) + + (test-assert "password-file backup completed" + (wait-for-file "/root/restic-password-file-repo/config" marionette + ;; Restic takes a second to run, give it a bit + ;; more time. + #:timeout 20)) + + (test-assert "password-comand backup completed" + (wait-for-file "/root/restic-password-file-repo/config" marionette + #:timeout 20)) + + (test-end))))) + + (gexp->derivation "restic-test" test)) + +(define %test-restic + (system-test + (name "restic") + (description "Basic tests for the restic service.") + (value (run-restic-test)))) -- 2.45.1
guix-patches <at> gnu.org
:bug#71639
; Package guix-patches
.
(Thu, 20 Jun 2024 03:46:06 GMT) Full text and rfc822 format available.Message #23 received at 71639 <at> debbugs.gnu.org (full text, mbox):
From: Richard Sent <richard <at> freakingpenguin.com> To: guix-patches <at> gnu.org, 71639 <at> debbugs.gnu.org Cc: Richard Sent <richard <at> freakingpenguin.com> Subject: [PATCH v2 0/5] Improve on restic-backup-service Date: Wed, 19 Jun 2024 23:44:11 -0400
Some minor adjustments to the WIP version, mostly cleanup and removing some TODOs that snuck in. Thinking on it there's no need to worry about marking this as WIP. If it winds up merged earlier than expected I can just submit another patch series. Richard Sent (5): services: backup: Support bootstrapping an initial restic backup services: backup: Add password-command support to restic-service services: backup: Add extra-packages field to restic-backup-job services: backup: Move restic package to restic-configuration tests: Add restic system test. doc/guix.texi | 21 ++++++- gnu/services/backup.scm | 129 +++++++++++++++++++++++++++++----------- gnu/tests/restic.scm | 119 ++++++++++++++++++++++++++++++++++++ 3 files changed, 231 insertions(+), 38 deletions(-) create mode 100644 gnu/tests/restic.scm base-commit: e32e3d0a03dc17c4c54a91aad053c9036998b601 -- 2.45.1
pelzflorian <at> pelzflorian.de, ludo <at> gnu.org, matt <at> excalamus.com, maxim.cournoyer <at> gmail.com, guix-patches <at> gnu.org
:bug#71639
; Package guix-patches
.
(Thu, 20 Jun 2024 03:46:06 GMT) Full text and rfc822 format available.Message #26 received at 71639 <at> debbugs.gnu.org (full text, mbox):
From: Richard Sent <richard <at> freakingpenguin.com> To: guix-patches <at> gnu.org, 71639 <at> debbugs.gnu.org Cc: Richard Sent <richard <at> freakingpenguin.com> Subject: [PATCH v2 1/5] services: backup: Support bootstrapping an initial restic backup Date: Wed, 19 Jun 2024 23:44:12 -0400
* gnu/services/backup.scm: (restic-backup-job): Add init? field. (restic-backup-job-program): Initialize repository if init? is set and repository does not already exist. * doc/guix.texi (Miscellaneous Services): Document it. Change-Id: I71d0cbaac646b9d160e662b69286f229b9a9f64d --- doc/guix.texi | 4 ++++ gnu/services/backup.scm | 19 +++++++++++++++++++ 2 files changed, 23 insertions(+) diff --git a/doc/guix.texi b/doc/guix.texi index 0102fd0fad..63c9cbd1a7 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -41353,6 +41353,10 @@ Miscellaneous Services The list of files or directories to be backed up. It must be a list of values that can be lowered to strings. +@item @code{init?} (default: @code{#f}) (type: boolean) +Whether restic-backup-service should check and (if it does not exist) +initialize the repository before running the backup. + @item @code{verbose?} (default: @code{#f}) (type: boolean) Whether to enable verbose output for the current backup job. diff --git a/gnu/services/backup.scm b/gnu/services/backup.scm index 555e9fc959..1279ece88f 100644 --- a/gnu/services/backup.scm +++ b/gnu/services/backup.scm @@ -1,5 +1,6 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2024 Giacomo Leidi <goodoldpaul <at> autistici.org> +;;; Copyright © 2024 Richard Sent <richard <at> freakingpenguin.com> ;;; ;;; This file is part of GNU Guix. ;;; @@ -38,6 +39,7 @@ (define-module (gnu services backup) restic-backup-job-password-file restic-backup-job-schedule restic-backup-job-files + restic-backup-job-init? restic-backup-job-verbose? restic-backup-job-extra-flags @@ -94,6 +96,9 @@ (define-configuration/no-serialization restic-backup-job (verbose? (boolean #f) "Whether to enable verbose output for the current backup job.") + (init? + (boolean #f) + "Whether to attempt to initialize a new repository for automated bootstrap purposes.") (extra-flags (list-of-lowerables '()) "A list of values that are lowered to strings. These will be passed as @@ -118,6 +123,8 @@ (define (restic-backup-job-program config) (restic-backup-job-files config)) (extra-flags (restic-backup-job-extra-flags config)) + (init? + (restic-backup-job-init? config)) (verbose (if (restic-backup-job-verbose? config) '("--verbose") @@ -130,6 +137,18 @@ (define (restic-backup-job-program config) (setenv "RESTIC_PASSWORD" (with-input-from-file #$password-file read-line)) + (when #$init? + ;; Use cat config to check if the repository exists. See + ;; https://github.com/restic/restic/issues/1690 and + ;; https://github.com/NixOS/nixpkgs/pull/307962. + ;; + ;; XXX: restic returns values other than 1 on failure. Check + ;; EXIT_SUCCESS instead of EXIT_FAILURE. + (unless (or (equal? EXIT_SUCCESS (system* #$restic "cat" "config" + "-r" #$repository)) + (equal? EXIT_SUCCESS (system* #$restic "init" + "-r" #$repository))) + (error "Failed to initialize restic repository: " #$repository))) (execlp #$restic #$restic #$@verbose "-r" #$repository #$@extra-flags -- 2.45.1
pelzflorian <at> pelzflorian.de, ludo <at> gnu.org, matt <at> excalamus.com, maxim.cournoyer <at> gmail.com, guix-patches <at> gnu.org
:bug#71639
; Package guix-patches
.
(Thu, 20 Jun 2024 03:46:07 GMT) Full text and rfc822 format available.Message #29 received at 71639 <at> debbugs.gnu.org (full text, mbox):
From: Richard Sent <richard <at> freakingpenguin.com> To: guix-patches <at> gnu.org, 71639 <at> debbugs.gnu.org Cc: Richard Sent <richard <at> freakingpenguin.com> Subject: [PATCH v2 2/5] services: backup: Add password-command support to restic-service Date: Wed, 19 Jun 2024 23:44:13 -0400
* gnu/services/backup.scm (restic-backup-job): Add password-command. (verify-restic-backup-job-configuration): Create. (restic-backup-job-program): Set either RESTIC_PASSWORD or RESTIC_PASSWORD_COMMAND depending on what is configured. * doc/guix.texi (Miscellaneous Services): Document it. Change-Id: Ice9cf85d1ee4485a2737f515c63c969918219df0 --- doc/guix.texi | 7 +++++++ gnu/services/backup.scm | 41 ++++++++++++++++++++++++++++++++++++----- 2 files changed, 43 insertions(+), 5 deletions(-) diff --git a/doc/guix.texi b/doc/guix.texi index 63c9cbd1a7..f22d679023 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -41344,6 +41344,13 @@ Miscellaneous Services that will be used to set the @env{RESTIC_PASSWORD} environment variable for the current job. +@item @code{password-command} (type: file-like) +String path or file-like object representing the executable file that +prints password to stdout. If a file-like object is used, it is placed +in the store globally executable and in plain text. The executable +should be designed such that it does not compromise the password if an +unauthorized user runs it. + @item @code{schedule} (type: gexp-or-string) A string or a gexp that will be passed as time specification in the mcron job specification (@pxref{Syntax, mcron job specifications,, diff --git a/gnu/services/backup.scm b/gnu/services/backup.scm index 1279ece88f..fd904bc9a9 100644 --- a/gnu/services/backup.scm +++ b/gnu/services/backup.scm @@ -66,6 +66,9 @@ (define (lowerable? value) (define list-of-lowerables? (list-of lowerable?)) +(define-maybe/no-serialization string) +(define-maybe/no-serialization file-like) + (define-configuration/no-serialization restic-backup-job (restic (package restic) @@ -80,10 +83,16 @@ (define-configuration/no-serialization restic-backup-job (string) "The restic repository target of this job.") (password-file - (string) + (maybe-string) "Name of the password file, readable by the configured @code{user}, that will be used to set the @code{RESTIC_PASSWORD} environment variable for the current job.") + (password-command + (maybe-file-like) + "An executable file who's path is stored in @code{RESTIC_PASSWORD_COMMAND}. +When run, the file writes the password to standard output. Due to the nature +of the store this command will be globally executable and should have external +protections to ensure unauthorized users cannot retrieve the password.") (schedule (gexp-or-string) "A string or a gexp that will be passed as time specification in the mcron @@ -104,6 +113,14 @@ (define-configuration/no-serialization restic-backup-job "A list of values that are lowered to strings. These will be passed as command-line arguments to the current job @command{restic backup} invokation.")) +(define (verify-restic-backup-job-configuration config) + (unless (or (maybe-value-set? (restic-backup-job-password-file config)) + (maybe-value-set? (restic-backup-job-password-command config))) + (error "either password-file or password-command must be configured.")) + (when (and (maybe-value-set? (restic-backup-job-password-file config)) + (maybe-value-set? (restic-backup-job-password-command config))) + (error "password-file and password-command can not be configured simultaneously."))) + (define list-of-restic-backup-jobs? (list-of restic-backup-job?)) @@ -113,12 +130,21 @@ (define-configuration/no-serialization restic-backup-configuration "The list of backup jobs for the current system.")) (define (restic-backup-job-program config) + (define (maybe-value-or-false maybe) + (if (maybe-value-set? maybe) + maybe + #f)) + + (verify-restic-backup-job-configuration config) + (let ((restic (file-append (restic-backup-job-restic config) "/bin/restic")) (repository (restic-backup-job-repository config)) (password-file - (restic-backup-job-password-file config)) + (maybe-value-or-false (restic-backup-job-password-file config))) + (password-command + (maybe-value-or-false (restic-backup-job-password-command config))) (files (restic-backup-job-files config)) (extra-flags @@ -134,9 +160,14 @@ (define (restic-backup-job-program config) #~(begin (use-modules (ice-9 popen) (ice-9 rdelim)) - (setenv "RESTIC_PASSWORD" - (with-input-from-file #$password-file read-line)) - + (or (and=> #$password-file (lambda (x) + (setenv "RESTIC_PASSWORD" + (with-input-from-file x read-line)))) + (and=> #$password-command (lambda (x) + (setenv "RESTIC_PASSWORD_COMMAND" x))) + ;; Have a backup error message in case + ;; verify-restic-backup-job-configuration is messed with + (error "Neither password-file or password-command set")) (when #$init? ;; Use cat config to check if the repository exists. See ;; https://github.com/restic/restic/issues/1690 and -- 2.45.1
pelzflorian <at> pelzflorian.de, ludo <at> gnu.org, matt <at> excalamus.com, maxim.cournoyer <at> gmail.com, guix-patches <at> gnu.org
:bug#71639
; Package guix-patches
.
(Thu, 20 Jun 2024 03:46:07 GMT) Full text and rfc822 format available.Message #32 received at 71639 <at> debbugs.gnu.org (full text, mbox):
From: Richard Sent <richard <at> freakingpenguin.com> To: guix-patches <at> gnu.org, 71639 <at> debbugs.gnu.org Cc: Richard Sent <richard <at> freakingpenguin.com> Subject: [PATCH v2 4/5] services: backup: Move restic package to restic-configuration Date: Wed, 19 Jun 2024 23:44:15 -0400
* gnu/services/backup.scm (restic-backup-configuration): Add restic field. (restic-backup-job-program): Add restic package to function signature. (restic-guix): Ditto. (restic-guix-wrapper-package): Ditto. Add restic package as package input. (restic-backup-service-profile): Remove excess lamba. * doc/guix.texi (Miscellaneous Services): Document it. Change-Id: I1e5f63c21cd072354225afe0ee270dca8d9d840b --- doc/guix.texi | 6 +++--- gnu/services/backup.scm | 38 ++++++++++++++++++++------------------ 2 files changed, 23 insertions(+), 21 deletions(-) diff --git a/doc/guix.texi b/doc/guix.texi index 32ce0c86b9..e4379c5e1c 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -41311,6 +41311,9 @@ Miscellaneous Services Available @code{restic-backup-configuration} fields are: @table @asis +@item @code{restic} (default: @code{restic}) (type: package) +The restic package to use for all jobs. + @item @code{jobs} (default: @code{'()}) (type: list-of-restic-backup-jobs) The list of backup jobs for the current system. @@ -41327,9 +41330,6 @@ Miscellaneous Services Available @code{restic-backup-job} fields are: @table @asis -@item @code{restic} (default: @code{restic}) (type: package) -The restic package to be used for the current job. - @item @code{user} (default: @code{"root"}) (type: string) The user used for running the current job. diff --git a/gnu/services/backup.scm b/gnu/services/backup.scm index bfbacfe590..5e2add088e 100644 --- a/gnu/services/backup.scm +++ b/gnu/services/backup.scm @@ -29,10 +29,10 @@ (define-module (gnu services backup) #:use-module (guix modules) #:use-module (guix packages) #:use-module (srfi srfi-1) + #:use-module (srfi srfi-26) #:export (restic-backup-job restic-backup-job? restic-backup-job-fields - restic-backup-job-restic restic-backup-job-user restic-backup-job-name restic-backup-job-repository @@ -46,6 +46,7 @@ (define-module (gnu services backup) restic-backup-configuration restic-backup-configuration? + restic-backup-configuration-restic restic-backup-configuration-fields restic-backup-configuration-jobs @@ -71,9 +72,6 @@ (define-maybe/no-serialization string) (define-maybe/no-serialization file-like) (define-configuration/no-serialization restic-backup-job - (restic - (package restic) - "The restic package to be used for the current job.") (user (string "root") "The user used for running the current job.") @@ -129,11 +127,14 @@ (define list-of-restic-backup-jobs? (list-of restic-backup-job?)) (define-configuration/no-serialization restic-backup-configuration + (restic + (package restic) + "The restic package to be used.") (jobs (list-of-restic-backup-jobs '()) "The list of backup jobs for the current system.")) -(define (restic-backup-job-program config) +(define (restic-backup-job-program restic-package config) (define (maybe-value-or-false maybe) (if (maybe-value-set? maybe) maybe @@ -142,7 +143,7 @@ (define (restic-backup-job-program config) (verify-restic-backup-job-configuration config) (let ((restic - (file-append (restic-backup-job-restic config) "/bin/restic")) + (file-append restic-package "/bin/restic")) (repository (restic-backup-job-repository config)) (password-file @@ -189,7 +190,7 @@ (define (restic-backup-job-program config) #$@extra-flags "backup" #$@files))))) -(define (restic-guix jobs) +(define (restic-guix restic-package jobs) (program-file "restic-guix" #~(begin @@ -197,7 +198,7 @@ (define (restic-guix jobs) (srfi srfi-1)) (define names '#$(map restic-backup-job-name jobs)) - (define programs '#$(map restic-backup-job-program jobs)) + (define programs '#$(map (cut restic-backup-job-program restic-package <>) jobs)) (define (get-program name) (define idx @@ -240,13 +241,13 @@ (define (restic-backup-job->mcron-job config) #$(string-append "restic-guix backup " name) #:user #$user))) -(define (restic-guix-wrapper-package jobs) +(define (restic-guix-wrapper-package restic jobs) (let ((extra-packages (append-map restic-backup-job-extra-packages jobs))) (package (name "restic-backup-service-wrapper") (version "0.0.0") - (source (restic-guix restic-package jobs)) + (source (restic-guix restic jobs)) (build-system copy-build-system) (arguments (list #:install-plan #~'(("./" "/bin")))) @@ -258,16 +259,17 @@ (define (restic-guix-wrapper-package jobs) by the @code{restic-backup-service-type}. It allows for easily interacting with Guix configured backup jobs, for example for manually triggering a backup without waiting for the scheduled job to run.") - (inputs extra-packages) + (inputs (cons restic extra-packages)) (license license:gpl3+)))) -(define restic-backup-service-profile - (lambda (config) - (define jobs (restic-backup-configuration-jobs config)) - (if (> (length jobs) 0) - (list - (restic-guix-wrapper-package jobs)) - '()))) +(define (restic-backup-service-profile config) + (let ((jobs (restic-backup-configuration-jobs config)) + (restic (restic-backup-configuration-restic config))) + ;; Even if we provide a wrapper we need to add restic to the profile so + ;; the service works in containers and VMs which only see a subset of + ;; /gnu/store. https://issues.guix.gnu.org/70553. Ergo pass restic to + ;; the wrapper package so it is added as an input. + (list (restic-guix-wrapper-package restic jobs)))) (define restic-backup-service-type (service-type (name 'restic-backup) -- 2.45.1
pelzflorian <at> pelzflorian.de, ludo <at> gnu.org, matt <at> excalamus.com, maxim.cournoyer <at> gmail.com, guix-patches <at> gnu.org
:bug#71639
; Package guix-patches
.
(Thu, 20 Jun 2024 03:46:07 GMT) Full text and rfc822 format available.Message #35 received at 71639 <at> debbugs.gnu.org (full text, mbox):
From: Richard Sent <richard <at> freakingpenguin.com> To: guix-patches <at> gnu.org, 71639 <at> debbugs.gnu.org Cc: Richard Sent <richard <at> freakingpenguin.com> Subject: [PATCH v2 3/5] services: backup: Add extra-packages field to restic-backup-job Date: Wed, 19 Jun 2024 23:44:14 -0400
* gnu/services/backup.scm (restic-backup-job): Create extra-packages. (restic-guix-wrapper-package): Add the extra-packages field of every job as inputs. * doc/guix.texi (Miscellaneous Services): Document it. Change-Id: I4f0b070bc6dc895553ba69256d14e45898291c02 --- doc/guix.texi | 4 ++++ gnu/services/backup.scm | 33 ++++++++++++++++++++------------- 2 files changed, 24 insertions(+), 13 deletions(-) diff --git a/doc/guix.texi b/doc/guix.texi index f22d679023..32ce0c86b9 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -41367,6 +41367,10 @@ Miscellaneous Services @item @code{verbose?} (default: @code{#f}) (type: boolean) Whether to enable verbose output for the current backup job. +@item @code{extra-packages} (default: @code{'()} (type: list-of-packages) +The list of extra packages needed for restic to run the current backup +job. + @item @code{extra-flags} (default: @code{'()}) (type: list-of-lowerables) A list of values that are lowered to strings. These will be passed as command-line arguments to the current job @command{restic backup} diff --git a/gnu/services/backup.scm b/gnu/services/backup.scm index fd904bc9a9..bfbacfe590 100644 --- a/gnu/services/backup.scm +++ b/gnu/services/backup.scm @@ -41,6 +41,7 @@ (define-module (gnu services backup) restic-backup-job-files restic-backup-job-init? restic-backup-job-verbose? + restic-backup-job-extra-packages restic-backup-job-extra-flags restic-backup-configuration @@ -102,6 +103,9 @@ (define-configuration/no-serialization restic-backup-job (list-of-lowerables '()) "The list of files or directories to be backed up. It must be a list of values that can be lowered to strings.") + (extra-packages + (list-of-packages '()) + "The list of extra packages needed for restic to run this backup job.") (verbose? (boolean #f) "Whether to enable verbose output for the current backup job.") @@ -237,22 +241,25 @@ (define (restic-backup-job->mcron-job config) #:user #$user))) (define (restic-guix-wrapper-package jobs) - (package - (name "restic-backup-service-wrapper") - (version "0.0.0") - (source (restic-guix jobs)) - (build-system copy-build-system) - (arguments - (list #:install-plan #~'(("./" "/bin")))) - (home-page "https://restic.net") - (synopsis - "Easily interact from the CLI with Guix configured backups") - (description - "This package provides a simple wrapper around @code{restic}, handled + (let ((extra-packages (append-map restic-backup-job-extra-packages + jobs))) + (package + (name "restic-backup-service-wrapper") + (version "0.0.0") + (source (restic-guix restic-package jobs)) + (build-system copy-build-system) + (arguments + (list #:install-plan #~'(("./" "/bin")))) + (home-page "https://restic.net") + (synopsis + "Easily interact from the CLI with Guix configured backups") + (description + "This package provides a simple wrapper around @code{restic}, handled by the @code{restic-backup-service-type}. It allows for easily interacting with Guix configured backup jobs, for example for manually triggering a backup without waiting for the scheduled job to run.") - (license license:gpl3+))) + (inputs extra-packages) + (license license:gpl3+)))) (define restic-backup-service-profile (lambda (config) -- 2.45.1
guix-patches <at> gnu.org
:bug#71639
; Package guix-patches
.
(Thu, 20 Jun 2024 03:46:08 GMT) Full text and rfc822 format available.Message #38 received at 71639 <at> debbugs.gnu.org (full text, mbox):
From: Richard Sent <richard <at> freakingpenguin.com> To: guix-patches <at> gnu.org, 71639 <at> debbugs.gnu.org Cc: Richard Sent <richard <at> freakingpenguin.com> Subject: [PATCH v2 5/5] tests: Add restic system test. Date: Wed, 19 Jun 2024 23:44:16 -0400
* gnu/tests/restic.scm: Create. Change-Id: Iad5472414c140b133d9b402855bb2f01e96bb0cc --- gnu/tests/restic.scm | 119 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 119 insertions(+) create mode 100644 gnu/tests/restic.scm diff --git a/gnu/tests/restic.scm b/gnu/tests/restic.scm new file mode 100644 index 0000000000..8d29ff441b --- /dev/null +++ b/gnu/tests/restic.scm @@ -0,0 +1,119 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright © 2024 Richard Sent <richard <at> freakingpenguin.com>. +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>. + +(define-module (gnu tests restic) + #:use-module (gnu bootloader) + #:use-module (gnu bootloader grub) + #:use-module (gnu packages) + #:use-module (gnu packages sync) ;rclone + #:use-module (gnu services) + #:use-module (gnu services backup) ;restic + #:use-module (gnu system) + #:use-module (gnu system vm) + #:use-module (gnu tests) + #:use-module (guix gexp) + #:use-module (guix modules) + #:use-module (srfi srfi-1) + #:export (%test-restic)) + +(define password "password") + +(define password-file + (plain-file "password-file" password)) + +(define password-command + (program-file "password-command" #~(display #$password))) + +(define (run-restic-test) + "Run tests in %restic-os." + + (define os + (marionette-operating-system + (simple-operating-system (extra-special-file "/root/.restic-test" + (plain-file "restic-test" + "Hello world!")) + ;; restic-backup-service only takes a string to avoid putting + ;; plaintext entries in the store. Ergo, symlink it. + (extra-special-file "/root/password-file" + password-file) + (service restic-backup-service-type + (restic-backup-configuration + (jobs + (list (restic-backup-job + (name "password-file-backup") + (repository "/root/restic-password-file-repo") + (schedule #~'(next-second '(0 15 30 45))) + (password-file "/root/password-file") + (files '("/root/.restic-test")) + (init? #t)) + (restic-backup-job + (name "password-command-backup") + (repository "/root/restic-password-command-repo") + (schedule #~'(next-second '(0 15 30 45))) + (password-command password-command) + (files '("/root/.restic-test")) + (init? #t))))))) + #:imported-modules '((gnu services herd) + (guix combinators)))) + + (define vm (virtual-machine + (operating-system os) + (memory-size 512))) + + (define test + (with-imported-modules (source-module-closure + '((gnu build marionette))) + #~(begin + (use-modules (gnu build marionette) + (srfi srfi-26) + (srfi srfi-64)) + + (let ((marionette (make-marionette (list #$vm)))) + + (test-runner-current (system-test-runner #$output)) + (test-begin "restic") + + (test-assert "backup-file-created" + (wait-for-file "/root/.restic-test" marionette)) + + (test-assert "mcron running" + (marionette-eval + '(begin + (use-modules (gnu services herd)) + (start-service 'mcron)) + marionette)) + + (test-assert "password-file backup completed" + (wait-for-file "/root/restic-password-file-repo/config" marionette + ;; Restic takes a second to run, give it a bit + ;; more time. + #:timeout 20)) + + (test-assert "password-comand backup completed" + (wait-for-file "/root/restic-password-file-repo/config" marionette + #:timeout 20)) + + (test-end))))) + + (gexp->derivation "restic-test" test)) + +(define %test-restic + (system-test + (name "restic") + (description "Basic tests for the restic service.") + (value (run-restic-test)))) -- 2.45.1
guix-patches <at> gnu.org
:bug#71639
; Package guix-patches
.
(Mon, 24 Jun 2024 22:50:01 GMT) Full text and rfc822 format available.Message #41 received at 71639 <at> debbugs.gnu.org (full text, mbox):
From: paul <goodoldpaul <at> autistici.org> To: 71639 <at> debbugs.gnu.org Cc: Richard Sent <richard <at> freakingpenguin.com> Subject: Re: [PATCHv2 0/5] Improve on restic-backup-service Date: Tue, 25 Jun 2024 00:49:07 +0200
Hi Richard, thank you for this work, it looks really useful and in general I think the patches are ok but I have some points I'd like to discuss. I think it would be nice to have an init action for the restic-guix command shipped with the service and move there the logic for initializing repositories. The service then could be adapted in a way that it would call the restic-guix init before calling restic-guix backup. Would you be interested in implementing this? At last, my use case for having a restic package field for each job is to have some critical jobs that I don't want to touch running with Guix's bootstrapped restic package and some personal jobs that I run with a restic 0.16 binary package I have in my personal channel . I'm not sure this warrants a field in each single job, but they are optional anyway. Anyway I wouldn't consider this a blocker and if the Guix project has some guidelines in this sense I'd say follow them. Cheers, giacomo
guix-patches <at> gnu.org
:bug#71639
; Package guix-patches
.
(Thu, 27 Jun 2024 03:58:02 GMT) Full text and rfc822 format available.Message #44 received at 71639 <at> debbugs.gnu.org (full text, mbox):
From: Richard Sent <richard <at> freakingpenguin.com> To: paul <goodoldpaul <at> autistici.org> Cc: 71639 <at> debbugs.gnu.org Subject: Re: [PATCHv2 0/5] Improve on restic-backup-service Date: Wed, 26 Jun 2024 23:56:55 -0400
Hi giacomo, > I think it would be nice to have an init action for the restic-guix > command shipped with the service and move there the logic for > initializing repositories. The service then could be adapted in a way > that it would call the restic-guix init before calling restic-guix > backup. Would you be interested in implementing this? I'll take a look at it and see what the code looks like. It might be a bit of effort to get that working cleanly while avoiding code dupe. For instance, setting RESTIC_PASSWORD(_COMMAND) in both init and backup program actions. Where do you think the appropriate place to check init? and run the init action is if it's no longer encapsulated in the backup action? A conditional in restic-backup-job->mcron-job before launching restic-guix backup? A one-shot shepherd service? The former will necessitate adding a job string to display when running $ herd schedule mcron. A downside of the latter is if the restic repository is deleted for one reason or another that backup job will fail until the system is rebooted, which isn't immediately obvious. Personally I prefer the mcron-job conditional. > At last, my use case for having a restic package field for each job is > to have some critical jobs that I don't want to touch running with > Guix's bootstrapped restic package and some personal jobs that I run > with a restic 0.16 binary package I have in my personal channel . I'm > not sure this warrants a field in each single job, but they are optional > anyway. Anyway I wouldn't consider this a blocker and if the Guix > project has some guidelines in this sense I'd say follow them. Ah, I see. To me this is where a per-job restic-override or similarly named field makes sense. This way we can have a default "global" restic package configured at a service level while still allowing individual jobs to use a custom restic package. I imagine this restic-override field would be a "maybe-file-like" either set to #f or a custom restic package. Thanks for the feedback! -- Take it easy, Richard Sent Making my computer weirder one commit at a time.
guix-patches <at> gnu.org
:bug#71639
; Package guix-patches
.
(Sun, 07 Jul 2024 20:42:01 GMT) Full text and rfc822 format available.Message #47 received at 71639 <at> debbugs.gnu.org (full text, mbox):
From: paul <goodoldpaul <at> autistici.org> To: Richard Sent <richard <at> freakingpenguin.com> Cc: 71639 <at> debbugs.gnu.org Subject: Re: [PATCHv2 0/5] Improve on restic-backup-service Date: Sun, 7 Jul 2024 22:40:55 +0200
[Message part 1 (text/plain, inline)]
Hi Richard, On 6/27/24 05:56, Richard Sent wrote: > I'll take a look at it and see what the code looks like. It might be a > bit of effort to get that working cleanly while avoiding code dupe. For > instance, setting RESTIC_PASSWORD(_COMMAND) in both init and backup > program actions. Yes, 100% agree. What I would do is (and probably there is a better way): - rename restic-backup-job-program to restic-action-job-program or something similar. the procedure would be the same but it would take two additional arguments: an action-name argument or similar that would replace all instances of "backup" and I think this new procedure should not access the files field of the job record but it instead it would take an action-arguments argument which would replace #$@files with #$@action-arguments in the execlp call. This way, I hope, it could be general enough to support additional actions besides init in the future, such as purge or restore. - create a new restic-backup-job-program and a restic-init-job-program or similar procedures that would call the new generic restic-action-job-program - inside the restic-guix procedure I'd rename the programs list to backup-programs and I would create a similar list for the init action with #f placeholders for jobs that do not require initializing, and I would check the conditional inside the backup procedure something like: (define (get-program name programs) [...]) (define (init args) [...]) (define (backup args) (define name (third args)) (define backup-program (get-program name backup-programs)) (define init-program (get-program name init-programs)) (when init-program (init args)) (execlp backup-program backup-program)) this requires the init action not to exec the script resulting from the gexp but I can't think of a reason why it could be problematic right now so I guess it is ok. Please take all of the above as a proposal there are for sure things to make better. > Thanks for the feedback! Thank you very much for your work, giacomo
[Message part 2 (text/html, inline)]
guix-patches <at> gnu.org
:bug#71639
; Package guix-patches
.
(Thu, 22 Aug 2024 17:45:02 GMT) Full text and rfc822 format available.Message #50 received at 71639 <at> debbugs.gnu.org (full text, mbox):
From: Fabio Natali <me <at> fabionatali.com> To: 71639 <at> debbugs.gnu.org Subject: Re: [bug#71639] [PATCHv2 0/5] Improve on restic-backup-service Date: Thu, 22 Aug 2024 18:43:51 +0100
Hi Giacomo, Thanks for adding the Restic Service to Guix. Hi Richard, Thanks for patch request 71639, which includes a few changes I'm looking forward to. For instance, I like the idea of exposing the 'RESTIC_PASSWORD_COMMAND' variable directly via the service configuration. I think I might still be able to use the 'extra-flags' field to specify a '--password-command' - if I understand it correctly - but direct access to the option would be neat indeed. Anything I can do to help with the patch? Could it make sense to break the patch down into a couple of stages, possibly leaving the most impactful changes for a later stage? Thanks, cheers, Fabio.
guix-patches <at> gnu.org
:bug#71639
; Package guix-patches
.
(Sun, 25 Aug 2024 14:14:01 GMT) Full text and rfc822 format available.Message #53 received at 71639 <at> debbugs.gnu.org (full text, mbox):
From: paul <goodoldpaul <at> autistici.org> To: 71639 <at> debbugs.gnu.org Subject: Re: [PATCHv2 0/5] Improve on restic-backup-service Date: Sun, 25 Aug 2024 16:12:51 +0200
[Message part 1 (text/plain, inline)]
Dear Fabio and Richard, I apologize for blocking this issue with my proposal. I think Fabio's proposal of splitting the changes in this patch is the best way forward, if you agree as well Richard. I submitted [0] to refactor the restic-guix procedure in a way that it can support many different commands. After it gets in it should be sufficient to add "init" to %restic-guix-supported-actions to have a working restic-guix init invokation. it should be then matter of understanding where it is better to put it. Lately I was thinking that may be best to have initialization as a one shot Shepherd service that check whether a given job is supposed to have its repository initialized and if that's the case it could run restic-guix init name-of the job. Please Richard let me know what you think of this approach and whether you would still be interested in implementing it, thank you very much! I think the following subdivision should match all requirements we stated until now: 1. Improve the restic backup system service . This can be done in the current issue #71639: services: backup: Add password-command support to restic-service services: backup: Add extra-packages field to restic-backup-job services: backup: Move restic package to restic-configuration 2. Refactor the restic-guix function to allow for more restic commands to be wrapped. This can be done in issue #72803: services: restic-backup: Add more restic commands to the restic-guix package. 3. Allow for repositories to be initialized with a restic-guix init command. This commit could be adapted and moved to a new branch based on #72803: services: backup: Support bootstrapping an initial restic backup What do you think? Could this be a suitable action plan? Thank you very much for your work, giacomo [0]: https://issues.guix.gnu.org/72803
[Message part 2 (text/html, inline)]
guix-patches <at> gnu.org
:bug#71639
; Package guix-patches
.
(Tue, 03 Sep 2024 16:45:02 GMT) Full text and rfc822 format available.Message #56 received at 71639 <at> debbugs.gnu.org (full text, mbox):
From: Fabio Natali <me <at> fabionatali.com> To: 71639 <at> debbugs.gnu.org Cc: Richard Sent <richard <at> freakingpenguin.com>, paul <goodoldpaul <at> autistici.org>, Fabio Natali <me <at> fabionatali.com> Subject: Re: [PATCH WIP 0/5] Improve on restic-backup-service Date: Tue, 03 Sep 2024 17:43:03 +0100
> I apologize for blocking this issue with my proposal. I think Fabio's > proposal of splitting the changes in this patch is the best way > forward, if you agree as well Richard. I submitted [0] to refactor the > restic-guix procedure in a way that it can support many different > commands. Hi Giacomo, Thanks for this! > 1. Improve the restic backup system service [...] > 2. Refactor the restic-guix function [...] > 3. Allow for repositories to be initialized with a restic-guix init [...] I think this is a perfectly sensible plan. For what it's worth, I've provided some little feedback around point 2 as part of #72803. As a further point, I also think we should manage to update Restic itself to 0.17.0 (while we're still at 0.9.6 which was released in Nov 2019). With such an out-of-date package, the service also risks to loose a little bit of its value. I've just sent a quick follow-up to #63019 in the hope that we might be able to get this sorted soon. Thanks, cheers, Fabio.
guix-patches <at> gnu.org
:bug#71639
; Package guix-patches
.
(Wed, 04 Sep 2024 15:51:02 GMT) Full text and rfc822 format available.Message #59 received at 71639 <at> debbugs.gnu.org (full text, mbox):
From: Richard Sent <richard <at> freakingpenguin.com> To: Fabio Natali <me <at> fabionatali.com> Cc: 71639 <at> debbugs.gnu.org, paul <goodoldpaul <at> autistici.org> Subject: Re: [bug#71639] [PATCH WIP 0/5] Improve on restic-backup-service Date: Wed, 04 Sep 2024 11:49:15 -0400
Hi Giacomo and Fabio! I've been struck by the the law of development: the backlog will always expand to exceed the time available. >> I apologize for blocking this issue with my proposal. I think Fabio's >> proposal of splitting the changes in this patch is the best way >> forward, if you agree as well Richard. I submitted [0] to refactor the >> restic-guix procedure in a way that it can support many different >> commands. That sounds fine with me. I'll try to rebase and resubmit as time is available (hopefully the next few days......). If my schedule is inconvenient feel free to do with my patches what you will. >> 1. Improve the restic backup system service [...] >> 2. Refactor the restic-guix function [...] >> 3. Allow for repositories to be initialized with a restic-guix init >> [...] Ideally, we'd also move the system tests to 1 so further work can be trivially verified by running $ make check-system TESTS=restic. That'll require writing an intermediary bootstrapping function in the test suite. > I think this is a perfectly sensible plan. For what it's worth, I've > provided some little feedback around point 2 as part of #72803. > > As a further point, I also think we should manage to update Restic > itself to 0.17.0 (while we're still at 0.9.6 which was released in Nov > 2019). With such an out-of-date package, the service also risks to loose > a little bit of its value. That would be lovely! One particular feature I'm hopeful for is compression support. Unfortunately restic stopped providing a vendor directory in their tarball after 0.9.6 and seemingly has no interest in providing it again [1]. Last time I tried guix import with restic I saw some odd behavior. Someone with more experience in Go may have better luck. >> Lately I was thinking that may be best to have initialization as a >> one shot Shepherd service that check whether a given job is supposed >> to have its repository initialized and if that's the case it could >> run restic-guix init name-of the job. Is the intent to have one "mega" one-shot service that checks every job, or a unique one-shot service per job? I greatly prefer the former as $ sudo herd status is crowded as it is. It's worth noting users will need the ability to specify custom Shepherd requirements so e.g. backup initialization to a remote file-share isn't triggered until the file-system-/my/remote/nas symbol is provisioned by Shepherd. On an unrelated note, Shepherd timers are interesting and may allow for better user management/error signaling instead of mcron. I'm not proposing we refactor to use them now (especially since they're not even in stable), but just thought I'd mention them. [2] [1]: https://github.com/restic/restic/issues/3945 [2]: https://lists.gnu.org/archive/html/guix-devel/2024-03/msg00247.html -- Take it easy, Richard Sent Making my computer weirder one commit at a time.
guix-patches <at> gnu.org
:bug#71639
; Package guix-patches
.
(Thu, 05 Sep 2024 10:11:02 GMT) Full text and rfc822 format available.Message #62 received at 71639 <at> debbugs.gnu.org (full text, mbox):
From: Fabio Natali <me <at> fabionatali.com> To: 71639 <at> debbugs.gnu.org Cc: Richard Sent <richard <at> freakingpenguin.com>, paul <goodoldpaul <at> autistici.org>, Fabio Natali <me <at> fabionatali.com> Subject: Re: [PATCH WIP 0/5] Improve on restic-backup-service Date: Thu, 05 Sep 2024 11:06:29 +0100
Hi Richard and Giacomo. > >> 1. Improve the restic backup system service [...] [...] > Ideally, we'd also move the system tests to 1 so further work can be > trivially verified by running $ make check-system > TESTS=restic. That'll require writing an intermediary bootstrapping > function in the test suite. +1 > > As a further point, I also think we should manage to update Restic > > itself to 0.17.0 (while we're still at 0.9.6 which was released in Nov > > 2019). With such an out-of-date package, the service also risks to loose > > a little bit of its value. > > That would be lovely! One particular feature I'm hopeful for is > compression support. > > Unfortunately restic stopped providing a vendor directory in their > tarball after 0.9.6 and seemingly has no interest in providing it > again [1]. Last time I tried guix import with restic I saw some odd > behavior. Someone with more experience in Go may have better luck. > [1]: https://github.com/restic/restic/issues/3945 Thanks for the link to the Restic's issue tracker. I think it's actually better to have the dependencies unbundled and built/imported individually - although this is causing us some pain at the moment. Some of the 'guix import go --recursive restic' errors seem to be caused by Go projects using a slightly different subfolder scheme? There seem to be various open issues (and semi-finalised patches) around this, such as: - https://issues.guix.gnu.org/issue/52362 - https://issues.guix.gnu.org/issue/63001 - https://issues.guix.gnu.org/issue/63647 - https://issues.guix.gnu.org/issue/64035 - https://issues.guix.gnu.org/issue/64036 I'll follow up on one of the threads above and report back here if I find anything useful. Thanks, cheers, Fabio.
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997,2003 nCipher Corporation Ltd,
1994-97 Ian Jackson.