GNU bug report logs - #77585
[PATCH] services: mumi: Add Debbugs rsync as shepherd timer.

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 77585 in the body.
You can then email your comments to 77585 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Arun Isaac <arunisaac <at> systemreboot.net>
To: guix-patches <at> gnu.org
Cc: Arun Isaac <arunisaac <at> systemreboot.net>,
 Ludovic Courtès <ludo <at> gnu.org>,
 Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Subject: [PATCH] services: mumi: Add Debbugs rsync as shepherd timer.
Date: Sun,  6 Apr 2025 20:30:27 +0100

* gnu/services/web.scm (<mumi-configuration>)[data-directory, rsync-remote,
rsync-flags]: New fields.
(%mumi-worker-log): Delete variable.
(%mumi-rsync-and-index-log): New variable.
(mumi-rsync-and-index-gexp): New function.
(mumi-shepherd-services): Remove mumi-worker service. Add mumi-rsync-and-index
service.
(mumi-service-type): Remove default value.
* doc/guix.texi (Web Services)[mumi]: Document data-directory, rsync-remote
and rsync-flags fields.

Co-authored-by: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
---
 doc/guix.texi        |  9 +++++++
 gnu/services/web.scm | 57 ++++++++++++++++++++++++++++++++------------
 2 files changed, 51 insertions(+), 15 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index 12f155e912..b6c0e64a53 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -33999,6 +33999,15 @@ Web Services
 @item @code{mumi} (default: @code{mumi})
 The Mumi package to use.
 
+@item @code{data-directory} (default: @file{"/var/mumi/data"})
+Data directory to store Debbugs data from upstream Debbugs instance.
+
+@item @code{rsync-remote}
+Remote rsync path from which to download Debbugs data.
+
+@item @code{rsync-flags} (default: @code{'()})
+Additional flags to pass to @command{rsync}.
+
 @item @code{mailer?} (default: @code{#true})
 Whether to enable or disable the mailer component.
 
diff --git a/gnu/services/web.scm b/gnu/services/web.scm
index f8cf06fb48..eea9a19426 100644
--- a/gnu/services/web.scm
+++ b/gnu/services/web.scm
@@ -18,6 +18,7 @@
 ;;; Copyright © 2023 Bruno Victal <mirai <at> makinata.eu>
 ;;; Copyright © 2023 Miguel Ángel Moreno <mail <at> migalmoreno.com>
 ;;; Copyright © 2024 Leo Nikkilä <hello <at> lnikki.la>
+;;; Copyright © 2025 Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -51,6 +52,7 @@ (define-module (gnu services web)
   #:use-module (gnu packages php)
   #:use-module (gnu packages python)
   #:use-module (gnu packages python-web)
+  #:use-module (gnu packages rsync)
   #:use-module (gnu packages gnupg)
   #:use-module (gnu packages guile)
   #:use-module (gnu packages logging)
@@ -295,6 +297,9 @@ (define-module (gnu services web)
             mumi-configuration
             mumi-configuration?
             mumi-configuration-mumi
+            mumi-configuration-data-directory
+            mumi-configuration-rsync-remote
+            mumi-configuration-rsync-flags
             mumi-configuration-mailer?
             mumi-configuration-sender
             mumi-configuration-smtp
@@ -2033,6 +2038,11 @@ (define-record-type* <mumi-configuration>
   mumi-configuration make-mumi-configuration
   mumi-configuration?
   (mumi    mumi-configuration-mumi (default mumi))
+  (data-directory mumi-configuration-data-directory
+                  (default "/var/mumi/data"))
+  (rsync-remote mumi-configuration-rsync-remote)
+  (rsync-flags mumi-configuration-rsync-flags
+               (default '()))
   (mailer? mumi-configuration-mailer? (default #t))
   (sender  mumi-configuration-sender (default #f))
   (smtp    mumi-configuration-smtp (default #f))
@@ -2080,7 +2090,7 @@ (define %mumi-log "/var/log/mumi.log")
 
 (define %mumi-mailer-log "/var/log/mumi.mailer.log")
 
-(define %mumi-worker-log "/var/log/mumi.worker.log")
+(define %mumi-rsync-and-index-log "/var/log/mumi.rsync-and-index.log")
 
 (define mumi-package-configuration->alist
   (match-record-lambda <mumi-package-configuration>
@@ -2118,6 +2128,23 @@ (define mumi-config-file
                                           packages)))
                               <>))))))
 
+(define (mumi-rsync-and-index-gexp config)
+  (match-record config <mumi-configuration>
+    (data-directory rsync-remote rsync-flags)
+    (with-imported-modules '((guix build utils))
+      #~(begin
+          (use-modules (guix build utils))
+
+          (invoke #$(file-append rsync "/bin/rsync")
+                  "--delete" "--archive" "--verbose"
+                  "--timeout" "120"
+                  #$@rsync-flags
+                  #$rsync-remote
+                  #$data-directory)
+          (invoke #$(file-append mumi "/bin/mumi") "fetch"
+                  (string-append "--config="
+                                 #$(mumi-config-file config)))))))
+
 (define (mumi-shepherd-services config)
   (define environment
     #~(list "LC_ALL=en_US.utf8"
@@ -2140,17 +2167,19 @@ (define (mumi-shepherd-services config)
                       #:log-file #$%mumi-log))
             (stop #~(make-kill-destructor)))
            (shepherd-service
-            (provision '(mumi-worker))
-            (documentation "Mumi bug-tracking web interface database worker.")
-            (requirement '(user-processes networking))
-            (start #~(make-forkexec-constructor
-                      `(#$(file-append mumi "/bin/mumi") "worker"
-                        ,(string-append "--config="
-                                        #$(mumi-config-file config)))
-                      #:environment-variables #$environment
-                      #:user "mumi" #:group "mumi"
-                      #:log-file #$%mumi-worker-log))
-            (stop #~(make-kill-destructor)))
+            (provision '(mumi-rsync-and-index))
+            (modules '((shepherd service timer)))
+            (start #~(make-timer-constructor
+                      (calendar-event)
+                      (command
+                       (program-file "mumi-rsync-and-index"
+                                     (mumi-rsync-and-index-gexp config)))
+                      #:log-file #$%mumi-rsync-and-index-log
+                      #:max-duration (* 60 60)
+                      #:wait-for-termination? #t))
+            (stop #~(make-timer-destructor))
+            (actions (list shepherd-trigger-action))
+            (documentation "Rsync and index the GNU Debbugs data"))
            (shepherd-service
             (provision '(mumi-mailer))
             (documentation "Mumi bug-tracking web interface mailer.")
@@ -2181,9 +2210,7 @@ (define mumi-service-type
           (service-extension shepherd-root-service-type
                              mumi-shepherd-services)))
    (description
-    "Run Mumi, a Web interface to the Debbugs bug-tracking server.")
-   (default-value
-     (mumi-configuration))))
+    "Run Mumi, a Web interface to the Debbugs bug-tracking server.")))
 
 (define %default-gmnisrv-config-file
   (plain-file "gmnisrv.ini" "

base-commit: ada14197fb465c1c90efbc450308c14f077ff167
-- 
2.49.0

Message #8 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Arun Isaac <arunisaac <at> systemreboot.net>
Cc: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>, guix-patches <at> gnu.org
Subject: Re: [PATCH] services: mumi: Add Debbugs rsync as shepherd timer.
Date: Sun, 06 Apr 2025 22:23:34 +0200

Hi Arun,

Arun Isaac <arunisaac <at> systemreboot.net> skribis:

> * gnu/services/web.scm (<mumi-configuration>)[data-directory, rsync-remote,
> rsync-flags]: New fields.
> (%mumi-worker-log): Delete variable.
> (%mumi-rsync-and-index-log): New variable.
> (mumi-rsync-and-index-gexp): New function.
> (mumi-shepherd-services): Remove mumi-worker service. Add mumi-rsync-and-index
> service.
> (mumi-service-type): Remove default value.
> * doc/guix.texi (Web Services)[mumi]: Document data-directory, rsync-remote
> and rsync-flags fields.
>
> Co-authored-by: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>

[...]

> +(define (mumi-rsync-and-index-gexp config)

As a matter of style, I would return a program
(‘mumi-rsync-and-index-program’) rather than a gexp; I find it clearer
and more robust (the gexp leaves it up to the caller to insert it in the
right context).

> +            (start #~(make-timer-constructor
> +                      (calendar-event)

You can add a margin comment say “every minute”, for clarity.

> +                      (command
> +                       (program-file "mumi-rsync-and-index"
> +                                     (mumi-rsync-and-index-gexp config)))

I think you’re missing #$ before (mumi-rsync-and-index-gexp config).

> +                      #:log-file #$%mumi-rsync-and-index-log
> +                      #:max-duration (* 60 60)
> +                      #:wait-for-termination? #t))

Every minute is a lot, especially since it may take several seconds to
establish the rsync-over-ssh connection.  It’s mitigated by the use of
#:wait-for-termination? but still sounds fairly aggressive to me.
(It would be ideal if the Debbugs instance could somehow notify mumi.)

Perhaps mention the polling frequency in the manual?

Thanks,
Ludo’.

Message #11 received at 77585 <at> debbugs.gnu.org (full text, mbox):

From: Arun Isaac <arunisaac <at> systemreboot.net>
To: 77585 <at> debbugs.gnu.org
Cc: Arun Isaac <arunisaac <at> systemreboot.net>,
 Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Subject: [PATCH v2] services: mumi: Add Debbugs rsync as shepherd timer.
Date: Sun,  6 Apr 2025 22:52:30 +0100

* gnu/services/web.scm (<mumi-configuration>)[data-directory, rsync-remote,
rsync-flags]: New fields.
(%mumi-worker-log): Delete variable.
(%mumi-rsync-and-index-log): New variable.
(mumi-rsync-and-index): New function.
(mumi-shepherd-services): Remove mumi-worker service. Add mumi-rsync-and-index
service.
(mumi-service-type): Remove default value.
* doc/guix.texi (Web Services)[mumi]: Document data-directory, rsync-remote
and rsync-flags fields.

Co-authored-by: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
---
 doc/guix.texi        | 11 +++++++++
 gnu/services/web.scm | 58 ++++++++++++++++++++++++++++++++------------
 2 files changed, 54 insertions(+), 15 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index 12f155e912..8aa0b5e3ba 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -33999,6 +33999,17 @@ Web Services
 @item @code{mumi} (default: @code{mumi})
 The Mumi package to use.
 
+@item @code{data-directory} (default: @file{"/var/mumi/data"})
+Data directory to store Debbugs data from upstream Debbugs instance.
+
+@item @code{rsync-remote}
+Remote rsync path from which to download Debbugs data.  The remote is
+polled every 2 minutes, unless a previous instance of rsync is already
+running.
+
+@item @code{rsync-flags} (default: @code{'()})
+Additional flags to pass to @command{rsync}.
+
 @item @code{mailer?} (default: @code{#true})
 Whether to enable or disable the mailer component.
 
diff --git a/gnu/services/web.scm b/gnu/services/web.scm
index f8cf06fb48..3f0eb8780a 100644
--- a/gnu/services/web.scm
+++ b/gnu/services/web.scm
@@ -18,6 +18,7 @@
 ;;; Copyright © 2023 Bruno Victal <mirai <at> makinata.eu>
 ;;; Copyright © 2023 Miguel Ángel Moreno <mail <at> migalmoreno.com>
 ;;; Copyright © 2024 Leo Nikkilä <hello <at> lnikki.la>
+;;; Copyright © 2025 Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -51,6 +52,7 @@ (define-module (gnu services web)
   #:use-module (gnu packages php)
   #:use-module (gnu packages python)
   #:use-module (gnu packages python-web)
+  #:use-module (gnu packages rsync)
   #:use-module (gnu packages gnupg)
   #:use-module (gnu packages guile)
   #:use-module (gnu packages logging)
@@ -295,6 +297,9 @@ (define-module (gnu services web)
             mumi-configuration
             mumi-configuration?
             mumi-configuration-mumi
+            mumi-configuration-data-directory
+            mumi-configuration-rsync-remote
+            mumi-configuration-rsync-flags
             mumi-configuration-mailer?
             mumi-configuration-sender
             mumi-configuration-smtp
@@ -2033,6 +2038,11 @@ (define-record-type* <mumi-configuration>
   mumi-configuration make-mumi-configuration
   mumi-configuration?
   (mumi    mumi-configuration-mumi (default mumi))
+  (data-directory mumi-configuration-data-directory
+                  (default "/var/mumi/data"))
+  (rsync-remote mumi-configuration-rsync-remote)
+  (rsync-flags mumi-configuration-rsync-flags
+               (default '()))
   (mailer? mumi-configuration-mailer? (default #t))
   (sender  mumi-configuration-sender (default #f))
   (smtp    mumi-configuration-smtp (default #f))
@@ -2080,7 +2090,7 @@ (define %mumi-log "/var/log/mumi.log")
 
 (define %mumi-mailer-log "/var/log/mumi.mailer.log")
 
-(define %mumi-worker-log "/var/log/mumi.worker.log")
+(define %mumi-rsync-and-index-log "/var/log/mumi.rsync-and-index.log")
 
 (define mumi-package-configuration->alist
   (match-record-lambda <mumi-package-configuration>
@@ -2118,6 +2128,24 @@ (define mumi-config-file
                                           packages)))
                               <>))))))
 
+(define (mumi-rsync-and-index config)
+  (match-record config <mumi-configuration>
+    (data-directory rsync-remote rsync-flags)
+    (program-file "mumi-rsync-and-index"
+                  (with-imported-modules '((guix build utils))
+                    #~(begin
+                        (use-modules (guix build utils))
+
+                        (invoke #$(file-append rsync "/bin/rsync")
+                                "--delete" "--archive" "--verbose"
+                                "--timeout" "120"
+                                #$@rsync-flags
+                                #$rsync-remote
+                                #$data-directory)
+                        (invoke #$(file-append mumi "/bin/mumi") "fetch"
+                                (string-append "--config="
+                                               #$(mumi-config-file config))))))))
+
 (define (mumi-shepherd-services config)
   (define environment
     #~(list "LC_ALL=en_US.utf8"
@@ -2140,17 +2168,19 @@ (define (mumi-shepherd-services config)
                       #:log-file #$%mumi-log))
             (stop #~(make-kill-destructor)))
            (shepherd-service
-            (provision '(mumi-worker))
-            (documentation "Mumi bug-tracking web interface database worker.")
-            (requirement '(user-processes networking))
-            (start #~(make-forkexec-constructor
-                      `(#$(file-append mumi "/bin/mumi") "worker"
-                        ,(string-append "--config="
-                                        #$(mumi-config-file config)))
-                      #:environment-variables #$environment
-                      #:user "mumi" #:group "mumi"
-                      #:log-file #$%mumi-worker-log))
-            (stop #~(make-kill-destructor)))
+            (provision '(mumi-rsync-and-index))
+            (modules '((shepherd service timer)))
+            (start #~(make-timer-constructor
+                      ;; Run every 2 minutes, unless an instance of this job
+                      ;; is already running.
+                      (calendar-event #:minutes (iota 30 0 2))
+                      (command #$(mumi-rsync-and-index config))
+                      #:log-file #$%mumi-rsync-and-index-log
+                      #:max-duration (* 60 60)
+                      #:wait-for-termination? #t))
+            (stop #~(make-timer-destructor))
+            (actions (list shepherd-trigger-action))
+            (documentation "Rsync and index the GNU Debbugs data"))
            (shepherd-service
             (provision '(mumi-mailer))
             (documentation "Mumi bug-tracking web interface mailer.")
@@ -2181,9 +2211,7 @@ (define mumi-service-type
           (service-extension shepherd-root-service-type
                              mumi-shepherd-services)))
    (description
-    "Run Mumi, a Web interface to the Debbugs bug-tracking server.")
-   (default-value
-     (mumi-configuration))))
+    "Run Mumi, a Web interface to the Debbugs bug-tracking server.")))
 
 (define %default-gmnisrv-config-file
   (plain-file "gmnisrv.ini" "

base-commit: ada14197fb465c1c90efbc450308c14f077ff167
-- 
2.49.0

Message #14 received at 77585 <at> debbugs.gnu.org (full text, mbox):

From: Arun Isaac <arunisaac <at> systemreboot.net>
To: 77585 <at> debbugs.gnu.org
Cc: Arun Isaac <arunisaac <at> systemreboot.net>,
 Ludovic Courtès <ludo <at> gnu.org>
Subject: Re: [PATCH] services: mumi: Add Debbugs rsync as shepherd timer.
Date: Sun, 06 Apr 2025 22:58:02 +0100

Hi Ludo,

> As a matter of style, I would return a program
> (‘mumi-rsync-and-index-program’) rather than a gexp; I find it clearer
> and more robust (the gexp leaves it up to the caller to insert it in the
> right context).

Good point, thanks! I was merely trying to save whitespace in the left.
Maybe we should indent program-file differently in the future?

But, fixed for now.

>> +                      (command
>> +                       (program-file "mumi-rsync-and-index"
>> +                                     (mumi-rsync-and-index-gexp config)))
>
> I think you’re missing #$ before (mumi-rsync-and-index-gexp config).

Good catch, thanks!

>> +                      #:log-file #$%mumi-rsync-and-index-log
>> +                      #:max-duration (* 60 60)
>> +                      #:wait-for-termination? #t))
>
> Every minute is a lot, especially since it may take several seconds to
> establish the rsync-over-ssh connection.  It’s mitigated by the use of
> #:wait-for-termination? but still sounds fairly aggressive to me.
> (It would be ideal if the Debbugs instance could somehow notify mumi.)

I have switched to a polling frequency of 2 minutes now. Still quite
high, but we can cut it down to 5 minutes if it's too much.

> Perhaps mention the polling frequency in the manual?

Done now.

When I deploy this on berlin, how do I get shepherd to pick up the new
timer service. Is there a `herd reload' or similar? Or, do I have to
reboot?

Thanks,
Arun

Message #17 received at 77585 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: Arun Isaac <arunisaac <at> systemreboot.net>
Cc: 77585 <at> debbugs.gnu.org
Subject: Re: [PATCH v2] services: mumi: Add Debbugs rsync as shepherd timer.
Date: Tue, 08 Apr 2025 10:56:39 +0900

Hi Arun,

Arun Isaac <arunisaac <at> systemreboot.net> writes:

> * gnu/services/web.scm (<mumi-configuration>)[data-directory, rsync-remote,
> rsync-flags]: New fields.
> (%mumi-worker-log): Delete variable.
> (%mumi-rsync-and-index-log): New variable.
> (mumi-rsync-and-index): New function.
> (mumi-shepherd-services): Remove mumi-worker service. Add mumi-rsync-and-index
> service.
> (mumi-service-type): Remove default value.
> * doc/guix.texi (Web Services)[mumi]: Document data-directory, rsync-remote
> and rsync-flags fields.
>
> Co-authored-by: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>

Reviewed-by: Maxim Cournoyer <maxim.cournoyer <at> gmail>

-- 
Thanks,
Maxim

Message #22 received at 77585-done <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: Arun Isaac <arunisaac <at> systemreboot.net>
Cc: Ludovic Courtès <ludo <at> gnu.org>, 77585-done <at> debbugs.gnu.org
Subject: Re: [bug#77585] [PATCH] services: mumi: Add Debbugs rsync as
 shepherd timer.
Date: Wed, 23 Apr 2025 11:19:09 +0900

Hi,

Arun Isaac <arunisaac <at> systemreboot.net> writes:

[...]

I think you'll see the new timer in 'herd status', e.g. on my machine:

Running timers:
 + log-cleanup
 + log-rotation

I assume they should be started automatically.

I'll close this issue as I'm assuming it's been taken care of already o
berlin.  Feel free to reopen if there are issues.

-- 
Thanks,
Maxim

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.