GNU bug report logs - #61750
[PATCH 1/2] gnu: shepherd: Build Shepherd from git.

Previous Next

To reply to this bug, email your comments to 61750 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Attila Lendvai <attila <at> lendvai.name>
To: guix-patches <at> gnu.org
Cc: Attila Lendvai <attila <at> lendvai.name>
Subject: [PATCH 1/2] gnu: shepherd: Build Shepherd from git.
Date: Fri, 24 Feb 2023 15:02:43 +0100

The rationale for this is that it makes it simpler to start VM's with a
Shepherd that is compiled from a local checkout.  This facilitates hacking on
Shepherd itself, and testing it in the Guix init environment, by speeding up
the edit-compile-test cycle.

* gnu/packages/admin.scm (shepherd-0.9): Source points to the git repo; adjust
native-inputs accordingly.  Also delete the strip build phase.
---

the first commit works, except when cross-compiling: help2man wants
to run the foreign binary, and fails.

the second commit is a WIP attempt to get the compiled man pages
from the native package when cross compiling. it's not ready,
but i got stuck at needing to gunzip the man pages, and where
to put them to convince the makefile to not execute help2man, etc.

my suggestion is to simply get rid of the not-very-useful
auto generated man page to fix cross-compilation.

if that's not desirable, then the second commit needs to be
polished, or implemented some other way.

 gnu/packages/admin.scm | 28 +++++++++++++++++-----------
 1 file changed, 17 insertions(+), 11 deletions(-)

diff --git a/gnu/packages/admin.scm b/gnu/packages/admin.scm
index d4cc8ca6ca..91f98bf48e 100644
--- a/gnu/packages/admin.scm
+++ b/gnu/packages/admin.scm
@@ -335,12 +335,13 @@ (define-public shepherd-0.9
     (inherit shepherd)
     (version "0.9.3")
     (source (origin
-              (method url-fetch)
-              (uri (string-append "mirror://gnu/shepherd/shepherd-"
-                                  version ".tar.gz"))
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://git.savannah.gnu.org/git/shepherd.git/")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name "shepherd" version))
               (sha256
-               (base32
-                "0qy2yq13xhf05an5ilz7grighdxicx56211yaarqq5qigiiybc32"))))
+               (base32 "1pvdpx2mfki8wvhcac54p8fn5idqszicww60ikswszz2cv5d0s11"))))
     (arguments
      (list #:configure-flags #~'("--localstatedir=/var")
            #:make-flags #~'("GUILE_AUTO_COMPILE=0")
@@ -360,12 +361,17 @@ (define-public shepherd-0.9
                                       (this-package-input "guile-fibers")
                                       "/lib/guile/3.0/site-ccache"))))))
                         #~%standard-phases)))
-
-    ;; Note: Use 'guile-3.0-latest' to address the continuation-related memory
-    ;; leak reported at <https://issues.guix.gnu.org/58631>.
-    (native-inputs (list pkg-config guile-3.0-latest
-                         guile-fibers-1.1))       ;for cross-compilation
-    (inputs (list guile-3.0-latest guile-fibers-1.1))))
+    (native-inputs
+     (list autoconf
+           automake
+           gettext-minimal
+           guile-3.0
+           guile-fibers-1.1             ; for cross-compilation
+           help2man
+           pkg-config
+           texinfo))
+    (inputs (list guile-3.0
+                  guile-fibers-1.1))))
 
 (define-public guile2.2-shepherd
   (package
-- 
2.38.1

Message #8 received at 61750 <at> debbugs.gnu.org (full text, mbox):

From: Attila Lendvai <attila <at> lendvai.name>
To: 61750 <at> debbugs.gnu.org
Cc: Attila Lendvai <attila <at> lendvai.name>
Subject: [PATCH 2/2] WIP failing attempt to get the man page while
 cross-compiling
Date: Fri, 24 Feb 2023 15:07:42 +0100

---
 gnu/packages/admin.scm | 48 +++++++++++++++++++++++++++++++++---------
 1 file changed, 38 insertions(+), 10 deletions(-)

diff --git a/gnu/packages/admin.scm b/gnu/packages/admin.scm
index 91f98bf48e..6e5082e612 100644
--- a/gnu/packages/admin.scm
+++ b/gnu/packages/admin.scm
@@ -343,7 +343,7 @@ (define-public shepherd-0.9
               (sha256
                (base32 "1pvdpx2mfki8wvhcac54p8fn5idqszicww60ikswszz2cv5d0s11"))))
     (arguments
-     (list #:configure-flags #~'("--localstatedir=/var")
+     (list #:configure-flags #~'("HELP2MAN=true --localstatedir=/var")
            #:make-flags #~'("GUILE_AUTO_COMPILE=0")
            #:phases (if (%current-target-system)
                         #~(modify-phases %standard-phases
@@ -359,17 +359,45 @@ (define-public shepherd-0.9
                                   (("%FIBERS_OBJECT_DIRECTORY%")
                                    #$(file-append
                                       (this-package-input "guile-fibers")
-                                      "/lib/guile/3.0/site-ccache"))))))
+                                      "/lib/guile/3.0/site-ccache")))))
+                            (add-after 'configure 'copy-man-pages
+                              (lambda* (#:key outputs native-inputs #:allow-other-keys)
+                                ;; Cannot run help2man when cross-compiling,
+                                ;; so let's copy the man pages from the native
+                                ;; build.
+                                (let ((gzip (search-input-file native-inputs
+                                                               "bin/gzip")))
+                                  ;; TODO
+                                  (pk #$(file-append
+                                         (this-package-input this-package)
+                                         "/man/shepherd.1.gz"))
+                                  (invoke gzip "--decompress" "--stdout"
+                                          #$(file-append
+                                             (this-package-input this-package)
+                                             "/man/shepherd.1.gz")
+                                          ">doc/shepherd.1"))
+                                #;(copy-file #$(file-append
+                                              (this-package-input this-package)
+                                              "/man/shepherd.1")
+                                           (string-append
+                                            (assoc-ref outputs "out")
+                                            "/man/shepherd.1"))
+                                #f)))
                         #~%standard-phases)))
     (native-inputs
-     (list autoconf
-           automake
-           gettext-minimal
-           guile-3.0
-           guile-fibers-1.1             ; for cross-compilation
-           help2man
-           pkg-config
-           texinfo))
+     (cons* autoconf
+            automake
+            gettext-minimal
+            guile-3.0
+            guile-fibers-1.1            ; for cross-compilation
+            help2man
+            pkg-config
+            texinfo
+            ;; for the help2man man page when cross-compiling
+            (if (%current-target-system)
+                (list gzip
+                      this-package)
+                '())))
     (inputs (list guile-3.0
                   guile-fibers-1.1))))
 
-- 
2.38.1

Message #11 received at 61750 <at> debbugs.gnu.org (full text, mbox):

From: Attila Lendvai <attila.lendvai <at> gmail.com>
To: 61750 <at> debbugs.gnu.org
Cc: Attila Lendvai <attila <at> lendvai.name>
Subject: [PATCH] gnu: shepherd: Build Shepherd from git.
Date: Sat, 29 Jul 2023 16:31:57 +0200

The rationale for this is that it makes it simpler to start VM's with a
Shepherd that is compiled from a local checkout.  This facilitates hacking on
Shepherd itself, and testing it in the Guix init environment, by speeding up
the edit-compile-test cycle.

* gnu/packages/admin.scm (shepherd-0.10): Source points to the git repo; adjust
native-inputs accordingly.  Also delete the strip build phase.
---

i've been running with this commit ever since i first submitted it, but
now it is causing quite a lot of local rebuilds, making it that much more
trouble.

please consider merging it if it makes senes.

note that it was not extensively tested by me WRT cross-compilation, or
non x86_64 platforms.

 gnu/packages/admin.scm | 38 +++++++++++++++++++++++---------------
 1 file changed, 23 insertions(+), 15 deletions(-)

diff --git a/gnu/packages/admin.scm b/gnu/packages/admin.scm
index ec32041055..77975654c9 100644
--- a/gnu/packages/admin.scm
+++ b/gnu/packages/admin.scm
@@ -337,12 +337,13 @@ (define-public shepherd-0.9
     (inherit shepherd-0.8)
     (version "0.9.3")
     (source (origin
-              (method url-fetch)
-              (uri (string-append "mirror://gnu/shepherd/shepherd-"
-                                  version ".tar.gz"))
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://git.savannah.gnu.org/git/shepherd.git/")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name "shepherd" version))
               (sha256
-               (base32
-                "0qy2yq13xhf05an5ilz7grighdxicx56211yaarqq5qigiiybc32"))))
+               (base32 "1pvdpx2mfki8wvhcac54p8fn5idqszicww60ikswszz2cv5d0s11"))))
     (arguments
      (list #:configure-flags #~'("--localstatedir=/var")
            #:make-flags #~'("GUILE_AUTO_COMPILE=0")
@@ -362,24 +363,31 @@ (define-public shepherd-0.9
                                       (this-package-input "guile-fibers")
                                       "/lib/guile/3.0/site-ccache"))))))
                         #~%standard-phases)))
-
-    ;; Note: Use 'guile-3.0-latest' to address the continuation-related memory
-    ;; leak reported at <https://issues.guix.gnu.org/58631>.
-    (native-inputs (list pkg-config guile-3.0-latest
-                         guile-fibers-1.1))       ;for cross-compilation
-    (inputs (list guile-3.0-latest guile-fibers-1.1))))
+    (native-inputs
+     (list autoconf
+           automake
+           gettext-minimal
+           guile-3.0
+           guile-fibers-1.1             ; for cross-compilation
+           help2man
+           pkg-config
+           texinfo))
+    (inputs (list guile-3.0
+                  guile-fibers-1.1))))
 
 (define-public shepherd-0.10
   (package
     (inherit shepherd-0.9)
     (version "0.10.2")
     (source (origin
-              (method url-fetch)
-              (uri (string-append "mirror://gnu/shepherd/shepherd-"
-                                  version ".tar.gz"))
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://git.savannah.gnu.org/git/shepherd.git/")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name "shepherd" version))
               (sha256
                (base32
-                "0v9ld9gbqdp5ya380fbkdsxa0iqr90gi6yk004ccz3n792nq6wlj"))))
+                "1kvfv4rkgs5jq2wir72m29idgk3n2i44qji9rq1hmmc8vkgqxh26"))))
     (native-inputs (modify-inputs (package-native-inputs shepherd-0.9)
                      (replace "guile-fibers" guile-fibers-1.3)))
     (inputs (modify-inputs (package-inputs shepherd-0.9)

base-commit: c7e45139faa27b60f2c7d0a4bc140f9793d97d47
prerequisite-patch-id: 5586be8a480bef6910bc7557e9491c5100f56a0e
prerequisite-patch-id: 0030a0ed3944c6bbbdefc9401325f095b92cc700
prerequisite-patch-id: 9f2204275da87bd4918b7c5f6d77643926e02afa
prerequisite-patch-id: 83a83f44b77fffb685e845ddc3ead64b711f4155
prerequisite-patch-id: 9a430e4316112cf97a40d220425dd8a5409a9e73
prerequisite-patch-id: 4b80e182f3cd3d48686f1faa15d9a3db3f354777
prerequisite-patch-id: 5b18a9e12c8eea82f8dc63dabc8c560588981a80
prerequisite-patch-id: f0ca061e8beb99c6305f38cef019cb8d66802ba7
prerequisite-patch-id: fd60adfe24e43be484f96e5403c3428ba3ef5fde
prerequisite-patch-id: 81af80e2d0821ffb9410c7c331a8ba3c4cb73aff
prerequisite-patch-id: 206663d5f2b364cf2c1598d382f8b34daedf0ba4
prerequisite-patch-id: cb5793fb638c7c086a1dbba94c115a51973f0cce
-- 
2.40.1

Message #14 received at 61750 <at> debbugs.gnu.org (full text, mbox):

From: Attila Lendvai <attila <at> lendvai.name>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 61750 <at> debbugs.gnu.org
Subject: Re: [PATCH] gnu: shepherd: Build Shepherd from git.
Date: Tue, 31 Oct 2023 17:03:57 +0000

Ludo, any hope that this gets merged?

i'm trying to debug something in shepherd (the SYSTEM call in shepherd misbehaves, but only when pulled on my server, not in a `guix system vm`). it will once again require me to use a custom shepherd, and pull/rebuild/reconfigure a gazillion times.

as for this patch: the only issue i know is that it cannot be cross compiled because of the help2man call. if you ask me, i would just get rid of that altogether. it's just added complexity for little gain.

--
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
“He who flatters a man is his enemy. He who tells him of his faults is his maker.”
	— Confucius (551–479 BC)

Message #17 received at 61750 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Attila Lendvai <attila <at> lendvai.name>
Cc: 61750 <at> debbugs.gnu.org
Subject: Re: [bug#61750] [PATCH] gnu: shepherd: Build Shepherd from git.
Date: Sun, 05 Nov 2023 15:47:03 +0100

Hi Attila,

Attila Lendvai <attila <at> lendvai.name> skribis:

> Ludo, any hope that this gets merged?

Apologies.  I’m not convinced about the patch, mainly because I’ve been
using Shepherd as a channel for testing, which I think makes fine
details about the ‘shepherd’ package in Guix less interesting.

For info on how to do that, see:

  https://git.savannah.gnu.org/cgit/shepherd.git/tree/README

The added bonus when doing this is that ‘guix system describe’ tells you
precisely which Shepherd commit you’re using.

WDYT?

Thanks,
Ludo’.

Message #20 received at 61750 <at> debbugs.gnu.org (full text, mbox):

From: Attila Lendvai <attila <at> lendvai.name>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 61750 <at> debbugs.gnu.org
Subject: Re: [bug#61750] [PATCH] gnu: shepherd: Build Shepherd from git.
Date: Mon, 13 Nov 2023 09:38:52 +0000

> Apologies. I’m not convinced about the patch, mainly because I’ve been
> using Shepherd as a channel for testing, which I think makes fine
> details about the ‘shepherd’ package in Guix less interesting.
> 
> For info on how to do that, see:
> 
> https://git.savannah.gnu.org/cgit/shepherd.git/tree/README
> 
> The added bonus when doing this is that ‘guix system describe’ tells you
> precisely which Shepherd commit you’re using.
> 
> WDYT?


ouch, we've been here before, sorry for my loss of memory!

but after having read the README once again, i'm still not sure how this would work. please confirm my assumption:

if i add shepherd as a channel, then i unavoidably need to issue a `guix pull` (slow!) before i can test any shepherd changes, even if the channel points to a local checkout. also, i need to record any pending changes that i want to test into a git commit.

this is my objective:
 - be able to edit files in a shepherd checkout
 - issue a command that runs a vm with my shepherd changes included
 - keep this cycle reasonably fast to avoid frustration

my ultimate objective:

fix a shepherd-action bug that doesn't manifest in my `guix system vm` tests, only when i pull stuff to my server (slow!). i have zero info on what goes wrong besides a non-zero return code from tar that my action invokes. to gain more info i'm planning to add extensive logging to shepherd, especially around its fork+exec-command calls.

this is how i used to hack shepherd, when i need to test it in its real environment:

i have a guix system test that i have extracted from the guix test suite. it defines an OPERATING-SYSTEM object that is run using `guix system vm`:

https://github.com/attila-lendvai/guix-crypto/blob/main/tests/swarm-tests.scm

a custom shepherd package is defined for this OS object by modifying its SHEPHERD-ROOT-SERVICE-TYPE.

my custom shepherd package inherits from the shepherd package from guix, and replaces its source to point to my own shepherd checkout. for this to work, my local guix contains the patch to compile shepherd from git.

this worked fine, but something broke my setup recently. it was that some heavy packages started to depend on shepherd, which means that if i include my patch to compile shepherd from git, then all those packages need to be recompiled locally, which is pohibitively slow (it's added to the edit-compile-run cycle).

a tangential: i think these new dependencies are only needed for the CLI tools provided by the shepherd package. splitting shepherd into two packages (CLI tools and service) would also resolve this problem of mine.

possible solutions:
 1) i learn some better way to hack on shepherd

 2) shepherd in guix is compiled from git, so that i can easily inherit from it

 3) i give up on getting this merged, and try to keep a completely standalone shepherd package in sync with what's in guix. i also give up on adding detailed instructions to the docs on how to hack on shepherd (https://issues.guix.gnu.org/54199).

i'd appreciate some guidance on how to proceed from here. i don't want to waste anyone's time further with this. if this change is not welcome in guix proper, then reject the patch and i'll try to find another way (probably option 3) to achieve my goals.

-- 
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
“Think what you do when you run in debt; you give to another power over your liberty.”
	— Benjamin Franklin (1706–1790), 'The Way to Wealth' (1758)

Message #23 received at 61750 <at> debbugs.gnu.org (full text, mbox):

From: Attila Lendvai <attila <at> lendvai.name>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 61750 <at> debbugs.gnu.org
Subject: Re: [bug#61750] [PATCH] gnu: shepherd: Build Shepherd from git.
Date: Fri, 17 Nov 2023 15:26:32 +0000

> but after having read the README once again, i'm still not sure how this would work. please confirm my assumption:
> 
> if i add shepherd as a channel, then i unavoidably need to issue a `guix pull` (slow!) before i can test any shepherd changes, even if the channel points to a local checkout. also, i need to record any pending changes that i want to test into a git commit.


FTR, i've made progress, and i think i'm back to my original routine.

what i was missing is adding this to my guix .envrc:

export GUILE_LOAD_PATH=~/workspace/guix/shepherd/.guix/modules:$GUILE_LOAD_PATH

this then enables the following:

(define *parent-shepherd-package* (@ (shepherd-package) shepherd))

with this shepherd package i can once again start a `guix system vm` with all the changes in my shepherd checkout compiled in, and with a bearable turnaround time.

note that this was not suggested in the README. i'll keep an eye on updating the docs to help others getting into shepherd hacking.

thanks for your patience and help Ludo!

-- 
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
“When a wise man points at the moon the imbecile examines the finger.”
	— Confucius (551–479 BC)

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.