GNU bug report logs - #76735
Shepherd parallel make help2man issue

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 76735 in the body.
You can then email your comments to 76735 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Simon Josefsson <simon <at> josefsson.org>
To: bug-guix <at> gnu.org
Subject: Shepherd parallel make help2man issue
Date: Tue, 04 Mar 2025 08:22:14 +0100

[Message part 1 (text/plain, inline)]

Hi

I got the build error below.  Presumably it happens because the jobs to
create the tools has not finished before the help2man jobs has started.

I will patch around this in the Debian packaging to serialize the build:

override_dh_auto_build:
	# build tools that help2man needs first
	dh_auto_build $(DH_BUILD_OPTS) -- shepherd herd halt reboot
	dh_auto_build $(DH_BUILD_OPTS)

But it seems like a upstream problem.  I know how annoying these kind of
bugs are, sorry for another report of this type :)

/Simon

   dh_auto_build
	make -j42
make[1]: Entering directory '/build/reproducible-path/shepherd-1.0.2'
/usr/bin/mkdir -p "`dirname modules/shepherd/config.scm`"
/usr/bin/sed -e 's,%PREFIX%,/usr,g' -e 's,%modsrcdir%,/build/reproducible-path/shepherd-1.0.2/modules,g' -e 's,%modbuilddir%,/build/reproducible-path/shepherd-1.0.2/modules,g' -e 's,%localstatedir%,/var,g' -e 's,%runstatedir%,/run,g' -e 's,%pkglibdir%,/usr/lib/x86_64-linux-gnu/shepherd,g' -e 's,%sysconfdir%,/etc,g' -e 's,%localedir%,/usr/share/locale,g' -e 's,%VERSION%,1.0.2,g' -e 's,%PACKAGE_BUGREPORT%,bug-guix <at> gnu.org,g' -e 's,%PACKAGE_NAME%,GNU Shepherd,g' -e 's,%PACKAGE_URL%,https://www.gnu.org/software/shepherd/,g' -e 's,%FIBERS_SOURCE_DIRECTORY%,/usr/share/guile/site/3.0,g' -e 's,%FIBERS_OBJECT_DIRECTORY%,/usr/lib/x86_64-linux-gnu/guile/3.0/site-ccache,g' -e 's,%GUILE%,/usr/bin/guile-3.0,g' < modules/shepherd/config.scm.in >modules/shepherd/config.scm
make  all-recursive
make[2]: Entering directory '/build/reproducible-path/shepherd-1.0.2'
Making all in po
make[3]: Entering directory '/build/reproducible-path/shepherd-1.0.2/po'
make[3]: Nothing to be done for 'all'.
make[3]: Leaving directory '/build/reproducible-path/shepherd-1.0.2/po'
make[3]: Entering directory '/build/reproducible-path/shepherd-1.0.2'
/usr/bin/mkdir -p "`dirname shepherd`" ;		\
/usr/bin/sed -e 's,%PREFIX%,/usr,g' -e 's,%modsrcdir%,/build/reproducible-path/shepherd-1.0.2/modules,g' -e 's,%modbuilddir%,/build/reproducible-path/shepherd-1.0.2/modules,g' -e 's,%localstatedir%,/var,g' -e 's,%runstatedir%,/run,g' -e 's,%pkglibdir%,/usr/lib/x86_64-linux-gnu/shepherd,g' -e 's,%sysconfdir%,/etc,g' -e 's,%localedir%,/usr/share/locale,g' -e 's,%VERSION%,1.0.2,g' -e 's,%PACKAGE_BUGREPORT%,bug-guix <at> gnu.org,g' -e 's,%PACKAGE_NAME%,GNU Shepherd,g' -e 's,%PACKAGE_URL%,https://www.gnu.org/software/shepherd/,g' -e 's,%FIBERS_SOURCE_DIRECTORY%,/usr/share/guile/site/3.0,g' -e 's,%FIBERS_OBJECT_DIRECTORY%,/usr/lib/x86_64-linux-gnu/guile/3.0/site-ccache,g' -e 's,%GUILE%,/usr/bin/guile-3.0,g' < shepherd.in >shepherd ;		\
 chmod +x shepherd
/usr/bin/mkdir -p "`dirname herd`" ;		\
/usr/bin/sed -e 's,%PREFIX%,/usr,g' -e 's,%modsrcdir%,/build/reproducible-path/shepherd-1.0.2/modules,g' -e 's,%modbuilddir%,/build/reproducible-path/shepherd-1.0.2/modules,g' -e 's,%localstatedir%,/var,g' -e 's,%runstatedir%,/run,g' -e 's,%pkglibdir%,/usr/lib/x86_64-linux-gnu/shepherd,g' -e 's,%sysconfdir%,/etc,g' -e 's,%localedir%,/usr/share/locale,g' -e 's,%VERSION%,1.0.2,g' -e 's,%PACKAGE_BUGREPORT%,bug-guix <at> gnu.org,g' -e 's,%PACKAGE_NAME%,GNU Shepherd,g' -e 's,%PACKAGE_URL%,https://www.gnu.org/software/shepherd/,g' -e 's,%FIBERS_SOURCE_DIRECTORY%,/usr/share/guile/site/3.0,g' -e 's,%FIBERS_OBJECT_DIRECTORY%,/usr/lib/x86_64-linux-gnu/guile/3.0/site-ccache,g' -e 's,%GUILE%,/usr/bin/guile-3.0,g' < herd.in >herd ;		\
 chmod +x herd
/usr/bin/mkdir -p "`dirname halt`" ;		\
/usr/bin/sed -e 's,%PREFIX%,/usr,g' -e 's,%modsrcdir%,/build/reproducible-path/shepherd-1.0.2/modules,g' -e 's,%modbuilddir%,/build/reproducible-path/shepherd-1.0.2/modules,g' -e 's,%localstatedir%,/var,g' -e 's,%runstatedir%,/run,g' -e 's,%pkglibdir%,/usr/lib/x86_64-linux-gnu/shepherd,g' -e 's,%sysconfdir%,/etc,g' -e 's,%localedir%,/usr/share/locale,g' -e 's,%VERSION%,1.0.2,g' -e 's,%PACKAGE_BUGREPORT%,bug-guix <at> gnu.org,g' -e 's,%PACKAGE_NAME%,GNU Shepherd,g' -e 's,%PACKAGE_URL%,https://www.gnu.org/software/shepherd/,g' -e 's,%FIBERS_SOURCE_DIRECTORY%,/usr/share/guile/site/3.0,g' -e 's,%FIBERS_OBJECT_DIRECTORY%,/usr/lib/x86_64-linux-gnu/guile/3.0/site-ccache,g' -e 's,%GUILE%,/usr/bin/guile-3.0,g' < halt.in >halt ;		\
 chmod +x halt
/usr/bin/mkdir -p "`dirname reboot`" ;		\
/usr/bin/sed -e 's,%PREFIX%,/usr,g' -e 's,%modsrcdir%,/build/reproducible-path/shepherd-1.0.2/modules,g' -e 's,%modbuilddir%,/build/reproducible-path/shepherd-1.0.2/modules,g' -e 's,%localstatedir%,/var,g' -e 's,%runstatedir%,/run,g' -e 's,%pkglibdir%,/usr/lib/x86_64-linux-gnu/shepherd,g' -e 's,%sysconfdir%,/etc,g' -e 's,%localedir%,/usr/share/locale,g' -e 's,%VERSION%,1.0.2,g' -e 's,%PACKAGE_BUGREPORT%,bug-guix <at> gnu.org,g' -e 's,%PACKAGE_NAME%,GNU Shepherd,g' -e 's,%PACKAGE_URL%,https://www.gnu.org/software/shepherd/,g' -e 's,%FIBERS_SOURCE_DIRECTORY%,/usr/share/guile/site/3.0,g' -e 's,%FIBERS_OBJECT_DIRECTORY%,/usr/lib/x86_64-linux-gnu/guile/3.0/site-ccache,g' -e 's,%GUILE%,/usr/bin/guile-3.0,g' < reboot.in >reboot ;		\
 chmod +x reboot
LANGUAGE= /bin/bash '/build/reproducible-path/shepherd-1.0.2/build-aux/missing' help2man --source=GNU --info-page=shepherd --output="doc/shepherd.1" "./`basename "doc/shepherd.1" .1`"
LANGUAGE= /bin/bash '/build/reproducible-path/shepherd-1.0.2/build-aux/missing' help2man --source=GNU --info-page=shepherd --output="doc/herd.1" "./`basename "doc/herd.1" .1`"
make[4]: Entering directory '/build/reproducible-path/shepherd-1.0.2'
help2man: can't get `--help' info from ./herd
Try `--no-discard-stderr' if option outputs to stderr
help2man: can't get `--help' info from ./shepherd
Try `--no-discard-stderr' if option outputs to stderr
WARNING: 'help2man' is missing on your system.
         You should only need it if you modified a dependency of a man page.
         You may want to install the GNU Help2man package:
         <https://www.gnu.org/software/help2man/>
WARNING: 'help2man' is missing on your system.
         You should only need it if you modified a dependency of a man page.
         You may want to install the GNU Help2man package:
         <https://www.gnu.org/software/help2man/>
make[3]: *** [Makefile:2252: doc/shepherd.1] Error 127
make[3]: *** Waiting for unfinished jobs....
make[3]: *** [Makefile:2255: doc/herd.1] Error 127
make[4]: Leaving directory '/build/reproducible-path/shepherd-1.0.2'
Updating ./doc/version.texi
make[3]: Leaving directory '/build/reproducible-path/shepherd-1.0.2'
make[2]: *** [Makefile:1470: all-recursive] Error 1
make[2]: Leaving directory '/build/reproducible-path/shepherd-1.0.2'
make[1]: *** [Makefile:871: all] Error 2
make[1]: Leaving directory '/build/reproducible-path/shepherd-1.0.2'
dh_auto_build: error: make -j42 returned exit code 2

[signature.asc (application/pgp-signature, inline)]

Message #8 received at 76735 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Simon Josefsson <simon <at> josefsson.org>
Cc: 76735 <at> debbugs.gnu.org
Subject: Re: bug#76735: Shepherd parallel make help2man issue
Date: Sun, 09 Mar 2025 21:32:31 +0100

[Message part 1 (text/plain, inline)]

Hi Simon,

Simon Josefsson <simon <at> josefsson.org> skribis:

> I got the build error below.  Presumably it happens because the jobs to
> create the tools has not finished before the help2man jobs has started.

Am I right that you’re building from a Git checkout?

The problem is that the Makefile rules for man pages are meant for
developers building the tarball; the tarball already includes man pages,
so someone building from the tarball doesn’t trigger any of these rules.

The changes below fix dependencies, but it also means that now everyone
will be building man pages from source, even if they’re already included
in the tarball.  That’s probably the right thing to do longer term, but
I’d rather not do it in the middle of a stable series.

WDYT?

Ludo’.

[Message part 2 (text/x-patch, inline)]

diff --git a/Makefile.am b/Makefile.am
index 23d1360..9d5e053 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -131,15 +131,18 @@ AM_V_HELP2MAN_0 = @echo "  HELP2MAN" $@;
 HELP2MANFLAGS = --source=GNU --info-page=$(PACKAGE_TARNAME)
 gen_man = $(AM_V_HELP2MAN)LANGUAGE= $(HELP2MAN) $(HELP2MANFLAGS)
 
-doc/shepherd.1: modules/shepherd.scm configure.ac
+doc/shepherd.1: shepherd $(nodist_shepherdsub_DATA) modules/shepherd.scm configure.ac
+	$(MKDIR_P) doc
 	$(gen_man) --name="Service manager for the GNU system" \
 	--output="$@" "$(top_builddir)/`basename "$@" .1`"
 
-doc/herd.1: modules/shepherd/scripts/herd.scm configure.ac
+doc/herd.1: herd $(nodist_shepherdsub_DATA) modules/shepherd/scripts/herd.scm configure.ac
+	$(MKDIR_P) doc
 	$(gen_man) --name="Control the shepherd(1) service manager" \
 	--output="$@" "$(top_builddir)/`basename "$@" .1`"
 
-doc/%.8: modules/shepherd/scripts/%.scm configure.ac
+doc/%.8: % $(nodist_shepherdsub_DATA) modules/shepherd/scripts/%.scm configure.ac
+	$(MKDIR_P) doc
 	$(gen_man) --section=8 --output="$@"	\
 	  "$(top_builddir)/`basename "$@" .8`"
 
@@ -223,13 +226,13 @@ instantiate =						\
 
 # Emit a 'chmod +x' command only if $@ matches one of the scripts.
 maybe_executable =					\
-   $(subst $@,chmod +x $@,				\
+   $(subst $@,chmod +x "$@.tmp",			\
       $(findstring $@,$(bin_SCRIPTS) $(sbin_SCRIPTS)))
 
 %: %.in Makefile
 	$(AM_V_GEN)$(MKDIR_P) "`dirname $@`" ;		\
-	$(SED) $(instantiate) < $< >$@ ;		\
-	$(maybe_executable)
+	$(SED) $(instantiate) < "$<" > "$@.tmp" ;	\
+	$(maybe_executable) ; mv "$@.tmp" "$@"
 
 modules/shepherd/config.scm: modules/shepherd/config.scm.in Makefile
 	$(MKDIR_P) "`dirname $@`"
diff --git a/halt.in b/halt.in
index 93ac83c..6516cf2 100644
--- a/halt.in
+++ b/halt.in
@@ -2,6 +2,6 @@
 -*- scheme -*-
 !#
 
-(set! %load-path (cons "%modsrcdir%" %load-path))
+(set! %load-path (append '("%modbuilddir%" "%modsrcdir%") %load-path))
 (set! %load-compiled-path (cons "%modbuilddir%" %load-compiled-path))
 (apply (@ (shepherd scripts halt) main) (cdr (command-line)))
diff --git a/herd.in b/herd.in
index e61d26b..09e2fd7 100644
--- a/herd.in
+++ b/herd.in
@@ -3,7 +3,7 @@
 !#
 
 (set! %load-path
-      (append '("%modsrcdir%" "%FIBERS_SOURCE_DIRECTORY%") %load-path))
+      (append '("%modbuilddir%" "%modsrcdir%" "%FIBERS_SOURCE_DIRECTORY%") %load-path))
 (set! %load-compiled-path
       (append '("%modbuilddir%" "%FIBERS_OBJECT_DIRECTORY%") %load-compiled-path))
 
diff --git a/reboot.in b/reboot.in
index 254b095..7bd93fa 100644
--- a/reboot.in
+++ b/reboot.in
@@ -2,6 +2,6 @@
 -*- scheme -*-
 !#
 
-(set! %load-path (cons "%modsrcdir%" %load-path))
+(set! %load-path (append '("%modbuilddir%" "%modsrcdir%") %load-path))
 (set! %load-compiled-path (cons "%modbuilddir%" %load-compiled-path))
 (apply (@ (shepherd scripts reboot) main) (cdr (command-line)))
diff --git a/shepherd.in b/shepherd.in
index 29a8227..b61fd35 100644
--- a/shepherd.in
+++ b/shepherd.in
@@ -3,7 +3,7 @@
 !#
 
 (set! %load-path
-      (append '("%modsrcdir%" "%FIBERS_SOURCE_DIRECTORY%") %load-path))
+      (append '("%modbuilddir%" "%modsrcdir%" "%FIBERS_SOURCE_DIRECTORY%") %load-path))
 (set! %load-compiled-path
       (append '("%modbuilddir%" "%FIBERS_OBJECT_DIRECTORY%") %load-compiled-path))
 

Message #11 received at 76735 <at> debbugs.gnu.org (full text, mbox):

From: Simon Josefsson <simon <at> josefsson.org>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 76735 <at> debbugs.gnu.org
Subject: Re: bug#76735: Shepherd parallel make help2man issue
Date: Sun, 09 Mar 2025 22:09:51 +0100

[Message part 1 (text/plain, inline)]

Ludovic Courtès <ludo <at> gnu.org> writes:

> Hi Simon,
>
> Simon Josefsson <simon <at> josefsson.org> skribis:
>
>> I got the build error below.  Presumably it happens because the jobs to
>> create the tools has not finished before the help2man jobs has started.
>
> Am I right that you’re building from a Git checkout?

This was during build of Debian packages of Shepherd.  They use the
tarballs.

> The problem is that the Makefile rules for man pages are meant for
> developers building the tarball; the tarball already includes man pages,
> so someone building from the tarball doesn’t trigger any of these
> rules.

Ah.  The Debian packaging intentionally removes the prebuilt manpages
because we want to know that we can rebuild them from source.  That is
what is triggering this problem!

execute_after_dh_auto_clean:
	rm -fv doc/shepherd.info doc/herd.1 doc/shepherd.1
	rm -fv doc/stamp-vti doc/version.texi

So maybe this is not so much of an upstream problem after all, and my
fault for trying to rebuild the manpages.

Maybe we should build from a 'git-archive' snapshot instead... but those
doesn't come with PGP signatures right now, although you could do that:

https://blog.josefsson.org/2024/04/13/reproducible-and-minimal-source-only-tarballs/

essentially just:

git archive --prefix=shepherd-vX.Y/ -o shepherd-vX.Y-src.tar.gz vX.Y
gpg --sign shepherd-vX.Y-src.tar.gz

> The changes below fix dependencies, but it also means that now everyone
> will be building man pages from source, even if they’re already included
> in the tarball.  That’s probably the right thing to do longer term, but
> I’d rather not do it in the middle of a stable series.

Thanks for developing that patch.  This problem is complex, and I'm not
sure what the best solution really is.  Your patch makes the man pages
depend on the binary, and I've had bad experiences with that.  If
someone is cross-compile Shepherd (not sure if that's a realistic
situation) then help2man can't run the binary, and in this situation it
would be preferable to fall back on the prebuilt manpages.  That doesn't
sound ideal.

Maybe an uglier solution is to try to run help2man, and if it works,
then fine, but if it doesn't, re-use the prebuilt manpage or fail.

/Simon

[signature.asc (application/pgp-signature, inline)]

Message #14 received at 76735 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Simon Josefsson <simon <at> josefsson.org>
Cc: 76735 <at> debbugs.gnu.org
Subject: Re: bug#76735: Shepherd parallel make help2man issue
Date: Mon, 10 Mar 2025 21:03:23 +0100

Hi Simon,

Simon Josefsson <simon <at> josefsson.org> skribis:

> Ah.  The Debian packaging intentionally removes the prebuilt manpages
> because we want to know that we can rebuild them from source.  That is
> what is triggering this problem!

Oh.  That makes perfect sense from a reproducible build perspective.

> Thanks for developing that patch.  This problem is complex, and I'm not
> sure what the best solution really is.  Your patch makes the man pages
> depend on the binary, and I've had bad experiences with that.  If
> someone is cross-compile Shepherd (not sure if that's a realistic
> situation) then help2man can't run the binary, and in this situation it
> would be preferable to fall back on the prebuilt manpages.  That doesn't
> sound ideal.

Right, I had forgotten about cross-compilation (we do rely on it in
Guix).

So in practice, we have no choice but to build the tarball natively and
then (cross-)build from the tarball.

> Maybe an uglier solution is to try to run help2man, and if it works,
> then fine, but if it doesn't, re-use the prebuilt manpage or fail.

Yeah.

Or, longer-term, extract command-line metadata from the Guile modules
and generate man pages without resorting to ‘help2man’.

Ludo’.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.