GNU bug report logs - #65732
[PATCH 0/2] Add the Build Farm Front-End

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 65732 in the body.
You can then email your comments to 65732 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Christopher Baines <mail <at> cbaines.net>
To: guix-patches <at> gnu.org
Subject: [PATCH 0/2] Add the Build Farm Front-End
Date: Mon, 04 Sep 2023 13:29:12 +0100

[Message part 1 (text/plain, inline)]

This is the package and service for the Build Farm Front-End, intended
to replace the Guix Build Coordinator queue builds script, and also
provide a web interface for build farms.


Christopher Baines (2):
  gnu: Add bffe.
  services: guix: Add bffe-service-type.

 doc/guix.texi                       |  59 +++++++++++++
 gnu/packages/package-management.scm |  51 +++++++++++
 gnu/services/guix.scm               | 131 +++++++++++++++++++++++++++-
 gnu/tests/guix.scm                  |  81 ++++++++++++++++-
 4 files changed, 320 insertions(+), 2 deletions(-)


base-commit: 8d70ff3a49f7cdf0eab93b3a1c54ec2f016afc4a
-- 
2.41.0

[signature.asc (application/pgp-signature, inline)]

Message #8 received at 65732 <at> debbugs.gnu.org (full text, mbox):

From: Christopher Baines <mail <at> cbaines.net>
To: 65732 <at> debbugs.gnu.org
Subject: [PATCH 1/2] gnu: Add bffe.
Date: Mon,  4 Sep 2023 13:32:46 +0100

* gnu/packages/package-management.scm (bffe): New variable.
---
 gnu/packages/package-management.scm | 51 +++++++++++++++++++++++++++++
 1 file changed, 51 insertions(+)

diff --git a/gnu/packages/package-management.scm b/gnu/packages/package-management.scm
index aa24798071..a49f302bb7 100644
--- a/gnu/packages/package-management.scm
+++ b/gnu/packages/package-management.scm
@@ -961,6 +961,57 @@ (define-public rpm
     ;; The whole is GPLv2+; librpm itself is dual-licensed LGPLv2+ | GPLv2+.
     (license license:gpl2+)))
 
+(define-public bffe
+  (let ((commit "3ce4613908bb4a42494323ef0597f6c3ae2dee24")
+        (revision "1"))
+    (package
+      (name "bffe")
+      (version (git-version "0" revision commit))
+      (source (origin
+                (method git-fetch)
+                (uri (git-reference
+                      (url "https://git.cbaines.net/guix/bffe")
+                      (commit commit)))
+                (sha256
+                 (base32
+                  "0g7arfl8rwybqpmw8vc16ilvrva1bb435lqgbqypx0nphbcv2pxn"))
+                (file-name (string-append name "-" version "-checkout"))))
+      (build-system gnu-build-system)
+      (native-inputs
+       (list pkg-config
+             autoconf
+             automake
+
+             ;; Guile libraries are needed here for cross-compilation.
+             (car (assoc-ref (package-native-inputs guix) "guile"))
+             guile-gnutls
+             guile-json-4
+             guix
+             guix-data-service
+             guix-build-coordinator
+             guile-fibers-1.3
+             guile-prometheus
+             guile-lib))
+      (propagated-inputs
+       (list guile-gnutls
+             guile-json-4
+             guix
+             guix-data-service
+             guix-build-coordinator
+             guile-fibers-1.3
+             guile-prometheus
+             guile-lib))
+      (home-page "https://git.cbaines.net/guix/bffe")
+      (synopsis "Build Farm Front-end for Guix")
+      (description
+       "The BFFE of Build Farm Front-end is an experimental frontend for Guix
+build farms.  It works together with the Guix Data Service and Guix Build
+Coordinator to submit builds and monitor the activity.
+
+It functions as a Guile library, with the @code{run-bffe-service} procedure in
+the @code{(bffe)} module as the entry point.")
+      (license license:gpl3+))))
+
 (define-public python-anaconda-client
   (package
     (name "python-anaconda-client")

base-commit: 8d70ff3a49f7cdf0eab93b3a1c54ec2f016afc4a
-- 
2.41.0

Message #11 received at 65732 <at> debbugs.gnu.org (full text, mbox):

From: Christopher Baines <mail <at> cbaines.net>
To: 65732 <at> debbugs.gnu.org
Subject: [PATCH 2/2] services: guix: Add bffe-service-type.
Date: Mon,  4 Sep 2023 13:32:47 +0100

This is intended to replace the functionality of the Guix Build Coordinator
queue builds script, and also provide a web interface for build farms.

* gnu/services/guix.scm (<bffe-configuration>): New record type.
(bffe-configuration, bffe-configuration?,
bffe-configuration-package,
bffe-configuration-user,
bffe-configuration-group,
bffe-configuration-arguments
bffe-configuration-extra-environment-variables): New procedures.
(bffe-service-type): New variable.
* gnu/tests/guix.scm (%test-bffe): New variable.
* doc/guix.texi (Guix Services): Document the new service.
---
 doc/guix.texi         |  59 +++++++++++++++++++
 gnu/services/guix.scm | 131 +++++++++++++++++++++++++++++++++++++++++-
 gnu/tests/guix.scm    |  81 +++++++++++++++++++++++++-
 3 files changed, 269 insertions(+), 2 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index a6b74ce9c7..bf12f8945d 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -38083,6 +38083,65 @@ PAM Mount Service
 @node Guix Services
 @subsection Guix Services
 
+@subsubheading Build Farm Front-End (BFFE)
+The @uref{https://git.cbaines.net/guix/bffe/,Build Farm Front-End}
+assists with building Guix packages in bulk.  It's responsible for
+submitting builds and displaying the status of the build farm.
+
+@defvar bffe-service-type
+Service type for the Build Farm Front-End.  Its value must be a
+@code{bffe-configuration} object.
+@end defvar
+
+@deftp {Data Type} bffe-configuration
+Data type representing the configuration of the Build Farm Front-End.
+
+@table @asis
+@item @code{package} (default: @code{bffe})
+The Build Farm Front-End package to use.
+
+@item @code{user} (default: @code{"bffe"})
+The system user to run the service as.
+
+@item @code{group} (default: @code{"bffe"})
+The system group to run the service as.
+
+@item @code{arguments}
+A list of arguments to the Build Farm Front-End.  These are passed to
+the @code{run-bffe-service} procedure when starting the service.
+
+For example, the following value directs the Build Farm Front-End to
+submit builds for derivations available from @code{data.guix.gnu.org} to
+the Build Coordinator instance assumed to be running on the same
+machine.
+
+@example
+(list
+ #:build
+ (list
+  (build-from-guix-data-service
+   (data-service-url "https://data.guix.gnu.org")
+   (build-coordinator-url "http://127.0.0.1:8746")
+   (branches '("master"))
+   (systems '("x86_64-linux" "i686-linux"))
+   (systems-and-targets
+    (map (lambda (target)
+           (cons "x86_64-linux" target))
+         '("aarch64-linux-gnu"
+           "i586-pc-gnu")))
+   (build-priority (const 0))))
+ #:web-server-args
+ '(#:event-source "https://example.com"
+   #:controller-args
+   (#:title "example.com build farm")))
+@end example
+
+@item @code{extra-environment-variables} (default: @var{'()})
+Extra environment variables to set via the shepherd service.
+
+@end table
+@end deftp
+
 @subsubheading Guix Build Coordinator
 The @uref{https://git.cbaines.net/guix/build-coordinator/,Guix Build
 Coordinator} aids in distributing derivation builds among machines
diff --git a/gnu/services/guix.scm b/gnu/services/guix.scm
index 99b21f52d8..e9db2a231d 100644
--- a/gnu/services/guix.scm
+++ b/gnu/services/guix.scm
@@ -140,7 +140,17 @@ (define-module (gnu services guix)
             nar-herder-cached-compression-configuration-type
             nar-herder-cached-compression-configuration-level
             nar-herder-cached-compression-configuration-directory
-            nar-herder-cached-compression-configuration-directory-max-size))
+            nar-herder-cached-compression-configuration-directory-max-size
+
+            bffe-configuration
+            bffe-configuration?
+            bffe-configuration-package
+            bffe-configuration-user
+            bffe-configuration-group
+            bffe-configuration-arguments
+            bffe-configuration-extra-environment-variables
+
+            bffe-service-type))
 
 ;;;; Commentary:
 ;;;
@@ -1030,3 +1040,122 @@ (define nar-herder-service-type
                         nar-herder-account)))
    (description
     "Run a Nar Herder server.")))
+
+
+;;;
+;;; Build Farm Front-end (BFFE)
+;;;
+
+(define-record-type* <bffe-configuration>
+  bffe-configuration make-bffe-configuration
+  bffe-configuration?
+  (package       bffe-configuration-package
+                 (default bffe))
+  (user          bffe-configuration-user
+                 (default "bffe"))
+  (group         bffe-configuration-group
+                 (default "bffe"))
+  (arguments     bffe-configuration-arguments)
+  (extra-environment-variables
+   bffe-configuration-extra-environment-variables
+   (default '())))
+
+(define (bffe-shepherd-services config)
+  (define bffe-package
+    (bffe-configuration-package config))
+
+  (define start-script
+    (program-file
+     "run-bffe"
+     (with-extensions (cons
+                       bffe-package
+                       ;; This is a poorly constructed Guile load path,
+                       ;; since it contains things that aren't Guile
+                       ;; libraries, but it means that the Guile
+                       ;; libraries needed for BFFE don't need to be
+                       ;; individually specified here.
+                       (map second (package-transitive-propagated-inputs
+                                    bffe-package)))
+       #~(begin
+           (use-modules (bffe)
+                        (bffe manage-builds))
+
+           (setvbuf (current-output-port) 'line)
+           (setvbuf (current-error-port) 'line)
+
+           (simple-format #t "starting the bffe:\n  ~A\n"
+                          (current-filename))
+
+           (apply run-bffe-service
+                  (append
+                   (list #:pid-file "/var/run/bffe/pid")
+                   #$(bffe-configuration-arguments config)))))
+     #:guile guile-3.0))
+
+  (match-record config <bffe-configuration>
+    (package user group arguments extra-environment-variables)
+
+    (list
+     (shepherd-service
+      (documentation "Build Farm Front-end")
+      (provision '(bffe))
+      (requirement '(networking))
+      (start #~(make-forkexec-constructor
+                (list #$start-script)
+                #:user #$user
+                #:group #$group
+                #:pid-file "/var/run/bffe/pid"
+                #:directory "/var/lib/bffe"
+                #:environment-variables
+                `(,(string-append
+                    "GUIX_LOCPATH=" #$glibc-utf8-locales "/lib/locale")
+                  "LC_ALL=en_US.utf8"
+                  #$@extra-environment-variables)
+                #:log-file "/var/log/bffe/server.log"))
+      (stop #~(make-kill-destructor))))))
+
+(define (bffe-activation config)
+  #~(begin
+      (use-modules (guix build utils))
+
+      (define %user
+        (getpw #$(bffe-configuration-user
+                  config)))
+
+      (chmod "/var/lib/bffe" #o755)
+
+      (mkdir-p "/var/log/bffe")
+
+      ;; Allow writing the PID file
+      (mkdir-p "/var/run/bffe")
+      (chown "/var/run/bffe"
+             (passwd:uid %user)
+             (passwd:gid %user))))
+
+(define (bffe-account config)
+  (match-record config <bffe-configuration>
+    (user group)
+    (list (user-group
+           (name group)
+           (system? #t))
+          (user-account
+           (name user)
+           (group group)
+           (system? #t)
+           (comment "BFFE user")
+           (home-directory "/var/lib/bffe")
+           (shell (file-append shadow "/sbin/nologin"))))))
+
+(define bffe-service-type
+  (service-type
+   (name 'bffe)
+   (extensions
+    (list
+     (service-extension shepherd-root-service-type
+                        bffe-shepherd-services)
+     (service-extension activation-service-type
+                        bffe-activation)
+     (service-extension account-service-type
+                        bffe-account)))
+   (description
+    "Run the Build Farm Front-end.")))
diff --git a/gnu/tests/guix.scm b/gnu/tests/guix.scm
index ad0980a10c..240ded4825 100644
--- a/gnu/tests/guix.scm
+++ b/gnu/tests/guix.scm
@@ -37,7 +37,8 @@ (define-module (gnu tests guix)
   #:use-module (ice-9 match)
   #:export (%test-guix-build-coordinator
             %test-guix-data-service
-            %test-nar-herder))
+            %test-nar-herder
+            %test-bffe))
 
 ;;;
 ;;; Guix Build Coordinator
@@ -325,3 +326,81 @@ (define %test-nar-herder
    (name "nar-herder")
    (description "Connect to a running Nar Herder server.")
    (value (run-nar-herder-test))))
+
+
+;;;
+;;; Build Farm Front-end
+;;;
+
+(define %bffe-os
+  (simple-operating-system
+   (service dhcp-client-service-type)
+   (service guix-build-coordinator-service-type)
+   (service bffe-service-type
+            (bffe-configuration
+             (arguments
+              #~(list
+                 #:web-server-args
+                 '(#:port 8767
+                   #:controller-args
+                   (#:title "Test title"))))))))
+
+(define (run-bffe-test)
+  (define os
+    (marionette-operating-system
+     %bffe-os
+     #:imported-modules '((gnu services herd)
+                          (guix combinators))))
+
+  (define forwarded-port 8767)
+
+  (define vm
+    (virtual-machine
+     (operating-system os)
+     (memory-size 1024)
+     (port-forwardings `((,forwarded-port . 8767)))))
+
+  (define test
+    (with-imported-modules '((gnu build marionette))
+      #~(begin
+          (use-modules (srfi srfi-11) (srfi srfi-64)
+                       (gnu build marionette)
+                       (web uri)
+                       (web client)
+                       (web response))
+
+          (define marionette
+            (make-marionette (list #$vm)))
+
+          (test-runner-current (system-test-runner #$output))
+          (test-begin "bffe")
+
+          (test-assert "service running"
+            (marionette-eval
+             '(begin
+                (use-modules (gnu services herd))
+                (match (start-service 'bffe)
+                  (#f #f)
+                  (('service response-parts ...)
+                   (match (assq-ref response-parts 'running)
+                     ((pid) (number? pid))))))
+             marionette))
+
+          (test-equal "http-get"
+            200
+            (let-values
+                (((response text)
+                  (http-get #$(simple-format
+                               #f "http://localhost:~A/" forwarded-port)
+                            #:decode-body? #t)))
+              (response-code response)))
+
+          (test-end))))
+
+  (gexp->derivation "bffe-test" test))
+
+(define %test-bffe
+  (system-test
+   (name "bffe")
+   (description "Connect to a running Build Farm Front-end.")
+   (value (run-bffe-test))))
-- 
2.41.0

Message #14 received at 65732 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Christopher Baines <mail <at> cbaines.net>
Cc: 65732 <at> debbugs.gnu.org
Subject: Re: bug#65732: [PATCH 0/2] Add the Build Farm Front-End
Date: Thu, 14 Sep 2023 16:58:46 +0200

Hello,

Christopher Baines <mail <at> cbaines.net> skribis:

> * gnu/packages/package-management.scm (bffe): New variable.


[...]

> +             ;; Guile libraries are needed here for cross-compilation.
> +             (car (assoc-ref (package-native-inputs guix) "guile"))

Rather: (lookup-package-native-input guix "guile")

Otherwise LGTM.

> +      (home-page "https://git.cbaines.net/guix/bffe")
> +      (synopsis "Build Farm Front-end for Guix")
> +      (description
> +       "The BFFE of Build Farm Front-end is an experimental frontend for Guix
> +build farms.  It works together with the Guix Data Service and Guix Build
> +Coordinator to submit builds and monitor the activity.

Looks nice!  Have you already deployed it somewhere?

Ludo’.

Message #17 received at 65732 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Christopher Baines <mail <at> cbaines.net>
Cc: 65732 <at> debbugs.gnu.org
Subject: Re: bug#65732: [PATCH 0/2] Add the Build Farm Front-End
Date: Thu, 14 Sep 2023 17:02:23 +0200

Christopher Baines <mail <at> cbaines.net> skribis:

> This is intended to replace the functionality of the Guix Build Coordinator
> queue builds script, and also provide a web interface for build farms.
>
> * gnu/services/guix.scm (<bffe-configuration>): New record type.
> (bffe-configuration, bffe-configuration?,
> bffe-configuration-package,
> bffe-configuration-user,
> bffe-configuration-group,
> bffe-configuration-arguments
> bffe-configuration-extra-environment-variables): New procedures.
> (bffe-service-type): New variable.
> * gnu/tests/guix.scm (%test-bffe): New variable.
> * doc/guix.texi (Guix Services): Document the new service.

Overall LGTM.

Nitpick: you can make lines a bit longer, for instance:

> +(define (bffe-activation config)
> +  #~(begin
> +      (use-modules (guix build utils))
> +
> +      (define %user
> +        (getpw #$(bffe-configuration-user
> +                  config)))

‘config’ can be moved to the previous line.

> +      (chown "/var/run/bffe"
> +             (passwd:uid %user)
> +             (passwd:gid %user))))

This can be a single line.

> +   (extensions
> +    (list
> +     (service-extension shepherd-root-service-type
> +                        bffe-shepherd-services)

And: (list (service-extension …

Ludo’.

Message #20 received at 65732 <at> debbugs.gnu.org (full text, mbox):

From: Christopher Baines <mail <at> cbaines.net>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 65732 <at> debbugs.gnu.org
Subject: Re: bug#65732: [PATCH 0/2] Add the Build Farm Front-End
Date: Fri, 15 Sep 2023 10:06:22 +0100

[Message part 1 (text/plain, inline)]

Ludovic Courtès <ludo <at> gnu.org> writes:

> Hello,
>
> Christopher Baines <mail <at> cbaines.net> skribis:
>
>> * gnu/packages/package-management.scm (bffe): New variable.
>
>
> [...]
>
>> +             ;; Guile libraries are needed here for cross-compilation.
>> +             (car (assoc-ref (package-native-inputs guix) "guile"))
>
> Rather: (lookup-package-native-input guix "guile")

I've actually changed this to guile-next now, as the bffe needs the new
scheme custom/soft ports that don't cause issues with fibers.

>> +      (home-page "https://git.cbaines.net/guix/bffe")
>> +      (synopsis "Build Farm Front-end for Guix")
>> +      (description
>> +       "The BFFE of Build Farm Front-end is an experimental frontend for Guix
>> +build farms.  It works together with the Guix Data Service and Guix Build
>> +Coordinator to submit builds and monitor the activity.
>
> Looks nice!  Have you already deployed it somewhere?

It's been running on bayfront for a while, and manages pages like:

  https://bordeaux.guix.gnu.org/
  https://bordeaux.guix.gnu.org/activity
  https://bordeaux.guix.gnu.org/build/21fa2ed8-b4d0-43ca-8531-e1fa33266708

As well as taking over the job of querying the data service for builds
and submitting them to the coordinator.

[signature.asc (application/pgp-signature, inline)]

Message #25 received at 65732-done <at> debbugs.gnu.org (full text, mbox):

From: Christopher Baines <mail <at> cbaines.net>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 65732-done <at> debbugs.gnu.org
Subject: Re: bug#65732: [PATCH 0/2] Add the Build Farm Front-End
Date: Fri, 15 Sep 2023 10:09:18 +0100

[Message part 1 (text/plain, inline)]

Ludovic Courtès <ludo <at> gnu.org> writes:

> Christopher Baines <mail <at> cbaines.net> skribis:
>
>> This is intended to replace the functionality of the Guix Build Coordinator
>> queue builds script, and also provide a web interface for build farms.
>>
>> * gnu/services/guix.scm (<bffe-configuration>): New record type.
>> (bffe-configuration, bffe-configuration?,
>> bffe-configuration-package,
>> bffe-configuration-user,
>> bffe-configuration-group,
>> bffe-configuration-arguments
>> bffe-configuration-extra-environment-variables): New procedures.
>> (bffe-service-type): New variable.
>> * gnu/tests/guix.scm (%test-bffe): New variable.
>> * doc/guix.texi (Guix Services): Document the new service.
>
> Overall LGTM.
>
> Nitpick: you can make lines a bit longer, for instance:
>
>> +(define (bffe-activation config)
>> +  #~(begin
>> +      (use-modules (guix build utils))
>> +
>> +      (define %user
>> +        (getpw #$(bffe-configuration-user
>> +                  config)))
>
> ‘config’ can be moved to the previous line.
>
>> +      (chown "/var/run/bffe"
>> +             (passwd:uid %user)
>> +             (passwd:gid %user))))
>
> This can be a single line.
>
>> +   (extensions
>> +    (list
>> +     (service-extension shepherd-root-service-type
>> +                        bffe-shepherd-services)
>
> And: (list (service-extension …

I've tweaked the formatting as suggested above and pushed this to master
as 82abf6ddadc6139148660440a064e60ae68f238e.

Thanks,

Chris

[signature.asc (application/pgp-signature, inline)]

Message #28 received at 65732 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Christopher Baines <mail <at> cbaines.net>
Cc: 65732 <at> debbugs.gnu.org
Subject: Re: bug#65732: [PATCH 0/2] Add the Build Farm Front-End
Date: Sun, 17 Sep 2023 12:11:49 +0200

Christopher Baines <mail <at> cbaines.net> skribis:

> It's been running on bayfront for a while, and manages pages like:
>
>   https://bordeaux.guix.gnu.org/
>   https://bordeaux.guix.gnu.org/activity
>   https://bordeaux.guix.gnu.org/build/21fa2ed8-b4d0-43ca-8531-e1fa33266708
>
> As well as taking over the job of querying the data service for builds
> and submitting them to the coordinator.

Oh, neat!

Ludo’.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.