GNU bug report logs - #36563
[PATCH] guix: Add directory to channel.

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 36563 in the body.
You can then email your comments to 36563 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Jan Nieuwenhuizen <janneke <at> gnu.org>
To: guix-patches <at> gnu.org
Subject: [PATCH] guix: Add directory to channel.
Date: Tue, 09 Jul 2019 21:26:00 +0200

[Message part 1 (text/plain, inline)]

Hi!

I commonly use a "guix/" subdirectory in upstream that contains updated
or modified packages.  The toplevel `guix.scm' then uses those package
definitions.

It would be nice if I could use that "guix/" subdirectory directly in a
channel.  This means a channel definition should look in a subdirectory
only: often an upstream source archive contains .scm files that do not
compile (guix.scm, build-aux/*.scm, etc.)

WDYT?

Greetings,
janneke

[0001-guix-Add-directory-to-channel.patch (text/x-patch, inline)]

From cf85280211ff0060b5283dc5a53cb15ee09a7998 Mon Sep 17 00:00:00 2001
From: Jan Nieuwenhuizen <janneke <at> gnu.org>
Date: Tue, 9 Jul 2019 21:01:11 +0200
Subject: [PATCH] guix: Add directory to channel.

Typical use

    (cons* (channel
            (name 'mes)
            (url "https://git.savannah.gnu.org/git/mes.git")
            (directory "guix")
            (branch "wip"))
           %default-channels)

* guix/channels.scm (<channel>): Add directory.
(read-channel-metadata): Fill directory slot.
(checkout->channel-instance): Add #:directory parameter.  Update callers.
(standard-module-derivation): Add directory parameter.  Update callers.
(build-channel-instance): Provide directory argument.
---
 guix/channels.scm | 58 ++++++++++++++++++++++++++++++-----------------
 1 file changed, 37 insertions(+), 21 deletions(-)

diff --git a/guix/channels.scm b/guix/channels.scm
index e6bb9b891b..bd64906832 100644
--- a/guix/channels.scm
+++ b/guix/channels.scm
@@ -1,6 +1,7 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2018, 2019 Ludovic Courtès <ludo <at> gnu.org>
 ;;; Copyright © 2018 Ricardo Wurmus <rekado <at> elephly.net>
+;;; Copyright © 2019 Jan (janneke) Nieuwenhuizen <janneke <at> gnu.org>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -50,6 +51,7 @@
             channel-branch
             channel-commit
             channel-location
+            channel-directory
 
             %default-channels
             guix-channel?
@@ -86,7 +88,8 @@
   (branch    channel-branch (default "master"))
   (commit    channel-commit (default #f))
   (location  channel-location
-             (default (current-source-location)) (innate)))
+             (default (current-source-location)) (innate))
+  (directory channel-directory (default #f)))
 
 (define %default-channels
   ;; Default list of channels.
@@ -141,7 +144,8 @@ file."
                         (name name)
                         (branch branch)
                         (url url)
-                        (commit (get 'commit))))))
+                        (commit (get 'commit))
+                        (directory (get 'directory))))))
                  dependencies))))))
 
 (define (channel-instance-dependencies instance)
@@ -205,13 +209,16 @@ of previously processed channels."
 
 (define* (checkout->channel-instance checkout
                                      #:key commit
-                                     (url checkout) (name 'guix))
+                                     (url checkout)
+                                     (name 'guix)
+                                     directory)
   "Return a channel instance for CHECKOUT, which is assumed to be a checkout
 of COMMIT at URL.  Use NAME as the channel name."
   (let* ((commit  (or commit (make-string 40 #\0)))
          (channel (channel (name name)
                            (commit commit)
-                           (url url))))
+                           (url url)
+                           (directory directory))))
     (channel-instance channel commit checkout)))
 
 (define %self-build-file
@@ -225,11 +232,12 @@ of COMMIT at URL.  Use NAME as the channel name."
   ;; place a set of compiled Guile modules in ~/.config/guix/latest.
   1)
 
-(define (standard-module-derivation name source core dependencies)
+(define (standard-module-derivation name source directory core dependencies)
   "Return a derivation that builds with CORE, a Guix instance, the Scheme
-modules in SOURCE and that depend on DEPENDENCIES, a list of lowerable
-objects.  The assumption is that SOURCE contains package modules to be added
-to '%package-module-path'."
+modules in SOURCE or if DIRECTORY in SOURCE/DIRECTORY and that depend on
+DEPENDENCIES, a list of lowerable objects.  The assumption is that SOURCE or
+SOURCE/DIRECTORY contains package modules to be added to
+'%package-module-path'."
   ;; FIXME: We should load, say SOURCE/.guix-channel.scm, which would allow
   ;; channel publishers to specify things such as the sub-directory where .scm
   ;; files live, files to exclude from the channel, preferred substitute URLs,
@@ -253,20 +261,27 @@ to '%package-module-path'."
             (string-append #$output "/share/guile/site/"
                            (effective-version)))
 
-          (compile-files #$source go
-                         (find-files #$source "\\.scm$"))
-          (mkdir-p (dirname scm))
-          (symlink #$source scm)
+          (let* ((subdir (if #$directory
+                             (string-append "/" #$directory)
+                             ""))
+                 (dir (string-append #$source subdir)))
+            (compile-files dir go
+                           (warn 'files (find-files dir "\\.scm$")))
+            (mkdir-p (dirname scm))
+            (symlink (string-append #$source subdir) scm))
+
           scm)))
 
   (gexp->derivation-in-inferior name build core))
 
 (define* (build-from-source name source
                             #:key core verbose? commit
-                            (dependencies '()))
+                            (dependencies '())
+                            directory)
   "Return a derivation to build Guix from SOURCE, using the self-build script
 contained therein; use COMMIT as the version string.  When CORE is true, build
-package modules under SOURCE using CORE, an instance of Guix."
+package modules under SOURCE or if DIRECTORY under SOURCE/DIRECTORY using
+CORE, an instance of Guix."
   ;; Running the self-build script makes it easier to update the build
   ;; procedure: the self-build script of the Guix-to-be-installed contains the
   ;; right dependencies, build procedure, etc., which the Guix-in-use may not
@@ -293,19 +308,20 @@ package modules under SOURCE using CORE, an instance of Guix."
                #:pull-version %pull-version))
 
       ;; Build a set of modules that extend Guix using the standard method.
-      (standard-module-derivation name source core dependencies)))
+      (standard-module-derivation name source directory core dependencies)))
 
 (define* (build-channel-instance instance
                                  #:optional core (dependencies '()))
   "Return, as a monadic value, the derivation for INSTANCE, a channel
 instance.  DEPENDENCIES is a list of extensions providing Guile modules that
 INSTANCE depends on."
-  (build-from-source (symbol->string
-                      (channel-name (channel-instance-channel instance)))
-                     (channel-instance-checkout instance)
-                     #:commit (channel-instance-commit instance)
-                     #:core core
-                     #:dependencies dependencies))
+  (let ((channel (channel-instance-channel instance)))
+    (build-from-source (symbol->string (channel-name channel))
+                       (channel-instance-checkout instance)
+                       #:commit (channel-instance-commit instance)
+                       #:core core
+                       #:dependencies dependencies
+                       #:directory (channel-directory channel))))
 
 (define (resolve-dependencies instances)
   "Return a procedure that, given one of the elements of INSTANCES, returns
-- 
2.21.0

[Message part 3 (text/plain, inline)]

-- 
Jan Nieuwenhuizen <janneke <at> gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Message #8 received at 36563 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Jan Nieuwenhuizen <janneke <at> gnu.org>
Cc: 36563 <at> debbugs.gnu.org
Subject: Re: [bug#36563] [PATCH] guix: Add directory to channel.
Date: Sat, 13 Jul 2019 12:28:41 +0200

Hello!

Jan Nieuwenhuizen <janneke <at> gnu.org> skribis:

> I commonly use a "guix/" subdirectory in upstream that contains updated
> or modified packages.  The toplevel `guix.scm' then uses those package
> definitions.
>
> It would be nice if I could use that "guix/" subdirectory directly in a
> channel.  This means a channel definition should look in a subdirectory
> only: often an upstream source archive contains .scm files that do not
> compile (guix.scm, build-aux/*.scm, etc.)

That makes a lot of sense to me.

I think that, instead of letting the user specify the right directory in
~/.config/guix/channels.scm, it should be the channel author that
specifies the sub-directory.

There’s already support for a ‘.guix-channel’ file so this is where this
‘directory’ thing could be added (actually there’s a FIXME in
channels.scm about this particular use case :-)).

WDYT?

Ludo’.

Message #11 received at 36563 <at> debbugs.gnu.org (full text, mbox):

From: Jan Nieuwenhuizen <janneke <at> gnu.org>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 36563 <at> debbugs.gnu.org
Subject: Re: [bug#36563] [PATCH] guix: Add directory to channel.
Date: Sat, 13 Jul 2019 17:37:07 +0200

[Message part 1 (text/plain, inline)]

Ludovic Courtès writes:

Hi!

> That makes a lot of sense to me.
>
> I think that, instead of letting the user specify the right directory in
> ~/.config/guix/channels.scm, it should be the channel author that
> specifies the sub-directory.
>
> There’s already support for a ‘.guix-channel’ file so this is where this
> ‘directory’ thing could be added (actually there’s a FIXME in
> channels.scm about this particular use case :-)).
>
> WDYT?

Yes, I agree that this should be the first option to support.  Attached
is a second attempt: I removed the FIXME and moved `directory' to
channel-metadata.

A downside could be that this could make a channel user dependent on
upstream to provide this `.guix-channel' file; probably best to worry
about that when the need arises :)

Greetings,
janneke

[0001-channels-Add-optional-directory-to-channel-metadata.patch (text/x-patch, inline)]

From 5d78a7eab391ea93c044d85736706b333c2dfee8 Mon Sep 17 00:00:00 2001
From: Jan Nieuwenhuizen <janneke <at> gnu.org>
Date: Sat, 13 Jul 2019 16:31:50 +0200
Subject: [PATCH] channels: Add optional directory to channel metadata.

* guix/channels.scm (<channel-metadata>): Add directory slot.  Update users.
(read-channel-metadata-from-source): New function.
(standard-module-derivation): Use it.
* doc/guix.texi (Package Modules in a Subdirectory): New subsection.
---
 doc/guix.texi     | 23 ++++++++++++
 guix/channels.scm | 93 ++++++++++++++++++++++++++---------------------
 2 files changed, 75 insertions(+), 41 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index 5a8ad7ebda..8bc00a5350 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -3921,6 +3921,29 @@ For the sake of reliability and maintainability, you should avoid dependencies
 on channels that you don't control, and you should aim to keep the number of
 dependencies to a minimum.
 
+@cindex subdirectory, channels
+@subsection Package Modules in a Subdirectory
+
+A developer may include a @file{guix} subdirectory in their upstream source
+that contains additional or modified Guix package descriptions.  Typically,
+their toplevel @file{guix.scm} then uses those package definitions:
+
+@lisp
+(define %source-dir (dirname (current-filename)))
+(add-to-load-path (string-append %source-dir "/guix"))
+(use-modules @dots{})
+@dots{}
+@end lisp
+
+By adding a meta-data file @file{.guix-channel} that specifies the
+subdirectory that contains the Guix package descriptions it can be used
+directly as a channel:
+
+@lisp
+(channel
+ (directory "guix"))
+@end lisp
+
 @subsection Replicating Guix
 
 @cindex pinning, channels
diff --git a/guix/channels.scm b/guix/channels.scm
index e6bb9b891b..614639f091 100644
--- a/guix/channels.scm
+++ b/guix/channels.scm
@@ -1,6 +1,7 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2018, 2019 Ludovic Courtès <ludo <at> gnu.org>
 ;;; Copyright © 2018 Ricardo Wurmus <rekado <at> elephly.net>
+;;; Copyright © 2019 Jan (janneke) Nieuwenhuizen <janneke <at> gnu.org>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -107,9 +108,10 @@
   (checkout  channel-instance-checkout))
 
 (define-record-type <channel-metadata>
-  (channel-metadata version dependencies)
+  (channel-metadata version directory dependencies)
   channel-metadata?
   (version       channel-metadata-version)
+  (directory     channel-metadata-directory)
   (dependencies  channel-metadata-dependencies))
 
 (define (channel-reference channel)
@@ -119,18 +121,18 @@
     (#f      `(branch . ,(channel-branch channel)))
     (commit  `(commit . ,(channel-commit channel)))))
 
-(define (read-channel-metadata instance)
-  "Return a channel-metadata record read from the channel INSTANCE's
-description file, or return #F if the channel instance does not include the
-file."
-  (let* ((source (channel-instance-checkout instance))
-         (meta-file (string-append source "/.guix-channel")))
+(define (read-channel-metadata-from-source source)
+  "Return a channel-metadata record read from channel's SOURCE/.guix-channel
+description file, or return #F if SOURCE/.guix-channel does not exist."
+  (let ((meta-file (string-append source "/.guix-channel")))
     (and (file-exists? meta-file)
-         (and-let* ((raw (call-with-input-file meta-file read))
-                    (version (and=> (assoc-ref raw 'version) first))
-                    (dependencies (or (assoc-ref raw 'dependencies) '())))
+         (let* ((raw (call-with-input-file meta-file read))
+                (version (and=> (assoc-ref raw 'version) first))
+                (directory (and=> (assoc-ref raw 'directory) first))
+                (dependencies (or (assoc-ref raw 'dependencies) '())))
            (channel-metadata
             version
+            directory
             (map (lambda (item)
                    (let ((get (lambda* (key #:optional default)
                                 (or (and=> (assoc-ref item key) first) default))))
@@ -144,12 +146,18 @@ file."
                         (commit (get 'commit))))))
                  dependencies))))))
 
+(define (read-channel-metadata instance)
+  "Return a channel-metadata record read from the channel INSTANCE's
+description file, or return #F if the channel instance does not include the
+file."
+  (read-channel-metadata-from-source (channel-instance-checkout instance)))
+
 (define (channel-instance-dependencies instance)
   "Return the list of channels that are declared as dependencies for the given
 channel INSTANCE."
   (match (read-channel-metadata instance)
     (#f '())
-    (($ <channel-metadata> version dependencies)
+    (($ <channel-metadata> version directory dependencies)
      dependencies)))
 
 (define* (latest-channel-instances store channels #:optional (previous-channels '()))
@@ -230,36 +238,39 @@ of COMMIT at URL.  Use NAME as the channel name."
 modules in SOURCE and that depend on DEPENDENCIES, a list of lowerable
 objects.  The assumption is that SOURCE contains package modules to be added
 to '%package-module-path'."
-  ;; FIXME: We should load, say SOURCE/.guix-channel.scm, which would allow
-  ;; channel publishers to specify things such as the sub-directory where .scm
-  ;; files live, files to exclude from the channel, preferred substitute URLs,
-  ;; etc.
-
-  (define build
-    ;; This is code that we'll run in CORE, a Guix instance, with its own
-    ;; modules and so on.  That way, we make sure these modules are built for
-    ;; the right Guile version, with the right dependencies, and that they get
-    ;; to see the right (gnu packages …) modules.
-    (with-extensions dependencies
-      #~(begin
-          (use-modules (guix build compile)
-                       (guix build utils)
-                       (srfi srfi-26))
-
-          (define go
-            (string-append #$output "/lib/guile/" (effective-version)
-                           "/site-ccache"))
-          (define scm
-            (string-append #$output "/share/guile/site/"
-                           (effective-version)))
-
-          (compile-files #$source go
-                         (find-files #$source "\\.scm$"))
-          (mkdir-p (dirname scm))
-          (symlink #$source scm)
-          scm)))
-
-  (gexp->derivation-in-inferior name build core))
+
+  (let* ((metadata (read-channel-metadata-from-source source))
+         (directory (and=> metadata channel-metadata-directory)))
+
+    (define build
+      ;; This is code that we'll run in CORE, a Guix instance, with its own
+      ;; modules and so on.  That way, we make sure these modules are built for
+      ;; the right Guile version, with the right dependencies, and that they get
+      ;; to see the right (gnu packages …) modules.
+      (with-extensions dependencies
+        #~(begin
+            (use-modules (guix build compile)
+                         (guix build utils)
+                         (srfi srfi-26))
+
+            (define go
+              (string-append #$output "/lib/guile/" (effective-version)
+                             "/site-ccache"))
+            (define scm
+              (string-append #$output "/share/guile/site/"
+                             (effective-version)))
+
+            (let* ((subdir (if #$directory
+                               (string-append "/" #$directory)
+                               ""))
+                   (dir (string-append #$source subdir)))
+              (compile-files dir go (find-files dir "\\.scm$"))
+              (mkdir-p (dirname scm))
+              (symlink (string-append #$source subdir) scm))
+
+            scm)))
+
+    (gexp->derivation-in-inferior name build core)))
 
 (define* (build-from-source name source
                             #:key core verbose? commit
-- 
2.21.0

[Message part 3 (text/plain, inline)]

-- 
Jan Nieuwenhuizen <janneke <at> gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Message #14 received at 36563 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Jan Nieuwenhuizen <janneke <at> gnu.org>
Cc: 36563 <at> debbugs.gnu.org
Subject: Re: [bug#36563] [PATCH] guix: Add directory to channel.
Date: Sat, 13 Jul 2019 22:49:36 +0200

Jan Nieuwenhuizen <janneke <at> gnu.org> skribis:

> Yes, I agree that this should be the first option to support.  Attached
> is a second attempt: I removed the FIXME and moved `directory' to
> channel-metadata.

That was fast!

> A downside could be that this could make a channel user dependent on
> upstream to provide this `.guix-channel' file; probably best to worry
> about that when the need arises :)

If upstream doesn’t provide this file, then the channel is effectively
unusable because of a file name/module name mismatch, no?

> From 5d78a7eab391ea93c044d85736706b333c2dfee8 Mon Sep 17 00:00:00 2001
> From: Jan Nieuwenhuizen <janneke <at> gnu.org>
> Date: Sat, 13 Jul 2019 16:31:50 +0200
> Subject: [PATCH] channels: Add optional directory to channel metadata.
>
> * guix/channels.scm (<channel-metadata>): Add directory slot.  Update users.
> (read-channel-metadata-from-source): New function.
> (standard-module-derivation): Use it.
> * doc/guix.texi (Package Modules in a Subdirectory): New subsection.

[...]

> +@cindex subdirectory, channels
> +@subsection Package Modules in a Subdirectory
> +
> +A developer may include a @file{guix} subdirectory in their upstream source
> +that contains additional or modified Guix package descriptions.  Typically,
> +their toplevel @file{guix.scm} then uses those package definitions:

What about framing it along these lines:

  As a channel author, you may want to keep your channel modules in a
  sub-directory.  In that case, blah…

?

> +By adding a meta-data file @file{.guix-channel} that specifies the
> +subdirectory that contains the Guix package descriptions it can be used
> +directly as a channel:
> +
> +@lisp
> +(channel
> + (directory "guix"))

It should be:

  (channel
    (version 0)
    (directory "something"))

(‘read-channel-metadata’ currently ignores ‘version’, but that’s a bug
that we should fix afterwards, IMO.)

> +            (let* ((subdir (if #$directory
> +                               (string-append "/" #$directory)
> +                               ""))
> +                   (dir (string-append #$source subdir)))

Maybe s/dir/source/?

Otherwise LGTM!

Thanks,
Ludo’.

Message #19 received at 36563-done <at> debbugs.gnu.org (full text, mbox):

From: Jan Nieuwenhuizen <janneke <at> gnu.org>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 36563-done <at> debbugs.gnu.org
Subject: Re: [bug#36563] [PATCH] guix: Add directory to channel.
Date: Sun, 14 Jul 2019 01:09:49 +0200

Ludovic Courtès writes:

Hi!

> What about framing it along these lines:
>
>   As a channel author, you may want to keep your channel modules in a
>   sub-directory.  In that case, blah…
>
> ?

Better, changed it to

--8<---------------cut here---------------start------------->8---
@cindex subdirectory, channels
@subsection Package Modules in a Sub-directory

As a channel author, you may want to keep your channel modules in a
sub-directory.  If your modules are in the sub-directory @file{guix}, you must
add a meta-data file @file{.guix-channel} that contains:

@lisp
(channel
  (version 0)
  (directory "guix"))
@end lisp
--8<---------------cut here---------------end--------------->8---

> (‘read-channel-metadata’ currently ignores ‘version’, but that’s a bug
> that we should fix afterwards, IMO.)

OK.

>> +            (let* ((subdir (if #$directory
>> +                               (string-append "/" #$directory)
>> +                               ""))
>> +                   (dir (string-append #$source subdir)))
>
> Maybe s/dir/source/?

Sure, done.

> Otherwise LGTM!

Great, thanks; pushed to master as 53f21642729e4786141c072dd835b04cb85dfe28

Greetings,
janneke

-- 
Jan Nieuwenhuizen <janneke <at> gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Message #22 received at 36563-done <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Jan Nieuwenhuizen <janneke <at> gnu.org>
Cc: 36563-done <at> debbugs.gnu.org
Subject: Re: [bug#36563] [PATCH] guix: Add directory to channel.
Date: Sun, 14 Jul 2019 15:24:40 +0200

Hi,

Jan Nieuwenhuizen <janneke <at> gnu.org> skribis:

> Great, thanks; pushed to master as 53f21642729e4786141c072dd835b04cb85dfe28

Awesome, thank you!

Ludo’.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.