GNU bug report logs - #33522
[PATCH 0/3] Add the '--with-branch' package transformation option

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 33522 in the body.
You can then email your comments to 33522 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: guix-patches <at> gnu.org
Cc: Ludovic Courtès <ludo <at> gnu.org>
Subject: [PATCH 0/3] Add the '--with-branch' package transformation option
Date: Tue, 27 Nov 2018 16:31:47 +0100

Hello,

This patch series adds a ‘git-checkout’ file-like construct along with
a ‘--with-branch’ package transformation option.

The ‘git-checkout’ construct may be used like this:

  (define-public guile-git
    (package
      (inherit guile-next)
      (version "git")
      (source (git-checkout
               (url "https://git.savannah.gnu.org/git/guile.git")
               (branch "master")))))

Then you can do this:

--8<---------------cut here---------------start------------->8---
$ guix build -L ~/.config/guix/personal/ guile-next <at> git
updating checkout of 'https://git.savannah.gnu.org/git/guile.git'...
retrieved commit 1f6786db06fee5f0bbdbd637acc6aa46aecd4169
building /gnu/store/7l168l4lgpjjg9jq7lh2416cc884r3s9-guile-next-git.drv...
[…]
--8<---------------cut here---------------end--------------->8---

IOW, every time you build the package, it gets built off the latest
commit of the specified branch.

The next patch add ‘--with-branch’ which makes that functionality
available from the command line.  For example:

  guix build --with-branch=guile-gcrypt=master cuirass

builds the latest guile-gcrypt, guix, and cuirass, using the latest
guile-gcrypt commit on ‘master’.

The ‘--with-branch’ option only works with packages that have a
‘git-fetch’ origin or a ‘git-checkout’ as their source.

I find it rather neat.  WDYT?  :-)

Ludo’.

Ludovic Courtès (3):
  git: 'latest-repository-commit' logs its progress.
  git: Add <git-checkout> record type.
  guix build: Add '--with-branch' transformation option.

 Makefile.am                |  1 +
 doc/guix.texi              | 27 ++++++++++++++++++
 guix/git.scm               | 41 ++++++++++++++++++++++++++--
 guix/scripts/build.scm     | 56 ++++++++++++++++++++++++++++++++++++--
 tests/guix-build-branch.sh | 48 ++++++++++++++++++++++++++++++++
 5 files changed, 168 insertions(+), 5 deletions(-)
 create mode 100644 tests/guix-build-branch.sh

-- 
2.19.2

Message #8 received at 33522 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: 33522 <at> debbugs.gnu.org
Cc: Ludovic Courtès <ludovic.courtes <at> inria.fr>
Subject: [PATCH 1/3] git: 'latest-repository-commit' logs its progress.
Date: Tue, 27 Nov 2018 16:34:00 +0100

From: Ludovic Courtès <ludovic.courtes <at> inria.fr>

* guix/git.scm (latest-repository-commit): Add #:log-port and honor it.
---
 guix/git.scm | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/guix/git.scm b/guix/git.scm
index d007916662..78645024c9 100644
--- a/guix/git.scm
+++ b/guix/git.scm
@@ -154,6 +154,7 @@ data, respectively [<branch name> | <sha1> | <tag name>]."
 
 (define* (latest-repository-commit store url
                                    #:key
+                                   (log-port (%make-void-port "w"))
                                    (cache-directory
                                     (%repository-cache-directory))
                                    (ref '(branch . "master")))
@@ -164,11 +165,14 @@ REF is pair whose key is [branch | commit | tag] and value the associated
 data, respectively [<branch name> | <sha1> | <tag name>].
 
 Git repositories are kept in the cache directory specified by
-%repository-cache-directory parameter."
+%repository-cache-directory parameter.
+
+Log progress and checkout info to LOG-PORT."
   (define (dot-git? file stat)
     (and (string=? (basename file) ".git")
          (eq? 'directory (stat:type stat))))
 
+  (format log-port "updating checkout of '~a'...~%" url)
   (let*-values
       (((checkout commit)
         (update-cached-checkout url
@@ -177,6 +181,7 @@ Git repositories are kept in the cache directory specified by
                                 (url-cache-directory url cache-directory)))
        ((name)
         (url+commit->name url commit)))
+    (format log-port "retrieved commit ~a~%" commit)
     (values (add-to-store store name #t "sha256" checkout
                           #:select? (negate dot-git?))
             commit)))
-- 
2.19.2

Message #11 received at 33522 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: 33522 <at> debbugs.gnu.org
Cc: Ludovic Courtès <ludovic.courtes <at> inria.fr>
Subject: [PATCH 2/3] git: Add <git-checkout> record type.
Date: Tue, 27 Nov 2018 16:34:01 +0100

From: Ludovic Courtès <ludovic.courtes <at> inria.fr>

* guix/git.scm (<git-checkout>): New record type.
(latest-repository-commit*): New procedure.
(git-checkout-compiler): New gexp compiler.
---
 guix/git.scm | 34 +++++++++++++++++++++++++++++++++-
 1 file changed, 33 insertions(+), 1 deletion(-)

diff --git a/guix/git.scm b/guix/git.scm
index 78645024c9..56cebb06ed 100644
--- a/guix/git.scm
+++ b/guix/git.scm
@@ -25,6 +25,8 @@
   #:use-module ((guix build utils) #:select (mkdir-p))
   #:use-module (guix store)
   #:use-module (guix utils)
+  #:use-module (guix records)
+  #:use-module (guix gexp)
   #:use-module (rnrs bytevectors)
   #:use-module (ice-9 match)
   #:use-module (srfi srfi-1)
@@ -33,7 +35,12 @@
   #:use-module (srfi srfi-35)
   #:export (%repository-cache-directory
             update-cached-checkout
-            latest-repository-commit))
+            latest-repository-commit
+
+            git-checkout
+            git-checkout?
+            git-checkout-url
+            git-checkout-branch))
 
 (define %repository-cache-directory
   (make-parameter (string-append (cache-directory #:ensure? #f)
@@ -185,3 +192,28 @@ Log progress and checkout info to LOG-PORT."
     (values (add-to-store store name #t "sha256" checkout
                           #:select? (negate dot-git?))
             commit)))
+
+
+;;;
+;;; Checkouts.
+;;;
+
+;; Representation of the "latest" checkout of a branch.
+(define-record-type* <git-checkout>
+  git-checkout make-git-checkout
+  git-checkout?
+  (url     git-checkout-url)
+  (branch  git-checkout-branch (default "master")))
+
+(define latest-repository-commit*
+  (store-lift latest-repository-commit))
+
+(define-gexp-compiler (git-checkout-compiler (checkout <git-checkout>)
+                                             system target)
+  ;; "Compile" CHECKOUT by updating the local checkout and adding it to the
+  ;; store.
+  (match checkout
+    (($ <git-checkout> url branch)
+     (latest-repository-commit* url
+                                #:ref `(branch . ,branch)
+                                #:log-port (current-error-port)))))
-- 
2.19.2

Message #14 received at 33522 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: 33522 <at> debbugs.gnu.org
Cc: Ludovic Courtès <ludovic.courtes <at> inria.fr>
Subject: [PATCH 3/3] guix build: Add '--with-branch' transformation option.
Date: Tue, 27 Nov 2018 16:34:02 +0100

From: Ludovic Courtès <ludovic.courtes <at> inria.fr>

* guix/scripts/build.scm (evaluate-git-replacement-specs)
(transform-package-source-branch): New procedures.
(%transformations, %transformation-options): Add 'with-branch'.
(show-transformation-options-help): Likewise.
* tests/guix-build-branch.sh: New file.
* Makefile.am (SH_TESTS): Add it.
* doc/guix.texi (Package Transformation Options): Document it.
---
 Makefile.am                |  1 +
 doc/guix.texi              | 27 ++++++++++++++++++
 guix/scripts/build.scm     | 56 ++++++++++++++++++++++++++++++++++++--
 tests/guix-build-branch.sh | 48 ++++++++++++++++++++++++++++++++
 4 files changed, 129 insertions(+), 3 deletions(-)
 create mode 100644 tests/guix-build-branch.sh

diff --git a/Makefile.am b/Makefile.am
index c5676b0b94..4f134ab174 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -407,6 +407,7 @@ endif
 
 SH_TESTS =					\
   tests/guix-build.sh				\
+  tests/guix-build-branch.sh			\
   tests/guix-download.sh			\
   tests/guix-gc.sh				\
   tests/guix-hash.sh				\
diff --git a/doc/guix.texi b/doc/guix.texi
index 917a3e9d57..18bc3508c1 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -6447,6 +6447,33 @@ must be compatible.  If @var{replacement} is somehow incompatible with
 @var{package}, then the resulting package may be unusable.  Use with
 care!
 
+@item --with-branch=@var{package}=@var{branch}
+@cindex Git, using the latest commit
+@cindex latest commit, building
+Build @var{package} from the latest commit of @var{branch}.  The @code{source}
+field of @var{package} must be an origin with the @code{git-fetch} method
+(@pxref{origin Reference}) or a @code{git-checkout} object; the repository URL
+is taken from that @code{source}.
+
+For instance, the following command builds @code{guile-sqlite3} from the
+latest commit of its @code{master} branch, and then builds @code{guix} (which
+depends on it) and @code{cuirass} (which depends on @code{guix}) against this
+specific @code{guile-sqlite3} build:
+
+@example
+guix build --with-branch=guile-sqlite3=master cuirass
+@end example
+
+@cindex continuous integration
+Obviously, since it uses the latest commit of the given branch, the result of
+such a command varies over time.  Nevertheless it is a convenient way to
+rebuild entire software stacks against the latest commit of one or more
+packages.  This is particularly useful in the context of continuous
+integration (CI).
+
+Checkouts are kept in a cache under @file{~/.cache/guix/checkouts} to speed up
+consecutive accesses to the same repository.  You may want to clean it up once
+in a while to save disk space.
 @end table
 
 @node Additional Build Options
diff --git a/guix/scripts/build.scm b/guix/scripts/build.scm
index 13978abb77..e8f2fe973d 100644
--- a/guix/scripts/build.scm
+++ b/guix/scripts/build.scm
@@ -45,6 +45,8 @@
   #:use-module (srfi srfi-37)
   #:autoload   (gnu packages) (specification->package %package-module-path)
   #:autoload   (guix download) (download-to-store)
+  #:autoload   (guix git-download) (git-reference?)
+  #:autoload   (guix git) (git-checkout?)
   #:use-module (guix status)
   #:use-module ((guix progress) #:select (current-terminal-columns))
   #:use-module ((guix build syscalls) #:select (terminal-columns))
@@ -270,6 +272,48 @@ current 'gnutls' package, after which version 3.5.4 is grafted onto them."
           (rewrite obj)
           obj))))
 
+(define (evaluate-git-replacement-specs specs)
+  "Parse SPECS, a list of strings like \"guile=stable-2.2\", and return a list
+of package pairs.  Raise an error if an element of SPECS uses invalid syntax,
+or if a package it refers to could not be found."
+  (define not-equal
+    (char-set-complement (char-set #\=)))
+
+  (map (lambda (spec)
+         (match (string-tokenize spec not-equal)
+           ((name branch)
+            (let* ((old    (specification->package name))
+                   (source (package-source old))
+                   (url    (cond ((and (origin? source)
+                                       (git-reference? (origin-uri source)))
+                                  (git-reference-url (origin-uri source)))
+                                 ((git-checkout? source)
+                                  (git-checkout-url source))
+                                 (else
+                                  (leave (G_ "the source of ~a is not a Git \
+reference~%")
+                                         (package-full-name old))))))
+              (cons old
+                    (package
+                      (inherit old)
+                      (version (string-append "git." branch))
+                      (source (git-checkout (url url) (branch branch)))))))
+           (x
+            (leave (G_ "invalid replacement specification: ~s~%") spec))))
+       specs))
+
+(define (transform-package-source-branch replacement-specs)
+  "Return a procedure that, when passed a package, replaces its direct
+dependencies according to REPLACEMENT-SPECS.  REPLACEMENT-SPECS is a list of
+strings like \"guile-next=stable-3.0\" meaning that packages are built using
+'guile-next' from the latest commit on its 'stable-3.0' branch."
+  (let* ((replacements (evaluate-git-replacement-specs replacement-specs))
+         (rewrite      (package-input-rewriting replacements)))
+    (lambda (store obj)
+      (if (package? obj)
+          (rewrite obj)
+          obj))))
+
 (define %transformations
   ;; Transformations that can be applied to things to build.  The car is the
   ;; key used in the option alist, and the cdr is the transformation
@@ -277,7 +321,8 @@ current 'gnutls' package, after which version 3.5.4 is grafted onto them."
   ;; things to build.
   `((with-source . ,transform-package-source)
     (with-input  . ,transform-package-inputs)
-    (with-graft  . ,transform-package-inputs/graft)))
+    (with-graft  . ,transform-package-inputs/graft)
+    (with-branch . ,transform-package-source-branch)))
 
 (define %transformation-options
   ;; The command-line interface to the above transformations.
@@ -291,7 +336,9 @@ current 'gnutls' package, after which version 3.5.4 is grafted onto them."
           (option '("with-input") #t #f
                   (parser 'with-input))
           (option '("with-graft") #t #f
-                  (parser 'with-graft)))))
+                  (parser 'with-graft))
+          (option '("with-branch") #t #f
+                  (parser 'with-branch)))))
 
 (define (show-transformation-options-help)
   (display (G_ "
@@ -302,7 +349,10 @@ current 'gnutls' package, after which version 3.5.4 is grafted onto them."
                          replace dependency PACKAGE by REPLACEMENT"))
   (display (G_ "
       --with-graft=PACKAGE=REPLACEMENT
-                         graft REPLACEMENT on packages that refer to PACKAGE")))
+                         graft REPLACEMENT on packages that refer to PACKAGE"))
+    (display (G_ "
+      --with-branch=PACKAGE=BRANCH
+                         build PACKAGE from the latest commit of BRANCH")))
 
 
 (define (options->transformation opts)
diff --git a/tests/guix-build-branch.sh b/tests/guix-build-branch.sh
new file mode 100644
index 0000000000..bc50d9c0ef
--- /dev/null
+++ b/tests/guix-build-branch.sh
@@ -0,0 +1,48 @@
+# GNU Guix --- Functional package management for GNU
+# Copyright © 2018 Ludovic Courtès <ludo <at> gnu.org>
+#
+# This file is part of GNU Guix.
+#
+# GNU Guix is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or (at
+# your option) any later version.
+#
+# GNU Guix is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+#
+# Test 'guix build --with-branch'.
+#
+
+guix build --version
+
+# 'guix build --with-branch' requires access to the network to clone the
+# Git repository below.
+
+if ! guile -c '(getaddrinfo "www.gnu.org" "80" AI_NUMERICSERV)' 2> /dev/null
+then
+    # Skipping.
+    exit 77
+fi
+
+orig_drv="`guix build guile-gcrypt -d`"
+latest_drv="`guix build guile-gcrypt --with-branch=guile-gcrypt=master -d`"
+test -n "$latest_drv"
+test "$orig_drv" != "$latest_drv"
+
+# FIXME: '-S' currently doesn't work with non-derivation source.
+# checkout="`guix build guile-gcrypt --with-branch=guile-gcrypt=master -S`"
+checkout="`guix gc --references "$latest_drv" | grep guile-gcrypt | grep -v -E '(-builder|\.drv)'`"
+test -d "$checkout"
+test -f "$checkout/COPYING"
+
+orig_drv="`guix build guix -d`"
+latest_drv="`guix build guix --with-branch=guile-gcrypt=master -d`"
+guix gc -R "$latest_drv" | grep guile-gcrypt-git.master
+test "$orig_drv" != "$latest_drv"
-- 
2.19.2

Message #19 received at 33522-done <at> debbugs.gnu.org (full text, mbox):

From: ludovic.courtes <at> inria.fr (Ludovic Courtès)
To: 33522-done <at> debbugs.gnu.org
Subject: Re: [bug#33522] [PATCH 0/3] Add the '--with-branch' package
 transformation option
Date: Fri, 30 Nov 2018 17:04:48 +0100

Hi there!

Ludovic Courtès <ludo <at> gnu.org> skribis:

> Ludovic Courtès (3):
>   git: 'latest-repository-commit' logs its progress.
>   git: Add <git-checkout> record type.
>   guix build: Add '--with-branch' transformation option.

I pushed these 3 commits and added a 4th one that implements
‘--with-commit’ along the same lines.

Enjoy!

Ludo’.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.