Package: guix-patches;
Reported by: Jørgen Kvalsvik <j <at> lambda.is>
Date: Tue, 13 May 2025 09:41:02 UTC
Severity: normal
Tags: patch
To reply to this bug, email your comments to 78404 AT debbugs.gnu.org.
Toggle the display of automated, internal messages from the tracker.
View this report as an mbox folder, status mbox, maintainer mbox
guix-patches <at> gnu.org:bug#78404; Package guix-patches.
(Tue, 13 May 2025 09:41:02 GMT) Full text and rfc822 format available.Jørgen Kvalsvik <j <at> lambda.is>:guix-patches <at> gnu.org.
(Tue, 13 May 2025 09:41:02 GMT) Full text and rfc822 format available.Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):
From: Jørgen Kvalsvik <j <at> lambda.is> To: guix-patches <at> gnu.org Cc: Jørgen Kvalsvik <j <at> lambda.is>, steve <at> futurile.net Subject: [PATCH 0/2] Go: Module aware build system Date: Tue, 13 May 2025 11:40:23 +0200
Hi,
This is a new build system for go, based on go modules. I'm submitting
this now to get an issue number, collect some feedback, and iron out the
last few wrinkles, but we have used an internal version of it for a
while and it works well for us. I have tweaked it a bit in preparation
for this patch series, and I do expect it to take a couple of
iterations.
The new build system can use packages built with go-build-system as
inputs, which I think is a prerequisite. There is already a large
collection of go libraries in guix, and we want to use that work
well. That being said, this build system should graudually replace the
go-build-system. Because they are (largely) compatible as inputs,
packages can be migrated gradually. The library or program being built does
not have to be "module aware" (have a go.mod file) to use this build system, a
new go.mod will be created unconditionally.
It addresses most of the shortcomings of the current go build
system. Here's the list from go.scm and how this system is different:
* Avoid copying dependencies into the build environment and / or avoid
using a tmpdir when creating the inputs union.
We still copy build dependencies into the build dir, so that go
build can "fetch" dependencies as it likes. There is a path where
this can be avoided by using replace directives in go.mod, but this
is a significant complication and requires build inputs to be proper
modules, which is not guaranteed.
* Use Go modules
Yes.
* Re-use compiled packages
Yes. We install the go build cache under $out/var, and seed
downstream builds with this cache.
* Avoid the go-inputs hack
Sort-of? We look at package name, but also the "shape" of the
package, namely the presence of the src/ directory. This seems to
work ok.
* Remove module packages, only offering the full Git repos? This is more
idiomatic, I think, because Go downloads Git repos, not modules.
There is the go-mod-fetch which downloads modules (as zips). The
build system does not particularly care about the source, and works
well both with git clone, hg clone, and url fetch.
* Figurie out how to passthrough --verbosity option to "build" and
"check"
Not addressed yet.
* Implement test-backend option, which would be similar to pyproject's
one, allowing to provide custom test runner.
Not really, but most go projects are just tested with `go test
./...` or `go test ./dir1 ./dir2 ...`. There are options for both
test flags and test targets. Anything else probably warrants a
custom check phase.
I have a prototype of guix import go-module ..., too, but it's not quite
ready yet. I will add onto this series a few packages to demonstrate the
build system, and port a few packages from go-build-system to
go-module-build-system.
Jørgen Kvalsvik (2):
guix: Add downloader for go modules from GOPROXY
guix: Add module-aware build system for go
Makefile.am | 3 +
guix/build-system/go-module.scm | 268 +++++++++++++++
guix/build/go-module-build-system.scm | 457 ++++++++++++++++++++++++++
guix/go-mod-download.scm | 126 +++++++
4 files changed, 854 insertions(+)
create mode 100644 guix/build-system/go-module.scm
create mode 100644 guix/build/go-module-build-system.scm
create mode 100644 guix/go-mod-download.scm
--
2.39.5
guix-patches <at> gnu.org:bug#78404; Package guix-patches.
(Tue, 13 May 2025 09:56:02 GMT) Full text and rfc822 format available.Message #8 received at 78404 <at> debbugs.gnu.org (full text, mbox):
From: Jørgen Kvalsvik <j <at> lambda.is> To: 78404 <at> debbugs.gnu.org Cc: Jørgen Kvalsvik <j <at> lambda.is>, steve <at> futurile.net Subject: [PATCH 2/2] guix: Add module-aware build system for go Date: Tue, 13 May 2025 11:55:22 +0200
Add a new build system for go, using go modules. This build system is partly
compatible with go-build-system; they can both be used as build inputs to each
other, but their options are incompatible.
The main departure from go-build-system is that go-build-system tries to build
a workspace [1], where go-module-build-system builds a goproxy + go.mod file
and lets `go build` do what it wants to.
Most go libraries should be straight forward to build. For example, this is
the package definition for golang.org/x/sync <at> 0.12:
(define-public go-golang-org-x-sync
(package
(name "go-golang-org-x-sync")
(version "0.12.0")
(source
(origin
(method go-mod-fetch)
(uri (go-mod-reference
(path "golang.org/x/sync")
(version version)))
(sha256
(base32 "00pd84ah4xd5sjax8rxv98xbnwrvkk8clazl3kq1xrbkmvjq2m53"))))
(build-system go-module-build-system)
(home-page "https://golang.org/x/sync")
(synopsis "Go Sync")
(description "This repository provides Go concurrency primitives
in addition to the ones provided by the language and \"sync\" and
\"sync/atomic\" packages.")
(license license:bsd-3)))
The build system also supports higher resolution build-, test-, and install
targets, re-use of compiled files, and options for common build tweaks.
[1] <https://golang.org/doc/code.html#Workspaces>
* guix/build/go-module-build-system.scm: New file.
* guix/build-system/go-module.scm: New file.
* Makefile.am (MODULES): Add them.
Change-Id: I47a028ab8f95fd3a338036480dbad6677e9c50a5
---
Makefile.am | 2 +
guix/build-system/go-module.scm | 268 +++++++++++++++
guix/build/go-module-build-system.scm | 459 ++++++++++++++++++++++++++
3 files changed, 729 insertions(+)
create mode 100644 guix/build-system/go-module.scm
create mode 100644 guix/build/go-module-build-system.scm
diff --git a/Makefile.am b/Makefile.am
index b5fb81f412..12446e6bb4 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -168,6 +168,7 @@ MODULES = \
guix/build-system/glib-or-gtk.scm \
guix/build-system/gnu.scm \
guix/build-system/go.scm \
+ guix/build-system/go-module.scm \
guix/build-system/guile.scm \
guix/build-system/haskell.scm \
guix/build-system/julia.scm \
@@ -227,6 +228,7 @@ MODULES = \
guix/build/minify-build-system.scm \
guix/build/font-build-system.scm \
guix/build/go-build-system.scm \
+ guix/build/go-module-build-system.scm \
guix/build/android-repo.scm \
guix/build/asdf-build-system.scm \
guix/build/bzr.scm \
diff --git a/guix/build-system/go-module.scm b/guix/build-system/go-module.scm
new file mode 100644
index 0000000000..5692e318d3
--- /dev/null
+++ b/guix/build-system/go-module.scm
@@ -0,0 +1,268 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2025 Jørgen Kvalsvik <j <at> lambda.is>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix build-system go-module)
+ #:use-module (guix gexp)
+ #:use-module (guix monads)
+ #:use-module (guix packages)
+ #:use-module (guix store)
+ #:use-module (guix utils)
+ #:use-module (guix search-paths)
+ #:use-module (guix build-system)
+ #:use-module (guix build-system gnu)
+ #:use-module ((guix build-system go) #:prefix go-build:)
+ #:use-module (srfi srfi-1)
+ #:export (%go-module-build-system-modules
+ go-module-build
+ go-module-build-system))
+
+;;; Commentary:
+;;;
+;;; Build procedure for packages using the module aware Go build system.
+;;;
+;;; Code:
+
+(define %go-module-build-system-modules
+ ;; Build-side modules imported by default.
+ `((guix build go-module-build-system)
+ (guix build union)
+ ,@%default-gnu-imported-modules))
+
+(define (default-go)
+ ;; Lazily resolve the binding to avoid a circular dependency.
+ (let ((go (resolve-interface '(gnu packages golang))))
+ (module-ref go 'go)))
+
+(define (default-gccgo)
+ ;; Lazily resolve the binding to avoid a circular dependency.
+ (let ((gcc (resolve-interface '(gnu packages gcc))))
+ (module-ref gcc 'gccgo-12)))
+
+(define (default-zip)
+ "Return the 'zip' package. This is a lazy reference so that we don't
+depend on (gnu packages compression)."
+ (let ((distro (resolve-interface '(gnu packages compression))))
+ (module-ref distro 'zip)))
+
+(define* (lower name
+ #:key source inputs native-inputs outputs system target
+ (go (if (supported-package? (default-go))
+ (default-go)
+ (default-gccgo)))
+ (zip (default-zip))
+ #:allow-other-keys
+ #:rest arguments)
+ "Return a bag for NAME."
+ (define private-keywords
+ '(#:target #:inputs #:native-inputs #:go #:zip))
+
+ (bag
+ (name name)
+ (system system)
+ (target target)
+ (build-inputs `(,@(if source
+ `(("source" ,source))
+ '())
+ ,@`(("go" ,go) ("zip" ,zip))
+ ,@inputs
+ ,@native-inputs
+ ,@(if target (standard-cross-packages target 'host) '())
+ ;; Keep the standard inputs of 'gnu-build-system'.
+ ,@(standard-packages)))
+ (target-inputs (if target (standard-cross-packages target 'target) '()))
+ (outputs outputs)
+ (build (if target go-cross-module-build go-module-build))
+ (arguments (strip-keyword-arguments private-keywords arguments))))
+
+(define* (go-module-build name inputs
+ #:key
+ source
+ (phases '%standard-phases)
+ (outputs '("out"))
+ (search-paths '())
+ (go-flags '())
+ (ld-flags '("-s" "-w"))
+ (tags '())
+ (build-targets '("./..."))
+ (test-targets '("./..."))
+ (install-targets '())
+ (test-flags '())
+ (module-path #f)
+ (trimpath? #t)
+ (cgo? #f)
+ (tests? #t)
+ (build-output-dir? #f)
+ (skip-build? #f)
+ (install-source? #t)
+ (install-cache? #t)
+ (parallel-build? #t)
+ (parallel-tests? #t)
+ (environment-variables '())
+ (system (%current-system))
+ (goarch #f)
+ (goos #f)
+ (guile #f)
+ (substitutable? #t)
+ (imported-modules %go-module-build-system-modules)
+ (modules '((guix build go-module-build-system)
+ (guix build utils))))
+
+ (define builder
+ (with-imported-modules
+ imported-modules
+ #~(begin
+ (use-modules #$@(sexp->gexp modules))
+ (go-module-build #:name #$name
+ #:source #+source
+ #:system #$system
+ #:go-flags '#$go-flags
+ #:ld-flags '#$ld-flags
+ #:tags '#$tags
+ #:build-targets '#$build-targets
+ #:test-targets '#$test-targets
+ #:install-targets '#$install-targets
+ #:test-flags '#$test-flags
+ #:module-path '#$module-path
+ #:trimpath? #$trimpath?
+ #:cgo? '#$cgo?
+ #:tests? #$tests?
+ #:build-output-dir? #$build-output-dir?
+ #:skip-build? #$skip-build?
+ #:install-source? #$install-source?
+ #:install-cache? #$install-cache?
+ #:parallel-build? #$parallel-build?
+ #:parallel-tests? #$parallel-tests?
+ #:environment-variables '#$environment-variables
+ #:goarch #$goarch
+ #:goos #$goos
+ #:phases #$phases
+ #:outputs #$(outputs->gexp outputs)
+ #:search-paths '#$(map
+ search-path-specification->sexp
+ search-paths)
+ #:inputs #$(input-tuples->gexp inputs)))))
+
+ (mlet %store-monad ((guile (package->derivation (or guile (default-guile))
+ system #:graft? #f)))
+ (gexp->derivation name builder
+ #:system system
+ #:guile-for-build guile)))
+
+(define* (go-cross-module-build name
+ #:key
+ source target
+ build-inputs target-inputs host-inputs
+ (phases '%standard-phases)
+ (outputs '("out"))
+ (search-paths '())
+ (native-search-paths '())
+ (go-flags '())
+ (ld-flags '("-s" "-w"))
+ (tags '())
+ (build-targets '("./..."))
+ (test-targets '())
+ (install-targets '())
+ (tests? #f) ; nothing can be done
+ (test-flags '())
+ (module-path #f)
+ (trimpath? #t)
+ (cgo? #f)
+ (build-output-dir? #f)
+ (skip-build? #f)
+ (install-source? #t)
+ (install-cache? #t)
+ (parallel-build? #t)
+ (parallel-tests? #f)
+ (environment-variables '())
+ (system (%current-system))
+ (goarch (if target (first (go-build:go-target target)) #f))
+ (goos (if target (last (go-build:go-target target)) #f))
+ (guile #f)
+ (imported-modules %go-module-build-system-modules)
+ (modules '((guix build go-module-build-system)
+ (guix build utils)))
+ (substitutable? #t))
+
+ (define builder
+ (with-imported-modules
+ imported-modules
+ #~(begin
+ (use-modules #$@(sexp->gexp modules))
+
+ (define %build-host-inputs
+ #+(input-tuples->gexp build-inputs))
+
+ (define %build-target-inputs
+ (append #$(input-tuples->gexp host-inputs)
+ #+(input-tuples->gexp target-inputs)))
+
+ (define %build-inputs
+ (append %build-host-inputs %build-target-inputs))
+
+ (go-module-build #:name #$name
+ #:source #+source
+ #:system #$system
+ #:go-flags '#$go-flags
+ #:ld-flags '#$ld-flags
+ #:tags '#$tags
+ #:build-targets '#$build-targets
+ #:test-targets '#$test-targets
+ #:install-targets '#$install-targets
+ #:test-flags '#$test-flags
+ #:module-path '#$module-path
+ #:trimpath? #$trimpath?
+ #:cgo? '#$cgo?
+ #:tests? #$tests?
+ #:build-output-dir? #$build-output-dir?
+ #:skip-build? #$skip-build?
+ #:install-source? #$install-source?
+ #:install-cache? #$install-cache?
+ #:parallel-build? #$parallel-build?
+ #:parallel-tests? #$parallel-tests?
+ #:environment-variables '#$environment-variables
+ #:target #$target
+ #:goarch #$goarch
+ #:goos #$goos
+ #:phases #$phases
+ #:outputs #$(outputs->gexp outputs)
+ #:make-dynamic-linker-cache? #f
+ #:search-paths '#$(map
+ search-path-specification->sexp
+ search-paths)
+ #:native-search-paths '#$(map
+ search-path-specification->sexp
+ native-search-paths)
+ #:native-inputs %build-host-inputs
+ #:inputs %build-inputs))))
+
+ (mlet %store-monad ((guile (package->derivation (or guile (default-guile))
+ system #:graft? #f)))
+ (gexp->derivation name builder
+ #:system system
+ #:target target
+ #:graft? #f
+ #:substitutable? substitutable?
+ #:guile-for-build guile)))
+
+(define go-module-build-system
+ (build-system
+ (name 'go-module)
+ (description "Go Module Build System")
+ (lower lower)))
+
+;;; go-module.scm ends here
diff --git a/guix/build/go-module-build-system.scm b/guix/build/go-module-build-system.scm
new file mode 100644
index 0000000000..8eeaac426c
--- /dev/null
+++ b/guix/build/go-module-build-system.scm
@@ -0,0 +1,459 @@
+(define-module (guix build go-module-build-system)
+ #:use-module ((guix build gnu-build-system) #:prefix gnu:)
+ #:use-module (guix build union)
+ #:use-module (guix build utils)
+ #:use-module (srfi srfi-71)
+ #:use-module (ice-9 rdelim)
+ #:use-module (ice-9 regex)
+ #:use-module (ice-9 match)
+ #:export (%standard-phases
+ go-module-build))
+
+;;; Commentary:
+;;;
+;;; Build procedure for packages using the module aware Go build
+;;; system. The go build system aggressively tries to fetch dependencies
+;;; or even compiler toolchains. While it may be possible to convince it to
+;;; not do that, we opt for not fighting it, and instead let it fetch
+;;; everything it wants to, served from the local filesystem in directories we
+;;; populate.
+;;;
+;;; The GOPROXY protocol [1] permits using file:// urls. From the manual:
+;;;
+;;; A module proxy is an HTTP server that can respond to GET requests for
+;;; paths specified below. The requests have no query parameters, and no
+;;; specific headers are required, so even a site serving from a fixed file
+;;; system (including a file:// URL) can be a module proxy.
+;;;
+;;; Go dependencies tend to be rigidly specified to very specific versions,
+;;; with hashes, which the go build tooling will figure out. This does not
+;;; work too well with guix' model, where we want to specify dependencies more
+;;; fludily (e.g. with input substitutions). Go modules also tend to specify
+;;; (minimum) toolchains which is not strictly necessary from a language
+;;; feature perspective, which breaks builds with older compilers.
+;;;
+;;; To address these problems, we always write a fresh go.mod file based on
+;;; the build-inputs. There is no guarantee that there even is a go.mod file
+;;; in the source, especially for older projects. Go build uses this file to
+;;; "download" from our just-assembled goproxy, which makes it happy. This
+;;; also clears any toolchain directive which makes the build accept the go
+;;; compiler through build-inputs. We populate the goproxy with just-in-time
+;;; built zips, version, and info files. This is a separate phase so that
+;;; additional build steps can be added between building the proxy and running go
+;;; build.
+;;;
+;;; The build system is compatible with go-build-system, in the sense that
+;;; go-build-system can be used as build-inputs, and vice versa, because they
+;;; both use the same $out/source/.
+;;;
+;;; We re-used compiled packages. The Go build system creates a
+;;; content-addressable build cache, which we install into build output, and
+;;; use to seed downstream builds. Go programs are (mostly) statically
+;;; linked, so this is roughly equivalent of installing lib.a. Note that this
+;;; only works when the build-input is built with go-module-build-system.
+;;;
+;;; [1] https://go.dev/ref/mod#goproxy-protocol
+;;;
+;;; Code:
+
+(define (find-single-file dir regex)
+ "Find the file in DIR matching the REGEX, and fail unless there is
+exactly one match."
+ (let ((files (find-files dir regex #:directories? #f)))
+ (unless (eq? 1 (length files))
+ (error "Expected exactly one file matching pattern, found:" files))
+ (car files)))
+
+(define (go-path-escape path)
+ "Escape a module path by replacing every uppercase letter with an
+exclamation mark followed with its lowercase equivalent, as per the module
+Escaped Paths specification (see:
+https://godoc.org/golang.org/x/mod/module#hdr-Escaped_Paths)."
+ (define (escape occurrence)
+ (string-append "!" (string-downcase (match:substring occurrence))))
+ (regexp-substitute/global #f "[A-Z]" path 'pre escape 'post))
+
+(define (call-with-append-file path f)
+ "call-with-output-file, but appends to the file if it exists rather
+than truncating it"
+ (let ((file (open-file path "a")))
+ (f file)
+ (close file)))
+
+(define (set-cache-action-epoch f)
+ "Set go build cache action entry timestamp to 0
+
+The go build cache action entries (xxxx-a) record a timestamp, which
+would break reproducibility of the build cache. Set it to all-zeros."
+ ;; The file has 5 columns, tand the timestamp is the rightmost one
+ ;; <version> <action-id> <output-id> <size> <timestamp>
+ ;;
+ ;; The timestamp seems to be in nanoseconds since epoch. We use
+ ;; replacement to avoid potential problems with whitespace
+ ;; sensitivity.
+ (let* ((file-line (read-first-line f))
+ (start-timestamp (string-skip-right file-line char-set:digit))
+ (end-timestamp (string-length file-line))
+ (base-line (substring file-line 0 start-timestamp))
+ (zero-timestamp (make-string (- end-timestamp start-timestamp) #\0)))
+ (call-with-output-file f
+ (lambda (port)
+ (format port "~a~a~%" base-line zero-timestamp)))))
+
+(define (copy-nonlink-recursively src-dir dst-dir)
+ "Recursively install files from src/ to dst/, skipping symlinks"
+ (copy-recursively src-dir dst-dir
+ #:log #f
+ #:copy-file
+ (lambda (src dst)
+ (unless (symbolic-link? src)
+ (copy-file src dst)))))
+
+(define (make-tags tags)
+ "Construct a -tags argument list.
+
+We accept tags both as a single string and a list of tags. go expects
+-tags tag1,tag2,..."
+ (cond ((and (list? tags) (not (null? tags)))
+ (list "-tags" (string-join tags ",")))
+ ((string? tags) (list "-tags" tags))
+ (else '())))
+
+(define (re-init-module module-path)
+ "Create a fresh go.mod file, replacing an old one if it exists."
+ ;; Wipe the go.mod if it exists, then create a new one. We might
+ ;; not use the exact same input set (versions or even modules) as
+ ;; upstream, e.g. when splitting an upstream package into multiple
+ ;; parts.
+ ;;
+ ;; Delete all go.sum files, if they exist. We do our own
+ ;; checksums so there is no safety here, and since our packages
+ ;; are differently sourced (and maybe differently versioned) they
+ ;; won't match upstream checksums.
+ (when (file-exists? "go.mod") (delete-file "go.mod"))
+ (when (file-exists? "go.sum") (delete-file "go.sum"))
+ (invoke/quiet "go" "mod" "init" module-path))
+
+(define (read-first-line f)
+ "Read the first line in a file"
+ (call-with-input-file f read-line))
+
+(define (filter-go-inputs inputs)
+ "Return the store paths of go library inputs.
+
+Inputs is a list of ('pkg' 'store-path') pairs, and returns a list of
+store paths.
+
+((zip . /gnu/store/x1c9w6dnmk23mpdfg08zyq379q26nd88-zip-3.0)
+ (unzip . /gnu/store/fmli224wbxrz1n0i2lhz6gy8a1ydcbp3-unzip-6.0)
+ (go-github-com-stretchr-testify . /gnu/store/7p6zk3zka35g3699b9kfl0njzwykimjm-go-github-com-stretchr-testify-1.10.0)
+ (go-golang-org-x-tools . /gnu/store/ax54x3d7fyywbppqvf0gmavsmxkz0h03-go-golang-org-x-tools-0.25.0))
+
+->
+
+(/gnu/store/7p6zk3zka35g3699b9kfl0njzwykimjm-go-github-com-stretchr-testify-1.10.0
+ /gnu/store/ax54x3d7fyywbppqvf0gmavsmxkz0h03-go-golang-org-x-tools-0.25.0)
+
+Sources installed with go-build-system and go-module-build-system have
+a /src directory. Packages realistically have dozens of inputs (the go
+compiler, coreutils, etc.) so this filtering is very much necessary."
+ (map cdr
+ (filter
+ (lambda (input)
+ (and (string-prefix? "go-" (car input))
+ (directory-exists? (string-append (cdr input) "/src"))))
+ inputs)))
+
+(define* (infer-module-path-from-dir dir #:optional (subdir "src"))
+ "Infer the go module 'github.com/user/module' from a store path.
+
+DIR should be a a store path <store>/<pkg>
+
+<store>/<pkg>/src/github.com/user/module/... -> github.com/user/module
+
+By default, this function assumes the store path is a go-build-system
+or go-module-build-system installed package with the sources installed
+under DIR/SUBDIR."
+ (with-directory-excursion (string-append dir "/" subdir)
+ (string-trim
+ (car
+ (sort! (map dirname (find-files "." #:fail-on-error? #t))
+ (lambda (x y) (< (string-length x) (string-length y)))))
+ char-set:punctuation)))
+
+(define* (module-path-from-file #:optional (go.mod "go.mod"))
+ "Read the module-path from a go.mod file.
+
+This assumes that go.mod exists and is well-defined. This can not be
+assumed in general until after the build stage, in which case go.mod
+should always have been generated in the root dir."
+ (cadr (string-split (read-first-line go.mod) #\space)))
+
+(define* (prepare-sources #:key module-path #:allow-other-keys)
+ ;; Remove any go.mod and go.sum files. The builder will write its
+ ;; own based on the build inputs, and any upstream checksums will be
+ ;; wrong. This has to be done before setup-proxy as it would detect
+ ;; the source dir's go.mod and go.sum files and fail because of
+ ;; checksum mismatches. It is in its own phase so that it can be
+ ;; overriden, if necessary.
+ ;;
+ ;; We want to init it with go init module so that it records the go
+ ;; version in order to not fall back to too old go versions:
+ ;; function instantiation requires go1.18 or later (-lang was set
+ ;; to go1.16; check go.mod)
+ ;;
+ ;; If the module path is explicitly set, use it. Otherwise,
+ ;; infer it from the go.mod file. If the go.mod file does not
+ ;; exist and module-path is not specified, fail the build.
+(let ((module-path (or module-path (module-path-from-file))))
+ (for-each delete-file (find-files "." "go\\.mod$"))
+ (for-each delete-file (find-files "." "go\\.sum$"))
+ (re-init-module module-path)))
+
+(define* (setup-go-env #:key outputs cgo? environment-variables
+ goarch goos #:allow-other-keys)
+ (let* ((go-proxy (string-append (getcwd) "/guix-go/proxy"))
+ (go-cache (string-append (getcwd) "/guix-go/cache"))
+ (go-mod-cache (string-append (getcwd) "/guix-go/modcache")))
+ (setenv "GOSUMDB" "off")
+ (setenv "GOPROXY" (string-append "file://" go-proxy))
+ (setenv "GOCACHE" go-cache)
+ (setenv "GOMODCACHE" go-mod-cache)
+ (setenv "GOBIN" (string-append (assoc-ref outputs "out") "/bin"))
+ (when cgo?
+ (setenv "CGO_ENABLED" "1"))
+
+ (setenv "GOARCH" (or goarch (getenv "GOHOSTARCH")))
+ (setenv "GOOS" (or goos (getenv "GOHOSTOS")))
+ (match goarch
+ ("arm"
+ (setenv "GOARM" "7"))
+ ((or "mips" "mipsel")
+ (setenv "GOMIPS" "hardfloat"))
+ ((or "mips64" "mips64le")
+ (setenv "GOMIPS64" "hardfloat"))
+ ((or "ppc64" "ppc64le")
+ (setenv "GOPPC64" "power8"))
+ (_ #t))
+
+ (for-each
+ (lambda (var) (setenv (car var) (cdr var)))
+ environment-variables)))
+
+(define (module-version-or-synthesized mod-input-path module-path)
+ "Resolve module or synthesize module version.
+
+Figure out what module version to use for go get to resolve, either
+the package version or a special syntesized one.
+
+This addresses a quirk of go module paths and the go module system. Go
+expects that if a package has a version >= v2.x.y, the module path is
+module/v2. When splitting a large package into smaller libraries that
+share import prefix, the module path no longer ends with the major
+version, and go get complains:
+
+ invalid version: should be v0 or v1, not v4
+
+The actual package versions used during the build matters little, and
+is an implementation detail for the builder. For the packages with
+version >= 2 with an \"unversioned\" module path, synthesize the special
+version 0.0.1-guix.
+
+MOD-INPUT-PATH should be the store path of the module as returned by
+filter-go-inputs. MODULE-PATH should be the module path as it is
+written in the go.mod, for example:
+
+(module-version-or-synthesized
+ \"/gnu/store/2v69cskzdjininks376wlw9cq3dv2gd1-go-github-com-stretchr-objx-0.5.2\"
+ \"github.com/stretchr/objx\")
+"
+ (let* ((_ pkg-version (package-name->name+version
+ (strip-store-file-name mod-input-path)))
+ (dir-version (basename module-path)))
+ (if (and (string-match "^[2-9]+\\." pkg-version)
+ (not (string-match "v[2-9]+$" dir-version)))
+ "0.0.1-guix"
+ pkg-version)))
+
+(define* (setup-goproxy #:key inputs #:allow-other-keys)
+ (let* (;; Remove file:// from the goproxy, we want the dir
+ (go-proxy (substring (getenv "GOPROXY") 7)))
+ (mkdir-p go-proxy)
+ (for-each
+ (lambda (mod-input-path)
+ (let* ((store-path (string-append mod-input-path "/src"))
+ (module-path (infer-module-path-from-dir mod-input-path))
+ (source-dir (string-append store-path "/" module-path))
+ (version (module-version-or-synthesized mod-input-path module-path))
+ (module-dir (format #f "~a <at> v~a" module-path version))
+ (proxy-dir (format #f "~a/~a/@v" go-proxy (go-path-escape module-path)))
+ (proxy/mod (string-append proxy-dir "/v" version ".mod"))
+ (proxy/zip (string-append proxy-dir "/v" version ".zip"))
+ (tmp-dir "guix-tmp")
+ (tmp-module (string-append tmp-dir "/" module-dir))
+ (tmp-mod (string-append tmp-module "/go.mod")))
+
+ ;; In some cases a module will show up twice, e.g. when
+ ;; breaking cyclic dependencies. In that case, don't install
+ ;; the second version. In that case the inputs have different
+ ;; store paths, and we can't rely at all on the package name.
+ (unless (file-exists? (string-append proxy-dir "/v" version ".mod"))
+ (mkdir-p tmp-module)
+ (copy-recursively source-dir tmp-module #:log #f)
+
+ ;; Delete all go.mod and go.sum files, and re-write the
+ ;; root one without dependencies and toolchain directives.
+ ;; Note that we cannot use re-init-module here. If there is
+ ;; a go.mod in the project root, it would be detected by
+ ;; re-init-module, and if it happened to contain a
+ ;; toolchain directive it would infect this module here.
+ (for-each delete-file (find-files tmp-module "go\\.mod$"))
+ (for-each delete-file (find-files tmp-module "go\\.sum$"))
+ (with-directory-excursion tmp-module
+ (re-init-module module-path))
+ (for-each
+ (lambda (f) (utime f 0 0 0 0 AT_SYMLINK_NOFOLLOW))
+ (find-files tmp-dir #:directories? #t))
+
+ ;; We need the -D flag, because go mod fails hard on any path
+ ;; that does not begin with $module <at> version, even if for
+ ;; sub-prefixes of $module
+ (mkdir-p proxy-dir)
+ (with-directory-excursion tmp-dir
+ (invoke "zip" "-r" "-q" "-o" "-D" "-X" proxy/zip "."))
+
+ (copy-file tmp-mod proxy/mod)
+ (delete-file-recursively tmp-dir)
+ (call-with-output-file (string-append proxy-dir "/v" version ".info")
+ (lambda (f) (format f "{~s:\"v~a\"}~%" "Version" version)))
+
+ (call-with-append-file (string-append proxy-dir "/list")
+ (lambda (f) (format f "v~a~%" version))))))
+ (filter-go-inputs inputs))))
+
+;; These paths must be consistent across different stages, so use
+;; symbols for them to ensure they're consistent
+(define guix-install-cache "guix-out-cache")
+(define var-build-cache "/var/cache/go/build")
+
+(define* (setup-gocache #:key inputs #:allow-other-keys)
+ (define (search-input-directories dir go-inputs)
+ (filter directory-exists?
+ (map (lambda (store) (string-append store "/" dir))
+ go-inputs)))
+
+ (union-build (getenv "GOCACHE")
+ (search-input-directories var-build-cache
+ (filter-go-inputs inputs))
+ ;; Creating all directories isn't that bad, because
+ ;; there are only ever 256 of them.
+ #:create-all-directories? #t
+ #:log-port (%make-void-port "w")))
+
+(define* (build #:key inputs go-flags tags build-targets
+ install-targets trimpath? build-output-dir? skip-build?
+ install-cache? install-source? parallel-build?
+ #:allow-other-keys)
+ (setenv "GOMAXPROCS"
+ (number->string
+ (if parallel-build? (parallel-job-count) 1)))
+
+ (for-each
+ (lambda (store-path)
+ (let* ((module (infer-module-path-from-dir store-path))
+ (version (module-version-or-synthesized store-path module)))
+ (invoke "go" "get" (string-append module "@v" version))))
+ (filter-go-inputs inputs))
+ ;; go.mod and go.sum have had their timestamps updated by go get,
+ ;; which will be snapshotted in the build cache and break it. From
+ ;; here on out these files should not need to change, so fix the
+ ;; timestamp. The sum will only exist if there are any
+ ;; dependencies.
+ (utime "go.mod" 0 0 0 0)
+ (when (file-exists? "go.sum")
+ (utime "go.sum" 0 0 0 0))
+
+ ;; If -o is used it must be the first flag to build. This flag is
+ ;; necessary when there is a command (program) with the same name
+ ;; as its directory, e.g. info/main.go instead of cmd/info.go
+ (unless skip-build?
+ (apply invoke "go" "build"
+ (append
+ (if build-output-dir?
+ (list "-o" (string-append (or (getenv "TMP") "/tmp")
+ "/go-build/")) '())
+ go-flags (make-tags tags)
+ (if trimpath? '("-trimpath") '())
+ build-targets install-targets)))
+
+ ;; Snapshot the cache before running tests. It is not interesting
+ ;; to snapshot test artifacts, and they may pollute the cache with
+ ;; non-reproducible artifacts.
+ (when (and install-cache? (not skip-build?))
+ (mkdir-p guix-install-cache)
+ (with-directory-excursion guix-install-cache
+ (copy-nonlink-recursively (getenv "GOCACHE") ".")
+ (delete-file "trim.txt")
+ (delete-file "README")
+ (for-each set-cache-action-epoch (find-files "." "-a$")))))
+
+(define* (check #:key inputs tests? go-flags tags test-flags
+ test-targets module-path parallel-tests? trimpath?
+ #:allow-other-keys)
+ (when tests?
+ (let ((njobs (if parallel-tests? (parallel-job-count) 1)))
+ (setenv "GOMAXPROCS" (number->string njobs))
+ (for-each
+ (lambda target
+ (apply invoke (append
+ (list "go" "test") go-flags (make-tags tags)
+ (if trimpath? '("-trimpath") '())
+ test-flags target)))
+ test-targets))))
+
+(define* (install #:key source inputs outputs go-flags tags
+ install-targets install-cache? install-source?
+ trimpath? #:allow-other-keys)
+ (for-each
+ (lambda target
+ (apply invoke (append (list "go" "install") go-flags (make-tags tags)
+ (if trimpath? '("-trimpath") '())
+ target)))
+ install-targets)
+
+ (when (directory-exists? guix-install-cache)
+ (with-directory-excursion guix-install-cache
+ (copy-recursively
+ "." (string-append (assoc-ref outputs "out") var-build-cache)
+ #:log #f)))
+
+ (when install-source?
+ (let* ((out (assoc-ref outputs "out"))
+ (name version (package-name->name+version (assoc-ref outputs "out")))
+ (module-line (read-first-line "go.mod"))
+ (module-path (cadr (string-split module-line #\space)))
+ (dst (format #f "~a/src/~a" out module-path)))
+ (copy-recursively source dst #:log #f)
+ (when (file-exists? (string-append dst "/go.mod"))
+ (make-file-writable (string-append dst "/go.mod")))
+ (install-file "go.mod" dst))))
+
+(define %standard-phases
+ (modify-phases gnu:%standard-phases
+ (delete 'bootstrap)
+ (delete 'configure)
+ (delete 'patch-generated-file-shebangs)
+ (add-after 'unpack 'prepare-sources prepare-sources)
+ (add-before 'build 'setup-go-env setup-go-env)
+ (add-after 'setup-go-env 'setup-goproxy setup-goproxy)
+ (add-after 'setup-goproxy 'setup-gocache setup-gocache)
+ (replace 'build build)
+ (replace 'check check)
+ (replace 'install install)))
+
+(define* (go-module-build #:key inputs (phases %standard-phases)
+ #:allow-other-keys #:rest args)
+ "Go Module Build System"
+ (apply gnu:gnu-build #:inputs inputs #:phases phases args))
+
+;;; go-module-build-system.scm ends here
--
2.39.5
guix-patches <at> gnu.org:bug#78404; Package guix-patches.
(Tue, 13 May 2025 09:56:02 GMT) Full text and rfc822 format available.Message #11 received at 78404 <at> debbugs.gnu.org (full text, mbox):
From: Jørgen Kvalsvik <j <at> lambda.is> To: 78404 <at> debbugs.gnu.org Cc: Jørgen Kvalsvik <j <at> lambda.is>, steve <at> futurile.net Subject: [PATCH 1/2] guix: Add downloader for go modules from GOPROXY Date: Tue, 13 May 2025 11:55:21 +0200
Add a new downloader which implements, approximately, the download step of go
get $module. This is a convenient way of downloading zips with go modules by
just specifying the version and import path, as an alternative to git clone,
or awkward https:// fetches. This is particularly useful for sources that are
processed before release (like autotools generated files in tarballs) or
generated modules.
* guix/go-mod-download.scm: New file.
* Makefile.am (MODULES): Add it.
Change-Id: Ibb3b3ee70833fd0ea0c64278c95b8cb96a0be639
---
Makefile.am | 1 +
guix/go-mod-download.scm | 126 +++++++++++++++++++++++++++++++++++++++
2 files changed, 127 insertions(+)
create mode 100644 guix/go-mod-download.scm
diff --git a/Makefile.am b/Makefile.am
index ec5220333e..b5fb81f412 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -102,6 +102,7 @@ MODULES = \
guix/android-repo-download.scm \
guix/bzr-download.scm \
guix/git-download.scm \
+ guix/go-mod-download.scm \
guix/hg-download.scm \
guix/hash.scm \
guix/swh.scm \
diff --git a/guix/go-mod-download.scm b/guix/go-mod-download.scm
new file mode 100644
index 0000000000..7024362318
--- /dev/null
+++ b/guix/go-mod-download.scm
@@ -0,0 +1,126 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2025 Jørgen Kvalsvik <j <at> lambda.is>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix go-mod-download)
+ #:use-module (guix gexp)
+ #:use-module (guix store)
+ #:use-module (guix packages)
+ #:use-module (guix monads)
+ #:use-module (guix records)
+ #:use-module (guix modules)
+ #:use-module (guix download)
+ #:use-module (ice-9 string-fun)
+ #:use-module (ice-9 regex)
+ #:autoload (guix build-system gnu) (standard-packages)
+ #:export (go-mod-reference
+ go-mod-reference?
+ go-mod-reference-path
+ go-mod-reference-version
+ go-mod-fetch))
+
+;;; Commentary:
+;;;
+;;; An <origin> method that fetches a go module [1] from a GOPROXY. A go
+;;; module is usually identified as a vcs (usually git) repository,
+;;; e.g. github.com/calmh/du or golang.org/x/net.
+;;;
+;;; This is mostly a regular http(s) fetch some custom url building. Unless
+;;; goproxy is specified, it fetches from the default goproxy
+;;; https://proxy.golang.org. This is mostly just a convenience -- the same
+;;; code could be fetched directly, but sometimes libraries are only
+;;; practically available through a goproxy. Such a module would be
+;;; https://pkg.go.dev/buf.build/gen/go/bufbuild/protovalidate/protocolbuffers/go
+;;;
+;;; [1] https://go.dev/ref/mod
+;;;
+;;; Code:
+
+(define-record-type* <go-mod-reference>
+ go-mod-reference make-go-mod-reference
+ go-mod-reference?
+ (path go-mod-reference-path)
+ (version go-mod-reference-version)
+ (goproxy go-mod-reference-goproxy (default "https://proxy.golang.org")))
+
+(define (default-unzip)
+ "Return the 'unzip' package. This is a lazy reference so that we don't
+depend on (gnu packages compression)."
+ (module-ref (resolve-interface '(gnu packages compression)) 'unzip))
+
+;; Fetch a go module e.g. golang.org/x/net from a goproxy.
+(define* (go-mod-fetch ref hash-algo hash
+ #:optional name
+ #:key (system (%current-system))
+ (guile (default-guile))
+ (unzip (default-unzip)))
+ (define inputs
+ `(("unzip" ,unzip)
+ ,@(standard-packages)))
+
+ (define (go-path-escape path)
+ "Escape a module path by replacing every uppercase letter with an
+exclamation mark followed with its lowercase equivalent, as per the module
+Escaped Paths specification (see:
+https://godoc.org/golang.org/x/mod/module#hdr-Escaped_Paths)."
+ (define (escape occurrence)
+ (string-append "!" (string-downcase (match:substring occurrence))))
+ (regexp-substitute/global #f "[A-Z]" path 'pre escape 'post))
+
+ (define modules
+ (source-module-closure '((guix build utils))))
+
+ (define (build mod.zip)
+ (with-imported-modules modules
+ #~(begin
+ (use-modules (guix build utils))
+ (let* ((pkg-path (getenv "go-mod path"))
+ (pkg-version (getenv "go-mod version"))
+ (pkg-root (string-append pkg-path "@v" pkg-version))
+ (go.mod (string-append pkg-root "/go.mod")))
+
+ (invoke (string-append #+unzip "/bin/unzip") "-q" #$mod.zip)
+ ;; The sources in the zip are in the subdir
+ ;; $path <at> v$version/, but we want our sources at root.
+ (copy-recursively pkg-root #$output)))))
+
+ (define path-as-store-name
+ (string-append
+ (string-replace-substring (go-mod-reference-path ref) "/" "-")
+ "-" (go-mod-reference-version ref)))
+
+ (define url/zip
+ (format #f "~a/~a/@v/v~a.zip"
+ (go-mod-reference-goproxy ref)
+ (go-path-escape (go-mod-reference-path ref))
+ (go-mod-reference-version ref)))
+
+ (mlet* %store-monad ((guile-for-build (package->derivation guile system))
+ (mod (url-fetch url/zip hash-algo hash
+ (or name (string-append path-as-store-name ".zip"))
+ #:system system
+ #:guile guile)))
+ (gexp->derivation (or name path-as-store-name) (build mod)
+ #:script-name "go-mod-fetch"
+ #:env-vars
+ `(("go-mod path" . ,(go-mod-reference-path ref))
+ ("go-mod version" . ,(go-mod-reference-version ref)))
+ #:leaked-env-vars '("http_proxy" "https_proxy"
+ "LC_ALL" "LC_MESSAGES" "LANG"
+ "COLUMNS")
+ #:system system
+ #:local-build? #t)))
--
2.39.5
guix-patches <at> gnu.org:bug#78404; Package guix-patches.
(Thu, 15 May 2025 08:18:02 GMT) Full text and rfc822 format available.Message #14 received at 78404 <at> debbugs.gnu.org (full text, mbox):
From: Jørgen Kvalsvik <j <at> lambda.is> To: 78404 <at> debbugs.gnu.org Cc: Jørgen Kvalsvik <j <at> lambda.is> Subject: [PATCH 1/4] guix: Add downloader for go modules from GOPROXY Date: Thu, 15 May 2025 10:16:32 +0200
Add a new downloader which implements, approximately, the download step of go
get $module. This is a convenient way of downloading zips with go modules by
just specifying the version and import path, as an alternative to git clone,
or awkward https:// fetches. This is particularly useful for sources that are
processed before release (like autotools generated files in tarballs) or
generated modules.
* guix/go-mod-download.scm: New file.
* Makefile.am (MODULES): Add it.
Change-Id: Ibb3b3ee70833fd0ea0c64278c95b8cb96a0be639
---
Makefile.am | 1 +
guix/go-mod-download.scm | 126 +++++++++++++++++++++++++++++++++++++++
2 files changed, 127 insertions(+)
create mode 100644 guix/go-mod-download.scm
diff --git a/Makefile.am b/Makefile.am
index ec5220333e..b5fb81f412 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -102,6 +102,7 @@ MODULES = \
guix/android-repo-download.scm \
guix/bzr-download.scm \
guix/git-download.scm \
+ guix/go-mod-download.scm \
guix/hg-download.scm \
guix/hash.scm \
guix/swh.scm \
diff --git a/guix/go-mod-download.scm b/guix/go-mod-download.scm
new file mode 100644
index 0000000000..7024362318
--- /dev/null
+++ b/guix/go-mod-download.scm
@@ -0,0 +1,126 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2025 Jørgen Kvalsvik <j <at> lambda.is>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix go-mod-download)
+ #:use-module (guix gexp)
+ #:use-module (guix store)
+ #:use-module (guix packages)
+ #:use-module (guix monads)
+ #:use-module (guix records)
+ #:use-module (guix modules)
+ #:use-module (guix download)
+ #:use-module (ice-9 string-fun)
+ #:use-module (ice-9 regex)
+ #:autoload (guix build-system gnu) (standard-packages)
+ #:export (go-mod-reference
+ go-mod-reference?
+ go-mod-reference-path
+ go-mod-reference-version
+ go-mod-fetch))
+
+;;; Commentary:
+;;;
+;;; An <origin> method that fetches a go module [1] from a GOPROXY. A go
+;;; module is usually identified as a vcs (usually git) repository,
+;;; e.g. github.com/calmh/du or golang.org/x/net.
+;;;
+;;; This is mostly a regular http(s) fetch some custom url building. Unless
+;;; goproxy is specified, it fetches from the default goproxy
+;;; https://proxy.golang.org. This is mostly just a convenience -- the same
+;;; code could be fetched directly, but sometimes libraries are only
+;;; practically available through a goproxy. Such a module would be
+;;; https://pkg.go.dev/buf.build/gen/go/bufbuild/protovalidate/protocolbuffers/go
+;;;
+;;; [1] https://go.dev/ref/mod
+;;;
+;;; Code:
+
+(define-record-type* <go-mod-reference>
+ go-mod-reference make-go-mod-reference
+ go-mod-reference?
+ (path go-mod-reference-path)
+ (version go-mod-reference-version)
+ (goproxy go-mod-reference-goproxy (default "https://proxy.golang.org")))
+
+(define (default-unzip)
+ "Return the 'unzip' package. This is a lazy reference so that we don't
+depend on (gnu packages compression)."
+ (module-ref (resolve-interface '(gnu packages compression)) 'unzip))
+
+;; Fetch a go module e.g. golang.org/x/net from a goproxy.
+(define* (go-mod-fetch ref hash-algo hash
+ #:optional name
+ #:key (system (%current-system))
+ (guile (default-guile))
+ (unzip (default-unzip)))
+ (define inputs
+ `(("unzip" ,unzip)
+ ,@(standard-packages)))
+
+ (define (go-path-escape path)
+ "Escape a module path by replacing every uppercase letter with an
+exclamation mark followed with its lowercase equivalent, as per the module
+Escaped Paths specification (see:
+https://godoc.org/golang.org/x/mod/module#hdr-Escaped_Paths)."
+ (define (escape occurrence)
+ (string-append "!" (string-downcase (match:substring occurrence))))
+ (regexp-substitute/global #f "[A-Z]" path 'pre escape 'post))
+
+ (define modules
+ (source-module-closure '((guix build utils))))
+
+ (define (build mod.zip)
+ (with-imported-modules modules
+ #~(begin
+ (use-modules (guix build utils))
+ (let* ((pkg-path (getenv "go-mod path"))
+ (pkg-version (getenv "go-mod version"))
+ (pkg-root (string-append pkg-path "@v" pkg-version))
+ (go.mod (string-append pkg-root "/go.mod")))
+
+ (invoke (string-append #+unzip "/bin/unzip") "-q" #$mod.zip)
+ ;; The sources in the zip are in the subdir
+ ;; $path <at> v$version/, but we want our sources at root.
+ (copy-recursively pkg-root #$output)))))
+
+ (define path-as-store-name
+ (string-append
+ (string-replace-substring (go-mod-reference-path ref) "/" "-")
+ "-" (go-mod-reference-version ref)))
+
+ (define url/zip
+ (format #f "~a/~a/@v/v~a.zip"
+ (go-mod-reference-goproxy ref)
+ (go-path-escape (go-mod-reference-path ref))
+ (go-mod-reference-version ref)))
+
+ (mlet* %store-monad ((guile-for-build (package->derivation guile system))
+ (mod (url-fetch url/zip hash-algo hash
+ (or name (string-append path-as-store-name ".zip"))
+ #:system system
+ #:guile guile)))
+ (gexp->derivation (or name path-as-store-name) (build mod)
+ #:script-name "go-mod-fetch"
+ #:env-vars
+ `(("go-mod path" . ,(go-mod-reference-path ref))
+ ("go-mod version" . ,(go-mod-reference-version ref)))
+ #:leaked-env-vars '("http_proxy" "https_proxy"
+ "LC_ALL" "LC_MESSAGES" "LANG"
+ "COLUMNS")
+ #:system system
+ #:local-build? #t)))
--
2.39.5
guix-patches <at> gnu.org:bug#78404; Package guix-patches.
(Thu, 15 May 2025 08:18:02 GMT) Full text and rfc822 format available.Message #17 received at 78404 <at> debbugs.gnu.org (full text, mbox):
From: Jørgen Kvalsvik <j <at> lambda.is> To: 78404 <at> debbugs.gnu.org Cc: Jørgen Kvalsvik <j <at> lambda.is> Subject: [PATCH 4/4] gnu: Build with go-module-build-system Date: Thu, 15 May 2025 10:16:35 +0200
* gnu/packages/golang-xyz.scm (go-github-com-burntsushi-toml): Use
go-module-build-system
Change-Id: Ic9222044eecc53053898d978d9bc4f79280fec8b
---
gnu/packages/golang-xyz.scm | 5 +----
1 file changed, 1 insertion(+), 4 deletions(-)
diff --git a/gnu/packages/golang-xyz.scm b/gnu/packages/golang-xyz.scm
index bcf758e8fe..0bae8577d8 100644
--- a/gnu/packages/golang-xyz.scm
+++ b/gnu/packages/golang-xyz.scm
@@ -2297,10 +2297,7 @@ (define-public go-github-com-burntsushi-toml
(file-name (git-file-name name version))
(sha256
(base32 "1vk0s7pcn80hkx0lcyws509gqs42c8y1rppv05zxiqj0yn2zrjnx"))))
- (build-system go-build-system)
- (arguments
- (list
- #:import-path "github.com/BurntSushi/toml"))
+ (build-system go-module-build-system)
(home-page "https://github.com/BurntSushi/toml")
(synopsis "Toml parser and encoder for Go")
(description
--
2.39.5
guix-patches <at> gnu.org:bug#78404; Package guix-patches.
(Thu, 15 May 2025 08:18:03 GMT) Full text and rfc822 format available.Message #20 received at 78404 <at> debbugs.gnu.org (full text, mbox):
From: Jørgen Kvalsvik <j <at> lambda.is> To: 78404 <at> debbugs.gnu.org Cc: Jørgen Kvalsvik <j <at> lambda.is> Subject: [PATCH 3/4] gnu: Build go-ulid with go-module-build system Date: Thu, 15 May 2025 10:16:34 +0200
* gnu/packages/golang-xyz.scm (go-ulid): Use go-module-build-system.
Change-Id: I4ee3ae8cdb5a17df1478aeef555ad8d6fcdbf437
---
gnu/packages/golang-xyz.scm | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/gnu/packages/golang-xyz.scm b/gnu/packages/golang-xyz.scm
index 21d2b05abc..bcf758e8fe 100644
--- a/gnu/packages/golang-xyz.scm
+++ b/gnu/packages/golang-xyz.scm
@@ -79,9 +79,11 @@
(define-module (gnu packages golang-xyz)
#:use-module ((guix licenses) #:prefix license:)
#:use-module (guix build-system go)
+ #:use-module (guix build-system go-module)
#:use-module (guix build-system copy)
#:use-module (guix gexp)
#:use-module (guix git-download)
+ #:use-module (guix go-mod-download)
#:use-module (guix packages)
#:use-module (guix utils)
#:use-module (gnu packages)
@@ -18217,11 +18219,13 @@ (define-public go-tomlv
(define-public go-ulid
(package/inherit go-github-com-oklog-ulid-v2
(name "go-ulid")
+ (build-system go-module-build-system)
(arguments
(list
#:install-source? #f
- #:import-path "github.com/oklog/ulid/v2/cmd/ulid"
- #:unpack-path "github.com/oklog/ulid/v2"))
+ #:install-cache? #f
+ #:build-targets '()
+ #:install-targets '("./cmd/ulid")))
(description
(string-append (package-description go-github-com-oklog-ulid-v2)
"\nThis package provides a command line interface (CLI)
--
2.39.5
guix-patches <at> gnu.org:bug#78404; Package guix-patches.
(Thu, 15 May 2025 08:18:03 GMT) Full text and rfc822 format available.Message #23 received at 78404 <at> debbugs.gnu.org (full text, mbox):
From: Jørgen Kvalsvik <j <at> lambda.is> To: 78404 <at> debbugs.gnu.org Cc: Jørgen Kvalsvik <j <at> lambda.is> Subject: [PATCH 0/4] Go module aware build system Date: Thu, 15 May 2025 10:16:31 +0200
Hi,
Here's a slightly revised patch series which fixes a couple of issues.
`go build` is now invoked in a loop, which allows specifying both patterns and
specific binaries. I needed this for a test case where a project built
internal/echo-plugin for use in tests. There is no real downside to doing this
in a loop as far as I am aware, and install is uses a loop.
I also fixed cross compiling. I did not test it extensively, but cross
compiling using the go-build-system doesn't seem to work well, because it sets
the GOBIN env var, which makes the compiler error out.
From my x86_64-linux system:
~$ file $(guix build go-ulid --target=x86_64-linux-gnu)/bin/ulid
/gnu/store/i258kcbn5g8s97bz26q2rkic7ar5667d-go-ulid-2.1.0/bin/ulid:
ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked,
Go BuildID=_VaiUxKWsiIAlkWq7ODq/4GeipEcLMBNQHuHSA3kh/kxL5wZXJ8V0hvXEoJaF4/hO3bq-ZKZ51CEH4uY7rc, stripped
~$ file $(guix build go-ulid --target=aarch64-linux-gnu)/bin/ulid
/gnu/store/f4yzrwd198xhk4194xrfvqdk5yqz0nac-go-ulid-2.1.0/bin/ulid:
ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked,
Go BuildID=Z_1o0etV-eeUchVexa3A/bUxxm0FmCf6iLc1enI_T/ET4MK0HAx8OyRYBFV73n/j-RubHuPPW1R02OqsNu9, stripped
~$ file $(guix build go-ulid --target=arm-linux-gnueabihf)/bin/ulid
/gnu/store/q6f1l82bl1lpvrciibkiffjfsxn8vz0x-go-ulid-2.1.0/bin/ulid: ELF
32-bit LSB executable, ARM, EABI5 version 1 (SYSV), statically linked,
Go BuildID=NVoNI56N4Ri60CnAR78t/9bQS_Y8jZabn5Y1v80w-/D1_8mW1gwE0Bdtx-heU-/aU1ZqDUxXoi_ToBSeHUL, stripped
I also added two demos, so it would be easier for you to try and to show the
compatibility between packages using the two build systems. One is an
executable, and one where is a library.
~$ guix build go-github-com-burntsushi-toml go-ulid
Jørgen Kvalsvik (4):
guix: Add downloader for go modules from GOPROXY
guix: Add module-aware build system for go
gnu: Build go-ulid with go-module-build system
gnu: Build with go-module-build-system
Makefile.am | 3 +
gnu/packages/golang-xyz.scm | 13 +-
guix/build-system/go-module.scm | 267 +++++++++++++++
guix/build-system/zig.scm | 4 +-
guix/build/go-module-build-system.scm | 473 ++++++++++++++++++++++++++
guix/go-mod-download.scm | 126 +++++++
6 files changed, 877 insertions(+), 9 deletions(-)
create mode 100644 guix/build-system/go-module.scm
create mode 100644 guix/build/go-module-build-system.scm
create mode 100644 guix/go-mod-download.scm
--
2.39.5
guix-patches <at> gnu.org:bug#78404; Package guix-patches.
(Thu, 15 May 2025 08:18:04 GMT) Full text and rfc822 format available.Message #26 received at 78404 <at> debbugs.gnu.org (full text, mbox):
From: Jørgen Kvalsvik <j <at> lambda.is> To: 78404 <at> debbugs.gnu.org Cc: Jørgen Kvalsvik <j <at> lambda.is> Subject: [PATCH 2/4] guix: Add module-aware build system for go Date: Thu, 15 May 2025 10:16:33 +0200
Add a new build system for go, using go modules. This build system is partly
compatible with go-build-system; they can both be used as build inputs to each
other, but their options are incompatible.
The main departure from go-build-system is that go-build-system tries to build
a workspace [1], where go-module-build-system builds a goproxy + go.mod file
and lets `go build` do what it wants to.
Most go libraries should be straight forward to build. For example, this is
the package definition for golang.org/x/sync <at> 0.12:
(define-public go-golang-org-x-sync
(package
(name "go-golang-org-x-sync")
(version "0.12.0")
(source
(origin
(method go-mod-fetch)
(uri (go-mod-reference
(path "golang.org/x/sync")
(version version)))
(sha256
(base32 "00pd84ah4xd5sjax8rxv98xbnwrvkk8clazl3kq1xrbkmvjq2m53"))))
(build-system go-module-build-system)
(home-page "https://golang.org/x/sync")
(synopsis "Go Sync")
(description "This repository provides Go concurrency primitives
in addition to the ones provided by the language and \"sync\" and
\"sync/atomic\" packages.")
(license license:bsd-3)))
The build system also supports higher resolution build-, test-, and install
targets, re-use of compiled files, and options for common build tweaks.
[1] <https://golang.org/doc/code.html#Workspaces>
* guix/build/go-module-build-system.scm: New file.
* guix/build-system/go-module.scm: New file.
* Makefile.am (MODULES): Add them.
Change-Id: I47a028ab8f95fd3a338036480dbad6677e9c50a5
---
Makefile.am | 2 +
guix/build-system/go-module.scm | 267 +++++++++++++++
guix/build-system/zig.scm | 4 +-
guix/build/go-module-build-system.scm | 473 ++++++++++++++++++++++++++
4 files changed, 743 insertions(+), 3 deletions(-)
create mode 100644 guix/build-system/go-module.scm
create mode 100644 guix/build/go-module-build-system.scm
diff --git a/Makefile.am b/Makefile.am
index b5fb81f412..12446e6bb4 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -168,6 +168,7 @@ MODULES = \
guix/build-system/glib-or-gtk.scm \
guix/build-system/gnu.scm \
guix/build-system/go.scm \
+ guix/build-system/go-module.scm \
guix/build-system/guile.scm \
guix/build-system/haskell.scm \
guix/build-system/julia.scm \
@@ -227,6 +228,7 @@ MODULES = \
guix/build/minify-build-system.scm \
guix/build/font-build-system.scm \
guix/build/go-build-system.scm \
+ guix/build/go-module-build-system.scm \
guix/build/android-repo.scm \
guix/build/asdf-build-system.scm \
guix/build/bzr.scm \
diff --git a/guix/build-system/go-module.scm b/guix/build-system/go-module.scm
new file mode 100644
index 0000000000..a61f591431
--- /dev/null
+++ b/guix/build-system/go-module.scm
@@ -0,0 +1,267 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2025 Jørgen Kvalsvik <j <at> lambda.is>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix build-system go-module)
+ #:use-module (guix gexp)
+ #:use-module (guix monads)
+ #:use-module (guix packages)
+ #:use-module (guix store)
+ #:use-module (guix utils)
+ #:use-module (guix search-paths)
+ #:use-module (guix build-system)
+ #:use-module (guix build-system gnu)
+ #:use-module ((guix build-system go) #:prefix go-build:)
+ #:use-module (srfi srfi-1)
+ #:export (%go-module-build-system-modules
+ go-module-build
+ go-module-build-system))
+
+;;; Commentary:
+;;;
+;;; Build procedure for packages using the module aware Go build system.
+;;;
+;;; Code:
+
+(define %go-module-build-system-modules
+ ;; Build-side modules imported by default.
+ `((guix build go-module-build-system)
+ (guix build union)
+ ,@%default-gnu-imported-modules))
+
+(define (default-go)
+ ;; Lazily resolve the binding to avoid a circular dependency.
+ (let ((go (resolve-interface '(gnu packages golang))))
+ (module-ref go 'go)))
+
+(define (default-gccgo)
+ ;; Lazily resolve the binding to avoid a circular dependency.
+ (let ((gcc (resolve-interface '(gnu packages gcc))))
+ (module-ref gcc 'gccgo-12)))
+
+(define (default-zip)
+ "Return the 'zip' package. This is a lazy reference so that we don't
+depend on (gnu packages compression)."
+ (let ((distro (resolve-interface '(gnu packages compression))))
+ (module-ref distro 'zip)))
+
+(define* (lower name
+ #:key source inputs native-inputs outputs system target
+ (go (if (supported-package? (default-go))
+ (default-go)
+ (default-gccgo)))
+ (zip (default-zip))
+ #:allow-other-keys
+ #:rest arguments)
+ "Return a bag for NAME."
+ (define private-keywords
+ '(#:target #:inputs #:native-inputs #:go #:zip))
+
+ (bag
+ (name name)
+ (system system)
+ (target target)
+ (build-inputs `(,@(if source `(("source" ,source)) '())
+ ,@`(("go" ,go) ("zip" ,zip))
+ ,@native-inputs
+ ,@(if target '() inputs)
+ ,@(if target (standard-cross-packages target 'host) '())
+ ;; Keep the standard inputs of 'gnu-build-system'.
+ ,@(standard-packages)))
+ (host-inputs (if target inputs '()))
+ (target-inputs (if target (standard-cross-packages target 'target) '()))
+ (outputs outputs)
+ (build (if target go-cross-module-build go-module-build))
+ (arguments (strip-keyword-arguments private-keywords arguments))))
+
+(define* (go-module-build name inputs
+ #:key
+ source
+ (phases '%standard-phases)
+ (outputs '("out"))
+ (search-paths '())
+ (go-flags '())
+ (ld-flags '("-s" "-w"))
+ (tags '())
+ (build-targets '("./..."))
+ (test-targets '("./..."))
+ (install-targets '())
+ (test-flags '())
+ (module-path #f)
+ (trimpath? #t)
+ (cgo? #f)
+ (tests? #t)
+ (build-output-dir? #f)
+ (skip-build? #f)
+ (install-source? #t)
+ (install-cache? #t)
+ (parallel-build? #t)
+ (parallel-tests? #t)
+ (environment-variables '())
+ (system (%current-system))
+ (goarch #f)
+ (goos #f)
+ (guile #f)
+ (substitutable? #t)
+ (imported-modules %go-module-build-system-modules)
+ (modules '((guix build go-module-build-system)
+ (guix build utils))))
+
+ (define builder
+ (with-imported-modules
+ imported-modules
+ #~(begin
+ (use-modules #$@(sexp->gexp modules))
+ (go-module-build #:name #$name
+ #:source #+source
+ #:system #$system
+ #:go-flags '#$go-flags
+ #:ld-flags '#$ld-flags
+ #:tags '#$tags
+ #:build-targets '#$build-targets
+ #:test-targets '#$test-targets
+ #:install-targets '#$install-targets
+ #:test-flags '#$test-flags
+ #:module-path '#$module-path
+ #:trimpath? #$trimpath?
+ #:cgo? '#$cgo?
+ #:tests? #$tests?
+ #:build-output-dir? #$build-output-dir?
+ #:skip-build? #$skip-build?
+ #:install-source? #$install-source?
+ #:install-cache? #$install-cache?
+ #:parallel-build? #$parallel-build?
+ #:parallel-tests? #$parallel-tests?
+ #:environment-variables '#$environment-variables
+ #:goarch #$goarch
+ #:goos #$goos
+ #:phases #$phases
+ #:outputs #$(outputs->gexp outputs)
+ #:search-paths '#$(map
+ search-path-specification->sexp
+ search-paths)
+ #:inputs #$(input-tuples->gexp inputs)))))
+
+ (mlet %store-monad ((guile (package->derivation (or guile (default-guile))
+ system #:graft? #f)))
+ (gexp->derivation name builder
+ #:system system
+ #:guile-for-build guile)))
+
+(define* (go-cross-module-build name
+ #:key
+ source target
+ build-inputs target-inputs host-inputs
+ (phases '%standard-phases)
+ (outputs '("out"))
+ (search-paths '())
+ (native-search-paths '())
+ (go-flags '())
+ (ld-flags '("-s" "-w"))
+ (tags '())
+ (build-targets '("./..."))
+ (test-targets '())
+ (install-targets '())
+ (tests? #f) ; nothing can be done
+ (test-flags '())
+ (module-path #f)
+ (trimpath? #t)
+ (cgo? #f)
+ (build-output-dir? #f)
+ (skip-build? #f)
+ (install-source? #t)
+ (install-cache? #t)
+ (parallel-build? #t)
+ (parallel-tests? #f)
+ (environment-variables '())
+ (system (%current-system))
+ (goarch (if target (first (go-build:go-target target)) #f))
+ (goos (if target (last (go-build:go-target target)) #f))
+ (guile #f)
+ (imported-modules %go-module-build-system-modules)
+ (modules '((guix build go-module-build-system)
+ (guix build utils)))
+ (substitutable? #t))
+
+ (define builder
+ (with-imported-modules
+ imported-modules
+ #~(begin
+ (use-modules #$@(sexp->gexp modules))
+
+ (define %build-host-inputs
+ #+(input-tuples->gexp build-inputs))
+
+ (define %build-target-inputs
+ (append #$(input-tuples->gexp host-inputs)
+ #+(input-tuples->gexp target-inputs)))
+
+ (define %build-inputs
+ (append %build-host-inputs %build-target-inputs))
+
+ (go-module-build #:name #$name
+ #:source #+source
+ #:system #$system
+ #:go-flags '#$go-flags
+ #:ld-flags '#$ld-flags
+ #:tags '#$tags
+ #:build-targets '#$build-targets
+ #:test-targets '#$test-targets
+ #:install-targets '#$install-targets
+ #:test-flags '#$test-flags
+ #:module-path '#$module-path
+ #:trimpath? #$trimpath?
+ #:cgo? '#$cgo?
+ #:tests? #$tests?
+ #:build-output-dir? #$build-output-dir?
+ #:skip-build? #$skip-build?
+ #:install-source? #$install-source?
+ #:install-cache? #$install-cache?
+ #:parallel-build? #$parallel-build?
+ #:parallel-tests? #$parallel-tests?
+ #:environment-variables '#$environment-variables
+ #:target #$target
+ #:goarch #$goarch
+ #:goos #$goos
+ #:phases #$phases
+ #:outputs #$(outputs->gexp outputs)
+ #:make-dynamic-linker-cache? #f
+ #:search-paths '#$(map
+ search-path-specification->sexp
+ search-paths)
+ #:native-search-paths '#$(map
+ search-path-specification->sexp
+ native-search-paths)
+ #:native-inputs %build-host-inputs
+ #:inputs %build-inputs))))
+
+ (mlet %store-monad ((guile (package->derivation (or guile (default-guile))
+ system #:graft? #f)))
+ (gexp->derivation name builder
+ #:system system
+ #:target target
+ #:graft? #f
+ #:substitutable? substitutable?
+ #:guile-for-build guile)))
+
+(define go-module-build-system
+ (build-system
+ (name 'go-module)
+ (description "Go Module Build System")
+ (lower lower)))
+
+;;; go-module.scm ends here
diff --git a/guix/build-system/zig.scm b/guix/build-system/zig.scm
index 43d6ee977c..238964eb22 100644
--- a/guix/build-system/zig.scm
+++ b/guix/build-system/zig.scm
@@ -206,9 +206,7 @@ (define private-keywords
;; Keep the standard inputs of 'gnu-build-system'.
,@(standard-packages)))
(host-inputs (if target inputs '()))
- (target-inputs (if target
- (standard-cross-packages target 'target)
- '()))
+ (target-inputs (if target (standard-cross-packages target 'target) '()))
(outputs outputs)
(build (if target zig-cross-build zig-build))
(arguments (strip-keyword-arguments private-keywords arguments))))
diff --git a/guix/build/go-module-build-system.scm b/guix/build/go-module-build-system.scm
new file mode 100644
index 0000000000..adc36df356
--- /dev/null
+++ b/guix/build/go-module-build-system.scm
@@ -0,0 +1,473 @@
+(define-module (guix build go-module-build-system)
+ #:use-module ((guix build gnu-build-system) #:prefix gnu:)
+ #:use-module (guix build union)
+ #:use-module (guix build utils)
+ #:use-module (srfi srfi-71)
+ #:use-module (ice-9 rdelim)
+ #:use-module (ice-9 regex)
+ #:use-module (ice-9 match)
+ #:export (%standard-phases
+ go-module-build))
+
+;;; Commentary:
+;;;
+;;; Build procedure for packages using the module aware Go build
+;;; system. The go build system aggressively tries to fetch dependencies
+;;; or even compiler toolchains. While it may be possible to convince it to
+;;; not do that, we opt for not fighting it, and instead let it fetch
+;;; everything it wants to, served from the local filesystem in directories we
+;;; populate.
+;;;
+;;; The GOPROXY protocol [1] permits using file:// urls. From the manual:
+;;;
+;;; A module proxy is an HTTP server that can respond to GET requests for
+;;; paths specified below. The requests have no query parameters, and no
+;;; specific headers are required, so even a site serving from a fixed file
+;;; system (including a file:// URL) can be a module proxy.
+;;;
+;;; Go dependencies tend to be rigidly specified to very specific versions,
+;;; with hashes, which the go build tooling will figure out. This does not
+;;; work too well with guix' model, where we want to specify dependencies more
+;;; fludily (e.g. with input substitutions). Go modules also tend to specify
+;;; (minimum) toolchains which is not strictly necessary from a language
+;;; feature perspective, which breaks builds with older compilers.
+;;;
+;;; To address these problems, we always write a fresh go.mod file based on
+;;; the build-inputs. There is no guarantee that there even is a go.mod file
+;;; in the source, especially for older projects. Go build uses this file to
+;;; "download" from our just-assembled goproxy, which makes it happy. This
+;;; also clears any toolchain directive which makes the build accept the go
+;;; compiler through build-inputs. We populate the goproxy with just-in-time
+;;; built zips, version, and info files. This is a separate phase so that
+;;; additional build steps can be added between building the proxy and running go
+;;; build.
+;;;
+;;; The build system is compatible with go-build-system, in the sense that
+;;; go-build-system can be used as build-inputs, and vice versa, because they
+;;; both use the same $out/source/.
+;;;
+;;; We re-used compiled packages. The Go build system creates a
+;;; content-addressable build cache, which we install into build output, and
+;;; use to seed downstream builds. Go programs are (mostly) statically
+;;; linked, so this is roughly equivalent of installing lib.a. Note that this
+;;; only works when the build-input is built with go-module-build-system.
+;;;
+;;; [1] https://go.dev/ref/mod#goproxy-protocol
+;;;
+;;; Code:
+
+(define (find-single-file dir regex)
+ "Find the file in DIR matching the REGEX, and fail unless there is
+exactly one match."
+ (let ((files (find-files dir regex #:directories? #f)))
+ (unless (eq? 1 (length files))
+ (error "Expected exactly one file matching pattern, found:" files))
+ (car files)))
+
+(define (go-path-escape path)
+ "Escape a module path by replacing every uppercase letter with an
+exclamation mark followed with its lowercase equivalent, as per the module
+Escaped Paths specification (see:
+https://godoc.org/golang.org/x/mod/module#hdr-Escaped_Paths)."
+ (define (escape occurrence)
+ (string-append "!" (string-downcase (match:substring occurrence))))
+ (regexp-substitute/global #f "[A-Z]" path 'pre escape 'post))
+
+(define (call-with-append-file path f)
+ "call-with-output-file, but appends to the file if it exists rather
+than truncating it"
+ (let ((file (open-file path "a")))
+ (f file)
+ (close file)))
+
+(define (set-cache-action-epoch f)
+ "Set go build cache action entry timestamp to 0
+
+The go build cache action entries (xxxx-a) record a timestamp, which
+would break reproducibility of the build cache. Set it to all-zeros."
+ ;; The file has 5 columns, tand the timestamp is the rightmost one
+ ;; <version> <action-id> <output-id> <size> <timestamp>
+ ;;
+ ;; The timestamp seems to be in nanoseconds since epoch. We use
+ ;; replacement to avoid potential problems with whitespace
+ ;; sensitivity.
+ (let* ((file-line (read-first-line f))
+ (start-timestamp (string-skip-right file-line char-set:digit))
+ (end-timestamp (string-length file-line))
+ (base-line (substring file-line 0 start-timestamp))
+ (zero-timestamp (make-string (- end-timestamp start-timestamp) #\0)))
+ (call-with-output-file f
+ (lambda (port)
+ (format port "~a~a~%" base-line zero-timestamp)))))
+
+(define (copy-nonlink-recursively src-dir dst-dir)
+ "Recursively install files from src/ to dst/, skipping symlinks"
+ (copy-recursively src-dir dst-dir
+ #:log #f
+ #:copy-file
+ (lambda (src dst)
+ (unless (symbolic-link? src)
+ (copy-file src dst)))))
+
+(define (make-tags tags)
+ "Construct a -tags argument list.
+
+We accept tags both as a single string and a list of tags. go expects
+-tags tag1,tag2,..."
+ (cond ((and (list? tags) (not (null? tags)))
+ (list "-tags" (string-join tags ",")))
+ ((string? tags) (list "-tags" tags))
+ (else '())))
+
+(define (re-init-module module-path)
+ "Create a fresh go.mod file, replacing an old one if it exists."
+ ;; Wipe the go.mod if it exists, then create a new one. We might
+ ;; not use the exact same input set (versions or even modules) as
+ ;; upstream, e.g. when splitting an upstream package into multiple
+ ;; parts.
+ ;;
+ ;; Delete all go.sum files, if they exist. We do our own
+ ;; checksums so there is no safety here, and since our packages
+ ;; are differently sourced (and maybe differently versioned) they
+ ;; won't match upstream checksums.
+ (when (file-exists? "go.mod") (delete-file "go.mod"))
+ (when (file-exists? "go.sum") (delete-file "go.sum"))
+ (invoke/quiet "go" "mod" "init" module-path))
+
+(define (read-first-line f)
+ "Read the first line in a file"
+ (call-with-input-file f read-line))
+
+(define (filter-go-inputs inputs)
+ "Return the store paths of go library inputs.
+
+Inputs is a list of ('pkg' 'store-path') pairs, and returns a list of
+store paths.
+
+((zip . /gnu/store/x1c9w6dnmk23mpdfg08zyq379q26nd88-zip-3.0)
+ (unzip . /gnu/store/fmli224wbxrz1n0i2lhz6gy8a1ydcbp3-unzip-6.0)
+ (go-github-com-stretchr-testify . /gnu/store/7p6zk3zka35g3699b9kfl0njzwykimjm-go-github-com-stretchr-testify-1.10.0)
+ (go-golang-org-x-tools . /gnu/store/ax54x3d7fyywbppqvf0gmavsmxkz0h03-go-golang-org-x-tools-0.25.0))
+
+->
+
+(/gnu/store/7p6zk3zka35g3699b9kfl0njzwykimjm-go-github-com-stretchr-testify-1.10.0
+ /gnu/store/ax54x3d7fyywbppqvf0gmavsmxkz0h03-go-golang-org-x-tools-0.25.0)
+
+Sources installed with go-build-system and go-module-build-system have
+a /src directory. Packages realistically have dozens of inputs (the go
+compiler, coreutils, etc.) so this filtering is very much necessary."
+ (map cdr
+ (filter
+ (lambda (input)
+ (and (string-prefix? "go-" (car input))
+ (directory-exists? (string-append (cdr input) "/src"))))
+ inputs)))
+
+(define* (infer-module-path-from-dir dir #:optional (subdir "src"))
+ "Infer the go module 'github.com/user/module' from a store path.
+
+DIR should be a a store path <store>/<pkg>
+
+<store>/<pkg>/src/github.com/user/module/... -> github.com/user/module
+
+By default, this function assumes the store path is a go-build-system
+or go-module-build-system installed package with the sources installed
+under DIR/SUBDIR."
+ (with-directory-excursion (string-append dir "/" subdir)
+ (string-trim
+ (car
+ (sort! (map dirname (find-files "." #:fail-on-error? #t))
+ (lambda (x y) (< (string-length x) (string-length y)))))
+ char-set:punctuation)))
+
+(define* (module-path-from-file #:optional (go.mod "go.mod"))
+ "Read the module-path from a go.mod file.
+
+This assumes that go.mod exists and is well-defined. This can not be
+assumed in general until after the build stage, in which case go.mod
+should always have been generated in the root dir."
+ (cadr (string-split (read-first-line go.mod) #\space)))
+
+(define* (prepare-sources #:key module-path #:allow-other-keys)
+ ;; Remove any go.mod and go.sum files. The builder will write its
+ ;; own based on the build inputs, and any upstream checksums will be
+ ;; wrong. This has to be done before setup-proxy as it would detect
+ ;; the source dir's go.mod and go.sum files and fail because of
+ ;; checksum mismatches. It is in its own phase so that it can be
+ ;; overriden, if necessary.
+ ;;
+ ;; We want to init it with go init module so that it records the go
+ ;; version in order to not fall back to too old go versions:
+ ;; function instantiation requires go1.18 or later (-lang was set
+ ;; to go1.16; check go.mod)
+ ;;
+ ;; If the module path is explicitly set, use it. Otherwise,
+ ;; infer it from the go.mod file. If the go.mod file does not
+ ;; exist and module-path is not specified, fail the build.
+(let ((module-path (or module-path (module-path-from-file))))
+ (for-each delete-file (find-files "." "go\\.mod$"))
+ (for-each delete-file (find-files "." "go\\.sum$"))
+ (re-init-module module-path)))
+
+(define* (setup-go-env #:key outputs cgo? environment-variables
+ goarch goos #:allow-other-keys)
+ (let* ((go-proxy (string-append (getcwd) "/guix-go/proxy"))
+ (go-cache (string-append (getcwd) "/guix-go/cache"))
+ (go-mod-cache (string-append (getcwd) "/guix-go/modcache")))
+ (setenv "GOSUMDB" "off")
+ (setenv "GOPROXY" (string-append "file://" go-proxy))
+ (setenv "GOCACHE" go-cache)
+ (setenv "GOMODCACHE" go-mod-cache)
+ (setenv "GOPATH" (string-append (assoc-ref outputs "out")))
+ (when cgo?
+ (setenv "CGO_ENABLED" "1"))
+
+ (setenv "GOARCH" (or goarch (getenv "GOHOSTARCH")))
+ (setenv "GOOS" (or goos (getenv "GOHOSTOS")))
+ (match goarch
+ ("arm"
+ (setenv "GOARM" "7"))
+ ((or "mips" "mipsel")
+ (setenv "GOMIPS" "hardfloat"))
+ ((or "mips64" "mips64le")
+ (setenv "GOMIPS64" "hardfloat"))
+ ((or "ppc64" "ppc64le")
+ (setenv "GOPPC64" "power8"))
+ (_ #t))
+
+ (for-each
+ (lambda (var) (setenv (car var) (cdr var)))
+ environment-variables)))
+
+(define (module-version-or-synthesized mod-input-path module-path)
+ "Resolve module or synthesize module version.
+
+Figure out what module version to use for go get to resolve, either
+the package version or a special syntesized one.
+
+This addresses a quirk of go module paths and the go module system. Go
+expects that if a package has a version >= v2.x.y, the module path is
+module/v2. When splitting a large package into smaller libraries that
+share import prefix, the module path no longer ends with the major
+version, and go get complains:
+
+ invalid version: should be v0 or v1, not v4
+
+The actual package versions used during the build matters little, and
+is an implementation detail for the builder. For the packages with
+version >= 2 with an \"unversioned\" module path, synthesize the special
+version 0.0.1-guix.
+
+MOD-INPUT-PATH should be the store path of the module as returned by
+filter-go-inputs. MODULE-PATH should be the module path as it is
+written in the go.mod, for example:
+
+(module-version-or-synthesized
+ \"/gnu/store/2v69cskzdjininks376wlw9cq3dv2gd1-go-github-com-stretchr-objx-0.5.2\"
+ \"github.com/stretchr/objx\")
+"
+ (let* ((_ pkg-version (package-name->name+version
+ (strip-store-file-name mod-input-path)))
+ (dir-version (basename module-path)))
+ (if (and (string-match "^[2-9]+\\." pkg-version)
+ (not (string-match "v[2-9]+$" dir-version)))
+ "0.0.1-guix"
+ pkg-version)))
+
+(define* (setup-goproxy #:key inputs #:allow-other-keys)
+ (let* (;; Remove file:// from the goproxy, we want the dir
+ (go-proxy (substring (getenv "GOPROXY") 7)))
+ (mkdir-p go-proxy)
+ (for-each
+ (lambda (mod-input-path)
+ (let* ((store-path (string-append mod-input-path "/src"))
+ (module-path (infer-module-path-from-dir mod-input-path))
+ (source-dir (string-append store-path "/" module-path))
+ (version (module-version-or-synthesized mod-input-path module-path))
+ (module-dir (format #f "~a <at> v~a" module-path version))
+ (proxy-dir (format #f "~a/~a/@v" go-proxy (go-path-escape module-path)))
+ (proxy/mod (string-append proxy-dir "/v" version ".mod"))
+ (proxy/zip (string-append proxy-dir "/v" version ".zip"))
+ (tmp-dir "guix-tmp")
+ (tmp-module (string-append tmp-dir "/" module-dir))
+ (tmp-mod (string-append tmp-module "/go.mod")))
+
+ ;; In some cases a module will show up twice, e.g. when
+ ;; breaking cyclic dependencies. In that case, don't install
+ ;; the second version. In that case the inputs have different
+ ;; store paths, and we can't rely at all on the package name.
+ (unless (file-exists? (string-append proxy-dir "/v" version ".mod"))
+ (mkdir-p tmp-module)
+ (copy-recursively source-dir tmp-module #:log #f)
+
+ ;; Delete all go.mod and go.sum files, and re-write the
+ ;; root one without dependencies and toolchain directives.
+ ;; Note that we cannot use re-init-module here. If there is
+ ;; a go.mod in the project root, it would be detected by
+ ;; re-init-module, and if it happened to contain a
+ ;; toolchain directive it would infect this module here.
+ (for-each delete-file (find-files tmp-module "go\\.mod$"))
+ (for-each delete-file (find-files tmp-module "go\\.sum$"))
+ (with-directory-excursion tmp-module
+ (re-init-module module-path))
+ (for-each
+ (lambda (f) (utime f 0 0 0 0 AT_SYMLINK_NOFOLLOW))
+ (find-files tmp-dir #:directories? #t))
+
+ ;; We need the -D flag, because go mod fails hard on any path
+ ;; that does not begin with $module <at> version, even if for
+ ;; sub-prefixes of $module
+ (mkdir-p proxy-dir)
+ (with-directory-excursion tmp-dir
+ (invoke "zip" "-r" "-q" "-o" "-D" "-X" proxy/zip "."))
+
+ (copy-file tmp-mod proxy/mod)
+ (delete-file-recursively tmp-dir)
+ (call-with-output-file (string-append proxy-dir "/v" version ".info")
+ (lambda (f) (format f "{~s:\"v~a\"}~%" "Version" version)))
+
+ (call-with-append-file (string-append proxy-dir "/list")
+ (lambda (f) (format f "v~a~%" version))))))
+ (filter-go-inputs inputs))))
+
+;; These paths must be consistent across different stages, so use
+;; symbols for them to ensure they're consistent
+(define guix-install-cache "guix-out-cache")
+(define var-build-cache "/var/cache/go/build")
+
+(define* (setup-gocache #:key inputs #:allow-other-keys)
+ (define (search-input-directories dir go-inputs)
+ (filter directory-exists?
+ (map (lambda (store) (string-append store "/" dir))
+ go-inputs)))
+
+ (union-build (getenv "GOCACHE")
+ (search-input-directories var-build-cache
+ (filter-go-inputs inputs))
+ ;; Creating all directories isn't that bad, because
+ ;; there are only ever 256 of them.
+ #:create-all-directories? #t
+ #:log-port (%make-void-port "w")))
+
+(define* (build #:key inputs go-flags tags build-targets
+ install-targets trimpath? build-output-dir? skip-build?
+ install-cache? install-source? parallel-build?
+ #:allow-other-keys)
+ (setenv "GOMAXPROCS"
+ (number->string
+ (if parallel-build? (parallel-job-count) 1)))
+
+ (for-each
+ (lambda (store-path)
+ (let* ((module (infer-module-path-from-dir store-path))
+ (version (module-version-or-synthesized store-path module)))
+ (invoke "go" "get" (string-append module "@v" version))))
+ (filter-go-inputs inputs))
+ ;; go.mod and go.sum have had their timestamps updated by go get,
+ ;; which will be snapshotted in the build cache and break it. From
+ ;; here on out these files should not need to change, so fix the
+ ;; timestamp. The sum will only exist if there are any
+ ;; dependencies.
+ (utime "go.mod" 0 0 0 0)
+ (when (file-exists? "go.sum")
+ (utime "go.sum" 0 0 0 0))
+
+ ;; If -o is used it must be the first flag to build. This flag is
+ ;; necessary when there is a command (program) with the same name
+ ;; as its directory, e.g. info/main.go instead of cmd/info.go
+ (unless skip-build?
+ (for-each
+ (lambda target
+ (apply invoke "go" "build"
+ (append
+ (if build-output-dir?
+ (list "-o" (string-append (or (getenv "TMP") "/tmp")
+ "/go-build/")) '())
+ go-flags (make-tags tags)
+ (if trimpath? '("-trimpath") '())
+ target)))
+ (append build-targets install-targets)))
+
+ ;; Snapshot the cache before running tests. It is not interesting
+ ;; to snapshot test artifacts, and they may pollute the cache with
+ ;; non-reproducible artifacts.
+ (when (and install-cache? (not skip-build?))
+ (mkdir-p guix-install-cache)
+ (with-directory-excursion guix-install-cache
+ (copy-nonlink-recursively (getenv "GOCACHE") ".")
+ (delete-file "trim.txt")
+ (delete-file "README")
+ (for-each set-cache-action-epoch (find-files "." "-a$")))))
+
+(define* (check #:key inputs tests? go-flags tags test-flags
+ test-targets module-path parallel-tests? trimpath?
+ #:allow-other-keys)
+ (when tests?
+ (let ((njobs (if parallel-tests? (parallel-job-count) 1)))
+ (setenv "GOMAXPROCS" (number->string njobs))
+ (for-each
+ (lambda target
+ (apply invoke (append
+ (list "go" "test") go-flags (make-tags tags)
+ (if trimpath? '("-trimpath") '())
+ test-flags target)))
+ test-targets))))
+
+(define* (install #:key source inputs outputs target go-flags tags
+ install-targets install-cache? install-source?
+ trimpath? #:allow-other-keys)
+ (for-each
+ (lambda target
+ (apply invoke (append (list "go" "install") go-flags (make-tags tags)
+ (if trimpath? '("-trimpath") '())
+ target)))
+ install-targets)
+
+ ;; When cross-compiling, binaries are installed into /bin/<arch>, so we need
+ ;; to move them up to /bin. However, when *not* cross-compiling they end up
+ ;; in /bin even when GOOS and GOARCH are set.
+ (when target
+ (let* ((bin (string-append (assoc-ref outputs "out") "/bin/"))
+ (platform (string-append (getenv "GOOS") "_" (getenv "GOARCH")))
+ (arch-dir (string-append bin platform)))
+ (when (directory-exists? arch-dir)
+ (copy-recursively arch-dir bin)
+ (delete-file-recursively arch-dir))))
+
+ (when (directory-exists? guix-install-cache)
+ (with-directory-excursion guix-install-cache
+ (copy-recursively
+ "." (string-append (assoc-ref outputs "out") var-build-cache)
+ #:log #f)))
+
+ (when install-source?
+ (let* ((out (assoc-ref outputs "out"))
+ (name version (package-name->name+version (assoc-ref outputs "out")))
+ (module-line (read-first-line "go.mod"))
+ (module-path (cadr (string-split module-line #\space)))
+ (dst (format #f "~a/src/~a" out module-path)))
+ (copy-recursively source dst #:log #f)
+ (when (file-exists? (string-append dst "/go.mod"))
+ (make-file-writable (string-append dst "/go.mod")))
+ (install-file "go.mod" dst))))
+
+(define %standard-phases
+ (modify-phases gnu:%standard-phases
+ (delete 'bootstrap)
+ (delete 'configure)
+ (delete 'patch-generated-file-shebangs)
+ (add-after 'unpack 'prepare-sources prepare-sources)
+ (add-before 'build 'setup-go-env setup-go-env)
+ (add-after 'setup-go-env 'setup-goproxy setup-goproxy)
+ (add-after 'setup-goproxy 'setup-gocache setup-gocache)
+ (replace 'build build)
+ (replace 'check check)
+ (replace 'install install)))
+
+(define* (go-module-build #:key inputs (phases %standard-phases)
+ #:allow-other-keys #:rest args)
+ "Go Module Build System"
+ (apply gnu:gnu-build #:inputs inputs #:phases phases args))
+
+;;; go-module-build-system.scm ends here
--
2.39.5
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997,2003 nCipher Corporation Ltd,
1994-97 Ian Jackson.