GNU bug report logs - #68423
[PATCH] guix: go-build-system: use trimpath go flag

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 68423 in the body.
You can then email your comments to 68423 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Ekaitz Zarraga <ekaitz <at> elenq.tech>
To: guix-patches <at> gnu.org
Cc: Picnoir <picnoir <at> alternativebit.fr>, Ekaitz Zarraga <ekaitz <at> elenq.tech>
Subject: [PATCH] guix: go-build-system: use trimpath go flag
Date: Sat, 13 Jan 2024 12:46:15 +0100

Go 1.13 introduced[1] a new "trimpath" flag in charge of removing the
references to the go toolchain in the produced ELFs. We used to remove
these references using the "remove-go-reference" function. This function
was executed after go install. By using this new trimpath flag, we don't
have to remove any store path from the ELFs produced by go install.

We're not using any go older than 1.13 anymore, it's safe to remove
these functions.

[1] https://go.dev/doc/go1.13

    -trimpath
        remove all file system paths from the resulting executable.
        Instead of absolute file system paths, the recorded file names
        will begin either a module path <at> version (when using modules),
        or a plain import path (when using the standard library, or
        GOPATH).

* guix/build/go-build-system.scm (build): Add -trimpath
(%standard-phases): Remove remove-go-references.
(remove-go-references): Remove.

Change-Id: Idcae366d226da5ce095693f81fd33133fd1d70d6
Co-authored-by: Picnoir <picnoir <at> alternativebit.fr>
---
 guix/build/go-build-system.scm | 59 +++-------------------------------
 1 file changed, 4 insertions(+), 55 deletions(-)

diff --git a/guix/build/go-build-system.scm b/guix/build/go-build-system.scm
index 7f25e05d0d..70ddcd07cd 100644
--- a/guix/build/go-build-system.scm
+++ b/guix/build/go-build-system.scm
@@ -6,6 +6,8 @@
 ;;; Copyright © 2020 Jakub Kądziołka <kuba <at> kadziolka.net>
 ;;; Copyright © 2020, 2021 Efraim Flashner <efraim <at> flashner.co.il>
 ;;; Copyright © 2021 Sarah Morgensen <iskarian <at> mgsn.dev>
+;;; Copyright © 2024 Ekaitz Zarraga <ekaitz <at> elenq.tech>
+;;; Copyright © 2024 Picnoir <picnoir <at> alternativebit.fr>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -90,7 +92,6 @@ (define-module (guix build go-build-system)
 ;; * Use Go modules [4]
 ;; * Re-use compiled packages [5]
 ;; * Avoid the go-inputs hack
-;; * Stop needing remove-go-references (-trimpath ? )
 ;; * Remove module packages, only offering the full Git repos? This is
 ;; more idiomatic, I think, because Go downloads Git repos, not modules.
 ;; What are the trade-offs?
@@ -265,6 +266,7 @@ (define* (build #:key import-path build-flags #:allow-other-keys)
               ;; Respectively, strip the symbol table and debug
               ;; information, and the DWARF symbol table.
               "-ldflags=-s -w"
+              "-trimpath"
               `(,@build-flags ,import-path)))
     (lambda (key . args)
       (display (string-append "Building '" import-path "' failed.\n"
@@ -304,58 +306,6 @@ (define* (install-license-files #:key unpack-path
                                                     unpack-path))
     (apply (assoc-ref gnu:%standard-phases 'install-license-files) args)))
 
-(define* (remove-store-reference file file-name
-                                  #:optional (store (%store-directory)))
-  "Remove from FILE occurrences of FILE-NAME in STORE; return #t when FILE-NAME
-is encountered in FILE, #f otherwise. This implementation reads FILE one byte at
-a time, which is slow. Instead, we should use the Boyer-Moore string search
-algorithm; there is an example in (guix build grafts)."
-  (define pattern
-    (string-take file-name
-                 (+ 34 (string-length (%store-directory)))))
-
-  (with-fluids ((%default-port-encoding #f))
-    (with-atomic-file-replacement file
-      (lambda (in out)
-        ;; We cannot use `regexp-exec' here because it cannot deal with
-        ;; strings containing NUL characters.
-        (format #t "removing references to `~a' from `~a'...~%" file-name file)
-        (setvbuf in 'block 65536)
-        (setvbuf out 'block 65536)
-        (fold-port-matches (lambda (match result)
-                             (put-bytevector out (string->utf8 store))
-                             (put-u8 out (char->integer #\/))
-                             (put-bytevector out
-                                             (string->utf8
-                                              "eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-"))
-                             #t)
-                           #f
-                           pattern
-                           in
-                           (lambda (char result)
-                             (put-u8 out (char->integer char))
-                             result))))))
-
-(define* (remove-go-references #:key allow-go-reference?
-                               inputs outputs #:allow-other-keys)
-  "Remove any references to the Go compiler from the compiled Go executable
-files in OUTPUTS."
-;; We remove this spurious reference to save bandwidth when installing Go
-;; executables. It would be better to not embed the reference in the first
-;; place, but I'm not sure how to do that. The subject was discussed at:
-;; <https://lists.gnu.org/archive/html/guix-devel/2017-10/msg00207.html>
-  (if allow-go-reference?
-    #t
-    (let ((go (assoc-ref inputs "go"))
-          (bin "/bin"))
-      (for-each (lambda (output)
-                  (when (file-exists? (string-append (cdr output)
-                                                     bin))
-                    (for-each (lambda (file)
-                                (remove-store-reference file go))
-                              (find-files (string-append (cdr output) bin)))))
-                outputs)
-      #t)))
 
 (define %standard-phases
   (modify-phases gnu:%standard-phases
@@ -367,8 +317,7 @@ (define %standard-phases
     (replace 'build build)
     (replace 'check check)
     (replace 'install install)
-    (replace 'install-license-files install-license-files)
-    (add-after 'install 'remove-go-references remove-go-references)))
+    (replace 'install-license-files install-license-files)))
 
 (define* (go-build #:key inputs (phases %standard-phases)
                       #:allow-other-keys #:rest args)

base-commit: c0b303aaa3d6154acbe054120d11467eb98e6d33
-- 
2.41.0

Message #8 received at 68423 <at> debbugs.gnu.org (full text, mbox):

From: Sharlatan Hellseher <sharlatanus <at> gmail.com>
To: Ekaitz Zarraga <ekaitz <at> elenq.tech>
Cc: 68423 <at> debbugs.gnu.org, Picnoir <picnoir <at> alternativebit.fr>,
 Katherine Cox-Buday <cox.katherine.e+guix <at> gmail.com>
Subject: Re: [bug#68423] [PATCH] guix: go-build-system: use trimpath go flag
Date: Sat, 13 Jan 2024 12:58:34 +0000

[Message part 1 (text/plain, inline)]

Hi Ekaitz,

Thanks for the patch.

I think this is more suitable for core updates brunch as triggers rebuild
the world for all golang packages as far as I see.

Regards,
Oleg

On Sat, 13 Jan 2024, 11:48 Ekaitz Zarraga, <ekaitz <at> elenq.tech> wrote:

> Go 1.13 introduced[1] a new "trimpath" flag in charge of removing the
> references to the go toolchain in the produced ELFs. We used to remove
> these references using the "remove-go-reference" function. This function
> was executed after go install. By using this new trimpath flag, we don't
> have to remove any store path from the ELFs produced by go install.
>
> We're not using any go older than 1.13 anymore, it's safe to remove
> these functions.
>
> [1] https://go.dev/doc/go1.13
>
>     -trimpath
>         remove all file system paths from the resulting executable.
>         Instead of absolute file system paths, the recorded file names
>         will begin either a module path <at> version (when using modules),
>         or a plain import path (when using the standard library, or
>         GOPATH).
>
> * guix/build/go-build-system.scm (build): Add -trimpath
> (%standard-phases): Remove remove-go-references.
> (remove-go-references): Remove.
>
> Change-Id: Idcae366d226da5ce095693f81fd33133fd1d70d6
> Co-authored-by: Picnoir <picnoir <at> alternativebit.fr>
> ---
>  guix/build/go-build-system.scm | 59 +++-------------------------------
>  1 file changed, 4 insertions(+), 55 deletions(-)
>
> diff --git a/guix/build/go-build-system.scm
> b/guix/build/go-build-system.scm
> index 7f25e05d0d..70ddcd07cd 100644
> --- a/guix/build/go-build-system.scm
> +++ b/guix/build/go-build-system.scm
> @@ -6,6 +6,8 @@
>  ;;; Copyright © 2020 Jakub Kądziołka <kuba <at> kadziolka.net>
>  ;;; Copyright © 2020, 2021 Efraim Flashner <efraim <at> flashner.co.il>
>  ;;; Copyright © 2021 Sarah Morgensen <iskarian <at> mgsn.dev>
> +;;; Copyright © 2024 Ekaitz Zarraga <ekaitz <at> elenq.tech>
> +;;; Copyright © 2024 Picnoir <picnoir <at> alternativebit.fr>
>  ;;;
>  ;;; This file is part of GNU Guix.
>  ;;;
> @@ -90,7 +92,6 @@ (define-module (guix build go-build-system)
>  ;; * Use Go modules [4]
>  ;; * Re-use compiled packages [5]
>  ;; * Avoid the go-inputs hack
> -;; * Stop needing remove-go-references (-trimpath ? )
>  ;; * Remove module packages, only offering the full Git repos? This is
>  ;; more idiomatic, I think, because Go downloads Git repos, not modules.
>  ;; What are the trade-offs?
> @@ -265,6 +266,7 @@ (define* (build #:key import-path build-flags
> #:allow-other-keys)
>                ;; Respectively, strip the symbol table and debug
>                ;; information, and the DWARF symbol table.
>                "-ldflags=-s -w"
> +              "-trimpath"
>                `(,@build-flags ,import-path)))
>      (lambda (key . args)
>        (display (string-append "Building '" import-path "' failed.\n"
> @@ -304,58 +306,6 @@ (define* (install-license-files #:key unpack-path
>                                                      unpack-path))
>      (apply (assoc-ref gnu:%standard-phases 'install-license-files) args)))
>
> -(define* (remove-store-reference file file-name
> -                                  #:optional (store (%store-directory)))
> -  "Remove from FILE occurrences of FILE-NAME in STORE; return #t when
> FILE-NAME
> -is encountered in FILE, #f otherwise. This implementation reads FILE one
> byte at
> -a time, which is slow. Instead, we should use the Boyer-Moore string
> search
> -algorithm; there is an example in (guix build grafts)."
> -  (define pattern
> -    (string-take file-name
> -                 (+ 34 (string-length (%store-directory)))))
> -
> -  (with-fluids ((%default-port-encoding #f))
> -    (with-atomic-file-replacement file
> -      (lambda (in out)
> -        ;; We cannot use `regexp-exec' here because it cannot deal with
> -        ;; strings containing NUL characters.
> -        (format #t "removing references to `~a' from `~a'...~%" file-name
> file)
> -        (setvbuf in 'block 65536)
> -        (setvbuf out 'block 65536)
> -        (fold-port-matches (lambda (match result)
> -                             (put-bytevector out (string->utf8 store))
> -                             (put-u8 out (char->integer #\/))
> -                             (put-bytevector out
> -                                             (string->utf8
> -
> "eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-"))
> -                             #t)
> -                           #f
> -                           pattern
> -                           in
> -                           (lambda (char result)
> -                             (put-u8 out (char->integer char))
> -                             result))))))
> -
> -(define* (remove-go-references #:key allow-go-reference?
> -                               inputs outputs #:allow-other-keys)
> -  "Remove any references to the Go compiler from the compiled Go
> executable
> -files in OUTPUTS."
> -;; We remove this spurious reference to save bandwidth when installing Go
> -;; executables. It would be better to not embed the reference in the first
> -;; place, but I'm not sure how to do that. The subject was discussed at:
> -;; <https://lists.gnu.org/archive/html/guix-devel/2017-10/msg00207.html>
> -  (if allow-go-reference?
> -    #t
> -    (let ((go (assoc-ref inputs "go"))
> -          (bin "/bin"))
> -      (for-each (lambda (output)
> -                  (when (file-exists? (string-append (cdr output)
> -                                                     bin))
> -                    (for-each (lambda (file)
> -                                (remove-store-reference file go))
> -                              (find-files (string-append (cdr output)
> bin)))))
> -                outputs)
> -      #t)))
>
>  (define %standard-phases
>    (modify-phases gnu:%standard-phases
> @@ -367,8 +317,7 @@ (define %standard-phases
>      (replace 'build build)
>      (replace 'check check)
>      (replace 'install install)
> -    (replace 'install-license-files install-license-files)
> -    (add-after 'install 'remove-go-references remove-go-references)))
> +    (replace 'install-license-files install-license-files)))
>
>  (define* (go-build #:key inputs (phases %standard-phases)
>                        #:allow-other-keys #:rest args)
>
> base-commit: c0b303aaa3d6154acbe054120d11467eb98e6d33
> --
> 2.41.0
>
>
>
>

[Message part 2 (text/html, inline)]

Message #11 received at 68423 <at> debbugs.gnu.org (full text, mbox):

From: Ekaitz Zarraga <ekaitz <at> elenq.tech>
To: Sharlatan Hellseher <sharlatanus <at> gmail.com>
Cc: 68423 <at> debbugs.gnu.org, Picnoir <picnoir <at> alternativebit.fr>,
 Katherine Cox-Buday <cox.katherine.e+guix <at> gmail.com>
Subject: Re: [bug#68423] [PATCH] guix: go-build-system: use trimpath go flag
Date: Sat, 13 Jan 2024 14:31:39 +0100

Hi,

On 2024-01-13 13:58, Sharlatan Hellseher wrote:
> Hi Ekaitz,
> 
> Thanks for the patch.
> 
> I think this is more suitable for core updates brunch as triggers 
> rebuild the world for all golang packages as far as I see.

Yeah, probably.

We also thought after the patch that would be interesting to provide a 
way to disable this behavior. Or maybe it's ok?

With the previous alternative the packager would just remove the phase, 
but hardcoding the option as we did doesn't provide any way to do 
that...  Some kind of flag should be enough.

Are you ok with the patch, thought?

If this is ok we'll make those small changes and send a v2.

Thanks

Message #14 received at 68423 <at> debbugs.gnu.org (full text, mbox):

From: Sharlatan Hellseher <sharlatanus <at> gmail.com>
To: Ekaitz Zarraga <ekaitz <at> elenq.tech>
Cc: 68423 <at> debbugs.gnu.org, Picnoir <picnoir <at> alternativebit.fr>,
 Katherine Cox-Buday <cox.katherine.e+guix <at> gmail.com>
Subject: Re: [bug#68423] [PATCH] guix: go-build-system: use trimpath go flag
Date: Tue, 23 Jan 2024 20:34:50 +0000

[Message part 1 (text/plain, inline)]

Hi Ekaitz,

> With the previous alternative the packager would just remove
> the phase, but hardcoding the option as we did doesn't provide
> any way to do that...  Some kind of flag should be enough.

It would be great to have more control over golang build envars
and build flags, like it happen in gnu-build-system or cmake-build-system
via #:make-flags and #:configure-flags.

Having some common defaults hardcoded and kind of #:build-flags as a way to
path anything else to compiler.

There is a similar thread referring go test flags, I've listed all in here:
https://mail.gnu.org/archive/html/guix-devel/2024-01/msg00235.html

Thanks,
Oleg

[Message part 2 (text/html, inline)]

Message #19 received at 68423-done <at> debbugs.gnu.org (full text, mbox):

From: Sharlatan Hellseher <sharlatanus <at> gmail.com>
To: 68423-done <at> debbugs.gnu.org
Subject: [PATCH] guix: go-build-system: use trimpath go flag
Date: Thu, 07 Mar 2024 12:21:06 +0000

[Message part 1 (text/plain, inline)]

Pushed as 998588599aa46e384b664ec576ab124b83fc16bd to go-team.

--
Oleg

[signature.asc (application/pgp-signature, inline)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.