GNU bug report logs - #27003
[PATCH 0/3] Generalized wrap phase for perl, python.

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 27003 in the body.
You can then email your comments to 27003 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Alex Sassmannshausen <alex.sassmannshausen <at> gmail.com>
To: bug-guix <at> gnu.org
Cc: Alex Sassmannshausen <alex <at> pompo.co>
Subject: [PATCH 0/3] Generalized wrap phase for perl, python.
Date: Sat, 20 May 2017 11:37:42 +0200

Hello,

This patch series goes back some time, but the need for it has not
disappeared.

It implements a general `wrap` phase that can be used in individual build
systems to make sure binaries in those languages have access to all their
propagated inputs at runtime, by setting an appropriate environment variable.

I tested it against perl and python packages.

It will need to go in core-updates due to the number of packages that will
need to be rebuilt.

Best wishes,

Alex

Alex Sassmannshausen (3):
  build/utils: Add 'program-wrapper'.
  build/perl-build-system: Add 'wrap' phase.
  build/python-build-system: Refactor 'wrap'.

 guix/build/perl-build-system.scm   | 15 +++++++++++++--
 guix/build/python-build-system.scm | 36 ++++++++----------------------------
 guix/build/utils.scm               | 36 ++++++++++++++++++++++++++++++++++++
 3 files changed, 57 insertions(+), 30 deletions(-)

-- 
2.12.2

Message #8 received at 27003 <at> debbugs.gnu.org (full text, mbox):

From: Alex Sassmannshausen <alex.sassmannshausen <at> gmail.com>
To: 27003 <at> debbugs.gnu.org
Cc: Alex Sassmannshausen <alex <at> pompo.co>
Subject: [PATCH 1/3] build/utils: Add 'program-wrapper'.
Date: Sat, 20 May 2017 11:40:28 +0200

* guix/build/utils.scm (program-wrapper): New procedure.
---
 guix/build/utils.scm | 36 ++++++++++++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)

diff --git a/guix/build/utils.scm b/guix/build/utils.scm
index e8efb0653..af5583651 100644
--- a/guix/build/utils.scm
+++ b/guix/build/utils.scm
@@ -78,6 +78,7 @@
             patch-/usr/bin/file
             fold-port-matches
             remove-store-references
+            program-wrapper
             wrap-program
 
             locale-category->string))
@@ -956,6 +957,41 @@ known as `nuke-refs' in Nixpkgs."
                              (put-u8 out (char->integer char))
                              result))))))
 
+(define (program-wrapper path-proc env-var)
+  "Return a procedure, which, invoked as part of a 'wrap' phase, is capable of
+wrapping executables inside an environment in which ENV-VAR is correctly set.
+
+The string ENV-VAR is the name of the environmental variable we are setting
+for the executable we are wrapping.  PATH-PROC is a procedure of 2 arguments,
+'inputs' and 'outputs', returning the value that we should set ENV-VAR to.
+
+This is a specialized version of 'wrap-program' below, intended specifically
+to grant all executables that are part of our output access to all libraries
+that were declared in our inputs.  This is of use for languages such as Perl,
+Python and Guile."
+  (define (list-of-files dir)
+    (map (cut string-append dir "/" <>)
+         (or (scandir dir (lambda (f)
+                            (let ((s (stat (string-append dir "/" f))))
+                              (eq? 'regular (stat:type s)))))
+             '())))
+  (lambda* (#:key inputs outputs #:allow-other-keys)
+    (define bindirs
+      (append-map (match-lambda
+                    ((_ . dir)
+                     (list (string-append dir "/bin")
+                           (string-append dir "/sbin"))))
+                  outputs))
+    (define vars
+      `(,env-var prefix ,(cons (path-proc inputs outputs)
+                               (search-path-as-string->list
+                                (or (getenv env-var) "")))))
+    (for-each (lambda (dir)
+                (let ((files (list-of-files dir)))
+                  (for-each (cut wrap-program <> vars)
+                            files)))
+              bindirs)))
+
 (define* (wrap-program prog #:rest vars)
   "Make a wrapper for PROG.  VARS should look like this:
 
-- 
2.12.2

Message #11 received at 27003 <at> debbugs.gnu.org (full text, mbox):

From: Alex Sassmannshausen <alex.sassmannshausen <at> gmail.com>
To: 27003 <at> debbugs.gnu.org
Cc: Alex Sassmannshausen <alex <at> pompo.co>
Subject: [PATCH 2/3] build/perl-build-system: Add 'wrap' phase.
Date: Sat, 20 May 2017 11:40:29 +0200

* guix/build/perl-build-system.scm (wrap): New procedure.
  (%standard-phases): Add 'wrap' phase.
---
 guix/build/perl-build-system.scm | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)

diff --git a/guix/build/perl-build-system.scm b/guix/build/perl-build-system.scm
index b2024e440..5bf5b2523 100644
--- a/guix/build/perl-build-system.scm
+++ b/guix/build/perl-build-system.scm
@@ -19,7 +19,7 @@
 (define-module (guix build perl-build-system)
   #:use-module ((guix build gnu-build-system) #:prefix gnu:)
   #:use-module (guix build utils)
-  #:use-module (ice-9 match)
+  #:use-module (srfi srfi-1)
   #:export (%standard-phases
             perl-build))
 
@@ -51,6 +51,15 @@
     (format #t "running `perl' with arguments ~s~%" args)
     (zero? (apply system* "perl" args))))
 
+;; Use 'program-wrapper' to return an executable wrapper for perl.
+(define wrap
+  (program-wrapper
+   (lambda (inputs outputs)
+     (string-append (assoc-ref outputs "out") "/lib/perl5/site_perl/"
+                    ;; As in python, assume version at end of `perl' string.
+                    (last (string-split (assoc-ref inputs "perl") #\-))))
+   "PERL5LIB"))
+
 (define-syntax-rule (define-w/gnu-fallback* (name args ...) body ...)
   (define* (name args ... #:rest rest)
     (if (access? "Build" X_OK)
@@ -74,9 +83,11 @@
 
 (define %standard-phases
   ;; Everything is as with the GNU Build System except for the `configure',
-  ;; `build', `check', and `install' phases.
+  ;; `build', `check', and `install' phases.  We also add a `wrap' phase to
+  ;; wrap perl binaries with a complete PERL5LIB path.
   (modify-phases gnu:%standard-phases
     (replace 'install install)
+    (add-after 'install 'wrap wrap)
     (replace 'check check)
     (replace 'build build)
     (replace 'configure configure)))
-- 
2.12.2

Message #14 received at 27003 <at> debbugs.gnu.org (full text, mbox):

From: Alex Sassmannshausen <alex.sassmannshausen <at> gmail.com>
To: 27003 <at> debbugs.gnu.org
Cc: Alex Sassmannshausen <alex <at> pompo.co>
Subject: [PATCH 3/3] build/python-build-system: Refactor 'wrap'.
Date: Sat, 20 May 2017 11:40:30 +0200

* guix/build/python-build-system.scm (wrap): Use
  'wrap-language-program'.
---
 guix/build/python-build-system.scm | 36 ++++++++----------------------------
 1 file changed, 8 insertions(+), 28 deletions(-)

diff --git a/guix/build/python-build-system.scm b/guix/build/python-build-system.scm
index dd07986b9..30d01f8cb 100644
--- a/guix/build/python-build-system.scm
+++ b/guix/build/python-build-system.scm
@@ -184,34 +184,14 @@ when running checks after installing the package."
                          configure-flags)))
     (call-setuppy "install" params use-setuptools?)))
 
-(define* (wrap #:key inputs outputs #:allow-other-keys)
-  (define (list-of-files dir)
-    (map (cut string-append dir "/" <>)
-         (or (scandir dir (lambda (f)
-                            (let ((s (stat (string-append dir "/" f))))
-                              (eq? 'regular (stat:type s)))))
-             '())))
-
-  (define bindirs
-    (append-map (match-lambda
-                 ((_ . dir)
-                  (list (string-append dir "/bin")
-                        (string-append dir "/sbin"))))
-                outputs))
-
-  (let* ((out  (assoc-ref outputs "out"))
-         (python (assoc-ref inputs "python"))
-         (var `("PYTHONPATH" prefix
-                ,(cons (string-append out "/lib/python"
-                                      (get-python-version python)
-                                      "/site-packages")
-                       (search-path-as-string->list
-                        (or (getenv "PYTHONPATH") ""))))))
-    (for-each (lambda (dir)
-                (let ((files (list-of-files dir)))
-                  (for-each (cut wrap-program <> var)
-                            files)))
-              bindirs)))
+;; Use 'program-wrapper' to return an executable wrapper for python.
+(define wrap
+  (program-wrapper
+   (lambda (inputs outputs)
+     (string-append (assoc-ref outputs "out") "/lib/python"
+                    (get-python-version (assoc-ref inputs "python"))
+                    "/site-packages"))
+   "PYTHONPATH"))
 
 (define* (rename-pth-file #:key name inputs outputs #:allow-other-keys)
   "Rename easy-install.pth to NAME.pth to avoid conflicts between packages
-- 
2.12.2

Message #17 received at 27003 <at> debbugs.gnu.org (full text, mbox):

From: Marius Bakke <mbakke <at> fastmail.com>
To: Alex Sassmannshausen <alex.sassmannshausen <at> gmail.com>,
 27003 <at> debbugs.gnu.org
Cc: Arun Isaac <arunisaac <at> systemreboot.net>
Subject: Re: bug#27003: [PATCH 0/3] Generalized wrap phase for perl, python.
Date: Sat, 20 May 2017 22:23:21 +0200

[Message part 1 (text/plain, inline)]

Alex Sassmannshausen <alex.sassmannshausen <at> gmail.com> writes:

> Hello,
>
> This patch series goes back some time, but the need for it has not
> disappeared.
>
> It implements a general `wrap` phase that can be used in individual build
> systems to make sure binaries in those languages have access to all their
> propagated inputs at runtime, by setting an appropriate environment variable.

Cool, thanks a lot for working on this. CCing Arun who was working on
something similar.

AFAICT this is also subject to <https://bugs.gnu.org/25235>. Can you
have a look at it?

[signature.asc (application/pgp-signature, inline)]

Message #20 received at 27003 <at> debbugs.gnu.org (full text, mbox):

From: Arun Isaac <arunisaac <at> systemreboot.net>
To: Marius Bakke <mbakke <at> fastmail.com>
Cc: Alex Sassmannshausen <alex.sassmannshausen <at> gmail.com>,
 27003 <at> debbugs.gnu.org
Subject: Re: bug#27003: [PATCH 0/3] Generalized wrap phase for perl, python.
Date: Sun, 28 May 2017 00:34:32 +0530

>> It implements a general `wrap` phase that can be used in individual build
>> systems to make sure binaries in those languages have access to all their
>> propagated inputs at runtime, by setting an appropriate environment variable.
>
> CCing Arun who was working on something similar.

I haven't actually worked on this problem for a long while. I'll test
Alex's patches, and get back to you soon.

> AFAICT this is also subject to <https://bugs.gnu.org/25235>. Can you
> have a look at it?

Yes, that's the python wrap phase bug report. And this is the one for
perl: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=26275

Message #23 received at 27003 <at> debbugs.gnu.org (full text, mbox):

From: Arun Isaac <arunisaac <at> systemreboot.net>
To: Marius Bakke <mbakke <at> fastmail.com>
Cc: Alex Sassmannshausen <alex.sassmannshausen <at> gmail.com>,
 27003 <at> debbugs.gnu.org
Subject: Re: bug#27003: [PATCH 0/3] Generalized wrap phase for perl, python.
Date: Mon, 29 May 2017 00:38:24 +0530

Building on my system is taking a long time. I'll need some more time to
properly test these patches. But, if I understand correctly, these
patches still wrap binaries with paths of native-inputs. That problem
remains to be solved.

Message #26 received at 27003 <at> debbugs.gnu.org (full text, mbox):

From: Alex Sassmannshausen <alex <at> pompo.co>
To: Arun Isaac <arunisaac <at> systemreboot.net>
Cc: Marius Bakke <mbakke <at> fastmail.com>, 27003 <at> debbugs.gnu.org
Subject: Re: bug#27003: [PATCH 0/3] Generalized wrap phase for perl, python.
Date: Mon, 29 May 2017 15:45:16 +0200

Hi Arun,

Thank you for testing my patches.

Arun Isaac writes:

> Building on my system is taking a long time. I'll need some more time to
> properly test these patches. But, if I understand correctly, these
> patches still wrap binaries with paths of native-inputs. That problem
> remains to be solved.

Yes, I certainly did nothing to resolve that particular issue, so indeed
it remains to be solved.

I cannot currently make guarantees on when I'll be able to wrap my head
around the entire problem, so if you come to conclusions for next steps
on the basis of your testing, we should go with those.

Else I will revisit this issue when I have time (which might be a
while).

Best wishes,

Alex

Message #29 received at 27003 <at> debbugs.gnu.org (full text, mbox):

From: Arun Isaac <arunisaac <at> systemreboot.net>
To: alex <at> pompo.co
Cc: Marius Bakke <mbakke <at> fastmail.com>, 27003 <at> debbugs.gnu.org
Subject: Re: bug#27003: [PATCH 0/3] Generalized wrap phase for perl, python.
Date: Tue, 30 May 2017 19:15:49 +0530

> I cannot currently make guarantees on when I'll be able to wrap my head
> around the entire problem, so if you come to conclusions for next steps
> on the basis of your testing, we should go with those.
>
> Else I will revisit this issue when I have time (which might be a
> while).

I don't have any ideas on how to fix this. And, I don't understand Guix
internals very well. So, I'll wait for you or someone else to come up
with a solution. But, I can help with testing patches.

Message #32 received at 27003 <at> debbugs.gnu.org (full text, mbox):

From: Marius Bakke <mbakke <at> fastmail.com>
To: Arun Isaac <arunisaac <at> systemreboot.net>, alex <at> pompo.co
Cc: 27003 <at> debbugs.gnu.org
Subject: Re: bug#27003: [PATCH 0/3] Generalized wrap phase for perl, python.
Date: Tue, 30 May 2017 17:17:55 +0200

[Message part 1 (text/plain, inline)]

Arun Isaac <arunisaac <at> systemreboot.net> writes:

>> I cannot currently make guarantees on when I'll be able to wrap my head
>> around the entire problem, so if you come to conclusions for next steps
>> on the basis of your testing, we should go with those.
>>
>> Else I will revisit this issue when I have time (which might be a
>> while).
>
> I don't have any ideas on how to fix this. And, I don't understand Guix
> internals very well. So, I'll wait for you or someone else to come up
> with a solution. But, I can help with testing patches.

I'm not sure if #25235 should block this issue though, since it's a
useful change regardless. But, not handling it adds "technical debt",
and a (minor) problem that is not currently present in perl modules.

We do have until the next 'core-updates' to think about it though ;-)

[signature.asc (application/pgp-signature, inline)]

Message #35 received at 27003 <at> debbugs.gnu.org (full text, mbox):

From: Brice Waegeneire <brice <at> waegenei.re>
To: 27003 <at> debbugs.gnu.org
Subject: Re: bug#27003: [PATCH 0/3] Generalized wrap phase for perl, python.
Date: Sat, 21 Mar 2020 14:23:32 +0000

Hello,

ludo <at> gnu.org (Ludovic Courtès) writes:
> BTW, there’s also this patch series on this topic:
> 
>   https://issues.guix.info/issue/27003
> 
> What to do?

A `wrap-script' procedure has been merged as of 
0fb9a8df429a7b9f40610ff15baaff0d8e31e8cf
thanks to Ricardo's patch from https://issues.guix.info/issue/27003, it 
seems to
provide similar functionality as the `program-wrapper' suggested here. 
Should we close
this issue?

Brice.

Message #38 received at 27003 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Brice Waegeneire <brice <at> waegenei.re>
Cc: 27003 <at> debbugs.gnu.org
Subject: Re: bug#27003: [PATCH 0/3] Generalized wrap phase for perl, python.
Date: Sat, 21 Mar 2020 17:32:56 +0100

Hi,

Brice Waegeneire <brice <at> waegenei.re> skribis:

> ludo <at> gnu.org (Ludovic Courtès) writes:
>> BTW, there’s also this patch series on this topic:
>>
>>   https://issues.guix.info/issue/27003
>>
>> What to do?
>
> A `wrap-script' procedure has been merged as of
> 0fb9a8df429a7b9f40610ff15baaff0d8e31e8cf
> thanks to Ricardo's patch from https://issues.guix.info/issue/27003,
> it seems to
> provide similar functionality as the `program-wrapper' suggested
> here. Should we close
> this issue?

Yes, it seems that the patch that Alex initially submitted shares the
same goals as Ricardo’s ‘wrap-script’, so I guess we can close it.

Thanks for your work on bug triage!

Ludo’.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.