GNU bug report logs - #46028
[PATCH core-updates 0/2] Farewell to PYTHONPATH.

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 46028 in the body.
You can then email your comments to 46028 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: guix-patches <at> gnu.org
Cc: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Subject: [PATCH core-updates 0/2] Farewell to PYTHONPATH.
Date: Thu, 21 Jan 2021 23:48:16 -0500

Farewell to PYTHONPATH.

Maxim Cournoyer (2):
  utils: Add a version-minor procedure.
  gnu: python: Replace PYTHONPATH by GUIX_PYTHONPATH_X_Y.

 gnu/packages/python.scm | 37 +++++++++++++++++++++++++++++++++----
 guix/utils.scm          | 15 ++++++++++++---
 2 files changed, 45 insertions(+), 7 deletions(-)

-- 
2.29.2

Message #8 received at 46028 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: 46028 <at> debbugs.gnu.org
Cc: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Subject: [PATCH core-updates 1/2] utils: Add a version-minor procedure.
Date: Thu, 21 Jan 2021 23:51:03 -0500

* guix/utils.scm (version-components): New procedure.
(version-prefix): Use it.
(version-major): Likewise.
(version-minor): New procedure.
---
 guix/utils.scm | 15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)

diff --git a/guix/utils.scm b/guix/utils.scm
index 678954dbfa..cd4e595f7e 100644
--- a/guix/utils.scm
+++ b/guix/utils.scm
@@ -8,7 +8,7 @@
 ;;; Copyright © 2017 Mathieu Othacehe <m.othacehe <at> gmail.com>
 ;;; Copyright © 2018, 2020 Marius Bakke <marius <at> gnu.org>
 ;;; Copyright © 2020 Efraim Flashner <efraim <at> flashner.co.il>
-;;; Copyright © 2020 Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
+;;; Copyright © 2020, 2021 Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -93,6 +93,7 @@
             version-major+minor+point
             version-major+minor
             version-major
+            version-minor
             guile-version>?
             version-prefix?
             string-replace-substring
@@ -573,10 +574,14 @@ or '= when they denote equal versions."
               ((negative? result) '<)
               (else '=))))))
 
+(define (version-components version-string)
+  "Return the dot-separated version components as a list."
+  (string-split version-string #\.))
+
 (define (version-prefix version-string num-parts)
   "Truncate version-string to the first num-parts components of the version.
 For example, (version-prefix \"2.1.47.4.23\" 3) returns \"2.1.47\""
-  (string-join (take (string-split version-string #\.) num-parts) "."))
+  (string-join (take (version-components version-string) num-parts) "."))
 
 (define (version-major+minor+point version-string)
   "Return \"major>.<minor>.<point>\", where major, minor and point are the
@@ -595,7 +600,11 @@ minor version numbers from version-string."
 
 (define (version-major version-string)
   "Return the major version number as string from the version-string."
-  (version-prefix version-string 1))
+  (first (version-components version-string)))
+
+(define (version-minor version-string)
+  "Return the minor version number as string from the version-string."
+  (second (version-components version-string)))
 
 (define (version>? a b)
   "Return #t when A denotes a version strictly newer than B."
-- 
2.29.2

Message #11 received at 46028 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: 46028 <at> debbugs.gnu.org
Cc: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Subject: [PATCH core-updates 2/2] gnu: python: Replace PYTHONPATH by
 GUIX_PYTHONPATH_X_Y.
Date: Thu, 21 Jan 2021 23:51:04 -0500

Using PYTHONPATH as a mean to discover the Python packages had the following
issues:

        1. It is not versioned, so different versions of Python would clash if
        installed in a shared profile.

        2. It would interfere with the host Python site on foreign
        distributions, sometimes preventing a a user to login their GDM
        session (!).

        3. It would take precedence over user installed Python packages
        installed through pip.

        4. It would leak into Python virtualenvs, which are supposed to create
        isolated Python environments.

This changes fixes the above issues by making use of a sitecustomize.py
module.  The newly introduced GUIX_PYTHONPATH_X_Y environment variable, where
X is the major and Y the minor version numbers, is read from the environment
and spliced in 'sys.path' just before Python's own site location, which
provides the expected behavior.

* gnu/packages/python.scm (python-2.7)[phases]{customize-site}: Add phase.
[native-search-paths]: Modify so that GUIX_PYTHONPATH_X_Y is used.
* gnu/packages/python.scm (python-3.9)[native-search-paths]: Likewise.
---
 gnu/packages/python.scm | 37 +++++++++++++++++++++++++++++++++----
 1 file changed, 33 insertions(+), 4 deletions(-)

diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index d3df1a4dab..79cea3d43a 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -313,7 +313,30 @@
                                     "/site-packages")))
                       (install-file tkinter.so target)
                       (delete-file tkinter.so)))))
-               #t))))))
+               #t)))
+         (add-after 'install 'customize-site
+           (lambda* (#:key outputs #:allow-other-keys)
+             (let* ((out (assoc-ref outputs "out"))
+                    (site-packages (string-append
+                                    out "/lib/python"
+                                    ,(version-major+minor version)
+                                    "/site-packages"))
+                    (pythonpath (format #f "GUIX_PYTHONPATH_~a_~a"
+                                        ,(version-major version)
+                                        ,(version-minor version)))
+                    (sitecustomize.py (string-append site-packages
+                                                     "/sitecustomize.py")))
+               (with-output-to-file sitecustomize.py
+                 (lambda _
+                   (format #t "\
+import os
+import sys
+
+# Insert the GUIX_PYTHONPATH_X_Y values just before Python's own site.
+sys_path_absolute = [os.path.realpath(p) for p in sys.path]
+index = sys_path_absolute.index(~s)
+sys.path = sys.path[:index] + os.environ[~s].split(':') + sys.path[index:]
+" site-packages pythonpath)))))))))
     (inputs
      `(("bzip2" ,bzip2)
        ("expat" ,expat)
@@ -334,8 +357,12 @@
              '())))
     (native-search-paths
      (list (search-path-specification
-            (variable "PYTHONPATH")
-            (files '("lib/python2.7/site-packages")))))
+            (variable (format #f "GUIX_PYTHONPATH_~a_~a"
+                              (version-major version)
+                              (version-minor version)))
+            (files (list (string-append "lib/python"
+                                        (version-major+minor version)
+                                        "/site-packages"))))))
     (home-page "https://www.python.org")
     (synopsis "High-level, dynamically-typed programming language")
     (description
@@ -472,7 +499,9 @@ data types.")
        ,@(package-native-inputs python-2)))
     (native-search-paths
      (list (search-path-specification
-            (variable "PYTHONPATH")
+            (variable (format #f "GUIX_PYTHONPATH_~a_~a"
+                              (version-major version)
+                              (version-minor version)))
             (files (list (string-append "lib/python"
                                         (version-major+minor version)
                                         "/site-packages"))))
-- 
2.29.2

Message #14 received at 46028 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: 46028 <at> debbugs.gnu.org
Cc: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Subject: [PATCH core-updates v2] gnu: python: Replace PYTHONPATH by
 GUIX_PYTHONPATH_X_Y.
Date: Fri, 22 Jan 2021 08:18:58 -0500

Using PYTHONPATH as a mean to discover the Python packages had the following
issues:

        1. It is not versioned, so different versions of Python would clash if
        installed in a shared profile.

        2. It would interfere with the host Python site on foreign
        distributions, sometimes preventing a a user to login their GDM
        session (!).

        3. It would take precedence over user installed Python packages
        installed through pip.

        4. It would leak into Python virtualenvs, which are supposed to create
        isolated Python environments.

This changes fixes the above issues by making use of a sitecustomize.py
module.  The newly introduced GUIX_PYTHONPATH_X_Y environment variable, where
X is the major and Y the minor version numbers, is read from the environment
and spliced in 'sys.path' just before Python's own site location, which
provides the expected behavior.

* gnu/packages/python.scm (customize-site)
(guix-pythonpath-search-path): New procedures.
* gnu/packages/python.scm (python-2.7)[phases]{customize-site}: New phase.
[native-search-paths]: Adjust.
* gnu/packages/python.scm (python-3.9)[native-search-paths]: Likewise.
[phases]{customize-site}: Override the inherited phase, so that it uses the
correct version.
* gnu/packages/commencement.scm (python-boot0)
[phases]{customize-site}: Likewise.
[native-search-paths]: Override the inherited one.
---
 gnu/packages/commencement.scm |  4 ++-
 gnu/packages/python.scm       | 57 +++++++++++++++++++++++++++--------
 2 files changed, 48 insertions(+), 13 deletions(-)

diff --git a/gnu/packages/commencement.scm b/gnu/packages/commencement.scm
index 0713bcb843..6d7593e223 100644
--- a/gnu/packages/commencement.scm
+++ b/gnu/packages/commencement.scm
@@ -3176,6 +3176,7 @@ memoized as a function of '%current-system'."
                      ;; built, since it requires Linux headers.
                      (("'linux', ") ""))))
                (delete 'set-TZDIR)
+               (replace 'customize-site ,(customize-site version))
                ,@(if (hurd-system?)
                      `((add-before 'build 'fix-regen
                          (lambda* (#:key inputs #:allow-other-keys)
@@ -3184,7 +3185,8 @@ memoized as a function of '%current-system'."
                                (("/usr/include/")
                                 (string-append libc "/include/")))))))
                      '())))
-           ((#:tests? _ #f) #f))))))
+           ((#:tests? _ #f) #f))))
+    (native-search-paths (list (guix-pythonpath-search-path version)))))
 
 (define/system-dependent ld-wrapper-boot0
   ;; The first 'ld' wrapper, defined with 'define/system-dependent' because
diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index d3df1a4dab..c3453d63a9 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -102,7 +102,44 @@
   #:use-module (guix build-system gnu)
   #:use-module (guix build-system trivial)
   #:use-module (srfi srfi-1)
-  #:use-module (srfi srfi-26))
+  #:use-module (srfi srfi-26)
+  #:export (customize-site
+            guix-pythonpath-search-path))
+
+(define* (customize-site version)
+  "Generate a sitecustomize.py phase, using VERSION."
+  `(lambda* (#:key outputs #:allow-other-keys)
+     (let* ((out (assoc-ref outputs "out"))
+            (site-packages (string-append
+                            out "/lib/python"
+                            ,(version-major+minor version)
+                            "/site-packages"))
+            (pythonpath (format #f "GUIX_PYTHONPATH_~a_~a"
+                                ,(version-major version)
+                                ,(version-minor version)))
+            (sitecustomize.py (string-append site-packages
+                                             "/sitecustomize.py")))
+       (with-output-to-file sitecustomize.py
+         (lambda _
+           (format #t "\
+import os
+import sys
+
+# Insert the GUIX_PYTHONPATH_X_Y values just before Python's own site.
+sys_path_absolute = [os.path.realpath(p) for p in sys.path]
+index = sys_path_absolute.index(~s)
+sys.path = sys.path[:index] + os.environ[~s].split(':') + sys.path[index:]
+" site-packages pythonpath))))))
+
+(define (guix-pythonpath-search-path version)
+  "Generate a GUIX_PYTHONPATH_X_Y search path specification, using VERSION."
+  (search-path-specification
+   (variable (format #f "GUIX_PYTHONPATH_~a_~a"
+                     (version-major version)
+                     (version-minor version)))
+   (files (list (string-append "lib/python"
+                               (version-major+minor version)
+                               "/site-packages")))))
 
 (define-public python-2.7
   (package
@@ -313,7 +350,9 @@
                                     "/site-packages")))
                       (install-file tkinter.so target)
                       (delete-file tkinter.so)))))
-               #t))))))
+               #t)))
+         (add-after 'install 'customize-site
+           ,(customize-site version)))))
     (inputs
      `(("bzip2" ,bzip2)
        ("expat" ,expat)
@@ -332,10 +371,7 @@
              `(("python2" ,this-package)
                ("which" ,which))
              '())))
-    (native-search-paths
-     (list (search-path-specification
-            (variable "PYTHONPATH")
-            (files '("lib/python2.7/site-packages")))))
+    (native-search-paths (list (guix-pythonpath-search-path version)))
     (home-page "https://www.python.org")
     (synopsis "High-level, dynamically-typed programming language")
     (description
@@ -463,7 +499,8 @@ data types.")
                                "-x" "lib2to3/.*"
                                ,out))))
                   (list "none" "-O" "-OO"))
-                 #t)))))))
+                 #t)))
+           (replace 'customize-site ,(customize-site version))))))
     (native-inputs
      `(("tzdata" ,tzdata-for-tests)
        ,@(if (%current-target-system)
@@ -471,11 +508,7 @@ data types.")
              '())
        ,@(package-native-inputs python-2)))
     (native-search-paths
-     (list (search-path-specification
-            (variable "PYTHONPATH")
-            (files (list (string-append "lib/python"
-                                        (version-major+minor version)
-                                        "/site-packages"))))
+     (list (guix-pythonpath-search-path version)
            ;; Used to locate tzdata by the zoneinfo module introduced in
            ;; Python 3.9.
            (search-path-specification
-- 
2.29.2

Message #17 received at 46028 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: 46028 <at> debbugs.gnu.org
Subject: Re: [PATCH core-updates v2] gnu: python: Replace PYTHONPATH by
 GUIX_PYTHONPATH_X_Y.
Date: Mon, 25 Jan 2021 02:08:14 -0500

Hi,

Maxim Cournoyer <maxim.cournoyer <at> gmail.com> writes:

> Using PYTHONPATH as a mean to discover the Python packages had the following
> issues:
>
>         1. It is not versioned, so different versions of Python would clash if
>         installed in a shared profile.
>
>         2. It would interfere with the host Python site on foreign
>         distributions, sometimes preventing a a user to login their GDM
>         session (!).
>
>         3. It would take precedence over user installed Python packages
>         installed through pip.
>
>         4. It would leak into Python virtualenvs, which are supposed to create
>         isolated Python environments.

I've adapted the most problematic uses of PYTHONPATH in the code base
(mostly (getenv "PYTHONPATH"), which would typically now return #f and
pushed the branch to cu/farewell-to-pythonpath (it contains ~160
commits, so I thought it'd be easier to review as a branch).

Thanks!

Maxim

Message #20 received at 46028 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: 46028 <at> debbugs.gnu.org
Subject: Re: bug#46028: [PATCH core-updates 0/2] Farewell to PYTHONPATH.
Date: Fri, 29 Jan 2021 12:12:47 -0500

Maxim Cournoyer <maxim.cournoyer <at> gmail.com> writes:

> Hi,
>
> Maxim Cournoyer <maxim.cournoyer <at> gmail.com> writes:
>
>> Using PYTHONPATH as a mean to discover the Python packages had the following
>> issues:
>>
>>         1. It is not versioned, so different versions of Python would clash if
>>         installed in a shared profile.
>>
>>         2. It would interfere with the host Python site on foreign
>>         distributions, sometimes preventing a a user to login their GDM
>>         session (!).
>>
>>         3. It would take precedence over user installed Python packages
>>         installed through pip.
>>
>>         4. It would leak into Python virtualenvs, which are supposed to create
>>         isolated Python environments.
>
> I've adapted the most problematic uses of PYTHONPATH in the code base
> (mostly (getenv "PYTHONPATH"), which would typically now return #f and
> pushed the branch to cu/farewell-to-pythonpath (it contains ~160
> commits, so I thought it'd be easier to review as a branch).

There's now a third version of this that I pushed to the
cu/farewell-to-pythonpath branch.  It now uses GUIX_PYTHONPATH instead
of GUIX_PYTHONPATH_X_Y, which makes it easier to use in any package
definition (as there's no longer a need to import the (guix build
python-build-system) module for packages not using the Python build
system).

I'll merge it soon if nobody has an objection.

Thanks,

Maxim

Message #25 received at 46028-done <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: 46028-done <at> debbugs.gnu.org
Subject: Re: bug#46028: [PATCH core-updates 0/2] Farewell to PYTHONPATH.
Date: Mon, 01 Feb 2021 13:35:52 -0500

Hello,

Maxim Cournoyer <maxim.cournoyer <at> gmail.com> writes:

> Maxim Cournoyer <maxim.cournoyer <at> gmail.com> writes:
>
>> Hi,
>>
>> Maxim Cournoyer <maxim.cournoyer <at> gmail.com> writes:
>>
>>> Using PYTHONPATH as a mean to discover the Python packages had the following
>>> issues:
>>>
>>>         1. It is not versioned, so different versions of Python would clash if
>>>         installed in a shared profile.
>>>
>>>         2. It would interfere with the host Python site on foreign
>>>         distributions, sometimes preventing a a user to login their GDM
>>>         session (!).
>>>
>>>         3. It would take precedence over user installed Python packages
>>>         installed through pip.
>>>
>>>         4. It would leak into Python virtualenvs, which are supposed to create
>>>         isolated Python environments.
>>
>> I've adapted the most problematic uses of PYTHONPATH in the code base
>> (mostly (getenv "PYTHONPATH"), which would typically now return #f and
>> pushed the branch to cu/farewell-to-pythonpath (it contains ~160
>> commits, so I thought it'd be easier to review as a branch).
>
> There's now a third version of this that I pushed to the
> cu/farewell-to-pythonpath branch.  It now uses GUIX_PYTHONPATH instead
> of GUIX_PYTHONPATH_X_Y, which makes it easier to use in any package
> definition (as there's no longer a need to import the (guix build
> python-build-system) module for packages not using the Python build
> system).
>
> I'll merge it soon if nobody has an objection.

I've now pushed this to core-updates.

Closing!

Maxim

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.