GNU bug report logs -
#53258
Python unable to find modules within a Singularity container created with guix pack
Previous Next
Reported by: Marek Felšöci <marek <at> felsoci.sk>
Date: Fri, 14 Jan 2022 16:08:01 UTC
Severity: normal
Done: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Bug is archived. No further changes may be made.
To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 53258 in the body.
You can then email your comments to 53258 AT debbugs.gnu.org in the normal way.
Toggle the display of automated, internal messages from the tracker.
Report forwarded
to
bug-guix <at> gnu.org
:
bug#53258
; Package
guix
.
(Fri, 14 Jan 2022 16:08:01 GMT)
Full text and
rfc822 format available.
Acknowledgement sent
to
Marek Felšöci <marek <at> felsoci.sk>
:
New bug report received and forwarded. Copy sent to
bug-guix <at> gnu.org
.
(Fri, 14 Jan 2022 16:08:01 GMT)
Full text and
rfc822 format available.
Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):
Hello folks,
Recently, I have come around this issue. I created a Singularity container using
the following `guix pack` command:
`guix pack -f squashfs bash coreutils python python-numpy`
Then, I run a bash shell within the container:
`singularity exec test.gz.squashfs bash`
Now, if I want to use numpy,
`python3 -m numpy`
it gives me the following error and the numpy module IS NOT found:
Error in sitecustomize; set PYTHONVERBOSE for traceback:
ValueError:
'/gnu/store/2dyd6rlcc9m68k3mvkw86k8dygkbg025-profile/bin/../../p5fgysbcnnp8b1d91mrvjvababmczga0-python-3.9.6/lib/python3.9/site-packages'
is not in list
/gnu/store/2dyd6rlcc9m68k3mvkw86k8dygkbg025-profile/bin/python3: No module named
numpy
However, if I do,
`export PYTHONPATH=$GUIX_PYTHONPATH`
the numpy module IS found:
Error in sitecustomize; set PYTHONVERBOSE for traceback:
ValueError:
'/gnu/store/2dyd6rlcc9m68k3mvkw86k8dygkbg025-profile/bin/../../p5fgysbcnnp8b1d91mrvjvababmczga0-python-3.9.6/lib/python3.9/site-packages'
is not in list
/gnu/store/2dyd6rlcc9m68k3mvkw86k8dygkbg025-profile/bin/python3: No module named
numpy.__main__; 'numpy' is a package and cannot be directly executed
Still, the sitecustomize error persists.
See the output of `guix describe` below (only the guix channel):
Pokolenie 3 14. január 2022 14:33:06 (súčasné)
guix 5c5d9e5
zdroj repozitára: https://git.savannah.gnu.org/git/guix.git
vetva: master
úprava: 5c5d9e5a3208fa51351c6ad8267eba3d96fa2232
Thanks,
Marek
Information forwarded
to
bug-guix <at> gnu.org
:
bug#53258
; Package
guix
.
(Fri, 14 Jan 2022 17:46:01 GMT)
Full text and
rfc822 format available.
Message #8 received at 53258 <at> debbugs.gnu.org (full text, mbox):
Hi Marek,
Marek Felšöci <marek <at> felsoci.sk> skribis:
> Recently, I have come around this issue. I created a Singularity
> container using the following `guix pack` command:
>
> `guix pack -f squashfs bash coreutils python python-numpy`
>
> Then, I run a bash shell within the container:
>
> `singularity exec test.gz.squashfs bash`
>
> Now, if I want to use numpy,
>
> `python3 -m numpy`
>
> it gives me the following error and the numpy module IS NOT found:
>
> Error in sitecustomize; set PYTHONVERBOSE for traceback:
> ValueError:
> '/gnu/store/2dyd6rlcc9m68k3mvkw86k8dygkbg025-profile/bin/../../p5fgysbcnnp8b1d91mrvjvababmczga0-python-3.9.6/lib/python3.9/site-packages'
> is not in list
> /gnu/store/2dyd6rlcc9m68k3mvkw86k8dygkbg025-profile/bin/python3: No
> module named numpy
I can kinda reproduce via a relocatable pack (relocatable packs and
Singularity packs have in common that they use relative symlinks in
their profile, to allow for relocation):
guix pack -RR python python-numpy -S /bin=bin -S /etc=etc
Then I unpack the thing:
mkdir /tmp/pack; cd /tmp/pack; tar xf /gnu/store/…-pack.tar.gz
And from there I use the same trick as in
‘tests/guix-pack-relocatable.sh’ to “hide” /gnu/store:
--8<---------------cut here---------------start------------->8---
$ unshare -mrf sh -c 'mount -t tmpfs -o ro none /gnu/store; . /tmp/pack/etc/profile; echo $GUIX_PYTHONPATH ; /tmp/pack/bin/python3 '
/gnu/store/w5cz9gbqbja99x0636zsg2fzksmfig4n-profile/lib/python3.9/site-packages
Python 3.9.6 (default, Jan 1 1970, 00:00:01)
[GCC 10.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'numpy'
>>> import os
>>> os.listdir("/gnu/store/w5cz9gbqbja99x0636zsg2fzksmfig4n-profile/lib/python3.9/site-packages")
['setuptools-56.0.0.dist-info', 'distutils-precedence.pth', '_distutils_hack', 'setuptools', 'pip-21.1.3.dist-info', 'sitecustomize.py', 'README.txt', 'pip', 'numpy-1.20.3-py3.9.egg-info', 'numpy', 'pkg_resources', '__pycache__']
>>> os.listdir("/gnu/store/w5cz9gbqbja99x0636zsg2fzksmfig4n-profile/lib/python3.9/site-packages/numpy")
['__init__.pyi', 'doc', 'matlib.py', 'ctypeslib.pyi', 'linalg', 'ctypeslib.py', 'compat', 'typing', '__init__.pxd', 'lib', 'char.pyi', 'version.py', '__config__.py', 'core', '_distributor_init.py', 'distutils', 'tests', 'emath.pyi', '_pytesttester.py', 'fft', '__init__.cython-30.pxd', 'conftest.py', '__init__.py', 'py.typed', '_globals.py', 'rec.pyi', 'testing', 'LICENSE.txt', 'polynomial', 'f2py', 'matrixlib', 'random', 'ma', 'dual.py', 'setup.py', '__pycache__']
--8<---------------cut here---------------end--------------->8---
So in this case ‘GUIX_PYTHONPATH’ is correctly set, but it looks as
though ‘sitecustomize.py’ wasn’t loaded. (I don’t get the “Error in
sitecustomize” message that Marek sees though, weird!)
From Marek’s message, it would seem that somewhere file names are
constructed by appending “..”. Given Unix’s weird dot-dot semantics,
relative symlinks could be interfering:
--8<---------------cut here---------------start------------->8---
$ ls -l /tmp/pack/gnu/store/w5cz9gbqbja99x0636zsg2fzksmfig4n-profile/lib/python3.9/site-packages/
totalo 48
lrwxrwxrwx 1 ludo users 102 Jan 1 1970 _distutils_hack -> ../../../../niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-packages/_distutils_hack
lrwxrwxrwx 1 ludo users 111 Jan 1 1970 distutils-precedence.pth -> ../../../../niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-packages/distutils-precedence.pth
lrwxrwxrwx 1 ludo users 99 Jan 1 1970 numpy -> ../../../../mlccgh05bf8cdinq0ilpvpdmsspq36pv-python-numpy-1.20.3R/lib/python3.9/site-packages/numpy
lrwxrwxrwx 1 ludo users 121 Jan 1 1970 numpy-1.20.3-py3.9.egg-info -> ../../../../mlccgh05bf8cdinq0ilpvpdmsspq36pv-python-numpy-1.20.3R/lib/python3.9/site-packages/numpy-1.20.3-py3.9.egg-info
lrwxrwxrwx 1 ludo users 90 Jan 1 1970 pip -> ../../../../niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-packages/pip
lrwxrwxrwx 1 ludo users 107 Jan 1 1970 pip-21.1.3.dist-info -> ../../../../niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-packages/pip-21.1.3.dist-info
lrwxrwxrwx 1 ludo users 100 Jan 1 1970 pkg_resources -> ../../../../niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-packages/pkg_resources
lrwxrwxrwx 1 ludo users 98 Jan 1 1970 __pycache__ -> ../../../../niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-packages/__pycache__
lrwxrwxrwx 1 ludo users 97 Jan 1 1970 README.txt -> ../../../../niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-packages/README.txt
lrwxrwxrwx 1 ludo users 97 Jan 1 1970 setuptools -> ../../../../niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-packages/setuptools
lrwxrwxrwx 1 ludo users 114 Jan 1 1970 setuptools-56.0.0.dist-info -> ../../../../niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-packages/setuptools-56.0.0.dist-info
lrwxrwxrwx 1 ludo users 103 Jan 1 1970 sitecustomize.py -> ../../../../niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-packages/sitecustomize.py
--8<---------------cut here---------------end--------------->8---
Now look below how dot-dot resolution whereas “lexical” dot-dot
resolution dose the right thing:
--8<---------------cut here---------------start------------->8---
$ ls -l /tmp/pack/gnu/store/2dyd6rlcc9m68k3mvkw86k8dygkbg025-profile/bin/../../p5fgysbcnnp8b1d91mrvjvababmczga0-python-3.9.6/lib/python3.9/site-packages
ls: ne eblas atingi '/tmp/pack/gnu/store/2dyd6rlcc9m68k3mvkw86k8dygkbg025-profile/bin/../../p5fgysbcnnp8b1d91mrvjvababmczga0-python-3.9.6/lib/python3.9/site-packages': Dosiero aŭ dosierujo ne ekzistas
$ ls -l /tmp/pack/gnu/store/p5fgysbcnnp8b1d91mrvjvababmczga0-python-3.9.6/lib/python3.9/site-packages
totalo 40
dr-xr-xr-x 3 ludo users 4096 Jan 1 1970 _distutils_hack/
-r--r--r-- 1 ludo users 152 Jan 1 1970 distutils-precedence.pth
dr-xr-xr-x 5 ludo users 4096 Jan 1 1970 pip/
dr-xr-xr-x 2 ludo users 4096 Jan 1 1970 pip-21.1.3.dist-info/
dr-xr-xr-x 6 ludo users 4096 Jan 1 1970 pkg_resources/
dr-xr-xr-x 2 ludo users 4096 Jan 1 1970 __pycache__/
-r--r--r-- 1 ludo users 119 Jan 1 1970 README.txt
dr-xr-xr-x 7 ludo users 4096 Jan 1 1970 setuptools/
dr-xr-xr-x 2 ludo users 4096 Jan 1 1970 setuptools-56.0.0.dist-info/
-r--r--r-- 1 ludo users 2172 Jan 1 1970 sitecustomize.py
--8<---------------cut here---------------end--------------->8---
In Python’s ‘site.py’, the ‘addsitedir’ method indirectly calls
‘os.path.abspath’, not sure if that is at fault.
I also see that ‘sys.path’ is in fact completely bogus, with “pack”
missing from every directory name:
--8<---------------cut here---------------start------------->8---
$ unshare -mrf sh -c 'mount -t tmpfs -o ro none /gnu/store; . /tmp/pack/etc/profile; echo $GUIX_PYTHONPATH ; /tmp/pack/bin/python3 '
/gnu/store/w5cz9gbqbja99x0636zsg2fzksmfig4n-profile/lib/python3.9/site-packages
Python 3.9.6 (default, Jan 1 1970, 00:00:01)
[GCC 10.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/tmp/niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python39.zip', '/tmp/niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9', '/tmp/niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/lib-dynload', '/tmp/niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-packages']
--8<---------------cut here---------------end--------------->8---
So I think something somewhere is manipulating file names and gets
confused.
Does that ring a bell, Maxim?
Thanks,
Ludo’.
Information forwarded
to
bug-guix <at> gnu.org
:
bug#53258
; Package
guix
.
(Fri, 04 Feb 2022 20:52:01 GMT)
Full text and
rfc822 format available.
Message #11 received at 53258 <at> debbugs.gnu.org (full text, mbox):
Hello!
Ludovic Courtès <ludovic.courtes <at> inria.fr> skribis:
>>From Marek’s message, it would seem that somewhere file names are
> constructed by appending “..”. Given Unix’s weird dot-dot semantics,
> relative symlinks could be interfering:
Some news! Here’s a recap of the IRC discussion earlier today:
‘sys.prefix’ looks like
/tmp/pack2/bin/../../zmv1alp7dzjd6hkc7l6fjz817n2mfzzw-python-3.9.9R.
‘sys.normpath’ does lexical dot-dot resolution (which is incorrect on
POSIX):
https://docs.python.org/3.11/library/os.path.html#os.path.normpath
Thus, passing the prefix above to ‘sys.normpath’ leads “pack2” to be
erased.
Josselin mentioned “path_hooks” as a possible solution, though I’m not
sure what that means; could you explain? :-)
Thanks!
Ludo’.
Information forwarded
to
bug-guix <at> gnu.org
:
bug#53258
; Package
guix
.
(Sat, 05 Feb 2022 22:29:01 GMT)
Full text and
rfc822 format available.
Message #14 received at 53258 <at> debbugs.gnu.org (full text, mbox):
Hello everyone,
After the IRC investigation, it turned out the use of abspath in
Lib/site.py was in fact the culprit. Replacing all its uses with
realpath and doing `./pre-inst-env guix pack -RR python python-numpy -S
/bin=bin -S /etc=etc` did in fact resolve the issue!
I've raised the issue upstream at [1]. The fix itself is pretty simple,
but it's theoretically a breaking change, so we'll see how Python
maintainers want to move forward with it. We could backport this fix
for the next core-updates cycle, whether or not it gets accepted
upstream, and we will also need to modify sitecustomize.py as well to
call os.path.realpath on python_site.
Best,
--
Josselin Poiret
Information forwarded
to
bug-guix <at> gnu.org
:
bug#53258
; Package
guix
.
(Sat, 05 Feb 2022 22:31:02 GMT)
Full text and
rfc822 format available.
Message #17 received at 53258 <at> debbugs.gnu.org (full text, mbox):
Silly me forgetting the link, here it is:
[1] https://bugs.python.org/issue46653
Apologies for the noise,
--
Josselin Poiret
Information forwarded
to
bug-guix <at> gnu.org
:
bug#53258
; Package
guix
.
(Mon, 07 Mar 2022 08:26:01 GMT)
Full text and
rfc822 format available.
Message #20 received at 53258 <at> debbugs.gnu.org (full text, mbox):
Hi Josselin,
Josselin Poiret <dev <at> jpoiret.xyz> skribis:
> After the IRC investigation, it turned out the use of abspath in
> Lib/site.py was in fact the culprit. Replacing all its uses with
> realpath and doing `./pre-inst-env guix pack -RR python python-numpy -S
> /bin=bin -S /etc=etc` did in fact resolve the issue!
>
> I've raised the issue upstream at [1]. The fix itself is pretty simple,
> but it's theoretically a breaking change, so we'll see how Python
> maintainers want to move forward with it. We could backport this fix
> for the next core-updates cycle, whether or not it gets accepted
> upstream, and we will also need to modify sitecustomize.py as well to
> call os.path.realpath on python_site.
> [1] https://bugs.python.org/issue46653
Thanks for the investigation and for reporting it upstream!
Is there a workaround we could apply locally, ideally one that does not
lead to a world rebuild?
Ludo’.
Information forwarded
to
bug-guix <at> gnu.org
:
bug#53258
; Package
guix
.
(Thu, 02 Feb 2023 14:45:02 GMT)
Full text and
rfc822 format available.
Message #23 received at 53258 <at> debbugs.gnu.org (full text, mbox):
Hi Ludo,
Ludovic Courtès <ludo <at> gnu.org> writes:
> Thanks for the investigation and for reporting it upstream!
>
> Is there a workaround we could apply locally, ideally one that does not
> lead to a world rebuild?
So, upstream hasn't replied at all to the above proposal. How do we move
forward? I have a patch that still applies on Python 3.9.9 that we can
put into core-updates. Would that be fine?
Best,
--
Josselin Poiret
Information forwarded
to
bug-guix <at> gnu.org
:
bug#53258
; Package
guix
.
(Thu, 02 Feb 2023 15:41:02 GMT)
Full text and
rfc822 format available.
Message #26 received at 53258 <at> debbugs.gnu.org (full text, mbox):
Hi Josselin,
Josselin Poiret <dev <at> jpoiret.xyz> writes:
> Hi Ludo,
>
> Ludovic Courtès <ludo <at> gnu.org> writes:
>
>> Thanks for the investigation and for reporting it upstream!
>>
>> Is there a workaround we could apply locally, ideally one that does not
>> lead to a world rebuild?
>
> So, upstream hasn't replied at all to the above proposal. How do we move
> forward? I have a patch that still applies on Python 3.9.9 that we can
> put into core-updates. Would that be fine?
I haven't revisited that patch, but if you are confident it wouldn't
break anything in the Python world, core-updates would be a fine place,
yes.
--
Thanks,
Maxim
Information forwarded
to
bug-guix <at> gnu.org
:
bug#53258
; Package
guix
.
(Mon, 06 Feb 2023 18:21:02 GMT)
Full text and
rfc822 format available.
Message #29 received at 53258 <at> debbugs.gnu.org (full text, mbox):
Hi,
On jeu., 02 févr. 2023 at 10:40, Maxim Cournoyer <maxim.cournoyer <at> gmail.com> wrote:
>> So, upstream hasn't replied at all to the above proposal. How do we move
>> forward? I have a patch that still applies on Python 3.9.9 that we can
>> put into core-updates. Would that be fine?
>
> I haven't revisited that patch, but if you are confident it wouldn't
> break anything in the Python world, core-updates would be a fine place,
> yes.
Maybe for testing this change, you could create one Python interpreter
variant containing the patch. Then using ’package-with-explicit-python’
package transformation from (guix build-system python) we could build
Python packages and check if it correctly fixes and also incrementally
check if nothing break. It would avoid a full world rebuild.
Cheers,
simon
Information forwarded
to
bug-guix <at> gnu.org
:
bug#53258
; Package
guix
.
(Thu, 04 Jan 2024 12:36:02 GMT)
Full text and
rfc822 format available.
Message #32 received at 53258 <at> debbugs.gnu.org (full text, mbox):
Hi everyone,
I found this issue while investigating a related one (the error
message). It looks like the core problem discussed here has been solved
in the meantime:
Singularity> python3 -m numpy
Error in sitecustomize; set PYTHONVERBOSE for traceback:
ValueError: '/gnu/store/kx98dz2vc3gdpjficy4nbd7rscnkvz4s-profile/bin/../../h855kddqbay0pcbwr8a7i8m6ilz67cfn-python-3.10.7/lib/python3.10/site-packages' is not in list
/gnu/store/kx98dz2vc3gdpjficy4nbd7rscnkvz4s-profile/bin/python3: No module named numpy.__main__; 'numpy' is a package and cannot be directly executed
This error message is to be expected, and NumPy can be imported:
Singularity> python3
Error in sitecustomize; set PYTHONVERBOSE for traceback:
ValueError: '/gnu/store/kx98dz2vc3gdpjficy4nbd7rscnkvz4s-profile/bin/../../h855kddqbay0pcbwr8a7i8m6ilz67cfn-python-3.10.7/lib/python3.10/site-packages' is not in list
Python 3.10.7 (main, Jan 1 1970, 00:00:01) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>>
What remains is the error message, which comes from a bug in Guix'
sitecustomize.py. It compares a raw path (sys.prefix) with paths that
have been normalized (os.path.normpath). That comparison fails if
sys.prefix contains "../", as it does when run from a Singularity
container. I will submit a patch for this.
Cheers,
Konrad
Information forwarded
to
bug-guix <at> gnu.org
:
bug#53258
; Package
guix
.
(Thu, 04 Jan 2024 13:02:02 GMT)
Full text and
rfc822 format available.
Message #35 received at 53258 <at> debbugs.gnu.org (full text, mbox):
Konrad Hinsen <konrad.hinsen <at> fastmail.net> writes:
> I will submit a patch for this.
Patch at https://issues.guix.gnu.org/68241
Cheers,
Konrad
Information forwarded
to
bug-guix <at> gnu.org
:
bug#53258
; Package
guix
.
(Fri, 05 Jan 2024 15:24:01 GMT)
Full text and
rfc822 format available.
Message #38 received at 53258 <at> debbugs.gnu.org (full text, mbox):
Konrad Hinsen <konrad.hinsen <at> fastmail.net> writes:
> Patch at https://issues.guix.gnu.org/68241
If you want to test it without rebuilding tons of packages, here is a
version that grafts a patched Python onto the existing ones (as substitutes):
https://codeberg.org/khinsen/guix/src/branch/graft-fix-python-sitecustomize
Cheers,
Konrad
Reply sent
to
Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
:
You have taken responsibility.
(Sun, 21 Jan 2024 18:32:02 GMT)
Full text and
rfc822 format available.
Notification sent
to
Marek Felšöci <marek <at> felsoci.sk>
:
bug acknowledged by developer.
(Sun, 21 Jan 2024 18:32:02 GMT)
Full text and
rfc822 format available.
Message #43 received at 53258-done <at> debbugs.gnu.org (full text, mbox):
Hi Konrad,
Konrad Hinsen <konrad.hinsen <at> fastmail.net> writes:
> Konrad Hinsen <konrad.hinsen <at> fastmail.net> writes:
>
>> Patch at https://issues.guix.gnu.org/68241
>
> If you want to test it without rebuilding tons of packages, here is a
> version that grafts a patched Python onto the existing ones (as substitutes):
>
> https://codeberg.org/khinsen/guix/src/branch/graft-fix-python-sitecustomize
I've applied your patch to core-updates. I trust that it fixed this
issue based on your own testing.
Closing; we can always reopen if we find it didn't fix every use cases
or create a fresh issue.
--
Thanks,
Maxim
bug archived.
Request was from
Debbugs Internal Request <help-debbugs <at> gnu.org>
to
internal_control <at> debbugs.gnu.org
.
(Mon, 19 Feb 2024 12:24:17 GMT)
Full text and
rfc822 format available.
This bug report was last modified 1 year and 80 days ago.
Previous Next
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997,2003 nCipher Corporation Ltd,
1994-97 Ian Jackson.