GNU bug report logs - #46569
python can't find internal pip modules in environment from manifest

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 46569 in the body.
You can then email your comments to 46569 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Jelle Licht <jlicht <at> fsfe.org>
To: bug-guix <at> gnu.org
Subject: python can't find internal pip modules in environment from manifest
Date: Tue, 16 Feb 2021 21:58:20 +0100

Python can not load all pip-related python modules when loaded from a
manifest file, yet seems to work fine when loaded 'directly' as part of
a `guix environment'-invocation.


Provided that we have a file in `hi.py':
--8<---------------cut here---------------start------------->8---
import pip
print("halfwaypoint")
import pip._internal.index.package_finder
print("I made it!")
--8<---------------cut here---------------end--------------->8---

and a `manifest.scm' with:

--8<---------------cut here---------------start------------->8---
(use-modules (gnu) (guix packages)
	     (gnu packages python)
	     (gnu packages python-xyz))

(packages->manifest (list python python-pip))
--8<---------------cut here---------------end--------------->8---

I can get the expected output by running:
`guix environment --pure --ad-hoc python python-pip -- python3 hi.py' =>
> halfwaypoint
> I made it!

Yet, when I run the following:
`guix environment --pure -m manifest.scm -- python3 hi.py' =>

--8<---------------cut here---------------start------------->8---
halfwaypoint
Traceback (most recent call last):
  File "hi.py", line 4, in <module>
    import pip._internal.index.package_finder
  File "/gnu/store/4m5vhlq61qnj36rq60l83xcmgj3mx92j-profile/lib/python3.8/site-packages/pip/_internal/__init__.py", line 40, in <module>
    from pip._internal.cli.autocompletion import autocomplete
  File "/gnu/store/4m5vhlq61qnj36rq60l83xcmgj3mx92j-profile/lib/python3.8/site-packages/pip/_internal/cli/autocompletion.py", line 8, in <module>
    from pip._internal.cli.main_parser import create_main_parser
  File "/gnu/store/4m5vhlq61qnj36rq60l83xcmgj3mx92j-profile/lib/python3.8/site-packages/pip/_internal/cli/main_parser.py", line 11, in <module>
    from pip._internal.commands import (
  File "/gnu/store/4m5vhlq61qnj36rq60l83xcmgj3mx92j-profile/lib/python3.8/site-packages/pip/_internal/commands/__init__.py", line 6, in <module>
    from pip._internal.commands.completion import CompletionCommand
  File "/gnu/store/4m5vhlq61qnj36rq60l83xcmgj3mx92j-profile/lib/python3.8/site-packages/pip/_internal/commands/completion.py", line 6, in <module>
    from pip._internal.cli.base_command import Command
  File "/gnu/store/4m5vhlq61qnj36rq60l83xcmgj3mx92j-profile/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 26, in <module>
    from pip._internal.index import PackageFinder
ImportError: cannot import name 'PackageFinder' from 'pip._internal.index' (/gnu/store/4m5vhlq61qnj36rq60l83xcmgj3mx92j-profile/lib/python3.8/site-packages/pip/_internal/index/__init__.py)
--8<---------------cut here---------------end--------------->8---

Why is there a difference in the first place? Shouldn't both approaches work?

- Jelle

Message #8 received at 46569 <at> debbugs.gnu.org (full text, mbox):

From: Jelle Licht <jlicht <at> fsfe.org>
To: 46569 <at> debbugs.gnu.org
Subject: Re: bug#46569: python can't find internal pip modules in
 environment from manifest
Date: Thu, 18 Feb 2021 14:49:28 +0100

Jelle Licht <jlicht <at> fsfe.org> writes:

> Python can not load all pip-related python modules when loaded from a
> manifest file, yet seems to work fine when loaded 'directly' as part of
> a `guix environment'-invocation.
>
>
> Provided that we have a file in `hi.py':
> --8<---------------cut here---------------start------------->8---
> import pip
> print("halfwaypoint")
> import pip._internal.index.package_finder
> print("I made it!")
> --8<---------------cut here---------------end--------------->8---
>
> and a `manifest.scm' with:
>
> --8<---------------cut here---------------start------------->8---
> (use-modules (gnu) (guix packages)
> 	     (gnu packages python)
> 	     (gnu packages python-xyz))
>
> (packages->manifest (list python python-pip))
> --8<---------------cut here---------------end--------------->8---
>
> I can get the expected output by running:
> `guix environment --pure --ad-hoc python python-pip -- python3 hi.py' =>
>> halfwaypoint
>> I made it!
>
> Yet, when I run the following:
> `guix environment --pure -m manifest.scm -- python3 hi.py' =>
>
> --8<---------------cut here---------------start------------->8---
> halfwaypoint
> Traceback (most recent call last):
>   File "hi.py", line 4, in <module>
>     import pip._internal.index.package_finder
>   File "/gnu/store/4m5vhlq61qnj36rq60l83xcmgj3mx92j-profile/lib/python3.8/site-packages/pip/_internal/__init__.py", line 40, in <module>
>     from pip._internal.cli.autocompletion import autocomplete
>   File "/gnu/store/4m5vhlq61qnj36rq60l83xcmgj3mx92j-profile/lib/python3.8/site-packages/pip/_internal/cli/autocompletion.py", line 8, in <module>
>     from pip._internal.cli.main_parser import create_main_parser
>   File "/gnu/store/4m5vhlq61qnj36rq60l83xcmgj3mx92j-profile/lib/python3.8/site-packages/pip/_internal/cli/main_parser.py", line 11, in <module>
>     from pip._internal.commands import (
>   File "/gnu/store/4m5vhlq61qnj36rq60l83xcmgj3mx92j-profile/lib/python3.8/site-packages/pip/_internal/commands/__init__.py", line 6, in <module>
>     from pip._internal.commands.completion import CompletionCommand
>   File "/gnu/store/4m5vhlq61qnj36rq60l83xcmgj3mx92j-profile/lib/python3.8/site-packages/pip/_internal/commands/completion.py", line 6, in <module>
>     from pip._internal.cli.base_command import Command
>   File "/gnu/store/4m5vhlq61qnj36rq60l83xcmgj3mx92j-profile/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 26, in <module>
>     from pip._internal.index import PackageFinder
> ImportError: cannot import name 'PackageFinder' from 'pip._internal.index' (/gnu/store/4m5vhlq61qnj36rq60l83xcmgj3mx92j-profile/lib/python3.8/site-packages/pip/_internal/index/__init__.py)
> --8<---------------cut here---------------end--------------->8---
>
> Why is there a difference in the first place? Shouldn't both approaches work?

Using diffoscope, I found the (practical) difference:

$GUIX_ENVIRONMENT/bin/pip3 is
│ │ -destination: /gnu/store/rz42ba0my9vrgbkjpkzr2drmnjk5ah50-python-3.8.2/bin/pip3
│ │ +destination: /gnu/store/d75bwzbla5ybhs0mdw80qy94mawnhhsw-python-pip-20.2.4/bin/pip3

If you make sure that `python-pip' precedes the `python' package in the
manifest, things work out fine. Likewise for the following --ad-hoc
example:

This fails:
guix environment --ad-hoc python-pip python -- python3 hi.py

This works:
guix environment --ad-hoc python python-pip -- python3 hi.py

So it seems the order in which guix sees packages when building profiles
determines how collisions are resolved, good to know :-).

Is the python-included pip package supposed to be broken like this, though?
 - Jelle

Message #11 received at 46569 <at> debbugs.gnu.org (full text, mbox):

From: Jelle Licht <jlicht <at> fsfe.org>
To: 46569 <at> debbugs.gnu.org
Subject: Re: bug#46569: python can't find internal pip modules in
 environment from manifest
Date: Thu, 18 Feb 2021 17:37:48 +0100

Jelle Licht <jlicht <at> fsfe.org> writes:

> Jelle Licht <jlicht <at> fsfe.org> writes:
> This fails:
> guix environment --ad-hoc python-pip python -- python3 hi.py
>
> This works:
> guix environment --ad-hoc python python-pip -- python3 hi.py

This should be the other way around, pasting error on my end.

Message #14 received at 46569 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: Jelle Licht <jlicht <at> fsfe.org>
Cc: 46569 <at> debbugs.gnu.org
Subject: Re: bug#46569: python can't find internal pip modules in
 environment from manifest
Date: Thu, 18 Feb 2021 13:05:47 -0500

Hi Jelle,

Jelle Licht <jlicht <at> fsfe.org> writes:

> Python can not load all pip-related python modules when loaded from a
> manifest file, yet seems to work fine when loaded 'directly' as part of
> a `guix environment'-invocation.
>
>
> Provided that we have a file in `hi.py':
>
> import pip
> print("halfwaypoint")
> import pip._internal.index.package_finder
> print("I made it!")
>
>
> and a `manifest.scm' with:
>
> (use-modules (gnu) (guix packages)
> 	     (gnu packages python)
> 	     (gnu packages python-xyz))
>
> (packages->manifest (list python python-pip))
>
>
> I can get the expected output by running:
> `guix environment --pure --ad-hoc python python-pip -- python3 hi.py' =>
>> halfwaypoint
>> I made it!

It seems that two pip versions don't cohabit very well.  The same
happens if you install pip via 'pip install --user pip' (e.g., to
~/.local/bin) and try to use it.

You could try using the pip version that comes bundled with Python
itself (e.g, not installing python-pip in the same profile); if it's the
only one, it should work.

HTH,

Maxim

Message #17 received at 46569 <at> debbugs.gnu.org (full text, mbox):

From: Jelle Licht <jlicht <at> fsfe.org>
To: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Cc: 46569 <at> debbugs.gnu.org
Subject: Re: bug#46569: python can't find internal pip modules in
 environment from manifest
Date: Thu, 18 Feb 2021 20:45:54 +0100

Maxim Cournoyer <maxim.cournoyer <at> gmail.com> writes:

> Hi Jelle,
>
> Jelle Licht <jlicht <at> fsfe.org> writes:
>
>> Python can not load all pip-related python modules when loaded from a
>> manifest file, yet seems to work fine when loaded 'directly' as part of
>> a `guix environment'-invocation.
>>
>>
>> Provided that we have a file in `hi.py':
>>
>> import pip
>> print("halfwaypoint")
>> import pip._internal.index.package_finder
>> print("I made it!")
>>
>>
>> and a `manifest.scm' with:
>>
>> (use-modules (gnu) (guix packages)
>> 	     (gnu packages python)
>> 	     (gnu packages python-xyz))
>>
>> (packages->manifest (list python python-pip))
>>
>>
>> I can get the expected output by running:
>> `guix environment --pure --ad-hoc python python-pip -- python3 hi.py' =>
>>> halfwaypoint
>>> I made it!
>
> It seems that two pip versions don't cohabit very well.  The same
> happens if you install pip via 'pip install --user pip' (e.g., to
> ~/.local/bin) and try to use it.
>
> You could try using the pip version that comes bundled with Python
> itself (e.g, not installing python-pip in the same profile); if it's the
> only one, it should work.

It doesn't, though, with the exact same error message as above.

Does my example (hi.py) work for you?

Message #22 received at 46569-done <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: Jelle Licht <jlicht <at> fsfe.org>
Cc: 46569-done <at> debbugs.gnu.org
Subject: Re: bug#46569: python can't find internal pip modules in
 environment from manifest
Date: Thu, 18 Feb 2021 16:18:47 -0500

Hello Jelle,

Jelle Licht <jlicht <at> fsfe.org> writes:

> Python can not load all pip-related python modules when loaded from a
> manifest file, yet seems to work fine when loaded 'directly' as part of
> a `guix environment'-invocation.
>
>
> Provided that we have a file in `hi.py':
>
> import pip
> print("halfwaypoint")
> import pip._internal.index.package_finder
> print("I made it!")

The problem seems to be related to the version of pip used.  The one
bundled with our Python on master is:

$ guix environment --pure --ad-hoc python -- pip3 --version
pip 19.2.3 from
/gnu/store/pw65c67dmzr2m9wbkj5mskhm6ljk6xal-profile/lib/python3.8/site-packages/pip
(python 3.8)

and

$ guix environment --pure --ad-hoc python -- python3 -c 'import pip._internal.index.package_finder'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named
'pip._internal.index.package_finder'; 'pip._internal.index' is not a
package

but

$ guix environment --pure --ad-hoc python python-pip -- pip3 --version
pip 20.2.4 from
/gnu/store/d75bwzbla5ybhs0mdw80qy94mawnhhsw-python-pip-20.2.4/lib/python3.8/site-packages/pip
(python 3.8)

and

$ guix environment --pure --ad-hoc python python-pip -- python3 -c 'import pip._internal.index.package_finder'
$ echo $?
0

HTH!

Closing.

Maxim

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.