GNU bug report logs - #30229
Python modules installed by pip in virtualenv can't find shared objects.

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 30229 in the body.
You can then email your comments to 30229 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Fis Trivial <ybbs.daans <at> hotmail.com>
To: "bug-guix <at> gnu.org" <bug-guix <at> gnu.org>
Subject: Python modules installed by pip in virtualenv can't find shared
 objects.
Date: Tue, 23 Jan 2018 11:49:59 +0000

When a python module needs to load a dynamic shared object, it looks in the
path provided by *LD_LIBRARY_PATH*(1), but guix doesn't modify this environment
variable to export the needed path for python.

* Backtrace
In my case, it's lightgbm (installed by pip), needs libgomp.so from gcc:lib for
openmp support. Here is the backtrace:

--8<---------------cut here---------------start------------->8---
Traceback (most recent call last):
  File "ex1.py", line 5, in <module>
    import lightgbm as lgb
  File "/home/fis/Workspace/tianchi/medical_treatment/lib/python3.5/site-packages/lightgbm/__init__.py", line 8, in <module>
    from .basic import Booster, Dataset
  File "/home/fis/Workspace/tianchi/medical_treatment/lib/python3.5/site-packages/lightgbm/basic.py", line 32, in <module>
    _LIB = _load_lib()
  File "/home/fis/Workspace/tianchi/medical_treatment/lib/python3.5/site-packages/lightgbm/basic.py", line 27, in _load_lib
    lib = ctypes.cdll.LoadLibrary(lib_path[0])
  File "/gnu/store/jb3n0bsdpkhvyb8y70jyr8fcx8fqssr9-python-3.5.3/lib/python3.5/ctypes/__init__.py", line 425, in LoadLibrary
    return self._dlltype(name)
  File "/gnu/store/jb3n0bsdpkhvyb8y70jyr8fcx8fqssr9-python-3.5.3/lib/python3.5/ctypes/__init__.py", line 347, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: libgomp.so.1: cannot open shared object file: No such file or directory
--8<---------------cut here---------------end--------------->8---

* Reproduce
To install lightgbm, simply use `pip install lightgbm`(in virtualenv).
Then in python shell:
import lightgbm as lgb

* Ad-hoc
export LD_LIBRARY_PATH=~/.guix-profile/lib:$LD_LIBRARY_PATH


[1]: https://stackoverflow.com/a/1100016

Message #8 received at 30229 <at> debbugs.gnu.org (full text, mbox):

From: Fis Trivial <ybbs.daans <at> hotmail.com>
To: Ricardo Wurmus <rekado <at> elephly.net>
Cc: "30229 <at> debbugs.gnu.org" <30229 <at> debbugs.gnu.org>
Subject: Re: bug#30229: Python modules installed by pip in virtualenv can't
 find shared objects.
Date: Tue, 23 Jan 2018 13:21:10 +0000

> 
> In general, though, I recommend using Guix for package management and
> development instead of virtualenv and pip.
> 
I understand the importance for having a self-contained dependency graph.

After trying guix a few months, I came to realize that, at the a theoretical
level, the environment variables are not treated as part of the dependency
graph in the Guix world. Dangling environment variables happens, and can not
be solved by installing everything with guix(1).

I really hope that we can come up with some methods to solve these problems
in a systematical way. I will keep reporting bugs caused by environment
variables, hoping that it might be helpful in future development.


[1]: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=30093

Message #11 received at 30229 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Fis Trivial <ybbs.daans <at> hotmail.com>
Cc: 30229 <at> debbugs.gnu.org
Subject: Re: bug#30229: Python modules installed by pip in virtualenv can't
 find shared objects.
Date: Wed, 24 Jan 2018 16:01:42 +0100

Hi,

Fis Trivial <ybbs.daans <at> hotmail.com> skribis:

> In my case, it's lightgbm (installed by pip), needs libgomp.so from gcc:lib for
> openmp support. Here is the backtrace:
>
> Traceback (most recent call last):
>   File "ex1.py", line 5, in <module>
>     import lightgbm as lgb
>   File "/home/fis/Workspace/tianchi/medical_treatment/lib/python3.5/site-packages/lightgbm/__init__.py", line 8, in <module>
>     from .basic import Booster, Dataset
>   File "/home/fis/Workspace/tianchi/medical_treatment/lib/python3.5/site-packages/lightgbm/basic.py", line 32, in <module>
>     _LIB = _load_lib()
>   File "/home/fis/Workspace/tianchi/medical_treatment/lib/python3.5/site-packages/lightgbm/basic.py", line 27, in _load_lib
>     lib = ctypes.cdll.LoadLibrary(lib_path[0])
>   File "/gnu/store/jb3n0bsdpkhvyb8y70jyr8fcx8fqssr9-python-3.5.3/lib/python3.5/ctypes/__init__.py", line 425, in LoadLibrary
>     return self._dlltype(name)
>   File "/gnu/store/jb3n0bsdpkhvyb8y70jyr8fcx8fqssr9-python-3.5.3/lib/python3.5/ctypes/__init__.py", line 347, in __init__
>     self._handle = _dlopen(self._name, mode)
> OSError: libgomp.so.1: cannot open shared object file: No such file or directory

I’d suggest hard-coding the absolute file name of libgomp.so.1 in
lightgbm/basic.py.

This is what we do, for instance, for Guile packages that use the
foreign function interface (FFI) and dynamically load shared libraries
(see ‘guile-gdbm-ffi’ as an example.)

HTH,
Ludo’.

Message #14 received at 30229 <at> debbugs.gnu.org (full text, mbox):

From: Ricardo Wurmus <rekado <at> elephly.net>
To: Fis Trivial <ybbs.daans <at> hotmail.com>
Cc: 30229 <at> debbugs.gnu.org
Subject: Re: bug#30229: Python modules installed by pip in virtualenv can't
 find shared objects.
Date: Tue, 23 Jan 2018 13:00:24 +0100

Hi,

> When a python module needs to load a dynamic shared object, it looks in the
> path provided by *LD_LIBRARY_PATH*(1), but guix doesn't modify this environment
> variable to export the needed path for python.

We cannot set this environment variable by default lest we break other
packages that may be installed.  LD_LIBRARY_PATH is dangerous as it
tells the runtime linker to prefer libraries in the specified
directories.

For Guix packages we use different means to embed store paths in
binaries, which are looked up at runtime.  For binaries that’s achieved
with RUNPATH; for others we patch the sources to ensure that libraries
are not looked up merely by name but by absolute path.

In your particular case (installing packages without Guix) I think the
best way is to manually set LD_LIBRARY_PATH on demand, or to set
LD_PRELOAD to the specific libraries that are required.

In general, though, I recommend using Guix for package management and
development instead of virtualenv and pip.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Message #17 received at 30229 <at> debbugs.gnu.org (full text, mbox):

From: zimoun <zimon.toutoune <at> gmail.com>
To: Ricardo Wurmus <rekado <at> elephly.net>
Cc: 30229 <at> debbugs.gnu.org, Fis Trivial <ybbs.daans <at> hotmail.com>
Subject: Re: bug#30229: Python modules installed by pip in virtualenv can't
 find shared objects.
Date: Thu, 25 Nov 2021 00:39:21 +0100

Hi,

On Tue, 23 Jan 2018 at 13:00, Ricardo Wurmus <rekado <at> elephly.net> wrote:

>> When a python module needs to load a dynamic shared object, it looks in the
>> path provided by *LD_LIBRARY_PATH*(1), but guix doesn't modify this environment
>> variable to export the needed path for python.
>
> We cannot set this environment variable by default lest we break other
> packages that may be installed.  LD_LIBRARY_PATH is dangerous as it
> tells the runtime linker to prefer libraries in the specified
> directories.
>
> For Guix packages we use different means to embed store paths in
> binaries, which are looked up at runtime.  For binaries that’s achieved
> with RUNPATH; for others we patch the sources to ensure that libraries
> are not looked up merely by name but by absolute path.
>
> In your particular case (installing packages without Guix) I think the
> best way is to manually set LD_LIBRARY_PATH on demand, or to set
> LD_PRELOAD to the specific libraries that are required.
>
> In general, though, I recommend using Guix for package management and
> development instead of virtualenv and pip.

Regarding the improvements of ’guix import pypi’ since 2018, and because
tweaking LD_LIBRARY_PATH is dangerous, I do not see what could be the
next action to solve this.

Therefore, I propose to close it.  WDYT?


Cheers,
simon

Message #22 received at 30229-done <at> debbugs.gnu.org (full text, mbox):

From: zimoun <zimon.toutoune <at> gmail.com>
To: Ricardo Wurmus <rekado <at> elephly.net>
Cc: Fis Trivial <ybbs.daans <at> hotmail.com>, 30229-done <at> debbugs.gnu.org
Subject: Re: bug#30229: Python modules installed by pip in virtualenv can't
 find shared objects.
Date: Thu, 13 Jan 2022 15:59:42 +0100

Hi,

On Thu, 25 Nov 2021 at 00:39, zimoun <zimon.toutoune <at> gmail.com> wrote:
> On Tue, 23 Jan 2018 at 13:00, Ricardo Wurmus <rekado <at> elephly.net> wrote:
>
>>> When a python module needs to load a dynamic shared object, it looks in the
>>> path provided by *LD_LIBRARY_PATH*(1), but guix doesn't modify this
>>> environment
>>> variable to export the needed path for python.
>>
>> We cannot set this environment variable by default lest we break other
>> packages that may be installed.  LD_LIBRARY_PATH is dangerous as it
>> tells the runtime linker to prefer libraries in the specified
>> directories.
>>
>> For Guix packages we use different means to embed store paths in
>> binaries, which are looked up at runtime.  For binaries that’s achieved
>> with RUNPATH; for others we patch the sources to ensure that libraries
>> are not looked up merely by name but by absolute path.
>>
>> In your particular case (installing packages without Guix) I think the
>> best way is to manually set LD_LIBRARY_PATH on demand, or to set
>> LD_PRELOAD to the specific libraries that are required.
>>
>> In general, though, I recommend using Guix for package management and
>> development instead of virtualenv and pip.
>
> Regarding the improvements of ’guix import pypi’ since 2018, and because
> tweaking LD_LIBRARY_PATH is dangerous, I do not see what could be the
> next action to solve this.
>
> Therefore, I propose to close it.  WDYT?

After 7 weeks of delay, I am closing.


Cheers,
simon

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.