GNU bug report logs -
#22876
Python can't use https with recent grafts
Previous Next
To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 22876 in the body.
You can then email your comments to 22876 AT debbugs.gnu.org in the normal way.
Toggle the display of automated, internal messages from the tracker.
Report forwarded
to
bug-guix <at> gnu.org
:
bug#22876
; Package
guix
.
(Wed, 02 Mar 2016 00:00:02 GMT)
Full text and
rfc822 format available.
Acknowledgement sent
to
Christopher Allan Webber <cwebber <at> dustycloud.org>
:
New bug report received and forwarded. Copy sent to
bug-guix <at> gnu.org
.
(Wed, 02 Mar 2016 00:00:03 GMT)
Full text and
rfc822 format available.
Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):
Most of Guix seems to be working just fine with the grafts support and
grafting of openssl. However, unlike most grafts that will be done
probably, this one removes a feature, and that seems to be creating
problems in Python land.
>>> from urllib.request import HTTPSHandler
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: cannot import name 'HTTPSHandler'
Notably, virtualenv no longer works:
$ guix environment --ad-hoc python-virtualenv
substitute: updating list of substitutes from 'http://hydra.gnu.org'... 100.0%
The following derivations will be built:
/gnu/store/mcxrh4ba9pf4855kcbdnz654r0xxf86b-profile.drv
/gnu/store/ii3ykjkidhz88ycx4p3gi2c7bhhn1vqz-ca-certificate-bundle.drv
/gnu/store/h6fzjn70ki8vk3sxd0863vqjwkds1723-info-dir.drv
$ virtualenv /tmp/try-virtualenv
Using base prefix '/gnu/store/1spkp48cbbzg6ic5qkv3qpm3mvsgwkys-python-3.4.3'
New python executable in /tmp/try-virtualenv/bin/python
Installing setuptools, pip, wheel...
Complete output from command /tmp/try-virtualenv/bin/python -c "import sys, pip; sys...d\"] + sys.argv[1:]))" setuptools pip wheel:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/gnu/store/h38982xp00s1g95nzr6lws31w8q8njb3-python-virtualenv-13.1.2/lib/python3.4/site-packages/virtualenv-13.1.2-py3.4.egg/virtualenv_support/pip-7.1.2-py2.py3-none-any.whl/pip/__init__.py", line 15, in <module>
File "/gnu/store/h38982xp00s1g95nzr6lws31w8q8njb3-python-virtualenv-13.1.2/lib/python3.4/site-packages/virtualenv-13.1.2-py3.4.egg/virtualenv_support/pip-7.1.2-py2.py3-none-any.whl/pip/vcs/subversion.py", line 9, in <module>
File "/gnu/store/h38982xp00s1g95nzr6lws31w8q8njb3-python-virtualenv-13.1.2/lib/python3.4/site-packages/virtualenv-13.1.2-py3.4.egg/virtualenv_support/pip-7.1.2-py2.py3-none-any.whl/pip/index.py", line 30, in <module>
File "/gnu/store/h38982xp00s1g95nzr6lws31w8q8njb3-python-virtualenv-13.1.2/lib/python3.4/site-packages/virtualenv-13.1.2-py3.4.egg/virtualenv_support/pip-7.1.2-py2.py3-none-any.whl/pip/wheel.py", line 35, in <module>
File "/gnu/store/h38982xp00s1g95nzr6lws31w8q8njb3-python-virtualenv-13.1.2/lib/python3.4/site-packages/virtualenv-13.1.2-py3.4.egg/virtualenv_support/pip-7.1.2-py2.py3-none-any.whl/pip/_vendor/distlib/scripts.py", line 14, in <module>
File "/gnu/store/h38982xp00s1g95nzr6lws31w8q8njb3-python-virtualenv-13.1.2/lib/python3.4/site-packages/virtualenv-13.1.2-py3.4.egg/virtualenv_support/pip-7.1.2-py2.py3-none-any.whl/pip/_vendor/distlib/compat.py", line 66, in <module>
ImportError: cannot import name 'HTTPSHandler'
----------------------------------------
...Installing setuptools, pip, wheel...done.
Traceback (most recent call last):
File "/gnu/store/h38982xp00s1g95nzr6lws31w8q8njb3-python-virtualenv-13.1.2/bin/.virtualenv-real", line 9, in <module>
load_entry_point('virtualenv==13.1.2', 'console_scripts', 'virtualenv')()
File "/gnu/store/h38982xp00s1g95nzr6lws31w8q8njb3-python-virtualenv-13.1.2/lib/python3.4/site-packages/virtualenv-13.1.2-py3.4.egg/virtualenv.py", line 832, in main
symlink=options.symlink)
File "/gnu/store/h38982xp00s1g95nzr6lws31w8q8njb3-python-virtualenv-13.1.2/lib/python3.4/site-packages/virtualenv-13.1.2-py3.4.egg/virtualenv.py", line 1004, in create_environment
install_wheel(to_install, py_executable, search_dirs)
File "/gnu/store/h38982xp00s1g95nzr6lws31w8q8njb3-python-virtualenv-13.1.2/lib/python3.4/site-packages/virtualenv-13.1.2-py3.4.egg/virtualenv.py", line 969, in install_wheel
'PIP_NO_INDEX': '1'
File "/gnu/store/h38982xp00s1g95nzr6lws31w8q8njb3-python-virtualenv-13.1.2/lib/python3.4/site-packages/virtualenv-13.1.2-py3.4.egg/virtualenv.py", line 910, in call_subprocess
% (cmd_desc, proc.returncode))
OSError: Command /tmp/try-virtualenv/bin/python -c "import sys, pip; sys...d\"] + sys.argv[1:]))" setuptools pip wheel failed with error code 1
I'm not really sure this is a problem with the new grafts system. It
might just be that a "fix" which tears parts of a library is going to
cause unexpected problems in some places for ABI incompatibility
reasons.
Not sure if we should just wait for the world-rebuild or what right
now...!
Information forwarded
to
bug-guix <at> gnu.org
:
bug#22876
; Package
guix
.
(Wed, 02 Mar 2016 00:14:01 GMT)
Full text and
rfc822 format available.
Message #8 received at 22876 <at> debbugs.gnu.org (full text, mbox):
On Tue, Mar 01, 2016 at 03:59:24PM -0800, Christopher Allan Webber wrote:
> Most of Guix seems to be working just fine with the grafts support and
> grafting of openssl. However, unlike most grafts that will be done
> probably, this one removes a feature, and that seems to be creating
> problems in Python land.
>
> >>> from urllib.request import HTTPSHandler
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> ImportError: cannot import name 'HTTPSHandler'
I suspect this has to do with the error message Mark shared on #guix:
ImportError: /gnu/store/f0jzhl04iyaqv56yj92cd9bk57p3inqx-python-2.7.10/lib/python2.7/lib-dynload/_ssl.so:
undefined symbol: SSLv2_method
[,,,]
> I'm not really sure this is a problem with the new grafts system. It
> might just be that a "fix" which tears parts of a library is going to
> cause unexpected problems in some places for ABI incompatibility
> reasons.
Yeah, I'm not surprised other packages are breaking as a result of this.
Can you give me a method to reproduce this bug? I can try building
Python against the new OpenSSL directly and see if the problem persists.
>
> Not sure if we should just wait for the world-rebuild or what right
> now...!
I guess that grafts of compatible updates can persist for a while but in
cases like this we should probably start rebuilding...
Information forwarded
to
bug-guix <at> gnu.org
:
bug#22876
; Package
guix
.
(Wed, 02 Mar 2016 00:14:02 GMT)
Full text and
rfc822 format available.
Message #11 received at 22876 <at> debbugs.gnu.org (full text, mbox):
Christopher Allan Webber writes:
> Most of Guix seems to be working just fine with the grafts support and
> grafting of openssl. However, unlike most grafts that will be done
> probably, this one removes a feature, and that seems to be creating
> problems in Python land.
>
> >>> from urllib.request import HTTPSHandler
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> ImportError: cannot import name 'HTTPSHandler'
As expected, this is for the same reasons offlineimap seemed to have
problems:
cwebber <at> oolong:~/devel/mediagoblin$ python3
Python 3.4.3 (default, Jan 1 1970, 00:00:01)
[GCC 4.9.3] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ssl
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/gnu/store/1spkp48cbbzg6ic5qkv3qpm3mvsgwkys-python-3.4.3/lib/python3.4/ssl.py", line 97, in <module>
import _ssl # if we can't import it, let the error propagate
ImportError: /gnu/store/1spkp48cbbzg6ic5qkv3qpm3mvsgwkys-python-3.4.3/lib/python3.4/lib-dynload/_ssl.cpython-34m.so: undefined symbol: SSLv2_method
This leads to my suspicion that it's not really grafting's fault here,
it's the *removal* of a piece of code, thus making things
abi-incompatible with the system we built. Hopefully most grafting
situations won't require this. I think that's right? :)
Unfortunately, I'd say that ssl and python isn't really optional!
Is it possible to graft on top of a graft? Could we rebuild Python
based on the grafted openssl, and then graft things on top of the
grafted Python? :)
Information forwarded
to
bug-guix <at> gnu.org
:
bug#22876
; Package
guix
.
(Wed, 02 Mar 2016 00:24:02 GMT)
Full text and
rfc822 format available.
Message #14 received at 22876 <at> debbugs.gnu.org (full text, mbox):
On Tue, Mar 01, 2016 at 04:13:41PM -0800, Christopher Allan Webber wrote:
> Christopher Allan Webber writes:
>
> > Most of Guix seems to be working just fine with the grafts support and
> > grafting of openssl. However, unlike most grafts that will be done
> > probably, this one removes a feature, and that seems to be creating
> > problems in Python land.
> >
> > >>> from urllib.request import HTTPSHandler
> > Traceback (most recent call last):
> > File "<stdin>", line 1, in <module>
> > ImportError: cannot import name 'HTTPSHandler'
>
> As expected, this is for the same reasons offlineimap seemed to have
> problems:
>
> cwebber <at> oolong:~/devel/mediagoblin$ python3
> Python 3.4.3 (default, Jan 1 1970, 00:00:01)
> [GCC 4.9.3] on linux
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import ssl
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> File "/gnu/store/1spkp48cbbzg6ic5qkv3qpm3mvsgwkys-python-3.4.3/lib/python3.4/ssl.py", line 97, in <module>
> import _ssl # if we can't import it, let the error propagate
> ImportError: /gnu/store/1spkp48cbbzg6ic5qkv3qpm3mvsgwkys-python-3.4.3/lib/python3.4/lib-dynload/_ssl.cpython-34m.so: undefined symbol: SSLv2_method
We need to get that SSLv2 out of there!
>
> This leads to my suspicion that it's not really grafting's fault here,
> it's the *removal* of a piece of code, thus making things
> abi-incompatible with the system we built. Hopefully most grafting
> situations won't require this. I think that's right? :)
It seems like most security updates are ABI compatible, thankfully. This
was a special case since it removed an insecure protocol completely.
>
> Unfortunately, I'd say that ssl and python isn't really optional!
> Is it possible to graft on top of a graft? Could we rebuild Python
> based on the grafted openssl, and then graft things on top of the
> grafted Python? :)
>
>
>
Information forwarded
to
bug-guix <at> gnu.org
:
bug#22876
; Package
guix
.
(Wed, 02 Mar 2016 00:41:01 GMT)
Full text and
rfc822 format available.
Message #17 received at 22876 <at> debbugs.gnu.org (full text, mbox):
On Tue, Mar 01, 2016 at 04:13:41PM -0800, Christopher Allan Webber wrote:
> This leads to my suspicion that it's not really grafting's fault here,
> it's the *removal* of a piece of code, thus making things
> abi-incompatible with the system we built. Hopefully most grafting
> situations won't require this. I think that's right? :)
At least we know grafting works now. :)
> Unfortunately, I'd say that ssl and python isn't really optional!
> Is it possible to graft on top of a graft? Could we rebuild Python
> based on the grafted openssl, and then graft things on top of the
> grafted Python? :)
I'm super curious now, are grafts composable? What if we need to graft multiple
packages for whatever reason (static linking comes to mind.)
Jookia.
Reply sent
to
Mark H Weaver <mhw <at> netris.org>
:
You have taken responsibility.
(Wed, 02 Mar 2016 09:18:02 GMT)
Full text and
rfc822 format available.
Notification sent
to
Christopher Allan Webber <cwebber <at> dustycloud.org>
:
bug acknowledged by developer.
(Wed, 02 Mar 2016 09:18:02 GMT)
Full text and
rfc822 format available.
Message #22 received at 22876-done <at> debbugs.gnu.org (full text, mbox):
In brief, I believe this is fixed by commit 03a0e682a8 on master.
See below for details.
Christopher Allan Webber <cwebber <at> dustycloud.org> writes:
> Christopher Allan Webber writes:
>
>> Most of Guix seems to be working just fine with the grafts support and
>> grafting of openssl. However, unlike most grafts that will be done
>> probably, this one removes a feature, and that seems to be creating
>> problems in Python land.
>>
>> >>> from urllib.request import HTTPSHandler
>> Traceback (most recent call last):
>> File "<stdin>", line 1, in <module>
>> ImportError: cannot import name 'HTTPSHandler'
>
> As expected, this is for the same reasons offlineimap seemed to have
> problems:
>
> cwebber <at> oolong:~/devel/mediagoblin$ python3
> Python 3.4.3 (default, Jan 1 1970, 00:00:01)
> [GCC 4.9.3] on linux
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import ssl
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> File "/gnu/store/1spkp48cbbzg6ic5qkv3qpm3mvsgwkys-python-3.4.3/lib/python3.4/ssl.py", line 97, in <module>
> import _ssl # if we can't import it, let the error propagate
> ImportError: /gnu/store/1spkp48cbbzg6ic5qkv3qpm3mvsgwkys-python-3.4.3/lib/python3.4/lib-dynload/_ssl.cpython-34m.so: undefined symbol: SSLv2_method
>
> This leads to my suspicion that it's not really grafting's fault here,
> it's the *removal* of a piece of code, thus making things
> abi-incompatible with the system we built.
That's exactly right, and it turns out that Guix is not the only one who
was bitten by this, e.g.:
https://bugzilla.redhat.com/show_bug.cgi?id=1313509
https://bodhi.fedoraproject.org/updates/openssl-1.0.2g-1.fc23#comment-395291
https://forums.gentoo.org/viewtopic-p-7886940.html
I believe this issue is fixed by commit 03a0e682a8 on master. In my
tests, I found that it fixes offlineimap, virtualenv, and importing
HTTPSHandler from urllib.request.
The fix is simply to add "enable-ssl2" to the arguments passed to the
OpenSSL ./config script. I concluded that this is safe based on the
following excerpt from the CHANGES file:
* Disable SSLv2 default build, default negotiation and weak ciphers. SSLv2
is by default disabled at build-time. Builds that are not configured with
"enable-ssl2" will not support SSLv2. Even if "enable-ssl2" is used,
users who want to negotiate SSLv2 via the version-flexible SSLv23_method()
will need to explicitly call either of:
SSL_CTX_clear_options(ctx, SSL_OP_NO_SSLv2);
or
SSL_clear_options(ssl, SSL_OP_NO_SSLv2);
as appropriate. Even if either of those is used, or the application
explicitly uses the version-specific SSLv2_method() or its client and
server variants, SSLv2 ciphers vulnerable to exhaustive search key
recovery have been removed. Specifically, the SSLv2 40-bit EXPORT
ciphers, and SSLv2 56-bit DES are no longer available.
(CVE-2016-0800)
[Viktor Dukhovni]
Note that the "enable-ssl2" option is only needed when grafting, because
that's the only case where we need to preserve ABI compatibility. I've
verified that 'offlineimap' works on the security-updates branch, where
openssl-1.0.2g has been updated in the normal way, without grafting and
without the "enable-ssl2" option.
> Hopefully most grafting situations won't require this. I think that's
> right? :)
Yes. When grafting, we must ensure ABI compatibility. The mistake here
was that the ABI of the grafted OpenSSL was different than the one it
replaced. Like Fedora and Gentoo, we expected upstream to ensure that
1.0.2g was ABI compatible with 1.0.2f. I believe this was a reasonable
expectation.
> Is it possible to graft on top of a graft?
Good question, I don't know! I guess we should test this, for the sake
of robustness, but on the other hand, I don't see a practical need for
this feature. In general, we can simply update the replacement package,
which is what I've done in 03a0e682a8.
I'm closing this bug, but feel free to re-open it if you find that
problems remain.
Thanks!
Mark
Information forwarded
to
bug-guix <at> gnu.org
:
bug#22876
; Package
guix
.
(Wed, 02 Mar 2016 09:23:02 GMT)
Full text and
rfc822 format available.
Message #25 received at 22876 <at> debbugs.gnu.org (full text, mbox):
Jookia <166291 <at> gmail.com> skribis:
> On Tue, Mar 01, 2016 at 04:13:41PM -0800, Christopher Allan Webber wrote:
>> This leads to my suspicion that it's not really grafting's fault here,
>> it's the *removal* of a piece of code, thus making things
>> abi-incompatible with the system we built. Hopefully most grafting
>> situations won't require this. I think that's right? :)
>
> At least we know grafting works now. :)
Yay, we can experience ABI breaks like a real distro now! :-)
>> Unfortunately, I'd say that ssl and python isn't really optional!
>> Is it possible to graft on top of a graft? Could we rebuild Python
>> based on the grafted openssl, and then graft things on top of the
>> grafted Python? :)
>
> I'm super curious now, are grafts composable? What if we need to graft multiple
> packages for whatever reason (static linking comes to mind.)
It’s supposed to Just Work! That is, any number of packages can have a
‘replacement’ field, and the resulting grafts will be applied as needed.
Ludo’.
bug archived.
Request was from
Debbugs Internal Request <help-debbugs <at> gnu.org>
to
internal_control <at> debbugs.gnu.org
.
(Wed, 30 Mar 2016 11:24:03 GMT)
Full text and
rfc822 format available.
This bug report was last modified 8 years and 270 days ago.
Previous Next
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997,2003 nCipher Corporation Ltd,
1994-97 Ian Jackson.