GNU bug report logs - #38043
Incorrect Python byte-compiling for Python 3 and PyPy3

Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.

Package: automake; Reported by: Michał Górny <mgorny@HIDDEN>; dated Sun, 3 Nov 2019 10:47:02 UTC; Maintainer for automake is bug-automake@HIDDEN.

Message received at 38043 <at> debbugs.gnu.org:


Received: (at 38043) by debbugs.gnu.org; 3 Nov 2019 11:34:42 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Nov 03 06:34:42 2019
Received: from localhost ([127.0.0.1]:33438 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1iRE9Z-0006ab-5c
	for submit <at> debbugs.gnu.org; Sun, 03 Nov 2019 06:34:42 -0500
Received: from smtp.gentoo.org ([140.211.166.183]:45812)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mgorny@HIDDEN>) id 1iRE9W-0006aK-JD
 for 38043 <at> debbugs.gnu.org; Sun, 03 Nov 2019 06:34:39 -0500
Received: from pomiot (c134-66.icpnet.pl [85.221.134.66])
 (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))
 (No client certificate requested) (Authenticated sender: mgorny)
 by smtp.gentoo.org (Postfix) with ESMTPSA id 0D49C34C788;
 Sun,  3 Nov 2019 11:34:30 +0000 (UTC)
Message-ID: <e65b90abb600ae6b200f089f5e12a78200ec5036.camel@HIDDEN>
Subject: Re: bug#38043: Acknowledgement (Incorrect Python byte-compiling for
 Python 3 and PyPy3)
From: =?UTF-8?Q?Micha=C5=82_G=C3=B3rny?= <mgorny@HIDDEN>
To: 38043 <at> debbugs.gnu.org
Date: Sun, 03 Nov 2019 12:34:26 +0100
In-Reply-To: <handler.38043.B.157277797220003.ack <at> debbugs.gnu.org>
References: <e8f105b6a5ce56f8957de114444570473506ebb0.camel@HIDDEN>
 <handler.38043.B.157277797220003.ack <at> debbugs.gnu.org>
Organization: Gentoo
Content-Type: multipart/signed; micalg="pgp-sha512";
 protocol="application/pgp-signature"; boundary="=-N3VT4FehSvnUoAI53u4H"
User-Agent: Evolution 3.32.4 
MIME-Version: 1.0
X-Spam-Score: -5.0 (-----)
X-Debbugs-Envelope-To: 38043
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -6.0 (------)


--=-N3VT4FehSvnUoAI53u4H
Content-Type: multipart/mixed; boundary="=-yBlW+Z/7L8F0NS7x/6En"


--=-yBlW+Z/7L8F0NS7x/6En
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

I'm attaching a reference patch that fixes the issue for me.  However,
I'm sorry but I don't have the time to work on adding a test for it.

--=20
Best regards,
Micha=C5=82 G=C3=B3rny


--=-yBlW+Z/7L8F0NS7x/6En
Content-Disposition: attachment;
	filename="0001-py-compile-Support-OO-for-py3.5-and-O-OO-for-pypy3.patch"
Content-Transfer-Encoding: base64
Content-Type: text/x-patch;
	name="0001-py-compile-Support-OO-for-py3.5-and-O-OO-for-pypy3.patch";
	charset="UTF-8"

RnJvbSBmYzAyMzk4ZmJiNWYxYTUwMzBhNzIwNWQyMjc5MGIwMjg4ZWM1YTJjIE1vbiBTZXAgMTcg
MDA6MDA6MDAgMjAwMQpGcm9tOiA9P1VURi04P3E/TWljaGE9QzU9ODI9MjBHPUMzPUIzcm55Pz0g
PG1nb3JueUBnZW50b28ub3JnPgpEYXRlOiBTdW4sIDMgTm92IDIwMTkgMTE6NTE6MTkgKzAxMDAK
U3ViamVjdDogW1BBVENIXSBweS1jb21waWxlOiBTdXBwb3J0IC1PTyBmb3IgcHkzLjUrLCBhbmQg
LU8mLU9PIGZvciBweXB5MwoKQWRkIGFuIGFkZGl0aW9uYWwgY29tcGlsYXRpb24gYmxvY2sgdG8g
cHJvZHVjZSAtT08gb3B0aW1pemVkIG1vZHVsZXMKZm9yIFB5dGhvbiAzLjUrLiAgRW5hYmxlIGJv
dGggLU8gYW5kIC1PTyBmb3IgUHlQeTMuNSsuICBSZW1vdmUKdGhlIHNlY29uZCBhcmd1bWVudCB0
byBpbXAuY2FjaGVfZnJvbV9zb3VyY2UoKSBpbiBib3RoIC1PIGFuZCAtT08gYmxvY2tzCmFzIHRo
ZXkgZG8gbm90IHNlZW0gbmVjZXNzYXJ5IGFuZCBicmVhayAtT08gYnl0ZS1jb21waWxhdGlvbi4K
CkZpeGVzIGF1dG9tYWtlIGJ1ZyAjMzgwNDMKLS0tCiBsaWIvcHktY29tcGlsZSB8IDI3ICsrKysr
KysrKysrKysrKysrKysrKysrKystLQogMSBmaWxlIGNoYW5nZWQsIDI1IGluc2VydGlvbnMoKyks
IDIgZGVsZXRpb25zKC0pCgpkaWZmIC0tZ2l0IGEvbGliL3B5LWNvbXBpbGUgYi9saWIvcHktY29t
cGlsZQppbmRleCAxYTNmNGQ2NDQuLjQ0YmU5N2ZjNCAxMDA3NTUKLS0tIGEvbGliL3B5LWNvbXBp
bGUKKysrIGIvbGliL3B5LWNvbXBpbGUKQEAgLTE0MCw3ICsxNDAsNyBAQCAkUFlUSE9OIC1PIC1j
ICIKIGltcG9ydCBzeXMsIG9zLCBweV9jb21waWxlLCBpbXAKIAogIyBweXB5IGRvZXMgbm90IHVz
ZSAucHlvIG9wdGltaXphdGlvbgotaWYgaGFzYXR0cihzeXMsICdweXB5X3RyYW5zbGF0aW9uX2lu
Zm8nKToKK2lmIGhhc2F0dHIoc3lzLCAncHlweV90cmFuc2xhdGlvbl9pbmZvJykgYW5kIHN5cy5o
ZXh2ZXJzaW9uIDwgMHgwMzA1MDAwMDoKICAgICBzeXMuZXhpdCgwKQogCiBmaWxlcyA9ICcnJyRm
aWxlcycnJwpAQCAtMTU0LDcgKzE1NCwzMCBAQCBmb3IgZmlsZSBpbiBmaWxlcy5zcGxpdCgpOgog
ICAgIHN5cy5zdGRvdXQud3JpdGUoZmlsZSkKICAgICBzeXMuc3Rkb3V0LmZsdXNoKCkKICAgICBp
ZiBoYXNhdHRyKGltcCwgJ2dldF90YWcnKToKLSAgICAgICAgcHlfY29tcGlsZS5jb21waWxlKGZp
bGVwYXRoLCBpbXAuY2FjaGVfZnJvbV9zb3VyY2UoZmlsZXBhdGgsIEZhbHNlKSwgcGF0aCkKKyAg
ICAgICAgcHlfY29tcGlsZS5jb21waWxlKGZpbGVwYXRoLCBpbXAuY2FjaGVfZnJvbV9zb3VyY2Uo
ZmlsZXBhdGgpLCBwYXRoKQorICAgIGVsc2U6CisgICAgICAgIHB5X2NvbXBpbGUuY29tcGlsZShm
aWxlcGF0aCwgZmlsZXBhdGggKyAnbycsIHBhdGgpCitzeXMuc3Rkb3V0LndyaXRlKCdcbicpIiAy
Pi9kZXYvbnVsbCB8fCA6CisKKyRQWVRIT04gLU9PIC1jICIKK2ltcG9ydCBzeXMsIG9zLCBweV9j
b21waWxlLCBpbXAKKworIyBweXRob248My41IGRvZXMgbm90IGhhdmUgc3BsaXQgZmlsZXMgZm9y
IC1PIGFuZCAtT08KK2lmIHN5cy5oZXh2ZXJzaW9uIDwgMHgwMzA1MDAwMDoKKyAgICBzeXMuZXhp
dCgwKQorCitmaWxlcyA9ICcnJyRmaWxlcycnJworc3lzLnN0ZG91dC53cml0ZSgnQnl0ZS1jb21w
aWxpbmcgcHl0aG9uIG1vZHVsZXMgKC1PTyB2ZXJzaW9ucykgLi4uXG4nKQorZm9yIGZpbGUgaW4g
ZmlsZXMuc3BsaXQoKToKKyAgICAkcGF0aHRyYW5zCisgICAgJGZpbGV0cmFucworICAgIGlmIG5v
dCBvcy5wYXRoLmV4aXN0cyhmaWxlcGF0aCkgb3Igbm90IChsZW4oZmlsZXBhdGgpID49IDMKKyAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgYW5kIGZpbGVwYXRoWy0z
Ol0gPT0gJy5weScpOgorCSAgICBjb250aW51ZQorICAgIHN5cy5zdGRvdXQud3JpdGUoZmlsZSkK
KyAgICBzeXMuc3Rkb3V0LmZsdXNoKCkKKyAgICBpZiBoYXNhdHRyKGltcCwgJ2dldF90YWcnKToK
KyAgICAgICAgcHlfY29tcGlsZS5jb21waWxlKGZpbGVwYXRoLCBpbXAuY2FjaGVfZnJvbV9zb3Vy
Y2UoZmlsZXBhdGgpLCBwYXRoKQogICAgIGVsc2U6CiAgICAgICAgIHB5X2NvbXBpbGUuY29tcGls
ZShmaWxlcGF0aCwgZmlsZXBhdGggKyAnbycsIHBhdGgpCiBzeXMuc3Rkb3V0LndyaXRlKCdcbicp
IiAyPi9kZXYvbnVsbCB8fCA6Ci0tIAoyLjIzLjAKCg==


--=-yBlW+Z/7L8F0NS7x/6En--

--=-N3VT4FehSvnUoAI53u4H
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: This is a digitally signed message part
Content-Transfer-Encoding: 7bit

-----BEGIN PGP SIGNATURE-----

iQGTBAABCgB9FiEEx2qEUJQJjSjMiybFY5ra4jKeJA4FAl2+u0NfFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEM3
NkE4NDUwOTQwOThEMjhDQzhCMjZDNTYzOUFEQUUyMzI5RTI0MEUACgkQY5ra4jKe
JA5khwf/URpC6A/6gNt23wEvLAfzv+5ReOMqL8tpbQhIo3/bdsZm1CPb4NXfSksn
N2Wj+DCF6/zAK/qJqVtaBg1/9hYST3f/NwC11sryCzRSFPh2mVg21m0o1AY8S/36
caz3cWO14LePWZTpGlbb4s4XrpzOOl38P/qtHiKKmHAVy85mzqpwYNp8MKdrLuHs
cDY1LU1OpykoHzsw3I2agLbE4ActiEp3luVjiHgZ0dVangEgqL3F00XkwuE3C1Ht
iR2EyXmog1/Neoji+OkrEr1mpZ695KADlUxX9A8FlLJ/96o/C+VTOtTprjD0cs+P
JxuLnimViD/3BGWfQROiwGPIiISNCg==
=caS4
-----END PGP SIGNATURE-----

--=-N3VT4FehSvnUoAI53u4H--





Information forwarded to bug-automake@HIDDEN:
bug#38043; Package automake. Full text available.

Message received at submit <at> debbugs.gnu.org:


Received: (at submit) by debbugs.gnu.org; 3 Nov 2019 10:46:12 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Nov 03 05:46:12 2019
Received: from localhost ([127.0.0.1]:33423 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1iRDOe-0005CZ-9M
	for submit <at> debbugs.gnu.org; Sun, 03 Nov 2019 05:46:12 -0500
Received: from lists.gnu.org ([209.51.188.17]:50597)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mgorny@HIDDEN>) id 1iRDOb-0005CP-Sd
 for submit <at> debbugs.gnu.org; Sun, 03 Nov 2019 05:46:10 -0500
Received: from eggs.gnu.org ([2001:470:142:3::10]:49987)
 by lists.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <mgorny@HIDDEN>) id 1iRDOP-0004nP-Sk
 for bug-automake@HIDDEN; Sun, 03 Nov 2019 05:46:04 -0500
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_50,RCVD_IN_DNSWL_HI,
 URIBL_BLOCKED autolearn=disabled version=3.3.2
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <mgorny@HIDDEN>) id 1iRDOL-0005Vg-JK
 for bug-automake@HIDDEN; Sun, 03 Nov 2019 05:45:57 -0500
Received: from smtp.gentoo.org ([140.211.166.183]:52868)
 by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_128_CBC_SHA1:16)
 (Exim 4.71) (envelope-from <mgorny@HIDDEN>) id 1iRDOL-0005RA-Af
 for bug-automake@HIDDEN; Sun, 03 Nov 2019 05:45:53 -0500
Received: from pomiot (c134-66.icpnet.pl [85.221.134.66])
 (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))
 (No client certificate requested) (Authenticated sender: mgorny)
 by smtp.gentoo.org (Postfix) with ESMTPSA id A3FE934C78D;
 Sun,  3 Nov 2019 10:45:43 +0000 (UTC)
Message-ID: <e8f105b6a5ce56f8957de114444570473506ebb0.camel@HIDDEN>
Subject: Incorrect Python byte-compiling for Python 3 and PyPy3
From: =?UTF-8?Q?Micha=C5=82_G=C3=B3rny?= <mgorny@HIDDEN>
To: bug-automake@HIDDEN
Date: Sun, 03 Nov 2019 11:45:39 +0100
Organization: Gentoo
Content-Type: multipart/signed; micalg="pgp-sha512";
 protocol="application/pgp-signature"; boundary="=-41qjJzF6FkmcyA/xg3DO"
User-Agent: Evolution 3.32.4 
MIME-Version: 1.0
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
 recognized.
X-Received-From: 140.211.166.183
X-Spam-Score: -1.6 (-)
X-Debbugs-Envelope-To: submit
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -2.6 (--)


--=-41qjJzF6FkmcyA/xg3DO
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Hello,

I've noticed that the logic in py-compile is built on assumptions from
Python 2 and does not fit Python 3 well.  Notably, there are two or
three bugs here:

1. .opt-2 (-OO) level is not compiled for py3.5+.

2. .opt-1 (-O) and .opt-2 (-OO) are wrongly skipped for PyPy3.


Firstly, the code in py-compile byte-compiles in two steps: without '-O'=
=20
and with '-O'.  This is correct for Python <3.5 since '-O' and '-OO'
share the same file (.pyo).  However, since Python 3.5 they are written
into separate '.opt-1.pyc' and '.opt-2.pyc' files and therefore can
coexist.

It is therefore necessary to perform another byte-compilation step with
'-OO', so that all levels of optimization are installed and the module
can be correctly loaded from cache independently of Python options used.


Secondly, the code in '-O' block skips PyPy based on the existence of
sys.pypy_translation_info.  This is correct for PyPy2 that doesn't write
optimized caches.  However, PyPy3 (in particular versions based
on Python 3.5 and newer, I'm not sure about the older versions) do use
the same split naming scheme as CPython.

Therefore, the check here needs to be modified to apply only to Python 2
versions of PyPy.


To reproduce:

touch mytest.py
cat > configure.ac <<EOF
AC_INIT([test], [0])
AM_INIT_AUTOMAKE([foreign])
AM_PATH_PYTHON
AC_CONFIG_FILES([Makefile])
AC_OUTPUT
EOF
cat > Makefile.am <<EOF
pyexec_PYTHON =3D mytest.py
EOF
autoreconf -i
./configure PYTHON=3D...
make DESTDIR=3D... install


With Python 3.5+, the following files are installed:
/tmp/z1/usr/local/lib64/python3.5/site-packages/mytest.py
/tmp/z1/usr/local/lib64/python3.5/site-packages/__pycache__/mytest.cpython-=
35.opt-1.pyc
/tmp/z1/usr/local/lib64/python3.5/site-packages/__pycache__/mytest.cpython-=
35.pyc

What should additionally be installed is:
/tmp/z1/usr/local/lib64/python3.5/site-packages/__pycache__/mytest.cpython-=
35.opt-2.pyc


With PyPy3.5+, the following files are installed:
/tmp/z2/usr/local/lib/python3.6/site-packages/mytest.py
/tmp/z2/usr/local/lib/python3.6/site-packages/__pycache__/mytest.pypy3-72.p=
yc

What should additionally be installed is:
/tmp/z2/usr/local/lib/python3.6/site-packages/__pycache__/mytest.pypy3-72.o=
pt-1.pyc
/tmp/z2/usr/local/lib/python3.6/site-packages/__pycache__/mytest.pypy3-72.o=
pt-2.pyc


Please let me know if you need any more details.

--=20
Best regards,
Micha=C5=82 G=C3=B3rny


--=-41qjJzF6FkmcyA/xg3DO
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: This is a digitally signed message part
Content-Transfer-Encoding: 7bit

-----BEGIN PGP SIGNATURE-----

iQGTBAABCgB9FiEEx2qEUJQJjSjMiybFY5ra4jKeJA4FAl2+r9NfFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEM3
NkE4NDUwOTQwOThEMjhDQzhCMjZDNTYzOUFEQUUyMzI5RTI0MEUACgkQY5ra4jKe
JA5FIggAvfuvu5/VTEndXVwIGeq6FRNDskWa0Uvl69pRxC/2fNdgAf4XhhL3FfJS
3QHp/rapzHMTJtroEMOuIWQ9505A91L7izELB7KOB+gU6iwcMbzFUMZzzq3n1zsn
17xkEyy6zbS1hpbzAi5cAZDsbJpe6GSdkH/XhBKgED1HnUX3SJ+P/MOnQ6I8Zok6
gzXKQbbOpk+Kh90lrobII5jV9O4RDB7JefC6PckdpWZj7m15pbuWEOneGsH8BglW
n+x0SaXjKuQmttgeJdhuOqLnv+DEQ679Qq1O2Qt2ehWtXvepyfgfCly4GQL0sOMD
L+hfgarncM7+yNnt3X0+FcyLZ3Kxow==
=Jkx8
-----END PGP SIGNATURE-----

--=-41qjJzF6FkmcyA/xg3DO--





Acknowledgement sent to Michał Górny <mgorny@HIDDEN>:
New bug report received and forwarded. Copy sent to bug-automake@HIDDEN. Full text available.
Report forwarded to bug-automake@HIDDEN:
bug#38043; Package automake. Full text available.
Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.
Last modified: Mon, 25 Nov 2019 12:00:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.