GNU bug report logs - #38043
Incorrect Python byte-compiling for Python 3.5+ and PyPy3

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 38043 in the body.
You can then email your comments to 38043 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Michał Górny <mgorny <at> gentoo.org>
To: bug-automake <at> gnu.org
Subject: Incorrect Python byte-compiling for Python 3 and PyPy3
Date: Sun, 03 Nov 2019 11:45:39 +0100

[Message part 1 (text/plain, inline)]

Hello,

I've noticed that the logic in py-compile is built on assumptions from
Python 2 and does not fit Python 3 well.  Notably, there are two or
three bugs here:

1. .opt-2 (-OO) level is not compiled for py3.5+.

2. .opt-1 (-O) and .opt-2 (-OO) are wrongly skipped for PyPy3.


Firstly, the code in py-compile byte-compiles in two steps: without '-O' 
and with '-O'.  This is correct for Python <3.5 since '-O' and '-OO'
share the same file (.pyo).  However, since Python 3.5 they are written
into separate '.opt-1.pyc' and '.opt-2.pyc' files and therefore can
coexist.

It is therefore necessary to perform another byte-compilation step with
'-OO', so that all levels of optimization are installed and the module
can be correctly loaded from cache independently of Python options used.


Secondly, the code in '-O' block skips PyPy based on the existence of
sys.pypy_translation_info.  This is correct for PyPy2 that doesn't write
optimized caches.  However, PyPy3 (in particular versions based
on Python 3.5 and newer, I'm not sure about the older versions) do use
the same split naming scheme as CPython.

Therefore, the check here needs to be modified to apply only to Python 2
versions of PyPy.


To reproduce:

touch mytest.py
cat > configure.ac <<EOF
AC_INIT([test], [0])
AM_INIT_AUTOMAKE([foreign])
AM_PATH_PYTHON
AC_CONFIG_FILES([Makefile])
AC_OUTPUT
EOF
cat > Makefile.am <<EOF
pyexec_PYTHON = mytest.py
EOF
autoreconf -i
./configure PYTHON=...
make DESTDIR=... install


With Python 3.5+, the following files are installed:
/tmp/z1/usr/local/lib64/python3.5/site-packages/mytest.py
/tmp/z1/usr/local/lib64/python3.5/site-packages/__pycache__/mytest.cpython-35.opt-1.pyc
/tmp/z1/usr/local/lib64/python3.5/site-packages/__pycache__/mytest.cpython-35.pyc

What should additionally be installed is:
/tmp/z1/usr/local/lib64/python3.5/site-packages/__pycache__/mytest.cpython-35.opt-2.pyc


With PyPy3.5+, the following files are installed:
/tmp/z2/usr/local/lib/python3.6/site-packages/mytest.py
/tmp/z2/usr/local/lib/python3.6/site-packages/__pycache__/mytest.pypy3-72.pyc

What should additionally be installed is:
/tmp/z2/usr/local/lib/python3.6/site-packages/__pycache__/mytest.pypy3-72.opt-1.pyc
/tmp/z2/usr/local/lib/python3.6/site-packages/__pycache__/mytest.pypy3-72.opt-2.pyc


Please let me know if you need any more details.

-- 
Best regards,
Michał Górny

[signature.asc (application/pgp-signature, inline)]

Message #8 received at 38043 <at> debbugs.gnu.org (full text, mbox):

From: Michał Górny <mgorny <at> gentoo.org>
To: 38043 <at> debbugs.gnu.org
Subject: Re: bug#38043: Acknowledgement (Incorrect Python byte-compiling for
 Python 3 and PyPy3)
Date: Sun, 03 Nov 2019 12:34:26 +0100

[Message part 1 (text/plain, inline)]

I'm attaching a reference patch that fixes the issue for me.  However,
I'm sorry but I don't have the time to work on adding a test for it.

-- 
Best regards,
Michał Górny

[0001-py-compile-Support-OO-for-py3.5-and-O-OO-for-pypy3.patch (text/x-patch, attachment)]

[signature.asc (application/pgp-signature, inline)]

Message #11 received at 38043 <at> debbugs.gnu.org (full text, mbox):

From: Mike Frysinger <vapier <at> gentoo.org>
To: Michał Górny <mgorny <at> gentoo.org>
Cc: 38043 <at> debbugs.gnu.org
Subject: Re: bug#38043: Incorrect Python byte-compiling for Python 3 and PyPy3
Date: Tue, 18 Jan 2022 01:51:35 -0500

On 03 Nov 2019 11:45, Michał Górny wrote:
> I've noticed that the logic in py-compile is built on assumptions from
> Python 2 and does not fit Python 3 well.  Notably, there are two or
> three bugs here:
> 
> 1. .opt-2 (-OO) level is not compiled for py3.5+.
> 
> 2. .opt-1 (-O) and .opt-2 (-OO) are wrongly skipped for PyPy3.

i'm inclined to drop support for <Python-3.6.  i know even 3.6 has reached
EOL upstream now (as of Sep 2021), but might as well pick a fairly stable
version as a base to cut off somewhere for now.
-mike

Message #14 received at 38043 <at> debbugs.gnu.org (full text, mbox):

From: Karl Berry <karl <at> freefriends.org>
To: vapier <at> gentoo.org
Cc: mgorny <at> gentoo.org, 38043 <at> debbugs.gnu.org
Subject: Re: bug#38043: Incorrect Python byte-compiling for Python 3 and PyPy3
Date: Tue, 18 Jan 2022 16:15:08 -0700

    i'm inclined to drop support for <Python-3.6.

My recollection is, plenty of people are still using <3.6, EOL or not.
Tempting though it is.

    Subject: bug#38043: Incorrect Python byte-compiling for Python 3 and PyPy3

For this particular bug, I have a half-baked version of py-compile
integrating the various changes that have come up. It is not fun, but it
doesn't look to me like it is so hard that we need to force no more
support for older python. I'll try to get back to it, or at least send
it to you for further work :). --thanks, karl.

Message #17 received at 38043 <at> debbugs.gnu.org (full text, mbox):

From: Mike Frysinger <vapier <at> gentoo.org>
To: Karl Berry <karl <at> freefriends.org>
Cc: mgorny <at> gentoo.org, 38043 <at> debbugs.gnu.org
Subject: Python version language support
Date: Tue, 18 Jan 2022 23:45:35 -0500

[Message part 1 (text/plain, inline)]

On 18 Jan 2022 16:15, Karl Berry wrote:
>     i'm inclined to drop support for <Python-3.6.
> 
> My recollection is, plenty of people are still using <3.6, EOL or not.
> Tempting though it is.

dissertation follows.  i know it's a lot, and i don't normally like writing
this much at a time, but i think this is something we need to agree to & write
down because it's a promise/commitment to users, and will have very long-term
implications for us.  if you want to skip to the end for "My proprosal", feel
free, and if you disagree with parts, you can backup and read my logic :p.

i don't think we can reasonably support every version of Python with the latest
automake release forever.  we don't even support every version now, and i don't
think we're going to backfill.  fairly certain we don't work with Python 0.x,
or Python 1.x, and i suspect Python 2.[0-6] is rotting.

so we already have a version cut off line.  the question is where do we draw
that line, and when do we move it ?

let's say that we never remove support for versions we've added.  i.e. once a
version is included, it'll always be here.  how do we make sure it continues
to work ?  practically speaking, it's impossible for us to test.  many distros
have already long ago deleted Python 3.[012], are in the process of deleting
Python 2.7 now, and it's only getting worse.  are automake developers expected
to manually download & build every Python version ?  and when those versions
stop compiling against current C runtimes (compilers/libraries), they have to
backport fixes to get them working ?  for a diff project, i've already done
this, and i know that old releases do not unpack+configure+compile+execute
without backports (as in, `./python` segfaults).

now you might argue that devs can install old distros.  doing that in a chroot
or in a container (e.g. docker) doesn't work as the kernel removes/disables
features that old binaries need.  i've seen this with CONFIG_COMPAT_VDSO being
removed -- old binaries now just segfault.  which means you need to downgrade
your kernel and/or re-enable settings that include known security bugs.

now you might argue that devs can always install old distros in VMs and thus
isolate themselves.  but those VMs will have known security issues at the
security level (i.e. OpenSSH and such have known exploits).  ok ok, you should
have those VMs be firewalled, and only accessible from localhost.

hopefully you can still find install media for these distros, and their network
packages can still be installed.  except their root CA store is going to be
expired, and they'll be unable to access any HTTPS sites, so they won't be able
to download anything from the network.

let's back up a little.  automake doesn't exist for its own sake, it exists
purely for its users (other developers).  we add & maintain features purely
because they want them.  so what do those developers want ?

do they want to support Python 2.[0-7] & 3.[0-9] in a single release of their
own software ?  in my experience, that pretty much never happens.  Python devs
tend to say "we've dropped support for Python X.Y with current versions, so if
you want that, you'll need to download an old release".

i have seen projects doing the Python 2.7 + Python 3.x support dance, but that
has largely evaporated with Python itself finally cutting Python 2 off.  now i
only see projects saying "if you want Python 2, grab an old release".

similarly, while Python 3.[0-9] has been done by some projects, even that is
evaporating.  i think that was largely predicated on the Python project being
very backwards compatible and not making breaking changes, and they were doing
that only as a carrot to get people off of Python 2.  now that they've killed
Python 2, they're starting to make breaking changes with Python 3.10+, and that
is only going to accelerate, which means projects are going to realize that the
effort to support Python 3.0 - 3.10+ in a single codebase is a lot harder than
it used to be.

now throw in alternative runtimes (cython, pypy, jython, etc...) and the variety
of virtual environments (venvs).  are we going to support those forever too ?

so if our users don't want every Python version, and keeping support for old
versions is impossible for us to validate, when can we move the post ?

to help the discussion, i indexed Python versions in Debian & Ubuntu releases
starting from ~2008.  from that we can <Python-2.5 is already gone, and that
<Python-2.7 is gone by 2016 (Debian Squeeze really dragged 2.6 out).  we see
that Python 3.0 never makes an appearance, and Python 3.1 is gone by 2016.
https://gerrit.googlesource.com/git-repo/+/HEAD/docs/release-process.md#Project-References

alternatively, do we take a similar position to Python projects ?  if you want
to support a Python version X.Y, use an old release of automake ?

# My proprosal

what if we said something like:
Automake guarantees releases will support all Python versions that are still
supported by the Python project at the time of the Automake release.  Support
for EOL versions of Python are not guaranteed, but will be considered as long
as it is not onerous to do so, and there are large supported distros including
them.  If you need to support older Python versions, please use a previous
Automake release.

that means right now:
* we have to support Python 3.7 (it goes EOL upstream in 2023)
* we can probably keep Python 3.6 easily enough
* Python 3.5 probably should be kept since it's in Ubuntu Xenial, and that
  doesn't go EOL until Apr 2024
* Python 3.4 might be kept since it's in Ubuntu Trusty, and that doesn't go
  EOL until Apr 2022
* Python 2.7 might be kept since it's in Debian Bullseye, and that doesn't go
  EOL until Apr 2026
* Python 3.[0-3] can be kept as long as they don't require unique changes
* Python 2.[0-6] can be kept as long as they don't require unique changes
* as soon as we get reports that Python 3.[0-3] or 2.[0-6] don't work, we
  respond by deleting them.  or if we make changes that seem like they won't
  work, we delete them.
* we update the manual to state clearly:
  * Python 2.7 & 3.4 - 3.10 are fully supported
  * Python 2.0 - 2.6 & 3.0 - 3.3 should work, but are not tested
  * Python <2.0 do not work are not supported at all
-mike

[signature.asc (application/pgp-signature, inline)]

Message #20 received at 38043 <at> debbugs.gnu.org (full text, mbox):

From: Karl Berry <karl <at> freefriends.org>
To: vapier <at> gentoo.org
Cc: mgorny <at> gentoo.org, 38043 <at> debbugs.gnu.org
Subject: Re: Python version language support
Date: Wed, 19 Jan 2022 19:32:50 -0700

    Automake guarantees releases will support all Python versions that
    are still supported by the Python project at the time of the
    Automake release.  Support for EOL versions of Python are not
    guaranteed, but will be considered as long as it is not onerous to
    do so, and there are large supported distros including them.  If you
    need to support older Python versions, please use a previous
    Automake release.

That sounds like a sensible approach to me. Jim?

    * we update the manual to state clearly:
      * Python 2.7 & 3.4 - 3.10 are fully supported
      * Python 2.0 - 2.6 & 3.0 - 3.3 should work, but are not tested
      * Python <2.0 do not work are not supported at all

I agree it would be useful to be explicit about Python versions in the
manual.

Personally, I would find it impossible to track what Python versions
require what and what we support, but as long as you can make the needed
updates to the manual at each release time (something to be added to the
release list ... Jim?), sounds good to me.

I would add that we don't object to installing patches for older
versions, or for whatever other reason, as long as the patches are
understandable and maintainable. In practice, that is what we (I) do
now.  I have little understanding of (or interest in) Python, but if
people send usable patches, I'm happy to install them. What any given
patch might mean for version support, I don't even begin to figure out.

Thanks,
Karl

Message #23 received at 38043 <at> debbugs.gnu.org (full text, mbox):

From: Mike Frysinger <vapier <at> gentoo.org>
To: 38043 <at> debbugs.gnu.org
Subject: [PATCH 1/6] py-compile: clean up usage info
Date: Sun,  6 Feb 2022 01:57:09 -0500

Include the full summary of options in the output.

* lib/py-compile: Update usage output.
* t/py-compile-usage.sh: Update test to match new output.
---
 lib/py-compile        | 8 +++++++-
 t/py-compile-usage.sh | 4 ++--
 2 files changed, 9 insertions(+), 3 deletions(-)

diff --git a/lib/py-compile b/lib/py-compile
index 0597c85b2147..4917cfed008c 100755
--- a/lib/py-compile
+++ b/lib/py-compile
@@ -62,13 +62,19 @@ while test $# -ne 0; do
       ;;
     -h|--help)
       cat <<\EOF
-Usage: py-compile [--help] [--version] [--basedir DIR] [--destdir DIR] FILES..."
+Usage: py-compile [options] FILES...
 
 Byte compile some python scripts FILES.  Use --destdir to specify any
 leading directory path to the FILES that you don't want to include in the
 byte compiled file.  Specify --basedir for any additional path information you
 do want to be shown in the byte compiled file.
 
+Options:
+  --basedir DIR   Prefix all FILES with DIR, and include in error messages.
+  --destdir DIR   Prefix all FILES with DIR before compiling.
+  -v, --version   Display version information.
+  -h, --help      This help screen.
+
 Example:
   py-compile --destdir /tmp/pkg-root --basedir /usr/share/test test.py test2.py
 
diff --git a/t/py-compile-usage.sh b/t/py-compile-usage.sh
index 94c0c4c9d7ba..f324ec9e6d52 100644
--- a/t/py-compile-usage.sh
+++ b/t/py-compile-usage.sh
@@ -29,8 +29,8 @@ cp "$am_scriptdir/py-compile" . \
 cat stdout
 test -s stderr && { cat stderr >&2; exit 1; }
 grep '^Usage: py-compile .' stdout
-$FGREP ' [--basedir DIR]' stdout
-$FGREP ' [--destdir DIR]' stdout
+$FGREP ' --basedir DIR ' stdout
+$FGREP ' --destdir DIR ' stdout
 
 # --version
 
-- 
2.34.1

Message #26 received at 38043 <at> debbugs.gnu.org (full text, mbox):

From: Mike Frysinger <vapier <at> gentoo.org>
To: 38043 <at> debbugs.gnu.org
Subject: [PATCH 2/6] py-compile: handle filenames with whitespace
Date: Sun,  6 Feb 2022 01:57:10 -0500

The list of files is put into a string and then split on whitespace.
Fix the way the list of files are passed to the compile script.

* lib/py-compile: Pass files as arguments, not as a string.
* t/py-compile-files.sh: New test.
---
 lib/py-compile        | 14 +++++---------
 t/py-compile-files.sh | 36 ++++++++++++++++++++++++++++++++++++
 2 files changed, 41 insertions(+), 9 deletions(-)
 create mode 100644 t/py-compile-files.sh

diff --git a/lib/py-compile b/lib/py-compile
index 4917cfed008c..0e1df24024e8 100755
--- a/lib/py-compile
+++ b/lib/py-compile
@@ -100,8 +100,7 @@ EOF
   shift
 done
 
-files=$*
-if test -z "$files"; then
+if test $# -eq 0; then
   usage_error "no files given"
 fi
 
@@ -143,10 +142,8 @@ fi
 $PYTHON -c "
 import sys, os, py_compile, $import_lib
 
-files = '''$files'''
-
 sys.stdout.write('Byte-compiling python modules...\n')
-for file in files.split():
+for file in sys.argv[1:]:
     $pathtrans
     $filetrans
     if not os.path.exists(filepath) or not (len(filepath) >= 3
@@ -158,7 +155,7 @@ for file in files.split():
         py_compile.compile(filepath, $import_call(filepath), path)
     else:
         py_compile.compile(filepath, filepath + 'c', path)
-sys.stdout.write('\n')" || exit $?
+sys.stdout.write('\n')" "$@" || exit $?
 
 # this will fail for python < 1.5, but that doesn't matter ...
 $PYTHON -O -c "
@@ -168,9 +165,8 @@ import sys, os, py_compile, $import_lib
 if hasattr(sys, 'pypy_translation_info'):
     sys.exit(0)
 
-files = '''$files'''
 sys.stdout.write('Byte-compiling python modules (optimized versions) ...\n')
-for file in files.split():
+for file in sys.argv[1:]:
     $pathtrans
     $filetrans
     if not os.path.exists(filepath) or not (len(filepath) >= 3
@@ -182,7 +178,7 @@ for file in files.split():
         py_compile.compile(filepath, $import_call(filepath$import_arg2), path)
     else:
         py_compile.compile(filepath, filepath + 'o', path)
-sys.stdout.write('\n')" 2>/dev/null || exit $?
+sys.stdout.write('\n')" "$@" 2>/dev/null || exit $?
 
 # Local Variables:
 # mode: shell-script
diff --git a/t/py-compile-files.sh b/t/py-compile-files.sh
new file mode 100644
index 000000000000..fa2dd3251372
--- /dev/null
+++ b/t/py-compile-files.sh
@@ -0,0 +1,36 @@
+#! /bin/sh
+# Copyright (C) 2022 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+# Verify 'py-compile' script can handle inputs with spaces, etc...
+
+required=python
+. test-init.sh
+
+cp "$am_scriptdir/py-compile" . \
+  || fatal_ "failed to fetch auxiliary script py-compile"
+
+# Create files that require proper quoting.
+mkdir "dir with spaces"
+touch "nospace.py" "has space.py" "*.py" "dir with spaces/|.py"
+
+./py-compile "nospace.py" "has space.py" "*.py" "dir with spaces/|.py"
+
+py_installed "nospace.pyc"
+py_installed "has space.pyc"
+py_installed "*.pyc"
+py_installed "dir with spaces/|.pyc"
+
+:
-- 
2.34.1

Message #29 received at 38043 <at> debbugs.gnu.org (full text, mbox):

From: Mike Frysinger <vapier <at> gentoo.org>
To: 38043 <at> debbugs.gnu.org
Subject: [PATCH 3/6] manual: document Python version support status
Date: Sun,  6 Feb 2022 01:57:11 -0500

Clarify to users what versions of Python are supported and until when.
This will make it easier for us to decide what versions to support.

* doc/automake.texi: Add Supported Python versions section.
---
 doc/automake.texi | 53 +++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 53 insertions(+)

diff --git a/doc/automake.texi b/doc/automake.texi
index df2b46dfaa12..4a2c17fa9044 100644
--- a/doc/automake.texi
+++ b/doc/automake.texi
@@ -7967,6 +7967,59 @@ manual has a section with more details on this topic
 Variables, autoconf, The Autoconf Manual}).  See also @ref{Hard-Coded
 Install Paths}.
 
+@menu
+* Supported Python versions::
+@end menu
+
+@node Supported Python versions
+@subsection Supported versions
+
+Automake guarantees releases will support all Python versions that are still
+supported by the Python project at the time of the Automake release.  Support
+for EOL versions of Python are not guaranteed, but will be considered as long
+as it is not onerous to do so, and there are large supported distros including
+them.  If you need to support older Python versions, please use a previous
+Automake release.
+
+Here are the current support plans.
+
+@multitable {2.0 -- 3.5} {Unsupported}
+@headitem Version @tab Status
+@item 0.x
+@tab Not supported
+@item 1.x
+@tab Not supported
+@item 2.0--2.6
+@tab Untested, but should work
+@item 2.7
+@c Debian Bullseye goes EOL in Apr 2026.
+@tab Supported (until Apr 2026)
+@item 3.0--3.3
+@tab Untested, but should work
+@item 3.4
+@c Ubuntu Trusty goes EOL in Apr 2022.
+@tab Supported (until Apr 2022)
+@item 3.5
+@c Ubuntu Xenial goes EOL in Apr 2024.
+@tab Supported (until Apr 2024)
+@item 3.6
+@c Ubuntu Bionic goes EOL in Apr 2028.
+@tab Supported (until Apr 2028)
+@item 3.7
+@c Debian Buster goes EOL in Apr 2024.
+@c But should support at least as long as previous Python version?
+@tab Supported (until Apr 2028)
+@item 3.8
+@c Ubuntu Focal goes EOL in Apr 2030
+@tab Supported (until Apr 2030)
+@item 3.9
+@c Debian Bullseye goes EOL in Apr 2026.
+@c But should support at least as long as previous Python version?
+@tab Supported (until Apr 2030)
+@item 3.10
+@c But should support at least as long as previous Python version?
+@tab Supported (until Apr 2030)
+@end multitable
 
 @node Documentation
 @chapter Building documentation
-- 
2.34.1

Message #32 received at 38043 <at> debbugs.gnu.org (full text, mbox):

From: Mike Frysinger <vapier <at> gentoo.org>
To: 38043 <at> debbugs.gnu.org
Subject: [PATCH 4/6] py-compile: drop support for Python 0.x & 1.x
Date: Sun,  6 Feb 2022 01:57:12 -0500

Python 2.0 was released in 2000.  There's really no way for us to check
those old versions, so let's just drop them.  No one will miss them.

* lib/py-compile: Abort if major version 0 or 1 is found.
* t/py-compile-env.sh: Rework slightly to handle new version probing.
---
 lib/py-compile      | 35 +++++++++++++----------------------
 t/py-compile-env.sh |  4 +---
 2 files changed, 14 insertions(+), 25 deletions(-)

diff --git a/lib/py-compile b/lib/py-compile
index 0e1df24024e8..2745d0b6b045 100755
--- a/lib/py-compile
+++ b/lib/py-compile
@@ -120,27 +120,19 @@ else
   filetrans="filepath = os.path.normpath('$destdir' + os.sep + path)"
 fi
 
-python_major=`$PYTHON -V 2>&1 | sed -e 's/.* //;s/\..*$//;1q'`
+python_major=`$PYTHON -c 'import sys; print(sys.version_info[0])'`
 if test -z "$python_major"; then
-  echo "$me: could not determine $PYTHON major version, guessing 3" >&2
-  python_major=3
+  usage_error "could not determine $PYTHON major version"
 fi
 
-# The old way to import libraries was deprecated.
-if test "$python_major" -le 2; then
-  import_lib=imp
-  import_test="hasattr(imp, 'get_tag')"
-  import_call=imp.cache_from_source
-  import_arg2=', False' # needed in one call and not the other
-else
-  import_lib=importlib
-  import_test="hasattr(sys.implementation, 'cache_tag')"
-  import_call=importlib.util.cache_from_source
-  import_arg2=
-fi
+case $python_major in
+[01])
+  usage_error "python version 0.x and 1.x not supported"
+  ;;
+esac
 
 $PYTHON -c "
-import sys, os, py_compile, $import_lib
+import sys, os, py_compile, importlib
 
 sys.stdout.write('Byte-compiling python modules...\n')
 for file in sys.argv[1:]:
@@ -151,15 +143,14 @@ for file in sys.argv[1:]:
 	    continue
     sys.stdout.write(file)
     sys.stdout.flush()
-    if $import_test:
-        py_compile.compile(filepath, $import_call(filepath), path)
+    if hasattr(sys.implementation, 'cache_tag'):
+        py_compile.compile(filepath, importlib.util.cache_from_source(filepath), path)
     else:
         py_compile.compile(filepath, filepath + 'c', path)
 sys.stdout.write('\n')" "$@" || exit $?
 
-# this will fail for python < 1.5, but that doesn't matter ...
 $PYTHON -O -c "
-import sys, os, py_compile, $import_lib
+import sys, os, py_compile, importlib
 
 # pypy does not use .pyo optimization
 if hasattr(sys, 'pypy_translation_info'):
@@ -174,8 +165,8 @@ for file in sys.argv[1:]:
 	    continue
     sys.stdout.write(file)
     sys.stdout.flush()
-    if $import_test:
-        py_compile.compile(filepath, $import_call(filepath$import_arg2), path)
+    if hasattr(sys.implementation, 'cache_tag'):
+        py_compile.compile(filepath, importlib.util.cache_from_source(filepath), path)
     else:
         py_compile.compile(filepath, filepath + 'o', path)
 sys.stdout.write('\n')" "$@" 2>/dev/null || exit $?
diff --git a/t/py-compile-env.sh b/t/py-compile-env.sh
index 2c7fe508488a..e7998589278e 100644
--- a/t/py-compile-env.sh
+++ b/t/py-compile-env.sh
@@ -23,6 +23,7 @@ cp "$am_scriptdir/py-compile" . \
 
 cat > my-py <<'END'
 #!/bin/sh
+echo 2
 : > my-py.run
 END
 chmod a+x my-py
@@ -30,9 +31,6 @@ chmod a+x my-py
 mkdir sub1
 cd sub1
 
-PYTHON=: ../py-compile foo.py
-ls | grep . && exit 1
-
 PYTHON=false ../py-compile foo.py && exit 1
 ls | grep . && exit 1
 
-- 
2.34.1

Message #35 received at 38043 <at> debbugs.gnu.org (full text, mbox):

From: Mike Frysinger <vapier <at> gentoo.org>
To: 38043 <at> debbugs.gnu.org
Subject: [PATCH 5/6] py-compile: fix display when compiling multiple files
Date: Sun,  6 Feb 2022 01:57:13 -0500

The compilation steps print the filename as it runs, but forgets to add
a space after it, so they all get squashed together:
$ ./py-compile 1.py 2.py 3.py
Byte-compiling python modules...
1.py2.py.3.py

* lib/py-compile: Add missing write.
---
 lib/py-compile | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/lib/py-compile b/lib/py-compile
index 2745d0b6b045..f72d4945da96 100755
--- a/lib/py-compile
+++ b/lib/py-compile
@@ -141,7 +141,7 @@ for file in sys.argv[1:]:
     if not os.path.exists(filepath) or not (len(filepath) >= 3
                                             and filepath[-3:] == '.py'):
 	    continue
-    sys.stdout.write(file)
+    sys.stdout.write(file + ' ')
     sys.stdout.flush()
     if hasattr(sys.implementation, 'cache_tag'):
         py_compile.compile(filepath, importlib.util.cache_from_source(filepath), path)
@@ -163,7 +163,7 @@ for file in sys.argv[1:]:
     if not os.path.exists(filepath) or not (len(filepath) >= 3
                                             and filepath[-3:] == '.py'):
 	    continue
-    sys.stdout.write(file)
+    sys.stdout.write(file + ' ')
     sys.stdout.flush()
     if hasattr(sys.implementation, 'cache_tag'):
         py_compile.compile(filepath, importlib.util.cache_from_source(filepath), path)
-- 
2.34.1

Message #38 received at 38043 <at> debbugs.gnu.org (full text, mbox):

From: Mike Frysinger <vapier <at> gentoo.org>
To: 38043 <at> debbugs.gnu.org
Subject: [PATCH 6/6] py-compile: fix optimized compiling for Python 3.5+
Date: Sun,  6 Feb 2022 01:57:14 -0500

Fixes automake bug https://bugs.gnu.org/38043.

Split the optimized compilation logic into a new section.  This avoids
trying to support multiple versions of major versions in a single script
as it gets harder to verify new changes don't break old versions as time
goes on.

Now for Python 3.5+, compile with -O2.

* THANKS: Add Michal Górny.
* lib/py-compile: Add new section for compiling Python 3.5+.
---
 THANKS         |  3 ++-
 lib/py-compile | 57 +++++++++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 58 insertions(+), 2 deletions(-)

diff --git a/THANKS b/THANKS
index 5a11cf4d25d9..554013ca87c3 100644
--- a/THANKS
+++ b/THANKS
@@ -244,7 +244,7 @@ Leonardo Boiko                  leoboiko <at> conectiva.com.br
 Libor Bukata                    libor.bukata <at> oracle.com
 Loulou Pouchet                  loulou <at> lrde.epita.fr
 Ludovic Courtès                 ludo <at> gnu.org
-Lukas Fleischer                        lfleischer <at> lfos.de
+Lukas Fleischer                 lfleischer <at> lfos.de
 Luo Yi                          luoyi.ly <at> gmail.com
 Maciej Stachowiak               mstachow <at> mit.edu
 Maciej W. Rozycki               macro <at> ds2.pg.gda.pl
@@ -285,6 +285,7 @@ Michael Daniels                 mdaniels <at> rim.com
 Michael Hofmann                 mhofma <at> googlemail.com
 Michael Ploujnikov              ploujj <at> gmail.com
 Michael Zucchi                  notzed <at> gmail.com
+Michał Górny                    mgorny <at> gentoo.org
 Michel de Ruiter                mdruiter <at> cs.vu.nl
 Mike Castle                     dalgoda <at> ix.netcom.com
 Mike Frysinger                  vapier <at> gentoo.org
diff --git a/lib/py-compile b/lib/py-compile
index f72d4945da96..d34bb2dd2364 100755
--- a/lib/py-compile
+++ b/lib/py-compile
@@ -131,6 +131,13 @@ case $python_major in
   ;;
 esac
 
+python_minor=`$PYTHON -c 'import sys; print(sys.version_info[1])'`
+
+# NB: When adding support for newer versions, prefer copying & adding new cases
+# rather than try to keep things merged with shell variables.
+
+# First byte compile (no optimization) all the modules.
+# This works for all currently known Python versions.
 $PYTHON -c "
 import sys, os, py_compile, importlib
 
@@ -149,7 +156,10 @@ for file in sys.argv[1:]:
         py_compile.compile(filepath, filepath + 'c', path)
 sys.stdout.write('\n')" "$@" || exit $?
 
-$PYTHON -O -c "
+# Then byte compile w/optimization all the modules.
+case $python_major.$python_minor in
+2.*)
+  $PYTHON -O -c "
 import sys, os, py_compile, importlib
 
 # pypy does not use .pyo optimization
@@ -170,6 +180,51 @@ for file in sys.argv[1:]:
     else:
         py_compile.compile(filepath, filepath + 'o', path)
 sys.stdout.write('\n')" "$@" 2>/dev/null || exit $?
+  ;;
+*)  # Python 3+
+  $PYTHON -O -c "
+import sys, os, py_compile, importlib
+
+print('Byte-compiling python modules (optimized versions) ...')
+for file in sys.argv[1:]:
+    $pathtrans
+    $filetrans
+    if not os.path.exists(filepath) or not (len(filepath) >= 3
+                                            and filepath[-3:] == '.py'):
+	    continue
+    print(file, end=' ', flush=True)
+    if hasattr(sys.implementation, 'cache_tag'):
+        py_compile.compile(filepath, importlib.util.cache_from_source(filepath), path)
+    else:
+        py_compile.compile(filepath, filepath + 'o', path)
+print()" "$@" 2>/dev/null || exit $?
+  ;;
+esac
+
+# Then byte compile w/more optimization.
+case $python_major.$python_minor in
+2.*|3.[0-4])
+  ;;
+*)  # Python 3.5+
+  # See https://bugs.gnu.org/38043 for background.
+  $PYTHON -OO -c "
+import sys, os, py_compile, imp
+
+print('Byte-compiling python modules (-OO version) ...')
+for file in sys.argv[1:]:
+    $pathtrans
+    $filetrans
+    if not os.path.exists(filepath) or not (len(filepath) >= 3
+                                            and filepath[-3:] == '.py'):
+        continue
+    print(file, end=' ', flush=True)
+    if hasattr(imp, 'get_tag'):
+        py_compile.compile(filepath, imp.cache_from_source(filepath), path)
+    else:
+        py_compile.compile(filepath, filepath + 'o', path)
+print()" "$@" 2>/dev/null || exit $?
+  ;;
+esac
 
 # Local Variables:
 # mode: shell-script
-- 
2.34.1

Message #43 received at 38043 <at> debbugs.gnu.org (full text, mbox):

From: Karl Berry <karl <at> freefriends.org>
To: vapier <at> gentoo.org
Cc: 38043 <at> debbugs.gnu.org
Subject: Re: bug#38043: [PATCH 6/6] py-compile: fix optimized compiling for
 Python 3.5+
Date: Sun, 6 Feb 2022 15:47:43 -0700

    Fixes automake bug https://bugs.gnu.org/38043.

All these changes look good to me (just casting my eyes over them ...),
and hopefully will result in something more future-maintainable. Thanks!

Message #46 received at 38043 <at> debbugs.gnu.org (full text, mbox):

From: Mike Frysinger <vapier <at> gentoo.org>
To: Karl Berry <karl <at> freefriends.org>
Cc: 38043 <at> debbugs.gnu.org
Subject: Re: bug#38043: [PATCH 6/6] py-compile: fix optimized compiling for
 Python 3.5+
Date: Sun, 6 Feb 2022 20:14:14 -0500

[Message part 1 (text/plain, inline)]

On 06 Feb 2022 15:47, Karl Berry wrote:
>     Fixes automake bug https://bugs.gnu.org/38043.
> 
> All these changes look good to me (just casting my eyes over them ...),
> and hopefully will result in something more future-maintainable. Thanks!

before i push, question about $scriptversion.  is the policy for handling
these documented somewhere ?  is it like m4 serial numbers ?  it should be
updated whenever there's a code change ?
-mike

[signature.asc (application/pgp-signature, inline)]

Message #49 received at 38043 <at> debbugs.gnu.org (full text, mbox):

From: Karl Berry <karl <at> freefriends.org>
To: vapier <at> gentoo.org
Cc: 38043 <at> debbugs.gnu.org
Subject: Re: bug#38043: [PATCH 6/6] py-compile: fix optimized compiling for
 Python 3.5+
Date: Mon, 7 Feb 2022 15:40:04 -0700

    before i push, question about $scriptversion.  is the policy for
    handling these documented somewhere ?  is it like m4 serial numbers
    ?  it should be updated whenever there's a code change ?

Yes. Standard practice is to update scriptversion whenever a new version
of the file is committed, I mean, "pushed".

This isn't documented anywhere in automake that I know of (feel free to
add it to HACKING if you like); maybe it's in the gnulib manual, but I
didn't check. --thanks, karl.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.