GNU bug report logs - #35322
AM_PATH_PYTHON incompatible with Python frameworks on macOS

Previous Next

Package: automake;

Reported by: Chih-Hsuan Yen <yan12125 <at> gmail.com>

Date: Fri, 19 Apr 2019 15:40:02 UTC

Severity: normal

Tags: confirmed, help

Done: Karl Berry <karl <at> freefriends.org>

Bug is archived. No further changes may be made.

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 35322 in the body.
You can then email your comments to 35322 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to bug-automake <at> gnu.org:
bug#35322; Package automake. (Fri, 19 Apr 2019 15:40:02 GMT) Full text and rfc822 format available.

Acknowledgement sent to Chih-Hsuan Yen <yan12125 <at> gmail.com>:
New bug report received and forwarded. Copy sent to bug-automake <at> gnu.org. (Fri, 19 Apr 2019 15:40:02 GMT) Full text and rfc822 format available.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Chih-Hsuan Yen <yan12125 <at> gmail.com>
To: bug-automake <at> gnu.org
Cc: Joshua Root <jmr <at> macports.org>
Subject: AM_PATH_PYTHON incompatible with Python frameworks on macOS
Date: Fri, 19 Apr 2019 23:38:55 +0800
Hi Automake experts,

On macOS, if CPython is configured with --enable-framework (default
for MacPorts [1]), Python packages should be installed in
${prefix}/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages.
However, AM_PATH_PYTHON gives
pythondir=${prefix}/lib/python3.7/site-packages.

Here's a minimal example configure.ac:

AC_PREREQ(2.57)
AC_INIT([automake-python-test], 0.1)
AM_INIT_AUTOMAKE([foreign])
AM_PATH_PYTHON([2.7])
AC_CONFIG_FILES([Makefile])

The issue appears to be that AM_PATH_PYTHON overwrites the 'base'
variable. If we remove this overwrite like [2], pythondir contains the
correct path.

Related bug report in MacPorts: https://trac.macports.org/ticket/57329

Regards,

Chih-Hsuan Yen

[1] https://github.com/macports/macports-ports/blob/834d761210be0d989076000c454a4a1e32209af5/lang/python37/Portfile#L57
[2] https://github.com/macports/macports-ports/blob/master/devel/automake/files/patch-issue57329.diff




Added tag(s) help and confirmed. Request was from Karl Berry <karl <at> freefriends.org> to control <at> debbugs.gnu.org. (Sat, 13 Feb 2021 02:47:02 GMT) Full text and rfc822 format available.

Information forwarded to bug-automake <at> gnu.org:
bug#35322; Package automake. (Sun, 14 Feb 2021 22:17:01 GMT) Full text and rfc822 format available.

Message #10 received at 35322 <at> debbugs.gnu.org (full text, mbox):

From: Karl Berry <karl <at> freefriends.org>
To: yan12125 <at> gmail.com
Cc: 35322 <at> debbugs.gnu.org, jmr <at> macports.org
Subject: Re: bug#35322: AM_PATH_PYTHON incompatible with Python frameworks on
 macOS
Date: Sun, 14 Feb 2021 15:16:18 -0700
Hi - thanks for this report from almost two years ago (sorry).
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=35322

    On macOS, if CPython is configured with --enable-framework (default
    for MacPorts [1]), Python packages should be installed in ...

Ok.

    Here's a minimal example configure.ac:

Thanks for sending.

    The issue appears to be that AM_PATH_PYTHON overwrites the 'base'
    variable. If we remove this overwrite like [2], pythondir contains
    the correct path.

Unfortunately, simply removing the change to the `base' assignment as
you did in your change, for instance the first hunk,

-    sitedir = sysconfig.get_path('purelib', vars={'base':'$am_py_prefix'})
+    sitedir = sysconfig.get_path('purelib')

will affect any use of AM_PATH_PYTHON in an (it seems to me)
incompatible way. That does not seem viable. It's not even OS-dependent,
since other Mac users might not use --enable-framework.

Evidently the current default for am_py_prefix is analogous to how
Automake treats "lispdir" and others, to make for a coherent system. So
it's not desirable to change these variables' default values in a
system-dependent way.

I don't know enough about Python to know if there is some clever way to
handle this. In my naive way, it seems like you just have to override
the directory for your case. I think setting the shell variable
$am_cv_python_pythondir before running configure would do this, although
I confess I haven't tried it.

Sorry not to have a helpful reply. If you (or anyone) can think of a
patch that will fix your case while not breaking others, I'd like to see.

Anyway, I'll leave this open, marked as needing help to resolve. --best, karl.

[2] https://github.com/macports/macports-ports/blob/master/devel/automake/files/patch-issue57329.diff




Information forwarded to bug-automake <at> gnu.org:
bug#35322; Package automake. (Mon, 15 Feb 2021 03:31:01 GMT) Full text and rfc822 format available.

Message #13 received at 35322 <at> debbugs.gnu.org (full text, mbox):

From: Joshua Root <jmr <at> macports.org>
To: Karl Berry <karl <at> freefriends.org>, yan12125 <at> gmail.com
Cc: 35322 <at> debbugs.gnu.org
Subject: Re: bug#35322: AM_PATH_PYTHON incompatible with Python frameworks on
 macOS
Date: Mon, 15 Feb 2021 14:10:21 +1100
On 2021-2-15 09:16 , Karl Berry wrote:
> Unfortunately, simply removing the change to the `base' assignment as
> you did in your change, for instance the first hunk,
> 
> -    sitedir = sysconfig.get_path('purelib', vars={'base':'$am_py_prefix'})
> +    sitedir = sysconfig.get_path('purelib')
> 
> will affect any use of AM_PATH_PYTHON in an (it seems to me)
> incompatible way. That does not seem viable. It's not even OS-dependent,
> since other Mac users might not use --enable-framework.
> 
> Evidently the current default for am_py_prefix is analogous to how
> Automake treats "lispdir" and others, to make for a coherent system. So
> it's not desirable to change these variables' default values in a
> system-dependent way.

A python installation configured with --disable-framework will have a 
different value for base. I guess it's a question of whether the goal is 
installing modules in a consistent location across different systems, or 
installing them somewhere where the python on this system will be able 
to find them.

The documentation says that pythondir is "The directory name for the 
site-packages subdirectory of the standard Python install tree." I 
suppose that could be interpreted different ways depending on your 
chosen value of "standard". Bear in mind that macOS may be unique in 
that it's normal for python's sys.prefix to be different (a few 
subdirectories deeper) than the --prefix passed to a configure script. I 
do wonder if the current approach was designed with the assumption that 
they would always be the same.

> I don't know enough about Python to know if there is some clever way to
> handle this. In my naive way, it seems like you just have to override
> the directory for your case. I think setting the shell variable
> $am_cv_python_pythondir before running configure would do this, although
> I confess I haven't tried it.

Just to be clear, --enable-framework is not just the default for 
MacPorts, it's the default on macOS in general in all recent python 
versions. It's how the Mac binary installers from python.org are 
configured. Python won't be able to find modules installed in the 
location currently chosen by AM_PATH_PYTHON unless its configuration is 
changed to look there.

I admit I seem not to totally understand the design here based on what 
you've said above, but it seems like it would be better to have defaults 
that result in a working installation.

- Josh




Information forwarded to bug-automake <at> gnu.org:
bug#35322; Package automake. (Sat, 20 Feb 2021 02:27:01 GMT) Full text and rfc822 format available.

Message #16 received at 35322 <at> debbugs.gnu.org (full text, mbox):

From: Karl Berry <karl <at> freefriends.org>
To: jmr <at> macports.org
Cc: yan12125 <at> gmail.com, 35322 <at> debbugs.gnu.org
Subject: Re: bug#35322: AM_PATH_PYTHON incompatible with Python frameworks on
 macOS
Date: Fri, 19 Feb 2021 19:26:33 -0700
Thanks for the follow-up.

    a question of whether the goal is installing modules in a consistent
    location across different systems, or installing them somewhere
    where the python on this system will be able to find them.

Yes. Ordinarily, install directories in Automake are supposed to be the
same across systems. But from what you quoted from the manual:

    The documentation says that pythondir is "The directory name for the 
    site-packages subdirectory of the standard Python install tree."

.. I tend to think pythondir was intended to be, or at least could
reasonably be, an exception.

Looking at the patch again (content pasted below for the record):
https://github.com/macports/macports-ports/blob/master/devel/automake/files/patch-issue57329.diff

It seems too dangerous to me to apply as is, i.e., simply removing all
the $am_py_prefix values. That would change the results of
AM_PYTHON_PATH on every system. Maybe in a good way, but maybe not, and
I, at least, am not prepared to induce that level of incompatible
change.  People could have their build systems set up to define the
prefix to be their desired directory.

What I think would be safer to apply is a patch that eliminates those
am_py_prefix values only on Macs. I'm not sure if am_py_prefix can be
defined (only on Macs) to something in the code above that sets it:

    [if test "x$prefix" = xNONE
     then
       am_py_prefix=$ac_default_prefix
     else
       am_py_prefix=$prefix
     fi

i.e.: if <on mac> then am_py_prefix=$something ...
or if it's necessary to make all those sysconfig.* calls conditional on
the system.

Wdyt? Can you provide a modified patch in whatever way seems best?
--thanks, karl.


diff --git m4/python.m4 m4/python.m4
index 63c0a0e04..f81684378 100644
--- m4/python.m4
+++ m4/python.m4
@@ -147,10 +147,10 @@ except ImportError:
      am_cv_python_pythondir=`$PYTHON -c "
 $am_python_setup_sysconfig
 if can_use_sysconfig:
-    sitedir = sysconfig.get_path('purelib', vars={'base':'$am_py_prefix'})
+    sitedir = sysconfig.get_path('purelib')
 else:
     from distutils import sysconfig
-    sitedir = sysconfig.get_python_lib(0, 0, prefix='$am_py_prefix')
+    sitedir = sysconfig.get_python_lib(0, 0)
 sys.stdout.write(sitedir)"`
      case $am_cv_python_pythondir in
      $am_py_prefix*)
@@ -189,10 +189,10 @@ sys.stdout.write(sitedir)"`
      am_cv_python_pyexecdir=`$PYTHON -c "
 $am_python_setup_sysconfig
 if can_use_sysconfig:
-    sitedir = sysconfig.get_path('platlib', vars={'platbase':'$am_py_prefix'})
+    sitedir = sysconfig.get_path('platlib')
 else:
     from distutils import sysconfig
-    sitedir = sysconfig.get_python_lib(1, 0, prefix='$am_py_prefix')
+    sitedir = sysconfig.get_python_lib(1, 0)
 sys.stdout.write(sitedir)"`
      case $am_cv_python_pyexecdir in
      $am_py_exec_prefix*)




Information forwarded to bug-automake <at> gnu.org:
bug#35322; Package automake. (Fri, 26 Feb 2021 19:20:02 GMT) Full text and rfc822 format available.

Message #19 received at 35322 <at> debbugs.gnu.org (full text, mbox):

From: Joshua Root <jmr <at> macports.org>
To: Karl Berry <karl <at> freefriends.org>
Cc: yan12125 <at> gmail.com, 35322 <at> debbugs.gnu.org
Subject: Re: bug#35322: AM_PATH_PYTHON incompatible with Python frameworks on
 macOS
Date: Sat, 27 Feb 2021 06:18:41 +1100
[Message part 1 (text/plain, inline)]
How about this? It bases the default values of PYTHON_PREFIX and 
PYTHON_EXEC_PREFIX on sys.prefix and sys.exec_prefix respectively, which 
the documentation has long said might happen. Behaviour should not 
change in the case that sys.prefix = $prefix. The variables can still be 
overridden; e.g. PYTHON_PREFIX is defined with reference to $prefix if 
the former is a subdir of (or the same as) the latter, and pythondir is 
defined with reference to $PYTHON_PREFIX.

There's also a fix for an apparent bug where $am_py_prefix was used 
instead of $am_py_exec_prefix when setting up the value of pyexecdir.

- Josh
[use-sys.prefix-for-PYTHON_PREFIX.patch (text/plain, attachment)]

Information forwarded to bug-automake <at> gnu.org:
bug#35322; Package automake. (Mon, 01 Mar 2021 23:15:01 GMT) Full text and rfc822 format available.

Message #22 received at 35322 <at> debbugs.gnu.org (full text, mbox):

From: Karl Berry <karl <at> freefriends.org>
To: jmr <at> macports.org
Cc: yan12125 <at> gmail.com, 35322 <at> debbugs.gnu.org
Subject: Re: bug#35322: AM_PATH_PYTHON incompatible with Python frameworks on
 macOS
Date: Mon, 1 Mar 2021 16:14:45 -0700
Hi Joshua,

    How about this? It bases the default values of PYTHON_PREFIX and 
    PYTHON_EXEC_PREFIX on sys.prefix and sys.exec_prefix respectively, ...

Thanks much for the patch (including fixing the tests). I don't see any
immediate comments. Jim and I will ponder for a couple days. --best, karl.




Information forwarded to bug-automake <at> gnu.org:
bug#35322; Package automake. (Tue, 02 Mar 2021 06:26:02 GMT) Full text and rfc822 format available.

Message #25 received at 35322 <at> debbugs.gnu.org (full text, mbox):

From: Joshua Root <jmr <at> macports.org>
To: Karl Berry <karl <at> freefriends.org>
Cc: yan12125 <at> gmail.com, 35322 <at> debbugs.gnu.org
Subject: Re: bug#35322: AM_PATH_PYTHON incompatible with Python frameworks on
 macOS
Date: Tue, 2 Mar 2021 17:25:28 +1100
On 2021-3-2 10:14 , Karl Berry wrote:
> Thanks much for the patch (including fixing the tests). I don't see any
> immediate comments. Jim and I will ponder for a couple days. --best, karl.

Thanks Karl. I'd certainly appreciate input from other packagers as well.

I wonder if it might also make sense to offer a configure option to set 
PYTHON_PREFIX?

- Josh




Information forwarded to bug-automake <at> gnu.org:
bug#35322; Package automake. (Tue, 02 Mar 2021 23:55:02 GMT) Full text and rfc822 format available.

Message #28 received at 35322 <at> debbugs.gnu.org (full text, mbox):

From: Karl Berry <karl <at> freefriends.org>
To: jmr <at> macports.org
Cc: yan12125 <at> gmail.com, 35322 <at> debbugs.gnu.org
Subject: Re: bug#35322: AM_PATH_PYTHON incompatible with Python frameworks on
 macOS
Date: Tue, 2 Mar 2021 16:54:03 -0700
    I wonder if it might also make sense to offer a configure option to set 
    PYTHON_PREFIX?

I think that would be great (--pythondir, --pythonexecdir or some
such). Would you mind looking into that? I've never played around with
the options Automake adds and have unfortunately limited time to
research the code. But I will eventually if needed.

    I'd certainly appreciate input from other packagers as well.

Indeed yes. --thanks, karl.




Information forwarded to bug-automake <at> gnu.org:
bug#35322; Package automake. (Fri, 05 Mar 2021 23:27:02 GMT) Full text and rfc822 format available.

Message #31 received at 35322 <at> debbugs.gnu.org (full text, mbox):

From: Karl Berry <karl <at> freefriends.org>
To: jmr <at> macports.org
Cc: yan12125 <at> gmail.com, 35322 <at> debbugs.gnu.org
Subject: Re: bug#35322: AM_PATH_PYTHON incompatible with Python frameworks on
 macOS
Date: Fri, 5 Mar 2021 16:26:40 -0700
Hi Joshua - did you run the full make check on your Mac?
I saw you patched some of the tests, which is great, so I'm guessing
you did, but just asking ...

On Jim's Mac system, many more tests (35) fail, but haven't tracked that
down yet. --thanks, karl.




Information forwarded to bug-automake <at> gnu.org:
bug#35322; Package automake. (Fri, 05 Mar 2021 23:38:01 GMT) Full text and rfc822 format available.

Message #34 received at 35322 <at> debbugs.gnu.org (full text, mbox):

From: Joshua Root <jmr <at> macports.org>
To: Karl Berry <karl <at> freefriends.org>
Cc: yan12125 <at> gmail.com, 35322 <at> debbugs.gnu.org
Subject: Re: bug#35322: AM_PATH_PYTHON incompatible with Python frameworks on
 macOS
Date: Sat, 6 Mar 2021 10:37:08 +1100
On 2021-3-6 10:26 , Karl Berry wrote:
> Hi Joshua - did you run the full make check on your Mac?
> I saw you patched some of the tests, which is great, so I'm guessing
> you did, but just asking ...
> 
> On Jim's Mac system, many more tests (35) fail, but haven't tracked that
> down yet. --thanks, karl.
> 

I did run a full make check. There were indeed quite a few tests that 
failed; I only fixed the python related ones.

- Josh




Information forwarded to bug-automake <at> gnu.org:
bug#35322; Package automake. (Sat, 13 Mar 2021 11:22:01 GMT) Full text and rfc822 format available.

Message #37 received at 35322 <at> debbugs.gnu.org (full text, mbox):

From: Joshua Root <jmr <at> macports.org>
To: Karl Berry <karl <at> freefriends.org>
Cc: yan12125 <at> gmail.com, 35322 <at> debbugs.gnu.org
Subject: Re: bug#35322: AM_PATH_PYTHON incompatible with Python frameworks on
 macOS
Date: Sat, 13 Mar 2021 22:21:01 +1100
[Message part 1 (text/plain, inline)]
On 2021-3-3 10:54 , Karl Berry wrote:
>      I wonder if it might also make sense to offer a configure option to set
>      PYTHON_PREFIX?
> 
> I think that would be great (--pythondir, --pythonexecdir or some
> such). Would you mind looking into that? I've never played around with
> the options Automake adds and have unfortunately limited time to
> research the code. But I will eventually if needed.

OK, here's a patch to add --with-python_prefix and 
--with-python_exec_prefix options. It applies on top of my previous patch.

Note that the paths given to these options aren't edited to use the 
${prefix} and ${exec_prefix} variables when they are subdirs thereof, 
since I figure if you set a specific value with these options you really 
mean it. Note also that if --with-python_prefix is given but 
--with-python_exec_prefix is not, PYTHON_EXEC_PREFIX takes the value 
that was given to --with-python_prefix. I think this is more convenient 
on the whole, since setting a custom PYTHON_PREFIX but wanting to keep 
the automatically determined value of PYTHON_EXEC_PREFIX seems like it 
would be a pretty rare scenario.

- Josh
[add-with-python_prefix-option.patch (text/plain, attachment)]

Information forwarded to bug-automake <at> gnu.org:
bug#35322; Package automake. (Sat, 13 Mar 2021 23:43:02 GMT) Full text and rfc822 format available.

Message #40 received at 35322 <at> debbugs.gnu.org (full text, mbox):

From: Karl Berry <karl <at> freefriends.org>
To: jmr <at> macports.org
Cc: yan12125 <at> gmail.com, 35322 <at> debbugs.gnu.org
Subject: Re: bug#35322: AM_PATH_PYTHON incompatible with Python frameworks on
 macOS
Date: Sat, 13 Mar 2021 16:42:52 -0700
    OK, here's a patch to add --with-python_prefix and 
    --with-python_exec_prefix options. It applies on top of my previous patch.

Thanks!  I think the option names would be normal with all hyphens
(e.g., the std option is --exec-prefix, not --exec_prefix), but
I'll just tweak that when I install, obviously trivial.

    setting a custom PYTHON_PREFIX but wanting to keep the automatically
    determined value of PYTHON_EXEC_PREFIX seems like it would be a
    pretty rare scenario.

Fully agreed. Thanks again. -k




Reply sent to Karl Berry <karl <at> freefriends.org>:
You have taken responsibility. (Tue, 18 May 2021 22:16:02 GMT) Full text and rfc822 format available.

Notification sent to Chih-Hsuan Yen <yan12125 <at> gmail.com>:
bug acknowledged by developer. (Tue, 18 May 2021 22:16:02 GMT) Full text and rfc822 format available.

Message #45 received at 35322-done <at> debbugs.gnu.org (full text, mbox):

From: Karl Berry <karl <at> freefriends.org>
To: jmr <at> macports.org
Cc: yan12125 <at> gmail.com, 35322 <at> debbugs.gnu.org
Subject: Re: bug#35322: AM_PATH_PYTHON incompatible with Python frameworks on
 macOS
Date: Tue, 18 May 2021 16:15:56 -0600
Hi Joshua and all - regarding https://bugs.gnu.org/35322, I have
(finally) pushed your patches. I tweaked the doc update a little bit and
created a new test to try out the --with-python_prefix and
--with-python_exec_prefix options. (I left the underscores in the name
since I saw there were other such cases already. Confusing either way,
so fine.)

Thanks much for doing all the work. --karl




Information forwarded to bug-automake <at> gnu.org:
bug#35322; Package automake. (Tue, 18 May 2021 22:17:02 GMT) Full text and rfc822 format available.

Information forwarded to bug-automake <at> gnu.org:
bug#35322; Package automake. (Wed, 19 May 2021 03:09:01 GMT) Full text and rfc822 format available.

Message #51 received at 35322 <at> debbugs.gnu.org (full text, mbox):

From: Joshua Root <jmr <at> macports.org>
To: Karl Berry <karl <at> freefriends.org>
Cc: yan12125 <at> gmail.com, 35322 <at> debbugs.gnu.org
Subject: Re: bug#35322: AM_PATH_PYTHON incompatible with Python frameworks on
 macOS
Date: Wed, 19 May 2021 13:08:48 +1000
On 2021-5-19 08:15 , Karl Berry wrote:
> Hi Joshua and all - regarding https://bugs.gnu.org/35322, I have
> (finally) pushed your patches. I tweaked the doc update a little bit and
> created a new test to try out the --with-python_prefix and
> --with-python_exec_prefix options. (I left the underscores in the name
> since I saw there were other such cases already. Confusing either way,
> so fine.)
> 
> Thanks much for doing all the work. --karl

Thanks Karl, much appreciated.

- Josh




bug archived. Request was from Debbugs Internal Request <help-debbugs <at> gnu.org> to internal_control <at> debbugs.gnu.org. (Wed, 16 Jun 2021 11:24:05 GMT) Full text and rfc822 format available.

This bug report was last modified 2 years and 286 days ago.

Previous Next


GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.