GNU bug report logs -
#46848
[PATCHES] [core-updates] PEP 517 python-build-system
Previous Next
Reported by: Lars-Dominik Braun <lars <at> 6xq.net>
Date: Mon, 1 Mar 2021 13:45:01 UTC
Severity: normal
Tags: patch
Done: Lars-Dominik Braun <lars <at> 6xq.net>
Bug is archived. No further changes may be made.
To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 46848 in the body.
You can then email your comments to 46848 AT debbugs.gnu.org in the normal way.
Toggle the display of automated, internal messages from the tracker.
Report forwarded
to
guix-patches <at> gnu.org:
bug#46848; Package
guix-patches.
(Mon, 01 Mar 2021 13:45:01 GMT)
Full text and
rfc822 format available.
Acknowledgement sent
to
Lars-Dominik Braun <lars <at> 6xq.net>:
New bug report received and forwarded. Copy sent to
guix-patches <at> gnu.org.
(Mon, 01 Mar 2021 13:45:01 GMT)
Full text and
rfc822 format available.
Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
Hi everyone,
the attached patches switch python-build-system to a PEP 517-based build
system using python-pypa-build.
As discussed previously Python is currently in the process of opening up
for build systems other than setuptools using the API specified in PEP
517. python-pypa-build is a simple tool for building packages using that
new API. It supports setup.py-based builds as a fallback, if no
pyproject.toml is present.
One downside is that this tool is not self-contained and has a few
dependencies. Thus first I bootstrap setuptools using itself (possible
because it bundles all of its own dependencies), then build
python-pypa-build’s dependencies using setuptools (which is fortunately
still possible) and then combine everything into a
python-toolchain(-for-build), which is then used by the build-process.
I can successfully build packages like python-pypa-build and
python-pytest and python-pep517-bootstrap. The latter is using flit as
its build backend. But other packages currently fail because I removed
some arguments.
Cheers,
Lars
[0001-build-python-Handle-missing-setuptools-in-sanity-che.patch (text/x-diff, attachment)]
[0002-gnu-python-pypa-build-Update-to-0.3.0.patch (text/x-diff, attachment)]
[0003-gnu-python-wheel-Install-entrypoint-scripts.patch (text/x-diff, attachment)]
[0004-gnu-python-setuptools-Bootstrap-using-itself.patch (text/x-diff, attachment)]
[0005-python-build-Switch-to-PEP-517-based-build.patch (text/x-diff, attachment)]
[0006-gnu-Add-python-pytoml.patch (text/x-diff, attachment)]
[0007-gnu-Add-python-flit-core.patch (text/x-diff, attachment)]
[0008-gnu-python-pep517-bootstrap-Build-using-flit-core.patch (text/x-diff, attachment)]
[0009-gnu-python-iniconfig-Add-missing-build-input.patch (text/x-diff, attachment)]
[0010-gnu-Add-python-u-msgpack.patch (text/x-diff, attachment)]
[0011-gnu-Add-python-pytest-expect.patch (text/x-diff, attachment)]
[0012-gnu-python-html5lib-Fix-tests-with-pytest-6.patch (text/x-diff, attachment)]
Information forwarded
to
guix-patches <at> gnu.org:
bug#46848; Package
guix-patches.
(Sat, 15 May 2021 09:33:01 GMT)
Full text and
rfc822 format available.
Message #8 received at 46848 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
Hi,
I rebased my changes on top of the current core-updates HEAD.
Cheers,
Lars
[0001-build-python-Handle-missing-setuptools-in-sanity-che.patch (text/x-diff, attachment)]
[0002-gnu-python-pypa-build-Update-to-0.3.0.patch (text/x-diff, attachment)]
[0003-gnu-python-wheel-Install-entrypoint-scripts.patch (text/x-diff, attachment)]
[0004-gnu-python-setuptools-Bootstrap-using-itself.patch (text/x-diff, attachment)]
[0005-python-build-Switch-to-PEP-517-based-build.patch (text/x-diff, attachment)]
[0006-gnu-Add-python-pytoml.patch (text/x-diff, attachment)]
[0007-gnu-Add-python-flit-core.patch (text/x-diff, attachment)]
[0008-gnu-python-pep517-bootstrap-Build-using-flit-core.patch (text/x-diff, attachment)]
[0009-gnu-python-iniconfig-Add-missing-build-input.patch (text/x-diff, attachment)]
[0010-gnu-Add-python-u-msgpack.patch (text/x-diff, attachment)]
[0011-gnu-Add-python-pytest-expect.patch (text/x-diff, attachment)]
[0012-gnu-python-html5lib-Fix-tests-with-pytest-6.patch (text/x-diff, attachment)]
[0013-gnu-python-libxml2-Fix-build.patch (text/x-diff, attachment)]
[0014-gnu-scons-Remove-obsolete-argument.patch (text/x-diff, attachment)]
Information forwarded
to
guix-patches <at> gnu.org:
bug#46848; Package
guix-patches.
(Sun, 06 Jun 2021 19:45:02 GMT)
Full text and
rfc822 format available.
Message #11 received at 46848 <at> debbugs.gnu.org (full text, mbox):
Hi Tanguy,
(cross-posting this to the issue itself too)
> Sorry if I'm (very) late, but apprently this hasn't made it to master
> yet, so… What the status? Do you still need a willing-but-maybe-not-qualified
> person to review or discuss your patch?
the patch set works, I can build many Python packages, although some
require changes. Still, multiple things need to be done before merging
is possible imo:
1) Validate the general idea of using pypa-build is viable and
sustainable in the long run – ideally through review by someone else
than me. We can’t touch python-build-system every week to solve
structural issues, so it needs to be bullet-proof.
2) Figure out how to run testing code. Currently python-build-system
just picks pytest, if available – not sure this is the best option we
have. How do we deal with other test systems? How do we pass options?
3) Determine the fate of Python 2, which is probably broken through this
patch set. Shall we remove it entirely? Is it worth to keep support?
4) Iron out minor details like including pip in the python package or
create a new python-toolchain package? What do we include in that
meta-package? pip? virtualenv? …?
5) Fix my awkward Scheme code, especially regarding unpacking of the
built wheels. Should we be using Python’s unzip module or can be
assumed unzip is available in the build environment? (Should we add
it?)
I’m by no means a Python packaging expert, so any help would be
appreciated, even if it’s just a question or thumbs-up/thumbs-down on my
ideas.
Cheers,
Lars
Information forwarded
to
guix-patches <at> gnu.org:
bug#46848; Package
guix-patches.
(Tue, 22 Jun 2021 07:02:02 GMT)
Full text and
rfc822 format available.
Message #14 received at 46848 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
Hi Lars,
sorry for being late for commenting on this (the time I can spend on
guix is rather limited atm).
Here are some general remarks on this patch-set (in order of appearance):
*
Not installing pip by default might break some user's environments.
Anyhow, since using pip in guix is not such a good idea anyway, this
should be okay.
*
"use-setuptools" is gone. There are still about 10 packages with
"#:use-setuptools #f" - which means they are (expected to be)
incompatible with setuptools for some reason. You might want to
check whether these packages actually still can't be packages with
setuptools.
*
setuptools-shim has been removed. I don't think this is a good idea,
since this peace of code enforces packages to be actually build with
setuptools instead of old distutils. This code is still in current
pip, so I assume it is still required.
(This shim ensures setuptools is used, even if setup.py only imports
distutils. And setuptools is required for some options like
""--single-version-externally-managed" - as the comment for the shim
says.)
*
set-SOURCE-DATE-EPOCH: Please keep the verbose rational. It's much
more helpful than the new one-line comment.
*
set-SOURCE-DATE-EPOCH: This implementation makes the code depend on
wheel and wheel being used for installation.
*
Why has rename-pth-file been removed? Are you sure .pth-files are
never created anymore nowerdays?
*
python-hashbang: Isn't this done already by the normal
"patch-shebangs" phase after install in gnu-build-system? (BTW:
these are called *she*bangs).
*
I suggest to have phase compile-bytecode still honor older versions
of python
> 1) Validate the general idea of using pypa-build is viable and
> sustainable in the long run – ideally through review by someone else
> than me. We can’t touch python-build-system every week to solve
> structural issues, so it needs to be bullet-proof.
pypa bulld is where the PyPA is pushing towards. Anyhow, as of today, as
far as I can see, adoption is low.
> 2) Figure out how to run testing code. Currently python-build-system
> just picks pytest, if available – not sure this is the best option we
> have. How do we deal with other test systems? How do we pass options?
AFAIK fhere is no standard way for running tests in python. pytest seems
to be the most modern test-system. Anyhow packages still use nose or tox
(which again might run pytest or nose, with parameters fetched from
tox.ini). So I'm afraid, there is no general rule.
Did the PyPA publish some recommendations or PEP on this?
> 4) Iron out minor details like including pip in the python package or
> create a new python-toolchain package? What do we include in that
> meta-package? pip? virtualenv? …?
As I Python developer I nowerdays would expect pip and venv (which is
part of the std-lib - but not the virualenv, which is a separate module)
to be availalbe when installing "python". Anyhow I could live with pip
being a separate package.
"python-toolchain" sounds oversized for me. Would this include the
C-compiler, too (which one? maybe I want to build cross). I'd rather not
have such a package.
> 5) Fix my awkward Scheme code, especially regarding unpacking of the
> built wheels. Should we be using Python’s unzip module or can be
> assumed unzip is available in the build environment? (Should we add
> it?)
The gnu-build-system already provides the "unzip" binary (used in phase
"unpack"). So we could simply use this. Otherwise I recommend using the
Python zip module, as this is what is used for creating the zip-archives
:-)
--
Regards
Hartmut Goebel
| Hartmut Goebel | h.goebel <at> crazy-compilers.com |
| www.crazy-compilers.com | compilers which you thought are impossible |
[Message part 2 (text/html, inline)]
Information forwarded
to
guix-patches <at> gnu.org:
bug#46848; Package
guix-patches.
(Mon, 28 Jun 2021 12:00:02 GMT)
Full text and
rfc822 format available.
Message #17 received at 46848 <at> debbugs.gnu.org (full text, mbox):
Hi Hartmut,
> sorry for being late for commenting on this (the time I can spend on
> guix is rather limited atm).
no problem, same thing on my side.
> *
>
> Not installing pip by default might break some user's environments.
> Anyhow, since using pip in guix is not such a good idea anyway, this
> should be okay.
True. We could rename python→python-minimal-interpreteronly (or similar;
given that python-minimal already exists) and python-toolchain→python to
work around that.
> *
>
> "use-setuptools" is gone. There are still about 10 packages with
> "#:use-setuptools #f" - which means they are (expected to be)
> incompatible with setuptools for some reason. You might want to
> check whether these packages actually still can't be packages with
> setuptools.
Yeah, I’ve seen those, but the number was too small to bother for now.
I’ll have a look later.
> *
>
> setuptools-shim has been removed. I don't think this is a good idea,
> since this peace of code enforces packages to be actually build with
> setuptools instead of old distutils. This code is still in current
> pip, so I assume it is still required.
>
> (This shim ensures setuptools is used, even if setup.py only imports
> distutils. And setuptools is required for some options like
> ""--single-version-externally-managed" - as the comment for the shim
> says.)
Is this relevant though? I doubt many packages are still importing
distutils and the few that do can be patched.
> *
>
> set-SOURCE-DATE-EPOCH: Please keep the verbose rational. It's much
> more helpful than the new one-line comment.
You mean the one from the now-removed ensure-no-mtimes-pre-1980? Sure.
> *
>
> set-SOURCE-DATE-EPOCH: This implementation makes the code depend on
> wheel and wheel being used for installation.
Technically it depends on the wheel builder understanding
SOURCE_DATE_EPOCH (not necessarily python-wheel). I’d say that’s
acceptable and it’d be preferable to fix build systems not respecting
this variable imo.
> *
>
> Why has rename-pth-file been removed? Are you sure .pth-files are
> never created anymore nowerdays?
Given that easy-install has been deprecated I think it’s safe to remove
this phase and flag any packages creating this easy-install.pth as
broken. (There are, however, legitimate packages creating files like
ruamel.yaml-0.15.83-py3.8-nspkg.pth.)
> *
>
> python-hashbang: Isn't this done already by the normal
> "patch-shebangs" phase after install in gnu-build-system? (BTW:
> these are called *she*bangs).
Afaik the function patch-shebang expects a leading slash and thus it
does not replace this “special” shebang (see
https://www.python.org/dev/peps/pep-0427/#installing-a-wheel-distribution-1-0-py32-none-any-whl;
Spread, point 3).
> *
>
> I suggest to have phase compile-bytecode still honor older versions
> of python
I’m not sure what you mean. compileall is also part of Python 2.
> pypa bulld is where the PyPA is pushing towards. Anyhow, as of today, as
> far as I can see, adoption is low.
Of pypa build? That is true.
> AFAIK fhere is no standard way for running tests in python. pytest seems
> to be the most modern test-system. Anyhow packages still use nose or tox
> (which again might run pytest or nose, with parameters fetched from
> tox.ini). So I'm afraid, there is no general rule.
>
> Did the PyPA publish some recommendations or PEP on this?
I’m not aware of any accepted PEP’s. There is a discussion about the
removal of `python setup.py test`:
https://github.com/pypa/setuptools/issues/931
And a proposal for pyproject.toml going nowhere:
https://discuss.python.org/t/proposal-for-tests-entry-point-in-pyproject-toml/2077/2
> As I Python developer I nowerdays would expect pip and venv (which is
> part of the std-lib - but not the virualenv, which is a separate module)
> to be availalbe when installing "python". Anyhow I could live with pip
> being a separate package.
If we keep setuptools/pip bundled, we don’t have to do any of this
pypa-build dance. We could also modernize python-build-system around
`pip install` and just be done with it. (I don’t have a proof-of-concept
for that yet.)
> "python-toolchain" sounds oversized for me. Would this include the
> C-compiler, too (which one? maybe I want to build cross). I'd rather not
> have such a package.
See suggestion above wrt renaming.
> The gnu-build-system already provides the "unzip" binary (used in phase
> "unpack"). So we could simply use this. Otherwise I recommend using the
> Python zip module, as this is what is used for creating the zip-archives
> :-)
I’m using Python’s zipfile module already.
Cheers,
Lars
Information forwarded
to
guix-patches <at> gnu.org:
bug#46848; Package
guix-patches.
(Mon, 28 Jun 2021 20:38:02 GMT)
Full text and
rfc822 format available.
Message #20 received at 46848 <at> debbugs.gnu.org (full text, mbox):
Hi Lars-Dominik,
Am 28.06.21 um 13:59 schrieb Lars-Dominik Braun:*
>> Not installing pip by default might break some user's environments.
>> Anyhow, since using pip in guix is not such a good idea anyway, this
>> should be okay.
> True. We could rename python→python-minimal-interpreteronly (or similar;
> given that python-minimal already exists) and python-toolchain→python to
> work around that.
What should be the use of having a package without pip? Anything else
than saving a few KB?
>> [setuptools-shim has been removed]
> Is this relevant though? I doubt many packages are still importing
> distutils and the few that do can be patched.
Was I wrote: This code is still in pip, so I assume it is still relevant.
I don't think patching is a good idea. It requires effort (implementing,
reviewing), which can be saved by keeping exisiting and tested code.
>> set-SOURCE-DATE-EPOCH: This implementation makes the code depend on
>> wheel and wheel being used for installation.
> Technically it depends on the wheel builder understanding
> SOURCE_DATE_EPOCH (not necessarily python-wheel). I’d say that’s
> acceptable and it’d be preferable to fix build systems not respecting
> this variable imo.
For this case please change the comment not not referring to wheel in
this way. More something like "we expect the builder to support
SOURCE_DATE_EPOCH, like wheel does"
Anyhow, *m not actually convinced that we should throw away the old
code. I can imagine in the next cuple of years quite some new
build-systems to arrive, most of which will most probably not support
SOURCE_DATE_EPOCH in the beginning, and thus making package's life harder.
>
>> Why has rename-pth-file been removed? Are you sure .pth-files are
>> never created anymore nowerdays?
> Given that easy-install has been deprecated I think it’s safe to remove
> this phase and flag any packages creating this easy-install.pth as
> broken. (There are, however, legitimate packages creating files like
> ruamel.yaml-0.15.83-py3.8-nspkg.pth.)
What exaclty do you mean with "flag as broken"? Will anybody (you? ;-)
verify *all* current packages to not be "broken" prior to merging this
change?
Anyhow, again, I'm not convinced we should remove this phase now.
.pth-file are deprecated only, but still supported. By removing this
phase we might create conflict cased we can not foresee. And I would
keep it even if one analyzes none of the current packages is "broken" -
just do be on the save side fpr avoiding user trouble. (These issues
will show up at the user, and are hard to track down, since noone will
think about .pth files)
>
>> python-hashbang: Isn't this done already by the normal
>> "patch-shebangs" phase after install in gnu-build-system? (BTW:
>> these are called *she*bangs).
> Afaik the function patch-shebang expects a leading slash and thus it
> does not replace this “special” shebang (see
> https://www.python.org/dev/peps/pep-0427/#installing-a-wheel-distribution-1-0-py32-none-any-whl;
> Spread, point 3).
IC. Please add a comment to make this clear (e.g. "handle shebang of
scripts generated by wheel missing leading slash")
>> *
>>
>> I suggest to have phase compile-bytecode still honor older versions
>> of python
> I’m not sure what you mean. compileall is also part of Python 2.
The old code did not compile the source for Python <3.7. Please see the
comment of the old code for rational.
>> As I Python developer I nowerdays would expect pip and venv (which is
>> part of the std-lib - but not the virualenv, which is a separate module)
>> to be availalbe when installing "python". Anyhow I could live with pip
>> being a separate package.
> If we keep setuptools/pip bundled, we don’t have to do any of this
> pypa-build dance. We could also modernize python-build-system around
> `pip install` and just be done with it. (I don’t have a proof-of-concept
> for that yet.)
AFAIK this might not be true if other build systems not using setuptools
at all might show up. And isn't this the main reason for all your work?
>
>> The gnu-build-system already provides the "unzip" binary (used in phase
>> "unpack"). So we could simply use this. Otherwise I recommend using the
>> Python zip module, as this is what is used for creating the zip-archives
>> :-)
> I’m using Python’s zipfile module already.
Fine, so you can safely remove the respective comment ;-)
--
Regards
Hartmut Goebel
| Hartmut Goebel | h.goebel <at> crazy-compilers.com |
| www.crazy-compilers.com | compilers which you thought are impossible |
Information forwarded
to
guix-patches <at> gnu.org:
bug#46848; Package
guix-patches.
(Tue, 29 Jun 2021 07:21:02 GMT)
Full text and
rfc822 format available.
Message #23 received at 46848 <at> debbugs.gnu.org (full text, mbox):
Hi Hartmut,
> What should be the use of having a package without pip? Anything else
> than saving a few KB?
saving some space and unvendoring components that we also have separate
packages for. (As I understand it, ensurepip, which installs both pip and
setuptools, is merely a convenience tool if you install Python from
source yourself – not for distributions.)
> AFAIK this might not be true if other build systems not using setuptools
> at all might show up. And isn't this the main reason for all your work?
No, try
git clone https://github.com/pypa/pep517.git
cd pep517
pip wheel --use-pep517 -v .
which has no setup.py and uses flit instead. pip can build it, because
it supports PEP 517-based builds. As I said, if we decide to keep it
bundled with our python package, there’s no good reason to choose
pypa-build.
Cheers,
Lars
Information forwarded
to
guix-patches <at> gnu.org:
bug#46848; Package
guix-patches.
(Tue, 06 Jul 2021 12:17:02 GMT)
Full text and
rfc822 format available.
Message #26 received at 46848 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
Hi again,
> No, try
>
> git clone https://github.com/pypa/pep517.git
> cd pep517
> pip wheel --use-pep517 -v .
>
> which has no setup.py and uses flit instead. pip can build it, because
> it supports PEP 517-based builds. As I said, if we decide to keep it
> bundled with our python package, there’s no good reason to choose
> pypa-build.
I now have a proof of concept for a pip-based python-build-system, see
1st patch. The changeset is quite small, but it does not handle testing
at all right now.
Note that alot of setuptools-based packages lack a dependency on
python-wheel (2nd patch; pardon the whitespace errors), which is
required when using pyproject.toml (and also declared there, but our
Python importer cannot read that file yet). But – as expected – that’s
really the only issue I’ve encountered while rebuilding alot of Python
packages so far.
Cheers,
Lars
[0001-dirty-build-Build-Python-packages-using-pip.patch (text/x-diff, attachment)]
[0002-dirty-Fix-build-errors.patch (text/x-diff, attachment)]
Information forwarded
to
guix-patches <at> gnu.org:
bug#46848; Package
guix-patches.
(Wed, 07 Jul 2021 15:02:01 GMT)
Full text and
rfc822 format available.
Message #29 received at 46848 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
Am 29.06.21 um 09:20 schrieb Lars-Dominik Braun:
>> AFAIK this might not be true if other build systems not using setuptools
>> at all might show up. And isn't this the main reason for all your work?
> No, try
Sorry, I've been inprecise on this:
There might still be quite some packages out there importing plain, old
distutils (and not setuptools) in their setup.py. These are what I meant
with "other build systems not using setuptools". For these setup.py to
understand the options we (and pip) need for installation, "import
distutils" has to be hacked to actually become "import setuptools" -
which is what setuptools-shim does
--
Regards
Hartmut Goebel
| Hartmut Goebel | h.goebel <at> crazy-compilers.com |
| www.crazy-compilers.com | compilers which you thought are impossible |
[Message part 2 (text/html, inline)]
Information forwarded
to
guix-patches <at> gnu.org:
bug#46848; Package
guix-patches.
(Wed, 25 Aug 2021 07:57:01 GMT)
Full text and
rfc822 format available.
Message #32 received at 46848 <at> debbugs.gnu.org (full text, mbox):
Hi,
> > What should be the use of having a package without pip? Anything else
> > than saving a few KB?
> saving some space and unvendoring components that we also have separate
> packages for. (As I understand it, ensurepip, which installs both pip and
> setuptools, is merely a convenience tool if you install Python from
> source yourself – not for distributions.)
there is another crucial one: Collisions. Right now our python-pip and
python-setuptools packages are rather useless, because they collide with
the Python-bundled one. I’m trying to update Jupyter currently and
python-jupyter-packaging depends on a newer setuptools (v46ish) than our
python package provides (v41). Adding python-setuptools (v52) to the
propagated inputs (yes, it’s a runtime dependency), would be the natural
solution, but the resulting setuptools package in the build environment
for packages depending on python-jupyter-packaging is a mix between v42
and v52 and does not work.
Cheers,
Lars
Information forwarded
to
guix-patches <at> gnu.org:
bug#46848; Package
guix-patches.
(Mon, 13 Dec 2021 20:11:02 GMT)
Full text and
rfc822 format available.
Message #35 received at 46848 <at> debbugs.gnu.org (full text, mbox):
Hi,
I’m working on version 3 of this patchset, removing the dependency
on python-pypa-build, which simplifies bootstrapping. A rough version
is available at
https://github.com/PromyLOPh/guix/commits/work-python-build-pep517
Cheers,
Lars
Information forwarded
to
guix-patches <at> gnu.org:
bug#46848; Package
guix-patches.
(Wed, 05 Jan 2022 14:52:02 GMT)
Full text and
rfc822 format available.
Message #38 received at 46848 <at> debbugs.gnu.org (full text, mbox):
Hi everyone,
I pushed the 3rd version of my new PEP 517-based python-build-system
into wip-python-pep517. I removed the dependency on python-pypa-build
for building packages and instead implemented building with a single
python invokation. This simplifies bootstrapping Python. The 'install
phase also learned how to create scripts in bin/ from an entry points
specification.
Currently 600 of 2212 packages using python-build-system are failing to
build. That’s alot, but most failures so far were related to testsuites,
which were not enabled correctly previously because 'check was not
replaced manually. And some fail because upstream does not separate
sources and tests and a name clash occurs. This also happens when C
extensions are build and there does not seem to be a workaround.
This number does not include Python 2 packages, which will probably
break with this new build system. Since Python 2 is EOL anyway I suggest
to entirely remove Python 2 support before merging this changeset. After
merging we should upgrade the entire Python ecosystem, because alot of
packages are already years old.
Cheers,
Lars
PS: Maxim: This is your ping for a review (via #52269). Thanks :)
Information forwarded
to
guix-patches <at> gnu.org:
bug#46848; Package
guix-patches.
(Thu, 20 Jan 2022 15:42:02 GMT)
Full text and
rfc822 format available.
Message #41 received at 46848 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
Lars-Dominik Braun <lars <at> 6xq.net> skriver:
> I pushed the 3rd version of my new PEP 517-based python-build-system
> into wip-python-pep517. I removed the dependency on python-pypa-build
> for building packages and instead implemented building with a single
> python invokation. This simplifies bootstrapping Python. The 'install
> phase also learned how to create scripts in bin/ from an entry points
> specification.
This is great, thank you.
> Currently 600 of 2212 packages using python-build-system are failing to
> build. That’s alot, but most failures so far were related to testsuites,
> which were not enabled correctly previously because 'check was not
> replaced manually. And some fail because upstream does not separate
> sources and tests and a name clash occurs. This also happens when C
> extensions are build and there does not seem to be a workaround.
Can you elaborate on the name clash issue?
> This number does not include Python 2 packages, which will probably
> break with this new build system. Since Python 2 is EOL anyway I suggest
> to entirely remove Python 2 support before merging this changeset. After
> merging we should upgrade the entire Python ecosystem, because alot of
> packages are already years old.
Unfortunately we need Python 2 for some time still. It is used to
bootstrap old versions of GHC, Node, and probably other things.
Do you think it is feasible to provide this as a new build system, say
pep517-build-system, during a transitional period? Then we can a) start
using it right away for the increasing amount of packages that lack a
setup.py; and b) flesh out most bugs before eventually merging it back
(possibly piecemeal) into python-build-system.
I've had a cursory look and it looks good overall. A few comments:
* Zipping a wheel just to unpack it afterwards is weird, but there seems
to be no way around it.
* I also think trying "python setup.py test" is unnecessary.
* It would be nice to support the "no tests" case without having to add
explicit #:tests? everywhere. Perhaps along the lines of...
(match use-test-backend
('pytest (apply invoke ...))
[...]
(_ (match (find-files "." "^test_.*\\.py$")
(() (format #t "no tests found~%"))
(_ (apply invoke `("python" "-m" "unittest" ,@test-flags))))))
WDYT?
Great work, and apologies for not chiming in earlier!
[signature.asc (application/pgp-signature, inline)]
Information forwarded
to
guix-patches <at> gnu.org:
bug#46848; Package
guix-patches.
(Thu, 20 Jan 2022 18:44:01 GMT)
Full text and
rfc822 format available.
Message #44 received at 46848 <at> debbugs.gnu.org (full text, mbox):
Hi Marius,
> Can you elaborate on the name clash issue?
the problem seems to be that most packages do not put the source code
into a subdirectory in the sdist. I.e. package foobar will have a foobar/
directory, instead of src/foobar. Thus Python tries to load the module
foobar.barbaz from that directory, instead of the store. Thus it’ll
also look there for C extensions and fail. I don’t know why it does
that, even when *not* using `python -m` – according to the documentation
it should not. If anyone has any pointers I’d like to have a look again.
> Unfortunately we need Python 2 for some time still. It is used to
> bootstrap old versions of GHC, Node, and probably other things.
But as far as I see these specific examples do not use
python-build-system. Only if some package using `#:python python-2` is
required during bootstrapping, then we’d need a way to support Python
2, right?
> Do you think it is feasible to provide this as a new build system, say
> pep517-build-system, during a transitional period? Then we can a) start
> using it right away for the increasing amount of packages that lack a
> setup.py; and b) flesh out most bugs before eventually merging it back
> (possibly piecemeal) into python-build-system.
Actually, I think that’s a good idea. I tried, but I cannot fix all
packages broken by this change on my own, so smoothing the transition
could help and – as you said – we could catch bugs early on.
I could prepare a patch, if there is interest in this path.
> * Zipping a wheel just to unpack it afterwards is weird, but there seems
> to be no way around it.
Indeed, that’s how PEP 517 works, which always builds a wheel
(i.e. ZIP file).
> * I also think trying "python setup.py test" is unnecessary.
It still works quite often, although its usefulness will decrease in
the future I guess. Another problem I see is that this command will not
fail if there are no tests.
> * It would be nice to support the "no tests" case without having to add
> explicit #:tests? everywhere. Perhaps along the lines of...
My idea was to force packagers to make and explain this decision
explicitly. If you don’t run tests, you have to add `#:tests? #f`
and leave a comment why they are disabled. I do see this could become
a hassle with packages that simply don’t have any tests. But my hope
is that it increases package quality.
> (_ (apply invoke `("python" "-m" "unittest" ,@test-flags))))))
If I remember correctly I tried this and it did not work for some
reason. I’ll have a look again.
Cheers,
Lars
Information forwarded
to
guix-patches <at> gnu.org:
bug#46848; Package
guix-patches.
(Thu, 20 Jan 2022 20:45:02 GMT)
Full text and
rfc822 format available.
Message #47 received at 46848 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
Lars-Dominik Braun <lars <at> 6xq.net> skriver:
> Hi Marius,
>
>> Can you elaborate on the name clash issue?
> the problem seems to be that most packages do not put the source code
> into a subdirectory in the sdist. I.e. package foobar will have a foobar/
> directory, instead of src/foobar. Thus Python tries to load the module
> foobar.barbaz from that directory, instead of the store. Thus it’ll
> also look there for C extensions and fail. I don’t know why it does
> that, even when *not* using `python -m` – according to the documentation
> it should not. If anyone has any pointers I’d like to have a look again.
Perhaps it could be worked around by building in a separate directory,
i.e. ./build/lib, like setuptools does? And ensure that the original
source directory does not end up on {,GUIX_}PYTHONPATH.
(I don't really understand the problem, just throwing ideas around.)
>> Unfortunately we need Python 2 for some time still. It is used to
>> bootstrap old versions of GHC, Node, and probably other things.
> But as far as I see these specific examples do not use
> python-build-system. Only if some package using `#:python python-2` is
> required during bootstrapping, then we’d need a way to support Python
> 2, right?
Right. If we don't need any Python 2 modules it should be fine to
remove support from the build system.
>> Do you think it is feasible to provide this as a new build system, say
>> pep517-build-system, during a transitional period? Then we can a) start
>> using it right away for the increasing amount of packages that lack a
>> setup.py; and b) flesh out most bugs before eventually merging it back
>> (possibly piecemeal) into python-build-system.
> Actually, I think that’s a good idea. I tried, but I cannot fix all
> packages broken by this change on my own, so smoothing the transition
> could help and – as you said – we could catch bugs early on.
>
> I could prepare a patch, if there is interest in this path.
It would lower the bar significantly for testing and contributing, so I
would appreciate it. :-)
>> * I also think trying "python setup.py test" is unnecessary.
> It still works quite often, although its usefulness will decrease in
> the future I guess. Another problem I see is that this command will not
> fail if there are no tests.
>
>> * It would be nice to support the "no tests" case without having to add
>> explicit #:tests? everywhere. Perhaps along the lines of...
> My idea was to force packagers to make and explain this decision
> explicitly. If you don’t run tests, you have to add `#:tests? #f`
> and leave a comment why they are disabled. I do see this could become
> a hassle with packages that simply don’t have any tests. But my hope
> is that it increases package quality.
My main concern is end-user experience when they just want to get some
random library working on their local channel. But it's arguably
something that can be solved on the importer level and not a strong
opinion.
Thinking about importers, perhaps they could start emitting git origins
when possible, as there is a trend to strip tests before uploading to
PyPI.
>> (_ (apply invoke `("python" "-m" "unittest" ,@test-flags))))))
> If I remember correctly I tried this and it did not work for some
> reason. I’ll have a look again.
Eh, it should be (apply invoke "python" "-m" "unittest" test-flags), but
you probably got that. :-P
Thanks!
[signature.asc (application/pgp-signature, inline)]
Information forwarded
to
guix-patches <at> gnu.org:
bug#46848; Package
guix-patches.
(Sun, 23 Jan 2022 05:30:02 GMT)
Full text and
rfc822 format available.
Message #50 received at 46848 <at> debbugs.gnu.org (full text, mbox):
Hi Lars,
Here's a review of patches 1 to 6.
Lars-Dominik Braun <lars <at> 6xq.net> writes:
> the attached patches switch python-build-system to a PEP 517-based build
> system using python-pypa-build.
Neat! Thank you for working on this!
> One downside is that this tool is not self-contained and has a few
> dependencies. Thus first I bootstrap setuptools using itself (possible
> because it bundles all of its own dependencies), then build
> python-pypa-build’s dependencies using setuptools (which is fortunately
> still possible) and then combine everything into a
> python-toolchain(-for-build), which is then used by the build-process.
>
> I can successfully build packages like python-pypa-build and
> python-pytest and python-pep517-bootstrap. The latter is using flit as
> its build backend. But other packages currently fail because I removed
> some arguments.
> Lars
>
>
>
>
>>From 61313d8ddba30772e2587e3e16ca30d1565d3c7e Mon Sep 17 00:00:00 2001
> From: Lars-Dominik Braun <lars <at> 6xq.net>
> Date: Sun, 28 Feb 2021 13:05:51 +0100
> Subject: [PATCH 04/12] gnu: python-setuptools: Bootstrap using itself
>
> * gnu/packages/python-xyz.scm (python-setuptools) [arguments]: Add phase
> setting GUIX_PYTHONPATH to source directory.
> ---
> gnu/packages/python-xyz.scm | 13 ++++++++++++-
> 1 file changed, 12 insertions(+), 1 deletion(-)
>
> diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm
> index f8afa13f33..79d01f700a 100644
> --- a/gnu/packages/python-xyz.scm
> +++ b/gnu/packages/python-xyz.scm
> @@ -1144,7 +1144,18 @@ other machines, such as over the network.")
> ;; FIXME: Tests require pytest, which itself relies on setuptools.
> ;; One could bootstrap with an internal untested setuptools.
> (arguments
> - `(#:tests? #f))
> + `(#:tests? #f
> + #:python ,python-wrapper
> + #:phases (modify-phases %standard-phases
> + ;; Use this setuptools’ sources to bootstrap themselves.
> + (add-before 'build 'set-PYTHONPATH
^ nitpick: GUIX_PYTHONPATH
> + (lambda _
> + (format #t "current working dir ~s~%" (getcwd))
> + (setenv "GUIX_PYTHONPATH"
> + (string-append ".:" (getenv "GUIX_PYTHONPATH")))
> + #t)))))
> + ;; Not required when not building a wheel
> + ;(propagated-inputs `(("python-wheel" ,python-wheel)))
> (home-page "https://pypi.org/project/setuptools/")
> (synopsis
> "Library designed to facilitate packaging Python projects")
> --
> 2.26.2
>
>
>>From 7a99aaa40e65fde58ee2e78ad7d3e0ccd6d169ae Mon Sep 17 00:00:00 2001
> From: Lars-Dominik Braun <lars <at> 6xq.net>
> Date: Sun, 28 Feb 2021 13:08:58 +0100
> Subject: [PATCH 05/12] python-build: Switch to PEP 517-based build
>
> * gnu/packages/python-commencement.scm: New file, containing
> python-toolchain.
> * gnu/local.mk: Add it.
> * gnu/packages/python.scm (python): Disable installing bundled
> pip/setuptools.
> * guix/build/python-build-system.scm: Rewrite using python-pypa-build.
> * guix/build-system/python.scm (default-python): Switch to
> python-toolchain
> (lower): Remove unused parameter.
>
> XXX: rationale
Forgotten XXX comment (perhaps just drop it).
> ---
> gnu/local.mk | 1 +
> gnu/packages/python-commencement.scm | 175 +++++++++++++++++++
> gnu/packages/python.scm | 2 +-
> guix/build-system/python.scm | 8 +-
> guix/build/python-build-system.scm | 249 ++++++++++++++++++---------
> 5 files changed, 342 insertions(+), 93 deletions(-)
> create mode 100644 gnu/packages/python-commencement.scm
[...]
> + (use-modules (ice-9 match)
> + (srfi srfi-1)
> + (srfi srfi-26)
> + (guix build union))
> +
> + (let ((out (assoc-ref %outputs "out")))
> + (union-build out (filter-map (match-lambda
> + ((_ . directory) directory))
> + %build-inputs))
> + #t))))
Note that returning #t after phases and snippets is obsolete; you can
safely take them all out now.
> + (inputs
> + `(("python" ,python-wrapper)
> + ("python-setuptools" ,python-setuptools)
> + ("python-pip" ,python-pip))) ; XXX Maybe virtualenv/venv too? It kind of
> + ; defeats the purpose of guix, but is used
> + ; alot in local development.
I think it's OK to have people explicitly add virtualenv if they want
it, rather than grow the set of minimal packages here. I'd simply
remove the above XXX comment.
> + (native-search-paths
> + (package-native-search-paths python))
> + (search-paths
> + (package-search-paths python))
> + (license (package-license python)) ; XXX
> + (synopsis "Python toolchain")
> + (description
> + "Python toolchain including Python itself, setuptools and pip. Use this
> +package if you need a fully-fledged Python toolchain instead of just the
> +interpreter.")
> + (home-page (package-home-page python))))
Following my above comment, perhaps s/fully-fledged/minimal/.
[...]
> diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
> index 8e8f46467b..ca5ce667ef 100644
> --- a/gnu/packages/python.scm
> +++ b/gnu/packages/python.scm
> @@ -182,7 +182,7 @@
> (list "--enable-shared" ;allow embedding
> "--with-system-expat" ;for XML support
> "--with-system-ffi" ;build ctypes
> - "--with-ensurepip=install" ;install pip and setuptools
> + "--with-ensurepip=no" ;do not install pip and setuptools
> "--enable-unicode=ucs4"
>
> ;; Prevent the installed _sysconfigdata.py from retaining a reference
> diff --git a/guix/build-system/python.scm b/guix/build-system/python.scm
> index 2bb6fa87ca..998ea9323d 100644
> --- a/guix/build-system/python.scm
> +++ b/guix/build-system/python.scm
> @@ -65,8 +65,8 @@ extension, such as '.tar.gz'."
> (define (default-python)
> "Return the default Python package."
> ;; Lazily resolve the binding to avoid a circular dependency.
> - (let ((python (resolve-interface '(gnu packages python))))
> - (module-ref python 'python-wrapper)))
> + (let ((python (resolve-interface '(gnu packages python-commencement))))
> + (module-ref python 'python-toolchain-for-build)))
>
> (define (default-python2)
> "Return the default Python 2 package."
> @@ -172,8 +172,6 @@ pre-defined variants."
> (define* (python-build store name inputs
> #:key
> (tests? #t)
> - (test-target "test")
> - (use-setuptools? #t)
> (configure-flags ''())
> (phases '(@ (guix build python-build-system)
> %standard-phases))
> @@ -199,9 +197,7 @@ provides a 'setup.py' file as its build system."
> source))
> #:configure-flags ,configure-flags
> #:system ,system
> - #:test-target ,test-target
> #:tests? ,tests?
> - #:use-setuptools? ,use-setuptools?
> #:phases ,phases
> #:outputs %outputs
> #:search-paths ',(map search-path-specification->sexp
> diff --git a/guix/build/python-build-system.scm b/guix/build/python-build-system.scm
> index 8ade1d5911..a5731511a9 100644
> --- a/guix/build/python-build-system.scm
> +++ b/guix/build/python-build-system.scm
> @@ -34,6 +34,7 @@
> #:use-module (ice-9 format)
> #:use-module (srfi srfi-1)
> #:use-module (srfi srfi-26)
> + #:use-module (srfi srfi-35)
> #:export (%standard-phases
> add-installed-pythonpath
> site-packages
> @@ -108,30 +109,17 @@
> ;; "--single-version-externally-managed" is set, thus the .egg-info directory
> ;; and the scripts defined in entry-points will always be created.
>
> +;; Base error type.
> +(define-condition-type &python-build-error &error
> + python-build-error?)
>
> -(define setuptools-shim
> - ;; Run setup.py with "setuptools" being imported, which will patch
> - ;; "distutils". This is needed for packages using "distutils" instead of
> - ;; "setuptools" since the former does not understand the
> - ;; "--single-version-externally-managed" flag.
> - ;; Python code taken from pip 9.0.1 pip/utils/setuptools_build.py
> - (string-append
> - "import setuptools, tokenize;__file__='setup.py';"
> - "f=getattr(tokenize, 'open', open)(__file__);"
> - "code=f.read().replace('\\r\\n', '\\n');"
> - "f.close();"
> - "exec(compile(code, __file__, 'exec'))"))
> -
> -(define (call-setuppy command params use-setuptools?)
> - (if (file-exists? "setup.py")
> - (begin
> - (format #t "running \"python setup.py\" with command ~s and parameters ~s~%"
> - command params)
> - (if use-setuptools?
> - (apply invoke "python" "-c" setuptools-shim
> - command params)
> - (apply invoke "python" "./setup.py" command params)))
> - (error "no setup.py found")))
> +;; Raised when 'check cannot find a valid test system in the inputs.
> +(define-condition-type &test-system-not-found &python-build-error
> + test-system-not-found?)
> +
> +;; Raised when multiple wheels are created by 'build.
> +(define-condition-type &cannot-extract-multiple-wheels &python-build-error
> + cannot-extract-multiple-wheels?)
>
> (define* (sanity-check #:key tests? inputs outputs #:allow-other-keys)
> "Ensure packages depending on this package via setuptools work properly,
> @@ -142,23 +130,51 @@ without errors."
> (with-directory-excursion "/tmp"
> (invoke "python" sanity-check.py (site-packages inputs outputs)))))
>
> -(define* (build #:key use-setuptools? #:allow-other-keys)
> +(define* (build #:key outputs #:allow-other-keys)
> "Build a given Python package."
> - (call-setuppy "build" '() use-setuptools?)
> +
> + (define pyproject-build (which "pyproject-build"))
> +
> + (define (build-pep517)
> + ;; XXX: should probably use a different path, outside of source directory,
> + ;; maybe secondary output “wheel”?
> + (mkdir-p "dist")
> + (invoke pyproject-build "--outdir" "dist" "--no-isolation" "--wheel" "."))
> +
> + ;; XXX Would be nice, if we could use bdist_wheel here to remove extra
> + ;; code path in 'install, but that depends on python-wheel.
> + (define (build-setuptools)
> + (invoke "python" "setup.py" "build"))
> +
> + (if pyproject-build
> + (build-pep517)
> + (build-setuptools))
> #t)
>
> -(define* (check #:key tests? test-target use-setuptools? #:allow-other-keys)
> +(define* (check #:key inputs outputs tests? #:allow-other-keys)
> "Run the test suite of a given Python package."
> (if tests?
> - ;; Running `setup.py test` creates an additional .egg-info directory in
> - ;; build/lib in some cases, e.g. if the source is in a sub-directory
> - ;; (given with `package_dir`). This will by copied to the output, too,
> - ;; so we need to remove.
> - (let ((before (find-files "build" "\\.egg-info$" #:directories? #t)))
> - (call-setuppy test-target '() use-setuptools?)
> - (let* ((after (find-files "build" "\\.egg-info$" #:directories? #t))
> - (inter (lset-difference string=? after before)))
> - (for-each delete-file-recursively inter)))
> + ;; Unfortunately with PEP 517 there is no common method to specify test
> + ;; systems. Guess test system based on inputs instead.
> + (let ((pytest (which "pytest"))
> + (have-setup-py (file-exists? "setup.py")))
^ indentation
> + ;; Prefer pytest
> + ;; XXX: support nose
You can remove this; nose is stale/deprecated.
> + (cond
> + (pytest
> + (begin
> + (format #t "using pytest~%")
> + (invoke pytest "-vv"))) ; XXX: support skipping tests based on name/extra arguments?
We could have a #:test-command argument to specify an arbitrary command
as a list of strings, such as used by the emacs-build-system; that'd
allow us to avoid overriding the phase just to add a '-k "not
this-test"' or similar.
> + ;; But fall back to setup.py, which should work for most
> + ;; packages. XXX: would be nice not to depend on setup.py here? fails
> + ;; more often than not to find any tests at all. Maybe we can run
> + ;; `python -m unittest`?
> + (have-setup-py
> + (begin
> + (format #t "using setup.py~%")
> + (invoke "python" "setup.py" "test" "-v")))
As Marius noted, falling back to 'python setup.py test' is not
desirable; it's scheduled to be removed already.
> + ;; The developer should explicitly disable tests in this case.
> + (#t (raise (condition (&test-system-not-found))))))
> (format #t "test suite not run~%"))
> #t)
>
> @@ -195,31 +211,109 @@ running checks after installing the package."
> "/bin:"
> (getenv "PATH"))))
>
> -(define* (install #:key inputs outputs (configure-flags '()) use-setuptools?
> - #:allow-other-keys)
> - "Install a given Python package."
> - (let* ((out (python-output outputs))
> - (python (assoc-ref inputs "python"))
> - (major-minor (map string->number
> - (take (string-split (python-version python) #\.) 2)))
> - (<3.7? (match major-minor
> - ((major minor)
> - (or (< major 3) (and (= major 3) (< minor 7))))))
> - (params (append (list (string-append "--prefix=" out)
> - "--no-compile")
> - (if use-setuptools?
> - ;; distutils does not accept these flags
> - (list "--single-version-externally-managed"
> - "--root=/")
> - '())
> - configure-flags)))
> - (call-setuppy "install" params use-setuptools?)
> - ;; Rather than produce potentially non-reproducible .pyc files on Pythons
> - ;; older than 3.7, whose 'compileall' module lacks the
> - ;; '--invalidation-mode' option, do not generate any.
> - (unless <3.7?
> - (invoke "python" "-m" "compileall" "--invalidation-mode=unchecked-hash"
> - out))))
> +(define* (install #:key inputs outputs (configure-flags '()) #:allow-other-keys)
> + "Install a wheel file according to PEP 427"
> + ;; See https://www.python.org/dev/peps/pep-0427/#installing-a-wheel-distribution-1-0-py32-none-any-whl
> + (let* ((site-dir (site-packages inputs outputs))
> + (out (assoc-ref outputs "out")))
> + (define (extract file)
> + "Extract wheel (ZIP file) into site-packages directory"
> + ;; Use Python’s zipfile to avoid extra dependency
> + (invoke "python" "-m" "zipfile" "-e" file site-dir))
> +
> + (define python-hashbang
> + (string-append "#!" (assoc-ref inputs "python") "/bin/python"))
> +
> + (define (move-data source destination)
> + (mkdir-p (dirname destination))
> + (rename-file source destination))
> +
> + (define (move-script source destination)
> + "Move executable script file from .data/scripts to out/bin and replace
> +temporary hashbang"
> + (move-data source destination)
> + ;; ZIP does not save/restore permissions, make executable
> + ;; XXX: might not be a file, but directory with subdirectories
> + (chmod destination #o755)
> + (substitute* destination (("#!python") python-hashbang)))
It seems the directory case should be handled? Otherwise the
substitute* call would error out upon encountering it.
> + ;; Python’s distutils.command.install defines this mapping from source to
> + ;; destination mapping.
> + (define install-schemes
> + `(("scripts" "bin" ,move-script)
> + ;; XXX: Why does Python not use share/ here?
> + ("data" "share" ,move-data)))
> +
> + (define (expand-data-directory directory)
> + "Move files from all .data subdirectories to their respective
> +destinations."
> + (for-each
> + (match-lambda ((source destination function)
> + (let ((source-path (string-append directory "/" source))
> + (destination-path (string-append out "/" destination)))
> + (when (file-exists? source-path)
> + (begin
> + ;; This assumes only files exist in the scripts/ directory.
> + (for-each
> + (lambda (file)
> + (apply
> + function
> + (list
> + (string-append source-path "/" file)
> + (string-append destination-path "/" file))))
> + (scandir source-path (negate (cut member <> '("." "..")))))
> + (rmdir source-path))))))
> + install-schemes))
> +
> + (define pyproject-build (which "pyproject-build"))
> +
> + (define (list-directories base predicate)
> + ;; Cannot use find-files here, because it’s recursive.
> + (scandir
> + base
> + (lambda (name)
> + (let ((stat (lstat (string-append base "/" name))))
> + (and
> + (not (member name '("." "..")))
> + (eq? (stat:type stat) 'directory)
> + (predicate name stat))))))
> +
> + (define (install-pep517)
> + "Install a wheel generated by a PEP 517-compatible builder."
> + (let ((wheels (find-files "dist" "\\.whl$"))) ; XXX: do not recurse
If we do not want to recurse, we should use scandir?
> + (when (> (length wheels) 1) ; This code does not support multiple wheels
> + ; yet, because their outputs would have to be
> + ; merged properly.
> + (raise (condition (&cannot-extract-multiple-wheels))))
> + (for-each extract wheels))
> + (let ((datadirs (map
> + (cut string-append site-dir "/" <>)
> + (list-directories site-dir (file-name-predicate "\\.data$")))))
> + (for-each (lambda (directory)
> + (expand-data-directory directory)
> + (rmdir directory))
> + datadirs)))
> +
> + (define (install-setuptools)
> + "Install using setuptools."
> + (let ((out (assoc-ref outputs "out")))
> + (invoke "python" "setup.py"
> + "install"
> + "--prefix" out
> + "--single-version-externally-managed"
> + "--root=/")))
> +
> + (if pyproject-build
> + (install-pep517)
> + (install-setuptools))
> + #t))
So, IIUC, this complicated install phase is because we no longer take
'pip' for granted and is only later available, built from this very
build system, right? Otherwise installing a wheel with pip would be
trivial (c.f. python-isort).
> +
> +(define* (compile-bytecode #:key inputs outputs (configure-flags '()) #:allow-other-keys)
> + "Compile installed byte-code in site-packages."
> + (let ((site-dir (site-packages inputs outputs)))
> + (invoke "python" "-m" "compileall" site-dir)
> + ;; XXX: We could compile with -O and -OO too here, at the cost of more space.
> + #t))
I think you can drop that comment; the default sounds reasonable:
-o OPT_LEVELS Optimization levels to run compilation with. Default is -1 which uses the
optimization level of the Python interpreter itself (see -O).
If we ever want to change we could globally change it for our Python.
I think we keep using "--invalidation-mode=unchecked-hash" though, for
performance [0]:
Unchecked hash-based .pyc files are a useful performance optimization
for environments where a system external to Python (e.g., the build
system) is responsible for keeping .pyc files up-to-date.
[0] https://docs.python.org/3.7/whatsnew/3.7.html#pep-552-hash-based-pyc-files
[...]
It looks rather good to me, with the above comments! I'll be looking
forward reviewing the rest of this series shortly.
Thank you!
Maxim
Information forwarded
to
guix-patches <at> gnu.org:
bug#46848; Package
guix-patches.
(Sun, 23 Jan 2022 10:22:02 GMT)
Full text and
rfc822 format available.
Message #53 received at 46848 <at> debbugs.gnu.org (full text, mbox):
Hi Maxim,
> Here's a review of patches 1 to 6.
thanks for the review. Unfortunately this is not the most recent
proposal and I have no way to retract the previous patches. I pushed v3
to the wip-python-pep517 branch, because of the sheer number of patches
and so the CI could build it (since it requires a rebuild of the entire
rust bootstrap chain).
> > + ;; Prefer pytest
> > + ;; XXX: support nose
>
> You can remove this; nose is stale/deprecated.
So it’s preferred to replace 'check in cases where python-nose is
still in use?
>
> > + (cond
> > + (pytest
> > + (begin
> > + (format #t "using pytest~%")
> > + (invoke pytest "-vv"))) ; XXX: support skipping tests based on name/extra arguments?
>
> We could have a #:test-command argument to specify an arbitrary command
> as a list of strings, such as used by the emacs-build-system; that'd
> allow us to avoid overriding the phase just to add a '-k "not
> this-test"' or similar.
I added #:test-flags in my v3 proposal.
> > + ;; But fall back to setup.py, which should work for most
> > + ;; packages. XXX: would be nice not to depend on setup.py here? fails
> > + ;; more often than not to find any tests at all. Maybe we can run
> > + ;; `python -m unittest`?
> > + (have-setup-py
> > + (begin
> > + (format #t "using setup.py~%")
> > + (invoke "python" "setup.py" "test" "-v")))
>
> As Marius noted, falling back to 'python setup.py test' is not
> desirable; it's scheduled to be removed already.
Sure, but using `python -m unittest` instead requires some investigation.
> > + (define (move-script source destination)
> > + "Move executable script file from .data/scripts to out/bin and replace
> > +temporary hashbang"
> > + (move-data source destination)
> > + ;; ZIP does not save/restore permissions, make executable
> > + ;; XXX: might not be a file, but directory with subdirectories
> > + (chmod destination #o755)
> > + (substitute* destination (("#!python") python-hashbang)))
>
> It seems the directory case should be handled? Otherwise the
> substitute* call would error out upon encountering it.
I have not seen anyone using subdirectories in bin/ yet. Is that
supported anywhere?
> So, IIUC, this complicated install phase is because we no longer take
> 'pip' for granted and is only later available, built from this very
> build system, right? Otherwise installing a wheel with pip would be
> trivial (c.f. python-isort).
If we want to bootstrap these two packages easily (and possibly start
unvendoring their vendored dependencies later), they cannot be part of
this build system and thus we need to implement building/installing
ourselves. I tried using pypa-build in an earlier version, but the
bootstrap chain is unmaintainable.
There also is a project called installer[1], but it does not have a
CLI yet.
Cheers,
Lars
[1] https://github.com/pradyunsg/installer
Information forwarded
to
guix-patches <at> gnu.org:
bug#46848; Package
guix-patches.
(Sat, 26 Feb 2022 14:11:01 GMT)
Full text and
rfc822 format available.
Message #56 received at 46848 <at> debbugs.gnu.org (full text, mbox):
Hi Lars-Dominik,
Lars-Dominik Braun <lars <at> 6xq.net> writes:
When you judge the branch ready to merge, could you please send a subset
of the patches (at least the ones touching the python-build-system
directly) to this issue (marked as v2 -- git send-email -v2) so that
they can be more easily commented?
Thanks,
Maxim
Information forwarded
to
guix-patches <at> gnu.org:
bug#46848; Package
guix-patches.
(Mon, 28 Feb 2022 19:26:01 GMT)
Full text and
rfc822 format available.
Message #59 received at 46848 <at> debbugs.gnu.org (full text, mbox):
Hi Maxim,
> When you judge the branch ready to merge, could you please send a subset
> of the patches (at least the ones touching the python-build-system
> directly) to this issue (marked as v2 -- git send-email -v2) so that
> they can be more easily commented?
as soon as time permits I can do that. Which route do we want to
take? Replace python-build-system entirely in one big merge or add a
new one and slowly migrate?
Cheers,
Lars
Information forwarded
to
guix-patches <at> gnu.org:
bug#46848; Package
guix-patches.
(Mon, 28 Feb 2022 22:34:01 GMT)
Full text and
rfc822 format available.
Message #62 received at 46848 <at> debbugs.gnu.org (full text, mbox):
Hi Lars,
Lars-Dominik Braun <lars <at> 6xq.net> writes:
> Hi Maxim,
>
>> When you judge the branch ready to merge, could you please send a subset
>> of the patches (at least the ones touching the python-build-system
>> directly) to this issue (marked as v2 -- git send-email -v2) so that
>> they can be more easily commented?
>
> as soon as time permits I can do that. Which route do we want to
> take? Replace python-build-system entirely in one big merge or add a
> new one and slowly migrate?
After the Berlin can be rebooted onto its new file system, I expect it
should allow us to keep the build farm much busier than in the past give
us the headroom to experiment with a PEP 517-focused world rebuilding
branch.
I'd aim to have the build system completely replaced; unless your
experience suggests it's going to take multiple weeks to fix iron out
the kinks?
Thanks,
Maxim
Information forwarded
to
guix-patches <at> gnu.org:
bug#46848; Package
guix-patches.
(Sun, 24 Apr 2022 09:14:02 GMT)
Full text and
rfc822 format available.
Message #65 received at 46848 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
Hi Maxim,
> When you judge the branch ready to merge, could you please send a subset
> of the patches (at least the ones touching the python-build-system
> directly) to this issue (marked as v2 -- git send-email -v2) so that
> they can be more easily commented?
I had some time to finish my work, so I pushed all of my changes to
wip-python-pep517 and attached changes that do not fix individual packages
to this email. Since my last version I added support for building Python
2 packages, although we should really phase out Python 2 asap. I kept
support for nose and setup.py’s test target, because they are still
in use/valuable, but we can also remove them if you prefer.
Please have a look when time permits. If all is good we can move on to
fix failing packages.
Thank you very much,
Lars
[v3-0001-build-python-Handle-missing-setuptools-in-sanity-.patch (text/plain, attachment)]
[v3-0002-gnu-python-2-setuptools-Move-to-python-build.patch (text/plain, attachment)]
[v3-0003-python-build-system-Use-PEP-517-compatible-builds.patch (text/plain, attachment)]
[v3-0004-gnu-Use-python-build-system-module-inside-importe.patch (text/plain, attachment)]
[v3-0005-gnu-Use-python2-toolchain-for-build-instead-of-py.patch (text/plain, attachment)]
[v3-0146-gnu-meson-Match-shebang-instead-of-setuptools-spe.patch (text/plain, attachment)]
Information forwarded
to
guix-patches <at> gnu.org:
bug#46848; Package
guix-patches.
(Sun, 24 Apr 2022 09:23:02 GMT)
Full text and
rfc822 format available.
Message #68 received at 46848 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
Hey again,
sorry for the noise. During one of my many rebases I missed to add a
file that should have been included in patch 3. See attached patch.
Cheers,
Lars
[v3-0151-dirty-squash-Add-missing-file.patch (text/plain, attachment)]
Information forwarded
to
guix-patches <at> gnu.org:
bug#46848; Package
guix-patches.
(Wed, 11 Jan 2023 15:42:02 GMT)
Full text and
rfc822 format available.
Message #71 received at 46848 <at> debbugs.gnu.org (full text, mbox):
Hello!
With the pyproject build system now in master, can we close this issue?
--
Thanks,
Maxim
Reply sent
to
Lars-Dominik Braun <lars <at> 6xq.net>:
You have taken responsibility.
(Fri, 10 Feb 2023 10:14:02 GMT)
Full text and
rfc822 format available.
Notification sent
to
Lars-Dominik Braun <lars <at> 6xq.net>:
bug acknowledged by developer.
(Fri, 10 Feb 2023 10:14:02 GMT)
Full text and
rfc822 format available.
Message #76 received at 46848-done <at> debbugs.gnu.org (full text, mbox):
Hi,
> With the pyproject build system now in master, can we close this issue?
yes, I think so.
Cheers,
Lars
bug archived.
Request was from
Debbugs Internal Request <help-debbugs <at> gnu.org>
to
internal_control <at> debbugs.gnu.org.
(Fri, 10 Mar 2023 12:24:08 GMT)
Full text and
rfc822 format available.
This bug report was last modified 1 year and 48 days ago.
Previous Next
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997,2003 nCipher Corporation Ltd,
1994-97 Ian Jackson.