GNU bug report logs - #47214
[core-updates] gnu: python: Remove unnecessary files

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 47214 in the body.
You can then email your comments to 47214 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Julien Lepiller <julien <at> lepiller.eu>
To: guix-patches <at> gnu.org
Subject: [core-updates] gnu: python: Remove unnecessary files
Date: Wed, 17 Mar 2021 13:41:02 +0100

[Message part 1 (text/plain, inline)]

Hi Guix!

this patch (going to core-updates) removes files from the output of
python2 and python3, in an attempt to save some space. First, I noticed
that python3 shipped windows binaries in its source distribution and
installs them to the output, in lib/python3.8/distutils/command/*.exe,
lib/python3.8/site-packages/*.exe and
lib/python3.8/site-packages/pip/_vendor/distlib/*.exe. I remove them
from the snippet, so they should not be installed anymore.

I also noticed that python installs files that were built during the
test phase, so I remove them, but we need to check they are not
actually needed. Other distros ship them in a separate package, so
maybe we could have a separate test output if this is causing issues?

I haven't yet tested the patch, because it needs to rebuild part of the
bootstrap chain. I'll report progress and update my patch depending
on the results!

[0001-gnu-python-Remove-unnecessary-files.patch (text/x-patch, attachment)]

Message #8 received at 47214 <at> debbugs.gnu.org (full text, mbox):

From: Efraim Flashner <efraim <at> flashner.co.il>
To: Julien Lepiller <julien <at> lepiller.eu>
Cc: 47214 <at> debbugs.gnu.org
Subject: Re: [bug#47214] [core-updates] gnu: python: Remove unnecessary files
Date: Wed, 17 Mar 2021 21:22:30 +0200

[Message part 1 (text/plain, inline)]

On Wed, Mar 17, 2021 at 01:41:02PM +0100, Julien Lepiller wrote:
> Hi Guix!
> 
> this patch (going to core-updates) removes files from the output of
> python2 and python3, in an attempt to save some space. First, I noticed
> that python3 shipped windows binaries in its source distribution and
> installs them to the output, in lib/python3.8/distutils/command/*.exe,
> lib/python3.8/site-packages/*.exe and
> lib/python3.8/site-packages/pip/_vendor/distlib/*.exe. I remove them
> from the snippet, so they should not be installed anymore.
> 
> I also noticed that python installs files that were built during the
> test phase, so I remove them, but we need to check they are not
> actually needed. Other distros ship them in a separate package, so
> maybe we could have a separate test output if this is causing issues?
> 
> I haven't yet tested the patch, because it needs to rebuild part of the
> bootstrap chain. I'll report progress and update my patch depending
> on the results!

I got stuck on python-minimal with test-asyncio but then it didn't hang
the second time I ran the build. I don't have a good pure python package
I can think of to test the change.

-- 
Efraim Flashner   <efraim <at> flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[signature.asc (application/pgp-signature, inline)]

Message #11 received at 47214 <at> debbugs.gnu.org (full text, mbox):

From: Julien Lepiller <julien <at> lepiller.eu>
To: 47214 <at> debbugs.gnu.org
Subject: Re: [bug#47214] [core-updates] gnu: python: Remove unnecessary files
Date: Fri, 19 Mar 2021 00:08:30 +0100

[Message part 1 (text/plain, inline)]

Le Wed, 17 Mar 2021 13:41:02 +0100,
Julien Lepiller <julien <at> lepiller.eu> a écrit :

> Hi Guix!
> 
> this patch (going to core-updates) removes files from the output of
> python2 and python3, in an attempt to save some space. First, I
> noticed that python3 shipped windows binaries in its source
> distribution and installs them to the output, in
> lib/python3.8/distutils/command/*.exe,
> lib/python3.8/site-packages/*.exe and
> lib/python3.8/site-packages/pip/_vendor/distlib/*.exe. I remove them
> from the snippet, so they should not be installed anymore.
> 
> I also noticed that python installs files that were built during the
> test phase, so I remove them, but we need to check they are not
> actually needed. Other distros ship them in a separate package, so
> maybe we could have a separate test output if this is causing issues?
> 
> I haven't yet tested the patch, because it needs to rebuild part of
> the bootstrap chain. I'll report progress and update my patch
> depending on the results!

I discovered other .exe in python3, so this updated patch deletes them.
Since I modify the python from commencement.scm by changing a phase in
python-2.7, I decided to also change the definition there to remove the
offending files.

I managed to build up to python-3.9 without any issue, but haven't
tested building more than a handful of python packages.

[0001-gnu-python-Remove-unnecessary-files.patch (text/x-patch, attachment)]

Message #14 received at 47214 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Julien Lepiller <julien <at> lepiller.eu>
Cc: 47214 <at> debbugs.gnu.org
Subject: Re: bug#47214: [core-updates] gnu: python: Remove unnecessary files
Date: Fri, 19 Mar 2021 13:27:06 +0100

Hi Julien,

Julien Lepiller <julien <at> lepiller.eu> skribis:

> Le Wed, 17 Mar 2021 13:41:02 +0100,
> Julien Lepiller <julien <at> lepiller.eu> a écrit :
>
>> Hi Guix!
>> 
>> this patch (going to core-updates) removes files from the output of
>> python2 and python3, in an attempt to save some space. First, I
>> noticed that python3 shipped windows binaries in its source
>> distribution and installs them to the output, in
>> lib/python3.8/distutils/command/*.exe,
>> lib/python3.8/site-packages/*.exe and
>> lib/python3.8/site-packages/pip/_vendor/distlib/*.exe. I remove them
>> from the snippet, so they should not be installed anymore.
>> 
>> I also noticed that python installs files that were built during the
>> test phase, so I remove them, but we need to check they are not
>> actually needed. Other distros ship them in a separate package, so
>> maybe we could have a separate test output if this is causing issues?
>> 
>> I haven't yet tested the patch, because it needs to rebuild part of
>> the bootstrap chain. I'll report progress and update my patch
>> depending on the results!
>
> I discovered other .exe in python3, so this updated patch deletes them.
> Since I modify the python from commencement.scm by changing a phase in
> python-2.7, I decided to also change the definition there to remove the
> offending files.

Woow, good catch.

> From 0d2d9a32cbbf24eadf74271b2fb1304b7a72d281 Mon Sep 17 00:00:00 2001
> From: Julien Lepiller <julien <at> lepiller.eu>
> Date: Wed, 17 Mar 2021 13:33:22 +0100
> Subject: [PATCH 1/2] gnu: python: Remove unnecessary files.
>
> * gnu/packages/python.scm (python2)[arguments]: Remove files built
> during the test phase.
> (python3)[arguments]: Also remove windows binaries shipped with the
> sources.

Unless it causes bootstrapping issues, I recommend removing those files
from a snippet so that ‘guix build -S python’ returns the cleaned-up
source.

WDYT?

Thanks,
Ludo’.

Message #17 received at 47214 <at> debbugs.gnu.org (full text, mbox):

From: Julien Lepiller <julien <at> lepiller.eu>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 47214 <at> debbugs.gnu.org
Subject: Re: bug#47214: [core-updates] gnu: python: Remove unnecessary files
Date: Thu, 3 Jun 2021 03:56:38 +0200

Le Fri, 19 Mar 2021 13:27:06 +0100,
Ludovic Courtès <ludo <at> gnu.org> a écrit :

> > From 0d2d9a32cbbf24eadf74271b2fb1304b7a72d281 Mon Sep 17 00:00:00
> > 2001 From: Julien Lepiller <julien <at> lepiller.eu>
> > Date: Wed, 17 Mar 2021 13:33:22 +0100
> > Subject: [PATCH 1/2] gnu: python: Remove unnecessary files.
> >
> > * gnu/packages/python.scm (python2)[arguments]: Remove files built
> > during the test phase.
> > (python3)[arguments]: Also remove windows binaries shipped with the
> > sources.  
> 
> Unless it causes bootstrapping issues, I recommend removing those
> files from a snippet so that ‘guix build -S python’ returns the
> cleaned-up source.
> 
> WDYT?
> 
> Thanks,
> Ludo’.

Hi, sorry for the late reply. I need to use unzip on these files to
remove some of their content, and rezip them later. How can I do that
from a snippet?

Message #20 received at 47214 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Julien Lepiller <julien <at> lepiller.eu>
Cc: 47214 <at> debbugs.gnu.org
Subject: Re: bug#47214: [core-updates] gnu: python: Remove unnecessary files
Date: Thu, 03 Jun 2021 11:46:44 +0200

Hi,

Julien Lepiller <julien <at> lepiller.eu> skribis:

> Le Fri, 19 Mar 2021 13:27:06 +0100,
> Ludovic Courtès <ludo <at> gnu.org> a écrit :
>
>> > From 0d2d9a32cbbf24eadf74271b2fb1304b7a72d281 Mon Sep 17 00:00:00
>> > 2001 From: Julien Lepiller <julien <at> lepiller.eu>
>> > Date: Wed, 17 Mar 2021 13:33:22 +0100
>> > Subject: [PATCH 1/2] gnu: python: Remove unnecessary files.
>> >
>> > * gnu/packages/python.scm (python2)[arguments]: Remove files built
>> > during the test phase.
>> > (python3)[arguments]: Also remove windows binaries shipped with the
>> > sources.  
>> 
>> Unless it causes bootstrapping issues, I recommend removing those
>> files from a snippet so that ‘guix build -S python’ returns the
>> cleaned-up source.

[...]

> Hi, sorry for the late reply. I need to use unzip on these files to
> remove some of their content, and rezip them later. How can I do that
> from a snippet?

Oh right, the bit that unzips/rezips .whl files cannot easily be done
from a snippet, so better keep it as a phase.

Deleting *.exe can be done from a snippet though.

BTW, the patch reads:

   (find-files "Lib/distutils/command" ".*.exe$")

but the regexp should be:

  "\\.exe$"

(“.*” is unnecessary, but the dot before “exe” must be protected.)

Likewise for:

  (find-files "Lib/ensurepip" ".*.whl$")

Thanks,
Ludo’.

Message #25 received at 47214-done <at> debbugs.gnu.org (full text, mbox):

From: Julien Lepiller <julien <at> lepiller.eu>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 47214-done <at> debbugs.gnu.org
Subject: Re: bug#47214: [core-updates] gnu: python: Remove unnecessary files
Date: Fri, 11 Jun 2021 03:01:26 +0200

Le Thu, 03 Jun 2021 11:46:44 +0200,
Ludovic Courtès <ludo <at> gnu.org> a écrit :

> Hi,
> 
> Julien Lepiller <julien <at> lepiller.eu> skribis:
> 
> > Le Fri, 19 Mar 2021 13:27:06 +0100,
> > Ludovic Courtès <ludo <at> gnu.org> a écrit :
> >  
> >> > From 0d2d9a32cbbf24eadf74271b2fb1304b7a72d281 Mon Sep 17 00:00:00
> >> > 2001 From: Julien Lepiller <julien <at> lepiller.eu>
> >> > Date: Wed, 17 Mar 2021 13:33:22 +0100
> >> > Subject: [PATCH 1/2] gnu: python: Remove unnecessary files.
> >> >
> >> > * gnu/packages/python.scm (python2)[arguments]: Remove files
> >> > built during the test phase.
> >> > (python3)[arguments]: Also remove windows binaries shipped with
> >> > the sources.    
> >> 
> >> Unless it causes bootstrapping issues, I recommend removing those
> >> files from a snippet so that ‘guix build -S python’ returns the
> >> cleaned-up source.  
> 
> [...]
> 
> > Hi, sorry for the late reply. I need to use unzip on these files to
> > remove some of their content, and rezip them later. How can I do
> > that from a snippet?  
> 
> Oh right, the bit that unzips/rezips .whl files cannot easily be done
> from a snippet, so better keep it as a phase.
> 
> Deleting *.exe can be done from a snippet though.
> 
> BTW, the patch reads:
> 
>    (find-files "Lib/distutils/command" ".*.exe$")
> 
> but the regexp should be:
> 
>   "\\.exe$"
> 
> (“.*” is unnecessary, but the dot before “exe” must be protected.)
> 
> Likewise for:
> 
>   (find-files "Lib/ensurepip" ".*.whl$")
> 
> Thanks,
> Ludo’.

Fixed, and pushed as 20d52562e5c943b30e48e310b58b94054c3a54e5 to
core-updates.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.