GNU bug report logs -
#47251
[core-updates] [patch] gnu: python: Enable optimizations.
Previous Next
Reported by: Julien Lepiller <julien <at> lepiller.eu>
Date: Thu, 18 Mar 2021 23:17:01 UTC
Severity: normal
Tags: patch
Done: Julien Lepiller <julien <at> lepiller.eu>
Bug is archived. No further changes may be made.
To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 47251 in the body.
You can then email your comments to 47251 AT debbugs.gnu.org in the normal way.
Toggle the display of automated, internal messages from the tracker.
Report forwarded
to
guix-patches <at> gnu.org:
bug#47251; Package
guix-patches.
(Thu, 18 Mar 2021 23:17:01 GMT)
Full text and
rfc822 format available.
Acknowledgement sent
to
Julien Lepiller <julien <at> lepiller.eu>:
New bug report received and forwarded. Copy sent to
guix-patches <at> gnu.org.
(Thu, 18 Mar 2021 23:17:01 GMT)
Full text and
rfc822 format available.
Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
Hi Guix!
This is the second patch I created in my exploration of our Python
package. Here, I noticed that our Python was very slow compared to
other distros and I tried to find which options they used. I found some
hints from Fedora. In this patch, I enable a few options and add a C/LD
flag to make python run faster.
I also attached a report of timing running the benchmark from
pyperformance on my Fedora machine. For comparison, I have also added
the results for Fedora's python3, which is somewhere in between.
I noticed that this increases the size of the python package by 20MB,
which is due entirely to --with-lto option, which also accounts for
half of the speedup. Overall, this python is almost twice as fast on
some benchmarks, and always faster on all of them. It's about 1.5 times
faster on average.
Strangely, the python package from Alpine is twice as small as this
package, and they use the --with-lto option. I wonder how they achieve
that.
Note that I modified python-2, as its flags are inherited by python-3
(and also the bootstrap python from commencement.scm).
[0002-gnu-python-Enable-optimizations.patch (text/x-patch, attachment)]
[python-report (application/octet-stream, attachment)]
Information forwarded
to
guix-patches <at> gnu.org:
bug#47251; Package
guix-patches.
(Wed, 31 Mar 2021 12:48:01 GMT)
Full text and
rfc822 format available.
Message #8 received at 47251 <at> debbugs.gnu.org (full text, mbox):
Hi Julien,
Julien Lepiller <julien <at> lepiller.eu> skribis:
>>From c72e34aa0eb0db589557e047c5db0128273d5d12 Mon Sep 17 00:00:00 2001
> From: Julien Lepiller <julien <at> lepiller.eu>
> Date: Thu, 18 Mar 2021 23:41:40 +0100
> Subject: [PATCH 2/2] gnu: python: Enable optimizations.
>
> As a result, python execution time of scripts is decreased by 6 to 40%.
>
> gnu/packages/python (python-2.7)[arguments]: Add configure and make flags to
> optimize the resulting python.
LGTM! Great that you found about these options.
I wonder what can be done to reduce the package size. On
‘core-updates’, we’re already stripping ELF binaries a bit more than on
‘master’.
What does ‘size …/bin/python’ say? Could it be there’s GIMPLE left in
the executable or in libpython.so as a result of using -flto?
Thanks,
Ludo’.
Information forwarded
to
guix-patches <at> gnu.org:
bug#47251; Package
guix-patches.
(Sat, 10 Apr 2021 20:56:02 GMT)
Full text and
rfc822 format available.
Message #11 received at 47251 <at> debbugs.gnu.org (full text, mbox):
Ping! :-)
Ludovic Courtès <ludo <at> gnu.org> skribis:
> Hi Julien,
>
> Julien Lepiller <julien <at> lepiller.eu> skribis:
>
>>>>From c72e34aa0eb0db589557e047c5db0128273d5d12 Mon Sep 17 00:00:00 2001
>> From: Julien Lepiller <julien <at> lepiller.eu>
>> Date: Thu, 18 Mar 2021 23:41:40 +0100
>> Subject: [PATCH 2/2] gnu: python: Enable optimizations.
>>
>> As a result, python execution time of scripts is decreased by 6 to 40%.
>>
>> gnu/packages/python (python-2.7)[arguments]: Add configure and make flags to
>> optimize the resulting python.
>
> LGTM! Great that you found about these options.
>
> I wonder what can be done to reduce the package size. On
> ‘core-updates’, we’re already stripping ELF binaries a bit more than on
> ‘master’.
>
> What does ‘size …/bin/python’ say? Could it be there’s GIMPLE left in
> the executable or in libpython.so as a result of using -flto?
>
> Thanks,
> Ludo’.
Information forwarded
to
guix-patches <at> gnu.org:
bug#47251; Package
guix-patches.
(Sat, 10 Apr 2021 20:56:02 GMT)
Full text and
rfc822 format available.
Message #14 received at 47251 <at> debbugs.gnu.org (full text, mbox):
Ping! :-)
Ludovic Courtès <ludo <at> gnu.org> skribis:
> Hi Julien,
>
> Julien Lepiller <julien <at> lepiller.eu> skribis:
>
>>>>From c72e34aa0eb0db589557e047c5db0128273d5d12 Mon Sep 17 00:00:00 2001
>> From: Julien Lepiller <julien <at> lepiller.eu>
>> Date: Thu, 18 Mar 2021 23:41:40 +0100
>> Subject: [PATCH 2/2] gnu: python: Enable optimizations.
>>
>> As a result, python execution time of scripts is decreased by 6 to 40%.
>>
>> gnu/packages/python (python-2.7)[arguments]: Add configure and make flags to
>> optimize the resulting python.
>
> LGTM! Great that you found about these options.
>
> I wonder what can be done to reduce the package size. On
> ‘core-updates’, we’re already stripping ELF binaries a bit more than on
> ‘master’.
>
> What does ‘size …/bin/python’ say? Could it be there’s GIMPLE left in
> the executable or in libpython.so as a result of using -flto?
>
> Thanks,
> Ludo’.
Information forwarded
to
guix-patches <at> gnu.org:
bug#47251; Package
guix-patches.
(Thu, 03 Jun 2021 02:07:01 GMT)
Full text and
rfc822 format available.
Message #17 received at 47251 <at> debbugs.gnu.org (full text, mbox):
Le Wed, 31 Mar 2021 14:47:44 +0200,
Ludovic Courtès <ludo <at> gnu.org> a écrit :
> Hi Julien,
>
> Julien Lepiller <julien <at> lepiller.eu> skribis:
>
> >>From c72e34aa0eb0db589557e047c5db0128273d5d12 Mon Sep 17 00:00:00
> >>2001
> > From: Julien Lepiller <julien <at> lepiller.eu>
> > Date: Thu, 18 Mar 2021 23:41:40 +0100
> > Subject: [PATCH 2/2] gnu: python: Enable optimizations.
> >
> > As a result, python execution time of scripts is decreased by 6 to
> > 40%.
> >
> > gnu/packages/python (python-2.7)[arguments]: Add configure and make
> > flags to optimize the resulting python.
>
> LGTM! Great that you found about these options.
>
> I wonder what can be done to reduce the package size. On
> ‘core-updates’, we’re already stripping ELF binaries a bit more than
> on ‘master’.
>
> What does ‘size …/bin/python’ say? Could it be there’s GIMPLE left in
> the executable or in libpython.so as a result of using -flto?
>
> Thanks,
> Ludo’.
Sorry for the late reply, I didn't see your message, and never received
the pings I see on issues.guix.gnu.org.
Here is the output of size on bin/python:
text data bss dec hex filename
2072 704 8 2784 ae0
/gnu/store/bcggflp7zszpazgk8qa4sr0kmp5a67pg-python-minimal-3.9.5/bin/python3
Same, for lib/libpython3.so:
text data bss dec hex filename
1154 576 8 1738 6ca
/gnu/store/bcggflp7zszpazgk8qa4sr0kmp5a67pg-python-minimal-3.9.5/lib/libpython3.so
Same for lib/libpython3.9.so.1.0:
text data bss dec hex filename
3614304 254928 139896 4009128 3d2ca8
/gnu/store/bcggflp7zszpazgk8qa4sr0kmp5a67pg-python-minimal-3.9.5/lib/libpython3.9.so.1.0
But I noticed another file gained a lot of weight:
lib/python3.9/config-3.9-x86_64-linux-gnu/libpython3.9.a weighs 39MB,
whereas on master (for 3.8.2) it weighs 6MB. Here is what size has to
say:
text data bss dec hex
298 0 52 350 15e
2642 0 0 2642 a52
370 0 100 470 1d6
481 0 0 481 1e1
1828 0 0 1828 724
1909 0 0 1909 775
1817 512 0 2329 919
24497 0 16032 40529 9e51
110286 976 0 111262 1b29e
12936 0 0 12936 3288
895 0 0 895 37f
1159 0 32 1191 4a7
4311 0 0 4311 10d7
29747 56 0 29803 746b
39941 344 0 40285 9d5d
1252 0 0 1252 4e4
1209 760 16 1985 7c1
16582 0 0 16582 40c6
56468 3856 1 60325 eba5
62176 3520 2080 67776 108c0
20674 0 0 20674 50c2
3160 408 0 3568 df0
1594 488 0 2082 822
4895 1392 16 6303 189f
17197 1384 16 18597 48a5
12586 1072 0 13658 355a
20052 6096 0 26148 6624
4232 1176 0 5408 1520
33447 28560 56 62063 f26f
4950 1096 0 6046 179e
18895 4928 1344 25167 624f
5088 880 0 5968 1750
30728 1496 452 32676 7fa4
9294 1016 16 10326 2856
11512 2328 8 13848 3618
1661 728 0 2389 955
2620 1040 0 3660 e4c
48674 2280 672 51626 c9aa
102026 2024 1240 105290 19b4a
69538 6296 1352 77186 12d82
14228 3728 0 17956 4624
30454 1968 0 32422 7ea6
6484 1304 0 7788 1e6c
9917 1368 8 11293 2c1d
2185 552 0 2737 ab1
31339 1816 0 33155 8183
19061 900 584 20545 5041
1886 528 0 2414 96e
12991 2240 0 15231 3b7f
46206 3624 8 49838 c2ae
5898 1176 8 7082 1baa
5952 248 0 6200 1838
13769 1264 256 15289 3bb9
111003 11760 98432 221195 3600b
280942 5056 2168 288166 465a6
141203 0 0 141203 22793
16406 2000 8 18414 47ee
13022 944 16 13982 369e
129842 2352 1712 133906 20b12
810 0 0 810 32a
71059 72 8 71139 115e3
24372 40 0 24412 5f5c
8726 0 48 8774 2246
41512 3992 0 45504 b1c0
61876 2460 4 64340 fb54
20340 360 8 20708 50e4
74371 88 80 74539 1232b
10273 2488 24 12785 31f1
48 0 0 48 30
26200 472 408 27080 69c8
1257 0 0 1257 4e9
1404 0 0 1404 57c
55841 0 8 55849 da29
117 0 0 117 75
412 0 0 412 19c
110 0 0 110 6e
192 0 250 442 1ba
6170 25384 0 31554 7b42
10561 3280 16 13857 3621
3590 0 0 3590 e06
30434 1392 72 31898 7c9a
2079 24 0 2103 837
36616 0 96 36712 8f68
27149 384 0 27533 6b8d
22163 0 8 22171 569b
494 0 0 494 1ee
4461 0 0 4461 116d
5822 0 48 5870 16ee
7570 0 0 7570 1d92
7739 0 32 7771 1e5b
1144 0 0 1144 478
1584 0 0 1584 630
113 0 0 113 71
4422 24 24 4470 1176
20465 416 672 21553 5431
402 0 0 402 192
14654 0 0 14654 393e
16034 696 0 16730 415a
7026 0 0 7026 1b72
1754 40 4 1798 706
2920 0 0 2920 b68
28827 768 56 29651 73d3
39065 2568 1792 43425 a9a1
4590 96 452 5138 1412
7585 952 0 8537 2159
1562 52 8 1622 656
316 0 0 316 13c
4609 56 0 4665 1239
1979 0 0 1979 7bb
36752 8 2400 39160 98f8
24453 0 0 24453 5f85
11769 12 0 11781 2e05
973 32 3104 4109 100d
299 528 0 827 33b
11964 0 0 11964 2ebc
6063 24 0 6087 17c7
18488 896 0 19384 4bb8
141834 17384 8 159226 26dfa
10041 160 32 10233 27f9
3542 416 0 3958 f76
81990 4624 0 86614 15256
27982 1600 16 29598 739e
1808 320 0 2128 850
12262 2776 8 15046 3ac6
18682 3320 0 22002 55f2
19569 4368 136 24073 5e09
11529 824 0 12353 3041
44792 11312 0 56104 db28
3189 328 0 3517 dbd
16392 968 1520 18880 49c0
6380 608 0 6988 1b4c
17490 1288 444 19222 4b16
17253 3088 424 20765 511d
7382 1456 0 8838 2286
19480 576 216 20272 4f30
11320 1936 0 13256 33c8
14121 1336 0 15457 3c61
12682 1720 0 14402 3842
29184 6088 8 35280 89d0
39623 4080 0 43703 aab7
10987 1112 0 12099 2f43
10034 1864 176 12074 2f2a
17528 488 248 18264 4758
1197 352 0 1549 60d
1614 192 0 1806 70e
31873 5840 440 38153 9509
1644 1504 0 3148 c4c
91445 317 0 91762 16672
I had to remove the file name it comes from, because of width issue
in email, but I don't really understand what size does, so I don't know
if that's relevant or not. Do we need to keep that static library
around?
Information forwarded
to
guix-patches <at> gnu.org:
bug#47251; Package
guix-patches.
(Thu, 03 Jun 2021 09:49:01 GMT)
Full text and
rfc822 format available.
Message #20 received at 47251 <at> debbugs.gnu.org (full text, mbox):
Hi,
Julien Lepiller <julien <at> lepiller.eu> skribis:
> But I noticed another file gained a lot of weight:
> lib/python3.9/config-3.9-x86_64-linux-gnu/libpython3.9.a weighs 39MB,
> whereas on master (for 3.8.2) it weighs 6MB. Here is what size has to
> say:
Yeah, the .o files in there likely contain serialized GIMPLE, as a
result of compiling with ‘-flto’.
How about removing *.a altogether in a separate patch? Or moving it to
a “static” output? This would be consistent with what’s done in other
packages.
Thanks,
Ludo’.
Information forwarded
to
guix-patches <at> gnu.org:
bug#47251; Package
guix-patches.
(Sat, 05 Jun 2021 15:44:02 GMT)
Full text and
rfc822 format available.
Message #23 received at 47251 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
Le Thu, 03 Jun 2021 11:48:40 +0200,
Ludovic Courtès <ludo <at> gnu.org> a écrit :
> Hi,
>
> Julien Lepiller <julien <at> lepiller.eu> skribis:
>
> > But I noticed another file gained a lot of weight:
> > lib/python3.9/config-3.9-x86_64-linux-gnu/libpython3.9.a weighs
> > 39MB, whereas on master (for 3.8.2) it weighs 6MB. Here is what
> > size has to say:
>
> Yeah, the .o files in there likely contain serialized GIMPLE, as a
> result of compiling with ‘-flto’.
>
> How about removing *.a altogether in a separate patch? Or moving it
> to a “static” output? This would be consistent with what’s done in
> other packages.
>
> Thanks,
> Ludo’.
Right, this is a good idea. I wrote the attached patches to: enable
optimizations, prevent building and installing the static library and
to install idle to a separate output, as it is not required in general.
With the patch from https://issues.guix.gnu.org/47214, python 3.9 on
coreupdates is ~1.7x faster than python 3.8 on master, and is ~25MB
smaller than 3.8 on master (I haven't measured the speedup for python2,
and it is ~10MB smaller with these changes than on master).
[0002-gnu-python-Enable-optimizations.patch (text/x-patch, attachment)]
[0003-gnu-python-Do-not-install-static-libraries.patch (text/x-patch, attachment)]
[0004-gnu-python-Move-idle-to-a-separate-output.patch (text/x-patch, attachment)]
Information forwarded
to
guix-patches <at> gnu.org:
bug#47251; Package
guix-patches.
(Sun, 06 Jun 2021 12:35:01 GMT)
Full text and
rfc822 format available.
Message #26 received at 47251 <at> debbugs.gnu.org (full text, mbox):
Hi,
Julien Lepiller <julien <at> lepiller.eu> skribis:
> Right, this is a good idea. I wrote the attached patches to: enable
> optimizations, prevent building and installing the static library and
> to install idle to a separate output, as it is not required in general.
>
> With the patch from https://issues.guix.gnu.org/47214, python 3.9 on
> coreupdates is ~1.7x faster than python 3.8 on master, and is ~25MB
> smaller than 3.8 on master (I haven't measured the speedup for python2,
> and it is ~10MB smaller with these changes than on master).
Yay!
> From 0c45d6e82a5f4d140840aef6b0de321122b6c4df Mon Sep 17 00:00:00 2001
> From: Julien Lepiller <julien <at> lepiller.eu>
> Date: Thu, 18 Mar 2021 23:41:40 +0100
> Subject: [PATCH 2/4] gnu: python: Enable optimizations.
>
> As a result, python execution time of scripts is decreased by 6 to 40%.
>
> gnu/packages/python (python-2.7)[arguments]: Add configure and make flags to
> optimize the resulting python.
[...]
> + "CFLAGS=-fno-semantic-interposition"
> (string-append "LDFLAGS=-Wl,-rpath="
> - (assoc-ref %outputs "out") "/lib"))
> + (assoc-ref %outputs "out") "/lib"
> + " -fno-semantic-interposition"))
Could you add a comment for ‘-fno-semantic-interposition’? Also, please
make sure that setting CFLAGS does not override useful defaults (for
Autoconf-based build systems, CFLAGS defaults to ‘-O2 -g’ and overriding
it means you lost both -O2 and -g; if we lose ‘-g’, ‘--with-debug-info’
won’t have any effect).
> From 1faeb01becd558f0f4eec16ff32df9dbf1adda7f Mon Sep 17 00:00:00 2001
> From: Julien Lepiller <julien <at> lepiller.eu>
> Date: Thu, 3 Jun 2021 22:10:09 +0200
> Subject: [PATCH 3/4] gnu: python: Do not install static libraries.
>
> This is something Fedora already does since 2010, so it should be safe,
> and saves a lot of space.
>
> * gnu/packages/patches/python-2.7-no-static-lib.patch: New file.
> * gnu/packages/patches/python-3-no-static-lib.patch: New file.
> * gnu/local.mk (dist_patch_DATA): Add them.
> * gnu/packages/python.scm (python-2.7, python-3): Use them.
LGTM.
> From c650ac48be47b373c6c8c0b3495f30d9fe21e68b Mon Sep 17 00:00:00 2001
> From: Julien Lepiller <julien <at> lepiller.eu>
> Date: Thu, 3 Jun 2021 22:14:14 +0200
> Subject: [PATCH 4/4] gnu: python: Move idle to a separate output.
>
> idle is a programming environment that is not useful for building
> packages, but it is still useful for end users. This saves 5MB in the
> default output.
>
> * gnu/packages/python.scm (python-2.7)[outputs]: Add `idle'.
> [arguments]: Move idle files in the new output.
> (wrap-python): Do not wrap idle since it is in a different output.
Good idea, LGTM.
Thank you!
Ludo’.
Reply sent
to
Julien Lepiller <julien <at> lepiller.eu>:
You have taken responsibility.
(Fri, 11 Jun 2021 01:04:01 GMT)
Full text and
rfc822 format available.
Notification sent
to
Julien Lepiller <julien <at> lepiller.eu>:
bug acknowledged by developer.
(Fri, 11 Jun 2021 01:04:01 GMT)
Full text and
rfc822 format available.
Message #31 received at 47251-done <at> debbugs.gnu.org (full text, mbox):
Le Sun, 06 Jun 2021 14:34:31 +0200,
Ludovic Courtès <ludo <at> gnu.org> a écrit :
>
> > + "CFLAGS=-fno-semantic-interposition"
> > (string-append "LDFLAGS=-Wl,-rpath="
> > - (assoc-ref %outputs "out") "/lib"))
> > + (assoc-ref %outputs "out") "/lib"
> > + " -fno-semantic-interposition"))
>
> Could you add a comment for ‘-fno-semantic-interposition’? Also,
> please make sure that setting CFLAGS does not override useful
> defaults (for Autoconf-based build systems, CFLAGS defaults to ‘-O2
> -g’ and overriding it means you lost both -O2 and -g; if we lose
> ‘-g’, ‘--with-debug-info’ won’t have any effect).
>
With the patch, python still builds with -g -O3.
Pushed as 99d450b085b7bf44d669093f6c9964391ce1b451 to
e4aa346d57f61956626032d0d1dfca6b3c96e06a to core-updates, thanks!
bug archived.
Request was from
Debbugs Internal Request <help-debbugs <at> gnu.org>
to
internal_control <at> debbugs.gnu.org.
(Fri, 09 Jul 2021 11:24:05 GMT)
Full text and
rfc822 format available.
This bug report was last modified 2 years and 285 days ago.
Previous Next
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997,2003 nCipher Corporation Ltd,
1994-97 Ian Jackson.