GNU bug report logs -
#49174
Poor 'guix substitute' performance when receiving Zstd-compressed substitutes
Previous Next
To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 49174 in the body.
You can then email your comments to 49174 AT debbugs.gnu.org in the normal way.
Toggle the display of automated, internal messages from the tracker.
Report forwarded
to
bug-guix <at> gnu.org
:
bug#49174
; Package
guix
.
(Tue, 22 Jun 2021 18:51:02 GMT)
Full text and
rfc822 format available.
Acknowledgement sent
to
Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
:
New bug report received and forwarded. Copy sent to
bug-guix <at> gnu.org
.
(Tue, 22 Jun 2021 18:51:02 GMT)
Full text and
rfc822 format available.
Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):
Hello,
It's something I've been observing for a while, but substitutes are very
IO intensive (as can be seen in iotop, the substitute process is waiting
on IO > 99% of the time) and is much slower than expected (3 minutes to
transfer 100 MiB uncompressed over a 50 mbps downstream link):
--8<---------------cut here---------------start------------->8---
TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
13934 be/4 root 1033.09 K/s 1485.06 K/s 0.00 % 93.36 % guile \ /gnu/store/vphx2839xv0qj9xwcwrb95592lzrrnx7-guix-1.3.0-3.50dfbbf/bin/guix substitute --substitute
--8<---------------cut here---------------end--------------->8---
The publisher (remote machine) is has its guix-daemon configured via:
--8<---------------cut here---------------start------------->8---
(service guix-publish-service-type
(guix-publish-configuration
(advertise? #t)
(compression '(("zstd" 3)))
(host "0.0.0.0"))) ;listen on all interfaces
--8<---------------cut here---------------end--------------->8---
CPU is idling during on both sides during the transfer (all the work
appears to be in the IO on the receiving end).
The above example was observed downloading
/gnu/store/4fcwwlv9bzfrraxiz41b4vcv131930fx-libstdc++-doc-9.4.0, but I
do not think it is substitute-specific.
Maxim
Information forwarded
to
bug-guix <at> gnu.org
:
bug#49174
; Package
guix
.
(Wed, 23 Jun 2021 13:27:02 GMT)
Full text and
rfc822 format available.
Message #8 received at 49174 <at> debbugs.gnu.org (full text, mbox):
Hi,
Maxim Cournoyer <maxim.cournoyer <at> gmail.com> skribis:
> It's something I've been observing for a while, but substitutes are very
> IO intensive (as can be seen in iotop, the substitute process is waiting
> on IO > 99% of the time) and is much slower than expected (3 minutes to
> transfer 100 MiB uncompressed over a 50 mbps downstream link):
>
> TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
> 13934 be/4 root 1033.09 K/s 1485.06 K/s 0.00 % 93.36 % guile \ /gnu/store/vphx2839xv0qj9xwcwrb95592lzrrnx7-guix-1.3.0-3.50dfbbf/bin/guix substitute --substitute
>
>
> The publisher (remote machine) is has its guix-daemon configured via:
>
> (service guix-publish-service-type
> (guix-publish-configuration
> (advertise? #t)
> (compression '(("zstd" 3)))
> (host "0.0.0.0"))) ;listen on all interfaces
Note that in this case nars are built and compressed on the fly on the
server side, which puts an upper bound on the bandwidth you can achieve.
I showed earlier how I profiled these things:
https://guix.gnu.org/en/blog/2021/getting-bytes-to-disk-more-quickly/
If the client is I/O-bound, that’s good: it means we can’t do any better
(unless we skip unpacking as demonstrated by distri).
If you can provide detailed profiles of either the server side or the
client side (but in that case, make sure the server is caching things),
that’d be great!
Otherwise I’m afraid this is not actionable. :-)
Thanks,
Ludo’.
Added tag(s) moreinfo.
Request was from
Ludovic Courtès <ludo <at> gnu.org>
to
control <at> debbugs.gnu.org
.
(Sat, 03 Jul 2021 16:11:01 GMT)
Full text and
rfc822 format available.
Reply sent
to
Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
:
You have taken responsibility.
(Fri, 07 Oct 2022 01:12:01 GMT)
Full text and
rfc822 format available.
Notification sent
to
Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
:
bug acknowledged by developer.
(Fri, 07 Oct 2022 01:12:01 GMT)
Full text and
rfc822 format available.
Message #15 received at 49174-done <at> debbugs.gnu.org (full text, mbox):
Hi,
Ludovic Courtès <ludo <at> gnu.org> writes:
> Hi,
>
> Maxim Cournoyer <maxim.cournoyer <at> gmail.com> skribis:
>
>> It's something I've been observing for a while, but substitutes are very
>> IO intensive (as can be seen in iotop, the substitute process is waiting
>> on IO > 99% of the time) and is much slower than expected (3 minutes to
>> transfer 100 MiB uncompressed over a 50 mbps downstream link):
>>
>> TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
>> 13934 be/4 root 1033.09 K/s 1485.06 K/s 0.00 % 93.36 % guile \ /gnu/store/vphx2839xv0qj9xwcwrb95592lzrrnx7-guix-1.3.0-3.50dfbbf/bin/guix substitute --substitute
>>
>>
>> The publisher (remote machine) is has its guix-daemon configured via:
>>
>> (service guix-publish-service-type
>> (guix-publish-configuration
>> (advertise? #t)
>> (compression '(("zstd" 3)))
>> (host "0.0.0.0"))) ;listen on all interfaces
>
> Note that in this case nars are built and compressed on the fly on the
> server side, which puts an upper bound on the bandwidth you can achieve.
>
> I showed earlier how I profiled these things:
>
> https://guix.gnu.org/en/blog/2021/getting-bytes-to-disk-more-quickly/
>
> If the client is I/O-bound, that’s good: it means we can’t do any better
> (unless we skip unpacking as demonstrated by distri).
>
> If you can provide detailed profiles of either the server side or the
> client side (but in that case, make sure the server is caching things),
> that’d be great!
>
> Otherwise I’m afraid this is not actionable. :-)
Since moving from HDDs to SSDs, I haven't seen this problem, so I
suspect the poor IO of the HDDs was really the culprit rather than
something to do with guile-zstd (and we had also benchmarked the late
some when I experimented with using zstd-compressed man pages).
Closing!
--
Thanks,
Maxim
bug archived.
Request was from
Debbugs Internal Request <help-debbugs <at> gnu.org>
to
internal_control <at> debbugs.gnu.org
.
(Fri, 04 Nov 2022 11:24:12 GMT)
Full text and
rfc822 format available.
This bug report was last modified 1 year and 167 days ago.
Previous Next
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997,2003 nCipher Corporation Ltd,
1994-97 Ian Jackson.