GNU bug report logs - #49174
Poor 'guix substitute' performance when receiving Zstd-compressed substitutes

Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.

Package: guix; Reported by: Maxim Cournoyer <maxim.cournoyer@HIDDEN>; Keywords: moreinfo; dated Tue, 22 Jun 2021 18:51:02 UTC; Maintainer for guix is bug-guix@HIDDEN.
Added tag(s) moreinfo. Request was from Ludovic Courtès <ludo@HIDDEN> to control <at> debbugs.gnu.org. Full text available.

Message received at 49174 <at> debbugs.gnu.org:


Received: (at 49174) by debbugs.gnu.org; 23 Jun 2021 13:26:53 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Wed Jun 23 09:26:53 2021
Received: from localhost ([127.0.0.1]:40603 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1lw2u5-0007vG-7a
	for submit <at> debbugs.gnu.org; Wed, 23 Jun 2021 09:26:53 -0400
Received: from eggs.gnu.org ([209.51.188.92]:35924)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1lw2u3-0007v3-H1
 for 49174 <at> debbugs.gnu.org; Wed, 23 Jun 2021 09:26:51 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:37170)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <ludo@HIDDEN>)
 id 1lw2ty-0002oP-6m; Wed, 23 Jun 2021 09:26:46 -0400
Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=33526 helo=ribbon)
 by fencepost.gnu.org with esmtpsa
 (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1)
 (envelope-from <ludo@HIDDEN>)
 id 1lw2tx-0004kj-Uq; Wed, 23 Jun 2021 09:26:46 -0400
From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>
To: Maxim Cournoyer <maxim.cournoyer@HIDDEN>
Subject: Re: bug#49174: Poor 'guix substitute' performance when receiving
 Zstd-compressed substitutes
References: <87fsx93f99.fsf@HIDDEN>
X-URL: http://www.fdn.fr/~lcourtes/
X-Revolutionary-Date: 5 Messidor an 229 de la =?utf-8?Q?R=C3=A9volution?=
X-PGP-Key-ID: 0x090B11993D9AEBB5
X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc
X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4  0CFB 090B 1199 3D9A EBB5
X-OS: x86_64-pc-linux-gnu
Date: Wed, 23 Jun 2021 15:26:43 +0200
In-Reply-To: <87fsx93f99.fsf@HIDDEN> (Maxim Cournoyer's message of "Tue, 22
 Jun 2021 14:50:42 -0400")
Message-ID: <87eecsyang.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 49174
Cc: 49174 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Hi,

Maxim Cournoyer <maxim.cournoyer@HIDDEN> skribis:

> It's something I've been observing for a while, but substitutes are very
> IO intensive (as can be seen in iotop, the substitute process is waiting
> on IO > 99% of the time) and is much slower than expected (3 minutes to
> transfer 100 MiB uncompressed over a 50 mbps downstream link):
>
> TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
> 13934 be/4 root     1033.09 K/s 1485.06 K/s  0.00 % 93.36 % guile \ /gnu/=
store/vphx2839xv0qj9xwcwrb95592lzrrnx7-guix-1.3.0-3.50dfbbf/bin/guix substi=
tute --substitute
>
>
> The publisher (remote machine) is has its guix-daemon configured via:
>
>       (service guix-publish-service-type
>                (guix-publish-configuration
>                 (advertise? #t)
>                 (compression '(("zstd" 3)))
>                 (host "0.0.0.0")))	;listen on all interfaces

Note that in this case nars are built and compressed on the fly on the
server side, which puts an upper bound on the bandwidth you can achieve.

I showed earlier how I profiled these things:

  https://guix.gnu.org/en/blog/2021/getting-bytes-to-disk-more-quickly/

If the client is I/O-bound, that=E2=80=99s good: it means we can=E2=80=99t =
do any better
(unless we skip unpacking as demonstrated by distri).

If you can provide detailed profiles of either the server side or the
client side (but in that case, make sure the server is caching things),
that=E2=80=99d be great!

Otherwise I=E2=80=99m afraid this is not actionable.  :-)

Thanks,
Ludo=E2=80=99.




Information forwarded to bug-guix@HIDDEN:
bug#49174; Package guix. Full text available.

Message received at submit <at> debbugs.gnu.org:


Received: (at submit) by debbugs.gnu.org; 22 Jun 2021 18:50:49 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Jun 22 14:50:49 2021
Received: from localhost ([127.0.0.1]:39882 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1lvlU1-0007he-9E
	for submit <at> debbugs.gnu.org; Tue, 22 Jun 2021 14:50:49 -0400
Received: from lists.gnu.org ([209.51.188.17]:58646)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <maxim.cournoyer@HIDDEN>) id 1lvlTz-0007hW-Nr
 for submit <at> debbugs.gnu.org; Tue, 22 Jun 2021 14:50:48 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:44130)
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <maxim.cournoyer@HIDDEN>)
 id 1lvlTz-00037z-DF
 for bug-guix@HIDDEN; Tue, 22 Jun 2021 14:50:47 -0400
Received: from mail-qt1-x831.google.com ([2607:f8b0:4864:20::831]:39775)
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128)
 (Exim 4.90_1) (envelope-from <maxim.cournoyer@HIDDEN>)
 id 1lvlTx-0005oB-Oq
 for bug-guix@HIDDEN; Tue, 22 Jun 2021 14:50:47 -0400
Received: by mail-qt1-x831.google.com with SMTP id 17so213411qtt.6
 for <bug-guix@HIDDEN>; Tue, 22 Jun 2021 11:50:44 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=from:to:subject:date:message-id:mime-version;
 bh=o2RRDeRl+CBINwa9K6J/YA6WHkrgnD5KAggvM6kcAW4=;
 b=WtAIf7c5u/MG0Z+xUnrNjjPDHQyzD9k7yKj0Qv+dJ8roq9B9hoJWDgDJLJ12a2s3Pp
 zCXrawuUjjnBBn8uv/wj/c/s0ApBlGHAAmZGOrICw1MiDTBJLdN98SnH9+FMida9kS5E
 dhCO45FlqF9/bELJTTIRFO00jeYVlMQBSmlmvSO+YVZ2Q0Khk6xqbirQwa4ou1hA3Mse
 848HYBECanGWsr6xytLGffBa1TjyRxFZ2iTbRnglrVMw9pXKJeWa3H61fwd28dXcCUKh
 92PtpTrnzgLyRTwuRJsV8INVezBp40RdmyUQyn1iFg3afMWGJKHo+DQsQCUM79TCS7i0
 cN7w==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:from:to:subject:date:message-id:mime-version;
 bh=o2RRDeRl+CBINwa9K6J/YA6WHkrgnD5KAggvM6kcAW4=;
 b=D2WN5N/+3fWTgz2+f5JpQhale15TAv/4auInTcG73hmUh0ro2VQRrmXTdSF43ErlUw
 ESTB5VBJjxD8naota6I4okDcx+Dmpe5DCMkGYVYAerrHBS4BK0YvgkXIstY3yRlAAZi0
 GikvDY8jhFLXdzhyaN5kVBQhG5nZhqZDFaaNkjMEjbK4Sfat4qKF7Qb7QHK31ER4r+Vp
 5baWaGGaT/0iavpeKHciRVlYhYFhriHYBbPOSlfszyd644M27hq6/ENVYzy5hYKTKEKC
 +sVdV4ufB/9+rGzzY0aP44G1eA9NZ+4OCHObCG1mZX1Dvyv3oKAo0MvVv+0W5iQ6f8V5
 LAbQ==
X-Gm-Message-State: AOAM530AF/rILdL4TntbjRh2irHkfc7zMMCVcL0Pm+qwJtalWVzuYq1L
 2p2acMD7GagtkIwA67vLfay0xSqkfIRgiA==
X-Google-Smtp-Source: ABdhPJyIz3N//jlnEYm5InkAZaT0dbds848gMsrFbijbTb9GNrwc2p7mF0nwWO2+qQ+GHX4BnGWonA==
X-Received: by 2002:ac8:4682:: with SMTP id g2mr221010qto.274.1624387843674;
 Tue, 22 Jun 2021 11:50:43 -0700 (PDT)
Received: from hurd (dsl-205-236-230-118.b2b2c.ca. [205.236.230.118])
 by smtp.gmail.com with ESMTPSA id p6sm10151984qkp.80.2021.06.22.11.50.43
 for <bug-guix@HIDDEN>
 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
 Tue, 22 Jun 2021 11:50:43 -0700 (PDT)
From: Maxim Cournoyer <maxim.cournoyer@HIDDEN>
To: bug-guix <bug-guix@HIDDEN>
Subject: Poor 'guix substitute' performance when receiving Zstd-compressed
 substitutes
Date: Tue, 22 Jun 2021 14:50:42 -0400
Message-ID: <87fsx93f99.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain
Received-SPF: pass client-ip=2607:f8b0:4864:20::831;
 envelope-from=maxim.cournoyer@HIDDEN; helo=mail-qt1-x831.google.com
X-Spam_score_int: -20
X-Spam_score: -2.1
X-Spam_bar: --
X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,
 DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001,
 RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001,
 SPF_PASS=-0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-Spam-Score: -1.3 (-)
X-Debbugs-Envelope-To: submit
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -2.3 (--)

Hello,

It's something I've been observing for a while, but substitutes are very
IO intensive (as can be seen in iotop, the substitute process is waiting
on IO > 99% of the time) and is much slower than expected (3 minutes to
transfer 100 MiB uncompressed over a 50 mbps downstream link):

--8<---------------cut here---------------start------------->8---
TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
13934 be/4 root     1033.09 K/s 1485.06 K/s  0.00 % 93.36 % guile \ /gnu/store/vphx2839xv0qj9xwcwrb95592lzrrnx7-guix-1.3.0-3.50dfbbf/bin/guix substitute --substitute
--8<---------------cut here---------------end--------------->8---

The publisher (remote machine) is has its guix-daemon configured via:

--8<---------------cut here---------------start------------->8---
      (service guix-publish-service-type
               (guix-publish-configuration
                (advertise? #t)
                (compression '(("zstd" 3)))
                (host "0.0.0.0")))	;listen on all interfaces
--8<---------------cut here---------------end--------------->8---

CPU is idling during on both sides during the transfer (all the work
appears to be in the IO on the receiving end).

The above example was observed downloading
/gnu/store/4fcwwlv9bzfrraxiz41b4vcv131930fx-libstdc++-doc-9.4.0, but I
do not think it is substitute-specific.

Maxim




Acknowledgement sent to Maxim Cournoyer <maxim.cournoyer@HIDDEN>:
New bug report received and forwarded. Copy sent to bug-guix@HIDDEN. Full text available.
Report forwarded to bug-guix@HIDDEN:
bug#49174; Package guix. Full text available.
Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.
Last modified: Sat, 3 Jul 2021 16:15:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.