GNU logs - #51787, boring messages


Message sent to bug-guix@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: bug#51787: GC takes more than 9 hours on berlin
Resent-From: Mathieu Othacehe <othacehe@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Fri, 12 Nov 2021 11:50:02 +0000
Resent-Message-ID: <handler.51787.B.163671775413756 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: report 51787
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: 51787 <at> debbugs.gnu.org
X-Debbugs-Original-To: bug-guix@HIDDEN
Received: via spool by submit <at> debbugs.gnu.org id=B.163671775413756
          (code B ref -1); Fri, 12 Nov 2021 11:50:02 +0000
Received: (at submit) by debbugs.gnu.org; 12 Nov 2021 11:49:14 +0000
Received: from localhost ([127.0.0.1]:43602 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mlV3S-0003Zo-5F
	for submit <at> debbugs.gnu.org; Fri, 12 Nov 2021 06:49:14 -0500
Received: from lists.gnu.org ([209.51.188.17]:49518)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <othacehe@HIDDEN>) id 1mlV3Q-0003Zg-GF
 for submit <at> debbugs.gnu.org; Fri, 12 Nov 2021 06:49:13 -0500
Received: from eggs.gnu.org ([209.51.188.92]:45604)
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <othacehe@HIDDEN>) id 1mlV3Q-0000Bq-8v
 for bug-guix@HIDDEN; Fri, 12 Nov 2021 06:49:12 -0500
Received: from [2001:470:142:3::e] (port=53196 helo=fencepost.gnu.org)
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <othacehe@HIDDEN>) id 1mlV3K-0003Du-VQ
 for bug-guix@HIDDEN; Fri, 12 Nov 2021 06:49:11 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=MIME-Version:Date:Subject:To:From:in-reply-to:
 references; bh=L6+QnlrYn47J+INqvi/3e31WWkGtQ/lDSyWSm+AW93k=; b=iReggNJs7bcFKh
 BBq5onFnOS7UGiCwIZ3dXGmUZ/UBu71mjK0u0A1iqhenHjSPFTalHWaOXRFWjRGzeg+hm60XAefrG
 socrDU2E1HwYxGaaJxyvCqLC6rqbVbORB0m4Ex8F5Lxx/9Q2062cMHP1hNHz49nBE4mN6fkm32N9L
 ZhRELQnln6cv2atOdfHB5+GtwYPbNuscVN58B7lIep2zfL0SEnAjmzR6PLszEq6mO4DjSCHYOI3rP
 9gCgplWoVxFh/8/S9Y6AgwlyvHf392gx6RYWymzO4oFAbjjj9495Wv8PhT/FfuvFRqWA/Xo9WTBsb
 PTXlsn7Nq5oODAaH8KOQ==;
Received: from [2a01:e0a:19b:d9a0:b4a5:c7aa:fee0:cc15] (port=60362 helo=meije)
 by fencepost.gnu.org with esmtpsa
 (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1)
 (envelope-from <othacehe@HIDDEN>) id 1mlV3K-0001zW-J2
 for bug-guix@HIDDEN; Fri, 12 Nov 2021 06:49:06 -0500
From: Mathieu Othacehe <othacehe@HIDDEN>
Date: Fri, 12 Nov 2021 11:49:04 +0000
Message-ID: <87o86pegr3.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)


Hello,

On berlin, the daily GC command is still running whereas it was started
9 hours ago.

--8<---------------cut here---------------start------------->8---
guix processes
[...]
SessionPID: 37231
ClientPID: 37195
ClientCommand: /gnu/store/49vfv8l1j96bbd73ssbqanpnjz83arss-guix-1.3.0-11.014f1b6/libexec/guix/guile \ /gnu/store/49vfv8l1j96bbd73ssbqanpnjz83arss-guix-1.3.0-11.014f1b6/bin/guix gc -F10995116277760
--8<---------------cut here---------------end--------------->8---

and

--8<---------------cut here---------------start------------->8---
mathieu@berlin ~$ ps auxww|grep 37195
root      37195  0.0  0.0 183260 33440 ?        Sl   03:59   0:00 /gnu/store/49vfv8l1j96bbd73ssbqanpnjz83arss-guix-1.3.0-11.014f1b6/libexec/guix/guile \ /gnu/store/49vfv8l1j96bbd73ssbqanpnjz83arss-guix-1.3.0-11.014f1b6/bin/guix gc -F10995116277760
--8<---------------cut here---------------end--------------->8---

That's really problematic as it is blocking some other berlin services
such as Cuirass, which has 4564 packages in its fetch queue:

--8<---------------cut here---------------start------------->8---
mathieu@berlin ~$ less /var/log/cuirass-remote-server.log
[...]
2021-11-12T12:47:01 period update: 0 resumable, 0 failed builds.
2021-11-12T12:47:01 period update: 4564 items in the fetch queue.
--8<---------------cut here---------------end--------------->8---

Thanks,

Mathieu




Message sent:


Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Mailer: MIME-tools 5.505 (Entity 5.505)
Content-Type: text/plain; charset=utf-8
X-Loop: help-debbugs@HIDDEN
From: help-debbugs@HIDDEN (GNU bug Tracking System)
To: Mathieu Othacehe <othacehe@HIDDEN>
Subject: bug#51787: Acknowledgement (GC takes more than 9 hours on berlin)
Message-ID: <handler.51787.B.163671775413756.ack <at> debbugs.gnu.org>
References: <87o86pegr3.fsf@HIDDEN>
X-Gnu-PR-Message: ack 51787
X-Gnu-PR-Package: guix
Reply-To: 51787 <at> debbugs.gnu.org
Date: Fri, 12 Nov 2021 11:50:02 +0000

Thank you for filing a new bug report with debbugs.gnu.org.

This is an automatically generated reply to let you know your message
has been received.

Your message is being forwarded to the package maintainers and other
interested parties for their attention; they will reply in due course.

Your message has been sent to the package maintainer(s):
 bug-guix@HIDDEN

If you wish to submit further information on this problem, please
send it to 51787 <at> debbugs.gnu.org.

Please do not send mail to help-debbugs@HIDDEN unless you wish
to report a problem with the Bug-tracking system.

--=20
51787: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D51787
GNU Bug Tracking System
Contact help-debbugs@HIDDEN with problems


Message sent to bug-guix@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: bug#51787: GC takes more than 9 hours on berlin
Resent-From: Mathieu Othacehe <othacehe@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Fri, 12 Nov 2021 19:18:01 +0000
Resent-Message-ID: <handler.51787.B51787.163674465412113 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 51787
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: 51787 <at> debbugs.gnu.org
Received: via spool by 51787-submit <at> debbugs.gnu.org id=B51787.163674465412113
          (code B ref 51787); Fri, 12 Nov 2021 19:18:01 +0000
Received: (at 51787) by debbugs.gnu.org; 12 Nov 2021 19:17:34 +0000
Received: from localhost ([127.0.0.1]:45986 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mlc3K-00039J-DZ
	for submit <at> debbugs.gnu.org; Fri, 12 Nov 2021 14:17:34 -0500
Received: from eggs.gnu.org ([209.51.188.92]:56042)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <othacehe@HIDDEN>) id 1mlc3I-000392-9X
 for 51787 <at> debbugs.gnu.org; Fri, 12 Nov 2021 14:17:33 -0500
Received: from [2001:470:142:3::e] (port=59220 helo=fencepost.gnu.org)
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <othacehe@HIDDEN>) id 1mlc3D-0005sP-3N
 for 51787 <at> debbugs.gnu.org; Fri, 12 Nov 2021 14:17:27 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To:
 From; bh=d1CaP/54GIHT1n78g1ZV/QjEqV0vXxA9aC8GmXtO+b4=; b=YLf/HHNk3KIrNR3FbDyq
 WTvZ1esOm4qrt2kknNViSaZkzYbhhKWXKUOqZoDQlRd8cLJv16qa9Rz0wC1O2rkmsYwxb1f6OktvC
 3h/Gzz+bi/Y65lqsf1qFJbCfZm2z91X289H/CF804D8DvqT1a45gxy2wfCKipiwzK6pjsUvuVPiVC
 WVBMzDd26FtY4rlRaTbE4ds2WoUdhrh9wfVxuV/CpPz/mY8JEQ4HyLaywncquh27xjILduYZ+f2B7
 p6C192M0sTareD+azFnGlnhMOGnpRMYYjzuAitTC/7xCTbvhnuF2zRS1EbdePWkbOJNULlvuUsbbp
 zHd8rRwzBMl3gA==;
Received: from [2a01:e0a:19b:d9a0:b4a5:c7aa:fee0:cc15] (port=60382 helo=meije)
 by fencepost.gnu.org with esmtpsa
 (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1)
 (envelope-from <othacehe@HIDDEN>) id 1mlc3C-0007DQ-Re
 for 51787 <at> debbugs.gnu.org; Fri, 12 Nov 2021 14:17:27 -0500
From: Mathieu Othacehe <othacehe@HIDDEN>
References: <87o86pegr3.fsf@HIDDEN>
Date: Fri, 12 Nov 2021 19:17:24 +0000
In-Reply-To: <87o86pegr3.fsf@HIDDEN> (Mathieu Othacehe's message of "Fri, 12
 Nov 2021 11:49:04 +0000")
Message-ID: <87ee7lji9n.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)


> mathieu@berlin ~$ ps auxww|grep 37195
> root      37195  0.0  0.0 183260 33440 ?        Sl   03:59   0:00 /gnu/store/49vfv8l1j96bbd73ssbqanpnjz83arss-guix-1.3.0-11.014f1b6/libexec/guix/guile \ /gnu/store/49vfv8l1j96bbd73ssbqanpnjz83arss-guix-1.3.0-11.014f1b6/bin/guix gc -F10995116277760

I just killed this process, it was running for more than 16 hours.

Mathieu




Message sent to bug-guix@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: bug#51787: GC takes more than 9 hours on berlin
Resent-From: zimoun <zimon.toutoune@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Mon, 22 Nov 2021 09:18:01 +0000
Resent-Message-ID: <handler.51787.B51787.163757262228129 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 51787
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: Mathieu Othacehe <othacehe@HIDDEN>
Cc: 51787 <at> debbugs.gnu.org
Received: via spool by 51787-submit <at> debbugs.gnu.org id=B51787.163757262228129
          (code B ref 51787); Mon, 22 Nov 2021 09:18:01 +0000
Received: (at 51787) by debbugs.gnu.org; 22 Nov 2021 09:17:02 +0000
Received: from localhost ([127.0.0.1]:46889 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mp5RY-0007JC-KH
	for submit <at> debbugs.gnu.org; Mon, 22 Nov 2021 04:17:02 -0500
Received: from mail-io1-f43.google.com ([209.85.166.43]:44659)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <zimon.toutoune@HIDDEN>) id 1mp5RV-0007Ih-5n
 for 51787 <at> debbugs.gnu.org; Mon, 22 Nov 2021 04:16:55 -0500
Received: by mail-io1-f43.google.com with SMTP id f9so22208395ioo.11
 for <51787 <at> debbugs.gnu.org>; Mon, 22 Nov 2021 01:16:53 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112;
 h=mime-version:references:in-reply-to:from:date:message-id:subject:to
 :cc; bh=NYCN6cozve7Z2bKbdcaLN05FFlPxkoWiVk0fgsNrHVg=;
 b=SgyXz5DYKuGoHuIeV86voH3w0b2IEs1Rmq8JMYhBJyTJZ00WAwBgmIuPVPvsfXsFs+
 k7K3RsJiFS+a64jXWDok6xnuJnvjf7nRZSMR/STZQUdOdeTuMhiQdqH2sStqzhC+Ryp1
 HhEcFarn609kJVjqKy4kImDDRnTJ59wMbC3KM0mP6YGpWmUgXrfMfT6u3JcavCf0k0TC
 /sxL2tkqzp7HD6XGHNS2Pq3tk9bNG+KcollRxTwHeBtYf0PeB+lMxe133KvxNFcdtZ8t
 t0tcXgYLo4xDxGkSzul4jeQpPvkD7+8Wgo5Mgvsd3lRGR/VFgYjUoY6/shFP/ls3gpo8
 VY1Q==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20210112;
 h=x-gm-message-state:mime-version:references:in-reply-to:from:date
 :message-id:subject:to:cc;
 bh=NYCN6cozve7Z2bKbdcaLN05FFlPxkoWiVk0fgsNrHVg=;
 b=VqPWvxQ7zWWZ+Fm1xyQ4pdyS0z/IZBB1rNgVq8w77AL/TZ+ynl0mXtA0eIHlSwgSUU
 zJvYVo0sJ561EsP44B8vDd2pnRDBr+a6sTzbCMkqKilgY0lzLRk0WMH5/e2WNm47zVD3
 o3lN2zJ4K9mh5j1VXuGNTyx7tGz1mwFYnUcwC0xuBwU2GsfElx/gPnPsu7WriGBtPu7E
 XBpmhHToXJahvz6Wo7eo2aXPn8DbcZzlQZDwBZywnSMqJx3RyXe3IAL+hj+1sFctPLvs
 Yd+wMLVAbGnsr2ubRnZK64SFXTBG/ak33I2jKxmXjrZ/SJToAC3h6cRtGl2pY37iAVyI
 3xJQ==
X-Gm-Message-State: AOAM531LeSLemOs9eIqyrdRHajjt7yUtpOFl/tb7yxxU9ssIv3fLAP95
 JwA14vgsOvAQ2Nv7isUaExs2CQUPHcJZrNECmCM=
X-Google-Smtp-Source: ABdhPJxXDPeoD8GKRtBplwk/UsVv2f608EnYiQud3/JtB60pknPiTY9ZfRYd4oZj29q6Uu5kQUQM7Wy0Wk34VSHCkfk=
X-Received: by 2002:a05:6638:2bb:: with SMTP id
 d27mr48187375jaq.66.1637572607699; 
 Mon, 22 Nov 2021 01:16:47 -0800 (PST)
MIME-Version: 1.0
References: <87o86pegr3.fsf@HIDDEN>
In-Reply-To: <87o86pegr3.fsf@HIDDEN>
From: zimoun <zimon.toutoune@HIDDEN>
Date: Mon, 22 Nov 2021 10:16:36 +0100
Message-ID: <CAJ3okZ3Y4WRm74jWEW9ax4QhRgimUKVxHiBr8d-CDN6UFh++fQ@HIDDEN>
Content-Type: text/plain; charset="UTF-8"
X-Spam-Score: 0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Hi,

On Fri, 12 Nov 2021 at 12:51, Mathieu Othacehe <othacehe@HIDDEN> wrote:

> On berlin, the daily GC command is still running whereas it was started
> 9 hours ago.
>
> --8<---------------cut here---------------start------------->8---
> guix processes
> [...]
> SessionPID: 37231
> ClientPID: 37195
> ClientCommand: /gnu/store/49vfv8l1j96bbd73ssbqanpnjz83arss-guix-1.3.0-11.014f1b6/libexec/guix/guile \ /gnu/store/49vfv8l1j96bbd73ssbqanpnjz83arss-guix-1.3.0-11.014f1b6/bin/guix gc -F10995116277760
> --8<---------------cut here---------------end--------------->8---
>
> and
>
> --8<---------------cut here---------------start------------->8---
> mathieu@berlin ~$ ps auxww|grep 37195
> root      37195  0.0  0.0 183260 33440 ?        Sl   03:59   0:00 /gnu/store/49vfv8l1j96bbd73ssbqanpnjz83arss-guix-1.3.0-11.014f1b6/libexec/guix/guile \ /gnu/store/49vfv8l1j96bbd73ssbqanpnjz83arss-guix-1.3.0-11.014f1b6/bin/guix gc -F10995116277760
> --8<---------------cut here---------------end--------------->8---
>
> That's really problematic as it is blocking some other berlin services
> such as Cuirass, which has 4564 packages in its fetch queue:
>
> --8<---------------cut here---------------start------------->8---
> mathieu@berlin ~$ less /var/log/cuirass-remote-server.log
> [...]
> 2021-11-12T12:47:01 period update: 0 resumable, 0 failed builds.
> 2021-11-12T12:47:01 period update: 4564 items in the fetch queue.
> --8<---------------cut here---------------end--------------->8---

How is it possible to investigate?

Cheers,
simon




Message received at control <at> debbugs.gnu.org:


Received: (at control) by debbugs.gnu.org; 23 Nov 2021 17:35:02 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Nov 23 12:35:02 2021
Received: from localhost ([127.0.0.1]:52203 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mpZh8-00042j-1r
	for submit <at> debbugs.gnu.org; Tue, 23 Nov 2021 12:35:02 -0500
Received: from eggs.gnu.org ([209.51.188.92]:50422)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1mpZh5-000426-T5
 for control <at> debbugs.gnu.org; Tue, 23 Nov 2021 12:35:01 -0500
Received: from [2001:470:142:3::e] (port=55454 helo=fencepost.gnu.org)
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ludo@HIDDEN>) id 1mpZh0-0004TU-Eo
 for control <at> debbugs.gnu.org; Tue, 23 Nov 2021 12:34:54 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=MIME-version:Subject:From:To:Date:in-reply-to:
 references; bh=F9/PEvoXH/MgqS5/ebo4POBFdVg4BJK3u7y/gVR7YCM=; b=sLQSITDuCWoK9d
 GCwH+w1VrD4ZoSsIMgacsMKU/CLtk4RJptey+srnnrMRjQ0GMSPBdAXtXThNUlKZ5Krj9Kp5gmBWb
 pSA4EFJxrB+tFx7Xa7N6Iu/6cGsaBrw/76K9IxZvAWh3efXtNnufswu95kKI1e5dcCbdtkUjNHkGf
 NTVNHxZk/G/xJmaKQT2qmRjPamkRrtnN3PXcLao/V/d9/+KTTRndKD76fRabOx5d7ieq0i9AJNt+u
 0LVAJsGaK2ryJEfxiuHQRtCCEBrk7A4pfKQc7t2q89i6OVDYwPNSiOJ64lSPwOy9UOs5EYeeUrJSQ
 de78KRDJrPUV2js9C8IQ==;
Received: from 91-160-117-201.subs.proxad.net ([91.160.117.201]:57424
 helo=ribbon)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ludo@HIDDEN>) id 1mpZh0-00060o-7R
 for control <at> debbugs.gnu.org; Tue, 23 Nov 2021 12:34:54 -0500
Date: Tue, 23 Nov 2021 18:34:52 +0100
Message-Id: <878rxeixmr.fsf@HIDDEN>
To: control <at> debbugs.gnu.org
From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>
Subject: control message for bug #51787
MIME-version: 1.0
Content-type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: control
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

severity 51787 important
quit





Message sent to bug-guix@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: bug#51787: GC takes more than 9 hours on berlin
Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Tue, 23 Nov 2021 17:49:02 +0000
Resent-Message-ID: <handler.51787.B51787.163768970025514 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 51787
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: Mathieu Othacehe <othacehe@HIDDEN>
Cc: Christopher Baines <mail@HIDDEN>, 51787 <at> debbugs.gnu.org
Received: via spool by 51787-submit <at> debbugs.gnu.org id=B51787.163768970025514
          (code B ref 51787); Tue, 23 Nov 2021 17:49:02 +0000
Received: (at 51787) by debbugs.gnu.org; 23 Nov 2021 17:48:20 +0000
Received: from localhost ([127.0.0.1]:52211 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mpZu0-0006dS-9X
	for submit <at> debbugs.gnu.org; Tue, 23 Nov 2021 12:48:20 -0500
Received: from eggs.gnu.org ([209.51.188.92]:53382)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1mpZtw-0006dA-GB
 for 51787 <at> debbugs.gnu.org; Tue, 23 Nov 2021 12:48:18 -0500
Received: from [2001:470:142:3::e] (port=56160 helo=fencepost.gnu.org)
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ludo@HIDDEN>)
 id 1mpZtr-0006D3-0U; Tue, 23 Nov 2021 12:48:11 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To:
 From; bh=D4+/pOhc4lcyyNqqozBds24SMdzwsCpWvMZZbi4l/uE=; b=WJCHv03maHI18x9v8FU8
 GwW2B9/+ubBRJfRxV4wCZyznKqHXoZdxUiPVsXCQfM8COgAsaOXZaMFTbIwP4+u6zRN99Wte3KydD
 l69PKn0Kkbec9DwSYhbxFruA8Rr9trvqsLj9f2TOFJKeGsSLPYS3OzBxb9sgAEW4fBxOZ63ok2DDH
 yaBmmXHKUQodSI4DGuCPnzhU5+6wbOluwCzZvhBlkyigXUldkq3gI/DJ5HMJcwXk0B+VFYdKnNuoL
 jnBHDu3c6KJfJRyUWzwz5gT8Vt4RBbu9WcXuRhH4YDe91U/GIZiGiotvcfbfZ7/yq2qfU1cXaniDK
 dKmK1nIB9icDCA==;
Received: from 91-160-117-201.subs.proxad.net ([91.160.117.201]:51615
 helo=ribbon)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ludo@HIDDEN>)
 id 1mpZtq-0007WN-K0; Tue, 23 Nov 2021 12:48:10 -0500
From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
References: <87o86pegr3.fsf@HIDDEN>
Date: Tue, 23 Nov 2021 18:48:08 +0100
In-Reply-To: <87o86pegr3.fsf@HIDDEN> (Mathieu Othacehe's message of "Fri, 12
 Nov 2021 11:49:04 +0000")
Message-ID: <87zgpuhig7.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Hello!

Mathieu Othacehe <othacehe@HIDDEN> skribis:

> On berlin, the daily GC command is still running whereas it was started
> 9 hours ago.

Some data points:

  =E2=80=A2 I deployed on berlin the new daemon featuring the faster =E2=80=
=9Cdeleting
    unused links=E2=80=9D phase from <https://issues.guix.gnu.org/24937> on
    Nov. 20.

    However, that part runs after the GC lock has been released, so it=E2=
=80=99s
    not really relevant (but it is relevant to I/O load and GC
    efficiency.)

  =E2=80=A2 When discussing together with Chris Baines yesterday on IRC, we
    found that the sqlite WAL file was 8=C2=A0GiB.  I later ran:

      PRAGMA wal_checkpoint(TRUNCATE);

    which emptied it immediately.  However, GC time wasn=E2=80=99t particul=
arly
    different today.

  =E2=80=A2 =E2=80=98db.sqlite=E2=80=99 weighs in at 19=C2=A0GiB (!) so per=
haps there=E2=80=99s something to
    do, like the =E2=80=9CVACUUM=E2=80=9D thing maybe.  Chris?

  =E2=80=A2 Stracing the session=E2=80=99s guix-daemon process during GC su=
ggests that
    most of the time goes into I/O from =E2=80=98db.sqlite=E2=80=99.  It=E2=
=80=99s not
    surprising because that GC phase is basically about browsing the
    database, but it does seem to take a little too long for each store
    item.

  =E2=80=A2 I haven=E2=80=99t checked recently but I recall that =E2=80=98g=
uix gc --list-roots=E2=80=99
    (or its C++ counterpart, =E2=80=98findRoots=E2=80=99) would take ages o=
n berlin
    because of all the GC roots Cuirass registers.  It may be that an
    hour or so goes into enumerating GC roots.

Collecting garbage,
Ludo=E2=80=99.




Message sent to bug-guix@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: bug#51787: GC takes more than 9 hours on berlin
Resent-From: Christopher Baines <mail@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Fri, 26 Nov 2021 12:11:01 +0000
Resent-Message-ID: <handler.51787.B51787.163792861929675 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 51787
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Cc: Mathieu Othacehe <othacehe@HIDDEN>, 51787 <at> debbugs.gnu.org
Received: via spool by 51787-submit <at> debbugs.gnu.org id=B51787.163792861929675
          (code B ref 51787); Fri, 26 Nov 2021 12:11:01 +0000
Received: (at 51787) by debbugs.gnu.org; 26 Nov 2021 12:10:19 +0000
Received: from localhost ([127.0.0.1]:58858 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mqa3N-0007iN-SM
	for submit <at> debbugs.gnu.org; Fri, 26 Nov 2021 07:10:19 -0500
Received: from mira.cbaines.net ([212.71.252.8]:47050)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mail@HIDDEN>) id 1mqa3L-0007iB-Vb
 for 51787 <at> debbugs.gnu.org; Fri, 26 Nov 2021 07:10:08 -0500
Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa])
 by mira.cbaines.net (Postfix) with ESMTPSA id 3386927BBE9;
 Fri, 26 Nov 2021 12:10:07 +0000 (GMT)
Received: from capella (localhost [127.0.0.1])
 by localhost (OpenSMTPD) with ESMTP id 03517c3f;
 Fri, 26 Nov 2021 12:10:06 +0000 (UTC)
References: <87o86pegr3.fsf@HIDDEN> <87zgpuhig7.fsf@HIDDEN>
User-agent: mu4e 1.6.6; emacs 27.2
From: Christopher Baines <mail@HIDDEN>
Date: Thu, 25 Nov 2021 13:24:17 +0000
In-reply-to: <87zgpuhig7.fsf@HIDDEN>
Message-ID: <87wnkv15k3.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="=-=-=";
 micalg=pgp-sha512; protocol="application/pgp-signature"
X-Spam-Score: 0.8 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.2 (/)

--=-=-=
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


Ludovic Court=C3=A8s <ludo@HIDDEN> writes:

>   =E2=80=A2 When discussing together with Chris Baines yesterday on IRC, =
we
>     found that the sqlite WAL file was 8=C2=A0GiB.  I later ran:
>
>       PRAGMA wal_checkpoint(TRUNCATE);
>
>     which emptied it immediately.  However, GC time wasn=E2=80=99t partic=
ularly
>     different today.

So, as I understand it, the WAL is made up of pages, and checking for
this db, I think they're the current default size of 4096 bytes.

sqlite> PRAGMA page_size;
4096

From=20looking at the code, the wal_autocheckpoint value is set to 40000:

    /* Increase the auto-checkpoint interval to 40000 pages.  This
       seems enough to ensure that instantiating the NixOS system
       derivation is done in a single fsync(). */
    if (mode =3D=3D "wal" && sqlite3_exec(db, "pragma wal_autocheckpoint =
=3D 40000;", 0, 0, 0) !=3D SQLITE_OK)
        throwSQLiteError(db, "setting autocheckpoint interval");

https://git.savannah.gnu.org/cgit/guix.git/tree/nix/libstore/local-store.cc=
#n253

This means you'd expect the WAL to be in the region of 40000*4096 bytes,
or ~160MB. Assuming the autocheckpointing is keeping up... it doesn't
look to be, since the file is now much larger than this.

As described here [1], the automatic checkpoints are PASSIVE ones, which
has the advantage of not interrupting any readers or writers, but can
also do nothing if it's being blocked by readers or writers.

1: https://www.sqlite.org/wal.html#application_initiated_checkpoints

What I've found while writing the Guix Build Coordinator, is that when
the service is busy (usually new builds being submitted, plus lots of
builds happening), the PASSIVE checkpoints aren't sufficient. To
supplement them, there's a regular check that looks at the size of the
WAL file, and runs a TRUNCATE checkpoint, which is a FULL checkpoint
(which blocks new writers), plus truncating the WAL file once it's
finished checkpointing the entire WAL. The truncating is mostly so that
it's easier to monitor the size of the WAL, by checking the size of the
file.

I feel like I need to defend SQLite at this point. Tuning the
configuration of a database to get acceptable performance is the norm, I
had to tune the PostgreSQL configuration for data.guix.gnu.org to
improve the performance. It's easier to get in to trouble with SQLite
because it's a lower level too, and requires you to actually initiate
things like checkpoints or periodic optimisation if you want them to
happen.

Unfortunately, I don't know enough about the internals of the daemon to
say anything specific though.

>   =E2=80=A2 =E2=80=98db.sqlite=E2=80=99 weighs in at 19=C2=A0GiB (!) so p=
erhaps there=E2=80=99s something to
>     do, like the =E2=80=9CVACUUM=E2=80=9D thing maybe.  Chris?

Doing a VACCUM might address some fragmentation and improve performance,
it's probably worth trying.

>   =E2=80=A2 Stracing the session=E2=80=99s guix-daemon process during GC =
suggests that
>     most of the time goes into I/O from =E2=80=98db.sqlite=E2=80=99.  It=
=E2=80=99s not
>     surprising because that GC phase is basically about browsing the
>     database, but it does seem to take a little too long for each store
>     item.

At least the way I've approached finding and fixing the poor performance
issues in the Guix Build Coordinator is through adding instrumentation,
so just recording the time that calling particular procedures takes, and
then logging if it's longer than some threshold.

Since this issue is about Cuirass, there's also the possibility of
avoiding the problems of a large store, by avoiding having a large
store. That's what bordeaux.guix.gnu.org does, and I thought it was part
of the plan for ci.guix.gnu.org (at least around a year ago)?

--=-=-=
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmGgzpxfFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF
ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh
aW5lcy5uZXQACgkQXiijOwuE9XesMg//X7T5tjRMpZwOF/VJ94u+m8rUYwZ10qkc
sttZnXJnXOLrw9kK1VgB2Dp5foYisDhRoXJ9LEb57tQbjwTWMcR3whD1Cs64/Urw
QZa2dYRzUDuJyVTenTNU+pOedakpKhEDG2dQbAIXxCIoT50cwTxg7JZrpgT7mQa4
Cx3vxBlpucM3SeQvQlBcg36tExLqsN633kMhye8GLdpeGj23vE3tc8doi87sFa/z
LuFyWXczLsor9xNuAPh9CHX+k1hRNS+7DXxV9Ie//ZQkRr3NCUIiPPswq6DyiDn5
dSWU2MgcgrksZZdNodUMLn+ohONnExpxgowVWC9jtTqFRtaym1Mh3CXF8gdoNPsq
j7w3x5zpGzSnUk7hyq1uXfUWikX+C3ps4gMcZRbk5a1kHBczDIn3VOY6ooISJjhr
eg6/SAujAdT9D/6Hb649EuMOUO+BgYXK1Jlmprh1IIxAifaQRp/6qaBigjnDgyJ4
EfYGgAzLhnLdQZzD0lAJP4pwnZJoRby7Ww+S4aCo2ohpe9P39lp9h3pD4dqXFROf
fB8/ev9wkBxx88R7sAyzDynjC7l20Fr/3tvnkC8Uqopq06uwvmQpPzmE75z8Mf4P
yRj4U5ZhJc7K6AgIqjxAzYbRHJiS8zOUCkfmj5fNLZeYh1CNnlYn2yiNyV4x6jw8
C7msGLYcll8=
=rYUT
-----END PGP SIGNATURE-----
--=-=-=--




Message sent to bug-guix@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: bug#51787: GC takes more than 9 hours on berlin
Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Sat, 27 Nov 2021 11:12:01 +0000
Resent-Message-ID: <handler.51787.B51787.163801148823775 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 51787
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: Mathieu Othacehe <othacehe@HIDDEN>
Cc: Christopher Baines <mail@HIDDEN>, 51787 <at> debbugs.gnu.org
Received: via spool by 51787-submit <at> debbugs.gnu.org id=B51787.163801148823775
          (code B ref 51787); Sat, 27 Nov 2021 11:12:01 +0000
Received: (at 51787) by debbugs.gnu.org; 27 Nov 2021 11:11:28 +0000
Received: from localhost ([127.0.0.1]:33043 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mqvc8-0006BP-4Y
	for submit <at> debbugs.gnu.org; Sat, 27 Nov 2021 06:11:28 -0500
Received: from eggs.gnu.org ([209.51.188.92]:54418)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1mqvbt-0006Av-0b
 for 51787 <at> debbugs.gnu.org; Sat, 27 Nov 2021 06:11:26 -0500
Received: from [2001:470:142:3::e] (port=55030 helo=fencepost.gnu.org)
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ludo@HIDDEN>)
 id 1mqvbn-0003Yl-Np; Sat, 27 Nov 2021 06:11:07 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To:
 From; bh=+3m2Sle1gqqBYI/7YhNSHeJ44maAx95sL3MLhqcFgzE=; b=rX4q/ZGLx2jVGAaCSRxr
 jD76ic/i+6chTG4wEOphTPy9p+iXUgcyH76ut1LXGEi5So9AUyViujWIt2+5WO4Rxt40UCVz0TLfR
 VNVp0canWMVqD3En5tHfS3MCbagVy1464gMwXkqR2JFbG/Obf8fUVJXGkjQrVlbWPrCDqsR1uoZv3
 6CnygRb6QJWv9HMkMI6nLwEJblgaPK2/axJ5XuuqBJjV94ngCFJBUBlGAcy4G2NmANWalOkUp7Tj0
 Ts6Z8wh97XQ/mRZz3kBdklzupCV4HvgDWFut2eSu9BHrK/VK44yrk6iR2D+QK+ByQ36+aPQbLXQdh
 CbVSn3Yi3vKczw==;
Received: from 91-160-117-201.subs.proxad.net ([91.160.117.201]:58908
 helo=ribbon)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ludo@HIDDEN>)
 id 1mqvbn-0003ll-EV; Sat, 27 Nov 2021 06:11:07 -0500
From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
References: <87o86pegr3.fsf@HIDDEN> <87zgpuhig7.fsf@HIDDEN>
Date: Sat, 27 Nov 2021 12:11:04 +0100
In-Reply-To: <87zgpuhig7.fsf@HIDDEN> ("Ludovic =?UTF-8?Q?Court=C3=A8s?="'s message of "Tue, 23 Nov 2021 18:48:08 +0100")
Message-ID: <87ilwdam5z.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: 0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Ludovic Court=C3=A8s <ludo@HIDDEN> skribis:

>   =E2=80=A2 Stracing the session=E2=80=99s guix-daemon process during GC =
suggests that
>     most of the time goes into I/O from =E2=80=98db.sqlite=E2=80=99.  It=
=E2=80=99s not
>     surprising because that GC phase is basically about browsing the
>     database, but it does seem to take a little too long for each store
>     item.

Stracing the client shows that the daemon spends several seconds on a
single store item occasionally:

--8<---------------cut here---------------start------------->8---
read(27, "gmlo\0\0\0\0", 8)             =3D 8 <0.363064>
read(27, "c\0\0\0\0\0\0\0", 8)          =3D 8 <0.000013>
read(27, "[95%] deleting '/gnu/store/p6r2jjy6frp682z3x94nvnmdh71p1p58-ecl-q=
uicksearch-0.01"..., 104) =3D 104 <0.000010>
write(2, "[95%] deleting '/gnu/store/p6r2jjy6frp682z3x94nvnmdh71p1p58-ecl-q=
uicksearch-0.01"..., 99) =3D 99 <0.000019>
read(27, "gmlo\0\0\0\0", 8)             =3D 8 <0.017863>
read(27, "^\0\0\0\0\0\0\0", 8)          =3D 8 <0.000019>
read(27, "[95%] deleting '/gnu/store/v6zd510kfmqd8j4w7q3zy9bid1fj96dk-sheph=
erd-guix-daemon"..., 96) =3D 96 <0.000007>
write(2, "[95%] deleting '/gnu/store/v6zd510kfmqd8j4w7q3zy9bid1fj96dk-sheph=
erd-guix-daemon"..., 94) =3D 94 <0.000012>
read(27, "gmlo\0\0\0\0", 8)             =3D 8 <5.861071>
read(27, "T\0\0\0\0\0\0\0", 8)          =3D 8 <0.000061>
read(27, "[95%] deleting '/gnu/store/0hpwig8cwdnzygjjzs9zjbxicvhif2vv-rust-=
bitvec-0.19.4.d"..., 88) =3D 88 <0.000087>
write(2, "[95%] deleting '/gnu/store/0hpwig8cwdnzygjjzs9zjbxicvhif2vv-rust-=
bitvec-0.19.4.d"..., 84) =3D 84 <0.000033>
--8<---------------cut here---------------end--------------->8---

(Notice =E2=80=98read=E2=80=99 taking 5.9s above.)

Ludo=E2=80=99.




Message sent to bug-guix@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: bug#51787: GC takes more than 9 hours on berlin
Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Sat, 27 Nov 2021 11:24:02 +0000
Resent-Message-ID: <handler.51787.B51787.163801219024941 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 51787
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: Christopher Baines <mail@HIDDEN>
Cc: Mathieu Othacehe <othacehe@HIDDEN>, 51787 <at> debbugs.gnu.org
Received: via spool by 51787-submit <at> debbugs.gnu.org id=B51787.163801219024941
          (code B ref 51787); Sat, 27 Nov 2021 11:24:02 +0000
Received: (at 51787) by debbugs.gnu.org; 27 Nov 2021 11:23:10 +0000
Received: from localhost ([127.0.0.1]:33059 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mqvnR-0006UC-Rp
	for submit <at> debbugs.gnu.org; Sat, 27 Nov 2021 06:23:10 -0500
Received: from eggs.gnu.org ([209.51.188.92]:56470)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1mqvnQ-0006Tz-MK
 for 51787 <at> debbugs.gnu.org; Sat, 27 Nov 2021 06:23:09 -0500
Received: from [2001:470:142:3::e] (port=55412 helo=fencepost.gnu.org)
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ludo@HIDDEN>)
 id 1mqvnL-0005Pf-Fb; Sat, 27 Nov 2021 06:23:03 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To:
 From; bh=5mvaxKCQSiBJX83BU6inKQlM7LkweTWP5L8gOQnMwCc=; b=MmD2mC3il8BhgZq6hOf2
 EtCKcl/6d7HcbF3XlMV4Ui9ZJDC9GyTyGLyHDROfTGZL373f5XrxfUodJjMAC0Goxm1LQEfXnhQFJ
 lh1X2UY3v+poPMNPlGnxKfvQRxIZk0ZUDyoP7pbxj80/s+eKV0ig01FcJXAmY3nGs9FBP8pvweCkA
 HBs3a9LSU/i+yDHHEUEi8HtZsI9203VznVcrTIf9kQKIuIVaVGSeRIaXStbi9Dptq2W6gCd80+ABt
 s6cMXd0KPHIlvZb/YJA7kYWibgQPMuUeDbIMzwaNjUl3IaskGbeUqvDjJ98rBDsK+aMsdjljOGB3P
 CG2BbbVuZqSVVQ==;
Received: from 91-160-117-201.subs.proxad.net ([91.160.117.201]:49494
 helo=ribbon)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ludo@HIDDEN>)
 id 1mqvnL-0003e9-5F; Sat, 27 Nov 2021 06:23:03 -0500
From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
References: <87o86pegr3.fsf@HIDDEN> <87zgpuhig7.fsf@HIDDEN>
 <87wnkv15k3.fsf@HIDDEN>
Date: Sat, 27 Nov 2021 12:23:00 +0100
In-Reply-To: <87wnkv15k3.fsf@HIDDEN> (Christopher Baines's message of
 "Thu, 25 Nov 2021 13:24:17 +0000")
Message-ID: <87zgpp971n.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Hi Chris,

Christopher Baines <mail@HIDDEN> skribis:

> So, as I understand it, the WAL is made up of pages, and checking for
> this db, I think they're the current default size of 4096 bytes.
>
> sqlite> PRAGMA page_size;
> 4096
>
> From looking at the code, the wal_autocheckpoint value is set to 40000:
>
>     /* Increase the auto-checkpoint interval to 40000 pages.  This
>        seems enough to ensure that instantiating the NixOS system
>        derivation is done in a single fsync(). */
>     if (mode =3D=3D "wal" && sqlite3_exec(db, "pragma wal_autocheckpoint =
=3D 40000;", 0, 0, 0) !=3D SQLITE_OK)
>         throwSQLiteError(db, "setting autocheckpoint interval");
>
> https://git.savannah.gnu.org/cgit/guix.git/tree/nix/libstore/local-store.=
cc#n253
>
> This means you'd expect the WAL to be in the region of 40000*4096 bytes,
> or ~160MB. Assuming the autocheckpointing is keeping up... it doesn't
> look to be, since the file is now much larger than this.
>
> As described here [1], the automatic checkpoints are PASSIVE ones, which
> has the advantage of not interrupting any readers or writers, but can
> also do nothing if it's being blocked by readers or writers.
>
> 1: https://www.sqlite.org/wal.html#application_initiated_checkpoints
>
> What I've found while writing the Guix Build Coordinator, is that when
> the service is busy (usually new builds being submitted, plus lots of
> builds happening), the PASSIVE checkpoints aren't sufficient. To
> supplement them, there's a regular check that looks at the size of the
> WAL file, and runs a TRUNCATE checkpoint, which is a FULL checkpoint
> (which blocks new writers), plus truncating the WAL file once it's
> finished checkpointing the entire WAL. The truncating is mostly so that
> it's easier to monitor the size of the WAL, by checking the size of the
> file.

OK.  That may well be what happens on berlin these days: the database is
kept busy all day long, so presumably checkpoints don=E2=80=99t happen and =
the
WAL file grows.

> I feel like I need to defend SQLite at this point. Tuning the
> configuration of a database to get acceptable performance is the norm, I
> had to tune the PostgreSQL configuration for data.guix.gnu.org to
> improve the performance. It's easier to get in to trouble with SQLite
> because it's a lower level too, and requires you to actually initiate
> things like checkpoints or periodic optimisation if you want them to
> happen.

Understood.  It=E2=80=99s really not about defending software X against Y, =
but
rather about finding ways to address the issues we experience.

>>   =E2=80=A2 =E2=80=98db.sqlite=E2=80=99 weighs in at 19=C2=A0GiB (!) so =
perhaps there=E2=80=99s something to
>>     do, like the =E2=80=9CVACUUM=E2=80=9D thing maybe.  Chris?
>
> Doing a VACCUM might address some fragmentation and improve performance,
> it's probably worth trying.

Alright, let=E2=80=99s give it a try.

>>   =E2=80=A2 Stracing the session=E2=80=99s guix-daemon process during GC=
 suggests that
>>     most of the time goes into I/O from =E2=80=98db.sqlite=E2=80=99.  It=
=E2=80=99s not
>>     surprising because that GC phase is basically about browsing the
>>     database, but it does seem to take a little too long for each store
>>     item.
>
> At least the way I've approached finding and fixing the poor performance
> issues in the Guix Build Coordinator is through adding instrumentation,
> so just recording the time that calling particular procedures takes, and
> then logging if it's longer than some threshold.

Yeah, that makes sense.  I think we always took these bits of the daemon
for granted because they=E2=80=99d been used on large stores even before Gu=
ix
existed.

> Since this issue is about Cuirass, there's also the possibility of
> avoiding the problems of a large store, by avoiding having a large
> store. That's what bordeaux.guix.gnu.org does, and I thought it was part
> of the plan for ci.guix.gnu.org (at least around a year ago)?

That=E2=80=99s indeed the case: the store is smaller than it used to be (but
still 27=C2=A0TiB), it=E2=80=99s GC=E2=80=99d more aggressively than before=
, and instead we
rely on /var/cache/guix/publish for long-term storage.

Perhaps we should go further and keep the store smaller though.

Thanks for your feedback!

Ludo=E2=80=99.




Message sent to bug-guix@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: bug#51787: GC takes more than 9 hours on berlin
Resent-From: Mathieu Othacehe <othacehe@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Fri, 03 Dec 2021 09:47:02 +0000
Resent-Message-ID: <handler.51787.B51787.163852476925995 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 51787
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Cc: Christopher Baines <mail@HIDDEN>, 51787 <at> debbugs.gnu.org
Received: via spool by 51787-submit <at> debbugs.gnu.org id=B51787.163852476925995
          (code B ref 51787); Fri, 03 Dec 2021 09:47:02 +0000
Received: (at 51787) by debbugs.gnu.org; 3 Dec 2021 09:46:09 +0000
Received: from localhost ([127.0.0.1]:50044 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mt58p-0006lC-VY
	for submit <at> debbugs.gnu.org; Fri, 03 Dec 2021 04:46:09 -0500
Received: from eggs.gnu.org ([209.51.188.92]:37814)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <othacehe@HIDDEN>) id 1mt58k-0006kP-Hf
 for 51787 <at> debbugs.gnu.org; Fri, 03 Dec 2021 04:46:06 -0500
Received: from [2001:470:142:3::e] (port=52632 helo=fencepost.gnu.org)
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <othacehe@HIDDEN>)
 id 1mt58f-0006yX-8P; Fri, 03 Dec 2021 04:45:57 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To:
 From; bh=xmwmbW9vXVr/ZtwjcCAoYK4+ewUBpIpM+LgIa8ApClg=; b=aMxx0sylfmKaraIDabWH
 Kp53oKd8T1sBKu1watkFVwPWYypqjtEalhTTYJgmltFGdSvNOK5ZDFH3u4a0dM+RpA0IdHZJtJy+b
 uXBSxAaIHmPCCcNbMDwOxetoosaejoRWGuhu76HoLpeqLL/xXA+TtwhfRonDdfuL70UCPaEOM6WJI
 tOZ2+8sQwuNGwqjBUU0mW8Yyctf5cYM7ZJmZqDQjegbOM7EUGqXc/OLztcwwJmYk4fRTn+WnSOrml
 gT0ATTkE2CK0x/i1u+hELbE1D9VAiiYmMOkMwWbDNVIhez8Ts8BxbUO/8LeFmgNYDmwhCSvY0p+LJ
 RvLNrwp7KtQCiw==;
Received: from [2a01:e0a:19b:d9a0:45b5:a14a:5c75:5737] (port=32910 helo=meije)
 by fencepost.gnu.org with esmtpsa
 (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1)
 (envelope-from <othacehe@HIDDEN>)
 id 1mt58e-0003JZ-8W; Fri, 03 Dec 2021 04:45:57 -0500
From: Mathieu Othacehe <othacehe@HIDDEN>
References: <87o86pegr3.fsf@HIDDEN> <87zgpuhig7.fsf@HIDDEN>
 <87wnkv15k3.fsf@HIDDEN> <87zgpp971n.fsf@HIDDEN>
Date: Fri, 03 Dec 2021 10:45:54 +0100
In-Reply-To: <87zgpp971n.fsf@HIDDEN> ("Ludovic =?UTF-8?Q?Court=C3=A8s?="'s message of "Sat, 27 Nov 2021 12:23:00 +0100")
Message-ID: <87pmqet419.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)


Hello,

> That=E2=80=99s indeed the case: the store is smaller than it used to be (=
but
> still 27=C2=A0TiB), it=E2=80=99s GC=E2=80=99d more aggressively than befo=
re, and instead we
> rely on /var/cache/guix/publish for long-term storage.
>
> Perhaps we should go further and keep the store smaller though.

That's what I did with 93adf7aaa693d234ee13240e9f4ff22a2dfef599 on
maintenance. It increases the GC threshold to 15TiB. Let's see if it
brings some improvements.

Thanks,

Mathieu




Message sent to bug-guix@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: bug#51787: GC takes more than 9 hours on berlin
Resent-From: Mathieu Othacehe <othacehe@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Fri, 10 Dec 2021 06:25:02 +0000
Resent-Message-ID: <handler.51787.B51787.16391174666736 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 51787
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Cc: 51787 <at> debbugs.gnu.org
Received: via spool by 51787-submit <at> debbugs.gnu.org id=B51787.16391174666736
          (code B ref 51787); Fri, 10 Dec 2021 06:25:02 +0000
Received: (at 51787) by debbugs.gnu.org; 10 Dec 2021 06:24:26 +0000
Received: from localhost ([127.0.0.1]:45305 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mvZKT-0001kZ-LS
	for submit <at> debbugs.gnu.org; Fri, 10 Dec 2021 01:24:25 -0500
Received: from eggs.gnu.org ([209.51.188.92]:35826)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <othacehe@HIDDEN>) id 1mvZKR-0001kM-9z
 for 51787 <at> debbugs.gnu.org; Fri, 10 Dec 2021 01:24:24 -0500
Received: from [2001:470:142:3::e] (port=43114 helo=fencepost.gnu.org)
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <othacehe@HIDDEN>)
 id 1mvZKL-0005h1-Ta; Fri, 10 Dec 2021 01:24:17 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To:
 From; bh=sqmiVqIAwj+IyqMPVIM7phDU/I4q3A1Acdo4zXzSmGo=; b=FLBy/2tkBklCPKlOBVbt
 dTGTdjLT19OcAJkVJltH8KnFH0PQr1aoYzaxf7fCcBUSnMVlJwL2cewJuxy80wc85E51l+kOUxsf5
 0LwjPmQJNVdsK2D3Z8vKiYX+SgxuSztSFfXSlVjmDrhar5ucZBrdBB9CFzQplS54Mjv7C65MdUj3V
 4As4Z4BJ6+HINU8r6w5sjbHlUv4hhCSMNd8eFFF/23RGelTjM2WDm5jYDcr61npemGiscMjDSeNvS
 52NTpV1kSfHc0nSgnArYWhI5kxXCy8MynbxJtsPfIKtwuUiR6P1VSxizrunHAGjGfxCjx1PY26SEI
 Myc+yEn3reRQvQ==;
Received: from [2a01:e0a:19b:d9a0:2ddb:d3d2:32e8:d31a] (port=33052 helo=meije)
 by fencepost.gnu.org with esmtpsa
 (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1)
 (envelope-from <othacehe@HIDDEN>)
 id 1mvZKL-0001xQ-R3; Fri, 10 Dec 2021 01:24:18 -0500
From: Mathieu Othacehe <othacehe@HIDDEN>
References: <87o86pegr3.fsf@HIDDEN> <87zgpuhig7.fsf@HIDDEN>
 <87wnkv15k3.fsf@HIDDEN> <87zgpp971n.fsf@HIDDEN>
 <87pmqet419.fsf@HIDDEN>
Date: Fri, 10 Dec 2021 07:24:15 +0100
In-Reply-To: <87pmqet419.fsf@HIDDEN> (Mathieu Othacehe's message of "Fri, 03
 Dec 2021 10:45:54 +0100")
Message-ID: <87czm57zao.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)


Hey,

New GC recap. The process that has been started yesterday at 04:00 is
still running. I killed the GC that was started today at 04:00 to keep
things clear.

From yesterday 11:00 when I started monitoring it to today when I'm
writing this email, 20 hours have elapsed and the GC is still in the
same phase: removing recursively the /gnu/store/trash directory content.

It corresponds to the following snippet for those of you who would like
to have a look to the corresponding code:

--8<---------------cut here---------------start------------->8---
if (state.shouldDelete) {
    if (pathExists(state.trashDir)) deleteGarbage(state, state.trashDir); // > 20 hours
    try {
        createDirs(state.trashDir);
    } catch (SysError & e) {
        if (e.errNo == ENOSPC) {
            printMsg(lvlInfo, format("note: can't create trash directory: %1%") % e.msg());
            state.moveToTrash = false;
        }
    }
}--8<---------------cut here---------------end--------------->8---

This is an early phase of the garbage collecting, where store items that
were moved to the trash directory by previous GC runs are effectively
removed.

Stracing the guix-daemon process associated with the GC process clearly
shows what's going on:

--8<---------------cut here---------------start------------->8---
chmod("/gnu/store/trash/272ibwb38i0kcbcl3n9v0ka1rsmd1104-guix-web-site/de/packages/rust-syntex-0.58.1", 040755) = 0 <0.000012>
openat(AT_FDCWD, "/gnu/store/trash/272ibwb38i0kcbcl3n9v0ka1rsmd1104-guix-web-site/de/packages/rust-syntex-0.58.1", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 13 <0.000011>
fstat(13, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 <0.000007>
getdents64(13, 0x397a510 /* 3 entries */, 32768) = 80 <0.005059>
getdents64(13, 0x397a510 /* 0 entries */, 32768) = 0 <0.000007>
close(13)                               = 0 <0.000008>
statx(AT_FDCWD, "/gnu/store/trash/272ibwb38i0kcbcl3n9v0ka1rsmd1104-guix-web-site/de/packages/rust-syntex-0.58.1/index.html", AT_STATX_SYNC_AS_STAT|AT_SYMLINK_NOFOLLOW, STATX_MODE|STATX_NLINK|STATX_SIZE, {stx_mask=STATX_BASIC_STATS|0x1000, stx_attributes=0, stx_mode=S_IFREG|0444, stx_size=10265, ...}) = 0 <0.000023>
unlink("/gnu/store/trash/272ibwb38i0kcbcl3n9v0ka1rsmd1104-guix-web-site/de/packages/rust-syntex-0.58.1/index.html") = 0 <0.000013>
rmdir("/gnu/store/trash/272ibwb38i0kcbcl3n9v0ka1rsmd1104-guix-web-site/de/packages/rust-syntex-0.58.1") = 0 <0.000028>
statx(AT_FDCWD, "/gnu/store/trash/272ibwb38i0kcbcl3n9v0ka1rsmd1104-guix-web-site/de/packages/lofreq-2.1.5", AT_STATX_
--8<---------------cut here---------------end--------------->8---

Several syscalls are involved to clean the trash directory: chmod,
openat, statx, unlink and rmdir. This does not seem particularly wrong.

What is problematic though is that in 20 hours, the free space has
bumped from 9.6T to 9.7T in the store partition. As the GC lock is
preventing most of Berlin services from running, almost all the machine
IO is dedicated to removing this directory, as shown by iotop.

I'm not sure to understand why this removing process is so long, but if
someone has an idea, I'm all ears. In the meantime, I plan to let the GC
run and keep monitoring it.

Thanks,

Mathieu




Message sent to bug-guix@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: bug#51787: GC takes more than 9 hours on berlin
Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Fri, 10 Dec 2021 10:23:01 +0000
Resent-Message-ID: <handler.51787.B51787.163913173031903 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 51787
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: Mathieu Othacehe <othacehe@HIDDEN>
Cc: 51787 <at> debbugs.gnu.org
Received: via spool by 51787-submit <at> debbugs.gnu.org id=B51787.163913173031903
          (code B ref 51787); Fri, 10 Dec 2021 10:23:01 +0000
Received: (at 51787) by debbugs.gnu.org; 10 Dec 2021 10:22:10 +0000
Received: from localhost ([127.0.0.1]:45586 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mvd2Y-0008IV-Gh
	for submit <at> debbugs.gnu.org; Fri, 10 Dec 2021 05:22:10 -0500
Received: from eggs.gnu.org ([209.51.188.92]:51934)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1mvd2U-0008IF-Ob
 for 51787 <at> debbugs.gnu.org; Fri, 10 Dec 2021 05:22:09 -0500
Received: from [2001:470:142:3::e] (port=51218 helo=fencepost.gnu.org)
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ludo@HIDDEN>) id 1mvd2P-00035U-I0
 for 51787 <at> debbugs.gnu.org; Fri, 10 Dec 2021 05:22:01 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To:
 From; bh=UZ3yiU183nsbMZaRxe9oH4t9kFvV2IAIZ7Nq5ZVFaL8=; b=a6++0wwYD43rCLU+NuUU
 r31nSPIQezPisvDT/zHJK1D07lZ2mOOKjG8NFF8jOcV7cF9aLmsonICJwwP+NTEA/iL3G1yfkCaEH
 w+btjGT2sAOKqmkVZyCvtqDqXfTnYRI6ZjHOpkiOBP4Z6n+v1BeaZyL4LymH2BwcwtfIhQaPha7+f
 38Vawd3jAXBIZZt4yc6aeKUSQuvSteyx9gvEtBXu24YgssXqzmcCAllqul11HCsgdao7LTvtSLrVb
 CIDI3U72X2a+DmcgOtdF2dCOlt6hok1WnWxnvVwwbq9lh0BvLX06UdZ6D6/9M2joOBt3EUVKVj6pv
 C8ND7Gb/gGgmAA==;
Received: from 91-160-117-201.subs.proxad.net ([91.160.117.201]:60560
 helo=ribbon)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ludo@HIDDEN>)
 id 1mvd2P-0003dT-B4; Fri, 10 Dec 2021 05:22:01 -0500
From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
References: <87o86pegr3.fsf@HIDDEN> <87zgpuhig7.fsf@HIDDEN>
 <87wnkv15k3.fsf@HIDDEN> <87zgpp971n.fsf@HIDDEN>
 <87pmqet419.fsf@HIDDEN> <87czm57zao.fsf@HIDDEN>
X-URL: http://www.fdn.fr/~lcourtes/
X-Revolutionary-Date: 20 Frimaire an 230 de la =?UTF-8?Q?R=C3=A9volution?=
X-PGP-Key-ID: 0x090B11993D9AEBB5
X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc
X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4  0CFB 090B 1199 3D9A EBB5
X-OS: x86_64-pc-linux-gnu
Date: Fri, 10 Dec 2021 11:21:58 +0100
In-Reply-To: <87czm57zao.fsf@HIDDEN> (Mathieu Othacehe's message of "Fri, 10
 Dec 2021 07:24:15 +0100")
Message-ID: <87czm4bvzt.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Hi,

Mathieu Othacehe <othacehe@HIDDEN> skribis:

> What is problematic though is that in 20 hours, the free space has
> bumped from 9.6T to 9.7T in the store partition. As the GC lock is
> preventing most of Berlin services from running, almost all the machine
> IO is dedicated to removing this directory, as shown by iotop.
>
> I'm not sure to understand why this removing process is so long, but if
> someone has an idea, I'm all ears. In the meantime, I plan to let the GC
> run and keep monitoring it.

This is the first time GC runs since we=E2=80=99ve increased the threshold =
from
10=C2=A0TiB to 15=C2=A0TiB free.  There are at least 5 more TiBs to delete =
than
usual, so it=E2=80=99s expected to take more time.

Still, I=E2=80=99m surprised a mere =E2=80=98rm -rf=E2=80=99 can take this =
long.  =E2=80=98strace -T=E2=80=99 on
the child guix-daemon process doesn=E2=80=99t reveal anything obviously wro=
ng,
pause times or similar.

Ludo=E2=80=99.




Message sent to bug-guix@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: bug#51787: GC takes more than 9 hours on berlin
Resent-From: Mathieu Othacehe <othacehe@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Fri, 10 Dec 2021 17:12:02 +0000
Resent-Message-ID: <handler.51787.B51787.163915629923203 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 51787
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Cc: 51787 <at> debbugs.gnu.org
Received: via spool by 51787-submit <at> debbugs.gnu.org id=B51787.163915629923203
          (code B ref 51787); Fri, 10 Dec 2021 17:12:02 +0000
Received: (at 51787) by debbugs.gnu.org; 10 Dec 2021 17:11:39 +0000
Received: from localhost ([127.0.0.1]:47818 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mvjQp-000627-10
	for submit <at> debbugs.gnu.org; Fri, 10 Dec 2021 12:11:39 -0500
Received: from eggs.gnu.org ([209.51.188.92]:41166)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <othacehe@HIDDEN>) id 1mvjQn-00061s-54
 for 51787 <at> debbugs.gnu.org; Fri, 10 Dec 2021 12:11:38 -0500
Received: from [2001:470:142:3::e] (port=36092 helo=fencepost.gnu.org)
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <othacehe@HIDDEN>)
 id 1mvjQh-0000fe-OT; Fri, 10 Dec 2021 12:11:31 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To:
 From; bh=9IkbFlV8mwax15N9cFcQRANKiSxTrz6R5ceiE2+33ao=; b=rn1rrLGx1Ug5myi24omF
 4yeEOlMS+ukzBMfcd1elW/8fkaCxpcDIqr+6oekr3OuHmzG2kUGpDqyM4umcQoTSAdqLyCcd2uTVu
 iA/fVIx2eiebSZ/A78EPupMa22+z8aKXxiR3xRwmU4/zfOwXfshmrgAr1KZIbJWWnMFKLyEtqnWd3
 YYKjOAg59Qtp4Bc3zcWEwaZKwgV6kBvaVCiXwuS38+b6xq8mDJREa0+/2r+feE5H9YYdarHqJcOgj
 ErGbeWJPCb5ngpkO147rddGZCHPeZMmvWuMl6vQkpbzB8k86+k8C8ufXaPqrtBY9f4Khmm/QiniBi
 uN8+tYOQmv3YTQ==;
Received: from [2a01:e0a:19b:d9a0:45b5:a14a:5c75:5737] (port=53954 helo=meije)
 by fencepost.gnu.org with esmtpsa
 (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1)
 (envelope-from <othacehe@HIDDEN>)
 id 1mvjQh-000857-Gn; Fri, 10 Dec 2021 12:11:31 -0500
From: Mathieu Othacehe <othacehe@HIDDEN>
References: <87o86pegr3.fsf@HIDDEN> <87zgpuhig7.fsf@HIDDEN>
 <87wnkv15k3.fsf@HIDDEN> <87zgpp971n.fsf@HIDDEN>
 <87pmqet419.fsf@HIDDEN> <87czm57zao.fsf@HIDDEN>
 <87czm4bvzt.fsf@HIDDEN>
Date: Fri, 10 Dec 2021 18:11:29 +0100
In-Reply-To: <87czm4bvzt.fsf@HIDDEN> ("Ludovic =?UTF-8?Q?Court=C3=A8s?="'s message of "Fri, 10 Dec 2021 11:21:58 +0100")
Message-ID: <877dcccrlq.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)


Hey,

> Still, I=E2=80=99m surprised a mere =E2=80=98rm -rf=E2=80=99 can take thi=
s long.  =E2=80=98strace -T=E2=80=99 on
> the child guix-daemon process doesn=E2=80=99t reveal anything obviously w=
rong,
> pause times or similar.

I noticed that the process isn't around anymore. Did anyone killed it,
or maybe it just crashed? Also noticed some trash removing commands were
running. Regardless of the root cause of the problem, getting rid of the
trash directory before the next evaluation seems like a good idea.

Thanks,

Mathieu




Message sent to bug-guix@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: bug#51787: GC takes more than 9 hours on berlin
Resent-From: Mathieu Othacehe <othacehe@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Fri, 10 Dec 2021 17:12:02 +0000
Resent-Message-ID: <handler.51787.B51787.163915630623222 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 51787
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Cc: 51787 <at> debbugs.gnu.org
Received: via spool by 51787-submit <at> debbugs.gnu.org id=B51787.163915630623222
          (code B ref 51787); Fri, 10 Dec 2021 17:12:02 +0000
Received: (at 51787) by debbugs.gnu.org; 10 Dec 2021 17:11:46 +0000
Received: from localhost ([127.0.0.1]:47822 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mvjQw-00062U-8P
	for submit <at> debbugs.gnu.org; Fri, 10 Dec 2021 12:11:46 -0500
Received: from eggs.gnu.org ([209.51.188.92]:41198)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <othacehe@HIDDEN>) id 1mvjQt-000626-Ku
 for 51787 <at> debbugs.gnu.org; Fri, 10 Dec 2021 12:11:44 -0500
Received: from [2001:470:142:3::e] (port=36100 helo=fencepost.gnu.org)
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <othacehe@HIDDEN>)
 id 1mvjQk-0000vS-T5; Fri, 10 Dec 2021 12:11:38 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=MIME-Version:Date:References:In-Reply-To:Subject:To:
 From; bh=9IkbFlV8mwax15N9cFcQRANKiSxTrz6R5ceiE2+33ao=; b=kD28yrLbWr5GDz9OUcTx
 1mn8y/akBuV6bbpQ+hLv68ryZp927R5a9nx5WdQ8ALNOsNaVHsTQ1YLFTjkWn8auk+Xy5pvi98CE8
 ImGByIFrYatN7g4rCmTYGAK2D5FcKPPIBhw+Oz8DJI9x8EmAC1fm1WjzRSOxxjDQUu0PO9t8bZ5O0
 UVSHllhQj7pKLYKhECrJTXAAdVeCLLX7ElSoH3P3FKmjXV5dCB7NIuDW8rJVw2FzY8eskhZXtnUGg
 dSrnvexFCPaWRafoS8mpvbfeFvIkPqiwnoMhjGTBgrWMEA6CUFoeLcu5WrsH/fUUD9QBNAQzruhku
 vCIvvNRlILSn2A==;
Received: from [2a01:e0a:19b:d9a0:45b5:a14a:5c75:5737] (port=53956 helo=meije)
 by fencepost.gnu.org with esmtpsa
 (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1)
 (envelope-from <othacehe@HIDDEN>)
 id 1mvjQk-00086L-QY; Fri, 10 Dec 2021 12:11:35 -0500
From: Mathieu Othacehe <othacehe@HIDDEN>
In-Reply-To: <87czm4bvzt.fsf@HIDDEN> ("Ludovic =?UTF-8?Q?Court=C3=A8s?="'s message of "Fri, 10 Dec 2021 11:21:58 +0100")
References: <87o86pegr3.fsf@HIDDEN> <87zgpuhig7.fsf@HIDDEN>
 <87wnkv15k3.fsf@HIDDEN> <87zgpp971n.fsf@HIDDEN>
 <87pmqet419.fsf@HIDDEN> <87czm57zao.fsf@HIDDEN>
 <87czm4bvzt.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux)
Date: Fri, 10 Dec 2021 18:11:32 +0100
Message-ID: <875yrwcrln.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)


Hey,

> Still, I=E2=80=99m surprised a mere =E2=80=98rm -rf=E2=80=99 can take thi=
s long.  =E2=80=98strace -T=E2=80=99 on
> the child guix-daemon process doesn=E2=80=99t reveal anything obviously w=
rong,
> pause times or similar.

I noticed that the process isn't around anymore. Did anyone killed it,
or maybe it just crashed? Also noticed some trash removing commands were
running. Regardless of the root cause of the problem, getting rid of the
trash directory before the next evaluation seems like a good idea.

Thanks,

Mathieu




Message sent to bug-guix@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: bug#51787: GC takes more than 9 hours on berlin
References: <87o86pegr3.fsf@HIDDEN>
In-Reply-To: <87o86pegr3.fsf@HIDDEN>
Resent-From: Ricardo Wurmus <rekado@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Fri, 10 Dec 2021 19:39:01 +0000
Resent-Message-ID: <handler.51787.B51787.16391651366201 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 51787
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: 51787 <at> debbugs.gnu.org
Received: via spool by 51787-submit <at> debbugs.gnu.org id=B51787.16391651366201
          (code B ref 51787); Fri, 10 Dec 2021 19:39:01 +0000
Received: (at 51787) by debbugs.gnu.org; 10 Dec 2021 19:38:56 +0000
Received: from localhost ([127.0.0.1]:47936 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mvljM-0001bw-21
	for submit <at> debbugs.gnu.org; Fri, 10 Dec 2021 14:38:56 -0500
Received: from sender3-of-o51.zoho.com ([136.143.184.51]:21785)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <rekado@HIDDEN>) id 1mvljG-0001bk-Ex
 for 51787 <at> debbugs.gnu.org; Fri, 10 Dec 2021 14:38:54 -0500
ARC-Seal: i=1; a=rsa-sha256; t=1639165127; cv=none; 
 d=zohomail.com; s=zohoarc; 
 b=Wi4lW8kBo6baRo6Zqeh48dfjQpPB3q7deffYdEywffgVnTtfKwGsNgE+gbzLvzOe7yFftGh8IcGbCf8m63pqeI/XG1PYwlxU64wtvEB5EUsUgbV9WK+dQfOitgydJUnTGx1/MxnttBRoii/5nVSlTsM5TOSWYayLJ9qVIHGEyxg=
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com;
 s=zohoarc; 
 t=1639165127; h=Content-Type:Date:From:MIME-Version:Message-ID:Subject:To; 
 bh=RX/BQH64PFJE4vOlQxDpH2fza9YjrrVBGZ8oc8f9/CI=; 
 b=X+nSEJJYyszXyPjLMB/WcDOSzanDQwuuE+wumOzlw8vgEXvv9lK0uz0ate/VsYDvMgKsZzqhdQVFFT9jMf8fhIh4Rc4Jg9A9THgRsTwO4e2vv0mrdftkQVMWasiu2VEvoFMfb2DSnRuS3jaaSKVJrkHiPs4c/bE5YVHcNuT+UHk=
ARC-Authentication-Results: i=1; mx.zohomail.com;
 dkim=pass  header.i=elephly.net;
 spf=pass  smtp.mailfrom=rekado@HIDDEN;
 dmarc=pass header.from=<rekado@HIDDEN>
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1639165127; 
 s=zoho; d=elephly.net; i=rekado@HIDDEN;
 h=From:To:Subject:Date:Message-ID:MIME-Version:Content-Type;
 bh=RX/BQH64PFJE4vOlQxDpH2fza9YjrrVBGZ8oc8f9/CI=;
 b=RAbMMHRX96gy3StJR7wJgz0zQ6xSlRBq/NECIjW3CMvbSUOKdzaL/pYzrDIP1vQ0
 RM9wWrw1tb8guh7lYszW/0KkZElalOhsB3/uM+0MP4AW8ocAWPTgK+KNsmiM4YxeJ5O
 2x/dV7aDL84E/4cI01UT6Hp9rzb6CL/8a7E+gN2E=
Received: from localhost (p54ad4e72.dip0.t-ipconnect.de [84.173.78.114]) by
 mx.zohomail.com with SMTPS id 1639165122215259.0267296351809;
 Fri, 10 Dec 2021 11:38:42 -0800 (PST)
User-agent: mu4e 1.6.6; emacs 27.2
From: Ricardo Wurmus <rekado@HIDDEN>
Date: Fri, 10 Dec 2021 19:38:11 +0000
X-URL: https://elephly.net
X-PGP-Key: https://elephly.net/rekado.pubkey
X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
Message-ID: <8735n0s11d.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain
X-ZohoMailClient: External
X-Spam-Score: -0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

> I noticed that the process isn't around anymore. Did anyone killed
> it,or maybe it just crashed?

I have not touched it.

-- 
Ricardo




Message sent to bug-guix@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: bug#51787: GC takes more than 9 hours on berlin
Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Fri, 10 Dec 2021 21:56:01 +0000
Resent-Message-ID: <handler.51787.B51787.163917330118923 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 51787
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: Mathieu Othacehe <othacehe@HIDDEN>
Cc: 51787 <at> debbugs.gnu.org
Received: via spool by 51787-submit <at> debbugs.gnu.org id=B51787.163917330118923
          (code B ref 51787); Fri, 10 Dec 2021 21:56:01 +0000
Received: (at 51787) by debbugs.gnu.org; 10 Dec 2021 21:55:01 +0000
Received: from localhost ([127.0.0.1]:48062 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mvnr3-0004v5-Bw
	for submit <at> debbugs.gnu.org; Fri, 10 Dec 2021 16:55:01 -0500
Received: from eggs.gnu.org ([209.51.188.92]:49242)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1mvnr2-0004uk-4D
 for 51787 <at> debbugs.gnu.org; Fri, 10 Dec 2021 16:55:00 -0500
Received: from [2001:470:142:3::e] (port=50382 helo=fencepost.gnu.org)
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ludo@HIDDEN>) id 1mvnqw-00029d-RJ
 for 51787 <at> debbugs.gnu.org; Fri, 10 Dec 2021 16:54:54 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To:
 From; bh=666QD3BrVVkj+diIOR1o7PctKtjSesXLghNLsDRB0+s=; b=DLBEXl7nzWOfryF81HKp
 6tvQuySdMrFuTpu9vkAhMLVGlRbBQgBftx8dufMqJUZ0OXGMN94F12BEVFftTzc32+N97dRWaIUM9
 jXd2eUDtUE9G1oaOi9R5JMA6uATq7zXasVHiVcdpxCClznlY4nDPrM6ztVqyuLoY3BiKCOAhUmp7u
 eJLt61yyB7JFHFe7P0CpTLlfqa0PSDzTxoB4BjjVnCIT4HWnqn3p82JWwIQjfyIjIZqxKsHijtQGL
 VMH2Y0rg4lGoIS3P/U5DgO7UWrdU00S88TBcHjaqpyJr1X3uedohIvquCStrc2yT9XQqGLEwya6Tt
 GkLjo1bgOzTLzw==;
Received: from 91-160-117-201.subs.proxad.net ([91.160.117.201]:63117
 helo=ribbon)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ludo@HIDDEN>)
 id 1mvnqw-0001yX-FA; Fri, 10 Dec 2021 16:54:54 -0500
From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
References: <87o86pegr3.fsf@HIDDEN> <87zgpuhig7.fsf@HIDDEN>
 <87wnkv15k3.fsf@HIDDEN> <87zgpp971n.fsf@HIDDEN>
 <87pmqet419.fsf@HIDDEN> <87czm57zao.fsf@HIDDEN>
 <87czm4bvzt.fsf@HIDDEN> <877dcccrlq.fsf@HIDDEN>
X-URL: http://www.fdn.fr/~lcourtes/
X-Revolutionary-Date: 20 Frimaire an 230 de la =?UTF-8?Q?R=C3=A9volution?=
X-PGP-Key-ID: 0x090B11993D9AEBB5
X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc
X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4  0CFB 090B 1199 3D9A EBB5
X-OS: x86_64-pc-linux-gnu
Date: Fri, 10 Dec 2021 22:54:51 +0100
In-Reply-To: <877dcccrlq.fsf@HIDDEN> (Mathieu Othacehe's message of "Fri, 10
 Dec 2021 18:11:29 +0100")
Message-ID: <87a6h86s7o.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Mathieu Othacehe <othacehe@HIDDEN> skribis:

>> Still, I=E2=80=99m surprised a mere =E2=80=98rm -rf=E2=80=99 can take th=
is long.  =E2=80=98strace -T=E2=80=99 on
>> the child guix-daemon process doesn=E2=80=99t reveal anything obviously =
wrong,
>> pause times or similar.
>
> I noticed that the process isn't around anymore. Did anyone killed it,
> or maybe it just crashed? Also noticed some trash removing commands were
> running. Regardless of the root cause of the problem, getting rid of the
> trash directory before the next evaluation seems like a good idea.

Yeah I don=E2=80=99t know what happened to the GC process but it disappeared
earlier today.  Then I started things like this:

  unshare -m sh -c 'mount --bind -o remount,rw /gnu/store; rm -rf --one-fil=
e-system /gnu/store/trash/[abc]*'

which is equally slow (perhaps slightly slower: it seems to do more
syscalls per file to remove than the guix-daemon code).

Anyway, I=E2=80=99ll let it run hoping it=E2=80=99ll be done by the next GC.

Ludo=E2=80=99.




Message sent to bug-guix@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: bug#51787: GC takes more than 9 hours on berlin
Resent-From: Mathieu Othacehe <othacehe@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Sat, 11 Dec 2021 09:43:02 +0000
Resent-Message-ID: <handler.51787.B51787.163921575922005 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 51787
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Cc: 51787 <at> debbugs.gnu.org
Received: via spool by 51787-submit <at> debbugs.gnu.org id=B51787.163921575922005
          (code B ref 51787); Sat, 11 Dec 2021 09:43:02 +0000
Received: (at 51787) by debbugs.gnu.org; 11 Dec 2021 09:42:39 +0000
Received: from localhost ([127.0.0.1]:48552 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mvytq-0005ir-VP
	for submit <at> debbugs.gnu.org; Sat, 11 Dec 2021 04:42:39 -0500
Received: from eggs.gnu.org ([209.51.188.92]:55402)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <othacehe@HIDDEN>) id 1mvytp-0005ie-1q
 for 51787 <at> debbugs.gnu.org; Sat, 11 Dec 2021 04:42:37 -0500
Received: from [2001:470:142:3::e] (port=54242 helo=fencepost.gnu.org)
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <othacehe@HIDDEN>)
 id 1mvytj-00071m-Py; Sat, 11 Dec 2021 04:42:31 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To:
 From; bh=LqKATR/Ne6Ec6COKq/S3LC1bNc+M/07AM+bhyC8XLxE=; b=i4cKFok48eFCyInyqHHd
 zlD/KIHVWJlqsAWN1AWtJtmgGAALZvEwnxEAAq2S+Wz+x03VXmmjJMcYEDBS9U7do4ifq+jn/T7b+
 ykYTWcf609VNAFMr6XlfhTr7joi6wx7emP8FKoFvmcGJP6/m07a5zAQPZG2wiVYfyzklLpr/UA3Pn
 +WjARzHQhkWXKk3uPbCet26GWukk2bnxmxqyol9gZBK0GqO1rZyuXns6p5uhS58nGq9g9MmV8Qj7X
 cn76xvf9SA1jnu1//PCZvTtOgLOB50/iG19yoCqCaXhEYg2iMNXbQ8tiF1ZeerVi8FrjmMXuED8c2
 JgE6eLkqHITI6g==;
Received: from [2a01:e0a:19b:d9a0:2ddb:d3d2:32e8:d31a] (port=33136 helo=meije)
 by fencepost.gnu.org with esmtpsa
 (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1)
 (envelope-from <othacehe@HIDDEN>)
 id 1mvytj-0002S3-Kg; Sat, 11 Dec 2021 04:42:31 -0500
From: Mathieu Othacehe <othacehe@HIDDEN>
References: <87o86pegr3.fsf@HIDDEN> <87zgpuhig7.fsf@HIDDEN>
 <87wnkv15k3.fsf@HIDDEN> <87zgpp971n.fsf@HIDDEN>
 <87pmqet419.fsf@HIDDEN> <87czm57zao.fsf@HIDDEN>
 <87czm4bvzt.fsf@HIDDEN> <877dcccrlq.fsf@HIDDEN>
 <87a6h86s7o.fsf@HIDDEN>
Date: Sat, 11 Dec 2021 10:42:30 +0100
In-Reply-To: <87a6h86s7o.fsf@HIDDEN> ("Ludovic =?UTF-8?Q?Court=C3=A8s?="'s message of "Fri, 10 Dec 2021 22:54:51 +0100")
Message-ID: <87bl1nxyt5.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)


Hey,

> Yeah I don=E2=80=99t know what happened to the GC process but it disappea=
red
> earlier today.  Then I started things like this:
>
>   unshare -m sh -c 'mount --bind -o remount,rw /gnu/store; rm -rf --one-f=
ile-system /gnu/store/trash/[abc]*'
>
> which is equally slow (perhaps slightly slower: it seems to do more
> syscalls per file to remove than the guix-daemon code).
>
> Anyway, I=E2=80=99ll let it run hoping it=E2=80=99ll be done by the next =
GC.

OK, thanks. Looks it just finished removing the trash directory
content. I started a GC process from my session to monitor it closely.

Mathieu




Message sent to bug-guix@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: bug#51787: GC takes more than 9 hours on berlin
Resent-From: Mathieu Othacehe <othacehe@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Sun, 12 Dec 2021 17:10:01 +0000
Resent-Message-ID: <handler.51787.B51787.16393289926642 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 51787
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: 51787 <at> debbugs.gnu.org
Received: via spool by 51787-submit <at> debbugs.gnu.org id=B51787.16393289926642
          (code B ref 51787); Sun, 12 Dec 2021 17:10:01 +0000
Received: (at 51787) by debbugs.gnu.org; 12 Dec 2021 17:09:52 +0000
Received: from localhost ([127.0.0.1]:53012 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mwSMB-0001j4-SF
	for submit <at> debbugs.gnu.org; Sun, 12 Dec 2021 12:09:52 -0500
Received: from eggs.gnu.org ([209.51.188.92]:41766)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <othacehe@HIDDEN>) id 1mwSM9-0001ir-Vg
 for 51787 <at> debbugs.gnu.org; Sun, 12 Dec 2021 12:09:50 -0500
Received: from [2001:470:142:3::e] (port=35650 helo=fencepost.gnu.org)
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <othacehe@HIDDEN>) id 1mwSM4-0002fl-QZ
 for 51787 <at> debbugs.gnu.org; Sun, 12 Dec 2021 12:09:44 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To:
 From; bh=q1IqzqZRjMyy8SGAcWVTq6J9mYkAHEDd3IDNK/2jRjk=; b=GDwFPaIk/oIPlwBlNcO8
 7PLNJ9gWKBo6HemDuLU8JisJqm8+/+yZFt8j/8I0xby61NnknQ5/MaPBDL+zgS+9Be6j9Wa7m8dX9
 +zp/nh64vNNuue3nmBYVaB6sWdijMWku1rlELItZrpN+j6BqApSn1XpIhOcvOCRx8S7ufubQxXIn0
 p30aehLKCn9a8pBuhuDmGOp5li1Ja2Ajq1K5GQ0iLZJ713SplAkq69w34Mz8oodamghqEjbq9ybRe
 dqfkDE6gCNAIT/d4XU2ftwnHM4bBjk311ceTPkeQpOBAXDGzb8Avm0MVNcb3qFR11e4JwwRE1UB94
 JlIKj2wPgJc3Lw==;
Received: from [2a01:e0a:19b:d9a0:45b5:a14a:5c75:5737] (port=53994 helo=meije)
 by fencepost.gnu.org with esmtpsa
 (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1)
 (envelope-from <othacehe@HIDDEN>) id 1mwSM4-00008D-NM
 for 51787 <at> debbugs.gnu.org; Sun, 12 Dec 2021 12:09:44 -0500
From: Mathieu Othacehe <othacehe@HIDDEN>
References: <87o86pegr3.fsf@HIDDEN> <87zgpuhig7.fsf@HIDDEN>
 <87wnkv15k3.fsf@HIDDEN> <87zgpp971n.fsf@HIDDEN>
 <87pmqet419.fsf@HIDDEN> <87czm57zao.fsf@HIDDEN>
 <87czm4bvzt.fsf@HIDDEN> <877dcccrlq.fsf@HIDDEN>
 <87a6h86s7o.fsf@HIDDEN> <87bl1nxyt5.fsf@HIDDEN>
Date: Sun, 12 Dec 2021 18:09:42 +0100
In-Reply-To: <87bl1nxyt5.fsf@HIDDEN> (Mathieu Othacehe's message of "Sat, 11
 Dec 2021 10:42:30 +0100")
Message-ID: <87o85lhhrd.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)


Hey,

> OK, thanks. Looks it just finished removing the trash directory
> content. I started a GC process from my session to monitor it closely.

Daily GC recap:

* The GC process I started yesterday, did collect 5.5TiB in
  approximately 24 hours, that are now in the /gnu/store/trash
  directory.
  
* The /gnu/store/trash directory contains 288910 entries. If those items
  are removed at the same rate than on the previous days, it will take
  days/months to delete them all.

* I noticed that the upstream Nix GC process can now operate without
  locking. I think it shouldn't be too hard to port it to our fork or
  maybe rewrite the process in Guile while we are at it.

  That will not fix the slow hard-drives issues though.

Thanks,

Mathieu




Message sent to bug-guix@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: bug#51787: GC takes more than 9 hours on berlin
Resent-From: Christian =?UTF-8?Q?Th=C3=A4ter?= <ct.guix@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Mon, 13 Dec 2021 16:16:01 +0000
Resent-Message-ID: <handler.51787.B.163941214824597 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 51787
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: 51787 <at> debbugs.gnu.org
X-Debbugs-Original-To: bug-guix@HIDDEN
Received: via spool by submit <at> debbugs.gnu.org id=B.163941214824597
          (code B ref -1); Mon, 13 Dec 2021 16:16:01 +0000
Received: (at submit) by debbugs.gnu.org; 13 Dec 2021 16:15:48 +0000
Received: from localhost ([127.0.0.1]:56625 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mwnzP-0006Of-Oy
	for submit <at> debbugs.gnu.org; Mon, 13 Dec 2021 11:15:47 -0500
Received: from lists.gnu.org ([209.51.188.17]:38810)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ct.guix@HIDDEN>) id 1mwnxR-0006Kr-35
 for submit <at> debbugs.gnu.org; Mon, 13 Dec 2021 11:13:48 -0500
Received: from eggs.gnu.org ([209.51.188.92]:44248)
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ct.guix@HIDDEN>)
 id 1mwnxQ-0003ZB-Uv
 for bug-guix@HIDDEN; Mon, 13 Dec 2021 11:13:44 -0500
Received: from mail.pipapo.org ([78.47.240.153]:59240)
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ct.guix@HIDDEN>)
 id 1mwnxO-0007cn-IL
 for bug-guix@HIDDEN; Mon, 13 Dec 2021 11:13:44 -0500
Received: from hermes.pipapo.org (imap.pipapo.org [10.10.40.10])
 by mail.pipapo.org (Postfix) with ESMTPS id 2D7945E45F
 for <bug-guix@HIDDEN>; Mon, 13 Dec 2021 16:13:37 +0000 (UTC)
Received: from wolke.pipapo.org (wolke.pipapo.org [10.100.20.10])
 by hermes.pipapo.org (Postfix) with ESMTPSA id 0FD199E02EAA
 for <bug-guix@HIDDEN>; Mon, 13 Dec 2021 16:13:42 +0000 (UTC)
Date: Mon, 13 Dec 2021 17:13:33 +0100
From: Christian =?UTF-8?Q?Th=C3=A4ter?= <ct.guix@HIDDEN>
Message-ID: <20211213171333.04105a3b@HIDDEN>
In-Reply-To: <87o85lhhrd.fsf@HIDDEN>
References: <87o86pegr3.fsf@HIDDEN> <87zgpuhig7.fsf@HIDDEN>
 <87wnkv15k3.fsf@HIDDEN> <87zgpp971n.fsf@HIDDEN>
 <87pmqet419.fsf@HIDDEN> <87czm57zao.fsf@HIDDEN>
 <87czm4bvzt.fsf@HIDDEN> <877dcccrlq.fsf@HIDDEN>
 <87a6h86s7o.fsf@HIDDEN> <87bl1nxyt5.fsf@HIDDEN>
 <87o85lhhrd.fsf@HIDDEN>
X-Mailer: Claws Mail 3.17.8 (GTK+ 2.24.33; x86_64-pc-linux-gnu)
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Received-SPF: none client-ip=78.47.240.153; envelope-from=ct.guix@HIDDEN;
 helo=mail.pipapo.org
X-Spam_score_int: -18
X-Spam_score: -1.9
X-Spam_bar: -
X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001,
 SPF_NONE=0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-Spam-Score: -2.3 (--)
X-Mailman-Approved-At: Mon, 13 Dec 2021 11:15:47 -0500
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

On 2021-12-12 18:09, Mathieu Othacehe wrote:

>Hey,
>
>> OK, thanks. Looks it just finished removing the trash directory
>> content. I started a GC process from my session to monitor it
>> closely.  
>
>Daily GC recap:
>
>* The GC process I started yesterday, did collect 5.5TiB in
>  approximately 24 hours, that are now in the /gnu/store/trash
>  directory.
>  
>* The /gnu/store/trash directory contains 288910 entries. If those
>items
>  are removed at the same rate than on the previous days, it will take
>  days/months to delete them all.
>
>* I noticed that the upstream Nix GC process can now operate without
>  locking. I think it shouldn't be too hard to port it to our fork or
>  maybe rewrite the process in Guile while we are at it.
>
>  That will not fix the slow hard-drives issues though.

While discussing this issue on IRC I came up with some idea:

'rmrfd' a system daemon that deletes huge trees in the background where
'-rf' stands for --really --fast :)

Actually this is an use case that happens for on my backup system too.
With that idea I just started coding and ran some experiments. For me
this looks quite feasible now and I will continue next days on this
small project. Any feedback or help would be welcomed!

The initial ideas and experiments are at https://github.com/cehteh/rmrfd

Note that the important part is that it will put some efforts into
freeing as much space as possible at begin of the freeing process,
Unlike just 'rm -rf' where space may only freed really late when the
last link count of the data goes to zero.

	Cheers
		Christian




Message sent to bug-guix@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: bug#51787: GC takes more than 9 hours on berlin
Resent-From: Christian =?UTF-8?Q?Th=C3=A4ter?= <ct@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Tue, 14 Dec 2021 03:32:02 +0000
Resent-Message-ID: <handler.51787.B.16394526987773 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 51787
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: 51787 <at> debbugs.gnu.org
X-Debbugs-Original-To: bug-guix@HIDDEN
Received: via spool by submit <at> debbugs.gnu.org id=B.16394526987773
          (code B ref -1); Tue, 14 Dec 2021 03:32:02 +0000
Received: (at submit) by debbugs.gnu.org; 14 Dec 2021 03:31:38 +0000
Received: from localhost ([127.0.0.1]:57288 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mwyXS-00021I-Gt
	for submit <at> debbugs.gnu.org; Mon, 13 Dec 2021 22:31:38 -0500
Received: from lists.gnu.org ([209.51.188.17]:52670)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ct@HIDDEN>) id 1mwyXO-000217-Iw
 for submit <at> debbugs.gnu.org; Mon, 13 Dec 2021 22:31:37 -0500
Received: from eggs.gnu.org ([209.51.188.92]:47354)
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ct@HIDDEN>) id 1mwyXM-0004Q7-4k
 for bug-guix@HIDDEN; Mon, 13 Dec 2021 22:31:33 -0500
Received: from mail.pipapo.org ([78.47.240.153]:59736)
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ct@HIDDEN>) id 1mwyXJ-0002ct-5k
 for bug-guix@HIDDEN; Mon, 13 Dec 2021 22:31:31 -0500
Received: from hermes.pipapo.org (imap.pipapo.org [10.10.40.10])
 by mail.pipapo.org (Postfix) with ESMTPS id 0D7A75E7EB
 for <bug-guix@HIDDEN>; Tue, 14 Dec 2021 03:31:25 +0000 (UTC)
Received: from wolke.pipapo.org (wolke.pipapo.org [10.100.20.10])
 by hermes.pipapo.org (Postfix) with ESMTPSA id 099D09E01211
 for <bug-guix@HIDDEN>; Tue, 14 Dec 2021 03:31:30 +0000 (UTC)
Date: Tue, 14 Dec 2021 04:31:23 +0100
From: Christian =?UTF-8?Q?Th=C3=A4ter?= <ct@HIDDEN>
Message-ID: <20211214043123.7b041b8a@HIDDEN>
In-Reply-To: <87o85lhhrd.fsf@HIDDEN>
References: <87o86pegr3.fsf@HIDDEN> <87zgpuhig7.fsf@HIDDEN>
 <87wnkv15k3.fsf@HIDDEN> <87zgpp971n.fsf@HIDDEN>
 <87pmqet419.fsf@HIDDEN> <87czm57zao.fsf@HIDDEN>
 <87czm4bvzt.fsf@HIDDEN> <877dcccrlq.fsf@HIDDEN>
 <87a6h86s7o.fsf@HIDDEN> <87bl1nxyt5.fsf@HIDDEN>
 <87o85lhhrd.fsf@HIDDEN>
X-Mailer: Claws Mail 3.17.8 (GTK+ 2.24.33; x86_64-pc-linux-gnu)
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Received-SPF: none client-ip=78.47.240.153; envelope-from=ct@HIDDEN;
 helo=mail.pipapo.org
X-Spam_score_int: -18
X-Spam_score: -1.9
X-Spam_bar: -
X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001,
 SPF_NONE=0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

On 2021-12-12 18:09, Mathieu Othacehe wrote:

>Hey,
>
>> OK, thanks. Looks it just finished removing the trash directory
>> content. I started a GC process from my session to monitor it
>> closely.  
>
>Daily GC recap:
>
>* The GC process I started yesterday, did collect 5.5TiB in
>  approximately 24 hours, that are now in the /gnu/store/trash
>  directory.
>  
>* The /gnu/store/trash directory contains 288910 entries. If those
>items
>  are removed at the same rate than on the previous days, it will take
>  days/months to delete them all.

On another note: when 'guix gc' determines that objects are dead, are
they really dead then or can it be that they are 'locally' dead but in
case the store serves as substitutes/offload server some external
clients may still request dead objects?

In the either case would make sense to run the GC more frequently, but
for the later case a --min-age option to preserve objects that just
died recently could be helping. Further it may consider the atime of
objects for removal.

And finally while I had this Idea: You mount the
guix store with 'relatime' or 'nodiratme', if not that could explain
the slow deletion process as well!

	Christian



>
>* I noticed that the upstream Nix GC process can now operate without
>  locking. I think it shouldn't be too hard to port it to our fork or
>  maybe rewrite the process in Guile while we are at it.
>
>  That will not fix the slow hard-drives issues though.
>
>Thanks,
>
>Mathieu
>
>
>





Message sent to bug-guix@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: bug#51787: GC takes more than 9 hours on berlin
Resent-From: Mathieu Othacehe <othacehe@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Fri, 17 Dec 2021 10:56:01 +0000
Resent-Message-ID: <handler.51787.B51787.163973853721483 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 51787
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: 51787 <at> debbugs.gnu.org
Cc: rekado@HIDDEN
Received: via spool by 51787-submit <at> debbugs.gnu.org id=B51787.163973853721483
          (code B ref 51787); Fri, 17 Dec 2021 10:56:01 +0000
Received: (at 51787) by debbugs.gnu.org; 17 Dec 2021 10:55:37 +0000
Received: from localhost ([127.0.0.1]:38098 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1myAtl-0005aR-1y
	for submit <at> debbugs.gnu.org; Fri, 17 Dec 2021 05:55:37 -0500
Received: from eggs.gnu.org ([209.51.188.92]:44560)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <othacehe@HIDDEN>) id 1myAti-0005a9-M1
 for 51787 <at> debbugs.gnu.org; Fri, 17 Dec 2021 05:55:35 -0500
Received: from [2001:470:142:3::e] (port=48796 helo=fencepost.gnu.org)
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <othacehe@HIDDEN>)
 id 1myAtc-0006BE-R1; Fri, 17 Dec 2021 05:55:29 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To:
 From; bh=UdydBjdTNVdollJXyJmFEtHnDfBaGwfw9N6HukRsDCU=; b=BsiOlDUc7QDyUiyko19M
 pQt5wkMwttedYr13iTY248Ir+ZxheEnAx+Y6ar/NDIz7Pg3tj42W964D4xVTlrZaERfYsY7HSu4j2
 vs7Oi61WaoxL5eu1GmmulKzJvOpU3W2bD1EKHmeV0BKg/Mt1LQwocuD856OtTvzldOJwJVO3b3Z72
 pxakDwhhPb/ylaIugVBdv9oL6gf6xhmchRSSMOcBeSyueabofc8XcvMKj8/y+ilwLLBhaIKFXcPlP
 NLnEruNbKvOOS50p+QkaXu+A6Rze+sORRpbmJt7y+sNsznPQMjJy8dAE18Q24Zt72lQPrI3rzz9BR
 kBFjLaa2HBcEaA==;
Received: from [2a01:e0a:19b:d9a0:45b5:a14a:5c75:5737] (port=54094 helo=meije)
 by fencepost.gnu.org with esmtpsa
 (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1)
 (envelope-from <othacehe@HIDDEN>)
 id 1myAtc-0002yz-K5; Fri, 17 Dec 2021 05:55:28 -0500
From: Mathieu Othacehe <othacehe@HIDDEN>
References: <87o86pegr3.fsf@HIDDEN> <87zgpuhig7.fsf@HIDDEN>
 <87wnkv15k3.fsf@HIDDEN> <87zgpp971n.fsf@HIDDEN>
 <87pmqet419.fsf@HIDDEN> <87czm57zao.fsf@HIDDEN>
 <87czm4bvzt.fsf@HIDDEN> <877dcccrlq.fsf@HIDDEN>
 <87a6h86s7o.fsf@HIDDEN> <87bl1nxyt5.fsf@HIDDEN>
 <87o85lhhrd.fsf@HIDDEN>
Date: Fri, 17 Dec 2021 11:55:26 +0100
In-Reply-To: <87o85lhhrd.fsf@HIDDEN> (Mathieu Othacehe's message of "Sun, 12
 Dec 2021 18:09:42 +0100")
Message-ID: <8735mrec0x.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="=-=-="
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

--=-=-=
Content-Type: text/plain


Hey,

New day, new benchmark. Berlin has two hard drives, which are roughly
used this way:

/dev/sda -> / (916G)
/dev/sdb -> /gnu (37T)

I ran the fio benchmark tool on both of them. See the reports
attached, and the following summary:

--8<---------------cut here---------------start------------->8---
|       | sda       | sdb       |
|-------+-----------+-----------|
| read  | 1565KiB/s | 9695KiB/s |
| write | 523KiB/s  | 3240KiB/s |
--8<---------------cut here---------------end--------------->8---

I'm not sure how slow those figures are relatively to the hard drives
technologies. Ricardo, any idea about that?

My own NVME hard drive has 294MiB/s read and 98.1MiB/s write with the
same test in comparison.

I also tried to benchmark file removal this way, but this is not really
conclusive:

--8<---------------cut here---------------start------------->8---
# 3.8s on sdb
time for i in {0..100000}; do echo 'test' > "fs_test/test${i}.txt"; done

# 2.7s on sdb
time rf -rf fs_test
--8<---------------cut here---------------end--------------->8---

Thanks,

Mathieu


--=-=-=
Content-Type: application/octet-stream
Content-Disposition: attachment; filename=fio_sda
Content-Transfer-Encoding: base64

bWF0aGlldUBiZXJsaW4gfiQgZmlvIC0tcmFuZHJlcGVhdD0xIC0taW9lbmdpbmU9bGliYWlvIC0t
ZGlyZWN0PTEgLS1ndG9kX3JlZHVjZT0xIC0tbmFtZT10ZXN0IC0tZmlsZW5hbWU9dGVzdCAtLWJz
PTRrIC0taW9kZXB0aD02NCAtLXNpemU9NEcgLS1yZWFkd3JpdGU9cmFuZHJ3IC0tcndtaXhyZWFk
PTc1CnRlc3Q6IChnPTApOiBydz1yYW5kcncsIGJzPShSKSA0MDk2Qi00MDk2QiwgKFcpIDQwOTZC
LTQwOTZCLCAoVCkgNDA5NkItNDA5NkIsIGlvZW5naW5lPWxpYmFpbywgaW9kZXB0aD02NApmaW8t
My4yOApTdGFydGluZyAxIHByb2Nlc3MKdGVzdDogTGF5aW5nIG91dCBJTyBmaWxlICgxIGZpbGUg
LyA0MDk2TWlCKQpKb2JzOiAxIChmPTEpOiBbbSgxKV1bMTAwLjAlXVtyPTE0MzdLaUIvcyx3PTQy
OEtpQi9zXVtyPTM1OSx3PTEwNyBJT1BTXVtldGEgMDBtOjAwc10KdGVzdDogKGdyb3VwaWQ9MCwg
am9icz0xKTogZXJyPSAwOiBwaWQ9NzE5MjQ6IEZyaSBEZWMgMTcgMTE6MTc6MTIgMjAyMQogIHJl
YWQ6IElPUFM9MzkxLCBCVz0xNTY1S2lCL3MgKDE2MDJrQi9zKSgzMDcwTWlCLzIwMDg4Njhtc2Vj
KQogICBidyAoICBLaUIvcyk6IG1pbj0gIDMwNCwgbWF4PSAzMTkwLCBwZXI9MTAwLjAwJSwgYXZn
PTE1NjYuMzAsIHN0ZGV2PTMyMy4yMywgc2FtcGxlcz00MDE0CiAgIGlvcHMgICAgICAgIDogbWlu
PSAgIDc2LCBtYXg9ICA3OTcsIGF2Zz0zOTEuNDgsIHN0ZGV2PTgwLjg0LCBzYW1wbGVzPTQwMTQK
ICB3cml0ZTogSU9QUz0xMzAsIEJXPTUyM0tpQi9zICg1MzZrQi9zKSgxMDI2TWlCLzIwMDg4Njht
c2VjKTsgMCB6b25lIHJlc2V0cwogICBidyAoICBLaUIvcyk6IG1pbj0gIDEyMCwgbWF4PSAxMTQ2
LCBwZXI9MTAwLjAwJSwgYXZnPTUyMy40Mywgc3RkZXY9MTI0LjIxLCBzYW1wbGVzPTQwMTQKICAg
aW9wcyAgICAgICAgOiBtaW49ICAgMzAsIG1heD0gIDI4NiwgYXZnPTEzMC44MCwgc3RkZXY9MzEu
MDYsIHNhbXBsZXM9NDAxNAogIGNwdSAgICAgICAgICA6IHVzcj0wLjE2JSwgc3lzPTAuNTglLCBj
dHg9MTAzOTQyNSwgbWFqZj0wLCBtaW5mPTU4OQogIElPIGRlcHRocyAgICA6IDE9MC4xJSwgMj0w
LjElLCA0PTAuMSUsIDg9MC4xJSwgMTY9MC4xJSwgMzI9MC4xJSwgPj02ND0xMDAuMCUKICAgICBz
dWJtaXQgICAgOiAwPTAuMCUsIDQ9MTAwLjAlLCA4PTAuMCUsIDE2PTAuMCUsIDMyPTAuMCUsIDY0
PTAuMCUsID49NjQ9MC4wJQogICAgIGNvbXBsZXRlICA6IDA9MC4wJSwgND0xMDAuMCUsIDg9MC4w
JSwgMTY9MC4wJSwgMzI9MC4wJSwgNjQ9MC4xJSwgPj02ND0wLjAlCiAgICAgaXNzdWVkIHJ3dHM6
IHRvdGFsPTc4NTkyMCwyNjI2NTYsMCwwIHNob3J0PTAsMCwwLDAgZHJvcHBlZD0wLDAsMCwwCiAg
ICAgbGF0ZW5jeSAgIDogdGFyZ2V0PTAsIHdpbmRvdz0wLCBwZXJjZW50aWxlPTEwMC4wMCUsIGRl
cHRoPTY0CgpSdW4gc3RhdHVzIGdyb3VwIDAgKGFsbCBqb2JzKToKICAgUkVBRDogYnc9MTU2NUtp
Qi9zICgxNjAya0IvcyksIDE1NjVLaUIvcy0xNTY1S2lCL3MgKDE2MDJrQi9zLTE2MDJrQi9zKSwg
aW89MzA3ME1pQiAoMzIxOU1CKSwgcnVuPTIwMDg4NjgtMjAwODg2OG1zZWMKICBXUklURTogYnc9
NTIzS2lCL3MgKDUzNmtCL3MpLCA1MjNLaUIvcy01MjNLaUIvcyAoNTM2a0Ivcy01MzZrQi9zKSwg
aW89MTAyNk1pQiAoMTA3Nk1CKSwgcnVuPTIwMDg4NjgtMjAwODg2OG1zZWMKCkRpc2sgc3RhdHMg
KHJlYWQvd3JpdGUpOgogIHNkYTogaW9zPTgwNTczNC8zNjQxNzQsIG1lcmdlPTcxNDMvMzcxMDAs
IHRpY2tzPTEzNzU4Njk2OS80MTMwOCwgaW5fcXVldWU9MTM3NjI4Mjc3LCB1dGlsPTk5LjcxJQoK
bWF0aGlldUBiZXJsaW4gfiQgZmlvIC0tcmFuZHJlcGVhdD0xIC0taW9lbmdpbmU9bGliYWlvIC0t
ZGlyZWN0PTEgLS1ndG9kX3JlZHVjZT0xIC0tbmFtZT10ZXN0IC0tZmlsZW5hbWU9L3Zhci9jYWNo
ZS90ZXN0IC0tYnM9NGsgLS1pb2RlcHRoPTY0IC0tc2l6ZT00RyAtLXJlYWR3cml0ZT1yYW5kcncg
LS1yd21peHJlYWQ9NzUKdGVzdDogKGc9MCk6IHJ3PXJhbmRydywgYnM9KFIpIDQwOTZCLTQwOTZC
LCAoVykgNDA5NkItNDA5NkIsIChUKSA0MDk2Qi00MDk2QiwgaW9lbmdpbmU9bGliYWlvLCBpb2Rl
cHRoPTY0CmZpby0zLjI4ClN0YXJ0aW5nIDEgcHJvY2Vzcwp0ZXN0OiBMYXlpbmcgb3V0IElPIGZp
bGUgKDEgZmlsZSAvIDQwOTZNaUIpCmZpbzogcGlkPTAsIGVycj0xMy9maWxlOmZpbGVzZXR1cC5j
OjE3NCwgZnVuYz1vcGVuLCBlcnJvcj1QZXJtaXNzaW9uIGRlbmllZAo=
--=-=-=
Content-Type: application/octet-stream
Content-Disposition: attachment; filename=fio_sdb
Content-Transfer-Encoding: base64

bWF0aGlldUBiZXJsaW4gfiQgZmlvIC0tcmFuZHJlcGVhdD0xIC0taW9lbmdpbmU9bGliYWlvIC0t
ZGlyZWN0PTEgLS1ndG9kX3JlZHVjZT0xIC0tbmFtZT10ZXN0IC0tZmlsZW5hbWU9L3Zhci9jYWNo
ZS90ZXN0IC0tYnM9NGsgLS1pb2RlcHRoPTY0IC0tc2l6ZT00RyAtLXJlYWR3cml0ZT1yYW5kcncg
LS1yd21peHJlYWQ9NzUKdGVzdDogKGc9MCk6IHJ3PXJhbmRydywgYnM9KFIpIDQwOTZCLTQwOTZC
LCAoVykgNDA5NkItNDA5NkIsIChUKSA0MDk2Qi00MDk2QiwgaW9lbmdpbmU9bGliYWlvLCBpb2Rl
cHRoPTY0CmZpby0zLjI4ClN0YXJ0aW5nIDEgcHJvY2Vzcwp0ZXN0OiBMYXlpbmcgb3V0IElPIGZp
bGUgKDEgZmlsZSAvIDQwOTZNaUIpCmZpbzogcGlkPTAsIGVycj0xMy9maWxlOmZpbGVzZXR1cC5j
OjE3NCwgZnVuYz1vcGVuLCBlcnJvcj1QZXJtaXNzaW9uIGRlbmllZAoKClJ1biBzdGF0dXMgZ3Jv
dXAgMCAoYWxsIGpvYnMpOgptYXRoaWV1QGJlcmxpbiB+JCBzdWRvIGZpbyAtLXJhbmRyZXBlYXQ9
MSAtLWlvZW5naW5lPWxpYmFpbyAtLWRpcmVjdD0xIC0tZ3RvZF9yZWR1Y2U9MSAtLW5hbWU9dGVz
dCAtLWZpbGVuYW1lPS92YXIvY2FjaGUvdGVzdCAtLWJzPTRrIC0taW9kZXB0aD02NCAtLXNpemU9
NEcgLS1yZWFkd3JpdGU9cmFuZHJ3IC0tcndtaXhyZWFkPTc1ClBhc3N3b3JkOiAKdGVzdDogKGc9
MCk6IHJ3PXJhbmRydywgYnM9KFIpIDQwOTZCLTQwOTZCLCAoVykgNDA5NkItNDA5NkIsIChUKSA0
MDk2Qi00MDk2QiwgaW9lbmdpbmU9bGliYWlvLCBpb2RlcHRoPTY0CmZpby0zLjI4ClN0YXJ0aW5n
IDEgcHJvY2Vzcwp0ZXN0OiBMYXlpbmcgb3V0IElPIGZpbGUgKDEgZmlsZSAvIDQwOTZNaUIpCkpv
YnM6IDEgKGY9MSk6IFttKDEpXVsxMDAuMCVdW3I9NjYxMEtpQi9zLHc9MjE5MEtpQi9zXVtyPTE2
NTIsdz01NDcgSU9QU11bZXRhIDAwbTowMHNdCnRlc3Q6IChncm91cGlkPTAsIGpvYnM9MSk6IGVy
cj0gMDogcGlkPTQ5Mjc0OiBGcmkgRGVjIDE3IDExOjMwOjM4IDIwMjEKICByZWFkOiBJT1BTPTI0
MjMsIEJXPTk2OTVLaUIvcyAoOTkyN2tCL3MpKDMwNzBNaUIvMzI0MjcwbXNlYykKICAgYncgKCAg
S2lCL3MpOiBtaW49IDIzOTIsIG1heD0xNzgwOCwgcGVyPTEwMC4wMCUsIGF2Zz05NzA0LjA2LCBz
dGRldj0yNDc2LjA4LCBzYW1wbGVzPTY0OAogICBpb3BzICAgICAgICA6IG1pbj0gIDU5OCwgbWF4
PSA0NDUyLCBhdmc9MjQyNS45Miwgc3RkZXY9NjE5LjAyLCBzYW1wbGVzPTY0OAogIHdyaXRlOiBJ
T1BTPTgwOSwgQlc9MzI0MEtpQi9zICgzMzE4a0IvcykoMTAyNk1pQi8zMjQyNzBtc2VjKTsgMCB6
b25lIHJlc2V0cwogICBidyAoICBLaUIvcyk6IG1pbj0gMTA4MCwgbWF4PSA1OTY4LCBwZXI9MTAw
LjAwJSwgYXZnPTMyNDMuMjcsIHN0ZGV2PTgzMy45Nywgc2FtcGxlcz02NDgKICAgaW9wcyAgICAg
ICAgOiBtaW49ICAyNzAsIG1heD0gMTQ5MiwgYXZnPTgxMC43MSwgc3RkZXY9MjA4LjQ5LCBzYW1w
bGVzPTY0OAogIGNwdSAgICAgICAgICA6IHVzcj0wLjU1JSwgc3lzPTIuODQlLCBjdHg9MTAwNDc0
OSwgbWFqZj0wLCBtaW5mPTEyOQogIElPIGRlcHRocyAgICA6IDE9MC4xJSwgMj0wLjElLCA0PTAu
MSUsIDg9MC4xJSwgMTY9MC4xJSwgMzI9MC4xJSwgPj02ND0xMDAuMCUKICAgICBzdWJtaXQgICAg
OiAwPTAuMCUsIDQ9MTAwLjAlLCA4PTAuMCUsIDE2PTAuMCUsIDMyPTAuMCUsIDY0PTAuMCUsID49
NjQ9MC4wJQogICAgIGNvbXBsZXRlICA6IDA9MC4wJSwgND0xMDAuMCUsIDg9MC4wJSwgMTY9MC4w
JSwgMzI9MC4wJSwgNjQ9MC4xJSwgPj02ND0wLjAlCiAgICAgaXNzdWVkIHJ3dHM6IHRvdGFsPTc4
NTkyMCwyNjI2NTYsMCwwIHNob3J0PTAsMCwwLDAgZHJvcHBlZD0wLDAsMCwwCiAgICAgbGF0ZW5j
eSAgIDogdGFyZ2V0PTAsIHdpbmRvdz0wLCBwZXJjZW50aWxlPTEwMC4wMCUsIGRlcHRoPTY0CgpS
dW4gc3RhdHVzIGdyb3VwIDAgKGFsbCBqb2JzKToKICAgUkVBRDogYnc9OTY5NUtpQi9zICg5OTI3
a0IvcyksIDk2OTVLaUIvcy05Njk1S2lCL3MgKDk5MjdrQi9zLTk5MjdrQi9zKSwgaW89MzA3ME1p
QiAoMzIxOU1CKSwgcnVuPTMyNDI3MC0zMjQyNzBtc2VjCiAgV1JJVEU6IGJ3PTMyNDBLaUIvcyAo
MzMxOGtCL3MpLCAzMjQwS2lCL3MtMzI0MEtpQi9zICgzMzE4a0Ivcy0zMzE4a0IvcyksIGlvPTEw
MjZNaUIgKDEwNzZNQiksIHJ1bj0zMjQyNzAtMzI0MjcwbXNlYwoKRGlzayBzdGF0cyAocmVhZC93
cml0ZSk6CiAgc2RiOiBpb3M9NzkwOTMwLzI3MDg0NiwgbWVyZ2U9MjUzOS8xMDY4MywgdGlja3M9
MjE5ODIwMjYvMjY0NjI5LCBpbl9xdWV1ZT0yMjI0NjY2NiwgdXRpbD0xMDAuMDAlCg==
--=-=-=--




Message sent to bug-guix@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: bug#51787: GC takes more than 9 hours on berlin
Resent-From: Ricardo Wurmus <rekado@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Fri, 17 Dec 2021 13:31:01 +0000
Resent-Message-ID: <handler.51787.B51787.163974784814513 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 51787
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: Mathieu Othacehe <othacehe@HIDDEN>
Cc: 51787 <at> debbugs.gnu.org
Received: via spool by 51787-submit <at> debbugs.gnu.org id=B51787.163974784814513
          (code B ref 51787); Fri, 17 Dec 2021 13:31:01 +0000
Received: (at 51787) by debbugs.gnu.org; 17 Dec 2021 13:30:48 +0000
Received: from localhost ([127.0.0.1]:38281 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1myDJv-0003m1-TH
	for submit <at> debbugs.gnu.org; Fri, 17 Dec 2021 08:30:48 -0500
Received: from sender3-of-o51.zoho.com ([136.143.184.51]:21772)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <rekado@HIDDEN>) id 1myDJt-0003ls-VU
 for 51787 <at> debbugs.gnu.org; Fri, 17 Dec 2021 08:30:46 -0500
ARC-Seal: i=1; a=rsa-sha256; t=1639747839; cv=none; 
 d=zohomail.com; s=zohoarc; 
 b=c4Roby/W35q0OwiPkK6+O2u+x43X8BQR/3xL7pD7cx1EfTUMsC5xo0xDDOGqtD5PKa9OymJqVNMyR/4nn1FNriqj1gK23MYJ1ALB0DRztqgTDWEbQkEBxqx3erTBOB7x2eswIfu7cLIu1fVGHk9/gpTiage7flVbRLPomh6JpKU=
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com;
 s=zohoarc; t=1639747839;
 h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:To;
 bh=/SXamSEv7clD/9M9R0Ma/Lyue7wr6H9gbgXwG2ZI7p0=; 
 b=T6yNtx4korOCHT/UB00kaJgCkEu0iub4Tir6OIQDn110rxToD7Jj+M6LYNv3L0N6voTn2NbLTK5VGtiJIBJuBsP/Cc7HgzeUlriRU3srZQPy+jKLlZk0MTVoFZChLiBxarRm35gK4cB1RkI4yNP7GNDcgIDEElT8vij7mmcA/hA=
ARC-Authentication-Results: i=1; mx.zohomail.com;
 dkim=pass  header.i=elephly.net;
 spf=pass  smtp.mailfrom=rekado@HIDDEN;
 dmarc=pass header.from=<rekado@HIDDEN>
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1639747839; 
 s=zoho; d=elephly.net; i=rekado@HIDDEN;
 h=References:From:To:Cc:Subject:Date:In-reply-to:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding;
 bh=/SXamSEv7clD/9M9R0Ma/Lyue7wr6H9gbgXwG2ZI7p0=;
 b=g+QWkNkozUJk7Z+FvDMBl3OLnETpeGtZkeTBETLWTmd9Wrk7ij85GVpNS0EAaN08
 S/d96qTafdKSGpyO7z1gNbJtzQZy8x25HkCAfTIOPT3xqRXv01aWAJPTVfYZn1uoGRm
 NR51ii0cb2Q7RSk+qoJcqwU+rcuXdlBbctRmrVME=
Received: from localhost (p54ad487d.dip0.t-ipconnect.de [84.173.72.125]) by
 mx.zohomail.com with SMTPS id 1639747836525936.8960152899299;
 Fri, 17 Dec 2021 05:30:36 -0800 (PST)
References: <87o86pegr3.fsf@HIDDEN> <87zgpuhig7.fsf@HIDDEN>
 <87wnkv15k3.fsf@HIDDEN> <87zgpp971n.fsf@HIDDEN>
 <87pmqet419.fsf@HIDDEN> <87czm57zao.fsf@HIDDEN> <87czm4bvzt.fsf@HIDDEN>
 <877dcccrlq.fsf@HIDDEN> <87a6h86s7o.fsf@HIDDEN> <87bl1nxyt5.fsf@HIDDEN>
 <87o85lhhrd.fsf@HIDDEN> <8735mrec0x.fsf@HIDDEN>
User-agent: mu4e 1.6.10; emacs 28.0.50
From: Ricardo Wurmus <rekado@HIDDEN>
Date: Fri, 17 Dec 2021 14:06:51 +0100
In-reply-to: <8735mrec0x.fsf@HIDDEN>
X-URL: https://elephly.net
X-PGP-Key: https://elephly.net/rekado.pubkey
X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
Message-ID: <87ilvnjr49.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-ZohoMailClient: External
X-Spam-Score: -0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)


Hi Mathieu,

> New day, new benchmark. Berlin has two hard drives, which are roughly
> used this way:
>
> /dev/sda -> / (916G)
> /dev/sdb -> /gnu (37T)

sda consists of two local hard disks that are combined to a RAID.
Here are the disk details:

--8<---------------cut here---------------start------------->8---
Disk.Bay.0:Enclosure.Internal.0-1:RAID.Slot.3-1
   Status                           =3D Ok=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20
   DeviceDescription                =3D Disk 0 in Backplane 1 of RAID Contr=
oller in Slot 3
   RollupStatus                     =3D Ok=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20
   Name                             =3D Physical Disk 0:1:0=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   State                            =3D Online=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20
   OperationState                   =3D Not Applicable=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   PowerStatus                      =3D Spun-Up=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   Size                             =3D 931.000 GB=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   FailurePredicted                 =3D NO=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20
   RemainingRatedWriteEndurance     =3D Not Applicable=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   SecurityStatus                   =3D Not Capable=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   BusProtocol                      =3D SATA=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20
   MediaType                        =3D HDD=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20
   UsedRaidDiskSpace                =3D 931.000 GB=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   AvailableRaidDiskSpace           =3D 0.001 GB=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   Hotspare                         =3D NO=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20
   Manufacturer                     =3D SEAGATE=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   ProductId                        =3D ST1000NX0443=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   Revision                         =3D NB33=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20
   SerialNumber                     =3D W470QK7K=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   PartNumber                       =3D CN08DN1YSGW0076S00L8A00=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   NegotiatedSpeed                  =3D 6.0 Gb/s=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   ManufacturedDay                  =3D 0=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20
   ManufacturedWeek                 =3D 0=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20
   ManufacturedYear                 =3D 0=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20
   ForeignKeyIdentifier             =3D null=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20
   SasAddress                       =3D 0x4433221106000000=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   FormFactor                       =3D 2.5 Inch=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   RaidNominalMediumRotationRate    =3D 7200=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20
   T10PICapability                  =3D Not Capable=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   BlockSizeInBytes                 =3D 512=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20
   MaxCapableSpeed                  =3D 6 Gb/s=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20
   RaidType                         =3D None=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20
   SystemEraseCapability            =3D CryptographicErasePD=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   SelfEncryptingDriveCapability    =3D Not Capable=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   EncryptionCapability             =3D Not Capable=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   CryptographicEraseCapability     =3D Capable=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
Disk.Bay.1:Enclosure.Internal.0-1:RAID.Slot.3-1
   Status                           =3D Ok=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20
   DeviceDescription                =3D Disk 1 in Backplane 1 of RAID Contr=
oller in Slot 3
   RollupStatus                     =3D Ok=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20
   Name                             =3D Physical Disk 0:1:1=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   State                            =3D Online=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20
   OperationState                   =3D Not Applicable=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   PowerStatus                      =3D Spun-Up=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   Size                             =3D 931.000 GB=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   FailurePredicted                 =3D NO=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20
   RemainingRatedWriteEndurance     =3D Not Applicable=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   SecurityStatus                   =3D Not Capable=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   BusProtocol                      =3D SATA=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20
   MediaType                        =3D HDD=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20
   UsedRaidDiskSpace                =3D 931.000 GB=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   AvailableRaidDiskSpace           =3D 0.001 GB=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   Hotspare                         =3D NO=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20
   Manufacturer                     =3D SEAGATE=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   ProductId                        =3D ST1000NX0443=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   Revision                         =3D NB33=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20
   SerialNumber                     =3D W470SYTP=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   PartNumber                       =3D CN08DN1YSGW0077F00FQA00=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   NegotiatedSpeed                  =3D 6.0 Gb/s=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   ManufacturedDay                  =3D 0=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20
   ManufacturedWeek                 =3D 0=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20
   ManufacturedYear                 =3D 0=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20
   ForeignKeyIdentifier             =3D null=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20
   SasAddress                       =3D 0x4433221107000000=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   FormFactor                       =3D 2.5 Inch=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   RaidNominalMediumRotationRate    =3D 7200=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20
   T10PICapability                  =3D Not Capable=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   BlockSizeInBytes                 =3D 512=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20
   MaxCapableSpeed                  =3D 6 Gb/s=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20
   RaidType                         =3D None=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20
   SystemEraseCapability            =3D CryptographicErasePD=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   SelfEncryptingDriveCapability    =3D Not Capable=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   EncryptionCapability             =3D Not Capable=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   CryptographicEraseCapability     =3D Capable=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20
--8<---------------cut here---------------end--------------->8---

sdb is an external storage array (Dell MD3400) filled with 10 hard disks
(SAS) in a RAID 10 configuration (36.36 TB effective capacity).  There
are two hot spares that are currently unassigned.  They are used
automatically when the RAID is degraded.  The two RAID controllers have
read and write caches enabled.  The enclosure has two redundant host
interfaces.

Berlin has two host based adapter cards of which *one* is connected to
the array.  Why only one?  Because we don=E2=80=99t have multipathd configu=
red
so that the system could *boot* off the external array with multipath.
Without multipath the storage would appear as one disk device per card,
but it would not be safe to mount them both at the same time.

If we wanted to make use of the redundant connection here: figure out
how to add multipathd to the initrd and set up multipath *before*
handing off control to Linux.  This would effectively double our
bandwidth to the storage.

My guess is that we=E2=80=99re not even close to saturating the available
bandwidth.

> I ran the fio benchmark tool on both of them. See the reports
> attached, and the following summary:
>
> |       | sda       | sdb       |
> |-------+-----------+-----------|
> | read  | 1565KiB/s | 9695KiB/s |
> | write | 523KiB/s  | 3240KiB/s |
>
>
> I'm not sure how slow those figures are relatively to the hard drives
> technologies. Ricardo, any idea about that?

It seems awfully slow.  Especially performance of sda is abysmal: this
is a local disk.  sdb is the fancy external disk array that=E2=80=99s hooke=
d up
to two HBA cards.  It should not perform *better* than sda.

I=E2=80=99ll run this on a few of the build nodes to get some more comparis=
ons.

--=20
Ricardo




Message sent to bug-guix@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: bug#51787: GC takes more than 9 hours on berlin
Resent-From: Ricardo Wurmus <rekado@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Fri, 17 Dec 2021 14:15:01 +0000
Resent-Message-ID: <handler.51787.B51787.163975047919107 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 51787
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: Mathieu Othacehe <othacehe@HIDDEN>
Cc: 51787 <at> debbugs.gnu.org
Received: via spool by 51787-submit <at> debbugs.gnu.org id=B51787.163975047919107
          (code B ref 51787); Fri, 17 Dec 2021 14:15:01 +0000
Received: (at 51787) by debbugs.gnu.org; 17 Dec 2021 14:14:39 +0000
Received: from localhost ([127.0.0.1]:38334 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1myE0N-0004y6-9R
	for submit <at> debbugs.gnu.org; Fri, 17 Dec 2021 09:14:39 -0500
Received: from sender3-of-o50.zoho.com ([136.143.184.50]:21669)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <rekado@HIDDEN>) id 1myE0L-0004xy-IG
 for 51787 <at> debbugs.gnu.org; Fri, 17 Dec 2021 09:14:38 -0500
ARC-Seal: i=1; a=rsa-sha256; t=1639750470; cv=none; 
 d=zohomail.com; s=zohoarc; 
 b=TFnzGphVLgNRb63se+z6jRBSlI5K+zGTs1jUCDrGbTgUzX26XPXZOHw88QAmTq3fGiVVNGPAPNaYvWAQT8s0cGjoQGPjaxlZFP8JuVjWAsJd9ut7VzCEn8CKPCL4KqIgx9eWDUapSYykYr2A0c/1Mwknm6AH7CYStMqcmghLrs8=
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com;
 s=zohoarc; t=1639750470;
 h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:To;
 bh=Ikt2TH0Dzv1tszYqaPfLsaj0ORKSesEW9ZdyfI56PkA=; 
 b=Fc71qshKV7Fru2UAMPctS83ul3U0VeLROkhndGD9ol47Rom45Uh2IA7sakiqJ7kyg7rZsHTHYCm4sFUHL6CMGV4hyp3id8uLq9xJvsvU0kAIAgTuwKrtqo9m8ie2D0ln1lznnSo6qO9Tsu2cIFNibpwAq3eUckbwpp0zMFtQJ6s=
ARC-Authentication-Results: i=1; mx.zohomail.com;
 dkim=pass  header.i=elephly.net;
 spf=pass  smtp.mailfrom=rekado@HIDDEN;
 dmarc=pass header.from=<rekado@HIDDEN>
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1639750470; 
 s=zoho; d=elephly.net; i=rekado@HIDDEN;
 h=References:From:To:Cc:Subject:Date:In-reply-to:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding;
 bh=Ikt2TH0Dzv1tszYqaPfLsaj0ORKSesEW9ZdyfI56PkA=;
 b=ibzHIMbK/FWYWk2nCyd86n70dMjWaQ1xCe4DHX9mODLFao+Dtt1c2rQ5qrnhogAq
 EHGaFytN2FRjklzc+saa6RyRkm0jJczYLgLDGY9QapThCOHsn4hiISN6bjd0lZbLZt0
 pN+ncr/VE2QtWHlY64hfnQCtM28jE4Yz+iWt0bBA=
Received: from localhost (p54ad487d.dip0.t-ipconnect.de [84.173.72.125]) by
 mx.zohomail.com with SMTPS id 1639750467533243.42228182621022;
 Fri, 17 Dec 2021 06:14:27 -0800 (PST)
References: <87o86pegr3.fsf@HIDDEN> <87zgpuhig7.fsf@HIDDEN>
 <87wnkv15k3.fsf@HIDDEN> <87zgpp971n.fsf@HIDDEN>
 <87pmqet419.fsf@HIDDEN> <87czm57zao.fsf@HIDDEN> <87czm4bvzt.fsf@HIDDEN>
 <877dcccrlq.fsf@HIDDEN> <87a6h86s7o.fsf@HIDDEN> <87bl1nxyt5.fsf@HIDDEN>
 <87o85lhhrd.fsf@HIDDEN> <8735mrec0x.fsf@HIDDEN>
 <87ilvnjr49.fsf@HIDDEN>
User-agent: mu4e 1.6.10; emacs 27.2
From: Ricardo Wurmus <rekado@HIDDEN>
Date: Fri, 17 Dec 2021 15:08:23 +0100
In-reply-to: <87ilvnjr49.fsf@HIDDEN>
X-URL: https://elephly.net
X-PGP-Key: https://elephly.net/rekado.pubkey
X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
Message-ID: <878rwjl3nj.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-ZohoMailClient: External
X-Spam-Score: -0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)


Ricardo Wurmus <rekado@HIDDEN> writes:

> I=E2=80=99ll run this on a few of the build nodes to get some more compar=
isons.

I ran =E2=80=9Cguix deploy=E2=80=9D for hydra-guix-107, a node that was bou=
ght at the
same time as the one that is now the head node of ci.guix.gnu.org.  I
copied over a current Guix and installed =E2=80=9Cfio=E2=80=9D
(/gnu/store/qs9cyy5s95n2fbjmxs48iccqvsvj6wxr-fio-3.28/bin/fio) there.

Here=E2=80=99s the output:

--8<---------------cut here---------------start------------->8---
root@hydra-guix-107 ~# fio --randrepeat=3D1 --ioengine=3Dlibaio --direct=3D=
1 --gtod_reduce=3D1 --name=3Dtest --filename=3Dtest --bs=3D4k --iodepth=3D6=
4 --size=3D4G --readwrite=3Drandrw --rwmixread=3D75
test: (g=3D0): rw=3Drandrw, bs=3D(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096=
B-4096B, ioengine=3Dlibaio, iodepth=3D64
fio-3.28
Starting 1 process
test: Laying out IO file (1 file / 4096MiB)
Jobs: 1 (f=3D1): [m(1)][100.0%][r=3D172MiB/s,w=3D56.9MiB/s][r=3D44.2k,w=3D1=
4.6k IOPS][eta 00m:00s]
test: (groupid=3D0, jobs=3D1): err=3D 0: pid=3D45365: Fri Dec 17 14:50:41 2=
021
  read: IOPS=3D42.9k, BW=3D167MiB/s (176MB/s)(3070MiB/18331msec)
   bw (  KiB/s): min=3D21208, max=3D199928, per=3D100.00%, avg=3D171621.56,=
 stdev=3D46624.05, samples=3D36
   iops        : min=3D 5302, max=3D49982, avg=3D42905.39, stdev=3D11656.01=
, samples=3D36
  write: IOPS=3D14.3k, BW=3D56.0MiB/s (58.7MB/s)(1026MiB/18331msec); 0 zone=
 resets
   bw (  KiB/s): min=3D 7424, max=3D66720, per=3D100.00%, avg=3D57364.22, s=
tdev=3D15612.42, samples=3D36
   iops        : min=3D 1856, max=3D16680, avg=3D14341.06, stdev=3D3903.10,=
 samples=3D36
  cpu          : usr=3D6.27%, sys=3D24.78%, ctx=3D121626, majf=3D0, minf=3D=
11
  IO depths    : 1=3D0.1%, 2=3D0.1%, 4=3D0.1%, 8=3D0.1%, 16=3D0.1%, 32=3D0.=
1%, >=3D64=3D100.0%
     submit    : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%, 32=3D0.0%, 64=
=3D0.0%, >=3D64=3D0.0%
     complete  : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%, 32=3D0.0%, 64=
=3D0.1%, >=3D64=3D0.0%
     issued rwts: total=3D785920,262656,0,0 short=3D0,0,0,0 dropped=3D0,0,0=
,0
     latency   : target=3D0, window=3D0, percentile=3D100.00%, depth=3D64

Run status group 0 (all jobs):
   READ: bw=3D167MiB/s (176MB/s), 167MiB/s-167MiB/s (176MB/s-176MB/s), io=
=3D3070MiB (3219MB), run=3D18331-18331msec
  WRITE: bw=3D56.0MiB/s (58.7MB/s), 56.0MiB/s-56.0MiB/s (58.7MB/s-58.7MB/s)=
, io=3D1026MiB (1076MB), run=3D18331-18331msec

Disk stats (read/write):
  sda: ios=3D779469/260494, merge=3D0/6, ticks=3D932265/195191, in_queue=3D=
1127456, util=3D99.50%
--8<---------------cut here---------------end--------------->8---

Here sda is RAID of two SSDs:

--8<---------------cut here---------------start------------->8---
racadm>>storage get pdisks -o
Disk.Bay.0:Enclosure.Internal.0-1:RAID.Integrated.1-1
   Status                           =3D Ok=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20
   DeviceDescription                =3D Disk 0 in Backplane 1 of Integrated=
 RAID Controller 1
   RollupStatus                     =3D Ok=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20
   Name                             =3D Solid State Disk 0:1:0=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   State                            =3D Online=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20
   OperationState                   =3D Not Applicable=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   PowerStatus                      =3D On=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20
   Size                             =3D 223.001 GB=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   FailurePredicted                 =3D NO=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20
   RemainingRatedWriteEndurance     =3D 99 %=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20
   SecurityStatus                   =3D Not Capable=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   BusProtocol                      =3D SATA=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20
   MediaType                        =3D SSD=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20
   UsedRaidDiskSpace                =3D 223.001 GB=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   AvailableRaidDiskSpace           =3D 0.001 GB=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   Hotspare                         =3D NO=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20
   Manufacturer                     =3D INTEL=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20
   ProductId                        =3D SSDSC2KG240G8R=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   Revision                         =3D XCV1DL67=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   SerialNumber                     =3D BTYG91520CHD240AGN=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   PartNumber                       =3D CN0T1WH8PESIT95302LTA01=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   NegotiatedSpeed                  =3D 6.0 Gb/s=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   ManufacturedDay                  =3D 0=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20
   ManufacturedWeek                 =3D 0=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20
   ManufacturedYear                 =3D 0=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20
   ForeignKeyIdentifier             =3D null=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20
   SasAddress                       =3D 0x4433221104000000=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   WWN                              =3D 0x4433221104000000=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   FormFactor                       =3D 2.5 Inch=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   RaidNominalMediumRotationRate    =3D 1=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20
   T10PICapability                  =3D Not Capable=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   BlockSizeInBytes                 =3D 512=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20
   MaxCapableSpeed                  =3D 6 Gb/s=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20
   RaidType                         =3D Unknown=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   SystemEraseCapability            =3D CryptographicErasePD=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   SelfEncryptingDriveCapability    =3D Not Capable=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   EncryptionCapability             =3D Not Capable=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   CryptographicEraseCapability     =3D Capable=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   Certified                        =3D Yes=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20
   NonRAIDDiskCachePolicy           =3D Not Applicable=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   EncryptionProtocol               =3D None=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20
Disk.Bay.1:Enclosure.Internal.0-1:RAID.Integrated.1-1
   Status                           =3D Ok=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20
   DeviceDescription                =3D Disk 1 in Backplane 1 of Integrated=
 RAID Controller 1
   RollupStatus                     =3D Ok=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20
   Name                             =3D Solid State Disk 0:1:1=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   State                            =3D Online=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20
   OperationState                   =3D Not Applicable=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   PowerStatus                      =3D On=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20
   Size                             =3D 223.001 GB=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   FailurePredicted                 =3D NO=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20
   RemainingRatedWriteEndurance     =3D 99 %=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20
   SecurityStatus                   =3D Not Capable=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   BusProtocol                      =3D SATA=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20
   MediaType                        =3D SSD=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20
   UsedRaidDiskSpace                =3D 223.001 GB=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   AvailableRaidDiskSpace           =3D 0.001 GB=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   Hotspare                         =3D NO=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20
   Manufacturer                     =3D INTEL=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20
   ProductId                        =3D SSDSC2KG240G8R=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   Revision                         =3D XCV1DL67=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   SerialNumber                     =3D BTYG915502D9240AGN=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   PartNumber                       =3D CN0T1WH8PESIT95303BSA01=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   NegotiatedSpeed                  =3D 6.0 Gb/s=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   ManufacturedDay                  =3D 0=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20
   ManufacturedWeek                 =3D 0=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20
   ManufacturedYear                 =3D 0=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20
   ForeignKeyIdentifier             =3D null=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20
   SasAddress                       =3D 0x4433221100000000=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   WWN                              =3D 0x4433221100000000=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   FormFactor                       =3D 2.5 Inch=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   RaidNominalMediumRotationRate    =3D 1=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20
   T10PICapability                  =3D Not Capable=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   BlockSizeInBytes                 =3D 512=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20
   MaxCapableSpeed                  =3D 6 Gb/s=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20
   RaidType                         =3D Unknown=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   SystemEraseCapability            =3D CryptographicErasePD=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   SelfEncryptingDriveCapability    =3D Not Capable=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   EncryptionCapability             =3D Not Capable=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   CryptographicEraseCapability     =3D Capable=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   Certified                        =3D Yes=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20
   NonRAIDDiskCachePolicy           =3D Not Applicable=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
   EncryptionProtocol               =3D None=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20
--8<---------------cut here---------------end--------------->8---

--=20
Ricardo




Message sent to bug-guix@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: bug#51787: Disk performance on ci.guix.gnu.org
Resent-From: Mathieu Othacehe <othacehe@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Mon, 20 Dec 2021 17:00:02 +0000
Resent-Message-ID: <handler.51787.B51787.164001958823426 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 51787
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: Ricardo Wurmus <rekado@HIDDEN>
Cc: 51787 <at> debbugs.gnu.org
Received: via spool by 51787-submit <at> debbugs.gnu.org id=B51787.164001958823426
          (code B ref 51787); Mon, 20 Dec 2021 17:00:02 +0000
Received: (at 51787) by debbugs.gnu.org; 20 Dec 2021 16:59:48 +0000
Received: from localhost ([127.0.0.1]:51290 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mzM0q-00065m-21
	for submit <at> debbugs.gnu.org; Mon, 20 Dec 2021 11:59:48 -0500
Received: from eggs.gnu.org ([209.51.188.92]:41468)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <othacehe@HIDDEN>) id 1mzM0l-00065U-95
 for 51787 <at> debbugs.gnu.org; Mon, 20 Dec 2021 11:59:46 -0500
Received: from [2001:470:142:3::e] (port=45860 helo=fencepost.gnu.org)
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <othacehe@HIDDEN>)
 id 1mzM0f-0006Qj-Mm; Mon, 20 Dec 2021 11:59:37 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To:
 From; bh=e7FuET/gbny9oP4pAlIHUNGsUk69aMrMMJUL//G4nUQ=; b=fjYMwiw2wZsJe8/E9pqK
 ZUEGvsxP/FZZ+ZyfhEAGmNzDPeDbRXeqbzi92IiwdBB3KWq92AOdalMtXD6ix2/42jlM/evAFg0TT
 YoSb9cAGG/A/I7oRYQWAjl0fawMbN5MhGiR2vY1pQYetzurRSXv0NT3McmDRMUrMCcUxmLGTmSwmT
 r7IypUWlKtxHAIwX+j1SvcnGRrVoQNKb18RYNfRnsME1mFvgCCkJ7q9aFKfTanLJaSCnDAmaa/2ga
 vqheSYcsZ8a62S+eKrKr28RCB21vdssGY/HRP9v17YKIDjxrSRXp5U4BRkj7Ayj2YxvmLb3h+suBy
 BO0FAPPLtTkeBg==;
Received: from [2a01:cb1e:72:d94a:2d41:6e45:df8f:384b] (port=51880 helo=meije)
 by fencepost.gnu.org with esmtpsa
 (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1)
 (envelope-from <othacehe@HIDDEN>)
 id 1mzM0e-0001dW-LZ; Mon, 20 Dec 2021 11:59:37 -0500
From: Mathieu Othacehe <othacehe@HIDDEN>
References: <875yrjpi1y.fsf@HIDDEN>
Date: Mon, 20 Dec 2021 17:59:33 +0100
In-Reply-To: <875yrjpi1y.fsf@HIDDEN> (Ricardo Wurmus's message of "Mon,
 20 Dec 2021 13:32:33 +0100")
Message-ID: <87o85bjjpm.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)


Hey,

> This is still pretty bad, but better than the <1M performance suggested
> by previous runs.

Mmh interesting, I also have a x10 speed up on sdb by increasing the
block size from 4k to 512k. I'm not sure what conclusion should we draw
from this observation.

In particular for our most urging matter, /gnu/store/trash
removal. Moving to a faster hard drive would definitely help here, but I
still don't understand if that disk performance regression comes from
Linux, the file-system fragmentation, or the disk itself.

>    READ: bw=1547MiB/s (1622MB/s), 1547MiB/s-1547MiB/s (1622MB/s-1622MB/s), io=3055MiB (3203MB), run=1975-1975msec
>   WRITE: bw=527MiB/s (553MB/s), 527MiB/s-527MiB/s (553MB/s-553MB/s), io=1042MiB (1092MB), run=1975-1975msec

Wooh that's fast! On test could be to copy the /gnu/store/trash content
to the SAN an observe how long that it takes for this operating to
complete.

Thanks for your support on that complex topic :)

Mathieu




Message sent to bug-guix@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: bug#51787: Disk performance on ci.guix.gnu.org
Resent-From: Ricardo Wurmus <rekado@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Mon, 20 Dec 2021 17:10:02 +0000
Resent-Message-ID: <handler.51787.B51787.164002015624584 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 51787
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: Mathieu Othacehe <othacehe@HIDDEN>
Cc: 51787 <at> debbugs.gnu.org
Received: via spool by 51787-submit <at> debbugs.gnu.org id=B51787.164002015624584
          (code B ref 51787); Mon, 20 Dec 2021 17:10:02 +0000
Received: (at 51787) by debbugs.gnu.org; 20 Dec 2021 17:09:16 +0000
Received: from localhost ([127.0.0.1]:51306 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mzMA0-0006OS-FV
	for submit <at> debbugs.gnu.org; Mon, 20 Dec 2021 12:09:16 -0500
Received: from sender3-of-o52.zoho.com ([136.143.184.52]:21825)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <rekado@HIDDEN>) id 1mzM9y-0006OH-Kk
 for 51787 <at> debbugs.gnu.org; Mon, 20 Dec 2021 12:09:15 -0500
ARC-Seal: i=1; a=rsa-sha256; t=1640020142; cv=none; 
 d=zohomail.com; s=zohoarc; 
 b=MvNsUk+rwORfasj6KHkegIkcbfSy2VRbpW6BqOSN26QYt6i0gcCoNhEVc05+wrgtbH1dKRUk/SORD6JXRadQPdTwZ9xvNvO6CYSHOe30FOsp9NvDN2PWRmdD2kTuQ8KchEi+5KwSUjL3hElKE/Z53ejA9709fwoMa3WnPZ8ti6Q=
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com;
 s=zohoarc; t=1640020142;
 h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:To;
 bh=Srh+d2fQ4E4iG4E1DV7TLhqynfH2bnQi/pkoK3qRZVI=; 
 b=SR0P0w3ddXAbXEPbZEzaqBzVAmaPtNh0Q7YUOqyjANJV9MdaYwRgQa2LSDPblCrZQcM6Ksjyh8g+fJEMvecu7eWjEL/sOivbmSPKX+mFJFZT8nS+YexyZziZ3Y9Qxh/VGMXKgMQzuRBT8Naq3n+elp/pjUlxGaawzRUd7WyHeCY=
ARC-Authentication-Results: i=1; mx.zohomail.com;
 dkim=pass  header.i=elephly.net;
 spf=pass  smtp.mailfrom=rekado@HIDDEN;
 dmarc=pass header.from=<rekado@HIDDEN>
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1640020142; 
 s=zoho; d=elephly.net; i=rekado@HIDDEN;
 h=References:From:To:Cc:Subject:Date:In-reply-to:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding;
 bh=Srh+d2fQ4E4iG4E1DV7TLhqynfH2bnQi/pkoK3qRZVI=;
 b=LiNjnmvZVB51GNT6VLu9VqCrs9ECLKkPv50zmtQvb1J1gZFeZ2Y9hnLJUsN6RNLD
 m6ptsIcU3DRPK5H0AM+GD7oCxzayyeUMinSFfc1FMKSjVc2R2Sui8grG4esFNNYQI8V
 IRuBSyGAUidTL3fUXDCeVVkJwJtH1sDsSPNP/Ecc=
Received: from localhost (p54ad4ec1.dip0.t-ipconnect.de [84.173.78.193]) by
 mx.zohomail.com with SMTPS id 164002014099628.77387882910216;
 Mon, 20 Dec 2021 09:09:00 -0800 (PST)
References: <875yrjpi1y.fsf@HIDDEN> <87o85bjjpm.fsf@HIDDEN>
User-agent: mu4e 1.6.10; emacs 27.2
From: Ricardo Wurmus <rekado@HIDDEN>
Date: Mon, 20 Dec 2021 18:05:08 +0100
In-reply-to: <87o85bjjpm.fsf@HIDDEN>
X-URL: https://elephly.net
X-PGP-Key: https://elephly.net/rekado.pubkey
X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
Message-ID: <871r27p5jq.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-ZohoMailClient: External
X-Spam-Score: -0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)


Mathieu Othacehe <othacehe@HIDDEN> writes:

>> This is still pretty bad, but better than the <1M performance suggested
>> by previous runs.
>
> Mmh interesting, I also have a x10 speed up on sdb by increasing the
> block size from 4k to 512k. I'm not sure what conclusion should we draw
> from this observation.

As a general rule, we want the block size to match that of the
configured disk layout =E2=80=94 if we care about getting the best numbers =
in
our benchmarks.  With real workloads things are always going to be
slower anyway.

> In particular for our most urging matter, /gnu/store/trash
> removal. Moving to a faster hard drive would definitely help here, but I
> still don't understand if that disk performance regression comes from
> Linux, the file-system fragmentation, or the disk itself.
>
>>    READ: bw=3D1547MiB/s (1622MB/s), 1547MiB/s-1547MiB/s (1622MB/s-1622MB=
/s), io=3D3055MiB (3203MB), run=3D1975-1975msec
>>   WRITE: bw=3D527MiB/s (553MB/s), 527MiB/s-527MiB/s (553MB/s-553MB/s), i=
o=3D1042MiB (1092MB), run=3D1975-1975msec
>
> Wooh that's fast! On test could be to copy the /gnu/store/trash content
> to the SAN an observe how long that it takes for this operating to
> complete.

Do you mean time the copy or time the removal from that storage?  You
know what, I=E2=80=99ll time both.  I=E2=80=99ll need to get more space fir=
st.  I think
the trash directory is larger than the 500G that I got for testing the
SAN.

> Thanks for your support on that complex topic :)

Hey, I=E2=80=99m just happy neither of us has to do this alone.  Thank you!

--=20
Ricardo




Message sent to bug-guix@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: bug#51787: Disk performance on ci.guix.gnu.org
Resent-From: Bengt Richter <bokr@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Mon, 20 Dec 2021 18:38:01 +0000
Resent-Message-ID: <handler.51787.B51787.16400254271534 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 51787
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: Mathieu Othacehe <othacehe@HIDDEN>
Cc: Ricardo Wurmus <rekado@HIDDEN>, 51787 <at> debbugs.gnu.org
Reply-To: Bengt Richter <bokr@HIDDEN>
Received: via spool by 51787-submit <at> debbugs.gnu.org id=B51787.16400254271534
          (code B ref 51787); Mon, 20 Dec 2021 18:38:01 +0000
Received: (at 51787) by debbugs.gnu.org; 20 Dec 2021 18:37:07 +0000
Received: from localhost ([127.0.0.1]:51437 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mzNX0-0000Og-Pp
	for submit <at> debbugs.gnu.org; Mon, 20 Dec 2021 13:37:06 -0500
Received: from imta-38.everyone.net ([216.200.145.38]:47114)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <bokr@HIDDEN>) id 1mzNWx-0000OW-Qu
 for 51787 <at> debbugs.gnu.org; Mon, 20 Dec 2021 13:37:05 -0500
Received: from pps.filterd (omta003.sj2.proofpoint.com [127.0.0.1])
 by imta-38.everyone.net (8.16.0.43/8.16.0.43) with SMTP id 1BKIU9TT007220;
 Mon, 20 Dec 2021 10:37:02 -0800
X-Eon-Originating-Account: u6V8_VNW-eR85gZdCrnb2L0l1GHZTv2hQA_FHe9vDfQ
X-Eon-Dm: m0116293.ppops.net
Received: by m0116293.mta.everyone.net (EON-AUTHRELAY2 - 53b9213a)
 id m0116293.6195d1a3.20c606; Mon, 20 Dec 2021 10:37:01 -0800
X-Eon-Sig: AQMHrIJhwM1NRFnBAgIAAAAD,701f634b381ee43731596c40407deecf
X-Eip: NcFhRzggRJtsCNG9cK0ReyYDVgZS-5eBCKPPAOxV6MQ
Date: Mon, 20 Dec 2021 19:36:51 +0100
From: Bengt Richter <bokr@HIDDEN>
Message-ID: <20211220183651.GA8380@LionPure>
References: <875yrjpi1y.fsf@HIDDEN>
 <87o85bjjpm.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
In-Reply-To: <87o85bjjpm.fsf@HIDDEN>
User-Agent: Mutt/1.10.1 (2018-07-13)
X-Proofpoint-GUID: 3ncA-nKkl22u3UFxoQbdMBaU_lvs-a8S
X-Proofpoint-ORIG-GUID: 3ncA-nKkl22u3UFxoQbdMBaU_lvs-a8S
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.425, 18.0.790
 definitions=2021-12-20_08:2021-12-20,
 2021-12-20 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0
 suspectscore=0
 mlxlogscore=432 clxscore=1034 lowpriorityscore=0 phishscore=0 adultscore=0
 impostorscore=0 malwarescore=0 priorityscore=1501 mlxscore=0 spamscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2110150000
 definitions=main-2112200103
X-Spam-Score: 0.2 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.8 (/)

On +2021-12-20 17:59:33 +0100, Mathieu Othacehe wrote:
> 
> Hey,
> 
> > This is still pretty bad, but better than the <1M performance suggested
> > by previous runs.
> 
> Mmh interesting, I also have a x10 speed up on sdb by increasing the
> block size from 4k to 512k. I'm not sure what conclusion should we draw
> from this observation.
> 
> In particular for our most urging matter, /gnu/store/trash
> removal. Moving to a faster hard drive would definitely help here, but I
> still don't understand if that disk performance regression comes from
> Linux, the file-system fragmentation, or the disk itself.
> 
> >    READ: bw=1547MiB/s (1622MB/s), 1547MiB/s-1547MiB/s (1622MB/s-1622MB/s), io=3055MiB (3203MB), run=1975-1975msec
> >   WRITE: bw=527MiB/s (553MB/s), 527MiB/s-527MiB/s (553MB/s-553MB/s), io=1042MiB (1092MB), run=1975-1975msec
> 
> Wooh that's fast! On test could be to copy the /gnu/store/trash content
> to the SAN an observe how long that it takes for this operating to
> complete.

also might be interesting to copy to /dev/null
to see read rate alone on /gnu/store?

> 
> Thanks for your support on that complex topic :)
> 
> Mathieu
> 
> 
> 

-- 
Regards,
Bengt Richter




Message sent to bug-guix@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: bug#51787: GC takes more than 9 hours on berlin
References: <87o86pegr3.fsf@HIDDEN>
In-Reply-To: <87o86pegr3.fsf@HIDDEN>
Resent-From: Ricardo Wurmus <rekado@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Mon, 20 Dec 2021 21:26:01 +0000
Resent-Message-ID: <handler.51787.B51787.164003552318886 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 51787
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: 51787 <at> debbugs.gnu.org
Received: via spool by 51787-submit <at> debbugs.gnu.org id=B51787.164003552318886
          (code B ref 51787); Mon, 20 Dec 2021 21:26:01 +0000
Received: (at 51787) by debbugs.gnu.org; 20 Dec 2021 21:25:23 +0000
Received: from localhost ([127.0.0.1]:51591 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mzQ9q-0004uX-Rl
	for submit <at> debbugs.gnu.org; Mon, 20 Dec 2021 16:25:23 -0500
Received: from sender3-of-o52.zoho.com ([136.143.184.52]:21817)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <rekado@HIDDEN>) id 1mzQ9o-0004uO-IK
 for 51787 <at> debbugs.gnu.org; Mon, 20 Dec 2021 16:25:21 -0500
ARC-Seal: i=1; a=rsa-sha256; t=1640035516; cv=none; 
 d=zohomail.com; s=zohoarc; 
 b=QHTqoZxDLzm2QLuJfLpwYtzrCl7TeciaKgkQDNvQ/XN60dEF2R6HnzKzYPsmOy1crw+bNXIJ8RLMDJktiXAh/b6qsp5BEIzArmBYlc418a4wPWtl4MLiZaVWnqPACwe8qifQB3EzpDQurn+kG6g1iTnLtvYht3jDEqBiMKBxawY=
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com;
 s=zohoarc; t=1640035516;
 h=Content-Type:Content-Transfer-Encoding:Date:From:MIME-Version:Message-ID:Subject:To;
 bh=xIbwBFNy3bt2yzc9rgHFU2pVXQ9mHZioEMGDDMM6YgY=; 
 b=DcyRK+jd/Be3+x/fvXnv5jWGMFu+ySaY/WEqjPBtuaqW1Eqn4aEj9puTPNKX8ActpwuiGjpkRl3CR+2vM7SVYl+qulaLNitHrFVrBqvtf3TidIpS7uW55IgbXvC2di9aKeafIlXadkWa6ap9PypIUDRwT+B3IexpR+N/9PiZVZo=
ARC-Authentication-Results: i=1; mx.zohomail.com;
 dkim=pass  header.i=elephly.net;
 spf=pass  smtp.mailfrom=rekado@HIDDEN;
 dmarc=pass header.from=<rekado@HIDDEN>
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1640035516; 
 s=zoho; d=elephly.net; i=rekado@HIDDEN;
 h=From:To:Subject:Date:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding;
 bh=xIbwBFNy3bt2yzc9rgHFU2pVXQ9mHZioEMGDDMM6YgY=;
 b=cSvmmhusJJIJ6kSr2f9y2LXKpc9lJC7xs+Mwu/TDCqMWdaChaZwjTmvp/Aop2h0y
 dtN7UmruJJIJTTeaP1hNxyc3Vsx3DEZtsctl/JTdHD6c5ds8PRw6nOmcRjoRHztQN3q
 YKf+f10E1tbpD6inQ1I4oRQ5vs1gBFmD8PAMFGh8=
Received: from localhost (p54ad4ec1.dip0.t-ipconnect.de [84.173.78.193]) by
 mx.zohomail.com with SMTPS id 1640035513433995.1509856250896;
 Mon, 20 Dec 2021 13:25:13 -0800 (PST)
User-agent: mu4e 1.6.10; emacs 27.2
From: Ricardo Wurmus <rekado@HIDDEN>
Date: Mon, 20 Dec 2021 22:12:46 +0100
X-URL: https://elephly.net
X-PGP-Key: https://elephly.net/rekado.pubkey
X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
Message-ID: <87czlrnf4a.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-ZohoMailClient: External
X-Spam-Score: -0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

My colleague extended the SAN slice to 5TB for more realistic testing.
I formatted the disk with btrfs, and mounted it like this:

    mount /dev/sdd /mnt_test/

Then I ran the test with block size 512k:

--8<---------------cut here---------------start------------->8---
root@berlin ~# fio --randrepeat=3D1 --ioengine=3Dlibaio --direct=3D1 --gtod=
_reduce=3D1 --name=3Dtest --filename=3D/mnt_test/test --bs=3D512k --iodepth=
=3D64 --size=3D4G --readwrite=3Drandrw --rwmixread=3D75
test: (g=3D0): rw=3Drandrw, bs=3D(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) =
512KiB-512KiB, ioengine=3Dlibaio, iodepth=3D64
fio-3.6
Starting 1 process
test: Laying out IO file (1 file / 4096MiB)
Jobs: 1 (f=3D1): [m(1)][100.0%][r=3D802MiB/s,w=3D274MiB/s][r=3D1603,w=3D547=
 IOPS][eta 00m:00s]
test: (groupid=3D0, jobs=3D1): err=3D 0: pid=3D16949: Mon Dec 20 22:18:28 2=
021
   read: IOPS=3D1590, BW=3D795MiB/s (834MB/s)(3055MiB/3842msec)
   bw (  KiB/s): min=3D747520, max=3D857088, per=3D99.83%, avg=3D812763.43,=
 stdev=3D44213.07, samples=3D7
   iops        : min=3D 1460, max=3D 1674, avg=3D1587.43, stdev=3D86.35, sa=
mples=3D7
  write: IOPS=3D542, BW=3D271MiB/s (284MB/s)(1042MiB/3842msec)
   bw (  KiB/s): min=3D262144, max=3D297984, per=3D100.00%, avg=3D278820.57=
, stdev=3D15115.88, samples=3D7
   iops        : min=3D  512, max=3D  582, avg=3D544.57, stdev=3D29.52, sam=
ples=3D7
  cpu          : usr=3D1.98%, sys=3D96.28%, ctx=3D1096, majf=3D0, minf=3D6
  IO depths    : 1=3D0.1%, 2=3D0.1%, 4=3D0.1%, 8=3D0.1%, 16=3D0.2%, 32=3D0.=
4%, >=3D64=3D99.2%
     submit    : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%, 32=3D0.0%, 64=
=3D0.0%, >=3D64=3D0.0%
     complete  : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%, 32=3D0.0%, 64=
=3D0.1%, >=3D64=3D0.0%
     issued rwts: total=3D6109,2083,0,0 short=3D0,0,0,0 dropped=3D0,0,0,0
     latency   : target=3D0, window=3D0, percentile=3D100.00%, depth=3D64

Run status group 0 (all jobs):
   READ: bw=3D795MiB/s (834MB/s), 795MiB/s-795MiB/s (834MB/s-834MB/s), io=
=3D3055MiB (3203MB), run=3D3842-3842msec
  WRITE: bw=3D271MiB/s (284MB/s), 271MiB/s-271MiB/s (284MB/s-284MB/s), io=
=3D1042MiB (1092MB), run=3D3842-3842msec
--8<---------------cut here---------------end--------------->8---

Because this is fun I reran it with the same arguments:

--8<---------------cut here---------------start------------->8---
root@berlin ~# fio --randrepeat=3D1 --ioengine=3Dlibaio --direct=3D1 --gtod=
_reduce=3D1 --name=3Dtest --filename=3D/mnt_test/test --bs=3D512k --iodepth=
=3D64 --size=3D4G --readwrite=3Drandrw --rwmixread=3D75
test: (g=3D0): rw=3Drandrw, bs=3D(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) =
512KiB-512KiB, ioengine=3Dlibaio, iodepth=3D64
fio-3.6
Starting 1 process
Jobs: 1 (f=3D0): [f(1)][-.-%][r=3D756MiB/s,w=3D260MiB/s][r=3D1511,w=3D519 I=
OPS][eta 00m:00s]
test: (groupid=3D0, jobs=3D1): err=3D 0: pid=3D17488: Mon Dec 20 22:18:56 2=
021
   read: IOPS=3D1647, BW=3D824MiB/s (864MB/s)(3055MiB/3708msec)
   bw (  KiB/s): min=3D738304, max=3D929792, per=3D99.28%, avg=3D837485.71,=
 stdev=3D73710.05, samples=3D7
   iops        : min=3D 1442, max=3D 1816, avg=3D1635.71, stdev=3D143.96, s=
amples=3D7
  write: IOPS=3D561, BW=3D281MiB/s (295MB/s)(1042MiB/3708msec)
   bw (  KiB/s): min=3D234496, max=3D320512, per=3D99.79%, avg=3D287012.57,=
 stdev=3D29009.60, samples=3D7
   iops        : min=3D  458, max=3D  626, avg=3D560.57, stdev=3D56.66, sam=
ples=3D7
  cpu          : usr=3D1.38%, sys=3D96.47%, ctx=3D1394, majf=3D0, minf=3D16=
420
  IO depths    : 1=3D0.1%, 2=3D0.1%, 4=3D0.1%, 8=3D0.1%, 16=3D0.2%, 32=3D0.=
4%, >=3D64=3D99.2%
     submit    : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%, 32=3D0.0%, 64=
=3D0.0%, >=3D64=3D0.0%
     complete  : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%, 32=3D0.0%, 64=
=3D0.1%, >=3D64=3D0.0%
     issued rwts: total=3D6109,2083,0,0 short=3D0,0,0,0 dropped=3D0,0,0,0
     latency   : target=3D0, window=3D0, percentile=3D100.00%, depth=3D64

Run status group 0 (all jobs):
   READ: bw=3D824MiB/s (864MB/s), 824MiB/s-824MiB/s (864MB/s-864MB/s), io=
=3D3055MiB (3203MB), run=3D3708-3708msec
  WRITE: bw=3D281MiB/s (295MB/s), 281MiB/s-281MiB/s (295MB/s-295MB/s), io=
=3D1042MiB (1092MB), run=3D3708-3708msec
--8<---------------cut here---------------end--------------->8---

Then I mounted with compression and space cache:

    mount /dev/sdd -o compress-force=3Dzstd,space_cache=3Dv2 /mnt_test/

The numbers don=E2=80=99t differ much at all.

--8<---------------cut here---------------start------------->8---
Run status group 0 (all jobs):
   READ: bw=3D882MiB/s (925MB/s), 882MiB/s-882MiB/s (925MB/s-925MB/s), io=
=3D3055MiB (3203MB), run=3D3464-3464msec
  WRITE: bw=3D301MiB/s (315MB/s), 301MiB/s-301MiB/s (315MB/s-315MB/s), io=
=3D1042MiB (1092MB), run=3D3464-3464msec
--8<---------------cut here---------------end--------------->8---


I then erased the file system and again put on a big ext4:

--8<---------------cut here---------------start------------->8---
root@berlin ~# fio --randrepeat=3D1 --ioengine=3Dlibaio --direct=3D1 --gtod=
_reduce=3D1 --name=3Dtest --filename=3D/mnt_test/test --bs=3D512k --iodepth=
=3D64 --size=3D4G --readwrite=3Drandrw --rwmixread=3D75
test: (g=3D0): rw=3Drandrw, bs=3D(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) =
512KiB-512KiB, ioengine=3Dlibaio, iodepth=3D64
fio-3.6
Starting 1 process
test: Laying out IO file (1 file / 4096MiB)
Jobs: 1 (f=3D1): [m(1)][-.-%][r=3D1539MiB/s,w=3D526MiB/s][r=3D3078,w=3D1052=
 IOPS][eta 00m:00s]
test: (groupid=3D0, jobs=3D1): err=3D 0: pid=3D20672: Mon Dec 20 22:23:29 2=
021
   read: IOPS=3D3077, BW=3D1539MiB/s (1614MB/s)(3055MiB/1985msec)
   bw (  MiB/s): min=3D 1530, max=3D 1548, per=3D100.00%, avg=3D1539.33, st=
dev=3D 9.02, samples=3D3
   iops        : min=3D 3060, max=3D 3096, avg=3D3078.67, stdev=3D18.04, sa=
mples=3D3
  write: IOPS=3D1049, BW=3D525MiB/s (550MB/s)(1042MiB/1985msec)
   bw (  KiB/s): min=3D533504, max=3D557056, per=3D100.00%, avg=3D546133.33=
, stdev=3D11868.39, samples=3D3
   iops        : min=3D 1042, max=3D 1088, avg=3D1066.67, stdev=3D23.18, sa=
mples=3D3
  cpu          : usr=3D2.17%, sys=3D11.24%, ctx=3D4787, majf=3D0, minf=3D8
  IO depths    : 1=3D0.1%, 2=3D0.1%, 4=3D0.1%, 8=3D0.1%, 16=3D0.2%, 32=3D0.=
4%, >=3D64=3D99.2%
     submit    : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%, 32=3D0.0%, 64=
=3D0.0%, >=3D64=3D0.0%
     complete  : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%, 32=3D0.0%, 64=
=3D0.1%, >=3D64=3D0.0%
     issued rwts: total=3D6109,2083,0,0 short=3D0,0,0,0 dropped=3D0,0,0,0
     latency   : target=3D0, window=3D0, percentile=3D100.00%, depth=3D64

Run status group 0 (all jobs):
   READ: bw=3D1539MiB/s (1614MB/s), 1539MiB/s-1539MiB/s (1614MB/s-1614MB/s)=
, io=3D3055MiB (3203MB), run=3D1985-1985msec
  WRITE: bw=3D525MiB/s (550MB/s), 525MiB/s-525MiB/s (550MB/s-550MB/s), io=
=3D1042MiB (1092MB), run=3D1985-1985msec

Disk stats (read/write):
  sdd: ios=3D5926/2087, merge=3D1/0, ticks=3D119183/3276, in_queue=3D122460=
, util=3D94.87%
--8<---------------cut here---------------end--------------->8---

No idea why btrfs performs so much worse in comparison.

I=E2=80=99ll copy over /gnu/store/trash next.

--=20
Ricardo




Message sent to bug-guix@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: bug#51787: Disk performance on ci.guix.gnu.org
Resent-From: Mathieu Othacehe <othacehe@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Mon, 20 Dec 2021 21:54:01 +0000
Resent-Message-ID: <handler.51787.B51787.164003724130380 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 51787
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: Ricardo Wurmus <rekado@HIDDEN>
Cc: 51787 <at> debbugs.gnu.org
Received: via spool by 51787-submit <at> debbugs.gnu.org id=B51787.164003724130380
          (code B ref 51787); Mon, 20 Dec 2021 21:54:01 +0000
Received: (at 51787) by debbugs.gnu.org; 20 Dec 2021 21:54:01 +0000
Received: from localhost ([127.0.0.1]:51634 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mzQbY-0007ts-Rv
	for submit <at> debbugs.gnu.org; Mon, 20 Dec 2021 16:54:01 -0500
Received: from eggs.gnu.org ([209.51.188.92]:60982)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <othacehe@HIDDEN>) id 1mzQbS-0007tX-9s
 for 51787 <at> debbugs.gnu.org; Mon, 20 Dec 2021 16:53:59 -0500
Received: from [2001:470:142:3::e] (port=55258 helo=fencepost.gnu.org)
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <othacehe@HIDDEN>)
 id 1mzQbM-0005ik-QZ; Mon, 20 Dec 2021 16:53:48 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To:
 From; bh=uTgwFC3wdEK6Obpgdn4wflDZoTIA19yh2eJfX+NHy/g=; b=BM28vqNDzd71YYXEtssX
 bABIGjr8gtvsckskOehLOUhXVajeSojnYi2F3Hiv/jslfNn9C97XdYcU3+4zK/t3FsGrlSL5Rz9C7
 cjc7k9j/hvgqnCCYT/kSaPDzryh9MKrPHOtnpQmtEf0BVWN8GdjTzgRoJWUwvfArqjg6FYpDqCvR6
 jVlNRISLBu/RvztBA/1d7kd887+fuopRzsDw6DjNKedFPSBu60ffM6vC7xdEE9IEfm1Y/dzrSMnmY
 ffLVNQlJ1+TJBA0hs9+hNBDYenrd0GqNNeN+JeAbYgz3Vlcfkz9UxdLaQJOoVIST996FJ6Ak+bxLC
 Ex73s/lZeOruTg==;
Received: from [2a01:cb18:832e:5f00:3563:417e:2a38:86d8] (port=49154
 helo=meije)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <othacehe@HIDDEN>)
 id 1mzQbM-00036T-2H; Mon, 20 Dec 2021 16:53:49 -0500
From: Mathieu Othacehe <othacehe@HIDDEN>
References: <875yrjpi1y.fsf@HIDDEN> <87o85bjjpm.fsf@HIDDEN>
 <871r27p5jq.fsf@HIDDEN>
Date: Mon, 20 Dec 2021 22:53:45 +0100
In-Reply-To: <871r27p5jq.fsf@HIDDEN> (Ricardo Wurmus's message of "Mon,
 20 Dec 2021 18:05:08 +0100")
Message-ID: <87a6gv3pue.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)


Hey,

> Do you mean time the copy or time the removal from that storage?  You
> know what, I=E2=80=99ll time both.  I=E2=80=99ll need to get more space f=
irst.  I think
> the trash directory is larger than the 500G that I got for testing the
> SAN.

Yeah I meant removal time :) I found this article[1] that suggests that
over time the ext4 fragmentation can cause a performance drop that is
very noticeable on hard drives.

I'm trying to determine how fragmented is the sdb1 file-system, by
running e4defrag and e2freefrag[2], but I'm not sure if they will
complete soon.

Copying and removing /gnu/store/trash on the SAN will be interesting but
the ultimate test would be to be able to re-create the ext4 file-system
directly on Berlin's sdb drive to evaluate the fragmentation role in
this funny business.

Thanks,

Mathieu

[1]: https://www.usenix.org/system/files/hotstorage19-paper-conway.pdf
[2]: https://cromwell-intl.com/open-source/performance-tuning/file-systems.=
html




Message sent to bug-guix@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: bug#51787: Disk performance on ci.guix.gnu.org
Resent-From: Ricardo Wurmus <rekado@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Tue, 21 Dec 2021 17:37:02 +0000
Resent-Message-ID: <handler.51787.B51787.164010820321995 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 51787
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: Mathieu Othacehe <othacehe@HIDDEN>
Cc: 51787 <at> debbugs.gnu.org
Received: via spool by 51787-submit <at> debbugs.gnu.org id=B51787.164010820321995
          (code B ref 51787); Tue, 21 Dec 2021 17:37:02 +0000
Received: (at 51787) by debbugs.gnu.org; 21 Dec 2021 17:36:43 +0000
Received: from localhost ([127.0.0.1]:55472 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mzj46-0005ig-Pu
	for submit <at> debbugs.gnu.org; Tue, 21 Dec 2021 12:36:43 -0500
Received: from sender3-of-o51.zoho.com ([136.143.184.51]:21738)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <rekado@HIDDEN>) id 1mzj45-0005iU-Ii
 for 51787 <at> debbugs.gnu.org; Tue, 21 Dec 2021 12:36:42 -0500
ARC-Seal: i=1; a=rsa-sha256; t=1640108191; cv=none; 
 d=zohomail.com; s=zohoarc; 
 b=JDqgyzsAypaq9CMTlFtILwmZ30LLNHb20sJHGPab7Qpu8Lm2oJGc1P6t4XAQD8XieoJQz5IbvAJ4ra9mbRWHlfLAc17K0yX/zdZ3E7vLIT3s8/FEATEhU0L8SLuWA0sS5yFP5TiZ0yzsB8B9rK9u0K/mT77OWw9MqsJzvo1NXa4=
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com;
 s=zohoarc; t=1640108191;
 h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:To;
 bh=jH1Dkp4lD2bICXPSZf+fwlKJ828YTBcl8g3NEzCp1q0=; 
 b=jgubDGrk5DkNle81PDiUxbQWfJIlvJltupTNvAmnlkQgFsokvBECnmfHStkhdInNuVuwGahATwHwKXyDzPx1qcoRYcUX5Gy/AUFelDAr4brQlWL0hSg/yRiF+HKaCIlF1CFqlD2Ks+k5OL8wxb9eYIaeiL/IXZ9Z+0NCeCIv/3o=
ARC-Authentication-Results: i=1; mx.zohomail.com;
 dkim=pass  header.i=elephly.net;
 spf=pass  smtp.mailfrom=rekado@HIDDEN;
 dmarc=pass header.from=<rekado@HIDDEN>
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1640108191; 
 s=zoho; d=elephly.net; i=rekado@HIDDEN;
 h=References:From:To:Cc:Subject:Date:In-reply-to:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding;
 bh=jH1Dkp4lD2bICXPSZf+fwlKJ828YTBcl8g3NEzCp1q0=;
 b=B+2PHfRBTtb7URbYVoHkYqVeXfSCxvGJp1UKt7NnuF2OPkf8lUsghJMJ20LWuHRk
 KawcQHxr8i0me0yjKWZLU8IwP4XKCQoLTvwS9ZZVXWAAjr3loVYejWlLsXK5P5OEIcs
 m0BWehO2IRsSIoZ5aYm3+4cs3pjqtNR+hgnlfKOE=
Received: from localhost (p54ad4c64.dip0.t-ipconnect.de [84.173.76.100]) by
 mx.zohomail.com with SMTPS id 1640108187448384.47795051636797;
 Tue, 21 Dec 2021 09:36:27 -0800 (PST)
References: <875yrjpi1y.fsf@HIDDEN> <87o85bjjpm.fsf@HIDDEN>
 <871r27p5jq.fsf@HIDDEN> <87a6gv3pue.fsf@HIDDEN>
User-agent: mu4e 1.6.10; emacs 27.2
From: Ricardo Wurmus <rekado@HIDDEN>
Date: Tue, 21 Dec 2021 18:26:03 +0100
In-reply-to: <87a6gv3pue.fsf@HIDDEN>
X-URL: https://elephly.net
X-PGP-Key: https://elephly.net/rekado.pubkey
X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
Message-ID: <87v8zhn9m1.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-ZohoMailClient: External
X-Spam-Score: -0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Today we discovered a few more things and discussed them on IRC.  Here=E2=
=80=99s
a summary.

/var/cache sits on the same storage as /gnu.  We mounted the 5TB ext4
file system that=E2=80=99s hosted by the SAN at /mnt_test and started copyi=
ng
over /var/cache to /mnt_test/var/cache.  Transfer speed was considerably
faster (not *great*, but reasonably fast) than the copy of
/gnu/store/trash to the same target.

This confirmed our suspicions that the problem is not with the storage
array but due to the fact that /gnu/store/trash (and also /gnu/store)
is an extremely large, flat directory.  /var/cache is not.

Here=E2=80=99s what we do now: continue copying /var/cache to the SAN, then
remount to serve substitutes from there.  This removes some pressure
from the file system as it will only be used for /gnu.  We=E2=80=99re
considering to dump the file system completely (i.e. reinstall the
server), thereby emptying /gnu, but leaving the stash of built
substitutes in /var/cache (hosted from the faster SAN).

We could take this opportunity to reformat /gnu with btrfs, which
performs quite a bit more poorly than ext4 but would be immune to
defragmentation.  It=E2=80=99s not clear that defragmentation matters here.=
  It
could just be that the problem is exclusively caused by having these
incredibly large, flat /gnu/store, /gnu/store/.links, and
/gnu/store/trash directories.

A possible alternative for this file system might also be XFS, which
performs well when presented with unreasonably large directories.

It may be a good idea to come up with realistic test scenarios that we
could test with each of these three file systems at scale.

Any ideas?

--=20
Ricardo




Message sent to bug-guix@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: bug#51787: Disk performance on ci.guix.gnu.org
Resent-From: Leo Famulari <leo@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Tue, 21 Dec 2021 17:52:01 +0000
Resent-Message-ID: <handler.51787.B51787.164010910623923 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 51787
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: Ricardo Wurmus <rekado@HIDDEN>
Cc: Mathieu Othacehe <othacehe@HIDDEN>, 51787 <at> debbugs.gnu.org
Received: via spool by 51787-submit <at> debbugs.gnu.org id=B51787.164010910623923
          (code B ref 51787); Tue, 21 Dec 2021 17:52:01 +0000
Received: (at 51787) by debbugs.gnu.org; 21 Dec 2021 17:51:46 +0000
Received: from localhost ([127.0.0.1]:55522 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mzjIg-0006Dn-5a
	for submit <at> debbugs.gnu.org; Tue, 21 Dec 2021 12:51:46 -0500
Received: from wout3-smtp.messagingengine.com ([64.147.123.19]:53019)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <leo@HIDDEN>) id 1mzjId-0006DR-Qv
 for 51787 <at> debbugs.gnu.org; Tue, 21 Dec 2021 12:51:44 -0500
Received: from compute4.internal (compute4.nyi.internal [10.202.2.44])
 by mailout.west.internal (Postfix) with ESMTP id B924632007D7;
 Tue, 21 Dec 2021 12:51:37 -0500 (EST)
Received: from mailfrontend2 ([10.202.2.163])
 by compute4.internal (MEProxy); Tue, 21 Dec 2021 12:51:38 -0500
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=famulari.name;
 h=date:from:to:cc:subject:message-id:references:mime-version
 :content-type:content-transfer-encoding:in-reply-to; s=mesmtp;
 bh=wpL1pBUfY4TyiCiU1BIcLzNCyFxMarIjym3Zp3IJZ80=; b=nizKUSfVlrEA
 lbtZ1a/QXvfgUvb89GY5Ya983slBHa8e8neN9Ew/rNZqaTj4wLwrLSQGE9Z5FKpz
 CXOvfZcDb124sMyCMzpWppq9XL0Ore7ncZdf1uThcsbSXhVIrvWruaR5j+At1B8q
 drr1DkltXItBPRKvdv1k4O+s5l7UEfo=
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:content-transfer-encoding:content-type
 :date:from:in-reply-to:message-id:mime-version:references
 :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender
 :x-sasl-enc; s=fm1; bh=wpL1pBUfY4TyiCiU1BIcLzNCyFxMarIjym3Zp3IJZ
 80=; b=Y7rEEaQKUiFRA6D5H823eLLUgl7AQMdT+WBuKFTaGpSAM54jhar53X4Zo
 raWui/XzT0FfzY9sMUH+Z0v/KP7zQq2CR9cZE1XPcefgQR3WBWKqjCGwOt/yGdXx
 XhULH1OhdhGUVB1GazlCsJGTsrit2Uk1gjRd65fcK3ErRfnXv/iUdVJKltV8xmLs
 3O89ICPgaz3ldskNYGaRoYFbtlwz5vSNM7uflkhzIWsQE3l1xjl6P8FH4Fi52DKt
 qu44e4+w/gIhjUDO6FcsGTDlA3HSMFsaPzxgfdJyaNuC6LiVjo29QHggKnryNjGI
 xgxJtRs3NDXvh+n5JCFY/hg5DheNw==
X-ME-Sender: <xms:KRTCYUYzZl-6PTqaAod37p7OCR8WZF8ePFFxhapIUYhpIy-rYmTt2A>
 <xme:KRTCYfbUErEeiRHZ0MF9KX67CdmyJm97yHjV1f1qAsHugt1vvvAF-phqk5e69iK56
 aSR6xshT6j5PpUzJQ>
X-ME-Received: <xmr:KRTCYe8Igu7RH2d1sRN3Cfu9l7k4w1Cgc1AQMxP9UzgodbPcPlCnHV5Zgx8z9b5VaeL8dxSta35D4MYc74jWGUWVag>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvuddruddtgedguddtjecutefuodetggdotefrod
 ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh
 necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd
 enucfjughrpeffhffvuffkfhggtggugfgjsehtkeertddttdejnecuhfhrohhmpefnvgho
 ucfhrghmuhhlrghrihcuoehlvghosehfrghmuhhlrghrihdrnhgrmhgvqeenucggtffrrg
 htthgvrhhnpefgudffteettdekkeduhffgfefgieefgeeuieetudejffelieduueeifffg
 udfgudenucffohhmrghinhepghhnuhdrohhrghenucevlhhushhtvghrufhiiigvpedtne
 curfgrrhgrmhepmhgrihhlfhhrohhmpehlvghosehfrghmuhhlrghrihdrnhgrmhgv
X-ME-Proxy: <xmx:KRTCYerzmyT_1iBVeigv-xiO-serKd5CQHlBt7ZIYP3-K2rYyD1bHQ>
 <xmx:KRTCYfrahrc2Pf-rjTE0blXAQE8Y-l_Ls13ds0bSiGniEuwxMxf1-Q>
 <xmx:KRTCYcR0hnhmLqizlfsaBMB3tXHuQgNigaA35zuXyUrwPc6id16XJw>
 <xmx:KRTCYYDSwUZOueLJkIQm5rBQfHDhC-kPGLZN5yPnfHVfiflZS32isA>
Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue,
 21 Dec 2021 12:51:36 -0500 (EST)
Date: Tue, 21 Dec 2021 12:51:35 -0500
From: Leo Famulari <leo@HIDDEN>
Message-ID: <YcIUJz37YySG/DtC@HIDDEN>
References: <875yrjpi1y.fsf@HIDDEN> <87o85bjjpm.fsf@HIDDEN>
 <871r27p5jq.fsf@HIDDEN> <87a6gv3pue.fsf@HIDDEN>
 <87v8zhn9m1.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <87v8zhn9m1.fsf@HIDDEN>
X-Spam-Score: -0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

On Tue, Dec 21, 2021 at 06:26:03PM +0100, Ricardo Wurmus wrote:
> We could take this opportunity to reformat /gnu with btrfs, which
> performs quite a bit more poorly than ext4 but would be immune to
> defragmentation.  It’s not clear that defragmentation matters here.  It
> could just be that the problem is exclusively caused by having these
> incredibly large, flat /gnu/store, /gnu/store/.links, and
> /gnu/store/trash directories.

My impression was that btrfs could also become fragmented. At least,
btrfs-progrs includes a command for defragmenting. Or do I
misunderstand?




Message sent to bug-guix@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: bug#51787: Disk performance on ci.guix.gnu.org
Resent-From: Mathieu Othacehe <othacehe@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Tue, 21 Dec 2021 18:24:01 +0000
Resent-Message-ID: <handler.51787.B51787.16401110193425 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 51787
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: Ricardo Wurmus <rekado@HIDDEN>
Cc: 51787 <at> debbugs.gnu.org
Received: via spool by 51787-submit <at> debbugs.gnu.org id=B51787.16401110193425
          (code B ref 51787); Tue, 21 Dec 2021 18:24:01 +0000
Received: (at 51787) by debbugs.gnu.org; 21 Dec 2021 18:23:39 +0000
Received: from localhost ([127.0.0.1]:55622 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mzjnX-0000tA-7z
	for submit <at> debbugs.gnu.org; Tue, 21 Dec 2021 13:23:39 -0500
Received: from eggs.gnu.org ([209.51.188.92]:48536)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <othacehe@HIDDEN>) id 1mzjnV-0000sy-4e
 for 51787 <at> debbugs.gnu.org; Tue, 21 Dec 2021 13:23:37 -0500
Received: from [2001:470:142:3::e] (port=60754 helo=fencepost.gnu.org)
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <othacehe@HIDDEN>)
 id 1mzjnP-0003PP-J4; Tue, 21 Dec 2021 13:23:31 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To:
 From; bh=C3dpeCCzYZiMUrYrYhZD0PYoQTvAt6/iG4SGGStFGsg=; b=IidPS+yIsd+MnOw2IzaI
 AUn+CyJG/DbN48PYDbxRHi/2L0B4clI1BQ/+g66VrIo65fabUsnp79AucanNriMn+d5b0jKOrPqLI
 okCWnTDDm5Je9hVRgPqIRUS/K/LYL39bbZJIa4mUiNQ5IVr4lAPGjmZ4XwQKpOMLlE3NRyZqCGPsy
 y6MN7BUNWUqBLSkQdglVRq+seqGFR08mv8vJm2x4qzcQXFfVRmtdZnhhYuCfS87NGVS2LkYe2mgWD
 DcJA9cH4QgOl6fWNtl2O/EOzz11xcXFKHXPvKXrF9jJfgKHBuf1qcvDjhjXkZcUW5MeqJUJ9E6qjx
 P/lI4D8M/7hMIA==;
Received: from [2a01:cb18:832e:5f00:3563:417e:2a38:86d8] (port=49210
 helo=meije)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <othacehe@HIDDEN>)
 id 1mzjnP-0005LD-GZ; Tue, 21 Dec 2021 13:23:31 -0500
From: Mathieu Othacehe <othacehe@HIDDEN>
References: <875yrjpi1y.fsf@HIDDEN> <87o85bjjpm.fsf@HIDDEN>
 <871r27p5jq.fsf@HIDDEN> <87a6gv3pue.fsf@HIDDEN>
 <87v8zhn9m1.fsf@HIDDEN>
Date: Tue, 21 Dec 2021 19:23:28 +0100
In-Reply-To: <87v8zhn9m1.fsf@HIDDEN> (Ricardo Wurmus's message of "Tue,
 21 Dec 2021 18:26:03 +0100")
Message-ID: <874k71g6lb.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)


Hey,

> Today we discovered a few more things and discussed them on IRC.  Here=E2=
=80=99s
> a summary.

Nice summary :)

> We could take this opportunity to reformat /gnu with btrfs, which
> performs quite a bit more poorly than ext4 but would be immune to
> defragmentation.  It=E2=80=99s not clear that defragmentation matters her=
e.  It
> could just be that the problem is exclusively caused by having these
> incredibly large, flat /gnu/store, /gnu/store/.links, and
> /gnu/store/trash directories.
>
> A possible alternative for this file system might also be XFS, which
> performs well when presented with unreasonably large directories.
>
> It may be a good idea to come up with realistic test scenarios that we
> could test with each of these three file systems at scale.

We could compare xfs, btrfs and ext4 performances on a store subset,
1TiB for instance that we would create on the SAN. Realistic test
scenario could be:

- Time the copy of new items to the test store.
- Time the removal of randomly picked items from the test store.
- Time the creation of nar archives from the test store.

That will allow us to choose the file-system that has the best
performances for our use-case, regardless of fragmentation.

Now fragmentation may or may not be a problem as you mentioned. What we
could do is repeat the same tests but on a test store that is created
and removed N times, to simulate file-system aging.

This is more or less what is done in this article[1] by "git pulling" N
times a repository and testing read performances. For them btrfs > xfs >
ext4 in term of performances, but we might draw different conclusions
for our specific use case.

Do you think it is realistic? If so, we can start working on some test
scripts.

Thanks,

Mathieu

[1]: https://www.usenix.org/system/files/hotstorage19-paper-conway.pdf




Message sent to bug-guix@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: bug#51787: Disk performance on ci.guix.gnu.org
Resent-From: Bengt Richter <bokr@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Tue, 21 Dec 2021 23:21:01 +0000
Resent-Message-ID: <handler.51787.B51787.164012884410471 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 51787
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: Ricardo Wurmus <rekado@HIDDEN>
Cc: Mathieu Othacehe <othacehe@HIDDEN>, 51787 <at> debbugs.gnu.org
Reply-To: Bengt Richter <bokr@HIDDEN>
Received: via spool by 51787-submit <at> debbugs.gnu.org id=B51787.164012884410471
          (code B ref 51787); Tue, 21 Dec 2021 23:21:01 +0000
Received: (at 51787) by debbugs.gnu.org; 21 Dec 2021 23:20:44 +0000
Received: from localhost ([127.0.0.1]:56160 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mzoR2-0002io-2r
	for submit <at> debbugs.gnu.org; Tue, 21 Dec 2021 18:20:44 -0500
Received: from imta-36.everyone.net ([216.200.145.36]:51576
 helo=imta-38.everyone.net)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <bokr@HIDDEN>) id 1mzoQz-0002ic-Hr
 for 51787 <at> debbugs.gnu.org; Tue, 21 Dec 2021 18:20:42 -0500
Received: from pps.filterd (omta002.sj2.proofpoint.com [127.0.0.1])
 by imta-38.everyone.net (8.16.0.43/8.16.0.43) with SMTP id 1BLNItAo029320;
 Tue, 21 Dec 2021 15:20:39 -0800
X-Eon-Originating-Account: i9_lel_50rLmNMFnfTwMF5dgksd13XQGCXzUboYiffs
X-Eon-Dm: m0117124.ppops.net
Received: by m0117124.mta.everyone.net (EON-AUTHRELAY2 - 53b92f98)
 id m0117124.6195d1ae.28b54c; Tue, 21 Dec 2021 15:20:37 -0800
X-Eon-Sig: AQMHrIJhwmFFNJJjkwIAAAAD,9402e946dd9158a37aa9c87c691908ec
X-Eip: pK5W_qA8Izdfz3RPTY6r50YbJvK_DaTYHrb3kbvsItQ
Date: Wed, 22 Dec 2021 00:20:24 +0100
From: Bengt Richter <bokr@HIDDEN>
Message-ID: <20211221232024.GA41746@LionPure>
References: <875yrjpi1y.fsf@HIDDEN> <87o85bjjpm.fsf@HIDDEN>
 <871r27p5jq.fsf@HIDDEN> <87a6gv3pue.fsf@HIDDEN>
 <87v8zhn9m1.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <87v8zhn9m1.fsf@HIDDEN>
User-Agent: Mutt/1.10.1 (2018-07-13)
X-Proofpoint-GUID: 3bv8ji_1oRleO6-mugcqY5_-fmQGCT6f
X-Proofpoint-ORIG-GUID: 3bv8ji_1oRleO6-mugcqY5_-fmQGCT6f
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.425, 18.0.790
 definitions=2021-12-21_07:2021-12-21,
 2021-12-21 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0
 impostorscore=0
 lowpriorityscore=0 adultscore=0 bulkscore=0 spamscore=0 phishscore=0
 mlxlogscore=804 mlxscore=0 malwarescore=0 clxscore=1034 priorityscore=1501
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2110150000
 definitions=main-2112210116
X-Spam-Score: 0.2 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.8 (/)

Hi Ricardo,

TL;DR: re: "Any ideas?" :)

Read this [0], and consider how file systems may be
interacting with with SSD wear-leveling algorithms.

Are some file systems dependent on successful speculative
transaction continuations, while others might slow down
waiting for signs that an SSD controller has committed one
of ITS transactions, e.g. in special cases where the user or
kernel file system wants to be sure metadata is
written/journaled for fs structural integrity, but maybe
cares less about data?

I guess this difference might show up in copying a large
file over-writing the same target file (slower) vs copying
to a series of new files (faster).

What happens if you use a contiguous file as swap space?

Or, if you use anonymous files as user data space buffers,
passing them to wayland as file handles, per its protocol,
can you do better than ignoring SSD controllers and/or
storage hardware altogether?

Reference [0] is from 2013, so probably much has happened
since then, and the paper mentions (which has probably not
gotten better), the following, referring to trade secrets
giving one manufacturer ability to produce longer-lasting
SSDs cheaper and better than others ...

--8<---------------cut here---------------start------------->8---
    This means that the SSD controller is dedicated to a
    single brand of NAND, and it means that the SSD maker
    can’t shop around among NAND suppliers for the best price.
    Furthermore, the NAND supplier won’t share this
    information unless it believes that there is some compelling
    reason to work the SSD manufacturer. Since there are
    hundreds of SSD makers it’s really difficult to get these
    companies to pay attention to you! The SSD manufacturers
    that have this kind of relationship with their flash
    suppliers are very rare and very special.
--8<---------------cut here---------------end--------------->8---

Well, maybe you will have to parameterize your file system
tuning with manufacturer ID and SSD controller firmware
version ;/

Mvh, Bengt

[0] https://www.snia.org/sites/default/files/SSSITECHNOTES_HowControllersMaximizeSSDLife.pdf

On +2021-12-21 18:26:03 +0100, Ricardo Wurmus wrote:
> Today we discovered a few more things and discussed them on IRC.  Here’s
> a summary.
> 
> /var/cache sits on the same storage as /gnu.  We mounted the 5TB ext4
> file system that’s hosted by the SAN at /mnt_test and started copying
> over /var/cache to /mnt_test/var/cache.  Transfer speed was considerably
> faster (not *great*, but reasonably fast) than the copy of
> /gnu/store/trash to the same target.
> 
> This confirmed our suspicions that the problem is not with the storage
> array but due to the fact that /gnu/store/trash (and also /gnu/store)
> is an extremely large, flat directory.  /var/cache is not.
> 
> Here’s what we do now: continue copying /var/cache to the SAN, then
> remount to serve substitutes from there.  This removes some pressure
> from the file system as it will only be used for /gnu.  We’re
> considering to dump the file system completely (i.e. reinstall the
> server), thereby emptying /gnu, but leaving the stash of built
> substitutes in /var/cache (hosted from the faster SAN).
> 
> We could take this opportunity to reformat /gnu with btrfs, which
> performs quite a bit more poorly than ext4 but would be immune to
> defragmentation.  It’s not clear that defragmentation matters here.  It
> could just be that the problem is exclusively caused by having these
> incredibly large, flat /gnu/store, /gnu/store/.links, and
> /gnu/store/trash directories.
> 
> A possible alternative for this file system might also be XFS, which
> performs well when presented with unreasonably large directories.
> 
> It may be a good idea to come up with realistic test scenarios that we
> could test with each of these three file systems at scale.
> 
> Any ideas?
> 
> -- 
> Ricardo
> 
> 
> 
(sorry, the top-post grew)
-- 
Regards,
Bengt Richter





Last modified: Tue, 21 Dec 2021 23:30:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.