Received: (at 44053) by debbugs.gnu.org; 19 Oct 2020 08:19:02 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Mon Oct 19 04:19:02 2020 Received: from localhost ([127.0.0.1]:39888 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1kUQNh-00038D-Ph for submit <at> debbugs.gnu.org; Mon, 19 Oct 2020 04:19:02 -0400 Received: from mail-qv1-f54.google.com ([209.85.219.54]:44235) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <zimon.toutoune@HIDDEN>) id 1kUQNg-00037r-30 for 44053 <at> debbugs.gnu.org; Mon, 19 Oct 2020 04:19:00 -0400 Received: by mail-qv1-f54.google.com with SMTP id s17so3946293qvr.11 for <44053 <at> debbugs.gnu.org>; Mon, 19 Oct 2020 01:19:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=Y6R7bUOsVCVV3n9qJcG28DjVtCwbDazSaRmy3kDaSG0=; b=GPb0RrK7C0iPgUGLcXbToDplXf/lHDFkl6/GB4/SybLqIoZSdBTU/QssQo2Gp3iV6T R0eNwU1dQmhMukU5u2zqoj6a5g5q5VjBLT4kITP7i6Qsf+XjAHXpV2WwZhW19/p1LPAA vZ5uwVbebCh/mluUqLUPqBvHIRFGjbQiilc0auf6To/v2fHq3J2D6mou1U5+llHJz+pP EWsLCpryO/EQQXeTOK4boQgqRd2pLxqqfFhsqBybTKWzxp7hkJYvW/0GdMxScXcVn29Y 747d41RxHtOrCiJH5P4Dsy7EAcYSroNxK7s9I4GEbUoqoP+cRRMqcl0Hr1C9Q9hWVgAw GSRQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=Y6R7bUOsVCVV3n9qJcG28DjVtCwbDazSaRmy3kDaSG0=; b=pll6TYty60k9tS9T9NHX8pTelUgjWrBK1iaxvaiZjelTlgCa+uqpGxDCIuqutSuIwl SVnIFjK92qsbdiog3ZvkK8I8HboBvw1alKenzcXD5AzAK+wODX3aVqX283Y3NW/rzavH /OLQwjCJIadpiT4CEhHSaJWTUSBL/WTqjvgf6aUPk6x4ZjBYWDz82Uav9Ru5l6n8MXEX 3dzX3Qa8wU13a/xvCK5oiDS4keatjwtKSXLunBnPlsx6oitQ9Al6DFMRnfoa9cFsdSVO Jqerc0eaQgyRcBTFn8ZlxUnhPLRydAkuuTOkXd/8cHEhDX7UWmKbB3zFCYufXLe489IT THHQ== X-Gm-Message-State: AOAM5329VsTu3w1ZDaS4OzBVRXTKWgT/wCMG7r2AQ2cw7lm/TKlYKUS0 Qr6MuIXz531vPrGbq/8ImL03qhpeuokFAMHNZYw= X-Google-Smtp-Source: ABdhPJzPPPUiakSLPW6PDy7lI0XGMlrd4bPjiC5t6AtHIfUbu9ssUE5NwE0BJYPgdIWcqZawcWaZWkkKA6k9ebNQ4yw= X-Received: by 2002:ad4:4eaf:: with SMTP id ed15mr16304955qvb.40.1603095534641; Mon, 19 Oct 2020 01:18:54 -0700 (PDT) MIME-Version: 1.0 References: <87zh4k435a.fsf@HIDDEN> In-Reply-To: <87zh4k435a.fsf@HIDDEN> From: zimoun <zimon.toutoune@HIDDEN> Date: Mon, 19 Oct 2020 10:18:43 +0200 Message-ID: <CAJ3okZ1AMUteo3qpAHb0m6RS3ApawCrdfQwVNkDg6Xtc515LcA@HIDDEN> Subject: Re: bug#44053: Poor profile generation performance on spinning disks To: Maxim Cournoyer <maxim.cournoyer@HIDDEN> Content-Type: text/plain; charset="UTF-8" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 44053 Cc: 44053 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) Hi Maxim, On Sun, 18 Oct 2020 at 05:37, Maxim Cournoyer <maxim.cournoyer@HIDDEN> wrote: > On my home machine, will is still relying on 2 x 1 TB spinning drives in > RAID1, rebuilding my user profile, which contains 182 entries, takes on > average about 20 minutes, even when there are no packages to be built: The database build time seems correlated to the number of packages in the very profile. Well, I have not benchmarked to see if it is linear, quadratic; or worse or better. > --8<---------------cut here---------------start------------->8--- [...] > building XDG MIME database... > successfully built /gnu/store/cir84qj587i6is4akgqand7ahg9bj938-xdg-mime-database.drv > successfully built /gnu/store/cir84qj587i6is4akgqand7ahg9bj938-xdg-mime-database.drv > /gnu/store/j0bznlj2ibnhirijhnwpkkxzz4qfk8wb-xdg-mime-database > > real 1m7.344s > user 0m1.331s > sys 0m0.053s > --8<---------------cut here---------------end--------------->8--- Well, it is all clear to me, but I do not know if we can do better because these 2 XDG updates seem relying on the external binaries "/bin/update-desktop-database" and "/bin/update-mime-database". I have not timed the function 'xdg-desktop-database' but all the time should be spent on these, I bet. :-) All the best, simon
bug-guix@HIDDEN:bug#44053; Package guix.
Full text available.Received: (at 44053) by debbugs.gnu.org; 18 Oct 2020 23:05:29 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Sun Oct 18 19:05:29 2020 Received: from localhost ([127.0.0.1]:39334 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1kUHk1-0001ZQ-Hm for submit <at> debbugs.gnu.org; Sun, 18 Oct 2020 19:05:29 -0400 Received: from mail-40141.protonmail.ch ([185.70.40.141]:28648) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <luis.felipe.la@HIDDEN>) id 1kUHjy-0001ZC-W4 for 44053 <at> debbugs.gnu.org; Sun, 18 Oct 2020 19:05:28 -0400 Date: Sun, 18 Oct 2020 23:05:10 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=protonmail.com; s=protonmail; t=1603062320; bh=NBffpbsu1l1qROCFvViBKCQyH7ZKm47CQ/8YUHWpvBw=; h=Date:To:From:Reply-To:Subject:From; b=Iz2M130LZlNFBHWgLS83LQZT7y5Q8JuTIeMu2/27/9RHO5TVFMDtb9XOdGm8Hbr8V TIbcH2fnUVXzwQAgYass8vu5X3oSGrJ4tckcbnDmQaKquTgbTdPksS8yvFComngvF+ OoFJbRK96iMyhavDPdkHl50CV1acbQjowABgeSfU= To: "44053 <at> debbugs.gnu.org" <44053 <at> debbugs.gnu.org> From: Luis Felipe <luis.felipe.la@HIDDEN> Subject: Poor profile generation performance on spinning disks Message-ID: <yuID-gVO5clS0T8k81R_AzM7g91lm_6kDlH9rJFE8OOv5XMxEsdFo6hCH_QGnJGMW4nyDfuduHS_cXzKOm0qov8BalGmi8NPGKwR_eInGFk=@protonmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Status: No, score=-1.2 required=10.0 tests=ALL_TRUSTED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM shortcircuit=no autolearn=disabled version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on mailout.protonmail.ch X-Spam-Score: 2.7 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: I've never used SDDs, so I've been experiencing this since I installed Guix (~5 years ago). And yes, building the profile seems to be the more resource intensive operation. I usually can't do anything [...] Content analysis details: (2.7 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_MSPIKE_H2 RBL: Average reputation (+2) [185.70.40.141 listed in wl.mailspike.net] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider (luis.felipe.la[at]protonmail.com) -0.0 SPF_HELO_PASS SPF: HELO matches SPF record 2.7 AC_FROM_MANY_DOTS Multiple periods in From user name X-Debbugs-Envelope-To: 44053 X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Reply-To: Luis Felipe <luis.felipe.la@HIDDEN> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) I've never used SDDs, so I've been experiencing this since I installed Guix= (~5 years ago). And yes, building the profile seems to be the more resourc= e intensive operation. I usually can't do anything else while that happens = because the computer becomes unresponsive. Running the same command Maxim mentioned took the following time in my case= : $ time guix package -i perl --max-jobs=3D1 Se instalar=C3=A1 el siguiente paquete: =C2=A0=C2=A0 perl 5.30.2 Se construir=C3=A1 la siguiente derivaci=C3=B3n: =C2=A0=C2=A0 /gnu/store/86g48nh8dhgdhzb8r9bxjk6mmszlssss-profile.drv construyendo empaquetado de certificados de CA... construyendo el directorio de tipograf=C3=ADas... generando la cach=C3=A9 de esquemas de GLib... creando la cach=C3=A9 de temas de iconos de GTK+... construyendo los ficheros de cach=C3=A9 para los m=C3=A9todos de entrada de= GTK+... construyendo el directorio de manuales Info... construyendo la base de datos de p=C3=A1ginas de manual... construyendo la cach=C3=A9 de ficheros desktop XDG... construyendo la base de datos MIME XDG... construyendo perfil con 87 paquetes... real=C2=A0=C2=A0=C2=A0 8m3,043s user=C2=A0=C2=A0=C2=A0 0m2,656s sys=C2=A0=C2=A0=C2=A0 0m0,239s My computer: Intel=C2=AE Core=E2=84=A2 i3-8100 CPU @ 3.60GHz =C3=97 4 RAM: 4 GiB 1 TB HDD
bug-guix@HIDDEN:bug#44053; Package guix.
Full text available.Received: (at submit) by debbugs.gnu.org; 18 Oct 2020 03:36:10 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Sat Oct 17 23:36:10 2020 Received: from localhost ([127.0.0.1]:35973 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1kTzUQ-0005c9-DS for submit <at> debbugs.gnu.org; Sat, 17 Oct 2020 23:36:10 -0400 Received: from lists.gnu.org ([209.51.188.17]:55440) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <maxim.cournoyer@HIDDEN>) id 1kTzUP-0005c1-3R for submit <at> debbugs.gnu.org; Sat, 17 Oct 2020 23:36:09 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:50610) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from <maxim.cournoyer@HIDDEN>) id 1kTzUO-0000FZ-TE for bug-guix@HIDDEN; Sat, 17 Oct 2020 23:36:08 -0400 Received: from mail-io1-xd31.google.com ([2607:f8b0:4864:20::d31]:33072) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from <maxim.cournoyer@HIDDEN>) id 1kTzUL-0003wx-8e for bug-guix@HIDDEN; Sat, 17 Oct 2020 23:36:08 -0400 Received: by mail-io1-xd31.google.com with SMTP id r4so9101837ioh.0 for <bug-guix@HIDDEN>; Sat, 17 Oct 2020 20:36:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id:mime-version; bh=EQL4wUvP0JzwEWDcs8iEsIPm/XDUsVG66N4fK8x/OEI=; b=o7GDX4qfbo3wYyLroWwt9kI3lXhypiQ1dvg8OkHLF8Y3cs7BvcKiEbcgS/i6ZyiUYK rO6q/gtm28vv87WqwGJUVkgIkhQKMxjS2u+zbKYzji/laPjoRnE3RuzeUh894wAQkRIf 29O4ywFs4X2XObC621gfkqq1yZiTIjbbFf6dJAJDKCGbjxOcQ5QChiWdYEqLm3r4fVoN YUHJXJi4nci2tmZq+Anyex+4bhovorzKpsp6/tMtAmwYLvmyPeEGFwQr1IKWppaJi9AV d51Q+cqwZdSnG5S6CQdwNfhN1vHqkObMYZICCJowO1Snh/blL6CxjlP/7eqVzk98jlHM uvXg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:mime-version; bh=EQL4wUvP0JzwEWDcs8iEsIPm/XDUsVG66N4fK8x/OEI=; b=Y3s32c0pf67qzdJy+7Q+pAFX7f1MFzqkJPTwnXS7RRWsJz3t4qjH0Gyu1+eC13H9lc vw/xbqmJso5CJ7ve8WKnY/YsRqcN+x7vupkwiRfEBp0zoyse/bGRxlveT7Urqxk+tIaD 3is05SdSsLngz8M9pK4AnPV9ZkAOfFQk3/VKFCqdKw1usYDHOn5xU/GKLYdi9jwsC7uM 73IWX2qw0nKiUcMdKOi9DFFamrDL3/UUyn1ejWFu72S/nNRZxS6yMU4Xpp9X4lZXpyTF cBnxcpc+aFbQwQp0w2/cefwdbmTC1s8kVoTgMdU6mlU8ol6dyIcqFs1g3QSx2bcU4+s7 7IYg== X-Gm-Message-State: AOAM531uVN26Ug9u08kGzjm7KV59z4ZgCw5gLLLkeBB/UbrP/9p4IfOs kddgHgSgCSYOeOUL0l459MiCARcWes4= X-Google-Smtp-Source: ABdhPJw8iYb3PoK5GTIrwuSgnOB5FZnuKUtOQOZc/z72kyn0mOKvUZsy85AYFr+8prXyTXydKo8D6g== X-Received: by 2002:a6b:1542:: with SMTP id 63mr7022979iov.64.1602992163761; Sat, 17 Oct 2020 20:36:03 -0700 (PDT) Received: from hurd (dsl-205-233-125-118.b2b2c.ca. [205.233.125.118]) by smtp.gmail.com with ESMTPSA id t10sm6408050iog.49.2020.10.17.20.36.02 for <bug-guix@HIDDEN> (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 17 Oct 2020 20:36:03 -0700 (PDT) From: Maxim Cournoyer <maxim.cournoyer@HIDDEN> To: bug-guix <bug-guix@HIDDEN> Subject: Poor profile generation performance on spinning disks Date: Sat, 17 Oct 2020 23:35:45 -0400 Message-ID: <87zh4k435a.fsf@HIDDEN> MIME-Version: 1.0 Content-Type: text/plain Received-SPF: pass client-ip=2607:f8b0:4864:20::d31; envelope-from=maxim.cournoyer@HIDDEN; helo=mail-io1-xd31.google.com X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache. That's all we know. X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Spam-Score: 0.7 (/) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -2.3 (--) Hello! I've noticed on multiple occasions that using Guix on traditional spinning drives can be quite slow. On my home machine, will is still relying on 2 x 1 TB spinning drives in RAID1, rebuilding my user profile, which contains 182 entries, takes on average about 20 minutes, even when there are no packages to be built: --8<---------------cut here---------------start------------->8--- $ time guix package -i perl --max-jobs=1 The following package will be upgraded: perl (dependencies or package changed) The following derivation will be built: /gnu/store/lhywla1z2zcz16df4hbvvvngr9zmswr7-profile.drv building CA certificate bundle... building fonts directory... generating GLib schema cache... creating GTK+ icon theme cache... building cache files for GTK+ input methods... building directory of Info manuals... building database for manual pages... building XDG desktop file cache... building XDG MIME database... building profile with 182 packages... real 19m0.126s user 0m5.648s sys 0m0.333s --8<---------------cut here---------------end--------------->8--- Most of the time remains spent after the message 'building profile with 182 package...'. That part seems IO-bound, with the spinning disks grinding heavily and the CPU mostly idling. The rest of the time (3 minutes), was used by the profile hooks. The same operation on a second, more modern machine equipped with M2 SSDs does much better and takes about 1 minute to accomplish the same, so it seems the bad performance can be mostly attributed to the much slower disk seek times of the spinning disks. On the older machine, two profile hooks are also sticking out w.r.t. the time they take (they take more than one minute opposed to a few seconds): --8<---------------cut here---------------start------------->8--- The following profile hook will be built: /gnu/store/08fanpydi7z4i3qnlqbr8iz23zdgsamw-manual-database.drv building database for manual pages... Creating manual page database... [2139/2139] building list of man-db entries... 175322 entries processed in 95.1 s successfully built /gnu/store/08fanpydi7z4i3qnlqbr8iz23zdgsamw-manual-database.drv successfully built /gnu/store/08fanpydi7z4i3qnlqbr8iz23zdgsamw-manual-database.drv /gnu/store/wzp4mk2r7r4ysciw74gqbfkyai0zmrcc-manual-database real 1m36.378s user 0m1.674s sys 0m0.108s The following profile hook will be built: /gnu/store/cir84qj587i6is4akgqand7ahg9bj938-xdg-mime-database.drv building XDG MIME database... successfully built /gnu/store/cir84qj587i6is4akgqand7ahg9bj938-xdg-mime-database.drv successfully built /gnu/store/cir84qj587i6is4akgqand7ahg9bj938-xdg-mime-database.drv /gnu/store/j0bznlj2ibnhirijhnwpkkxzz4qfk8wb-xdg-mime-database real 1m7.344s user 0m1.331s sys 0m0.053s --8<---------------cut here---------------end--------------->8--- So we should profile what's going on while generating the profile (no pun intended) and try to improve this at first since this is where most of the time is spent on spinning drives (17 minutes out of the 20 in the above example). After that we could look into the two above profile hooks. Thanks, Maxim
Maxim Cournoyer <maxim.cournoyer@HIDDEN>:bug-guix@HIDDEN.
Full text available.bug-guix@HIDDEN:bug#44053; Package guix.
Full text available.
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997 nCipher Corporation Ltd,
1994-97 Ian Jackson.