X-Loop: help-debbugs@HIDDEN
Subject: bug#48005: low performance when building profile of already built packages
Resent-From: Ricardo Wurmus <rekado@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Sat, 24 Apr 2021 20:40:02 +0000
Resent-Message-ID: <handler.48005.B.161929675628125 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: report 48005
X-GNU-PR-Package: guix
X-GNU-PR-Keywords:
To: 48005 <at> debbugs.gnu.org
X-Debbugs-Original-To: bug-guix@HIDDEN
Received: via spool by submit <at> debbugs.gnu.org id=B.161929675628125
(code B ref -1); Sat, 24 Apr 2021 20:40:02 +0000
Received: (at submit) by debbugs.gnu.org; 24 Apr 2021 20:39:16 +0000
Received: from localhost ([127.0.0.1]:41580 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1laP3b-0007JZ-Qb
for submit <at> debbugs.gnu.org; Sat, 24 Apr 2021 16:39:16 -0400
Received: from lists.gnu.org ([209.51.188.17]:41314)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <rekado@HIDDEN>) id 1laP3Y-0007JQ-8W
for submit <at> debbugs.gnu.org; Sat, 24 Apr 2021 16:39:14 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:39280)
by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
(Exim 4.90_1) (envelope-from <rekado@HIDDEN>)
id 1laP3X-0008DT-SG
for bug-guix@HIDDEN; Sat, 24 Apr 2021 16:39:12 -0400
Received: from sender4-of-o51.zoho.com ([136.143.188.51]:21154)
by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
(Exim 4.90_1) (envelope-from <rekado@HIDDEN>)
id 1laP3V-0004Q0-Pq
for bug-guix@HIDDEN; Sat, 24 Apr 2021 16:39:11 -0400
ARC-Seal: i=1; a=rsa-sha256; t=1619296743; cv=none;
d=zohomail.com; s=zohoarc;
b=R3t2kuMFLHPUvYA7S8aodivRH/PrhHdXntQhQEBmUC/hBcwujNUFb1/9zpuPk4BMVu8cHS/N51uo46tHS9Ta7VjICb8UltAOZYaWd9/gt7TWfXiqRjW91o/OEij1e4gMeyLpN7lqw81r5eFmVqaz1tdJai7IeOHcYqybJJO9Pi0=
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com;
s=zohoarc; t=1619296743;
h=Content-Type:Content-Transfer-Encoding:Date:From:MIME-Version:Message-ID:Subject:To;
bh=sI0vQYrIZpHYJmB4h9CVDSgtiWmeKqAl4eoi3lHCYXo=;
b=Y8mJ5OiZl+bm5sEOjIU7M+NvFnQeO3nUeGX8QDKAIH+/YFyuMG6NRmGB3Z+HreyT12dsWwI9i3hSFNtj+NqJJMttFNbUPrMTX+S9FQemB+/a+NMwbUUHZUgAoWTnJZ5EnGU++xmdv5HY1RgK7BimOOGnRWc0dh5nwePgQUDY90Q=
ARC-Authentication-Results: i=1; mx.zohomail.com;
dkim=pass header.i=elephly.net;
spf=pass smtp.mailfrom=rekado@HIDDEN;
dmarc=pass header.from=<rekado@HIDDEN> header.from=<rekado@HIDDEN>
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1619296743;
s=zoho; d=elephly.net; i=rekado@HIDDEN;
h=From:To:Subject:Date:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding;
bh=sI0vQYrIZpHYJmB4h9CVDSgtiWmeKqAl4eoi3lHCYXo=;
b=aMWInhCrxsRskEgeLrVKlPQ8WWgip4GFSurjzjh/6IV9oqYMIrau5vus240dumOe
errVOq1f3YHoNh7z0ulbPpB7gHoxACV9SolEvNQAdTSnA12EmJNC0OVpr0iPbIU1mlc
6sV5GXei1PUk9/B6X5gsd2Z4RcEyICpZDjIFNRNw=
Received: from localhost (p4fd5ab0c.dip0.t-ipconnect.de [79.213.171.12]) by
mx.zohomail.com with SMTPS id 1619296741170139.57686629621355;
Sat, 24 Apr 2021 13:39:01 -0700 (PDT)
User-agent: mu4e 1.4.15; emacs 27.2
From: Ricardo Wurmus <rekado@HIDDEN>
X-URL: https://elephly.net
X-PGP-Key: https://elephly.net/rekado.pubkey
X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC
Date: Sat, 24 Apr 2021 22:38:58 +0200
Message-ID: <87zgxnpgd9.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: quoted-printable
X-ZohoMailClient: External
Received-SPF: pass client-ip=136.143.188.51; envelope-from=rekado@HIDDEN;
helo=sender4-of-o51.zoho.com
X-Spam_score_int: -20
X-Spam_score: -2.1
X-Spam_bar: --
X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,
DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1,
RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001,
SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-Spam-Score: -1.3 (-)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -2.3 (--)
Here is a manifest that isn=E2=80=99t unusual for a bioinformatics=20
workflow:
--8<---------------cut here---------------start------------->8---
(define packages
(list "bash-minimal"
"fastqc"
"gzip"
"htseq"
"multiqc"
"r-corrplot"
"r-crosstalk"
"r-data-table"
"r-deseq2"
"r-dt"
"r-genomicalignments"
"r-genomicranges"
"r-ggplot2"
"r-ggrepel"
"r-gprofiler"
"r-knitr"
"r-pheatmap"
"r-plotly"
"r-reshape2"
"r-rmarkdown"
"r-rsamtools"
"r-rtracklayer"
"r-s4vectors"
"r-scales"
"r-summarizedexperiment"
"r-tximport"
"samtools"
"sed"
"trim-galore"))
(specifications->manifest packages)
--8<---------------cut here---------------end--------------->8---
When all these packages already exist and nothing is to be done=20
when Guix is asked to instantiate this manifest, more than 13=20
seconds pass.
--8<---------------cut here---------------start------------->8---
$ [env] export GUIX_PROFILING=3D"object-cache=20
add-data-to-store-cache rpc"
$ [env] time guix package -m manifest-test.scm -p /tmp/foo
The following packages will be installed:
bash-minimal 5.0.16
fastqc 0.11.5
gzip 1.10
htseq 0.9.1
multiqc 1.5
r-corrplot 0.84
r-crosstalk 1.1.1
r-data-table 1.14.0
r-deseq2 1.30.1
r-dt 0.17
r-genomicalignments 1.26.0
r-genomicranges 1.42.0
r-ggplot2 3.3.3
r-ggrepel 0.9.1
r-gprofiler 0.7.0
r-knitr 1.31
r-pheatmap 1.0.12
r-plotly 4.9.3
r-reshape2 1.4.4
r-rmarkdown 2.7
r-rsamtools 2.6.0
r-rtracklayer 1.50.0
r-s4vectors 0.28.1
r-scales 1.1.1
r-summarizedexperiment 1.20.0
r-tximport 1.18.0
samtools 1.11
sed 4.8
trim-galore 0.6.6
[[most of the time passes here]]
nothing to be done
Store object cache:
fresh caches: 2
lookups: 45493
hits: 43943 (96.6%)
cache size: 1549 entries
'add-data-to-store' cache:
lookups: 7755
hits: 3411 (44.0%)
.drv files: 4357 (56.2%)
Scheme files: 3343 (43.1%)
Remote procedure call summary: 5283 RPCs
built-in-builders ... 1
add-to-store/tree ... 27
add-to-store ... 213
query-references ... 698
add-text-to-store ... 4344
real 0m13.379s
user 0m16.414s
sys 0m0.253s
--8<---------------cut here---------------end--------------->8---
Some of these R packages have a lot of transitive inputs, such as=20
r-deseq2 with 81 transitive propagated inputs. So there=E2=80=99s a lot=20
of propagation here, which I suspect contributes to things being=20
so unexpectedly slow.
This report may be related to performance issues reported for=20
=E2=80=9Cguix environment=E2=80=9D in https://issues.guix.gnu.org/41702, wh=
ich=20
also involves R packages (such as r-learnr with 46 transitive=20
propagated inputs).
--=20
Ricardo
Content-Disposition: inline Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Mailer: MIME-tools 5.505 (Entity 5.505) Content-Type: text/plain; charset=utf-8 X-Loop: help-debbugs@HIDDEN From: help-debbugs@HIDDEN (GNU bug Tracking System) To: Ricardo Wurmus <rekado@HIDDEN> Subject: bug#48005: Acknowledgement (low performance when building profile of already built packages) Message-ID: <handler.48005.B.161929675628125.ack <at> debbugs.gnu.org> References: <87zgxnpgd9.fsf@HIDDEN> X-Gnu-PR-Message: ack 48005 X-Gnu-PR-Package: guix Reply-To: 48005 <at> debbugs.gnu.org Date: Sat, 24 Apr 2021 20:40:02 +0000 Thank you for filing a new bug report with debbugs.gnu.org. This is an automatically generated reply to let you know your message has been received. Your message is being forwarded to the package maintainers and other interested parties for their attention; they will reply in due course. Your message has been sent to the package maintainer(s): bug-guix@HIDDEN If you wish to submit further information on this problem, please send it to 48005 <at> debbugs.gnu.org. Please do not send mail to help-debbugs@HIDDEN unless you wish to report a problem with the Bug-tracking system. --=20 48005: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D48005 GNU Bug Tracking System Contact help-debbugs@HIDDEN with problems
Received: (at control) by debbugs.gnu.org; 30 Apr 2021 15:46:01 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Fri Apr 30 11:46:01 2021 Received: from localhost ([127.0.0.1]:59615 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1lcVL7-0005aw-5f for submit <at> debbugs.gnu.org; Fri, 30 Apr 2021 11:46:01 -0400 Received: from eggs.gnu.org ([209.51.188.92]:40666) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <ludo@HIDDEN>) id 1lcVL5-0005Uc-IM for control <at> debbugs.gnu.org; Fri, 30 Apr 2021 11:45:59 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:49811) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from <ludo@HIDDEN>) id 1lcVL0-0007b2-CT for control <at> debbugs.gnu.org; Fri, 30 Apr 2021 11:45:54 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=40476 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from <ludo@HIDDEN>) id 1lcVKz-0000Jg-Rn for control <at> debbugs.gnu.org; Fri, 30 Apr 2021 11:45:54 -0400 Date: Fri, 30 Apr 2021 17:45:52 +0200 Message-Id: <87lf8zbwsv.fsf@HIDDEN> To: control <at> debbugs.gnu.org From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN> Subject: control message for bug #48005 MIME-version: 1.0 Content-type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: control X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -3.3 (---) severity 48005 important quit
Received: (at control) by debbugs.gnu.org; 27 May 2021 16:32:57 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Thu May 27 12:32:57 2021 Received: from localhost ([127.0.0.1]:52678 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1lmIwL-0002RM-BM for submit <at> debbugs.gnu.org; Thu, 27 May 2021 12:32:57 -0400 Received: from mail3-relais-sop.national.inria.fr ([192.134.164.104]:38923) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <ludo@HIDDEN>) id 1lmIwJ-0002R4-Np for control <at> debbugs.gnu.org; Thu, 27 May 2021 12:32:56 -0400 X-IronPort-AV: E=Sophos;i="5.83,227,1616454000"; d="scan'208";a="382669827" Received: from 91-160-117-201.subs.proxad.net (HELO ribbon) ([91.160.117.201]) by mail3-relais-sop.national.inria.fr with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 27 May 2021 18:32:49 +0200 Date: Thu, 27 May 2021 18:32:48 +0200 Message-Id: <87wnrkjfwv.fsf@HIDDEN> To: control <at> debbugs.gnu.org From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN> Subject: control message for bug #41702 MIME-version: 1.0 Content-type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -1.3 (-) X-Debbugs-Envelope-To: control X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -2.3 (--) merge 41702 48005 quit
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997 nCipher Corporation Ltd,
1994-97 Ian Jackson.