GNU bug report logs - #44175
[optimization] Grafting is too slow

Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.
Package: guix; Reported by: Maxim Cournoyer <maxim.cournoyer@HIDDEN>; dated Fri, 23 Oct 2020 13:08:01 UTC; Maintainer for guix is bug-guix@HIDDEN.

Message received at 44175 <at> debbugs.gnu.org:


Received: (at 44175) by debbugs.gnu.org; 25 Oct 2020 16:43:23 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Oct 25 12:43:23 2020
Received: from localhost ([127.0.0.1]:37484 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1kWj75-0004fL-8u
	for submit <at> debbugs.gnu.org; Sun, 25 Oct 2020 12:43:23 -0400
Received: from eggs.gnu.org ([209.51.188.92]:60102)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1kWj73-0004f7-59
 for 44175 <at> debbugs.gnu.org; Sun, 25 Oct 2020 12:43:21 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:37194)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <ludo@HIDDEN>)
 id 1kWj6x-0003hk-Mu; Sun, 25 Oct 2020 12:43:15 -0400
Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=57270 helo=ribbon)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <ludo@HIDDEN>)
 id 1kWj6w-0006km-M4; Sun, 25 Oct 2020 12:43:15 -0400
From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>
To: Maxim Cournoyer <maxim.cournoyer@HIDDEN>
Subject: Re: bug#44175: [optimization] Grafting is too slow
References: <87y2o3zf00.fsf@HIDDEN> <20200701112453.GB25179@zpidnp36>
 <87eepuyk99.fsf@HIDDEN> <20200702070000.GA4532@zpidnp36>
 <20200702120333.GB4532@zpidnp36> <87k0zhkoyf.fsf@HIDDEN>
 <20200706125800.GA3176@zpidnp36> <20200720095059.GB3143@zpidnp36>
 <87mu3tq2hh.fsf@HIDDEN> <87r1pqc6sy.fsf@HIDDEN>
 <20201023072621.GA3115@zpidnp36> <875z71ccq0.fsf_-_@HIDDEN>
 <87lffwk5fm.fsf@HIDDEN> <87o8kr2fx7.fsf@HIDDEN>
X-URL: http://www.fdn.fr/~lcourtes/
X-Revolutionary-Date: 4 Brumaire an 229 de la =?utf-8?Q?R=C3=A9volution?=
X-PGP-Key-ID: 0x090B11993D9AEBB5
X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc
X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4  0CFB 090B 1199 3D9A EBB5
X-OS: x86_64-pc-linux-gnu
Date: Sun, 25 Oct 2020 17:43:12 +0100
In-Reply-To: <87o8kr2fx7.fsf@HIDDEN> (Maxim Cournoyer's message of "Sat, 24
 Oct 2020 22:33:08 -0400")
Message-ID: <87wnzedzof.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 44175
Cc: 44175 <at> debbugs.gnu.org, Lars-Dominik Braun <ldb@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Hi!

Maxim Cournoyer <maxim.cournoyer@HIDDEN> skribis:

>>>> $ time guix environment --ad-hoc  --search-paths r-learnr
>>>> guix environment --ad-hoc --search-paths r-learnr  5,90s user 0,09s sy=
stem 210% cpu 2,844 total
>>>> $ time guix environment --ad-hoc  --search-paths r-learnr --no-grafts
>>>> guix environment --ad-hoc --search-paths r-learnr --no-grafts  2,03s u=
ser 0,08s system 164% cpu 1,277 total
>>>
>>> I'm opening a new issue to track optimizing the grafting code, since
>>> it's independent of environments (grafts are applied anytime a
>>> derivation is built, AFAICT).  Grafting is inherently IO-bound,
>>
>> What is slow above is not grafting itself: it=E2=80=99s determining what=
 to
>> graft that takes CPU time.
>
> On my system, grafting seems IO rather than CPU bound, I'm guessing
> because of the need to scan all the files for strings to replace in the
> graft process.

Yes, you=E2=80=99re right of course.  But I think in the example above Lars=
 was
reporting the run time of the =E2=80=98guix=E2=80=99 command when the graft=
 itself is
already built.  Just the extra code to compute the graft derivation (not
actually building them) leads to x2 wall-clock time or so.

>> I had reopened the initial bug at <https://issues.guix.gnu.org/41702>;
>> should we close this one?
>
> Many optimizations were made in the above issue that were not related to
> the grafting process, so to me a fresh entry such as this one is clearer
> to follow.  That said, feel free to proceed as you see fit, being the
> issue "owner" :-).

Alright, thanks!

Ludo=E2=80=99.
Information forwarded to bug-guix@HIDDEN:
bug#44175; Package guix. Full text available.

Message received at 44175 <at> debbugs.gnu.org:


Received: (at 44175) by debbugs.gnu.org; 25 Oct 2020 02:33:17 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Oct 24 22:33:17 2020
Received: from localhost ([127.0.0.1]:35561 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1kWVqP-0001rI-AM
	for submit <at> debbugs.gnu.org; Sat, 24 Oct 2020 22:33:17 -0400
Received: from mail-qt1-f179.google.com ([209.85.160.179]:35696)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <maxim.cournoyer@HIDDEN>) id 1kWVqN-0001r4-GI
 for 44175 <at> debbugs.gnu.org; Sat, 24 Oct 2020 22:33:16 -0400
Received: by mail-qt1-f179.google.com with SMTP id c15so4380015qtc.2
 for <44175 <at> debbugs.gnu.org>; Sat, 24 Oct 2020 19:33:15 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=from:to:cc:subject:references:date:in-reply-to:message-id
 :user-agent:mime-version:content-transfer-encoding;
 bh=O67LJmwuJroGsWbSJUZdoqLA+n2Tp4+T1z0Qt+f0Qio=;
 b=LaAnhgoycQHd4E4BXwO7ZizQpq+beCe7p0TfBpQalOSjPhpEJjfpQH+zj4ssURAfjK
 EHHH7PRkKvMybhVeOyJcTIRQfIs4xC/CzFGEI70OpEKY9flpqmNTRWMKrciW/wqdzybN
 IMv6E0wbESZsC0znyaMM62miqSDbDlypiRgKtIM19+RLz9cbbxp2GRG4+ZP4G/+eJev0
 qedvGKun4PGC1KrOvoObSebmu6SHNahY3CUD0nDT/JhkTUlL6/mmyNc/8/+8/WamhBmV
 6vKXDOWMi4bc3HqZGsUG4MrQEGSnNfSgh4DajKg3FReb4oz9hVw+Co2sQPkemfDLwYpc
 FOqA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to
 :message-id:user-agent:mime-version:content-transfer-encoding;
 bh=O67LJmwuJroGsWbSJUZdoqLA+n2Tp4+T1z0Qt+f0Qio=;
 b=TkIsD2TGCUDg1pEsF/aoPTpQ9+fqm4Bm1V6pi2RmH8eVLJvHttkxgqpedUD8yXnZiX
 VhaRWxt7Rs4cOKvA8PZnmuFpVex5q/4YZ76fulQBjX0l+g9KLWK8p67YpPlIyIvPZUuQ
 2RXxwZH1OjE9isDi0vOyxr4rYMh3HTsupmQ03zOm6UvPxCmsyDBpc2v/aFGMBE5AysKI
 bALMBbeqgdv7VyDngwUz9NG5Adv2A7mMEKk15A8Y/K6ifUfDAmN9A6jMzLnB/Ri7xgTe
 VWhzCs2+ljvOZtYjX333TcZoElhjwN4ea/1ysmGDuRC6v+kDdR9lh+fFquyw9wo5jmTw
 7iZQ==
X-Gm-Message-State: AOAM532zEhYXcJTXNI3vHtCMyuCx5hras9R8ZTrzVcTEzjpth2Im0itN
 5OaKE6zxlP+yGvmrfZxeWIAQ/aD45BA=
X-Google-Smtp-Source: ABdhPJwkp/6SqD6i3nMbM2t1Z7uesNsvpwnX+rvhDS/oTCuctfW3fm2C8R+6Zp7alZ1/FjNrlG8UVw==
X-Received: by 2002:ac8:64d:: with SMTP id e13mr10372093qth.23.1603593189822; 
 Sat, 24 Oct 2020 19:33:09 -0700 (PDT)
Received: from hurd (dsl-159-173.b2b2c.ca. [66.158.159.173])
 by smtp.gmail.com with ESMTPSA id t70sm3879298qke.119.2020.10.24.19.33.08
 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
 Sat, 24 Oct 2020 19:33:09 -0700 (PDT)
From: Maxim Cournoyer <maxim.cournoyer@HIDDEN>
To: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Subject: Re: bug#44175: [optimization] Grafting is too slow
References: <87y2o3zf00.fsf@HIDDEN> <20200701112453.GB25179@zpidnp36>
 <87eepuyk99.fsf@HIDDEN> <20200702070000.GA4532@zpidnp36>
 <20200702120333.GB4532@zpidnp36> <87k0zhkoyf.fsf@HIDDEN>
 <20200706125800.GA3176@zpidnp36> <20200720095059.GB3143@zpidnp36>
 <87mu3tq2hh.fsf@HIDDEN> <87r1pqc6sy.fsf@HIDDEN>
 <20201023072621.GA3115@zpidnp36> <875z71ccq0.fsf_-_@HIDDEN>
 <87lffwk5fm.fsf@HIDDEN>
Date: Sat, 24 Oct 2020 22:33:08 -0400
In-Reply-To: <87lffwk5fm.fsf@HIDDEN> ("Ludovic =?utf-8?Q?Court=C3=A8s=22'?=
 =?utf-8?Q?s?= message of "Fri, 23 Oct 2020 23:17:49 +0200")
Message-ID: <87o8kr2fx7.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 44175
Cc: 44175 <at> debbugs.gnu.org, Lars-Dominik Braun <ldb@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Hello!

Ludovic Court=C3=A8s <ludo@HIDDEN> writes:

> Maxim Cournoyer <maxim.cournoyer@HIDDEN> skribis:
>
>> Lars-Dominik Braun <ldb@HIDDEN> writes:
>>
>>> Hi Maxim,
>>>
>>>> Judging from the above, it seems this issue has been resolved.
>>> grafting is still a performance issue imo. Compare for example:
>>>
>>> $ time guix environment --ad-hoc  --search-paths r-learnr
>>> guix environment --ad-hoc --search-paths r-learnr  5,90s user 0,09s sys=
tem 210% cpu 2,844 total
>>> $ time guix environment --ad-hoc  --search-paths r-learnr --no-grafts
>>> guix environment --ad-hoc --search-paths r-learnr --no-grafts  2,03s us=
er 0,08s system 164% cpu 1,277 total
>>
>> I'm opening a new issue to track optimizing the grafting code, since
>> it's independent of environments (grafts are applied anytime a
>> derivation is built, AFAICT).  Grafting is inherently IO-bound,
>
> What is slow above is not grafting itself: it=E2=80=99s determining what =
to
> graft that takes CPU time.

On my system, grafting seems IO rather than CPU bound, I'm guessing
because of the need to scan all the files for strings to replace in the
graft process.

> I had reopened the initial bug at <https://issues.guix.gnu.org/41702>;
> should we close this one?

Many optimizations were made in the above issue that were not related to
the grafting process, so to me a fresh entry such as this one is clearer
to follow.  That said, feel free to proceed as you see fit, being the
issue "owner" :-).

Thanks,

Maxim
Information forwarded to bug-guix@HIDDEN:
bug#44175; Package guix. Full text available.

Message received at 44175 <at> debbugs.gnu.org:


Received: (at 44175) by debbugs.gnu.org; 23 Oct 2020 21:18:00 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Oct 23 17:18:00 2020
Received: from localhost ([127.0.0.1]:58549 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1kW4Rk-0008Jd-GH
	for submit <at> debbugs.gnu.org; Fri, 23 Oct 2020 17:18:00 -0400
Received: from eggs.gnu.org ([209.51.188.92]:51372)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1kW4Rh-0008JH-Uo
 for 44175 <at> debbugs.gnu.org; Fri, 23 Oct 2020 17:17:59 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:60194)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <ludo@HIDDEN>)
 id 1kW4Rc-0003ea-IK; Fri, 23 Oct 2020 17:17:52 -0400
Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=59378 helo=ribbon)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <ludo@HIDDEN>)
 id 1kW4Rc-0006bz-3u; Fri, 23 Oct 2020 17:17:52 -0400
From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>
To: Maxim Cournoyer <maxim.cournoyer@HIDDEN>
Subject: Re: bug#44175: [optimization] Grafting is too slow
References: <87y2o3zf00.fsf@HIDDEN> <20200701112453.GB25179@zpidnp36>
 <87eepuyk99.fsf@HIDDEN> <20200702070000.GA4532@zpidnp36>
 <20200702120333.GB4532@zpidnp36> <87k0zhkoyf.fsf@HIDDEN>
 <20200706125800.GA3176@zpidnp36> <20200720095059.GB3143@zpidnp36>
 <87mu3tq2hh.fsf@HIDDEN> <87r1pqc6sy.fsf@HIDDEN>
 <20201023072621.GA3115@zpidnp36> <875z71ccq0.fsf_-_@HIDDEN>
Date: Fri, 23 Oct 2020 23:17:49 +0200
In-Reply-To: <875z71ccq0.fsf_-_@HIDDEN> (Maxim Cournoyer's message of "Fri, 
 23 Oct 2020 09:07:35 -0400")
Message-ID: <87lffwk5fm.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 44175
Cc: 44175 <at> debbugs.gnu.org, Lars-Dominik Braun <ldb@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Maxim Cournoyer <maxim.cournoyer@HIDDEN> skribis:

> Lars-Dominik Braun <ldb@HIDDEN> writes:
>
>> Hi Maxim,
>>
>>> Judging from the above, it seems this issue has been resolved.
>> grafting is still a performance issue imo. Compare for example:
>>
>> $ time guix environment --ad-hoc  --search-paths r-learnr
>> guix environment --ad-hoc --search-paths r-learnr  5,90s user 0,09s syst=
em 210% cpu 2,844 total
>> $ time guix environment --ad-hoc  --search-paths r-learnr --no-grafts
>> guix environment --ad-hoc --search-paths r-learnr --no-grafts  2,03s use=
r 0,08s system 164% cpu 1,277 total
>
> I'm opening a new issue to track optimizing the grafting code, since
> it's independent of environments (grafts are applied anytime a
> derivation is built, AFAICT).  Grafting is inherently IO-bound,

What is slow above is not grafting itself: it=E2=80=99s determining what to
graft that takes CPU time.

I had reopened the initial bug at <https://issues.guix.gnu.org/41702>;
should we close this one?

Thanks,
Ludo=E2=80=99.
Information forwarded to bug-guix@HIDDEN:
bug#44175; Package guix. Full text available.

Message received at submit <at> debbugs.gnu.org:


Received: (at submit) by debbugs.gnu.org; 23 Oct 2020 13:07:44 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Oct 23 09:07:44 2020
Received: from localhost ([127.0.0.1]:56309 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1kVwnI-0003Zn-Dn
	for submit <at> debbugs.gnu.org; Fri, 23 Oct 2020 09:07:44 -0400
Received: from lists.gnu.org ([209.51.188.17]:44862)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <maxim.cournoyer@HIDDEN>) id 1kVwnG-0003Zf-Kd
 for submit <at> debbugs.gnu.org; Fri, 23 Oct 2020 09:07:43 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:50314)
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <maxim.cournoyer@HIDDEN>)
 id 1kVwnG-0003XY-Fu
 for bug-guix@HIDDEN; Fri, 23 Oct 2020 09:07:42 -0400
Received: from mail-qv1-xf32.google.com ([2607:f8b0:4864:20::f32]:44970)
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128)
 (Exim 4.90_1) (envelope-from <maxim.cournoyer@HIDDEN>)
 id 1kVwnE-0007gH-QA; Fri, 23 Oct 2020 09:07:42 -0400
Received: by mail-qv1-xf32.google.com with SMTP id s17so605256qvr.11;
 Fri, 23 Oct 2020 06:07:39 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=from:to:cc:subject:references:date:in-reply-to:message-id
 :user-agent:mime-version;
 bh=HNPJPbwHB+tUEYnv62k4WakSz/x7WlEkRe9wVFNPQOg=;
 b=NQcw0p7LC5EBS9YpJDk3OTy9qvha4NTkjgKTeIufc7s4pv2tazrI6fcE7sUnFbz33p
 n9ld+SOHYOjeCSYyQFjWU5UN8Bsb4Vngy/jGSicdmlUWYMJi29hDqQgsnvGB76j0+3XB
 tpELpFpH+A2dT6CamXytuDo1eVwjTh6Djw1uaGWPSHi3kBD/Jo+olIpDZ3nS6KDFL6WU
 ScgI8CtngqjqgpraKEjxq2ShPNz/jAz4WGDk3gUOiuibRuS+pjJ8TnWcRyeYEF8tlOr6
 RZoqSs9ocKYNj+WqY/6WHlWcUUaAK1wihparRVlAvN0jd8GWIbzior3XQbFrm06Ib8fH
 KeNQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to
 :message-id:user-agent:mime-version;
 bh=HNPJPbwHB+tUEYnv62k4WakSz/x7WlEkRe9wVFNPQOg=;
 b=UEziSPDrCtjJqgyPlM+MYAXTpuwakgiazb+BwazvoNyVEujWylGbPjcGhEAEgj53c4
 RZUyWK+EFX/N4qCyuKgJYDtwzQQVB9Z2Ni0Ydiu48YV2V+elN1oLM388nm4tenQ+nRdU
 vs3EeRVG6UaBgxFi4m2IXXICKQvcsnh3vpELNoSqPvW4CDVfbu/h+fOPDyySLf7TGs0n
 4dXj1iBxQSlxWj/UUbJ/K3Mrjlt8qGlKcGETXi4JVgXin8lz7eNy7x41ZzO47ZW1lIEG
 2gfXuNM4A9Gw4KqRpd6cZLkgupPqt22Du+4caNxr3BTvGPT/QeBIxfqOfmv/e4v43R2+
 387Q==
X-Gm-Message-State: AOAM531colYN4pQ4Je5fxyPnLPK42zXjG18b69raW7S0GTvhaYIrHaxk
 F7TmD4plh5kAUyQFr9mfbL4W61QOgw63Cw==
X-Google-Smtp-Source: ABdhPJyuCJqg7n6R0sKqhfg+IGr5o6A8jHacZm029jDCm1PFzaYviqxb+B7fUh5fve+ZqRrk3ggz8w==
X-Received: by 2002:a0c:c985:: with SMTP id b5mr2385243qvk.10.1603458457352;
 Fri, 23 Oct 2020 06:07:37 -0700 (PDT)
Received: from hurd (dsl-159-173.b2b2c.ca. [66.158.159.173])
 by smtp.gmail.com with ESMTPSA id u2sm770111qtw.40.2020.10.23.06.07.36
 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
 Fri, 23 Oct 2020 06:07:36 -0700 (PDT)
From: Maxim Cournoyer <maxim.cournoyer@HIDDEN>
To: Lars-Dominik Braun <ldb@HIDDEN>
Subject: [optimization] Grafting is too slow
References: <87y2o3zf00.fsf@HIDDEN> <20200701112453.GB25179@zpidnp36>
 <87eepuyk99.fsf@HIDDEN> <20200702070000.GA4532@zpidnp36>
 <20200702120333.GB4532@zpidnp36> <87k0zhkoyf.fsf@HIDDEN>
 <20200706125800.GA3176@zpidnp36> <20200720095059.GB3143@zpidnp36>
 <87mu3tq2hh.fsf@HIDDEN> <87r1pqc6sy.fsf@HIDDEN>
 <20201023072621.GA3115@zpidnp36>
Date: Fri, 23 Oct 2020 09:07:35 -0400
In-Reply-To: <20201023072621.GA3115@zpidnp36> (Lars-Dominik Braun's message of
 "Fri, 23 Oct 2020 09:26:21 +0200")
Message-ID: <875z71ccq0.fsf_-_@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
Received-SPF: pass client-ip=2607:f8b0:4864:20::f32;
 envelope-from=maxim.cournoyer@HIDDEN; helo=mail-qv1-xf32.google.com
X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache.
 That's all we know.
X-Spam_score_int: -20
X-Spam_score: -2.1
X-Spam_bar: --
X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,
 DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001,
 RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001,
 SPF_PASS=-0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-Spam-Score: -1.3 (-)
X-Debbugs-Envelope-To: submit
Cc: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>,
 bug-guix <bug-guix@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -2.3 (--)

Hello Lars,

Lars-Dominik Braun <ldb@HIDDEN> writes:

> Hi Maxim,
>
>> Judging from the above, it seems this issue has been resolved.
> grafting is still a performance issue imo. Compare for example:
>
> $ time guix environment --ad-hoc  --search-paths r-learnr
> guix environment --ad-hoc --search-paths r-learnr  5,90s user 0,09s system 210% cpu 2,844 total
> $ time guix environment --ad-hoc  --search-paths r-learnr --no-grafts
> guix environment --ad-hoc --search-paths r-learnr --no-grafts  2,03s user 0,08s system 164% cpu 1,277 total

I'm opening a new issue to track optimizing the grafting code, since
it's independent of environments (grafts are applied anytime a
derivation is built, AFAICT).  Grafting is inherently IO-bound, since it
must scan all the built artifacts of a grafted package (from its
dependency that was 'replaced' all the way up), if I understood
correctly.  "info (guix) Security Updates" gives some information.

Perhaps we could benchmark how fast our code currently is for grafting,
and compare it with the fastest sed-like utility out there, as a
starting point, to have an idea of how much there is to gain from
optimization.

Maxim
Acknowledgement sent to Maxim Cournoyer <maxim.cournoyer@HIDDEN>:
New bug report received and forwarded. Copy sent to bug-guix@HIDDEN. Full text available.
Report forwarded to bug-guix@HIDDEN:
bug#44175; Package guix. Full text available.
Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.
Last modified: Sun, 25 Oct 2020 16:45:01 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.