GNU logs - #44175, boring messages

Message sent to bug-guix@HIDDEN:

Subject: bug#44175: [optimization] Grafting is too slow
Resent-From: Maxim Cournoyer <maxim.cournoyer@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Fri, 23 Oct 2020 13:08:01 +0000
Resent-Message-ID: <handler.44175.B.160345846413755 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: Lars-Dominik Braun <ldb@HIDDEN>
Cc: ludo@HIDDEN, 44175 <at> debbugs.gnu.org
From: Maxim Cournoyer <maxim.cournoyer@HIDDEN>
References: <87y2o3zf00.fsf@HIDDEN> <20200701112453.GB25179@zpidnp36>
 <87eepuyk99.fsf@HIDDEN> <20200702070000.GA4532@zpidnp36>
 <20200702120333.GB4532@zpidnp36> <87k0zhkoyf.fsf@HIDDEN>
 <20200706125800.GA3176@zpidnp36> <20200720095059.GB3143@zpidnp36>
 <87mu3tq2hh.fsf@HIDDEN> <87r1pqc6sy.fsf@HIDDEN>
 <20201023072621.GA3115@zpidnp36>
Date: Fri, 23 Oct 2020 09:07:35 -0400
In-Reply-To: <20201023072621.GA3115@zpidnp36> (Lars-Dominik Braun's message of
 "Fri, 23 Oct 2020 09:26:21 +0200")
Message-ID: <875z71ccq0.fsf_-_@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
Received-SPF: pass client-ip=2607:f8b0:4864:20::f32;
 envelope-from=maxim.cournoyer@HIDDEN; helo=mail-qv1-xf32.google.com
X-Spam_score_int: -20
X-Spam_score: -2.1
X-Spam_bar: --
X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,
 DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001,
 RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001,
 SPF_PASS=-0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-Spam-Score: -1.3 (-)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -2.3 (--)

Hello Lars,

Lars-Dominik Braun <ldb@HIDDEN> writes:

> Hi Maxim,
>
>> Judging from the above, it seems this issue has been resolved.
> grafting is still a performance issue imo. Compare for example:
>
> $ time guix environment --ad-hoc  --search-paths r-learnr
> guix environment --ad-hoc --search-paths r-learnr  5,90s user 0,09s system 210% cpu 2,844 total
> $ time guix environment --ad-hoc  --search-paths r-learnr --no-grafts
> guix environment --ad-hoc --search-paths r-learnr --no-grafts  2,03s user 0,08s system 164% cpu 1,277 total

I'm opening a new issue to track optimizing the grafting code, since
it's independent of environments (grafts are applied anytime a
derivation is built, AFAICT).  Grafting is inherently IO-bound, since it
must scan all the built artifacts of a grafted package (from its
dependency that was 'replaced' all the way up), if I understood
correctly.  "info (guix) Security Updates" gives some information.

Perhaps we could benchmark how fast our code currently is for grafting,
and compare it with the fastest sed-like utility out there, as a
starting point, to have an idea of how much there is to gain from
optimization.

Maxim

Message sent:

Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Mailer: MIME-tools 5.505 (Entity 5.505)
Content-Type: text/plain; charset=utf-8
X-Loop: help-debbugs@HIDDEN
From: help-debbugs@HIDDEN (GNU bug Tracking System)
To: Maxim Cournoyer <maxim.cournoyer@HIDDEN>
Subject: bug#44175: Acknowledgement ([optimization] Grafting is too slow)
Message-ID: <handler.44175.B.160345846413755.ack <at> debbugs.gnu.org>
References: <875z71ccq0.fsf_-_@HIDDEN>
X-Gnu-PR-Message: ack 44175
X-Gnu-PR-Package: guix
Reply-To: 44175 <at> debbugs.gnu.org
Date: Fri, 23 Oct 2020 13:08:02 +0000

Thank you for filing a new bug report with debbugs.gnu.org.

This is an automatically generated reply to let you know your message
has been received.

Your message is being forwarded to the package maintainers and other
interested parties for their attention; they will reply in due course.

Your message has been sent to the package maintainer(s):
 bug-guix@HIDDEN

If you wish to submit further information on this problem, please
send it to 44175 <at> debbugs.gnu.org.

Please do not send mail to help-debbugs@HIDDEN unless you wish
to report a problem with the Bug-tracking system.

--=20
44175: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D44175
GNU Bug Tracking System
Contact help-debbugs@HIDDEN with problems

Message sent to bug-guix@HIDDEN:

Subject: bug#44175: [optimization] Grafting is too slow
Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Fri, 23 Oct 2020 21:18:02 +0000
Resent-Message-ID: <handler.44175.B44175.160348788031973 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: Maxim Cournoyer <maxim.cournoyer@HIDDEN>
Cc: 44175 <at> debbugs.gnu.org, Lars-Dominik Braun <ldb@HIDDEN>
From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
References: <87y2o3zf00.fsf@HIDDEN> <20200701112453.GB25179@zpidnp36>
 <87eepuyk99.fsf@HIDDEN> <20200702070000.GA4532@zpidnp36>
 <20200702120333.GB4532@zpidnp36> <87k0zhkoyf.fsf@HIDDEN>
 <20200706125800.GA3176@zpidnp36> <20200720095059.GB3143@zpidnp36>
 <87mu3tq2hh.fsf@HIDDEN> <87r1pqc6sy.fsf@HIDDEN>
 <20201023072621.GA3115@zpidnp36> <875z71ccq0.fsf_-_@HIDDEN>
Date: Fri, 23 Oct 2020 23:17:49 +0200
In-Reply-To: <875z71ccq0.fsf_-_@HIDDEN> (Maxim Cournoyer's message of "Fri, 
 23 Oct 2020 09:07:35 -0400")
Message-ID: <87lffwk5fm.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

Maxim Cournoyer <maxim.cournoyer@HIDDEN> skribis:

> Lars-Dominik Braun <ldb@HIDDEN> writes:
>
>> Hi Maxim,
>>
>>> Judging from the above, it seems this issue has been resolved.
>> grafting is still a performance issue imo. Compare for example:
>>
>> $ time guix environment --ad-hoc  --search-paths r-learnr
>> guix environment --ad-hoc --search-paths r-learnr  5,90s user 0,09s syst=
em 210% cpu 2,844 total
>> $ time guix environment --ad-hoc  --search-paths r-learnr --no-grafts
>> guix environment --ad-hoc --search-paths r-learnr --no-grafts  2,03s use=
r 0,08s system 164% cpu 1,277 total
>
> I'm opening a new issue to track optimizing the grafting code, since
> it's independent of environments (grafts are applied anytime a
> derivation is built, AFAICT).  Grafting is inherently IO-bound,

What is slow above is not grafting itself: it=E2=80=99s determining what to
graft that takes CPU time.

I had reopened the initial bug at <https://issues.guix.gnu.org/41702>;
should we close this one?

Thanks,
Ludo=E2=80=99.

Message sent to bug-guix@HIDDEN:

Subject: bug#44175: [optimization] Grafting is too slow
Resent-From: Maxim Cournoyer <maxim.cournoyer@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Sun, 25 Oct 2020 02:34:01 +0000
Resent-Message-ID: <handler.44175.B44175.16035931977152 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Cc: 44175 <at> debbugs.gnu.org, Lars-Dominik Braun <ldb@HIDDEN>
From: Maxim Cournoyer <maxim.cournoyer@HIDDEN>
References: <87y2o3zf00.fsf@HIDDEN> <20200701112453.GB25179@zpidnp36>
 <87eepuyk99.fsf@HIDDEN> <20200702070000.GA4532@zpidnp36>
 <20200702120333.GB4532@zpidnp36> <87k0zhkoyf.fsf@HIDDEN>
 <20200706125800.GA3176@zpidnp36> <20200720095059.GB3143@zpidnp36>
 <87mu3tq2hh.fsf@HIDDEN> <87r1pqc6sy.fsf@HIDDEN>
 <20201023072621.GA3115@zpidnp36> <875z71ccq0.fsf_-_@HIDDEN>
 <87lffwk5fm.fsf@HIDDEN>
Date: Sat, 24 Oct 2020 22:33:08 -0400
In-Reply-To: <87lffwk5fm.fsf@HIDDEN> ("Ludovic =?UTF-8?Q?Court=C3=A8s?="'s message of "Fri, 23 Oct 2020 23:17:49 +0200")
Message-ID: <87o8kr2fx7.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

Hello!

Ludovic Court=C3=A8s <ludo@HIDDEN> writes:

> Maxim Cournoyer <maxim.cournoyer@HIDDEN> skribis:
>
>> Lars-Dominik Braun <ldb@HIDDEN> writes:
>>
>>> Hi Maxim,
>>>
>>>> Judging from the above, it seems this issue has been resolved.
>>> grafting is still a performance issue imo. Compare for example:
>>>
>>> $ time guix environment --ad-hoc  --search-paths r-learnr
>>> guix environment --ad-hoc --search-paths r-learnr  5,90s user 0,09s sys=
tem 210% cpu 2,844 total
>>> $ time guix environment --ad-hoc  --search-paths r-learnr --no-grafts
>>> guix environment --ad-hoc --search-paths r-learnr --no-grafts  2,03s us=
er 0,08s system 164% cpu 1,277 total
>>
>> I'm opening a new issue to track optimizing the grafting code, since
>> it's independent of environments (grafts are applied anytime a
>> derivation is built, AFAICT).  Grafting is inherently IO-bound,
>
> What is slow above is not grafting itself: it=E2=80=99s determining what =
to
> graft that takes CPU time.

On my system, grafting seems IO rather than CPU bound, I'm guessing
because of the need to scan all the files for strings to replace in the
graft process.

> I had reopened the initial bug at <https://issues.guix.gnu.org/41702>;
> should we close this one?

Many optimizations were made in the above issue that were not related to
the grafting process, so to me a fresh entry such as this one is clearer
to follow.  That said, feel free to proceed as you see fit, being the
issue "owner" :-).

Thanks,

Maxim

Message sent to bug-guix@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#44175: [optimization] Grafting is too slow
Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Sun, 25 Oct 2020 16:44:02 +0000
Resent-Message-ID: <handler.44175.B44175.160364420317944 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 44175
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: Maxim Cournoyer <maxim.cournoyer@HIDDEN>
Cc: 44175 <at> debbugs.gnu.org, Lars-Dominik Braun <ldb@HIDDEN>
Received: via spool by 44175-submit <at> debbugs.gnu.org id=B44175.160364420317944
          (code B ref 44175); Sun, 25 Oct 2020 16:44:02 +0000
Received: (at 44175) by debbugs.gnu.org; 25 Oct 2020 16:43:23 +0000
Received: from localhost ([127.0.0.1]:37484 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1kWj75-0004fL-8u
	for submit <at> debbugs.gnu.org; Sun, 25 Oct 2020 12:43:23 -0400
Received: from eggs.gnu.org ([209.51.188.92]:60102)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1kWj73-0004f7-59
 for 44175 <at> debbugs.gnu.org; Sun, 25 Oct 2020 12:43:21 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:37194)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <ludo@HIDDEN>)
 id 1kWj6x-0003hk-Mu; Sun, 25 Oct 2020 12:43:15 -0400
Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=57270 helo=ribbon)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <ludo@HIDDEN>)
 id 1kWj6w-0006km-M4; Sun, 25 Oct 2020 12:43:15 -0400
From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
References: <87y2o3zf00.fsf@HIDDEN> <20200701112453.GB25179@zpidnp36>
 <87eepuyk99.fsf@HIDDEN> <20200702070000.GA4532@zpidnp36>
 <20200702120333.GB4532@zpidnp36> <87k0zhkoyf.fsf@HIDDEN>
 <20200706125800.GA3176@zpidnp36> <20200720095059.GB3143@zpidnp36>
 <87mu3tq2hh.fsf@HIDDEN> <87r1pqc6sy.fsf@HIDDEN>
 <20201023072621.GA3115@zpidnp36> <875z71ccq0.fsf_-_@HIDDEN>
 <87lffwk5fm.fsf@HIDDEN> <87o8kr2fx7.fsf@HIDDEN>
X-URL: http://www.fdn.fr/~lcourtes/
X-Revolutionary-Date: 4 Brumaire an 229 de la =?UTF-8?Q?R=C3=A9volution?=
X-PGP-Key-ID: 0x090B11993D9AEBB5
X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc
X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4  0CFB 090B 1199 3D9A EBB5
X-OS: x86_64-pc-linux-gnu
Date: Sun, 25 Oct 2020 17:43:12 +0100
In-Reply-To: <87o8kr2fx7.fsf@HIDDEN> (Maxim Cournoyer's message of "Sat, 24
 Oct 2020 22:33:08 -0400")
Message-ID: <87wnzedzof.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Hi!

Maxim Cournoyer <maxim.cournoyer@HIDDEN> skribis:

>>>> $ time guix environment --ad-hoc  --search-paths r-learnr
>>>> guix environment --ad-hoc --search-paths r-learnr  5,90s user 0,09s sy=
stem 210% cpu 2,844 total
>>>> $ time guix environment --ad-hoc  --search-paths r-learnr --no-grafts
>>>> guix environment --ad-hoc --search-paths r-learnr --no-grafts  2,03s u=
ser 0,08s system 164% cpu 1,277 total
>>>
>>> I'm opening a new issue to track optimizing the grafting code, since
>>> it's independent of environments (grafts are applied anytime a
>>> derivation is built, AFAICT).  Grafting is inherently IO-bound,
>>
>> What is slow above is not grafting itself: it=E2=80=99s determining what=
 to
>> graft that takes CPU time.
>
> On my system, grafting seems IO rather than CPU bound, I'm guessing
> because of the need to scan all the files for strings to replace in the
> graft process.

Yes, you=E2=80=99re right of course.  But I think in the example above Lars=
 was
reporting the run time of the =E2=80=98guix=E2=80=99 command when the graft=
 itself is
already built.  Just the extra code to compute the graft derivation (not
actually building them) leads to x2 wall-clock time or so.

>> I had reopened the initial bug at <https://issues.guix.gnu.org/41702>;
>> should we close this one?
>
> Many optimizations were made in the above issue that were not related to
> the grafting process, so to me a fresh entry such as this one is clearer
> to follow.  That said, feel free to proceed as you see fit, being the
> issue "owner" :-).

Alright, thanks!

Ludo=E2=80=99.

Last modified: Sun, 25 Oct 2020 16:45:01 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.