GNU bug report logs - #46402
Cuirass rebuilds the same packae multiple times

Previous Next

Package: guix;

Reported by: Julien Lepiller <julien <at> lepiller.eu>

Date: Tue, 9 Feb 2021 13:20:02 UTC

Severity: normal

Done: Mathieu Othacehe <othacehe <at> gnu.org>

Bug is archived. No further changes may be made.

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 46402 in the body.
You can then email your comments to 46402 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to bug-guix <at> gnu.org:
bug#46402; Package guix. (Tue, 09 Feb 2021 13:20:02 GMT) Full text and rfc822 format available.

Acknowledgement sent to Julien Lepiller <julien <at> lepiller.eu>:
New bug report received and forwarded. Copy sent to bug-guix <at> gnu.org. (Tue, 09 Feb 2021 13:20:02 GMT) Full text and rfc822 format available.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Julien Lepiller <julien <at> lepiller.eu>
To: bug-guix <at> gnu.org
Subject: Cuirass rebuilds the same packae multiple times
Date: Tue, 9 Feb 2021 14:19:15 +0100
Hi!

I've updated php yesterday and I noticed that cuirass is now busy
building dependents on aarch64. Looking at the log of some of them, it
seems that the workers are independently building the same php
derivation, and not the dependents, at the same time. This is extremely
wasteful as php is very long to build (and might even eventually fail).

here is an example: https://ci.guix.gnu.org/build/287478/details and
https://ci.guix.gnu.org/build/287476/details are being built at the
same time, and the logs currently show they are both running the test
phase of the php package.

Shouldn't cuirass first schedule builds for dependencies before it
builds dependents?




Information forwarded to bug-guix <at> gnu.org:
bug#46402; Package guix. (Tue, 09 Feb 2021 15:43:02 GMT) Full text and rfc822 format available.

Message #8 received at 46402 <at> debbugs.gnu.org (full text, mbox):

From: Mathieu Othacehe <othacehe <at> gnu.org>
To: Julien Lepiller <julien <at> lepiller.eu>
Cc: Ludovic Courtès <ludo <at> gnu.org>, 46402 <at> debbugs.gnu.org
Subject: Re: bug#46402: Cuirass rebuilds the same packae multiple times
Date: Tue, 09 Feb 2021 16:42:46 +0100
Hello Julien,

> here is an example: https://ci.guix.gnu.org/build/287478/details and
> https://ci.guix.gnu.org/build/287476/details are being built at the
> same time, and the logs currently show they are both running the test
> phase of the php package.

Thanks for the report. This problem has been briefly discussed
yesterday. It has been introduced by the new remote building mechanism
in Cuirass. Hydra solves it by breaking each build into buildsteps
corresponding to the derivation inputs.

The buildsteps are then submitted to the workers in a logical order. I
proposed to introduce a similar mechanism in Cuirass but Ludo expressed
doubts. Ludo do you think this problem could be solved otherwise?

Thanks,

Mathieu




Information forwarded to bug-guix <at> gnu.org:
bug#46402; Package guix. (Wed, 10 Feb 2021 10:48:01 GMT) Full text and rfc822 format available.

Message #11 received at 46402 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Mathieu Othacehe <othacehe <at> gnu.org>
Cc: Julien Lepiller <julien <at> lepiller.eu>, 46402 <at> debbugs.gnu.org
Subject: Re: bug#46402: Cuirass rebuilds the same packae multiple times
Date: Wed, 10 Feb 2021 11:46:51 +0100
Hi,

Mathieu Othacehe <othacehe <at> gnu.org> skribis:

>> here is an example: https://ci.guix.gnu.org/build/287478/details and
>> https://ci.guix.gnu.org/build/287476/details are being built at the
>> same time, and the logs currently show they are both running the test
>> phase of the php package.
>
> Thanks for the report. This problem has been briefly discussed
> yesterday. It has been introduced by the new remote building mechanism
> in Cuirass. Hydra solves it by breaking each build into buildsteps
> corresponding to the derivation inputs.
>
> The buildsteps are then submitted to the workers in a logical order. I
> proposed to introduce a similar mechanism in Cuirass but Ludo expressed
> doubts. Ludo do you think this problem could be solved otherwise?

I’m not sure exactly but I can share my feelings.  :-)

Seems to me that ‘BuildSteps’ is an orthogonal concern that has little
to do with Cuirass’ job and with its data model.  In Hydra I saw that as
a (necessary) kludge.

I like the way the Coordinator does it, and AIUI it’s pretty much the
same as what the daemon is doing: submit build requests in topological
order, such that when a derivation build is submitted, its prerequisites
are known to be built already.

I suppose what makes it more difficult here is that we have this extra
“job” abstraction on top of derivations; everything in Cuirass revolves
around jobs, which leads to this impedance mismatch.

If Cuirass would instead delegate derivation build requests to a
Coordinator/daemon-like thing, it wouldn’t have to worry about those
details.  That would better separate concerns.

This is quite a hand-wavy reply but I hope it’s useful!

Thanks,
Ludo’.




Information forwarded to bug-guix <at> gnu.org:
bug#46402; Package guix. (Wed, 10 Feb 2021 11:25:01 GMT) Full text and rfc822 format available.

Message #14 received at 46402 <at> debbugs.gnu.org (full text, mbox):

From: Mathieu Othacehe <othacehe <at> gnu.org>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: Julien Lepiller <julien <at> lepiller.eu>, 46402 <at> debbugs.gnu.org
Subject: Re: bug#46402: Cuirass rebuilds the same packae multiple times
Date: Wed, 10 Feb 2021 12:24:09 +0100
Hey Ludo,

Thanks for sharing your thoughts, it's always useful :).

> Seems to me that ‘BuildSteps’ is an orthogonal concern that has little
> to do with Cuirass’ job and with its data model.  In Hydra I saw that as
> a (necessary) kludge.

I'm not sure to follow you here. Cuirass and Hydra have an almost
identical database schema and are now working very similarly from what I
understand.

In Hydra, a JobSet (Specification in Cuirass) has several Builds. Each
Build can be broken in several BuildSteps, corresponding to transitive
derivation inputs that must be built.

Hydra manages to get those BuildSteps to be built in a topological
order, in the same way as the Guix Build Coordinator.

This makes me think that we could implement this exact same mechanism in
Cuirass but I'm maybe missing something.

> If Cuirass would instead delegate derivation build requests to a
> Coordinator/daemon-like thing, it wouldn’t have to worry about those
> details.  That would better separate concerns.

I think that having Cuirass delegating its builds to the Coordinator is
not the right move. That would mean doubling the size of the CI code
base, doubling the number of databases, for a feature that we could
implement in Cuirass, just by making it catch-up on Hydra.

Thanks,

Mathieu




Information forwarded to bug-guix <at> gnu.org:
bug#46402; Package guix. (Wed, 17 Feb 2021 14:23:02 GMT) Full text and rfc822 format available.

Message #17 received at 46402 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Mathieu Othacehe <othacehe <at> gnu.org>
Cc: Julien Lepiller <julien <at> lepiller.eu>, 46402 <at> debbugs.gnu.org
Subject: Re: bug#46402: Cuirass rebuilds the same packae multiple times
Date: Wed, 17 Feb 2021 15:22:21 +0100
Howdy!

Mathieu Othacehe <othacehe <at> gnu.org> skribis:

>> Seems to me that ‘BuildSteps’ is an orthogonal concern that has little
>> to do with Cuirass’ job and with its data model.  In Hydra I saw that as
>> a (necessary) kludge.
>
> I'm not sure to follow you here. Cuirass and Hydra have an almost
> identical database schema and are now working very similarly from what I
> understand.
>
> In Hydra, a JobSet (Specification in Cuirass) has several Builds. Each
> Build can be broken in several BuildSteps, corresponding to transitive
> derivation inputs that must be built.
>
> Hydra manages to get those BuildSteps to be built in a topological
> order, in the same way as the Guix Build Coordinator.
>
> This makes me think that we could implement this exact same mechanism in
> Cuirass but I'm maybe missing something.

When Cuirass was started, I wanted to avoid what I perceived as a
shortcoming of Hydra’s design: one daemon connection per job and build
steps, which kinda replicate what the daemon is doing.

So I suggested going for one connection for all the jobs and passing all
the derivations to the daemon so that the daemon can see the big picture,
make better scheduling decisions, and so we don’t have to re-implement
“build steps”.

But as you know, this strategy didn’t work out as expected because of
scalability issues in the daemon.


Regardless, it seems to me that ‘BuildSteps’ is a low-level thing
compared to the rest of the Cuirass database: it reifies part of the
derivation graph whereas the rest of the database is all about “jobs”
and “builds” thereof.  It’s not the same abstraction level.

I realize it’s somewhat subjective though and I don’t want to impede
progress!

>> If Cuirass would instead delegate derivation build requests to a
>> Coordinator/daemon-like thing, it wouldn’t have to worry about those
>> details.  That would better separate concerns.
>
> I think that having Cuirass delegating its builds to the Coordinator is
> not the right move. That would mean doubling the size of the CI code
> base, doubling the number of databases, for a feature that we could
> implement in Cuirass, just by making it catch-up on Hydra.

I see.  Generally speaking, I think better separation of concerns may
sometimes be worth extra code, insomuch as it makes it easier to reason
about things, to debug, and to add new features.  Of course it’s a
tradeoff; adding too much code just for the beauty of abstractions isn’t
reasonable either.

I wonder if having two databases instead of single one (which would
essentially be the union of those two databases) is a problem.  I guess
one problem is if that makes it hard to make commonly-needed “joins”
across the two databases.

Regarding features, one thing I like about the Coordinator is its
support for retrying builds, which could serve to detect flaky builds or
build processes that are kernel- or hardware-dependent.  I think it’s a
feature we’d want eventually, but I wonder if it should be Cuirass’s
job.


It’d be nice to focus on a single code base for “distributed builds” in
general, and I was hoping for a Coordinator/Cuirass convergence on this
aspect.  But at the end of the day, what matters most is what we
achieve.  Cuirass has been doing so much better on many fronts over the
last few weeks, including reliability, build throughput, and monitoring.
At the same time, the Coordinator proves useful and easy to deploy in
more experimental setups; I think Chris’s instance now aggregates
results from a variety of machines, including POWER and GNU/Hurd, and
that seemed quite easy to do.  I’m not going to complain about
over-success in this area!  :-)

Ludo’.




Reply sent to Mathieu Othacehe <othacehe <at> gnu.org>:
You have taken responsibility. (Fri, 28 May 2021 12:52:02 GMT) Full text and rfc822 format available.

Notification sent to Julien Lepiller <julien <at> lepiller.eu>:
bug acknowledged by developer. (Fri, 28 May 2021 12:52:02 GMT) Full text and rfc822 format available.

Message #22 received at 46402-done <at> debbugs.gnu.org (full text, mbox):

From: Mathieu Othacehe <othacehe <at> gnu.org>
To: Julien Lepiller <julien <at> lepiller.eu>
Cc: 46402-done <at> debbugs.gnu.org
Subject: Re: bug#46402: Cuirass rebuilds the same packae multiple times
Date: Fri, 28 May 2021 14:51:48 +0200
Hello,

> Shouldn't cuirass first schedule builds for dependencies before it
> builds dependents?

Yes fixed with
https://git.savannah.gnu.org/cgit/guix/guix-cuirass.git/commit/?id=d1a95e8b33b454a45bda506a22a8b9d9d2c8b16e.

Thanks,

Mathieu




Information forwarded to bug-guix <at> gnu.org:
bug#46402; Package guix. (Fri, 28 May 2021 18:41:01 GMT) Full text and rfc822 format available.

Message #25 received at 46402 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: 46402 <at> debbugs.gnu.org
Cc: othacehe <at> gnu.org, julien <at> lepiller.eu
Subject: Re: bug#46402: Cuirass rebuilds the same packae multiple times
Date: Fri, 28 May 2021 14:40:36 -0400
Hi,

Mathieu Othacehe <othacehe <at> gnu.org> writes:

> Hello,
>
>> Shouldn't cuirass first schedule builds for dependencies before it
>> builds dependents?
>
> Yes fixed with
> https://git.savannah.gnu.org/cgit/guix/guix-cuirass.git/commit/?id=d1a95e8b33b454a45bda506a22a8b9d9d2c8b16e.

Thumbs up!  :-)

Maxim




bug archived. Request was from Debbugs Internal Request <help-debbugs <at> gnu.org> to internal_control <at> debbugs.gnu.org. (Sat, 26 Jun 2021 11:24:04 GMT) Full text and rfc822 format available.

This bug report was last modified 2 years and 277 days ago.

Previous Next


GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.