GNU bug report logs - #49810
source tarballs potentially built for each derivation

Previous Next

To reply to this bug, email your comments to 49810 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Vagrant Cascadian <vagrant <at> debian.org>
To: bug-guix <at> gnu.org
Cc: guix-devel <guix-devel <at> gnu.org>
Subject: source tarballs potentially built for each derivation
Date: Sun, 01 Aug 2021 13:45:41 -0700

[Message part 1 (text/plain, inline)]

Turning this conversation into a bug, original thread around here:

  https://lists.gnu.org/archive/html/guix-devel/2021-05/threads.html#00427

On 2021-05-29, Vagrant Cascadian wrote:
> On 2021-05-01, Leo Famulari wrote:
>> On Sat, May 01, 2021 at 06:45:32PM -0700, Vagrant Cascadian wrote:
>>> Pragmatically speaking, on slower platforms this is a huge resource
>>> overhead. So much so that ci.guix.gnu.org *usually* times out when
>>> generating the linux-libre aarch64 tarballs:
>>> 
>>>   https://ci.guix.gnu.org/search?query=system%3Aaarch64-linux+linux-libre-arm64-generic
>>
>> Thanks for letting me know. I didn't know this was happening.
>>
>> The immediate solution is for me to make sure the tarballs have built
>> before committing the updates. I already do this for x86_64 and I can
>> start doing it for aarch64 too.
>
> This has definitely helped sometimes, thanks! I even saw a substitute of
> linux-libre for aarch64 earlier today! :)
>
> Still, I'm noticing another problem with the way way the tarballs are
> generated on ci.guix.gnu.org ...
>
> When it generates a tarball, all the various packages independently try
> to recreate the source tarball; so you have at least fours jobs
> ("linux-libre", "linux-libre-arm64-generic", "linux-libre-headers",
> "linux-libre-bpf") all concurrently trying to build the very same
> very-slow-to-build tarball on ci.guix.gnu.org. Sometimes one of them
> might succeed, but the others may not, and even though one of them
> succeeded, none of the failing ones retry...
>
> Not knowing exactly how ci.guix.gnu.org works, would it make sense to
> create a tarball package instead of the ... computed origin(?) tarball,
> so it could be better represented in the package dependency graph, and
> the various linux-libre-* packages can wait till it is available rather
> than all trying to recreate the same thing?
>
> That still requires the tarball generation to not time out in the first
> place, but maybe it would help with the resource limitations a bit to
> only build the source tarball once per architecture?

This seems to still be an issue for ci.guix.gnu.org, but the
linux-libre* substitutes for aarch64 seem to be available on
https://bordeaux.guix.gnu.org ...

$ guix weather linux-libre linux-libre-arm64-generic
computing 2 package derivations for aarch64-linux...
looking for 2 store items on https://ci.guix.gnu.org...
https://ci.guix.gnu.org
  0.0% substitutes available (0 out of 2)
  unknown substitute sizes
  0.0 MiB on disk (uncompressed)
  0.740 seconds per request (0.7 seconds in total)
  1.4 requests per second

  0.0% (0 out of 2) of the missing items are queued
  1 queued builds
      aarch64-linux: 1 (100.0%)
  build rate: .00 builds per hour
      x86_64-linux: 0.00 builds per hour
      aarch64-linux: 0.00 builds per hour
      i686-linux: 0.00 builds per hour
looking for 2 store items on https://bordeaux.guix.gnu.org...
https://bordeaux.guix.gnu.org
  100.0% substitutes available (2 out of 2)
  83.9 MiB of nars (compressed)
  202.2 MiB on disk (uncompressed)
  (continuous integration information unavailable)


live well,
  vagrant

[signature.asc (application/pgp-signature, inline)]

Message #8 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Vagrant Cascadian <vagrant <at> debian.org>
To: bug-guix <at> gnu.org
Cc: guix-devel <guix-devel <at> gnu.org>
Subject: Re: source tarballs potentially built for each derivation
Date: Sun, 01 Aug 2021 14:04:52 -0700

[Message part 1 (text/plain, inline)]

On 2021-08-01, Vagrant Cascadian wrote:
> Turning this conversation into a bug, original thread around here:
>
>   https://lists.gnu.org/archive/html/guix-devel/2021-05/threads.html#00427

For reference:

  bug#49810: source tarballs potentially built for each derivation


live well,
  vagrant

[signature.asc (application/pgp-signature, inline)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.