GNU bug report logs - #48963
"guix build guix" halts after exhausting memory

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 48963 in the body.
You can then email your comments to 48963 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Julien Lepiller <julien <at> lepiller.eu>
To: bug-guix <at> gnu.org
Subject: "guix build guix" halts after exhausting memory
Date: Sat, 12 Jun 2021 00:23:30 +0200

Hi Guix!

I tried updating my system on my armhf board (2GB of RAM), but during
"guix system reconfigure", guix tries to build itself (the guix package
from (gnu packages package-management)). This package uses too much
memory to build, and I start getting GC warnings like so:

GC Warning: Out of memory - trying to allocate requested amount (552
bytes)...
GC Warning: Header allocation failed: dropping block
GC Warning: Out of Memory! Heap size: 2571 MiB. Returning NULL!
Warning: Unwind-only out of memory exception; skipping pre-unwind
handler.

after some more progress and a lot more warnings, the build stops (CPU
is at 2%), but the memory is not freed. The build stayed stuck for a
long time until I decided to cancel it.

As a work-around, I tried using the guix-daemon package instead (by
changing the guix field in guix-configuration), but "guix system" still
wants to build the guix package anyway. I tried finding usages of the
guix package (grepping for "package-management" yields a relatively
short list of files, and none of them seems to be used by my config,
except (gnu services base) which always uses the guix from the
configuration). Why?

Can we instead break the build (at the Makefile level) into multiple
smaller chunks, that require less memory, in the same way (guix self)
works?

Message #8 received at 48963 <at> debbugs.gnu.org (full text, mbox):

From: Julien Lepiller <julien <at> lepiller.eu>
To: 48963 <at> debbugs.gnu.org
Subject: Re: bug#48963: "guix build guix" halts after exhausting memory
Date: Sat, 12 Jun 2021 00:30:14 +0200

Le Sat, 12 Jun 2021 00:23:30 +0200,
Julien Lepiller <julien <at> lepiller.eu> a écrit :

> Hi Guix!
> 
> I tried updating my system on my armhf board (2GB of RAM), but during
> "guix system reconfigure", guix tries to build itself (the guix
> package from (gnu packages package-management)). This package uses
> too much memory to build, and I start getting GC warnings like so:
> 
> GC Warning: Out of memory - trying to allocate requested amount (552
> bytes)...
> GC Warning: Header allocation failed: dropping block
> GC Warning: Out of Memory! Heap size: 2571 MiB. Returning NULL!
> Warning: Unwind-only out of memory exception; skipping pre-unwind
> handler.
> 
> after some more progress and a lot more warnings, the build stops (CPU
> is at 2%), but the memory is not freed. The build stayed stuck for a
> long time until I decided to cancel it.
> 
> As a work-around, I tried using the guix-daemon package instead (by
> changing the guix field in guix-configuration), but "guix system"
> still wants to build the guix package anyway. I tried finding usages
> of the guix package (grepping for "package-management" yields a
> relatively short list of files, and none of them seems to be used by
> my config, except (gnu services base) which always uses the guix from
> the configuration). Why?
> 
> Can we instead break the build (at the Makefile level) into multiple
> smaller chunks, that require less memory, in the same way (guix self)
> works?
> 
> 
> 

also note it's not an OOM issue at the system level, because guile
doesn't seem to care I have 8GB of free swap it could use. Top reports
1.8GB of resident memory being used by the process. I tried to stop all
services at the time in the hope it would save some memory and help,
but guile just kept swallowing all that free memory I gave it.

Message #13 received at 48963 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Julien Lepiller <julien <at> lepiller.eu>
Cc: 48963 <at> debbugs.gnu.org
Subject: Re: bug#48963: "guix build guix" halts after exhausting memory
Date: Wed, 16 Jun 2021 22:01:20 +0200

[Message part 1 (text/plain, inline)]

Hi,

Julien Lepiller <julien <at> lepiller.eu> skribis:

> I tried updating my system on my armhf board (2GB of RAM), but during
> "guix system reconfigure", guix tries to build itself (the guix package
> from (gnu packages package-management)). This package uses too much
> memory to build, and I start getting GC warnings like so:
>
> GC Warning: Out of memory - trying to allocate requested amount (552
> bytes)...
> GC Warning: Header allocation failed: dropping block
> GC Warning: Out of Memory! Heap size: 2571 MiB. Returning NULL!
> Warning: Unwind-only out of memory exception; skipping pre-unwind
> handler.
>
> after some more progress and a lot more warnings, the build stops (CPU
> is at 2%), but the memory is not freed. The build stayed stuck for a
> long time until I decided to cancel it.

This is ridiculous.  :-/

> Can we instead break the build (at the Makefile level) into multiple
> smaller chunks, that require less memory, in the same way (guix self)
> works?

Yes, that’s a good idea.

Could you check the extent to which the attached patch helps on this
machine?

It doesn’t split as much as (guix self) does, only in three pieces, but
hopefully that helps a bit.  A side effect is that progress report is
now off, but we can fix it later.

Thanks,
Ludo’.

[Message part 2 (text/x-patch, inline)]

diff --git a/Makefile.am b/Makefile.am
index aa21b5383b..758d8b9b8a 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -663,7 +663,11 @@ CLEANFILES =					\
 # the whole thing.  Likewise, set 'XDG_CACHE_HOME' to avoid loading possibly
 # stale files from ~/.cache/guile/ccache.
 %.go: make-go ; @:
-make-go: $(MODULES) guix/config.scm $(dist_noinst_DATA)
+make-go: make-core-go make-packages-go make-system-go
+
+define guile-compilation-rule 
+
+$(1): $(2)
 	$(AM_V_at)echo "Compiling Scheme modules..." ;			\
 	unset GUILE_LOAD_COMPILED_PATH ;				\
 	XDG_CACHE_HOME=/nowhere						\
@@ -671,7 +675,19 @@ make-go: $(MODULES) guix/config.scm $(dist_noinst_DATA)
 	$(top_builddir)/pre-inst-env					\
 	$(GUILE) -L "$(top_builddir)" -L "$(top_srcdir)"		\
 	  --no-auto-compile 						\
-	  -s "$(top_srcdir)"/build-aux/compile-all.scm $^
+	  -s "$(top_srcdir)"/build-aux/compile-all.scm $$(filter %.scm,$$^)
+
+.PHONY: $(1)
+
+endef
+
+MODULES_CORE     = $(filter guix/%,$(MODULES))
+MODULES_PACKAGES = $(filter gnu/packages/%,$(MODULES))
+MODULES_SYSTEM   = $(filter-out gnu/packages/%,$(filter gnu/%,$(MODULES)))
+
+$(eval $(call guile-compilation-rule,make-core-go,$(MODULES_CORE) guix/config.scm $(dist_noinst_DATA)))
+$(eval $(call guile-compilation-rule,make-packages-go,$(MODULES_PACKAGES) make-core-go))
+$(eval $(call guile-compilation-rule,make-system-go,$(MODULES_SYSTEM) make-packages-go make-core-go))
 
 SUFFIXES = .go
 

Message #16 received at 48963 <at> debbugs.gnu.org (full text, mbox):

From: Julien Lepiller <julien <at> lepiller.eu>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 48963 <at> debbugs.gnu.org
Subject: Re: bug#48963: "guix build guix" halts after exhausting memory
Date: Thu, 17 Jun 2021 09:58:09 -0400

[Message part 1 (text/plain, inline)]

Hi Ludo,

I tried your patch by creating a variant of the guix package. My first attempt was a failure because it's still using almost all my memory and I forgot to enable my swap. In the second attempt, the build phase succeeded, but the build failed during the test phase (test/inferior). I've started a new build hoping it's a non-deterministic failure.

Le 16 juin 2021 16:01:20 GMT-04:00, "Ludovic Courtès" <ludo <at> gnu.org> a écrit :
>Hi,
>
>Julien Lepiller <julien <at> lepiller.eu> skribis:
>
>> I tried updating my system on my armhf board (2GB of RAM), but during
>> "guix system reconfigure", guix tries to build itself (the guix
>package
>> from (gnu packages package-management)). This package uses too much
>> memory to build, and I start getting GC warnings like so:
>>
>> GC Warning: Out of memory - trying to allocate requested amount (552
>> bytes)...
>> GC Warning: Header allocation failed: dropping block
>> GC Warning: Out of Memory! Heap size: 2571 MiB. Returning NULL!
>> Warning: Unwind-only out of memory exception; skipping pre-unwind
>> handler.
>>
>> after some more progress and a lot more warnings, the build stops
>(CPU
>> is at 2%), but the memory is not freed. The build stayed stuck for a
>> long time until I decided to cancel it.
>
>This is ridiculous.  :-/
>
>> Can we instead break the build (at the Makefile level) into multiple
>> smaller chunks, that require less memory, in the same way (guix self)
>> works?
>
>Yes, that’s a good idea.
>
>Could you check the extent to which the attached patch helps on this
>machine?
>
>It doesn’t split as much as (guix self) does, only in three pieces, but
>hopefully that helps a bit.  A side effect is that progress report is
>now off, but we can fix it later.
>
>Thanks,
>Ludo’.

[Message part 2 (text/html, inline)]

Message #19 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Julien Lepiller <julien <at> lepiller.eu>
To: bug-guix <at> gnu.org
Subject: Re: bug#48963: "guix build guix" halts after exhausting memory
Date: Fri, 18 Jun 2021 00:17:37 +0200

Le Thu, 17 Jun 2021 09:58:09 -0400,
Julien Lepiller <julien <at> lepiller.eu> a écrit :

> Hi Ludo,
> 
> I tried your patch by creating a variant of the guix package. My
> first attempt was a failure because it's still using almost all my
> memory and I forgot to enable my swap. In the second attempt, the
> build phase succeeded, but the build failed during the test phase
> (test/inferior). I've started a new build hoping it's a
> non-deterministic failure.

I'm afraid after testing 2 more times that the test failure is
happening consistently, at least of armhf.

Message #22 received at 48963 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Julien Lepiller <julien <at> lepiller.eu>
Cc: 48963 <at> debbugs.gnu.org
Subject: Re: bug#48963: "guix build guix" halts after exhausting memory
Date: Fri, 18 Jun 2021 11:51:14 +0200

Hi,

Julien Lepiller <julien <at> lepiller.eu> skribis:

> Le Thu, 17 Jun 2021 09:58:09 -0400,
> Julien Lepiller <julien <at> lepiller.eu> a écrit :
>
>> Hi Ludo,
>> 
>> I tried your patch by creating a variant of the guix package. My
>> first attempt was a failure because it's still using almost all my
>> memory and I forgot to enable my swap. In the second attempt, the
>> build phase succeeded, but the build failed during the test phase
>> (test/inferior). I've started a new build hoping it's a
>> non-deterministic failure.
>
> I'm afraid after testing 2 more times that the test failure is
> happening consistently, at least of armhf.

(The _build_ failure, right?)

Have you been able to see how this affects max RSS?

Does it go any further than without the patch?

Thanks,
Ludo’.

Message #25 received at 48963 <at> debbugs.gnu.org (full text, mbox):

From: Julien Lepiller <julien <at> lepiller.eu>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 48963 <at> debbugs.gnu.org
Subject: Re: bug#48963: "guix build guix" halts after exhausting memory
Date: Fri, 18 Jun 2021 06:52:10 -0400

[Message part 1 (text/plain, inline)]

No, the _test_ failure is consistent. The build itself now always completes, which is much better that before.

Le 18 juin 2021 05:51:14 GMT-04:00, "Ludovic Courtès" <ludo <at> gnu.org> a écrit :
>Hi,
>
>Julien Lepiller <julien <at> lepiller.eu> skribis:
>
>> Le Thu, 17 Jun 2021 09:58:09 -0400,
>> Julien Lepiller <julien <at> lepiller.eu> a écrit :
>>
>>> Hi Ludo,
>>> 
>>> I tried your patch by creating a variant of the guix package. My
>>> first attempt was a failure because it's still using almost all my
>>> memory and I forgot to enable my swap. In the second attempt, the
>>> build phase succeeded, but the build failed during the test phase
>>> (test/inferior). I've started a new build hoping it's a
>>> non-deterministic failure.
>>
>> I'm afraid after testing 2 more times that the test failure is
>> happening consistently, at least of armhf.
>
>(The _build_ failure, right?)
>
>Have you been able to see how this affects max RSS?
>
>Does it go any further than without the patch?
>
>Thanks,
>Ludo’.

[Message part 2 (text/html, inline)]

Message #28 received at 48963 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Julien Lepiller <julien <at> lepiller.eu>
Cc: 48963 <at> debbugs.gnu.org
Subject: Re: bug#48963: "guix build guix" halts after exhausting memory
Date: Sun, 20 Jun 2021 23:00:53 +0200

Hi!

Julien Lepiller <julien <at> lepiller.eu> skribis:

> No, the _test_ failure is consistent. The build itself now always completes, which is much better that before.

Ah, that’s great!  I can work on a variant of this patch that shows
correct completion numbers.

Could you share ‘tests/inferiors.log’ if that’s the one that’s failing?
Is it also failing for you on x86_64?

Thanks,
Ludo’.

Message #31 received at 48963 <at> debbugs.gnu.org (full text, mbox):

From: Julien Lepiller <julien <at> lepiller.eu>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 48963 <at> debbugs.gnu.org
Subject: Re: bug#48963: "guix build guix" halts after exhausting memory
Date: Mon, 21 Jun 2021 13:05:10 +0200

Le Sun, 20 Jun 2021 23:00:53 +0200,
Ludovic Courtès <ludo <at> gnu.org> a écrit :

> Hi!
> 
> Julien Lepiller <julien <at> lepiller.eu> skribis:
> 
> > No, the _test_ failure is consistent. The build itself now always
> > completes, which is much better that before.  
> 
> Ah, that’s great!  I can work on a variant of this patch that shows
> correct completion numbers.
> 
> Could you share ‘tests/inferiors.log’ if that’s the one that’s
> failing? Is it also failing for you on x86_64?
> 
> Thanks,
> Ludo’.

Here is the content of `tests/inferiors.log`. The failure is the same
on armhf and x86_64:

Backtrace:
           2 (primitive-load-path "tests/inferior.scm")
In ice-9/eval.scm:
   626:19  1 (_ #<directory (test-inferior) 7ffff5d7ad20>)
In unknown file:
           0 (dirname #f)

ERROR: In procedure dirname:
In procedure scm_to_utf8_stringn: Wrong type argument in position 1
(expecting string): #f

Message #36 received at 48963-done <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Julien Lepiller <julien <at> lepiller.eu>
Cc: 48963-done <at> debbugs.gnu.org
Subject: Re: bug#48963: "guix build guix" halts after exhausting memory
Date: Wed, 23 Jun 2021 23:43:55 +0200

Hi,

Ludovic Courtès <ludo <at> gnu.org> skribis:

> Julien Lepiller <julien <at> lepiller.eu> skribis:

>> Can we instead break the build (at the Makefile level) into multiple
>> smaller chunks, that require less memory, in the same way (guix self)
>> works?
>
> Yes, that’s a good idea.
>
> Could you check the extent to which the attached patch helps on this
> machine?
>
> It doesn’t split as much as (guix self) does, only in three pieces, but
> hopefully that helps a bit.  A side effect is that progress report is
> now off, but we can fix it later.

I pushed a variant of this patch, resorting to an ugly hack so that
compile-all.scm can still estimate progress.  The estimate is correct if
you’re building all the .go files, not so much if you just do, say:

  make
  rm guix/cpio.go && make

because guix/cpio.scm is in the first module set, so it’ll think it’s
starting from scratch and display “0%”.

Fixing it is left as an exercise to the reader.  :-)

It seemed more important to me to fix the memory exhaustion issue first.

Thanks,
Ludo’.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.