GNU logs - #59365, boring messages

Message sent to bug-guix@HIDDEN:

Subject: bug#59365: make-dynamic-linker-cache OOMs for LLVM 15 on i686-linux
Resent-From: Marius Bakke <marius@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Fri, 18 Nov 2022 18:42:02 +0000
Resent-Message-ID: <handler.59365.B.166879688615077 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: 59365 <at> debbugs.gnu.org
From: Marius Bakke <marius@HIDDEN>
Date: Fri, 18 Nov 2022 19:41:18 +0100
Message-ID: <87mt8o14xd.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="=-=-=";
 micalg=pgp-sha512; protocol="application/pgp-signature"
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

--=-=-=
Content-Type: text/plain

Hello,

LLVM 15.0.4 fails on i686-linux:

  https://ci.guix.gnu.org/build/1702995/details

Because the 'make-dynamic-linker-cache' phase runs out of memory:

  starting phase `make-dynamic-linker-cache'
  GC Warning: Repeated allocation of very large block (appr. size 268439552):
	May lead to memory leak and poor performance
  GC Warning: Repeated allocation of very large block (appr. size 134221824):
	May lead to memory leak and poor performance
  GC Warning: Repeated allocation of very large block (appr. size 268439552):
  	May lead to memory leak and poor performance
  GC Warning: Failed to expand heap by 285216768 bytes
  GC Warning: Failed to expand heap by 268439552 bytes
  GC Warning: Out of Memory! Heap size: 3620 MiB. Returning NULL!
  Warning: Unwind-only out of memory exception; skipping pre-unwind handler.
  Warning: Unwind-only out of memory exception; skipping pre-unwind handler.
  Warning: Unwind-only out of memory exception; skipping pre-unwind handler.

(excerpt from https://ci.guix.gnu.org/build/1702995/log/raw)

Not sure why this phase uses so much memory.  Ideas?

--=-=-=
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iIUEARYKAC0WIQRNTknu3zbaMQ2ddzTocYulkRQQdwUCY3fRzg8cbWFyaXVzQGdu
dS5vcmcACgkQ6HGLpZEUEHdg4AEAlgoHstuJbIvzu6VNET3NpLOzagBsqFCMMulA
3qOfm/cBAKpm1PeRtKIKv40YW/HnXhVkUNKzNVizYDfDt8pFi0kA
=OSPc
-----END PGP SIGNATURE-----
--=-=-=--

Message sent:

Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Mailer: MIME-tools 5.505 (Entity 5.505)
Content-Type: text/plain; charset=utf-8
X-Loop: help-debbugs@HIDDEN
From: help-debbugs@HIDDEN (GNU bug Tracking System)
To: Marius Bakke <marius@HIDDEN>
Subject: bug#59365: Acknowledgement (make-dynamic-linker-cache OOMs for
 LLVM 15 on i686-linux)
Message-ID: <handler.59365.B.166879688615077.ack <at> debbugs.gnu.org>
References: <87mt8o14xd.fsf@HIDDEN>
X-Gnu-PR-Message: ack 59365
X-Gnu-PR-Package: guix
Reply-To: 59365 <at> debbugs.gnu.org
Date: Fri, 18 Nov 2022 18:42:02 +0000

Thank you for filing a new bug report with debbugs.gnu.org.

This is an automatically generated reply to let you know your message
has been received.

Your message is being forwarded to the package maintainers and other
interested parties for their attention; they will reply in due course.

Your message has been sent to the package maintainer(s):
 bug-guix@HIDDEN

If you wish to submit further information on this problem, please
send it to 59365 <at> debbugs.gnu.org.

Please do not send mail to help-debbugs@HIDDEN unless you wish
to report a problem with the Bug-tracking system.

--=20
59365: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D59365
GNU Bug Tracking System
Contact help-debbugs@HIDDEN with problems

Message sent to bug-guix@HIDDEN:

Subject: bug#59365: make-dynamic-linker-cache OOMs for LLVM 15 on i686-linux
Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Tue, 22 Nov 2022 08:43:02 +0000
Resent-Message-ID: <handler.59365.B59365.166910655421619 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: Marius Bakke <marius@HIDDEN>
Cc: 59365 <at> debbugs.gnu.org, Maxim Cournoyer <maxim.cournoyer@HIDDEN>, Greg Hogan <code@HIDDEN>
From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
References: <87mt8o14xd.fsf@HIDDEN>
Date: Tue, 22 Nov 2022 09:42:19 +0100
In-Reply-To: <87mt8o14xd.fsf@HIDDEN> (Marius Bakke's message of "Fri, 18 Nov
 2022 19:41:18 +0100")
Message-ID: <877cznidn8.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="=-=-="
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

--=-=-=
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

Hi,

(Cc: Maxim and Greg for LLVM packaging questions below.)

Marius Bakke <marius@HIDDEN> skribis:

> LLVM 15.0.4 fails on i686-linux:
>
>   https://ci.guix.gnu.org/build/1702995/details
>
> Because the 'make-dynamic-linker-cache' phase runs out of memory:
>
>   starting phase `make-dynamic-linker-cache'
>   GC Warning: Repeated allocation of very large block (appr. size 2684395=
52):
> 	May lead to memory leak and poor performance
>   GC Warning: Repeated allocation of very large block (appr. size 1342218=
24):
> 	May lead to memory leak and poor performance
>   GC Warning: Repeated allocation of very large block (appr. size 2684395=
52):
>   	May lead to memory leak and poor performance
>   GC Warning: Failed to expand heap by 285216768 bytes
>   GC Warning: Failed to expand heap by 268439552 bytes
>   GC Warning: Out of Memory! Heap size: 3620 MiB. Returning NULL!
>   Warning: Unwind-only out of memory exception; skipping pre-unwind handl=
er.
>   Warning: Unwind-only out of memory exception; skipping pre-unwind handl=
er.
>   Warning: Unwind-only out of memory exception; skipping pre-unwind handl=
er.
>
> (excerpt from https://ci.guix.gnu.org/build/1702995/log/raw)
>
> Not sure why this phase uses so much memory.  Ideas?

Yes: the gremlin.scm code uses =E2=80=98file-dynamic-info=E2=80=99, which l=
oads the
whole file in memory.  Ridiculous.

We should instead mmap it (but there are no =E2=80=98mmap=E2=80=99 bindings=
 in Guile,
yet) or arrange to load just the relevant parts (we=E2=80=99ll have to chec=
k but
maybe =E2=80=98file-dynamic-info=E2=80=99 can find everything it needs at t=
he beginning
of a file, the PT_DYNAMIC segment.)

For example, with the patch below, things still appear to be fine with
LLVM:

--8<---------------cut here---------------start------------->8---
scheme@(guix build gremlin)> (file-dynamic-info "/gnu/store/mj14k58lfc88jhc=
n6va0s2fpwkv3s35c-llvm-13.0.1/lib/libLLVMScalarOpts.so")
$11 =3D #<<elf-dynamic-info> soname: "libLLVMScalarOpts.so.13" needed: ("li=
bLLVMAggressiveInstCombine.so.13" "libLLVMInstCombine.so.13" "libLLVMTransf=
ormUtils.so.13" "libLLVMAnalysis.so.13" "libLLVMCore.so.13" "libLLVMSupport=
.so.13" "libstdc++.so.6" "libm.so.6" "libgcc_s.so.1" "libc.so.6" "ld-linux-=
x86-64.so.2") rpath: () runpath: ("/gnu/store/mj14k58lfc88jhcn6va0s2fpwkv3s=
35c-llvm-13.0.1/lib" "/gnu/store/5h2w4qi9hk1qzzgi1w83220ydslinr4s-glibc-2.3=
3/lib" "/gnu/store/094bbaq6glba86h1d4cj16xhdi6fk2jl-gcc-10.3.0-lib/lib" "/g=
nu/store/094bbaq6glba86h1d4cj16xhdi6fk2jl-gcc-10.3.0-lib/lib/gcc/x86_64-unk=
nown-linux-gnu/10.3.0/../../..")>
scheme@(guix build gremlin)> (file-dynamic-info "/gnu/store/mj14k58lfc88jhc=
n6va0s2fpwkv3s35c-llvm-13.0.1/lib/libLLVMX86CodeGen.so.13")
$12 =3D #<<elf-dynamic-info> soname: "libLLVMX86CodeGen.so.13" needed: ("li=
bLLVMAsmPrinter.so.13" "libLLVMX86Desc.so.13" "libLLVMX86Info.so.13" "libLL=
VMGlobalISel.so.13" "libLLVMCFGuard.so.13" "libLLVMSelectionDAG.so.13" "lib=
LLVMCodeGen.so.13" "libLLVMTarget.so.13" "libLLVMTransformUtils.so.13" "lib=
LLVMAnalysis.so.13" "libLLVMProfileData.so.13" "libLLVMMC.so.13" "libLLVMCo=
re.so.13" "libLLVMSupport.so.13" "libstdc++.so.6" "libm.so.6" "libgcc_s.so.=
1" "libc.so.6" "ld-linux-x86-64.so.2") rpath: () runpath: ("/gnu/store/mj14=
k58lfc88jhcn6va0s2fpwkv3s35c-llvm-13.0.1/lib" "/gnu/store/5h2w4qi9hk1qzzgi1=
w83220ydslinr4s-glibc-2.33/lib" "/gnu/store/094bbaq6glba86h1d4cj16xhdi6fk2j=
l-gcc-10.3.0-lib/lib" "/gnu/store/094bbaq6glba86h1d4cj16xhdi6fk2jl-gcc-10.3=
.0-lib/lib/gcc/x86_64-unknown-linux-gnu/10.3.0/../../..")>
--8<---------------cut here---------------end--------------->8---

We could temporarily delete this phase for all 32-bit builds of LLVM.

But the crux of the problem is that llvm@15 has a single huge shared
library, unlike previous versions:

--8<---------------cut here---------------start------------->8---
$ du -hL  /gnu/store/bgqdvvi7k6l255332rfawgjmn2hpn13r-llvm-15.0.4/lib/*.so
133M    /gnu/store/bgqdvvi7k6l255332rfawgjmn2hpn13r-llvm-15.0.4/lib/libLLVM=
-15.0.4.so
96K     /gnu/store/bgqdvvi7k6l255332rfawgjmn2hpn13r-llvm-15.0.4/lib/libLTO.=
so
16K     /gnu/store/bgqdvvi7k6l255332rfawgjmn2hpn13r-llvm-15.0.4/lib/libRema=
rks.so
--8<---------------cut here---------------end--------------->8---

(It also has tons of .a files, which shouldn=E2=80=99t be there.)

Is that big LLVM.so due to different build options on our side?  Or is
it a radical upstream change (sounds unlikely, but who knows)?

Thanks,
Ludo=E2=80=99.


--=-=-=
Content-Type: text/x-patch
Content-Disposition: inline

diff --git a/guix/build/gremlin.scm b/guix/build/gremlin.scm
index 2a74d51dd9..8a38dde1eb 100644
--- a/guix/build/gremlin.scm
+++ b/guix/build/gremlin.scm
@@ -250,7 +250,11 @@ (define (file-dynamic-info file)
 info."
   (call-with-input-file file
     (lambda (port)
-      (elf-dynamic-info (parse-elf (get-bytevector-all port))))))
+      (elf-dynamic-info (parse-elf
+                         ;; Read at most 10 MiB in memory, which should be
+                         ;; enough to get the PT_DYNAMIC segment.
+                         ;; TODO: mmap the whole file instead.
+                         (get-bytevector-n port (* 10 (expt 2 20))))))))
 
 (define (file-runpath file)
   "Return the DT_RUNPATH dynamic entry of FILE as a list of strings, or #f if

--=-=-=--

Message sent to bug-guix@HIDDEN:

Subject: bug#59365: make-dynamic-linker-cache OOMs for LLVM 15 on i686-linux
Resent-From: Maxim Cournoyer <maxim.cournoyer@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Tue, 22 Nov 2022 10:23:02 +0000
Resent-Message-ID: <handler.59365.B59365.166911257831784 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Cc: 59365 <at> debbugs.gnu.org, Greg Hogan <code@HIDDEN>, Marius Bakke <marius@HIDDEN>
From: Maxim Cournoyer <maxim.cournoyer@HIDDEN>
References: <87mt8o14xd.fsf@HIDDEN> <877cznidn8.fsf@HIDDEN>
Date: Tue, 22 Nov 2022 05:22:48 -0500
In-Reply-To: <877cznidn8.fsf@HIDDEN> ("Ludovic =?UTF-8?Q?Court=C3=A8s?="'s message of "Tue, 22 Nov 2022 09:42:19 +0100")
Message-ID: <874jurp9tz.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

Hi Ludovic,

Ludovic Court=C3=A8s <ludo@HIDDEN> writes:

> Hi,
>
> (Cc: Maxim and Greg for LLVM packaging questions below.)
>
> Marius Bakke <marius@HIDDEN> skribis:
>
>> LLVM 15.0.4 fails on i686-linux:
>>
>>   https://ci.guix.gnu.org/build/1702995/details
>>
>> Because the 'make-dynamic-linker-cache' phase runs out of memory:
>>
>>   starting phase `make-dynamic-linker-cache'
>>   GC Warning: Repeated allocation of very large block (appr. size 268439=
552):
>> 	May lead to memory leak and poor performance
>>   GC Warning: Repeated allocation of very large block (appr. size 134221=
824):
>> 	May lead to memory leak and poor performance
>>   GC Warning: Repeated allocation of very large block (appr. size 268439=
552):
>>   	May lead to memory leak and poor performance
>>   GC Warning: Failed to expand heap by 285216768 bytes
>>   GC Warning: Failed to expand heap by 268439552 bytes
>>   GC Warning: Out of Memory! Heap size: 3620 MiB. Returning NULL!
>>   Warning: Unwind-only out of memory exception; skipping pre-unwind hand=
ler.
>>   Warning: Unwind-only out of memory exception; skipping pre-unwind hand=
ler.
>>   Warning: Unwind-only out of memory exception; skipping pre-unwind hand=
ler.
>>
>> (excerpt from https://ci.guix.gnu.org/build/1702995/log/raw)
>>
>> Not sure why this phase uses so much memory.  Ideas?
>
> Yes: the gremlin.scm code uses =E2=80=98file-dynamic-info=E2=80=99, which=
 loads the
> whole file in memory.  Ridiculous.

If it loaded just that file, it should be fine, no?  It weighs 133 MiB,
as you've shown below:

> But the crux of the problem is that llvm@15 has a single huge shared
> library, unlike previous versions:
>
> $ du -hL  /gnu/store/bgqdvvi7k6l255332rfawgjmn2hpn13r-llvm-15.0.4/lib/*.so
> 133M    /gnu/store/bgqdvvi7k6l255332rfawgjmn2hpn13r-llvm-15.0.4/lib/libLL=
VM-15.0.4.so
> 96K     /gnu/store/bgqdvvi7k6l255332rfawgjmn2hpn13r-llvm-15.0.4/lib/libLT=
O.so
> 16K     /gnu/store/bgqdvvi7k6l255332rfawgjmn2hpn13r-llvm-15.0.4/lib/libRe=
marks.so
> (It also has tons of .a files, which shouldn=E2=80=99t be there.)

The static files are needed at least to build the clang runtime.  I had
tried to get rid of them without success.  Perhaps they could be moved
to a "static" output if they're needed only at that time.

> Is that big LLVM.so due to different build options on our side?  Or is
> it a radical upstream change (sounds unlikely, but who knows)?

It's caused by -DLLVM_LINK_LLVM_DYLIB=3DON and -DLLVM_BUILD_LLVM_DYLIB=3DON,
which is the supported configuration to build a shared library of LLVM
(-DBUILD_SHARED_LIBS=3DON is obsolete/deprecated) [0].

It also makes things conveniently easy to link to LLVM; you just need to
link to '-lLLVM', and everything it needs is available.

[0]  https://llvm.org/docs/BuildingADistribution.html#special-notes-for-lib=
rary-only-distributions

--=20
Thanks,
Maxim

Message sent to bug-guix@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#59365: make-dynamic-linker-cache OOMs for LLVM 15 on i686-linux
Resent-From: Maxim Cournoyer <maxim.cournoyer@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Mon, 20 Nov 2023 19:34:02 +0000
Resent-Message-ID: <handler.59365.B59365.170050883817660 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 59365
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Cc: 59365 <at> debbugs.gnu.org, Greg Hogan <code@HIDDEN>, Marius Bakke <marius@HIDDEN>
Received: via spool by 59365-submit <at> debbugs.gnu.org id=B59365.170050883817660
          (code B ref 59365); Mon, 20 Nov 2023 19:34:02 +0000
Received: (at 59365) by debbugs.gnu.org; 20 Nov 2023 19:33:58 +0000
Received: from localhost ([127.0.0.1]:54639 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1r5A1t-0004al-Se
	for submit <at> debbugs.gnu.org; Mon, 20 Nov 2023 14:33:58 -0500
Received: from mail-ot1-x32f.google.com ([2607:f8b0:4864:20::32f]:54402)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <maxim.cournoyer@HIDDEN>) id 1r5A1r-0004aR-8K
 for 59365 <at> debbugs.gnu.org; Mon, 20 Nov 2023 14:33:57 -0500
Received: by mail-ot1-x32f.google.com with SMTP id
 46e09a7af769-6cd0963c61cso2534806a34.0
 for <59365 <at> debbugs.gnu.org>; Mon, 20 Nov 2023 11:33:52 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=gmail.com; s=20230601; t=1700508827; x=1701113627; darn=debbugs.gnu.org;
 h=mime-version:user-agent:message-id:date:references:in-reply-to
 :subject:cc:to:from:from:to:cc:subject:date:message-id:reply-to;
 bh=SkEiuFoOLf5/xhpAe0vnn9QdkTTJTJtQzEeJ2ksdA9M=;
 b=g/bQ+kxSz7wRGq710Nk4H8ae0eT3sM3fVz8jHFDQNwb381OfP4ybt5XcIBu2nRNUt0
 53uBXeaOrv578a2UP84SHgWZ9EtyYZugtWiJ23WarxPx6MiUsO87QGO7Y3l2v8jwE+jc
 7WaS2IJIwN4rCRWjkgeEfIzDZyYovoPWENSm2u37kIDQAAH6cqFIfHoUNOOYh/zHwQJu
 RNmQIXU3fQ40988nT3mkKtb8uiyT/pke/HWPBU9rg/2eze7k6//chtGubSEoMBbO4qtn
 PGLxDHtq9VcnDapZ6CGKX001sh1Tz84345lcmsazvqBPQq+mRJxAxc8s9veXl468WoGl
 5ovg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20230601; t=1700508827; x=1701113627;
 h=mime-version:user-agent:message-id:date:references:in-reply-to
 :subject:cc:to:from:x-gm-message-state:from:to:cc:subject:date
 :message-id:reply-to;
 bh=SkEiuFoOLf5/xhpAe0vnn9QdkTTJTJtQzEeJ2ksdA9M=;
 b=EKzqh8FE0fJcWTaE4wJw7Lvfs0kbLWviCO/dlB56RUGcYe0P/x1wplcEIuyPGPnbjl
 jhT8fFzR2qvZZYvvVFyJ510vktzgyDqSi0wmKHWr8s7+BodIZTMUeUVfrWzNPvmZpNTI
 tGhjqGvquEEY4gA3/a/4GEp4toEdgXYgMg+eP5OaWDbyi9+CMOmSuiPS/Y7iG5QtKBpA
 mfyXwmGRPG7yEDkg+OkSoHHUowxulW0zKMDV0/Txwaj9JLAjzp0ykQFxbuBNxveo/sc1
 jFwE6Hw6cALm8HxWGw84/6yMVsO7SSoDBbpe1QiUHcWu1d7z9Uq6j7R/ViCqXLIEfQRi
 KWWw==
X-Gm-Message-State: AOJu0YzngwtaIDQgrtrrXFeEEqT8g0jXTqzPRBZjGjgmdGfrdao0f+d4
 vnis7mh/Pkq316H4i7GJ92I=
X-Google-Smtp-Source: AGHT+IEHclmGSoYgxxzlHjV4AoA1o8UWAJhLV0GKKhuTaFeghiTLibujE3smbUepBSDHsNwQkqelvQ==
X-Received: by 2002:a05:6830:1495:b0:6bd:a47:7bb6 with SMTP id
 s21-20020a056830149500b006bd0a477bb6mr6976658otq.14.1700508827239; 
 Mon, 20 Nov 2023 11:33:47 -0800 (PST)
Received: from hurd (dsl-154-9.b2b2c.ca. [66.158.154.9])
 by smtp.gmail.com with ESMTPSA id
 pr15-20020a056214140f00b0065823d20381sm3155602qvb.8.2023.11.20.11.33.46
 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
 Mon, 20 Nov 2023 11:33:46 -0800 (PST)
From: Maxim Cournoyer <maxim.cournoyer@HIDDEN>
In-Reply-To: <874jurp9tz.fsf@HIDDEN> (Maxim Cournoyer's message of "Tue, 22
 Nov 2022 05:22:48 -0500")
References: <87mt8o14xd.fsf@HIDDEN> <877cznidn8.fsf@HIDDEN>
 <874jurp9tz.fsf@HIDDEN>
Date: Mon, 20 Nov 2023 14:33:45 -0500
Message-ID: <87bkbo9q3q.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Hi,

This still happens:

--8<---------------cut here---------------start------------->8---
starting phase `make-dynamic-linker-cache'
GC Warning: Repeated allocation of very large block (appr. size 16781312):
	May lead to memory leak and poor performance
GC Warning: Repeated allocation of very large block (appr. size 67112960):
	May lead to memory leak and poor performance
GC Warning: Repeated allocation of very large block (appr. size 134221824):
	May lead to memory leak and poor performance
GC Warning: Repeated allocation of very large block (appr. size 67112960):
	May lead to memory leak and poor performance
GC Warning: Repeated allocation of very large block (appr. size 67112960):
	May lead to memory leak and poor performance
GC Warning: Repeated allocation of very large block (appr. size 134221824):
	May lead to memory leak and poor performance
GC Warning: Repeated allocation of very large block (appr. size 134221824):
	May lead to memory leak and poor performance
GC Warning: Repeated allocation of very large block (appr. size 134221824):
	May lead to memory leak and poor performance
GC Warning: Repeated allocation of very large block (appr. size 268439552):
	May lead to memory leak and poor performance
GC Warning: Repeated allocation of very large block (appr. size 33558528):
	May lead to memory leak and poor performance
GC Warning: Repeated allocation of very large block (appr. size 67112960):
	May lead to memory leak and poor performance
GC Warning: Repeated allocation of very large block (appr. size 67112960):
	May lead to memory leak and poor performance
GC Warning: Repeated allocation of very large block (appr. size 268439552):
	May lead to memory leak and poor performance
GC Warning: Repeated allocation of very large block (appr. size 268439552):
	May lead to memory leak and poor performance
GC Warning: Repeated allocation of very large block (appr. size 268439552):
	May lead to memory leak and poor performance
GC Warning: Failed to expand heap by 285216768 bytes
GC Warning: Failed to expand heap by 268439552 bytes
GC Warning: Out of Memory! Heap size: 3362 MiB. Returning NULL!
Warning: Unwind-only out of memory exception; skipping pre-unwind handler.
Warning: Unwind-only out of memory exception; skipping pre-unwind handler.
Warning: Unwind-only out of memory exception; skipping pre-unwind handler.
builder for
`/gnu/store/j4w1wrhgpjjcfqf2jskklr95r6hpy51i-llvm-15.0.7.drv' failed
with exit code 1
--8<---------------cut here---------------end--------------->8---

See this recent build: https://ci.guix.gnu.org/build/2602334/details

-- 
Thanks,
Maxim

Message sent to bug-guix@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#59365: make-dynamic-linker-cache OOMs for LLVM 15 on i686-linux
Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludovic.courtes@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Fri, 12 Apr 2024 09:37:03 +0000
Resent-Message-ID: <handler.59365.B59365.171291457111638 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 59365
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: 59365 <at> debbugs.gnu.org
Cc: Romain Garbage <romain.garbage@HIDDEN>, Greg Hogan <code@HIDDEN>, Maxim Cournoyer <maxim.cournoyer@HIDDEN>, Marius Bakke <marius@HIDDEN>
Received: via spool by 59365-submit <at> debbugs.gnu.org id=B59365.171291457111638
          (code B ref 59365); Fri, 12 Apr 2024 09:37:03 +0000
Received: (at 59365) by debbugs.gnu.org; 12 Apr 2024 09:36:11 +0000
Received: from localhost ([127.0.0.1]:57890 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1rvDKM-00031R-1Y
	for submit <at> debbugs.gnu.org; Fri, 12 Apr 2024 05:36:11 -0400
Received: from mail3-relais-sop.national.inria.fr ([192.134.164.104]:58021)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludovic.courtes@HIDDEN>) id 1rvDKF-0002zU-9m
 for 59365 <at> debbugs.gnu.org; Fri, 12 Apr 2024 05:36:07 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=inria.fr; s=dc;
 h=from:to:cc:subject:in-reply-to:references:date:
 message-id:mime-version:content-transfer-encoding;
 bh=mkSSTBDlziE099EF+00sj6ds+WxmMVU8AyZInxGilnc=;
 b=ruKAaKjt7yMNp9GOR30ggdc5zVbFW9drYw7wzDS0R2JpqJZKBm9PqE92
 h/tKEP/bVvI6uangtLLLUghdY9BqMIM1e6eeSKCDHC4pHwhQa1nMW62OX
 dPcX/O1Oby3KjXyoKu02ClI4dV5SbABHmN0bNTfXNYgJwdNt5PJIbfc9w 8=;
Authentication-Results: mail3-relais-sop.national.inria.fr;
 dkim=none (message not signed) header.i=none;
 spf=SoftFail smtp.mailfrom=ludovic.courtes@HIDDEN;
 dmarc=fail (p=none dis=none) d=inria.fr
X-IronPort-AV: E=Sophos;i="6.07,195,1708383600"; d="scan'208";a="84585832"
Received: from unknown (HELO ribbon) ([193.50.110.177])
 by mail3-relais-sop.national.inria.fr with
 ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Apr 2024 11:35:47 +0200
From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludovic.courtes@HIDDEN>
In-Reply-To: <877cznidn8.fsf@HIDDEN> ("Ludovic =?UTF-8?Q?Court=C3=A8s?="'s message of "Tue, 22 Nov 2022 09:42:19 +0100")
References: <87mt8o14xd.fsf@HIDDEN> <877cznidn8.fsf@HIDDEN>
Date: Fri, 12 Apr 2024 11:35:47 +0200
Message-ID: <87le5igbyk.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Hello,

Ludovic Court=C3=A8s <ludo@HIDDEN> skribis:

>>   GC Warning: Failed to expand heap by 285216768 bytes
>>   GC Warning: Failed to expand heap by 268439552 bytes
>>   GC Warning: Out of Memory! Heap size: 3620 MiB. Returning NULL!
>>   Warning: Unwind-only out of memory exception; skipping pre-unwind hand=
ler.
>>   Warning: Unwind-only out of memory exception; skipping pre-unwind hand=
ler.
>>   Warning: Unwind-only out of memory exception; skipping pre-unwind hand=
ler.
>>
>> (excerpt from https://ci.guix.gnu.org/build/1702995/log/raw)
>>
>> Not sure why this phase uses so much memory.  Ideas?
>
> Yes: the gremlin.scm code uses =E2=80=98file-dynamic-info=E2=80=99, which=
 loads the
> whole file in memory.  Ridiculous.
>
> We should instead mmap it (but there are no =E2=80=98mmap=E2=80=99 bindin=
gs in Guile,
> yet) or arrange to load just the relevant parts (we=E2=80=99ll have to ch=
eck but
> maybe =E2=80=98file-dynamic-info=E2=80=99 can find everything it needs at=
 the beginning
> of a file, the PT_DYNAMIC segment.)

Another instance of the problem that we just stumbled upon is =E2=80=98guix=
 pack -RR=E2=80=99:
that too tries to load entire ELF files in memory, in
=E2=80=98elf-loader-compile-flags=E2=80=99.

Mmap!

Ludo=E2=80=99.

Message received at control <at> debbugs.gnu.org:

Received: (at control) by debbugs.gnu.org; 12 Apr 2024 09:36:17 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Apr 12 05:36:16 2024
Received: from localhost ([127.0.0.1]:57892 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1rvDKS-00032Y-KU
	for submit <at> debbugs.gnu.org; Fri, 12 Apr 2024 05:36:16 -0400
Received: from mail3-relais-sop.national.inria.fr ([192.134.164.104]:58021)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1rvDKK-0002zU-3E
 for control <at> debbugs.gnu.org; Fri, 12 Apr 2024 05:36:09 -0400
Authentication-Results: mail3-relais-sop.national.inria.fr;
 dkim=none (message not signed) header.i=none;
 spf=SoftFail smtp.mailfrom=ludo@HIDDEN;
 dmarc=fail (p=none dis=none) d=gnu.org
X-IronPort-AV: E=Sophos;i="6.07,195,1708383600"; d="scan'208";a="84585878"
Received: from unknown (HELO ribbon) ([193.50.110.177])
 by mail3-relais-sop.national.inria.fr with
 ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Apr 2024 11:35:53 +0200
Date: Fri, 12 Apr 2024 11:35:53 +0200
Message-Id: <87jzl2gbye.fsf@HIDDEN>
To: control <at> debbugs.gnu.org
From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>
Subject: control message for bug #59365
MIME-version: 1.0
Content-type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
X-Spam-Score: 1.0 (+)
X-Debbugs-Envelope-To: control
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.0 (/)

severity 59365 important
quit

Last modified: Fri, 12 Apr 2024 09:45:01 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.