GNU bug report logs - #59365
make-dynamic-linker-cache OOMs for LLVM 15 on i686-linux

Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.
Package: guix; Severity: important; Reported by: Marius Bakke <marius@HIDDEN>; dated Fri, 18 Nov 2022 18:42:02 UTC; Maintainer for guix is bug-guix@HIDDEN.
Severity set to 'important' from 'normal' Request was from Ludovic Courtès <ludo@HIDDEN> to control <at> debbugs.gnu.org. Full text available.

Message received at 59365 <at> debbugs.gnu.org:


Received: (at 59365) by debbugs.gnu.org; 12 Apr 2024 09:36:11 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Apr 12 05:36:11 2024
Received: from localhost ([127.0.0.1]:57890 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1rvDKM-00031R-1Y
	for submit <at> debbugs.gnu.org; Fri, 12 Apr 2024 05:36:11 -0400
Received: from mail3-relais-sop.national.inria.fr ([192.134.164.104]:58021)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludovic.courtes@HIDDEN>) id 1rvDKF-0002zU-9m
 for 59365 <at> debbugs.gnu.org; Fri, 12 Apr 2024 05:36:07 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=inria.fr; s=dc;
 h=from:to:cc:subject:in-reply-to:references:date:
 message-id:mime-version:content-transfer-encoding;
 bh=mkSSTBDlziE099EF+00sj6ds+WxmMVU8AyZInxGilnc=;
 b=ruKAaKjt7yMNp9GOR30ggdc5zVbFW9drYw7wzDS0R2JpqJZKBm9PqE92
 h/tKEP/bVvI6uangtLLLUghdY9BqMIM1e6eeSKCDHC4pHwhQa1nMW62OX
 dPcX/O1Oby3KjXyoKu02ClI4dV5SbABHmN0bNTfXNYgJwdNt5PJIbfc9w 8=;
Authentication-Results: mail3-relais-sop.national.inria.fr;
 dkim=none (message not signed) header.i=none;
 spf=SoftFail smtp.mailfrom=ludovic.courtes@HIDDEN;
 dmarc=fail (p=none dis=none) d=inria.fr
X-IronPort-AV: E=Sophos;i="6.07,195,1708383600"; d="scan'208";a="84585832"
Received: from unknown (HELO ribbon) ([193.50.110.177])
 by mail3-relais-sop.national.inria.fr with
 ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Apr 2024 11:35:47 +0200
From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludovic.courtes@HIDDEN>
To: 59365 <at> debbugs.gnu.org
Subject: Re: bug#59365: make-dynamic-linker-cache OOMs for LLVM 15 on
 i686-linux
In-Reply-To: <877cznidn8.fsf@HIDDEN> ("Ludovic =?utf-8?Q?Court=C3=A8s=22'?=
 =?utf-8?Q?s?= message of "Tue, 22 Nov 2022 09:42:19 +0100")
References: <87mt8o14xd.fsf@HIDDEN> <877cznidn8.fsf@HIDDEN>
Date: Fri, 12 Apr 2024 11:35:47 +0200
Message-ID: <87le5igbyk.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 59365
Cc: Romain Garbage <romain.garbage@HIDDEN>, Greg Hogan <code@HIDDEN>,
 Maxim Cournoyer <maxim.cournoyer@HIDDEN>, Marius Bakke <marius@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Hello,

Ludovic Court=C3=A8s <ludo@HIDDEN> skribis:

>>   GC Warning: Failed to expand heap by 285216768 bytes
>>   GC Warning: Failed to expand heap by 268439552 bytes
>>   GC Warning: Out of Memory! Heap size: 3620 MiB. Returning NULL!
>>   Warning: Unwind-only out of memory exception; skipping pre-unwind hand=
ler.
>>   Warning: Unwind-only out of memory exception; skipping pre-unwind hand=
ler.
>>   Warning: Unwind-only out of memory exception; skipping pre-unwind hand=
ler.
>>
>> (excerpt from https://ci.guix.gnu.org/build/1702995/log/raw)
>>
>> Not sure why this phase uses so much memory.  Ideas?
>
> Yes: the gremlin.scm code uses =E2=80=98file-dynamic-info=E2=80=99, which=
 loads the
> whole file in memory.  Ridiculous.
>
> We should instead mmap it (but there are no =E2=80=98mmap=E2=80=99 bindin=
gs in Guile,
> yet) or arrange to load just the relevant parts (we=E2=80=99ll have to ch=
eck but
> maybe =E2=80=98file-dynamic-info=E2=80=99 can find everything it needs at=
 the beginning
> of a file, the PT_DYNAMIC segment.)

Another instance of the problem that we just stumbled upon is =E2=80=98guix=
 pack -RR=E2=80=99:
that too tries to load entire ELF files in memory, in
=E2=80=98elf-loader-compile-flags=E2=80=99.

Mmap!

Ludo=E2=80=99.
Information forwarded to bug-guix@HIDDEN:
bug#59365; Package guix. Full text available.

Message received at 59365 <at> debbugs.gnu.org:


Received: (at 59365) by debbugs.gnu.org; 20 Nov 2023 19:33:58 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Nov 20 14:33:58 2023
Received: from localhost ([127.0.0.1]:54639 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1r5A1t-0004al-Se
	for submit <at> debbugs.gnu.org; Mon, 20 Nov 2023 14:33:58 -0500
Received: from mail-ot1-x32f.google.com ([2607:f8b0:4864:20::32f]:54402)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <maxim.cournoyer@HIDDEN>) id 1r5A1r-0004aR-8K
 for 59365 <at> debbugs.gnu.org; Mon, 20 Nov 2023 14:33:57 -0500
Received: by mail-ot1-x32f.google.com with SMTP id
 46e09a7af769-6cd0963c61cso2534806a34.0
 for <59365 <at> debbugs.gnu.org>; Mon, 20 Nov 2023 11:33:52 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=gmail.com; s=20230601; t=1700508827; x=1701113627; darn=debbugs.gnu.org;
 h=mime-version:user-agent:message-id:date:references:in-reply-to
 :subject:cc:to:from:from:to:cc:subject:date:message-id:reply-to;
 bh=SkEiuFoOLf5/xhpAe0vnn9QdkTTJTJtQzEeJ2ksdA9M=;
 b=g/bQ+kxSz7wRGq710Nk4H8ae0eT3sM3fVz8jHFDQNwb381OfP4ybt5XcIBu2nRNUt0
 53uBXeaOrv578a2UP84SHgWZ9EtyYZugtWiJ23WarxPx6MiUsO87QGO7Y3l2v8jwE+jc
 7WaS2IJIwN4rCRWjkgeEfIzDZyYovoPWENSm2u37kIDQAAH6cqFIfHoUNOOYh/zHwQJu
 RNmQIXU3fQ40988nT3mkKtb8uiyT/pke/HWPBU9rg/2eze7k6//chtGubSEoMBbO4qtn
 PGLxDHtq9VcnDapZ6CGKX001sh1Tz84345lcmsazvqBPQq+mRJxAxc8s9veXl468WoGl
 5ovg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20230601; t=1700508827; x=1701113627;
 h=mime-version:user-agent:message-id:date:references:in-reply-to
 :subject:cc:to:from:x-gm-message-state:from:to:cc:subject:date
 :message-id:reply-to;
 bh=SkEiuFoOLf5/xhpAe0vnn9QdkTTJTJtQzEeJ2ksdA9M=;
 b=EKzqh8FE0fJcWTaE4wJw7Lvfs0kbLWviCO/dlB56RUGcYe0P/x1wplcEIuyPGPnbjl
 jhT8fFzR2qvZZYvvVFyJ510vktzgyDqSi0wmKHWr8s7+BodIZTMUeUVfrWzNPvmZpNTI
 tGhjqGvquEEY4gA3/a/4GEp4toEdgXYgMg+eP5OaWDbyi9+CMOmSuiPS/Y7iG5QtKBpA
 mfyXwmGRPG7yEDkg+OkSoHHUowxulW0zKMDV0/Txwaj9JLAjzp0ykQFxbuBNxveo/sc1
 jFwE6Hw6cALm8HxWGw84/6yMVsO7SSoDBbpe1QiUHcWu1d7z9Uq6j7R/ViCqXLIEfQRi
 KWWw==
X-Gm-Message-State: AOJu0YzngwtaIDQgrtrrXFeEEqT8g0jXTqzPRBZjGjgmdGfrdao0f+d4
 vnis7mh/Pkq316H4i7GJ92I=
X-Google-Smtp-Source: AGHT+IEHclmGSoYgxxzlHjV4AoA1o8UWAJhLV0GKKhuTaFeghiTLibujE3smbUepBSDHsNwQkqelvQ==
X-Received: by 2002:a05:6830:1495:b0:6bd:a47:7bb6 with SMTP id
 s21-20020a056830149500b006bd0a477bb6mr6976658otq.14.1700508827239; 
 Mon, 20 Nov 2023 11:33:47 -0800 (PST)
Received: from hurd (dsl-154-9.b2b2c.ca. [66.158.154.9])
 by smtp.gmail.com with ESMTPSA id
 pr15-20020a056214140f00b0065823d20381sm3155602qvb.8.2023.11.20.11.33.46
 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
 Mon, 20 Nov 2023 11:33:46 -0800 (PST)
From: Maxim Cournoyer <maxim.cournoyer@HIDDEN>
To: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Subject: Re: bug#59365: make-dynamic-linker-cache OOMs for LLVM 15 on
 i686-linux
In-Reply-To: <874jurp9tz.fsf@HIDDEN> (Maxim Cournoyer's message of "Tue, 22
 Nov 2022 05:22:48 -0500")
References: <87mt8o14xd.fsf@HIDDEN> <877cznidn8.fsf@HIDDEN>
 <874jurp9tz.fsf@HIDDEN>
Date: Mon, 20 Nov 2023 14:33:45 -0500
Message-ID: <87bkbo9q3q.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 59365
Cc: 59365 <at> debbugs.gnu.org, Greg Hogan <code@HIDDEN>,
 Marius Bakke <marius@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Hi,

This still happens:

--8<---------------cut here---------------start------------->8---
starting phase `make-dynamic-linker-cache'
GC Warning: Repeated allocation of very large block (appr. size 16781312):
	May lead to memory leak and poor performance
GC Warning: Repeated allocation of very large block (appr. size 67112960):
	May lead to memory leak and poor performance
GC Warning: Repeated allocation of very large block (appr. size 134221824):
	May lead to memory leak and poor performance
GC Warning: Repeated allocation of very large block (appr. size 67112960):
	May lead to memory leak and poor performance
GC Warning: Repeated allocation of very large block (appr. size 67112960):
	May lead to memory leak and poor performance
GC Warning: Repeated allocation of very large block (appr. size 134221824):
	May lead to memory leak and poor performance
GC Warning: Repeated allocation of very large block (appr. size 134221824):
	May lead to memory leak and poor performance
GC Warning: Repeated allocation of very large block (appr. size 134221824):
	May lead to memory leak and poor performance
GC Warning: Repeated allocation of very large block (appr. size 268439552):
	May lead to memory leak and poor performance
GC Warning: Repeated allocation of very large block (appr. size 33558528):
	May lead to memory leak and poor performance
GC Warning: Repeated allocation of very large block (appr. size 67112960):
	May lead to memory leak and poor performance
GC Warning: Repeated allocation of very large block (appr. size 67112960):
	May lead to memory leak and poor performance
GC Warning: Repeated allocation of very large block (appr. size 268439552):
	May lead to memory leak and poor performance
GC Warning: Repeated allocation of very large block (appr. size 268439552):
	May lead to memory leak and poor performance
GC Warning: Repeated allocation of very large block (appr. size 268439552):
	May lead to memory leak and poor performance
GC Warning: Failed to expand heap by 285216768 bytes
GC Warning: Failed to expand heap by 268439552 bytes
GC Warning: Out of Memory! Heap size: 3362 MiB. Returning NULL!
Warning: Unwind-only out of memory exception; skipping pre-unwind handler.
Warning: Unwind-only out of memory exception; skipping pre-unwind handler.
Warning: Unwind-only out of memory exception; skipping pre-unwind handler.
builder for
`/gnu/store/j4w1wrhgpjjcfqf2jskklr95r6hpy51i-llvm-15.0.7.drv' failed
with exit code 1
--8<---------------cut here---------------end--------------->8---

See this recent build: https://ci.guix.gnu.org/build/2602334/details

-- 
Thanks,
Maxim
Information forwarded to bug-guix@HIDDEN:
bug#59365; Package guix. Full text available.

Message received at 59365 <at> debbugs.gnu.org:


Received: (at 59365) by debbugs.gnu.org; 22 Nov 2022 10:22:58 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Nov 22 05:22:58 2022
Received: from localhost ([127.0.0.1]:49921 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1oxQQc-0008Ga-6e
	for submit <at> debbugs.gnu.org; Tue, 22 Nov 2022 05:22:58 -0500
Received: from mail-qk1-f170.google.com ([209.85.222.170]:36670)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <maxim.cournoyer@HIDDEN>) id 1oxQQZ-0008GL-Gc
 for 59365 <at> debbugs.gnu.org; Tue, 22 Nov 2022 05:22:56 -0500
Received: by mail-qk1-f170.google.com with SMTP id d7so9891074qkk.3
 for <59365 <at> debbugs.gnu.org>; Tue, 22 Nov 2022 02:22:55 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112;
 h=content-transfer-encoding:mime-version:user-agent:message-id
 :in-reply-to:date:references:subject:cc:to:from:from:to:cc:subject
 :date:message-id:reply-to;
 bh=uF3Z4Qoxr0txlqMqI3SqVBuR80tu5lxgTIvHDXTSHIk=;
 b=kkUBO6ccGZb+LeQlfS1s7t2CFg1rXZyyRbuL6IAlZ2qd8bBq7LPWjE+Tkg64Qj5REs
 AOHYzwezpd3xn5BKESFsJwKHhMZ6GebfkSoak8ovTLvuD2rcNyGhM/m+6cuUXguPPXsf
 ALpUNJNWeQEgvOD9nzvEJ6fgRuiEB7diQY9VfYdO07S0+h8fKK5VcapUN/EmxFx4TbLP
 JRfMZEvtNiEu+CGRrH08iIV3wuljHYUSwfZUkF2aA0FqnPlRRjYfpWu/kXz3QBlLDdnD
 pmH5tcnlER8/jAPO4mReIm/2kNmDNz3KJ9Yu2Lf9+uTXGmbZC5u2m0Y9xXHSBZSTULXz
 W13w==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20210112;
 h=content-transfer-encoding:mime-version:user-agent:message-id
 :in-reply-to:date:references:subject:cc:to:from:x-gm-message-state
 :from:to:cc:subject:date:message-id:reply-to;
 bh=uF3Z4Qoxr0txlqMqI3SqVBuR80tu5lxgTIvHDXTSHIk=;
 b=aX3RBT9AVUvgTefeFjfT1DBXnhVce2cgbl/AW7ptgbS3gKDl34ByjgPMxAp00t06b4
 asEDy8o3QPO+W+nqH7heHHX7cuQNKvik+kORqOg+fNfRBw1MTvQkDf29q02Grn9S9moC
 56hSIa79C14KRTIpAEgQen1o7/LN5Kj7JJ6tOQR9U0XO/WyNlvu2hkUOB8S133lWf1Fw
 /DcxKaqGBXdTMPLkzyxncbiPGBrVtAPp/iqTl0oM8AqSa4ro0BXV8XTaOyv63pcgbi4U
 ZTyzoW1uEgMlMBZyp8bdmuZm5r234gyKXn6uTnjXAC+zit1oxkmFXoRiY//Rp0en6W1y
 C8NA==
X-Gm-Message-State: ANoB5pkDgiLaMR6WsBXIoIYT63jBZJh4GE9SX0iBSoHdStDSm5XJ2DcB
 2pD9dqxQNEOr6Qbf+f7lszH2xmIhBJ9CjQ==
X-Google-Smtp-Source: AA0mqf7FFGiknGYnM4n2Ki2O8V/weHyCTzDRMgu5WThSrlTFtgM5zEpejK3kBh7ErwgKMWFhytfcIQ==
X-Received: by 2002:a37:bc9:0:b0:6fa:163f:dad8 with SMTP id
 192-20020a370bc9000000b006fa163fdad8mr20106095qkl.575.1669112569791; 
 Tue, 22 Nov 2022 02:22:49 -0800 (PST)
Received: from hurd ([2607:fad8:4:3::1003]) by smtp.gmail.com with ESMTPSA id
 s3-20020a05620a0bc300b006fa84082b6dsm10071454qki.128.2022.11.22.02.22.49
 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
 Tue, 22 Nov 2022 02:22:49 -0800 (PST)
From: Maxim Cournoyer <maxim.cournoyer@HIDDEN>
To: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Subject: Re: bug#59365: make-dynamic-linker-cache OOMs for LLVM 15 on
 i686-linux
References: <87mt8o14xd.fsf@HIDDEN> <877cznidn8.fsf@HIDDEN>
Date: Tue, 22 Nov 2022 05:22:48 -0500
In-Reply-To: <877cznidn8.fsf@HIDDEN> ("Ludovic =?utf-8?Q?Court=C3=A8s=22'?=
 =?utf-8?Q?s?= message of "Tue, 22 Nov 2022 09:42:19 +0100")
Message-ID: <874jurp9tz.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 59365
Cc: 59365 <at> debbugs.gnu.org, Greg Hogan <code@HIDDEN>,
 Marius Bakke <marius@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Hi Ludovic,

Ludovic Court=C3=A8s <ludo@HIDDEN> writes:

> Hi,
>
> (Cc: Maxim and Greg for LLVM packaging questions below.)
>
> Marius Bakke <marius@HIDDEN> skribis:
>
>> LLVM 15.0.4 fails on i686-linux:
>>
>>   https://ci.guix.gnu.org/build/1702995/details
>>
>> Because the 'make-dynamic-linker-cache' phase runs out of memory:
>>
>>   starting phase `make-dynamic-linker-cache'
>>   GC Warning: Repeated allocation of very large block (appr. size 268439=
552):
>> 	May lead to memory leak and poor performance
>>   GC Warning: Repeated allocation of very large block (appr. size 134221=
824):
>> 	May lead to memory leak and poor performance
>>   GC Warning: Repeated allocation of very large block (appr. size 268439=
552):
>>   	May lead to memory leak and poor performance
>>   GC Warning: Failed to expand heap by 285216768 bytes
>>   GC Warning: Failed to expand heap by 268439552 bytes
>>   GC Warning: Out of Memory! Heap size: 3620 MiB. Returning NULL!
>>   Warning: Unwind-only out of memory exception; skipping pre-unwind hand=
ler.
>>   Warning: Unwind-only out of memory exception; skipping pre-unwind hand=
ler.
>>   Warning: Unwind-only out of memory exception; skipping pre-unwind hand=
ler.
>>
>> (excerpt from https://ci.guix.gnu.org/build/1702995/log/raw)
>>
>> Not sure why this phase uses so much memory.  Ideas?
>
> Yes: the gremlin.scm code uses =E2=80=98file-dynamic-info=E2=80=99, which=
 loads the
> whole file in memory.  Ridiculous.

If it loaded just that file, it should be fine, no?  It weighs 133 MiB,
as you've shown below:

> But the crux of the problem is that llvm@15 has a single huge shared
> library, unlike previous versions:
>
> $ du -hL  /gnu/store/bgqdvvi7k6l255332rfawgjmn2hpn13r-llvm-15.0.4/lib/*.so
> 133M    /gnu/store/bgqdvvi7k6l255332rfawgjmn2hpn13r-llvm-15.0.4/lib/libLL=
VM-15.0.4.so
> 96K     /gnu/store/bgqdvvi7k6l255332rfawgjmn2hpn13r-llvm-15.0.4/lib/libLT=
O.so
> 16K     /gnu/store/bgqdvvi7k6l255332rfawgjmn2hpn13r-llvm-15.0.4/lib/libRe=
marks.so
> (It also has tons of .a files, which shouldn=E2=80=99t be there.)

The static files are needed at least to build the clang runtime.  I had
tried to get rid of them without success.  Perhaps they could be moved
to a "static" output if they're needed only at that time.

> Is that big LLVM.so due to different build options on our side?  Or is
> it a radical upstream change (sounds unlikely, but who knows)?

It's caused by -DLLVM_LINK_LLVM_DYLIB=3DON and -DLLVM_BUILD_LLVM_DYLIB=3DON,
which is the supported configuration to build a shared library of LLVM
(-DBUILD_SHARED_LIBS=3DON is obsolete/deprecated) [0].

It also makes things conveniently easy to link to LLVM; you just need to
link to '-lLLVM', and everything it needs is available.

[0]  https://llvm.org/docs/BuildingADistribution.html#special-notes-for-lib=
rary-only-distributions

--=20
Thanks,
Maxim
Information forwarded to bug-guix@HIDDEN:
bug#59365; Package guix. Full text available.

Message received at 59365 <at> debbugs.gnu.org:


Received: (at 59365) by debbugs.gnu.org; 22 Nov 2022 08:42:34 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Nov 22 03:42:34 2022
Received: from localhost ([127.0.0.1]:49691 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1oxOrR-0005cd-MS
	for submit <at> debbugs.gnu.org; Tue, 22 Nov 2022 03:42:34 -0500
Received: from eggs.gnu.org ([209.51.188.92]:60202)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1oxOrP-0005cP-F5
 for 59365 <at> debbugs.gnu.org; Tue, 22 Nov 2022 03:42:32 -0500
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ludo@HIDDEN>)
 id 1oxOrJ-0006Kg-Rg; Tue, 22 Nov 2022 03:42:25 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To:
 From; bh=GRWNBFaH+l9SToWarPpluIlsJ8gyxSbMonB3FaZY5h8=; b=hjqCsnLDZgR3s1+qrH7t
 kyz6sAzIRev0KHqUPbzwrTf98SyCCp2rnc2s79wpUPmpee8YIA+SaaAMMrVSOaeSa5gBIW8MWCC82
 wovN2pqKjrMVjtsc0EeJHEFnTCADXGUMRU3MqHJWHvKLHs+f3Nk9W4nmoXpcN2PAJ3ueRGABcDSyb
 /T38rKPP8Lz+MUPZpyitUiinQqi5EFVzqEwPHrDFB9GR+mu2F6LETjcx/zXRvEbf30d9U/IZ1wvWL
 fdo/GmaT0pEzm4SBf7ZMnOchpH8Y+mI6hQkmOhHjpmIoZF8P/DRVPiMjZSZK+SxibexEcI8pclhp2
 QmAiev2QPi+KEQ==;
Received: from [89.207.171.76] (helo=ribbon)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ludo@HIDDEN>)
 id 1oxOrG-0003PM-Q5; Tue, 22 Nov 2022 03:42:25 -0500
From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>
To: Marius Bakke <marius@HIDDEN>
Subject: Re: bug#59365: make-dynamic-linker-cache OOMs for LLVM 15 on
 i686-linux
References: <87mt8o14xd.fsf@HIDDEN>
Date: Tue, 22 Nov 2022 09:42:19 +0100
In-Reply-To: <87mt8o14xd.fsf@HIDDEN> (Marius Bakke's message of "Fri, 18 Nov
 2022 19:41:18 +0100")
Message-ID: <877cznidn8.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="=-=-="
X-Spam-Score: -1.8 (-)
X-Debbugs-Envelope-To: 59365
Cc: 59365 <at> debbugs.gnu.org, Maxim Cournoyer <maxim.cournoyer@HIDDEN>,
 Greg Hogan <code@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -2.8 (--)

--=-=-=
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

Hi,

(Cc: Maxim and Greg for LLVM packaging questions below.)

Marius Bakke <marius@HIDDEN> skribis:

> LLVM 15.0.4 fails on i686-linux:
>
>   https://ci.guix.gnu.org/build/1702995/details
>
> Because the 'make-dynamic-linker-cache' phase runs out of memory:
>
>   starting phase `make-dynamic-linker-cache'
>   GC Warning: Repeated allocation of very large block (appr. size 2684395=
52):
> 	May lead to memory leak and poor performance
>   GC Warning: Repeated allocation of very large block (appr. size 1342218=
24):
> 	May lead to memory leak and poor performance
>   GC Warning: Repeated allocation of very large block (appr. size 2684395=
52):
>   	May lead to memory leak and poor performance
>   GC Warning: Failed to expand heap by 285216768 bytes
>   GC Warning: Failed to expand heap by 268439552 bytes
>   GC Warning: Out of Memory! Heap size: 3620 MiB. Returning NULL!
>   Warning: Unwind-only out of memory exception; skipping pre-unwind handl=
er.
>   Warning: Unwind-only out of memory exception; skipping pre-unwind handl=
er.
>   Warning: Unwind-only out of memory exception; skipping pre-unwind handl=
er.
>
> (excerpt from https://ci.guix.gnu.org/build/1702995/log/raw)
>
> Not sure why this phase uses so much memory.  Ideas?

Yes: the gremlin.scm code uses =E2=80=98file-dynamic-info=E2=80=99, which l=
oads the
whole file in memory.  Ridiculous.

We should instead mmap it (but there are no =E2=80=98mmap=E2=80=99 bindings=
 in Guile,
yet) or arrange to load just the relevant parts (we=E2=80=99ll have to chec=
k but
maybe =E2=80=98file-dynamic-info=E2=80=99 can find everything it needs at t=
he beginning
of a file, the PT_DYNAMIC segment.)

For example, with the patch below, things still appear to be fine with
LLVM:

--8<---------------cut here---------------start------------->8---
scheme@(guix build gremlin)> (file-dynamic-info "/gnu/store/mj14k58lfc88jhc=
n6va0s2fpwkv3s35c-llvm-13.0.1/lib/libLLVMScalarOpts.so")
$11 =3D #<<elf-dynamic-info> soname: "libLLVMScalarOpts.so.13" needed: ("li=
bLLVMAggressiveInstCombine.so.13" "libLLVMInstCombine.so.13" "libLLVMTransf=
ormUtils.so.13" "libLLVMAnalysis.so.13" "libLLVMCore.so.13" "libLLVMSupport=
.so.13" "libstdc++.so.6" "libm.so.6" "libgcc_s.so.1" "libc.so.6" "ld-linux-=
x86-64.so.2") rpath: () runpath: ("/gnu/store/mj14k58lfc88jhcn6va0s2fpwkv3s=
35c-llvm-13.0.1/lib" "/gnu/store/5h2w4qi9hk1qzzgi1w83220ydslinr4s-glibc-2.3=
3/lib" "/gnu/store/094bbaq6glba86h1d4cj16xhdi6fk2jl-gcc-10.3.0-lib/lib" "/g=
nu/store/094bbaq6glba86h1d4cj16xhdi6fk2jl-gcc-10.3.0-lib/lib/gcc/x86_64-unk=
nown-linux-gnu/10.3.0/../../..")>
scheme@(guix build gremlin)> (file-dynamic-info "/gnu/store/mj14k58lfc88jhc=
n6va0s2fpwkv3s35c-llvm-13.0.1/lib/libLLVMX86CodeGen.so.13")
$12 =3D #<<elf-dynamic-info> soname: "libLLVMX86CodeGen.so.13" needed: ("li=
bLLVMAsmPrinter.so.13" "libLLVMX86Desc.so.13" "libLLVMX86Info.so.13" "libLL=
VMGlobalISel.so.13" "libLLVMCFGuard.so.13" "libLLVMSelectionDAG.so.13" "lib=
LLVMCodeGen.so.13" "libLLVMTarget.so.13" "libLLVMTransformUtils.so.13" "lib=
LLVMAnalysis.so.13" "libLLVMProfileData.so.13" "libLLVMMC.so.13" "libLLVMCo=
re.so.13" "libLLVMSupport.so.13" "libstdc++.so.6" "libm.so.6" "libgcc_s.so.=
1" "libc.so.6" "ld-linux-x86-64.so.2") rpath: () runpath: ("/gnu/store/mj14=
k58lfc88jhcn6va0s2fpwkv3s35c-llvm-13.0.1/lib" "/gnu/store/5h2w4qi9hk1qzzgi1=
w83220ydslinr4s-glibc-2.33/lib" "/gnu/store/094bbaq6glba86h1d4cj16xhdi6fk2j=
l-gcc-10.3.0-lib/lib" "/gnu/store/094bbaq6glba86h1d4cj16xhdi6fk2jl-gcc-10.3=
.0-lib/lib/gcc/x86_64-unknown-linux-gnu/10.3.0/../../..")>
--8<---------------cut here---------------end--------------->8---

We could temporarily delete this phase for all 32-bit builds of LLVM.

But the crux of the problem is that llvm@15 has a single huge shared
library, unlike previous versions:

--8<---------------cut here---------------start------------->8---
$ du -hL  /gnu/store/bgqdvvi7k6l255332rfawgjmn2hpn13r-llvm-15.0.4/lib/*.so
133M    /gnu/store/bgqdvvi7k6l255332rfawgjmn2hpn13r-llvm-15.0.4/lib/libLLVM=
-15.0.4.so
96K     /gnu/store/bgqdvvi7k6l255332rfawgjmn2hpn13r-llvm-15.0.4/lib/libLTO.=
so
16K     /gnu/store/bgqdvvi7k6l255332rfawgjmn2hpn13r-llvm-15.0.4/lib/libRema=
rks.so
--8<---------------cut here---------------end--------------->8---

(It also has tons of .a files, which shouldn=E2=80=99t be there.)

Is that big LLVM.so due to different build options on our side?  Or is
it a radical upstream change (sounds unlikely, but who knows)?

Thanks,
Ludo=E2=80=99.


--=-=-=
Content-Type: text/x-patch
Content-Disposition: inline

diff --git a/guix/build/gremlin.scm b/guix/build/gremlin.scm
index 2a74d51dd9..8a38dde1eb 100644
--- a/guix/build/gremlin.scm
+++ b/guix/build/gremlin.scm
@@ -250,7 +250,11 @@ (define (file-dynamic-info file)
 info."
   (call-with-input-file file
     (lambda (port)
-      (elf-dynamic-info (parse-elf (get-bytevector-all port))))))
+      (elf-dynamic-info (parse-elf
+                         ;; Read at most 10 MiB in memory, which should be
+                         ;; enough to get the PT_DYNAMIC segment.
+                         ;; TODO: mmap the whole file instead.
+                         (get-bytevector-n port (* 10 (expt 2 20))))))))
 
 (define (file-runpath file)
   "Return the DT_RUNPATH dynamic entry of FILE as a list of strings, or #f if

--=-=-=--
Information forwarded to bug-guix@HIDDEN:
bug#59365; Package guix. Full text available.

Message received at submit <at> debbugs.gnu.org:


Received: (at submit) by debbugs.gnu.org; 18 Nov 2022 18:41:26 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Nov 18 13:41:26 2022
Received: from localhost ([127.0.0.1]:38236 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1ow6In-0003v7-WD
	for submit <at> debbugs.gnu.org; Fri, 18 Nov 2022 13:41:26 -0500
Received: from lists.gnu.org ([209.51.188.17]:55444)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <marius@HIDDEN>) id 1ow6Im-0003uz-7l
 for submit <at> debbugs.gnu.org; Fri, 18 Nov 2022 13:41:25 -0500
Received: from eggs.gnu.org ([2001:470:142:3::10])
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <marius@HIDDEN>) id 1ow6Im-0005sf-2d
 for bug-guix@HIDDEN; Fri, 18 Nov 2022 13:41:24 -0500
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <marius@HIDDEN>) id 1ow6Il-0002iM-Ql
 for bug-guix@HIDDEN; Fri, 18 Nov 2022 13:41:23 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=MIME-Version:Date:Subject:To:From:in-reply-to:
 references; bh=6IUdVkfJ5rNlJhbwYxhmky1eFUautfiWfHpP1NMLwLg=; b=C751cz33nKVStT
 vYphho4te5+MiZK4IRHy0wdln/bVGRkJvjpiBLxbBFo5aMng0/rt5vemd1E9ePlyk968mtFOn6EXZ
 hjHvgL5Et3N0/f07Fr5K5qrjXmxtaHtm0httlYjr2lxzN1O6qVveVnz9KvnVmCxI/0mwljYeZyolp
 oTB1ShsfJi1FNh4GQ9jua8FBaNpDaKuwBcplV9++DejcbwWEOkepxiQiHUEaQJU9OWC0or0OcfDMH
 Be9AlQwg+CYROw2TPVYYg5eMS5HLqANtH3VK9uzfufhDnSuKt+YaXhHmEC9nXenasyYUj3JkQyCQI
 IgHvlxaDjE12C2Mtu4Dg==;
Received: from [84.214.173.6] (helo=localhost)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <marius@HIDDEN>) id 1ow6Il-0001lF-2S
 for bug-guix@HIDDEN; Fri, 18 Nov 2022 13:41:23 -0500
From: Marius Bakke <marius@HIDDEN>
To: bug-guix@HIDDEN
Subject: make-dynamic-linker-cache OOMs for LLVM 15 on i686-linux
Date: Fri, 18 Nov 2022 19:41:18 +0100
Message-ID: <87mt8o14xd.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="=-=-=";
 micalg=pgp-sha512; protocol="application/pgp-signature"
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: submit
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

--=-=-=
Content-Type: text/plain

Hello,

LLVM 15.0.4 fails on i686-linux:

  https://ci.guix.gnu.org/build/1702995/details

Because the 'make-dynamic-linker-cache' phase runs out of memory:

  starting phase `make-dynamic-linker-cache'
  GC Warning: Repeated allocation of very large block (appr. size 268439552):
	May lead to memory leak and poor performance
  GC Warning: Repeated allocation of very large block (appr. size 134221824):
	May lead to memory leak and poor performance
  GC Warning: Repeated allocation of very large block (appr. size 268439552):
  	May lead to memory leak and poor performance
  GC Warning: Failed to expand heap by 285216768 bytes
  GC Warning: Failed to expand heap by 268439552 bytes
  GC Warning: Out of Memory! Heap size: 3620 MiB. Returning NULL!
  Warning: Unwind-only out of memory exception; skipping pre-unwind handler.
  Warning: Unwind-only out of memory exception; skipping pre-unwind handler.
  Warning: Unwind-only out of memory exception; skipping pre-unwind handler.

(excerpt from https://ci.guix.gnu.org/build/1702995/log/raw)

Not sure why this phase uses so much memory.  Ideas?

--=-=-=
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iIUEARYKAC0WIQRNTknu3zbaMQ2ddzTocYulkRQQdwUCY3fRzg8cbWFyaXVzQGdu
dS5vcmcACgkQ6HGLpZEUEHdg4AEAlgoHstuJbIvzu6VNET3NpLOzagBsqFCMMulA
3qOfm/cBAKpm1PeRtKIKv40YW/HnXhVkUNKzNVizYDfDt8pFi0kA
=OSPc
-----END PGP SIGNATURE-----
--=-=-=--
Acknowledgement sent to Marius Bakke <marius@HIDDEN>:
New bug report received and forwarded. Copy sent to bug-guix@HIDDEN. Full text available.
Report forwarded to bug-guix@HIDDEN:
bug#59365; Package guix. Full text available.
Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.
Last modified: Fri, 12 Apr 2024 09:45:01 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.