GNU bug report logs -
#63852
‘guix-package-cache.drv’ eats all your memory with ‘guix-cran’
Previous Next
To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 63852 in the body.
You can then email your comments to 63852 AT debbugs.gnu.org in the normal way.
Toggle the display of automated, internal messages from the tracker.
Report forwarded
to
lars <at> 6xq.net, bug-guix <at> gnu.org:
bug#63852; Package
guix.
(Fri, 02 Jun 2023 16:23:02 GMT)
Full text and
rfc822 format available.
Acknowledgement sent
to
Ludovic Courtès <ludovic.courtes <at> inria.fr>:
New bug report received and forwarded. Copy sent to
lars <at> 6xq.net, bug-guix <at> gnu.org.
(Fri, 02 Jun 2023 16:23:02 GMT)
Full text and
rfc822 format available.
Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):
Consider this:
--8<---------------cut here---------------start------------->8---
$ cat /tmp/cran.scm
(cons (channel
(name 'guix-cran)
(url "https://github.com/guix-science/guix-cran.git"))
%default-channels)
$ guix time-machine -C /tmp/cran.scm -- describe
Updating channel 'guix-cran' from Git repository at 'https://github.com/guix-science/guix-cran.git'...
Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'...
[...]
The following derivations will be built:
/gnu/store/90qiphp9048zgvi96lj8zqg8zhwi4js0-profile.drv
/gnu/store/2lv56bl8ghyf2kvfkwj5iy3mrlfik4la-inferior-script.scm.drv
/gnu/store/vnjf8gf6yf5gjm1syvz7yyfn71mmvy27-profile.drv
/gnu/store/8x49757i1s0r6igrah8ramn9zc1v577b-guix-69dfdb7bd.drv
/gnu/store/ll5x27npc3znyqv0rkjh945x9cr41k9q-guix-command.drv
/gnu/store/m03hp1n62bv8gkid53g8igizbw012gai-guix-module-union.drv
/gnu/store/pw8k2jwwyj2nwvqjcd1w3xjvz2n5zpbg-guix-69dfdb7bd-modules.drv
/gnu/store/a6si5zgzsxyha2466v4q45hqimg31mjg-guix-config-modules.drv
/gnu/store/3pm8aivzg3hfv95zlffxjrxvf8hfbg5n-guix-config-source.drv
/gnu/store/rffsvls17s21v2h6gw7agah5vczwi3rm-config.scm.drv
/gnu/store/v61jlksbgfslh27y8dbaj1jcajxz5c8z-guix-config.drv
/gnu/store/z6r6xdcq9dyxlk8ln8cq1ka2gspxp83i-guix-daemon.drv
/gnu/store/fshds3i961v1p3x8f7awh55lix0f7vkl-guix-cran.drv
/gnu/store/xsnp3znb0h630kipjxd198qfd3jr2ww5-inferior-script.scm.drv
building /gnu/store/xsnp3znb0h630kipjxd198qfd3jr2ww5-inferior-script.scm.drv...
building /gnu/store/rffsvls17s21v2h6gw7agah5vczwi3rm-config.scm.drv...
building /gnu/store/3pm8aivzg3hfv95zlffxjrxvf8hfbg5n-guix-config-source.drv...
building /gnu/store/v61jlksbgfslh27y8dbaj1jcajxz5c8z-guix-config.drv...
building /gnu/store/a6si5zgzsxyha2466v4q45hqimg31mjg-guix-config-modules.drv...
building /gnu/store/pw8k2jwwyj2nwvqjcd1w3xjvz2n5zpbg-guix-69dfdb7bd-modules.drv...
building /gnu/store/m03hp1n62bv8gkid53g8igizbw012gai-guix-module-union.drv...
building /gnu/store/ll5x27npc3znyqv0rkjh945x9cr41k9q-guix-command.drv...
building /gnu/store/z6r6xdcq9dyxlk8ln8cq1ka2gspxp83i-guix-daemon.drv...
building /gnu/store/8x49757i1s0r6igrah8ramn9zc1v577b-guix-69dfdb7bd.drv...
building /gnu/store/fshds3i961v1p3x8f7awh55lix0f7vkl-guix-cran.drv...
building CA certificate bundle...
listing Emacs sub-directories...
building fonts directory...
building directory of Info manuals...
building profile with 2 packages...
building /gnu/store/2lv56bl8ghyf2kvfkwj5iy3mrlfik4la-inferior-script.scm.drv...
building package cache...
--8<---------------cut here---------------end--------------->8---
It goes OK until building /gnu/store/6rfaqfq693vda59a55asc4wjjg52ilns-guix-package-cache.drv, which
consumes memory until it gets OOM-killed.
It used to work fine a few months ago.
Ludo’.
Severity set to 'important' from 'normal'
Request was from
Ludovic Courtès <ludo <at> gnu.org>
to
control <at> debbugs.gnu.org.
(Fri, 02 Jun 2023 20:56:01 GMT)
Full text and
rfc822 format available.
Information forwarded
to
bug-guix <at> gnu.org:
bug#63852; Package
guix.
(Sat, 03 Jun 2023 14:06:02 GMT)
Full text and
rfc822 format available.
Message #10 received at 63852 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
Ludovic Courtès <ludovic.courtes <at> inria.fr> skribis:
> It goes OK until building /gnu/store/6rfaqfq693vda59a55asc4wjjg52ilns-guix-package-cache.drv, which
> consumes memory until it gets OOM-killed.
The culprit is easily found (here on just the 22K packages of ‘guix’, so
less than half of what you get with ‘guix-cran’):
--8<---------------cut here---------------start------------->8---
$ guix repl
GNU Guile 3.0.9
Copyright (C) 1995-2023 Free Software Foundation, Inc.
Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
This program is free software, and you are welcome to redistribute it
under certain conditions; type `,show c' for details.
Enter `,help' for help.
scheme@(guix-user)> ,m(gnu packages)
scheme@(gnu packages)> ,use(statprof)
scheme@(gnu packages)> (gcprof (lambda () (generate-package-cache "/tmp/t")))
% cumulative self
time seconds seconds procedure
50.00 13.77 13.77 ice-9/boot-9.scm:2498:4
6.94 1.91 1.91 display
5.56 1.53 1.53 object->string
2.78 0.76 0.76 reverse
2.78 0.76 0.76 gensym
2.78 0.76 0.76 ice-9/boot-9.scm:2217:0:%load-announce
1.39 6.88 0.38 gnu/packages.scm:388:4:expand-cache
1.39 0.76 0.38 srfi/srfi-1.scm:1028:0:lset-intersection
1.39 0.38 0.38 number->string
1.39 0.38 0.38 gnu/packages/crates-io.scm:17783:2:arguments
1.39 0.38 0.38 open-output-string
1.39 0.38 0.38 gnu/packages/crates-io.scm:18450:2:arguments
1.39 0.38 0.38 gnu/packages/crates-io.scm:36208:2:arguments
1.39 0.38 0.38 guix/packages.scm:1320:8:mproc
1.39 0.38 0.38 gnu/packages/crates-io.scm:25101:2:arguments
1.39 0.38 0.38 gnu/packages/crates-io.scm:29972:2:arguments
1.39 0.38 0.38 gnu/packages/crates-io.scm:883:2:arguments
1.39 0.38 0.38 ice-9/vlist.scm:449:0:vhash-cons
1.39 0.38 0.38 append
1.39 0.38 0.38 string-append
1.39 0.38 0.38 system/vm/assembler.scm:2175:8
1.39 0.38 0.38 guix/build/syscalls.scm:1168:0:read-dirent-header/linux
1.39 0.38 0.38 srfi/srfi-1.scm:1033:17
1.39 0.38 0.38 gnu/packages/crates-io.scm:9336:2:arguments
1.39 0.38 0.38 gnu/packages/crates-io.scm:64009:2:arguments
1.39 0.38 0.38 guix/build-system/cargo.scm:229:4
1.39 0.38 0.38 ice-9/boot-9.scm:2759:0:module-make-local-var!
0.00 7328.64 0.00 system/vm/assembler.scm:1258:0:intern-constant
0.00 1423.97 0.00 ice-9/boot-9.scm:220:5:map1
0.00 201.90 0.00 ice-9/threads.scm:388:4
[...]
---
Sample count: 72
Total time: 27.531145665 seconds (14.056589389 seconds in GC)
--8<---------------cut here---------------end--------------->8---
The assembler allocates a huge constant table in ‘intern-constant’
(‘generate-package-cache’ produces one literal list containing one small
vector per package, each of which contains strings and other constants).
The attached file emulates what ‘generate-package-cache’ does. Each
vector in the list looks like this:
#(0 "xyzxyz-0" "1.0.0" (gnu packages xyz0) xyzxyz-0 ("out") #t #f "gnu/packages/xyz0" 0 1)
The program terminates with roughly a 250 MiB heap for 22K entries—about
10 MiB per vector.
The heap profile on completion looks like this:
--8<---------------cut here---------------start------------->8---
% type self avg obj size
57.5 bytevector 8,205,632 455868.4
23.9 vector 3,413,857 549.3
7.7 struct 1,093,840 40.8
5.3 pair 753,376 16.0
2.7 stringbuf 385,856 51.1
1.1 symbol 151,424 32.0
0.4 program 59,424 47.3
0.3 string 49,152 32.0
0.2 heap-number 23,520 32.0
0.2 variable 21,696 30.3
0.1 smob 21,056 32.0
0.1 vm-continuation 20,864 32.0
0.1 atomic-box 20,736 32.0
0.1 weak-table 19,584 31.8
0.1 unknown 19,040 32.0
0.0 pointer 2,000 16.0
0.0 hash-table 896 32.0
0.0 primitive 144 16.0
0.0 weak-vector 96 16.0
0.0 frame 96 32.0
0.0 primitive-generic 64 32.0
0.0 keyword 16 16.0
sampled heap: 13.60166 MiB (heap size: 233.16016 MiB)
--8<---------------cut here---------------end--------------->8---
It’s likely that the ‘buf’ field of <asm> is what’s taking space
On that topic, see also:
https://lists.gnu.org/archive/html/guile-devel/2023-01/msg00013.html
Ludo’.
[assembler-memory-consumption.scm (text/plain, inline)]
;; https://issues.guix.gnu.org/63852
(use-modules (system base compile)
(language tree-il)
(srfi srfi-1)
(srfi srfi-26)
(ice-9 match)
(ice-9 time)
(statprof))
(define lst
(unfold (cut > <> 22000)
(lambda (i)
;; Mimic 'generate-package-cache'.
(define name
(string-append "xyzxyz-" (number->string i 8)))
(vector i name
(string-append "1.0."
(number->string (modulo i 300)))
(list 'gnu 'packages
(string->symbol
(string-append "xyz" (number->string i))))
(string->symbol name)
'("out")
#t
#f
(string-append "gnu/packages/xyz"
(number->string (modulo i 300)))
(modulo i 1000)
1))
1+
0))
(pk 'go! (gc-stats))
(let ((exp (match (or (getenv "VARIANT") "0")
("0" `',lst)
("1" `(list ,@lst))
("2" (make-const #f lst)) ;avoid psyntax overhead
("3" (make-primcall #f 'list
(map (lambda (vector)
(make-const #f vector))
lst))))))
(compile exp #:to 'bytecode
#:from (if (pair? exp)
'scheme
'tree-il)
#:optimization-level 0
#:opts '(#:to-file? #t)))
(pk (gc-stats))
Information forwarded
to
bug-guix <at> gnu.org:
bug#63852; Package
guix.
(Mon, 05 Jun 2023 16:36:01 GMT)
Full text and
rfc822 format available.
Message #13 received at 63852 <at> debbugs.gnu.org (full text, mbox):
Ludovic Courtès <ludo <at> gnu.org> skribis:
> Ludovic Courtès <ludovic.courtes <at> inria.fr> skribis:
>
>> It goes OK until building /gnu/store/6rfaqfq693vda59a55asc4wjjg52ilns-guix-package-cache.drv, which
>> consumes memory until it gets OOM-killed.
>
> The culprit is easily found (here on just the 22K packages of ‘guix’, so
> less than half of what you get with ‘guix-cran’):
(Beware of “easily” when the guy doesn’t know what he’s talking about.)
Turns out the problem is more pedestrian: ‘r-future-tests’ depends on
itself.
--8<---------------cut here---------------start------------->8---
$ guix import cran future.tests
Starting download of /tmp/guix-file.TOBjcf
From http://cran.r-project.org/src/contrib/future.tests_0.7.0.tar.gz...
…_0.7.0.tar.gz 186KiB 1.2MiB/s 00:00 ▕██████████████████▏ 100.0%
(package
(name "r-future-tests")
(version "0.7.0")
(source (origin
(method url-fetch)
(uri (cran-uri "future.tests" version))
(sha256
(base32
"10g1w99xqr7l0cn27642aphqcvfidgpas38f84r815yy6k1ryrlx"))))
(properties `((upstream-name . "future.tests")))
(build-system r-build-system)
(propagated-inputs (list r-cli r-crayon r-future r-prettyunits r-sessioninfo))
(native-inputs (list r-future-tests))
(home-page "https://future.tests.futureverse.org")
(synopsis "Test Suite for 'Future API' Backends")
(description
"Backends implementing the Future API, as defined by the future package, should
use the tests provided by this package to validate that they meet the minimal
requirements of the Future API. The tests can be performed easily from within R
or from outside of R from the command line making it straightforward to include
them in package tests and in Continuous Integration (CI) pipelines.")
$ guix describe
Generation 265 Jun 04 2023 23:48:31 (current)
guix eed55a6
repository URL: https://git.savannah.gnu.org/git/guix.git
branch: master
commit: eed55a6544d5bda2245ec853e5fa4b28e1865bea
--8<---------------cut here---------------end--------------->8---
To be fixed in the importer.
Ludo’.
Information forwarded
to
bug-guix <at> gnu.org:
bug#63852; Package
guix.
(Mon, 05 Jun 2023 21:59:02 GMT)
Full text and
rfc822 format available.
Message #16 received at 63852 <at> debbugs.gnu.org (full text, mbox):
Ludovic Courtès <ludo <at> gnu.org> skribis:
> Ludovic Courtès <ludo <at> gnu.org> skribis:
>
>> Ludovic Courtès <ludovic.courtes <at> inria.fr> skribis:
>>
>>> It goes OK until building /gnu/store/6rfaqfq693vda59a55asc4wjjg52ilns-guix-package-cache.drv, which
>>> consumes memory until it gets OOM-killed.
>>
>> The culprit is easily found (here on just the 22K packages of ‘guix’, so
>> less than half of what you get with ‘guix-cran’):
>
> (Beware of “easily” when the guy doesn’t know what he’s talking about.)
>
> Turns out the problem is more pedestrian: ‘r-future-tests’ depends on
> itself.
Here’s a patch to prevent that from happening again:
https://issues.guix.gnu.org/63917
(We also need to fix the importer.)
Ludo’.
Reply sent
to
Ludovic Courtès <ludo <at> gnu.org>:
You have taken responsibility.
(Thu, 15 Jun 2023 13:14:01 GMT)
Full text and
rfc822 format available.
Notification sent
to
Ludovic Courtès <ludovic.courtes <at> inria.fr>:
bug acknowledged by developer.
(Thu, 15 Jun 2023 13:14:01 GMT)
Full text and
rfc822 format available.
Message #21 received at 63852-done <at> debbugs.gnu.org (full text, mbox):
Ludovic Courtès <ludo <at> gnu.org> skribis:
> Ludovic Courtès <ludo <at> gnu.org> skribis:
>
>> Ludovic Courtès <ludovic.courtes <at> inria.fr> skribis:
>>
>>> It goes OK until building /gnu/store/6rfaqfq693vda59a55asc4wjjg52ilns-guix-package-cache.drv, which
>>> consumes memory until it gets OOM-killed.
>>
>> The culprit is easily found (here on just the 22K packages of ‘guix’, so
>> less than half of what you get with ‘guix-cran’):
>
> (Beware of “easily” when the guy doesn’t know what he’s talking about.)
>
> Turns out the problem is more pedestrian: ‘r-future-tests’ depends on
> itself.
With commit e4259d4e9e3251e4c4b45d1cce4008ac32b504c8, such cycles are
detected early on.
Closing this bug.
We should still improve the memory footprint of the assembler though.
Ludo’.
bug archived.
Request was from
Debbugs Internal Request <help-debbugs <at> gnu.org>
to
internal_control <at> debbugs.gnu.org.
(Fri, 14 Jul 2023 11:24:06 GMT)
Full text and
rfc822 format available.
This bug report was last modified 287 days ago.
Previous Next
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997,2003 nCipher Corporation Ltd,
1994-97 Ian Jackson.