GNU bug report logs - #59021
Unbounded heap growth when combining dynamic states & delimited continuation

Previous Next

Package: guile;

Reported by: Ludovic Courtès <ludo <at> gnu.org>

Date: Fri, 4 Nov 2022 18:25:02 UTC

Severity: important

Done: Ludovic Courtès <ludo <at> gnu.org>

Bug is archived. No further changes may be made.

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 59021 in the body.
You can then email your comments to 59021 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to bug-guile <at> gnu.org:
bug#59021; Package guile. (Fri, 04 Nov 2022 18:25:02 GMT) Full text and rfc822 format available.

Acknowledgement sent to Ludovic Courtès <ludo <at> gnu.org>:
New bug report received and forwarded. Copy sent to bug-guile <at> gnu.org. (Fri, 04 Nov 2022 18:25:02 GMT) Full text and rfc822 format available.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: bug-guile <at> gnu.org
Subject: Unbounded heap growth when combining dynamic states & delimited
 continuation
Date: Fri, 04 Nov 2022 19:24:50 +0100
(This is a followup to <https://github.com/wingo/fibers/issues/65>,
itself a followup to <https://issues.guix.gnu.org/58631>.)

Consider this code:

--8<---------------cut here---------------start------------->8---
;; https://issues.guix.gnu.org/58631
;; https://github.com/wingo/fibers/issues/65

(define loss
  (make-vector 1000000))

(let ((tag (make-prompt-tag "my prompt")))
  (define handler
    (lambda (k i)
      (when (zero? (modulo i 2000000))
        (pk 'heap-size (assoc-ref (gc-stats) 'heap-size)))

      (call-with-prompt tag
        (lambda ()
          (k (modulo (+ 1 i) 10000000)))
        handler)))

  (call-with-prompt tag
    (let ((state (current-dynamic-state)))
      (lambda ()
        ;; (define (with-dynamic-state state thunk)
        ;;   (let ((previous #f))
        ;;     (dynamic-wind
        ;;       (lambda () (set! previous (set-current-dynamic-state state)))
        ;;       thunk
        ;;       (lambda () (set-current-dynamic-state previous)))))
        (with-dynamic-state state
                            (lambda ()
                              (let loop ((i 0))
                                (loop (abort-to-prompt tag i)))))))
    handler))
--8<---------------cut here---------------end--------------->8---

On Guile 3.0.8, this program exhibits seemingly unbounded heap growth.
Uncommenting the local ‘with-dynamic-state’ definition fixes the
problem.

Ludo’.




Severity set to 'important' from 'normal' Request was from Ludovic Courtès <ludo <at> gnu.org> to control <at> debbugs.gnu.org. (Fri, 04 Nov 2022 20:26:03 GMT) Full text and rfc822 format available.

Information forwarded to bug-guile <at> gnu.org:
bug#59021; Package guile. (Sat, 05 Nov 2022 22:05:01 GMT) Full text and rfc822 format available.

Message #10 received at 59021 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: 59021 <at> debbugs.gnu.org
Cc: Andy Wingo <wingo <at> pobox.com>
Subject: Re: bug#59021: Unbounded heap growth when combining dynamic states
 & delimited continuation
Date: Sat, 05 Nov 2022 23:04:22 +0100
[Message part 1 (text/plain, inline)]
Ludovic Courtès <ludo <at> gnu.org> skribis:

> Consider this code:
>
> ;; https://issues.guix.gnu.org/58631
> ;; https://github.com/wingo/fibers/issues/65
>
> (define loss
>   (make-vector 1000000))
>
> (let ((tag (make-prompt-tag "my prompt")))
>   (define handler
>     (lambda (k i)
>       (when (zero? (modulo i 2000000))
>         (pk 'heap-size (assoc-ref (gc-stats) 'heap-size)))
>
>       (call-with-prompt tag
>         (lambda ()
>           (k (modulo (+ 1 i) 10000000)))
>         handler)))
>
>   (call-with-prompt tag
>     (let ((state (current-dynamic-state)))
>       (lambda ()
>         ;; (define (with-dynamic-state state thunk)
>         ;;   (let ((previous #f))
>         ;;     (dynamic-wind
>         ;;       (lambda () (set! previous (set-current-dynamic-state state)))
>         ;;       thunk
>         ;;       (lambda () (set-current-dynamic-state previous)))))
>         (with-dynamic-state state
>                             (lambda ()
>                               (let loop ((i 0))
>                                 (loop (abort-to-prompt tag i)))))))
>     handler))
>
> On Guile 3.0.8, this program exhibits seemingly unbounded heap growth.

This is fixed by the patch below (tested against the test case above and
the Fibers and Shepherd test cases mentioned before):

[Message part 2 (text/x-patch, inline)]
diff --git a/libguile/vm.c b/libguile/vm.c
index 6fd5c554f..516bae773 100644
--- a/libguile/vm.c
+++ b/libguile/vm.c
@@ -165,11 +165,13 @@ capture_stack (union scm_vm_stack_element *stack_top,
                scm_t_dynstack *dynstack, uint32_t flags)
 {
   struct scm_vm_cont *p;
+  size_t stack_size;
 
-  p = scm_gc_malloc (sizeof (*p), "capture_vm_cont");
-  p->stack_size = stack_top - sp;
-  p->stack_bottom = scm_gc_malloc (p->stack_size * sizeof (*p->stack_bottom),
-                                   "capture_vm_cont");
+  stack_size = stack_top - sp;
+  p = scm_gc_malloc (sizeof (*p) + stack_size * sizeof (*p->stack_bottom),
+                     "capture_vm_cont");
+  p->stack_size = stack_size;
+  p->stack_bottom = (void *) ((char *) p + sizeof (*p));
   p->vra = vra;
   p->mra = mra;
   p->fp_offset = stack_top - fp;
[Message part 3 (text/plain, inline)]
Using a simple heap profiler (more on that later), I noticed that the
stacks allocated at ‘p->stack_bottom’ would be partly retained,
explaining the heap growth.

I couldn’t pinpoint what exactly is keeping a pointer to the stack, but
what I can tell is that the trick above makes that impossible (because
we disable interior pointer tracing), hence the difference.

Also, why changing the SCM_DYNSTACK_TYPE_DYNAMIC_STATE entry to an
SCM_DYNSTACK_TYPE_UNWINDER entry would make a difference remains a
mystery to me.

I’m interested in theories that would explain all this in more detail!
I’ll go ahead with the fix above if there are no objections.

It’s not fully satisfying but still it’s a relief.

Ludo’.

Information forwarded to bug-guile <at> gnu.org:
bug#59021; Package guile. (Mon, 07 Nov 2022 16:04:02 GMT) Full text and rfc822 format available.

Message #13 received at 59021 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 59021 <at> debbugs.gnu.org
Subject: Re: bug#59021: Unbounded heap growth when combining dynamic states
 & delimited continuation
Date: Mon, 07 Nov 2022 11:03:26 -0500
Hi Ludovic,

Ludovic Courtès <ludo <at> gnu.org> writes:

> (This is a followup to <https://github.com/wingo/fibers/issues/65>,
> itself a followup to <https://issues.guix.gnu.org/58631>.)
>
> Consider this code:
>
> ;; https://issues.guix.gnu.org/58631
> ;; https://github.com/wingo/fibers/issues/65
>
> (define loss
>   (make-vector 1000000))
>
> (let ((tag (make-prompt-tag "my prompt")))
>   (define handler
>     (lambda (k i)
>       (when (zero? (modulo i 2000000))
>         (pk 'heap-size (assoc-ref (gc-stats) 'heap-size)))
>
>       (call-with-prompt tag
>         (lambda ()
>           (k (modulo (+ 1 i) 10000000)))
>         handler)))
>
>   (call-with-prompt tag
>     (let ((state (current-dynamic-state)))
>       (lambda ()
>         ;; (define (with-dynamic-state state thunk)
>         ;;   (let ((previous #f))
>         ;;     (dynamic-wind
>         ;;       (lambda () (set! previous (set-current-dynamic-state state)))
>         ;;       thunk
>         ;;       (lambda () (set-current-dynamic-state previous)))))
>         (with-dynamic-state state
>                             (lambda ()
>                               (let loop ((i 0))
>                                 (loop (abort-to-prompt tag i)))))))
>     handler))
>
> On Guile 3.0.8, this program exhibits seemingly unbounded heap growth.
> Uncommenting the local ‘with-dynamic-state’ definition fixes the
> problem.

I've tested both 3.0.8 from Guix on multiple machines (including Berlin)
and 2.2 from Debian 10, and ran the above snippet; it grows initially
but stabilize quickly and then doesn't budge.  I've let it run for more
than an hour.

So there's a problem there (?), but it doesn't seem like an unbound leak
from my experiments.  Perhaps the reproducer needs to be tweaked to
mimic better what is happening in Shepherd?

-- 
Thanks,
Maxim




Information forwarded to bug-guile <at> gnu.org:
bug#59021; Package guile. (Mon, 07 Nov 2022 21:53:01 GMT) Full text and rfc822 format available.

Message #16 received at 59021 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 59021 <at> debbugs.gnu.org
Subject: Re: bug#59021: Unbounded heap growth when combining dynamic states
 & delimited continuation
Date: Mon, 07 Nov 2022 16:52:30 -0500
Hi,

Maxim Cournoyer <maxim.cournoyer <at> gmail.com> writes:

[...]

> I've tested both 3.0.8 from Guix on multiple machines (including Berlin)
> and 2.2 from Debian 10, and ran the above snippet; it grows initially
> but stabilize quickly and then doesn't budge.  I've let it run for more
> than an hour.

Actually, it does grow, it just takes a lot of time.  It's now at
150751472 from the original "stable" value of 87846912, so it seems like
an unbound leak after all.

-- 
Thanks,
Maxim




Reply sent to Ludovic Courtès <ludo <at> gnu.org>:
You have taken responsibility. (Sun, 20 Nov 2022 17:29:02 GMT) Full text and rfc822 format available.

Notification sent to Ludovic Courtès <ludo <at> gnu.org>:
bug acknowledged by developer. (Sun, 20 Nov 2022 17:29:02 GMT) Full text and rfc822 format available.

Message #21 received at 59021-done <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: 59021-done <at> debbugs.gnu.org
Cc: Andy Wingo <wingo <at> pobox.com>
Subject: Re: bug#59021: Unbounded heap growth when combining dynamic states
 & delimited continuation
Date: Sun, 20 Nov 2022 18:28:13 +0100
Hi,

Ludovic Courtès <ludo <at> gnu.org> skribis:

> Ludovic Courtès <ludo <at> gnu.org> skribis:
>
>> Consider this code:
>>
>> ;; https://issues.guix.gnu.org/58631
>> ;; https://github.com/wingo/fibers/issues/65
>>
>> (define loss
>>   (make-vector 1000000))
>>
>> (let ((tag (make-prompt-tag "my prompt")))
>>   (define handler
>>     (lambda (k i)
>>       (when (zero? (modulo i 2000000))
>>         (pk 'heap-size (assoc-ref (gc-stats) 'heap-size)))
>>
>>       (call-with-prompt tag
>>         (lambda ()
>>           (k (modulo (+ 1 i) 10000000)))
>>         handler)))
>>
>>   (call-with-prompt tag
>>     (let ((state (current-dynamic-state)))
>>       (lambda ()
>>         ;; (define (with-dynamic-state state thunk)
>>         ;;   (let ((previous #f))
>>         ;;     (dynamic-wind
>>         ;;       (lambda () (set! previous (set-current-dynamic-state state)))
>>         ;;       thunk
>>         ;;       (lambda () (set-current-dynamic-state previous)))))
>>         (with-dynamic-state state
>>                             (lambda ()
>>                               (let loop ((i 0))
>>                                 (loop (abort-to-prompt tag i)))))))
>>     handler))
>>
>> On Guile 3.0.8, this program exhibits seemingly unbounded heap growth.
>
> This is fixed by the patch below (tested against the test case above and
> the Fibers and Shepherd test cases mentioned before):

Pushed as e47a153317c046ea5d335940412999e7dc604c33.

> Using a simple heap profiler (more on that later), I noticed that the
> stacks allocated at ‘p->stack_bottom’ would be partly retained,
> explaining the heap growth.
>
> I couldn’t pinpoint what exactly is keeping a pointer to the stack, but
> what I can tell is that the trick above makes that impossible (because
> we disable interior pointer tracing), hence the difference.
>
> Also, why changing the SCM_DYNSTACK_TYPE_DYNAMIC_STATE entry to an
> SCM_DYNSTACK_TYPE_UNWINDER entry would make a difference remains a
> mystery to me.
>
> I’m interested in theories that would explain all this in more detail!
> I’ll go ahead with the fix above if there are no objections.

I still am.  :-)

Ludo’.




bug archived. Request was from Debbugs Internal Request <help-debbugs <at> gnu.org> to internal_control <at> debbugs.gnu.org. (Mon, 19 Dec 2022 12:24:09 GMT) Full text and rfc822 format available.

This bug report was last modified 1 year and 101 days ago.

Previous Next


GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.