GNU logs - #14756, boring messages

Message sent to bug-guile@HIDDEN:

Subject: bug#14756: threads - par-map - multicore issue
Resent-From: David Pirotte <david@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guile@HIDDEN
Resent-Date: Sun, 30 Jun 2013 18:02:02 +0000
Resent-Message-ID: <handler.14756.B.137261529626406 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: 14756 <at> debbugs.gnu.org
Date: Sun, 30 Jun 2013 15:00:54 -0300
From: David Pirotte <david@HIDDEN>
Message-ID: <20130630150054.07f19f35@capac>
Mime-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

Hello,

	guile --version
	guile (GNU Guile) 2.0.9.20-10454

It seems that the par-map not using all cores problem has some how been reintroduced?

	guile -c '(begin (use-modules (ice-9 threads)) (par-map 1+ (iota 400000)))'

only uses 1 core [it seems it uses some other [maybe all, i can't tell] a couple of
milliseconds, then drops to 1 core only.

Thanks,
David

;; -- 

david@idefix:~ 16 $ guile -c '(begin
>     (use-modules (ice-9 threads))
>     (par-map 1+ (iota 400))
>     (display (current-processor-count)) (display "\n")
>     (display (length (@@ (ice-9 futures) %workers))) (display "\n"))'
12
11

Message sent:

Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Mailer: MIME-tools 5.503 (Entity 5.503)
Content-Type: text/plain; charset=utf-8
X-Loop: help-debbugs@HIDDEN
From: help-debbugs@HIDDEN (GNU bug Tracking System)
To: David Pirotte <david@HIDDEN>
Subject: bug#14756: Acknowledgement (threads - par-map - multicore issue)
Message-ID: <handler.14756.B.137261529626406.ack <at> debbugs.gnu.org>
References: <20130630150054.07f19f35@capac>
X-Gnu-PR-Message: ack 14756
X-Gnu-PR-Package: guile
Reply-To: 14756 <at> debbugs.gnu.org
Date: Sun, 30 Jun 2013 18:02:03 +0000

Thank you for filing a new bug report with debbugs.gnu.org.

This is an automatically generated reply to let you know your message
has been received.

Your message is being forwarded to the package maintainers and other
interested parties for their attention; they will reply in due course.

Your message has been sent to the package maintainer(s):
 bug-guile@HIDDEN

If you wish to submit further information on this problem, please
send it to 14756 <at> debbugs.gnu.org.

Please do not send mail to help-debbugs@HIDDEN unless you wish
to report a problem with the Bug-tracking system.

--=20
14756: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D14756
GNU Bug Tracking System
Contact help-debbugs@HIDDEN with problems

Message sent to bug-guile@HIDDEN:

Subject: bug#14756: threads - par-map - multicore issue
Resent-From: Andy Wingo <wingo@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guile@HIDDEN
Resent-Date: Tue, 21 Jun 2016 06:52:01 +0000
Resent-Message-ID: <handler.14756.B14756.1466491884753 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: ludo@HIDDEN
Cc: 14756 <at> debbugs.gnu.org, David Pirotte <david@HIDDEN>
DomainKey-Signature: a=rsa-sha1; c=nofws; d=pobox.com; h=from:to:cc
 :subject:references:date:in-reply-to:message-id:mime-version
 :content-type; q=dns; s=sasl; b=Ij7AdWal67OeoJ5OZ/psuQCWE2l7XIrK
 j7BrpULVnzD+PsEwoefk2fQbRsJvn1pGLBQoFPRl67PYO+yqSCXyVx7xXb/aRDqv
 1joaJjDHhYA1LHxRmP/lzOfPdJzjOi0KHdqPVwKyOiVMUzR+QX9LjLYSE67UGaId
 lH9Yr8QTRFo=
From: Andy Wingo <wingo@HIDDEN>
References: <20130630150054.07f19f35@capac>
Date: Tue, 21 Jun 2016 08:51:09 +0200
In-Reply-To: <20130630150054.07f19f35@capac> (David Pirotte's message of "Sun, 
 30 Jun 2013 15:00:54 -0300")
Message-ID: <87lh1z592a.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

I see this, but I'm not quite sure what's going on.  What I do see is
that par-map of 1+ on a list is horribly slow, both on 2.0 and master.
Ludovic do you know what's going on here?

Andy

On Sun 30 Jun 2013 20:00, David Pirotte <david@HIDDEN> writes:

> Hello,
>
> 	guile --version
> 	guile (GNU Guile) 2.0.9.20-10454
>
> It seems that the par-map not using all cores problem has some how been reintroduced?
>
> 	guile -c '(begin (use-modules (ice-9 threads)) (par-map 1+ (iota 400000)))'
>
> only uses 1 core [it seems it uses some other [maybe all, i can't tell] a couple of
> milliseconds, then drops to 1 core only.
>
> Thanks,
> David
>
> ;; -- 
>
> david@idefix:~ 16 $ guile -c '(begin
>>     (use-modules (ice-9 threads))
>>     (par-map 1+ (iota 400))
>>     (display (current-processor-count)) (display "\n")
>>     (display (length (@@ (ice-9 futures) %workers))) (display "\n"))'
> 12
> 11

Message sent to bug-guile@HIDDEN:

Subject: bug#14756: threads - par-map - multicore issue
Resent-From: ludo@HIDDEN (Ludovic =?UTF-8?Q?Court=C3=A8s?=)
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guile@HIDDEN
Resent-Date: Tue, 21 Jun 2016 08:35:02 +0000
Resent-Message-ID: <handler.14756.B14756.146649807110701 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: Andy Wingo <wingo@HIDDEN>
Cc: 14756 <at> debbugs.gnu.org, David Pirotte <david@HIDDEN>
From: ludo@HIDDEN (Ludovic =?UTF-8?Q?Court=C3=A8s?=)
References: <20130630150054.07f19f35@capac> <87lh1z592a.fsf@HIDDEN>
Date: Tue, 21 Jun 2016 10:33:47 +0200
In-Reply-To: <87lh1z592a.fsf@HIDDEN> (Andy Wingo's message of "Tue, 21 Jun
 2016 08:51:09 +0200")
Message-ID: <87vb13c55g.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

Andy Wingo <wingo@HIDDEN> skribis:

> I see this, but I'm not quite sure what's going on.  What I do see is
> that par-map of 1+ on a list is horribly slow, both on 2.0 and master.
> Ludovic do you know what's going on here?

As David put it, only one core is being used, which is clearly a bug.

I believe the bug was introduced by
8a177d316c0062afe74f9a761ef460e297435e59 (however, before that commit,
you would hit a stack overflow when doing =E2=80=98par-map=E2=80=99 on a la=
rge-enough
list.)

What happens is that =E2=80=98par-mapper=E2=80=99 creates nested futures wh=
ose
dependency graph forms a comb-shaped tree; thus we quickly hit
%MAX-NESTING-LEVEL.

This is fine in itself, but for some reason, it ends up evaluating most
of those futures in one thread while the other threads apparently remain
stuck in =E2=80=98wait-condition-variable=E2=80=99 in =E2=80=98process-futu=
res=E2=80=99.

I=E2=80=99ve looked into it a bit but that needs more time=E2=80=A6

Ludo=E2=80=99.

Message sent to bug-guile@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#14756: threads - par-map - multicore issue
Resent-From: Andy Wingo <wingo@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guile@HIDDEN
Resent-Date: Tue, 28 Feb 2017 09:54:01 +0000
Resent-Message-ID: <handler.14756.B14756.148827563231186 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 14756
X-GNU-PR-Package: guile
X-GNU-PR-Keywords: 
To: ludo@HIDDEN (Ludovic =?UTF-8?Q?Court=C3=A8s?=)
Cc: 14756 <at> debbugs.gnu.org, David Pirotte <david@HIDDEN>
Received: via spool by 14756-submit <at> debbugs.gnu.org id=B14756.148827563231186
          (code B ref 14756); Tue, 28 Feb 2017 09:54:01 +0000
Received: (at 14756) by debbugs.gnu.org; 28 Feb 2017 09:53:52 +0000
Received: from localhost ([127.0.0.1]:59070 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1cieTg-00086v-Aj
	for submit <at> debbugs.gnu.org; Tue, 28 Feb 2017 04:53:52 -0500
Received: from pb-sasl1.pobox.com ([64.147.108.66]:54793
 helo=sasl.smtp.pobox.com) by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <wingo@HIDDEN>) id 1cieTe-00086k-CW
 for 14756 <at> debbugs.gnu.org; Tue, 28 Feb 2017 04:53:50 -0500
Received: from sasl.smtp.pobox.com (unknown [127.0.0.1])
 by pb-sasl1.pobox.com (Postfix) with ESMTP id 7D9D25F838;
 Tue, 28 Feb 2017 04:53:48 -0500 (EST)
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=pobox.com; h=from:to:cc
 :subject:references:date:in-reply-to:message-id:mime-version
 :content-type:content-transfer-encoding; s=sasl; bh=J8TMkdKRAO7N
 t6ObTCVsSe8mhSU=; b=o5wbuiVF9AXk7nGEXUy8rDehPEK4hDiXkwVx2yMz1zYn
 1678CVeHuFtMlZAWYlP7XxfRePWCSBlGaC0L/agdwds76x81EOysNPri2ktmrptM
 be5gbMEaWOzMJA4YU1vIC+8dqLhHZYFhDntPToFnQ+9g4LlqwPi3DEqhguhm6UM=
DomainKey-Signature: a=rsa-sha1; c=nofws; d=pobox.com; h=from:to:cc
 :subject:references:date:in-reply-to:message-id:mime-version
 :content-type:content-transfer-encoding; q=dns; s=sasl; b=DDIIvg
 1+FcpixOsssE9EMEhl07geKt/Iwn0jDqlOnLOr4bNwPrGUqoPJb9e9D9S1Qh+RVq
 WkQEUOEeFl7Kp0kwYgu0YFcd9EJOszbS5PGAL4JnZ1AxuIDtKGjNTO9A3hwuKuie
 0q8ZhgzBc9cUlcmaS/ViryXhkxv8qMS91Sshw=
Received: from pb-sasl1.nyi.icgroup.com (unknown [127.0.0.1])
 by pb-sasl1.pobox.com (Postfix) with ESMTP id 6AAA75F836;
 Tue, 28 Feb 2017 04:53:48 -0500 (EST)
Received: from clucks (unknown [88.160.190.192])
 (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits))
 (No client certificate requested)
 by pb-sasl1.pobox.com (Postfix) with ESMTPSA id 1F93F5F835;
 Tue, 28 Feb 2017 04:53:47 -0500 (EST)
From: Andy Wingo <wingo@HIDDEN>
References: <20130630150054.07f19f35@capac> <87lh1z592a.fsf@HIDDEN>
 <87vb13c55g.fsf@HIDDEN>
Date: Tue, 28 Feb 2017 10:53:39 +0100
In-Reply-To: <87vb13c55g.fsf@HIDDEN> ("Ludovic =?UTF-8?Q?Court=C3=A8s?="'s message of "Tue, 21 Jun 2016 10:33:47 +0200")
Message-ID: <87fuiyproc.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.1 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Pobox-Relay-ID: CC81A62A-FD9B-11E6-8C80-CDEC6462E9F6-02397024!pb-sasl1.pobox.com
X-Spam-Score: 0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: 0.0 (/)

On Tue 21 Jun 2016 10:33, ludo@HIDDEN (Ludovic Court=C3=A8s) writes:

> Andy Wingo <wingo@HIDDEN> skribis:
>
>> I see this, but I'm not quite sure what's going on.  What I do see is
>> that par-map of 1+ on a list is horribly slow, both on 2.0 and master.
>> Ludovic do you know what's going on here?
>
> As David put it, only one core is being used, which is clearly a bug.
>
> I believe the bug was introduced by
> 8a177d316c0062afe74f9a761ef460e297435e59 (however, before that commit,
> you would hit a stack overflow when doing =E2=80=98par-map=E2=80=99 on a =
large-enough
> list.)

Given that Guile 2.2. doesn't have a stack limit problem, I have
reverted this commit on master (though I kept the tests).

FWIW Guile 2.0 with this test

   $ time ../guile-2.0/meta/guile -c '(begin (use-modules (ice-9 threads)) =
(par-map 1+ (iota 40000)))'

   real	1m45.282s
   user	1m45.208s
   sys	0m0.036s


Guile 2.1.x with the stack-limit stuff:

   $ time /opt/guile/bin/guile -c '(begin (use-modules (ice-9 threads)) (pa=
r-map 1+ (iota 40000)))'

   real	0m51.738s
   user	1m2.720s
   sys	0m0.116s

Guile 2.1.x after reverting the patch:

   $ time meta/guile -c '(begin (use-modules (ice-9 threads)) (par-map 1+ (=
iota 40000)))'

   real	0m1.403s
   user	0m1.396s
   sys	0m0.024s

Note that I took a zero off the original test in all examples above.
However!  I still have the problem that mostly only one core is used.  I
would imagine that is because the thread that builds the spine is more
costly than the threads that actually do the workload (the 1+ in this
case).  But maybe that is wrong.  Certainly there are improvements that
can be made in the futures implementation in 2.2 with atomic boxes.

Andy

Last modified: Mon, 25 Nov 2019 12:00:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.