GNU bug report logs - #22608
Module system thread unsafety and .go compilation

Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.
Package: guix; Severity: important; Reported by: taylanbayirli@HIDDEN (Taylan Ulrich Bayırlı/Kammer); dated Tue, 9 Feb 2016 20:03:01 UTC; Maintainer for guix is bug-guix@HIDDEN.

Message received at 22608 <at> debbugs.gnu.org:


Received: (at 22608) by debbugs.gnu.org; 3 Jul 2018 22:10:23 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Jul 03 18:10:23 2018
Received: from localhost ([127.0.0.1]:45712 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1faTV9-00037R-01
	for submit <at> debbugs.gnu.org; Tue, 03 Jul 2018 18:10:23 -0400
Received: from eggs.gnu.org ([208.118.235.92]:38870)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1faTV7-00037A-Mb
 for 22608 <at> debbugs.gnu.org; Tue, 03 Jul 2018 18:10:22 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <ludo@HIDDEN>) id 1faTV1-0002vP-HG
 for 22608 <at> debbugs.gnu.org; Tue, 03 Jul 2018 18:10:16 -0400
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=-0.0 required=5.0 tests=BAYES_20 autolearn=disabled
 version=3.3.2
Received: from fencepost.gnu.org ([2001:4830:134:3::e]:56524)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <ludo@HIDDEN>)
 id 1faTUz-0002tM-BF; Tue, 03 Jul 2018 18:10:13 -0400
Received: from [2a01:e0a:1d:7270:6a6c:dc17:fc02:cfda] (port=48332 helo=ribbon)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <ludo@HIDDEN>)
 id 1faTUx-0001Ur-Lf; Tue, 03 Jul 2018 18:10:12 -0400
From: ludo@HIDDEN (Ludovic =?utf-8?Q?Court=C3=A8s?=)
To: taylanbayirli@HIDDEN (Taylan Ulrich =?utf-8?Q?=22Bay=C4=B1rl=C4=B1?=
 =?utf-8?Q?=2FKammer=22?=)
Subject: Re: bug#22608: Module system thread unsafety and .go compilation
References: <8737t1k5yk.fsf@HIDDEN>
Date: Wed, 04 Jul 2018 00:10:10 +0200
In-Reply-To: <8737t1k5yk.fsf@HIDDEN> ("Taylan Ulrich
 \=\?utf-8\?Q\?\=5C\=22Bay\=C4\=B1rl\=C4\=B1\=2FKammer\=5C\=22\=22's\?\=
 message of "Tue, 09 Feb 2016 21:02:27 +0100")
Message-ID: <87d0w4ezst.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2001:4830:134:3::e
X-Spam-Score: -5.0 (-----)
X-Debbugs-Envelope-To: 22608
Cc: 22608 <at> debbugs.gnu.org, guile-devel@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -6.0 (------)

Hello,

taylanbayirli@HIDDEN (Taylan Ulrich "Bay=C4=B1rl=C4=B1/Kammer") skribis:

> To speed up the compilation of the many Scheme files in Guix, we use a
> script that first loads all modules to be compiled into the Guile
> process (by calling 'resolve-interface' on the module names), and then
> the corresponding Scheme files are compiled in a par-for-each.
>
> While Guile's module system is known to be thread unsafe, the idea was
> that all mutation should happen in the serial loading phase, and the
> parallel compile-file calls should then be thread safe.
>
> Sadly that assumption isn't met when autoloads are involved.

For the record, these issues should be fixed in Guile 2.2.4:

533e3ff17 * Serialize accesses to submodule hash tables.
46bcbfa56 * Module import obarrays are accessed in a critical section.
761cf0fb8 * Make module autoloading thread-safe.

=E2=80=98guix pull=E2=80=99 now defaults to 2.2.4, so we=E2=80=99ll see if =
indeed those crashes
disappear.

Ludo=E2=80=99.
Information forwarded to bug-guix@HIDDEN:
bug#22608; Package guix. Full text available.
Severity set to 'important' from 'normal' Request was from ludo@HIDDEN (Ludovic Courtès) to control <at> debbugs.gnu.org. Full text available.

Message received at 22608 <at> debbugs.gnu.org:


Received: (at 22608) by debbugs.gnu.org; 10 Feb 2016 13:50:57 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Wed Feb 10 08:50:57 2016
Received: from localhost ([127.0.0.1]:34330 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1aTVAX-0006b3-4D
	for submit <at> debbugs.gnu.org; Wed, 10 Feb 2016 08:50:57 -0500
Received: from eggs.gnu.org ([208.118.235.92]:59837)
 by debbugs.gnu.org with esmtp (Exim 4.84)
 (envelope-from <ludo@HIDDEN>) id 1aTVAW-0006ac-8D
 for 22608 <at> debbugs.gnu.org; Wed, 10 Feb 2016 08:50:56 -0500
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <ludo@HIDDEN>) id 1aTVAO-0004JO-0Y
 for 22608 <at> debbugs.gnu.org; Wed, 10 Feb 2016 08:50:51 -0500
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=0.6 required=5.0 tests=BAYES_50,RP_MATCHES_RCVD
 autolearn=disabled version=3.3.2
Received: from fencepost.gnu.org ([2001:4830:134:3::e]:36382)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <ludo@HIDDEN>)
 id 1aTVAL-0004FW-2Z; Wed, 10 Feb 2016 08:50:45 -0500
Received: from reverse-83.fdn.fr ([80.67.176.83]:58368 helo=pluto)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_128_CBC_SHA1:128)
 (Exim 4.82) (envelope-from <ludo@HIDDEN>)
 id 1aTVAK-0001rL-8k; Wed, 10 Feb 2016 08:50:44 -0500
From: ludo@HIDDEN (Ludovic =?utf-8?Q?Court=C3=A8s?=)
To: taylanbayirli@HIDDEN (Taylan Ulrich =?utf-8?Q?=22Bay=C4=B1rl=C4=B1?=
 =?utf-8?Q?=2FKammer=22?=)
Subject: Re: bug#22608: Module system thread unsafety and .go compilation
In-Reply-To: <8737t1k5yk.fsf@HIDDEN> ("Taylan Ulrich
 \=\?utf-8\?Q\?\=5C\=22Bay\=C4\=B1rl\=C4\=B1\=2FKammer\=5C\=22\=22's\?\=
 message of "Tue, 09 Feb 2016 21:02:27 +0100")
References: <8737t1k5yk.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux)
Date: Wed, 10 Feb 2016 14:50:42 +0100
Message-ID: <87egckn07h.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2001:4830:134:3::e
X-Spam-Score: -5.3 (-----)
X-Debbugs-Envelope-To: 22608
Cc: 22608 <at> debbugs.gnu.org, guile-devel@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -5.3 (-----)

taylanbayirli@HIDDEN (Taylan Ulrich "Bay=C4=B1rl=C4=B1/Kammer") skribis:

> Sadly that assumption isn't met when autoloads are involved.
> Minimal-ish test-case:
>
> - Check out 0889321.
>
> - Build it.
>
> - Edit gnu/build/activation.scm and gnu/build/linux-boot.scm to contain
>   merely the following expressions, respectively:
>
> (define-module (gnu build activation)
>   #:use-module (gnu build linux-boot))
>
> (define-module (gnu build linux-boot)
>   #:autoload   (system base compile) (compile-file))
>
> - Run make again.
>
> If you're on a multi-core system, you will probably get an error saying
> something weird like "no such language scheme".

Do you have a clear explanation of why this happens?  I would expect
(system base compile) to already be loaded for instance, so it=E2=80=99s not
clear to me what=E2=80=99s going on.  Or is it just the mutation of (gnu bu=
ild
linux-boot) that=E2=80=99s causing problems?

> Solution proposals:
>
> 1. s/par-for-each/for-each/.  Will make compilation slower on multi-core
>    machines.  We would do the same for guix pull, which is a bit sad
>    because it's so fast right now.  Very simple solution though.
>
> 2. We find out some partitioning of the Scheme modules such that there
>    is minimal overlap in total loaded modules when the modules in one
>    subset are each loaded by one Guile process.  Then each Guile process
>    loads & compiles the modules in its given subset serially, but these
>    Guile processes run in parallel.  This could speed things up even
>    more than now because the module-loading phases of the processes
>    would be parallel too.  It also has the side-effect that less memory
>    is consumed the fewer cores you have (because less Scheme modules
>    loaded into memory at once).  If someone (Ludo?)  has a good general
>    overview of Guix's module graph then maybe they can come up with a
>    sensible partitioning of the modules, say into 4 subsets (maxing out
>    benefits at quad-core), such that loading all modules in one subset
>    loads a minimal amount of modules that are outside that subset.  That
>    should be the only challenging part of this solution.
>
> 3. We do nothing for now since this bug triggers rarely, and can be
>    worked around by simply re-running make.  (We just have to hope that
>    it doesn't trigger on guix pull or on clean builds after some commit;
>    there's no "just rerun make" in guix pull or an automated build of
>    Guix.)  AFAIU Wingo expressed motivation to make Guile's module
>    system thread safe, so this problem would then truly disappear.

Short-term, I=E2=80=99d do #1 or #3; probably #1 though, because random fai=
lures
are no fun, and we know they can happen.

Longer-term, I=E2=80=99m not convinced by #2.  I think I would instead build
packages in reverse topological order, probably serially at first, which
would address <http://bugs.gnu.org/15602> (with the caveat that the (gnu
packages =E2=80=A6) modules cannot be topologically-sorted, but OTOH they
typically don=E2=80=99t use macros, so we=E2=80=99re fine.)

That would require a tool to extract and the =E2=80=98define-module=E2=80=
=99 forms and
build a graph from there.

But really, we must fix <http://bugs.gnu.org/15602>, an in particular,
=E2=80=98compile-file=E2=80=99 should not mutate the global module name spa=
ce.  I think
we could do something like:

  (define (compile-file* =E2=80=A6)
    (let ((root the-root-module)
          (compile-root (copy-module the-root-module)))
      (dynamic-wind
        (lambda ()
          (set! the-root-module compile-root)
          ;; ditto with the-scm-module
          )
        (lambda ()
          (compile-file =E2=80=A6))
        (lambda ()
          (set! the-root-module root)
          ;; =E2=80=A6
          ))))

It=E2=80=99s unclear how costly =E2=80=98copy-module=E2=80=99 would be, and=
 the whole strategy
depends on it.

Eventually it seems clear that Guile proper needs to address this use
case, and needs to provide thread-safe modules.

Ludo=E2=80=99.
Information forwarded to bug-guix@HIDDEN:
bug#22608; Package guix. Full text available.

Message received at 22608 <at> debbugs.gnu.org:


Received: (at 22608) by debbugs.gnu.org; 10 Feb 2016 13:50:54 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Wed Feb 10 08:50:54 2016
Received: from localhost ([127.0.0.1]:34327 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1aTVAT-0006ao-Rq
	for submit <at> debbugs.gnu.org; Wed, 10 Feb 2016 08:50:54 -0500
Received: from eggs.gnu.org ([208.118.235.92]:59771)
 by debbugs.gnu.org with esmtp (Exim 4.84)
 (envelope-from <ludo@HIDDEN>) id 1aTVAS-0006aX-BX
 for 22608 <at> debbugs.gnu.org; Wed, 10 Feb 2016 08:50:52 -0500
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <ludo@HIDDEN>) id 1aTVAL-0004Fb-C9
 for 22608 <at> debbugs.gnu.org; Wed, 10 Feb 2016 08:50:47 -0500
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=0.6 required=5.0 tests=BAYES_50,RP_MATCHES_RCVD
 autolearn=disabled version=3.3.2
Received: from fencepost.gnu.org ([2001:4830:134:3::e]:36381)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <ludo@HIDDEN>)
 id 1aTVAE-0004Ej-U5; Wed, 10 Feb 2016 08:50:38 -0500
Received: from reverse-83.fdn.fr ([80.67.176.83]:58366 helo=pluto)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_128_CBC_SHA1:128)
 (Exim 4.82) (envelope-from <ludo@HIDDEN>)
 id 1aTVAE-0001qr-6A; Wed, 10 Feb 2016 08:50:38 -0500
From: ludo@HIDDEN (Ludovic =?utf-8?Q?Court=C3=A8s?=)
To: taylanbayirli@HIDDEN (Taylan Ulrich =?utf-8?Q?=22Bay=C4=B1rl=C4=B1?=
 =?utf-8?Q?=2FKammer=22?=)
Subject: Re: bug#22608: Module system thread unsafety and .go compilation
References: <8737t1k5yk.fsf@HIDDEN>
Date: Wed, 10 Feb 2016 14:50:35 +0100
In-Reply-To: <8737t1k5yk.fsf@HIDDEN> ("Taylan Ulrich
 \=\?utf-8\?Q\?\=5C\=22Bay\=C4\=B1rl\=C4\=B1\=2FKammer\=5C\=22\=22's\?\=
 message of "Tue, 09 Feb 2016 21:02:27 +0100")
Message-ID: <87fux0n07o.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2001:4830:134:3::e
X-Spam-Score: -5.3 (-----)
X-Debbugs-Envelope-To: 22608
Cc: 22608 <at> debbugs.gnu.org, guile-devel@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -5.3 (-----)

taylanbayirli@HIDDEN (Taylan Ulrich "Bay=C4=B1rl=C4=B1/Kammer") skribis:

> Sadly that assumption isn't met when autoloads are involved.
> Minimal-ish test-case:
>
> - Check out 0889321.
>
> - Build it.
>
> - Edit gnu/build/activation.scm and gnu/build/linux-boot.scm to contain
>   merely the following expressions, respectively:
>
> (define-module (gnu build activation)
>   #:use-module (gnu build linux-boot))
>
> (define-module (gnu build linux-boot)
>   #:autoload   (system base compile) (compile-file))
>
> - Run make again.
>
> If you're on a multi-core system, you will probably get an error saying
> something weird like "no such language scheme".

Do you have a clear explanation of why this happens?  I would expect
(system base compile) to already be loaded for instance, so it=E2=80=99s not
clear to me what=E2=80=99s going on.  Or is it just the mutation of (gnu bu=
ild
linux-boot) that=E2=80=99s causing problems?

> Solution proposals:
>
> 1. s/par-for-each/for-each/.  Will make compilation slower on multi-core
>    machines.  We would do the same for guix pull, which is a bit sad
>    because it's so fast right now.  Very simple solution though.
>
> 2. We find out some partitioning of the Scheme modules such that there
>    is minimal overlap in total loaded modules when the modules in one
>    subset are each loaded by one Guile process.  Then each Guile process
>    loads & compiles the modules in its given subset serially, but these
>    Guile processes run in parallel.  This could speed things up even
>    more than now because the module-loading phases of the processes
>    would be parallel too.  It also has the side-effect that less memory
>    is consumed the fewer cores you have (because less Scheme modules
>    loaded into memory at once).  If someone (Ludo?)  has a good general
>    overview of Guix's module graph then maybe they can come up with a
>    sensible partitioning of the modules, say into 4 subsets (maxing out
>    benefits at quad-core), such that loading all modules in one subset
>    loads a minimal amount of modules that are outside that subset.  That
>    should be the only challenging part of this solution.
>
> 3. We do nothing for now since this bug triggers rarely, and can be
>    worked around by simply re-running make.  (We just have to hope that
>    it doesn't trigger on guix pull or on clean builds after some commit;
>    there's no "just rerun make" in guix pull or an automated build of
>    Guix.)  AFAIU Wingo expressed motivation to make Guile's module
>    system thread safe, so this problem would then truly disappear.

Short-term, I=E2=80=99d do #1 or #3; probably #1 though, because random fai=
lures
are no fun, and we know they can happen.

Longer-term, I=E2=80=99m not convinced by #2.  I think I would instead build
packages in reverse topological order, probably serially at first, which
would address <http://bugs.gnu.org/15602> (with the caveat that the (gnu
packages =E2=80=A6) modules cannot be topologically-sorted, but OTOH they
typically don=E2=80=99t use macros, so we=E2=80=99re fine.)

That would require a tool to extract and the =E2=80=98define-module=E2=80=
=99 forms and
build a graph from there.

But really, we must fix <http://bugs.gnu.org/15602>, an in particular,
=E2=80=98compile-file=E2=80=99 should not mutate the global module name spa=
ce.  I think
we could do something like:

  (define (compile-file* =E2=80=A6)
    (let ((root the-root-module)
          (compile-root (copy-module the-root-module)))
      (dynamic-wind
        (lambda ()
          (set! the-root-module compile-root)
          ;; ditto with the-scm-module
          )
        (lambda ()
          (compile-file =E2=80=A6))
        (lambda ()
          (set! the-root-module root)
          ;; =E2=80=A6
          ))))

It=E2=80=99s unclear how costly =E2=80=98copy-module=E2=80=99 would be, and=
 the whole strategy
depends on it.

Eventually it seems clear that Guile proper needs to address this use
case, and needs to provide thread-safe modules.

Ludo=E2=80=99.
Information forwarded to bug-guix@HIDDEN:
bug#22608; Package guix. Full text available.

Message received at submit <at> debbugs.gnu.org:


Received: (at submit) by debbugs.gnu.org; 9 Feb 2016 20:02:47 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Feb 09 15:02:47 2016
Received: from localhost ([127.0.0.1]:33885 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1aTEUo-0004Tu-U6
	for submit <at> debbugs.gnu.org; Tue, 09 Feb 2016 15:02:47 -0500
Received: from eggs.gnu.org ([208.118.235.92]:34697)
 by debbugs.gnu.org with esmtp (Exim 4.84)
 (envelope-from <taylanbayirli@HIDDEN>) id 1aTEUn-0004Ti-AF
 for submit <at> debbugs.gnu.org; Tue, 09 Feb 2016 15:02:45 -0500
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <taylanbayirli@HIDDEN>) id 1aTEUd-00064n-76
 for submit <at> debbugs.gnu.org; Tue, 09 Feb 2016 15:02:40 -0500
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50,FREEMAIL_FROM,
 T_DKIM_INVALID autolearn=disabled version=3.3.2
Received: from lists.gnu.org ([2001:4830:134:3::11]:56758)
 by eggs.gnu.org with esmtp (Exim 4.71)
 (envelope-from <taylanbayirli@HIDDEN>) id 1aTEUd-00064j-42
 for submit <at> debbugs.gnu.org; Tue, 09 Feb 2016 15:02:35 -0500
Received: from eggs.gnu.org ([2001:4830:134:3::10]:53804)
 by lists.gnu.org with esmtp (Exim 4.71)
 (envelope-from <taylanbayirli@HIDDEN>) id 1aTEUc-0001ra-12
 for bug-guix@HIDDEN; Tue, 09 Feb 2016 15:02:35 -0500
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <taylanbayirli@HIDDEN>) id 1aTEUa-000643-JX
 for bug-guix@HIDDEN; Tue, 09 Feb 2016 15:02:33 -0500
Received: from mail-wm0-x22f.google.com ([2a00:1450:400c:c09::22f]:33640)
 by eggs.gnu.org with esmtp (Exim 4.71)
 (envelope-from <taylanbayirli@HIDDEN>)
 id 1aTEUY-00063T-BE; Tue, 09 Feb 2016 15:02:30 -0500
Received: by mail-wm0-x22f.google.com with SMTP id g62so189072111wme.0;
 Tue, 09 Feb 2016 12:02:29 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;
 h=from:to:cc:subject:date:message-id:user-agent:mime-version
 :content-type; bh=vahbhIvWVj41OaVJM9OS+1iR9q+B+w1Da14IMPLAj4s=;
 b=ISRUc9M5+uAi1iMiEW0EewNz75z1mJh11oy7JiLNBFEkfRIbAUdNJ9kGlBgFDNjIO/
 OmOPeg2BbEuTJdkMO2WF0Qo0hckzc6CfApggGw6o5PBxp5B1khlZbilS7WS2eapY7ay9
 zW5rA21RZzmgI4NaVrhbDC9+zeBM32lS0c9P0YWqXjR/gSw7IiVqXI+mY5Tq57po4jgk
 KInB7Z8aiWWgDU2Y0Ls5iRIqi79FqyKnvAz75d+yaxEifLSWGYJEMOEd10crWtZMJkR6
 mNromyxL2OFSYT73AKNUDA4nGKl1V94kFnpSyMcNeEjSxjY3IGCUSIG4fdswr0sZ31MF
 xkXg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20130820;
 h=x-gm-message-state:from:to:cc:subject:date:message-id:user-agent
 :mime-version:content-type;
 bh=vahbhIvWVj41OaVJM9OS+1iR9q+B+w1Da14IMPLAj4s=;
 b=OM5p1VU1oYDlz8JzcCm9vh0xXGWPZz/JpianWuMQEfaJdSTqzRbYGOHnNrDacpgEuS
 s3m7BWyZ10E33pI/3lqXklFtfMQhEUmJKvjqMc18un2E9Sffn7+vagiBOw/Eb1ciSU09
 VqmVGqAxxU1QnHyyOlWNg4eFbMQscQ39gvZIcddwUeryQksoWRGUrrp6SHFtvOX68kFf
 DAUzXJsMT6TCQTQ3Hq//bvDGIFVAEWJO+IWU55uhpHFBJMjslY0CMFuN61dwn065AO6r
 KrS1Ra5j/ZdlzPcpk8UybBpBhvzS1+1qTQC16ozS6Lg9wjhpRUfR2E09cogdXuLTc45G
 QJ5w==
X-Gm-Message-State: AG10YOTNHBwF8CyJ4mBIIzKC1mYNNLlqmox6MXBxmgGzWZ5hUq8n+6MzLdriUC/Ij9AuPA==
X-Received: by 10.28.103.5 with SMTP id b5mr7257922wmc.5.1455048149355;
 Tue, 09 Feb 2016 12:02:29 -0800 (PST)
Received: from T420.taylan ([2a02:908:c30:3ec0:221:ccff:fe66:68f0])
 by smtp.gmail.com with ESMTPSA id y188sm164723wmy.11.2016.02.09.12.02.27
 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);
 Tue, 09 Feb 2016 12:02:28 -0800 (PST)
From: taylanbayirli@HIDDEN (Taylan Ulrich =?utf-8?Q?Bay=C4=B1rl=C4=B1?=
 =?utf-8?Q?=2FKammer?=)
To: bug-guix@HIDDEN
Subject: Module system thread unsafety and .go compilation
Date: Tue, 09 Feb 2016 21:02:27 +0100
Message-ID: <8737t1k5yk.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x
X-Received-From: 2001:4830:134:3::11
X-Spam-Score: -4.0 (----)
X-Debbugs-Envelope-To: submit
Cc: guile-devel@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -4.0 (----)

To speed up the compilation of the many Scheme files in Guix, we use a
script that first loads all modules to be compiled into the Guile
process (by calling 'resolve-interface' on the module names), and then
the corresponding Scheme files are compiled in a par-for-each.

While Guile's module system is known to be thread unsafe, the idea was
that all mutation should happen in the serial loading phase, and the
parallel compile-file calls should then be thread safe.

Sadly that assumption isn't met when autoloads are involved.
Minimal-ish test-case:

- Check out 0889321.

- Build it.

- Edit gnu/build/activation.scm and gnu/build/linux-boot.scm to contain
  merely the following expressions, respectively:

(define-module (gnu build activation)
  #:use-module (gnu build linux-boot))

(define-module (gnu build linux-boot)
  #:autoload   (system base compile) (compile-file))

- Run make again.

If you're on a multi-core system, you will probably get an error saying
something weird like "no such language scheme".

Note: when you then run make *again* it succeeds.


Solution proposals:

1. s/par-for-each/for-each/.  Will make compilation slower on multi-core
   machines.  We would do the same for guix pull, which is a bit sad
   because it's so fast right now.  Very simple solution though.

2. We find out some partitioning of the Scheme modules such that there
   is minimal overlap in total loaded modules when the modules in one
   subset are each loaded by one Guile process.  Then each Guile process
   loads & compiles the modules in its given subset serially, but these
   Guile processes run in parallel.  This could speed things up even
   more than now because the module-loading phases of the processes
   would be parallel too.  It also has the side-effect that less memory
   is consumed the fewer cores you have (because less Scheme modules
   loaded into memory at once).  If someone (Ludo?)  has a good general
   overview of Guix's module graph then maybe they can come up with a
   sensible partitioning of the modules, say into 4 subsets (maxing out
   benefits at quad-core), such that loading all modules in one subset
   loads a minimal amount of modules that are outside that subset.  That
   should be the only challenging part of this solution.

3. We do nothing for now since this bug triggers rarely, and can be
   worked around by simply re-running make.  (We just have to hope that
   it doesn't trigger on guix pull or on clean builds after some commit;
   there's no "just rerun make" in guix pull or an automated build of
   Guix.)  AFAIU Wingo expressed motivation to make Guile's module
   system thread safe, so this problem would then truly disappear.

I think #2 is a pretty good solution.  The only thing worrying me is
that we might not be able to sensibly partition the Scheme modules
according to any simple logic that can be automated (like guix/ is one
subset, gnu/packages/ is another, etc.).  Maintaining the subsets
manually in the Makefile would be pretty ugly.  But maybe some simple
logic, possibly combined with few special-cases in the code, would be
good enough.

Thoughts?

Taylan
Acknowledgement sent to taylanbayirli@HIDDEN (Taylan Ulrich Bayırlı/Kammer):
New bug report received and forwarded. Copy sent to bug-guix@HIDDEN. Full text available.
Report forwarded to bug-guix@HIDDEN:
bug#22608; Package guix. Full text available.
Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.
Last modified: Mon, 25 Nov 2019 12:00:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.