GNU bug report logs - #36630
[PATCH] guix: parallelize building the manual-database

Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.

Package: guix-patches; Reported by: arne_bab@HIDDEN; Keywords: patch; dated Fri, 12 Jul 2019 21:44:01 UTC; Maintainer for guix-patches is guix-patches@HIDDEN.

Message received at 36630 <at> debbugs.gnu.org:


Received: (at 36630) by debbugs.gnu.org; 27 Oct 2019 16:28:21 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Oct 27 12:28:21 2019
Received: from localhost ([127.0.0.1]:43430 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1iOlOv-0004F1-Ha
	for submit <at> debbugs.gnu.org; Sun, 27 Oct 2019 12:28:21 -0400
Received: from mout.web.de ([212.227.17.12]:40003)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <arne_bab@HIDDEN>) id 1iOlOt-0004Eo-Gi
 for 36630 <at> debbugs.gnu.org; Sun, 27 Oct 2019 12:28:20 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=web.de;
 s=dbaedf251592; t=1572193693;
 bh=pqxV/DTGr4V8YDf4swbiSrj5VJ0+V4lecEdV1REM3Mo=;
 h=X-UI-Sender-Class:From:To:Cc:Subject:Date:Reply-To;
 b=XA9q28MTAeq0sKCZaq01KC9mgJeqFa2ax8MmURNKjLizIPTZLdk+Ic6vRY2m5lTf2
 H/6AFo19vwcMg+bqFF0LP02QJTycrLYDWJaX21D40nAfA8mZTH7wRYT962SKK2cgE4
 nBgUzW2ktmpqU4fjsLsanpSiZNfGtbo8WIV1Vlh8=
X-UI-Sender-Class: c548c8c5-30a9-4db5-a2e7-cb6cb037b8f9
Received: from fluss.speedport.ip ([80.136.28.233]) by smtp.web.de (mrweb103
 [213.165.67.124]) with ESMTPSA (Nemesis) id 0Lx7Ir-1hvk7p3dIP-016eGV; Sun, 27
 Oct 2019 17:28:12 +0100
From: Arne Babenhauserheide <arne_bab@HIDDEN>
To: 36630 <at> debbugs.gnu.org
Subject: [PATCH] use two threads to build man-pages and secure output msgs
 with mutex
Date: Sun, 27 Oct 2019 17:27:57 +0100
Message-Id: <20191027162757.32295-1-arne_bab@HIDDEN>
X-Mailer: git-send-email 2.23.0
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable
X-Provags-ID: V03:K1:MDzkB42nxTDLwCiU+4hCseRPTZcPet9JXARYgejWT4ldFEZyBXk
 LwqPKk0GIMusRdgVmx2kovg6bc5EgfXAbETwi8pEr5cjfIX/Cpm3RlsbhFjAn0hlh93hnAe
 1O4DAX+mCFnuoqzX71/H38t0PfCDPqnOY7BDUKnU2dIpfQGymOuWG/NJ7Cei84WmW0JSJ6O
 qNYYaU+BN6OsterZS33Iw==
X-Spam-Flag: NO
X-UI-Out-Filterresults: notjunk:1;V03:K0:w49cmtcv31Y=:PgFOMq8i7BC3ko1KWCdD/9
 Mx0WS7igsNEm7EaJ/GHu5JcocJ9/AWf05sASki8u/J/hf2Zvmb2g7LokNBagKjWTA7q86S7if
 rkqMqC/tx5sQ6Hm+gMf+rZt1CAPQxlpEvQvQXSs6kJrOsCjIOAGnj8028CZspJ/DyxuXJ8W9w
 WspVnEewI8bvUWcxTa+jsCt6/5BC2uk+fUmgIbv6Zc+B/gja0ClOkn0Dku+tgw395d83nsO4v
 ChgVlQ65auK8e71DaivV1paSX/nj3f0QjfocOEUxnwF8hvKAYOVtZMzk9AVXxDzS2ZEL+aQrz
 KHvvN5Vs+b7uTltOjf2RgM4CfSIY5V3CHbqWNYH2NrKFqH8PjRiE2qRdsnd1wdUR7bnmNRN9S
 G5o5nD9QNCvIF2aaEw44scDyFYQpGNnzxS0JCJGvuu+0MH0V+O5XBhnFk9Ip8aZGwmvSG0MDR
 hBMg6fLpqWQZdW10vo3WM9EdoK1skH3ywZbpVh/gMzMuPoMGFF8A0v9FjdJ9LvRvx739KTWIX
 vIdD2cuSNNU9cxXVADHyNz39OvNBqbdJrYluN1Ve2pmLrD6fpybRI2HFRMaoTp+tJ7LrCItrW
 sHpU0k9U+ndVPdyUZmf+XZEvMEu/MLDskU0XA3PQs4mZhbBWuZk+W8PppCDtWrhGof8JMzepH
 Z/ExloHLD00uF5N9UTuUGZiH3nmurAwYFb+icVX3szAIcF2SX3Wb7TbQiYumVVEKpR5P01YSj
 ypG86hdOejvpMPOe1ukLxfYYj2zHw0UuylE5eqc5io482CmYXq496CQrYphaHeeZD+AIzYi24
 oLVtjATgvRUPSmRjUDOg1pI/Q4pxV/uHuF3cicMe/WVElMSOg765g/8PWECTfYgJkZrCPjJsS
 lwXosPUzYmIHSJ1y2+5p9ugfl8bxStaGi3dcflpR1GWNY0jYHRJZlAzBD16DAADqMzC2wWXNW
 X7Uv+y5bXz/GvAxYh3UxrXPSIlZVt5Zx7JeYaDjvZBTdtMEQnme6igRqmeu3Nnu1SJkk1M+Kg
 Mbz0ckNkcIBzwmV8oa49w/pVyvA38hi0nHk60XVVdDasUmOJd/Z2u8YjycG9yao2I0JF9Drwg
 uD0zgte098oiTrRRKLjLCNyjlvfnLqkn1/GvFr2g02LhJP9tnhCnZfkbmwRxIFN7OG4zwM905
 n0skzG743s979YsRApNzGrD7J+y3iq2xiHBl5mhLP8HL0CcUPhX/xMrMhpZL8p2b27R9SXKTt
 edIzZi13beYdyaWHcHqsEI4+g9TAxMJTl7j22cw==
X-Spam-Score: -0.7 (/)
X-Debbugs-Envelope-To: 36630
Cc: Arne Babenhauserheide <arne_bab@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Reply-To: arne_bab@HIDDEN
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

=2D--
 guix/profiles.scm | 15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)

diff --git a/guix/profiles.scm b/guix/profiles.scm
index a0ae9a4c7e..9d2133df2d 100644
=2D-- a/guix/profiles.scm
+++ b/guix/profiles.scm
@@ -1315,11 +1315,18 @@ the entries in MANIFEST."
                          (ice-9 threads)
                          (srfi srfi-1)
                          (srfi srfi-19))
+            ;; ensure conflict free user-output
+            (define status-output-mutex (make-mutex))
+
+            (define (status-message msg)
+              (lock-mutex status-output-mutex)
+              (display msg)
+              (unlock-mutex status-output-mutex))

             (define (compute-entry directory count total)
-                              (format #t "\r[~3d/~3d] building list of \
+                              (status-message (format #f "\r[~3d/~3d] bui=
lding list of \
 man-db entries..."
-                                      count total)
+                                      count total))
                               (force-output)
                               (let ((man (string-append directory
                                                         "/share/man")))
@@ -1332,7 +1339,9 @@ man-db entries..."
               ;; decompression), so report progress as we traverse INPUTS=
.
               (let* ((inputs '#$(manifest-inputs manifest))
                      (total  (length inputs)))
-                (apply append (par-map compute-entry
+                ;; experimenting shows that two threads suffice to remove=
 most
+                ;; waiting
+                (apply append (n-par-map 2 compute-entry
                                        inputs
                                        (iota total 1)
                                        (make-list total total)))))
=2D-
2.23.0





Information forwarded to guix-patches@HIDDEN:
bug#36630; Package guix-patches. Full text available.

Message received at 36630 <at> debbugs.gnu.org:


Received: (at 36630) by debbugs.gnu.org; 23 Oct 2019 20:02:06 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Wed Oct 23 16:02:06 2019
Received: from localhost ([127.0.0.1]:35059 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1iNMpY-0005BA-Ki
	for submit <at> debbugs.gnu.org; Wed, 23 Oct 2019 16:02:06 -0400
Received: from mout.web.de ([212.227.15.3]:57587)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <arne_bab@HIDDEN>) id 1iNMpV-0005Ae-0A
 for 36630 <at> debbugs.gnu.org; Wed, 23 Oct 2019 16:02:01 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=web.de;
 s=dbaedf251592; t=1571860913;
 bh=AwhukiMjGjBrM4o5hamKsMz0psS6f+r++4m/7y5Udp8=;
 h=X-UI-Sender-Class:References:From:To:Cc:Subject:In-reply-to:Date;
 b=YbzrTehSCG/hQ+IElCmBusw88k9jFrzz6EnHpanukXT3OogVqeL9dtIYvMmpC4Jjs
 vf+RHKPzvKP72eChF+H1r3vydeWe3NgMJZGedDKkwL08auEtNnTeLyRiFjwUWaLyfF
 DosQnZ+UsPbAW5akCOv1RzuFUPVirRZZua7HFNlI=
X-UI-Sender-Class: c548c8c5-30a9-4db5-a2e7-cb6cb037b8f9
Received: from fluss ([80.136.28.233]) by smtp.web.de (mrweb003
 [213.165.67.108]) with ESMTPSA (Nemesis) id 0Lfzwp-1hdoap3C6l-00pZ42; Wed, 23
 Oct 2019 22:01:52 +0200
References: <20190712214245.23857-1-arne_bab@HIDDEN> <878sszl1jo.fsf@HIDDEN>
 <877e8ig9gh.fsf@HIDDEN> <87zhldel6f.fsf@HIDDEN> <878ssw8ger.fsf@HIDDEN>
 <875znzhg91.fsf@HIDDEN> <87k1cfpq0k.fsf@HIDDEN> <87zhlbe9qt.fsf@HIDDEN>
 <87d0i7p0ug.fsf@HIDDEN>
User-agent: mu4e 1.2.0; emacs 26.1
From: Arne Babenhauserheide <arne_bab@HIDDEN>
To: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Subject: Re: [bug#36630] [PATCH] guix: parallelize building the manual-database
In-reply-to: <87d0i7p0ug.fsf@HIDDEN>
Date: Wed, 23 Oct 2019 22:01:46 +0200
Message-ID: <87d0entemt.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="=-=-=";
 micalg=pgp-sha256; protocol="application/pgp-signature"
X-Provags-ID: V03:K1:HDoXsw0bY6XmcR7hDxUrtcNImI6OnXi/pEl8xjEahWDN3wRkYSg
 eT2++0xIc2MALltMloCmeXLXiBIz7AgVl9y9BqlobDfnOpb57t/+/cQUf4HyoIlpM6slgzK
 uCiuZJdOR73t7OyrV2lp8HBiyodWh53tD7bebwCtT58GKwA4bxU09c5IKRtqeQsBXGVvfpV
 p/tqBUX6BUc14K+dT6FrQ==
X-Spam-Flag: NO
X-UI-Out-Filterresults: notjunk:1;V03:K0:ONibKfXLCrA=:lDnJ2pD0k11OY+1v9ij7wK
 fqtEkGGLoTKd9GMuwliZZ7pU2+o4FaSbu98upMIIEk1FB4EZ7ZVBZZV9f4M8NOcFIAxvYokz6
 oo+Sxr/HVaSSj5DCvj7yXBLXAKEEeHi0aHxpRbzPJba0d9ZzZSD9trG9knYik6aNb78YwxnuY
 zugaRhcKPhOBwwx2u76HVqlZGMqBJCB63miaj9JdUQKm30hatkyzilZyJ9cTPIICx1Om3EbRc
 s+TL1A6EcabO+LGnDxpeqoloAHdk8ChKsnD20hRSnFihPVFKSYjt3OUYJa+GPnWLJKfvz8yWa
 KyJDnbFMd86WQJ2D/TcXMvDVGBVVeHtLzbHNEIg4eFudPVHMwWR67kcRWyy4hqucosWcta/7Z
 Z+4OELCB5CZRSC1GgELVkCW34FWrf65S1ZctMG3fhuCDJfKRO3z4W/BwqdlyH9cBuh9PvQoFd
 8kv+d64ipdL9AELk0zo5sljcS7daGXKGMUy6Zcx7jM0mHPxvJUINJy0QZIELBKPZMx5RiQBpa
 e5izV14VK/i9hrpkz9wdcbC8D0v/jlernfy6JawaK6I9Pnn0XtR//6qoL2E1HUXz7A/mJyBDy
 7zzqCJ3p3Nk0qPzLMPb4aiW+11Zz3PGfuMTO4NUXBz6W+AkyMgrbZkFWfeXv1e580xndh21b7
 XyMLdVdREPeMWzcly+8Qqnmg2fwXZ15oP03m3v2kIlDSKURXnB0PjONhdHxliTMeuLgY2a5MC
 3jnvfa8jLN/E1UmsdlGFOo7f6/mk6ouf1iSObua6KUxsLlQphc2XSJHwJuaYaL7a7FI7T5RsB
 102oy2RfK6eH9OpT6HR6fUjQ8Y3VhXio3AZAdblXH9fdu4Cd9hWkrmt+o1wZctB0L2W80oSAW
 ml4zPdmGAPft29BdqTVt7LO/3GEvRGfDEuxGyrNODKAs0uRn061tKeFedwjzYJ2vRv0qj8eJP
 Ncbmy04x2b60dbji9Ib2H2L3lZ38CXkpD9J0I2ZpmgiEL6zYVxGXGsQKGzjwTKcsSBKzf7fiz
 Ij364ag2AefRz7770vCsEUiypcGxAHJ7lG6nhXkdBJk/UWwanxMhMDrvOk498WnimjJwBLVsf
 A2q37lbwvf2I0d8/hcUg4SaD32NADZ8Cv4/SY0XGIFpZyhBN9MbYUy5W/CiEHiHghi//efAVV
 fItGdsYcp8+lUIa97sAqGpS++t3ByLB9FaBKmEHYmXcqjbyCMr811WQBva9f8dhJaD/srIGgd
 O6NuRyC6CeEjubY5bsb04bgCT0lLFAroJgCZJuQ==
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 36630
Cc: 36630 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

--=-=-=
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


Arne Babenhauserheide <arne_bab@HIDDEN> writes:
>> Right, sorry for being unclear: you need to run
>>
>>   ./pre-inst-env guix environment -n =E2=80=A6
>>
>> That will run you modified code and thus create a manual-database.drv
>> that uses your code; it=E2=80=99s this manual-database.drv that you shou=
ld pass
>> to =E2=80=98guix build=E2=80=99.
>
> That works now =E2=80=94 thank you!
=E2=80=A6
> With the change:
> 185552 entries processed in 108.2 s
> Before the change:
> 185552 entries processed in 220.1 s
>
> The exact commands I=E2=80=99m running:
> cd Dokumente/Guix/guix
> git checkout master # with the change
> ./pre-inst-env guix environment --ad-hoc jupyter python-ipython python-ip=
ykernel -n
> time guix build /gnu/...-manual-database.drv
> git checkout 97bf46e64c11c64a968fdb833983ede6bdafbc00
> ./pre-inst-env guix environment --ad-hoc jupyter python-ipython python-ip=
ykernel -n
> time guix build /gnu/...-manual-database.drv

I now reduced the thread count to exactly 2 (to avoid running into
resource troubles; I hope that two should be safe) and added a mutex for
status messages to ensure that writes don=E2=80=99t overlap.

The patch should arrive shortly.

Best wishes,
Arne
=2D-
Unpolitisch sein
hei=C3=9Ft politisch sein
ohne es zu merken

--=-=-=
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEE801qEjXQSQPNItXAE++NRSQDw+sFAl2wsa0ACgkQE++NRSQD
w+sN7g//fy4HDFaVlUVHxACX9sIYsFi+R6xR654u+Q8UujdtryKizroo4vGCY4Fr
QmWPXonFn/npwUMni1IFPgQ7JsWRvyXghZfs4hHG45y3tc8AGIVfs3Z5yC4WqVi1
07cIvTHnmoL0/XQW4U38BGwWfY1MH1eypVTPp3mvwJZgGvSB2yjmRCVKVxxbQwX+
t1KzpdWq/guBBHoMl4qkNv5BBWHmoOO/1KQY/92BwMNorwtL95MJFqjqcwCfOdpo
6HLXg6UgBdAyoxoxmUoGhJqAtJhrkP4sC60syg/vl+Pi3pFL3wvPOP3gzRnIAwU4
31ZurLSsemlVESsI2w5u0zuhakup+k6emHwgXIkbTE4WYMySMgw1I57G8qnPlLOW
jsiajBlOx4F0MQnpsWRovhaSfG7M1QFfY8c+FR1MR/XkiKTSKYrDXK0PyX30Orm8
Xy2RWJ/dFaZqp3kSzvWxrIX6GOQdU/wAlzI/1FYQTav0qEInXYNOvIfMeg55RKUV
wdneE6d7/CGpZaRGV3SdYZ6roQdCHnldA1xaaUsmC/Xf46yVhUzhhktkkkNA4K6C
uOXHgebZUUuslMTEl3DNPXy3EdYmVvtWJoiES64Rq/PY7R1G/mCxNJ1wN5TLsDOu
AjOxsLENBJKiaqRDuGM+1arSGIN/HspFvRKbHyLUBT9a3KItfzmIswQBAQgAHRYh
BN0ovebZh1yrzkqLHdzPDbMLwQVIBQJdsLGtAAoJENzPDbMLwQVIGKMEAJlgj9AX
iJ+O8cUqr60V9ZneVgaNqMmpLc2BuCPFHcAuHmym0tPDvWWp7dcdGsQWbXgWVXKt
01iilzh/dLe1er6TeKWbevfcAZx2ZB5gCn9SSvtP0CwA6HJsDnIht5QzHV4YeuI/
0tRDgXdtxGjgrgUCdLZh474pd7Ri66bvVL7a
=IW/g
-----END PGP SIGNATURE-----
--=-=-=--




Information forwarded to guix-patches@HIDDEN:
bug#36630; Package guix-patches. Full text available.

Message received at 36630 <at> debbugs.gnu.org:


Received: (at 36630) by debbugs.gnu.org; 23 Oct 2019 19:59:42 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Wed Oct 23 15:59:42 2019
Received: from localhost ([127.0.0.1]:35055 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1iNMnG-00055w-2u
	for submit <at> debbugs.gnu.org; Wed, 23 Oct 2019 15:59:42 -0400
Received: from mout.web.de ([212.227.15.14]:56111)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <arne_bab@HIDDEN>) id 1iNMnE-00055j-9N
 for 36630 <at> debbugs.gnu.org; Wed, 23 Oct 2019 15:59:41 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=web.de;
 s=dbaedf251592; t=1571860769;
 bh=pqxV/DTGr4V8YDf4swbiSrj5VJ0+V4lecEdV1REM3Mo=;
 h=X-UI-Sender-Class:From:To:Cc:Subject:Date:Reply-To;
 b=jzKJIbb5R3S49V1f/Smt6hWyt69z1OjdN3m8gMoz/OWmrVVE02bRKtM1n4fEwDRid
 ZWkV89fe5hPGqCy4S+dPr5jnzGjD2mDUXWX825Pkllz/XR4jrcdxFsq8yIT6aLSIYx
 DU8k6N/ePUMs+978xeh0/S20dFu7SsRyXsfNzEMc=
X-UI-Sender-Class: c548c8c5-30a9-4db5-a2e7-cb6cb037b8f9
Received: from fluss.speedport.ip ([80.136.28.233]) by smtp.web.de (mrweb004
 [213.165.67.108]) with ESMTPSA (Nemesis) id 0Lpyh5-1hlXs62hvj-00fhPa; Wed, 23
 Oct 2019 21:59:29 +0200
From: Arne Babenhauserheide <arne_bab@HIDDEN>
To: 36630 <at> debbugs.gnu.org
Subject: [PATCH] use two threads to build man-pages and secure output msgs
 with mutex
Date: Wed, 23 Oct 2019 21:59:18 +0200
Message-Id: <20191023195918.26514-1-arne_bab@HIDDEN>
X-Mailer: git-send-email 2.23.0
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable
X-Provags-ID: V03:K1:QVHHxoGzswf3MQ+89avBnUIW9/QYe8oHGjoaYSHcVY2yqjiEjkz
 io+0RmEoIGJa/Is3pKfMTq5vFu78sl9No4hR+sW1slm3TsmQhuFQxRN5hyuBO57e4JaEIRv
 rSlw/qrE0B0OAmiHdj5Or+eNto95reA9OyYuy5f7bVAO45r5xZsE8iM0MeMswE3hkVKRJsx
 u5LtkRaWXLorUpmSDIeBQ==
X-Spam-Flag: NO
X-UI-Out-Filterresults: notjunk:1;V03:K0:7htekj3tE2U=:hgN19edpyHpr47v/Q+k+Gn
 i4MuZ1OLKvt+s6zYia7uQn+U6PV9PQM1cKzR4yXAc3NuOMAb8PxVn7q+kZEnYX7qXfTntKWPk
 3eoS7xu0T2YlWxqg3l1cEux73QAs4hWRm9HBRhRRtXg3zCEYRK+G0x1RHO6EzyovdAGw435Aq
 nrs/ZGvQKkROxGxJ4cN17ZRhwCnOOvE01M70gJDQty3kY2cFjmkl3rGRPE4N1venTnDWp+Xk2
 W/ekKwM8+qgv4dc05ky7GXFkGAitAuGARaA6N8QbEEi5J6FeQFB08o9cx40wt3qxUDfI8op3w
 6ru94rUC7NCBA41V9gw8+LpIwd7CVV124oIGZUlQtDJQoF1H5UlvR/pkjd4EjKzfGe0lpKrX3
 egAFYhCdc6mkP+GegO8osk69lbPTsFUbT18DuDRwjaoKvC/9Ho2pEv4Vl817A1mptN697ONyV
 g1Dey90SG+3tMafV5KNVMRMZpl+Rs0gxUFlRsXRpV4JY09AbAADtn0UUdGinSsUHCXwin9K0F
 +6R5ZcIWkMvhntBTy2mrKfcuURIq1J6q/PTUMmKxKjDybtVz+SPl2kQ0O/MuBZ80YQVDbqJIl
 Rssue75xDg9MkGtOeoBhULQClST+pAMHxNHCT3yr0aQlC1gMuWf9oAZXYHeCLaB2GkIxpaJij
 fo9kGhKFCSsckbMxY/4oP7pU5BFWVJfPIMaQJ8C7aY2JQAEqlUMuHq0k2zCJMhYW3Z5Yk5JGZ
 PFtgQcCQ/2zV4t32W4s2KL90zj7esCqowi2dgxOxYIjywGJ3yfjMZrpxHXlWL+jsgNHnd4Ql+
 uGOx9KbYikzUDYx7q5riCfuP+Aaxtxbcm4kiBMjV2R2WszBNjP4KrU6kCiQs4zmOMQYdbgI2Z
 aQ8LU1ja4dF3hQnFCTOdsvxjbBiLLAqAlQLg3246jYGzwdfVDwm49NOlZT97UIUXXb97VwPzF
 x5a/t8JMsCyvmVja9vFS7tZXkEkmAiaUThGBSZ9ZMrhCGeF5LpP3lo/3axIldhUpFaNskIyv3
 zYIkekTE6jtpZarHxbHtonBxFBqj8fzHwsyQD73DwQZpl+Qo1/nkgzrL+OJTwQqxGqkQs53vZ
 XlbjEkW0AOTpYTtv4jlgvr9PAK4ZFAL+evLaHqfunPbqisNfm2s1AUa5L7DZY4EKHlBRyd6wa
 mM3pT1DA2oqVSiZDjeD7i6CE/GOtdTyawHf99+A1V4KZa01TVD/wh9xdktwvBSMLXZ6csja38
 9JSzOYv+WxK/Be6qTb6rDq/WS65OL4Xb4O0apBg==
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 36630
Cc: Arne Babenhauserheide <arne_bab@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Reply-To: arne_bab@HIDDEN
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

=2D--
 guix/profiles.scm | 15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)

diff --git a/guix/profiles.scm b/guix/profiles.scm
index a0ae9a4c7e..9d2133df2d 100644
=2D-- a/guix/profiles.scm
+++ b/guix/profiles.scm
@@ -1315,11 +1315,18 @@ the entries in MANIFEST."
                          (ice-9 threads)
                          (srfi srfi-1)
                          (srfi srfi-19))
+            ;; ensure conflict free user-output
+            (define status-output-mutex (make-mutex))
+
+            (define (status-message msg)
+              (lock-mutex status-output-mutex)
+              (display msg)
+              (unlock-mutex status-output-mutex))

             (define (compute-entry directory count total)
-                              (format #t "\r[~3d/~3d] building list of \
+                              (status-message (format #f "\r[~3d/~3d] bui=
lding list of \
 man-db entries..."
-                                      count total)
+                                      count total))
                               (force-output)
                               (let ((man (string-append directory
                                                         "/share/man")))
@@ -1332,7 +1339,9 @@ man-db entries..."
               ;; decompression), so report progress as we traverse INPUTS=
.
               (let* ((inputs '#$(manifest-inputs manifest))
                      (total  (length inputs)))
-                (apply append (par-map compute-entry
+                ;; experimenting shows that two threads suffice to remove=
 most
+                ;; waiting
+                (apply append (n-par-map 2 compute-entry
                                        inputs
                                        (iota total 1)
                                        (make-list total total)))))
=2D-
2.23.0





Information forwarded to guix-patches@HIDDEN:
bug#36630; Package guix-patches. Full text available.

Message received at 36630 <at> debbugs.gnu.org:


Received: (at 36630) by debbugs.gnu.org; 18 Jul 2019 22:49:22 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Jul 18 18:49:22 2019
Received: from localhost ([127.0.0.1]:54808 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1hoFDG-0002so-FC
	for submit <at> debbugs.gnu.org; Thu, 18 Jul 2019 18:49:22 -0400
Received: from mout.web.de ([212.227.15.4]:44253)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <arne_bab@HIDDEN>) id 1hoFDE-0002sZ-U7
 for 36630 <at> debbugs.gnu.org; Thu, 18 Jul 2019 18:49:21 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=web.de;
 s=dbaedf251592; t=1563490150;
 bh=8rdjocyH0SL+ZjGXhU68dGzMDUtUNxluyicHnNbUpuk=;
 h=X-UI-Sender-Class:References:From:To:Cc:Subject:In-reply-to:Date;
 b=kRRg6NkL3JqtsK0jb38vIyzpG4UnREphY1UEXOXOrPXTL6xjnVJDqooA5XaGuIIju
 M+D0S+d/D+TLjAXV/e88ONUJW4AG3Lw8pS+ddQSW5N/Nz43f4RIaf1FTO131BZCTuS
 OpOgFkb9qby35oFRHAtx9TTCNvvJ8rZyb6x9wCn8=
X-UI-Sender-Class: c548c8c5-30a9-4db5-a2e7-cb6cb037b8f9
Received: from fluss ([84.149.83.171]) by smtp.web.de (mrweb001
 [213.165.67.108]) with ESMTPSA (Nemesis) id 0MNc1S-1hmXvw1zPW-007B7z; Thu, 18
 Jul 2019 22:03:18 +0200
References: <20190712214245.23857-1-arne_bab@HIDDEN> <878sszl1jo.fsf@HIDDEN>
 <877e8ig9gh.fsf@HIDDEN> <87zhldel6f.fsf@HIDDEN> <878ssw8ger.fsf@HIDDEN>
 <875znzhg91.fsf@HIDDEN> <87k1cfpq0k.fsf@HIDDEN> <87zhlbe9qt.fsf@HIDDEN>
User-agent: mu4e 1.2.0; emacs 26.1
From: Arne Babenhauserheide <arne_bab@HIDDEN>
To: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Subject: Re: [bug#36630] [PATCH] guix: parallelize building the manual-database
In-reply-to: <87zhlbe9qt.fsf@HIDDEN>
Date: Thu, 18 Jul 2019 22:03:03 +0200
Message-ID: <87d0i7p0ug.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="=-=-=";
 micalg=pgp-sha256; protocol="application/pgp-signature"
X-Provags-ID: V03:K1:hVHqX+XcTIQtaBVXm++v/uzGbo4/4JtVGhXa3fyVcrUOkQEugxt
 2eXM8OvbEM2Wrx+XtvBQ7EVacW91T9PF2RqlW4Ts7AeoMRoVMV2d/rhelUJgWPaPCZt0Lo1
 upWYhf1a6srfKcPGsccZXDdHo8dqLGIjzuc0HNtoAuqiA9KbjSRsF9jL6/7NvgW40vBW8YF
 LSI6PcRbE5ksy8uXY3uFw==
X-Spam-Flag: NO
X-UI-Out-Filterresults: notjunk:1;V03:K0:/EO14+QPL1c=:nwONDV7YFty9dHgRZ1gcUP
 piIC6zDIKE+q+IyVTlJosDuLy5zhLbXuOHFMnbIS9tUHwLTp+kndg3pwWN5/BXej/a91dWA02
 pY4IP8UO3cTsps6p08TkTkXaLs9uCk/dJ4G5aWc/+1YGLis2LLvcHVoHaR+dvQqr8sPFNpvJZ
 BLA985aZticvZcMvwLLfWuB8TDUTgBAKr3uhB39NsFL9EU7P75FmhvYZXhFgKKFAKcuGAByzE
 mSrs8OpAAUBWQPmwCNMl01Yy+iI5qgK32Mb7PbSRZ4iMUc2/J5zOJdDICV2SpbPRTtLpZljiJ
 EGI/PxPiYjjK1J1tILTsyEY/76SDWDGIu5aCpDj09gsuLOmHOyhuqEMorN5piOFGfli1cQ0mn
 LFRfVPTX6mtiJmpd4RN46T3b+HF2NT6/h+B/Q9WUutr+JHFnHcUBytb88nkCiOVvzTtTTmJYf
 sX1BDFWzdi/jL4Ln7f6I4RxEWzebq/AI/Uj8Oi4TzNvauw1vYEPPpf7Wv4p7HI4VSQWtCYItl
 QsVkYdGmTBCdHaZtC0eF6U2kHgDsLcukM5WxcRKtDWWVo+KTT9ReIbtVVtSFcCO2NKDIMOqd+
 V2FfoOTj4OC7BFgmPBaiaks9PVO7lIiOkuZ4x00RxtMwoW/JTLBE4Ma5cmbftK0yOJWo7nCdz
 dCAQ=
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 36630
Cc: 36630 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

--=-=-=
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


Ludovic Court=C3=A8s <ludo@HIDDEN> writes:

> Arne Babenhauserheide <arne_bab@HIDDEN> skribis:
>
>> Ludovic Court=C3=A8s <ludo@HIDDEN> writes:
>>
>>> Did you try the =E2=80=98guix environment -n=E2=80=99 command above?  D=
oesn=E2=80=99t it show
>>> the manual-database.drv?
>>
>> It does show the manual database, but then running guix build
>> /gnu/....drv --check does not run my changed code.
>>
>> I=E2=80=99m doing
>>
>> time ./pre-inst-env guix build /gnu/store/jnkxwwxk71n07fs6naa11fxmg3vpnn=
b3-manual-database.drv --check
>>
>> But it runs the installed guix, not the local changes to profile.scm.
>
> Right, sorry for being unclear: you need to run
>
>   ./pre-inst-env guix environment -n =E2=80=A6
>
> That will run you modified code and thus create a manual-database.drv
> that uses your code; it=E2=80=99s this manual-database.drv that you shoul=
d pass
> to =E2=80=98guix build=E2=80=99.

That works now =E2=80=94 thank you!

With the change:
185552 entries processed in 108.2 s
Before the change:
185552 entries processed in 220.1 s

The exact commands I=E2=80=99m running:
cd Dokumente/Guix/guix
git checkout master # with the change
./pre-inst-env guix environment --ad-hoc jupyter python-ipython python-ipyk=
ernel -n
time guix build /gnu/...-manual-database.drv
git checkout 97bf46e64c11c64a968fdb833983ede6bdafbc00
./pre-inst-env guix environment --ad-hoc jupyter python-ipython python-ipyk=
ernel -n
time guix build /gnu/...-manual-database.drv

So I also see roughly factor 2 speedup, which means a limit to 4 threads
should work.

(what I saw is that I only see the =E2=80=A6manual-database.drv before I bu=
ild
it, after it=E2=80=99s built, I no longer see it in the environment output)

How do I get the defined limit of cores and threads?

Best wishes,
Arne
=2D-=20
Unpolitisch sein
hei=C3=9Ft politisch sein
ohne es zu merken

--=-=-=
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEE801qEjXQSQPNItXAE++NRSQDw+sFAl0w0HkACgkQE++NRSQD
w+sPHhAA0Mr9BAzh9hjHgDq+Ji5KQFRXBCEvHyyQV2TYJBs0TZ04oXmNiHZHSWQp
coSGnmnPvtx1aGkXI7edHuuy/rDwKBcUpJtWZLmFiZ/89baFL+hJnRteZSJILj9P
DFGFFODAg/2MRyDZQhpMcEx+ea9hZKUJ068aGPW66DkpN35hJoSu/w0qetAqD8Pm
+o1WyB0z5Y/5vuUPgGcN63EVh+BMMAQABGJLlYBJjEao2n5EsrAh6q+c4yMn5JjO
PLybFvT/MLkWfUylNpvlIbRffJaphgwbXhLlZBbzPivyZ/ucZTek3J1439JW65oN
u4IziODcVZwsZfjluR0koRNvJx73CAM7eflfQp+A/lwNRLDQf96Qqc6VYJO/2HdJ
YhVNX3RqFCzgVAdo6fosjtCkPQUJAkfiq4gmNIPAR6Q9ZMn7jw/hD6GDGRJMlFZ8
aqN9QlWVdkLS4Bn9TJ4mZpNcA7F+YF9xusFhk3yC/ViYe5O9iaCtACml4wGVCRzL
q1He4D/ZVjXNusfv734vGjxW89gfRuF9/J+CClEwCwaOA8N0p0lK+MFqRInzjvDI
aA/i/Bf20RVtjl8tZA6/lZi/bwubwJGU21D2mEcNrdFZFCV73Zl11vVvEc0bsaLH
2LAp/39saKigMZO3jVGrQ9vnVjCAjIv2fnUB97W51463FR3fKUyIswQBAQgAHRYh
BN0ovebZh1yrzkqLHdzPDbMLwQVIBQJdMNB5AAoJENzPDbMLwQVIcBMD/i6C+ujg
+M7Y7GC2YXgXEHmnZQGRMQ6MJGdF6nqy9jTQFXx5ZyCALtgGSi0JKy2dExF28a3n
xzYs9MNftZ03sMxEKl1sOh9GNbLFEPAZNCkTBX2yxGBM7AnoqHjUn7YHLtdtPfmQ
AFu5xtjKSUTYWsPlfaX125ExhcPnKUPLu3uS
=UQ9d
-----END PGP SIGNATURE-----
--=-=-=--




Information forwarded to guix-patches@HIDDEN:
bug#36630; Package guix-patches. Full text available.

Message received at 36630 <at> debbugs.gnu.org:


Received: (at 36630) by debbugs.gnu.org; 18 Jul 2019 13:46:30 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Jul 18 09:46:30 2019
Received: from localhost ([127.0.0.1]:53638 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1ho6ju-000113-Dk
	for submit <at> debbugs.gnu.org; Thu, 18 Jul 2019 09:46:30 -0400
Received: from eggs.gnu.org ([209.51.188.92]:55185)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1ho6jr-00010p-4V
 for 36630 <at> debbugs.gnu.org; Thu, 18 Jul 2019 09:46:27 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:55953)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <ludo@HIDDEN>)
 id 1ho6jl-0002KU-JE; Thu, 18 Jul 2019 09:46:21 -0400
Received: from [2001:660:6102:320:e120:2c8f:8909:cdfe] (port=42776 helo=ribbon)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <ludo@HIDDEN>)
 id 1ho6jl-0006TE-2r; Thu, 18 Jul 2019 09:46:21 -0400
From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>
To: Arne Babenhauserheide <arne_bab@HIDDEN>
Subject: Re: [bug#36630] [PATCH] guix: parallelize building the manual-database
References: <20190712214245.23857-1-arne_bab@HIDDEN> <878sszl1jo.fsf@HIDDEN>
 <877e8ig9gh.fsf@HIDDEN> <87zhldel6f.fsf@HIDDEN>
 <878ssw8ger.fsf@HIDDEN> <875znzhg91.fsf@HIDDEN>
 <87k1cfpq0k.fsf@HIDDEN>
X-URL: http://www.fdn.fr/~lcourtes/
X-Revolutionary-Date: 30 Messidor an 227 de la =?utf-8?Q?R=C3=A9volution?=
X-PGP-Key-ID: 0x090B11993D9AEBB5
X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc
X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4  0CFB 090B 1199 3D9A EBB5
X-OS: x86_64-pc-linux-gnu
Date: Thu, 18 Jul 2019 15:46:18 +0200
In-Reply-To: <87k1cfpq0k.fsf@HIDDEN> (Arne Babenhauserheide's message of "Thu, 
 18 Jul 2019 12:59:23 +0200")
Message-ID: <87zhlbe9qt.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 36630
Cc: 36630 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Arne Babenhauserheide <arne_bab@HIDDEN> skribis:

> Ludovic Court=C3=A8s <ludo@HIDDEN> writes:
>
>> Did you try the =E2=80=98guix environment -n=E2=80=99 command above?  Do=
esn=E2=80=99t it show
>> the manual-database.drv?
>
> It does show the manual database, but then running guix build
> /gnu/....drv --check does not run my changed code.
>
> I=E2=80=99m doing
>
> time ./pre-inst-env guix build /gnu/store/jnkxwwxk71n07fs6naa11fxmg3vpnnb=
3-manual-database.drv --check
>
> But it runs the installed guix, not the local changes to profile.scm.

Right, sorry for being unclear: you need to run

  ./pre-inst-env guix environment -n =E2=80=A6

That will run you modified code and thus create a manual-database.drv
that uses your code; it=E2=80=99s this manual-database.drv that you should =
pass
to =E2=80=98guix build=E2=80=99.

HTH,
Ludo=E2=80=99.




Information forwarded to guix-patches@HIDDEN:
bug#36630; Package guix-patches. Full text available.

Message received at 36630 <at> debbugs.gnu.org:


Received: (at 36630) by debbugs.gnu.org; 18 Jul 2019 10:59:42 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Jul 18 06:59:42 2019
Received: from localhost ([127.0.0.1]:53489 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1ho48U-00015s-46
	for submit <at> debbugs.gnu.org; Thu, 18 Jul 2019 06:59:42 -0400
Received: from mout.web.de ([217.72.192.78]:38149)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <arne_bab@HIDDEN>) id 1ho48S-00015f-7q
 for 36630 <at> debbugs.gnu.org; Thu, 18 Jul 2019 06:59:40 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=web.de;
 s=dbaedf251592; t=1563447571;
 bh=HcZzKeJnT1IV+bkjllI29jtpYhBxxhVyP3TCEkCeE/g=;
 h=X-UI-Sender-Class:References:From:To:Cc:Subject:In-reply-to:Date;
 b=IcN/L76XUkO9lDMQm0A+CuXHT8nB+VptXyAJEotJRzenWzU0aDRiNOZzeVFVkcLhj
 XIbvv+2ZDkMZjmuqQVPGgGnd16E8mtOMYZFd5J0rX7DwRnR0N0nRQGf3pSrEXK2XuO
 qnYguGNe3Gm5KR+OEM2sbp03aqqAqxQyfxUBpU5Y=
X-UI-Sender-Class: c548c8c5-30a9-4db5-a2e7-cb6cb037b8f9
Received: from fluss ([84.149.83.171]) by smtp.web.de (mrweb103
 [213.165.67.124]) with ESMTPSA (Nemesis) id 0LjJH1-1iRXYL3fbD-00daZu; Thu, 18
 Jul 2019 12:59:31 +0200
References: <20190712214245.23857-1-arne_bab@HIDDEN> <878sszl1jo.fsf@HIDDEN>
 <877e8ig9gh.fsf@HIDDEN> <87zhldel6f.fsf@HIDDEN> <878ssw8ger.fsf@HIDDEN>
 <875znzhg91.fsf@HIDDEN>
User-agent: mu4e 1.2.0; emacs 26.1
From: Arne Babenhauserheide <arne_bab@HIDDEN>
To: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Subject: Re: [bug#36630] [PATCH] guix: parallelize building the manual-database
In-reply-to: <875znzhg91.fsf@HIDDEN>
Date: Thu, 18 Jul 2019 12:59:23 +0200
Message-ID: <87k1cfpq0k.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="=-=-=";
 micalg=pgp-sha256; protocol="application/pgp-signature"
X-Provags-ID: V03:K1:aE3uPQTK6sDvqGJ+OnjsPAO6yJqND8GOL2bTawbtuSwWdhCKJc2
 HBvsdFMW8TXxt1FKD4GPmc+7fikwa6wb9ap+ggfsp4fn7OCisVdxLOFFa4Y0x5mAYLoTmwY
 9T7mBp96sjgmNa04WTIuZDlHFmJjMEC7YRUkWZsrFqjl9Ms/K+8zZNusqcqU+RlXPdhCQLT
 t9z/lp6j6Fk3J4JWHpOSw==
X-Spam-Flag: NO
X-UI-Out-Filterresults: notjunk:1;V03:K0:XNLPMwLmP+I=:nH3gyGvhadNSA3BVmMYXKA
 V2TlV1py/soZJqSR+nbEXYyw5ZRAANH3mReGtf0XxLLVLgb1i+W63j0Lir8X+aIGEablIK/FG
 esTnPKjYYhqK7vPtRVt8PXvRahWk768P5jggVk9176Hj3A+6ENo6Sx1ttWUYNiBlaWKnD/aoN
 VImx4eDjQcQpo5qM9AO2nzaNuFr8DnUiB5OkRqpk4+rqV25c9hN3p8NmaXWKjoJXIn6rhMvfz
 BdLF+ScKUZ73Q1vvcJIdPMPre45vuMsXGwIR/1h4ny5j04seja73MPslqbIWwm/6YouG/eB5H
 9pwugb7NEarSVEFCDioa6VVGYN04CLVRo9oAoT55TH0Vt+Lz396HhOSTjIPi9zA9dXtQZ/iuE
 31sRX0rIk8mphD84hXtnCWzHN4HLyrW5hNBr8J081/eAbLBoyw4htj/jmwW00cRGVWSw6/xGN
 MNQdmCrf+QeskMQUZp6LcAewuwmpNUoW+yOXhyMJ6tnUvIdyOMB9ktiqpKZa8ECcUa4QGBmAC
 0yZeO8lncUryczRpPZ21GltrQ29bVa6+MS2wFUzgBFDzb/fgwCycCyOSYGngX96dQSb7Mub1J
 QsUr+DFdaZaykeF1joc6Su/QvzYjEILuEH/7Y3BkDtZUBEs9ty6KoOZhCgdyP26W9NW4W9tY3
 cxEo=
X-Spam-Score: -0.7 (/)
X-Debbugs-Envelope-To: 36630
Cc: 36630 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

--=-=-=
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


Ludovic Court=C3=A8s <ludo@HIDDEN> writes:

> Did you try the =E2=80=98guix environment -n=E2=80=99 command above?  Doe=
sn=E2=80=99t it show
> the manual-database.drv?

It does show the manual database, but then running guix build
/gnu/....drv --check does not run my changed code.

I=E2=80=99m doing

time ./pre-inst-env guix build /gnu/store/jnkxwwxk71n07fs6naa11fxmg3vpnnb3-=
manual-database.drv --check

But it runs the installed guix, not the local changes to profile.scm.

Best wishes,
Arne
=2D-=20
Unpolitisch sein
hei=C3=9Ft politisch sein
ohne es zu merken

--=-=-=
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEE801qEjXQSQPNItXAE++NRSQDw+sFAl0wUQ4ACgkQE++NRSQD
w+upYhAAueygBlMf1kVpYmi/9zrfTCk2KjSv36H4PxlILes3F0KYg//wyFafMQc+
c/gP3PK2+nj2KNLcTrEF7eUwLwXDhVM7LkXVQSZQSBlaRnTRxxlqwEEkq2YMydeC
aEMOaIV1egUD6Chriwl29hE0JUxT4gNxQoo+EbKenK64bouhQmtLHzkZDgLiZld3
dF+k1zWeKNUP/r9Iege6ZYrFTjS5sRqlYRLo9h410meP/j8w3kTA7X64oBqBbcp3
9Q+PfldIh7i+2m5N0cnoI3QxXYK+Z9kmfGHI+1djPCPpJD2Kvi4CGQZLSg6VCMJw
x132r0BbKFtmNtFFRicCytofx9Y1CKZ3+xwMU9nwxQ73iKU2raiWRSWYtBelbqRE
Mzm272aWPZxPN8Ak1YkpukLYaq9SaxZAfmJ0ZzBD5yS8/vcqUPcNzB0rT/FIQ0V8
U8Apxn2TyjZu1hXjUq8nhwkYyxptX/QDiHiKPIPScBlBnGDcmz6BGZW1jP8rB2A4
tbWpecSY+fyf/JE2DPLAgoRksQvlTimISkIn+FptduYKNxSP1fNqx95cM7T46XML
omPGk8VAaZVrXKs2C5nw4JhZgxpCczDrwwUqzwXAUx5+XidBE18+lhZrsJ/cMNH/
WDAOXHWgGlRppW3uxTxSFMccZupIT51V+VTSzWizojVigcD0K+CIswQBAQgAHRYh
BN0ovebZh1yrzkqLHdzPDbMLwQVIBQJdMFEOAAoJENzPDbMLwQVImLYD/0IO8lOZ
mq65FrAx3IFt9ArzcH3/9vYFVmQGcw9z7BsKQLlLAuFkv93koPnQqEvH+wcRwh8f
SJH3wVaZ4Y+WJBvR46PV8OLQq2SHFvIDROTIu6UWeXsyDHZLuD8eneQPFekYt2K9
cYFA4x8vd0uOkxJ2zJ/OoU6EudvWYIJOGpag
=asMy
-----END PGP SIGNATURE-----
--=-=-=--




Information forwarded to guix-patches@HIDDEN:
bug#36630; Package guix-patches. Full text available.

Message received at 36630 <at> debbugs.gnu.org:


Received: (at 36630) by debbugs.gnu.org; 18 Jul 2019 08:58:25 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Jul 18 04:58:25 2019
Received: from localhost ([127.0.0.1]:53379 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1ho2F7-0004Jz-7L
	for submit <at> debbugs.gnu.org; Thu, 18 Jul 2019 04:58:25 -0400
Received: from eggs.gnu.org ([209.51.188.92]:48104)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1ho2F5-0004Jc-Jd
 for 36630 <at> debbugs.gnu.org; Thu, 18 Jul 2019 04:58:23 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:50522)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <ludo@HIDDEN>)
 id 1ho2EM-0008Dd-Q6; Thu, 18 Jul 2019 04:58:18 -0400
Received: from nat-eduroam-36-gw-01-bso.bordeaux.inria.fr
 ([194.199.1.36]:58410 helo=ribbon)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <ludo@HIDDEN>)
 id 1ho2Cs-0007km-BH; Thu, 18 Jul 2019 04:57:38 -0400
From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>
To: Arne Babenhauserheide <arne_bab@HIDDEN>
Subject: Re: [bug#36630] [PATCH] guix: parallelize building the manual-database
References: <20190712214245.23857-1-arne_bab@HIDDEN> <878sszl1jo.fsf@HIDDEN>
 <877e8ig9gh.fsf@HIDDEN> <87zhldel6f.fsf@HIDDEN>
 <878ssw8ger.fsf@HIDDEN>
X-URL: http://www.fdn.fr/~lcourtes/
X-Revolutionary-Date: 30 Messidor an 227 de la =?utf-8?Q?R=C3=A9volution?=
X-PGP-Key-ID: 0x090B11993D9AEBB5
X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc
X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4  0CFB 090B 1199 3D9A EBB5
X-OS: x86_64-pc-linux-gnu
Date: Thu, 18 Jul 2019 10:55:19 +0200
In-Reply-To: <878ssw8ger.fsf@HIDDEN> (Arne Babenhauserheide's message of "Thu, 
 18 Jul 2019 00:06:36 +0200")
Message-ID: <877e8fhgco.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 36630
Cc: 36630 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Hello,

Arne Babenhauserheide <arne_bab@HIDDEN> skribis:

> Ludovic Court=C3=A8s <ludo@HIDDEN> skribis:
>> Arne Babenhauserheide <arne_bab@HIDDEN> skribis:
> Offtopic: I love reading Esperanto here!
>>> Ludovic Court=C3=A8s <ludo@HIDDEN> writes:
>
>>>>   guix environment --ad-hoc jupyter python-ipython python-ipykernel -n
>
>>> How exactly did you run the derivation?
>
>> If you run the command above, it=E2=80=99ll list
>> /gnu/store/=E2=80=A6-manual-database.drv.  So you can just run:
>>
>>   guix build /gnu/store/=E2=80=A6-manual-database.drv
>>
>> or:
>>
>>   guix build /gnu/store/=E2=80=A6-manual-database.drv --check
>>
>> if it had already been built before.
>
> Somehow I can=E2=80=99t get guix to actually run my changed code with this
> command, so I=E2=80=99m not sure I tested the right thing.

Did you try the =E2=80=98guix environment -n=E2=80=99 command above?  Doesn=
=E2=80=99t it show
the manual-database.drv?

Alternately, you can also do something like:

  guix install -p /tmp/foo jupyter python-ipython python-ipykernel -n

HTH,
Ludo=E2=80=99.




Information forwarded to guix-patches@HIDDEN:
bug#36630; Package guix-patches. Full text available.

Message received at 36630 <at> debbugs.gnu.org:


Received: (at 36630) by debbugs.gnu.org; 18 Jul 2019 08:58:25 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Jul 18 04:58:25 2019
Received: from localhost ([127.0.0.1]:53377 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1ho2F6-0004Jx-Um
	for submit <at> debbugs.gnu.org; Thu, 18 Jul 2019 04:58:25 -0400
Received: from eggs.gnu.org ([209.51.188.92]:48106)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1ho2F5-0004Jd-JW
 for 36630 <at> debbugs.gnu.org; Thu, 18 Jul 2019 04:58:23 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:50521)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <ludo@HIDDEN>)
 id 1ho2EG-0008Bt-U8; Thu, 18 Jul 2019 04:58:18 -0400
Received: from nat-eduroam-36-gw-01-bso.bordeaux.inria.fr
 ([194.199.1.36]:58448 helo=ribbon)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <ludo@HIDDEN>)
 id 1ho2EG-00085K-8i; Thu, 18 Jul 2019 04:57:32 -0400
From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>
To: Arne Babenhauserheide <arne_bab@HIDDEN>
Subject: Re: [bug#36630] [PATCH] guix: parallelize building the manual-database
In-Reply-To: <878ssw8ger.fsf@HIDDEN> (Arne Babenhauserheide's message of "Thu, 
 18 Jul 2019 00:06:36 +0200")
References: <20190712214245.23857-1-arne_bab@HIDDEN> <878sszl1jo.fsf@HIDDEN>
 <877e8ig9gh.fsf@HIDDEN> <87zhldel6f.fsf@HIDDEN>
 <878ssw8ger.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.2 (gnu/linux)
X-URL: http://www.fdn.fr/~lcourtes/
X-Revolutionary-Date: 30 Messidor an 227 de la =?utf-8?Q?R=C3=A9volution?=
X-PGP-Key-ID: 0x090B11993D9AEBB5
X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc
X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4  0CFB 090B 1199 3D9A EBB5
X-OS: x86_64-pc-linux-gnu
Date: Thu, 18 Jul 2019 10:57:30 +0200
Message-ID: <875znzhg91.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 36630
Cc: 36630 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Hello,

Arne Babenhauserheide <arne_bab@HIDDEN> skribis:

> Ludovic Court=C3=A8s <ludo@HIDDEN> skribis:
>> Arne Babenhauserheide <arne_bab@HIDDEN> skribis:
> Offtopic: I love reading Esperanto here!
>>> Ludovic Court=C3=A8s <ludo@HIDDEN> writes:
>
>>>>   guix environment --ad-hoc jupyter python-ipython python-ipykernel -n
>
>>> How exactly did you run the derivation?
>
>> If you run the command above, it=E2=80=99ll list
>> /gnu/store/=E2=80=A6-manual-database.drv.  So you can just run:
>>
>>   guix build /gnu/store/=E2=80=A6-manual-database.drv
>>
>> or:
>>
>>   guix build /gnu/store/=E2=80=A6-manual-database.drv --check
>>
>> if it had already been built before.
>
> Somehow I can=E2=80=99t get guix to actually run my changed code with this
> command, so I=E2=80=99m not sure I tested the right thing.

Did you try the =E2=80=98guix environment -n=E2=80=99 command above?  Doesn=
=E2=80=99t it show
the manual-database.drv?

Alternately, you can also do something like:

  guix install -p /tmp/foo jupyter python-ipython python-ipykernel -n

HTH,
Ludo=E2=80=99.




Information forwarded to guix-patches@HIDDEN:
bug#36630; Package guix-patches. Full text available.

Message received at 36630 <at> debbugs.gnu.org:


Received: (at 36630) by debbugs.gnu.org; 17 Jul 2019 22:06:54 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Wed Jul 17 18:06:54 2019
Received: from localhost ([127.0.0.1]:53120 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1hns4c-000162-0s
	for submit <at> debbugs.gnu.org; Wed, 17 Jul 2019 18:06:54 -0400
Received: from mout.web.de ([212.227.15.4]:51039)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <arne_bab@HIDDEN>) id 1hns4a-00015m-2U
 for 36630 <at> debbugs.gnu.org; Wed, 17 Jul 2019 18:06:52 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=web.de;
 s=dbaedf251592; t=1563401203;
 bh=cDHhpIPNoBGX5FC90euBJCs+zrFX24Gq8Gf+RqRemEY=;
 h=X-UI-Sender-Class:References:From:To:Cc:Subject:In-reply-to:Date;
 b=nxlQ8aUsKLuHtp1e2BnYGSqNbfWO2juxNFPTKRyTjMFj2Ps/+4znKyGG/OnN6pTy4
 lkrd+HvOJGmIecv7ZRblsX3ZGTlPtl6h/jfRQw0bk+Ow6eCatvfB1Y6liORfg0pAtt
 7AxDCFPdMaWbJlvV+EdKR1IdRvGhJFsnjQNss5Dk=
X-UI-Sender-Class: c548c8c5-30a9-4db5-a2e7-cb6cb037b8f9
Received: from fluss ([84.149.83.171]) by smtp.web.de (mrweb004
 [213.165.67.108]) with ESMTPSA (Nemesis) id 0Lgs1W-1iGtsr1Yc2-00oHlL; Thu, 18
 Jul 2019 00:06:43 +0200
References: <20190712214245.23857-1-arne_bab@HIDDEN> <878sszl1jo.fsf@HIDDEN>
 <877e8ig9gh.fsf@HIDDEN> <87zhldel6f.fsf@HIDDEN>
User-agent: mu4e 1.2.0; emacs 26.1
From: Arne Babenhauserheide <arne_bab@HIDDEN>
To: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Subject: Re: [bug#36630] [PATCH] guix: parallelize building the manual-database
In-reply-to: <87zhldel6f.fsf@HIDDEN>
Date: Thu, 18 Jul 2019 00:06:36 +0200
Message-ID: <878ssw8ger.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="=-=-=";
 micalg=pgp-sha256; protocol="application/pgp-signature"
X-Provags-ID: V03:K1:FQNI/0k6ZfN9fqnPlinpQi5oU2M1Gt6S/a132fgqXpbDfR/DvXe
 KdhQtY11+eYC11M0SnaBZPmc8Lk5kMhmxAo5O1Jw1hocSDAVu+iwgw2aC6gcGSGCKIpvGEW
 fN0ei5VxFShYjc5zhC1ucm/qJxQo2m3Kem/U2kEYNwlKDyz+hHlJgGS07hFbJN5M4uPl2Ik
 Xb3lieSW86an0Gb4gAmkw==
X-Spam-Flag: NO
X-UI-Out-Filterresults: notjunk:1;V03:K0:WVi4h1ny/wk=:ynqdMADmxec9CxQ7fRJTym
 PoSPtJKJwkGeFU3OgUlBGlDPVc7CqzpArBKg130kM1LSuZMXxwQYrVnTenc77DoC3Kvud5ukl
 7vlJdfVBBmNr+IApCJu2ub9Ew4w/L9wUd89Wdqucq0Ol8K1XEMLd82Xivl/W+1PoyaFJM5UPX
 YSl9+UZPOWNCu0CtlGwTOzU3UkQu26mNcho2RGofCbgnIw7wHC5YkHasGze0CvwwpXdkWZKj1
 WSBZwmtORrR/v8q2FutlNtMRvfntfbGo5KrMAmZ8dYEAARYBhJywo/Xaca/ufTRvWWlSvep/g
 TsAdFJ1C8MjEzqJAhbPoDObyv6YcFFDFBXrbzP7bS71+LyCtyOASVXQFKu52RtlwhsbbTOagf
 zTbNKkWQX4PI5BVuI4W+KHIdvRUWpJ5BzjrL7cigVRmO14JHsG7zDNgemqUhVwgfoRylq2wmY
 m5PqOH5zwFRPYx4JSfWwokbVUvnZ49ggtYay68cBc+uAj7m1Ek4NBJNHZiF5ZqTgM69f2WUHc
 0TSu+nLJvGxO0zDGve6Css1QkxqyY8mHIxxEezyLTVwomyUCW5TWUEnlRn6UxED2WCcls8ier
 aBBSWK9bf2IX6HXkkwzXqC32NFlgAMS1uPMW6z9o+oyRtzYe95UrANASJ4TaRPQNrD50jrpDF
 o3sY=
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 36630
Cc: 36630 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

--=-=-=
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

Hi,

Ludovic Court=C3=A8s <ludo@HIDDEN> skribis:
> Arne Babenhauserheide <arne_bab@HIDDEN> skribis:
Offtopic: I love reading Esperanto here!
>> Ludovic Court=C3=A8s <ludo@HIDDEN> writes:

>>>   guix environment --ad-hoc jupyter python-ipython python-ipykernel -n

>> How exactly did you run the derivation?

> If you run the command above, it=E2=80=99ll list
> /gnu/store/=E2=80=A6-manual-database.drv.  So you can just run:
>
>   guix build /gnu/store/=E2=80=A6-manual-database.drv
>
> or:
>
>   guix build /gnu/store/=E2=80=A6-manual-database.drv --check
>
> if it had already been built before.

Somehow I can=E2=80=99t get guix to actually run my changed code with this
command, so I=E2=80=99m not sure I tested the right thing.

What is the clean approach to run the profile.scm from git?

>>> On a SSD and with a hot cache, on my 4-core laptop, I get 74s with
>>> =E2=80=98master=E2=80=99, and 53s with this patch.
>>
>> I=E2=80=99m using a machine with 6 physical cores, hyperthreading, and a=
n NVMe
>> M.2 disk, so it is likely that it would not be disk-bound for me at 4
>> threads.
>
> The result may be entirely different with a spinning disk.  :-)
>
> I=E2=80=99m not saying we should optimize for spinning disks, just that w=
hat you
> see is at one end of the spectrum.

That=E2=80=99s right, yes.

> But anyway, another issue is that we would need to honor
> =E2=80=98parallel-job-count=E2=80=99, which means using =E2=80=98n-par-ma=
p=E2=80=99, which doesn=E2=80=99t use
> futures.

Ouch, yes. That=E2=80=99s an issue =E2=80=A6=20

Thank you for bringing it up!

Best wishes,
Arne
=2D-=20
Unpolitisch sein
hei=C3=9Ft politisch sein
ohne es zu merken

--=-=-=
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEE801qEjXQSQPNItXAE++NRSQDw+sFAl0vm+4ACgkQE++NRSQD
w+uICBAAwiQygjoFRmEjJDl6Y8Ny6AdeTGmLXfOVb2FH9s/FEF2zhJtLf0atKWUN
EDqoj5mzqQ1IEoyKhcgzBbw/kWdx/eNj4qp/yJpj/Tp5z39Rr5QZFtbgX+oEQUur
d1T4zgCb2tpll93W3IC4BKGNtTBbubYM0SlPxlELU+x0yGRKTccYfSaYIHynWgC1
3Ij5tYZoU34m1Tk29H5AGdjlPjMfF5oNapmwwNk/JGlUWtg9Vaquf1S9HFqOcn3d
mezgEMz22wyXOdMXpUkuIxjwtvwJ5qvbxb0mlfjejt3eiShl0R8eThYhebzKgDNq
DTZ//3XLGaSf6Ih99Up4E03HUXOhZBCdQfM8aucU9LUR0okMGm5N9Nc+C4v0AI/N
hWzyuLC0wTKBdDLOqBJx5WTUBvKnuKIGKwOq0n3PQYVzVhye/eLFdTMAojPMXWaI
CK7edZP5HzKHnrRWxeTIIGkUrcccfQa1adVDH2d7xF1MzOSTAhkTSPGKGW7EZ2Ce
aeu0/DyGY8LGQD55sAoIeSKKfRkt1zkCA/AYsLC4ZkhuwdfoLQ9kvgp/DqvorUKl
bDMgFoMcDOo0mBHm409N5HQudca10FYPuaw4LwBIbKKxebY5Iiv75NA3WFehCDOs
ub4EX4C5lraADIqf9NY1AYo5CNYHpiegaeuA0jWT7jfEcNXCkXWIswQBAQgAHRYh
BN0ovebZh1yrzkqLHdzPDbMLwQVIBQJdL5vuAAoJENzPDbMLwQVIWr8D/1A6Kdyf
dQOA+QugrHNqVeYC5MqT/GvisnlikfM0N1mU+KrNX59CnoHZetKzKrYS8dh07eOU
3Ad3tdvSCErAQoUIhP6zACDeSudAu4ofuCxI9b5aQM4hNvDdMRQjO3VV+wSEA6a6
qMI4FopRdngMbwtxqrZu/bE4kt5R+4PCiqrO
=Uob2
-----END PGP SIGNATURE-----
--=-=-=--




Information forwarded to guix-patches@HIDDEN:
bug#36630; Package guix-patches. Full text available.

Message received at 36630 <at> debbugs.gnu.org:


Received: (at 36630) by debbugs.gnu.org; 16 Jul 2019 21:14:57 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Jul 16 17:14:57 2019
Received: from localhost ([127.0.0.1]:51256 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1hnUmn-00088l-9x
	for submit <at> debbugs.gnu.org; Tue, 16 Jul 2019 17:14:57 -0400
Received: from eggs.gnu.org ([209.51.188.92]:40415)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1hnUml-00088X-W4
 for 36630 <at> debbugs.gnu.org; Tue, 16 Jul 2019 17:14:56 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:46432)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <ludo@HIDDEN>)
 id 1hnUmg-0002WG-OS; Tue, 16 Jul 2019 17:14:50 -0400
Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=60040 helo=ribbon)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <ludo@HIDDEN>)
 id 1hnUmg-0002tl-AN; Tue, 16 Jul 2019 17:14:50 -0400
From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>
To: Arne Babenhauserheide <arne_bab@HIDDEN>
Subject: Re: [bug#36630] [PATCH] guix: parallelize building the manual-database
References: <20190712214245.23857-1-arne_bab@HIDDEN> <878sszl1jo.fsf@HIDDEN>
 <877e8ig9gh.fsf@HIDDEN>
X-URL: http://www.fdn.fr/~lcourtes/
X-Revolutionary-Date: 28 Messidor an 227 de la =?utf-8?Q?R=C3=A9volution?=
X-PGP-Key-ID: 0x090B11993D9AEBB5
X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc
X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4  0CFB 090B 1199 3D9A EBB5
X-OS: x86_64-pc-linux-gnu
Date: Tue, 16 Jul 2019 23:14:48 +0200
In-Reply-To: <877e8ig9gh.fsf@HIDDEN> (Arne Babenhauserheide's message of "Tue, 
 16 Jul 2019 01:32:46 +0200")
Message-ID: <87zhldel6f.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 36630
Cc: 36630 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Hello,

Arne Babenhauserheide <arne_bab@HIDDEN> skribis:

> Ludovic Court=C3=A8s <ludo@HIDDEN> writes:

[...]

>> I picked the manual-database derivation returned for:
>>   guix environment --ad-hoc jupyter python-ipython python-ipykernel -n
>> (It has 3,046 entries.)
>
> How exactly did you run the derivation? I=E2=80=99d like to check it if y=
ou can
> give me the exact commandline to run (a command I can run repeatedly).

If you run the command above, it=E2=80=99ll list
/gnu/store/=E2=80=A6-manual-database.drv.  So you can just run:

  guix build /gnu/store/=E2=80=A6-manual-database.drv

or:

  guix build /gnu/store/=E2=80=A6-manual-database.drv --check

if it had already been built before.

>> On a SSD and with a hot cache, on my 4-core laptop, I get 74s with
>> =E2=80=98master=E2=80=99, and 53s with this patch.
>
> I=E2=80=99m using a machine with 6 physical cores, hyperthreading, and an=
 NVMe
> M.2 disk, so it is likely that it would not be disk-bound for me at 4
> threads.

The result may be entirely different with a spinning disk.  :-)

I=E2=80=99m not saying we should optimize for spinning disks, just that wha=
t you
see is at one end of the spectrum.

>> However, it will definitely not scale linearly, so we should probably
>> cap at 2 or 4 threads.  WDYT?
>
> Looking at the underlying action, this seems to be a task that scales
> pretty well. It just unpacks files into the disk-cache.
>
> It should also not consume much memory, so I don=E2=80=99t see a reason to
> artificially limit the number of threads.

On a many-core machine like we have in our build farm, with spinning
disks, I believe that using one thread per core would be
counterproductive.

>> Another issue with the patch is that the [n/total] counter does not grow
>> monotically now: it might temporally go backwards.  Consequently, at
>> -v1, users will see a progress bar that hesitates and occasionally goes
>> backward, which isn=E2=80=99t great.
>
> It typically jumps forward in the beginning and then stalls until the
> first manual page is finished.
>
> Since par-map uses a global queue of futures to process, and since the
> output is the first part of (compute-entry =E2=80=A6), I don=E2=80=99t ex=
pect the
> progress to move backwards in ways a user sees: It could only move
> backwards during the initial step where all threads start at the same
> time, and there the initial output should be overwritten fast enough to
> not be noticeable.

Hmm, maybe.  I=E2=80=99m sure we=E2=80=99ll get reports saying this looks w=
eird and
Something Must Absolutely Be Done About It.  :-)

But anyway, another issue is that we would need to honor
=E2=80=98parallel-job-count=E2=80=99, which means using =E2=80=98n-par-map=
=E2=80=99, which doesn=E2=80=99t use
futures.

> Given that building manual pages is the most timeconsuming part when
> installing a small tool into my profile, I think it is worth the
> complexity. Especially because most of the complexity is being taken
> care of by (ice-9 threads par-map).

Just today I realized that the example above (with Jupyter) has so many
entries because of propagated inputs; in particular libxext along brings
1,000+ man pages.  We should definitely do something about these
packages.

Needs more thought=E2=80=A6

Thanks,
Ludo=E2=80=99.




Information forwarded to guix-patches@HIDDEN:
bug#36630; Package guix-patches. Full text available.

Message received at 36630 <at> debbugs.gnu.org:


Received: (at 36630) by debbugs.gnu.org; 15 Jul 2019 23:33:04 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Jul 15 19:33:04 2019
Received: from localhost ([127.0.0.1]:49391 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1hnASu-0008PX-8O
	for submit <at> debbugs.gnu.org; Mon, 15 Jul 2019 19:33:04 -0400
Received: from mout.web.de ([217.72.192.78]:53605)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <arne_bab@HIDDEN>) id 1hnASs-0008Ou-Dn
 for 36630 <at> debbugs.gnu.org; Mon, 15 Jul 2019 19:33:03 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=web.de;
 s=dbaedf251592; t=1563233573;
 bh=t2olDhcKTruwjH+4XQInq7SYnJpZsx1v0zsRS5Ck75w=;
 h=X-UI-Sender-Class:References:From:To:Cc:Subject:In-reply-to:Date;
 b=mf20X3huvBVfsrsIkWDoU6PQw9oOgUfZ8VIj+AvFvITHA7Zv5tOnl6vc3wClmwEHu
 vbmWJFbrC9D0h3Rh0FvFVbe1apgMwFOPR2OiJDRof5uehND4x2SK40dBLSYQFCQdnq
 u9Re55niSI+AdDMdidLLLpRr0pxJCvM7Ta0OCfJQ=
X-UI-Sender-Class: c548c8c5-30a9-4db5-a2e7-cb6cb037b8f9
Received: from fluss ([84.149.83.171]) by smtp.web.de (mrweb103
 [213.165.67.124]) with ESMTPSA (Nemesis) id 0LmuAa-1iNbkP0b08-00h8uj; Tue, 16
 Jul 2019 01:32:53 +0200
References: <20190712214245.23857-1-arne_bab@HIDDEN> <878sszl1jo.fsf@HIDDEN>
User-agent: mu4e 1.2.0; emacs 26.2
From: Arne Babenhauserheide <arne_bab@HIDDEN>
To: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Subject: Re: [bug#36630] [PATCH] guix: parallelize building the manual-database
In-reply-to: <878sszl1jo.fsf@HIDDEN>
Date: Tue, 16 Jul 2019 01:32:46 +0200
Message-ID: <877e8ig9gh.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="=-=-=";
 micalg=pgp-sha256; protocol="application/pgp-signature"
X-Provags-ID: V03:K1:bkrVH6QjhXkXv1Wpnuww+iiwTSjJcnwRnom0S8WaiUT8DVm44Ey
 1YNfHajak8BqHsb7l262XinfTmoTUC5SHTNGbrEYQ7sTivyEGHZ9xlwV2Bt+XcLkMerjdJq
 lnBoUsonVnfzeN1xyTBxEUz3/2mxxJvd8GjZ4L+ycxrKmwCSPLMIQ//IgPuJ3FTAzUCpudu
 EupC/FfQY/EDBYzdcyCGQ==
X-Spam-Flag: NO
X-UI-Out-Filterresults: notjunk:1;V03:K0:ONnhTmU/Bqk=:VLTjzf3RwY0r1BmNYkLdzV
 vCM0rmghWe6sOG0SIqE1vk9Dt6BUBUnbecdyl4ulBpaqQP/eHbWErluXGA7UbGc5ntqNfaEFd
 ukuarguqqsV62+2gzhaSSTpJ0kFEvACKhvr7M3V4D2V460/4ADR4Li/LwFYPOoj1ozTqiHLs/
 PSpVHqNnxrXRdRqaMcSQn+TVjex5huqXl2pA/NpDVLJ0pFkubvSZS17avhHj+h/alb6rpB3R+
 R3KkGrHeCDuo0fKklBzP8fgZXQAMaeZ6iuA67CrO9bVAauGWcyUEg/PrmdYg40DSJSxHRgGRd
 Xl6S3NRU6KYCPH0KQw7R7oONjdcAlR2RqoRcVAyoABsdT4hpZB4CqLHjNPHO6KhJ5+O43OGxm
 S3rnhFcypqq6FgpHPpC036aq9nBRL0fJJfjz3rsgT6RrUcahbP95K3ubDwnHXhx+ia3PADK2Q
 VB40betV5x66AaBz5cPrd9cSm9Yqt9hZjHCgL4WYecFNtE1eDWRqsCKahiMWaclgkLf0mutmH
 uEBBKvfJsMTnk00L2XUHRxuuGfLip+IP9mBo0Kv1VmELu4eXxJDsXTyWgS0l7TEW7Q+pdcYVV
 dgjQo0dYtVaEUu7NeUOuyUxFj71TehRVK5phKehIKO4FcoRmFlhr60PzQtlXi8UQKLZmwJotM
 8tkc=
X-Spam-Score: -0.7 (/)
X-Debbugs-Envelope-To: 36630
Cc: 36630 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

--=-=-=
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

Hi Ludo=E2=80=99,

Ludovic Court=C3=A8s <ludo@HIDDEN> writes:
>> * guix/profiles.scm (manual-database): par-map over the entries.  This
>> distributes the load roughly equally over all cores and avoids blocking =
on
>> I/O.  The order of the entries stays the same since write-mandb-database=
 sorts
>> them.
>
> I would think the whole process is largely I/O-bound.  Did you try
> measuring differences?

I did not measure the difference in build-time, but I did check the
system load. Without this patch, one of my cores is under full
load. With this patch all 12 hyperthreads have a mean load of 50%.

> I picked the manual-database derivation returned for:
>   guix environment --ad-hoc jupyter python-ipython python-ipykernel -n
> (It has 3,046 entries.)

How exactly did you run the derivation? I=E2=80=99d like to check it if you=
 can
give me the exact commandline to run (a command I can run repeatedly).


> On a SSD and with a hot cache, on my 4-core laptop, I get 74s with
> =E2=80=98master=E2=80=99, and 53s with this patch.

I=E2=80=99m using a machine with 6 physical cores, hyperthreading, and an N=
VMe
M.2 disk, so it is likely that it would not be disk-bound for me at 4
threads.

> However, it will definitely not scale linearly, so we should probably
> cap at 2 or 4 threads.  WDYT?

Looking at the underlying action, this seems to be a task that scales
pretty well. It just unpacks files into the disk-cache.

It should also not consume much memory, so I don=E2=80=99t see a reason to
artificially limit the number of threads.

> Another issue with the patch is that the [n/total] counter does not grow
> monotically now: it might temporally go backwards.  Consequently, at
> -v1, users will see a progress bar that hesitates and occasionally goes
> backward, which isn=E2=80=99t great.

It typically jumps forward in the beginning and then stalls until the
first manual page is finished.

Since par-map uses a global queue of futures to process, and since the
output is the first part of (compute-entry =E2=80=A6), I don=E2=80=99t expe=
ct the
progress to move backwards in ways a user sees: It could only move
backwards during the initial step where all threads start at the same
time, and there the initial output should be overwritten fast enough to
not be noticeable.

> This would need to fix it with a mutex-protected global counter.

A global counter would be pretty bad for scaling. As it is, this code
needs no communication between processes besides returning the final
result, so it behaves exactly like a normal map, aside from being
faster. So I=E2=80=99d prefer to accept the forward-jumping.

> All in all, I=E2=80=99m not sure this is worth the complexity.
>
> WDYT?

Given that building manual pages is the most timeconsuming part when
installing a small tool into my profile, I think it is worth the
complexity. Especially because most of the complexity is being taken
care of by (ice-9 threads par-map).

Best wishes,
Arne
=2D-
Unpolitisch sein
hei=C3=9Ft politisch sein
ohne es zu merken

--=-=-=
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEE801qEjXQSQPNItXAE++NRSQDw+sFAl0tDSAACgkQE++NRSQD
w+uuzQ//SIjIPq5kpaNC/3EgntRa5nIt+yKfT23K2juzCaYCAVUjnwHw4IXCCcSi
c2vX8heNXMFcLeML65E6ldao72WejeJ5tyXdbwjvexIQgvddUvG5vXpJfVVnrfPb
JPsjgaNj3DWdZRdq1lTMghlIpGrnRym+XIlzMB4BFHaiI1V5mBJ6IUk3ICddSqQo
QxOvEcI3oHuDh6C55UkQrrClcO0KDYNxG631EXB6/jCLNd7UzSq3DKtQNmHGRiQq
JyN2g29XipNyFp5HZ284FlegBoiO/OWseZoJ9s61KaWb9TKHeDeb7j4Yroj+MHxc
lpYoD58hVyJHb0Cgy0PtWIaQIa6WwWVG1OGStDKTTkLD5a8aZqwk0I8luuzvftV2
NdZYf047HYpbABQuVR+VLpS+ofkRy5GJCpSgJwpYzko8aI71LWDexipIs34LJTEE
RJ2QZ8WM7vUGBFRqjztOi0kxOYArRAkBR+aBY+dTOG+DAU/5Bitm8goVxuq9THob
MZQeRRCC4b/KC4lMB3QkREXf2U6FfLOpC0Y0+OHQMV4WBs/5Wf+3Qhqa3oy/NIZW
oodu4nAV4sytcdT9lVJuJqXmW6Vqy1UdcDry/FQbUjocJa4HN/BoRg2q5C5TL9Om
+B4CEXoxjfa7KLEL1dxEEqt+xcXajX6ys1t14FELqxr/OKJGdYmIswQBAQgAHRYh
BN0ovebZh1yrzkqLHdzPDbMLwQVIBQJdLQ0gAAoJENzPDbMLwQVICe0EAJarRMWw
leLwx4Q6quii/FUJK85wPqbB3cAPupTyS2v9pso/QIGkd2ku/NkksWEowa1PbhRx
V1PhD1AVFZ8aW1eQjz5aeh49g1udv0+IrOzetdm9Bzsv+zuweswzhZUJXgM1MMni
ILP1PcMcQEMPcYkC1+BiYOxZMgEip+3U72fN
=883H
-----END PGP SIGNATURE-----
--=-=-=--




Information forwarded to guix-patches@HIDDEN:
bug#36630; Package guix-patches. Full text available.

Message received at 36630 <at> debbugs.gnu.org:


Received: (at 36630) by debbugs.gnu.org; 15 Jul 2019 16:12:44 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Jul 15 12:12:44 2019
Received: from localhost ([127.0.0.1]:48839 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1hn3am-0007kC-0Y
	for submit <at> debbugs.gnu.org; Mon, 15 Jul 2019 12:12:44 -0400
Received: from eggs.gnu.org ([209.51.188.92]:45191)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1hn3ae-0007jj-Sk
 for 36630 <at> debbugs.gnu.org; Mon, 15 Jul 2019 12:12:41 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:51429)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <ludo@HIDDEN>)
 id 1hn3aZ-00023b-5Z; Mon, 15 Jul 2019 12:12:31 -0400
Received: from [2001:660:6102:320:e120:2c8f:8909:cdfe] (port=40746 helo=ribbon)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <ludo@HIDDEN>)
 id 1hn3aY-0007pt-Km; Mon, 15 Jul 2019 12:12:30 -0400
From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>
To: Arne Babenhauserheide <arne_bab@HIDDEN>
Subject: Re: [bug#36630] [PATCH] guix: parallelize building the manual-database
References: <20190712214245.23857-1-arne_bab@HIDDEN>
Date: Mon, 15 Jul 2019 18:12:27 +0200
In-Reply-To: <20190712214245.23857-1-arne_bab@HIDDEN> (Arne Babenhauserheide's
 message of "Fri, 12 Jul 2019 23:42:45 +0200")
Message-ID: <878sszl1jo.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 36630
Cc: 36630 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Hi Arne,

Arne Babenhauserheide <arne_bab@HIDDEN> skribis:

> * guix/profiles.scm (manual-database): par-map over the entries.  This
> distributes the load roughly equally over all cores and avoids blocking on
> I/O.  The order of the entries stays the same since write-mandb-database =
sorts
> them.

I would think the whole process is largely I/O-bound.  Did you try
measuring differences?

I picked the manual-database derivation returned for:

  guix environment --ad-hoc jupyter python-ipython python-ipykernel -n

(It has 3,046 entries.)

On a SSD and with a hot cache, on my 4-core laptop, I get 74s with
=E2=80=98master=E2=80=99, and 53s with this patch.

However, it will definitely not scale linearly, so we should probably
cap at 2 or 4 threads.  WDYT?

Another issue with the patch is that the [n/total] counter does not grow
monotically now: it might temporally go backwards.  Consequently, at
-v1, users will see a progress bar that hesitates and occasionally goes
backward, which isn=E2=80=99t great.

This would need to fix it with a mutex-protected global counter.

All in all, I=E2=80=99m not sure this is worth the complexity.

WDYT?

Thanks,
Ludo=E2=80=99.




Information forwarded to guix-patches@HIDDEN:
bug#36630; Package guix-patches. Full text available.

Message received at submit <at> debbugs.gnu.org:


Received: (at submit) by debbugs.gnu.org; 12 Jul 2019 21:43:19 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Jul 12 17:43:19 2019
Received: from localhost ([127.0.0.1]:40861 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1hm3K2-0006Ni-Os
	for submit <at> debbugs.gnu.org; Fri, 12 Jul 2019 17:43:19 -0400
Received: from lists.gnu.org ([209.51.188.17]:38274)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <arne_bab@HIDDEN>) id 1hm3K0-0006Na-V8
 for submit <at> debbugs.gnu.org; Fri, 12 Jul 2019 17:43:17 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:52943)
 by lists.gnu.org with esmtp (Exim 4.86_2)
 (envelope-from <arne_bab@HIDDEN>) id 1hm3Jz-0003qD-Tr
 for guix-patches@HIDDEN; Fri, 12 Jul 2019 17:43:16 -0400
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=-0.7 required=5.0 tests=BAYES_20,FREEMAIL_FROM,
 RCVD_IN_DNSWL_LOW autolearn=disabled version=3.3.2
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <arne_bab@HIDDEN>) id 1hm3Jy-0005gA-Sn
 for guix-patches@HIDDEN; Fri, 12 Jul 2019 17:43:15 -0400
Received: from mout.web.de ([217.72.192.78]:33057)
 by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_128_CBC_SHA1:16)
 (Exim 4.71) (envelope-from <arne_bab@HIDDEN>) id 1hm3Jw-0005dr-Po
 for guix-patches@HIDDEN; Fri, 12 Jul 2019 17:43:13 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=web.de;
 s=dbaedf251592; t=1562967788;
 bh=JSbsKdEiZ7UE+6euXjjDF1YUCnSrh1qriFSJCajp9WE=;
 h=X-UI-Sender-Class:From:To:Cc:Subject:Date:Reply-To;
 b=HoCYmKd1W4wBxRFYm8LJwncDR4fvwHBY4FaLxJ6UXivGXFx3ICocirIVphZV0HNGf
 7WHyjjwSmIRZDdGhilPbhVzHrVAGZ+ai/OQK8XTXOfezPVc2jn49H68/x25PLI5eE8
 jmISwfqE9tYTvswKN3c5po9m5rK34PTrS7hlafoA=
X-UI-Sender-Class: c548c8c5-30a9-4db5-a2e7-cb6cb037b8f9
Received: from fluss.speedport.ip ([84.149.83.171]) by smtp.web.de (mrweb103
 [213.165.67.124]) with ESMTPSA (Nemesis) id 0MRlED-1hsoej20IT-00SuF8; Fri, 12
 Jul 2019 23:43:08 +0200
From: Arne Babenhauserheide <arne_bab@HIDDEN>
To: guix-patches@HIDDEN
Subject: [PATCH] guix: parallelize building the manual-database
Date: Fri, 12 Jul 2019 23:42:45 +0200
Message-Id: <20190712214245.23857-1-arne_bab@HIDDEN>
X-Mailer: git-send-email 2.22.0
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable
X-Provags-ID: V03:K1:M6QunW6vh03fYfPLecXtdY1IdXpj1J2+g7gqzxoq5hpplzKG1rj
 w1WR1Y/83NNBgPaIt91gT4b4JdYf27w5rrPSJDG4Wblhyaypo5MOGGij8a7m6jT1EU4qOnE
 WxI+RBQBWKUMHIY+GQYAdqbhXSdjKF7/9KZoLJ1LlKfxN+5a7u0PqoVSwNJ0FR/GB6M6DDf
 xq2cNNdlLw9TB0+MEltRA==
X-UI-Out-Filterresults: notjunk:1;V03:K0:Z4qCFm3X6ts=:tE0LYddfyeMyARB0+HN+N7
 V4S/ZxVv0r/GKADlYF2UkWqdflqj2kIpgXuApr6PALMxwUe/flEeQZpuNffSUBnoQTZnryq4g
 89CNK2YGk8l5lwck+deV+E+fsclC9z6TuVKqCZtLdHwwtjTCh4G955cYctvLKvFLNDlcT3aZ6
 lxrQFWxt/aq/gjsgGeDd8z9n8AifbHOMrP62vgQeN/0A4E6ktisuM9YPCJj8D/p+b59/jFOe7
 xD1eJAZ6nYwsKE0LeKFTv7DzL266J/x958RUYI5JfyrYiEH7BQebx2Ooelqy+HoZVTUyjNW9T
 hrreQkWR6AnlAnQ7e7aHV44zumDeZ1Y7OmjERZ8OjobZac3p7emjPX/HNXqzhZYQhLyVVDOI9
 JWYIo9s/mVPhw3SwNA/5zbsx4OiKClv/tZfIbCrvOhBwpa5srz4kJfGJNpepUZARPdwBWhFzW
 qqdS3DkDhmrAdOfp4xRXLxH8P3NQugkxq4S4YVOarXiblg+pvWCp8v2WV5jJKAvn/ExQenBdh
 p6Vr9HljPxNZVEkXbyoNC9CSq5Ldq0KQOM381bRi0fvajoN68gtNe2WwWgePMqcNOiUejDD0m
 iZw5yMNrThofgPQMGfUJTaFBwC5Gbs2+XzlzRoAUmyyHsyS+M5xV8HELUB3NrzvSYwTtOu/UQ
 bfzQ=
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 217.72.192.78
X-Spam-Score: -1.4 (-)
X-Debbugs-Envelope-To: submit
Cc: Arne Babenhauserheide <arne_bab@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Reply-To: arne_bab@HIDDEN
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -2.4 (--)

* guix/profiles.scm (manual-database): par-map over the entries.  This
distributes the load roughly equally over all cores and avoids blocking on
I/O.  The order of the entries stays the same since write-mandb-database s=
orts
them.
=2D--
 guix/profiles.scm | 20 ++++++++++++--------
 1 file changed, 12 insertions(+), 8 deletions(-)

diff --git a/guix/profiles.scm b/guix/profiles.scm
index f5c863945c..374f0f8a90 100644
=2D-- a/guix/profiles.scm
+++ b/guix/profiles.scm
@@ -1312,15 +1312,11 @@ the entries in MANIFEST."
         #~(begin
             (use-modules (guix man-db)
                          (guix build utils)
+                         (ice-9 threads)
                          (srfi srfi-1)
                          (srfi srfi-19))

-            (define (compute-entries)
-              ;; This is the most expensive part (I/O and CPU, due to
-              ;; decompression), so report progress as we traverse INPUTS=
.
-              (let* ((inputs '#$(manifest-inputs manifest))
-                     (total  (length inputs)))
-                (append-map (lambda (directory count)
+            (define (compute-entry directory count total)
                               (format #t "\r[~3d/~3d] building list of \
 man-db entries..."
                                       count total)
@@ -1330,8 +1326,16 @@ man-db entries..."
                                 (if (directory-exists? man)
                                     (mandb-entries man)
                                     '())))
-                            inputs
-                            (iota total 1))))
+
+            (define (compute-entries)
+              ;; This is the most expensive part (I/O and CPU, due to
+              ;; decompression), so report progress as we traverse INPUTS=
.
+              (let* ((inputs '#$(manifest-inputs manifest))
+                     (total  (length inputs)))
+                (apply append (par-map compute-entry
+                                       inputs
+                                       (iota total 1)
+                                       (make-list total total)))))

             (define man-directory
               (string-append #$output "/share/man"))
=2D-
2.22.0





Acknowledgement sent to arne_bab@HIDDEN:
New bug report received and forwarded. Copy sent to guix-patches@HIDDEN. Full text available.
Report forwarded to guix-patches@HIDDEN:
bug#36630; Package guix-patches. Full text available.
Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.
Last modified: Mon, 25 Nov 2019 12:00:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.