GNU bug report logs - #40628
Optimize admin/nt dependency computation

Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.

Package: emacs; Severity: wishlist; Reported by: Noam Postavsky <npostavs@HIDDEN>; Keywords: patch; Done: Lars Ingebrigtsen <larsi@HIDDEN>; Maintainer for emacs is bug-gnu-emacs@HIDDEN.
bug marked as fixed in version 29.1, send any further explanations to 40628 <at> debbugs.gnu.org and Noam Postavsky <npostavs@HIDDEN> Request was from Lars Ingebrigtsen <larsi@HIDDEN> to control <at> debbugs.gnu.org. Full text available.

Message received at 40628 <at> debbugs.gnu.org:


Received: (at 40628) by debbugs.gnu.org; 12 Nov 2021 09:23:33 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Nov 12 04:23:33 2021
Received: from localhost ([127.0.0.1]:43445 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mlSmT-0005r1-6L
	for submit <at> debbugs.gnu.org; Fri, 12 Nov 2021 04:23:33 -0500
Received: from quimby.gnus.org ([95.216.78.240]:35400)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <larsi@HIDDEN>) id 1mlSmR-0005ql-4E
 for 40628 <at> debbugs.gnu.org; Fri, 12 Nov 2021 04:23:32 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnus.org;
 s=20200322; h=Content-Type:MIME-Version:Message-ID:In-Reply-To:Date:
 References:Subject:Cc:To:From:Sender:Reply-To:Content-Transfer-Encoding:
 Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender:
 Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:
 List-Subscribe:List-Post:List-Owner:List-Archive;
 bh=sZpSZGr6nQXvCchb5Q8bE3poCjU1EILcExjiPHiFDyk=; b=O5lnsiuUJBOg2EAOFxfUu0Tpx4
 iA4+Vm/VfdUJn2WtcRYMX9jJRRX3ifS3YS73uKgQ6lMzUbYePZQWk/MDuAqIhU+gSjxittv37lrys
 zftgBSi0+jDEh5SBXWlEcarw9ys40b9pLDJmhMDsNnK53zx15E+fvQ2gWhcsRfbs9HDk=;
Received: from [84.212.220.105] (helo=xo)
 by quimby.gnus.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.92) (envelope-from <larsi@HIDDEN>)
 id 1mlSmE-00065P-Uj; Fri, 12 Nov 2021 10:23:22 +0100
From: Lars Ingebrigtsen <larsi@HIDDEN>
To: Noam Postavsky <npostavs@HIDDEN>
Subject: Re: bug#40628: Optimize admin/nt dependency computation
References: <854ktmyq7s.fsf@HIDDEN> <87k0y91ejn.fsf@HIDDEN>
 <87lfickqd3.fsf@HIDDEN>
X-Now-Playing: Sonic Youth's _Sonic Nurse_: "Pattern Recognition"
Date: Fri, 12 Nov 2021 10:23:17 +0100
In-Reply-To: <87lfickqd3.fsf@HIDDEN> (Lars Ingebrigtsen's message of "Tue,
 18 Aug 2020 15:58:32 +0200")
Message-ID: <87czn5u3qy.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/29.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Report: Spam detection software, running on the system "quimby.gnus.org",
 has NOT identified this incoming email as spam.  The original
 message has been attached to this so you can view it or label
 similar future email.  If you have any questions, see
 @@CONTACT_ADDRESS@@ for details.
 Content preview: Lars Ingebrigtsen <larsi@HIDDEN> writes: > Lars
 Ingebrigtsen
 <larsi@HIDDEN> writes: > >> Noam Postavsky <npostavs@HIDDEN> writes:
 >> >>> While checking the patch I suggested in >>>
 https://debbugs.gnu.org/40003#11, I noticed the depende [...] 
 Content analysis details:   (-2.9 points, 5.0 required)
 pts rule name              description
 ---- ---------------------- --------------------------------------------------
 -1.0 ALL_TRUSTED            Passed through trusted hosts only via SMTP
 -1.9 BAYES_00               BODY: Bayes spam probability is 0 to 1%
 [score: 0.0000]
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 40628
Cc: 40628 <at> debbugs.gnu.org, phillip lord <phillip.lord@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Lars Ingebrigtsen <larsi@HIDDEN> writes:

> Lars Ingebrigtsen <larsi@HIDDEN> writes:
>
>> Noam Postavsky <npostavs@HIDDEN> writes:
>>
>>> While checking the patch I suggested in
>>> https://debbugs.gnu.org/40003#11, I noticed the dependency extraction
>>> seemed pretty slow.  With the patch attached below it's much faster
>>> (1m20.437s to 0m3.695s according to 'time ./build-dep-zips.py -l').
>
> [...]
>
>> This was four months ago, but the patch was apparently not applied?
>
> I don't have any Windows machines -- has anybody given this patch a test
> to see whether it introduces any problems?

I tested it myself, and it brought the generation time down from ~3
minutes to about 20 seconds, so I've applied the patch.  I don't see any
difference in the results, but then again, if failed with this both
before and after the change:

subprocess.CalledProcessError: Command 'wget -a ../download.log -O ../emacs-src-cache/mingw-w64-libtiff-4.3.0-6.src.tar.gz https://sourceforge.net/projects/msys2/files/REPOS/MINGW/Sources/mingw-w64-libtiff-4.3.0-6.src.tar.gz/download' returned non-zero exit status 8.


-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#40628; Package emacs. Full text available.

Message received at 40628 <at> debbugs.gnu.org:


Received: (at 40628) by debbugs.gnu.org; 18 Aug 2020 13:58:44 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Aug 18 09:58:44 2020
Received: from localhost ([127.0.0.1]:36532 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1k828S-0007ta-BL
	for submit <at> debbugs.gnu.org; Tue, 18 Aug 2020 09:58:44 -0400
Received: from quimby.gnus.org ([95.216.78.240]:32952)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <larsi@HIDDEN>) id 1k828Q-0007tK-RD
 for 40628 <at> debbugs.gnu.org; Tue, 18 Aug 2020 09:58:43 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnus.org;
 s=20200322; h=Content-Type:MIME-Version:Message-ID:In-Reply-To:Date:
 References:Subject:Cc:To:From:Sender:Reply-To:Content-Transfer-Encoding:
 Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender:
 Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:
 List-Subscribe:List-Post:List-Owner:List-Archive;
 bh=8gJ+aKfsS3EgM5C6pY8MW5YdeHH9mKXSAP3LO/NASZw=; b=alKf79Cc1fsMh12/2NgiSgCWCW
 nNPUDdaZEZesTvOIl4zvzzLqMts3z9lkko0Byp9Dbece25atCqQiw+QBS/3AKPDL7KblMcgRsSP41
 mCjQIy0+Fwo4HKmjNLvWZwGqHNbMPYBDlk7ReXqzsqlv2E5sMFpw68SWx0W662iGds6k=;
Received: from cm-84.212.202.86.getinternet.no ([84.212.202.86] helo=xo)
 by quimby with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.92) (envelope-from <larsi@HIDDEN>)
 id 1k828I-0007bs-3b; Tue, 18 Aug 2020 15:58:36 +0200
From: Lars Ingebrigtsen <larsi@HIDDEN>
To: Noam Postavsky <npostavs@HIDDEN>
Subject: Re: bug#40628: Optimize admin/nt dependency computation
References: <854ktmyq7s.fsf@HIDDEN> <87k0y91ejn.fsf@HIDDEN>
X-Now-Playing: Dead Can Dance's _Spleen and Ideal_: "De Profundis (Out of
 the Depths of Sorrow)"
Date: Tue, 18 Aug 2020 15:58:32 +0200
In-Reply-To: <87k0y91ejn.fsf@HIDDEN> (Lars Ingebrigtsen's message of "Sat,
 08 Aug 2020 14:57:00 +0200")
Message-ID: <87lfickqd3.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Report: Spam detection software, running on the system "quimby.gnus.org",
 has NOT identified this incoming email as spam.  The original
 message has been attached to this so you can view it or label
 similar future email.  If you have any questions, see
 @@CONTACT_ADDRESS@@ for details.
 Content preview: Lars Ingebrigtsen <larsi@HIDDEN> writes: > Noam Postavsky
 <npostavs@HIDDEN> writes: > >> While checking the patch I suggested in
 >> https://debbugs.gnu.org/40003#11, I noticed the dependency extraction
 >> seemed pretty slow. With the patc [...] 
 Content analysis details:   (-2.9 points, 5.0 required)
 pts rule name              description
 ---- ---------------------- --------------------------------------------------
 -1.0 ALL_TRUSTED            Passed through trusted hosts only via SMTP
 -1.9 BAYES_00               BODY: Bayes spam probability is 0 to 1%
 [score: 0.0000]
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 40628
Cc: 40628 <at> debbugs.gnu.org, phillip lord <phillip.lord@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Lars Ingebrigtsen <larsi@HIDDEN> writes:

> Noam Postavsky <npostavs@HIDDEN> writes:
>
>> While checking the patch I suggested in
>> https://debbugs.gnu.org/40003#11, I noticed the dependency extraction
>> seemed pretty slow.  With the patch attached below it's much faster
>> (1m20.437s to 0m3.695s according to 'time ./build-dep-zips.py -l').

[...]

> This was four months ago, but the patch was apparently not applied?

I don't have any Windows machines -- has anybody given this patch a test
to see whether it introduces any problems?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#40628; Package emacs. Full text available.

Message received at 40628 <at> debbugs.gnu.org:


Received: (at 40628) by debbugs.gnu.org; 8 Aug 2020 12:57:13 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Aug 08 08:57:13 2020
Received: from localhost ([127.0.0.1]:58476 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1k4OPQ-0004xC-Sa
	for submit <at> debbugs.gnu.org; Sat, 08 Aug 2020 08:57:13 -0400
Received: from quimby.gnus.org ([95.216.78.240]:51436)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <larsi@HIDDEN>) id 1k4OPP-0004wz-EC
 for 40628 <at> debbugs.gnu.org; Sat, 08 Aug 2020 08:57:11 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnus.org;
 s=20200322; h=Content-Type:MIME-Version:Message-ID:In-Reply-To:Date:
 References:Subject:Cc:To:From:Sender:Reply-To:Content-Transfer-Encoding:
 Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender:
 Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:
 List-Subscribe:List-Post:List-Owner:List-Archive;
 bh=SPBHvh7N4kgRMXmVaZ2VfhpyYHeUCYSr/XTnc2VF3uQ=; b=BxJKVXzwkWH0rxi8VUhnR0mZgt
 IZa4re9MQw6FUTlidwfMszwnl4ZAS0xC3R5LrD7sgcAdQbCmWlBJ7wyC3vh3JMokvLjMnZ0OMG/Zl
 rttHqmy8I32TlFyoVffKPoWtiHFUvIZummO1Xu03HUHfp0GFWGOhnkdI87X5VvB4CR9E=;
Received: from cm-84.212.202.86.getinternet.no ([84.212.202.86] helo=xo)
 by quimby with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.92) (envelope-from <larsi@HIDDEN>)
 id 1k4OPG-0004YC-Il; Sat, 08 Aug 2020 14:57:05 +0200
From: Lars Ingebrigtsen <larsi@HIDDEN>
To: Noam Postavsky <npostavs@HIDDEN>
Subject: Re: bug#40628: Optimize admin/nt dependency computation
References: <854ktmyq7s.fsf@HIDDEN>
Date: Sat, 08 Aug 2020 14:57:00 +0200
In-Reply-To: <854ktmyq7s.fsf@HIDDEN> (Noam Postavsky's message of "Tue, 14
 Apr 2020 12:41:43 -0400")
Message-ID: <87k0y91ejn.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Report: Spam detection software, running on the system "quimby.gnus.org",
 has NOT identified this incoming email as spam.  The original
 message has been attached to this so you can view it or label
 similar future email.  If you have any questions, see
 @@CONTACT_ADDRESS@@ for details.
 Content preview: Noam Postavsky <npostavs@HIDDEN> writes: > While checking
 the patch I suggested in > https://debbugs.gnu.org/40003#11, I noticed the
 dependency extraction > seemed pretty slow. With the patch attached below
 it's much faster > (1m20.437s to 0 [...] 
 Content analysis details:   (-2.9 points, 5.0 required)
 pts rule name              description
 ---- ---------------------- --------------------------------------------------
 -1.0 ALL_TRUSTED            Passed through trusted hosts only via SMTP
 -1.9 BAYES_00               BODY: Bayes spam probability is 0 to 1%
 [score: 0.0000]
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 40628
Cc: 40628 <at> debbugs.gnu.org, phillip lord <phillip.lord@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Noam Postavsky <npostavs@HIDDEN> writes:

> While checking the patch I suggested in
> https://debbugs.gnu.org/40003#11, I noticed the dependency extraction
> seemed pretty slow.  With the patch attached below it's much faster
> (1m20.437s to 0m3.695s according to 'time ./build-dep-zips.py -l').
>
>>From 5312f51e315cc147ca901d499aa3c3cc64aa3340 Mon Sep 17 00:00:00 2001
> From: Noam Postavsky <npostavs@HIDDEN>
> Date: Tue, 14 Apr 2020 12:27:34 -0400
> Subject: [PATCH] Optimize admin/nt dependency computation
>
> admin/nt/dist-build/build-dep-zips.py (immediate_deps)
> (extract_deps): Gather package dependency info in batches, rather than
> one at a time.  This reduces the number of invocations of 'pacman -Si
> ...' to the depth of the dependency tree, rather than the number of
> dependent packages.
> (top-level): Don't call 'extract_deps' when given the '-l' option.

This was four months ago, but the patch was apparently not applied?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#40628; Package emacs. Full text available.

Message received at submit <at> debbugs.gnu.org:


Received: (at submit) by debbugs.gnu.org; 14 Apr 2020 16:41:53 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Apr 14 12:41:53 2020
Received: from localhost ([127.0.0.1]:34611 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1jOOdF-0001SW-D0
	for submit <at> debbugs.gnu.org; Tue, 14 Apr 2020 12:41:53 -0400
Received: from lists.gnu.org ([209.51.188.17]:34038)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <npostavs@HIDDEN>) id 1jOOdE-0001SP-46
 for submit <at> debbugs.gnu.org; Tue, 14 Apr 2020 12:41:52 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:57386)
 by lists.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <npostavs@HIDDEN>) id 1jOOdC-0000BR-2I
 for bug-gnu-emacs@HIDDEN; Tue, 14 Apr 2020 12:41:51 -0400
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50,FREEMAIL_FROM,
 URIBL_BLOCKED autolearn=disabled version=3.3.2
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <npostavs@HIDDEN>) id 1jOOdA-0006Ax-Kw
 for bug-gnu-emacs@HIDDEN; Tue, 14 Apr 2020 12:41:49 -0400
Received: from mail-qv1-xf2f.google.com ([2607:f8b0:4864:20::f2f]:32832)
 by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16)
 (Exim 4.71) (envelope-from <npostavs@HIDDEN>) id 1jOOdA-0006AZ-Ef
 for bug-gnu-emacs@HIDDEN; Tue, 14 Apr 2020 12:41:48 -0400
Received: by mail-qv1-xf2f.google.com with SMTP id p19so169847qve.0
 for <bug-gnu-emacs@HIDDEN>; Tue, 14 Apr 2020 09:41:48 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=from:to:subject:date:message-id:mime-version;
 bh=KpWw2VQWkuWbTHN9yy2qkLWICjE/RHWddZ9XZdDMO+E=;
 b=txixhTyB2NGKvulai/QixH7yn3ph+XMDDE8pAK6FAzye3/bCHgQIbUATPqcPnnno1J
 SU8OrUzXr7PBdHTaUacj9RagIhmemw8wBY2OHwUbn/Nf6muf8YKqM8roXj6buxaOffLs
 U2Sr5gQrRd/9k1ecWbyndhbLuWidDVAOhybEK7fnShRGn28bwUbNWXUvYtciEnJnGf4W
 1B4UUcaPKxGAFiiYBTgGSzySnKm9/oGohOAbSWff+ymWyOAj2vtQ9NUWET95Z+IPPVL/
 Da/j0OarxbtBOcYmkEYKUibCXU79nnT7LkNOumPL006fslWvJoCF/upuRCHQNB3vZjny
 Eg+A==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:from:to:subject:date:message-id:mime-version;
 bh=KpWw2VQWkuWbTHN9yy2qkLWICjE/RHWddZ9XZdDMO+E=;
 b=SxGvGldukW4Hs8zk754HHFAMoCHavAYHIpzNsm/vRyzmffkKTAsRbCUBJjVcXc11A0
 qE/IY16E6zJf2qsamy7nQFesXgwf9QDvVHQRVRP/0cPVjQ8966HxII7vEOevlIqfHL5i
 SJN+tL7vF1/9a7GkGVYY1xHkTfcJlxRr7HAZkUuxHjy6hVZDQicGc9K5IsCbwWMJbSzL
 xZRHZ/qlyYEravNnlB5ARgEQ2dqy2lHPyH8P3Y94DdIykGSysr1OduqhXi4pc3630urw
 L13zPnvf0HQV/gaRQAZ7sVJhKEAmASFy1PlzQasZfJ0w9i0m0lMRC1dFGIoqzx/Mmr58
 rjzg==
X-Gm-Message-State: AGi0PuYD8v4uUAX1k8SR9jIM6gz6rJb7dA1LaHpeyLToeRGhEm3xu9IR
 q8yEev6vMuI3mGzmR64q34Oic4Id0LE=
X-Google-Smtp-Source: APiQypKe+/GHgvprOfyleCmaOMXf9YlffpVW0x4RWoAtK61xFIB3EGwOGtGrZCVNvfx3EC+FD+oAgg==
X-Received: by 2002:a0c:fa08:: with SMTP id q8mr855492qvn.132.1586882504646;
 Tue, 14 Apr 2020 09:41:44 -0700 (PDT)
Received: from vhost2
 (CPE001143542e1f-CMf81d0f809fa0.cpe.net.cable.rogers.com. [99.230.38.42])
 by smtp.gmail.com with ESMTPSA id v62sm1773278qkb.85.2020.04.14.09.41.43
 for <bug-gnu-emacs@HIDDEN>
 (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256);
 Tue, 14 Apr 2020 09:41:43 -0700 (PDT)
From: Noam Postavsky <npostavs@HIDDEN>
To: bug-gnu-emacs@HIDDEN
Subject: Optimize admin/nt dependency computation
Date: Tue, 14 Apr 2020 12:41:43 -0400
Message-ID: <854ktmyq7s.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="=-=-="
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
 recognized.
X-Received-From: 2607:f8b0:4864:20::f2f
X-Spam-Score: 2.3 (++)
X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org",
 has NOT identified this incoming email as spam.  The original
 message has been attached to this so you can view it or label
 similar future email.  If you have any questions, see
 the administrator of that system for details.
 Content preview:  Severity: wishlist Tags: patch X-Debbugs-CC: Phillip Lord
 While checking the patch I suggested in https://debbugs.gnu.org/40003#11,
 I noticed the dependency extraction seemed pretty slow. With the patch
 attached below it's much faster (1m20.437s to 0m3.695s [...] 
 Content analysis details:   (2.3 points, 10.0 required)
 pts rule name              description
 ---- ---------------------- --------------------------------------------------
 0.0 URIBL_BLOCKED          ADMINISTRATOR NOTICE: The query to URIBL was
 blocked.  See
 http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block
 for more information. [URIs: build-dep-zips.py]
 0.0 SPF_HELO_NONE          SPF: HELO does not publish an SPF Record
 0.0 FREEMAIL_FROM          Sender email is commonly abused enduser mail
 provider (npostavs[at]gmail.com)
 1.0 SPF_SOFTFAIL           SPF: sender does not match SPF record (softfail)
 -0.7 RCVD_IN_DNSWL_LOW      RBL: Sender listed at https://www.dnswl.org/,
 low trust [209.51.188.17 listed in list.dnswl.org]
 2.0 SPOOFED_FREEMAIL       No description available.
X-Debbugs-Envelope-To: submit
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.7 (/)

--=-=-=
Content-Type: text/plain

Severity: wishlist
Tags: patch
X-Debbugs-CC: Phillip Lord <phillip.lord@HIDDEN>

While checking the patch I suggested in
https://debbugs.gnu.org/40003#11, I noticed the dependency extraction
seemed pretty slow.  With the patch attached below it's much faster
(1m20.437s to 0m3.695s according to 'time ./build-dep-zips.py -l').


--=-=-=
Content-Type: text/plain
Content-Disposition: attachment;
 filename=0001-Optimize-admin-nt-dependency-computation.patch
Content-Description: patch

From 5312f51e315cc147ca901d499aa3c3cc64aa3340 Mon Sep 17 00:00:00 2001
From: Noam Postavsky <npostavs@HIDDEN>
Date: Tue, 14 Apr 2020 12:27:34 -0400
Subject: [PATCH] Optimize admin/nt dependency computation

admin/nt/dist-build/build-dep-zips.py (immediate_deps)
(extract_deps): Gather package dependency info in batches, rather than
one at a time.  This reduces the number of invocations of 'pacman -Si
...' to the depth of the dependency tree, rather than the number of
dependent packages.
(top-level): Don't call 'extract_deps' when given the '-l' option.
---
 admin/nt/dist-build/build-dep-zips.py | 45 ++++++++++++++-------------
 1 file changed, 23 insertions(+), 22 deletions(-)

diff --git a/admin/nt/dist-build/build-dep-zips.py b/admin/nt/dist-build/build-dep-zips.py
index b0345a42cf3..dd7dc5e99dc 100755
--- a/admin/nt/dist-build/build-dep-zips.py
+++ b/admin/nt/dist-build/build-dep-zips.py
@@ -22,6 +22,8 @@
 import os
 import shutil
 import re
+import functools
+import operator
 
 from subprocess import check_output
 
@@ -47,7 +49,7 @@
 ## Packages to fiddle with
 ## Source for gcc-libs is part of gcc
 SKIP_SRC_PKGS=["mingw-w64-gcc-libs"]
-SKIP_DEP_PKGS=["mingw-w64-x86_64-glib2"]
+SKIP_DEP_PKGS=frozenset(["mingw-w64-x86_64-glib2"])
 MUNGE_SRC_PKGS={"mingw-w64-libwinpthread-git":"mingw-w64-winpthreads-git"}
 MUNGE_DEP_PKGS={
     "mingw-w64-i686-libwinpthread":"mingw-w64-i686-libwinpthread-git",
@@ -68,16 +70,14 @@ def check_output_maybe(*args,**kwargs):
     else:
         return check_output(*args,**kwargs)
 
-def immediate_deps(pkg):
-    package_info = check_output(["pacman", "-Si", pkg]).decode("utf-8").split("\n")
+def immediate_deps(pkgs):
+    package_info = check_output(["pacman", "-Si"] + pkgs).decode("utf-8").splitlines()
 
-    ## Extract the "Depends On" line
-    depends_on = [x for x in package_info if x.startswith("Depends On")][0]
-    ## Remove "Depends On" prefix
-    dependencies = depends_on.split(":")[1]
-
-    ## Split into dependencies
-    dependencies = dependencies.strip().split(" ")
+    ## Extract the packages listed for "Depends On:" lines.
+    dependencies = [line.split(":")[1].split() for line in package_info
+                    if line.startswith("Depends On")]
+    ## Flatten dependency lists from multiple packages into one list.
+    dependencies = functools.reduce(operator.iconcat, dependencies, [])
 
     ## Remove > signs TODO can we get any other punctuation here?
     dependencies = [d.split(">")[0] for d in dependencies if d]
@@ -92,18 +92,19 @@ def extract_deps():
     print( "Extracting deps" )
 
     # Get a list of all dependencies needed for packages mentioned above.
-    pkgs = PKG_REQ[:]
+    pkgs = set(PKG_REQ)
     print("Initial pkgs", pkgs)
-    n = 0
-    while n < len(pkgs):
-        subdeps = immediate_deps(pkgs[n])
-        for p in subdeps:
-            if not (p in pkgs or p in SKIP_DEP_PKGS):
-                print("adding", p)
-                pkgs.append(p)
-        n = n + 1
-
-    return sorted(pkgs)
+    newdeps = pkgs
+    print("adding...")
+    while True:
+        subdeps = frozenset(immediate_deps(list(newdeps)))
+        newdeps = subdeps - SKIP_DEP_PKGS - pkgs
+        if not newdeps:
+            break
+        print('\n'.join(newdeps))
+        pkgs |= newdeps
+
+    return list(pkgs)
 
 def gather_deps(deps, arch, directory):
 
@@ -261,7 +262,7 @@ def clean():
 
 if( args.l ):
     print("List of dependencies")
-    print( extract_deps() )
+    print( deps )
     exit(0)
 
 if args.s:
-- 
2.23.0.windows.1


--=-=-=--




Acknowledgement sent to Noam Postavsky <npostavs@HIDDEN>:
New bug report received and forwarded. Copy sent to phillip.lord@HIDDEN, bug-gnu-emacs@HIDDEN. Full text available.
Report forwarded to phillip.lord@HIDDEN, bug-gnu-emacs@HIDDEN:
bug#40628; Package emacs. Full text available.
Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.
Last modified: Fri, 12 Nov 2021 09:30:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.