X-Loop: help-debbugs@HIDDEN
Subject: [bug#59078] [PATCH] lint: Split the derivation lint checker by system.
Resent-From: Christopher Baines <mail@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: guix-patches@HIDDEN
Resent-Date: Sun, 06 Nov 2022 13:56:02 +0000
Resent-Message-ID: <handler.59078.B.166774294618285 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: report 59078
X-GNU-PR-Package: guix-patches
X-GNU-PR-Keywords: patch
To: 59078 <at> debbugs.gnu.org
X-Debbugs-Original-To: guix-patches@HIDDEN
Received: via spool by submit <at> debbugs.gnu.org id=B.166774294618285
(code B ref -1); Sun, 06 Nov 2022 13:56:02 +0000
Received: (at submit) by debbugs.gnu.org; 6 Nov 2022 13:55:46 +0000
Received: from localhost ([127.0.0.1]:58966 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1org7l-0004kr-3O
for submit <at> debbugs.gnu.org; Sun, 06 Nov 2022 08:55:45 -0500
Received: from lists.gnu.org ([209.51.188.17]:46312)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <mail@HIDDEN>) id 1org7i-0004kh-0G
for submit <at> debbugs.gnu.org; Sun, 06 Nov 2022 08:55:42 -0500
Received: from eggs.gnu.org ([2001:470:142:3::10])
by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
(Exim 4.90_1) (envelope-from <mail@HIDDEN>) id 1org7h-000693-Qk
for guix-patches@HIDDEN; Sun, 06 Nov 2022 08:55:41 -0500
Received: from mira.cbaines.net ([212.71.252.8])
by eggs.gnu.org with esmtp (Exim 4.90_1)
(envelope-from <mail@HIDDEN>) id 1org7f-0003fv-BX
for guix-patches@HIDDEN; Sun, 06 Nov 2022 08:55:41 -0500
Received: from localhost (host-82-184-251-30.business.telecomitalia.it
[82.184.251.30])
by mira.cbaines.net (Postfix) with ESMTPSA id EDA3827BBE9
for <guix-patches@HIDDEN>; Sun, 6 Nov 2022 13:55:35 +0000 (GMT)
Received: from localhost (localhost [local])
by localhost (OpenSMTPD) with ESMTPA id 61b65099
for <guix-patches@HIDDEN>; Sun, 6 Nov 2022 13:55:32 +0000 (UTC)
From: Christopher Baines <mail@HIDDEN>
Date: Sun, 6 Nov 2022 14:55:32 +0100
Message-Id: <20221106135532.5724-1-mail@HIDDEN>
X-Mailer: git-send-email 2.37.3
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Received-SPF: pass client-ip=212.71.252.8; envelope-from=mail@HIDDEN;
helo=mira.cbaines.net
X-Spam_score_int: -18
X-Spam_score: -1.9
X-Spam_bar: -
X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001,
SPF_PASS=-0.001, UNPARSEABLE_RELAY=0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-Spam-Score: -1.4 (-)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -2.4 (--)
Currently, if you attempt to run the derivation checker on all packages, the
Guile process will run out of memory. I think a contributing factor to this is
that the checker forces an inefficient order when you want to generate
derivations for all the supported systems of each package, constantly
switching system then package.
This problem also impacts the Guix Data Service, since it tries to run the
derivation checker for all packages.
The changes in this commit to split the derivation lint checker in to several,
one for each system, means that you can now treat each system separately,
which should be better for caching purposes.
If it's desirable to keep some notion of checking all supported systems for a
single package, I think lint checker groups could be added, so that you could
ask for the "derivation" checker, and this would run all the derivation
checkers.
* guix/lint.scm (check-derivation): Adapt to make-check-derivation-for-system.
(%derivation-checkers): New variable.
(%local-checkers): Include all %derivation-checkers.
* doc/guix.texi (Invoking guix lint): Update.
---
doc/guix.texi | 4 +-
guix/lint.scm | 139 +++++++++++++++++++++++++++++---------------------
2 files changed, 83 insertions(+), 60 deletions(-)
diff --git a/doc/guix.texi b/doc/guix.texi
index 7806b21a0f..8d4989a60c 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -14422,9 +14422,9 @@ Parse the @code{source} URL to determine if a tarball from GitHub is
autogenerated or if it is a release tarball. Unfortunately GitHub's
autogenerated tarballs are sometimes regenerated.
-@item derivation
+@item derivation/SYSTEM
Check that the derivation of the given packages can be successfully
-computed for all the supported systems (@pxref{Derivations}).
+computed for the specified system (@pxref{Derivations}).
@item profile-collisions
Check whether installing the given packages in a profile would lead to
diff --git a/guix/lint.scm b/guix/lint.scm
index 8e3976171f..f692856f42 100644
--- a/guix/lint.scm
+++ b/guix/lint.scm
@@ -52,6 +52,7 @@ (define-module (guix lint)
#:use-module (guix memoization)
#:use-module (guix profiles)
#:use-module (guix monads)
+ #:use-module (guix platform)
#:use-module (guix scripts)
#:use-module ((guix ui) #:select (texi->plain-text fill-paragraph))
#:use-module (guix gnu-maintenance)
@@ -98,7 +99,6 @@ (define-module (guix lint)
check-patch-file-names
check-patch-headers
check-synopsis-style
- check-derivation
check-home-page
check-name
check-source
@@ -116,6 +116,8 @@ (define-module (guix lint)
check-haskell-stackage
check-tests-true
+ make-check-derivation-for-system
+
lint-warning
lint-warning?
lint-warning-package
@@ -1369,56 +1371,6 @@ (define (check-phases-deltas deltas)
(append-map check-phases-delta deltas))
(find-phase-deltas package check-phases-deltas))
-(define* (check-derivation package #:key store)
- "Emit a warning if we fail to compile PACKAGE to a derivation."
- (define (try store system)
- (guard (c ((store-protocol-error? c)
- (make-warning package
- (G_ "failed to create ~a derivation: ~a")
- (list system
- (store-protocol-error-message c))))
- ((exception-with-kind-and-args? c)
- (make-warning package
- (G_ "failed to create ~a derivation: ~s")
- (list system
- (cons (exception-kind c)
- (exception-args c)))))
- ((message-condition? c)
- (make-warning package
- (G_ "failed to create ~a derivation: ~a")
- (list system
- (condition-message c))))
- ((formatted-message? c)
- (let ((str (apply format #f
- (formatted-message-string c)
- (formatted-message-arguments c))))
- (make-warning package
- (G_ "failed to create ~a derivation: ~a")
- (list system str))))
- (else
- (make-warning package
- (G_ "failed to create ~a derivation: ~a")
- (list system c))))
- (parameterize ((%graft? #f))
- (package-derivation store package system #:graft? #f)
-
- ;; If there's a replacement, make sure we can compute its
- ;; derivation.
- (match (package-replacement package)
- (#f #t)
- (replacement
- (package-derivation store replacement system
- #:graft? #f))))))
-
- (define (check-with-store store)
- (filter lint-warning?
- (map (cut try store <>) (package-supported-systems package))))
-
- ;; For backwards compatability, don't rely on store being set
- (or (and=> store check-with-store)
- (with-store store
- (check-with-store store))))
-
(define* (check-profile-collisions package #:key store)
"Check for collisions that would occur when installing PACKAGE as a result
of the propagated inputs it pulls in."
@@ -1843,13 +1795,88 @@ (define (check-formatting package)
(G_ "source file not found"))))))))
'())))
+(define (make-check-derivation-for-system system)
+ (define (try package proc)
+ (guard (c ((store-protocol-error? c)
+ (make-warning package
+ (G_ "failed to create ~a derivation: ~a")
+ (list system
+ (store-protocol-error-message c))))
+ ((exception-with-kind-and-args? c)
+ (make-warning package
+ (G_ "failed to create ~a derivation: ~s")
+ (list system
+ (cons (exception-kind c)
+ (exception-args c)))))
+ ((message-condition? c)
+ (make-warning package
+ (G_ "failed to create ~a derivation: ~a")
+ (list system
+ (condition-message c))))
+ ((formatted-message? c)
+ (let ((str (apply format #f
+ (formatted-message-string c)
+ (formatted-message-arguments c))))
+ (make-warning package
+ (G_ "failed to create ~a derivation: ~a")
+ (list system str))))
+ (else
+ (make-warning package
+ (G_ "failed to create ~a derivation: ~a")
+ (list system c))))
+ (proc)))
+
+
+
+ (lambda* (package #:key store)
+ "Emit a warning if we fail to compile PACKAGE to a derivation."
+
+ (define (check-with-store store)
+ (if (member system (package-supported-systems package))
+ (filter
+ lint-warning?
+ (map (cut try package <>)
+ (list
+ (lambda ()
+ (parameterize ((%graft? #f))
+ (package-derivation store package system #:graft? #f)))
+ (lambda ()
+ ;; If there's a replacement, make sure we can compute its
+ ;; derivation.
+ (match (package-replacement package)
+ (#f #t)
+ (replacement
+ (parameterize ((%graft? #f))
+ (package-derivation store replacement system
+ #:graft? #f))))))))
+ '()))
+
+ ;; For backwards compatability, don't rely on store being set
+ (or (and=> store check-with-store)
+ (with-store store
+ (check-with-store store)))))
+
;;;
;;; List of checkers.
;;;
+(define %derivation-checkers
+ (map (lambda (system)
+ (lint-checker
+ (name (string->symbol
+ (simple-format #f "derivation/~A" system)))
+ (description
+ (simple-format
+ #f
+ "Report failure to compile a package to a derivation for ~A"
+ system))
+ (check (make-check-derivation-for-system system))
+ (requires-store? #t)))
+ (systems)))
+
(define %local-checkers
- (list
+ (cons*
(lint-checker
(name 'name)
(description "Validate package names")
@@ -1901,11 +1928,6 @@ (define %local-checkers
(name 'source-unstable-tarball)
(description "Check for autogenerated tarballs")
(check check-source-unstable-tarball))
- (lint-checker
- (name 'derivation)
- (description "Report failure to compile a package to a derivation")
- (check check-derivation)
- (requires-store? #t))
(lint-checker
(name 'profile-collisions)
(description "Report collisions that would occur due to propagated inputs")
@@ -1922,7 +1944,8 @@ (define %local-checkers
(lint-checker
(name 'formatting)
(description "Look for formatting issues in the source")
- (check check-formatting))))
+ (check check-formatting))
+ %derivation-checkers))
(define %network-dependent-checkers
(list
--
2.37.3
Content-Disposition: inline Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Mailer: MIME-tools 5.505 (Entity 5.505) Content-Type: text/plain; charset=utf-8 X-Loop: help-debbugs@HIDDEN From: help-debbugs@HIDDEN (GNU bug Tracking System) To: Christopher Baines <mail@HIDDEN> Subject: bug#59078: Acknowledgement ([PATCH] lint: Split the derivation lint checker by system.) Message-ID: <handler.59078.B.166774294618285.ack <at> debbugs.gnu.org> References: <20221106135532.5724-1-mail@HIDDEN> X-Gnu-PR-Message: ack 59078 X-Gnu-PR-Package: guix-patches X-Gnu-PR-Keywords: patch Reply-To: 59078 <at> debbugs.gnu.org Date: Sun, 06 Nov 2022 13:56:02 +0000 Thank you for filing a new bug report with debbugs.gnu.org. This is an automatically generated reply to let you know your message has been received. Your message is being forwarded to the package maintainers and other interested parties for their attention; they will reply in due course. Your message has been sent to the package maintainer(s): guix-patches@HIDDEN If you wish to submit further information on this problem, please send it to 59078 <at> debbugs.gnu.org. Please do not send mail to help-debbugs@HIDDEN unless you wish to report a problem with the Bug-tracking system. --=20 59078: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D59078 GNU Bug Tracking System Contact help-debbugs@HIDDEN with problems
X-Loop: help-debbugs@HIDDEN
Subject: [bug#59078] [PATCH] lint: Split the derivation lint checker by system.
Resent-From: Christopher Baines <mail@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: guix-patches@HIDDEN
Resent-Date: Mon, 07 Nov 2022 17:47:01 +0000
Resent-Message-ID: <handler.59078.B59078.166784321911294 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 59078
X-GNU-PR-Package: guix-patches
X-GNU-PR-Keywords: patch
To: Christopher Baines <mail@HIDDEN>
Cc: 59078 <at> debbugs.gnu.org
X-Debbugs-Original-Cc: 59078 <at> debbugs.gnu.org, guix-patches@HIDDEN
Received: via spool by 59078-submit <at> debbugs.gnu.org id=B59078.166784321911294
(code B ref 59078); Mon, 07 Nov 2022 17:47:01 +0000
Received: (at 59078) by debbugs.gnu.org; 7 Nov 2022 17:46:59 +0000
Received: from localhost ([127.0.0.1]:35274 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1os6D5-0002w6-Du
for submit <at> debbugs.gnu.org; Mon, 07 Nov 2022 12:46:59 -0500
Received: from mira.cbaines.net ([212.71.252.8]:41800)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <mail@HIDDEN>) id 1os6D3-0002vx-E1
for 59078 <at> debbugs.gnu.org; Mon, 07 Nov 2022 12:46:57 -0500
Received: from localhost (93-42-6-17.ip84.fastwebnet.it [93.42.6.17])
by mira.cbaines.net (Postfix) with ESMTPSA id D3C1027BBE9;
Mon, 7 Nov 2022 17:46:55 +0000 (GMT)
Received: from felis (localhost [127.0.0.1])
by localhost (OpenSMTPD) with ESMTP id a456c9c1;
Mon, 7 Nov 2022 17:46:54 +0000 (UTC)
References: <20221106135532.5724-1-mail@HIDDEN>
User-agent: mu4e 1.8.9; emacs 28.1
From: Christopher Baines <mail@HIDDEN>
Date: Mon, 07 Nov 2022 18:37:32 +0100
In-reply-to: <20221106135532.5724-1-mail@HIDDEN>
Message-ID: <87cz9ypshf.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="=-=-=";
micalg=pgp-sha512; protocol="application/pgp-signature"
X-Spam-Score: -0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)
--=-=-=
Content-Type: text/plain
Christopher Baines <mail@HIDDEN> writes:
> This problem also impacts the Guix Data Service, since it tries to run the
> derivation checker for all packages.
This patch has now been processed by qa.guix.gnu.org. Looking at the
logs for the Guix Data Service processing the base and target revision,
and the change is more significant than I'd imagined:
Base:
inferior heap after cleanup: 1739.0 MiB used (5160.0 MiB heap)
debug: Finished getting formatting lint warnings, took 349 seconds
debug: Finished fetching inferior lint warnings, took 3782 seconds
Target:
inferior heap after cleanup: 1152.0 MiB used (1778.0 MiB heap)
debug: Finished getting derivation/aarch64-linux lint warnings, took 334 seconds
debug: Finished fetching inferior lint warnings, took 3285 seconds
So with the changes, it's a little faster, but the main difference is
that the heap ~3GiB smaller, so ~34% of what it was previously.
I did notice that this also subtly differs from how the linter behaved
previously, since some packages define support for systems not defined
through the platform module.
https://data.qa.guix.gnu.org/compare?base_commit=a60dc46c2bb5de196858594b72b00d5f86ca7e98&target_commit=4e152714f55337015991e62e51e8dea15e889b9f
Personally, I think this change is still a good one. Maybe we can add a
separate linter to go round and check that packages don't declare
support for systems that aren't in the platform module.
Unless anyone objects, I'll like to push this sooner rather than later,
as I think the excessive heap size in the inferior process is not ideal.
Thanks,
Chris
--=-=-=
Content-Type: application/pgp-signature; name="signature.asc"
-----BEGIN PGP SIGNATURE-----
iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmNpRIxfFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF
ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh
aW5lcy5uZXQACgkQXiijOwuE9XflUg//bzvSQ+wbaAL3FO9EQz6zGmVeDOgqbkEe
rX+iCqLk0ZvYAzTMUYmz+z55ozfJChkhB5TwA71s68hOKZQFEvkI6lwUylaryy4a
ovkttrcn6hEtr5E0KglToltSj5F8IHlQ3BT4uc0re2/zWIfrsFUEbjI2HLH99xyA
CKxH1ENy0GboVZBH8SHYRLTTWefMRkMUCq6c2hWpZeOSYHuqf4Gk6KXc6oEln6js
FBkFeH9N1V2607/KakrCY851zevbjXc5LD+0OXXN6ALR9qXzrb+gp00U/5gJ54zd
pNlzZSanRC8R+LM3bAKOQxe1womVv3/QIczJCTclXL7NTVT/5a3WYlrbIldoPMhN
ET2XVjRzHz4WheprtTzkm/F2JdcbFPNsra9K+uDzRHX2AH98JzklDhq58AzsxF+s
hXge0BCv3rsJpjgQHtp8sW26VdyBssuZYMqEE9XbX+Jzvr++DtN4rgDu2nXWJOX/
RMns3KiK6U5yP3Gq6dI8YCtdxreHPbAPjfH829H5/gPCqE4wDcyinb1YCCAwsy7K
4ELkFjg/L7CEFfiuB5N+wuGq2bpxF2YNwWMs5QMq/tlGGSSXkGVWDwsaxKS+Y0XI
GBzO77Hsjxm1fNpuBowP0ZaQKYnBtEwl5gbUYkeAnIDBqgdZECcGPTGOEh/jEQFN
KQljNj7kxtE=
=ALR6
-----END PGP SIGNATURE-----
--=-=-=--
X-Loop: help-debbugs@HIDDEN
Subject: [bug#59078] [PATCH] lint: Split the derivation lint checker by system.
Resent-From: Christopher Baines <mail@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: guix-patches@HIDDEN
Resent-Date: Mon, 07 Nov 2022 17:48:02 +0000
Resent-Message-ID: <handler.59078.B.166784323711350 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 59078
X-GNU-PR-Package: guix-patches
X-GNU-PR-Keywords: patch
To: Christopher Baines <mail@HIDDEN>
Cc: 59078 <at> debbugs.gnu.org
X-Debbugs-Original-Cc: 59078 <at> debbugs.gnu.org, guix-patches@HIDDEN
Received: via spool by submit <at> debbugs.gnu.org id=B.166784323711350
(code B ref -1); Mon, 07 Nov 2022 17:48:02 +0000
Received: (at submit) by debbugs.gnu.org; 7 Nov 2022 17:47:17 +0000
Received: from localhost ([127.0.0.1]:35278 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1os6DM-0002x0-RZ
for submit <at> debbugs.gnu.org; Mon, 07 Nov 2022 12:47:17 -0500
Received: from lists.gnu.org ([209.51.188.17]:47796)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <mail@HIDDEN>) id 1os6DK-0002ws-S0
for submit <at> debbugs.gnu.org; Mon, 07 Nov 2022 12:47:15 -0500
Received: from eggs.gnu.org ([2001:470:142:3::10])
by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
(Exim 4.90_1) (envelope-from <mail@HIDDEN>) id 1os6DD-0004AL-CJ
for guix-patches@HIDDEN; Mon, 07 Nov 2022 12:47:12 -0500
Received: from mira.cbaines.net ([212.71.252.8])
by eggs.gnu.org with esmtp (Exim 4.90_1)
(envelope-from <mail@HIDDEN>) id 1os6DA-0008Rg-I1
for guix-patches@HIDDEN; Mon, 07 Nov 2022 12:47:07 -0500
Received: from localhost (93-42-6-17.ip84.fastwebnet.it [93.42.6.17])
by mira.cbaines.net (Postfix) with ESMTPSA id D3C1027BBE9;
Mon, 7 Nov 2022 17:46:55 +0000 (GMT)
Received: from felis (localhost [127.0.0.1])
by localhost (OpenSMTPD) with ESMTP id a456c9c1;
Mon, 7 Nov 2022 17:46:54 +0000 (UTC)
References: <20221106135532.5724-1-mail@HIDDEN>
User-agent: mu4e 1.8.9; emacs 28.1
From: Christopher Baines <mail@HIDDEN>
Date: Mon, 07 Nov 2022 18:37:32 +0100
In-reply-to: <20221106135532.5724-1-mail@HIDDEN>
Message-ID: <87cz9ypshf.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="=-=-=";
micalg=pgp-sha512; protocol="application/pgp-signature"
Received-SPF: pass client-ip=212.71.252.8; envelope-from=mail@HIDDEN;
helo=mira.cbaines.net
X-Spam_score_int: -18
X-Spam_score: -1.9
X-Spam_bar: -
X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001,
SPF_PASS=-0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-Spam-Score: -1.4 (-)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -2.4 (--)
--=-=-=
Content-Type: text/plain
Christopher Baines <mail@HIDDEN> writes:
> This problem also impacts the Guix Data Service, since it tries to run the
> derivation checker for all packages.
This patch has now been processed by qa.guix.gnu.org. Looking at the
logs for the Guix Data Service processing the base and target revision,
and the change is more significant than I'd imagined:
Base:
inferior heap after cleanup: 1739.0 MiB used (5160.0 MiB heap)
debug: Finished getting formatting lint warnings, took 349 seconds
debug: Finished fetching inferior lint warnings, took 3782 seconds
Target:
inferior heap after cleanup: 1152.0 MiB used (1778.0 MiB heap)
debug: Finished getting derivation/aarch64-linux lint warnings, took 334 seconds
debug: Finished fetching inferior lint warnings, took 3285 seconds
So with the changes, it's a little faster, but the main difference is
that the heap ~3GiB smaller, so ~34% of what it was previously.
I did notice that this also subtly differs from how the linter behaved
previously, since some packages define support for systems not defined
through the platform module.
https://data.qa.guix.gnu.org/compare?base_commit=a60dc46c2bb5de196858594b72b00d5f86ca7e98&target_commit=4e152714f55337015991e62e51e8dea15e889b9f
Personally, I think this change is still a good one. Maybe we can add a
separate linter to go round and check that packages don't declare
support for systems that aren't in the platform module.
Unless anyone objects, I'll like to push this sooner rather than later,
as I think the excessive heap size in the inferior process is not ideal.
Thanks,
Chris
--=-=-=
Content-Type: application/pgp-signature; name="signature.asc"
-----BEGIN PGP SIGNATURE-----
iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmNpRIxfFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF
ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh
aW5lcy5uZXQACgkQXiijOwuE9XflUg//bzvSQ+wbaAL3FO9EQz6zGmVeDOgqbkEe
rX+iCqLk0ZvYAzTMUYmz+z55ozfJChkhB5TwA71s68hOKZQFEvkI6lwUylaryy4a
ovkttrcn6hEtr5E0KglToltSj5F8IHlQ3BT4uc0re2/zWIfrsFUEbjI2HLH99xyA
CKxH1ENy0GboVZBH8SHYRLTTWefMRkMUCq6c2hWpZeOSYHuqf4Gk6KXc6oEln6js
FBkFeH9N1V2607/KakrCY851zevbjXc5LD+0OXXN6ALR9qXzrb+gp00U/5gJ54zd
pNlzZSanRC8R+LM3bAKOQxe1womVv3/QIczJCTclXL7NTVT/5a3WYlrbIldoPMhN
ET2XVjRzHz4WheprtTzkm/F2JdcbFPNsra9K+uDzRHX2AH98JzklDhq58AzsxF+s
hXge0BCv3rsJpjgQHtp8sW26VdyBssuZYMqEE9XbX+Jzvr++DtN4rgDu2nXWJOX/
RMns3KiK6U5yP3Gq6dI8YCtdxreHPbAPjfH829H5/gPCqE4wDcyinb1YCCAwsy7K
4ELkFjg/L7CEFfiuB5N+wuGq2bpxF2YNwWMs5QMq/tlGGSSXkGVWDwsaxKS+Y0XI
GBzO77Hsjxm1fNpuBowP0ZaQKYnBtEwl5gbUYkeAnIDBqgdZECcGPTGOEh/jEQFN
KQljNj7kxtE=
=ALR6
-----END PGP SIGNATURE-----
--=-=-=--
X-Loop: help-debbugs@HIDDEN
Subject: [bug#59078] [PATCH] lint: Split the derivation lint checker by system.
Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: guix-patches@HIDDEN
Resent-Date: Fri, 11 Nov 2022 21:58:02 +0000
Resent-Message-ID: <handler.59078.B59078.166820386812514 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 59078
X-GNU-PR-Package: guix-patches
X-GNU-PR-Keywords: patch
To: Christopher Baines <mail@HIDDEN>
Cc: 59078 <at> debbugs.gnu.org
Received: via spool by 59078-submit <at> debbugs.gnu.org id=B59078.166820386812514
(code B ref 59078); Fri, 11 Nov 2022 21:58:02 +0000
Received: (at 59078) by debbugs.gnu.org; 11 Nov 2022 21:57:48 +0000
Received: from localhost ([127.0.0.1]:46914 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1otc20-0003Fm-3G
for submit <at> debbugs.gnu.org; Fri, 11 Nov 2022 16:57:48 -0500
Received: from eggs.gnu.org ([209.51.188.92]:32934)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <ludo@HIDDEN>) id 1otc1x-0003FY-Mb
for 59078 <at> debbugs.gnu.org; Fri, 11 Nov 2022 16:57:46 -0500
Received: from fencepost.gnu.org ([2001:470:142:3::e])
by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
(Exim 4.90_1) (envelope-from <ludo@HIDDEN>)
id 1otc1s-0000f9-Fd; Fri, 11 Nov 2022 16:57:40 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To:
From; bh=cuVrb0nyfLdB1+jaQibg2Rr8Wzu0A8bSFxThvMvT2nc=; b=BAFpAcm0760ubEHe91NV
dvQN8JFS+pGM0LYBU6Nx1mqrd+zKiJdP4P+AWGJ5Wql78UCQt7W6SRdscROqhyiRtTi1HhL91bM0d
FeSU6Cva/s9hUnHEYtC8G4I2QTl7kCT8xYyPQsX5piUl5ri9NYm9oBqXoyeVRFjNX678Hce7Upxs+
mtVvXbhm+Zb722mjqBpwIEXMYfg8DTNY58MicdPClTubqxMssGFcIQanWr5ea1o11CNiMGRRjZved
XgC8KsDVRhAioyZzB0vXCsDVEQtrSBUIPeQDprBrhAwFtMiP4+8Mbtg1xv9u8UEc5xXMVuqK1wTpY
PHxCCljAdHgG5g==;
Received: from 91-160-117-201.subs.proxad.net ([91.160.117.201] helo=ribbon)
by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
(Exim 4.90_1) (envelope-from <ludo@HIDDEN>)
id 1otc1q-0007vv-MM; Fri, 11 Nov 2022 16:57:40 -0500
From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
References: <20221106135532.5724-1-mail@HIDDEN>
Date: Fri, 11 Nov 2022 22:57:36 +0100
In-Reply-To: <20221106135532.5724-1-mail@HIDDEN> (Christopher Baines's
message of "Sun, 6 Nov 2022 14:55:32 +0100")
Message-ID: <87k041dui7.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="=-=-="
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)
--=-=-=
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Hi,
Christopher Baines <mail@HIDDEN> skribis:
> Currently, if you attempt to run the derivation checker on all packages, =
the
> Guile process will run out of memory. I think a contributing factor to th=
is is
> that the checker forces an inefficient order when you want to generate
> derivations for all the supported systems of each package, constantly
> switching system then package.
>
> This problem also impacts the Guix Data Service, since it tries to run the
> derivation checker for all packages.
>
> The changes in this commit to split the derivation lint checker in to sev=
eral,
> one for each system, means that you can now treat each system separately,
> which should be better for caching purposes.
>
> If it's desirable to keep some notion of checking all supported systems f=
or a
> single package, I think lint checker groups could be added, so that you c=
ould
> ask for the "derivation" checker, and this would run all the derivation
> checkers.
The =E2=80=98derivation=E2=80=99 checker was added for this purpose: making=
sure that a
package=E2=80=99s derivation can be computed for all the supported systems.
Previously it was easy to overlook that kind of breakage.
I think it=E2=80=99s important to keep a =E2=80=98derivation=E2=80=99 check=
er that does this.
Now, the memory consumption you report is unacceptable and this needs to
be addressed.
Most (all?) caches are now per-session (associated with
<store-connection>). Since =E2=80=98guix lint=E2=80=99 uses a single sessi=
on, those
caches keep growing because there=E2=80=99s no eviction mechanism in place.
A hack like the one below should work around that. Could you check how
well it works for you? It can also be helpful to set:
GUIX_PROFILING=3Dgc GUIX_PROFILING_EVENTS=3Dafter-gc
Longer-term we should have a proper cache eviction mechanism in place.
It=E2=80=99s not been a priority because =E2=80=9Cnormal=E2=80=9D applicati=
ons such as =E2=80=98guix
package=E2=80=99 don=E2=80=99t need it.
Thanks,
Ludo=E2=80=99.
--=-=-=
Content-Type: text/x-patch
Content-Disposition: inline
diff --git a/guix/scripts/lint.scm b/guix/scripts/lint.scm
index 9920c3ee62..6d9b9e96a9 100644
--- a/guix/scripts/lint.scm
+++ b/guix/scripts/lint.scm
@@ -209,23 +209,31 @@ (define (parse-options)
(list-checkers-and-exit checkers))
(with-error-handling
- (let ((any-lint-checker-requires-store?
- (any lint-checker-requires-store? checkers)))
+ (let ((store-needed? (any lint-checker-requires-store? checkers)))
- (define (call-maybe-with-store proc)
- (if any-lint-checker-requires-store?
- (with-store store
- (proc store))
- (proc #f)))
-
- (call-maybe-with-store
- (lambda (store)
- (cond
- ((null? args)
- (fold-packages (lambda (p r) (run-checkers p checkers
- #:store store)) '()))
- (else
- (for-each (lambda (package)
- (run-checkers package checkers
- #:store store))
- args)))))))))
+ (cond
+ ((null? args)
+ (let loop ((packages (fold-packages cons '()))
+ (processed 0)
+ (store #f))
+ (match packages
+ ((package . rest)
+ (let ((store (and store-needed?
+ (if store
+ ;; XXX: Periodically start a new session
+ ;; with empty caches to avoid unbounded
+ ;; heap growth caused by the 'derivation'
+ ;; checker.
+ (if (zero? (modulo processed 1000))
+ (begin
+ (close-connection store)
+ (open-connection))
+ store)
+ (open-connection)))))
+ (run-checkers package checkers #:store store)
+ (loop rest (+ 1 processed) store))))))
+ (else
+ (for-each (lambda (package)
+ (run-checkers package checkers
+ #:store store))
+ args)))))))
--=-=-=--
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997 nCipher Corporation Ltd,
1994-97 Ian Jackson.