Hartmut Goebel <h.goebel@HIDDEN>
to control <at> debbugs.gnu.org.
Full text available.
Received: (at submit) by debbugs.gnu.org; 16 Nov 2020 19:08:14 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Nov 16 14:08:14 2020
Received: from localhost ([127.0.0.1]:57279 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1kejrJ-0002uF-GU
for submit <at> debbugs.gnu.org; Mon, 16 Nov 2020 14:08:13 -0500
Received: from lists.gnu.org ([209.51.188.17]:47050)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <h.goebel@HIDDEN>) id 1kejrI-0002u8-Ar
for submit <at> debbugs.gnu.org; Mon, 16 Nov 2020 14:08:12 -0500
Received: from eggs.gnu.org ([2001:470:142:3::10]:46316)
by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
(Exim 4.90_1) (envelope-from <h.goebel@HIDDEN>)
id 1kejrH-0001zD-1B
for guix-patches@HIDDEN; Mon, 16 Nov 2020 14:08:12 -0500
Received: from mail-out.m-online.net ([212.18.0.9]:37904)
by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
(Exim 4.90_1) (envelope-from <h.goebel@HIDDEN>)
id 1kejrB-0001Ew-FO
for guix-patches@HIDDEN; Mon, 16 Nov 2020 14:08:10 -0500
Received: from frontend01.mail.m-online.net (unknown [192.168.8.182])
by mail-out.m-online.net (Postfix) with ESMTP id 4CZdrL3d8wz1qs3l;
Mon, 16 Nov 2020 20:08:02 +0100 (CET)
Received: from localhost (dynscan1.mnet-online.de [192.168.6.70])
by mail.m-online.net (Postfix) with ESMTP id 4CZdrL3M4Xz1qqkX;
Mon, 16 Nov 2020 20:08:02 +0100 (CET)
X-Virus-Scanned: amavisd-new at mnet-online.de
Received: from mail.mnet-online.de ([192.168.8.182])
by localhost (dynscan1.mail.m-online.net [192.168.6.70]) (amavisd-new,
port 10024)
with ESMTP id NPa1t-ZMq-AK; Mon, 16 Nov 2020 20:08:00 +0100 (CET)
Received: from hermia.goebel-consult.de
(ppp-188-174-59-253.dynamic.mnet-online.de [188.174.59.253])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(No client certificate requested)
by mail.mnet-online.de (Postfix) with ESMTPS;
Mon, 16 Nov 2020 20:08:00 +0100 (CET)
Received: from thisbe.fritz.box (thisbe.fritz.box [192.168.110.24])
by hermia.goebel-consult.de (Postfix) with ESMTP id F189060505;
Mon, 16 Nov 2020 20:08:28 +0100 (CET)
From: Hartmut Goebel <h.goebel@HIDDEN>
To: 38408 <at> debbugs.gnu.org,
guix-patches@HIDDEN
Subject: [PATCH v17 8/8] import: crate: Use existing package satisfying semver
requirement.
Date: Mon, 16 Nov 2020 20:07:28 +0100
Message-Id: <7a551f3ffbd32779fdc46f0e39198ba55c2cefbe.1605552418.git.h.goebel@HIDDEN>
X-Mailer: git-send-email 2.21.3
In-Reply-To: <cover.1605552418.git.h.goebel@HIDDEN>
References: <cover.1605552418.git.h.goebel@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Received-SPF: none client-ip=212.18.0.9;
envelope-from=h.goebel@HIDDEN; helo=mail-out.m-online.net
X-detected-operating-system: by eggs.gnu.org: First seen = 2020/11/16 14:08:02
X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy]
X-Spam_score_int: -25
X-Spam_score: -2.6
X-Spam_bar: --
X-Spam_report: (-2.6 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_LOW=-0.7,
RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001,
SPF_NONE=0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: submit
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)
If a package satisfying the dependency's semver requirement already exists,
use it. Prior to this change the highest version matching the semver
requirement was used (and imported in case it was not defined as package
already).
When resolving a dependency (now done in `sort-map-dependencies`), first
search for a package matching the semver requirement and only if this fails
reach out for a crate.
* guix/import/crate.scm (crate->guix-package)[find-package-version]: New
function. [dependency-name+version]: New function.
[sort-map-dependencies]: Use it instead of lambda function.
* tests/crate.scm (test-doctool-crate, test-doctool-dependencies): New
variables.
("self-test …", "cargo-recursive-import-hoors-existing-packages"): New
tests.
---
guix/import/crate.scm | 37 ++++++++++++++-----
tests/crate.scm | 83 +++++++++++++++++++++++++++++++++++++++++++
2 files changed, 112 insertions(+), 8 deletions(-)
diff --git a/guix/import/crate.scm b/guix/import/crate.scm
index b133529ba7..3bc261b04e 100644
--- a/guix/import/crate.scm
+++ b/guix/import/crate.scm
@@ -32,6 +32,7 @@
#:use-module (guix packages)
#:use-module (guix upstream)
#:use-module (guix utils)
+ #:use-module (gnu packages)
#:use-module (ice-9 match)
#:use-module (ice-9 regex)
#:use-module (json)
@@ -92,7 +93,7 @@
(requirement crate-dependency-requirement "req")) ;string
(module-autoload! (current-module)
- '(semver) '(string->semver semver<?))
+ '(semver) '(string->semver semver->string semver<?))
(module-autoload! (current-module)
'(semver ranges) '(string->semver-range semver-range-contains?))
@@ -235,6 +236,21 @@ look up the development dependencs for the given crate."
(or version
(crate-latest-version crate))))
+ ;; find the highest existing package that fulfills the semver <range>
+ (define (find-package-version name range)
+ (let* ((semver-range (string->semver-range range))
+ (versions
+ (sort
+ (filter (lambda (version)
+ (semver-range-contains? semver-range version))
+ (map (lambda (pkg)
+ (string->semver (package-version pkg)))
+ (find-packages-by-name
+ (crate-name->package-name name))))
+ semver<?)))
+ (and (not (null-list? versions))
+ (semver->string (last versions)))))
+
;; find the highest version of a crate that fulfills the semver <range>
(define (find-crate-version crate range)
(let* ((semver-range (string->semver-range range))
@@ -251,6 +267,17 @@ look up the development dependencs for the given crate."
(and (not (null-list? versions))
(second (last versions)))))
+ (define (dependency-name+version dep)
+ (let* ((name (crate-dependency-id dep))
+ (req (crate-dependency-requirement dep))
+ (existing-version (find-package-version name req)))
+ (if existing-version
+ (list name existing-version)
+ (let* ((crate (lookup-crate* name))
+ (ver (find-crate-version crate req)))
+ (list name
+ (crate-version-number ver))))))
+
(define version*
(and crate
(find-crate-version crate version-number)))
@@ -258,13 +285,7 @@ look up the development dependencs for the given crate."
;; sort and map the dependencies to a list containing
;; pairs of (name version)
(define (sort-map-dependencies deps)
- (sort (map (lambda (dep)
- (let* ((name (crate-dependency-id dep))
- (crate (lookup-crate* name))
- (req (crate-dependency-requirement dep))
- (ver (find-crate-version crate req)))
- (list name
- (crate-version-number ver))))
+ (sort (map dependency-name+version
deps)
(match-lambda* (((name _) ...)
(apply string-ci<? name)))))
diff --git a/tests/crate.scm b/tests/crate.scm
index 1506daeadd..a24f734093 100644
--- a/tests/crate.scm
+++ b/tests/crate.scm
@@ -25,6 +25,7 @@
#:use-module (guix build-system cargo)
#:use-module (gcrypt hash)
#:use-module (guix tests)
+ #:use-module (gnu packages)
#:use-module (ice-9 iconv)
#:use-module (ice-9 match)
#:use-module (srfi srfi-64))
@@ -312,6 +313,7 @@
\"dependencies\": []
}")
+
(define test-source-hash
"")
@@ -572,4 +574,85 @@
'(license:expat license:asl2.0)
(string->license "MIT/Apache-2.0"))
+
+
+(define test-doctool-crate
+ "{
+ \"crate\": {
+ \"max_version\": \"2.2.2\",
+ \"name\": \"leaf-bob\",
+ \"description\": \"summary\",
+ \"homepage\": \"http://example.com\",
+ \"repository\": \"http://example.com\",
+ \"keywords\": [\"dummy\", \"test\"],
+ \"categories\": [\"test\"]
+ \"actual_versions\": [
+ { \"id\": 234280,
+ \"num\": \"2.2.2\",
+ \"license\": \"MIT OR Apache-2.0\",
+ \"links\": {
+ \"dependencies\": \"/api/v1/crates/doctool/2.2.2/dependencies\"
+ }
+ }
+ ]
+ }
+}")
+
+;; FIXME: This test depends on some existing packages
+(define test-doctool-dependencies
+ "{
+ \"dependencies\": [
+ {
+ \"crate_id\": \"docopt\",
+ \"kind\": \"normal\",
+ \"req\": \"^0.8.1\"
+ }
+ ]
+}")
+
+
+(test-assert "self-test: rust-docopt 0.8.x is gone, please adjust the test case"
+ (not (null? (find-packages-by-name "rust-docopt" "0.8"))))
+
+(test-assert "cargo-recursive-import-hoors-existing-packages"
+ (mock ((guix http-client) http-fetch
+ (lambda (url . rest)
+ (match url
+ ("https://crates.io/api/v1/crates/doctool"
+ (open-input-string test-doctool-crate))
+ ("https://crates.io/api/v1/crates/doctool/2.2.2/download"
+ (set! test-source-hash
+ (bytevector->nix-base32-string
+ (sha256 (string->bytevector "empty file\n" "utf-8"))))
+ (open-input-string "empty file\n"))
+ ("https://crates.io/api/v1/crates/doctool/2.2.2/dependencies"
+ (open-input-string test-doctool-dependencies))
+ (_ (error "Unexpected URL: " url)))))
+ (match (crate-recursive-import "doctool")
+ (((define-public 'rust-doctool-2
+ (package
+ (name "rust-doctool")
+ (version "2.2.2")
+ (source
+ (origin
+ (method url-fetch)
+ (uri (crate-uri "doctool" version))
+ (file-name
+ (string-append name "-" version ".tar.gz"))
+ (sha256
+ (base32
+ (? string? hash)))))
+ (build-system cargo-build-system)
+ (arguments
+ ('quasiquote (#:cargo-inputs
+ (("rust-docopt"
+ ('unquote 'rust-docopt-0.8))))))
+ (home-page "http://example.com")
+ (synopsis "summary")
+ (description "summary")
+ (license (list license:expat license:asl2.0)))))
+ #t)
+ (x
+ (pk 'fail x #f)))))
+
(test-end "crate")
--
2.21.3
guix-patches@HIDDEN:bug#44694; Package guix-patches.
Full text available.
Received: (at submit) by debbugs.gnu.org; 16 Nov 2020 19:07:23 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Nov 16 14:07:22 2020
Received: from localhost ([127.0.0.1]:57236 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1kejqU-0002pW-Kf
for submit <at> debbugs.gnu.org; Mon, 16 Nov 2020 14:07:22 -0500
Received: from lists.gnu.org ([209.51.188.17]:45588)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <h.goebel@HIDDEN>) id 1kejqQ-0002pN-SY
for submit <at> debbugs.gnu.org; Mon, 16 Nov 2020 14:07:21 -0500
Received: from eggs.gnu.org ([2001:470:142:3::10]:46112)
by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
(Exim 4.90_1) (envelope-from <h.goebel@HIDDEN>)
id 1kejqP-0001HN-3o
for guix-patches@HIDDEN; Mon, 16 Nov 2020 14:07:18 -0500
Received: from mail-out.m-online.net ([212.18.0.10]:47951)
by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
(Exim 4.90_1) (envelope-from <h.goebel@HIDDEN>)
id 1kejqL-0000y1-KQ
for guix-patches@HIDDEN; Mon, 16 Nov 2020 14:07:16 -0500
Received: from frontend01.mail.m-online.net (unknown [192.168.8.182])
by mail-out.m-online.net (Postfix) with ESMTP id 4CZdqK3mGGz1rwDW;
Mon, 16 Nov 2020 20:07:09 +0100 (CET)
Received: from localhost (dynscan1.mnet-online.de [192.168.6.70])
by mail.m-online.net (Postfix) with ESMTP id 4CZdqK26kjz1qqkX;
Mon, 16 Nov 2020 20:07:09 +0100 (CET)
X-Virus-Scanned: amavisd-new at mnet-online.de
Received: from mail.mnet-online.de ([192.168.8.182])
by localhost (dynscan1.mail.m-online.net [192.168.6.70]) (amavisd-new,
port 10024)
with ESMTP id wlWDapa_6NIp; Mon, 16 Nov 2020 20:07:08 +0100 (CET)
Received: from hermia.goebel-consult.de
(ppp-188-174-59-253.dynamic.mnet-online.de [188.174.59.253])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(No client certificate requested)
by mail.mnet-online.de (Postfix) with ESMTPS;
Mon, 16 Nov 2020 20:07:08 +0100 (CET)
Received: from thisbe.fritz.box (thisbe.fritz.box [192.168.110.24])
by hermia.goebel-consult.de (Postfix) with ESMTP id E988960052;
Mon, 16 Nov 2020 20:08:04 +0100 (CET)
From: Hartmut Goebel <h.goebel@HIDDEN>
To: guix-patches@HIDDEN,
38408 <at> debbugs.gnu.org
Subject: [PATCH v17 0/8] New take continued: Semantic version aware recursive
Date: Mon, 16 Nov 2020 20:07:05 +0100
Message-Id: <cover.1605552418.git.h.goebel@HIDDEN>
X-Mailer: git-send-email 2.21.3
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Received-SPF: none client-ip=212.18.0.10;
envelope-from=h.goebel@HIDDEN; helo=mail-out.m-online.net
X-detected-operating-system: by eggs.gnu.org: First seen = 2020/11/16 14:07:09
X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy]
X-Spam_score_int: -25
X-Spam_score: -2.6
X-Spam_bar: --
X-Spam_report: (-2.6 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_LOW=-0.7,
RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001,
SPF_NONE=0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: submit
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)
This is another revision of the patch set. Major changes compared to v16 are:
* When searching dependencies, prefer an existing package satisfying
semver requirement over importing the highest version create.
This saves adding a lot of new packages. As an example: When importing
sequoia-openpgp@HIDDEN this only imports 19 crates now, compared to 96
using the former method.
* Package names are not trimmed to the first no-zero part.
* Test cases have been improved so packages have different versions. THis
actually showed some bubs I solved (details see below).
Notable bug-fixes:
* Change selection of package version: use the highest version matching the
required range instead of first one.
* Sort dependencies by name (was version)
Some details about the improved test cases
* Change crate names to avoid possible confusion
* Use different version (not 1.0.0 for all)
* Add some versions to some of the crates to test selecting the version.
* ids of create version entries are numbers.
* Document crate versions and dependencies used in tests
* Actually define some dependencies using caret semver.
Hartmut Goebel (3):
import: utils: Trim patch version from names.
import: crate: Trim version for names after left-most non-zero part.
import: crate: Use existing package satisfying semver requirement.
Martin Becze (5):
guix: self: Add guile-semver as a depenedency.
import: utils: 'recursive-import' accepts an optional version
parameter.
import: crate: Use guile-semver to resolve module versions.
import: crate: Memorize crate->guix-package.
import: crate: Parameterized importing of dev dependencies.
guix/import/cran.scm | 8 +-
guix/import/crate.scm | 150 +++++++--
guix/import/elpa.scm | 6 +-
guix/import/gem.scm | 6 +-
guix/import/opam.scm | 8 +-
guix/import/pypi.scm | 8 +-
guix/import/stackage.scm | 5 +-
guix/import/utils.scm | 84 +++--
guix/scripts/import/cran.scm | 5 +-
guix/scripts/import/crate.scm | 13 +-
guix/scripts/import/elpa.scm | 4 +-
guix/self.scm | 8 +-
tests/crate.scm | 581 +++++++++++++++++++++++-----------
tests/elpa.scm | 3 +-
tests/import-utils.scm | 8 +-
15 files changed, 608 insertions(+), 289 deletions(-)
--
2.21.3
Hartmut Goebel <h.goebel@HIDDEN>:guix-patches@HIDDEN.
Full text available.guix-patches@HIDDEN:bug#44694; Package guix-patches.
Full text available.
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997 nCipher Corporation Ltd,
1994-97 Ian Jackson.