GNU bug report logs - #50854
[PATCH 0/2] Make the PyPI importer honor the ‘upstream-name’ package property

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 50854 in the body.
You can then email your comments to 50854 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Xinglu Chen <public <at> yoctocell.xyz>
To: guix-patches <at> gnu.org
Subject: [PATCH 0/2] Make the PyPI importer honor the ‘upstream-name’ package property
Date: Mon, 27 Sep 2021 22:09:21 +0200

[Message part 1 (text/plain, inline)]

If a PyPI package contains a “-” followed by one or more digits, e.g.,
“AV-98”, the PyPI importer/updater will think that the digits after the
“-” is the version of the package, and instead of trying to update
“AV-98” it will try to update “AV”.

--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env guix refresh av-98
following redirection to `https://pypi.org/pypi/av/json'...
/home/yoctocell/src/guix/gnu/packages/web-browsers.scm:914:13: av-98 would be upgraded from 1.0.1 to 8.0.3
--8<---------------cut here---------------end--------------->8---

The first patch makes the PyPI importer honor the ‘upstream-name’
property; it will also generate a package with an ‘upstream-name’
property if the package name contains a “-” followed by one or more
digits.

The second patch sets the ‘upstream-name’ property for the ‘av-98’
package.

Xinglu Chen (2):
  import: pypi: Honor the 'upstream-name' package property.
  gnu: av-98: Set 'upstream-name' property.

 gnu/packages/web-browsers.scm |   2 +
 guix/import/pypi.scm          |  20 +++++--
 tests/pypi.scm                | 106 +++++++++++++++++++++++++++++++---
 3 files changed, 115 insertions(+), 13 deletions(-)


base-commit: 6ae4644984608b7eff7ab54d3a5787c661d85b2e
-- 
2.33.0

[signature.asc (application/pgp-signature, inline)]

Message #8 received at 50854 <at> debbugs.gnu.org (full text, mbox):

From: Xinglu Chen <public <at> yoctocell.xyz>
To: 50854 <at> debbugs.gnu.org
Subject: [PATCH 1/2] import: pypi: Honor the 'upstream-name' package property.
Date: Mon, 27 Sep 2021 22:11:20 +0200

Previously, when a PyPI package had a “-” followed by one or more digits in
its name, e.g., “AV-98”, the importer would interpret “98” as the version of
the package and thus mistake the “AV-98” package for the “av” package on PyPI.

  $ ./pre-inst-env guix refresh av-98
  following redirection to `https://pypi.org/pypi/av/json'...
  /home/yoctocell/src/guix/gnu/packages/web-browsers.scm:914:13: av-98 would be upgraded from 1.0.1 to 8.0.3

Setting the ‘upstream-name’ property to “AV-98” would solve the problem.

  $ ./pre-inst-env guix refresh av-98
  /home/yoctocell/src/guix/gnu/packages/web-browsers.scm:914:13: 1.0.1 is already the latest version of av-98

* guix/import/pypi.scm (guix-package->pypi-name): Honor ‘upstream-name’
property.
(make-pypi-sexp): Set ‘upstream-name’ property when appropriate.
* tests/pypi.scm (test-json): Rename to ...
(test-json-1): ... this.
(test-json-2): New variable
("guix-package->pypi-name, honor 'upstream-name'"): New test.
("pypi->guix-package, package name contains \"-\" followed by digits"):
Likewise.
---
 guix/import/pypi.scm |  20 +++++---
 tests/pypi.scm       | 106 ++++++++++++++++++++++++++++++++++++++++---
 2 files changed, 113 insertions(+), 13 deletions(-)

diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm
index 6731d50891..b7859c8341 100644
--- a/guix/import/pypi.scm
+++ b/guix/import/pypi.scm
@@ -9,6 +9,7 @@
 ;;; Copyright © 2020 Lars-Dominik Braun <ldb <at> leibniz-psychology.org>
 ;;; Copyright © 2020 Arun Isaac <arunisaac <at> systemreboot.net>
 ;;; Copyright © 2020 Martin Becze <mjbecze <at> riseup.net>
+;;; Copyright © 2021 Xinglu Chen <public <at> yoctocell.xyz>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -163,12 +164,13 @@ (define (url->pypi-name url)
     (hyphen-package-name->name+version
      (basename (file-sans-extension url))))
 
-  (match (and=> (package-source package) origin-uri)
-    ((? string? url)
-     (url->pypi-name url))
-    ((lst ...)
-     (any url->pypi-name lst))
-    (#f #f)))
+  (or (assoc-ref (package-properties package) 'upstream-name)
+      (match (and=> (package-source package) origin-uri)
+        ((? string? url)
+         (url->pypi-name url))
+        ((lst ...)
+         (any url->pypi-name lst))
+        (#f #f))))
 
 (define (wheel-url->extracted-directory wheel-url)
   (match (string-split (basename wheel-url) #\-)
@@ -423,6 +425,11 @@ (define (make-pypi-sexp name version source-url wheel-url home-page synopsis
                         description license)
   "Return the `package' s-expression for a python package with the given NAME,
 VERSION, SOURCE-URL, HOME-PAGE, SYNOPSIS, DESCRIPTION, and LICENSE."
+  (define (maybe-upstream-name name)
+    (if (string-match ".*\\-[0-9]+" (pk name))
+        `((properties ,`'(("upstream-name" . ,name))))
+        '()))
+  
   (call-with-temporary-output-file
    (lambda (temp port)
      (and (url-fetch source-url temp)
@@ -461,6 +468,7 @@ (define (make-pypi-sexp name version source-url wheel-url home-page synopsis
                       (sha256
                        (base32
                         ,(guix-hash-url temp)))))
+                   ,@(maybe-upstream-name name)
                    (build-system python-build-system)
                    ,@(maybe-inputs required-inputs 'propagated-inputs)
                    ,@(maybe-inputs native-inputs 'native-inputs)
diff --git a/tests/pypi.scm b/tests/pypi.scm
index f421d6d9df..70f4298a90 100644
--- a/tests/pypi.scm
+++ b/tests/pypi.scm
@@ -2,6 +2,7 @@
 ;;; Copyright © 2014 David Thompson <davet <at> gnu.org>
 ;;; Copyright © 2016 Ricardo Wurmus <rekado <at> elephly.net>
 ;;; Copyright © 2019 Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
+;;; Copyright © 2021 Xinglu Chen <public <at> yoctocell.xyz>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -29,7 +30,7 @@ (define-module (test-pypi)
   #:use-module (srfi srfi-64)
   #:use-module (ice-9 match))
 
-(define test-json
+(define test-json-1
   "{
   \"info\": {
     \"version\": \"1.0.0\",
@@ -57,6 +58,34 @@ (define test-json
   }
 }")
 
+(define test-json-2
+  "{
+  \"info\": {
+    \"version\": \"1.0.0\",
+    \"name\": \"foo-99\",
+    \"license\": \"GNU LGPL\",
+    \"summary\": \"summary\",
+    \"home_page\": \"http://example.com\",
+    \"classifiers\": [],
+    \"download_url\": \"\"
+  },
+  \"urls\": [],
+  \"releases\": {
+    \"1.0.0\": [
+      {
+        \"url\": \"https://example.com/foo-99-1.0.0.egg\",
+        \"packagetype\": \"bdist_egg\"
+      }, {
+        \"url\": \"https://example.com/foo-99-1.0.0.tar.gz\",
+        \"packagetype\": \"sdist\"
+      }, {
+        \"url\": \"https://example.com/foo-99-1.0.0-py2.py3-none-any.whl\",
+        \"packagetype\": \"bdist_wheel\"
+      }
+    ]
+  }
+}")
+
 (define test-source-hash
   "")
 
@@ -147,6 +176,13 @@ (define test-metadata-with-extras-jedi "\
                     (uri (list "https://bitheap.org/cram/cram-0.7.tar.gz"
                                (pypi-uri "cram" "0.7"))))))))
 
+(test-equal "guix-package->pypi-name, honor 'upstream-name'"
+  "bar-3"
+  (guix-package->pypi-name
+   (dummy-package "foo"
+                  (properties
+                   '((upstream-name . "bar-3"))))))
+
 (test-equal "specification->requirement-name"
   '("Fizzy" "PickyThing" "SomethingWithMarker" "requests" "pip")
   (map specification->requirement-name test-specifications))
@@ -198,8 +234,8 @@ (define test-metadata-with-extras-jedi "\
                  (lambda (url . rest)
                    (match url
                      ("https://pypi.org/pypi/foo/json"
-                      (values (open-input-string test-json)
-                              (string-length test-json)))
+                      (values (open-input-string test-json-1)
+                              (string-length test-json-1)))
                      ("https://example.com/foo-1.0.0-py2.py3-none-any.whl" #f)
                      (_ (error "Unexpected URL: " url)))))
                 (match (pypi->guix-package "foo")
@@ -264,8 +300,8 @@ (define test-metadata-with-extras-jedi "\
                (lambda (url . rest)
                  (match url
                    ("https://pypi.org/pypi/foo/json"
-                    (values (open-input-string test-json)
-                            (string-length test-json)))
+                    (values (open-input-string test-json-1)
+                            (string-length test-json-1)))
                    ("https://example.com/foo-1.0.0-py2.py3-none-any.whl" #f)
                    (_ (error "Unexpected URL: " url)))))
               ;; Not clearing the memoization cache here would mean returning the value
@@ -317,8 +353,8 @@ (define test-metadata-with-extras-jedi "\
                (lambda (url . rest)
                  (match url
                    ("https://pypi.org/pypi/foo/json"
-                    (values (open-input-string test-json)
-                            (string-length test-json)))
+                    (values (open-input-string test-json-1)
+                            (string-length test-json-1)))
                    ("https://example.com/foo-1.0.0-py2.py3-none-any.whl" #f)
                    (_ (error "Unexpected URL: " url)))))
               ;; Not clearing the memoization cache here would mean returning the value
@@ -345,4 +381,60 @@ (define test-metadata-with-extras-jedi "\
                 (x
                  (pk 'fail x #f))))))
 
+(test-assert "pypi->guix-package, package name contains \"-\" followed by digits"
+  ;; Replace network resources with sample data.
+  (mock ((guix import utils) url-fetch
+         (lambda (url file-name)
+           (match url
+             ("https://example.com/foo-99-1.0.0.tar.gz"
+              (begin
+                ;; Unusual requires.txt location should still be found.
+                (mkdir-p "foo-99-1.0.0/src/bizarre.egg-info")
+                (with-output-to-file "foo-99-1.0.0/src/bizarre.egg-info/requires.txt"
+                  (lambda ()
+                    (display test-requires.txt)))
+                (parameterize ((current-output-port (%make-void-port "rw+")))
+                  (system* "tar" "czvf" file-name "foo-99-1.0.0/"))
+                (delete-file-recursively "foo-99-1.0.0")
+                (set! test-source-hash
+                  (call-with-input-file file-name port-sha256))))
+             ("https://example.com/foo-99-1.0.0-py2.py3-none-any.whl" #f)
+             (_ (error "Unexpected URL: " url)))))
+        (mock ((guix http-client) http-fetch
+               (lambda (url . rest)
+                 (match url
+                   ("https://pypi.org/pypi/foo-99/json"
+                    (values (open-input-string test-json-2)
+                            (string-length test-json-2)))
+                   ("https://example.com/foo-99-1.0.0-py2.py3-none-any.whl" #f)
+                   (_ (error "Unexpected URL: " url)))))
+              (match (pypi->guix-package "foo-99")
+                (('package
+                   ('name "python-foo-99")
+                   ('version "1.0.0")
+                   ('source ('origin
+                              ('method 'url-fetch)
+                              ('uri ('pypi-uri "foo-99" 'version))
+                              ('sha256
+                               ('base32
+                                (? string? hash)))))
+                   ('properties ('quote (("upstream-name" . "foo-99"))))
+                   ('build-system 'python-build-system)
+                   ('propagated-inputs
+                    ('quasiquote
+                     (("python-bar" ('unquote 'python-bar))
+                      ("python-foo" ('unquote 'python-foo)))))
+                   ('native-inputs
+                    ('quasiquote
+                     (("python-pytest" ('unquote 'python-pytest)))))
+                   ('home-page "http://example.com")
+                   ('synopsis "summary")
+                   ('description "summary")
+                   ('license 'license:lgpl2.0))
+                 (string=? (bytevector->nix-base32-string
+                            test-source-hash)
+                           hash))
+                (x
+                 (pk 'fail x #f))))))
+
 (test-end "pypi")
-- 
2.33.0

Message #11 received at 50854 <at> debbugs.gnu.org (full text, mbox):

From: Xinglu Chen <public <at> yoctocell.xyz>
To: 50854 <at> debbugs.gnu.org
Subject: [PATCH 2/2] gnu: av-98: Set 'upstream-name' property.
Date: Mon, 27 Sep 2021 22:11:26 +0200

* gnu/packages/web-browsers.scm (av-98)[properties]: Set ‘upstream-name’.
---
 gnu/packages/web-browsers.scm | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/gnu/packages/web-browsers.scm b/gnu/packages/web-browsers.scm
index f0aefa3eb4..596d9f7f2b 100644
--- a/gnu/packages/web-browsers.scm
+++ b/gnu/packages/web-browsers.scm
@@ -912,6 +912,8 @@ (define-public av-98
   (package
     (name "av-98")
     (version "1.0.1")
+    (properties
+     '((upstream-name . "AV-98")))
     (source
      (origin
        (method url-fetch)
-- 
2.33.0

Message #16 received at 50854-done <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Xinglu Chen <public <at> yoctocell.xyz>
Cc: 50854-done <at> debbugs.gnu.org
Subject: Re: bug#50854: [PATCH 0/2] Make the PyPI importer honor the
 ‘upstream-name’ package property
Date: Thu, 30 Sep 2021 23:24:12 +0200

Xinglu Chen <public <at> yoctocell.xyz> skribis:

> If a PyPI package contains a “-” followed by one or more digits, e.g.,
> “AV-98”, the PyPI importer/updater will think that the digits after the
> “-” is the version of the package, and instead of trying to update
> “AV-98” it will try to update “AV”.
>
> $ ./pre-inst-env guix refresh av-98
> following redirection to `https://pypi.org/pypi/av/json'...
> /home/yoctocell/src/guix/gnu/packages/web-browsers.scm:914:13: av-98 would be upgraded from 1.0.1 to 8.0.3
>
> The first patch makes the PyPI importer honor the ‘upstream-name’
> property; it will also generate a package with an ‘upstream-name’
> property if the package name contains a “-” followed by one or more
> digits.
>
> The second patch sets the ‘upstream-name’ property for the ‘av-98’
> package.

Nice.

> Xinglu Chen (2):
>   import: pypi: Honor the 'upstream-name' package property.
>   gnu: av-98: Set 'upstream-name' property.

Applied, thanks!

Ludo'.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.