GNU bug report logs - #44030
guix import pypi foo@1.2.3 breaks

Previous Next

Package: guix;

Reported by: zimoun <zimon.toutoune <at> gmail.com>

Date: Fri, 16 Oct 2020 13:49:01 UTC

Severity: normal

Tags: easy

Done: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>

Bug is archived. No further changes may be made.

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 44030 in the body.
You can then email your comments to 44030 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to bug-guix <at> gnu.org:
bug#44030; Package guix. (Fri, 16 Oct 2020 13:49:02 GMT) Full text and rfc822 format available.

Acknowledgement sent to zimoun <zimon.toutoune <at> gmail.com>:
New bug report received and forwarded. Copy sent to bug-guix <at> gnu.org. (Fri, 16 Oct 2020 13:49:02 GMT) Full text and rfc822 format available.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: zimoun <zimon.toutoune <at> gmail.com>
To: bug-guix <at> gnu.org
Subject: guix import pypi foo <at> 1.2.3 breaks
Date: Fri, 16 Oct 2020 15:48:19 +0200
Dear,

The syntax across the different importers are not uniform.  Especially,
compare:

   guix import hackage mtl <at> 2.1.3.1
   guix import pypi itsdangerous <at> 1.1.0

and worse, the option ’--recursive’ leads to an error for the latter.
Note that instead:

   guix import pypi itsdangerous/1.1.0

perfectly works, even the option ’--recursive’.

All the information is there, and the UI has to be fixed.  It is a
perfect first contribution fixes.


All the best,
simon

PS:
Reported by Zelphir Kaltstahl.
<https://lists.gnu.org/archive/html/help-guix/2020-10/msg00055.html>







Added tag(s) easy. Request was from zimoun <zimon.toutoune <at> gmail.com> to control <at> debbugs.gnu.org. (Fri, 16 Oct 2020 13:52:02 GMT) Full text and rfc822 format available.

Information forwarded to bug-guix <at> gnu.org:
bug#44030; Package guix. (Sun, 25 Oct 2020 23:29:03 GMT) Full text and rfc822 format available.

Message #10 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Lulu <me <at> erkin.party>
To: "guix-patches <at> gnu.org" <guix-patches <at> gnu.org>
Subject: [bug#44030] [PATCH] guix: import: Add versioning syntax to pypi
 importer.
Date: Sun, 25 Oct 2020 22:09:41 +0300 (TRT)
Use @ as the unified versioning syntax, as per crate and
hackage importers, plus minor spacing fixes.


diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm
index 15116e349d..add2a7ed7a 100644
--- a/guix/import/pypi.scm
+++ b/guix/import/pypi.scm
@@ -118,13 +118,15 @@
 
 (define (pypi-fetch name)
   "Return a <pypi-project> record for package NAME, or #f on failure."
-  (and=> (json-fetch (string-append "https://pypi.org/pypi/" name "/json"))
-         json->pypi-project))
+  ;; Convert @ in package name to / to access the correct URL.
+  (let ((versioned-name (string-join (string-split name #\@) "/")))
+    (and=> (json-fetch (string-append "https://pypi.org/pypi/" name "/json"))
+           json->pypi-project)))
 
 ;; For packages found on PyPI that lack a source distribution.
 (define-condition-type &missing-source-error &error
   missing-source-error?
-  (package  missing-source-error-package))
+  (package missing-source-error-package))
 
 (define (latest-source-release pypi-package)
   "Return the latest source release for PYPI-PACKAGE."
@@ -371,7 +373,7 @@ be extracted in a temporary directory."
                  (invoke "tar" "xf" archive "-C" dir)))
            (let ((requires.txt-files
                   (find-files dir (lambda (abs-file-name _)
-		                    (string-match "\\.egg-info/requires.txt$"
+                                    (string-match "\\.egg-info/requires.txt$"
                                                   abs-file-name)))))
              (match requires.txt-files
                (()




Information forwarded to bug-guix <at> gnu.org:
bug#44030; Package guix. (Sun, 25 Oct 2020 23:29:03 GMT) Full text and rfc822 format available.

Message #13 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Lulu <me <at> erkin.party>
To: "guix-patches <at> gnu.org" <guix-patches <at> gnu.org>
Subject: Re: [bug#44030] [PATCH] guix: import: Add versioning syntax to pypi
 importer.
Date: Sun, 25 Oct 2020 23:04:32 +0300 (TRT)
Ah, heck, I used an old revision of the file when I generated the diff.
Sorry about that. Here's the properly working patch:


diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm
index 15116e349d..1ec1ecbfa1 100644
--- a/guix/import/pypi.scm
+++ b/guix/import/pypi.scm
@@ -118,13 +118,15 @@
 
 (define (pypi-fetch name)
   "Return a <pypi-project> record for package NAME, or #f on failure."
-  (and=> (json-fetch (string-append "https://pypi.org/pypi/" name "/json"))
-         json->pypi-project))
+  ;; Convert @ in package name to / to access the correct URL.
+  (let ((versioned-name (string-join (string-split name #\@) "/")))
+    (and=> (json-fetch (string-append "https://pypi.org/pypi/" versioned-name "/json"))
+           json->pypi-project)))
 
 ;; For packages found on PyPI that lack a source distribution.
 (define-condition-type &missing-source-error &error
   missing-source-error?
-  (package  missing-source-error-package))
+  (package missing-source-error-package))
 
 (define (latest-source-release pypi-package)
   "Return the latest source release for PYPI-PACKAGE."
@@ -371,7 +373,7 @@ be extracted in a temporary directory."
                  (invoke "tar" "xf" archive "-C" dir)))
            (let ((requires.txt-files
                   (find-files dir (lambda (abs-file-name _)
-		                    (string-match "\\.egg-info/requires.txt$"
+                                    (string-match "\\.egg-info/requires.txt$"
                                                   abs-file-name)))))
              (match requires.txt-files
                (()




Information forwarded to bug-guix <at> gnu.org:
bug#44030; Package guix. (Sun, 25 Oct 2020 23:29:04 GMT) Full text and rfc822 format available.

Message #16 received at 44030 <at> debbugs.gnu.org (full text, mbox):

From: Lulu <me <at> erkin.party>
To: "44030 <at> debbugs.gnu.org" <44030 <at> debbugs.gnu.org>
Subject: guix import pypi foo <at> 1.2.3 breaks
Date: Sun, 25 Oct 2020 23:50:32 +0300 (TRT)
I just sent in a patch to fix this for the pypi importer, although
ideally we'd want versioning support for all importers with a uniform
syntax.

I poked and prodded at the RubyGems API to see if it can do what PyPI
can. It's unfortunately much more limited: API v1 provides a method
for querying all versions of a gem [1] and v2 provides a method for
querying specific versions of a gem [2], although neither of them
provides information about dependency versions, so we can't rely on it
for recursive imports (since it would try to import latest versions of
all dependencies). There's a method that fetches dependency and
versioning info for all versions of all given gems but (probably due
to the verbosity of JSON/YAML) it's in binary Ruby serialisation
format [3]. I can try to hack a parser that surgically extracts
dependency info from a given gem version. What do you think?

ELPA runs into a similar problem in that it provides tarballs/files
for older versions but dependency info is only provided in the repo
file. (MELPA tries to directly peg all packages to their respective
Git repos' trunks.)

I presume it's much simpler with the gnu importer, as it's only a
matter of pointing the FTP fetch in the right direction, although I
couldn't confirm it, as the gnu importer doesn't work for me since my
ISP blocks PGP keyserver ports.

I need to take a closer look at CRAN, CPAN, TeXLive and opam.

--
Lulu

[1]: https://guides.rubygems.org/rubygems-org-api/#gem-version-methods
[2]: https://guides.rubygems.org/rubygems-org-api-v2/
[3]: https://guides.rubygems.org/rubygems-org-api/#misc-methods




Information forwarded to bug-guix <at> gnu.org:
bug#44030; Package guix. (Mon, 26 Oct 2020 15:50:01 GMT) Full text and rfc822 format available.

Message #19 received at submit <at> debbugs.gnu.org (full text, mbox):

From: zimoun <zimon.toutoune <at> gmail.com>
To: Lulu <me <at> erkin.party>, "guix-patches <at> gnu.org" <guix-patches <at> gnu.org>
Subject: Re: bug#44030: [PATCH] guix: import: Add versioning syntax to pypi
 importer.
Date: Mon, 26 Oct 2020 16:49:14 +0100
Hi,

Thank you for working on this.

On Sun, 25 Oct 2020 at 23:04, Lulu <me <at> erkin.party> wrote:
> Ah, heck, I used an old revision of the file when I generated the diff.
> Sorry about that. Here's the properly working patch:

That’s fine.  Usually to ease the reading, it is a good habit to
increase the version of the patch, (see the option ’reroll-count’ of
git-format-patch). 


However, the 2 patches are not proper commit, right?  You can locally
commit your changes and send them.  Basically, it means the same thing
but including the commit message.  Here it looks like:

--8<---------------cut here---------------start------------->8---
import: pypi: Allow ’@’ for specifying the version.

Fixes <https://bugs.gnu.org/44030>.

* guix/import/pypi.json (pypi-fetch): Allow ’@’ for specifying the
  version.
--8<---------------cut here---------------end--------------->8---

Well, I let you find more inspiration in previous commit messages. ;-)


> diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm
> index 15116e349d..1ec1ecbfa1 100644
> --- a/guix/import/pypi.scm
> +++ b/guix/import/pypi.scm
> @@ -118,13 +118,15 @@
>  
>  (define (pypi-fetch name)
>    "Return a <pypi-project> record for package NAME, or #f on failure."
> -  (and=> (json-fetch (string-append "https://pypi.org/pypi/" name "/json"))
> -         json->pypi-project))
> +  ;; Convert @ in package name to / to access the correct URL.
> +  (let ((versioned-name (string-join (string-split name #\@) "/")))
> +    (and=> (json-fetch (string-append "https://pypi.org/pypi/" versioned-name "/json"))
> +           json->pypi-project)))

If you feel adventurous, you could try to add a test in the file
“tests/pypi.scm”. :-)

>  ;; For packages found on PyPI that lack a source distribution.
>  (define-condition-type &missing-source-error &error
>    missing-source-error?
> -  (package  missing-source-error-package))
> +  (package missing-source-error-package))

Why is this line modified?


>  (define (latest-source-release pypi-package)
>    "Return the latest source release for PYPI-PACKAGE."
> @@ -371,7 +373,7 @@ be extracted in a temporary directory."
>                   (invoke "tar" "xf" archive "-C" dir)))
>             (let ((requires.txt-files
>                    (find-files dir (lambda (abs-file-name _)
> -		                    (string-match "\\.egg-info/requires.txt$"
> +                                    (string-match "\\.egg-info/requires.txt$"

Idem.


All the best,
simon





Information forwarded to bug-guix <at> gnu.org:
bug#44030; Package guix. (Mon, 26 Oct 2020 16:00:02 GMT) Full text and rfc822 format available.

Message #22 received at 44030 <at> debbugs.gnu.org (full text, mbox):

From: zimoun <zimon.toutoune <at> gmail.com>
To: Lulu <me <at> erkin.party>, "44030 <at> debbugs.gnu.org" <44030 <at> debbugs.gnu.org>
Subject: Re: bug#44030: guix import pypi foo <at> 1.2.3 breaks
Date: Mon, 26 Oct 2020 16:58:58 +0100
Hi,

On Sun, 25 Oct 2020 at 23:50, Lulu <me <at> erkin.party> wrote:
> I just sent in a patch to fix this for the pypi importer, although
> ideally we'd want versioning support for all importers with a uniform
> syntax.

The uniform syntax is ’@’.  However, it is importer by importer (case by
case) since the external API is different.  AFAIU.


Speaking about “guix import pypi”, the “--recursive” options still
ungracefully fails; even with you patch.


> I poked and prodded at the RubyGems API to see if it can do what PyPI
> can. It's unfortunately much more limited: API v1 provides a method
> for querying all versions of a gem [1] and v2 provides a method for
> querying specific versions of a gem [2], although neither of them
> provides information about dependency versions, so we can't rely on it
> for recursive imports (since it would try to import latest versions of
> all dependencies). There's a method that fetches dependency and
> versioning info for all versions of all given gems but (probably due
> to the verbosity of JSON/YAML) it's in binary Ruby serialisation
> format [3]. I can try to hack a parser that surgically extracts
> dependency info from a given gem version. What do you think?
>
> ELPA runs into a similar problem in that it provides tarballs/files
> for older versions but dependency info is only provided in the repo
> file. (MELPA tries to directly peg all packages to their respective
> Git repos' trunks.)
>
> I presume it's much simpler with the gnu importer, as it's only a
> matter of pointing the FTP fetch in the right direction, although I
> couldn't confirm it, as the gnu importer doesn't work for me since my
> ISP blocks PGP keyserver ports.
>
> I need to take a closer look at CRAN, CPAN, TeXLive and opam.

Hum?  You are discussing this bug [1], right?  So please could you
discuss overthere.

1: <http://issues.guix.gnu.org/issue/44115>


All the best,
simon




Information forwarded to bug-guix <at> gnu.org:
bug#44030; Package guix. (Wed, 28 Oct 2020 19:33:01 GMT) Full text and rfc822 format available.

Message #25 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Lulu <me <at> erkin.party>
To: "guix-patches <at> gnu.org" <guix-patches <at> gnu.org>
Subject: [bug#44030] [PATCH] import: pypi: Add '@' syntax for specifying the
 package version.
Date: Wed, 28 Oct 2020 22:32:27 +0300 (TRT)
* guix/import/pypi.scm (pypi-fetch): Add ’@’ syntax for specifying the package version.
---
  guix/import/pypi.scm | 6 ++++--
  1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm
index 15116e349d..559be4a98b 100644
--- a/guix/import/pypi.scm
+++ b/guix/import/pypi.scm
@@ -118,8 +118,10 @@
  
  (define (pypi-fetch name)
     "Return a <pypi-project>  record for package NAME, or #f on failure."
-   (and=>  (json-fetch (string-append "https://pypi.org/pypi/"  name "/json"))
-             json->pypi-project))
+   ;; Convert @ in package name to / to access the correct URL.
+   (let ((versioned-name (string-join (string-split name #\@) "/")))
+      (and=>  (json-fetch (string-append "https://pypi.org/pypi/"  versioned-name "/json"))
+                json->pypi-project)))
  
  ;; For packages found on PyPI that lack a source distribution.
  (define-condition-type &missing-source-error &error
-- 
2.29.1




Information forwarded to bug-guix <at> gnu.org:
bug#44030; Package guix. (Thu, 29 Oct 2020 22:10:02 GMT) Full text and rfc822 format available.

Message #28 received at 44030 <at> debbugs.gnu.org (full text, mbox):

From: Lulu <me <at> erkin.party>
To: "44030 <at> debbugs.gnu.org" <44030 <at> debbugs.gnu.org>
Subject: guix import pypi foo <at> 1.2.3 breaks
Date: Fri, 30 Oct 2020 01:09:40 +0300 (TRT)
Okay, I tested this a bit and it seems to work fine. I'm going to write formal
tests for it (and other importers, if possible) later.

--
Lulu




Information forwarded to bug-guix <at> gnu.org:
bug#44030; Package guix. (Mon, 02 Nov 2020 12:16:02 GMT) Full text and rfc822 format available.

Message #31 received at 44030 <at> debbugs.gnu.org (full text, mbox):

From: zimoun <zimon.toutoune <at> gmail.com>
To: Lulu <me <at> erkin.party>
Cc: "44030 <at> debbugs.gnu.org" <44030 <at> debbugs.gnu.org>
Subject: Re: bug#44030: guix import pypi foo <at> 1.2.3 breaks
Date: Mon, 02 Nov 2020 13:15:50 +0100
Hi,

On Fri, 30 Oct 2020 at 01:09, Lulu <me <at> erkin.party> wrote:
> Okay, I tested this a bit and it seems to work fine. I'm going to write formal
> tests for it (and other importers, if possible) later.

Usually, we try to use the option ’--reroll-count’ of git-format-patch
for the second (or third or more) version.  It helps the review to know
– from the subject – which is the email to read, etc.  Well, next
time. :-)


What do you mean by write “formal tests”?


All the best,
simon





Reply sent to Maxim Cournoyer <maxim.cournoyer <at> gmail.com>:
You have taken responsibility. (Sun, 21 Nov 2021 06:15:02 GMT) Full text and rfc822 format available.

Notification sent to zimoun <zimon.toutoune <at> gmail.com>:
bug acknowledged by developer. (Sun, 21 Nov 2021 06:15:02 GMT) Full text and rfc822 format available.

Message #36 received at 44030-done <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: zimoun <zimon.toutoune <at> gmail.com>
Cc: 44030-done <at> debbugs.gnu.org
Subject: Re: bug#44030: guix import pypi foo <at> 1.2.3 breaks
Date: Sun, 21 Nov 2021 01:14:01 -0500
Hi,

zimoun <zimon.toutoune <at> gmail.com> writes:

> Dear,
>
> The syntax across the different importers are not uniform.  Especially,
> compare:
>
>    guix import hackage mtl <at> 2.1.3.1
>    guix import pypi itsdangerous <at> 1.1.0
>
> and worse, the option ’--recursive’ leads to an error for the latter.
> Note that instead:
>
>    guix import pypi itsdangerous/1.1.0
>
> perfectly works, even the option ’--recursive’.
>
> All the information is there, and the UI has to be fixed.  It is a
> perfect first contribution fixes.
>
>
> All the best,
> simon

This works now:

$ ./pre-inst-env guix import pypi ansible <at> 4.4.0
Starting download of /tmp/guix-file.mtcWzL
From https://files.pythonhosted.org/packages/b5/01/dd6bf3cb7d834c7493d10fa1f0720c34c7703fc9bf12c93f294def465bb0/ansible-4.4.0.tar.gz...
 …4.0.tar.gz  33.7MiB                49.2MiB/s 00:01 [##################] 100.0%
(package
  (name "python-ansible")
  (version "4.4.0")
  (source
    (origin
      (method url-fetch)
      (uri (pypi-uri "ansible" version))
      (sha256
        (base32 "031n22j0lsmh69x6i6gkva81j68b4yzh1pbg3q2h4bknl85q46ag"))))
  (build-system python-build-system)
  (propagated-inputs (list python-ansible-core))
  (home-page "https://ansible.com/")
  (synopsis "Radically simple IT automation")
  (description "Radically simple IT automation")
  (license #f))

$ ./pre-inst-env guix import pypi ansible <at> 4.4.1
guix import: error: no source release for pypi package ansible 4.4.1

Closing!

Maxim




Information forwarded to bug-guix <at> gnu.org:
bug#44030; Package guix. (Mon, 22 Nov 2021 09:08:01 GMT) Full text and rfc822 format available.

Message #39 received at 44030 <at> debbugs.gnu.org (full text, mbox):

From: zimoun <zimon.toutoune <at> gmail.com>
To: 44030 <at> debbugs.gnu.org, <maxim.cournoyer <at> gmail.com>
Subject: Re: bug#44030: guix import pypi foo <at> 1.2.3 breaks
Date: Mon, 22 Nov 2021 10:07:09 +0100
Hi Maxim,

On Sun, 21 Nov 2021 at 07:14, Maxim Cournoyer <maxim.cournoyer <at> gmail.com> wrote:
> zimoun <zimon.toutoune <at> gmail.com> writes:

>> The syntax across the different importers are not uniform.  Especially,
>> compare:
>>
>>    guix import hackage mtl <at> 2.1.3.1
>>    guix import pypi itsdangerous <at> 1.1.0
>>
>> and worse, the option ’--recursive’ leads to an error for the latter.

Yes, this is indeed fixed.


> > Note that instead:
> >
> >    guix import pypi itsdangerous/1.1.0
> >
> > perfectly works, even the option ’--recursive’.

Not this one.  Well, except if we consider it is a feature. ;-)



Cheers,
simon




bug archived. Request was from Debbugs Internal Request <help-debbugs <at> gnu.org> to internal_control <at> debbugs.gnu.org. (Mon, 20 Dec 2021 12:24:10 GMT) Full text and rfc822 format available.

This bug report was last modified 2 years and 89 days ago.

Previous Next


GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.