GNU bug report logs - #47746
pypi importer generates wrong pypi-uri without underscore

Previous Next

To reply to this bug, email your comments to 47746 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Ricardo Wurmus <rekado <at> elephly.net>
To: bug-guix <at> gnu.org
Subject: pypi importer generates wrong pypi-uri without underscore
Date: Tue, 13 Apr 2021 10:17:06 +0200

When importing a package from Pypi whose name includes an 
underscore, the importer will generate a pypi-uri expression where 
the underscore has been replaced with a dash.

-- 
Ricardo

Message #8 received at 47746 <at> debbugs.gnu.org (full text, mbox):

From: Xinglu Chen <public <at> yoctocell.xyz>
To: Ricardo Wurmus <rekado <at> elephly.net>, 47746 <at> debbugs.gnu.org
Subject: Re: bug#47746: pypi importer generates wrong pypi-uri without
 underscore
Date: Tue, 13 Apr 2021 20:54:56 +0200

On Tue, Apr 13 2021, Ricardo Wurmus wrote:

> When importing a package from Pypi whose name includes an 
> underscore, the importer will generate a pypi-uri expression where 
> the underscore has been replaced with a dash.

Hmm, I just grabbed a random package from PyPi, and the URI seems fine
to me.

--8<---------------cut here---------------start------------->8---
~/src/guix $ ./pre-inst-env guix import pypi angreal_data_science_project
Starting download of /tmp/guix-file.4MH43L
From
https://files.pythonhosted.org/packages/28/5a/6c77f8f5c2333ab2d3e83c27615358a0badcfeb6513a36079ba0fee78524/angreal_data_science_project-0.5.5.tar.gz...
 …_project-0.5.5.tar.gz  15KiB        378KiB/s 00:00 [##################] 100.0%

Starting download of /tmp/guix-file.hrC4cL
From
https://files.pythonhosted.org/packages/f4/18/e4747e2f3232ca1fbbd1286455fe57f3390a94b14c26a37dba72eb829523/angreal_data_science_project-0.5.5-py3-none-any.whl...
 …ect-0.5.5-py3-none-any.whl  28KiB   244KiB/s 00:00 [##################] 100.0%
(package
  (name "python-angreal-data-science-project")
  (version "0.5.5")
  (source
    (origin
      (method url-fetch)
      (uri (pypi-uri "angreal_data_science_project" version))
      (sha256
        (base32
          "1krvhwmlmv77j8w3sxaxlkmhd0xzmr24f4n873p00631lacwmpck"))))
  (build-system python-build-system)
  (home-page
    "https://gitlab.com/angreal/data-science-project")
  (synopsis
    "An angreal template for data science projects.")
  (description
    "An angreal template for data science projects.")
  (license #f))
--8<---------------cut here---------------end--------------->8---

How would I reproduce the bug?

Message #11 received at 47746 <at> debbugs.gnu.org (full text, mbox):

From: Ricardo Wurmus <rekado <at> elephly.net>
To: Xinglu Chen <public <at> yoctocell.xyz>
Cc: 47746 <at> debbugs.gnu.org
Subject: Re: bug#47746: pypi importer generates wrong pypi-uri without
 underscore
Date: Tue, 13 Apr 2021 23:39:32 +0200

Xinglu Chen <public <at> yoctocell.xyz> writes:

> On Tue, Apr 13 2021, Ricardo Wurmus wrote:
>
>> When importing a package from Pypi whose name includes an 
>> underscore, the importer will generate a pypi-uri expression 
>> where 
>> the underscore has been replaced with a dash.
>
> Hmm, I just grabbed a random package from PyPi, and the URI 
> seems fine
> to me. […]
> How would I reproduce the bug?

You’re right, this only affects some packages.  Here are two 
examples:

--8<---------------cut here---------------start------------->8---
./pre-inst-env guix import pypi -r jupyterlab_widgets
following redirection to 
`https://pypi.org/pypi/jupyterlab-widgets/json'...
[…] (uri (pypi-uri "jupyterlab-widgets" version)) […]
--8<---------------cut here---------------end--------------->8---

--8<---------------cut here---------------start------------->8---
./pre-inst-env guix import pypi -r jupyter_packaging
following redirection to 
`https://pypi.org/pypi/jupyter-packaging/json'...
[…] (uri (pypi-uri "jupyter-packaging" version)) […]
--8<---------------cut here---------------end--------------->8---

They both have a redirection in common, but the pypi-uri is 
incorrect as it needs to have the underscore.

-- 
Ricardo

Message #14 received at 47746 <at> debbugs.gnu.org (full text, mbox):

From: Xinglu Chen <public <at> yoctocell.xyz>
To: Ricardo Wurmus <rekado <at> elephly.net>
Cc: 47746 <at> debbugs.gnu.org
Subject: Re: bug#47746: pypi importer generates wrong pypi-uri without
 underscore
Date: Wed, 14 Apr 2021 14:45:11 +0200

[Message part 1 (text/plain, inline)]

On Tue, Apr 13 2021, Ricardo Wurmus wrote:

> You’re right, this only affects some packages.  Here are two 
> examples:
>
> [...]
>
> They both have a redirection in common, but the pypi-uri is 
> incorrect as it needs to have the underscore.

Looking at the JSON object at
<https://pypi.org/pypi/jupyterlab-widgets/json>, it specifies
“jupyterlabs-widgets” as ‘info.downloads.name’, which is what the PyPi
importer reads to determine the PyPi URI.

Would it make sense to just make it read the name that the user
specifies in the shell, i.e. ‘./pre-inst-env guix import pypi
PACKAGE-NAME’ (see the patch below)?  Or will this potentially mess up
other PyPi URIs?

[0001-import-pypi-Use-package-name-from-the-shell-to-const.patch (text/x-patch, inline)]

From ec5612e9d4c2026541874c6a3eaf7c9009ce4a31 Mon Sep 17 00:00:00 2001
Message-Id: <ec5612e9d4c2026541874c6a3eaf7c9009ce4a31.1618404153.git.public <at> yoctocell.xyz>
From: Xinglu Chen <public <at> yoctocell.xyz>
Date: Wed, 14 Apr 2021 14:37:01 +0200
Subject: [PATCH] import: pypi: Use package name from the shell to construct
 the URI.

Some packages will list an incorrect package name in the JSON object,
resulting in PyPi URIs that lead to nowhere.

* guix/import/pypi.scm (pypi->guix-package): Use the package name specified in
the shell (guix import pypi PACKAGE-NAME) to construct the PyPi URI.

Fixes <https://issues.guix.gnu.org/47746>.
Reported by Ricardo Wurmus <rekado <at> elephly.net>.
---
 guix/import/pypi.scm | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm
index bf4dc50138..be9470ad26 100644
--- a/guix/import/pypi.scm
+++ b/guix/import/pypi.scm
@@ -9,6 +9,7 @@
 ;;; Copyright © 2020 Lars-Dominik Braun <ldb <at> leibniz-psychology.org>
 ;;; Copyright © 2020 Arun Isaac <arunisaac <at> systemreboot.net>
 ;;; Copyright © 2020 Martin Becze <mjbecze <at> riseup.net>
+;;; Copyright © 2021 Xinglu Chen <public <at> yoctocell.xyz>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -483,7 +484,7 @@ VERSION, SOURCE-URL, HOME-PAGE, SYNOPSIS, DESCRIPTION, and LICENSE."
                          (leave (G_ "no source release for pypi package ~a ~a~%")
                                 (project-info-name info)
                                 (project-info-version info)))))
-              (make-pypi-sexp (project-info-name info)
+              (make-pypi-sexp package-name
                               (project-info-version info)
                               (and=> (latest-source-release project)
                                      distribution-url)

base-commit: 82543e9649da2da9a5285ede4ec4f718fd740fcb
-- 
2.31.1

Message #17 received at 47746 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: Xinglu Chen <public <at> yoctocell.xyz>
Cc: Ricardo Wurmus <rekado <at> elephly.net>, 47746 <at> debbugs.gnu.org
Subject: Re: bug#47746: pypi importer generates wrong pypi-uri without
 underscore
Date: Sun, 18 Apr 2021 22:37:16 -0400

Hi,

Xinglu Chen <public <at> yoctocell.xyz> writes:

> On Tue, Apr 13 2021, Ricardo Wurmus wrote:
>
>> You’re right, this only affects some packages.  Here are two 
>> examples:
>>
>> [...]
>>
>> They both have a redirection in common, but the pypi-uri is 
>> incorrect as it needs to have the underscore.
>
> Looking at the JSON object at
> <https://pypi.org/pypi/jupyterlab-widgets/json>, it specifies
> “jupyterlabs-widgets” as ‘info.downloads.name’, which is what the PyPi
> importer reads to determine the PyPi URI.
>
> Would it make sense to just make it read the name that the user
> specifies in the shell, i.e. ‘./pre-inst-env guix import pypi
> PACKAGE-NAME’ (see the patch below)?  Or will this potentially mess up
> other PyPi URIs?
>
>>From ec5612e9d4c2026541874c6a3eaf7c9009ce4a31 Mon Sep 17 00:00:00 2001
> Message-Id: <ec5612e9d4c2026541874c6a3eaf7c9009ce4a31.1618404153.git.public <at> yoctocell.xyz>
> From: Xinglu Chen <public <at> yoctocell.xyz>
> Date: Wed, 14 Apr 2021 14:37:01 +0200
> Subject: [PATCH] import: pypi: Use package name from the shell to construct
>  the URI.
>
> Some packages will list an incorrect package name in the JSON object,
> resulting in PyPi URIs that lead to nowhere.

In my opinion, if a package contains wrong values in their metadata,
it's a problem that should be reported to that package.  As with other
things, garbage in -> garbage out; Guix cannot do much here.  I seem to
recall that some older PyPI mirror had that problem.  It may be that
their metadata name -> URI is not consistent.  We could try to find what
that 2nd scheme is and attempt both, although that's not very elegant
and it probably points to packages being very outdated to start with.

Thanks,

Maxim

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.