GNU bug report logs - #65114
[PATCH 0/6] Update Scrapy.

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 65114 in the body.
You can then email your comments to 65114 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Vinicius Monego <monego <at> posteo.net>
To: guix-patches <at> gnu.org
Cc: Vinicius Monego <monego <at> posteo.net>
Subject: [PATCH 0/6] Update Scrapy.
Date: Sun,  6 Aug 2023 15:31:59 +0000

Vinicius Monego (6):
  gnu: python-w3lib: Update to 2.1.2.
  gnu: python-cssselect: Update to 1.2.0.
  gnu: python-parsel: Update to 1.8.1.
  gnu: python-itemadapter: Update to 0.8.0.
  gnu: python-itemloaders: Update to 1.1.0.
  gnu: python-scrapy: Update to 2.10.0.

 gnu/local.mk                                  |  1 -
 .../python-w3lib-fix-test-failure.patch       | 60 --------------
 gnu/packages/python-web.scm                   | 80 +++++++------------
 gnu/packages/python-xyz.scm                   | 13 +--
 4 files changed, 38 insertions(+), 116 deletions(-)
 delete mode 100644 gnu/packages/patches/python-w3lib-fix-test-failure.patch

-- 
2.34.1

Message #8 received at 65114 <at> debbugs.gnu.org (full text, mbox):

From: Vinicius Monego <monego <at> posteo.net>
To: 65114 <at> debbugs.gnu.org
Cc: Vinicius Monego <monego <at> posteo.net>
Subject: [PATCH 1/6] gnu: python-w3lib: Update to 2.1.2.
Date: Sun,  6 Aug 2023 15:33:12 +0000

* gnu/packages/python-web.scm (python-w3lib): Update to 2.1.2.
[source]: Remove patch.
[build-system]: Use pyproject-build-system.
[arguments]: Do not override the check phase.
[native-inputs]: Remove python-six.
* gnu/packages/patches/python-w3lib-fix-test-failure.patch: Delete file.
* gnu/local.mk (dist_patch_DATA): Remove it.
---
 gnu/local.mk                                  |  1 -
 .../python-w3lib-fix-test-failure.patch       | 60 -------------------
 gnu/packages/python-web.scm                   | 16 ++---
 3 files changed, 4 insertions(+), 73 deletions(-)
 delete mode 100644 gnu/packages/patches/python-w3lib-fix-test-failure.patch

diff --git a/gnu/local.mk b/gnu/local.mk
index 77707127a7..e091b310c2 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -1730,7 +1730,6 @@ dist_patch_DATA =						\
   %D%/packages/patches/python-shiboken-2-compat.patch		\
   %D%/packages/patches/python-telingo-fix-comparison.patch	\
   %D%/packages/patches/python-typeguard-python3.10.patch	\
-  %D%/packages/patches/python-w3lib-fix-test-failure.patch	\
   %D%/packages/patches/python-wxwidgets-type-errors.patch	\
   %D%/packages/patches/quodlibet-fix-invalid-glob.patch	\
   %D%/packages/patches/quodlibet-fix-mtime-tests.patch		\
diff --git a/gnu/packages/patches/python-w3lib-fix-test-failure.patch b/gnu/packages/patches/python-w3lib-fix-test-failure.patch
deleted file mode 100644
index f38fc749df..0000000000
--- a/gnu/packages/patches/python-w3lib-fix-test-failure.patch
+++ /dev/null
@@ -1,60 +0,0 @@
-From fae6cc40e112cd13697cb0e8d79976f32c72491d Mon Sep 17 00:00:00 2001
-From: Eugenio Lacuesta <eugenio.lacuesta <at> gmail.com>
-Date: Wed, 10 Mar 2021 12:31:05 -0300
-Subject: [PATCH] [CI] Mark single add_or_replace_parameter test as xfail
-
----
-This patch is based on upstream commit
-fae6cc40e112cd13697cb0e8d79976f32c72491d which is part of
-https://github.com/scrapy/w3lib/pull/166.
-
- .gitignore        |  3 ++-
- tests/test_url.py | 25 ++++++++++++++-----------
- 2 files changed, 16 insertions(+), 12 deletions(-)
-
-diff --git a/tests/test_url.py b/tests/test_url.py
-index 8b07c00..0f7458e 100644
---- a/tests/test_url.py
-+++ b/tests/test_url.py
-@@ -1,12 +1,14 @@
--# -*- coding: utf-8 -*-
- from __future__ import absolute_import
- import os
- import unittest
-+
-+import pytest
-+from six.moves.urllib.parse import urlparse
-+
- from w3lib.url import (is_url, safe_url_string, safe_download_url,
-     url_query_parameter, add_or_replace_parameter, url_query_cleaner,
-     file_uri_to_path, parse_data_uri, path_to_file_uri, any_to_uri,
-     urljoin_rfc, canonicalize_url, parse_url, add_or_replace_parameters)
--from six.moves.urllib.parse import urlparse
- 
- 
- class UrlTests(unittest.TestCase):
-@@ -310,10 +311,6 @@ def test_add_or_replace_parameter(self):
-         self.assertEqual(add_or_replace_parameter(url, 'arg3', 'nv3'),
-                          'http://domain/test?arg1=v1&arg2=v2&arg3=nv3')
- 
--        url = 'http://domain/test?arg1=v1;arg2=v2'
--        self.assertEqual(add_or_replace_parameter(url, 'arg1', 'v3'),
--                         'http://domain/test?arg1=v3&arg2=v2')
--
-         self.assertEqual(add_or_replace_parameter("http://domain/moreInfo.asp?prodID=", 'prodID', '20'),
-                          'http://domain/moreInfo.asp?prodID=20')
-         url = 'http://rmc-offers.co.uk/productlist.asp?BCat=2%2C60&CatID=60'
-@@ -338,6 +335,13 @@ def test_add_or_replace_parameter(self):
-         self.assertEqual(add_or_replace_parameter(url, 'arg1', 'v3'),
-                          'http://domain/test?arg1=v3&arg2=v2')
- 
-+    @pytest.mark.xfail(reason="https://github.com/scrapy/w3lib/issues/164")
-+    def test_add_or_replace_parameter_fail(self):
-+        self.assertEqual(
-+            add_or_replace_parameter('http://domain/test?arg1=v1;arg2=v2', 'arg1', 'v3'),
-+            'http://domain/test?arg1=v3&arg2=v2'
-+        )
-+
-     def test_add_or_replace_parameters(self):
-         url = 'http://domain/test'
-         self.assertEqual(add_or_replace_parameters(url, {'arg': 'v'}),
diff --git a/gnu/packages/python-web.scm b/gnu/packages/python-web.scm
index 17c9563cfd..24071a8553 100644
--- a/gnu/packages/python-web.scm
+++ b/gnu/packages/python-web.scm
@@ -7752,25 +7752,17 @@ (define-public python-smart-open
 (define-public python-w3lib
   (package
     (name "python-w3lib")
-    (version "1.22.0")
+    (version "2.1.2")
     (source
      (origin
        (method url-fetch)
        (uri (pypi-uri "w3lib" version))
-       (patches (search-patches "python-w3lib-fix-test-failure.patch"))
        (sha256
         (base32
-         "1pv02lvvmgz2qb61vz1jkjc04fgm4hpfvaj5zm4i3mjp64hd1mha"))))
-    (build-system python-build-system)
-    (arguments
-     `(#:phases
-        (modify-phases %standard-phases
-          (replace 'check
-            (lambda* (#:key tests? #:allow-other-keys)
-              (when tests?
-                (invoke "pytest")))))))
+         "1cd4b3w5g3pfccsg79kjj27fwi216ip927rjq7isp8pfjzlp8nzd"))))
+    (build-system pyproject-build-system)
     (native-inputs
-     (list python-pytest python-six))
+     (list python-pytest))
     (home-page "https://github.com/scrapy/w3lib")
     (synopsis "Python library of web-related functions")
     (description
-- 
2.34.1

Message #11 received at 65114 <at> debbugs.gnu.org (full text, mbox):

From: Vinicius Monego <monego <at> posteo.net>
To: 65114 <at> debbugs.gnu.org
Cc: Vinicius Monego <monego <at> posteo.net>
Subject: [PATCH 2/6] gnu: python-cssselect: Update to 1.2.0.
Date: Sun,  6 Aug 2023 15:33:13 +0000

* gnu/packages/python-web.scm (python-cssselect): Update to 1.2.0.
[build-system]: Use pyproject-build-system.
[arguments]: Do not override the check phase.
---
 gnu/packages/python-web.scm | 11 +++--------
 1 file changed, 3 insertions(+), 8 deletions(-)

diff --git a/gnu/packages/python-web.scm b/gnu/packages/python-web.scm
index 24071a8553..9ecd9f35a3 100644
--- a/gnu/packages/python-web.scm
+++ b/gnu/packages/python-web.scm
@@ -2095,7 +2095,7 @@ (define-public python-css-parser
 (define-public python-cssselect
   (package
     (name "python-cssselect")
-    (version "1.1.0")
+    (version "1.2.0")
     (source (origin
               ;; The PyPI release does not contain tests.
               (method git-fetch)
@@ -2105,13 +2105,8 @@ (define-public python-cssselect
               (file-name (git-file-name name version))
               (sha256
                (base32
-                "0xslrnhbrmgakp4xg6k26qffay3kqffp3a2z2sk27c65rwxa79kc"))))
-    (build-system python-build-system)
-    (arguments
-     `(#:phases (modify-phases %standard-phases
-                  (replace 'check
-                    (lambda _
-                      (invoke "pytest" "-vv"))))))
+                "1x4nrvb1p1byi1whmspik7lbh303akdlh762dayfxam3hycsh5kk"))))
+    (build-system pyproject-build-system)
     (native-inputs
      (list python-lxml python-pytest))
     (home-page "https://github.com/scrapy/cssselect")
-- 
2.34.1

Message #14 received at 65114 <at> debbugs.gnu.org (full text, mbox):

From: Vinicius Monego <monego <at> posteo.net>
To: 65114 <at> debbugs.gnu.org
Cc: Vinicius Monego <monego <at> posteo.net>
Subject: [PATCH 3/6] gnu: python-parsel: Update to 1.8.1.
Date: Sun,  6 Aug 2023 15:33:14 +0000

* gnu/packages/python-web.scm (python-parsel): Update to 1.8.1.
[build-system]: Use pyproject-build-system.
[propagated-inputs]: Add python-jmespath, python-typing-extensions.  Remove
python-six.
[native-inputs]: Add python-psutil.  Remove python-pytest-runner.
---
 gnu/packages/python-web.scm | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/gnu/packages/python-web.scm b/gnu/packages/python-web.scm
index 9ecd9f35a3..70a1ab082f 100644
--- a/gnu/packages/python-web.scm
+++ b/gnu/packages/python-web.scm
@@ -8039,21 +8039,22 @@ (define-public python-protego
 (define-public python-parsel
   (package
     (name "python-parsel")
-    (version "1.6.0")
+    (version "1.8.1")
     (source
       (origin
         (method url-fetch)
         (uri (pypi-uri "parsel" version))
         (sha256
-          (base32 "0yawf9r3r863lwxj0n89i7h3n8xjbsl5b7n6xg76r68scl5yzvvh"))))
-    (build-system python-build-system)
+          (base32 "0f8yh30y3961a7kqwcnp4j3s7044ilakykiavc0skwdkr5l8xwmg"))))
+    (build-system pyproject-build-system)
     (propagated-inputs
       (list python-cssselect
+            python-jmespath
             python-lxml
-            python-six
+            python-typing-extensions
             python-w3lib))
     (native-inputs
-      (list python-pytest python-pytest-runner))
+      (list python-psutil python-pytest))
     (home-page "https://github.com/scrapy/parsel")
     (synopsis "Extract data from HTML and XML using XPath and CSS selectors")
     (description "Parsel is a library to extract and remove data from
-- 
2.34.1

Message #17 received at 65114 <at> debbugs.gnu.org (full text, mbox):

From: Vinicius Monego <monego <at> posteo.net>
To: 65114 <at> debbugs.gnu.org
Cc: Vinicius Monego <monego <at> posteo.net>
Subject: [PATCH 4/6] gnu: python-itemadapter: Update to 0.8.0.
Date: Sun,  6 Aug 2023 15:33:15 +0000

* gnu/packages/python-xyz.scm (python-itemadapter): Update to 0.8.0.
---
 gnu/packages/python-xyz.scm | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm
index 884c37e975..e6cbe7d333 100644
--- a/gnu/packages/python-xyz.scm
+++ b/gnu/packages/python-xyz.scm
@@ -32504,13 +32504,13 @@ (define-public python-posix-ipc
 (define-public python-itemadapter
   (package
     (name "python-itemadapter")
-    (version "0.5.0")
+    (version "0.8.0")
     (source
       (origin
         (method url-fetch)
         (uri (pypi-uri "itemadapter" version))
         (sha256
-          (base32 "083wp3h2brh8x19jbdr8rz3biqwp3jlqd0rfzcyrjyhssffsgdh5"))))
+          (base32 "1aa898gjgwy3axxfrgsh4kdvhp6n6wz3ccdishq0gh8azf2q8xbp"))))
     (build-system python-build-system)
     (home-page "https://github.com/scrapy/itemadapter")
     (synopsis "Common interface for data container classes")
-- 
2.34.1

Message #20 received at 65114 <at> debbugs.gnu.org (full text, mbox):

From: Vinicius Monego <monego <at> posteo.net>
To: 65114 <at> debbugs.gnu.org
Cc: Vinicius Monego <monego <at> posteo.net>
Subject: [PATCH 5/6] gnu: python-itemloaders: Update to 1.1.0.
Date: Sun,  6 Aug 2023 15:33:16 +0000

* gnu/packages/python-xyz.scm (python-itemloaders): Update to 1.1.0.
[build-system]: Use pyproject-build-system.
[native-inputs]: Add python-pytest.
---
 gnu/packages/python-xyz.scm | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm
index e6cbe7d333..c7bce1b135 100644
--- a/gnu/packages/python-xyz.scm
+++ b/gnu/packages/python-xyz.scm
@@ -32534,16 +32534,17 @@ (define-public python-itemadapter
 (define-public python-itemloaders
   (package
     (name "python-itemloaders")
-    (version "1.0.4")
+    (version "1.1.0")
     (source
       (origin
         (method url-fetch)
         (uri (pypi-uri "itemloaders" version))
         (sha256
-          (base32 "15hc78h90qhwass1bga1c3xar2dd6j8sxg61zg6jvh74lf6csxqj"))))
-    (build-system python-build-system)
+          (base32 "0j2aw4ipalj208594x80blpgkh1i63gqqa4nb67b823av9hirn11"))))
+    (build-system pyproject-build-system)
+    (native-inputs (list python-pytest))
     (propagated-inputs
-      (list python-itemadapter python-jmespath python-parsel python-w3lib))
+     (list python-itemadapter python-jmespath python-parsel python-w3lib))
     (home-page "https://github.com/scrapy/itemloaders")
     (synopsis "Base library for scrapy's ItemLoader")
     (description "Itemloaders is a library that helps you collect data
-- 
2.34.1

Message #23 received at 65114 <at> debbugs.gnu.org (full text, mbox):

From: Vinicius Monego <monego <at> posteo.net>
To: 65114 <at> debbugs.gnu.org
Cc: Vinicius Monego <monego <at> posteo.net>
Subject: [PATCH 6/6] gnu: python-scrapy: Update to 2.10.0.
Date: Sun,  6 Aug 2023 15:33:17 +0000

* gnu/packages/python-web.scm (python-scrapy): Update to 2.10.0.
[build-system]: Use pyproject-build-system.
[arguments]: Use G-Expressions.  Do not override the check phase, skip tests
in #:test-flags instead.  Run tests sequentially and skip fewer tests.
---
 gnu/packages/python-web.scm | 42 ++++++++++++++++---------------------
 1 file changed, 18 insertions(+), 24 deletions(-)

diff --git a/gnu/packages/python-web.scm b/gnu/packages/python-web.scm
index 70a1ab082f..c12740b587 100644
--- a/gnu/packages/python-web.scm
+++ b/gnu/packages/python-web.scm
@@ -8065,36 +8065,30 @@ (define-public python-parsel
 (define-public python-scrapy
   (package
     (name "python-scrapy")
-    (version "2.7.1")
+    (version "2.10.0")
     (source
      (origin
        (method url-fetch)
        (uri (pypi-uri "Scrapy" version))
        (sha256
-        (base32 "0kpi3hg2ycs6s8cg41r2zc1axd0rpnps8bnzg7wisjyjaf1l1yih"))))
-    (build-system python-build-system)
+        (base32 "0fdf1kx39zy7b0v5gkwwqvm58ww63z5bvm78yjymhfn0r0zs65jf"))))
+    (build-system pyproject-build-system)
     (arguments
-     `(#:phases
-       (modify-phases %standard-phases
-         (replace 'check
-           (lambda* (#:key tests? #:allow-other-keys)
-             (when tests?
-               (invoke "pytest"
-                       "-n" (number->string (parallel-job-count))
-                       ;; These tests fail when run in parallel (see:
-                       ;; https://github.com/scrapy/scrapy/issues/5502).
-                       "--ignore" "tests/test_engine.py"
-                       "--ignore" "tests/test_engine_stop_download_bytes.py"
-                       "--ignore" "tests/test_engine_stop_download_headers.py"
-                       ;; This test require network access.
-                       "--ignore" "tests/test_command_check.py"
-                       "-k"
-                       (string-append
-                        ;; The followin tests fail for unknown reasons.
-                        "not test_server_set_cookie_domain_suffix_public_private"
-                        " and not test_user_set_cookie_domain_suffix_public_private"
-                        " and not test_pformat")
-                       "tests")))))))
+     (list #:test-flags
+           ;; Tests fail with DNS lookup or need a display.
+           #~(list "-k" (string-append
+                         "not test_SCRAPY_CHECK_set"
+                         " and not "
+                         (string-join
+                          (list "test_check_all_default_contracts"
+                                "test_check_cb_kwargs_contract"
+                                "test_check_returns_items_contract"
+                                "test_check_returns_requests_contract"
+                                "test_check_scrapes_contract"
+                                "test_pformat"
+                                "test_pformat_old_windows"
+                                "test_pformat_windows")
+                          " and not ")))))
     (propagated-inputs
      (list python-botocore              ; Optional: For S3FeedStorage class.
            python-cryptography
-- 
2.34.1

Message #26 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Christopher Baines <mail <at> cbaines.net>
To: Vinicius Monego <monego <at> posteo.net>
Cc: 65114 <at> debbugs.gnu.org, guix-patches <at> gnu.org
Subject: Re: [bug#65114] [PATCH 0/6] Update Scrapy.
Date: Sun, 27 Aug 2023 19:14:57 +0100

[Message part 1 (text/plain, inline)]

Vinicius Monego <monego <at> posteo.net> writes:

> Vinicius Monego (6):
>   gnu: python-w3lib: Update to 2.1.2.
>   gnu: python-cssselect: Update to 1.2.0.
>   gnu: python-parsel: Update to 1.8.1.
>   gnu: python-itemadapter: Update to 0.8.0.
>   gnu: python-itemloaders: Update to 1.1.0.
>   gnu: python-scrapy: Update to 2.10.0.
>
>  gnu/local.mk                                  |  1 -
>  .../python-w3lib-fix-test-failure.patch       | 60 --------------
>  gnu/packages/python-web.scm                   | 80 +++++++------------
>  gnu/packages/python-xyz.scm                   | 13 +--
>  4 files changed, 38 insertions(+), 116 deletions(-)
>  delete mode 100644 gnu/packages/patches/python-w3lib-fix-test-failure.patch

These changes look good to me +1

Thanks,

Chris

[signature.asc (application/pgp-signature, inline)]

Message #34 received at 65114-done <at> debbugs.gnu.org (full text, mbox):

From: Vinicius Monego <monego <at> posteo.net>
To: Christopher Baines <mail <at> cbaines.net>
Cc: 65114-done <at> debbugs.gnu.org
Subject: Re: [bug#65114] [PATCH 0/6] Update Scrapy.
Date: Sat, 02 Sep 2023 17:59:49 +0000

Em dom, 2023-08-27 às 19:14 +0100, Christopher Baines escreveu:


[...]

> These changes look good to me +1
> 

[...]

Applied with a minor update to Scrapy, thanks.

Vinicius

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.