GNU bug report logs - #48971
[PATCH 0/2] Add SWH support for Hg repositories

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 48971 in the body.
You can then email your comments to 48971 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Xinglu Chen <public <at> yoctocell.xyz>
To: guix-patches <at> gnu.org
Subject: [PATCH 0/2] Add SWH support for Hg repositories
Date: Sat, 12 Jun 2021 13:52:23 +0200

This series adds SWH support for Hg repositories, meaning that if the
upstream Hg repository is missing, Guix will try to fetch the source
from SWH.  SWH has archived more than 250 000 Hg repositories that were
previously hosted od BitBucket, so I think will be a useful feature.  :)

  https://www.softwareheritage.org/2020/09/17/250-000-endangered-mercurial-repositories-rescued/

Xinglu Chen (2):
  swh: Support lookup of Hg tags.
  hg-download: Support falling back to SWH.

 guix/hg-download.scm | 31 ++++++++++++++++++++++++++++---
 guix/swh.scm         | 10 ++++++++--
 2 files changed, 36 insertions(+), 5 deletions(-)


base-commit: 00727470b92e0b8ab0e00dd2cd87495c0c136341
-- 
2.32.0

Message #8 received at 48971 <at> debbugs.gnu.org (full text, mbox):

From: Xinglu Chen <public <at> yoctocell.xyz>
To: 48971 <at> debbugs.gnu.org
Subject: [PATCH 1/2] swh: Support lookup of Hg tags.
Date: Sat, 12 Jun 2021 13:57:19 +0200

* guix/swh.scm (lookup-origin-revision): Support lookup of Hg tags, not just
Git tags.
---
I noticed that the ‘commit-id?’ procedure checks if REFERENCE is of
length 40, this will result in false negatives because some packages
that use ‘hg-fetch’ only specify the first ~10 or ~15 characters of the
changeset revision.  We should probably fix that.

 guix/swh.scm | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/guix/swh.scm b/guix/swh.scm
index f6d5241e06..b5c800011d 100644
--- a/guix/swh.scm
+++ b/guix/swh.scm
@@ -1,6 +1,7 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2018, 2019, 2020, 2021 Ludovic Courtès <ludo <at> gnu.org>
 ;;; Copyright © 2020 Jakub Kądziołka <kuba <at> kadziolka.net>
+;;; Copyright © 2021 Xinglu Chen <public <at> yoctocell.xyz>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -455,8 +456,13 @@ URL could not be found."
         ((visit . _)
          (let ((snapshot (visit-snapshot visit)))
            (match (and=> (find (lambda (branch)
-                                 (string=? (string-append "refs/tags/" tag)
-                                           (branch-name branch)))
+                                 (or
+                                  ;; Git specific.
+                                  (string=? (string-append "refs/tags/" tag)
+                                            (branch-name branch))
+                                  ;; Hg specific.
+                                  (string=? tag
+                                            (branch-name branch))))
                                (snapshot-branches snapshot))
                          branch-target)
              ((? release? release)
-- 
2.32.0

Message #11 received at 48971 <at> debbugs.gnu.org (full text, mbox):

From: Xinglu Chen <public <at> yoctocell.xyz>
To: 48971 <at> debbugs.gnu.org
Subject: [PATCH 2/2] hg-download: Support falling back to SWH.
Date: Sat, 12 Jun 2021 13:57:22 +0200

* guix/hg-download.scm (hg-fetch): Fall back to fetching the source from SWH
if the upstream source is missing.
---
 guix/hg-download.scm | 31 ++++++++++++++++++++++++++++---
 1 file changed, 28 insertions(+), 3 deletions(-)

diff --git a/guix/hg-download.scm b/guix/hg-download.scm
index eb7c345489..c386d2f5f3 100644
--- a/guix/hg-download.scm
+++ b/guix/hg-download.scm
@@ -67,6 +67,13 @@
   "Return a fixed-output derivation that fetches REF, a <hg-reference>
 object.  The output is expected to have recursive hash HASH of type
 HASH-ALGO (a symbol).  Use NAME as the file name, or a generic name if #f."
+  (define inputs
+    ;; The 'swh-download' procedure requires tar and gzip.
+    `(("gzip" ,(module-ref (resolve-interface '(gnu packages compression))
+                           'gzip))
+      ("tar" ,(module-ref (resolve-interface '(gnu packages base))
+                          'tar))))
+
   (define guile-zlib
     (module-ref (resolve-interface '(gnu packages guile)) 'guile-zlib))
 
@@ -79,7 +86,8 @@ HASH-ALGO (a symbol).  Use NAME as the file name, or a generic name if #f."
   (define modules
     (delete '(guix config)
             (source-module-closure '((guix build hg)
-                                     (guix build download-nar)))))
+                                     (guix build download-nar)
+                                     (guix swh)))))
 
   (define build
     (with-imported-modules modules
@@ -87,13 +95,30 @@ HASH-ALGO (a symbol).  Use NAME as the file name, or a generic name if #f."
                              guile-zlib)
         #~(begin
             (use-modules (guix build hg)
-                         (guix build download-nar))
+                         (guix build utils) ;for `set-path-environment-variable'
+                         (guix build download-nar)
+                         (guix swh)
+                         (ice-9 match))
+
+            (set-path-environment-variable "PATH" '("bin")
+                                           (match '#+inputs
+                                             (((names dirs outputs ...) ...)
+                                              dirs)))
 
             (or (hg-fetch '#$(hg-reference-url ref)
                           '#$(hg-reference-changeset ref)
                           #$output
                           #:hg-command (string-append #+hg "/bin/hg"))
-                (download-nar #$output))))))
+                (download-nar #$output)
+                ;; As a last resort, attempt to download from Software Heritage.
+                ;; Disable X.509 certificate verification to avoid depending
+                ;; on nss-certs--we're authenticating the checkout anyway.
+                (parameterize ((%verify-swh-certificate? #f))
+                  (format (current-error-port)
+                          "Trying to download from Software Heritage...~%")
+                  (swh-download #$(hg-reference-url ref)
+                                #$(hg-reference-changeset ref)
+                                #$output)))))))
 
   (mlet %store-monad ((guile (package->derivation guile system)))
     (gexp->derivation (or name "hg-checkout") build
-- 
2.32.0

Message #14 received at 48971 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Xinglu Chen <public <at> yoctocell.xyz>
Cc: 48971 <at> debbugs.gnu.org
Subject: Re: bug#48971: [PATCH 0/2] Add SWH support for Hg repositories
Date: Mon, 14 Jun 2021 18:44:40 +0200

Hello,

Xinglu Chen <public <at> yoctocell.xyz> skribis:

> Xinglu Chen (2):
>   swh: Support lookup of Hg tags.
>   hg-download: Support falling back to SWH.

Pushed as 69d7333217ce85d9d1643a8349757b6b170d3b1f!

Note that currently this only works for (1) repositories that are
archived, and (2) origins that refer to hg tags, not commit IDs.

I found an example that meets these criteria: python-openpyxl.  :-)
I commented out the other fallbacks in (guix hg-download) and ran:

--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env guix build python-openpyxl -S --no-substitutes 
The following derivation will be built:
   /gnu/store/71xjlgdqw50a76ycqrmjrqh7b4mdi5s7-python-openpyxl-3.0.5-checkout.drv
building /gnu/store/71xjlgdqw50a76ycqrmjrqh7b4mdi5s7-python-openpyxl-3.0.5-checkout.drv...
guile: warning: failed to install locale
environment variable `PATH' set to `/gnu/store/378zjf2kgajcfd7mfr98jn5xyc5wa3qv-gzip-1.10/bin:/gnu/store/sf3rbvb6iqcphgm1afbplcs72hsywg25-tar-1.32/bin'
Trying to download from Software Heritage...
SWH: found revision 4d4e19e58314de481b98f4c239c3b3e9fd080613 with directory at 'https://archive.softwareheritage.org/api/1/directory/fb5bbba156e6b8e2bb94ea1dd5807ef2c8931864/'
SWH vault: requested bundle cooking, waiting for completion...
SWH vault: requested bundle cooking, waiting for completion...
SWH vault: requested bundle cooking, waiting for completion...
SWH vault: requested bundle cooking, waiting for completion...
SWH vault: requested bundle cooking, waiting for completion...
SWH vault: requested bundle cooking, waiting for completion...
SWH vault: requested bundle cooking, waiting for completion...
--8<---------------cut here---------------end--------------->8---

It’s still waiting for the bundle…

Regarding #2, there are some good news:
<https://forge.softwareheritage.org/T2849> is about implementing a
mapping from “VCS IDs” (hg commit IDs in this case) to SWHIDs.  Thus,
eventually, ‘lookup-origin-revision’ should be able to take an hg commit
ID and to map it to an SWHID.  That JSON API isn’t exposed yet.

Thanks,
Ludo’.

Message #19 received at 48971 <at> debbugs.gnu.org (full text, mbox):

From: Xinglu Chen <public <at> yoctocell.xyz>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 48971 <at> debbugs.gnu.org
Subject: Re: [bug#48971] [PATCH 0/2] Add SWH support for Hg repositories
Date: Tue, 15 Jun 2021 07:37:27 +0200

[Message part 1 (text/plain, inline)]

On Mon, Jun 14 2021, Ludovic Courtès wrote:

> Hello,
>
> Xinglu Chen <public <at> yoctocell.xyz> skribis:
>
>> Xinglu Chen (2):
>>   swh: Support lookup of Hg tags.
>>   hg-download: Support falling back to SWH.
>
> Pushed as 69d7333217ce85d9d1643a8349757b6b170d3b1f!
>
> Note that currently this only works for (1) repositories that are
> archived, and (2) origins that refer to hg tags, not commit IDs.
>
> I found an example that meets these criteria: python-openpyxl.  :-)
> I commented out the other fallbacks in (guix hg-download) and ran:
>
> --8<---------------cut here---------------start------------->8---
> $ ./pre-inst-env guix build python-openpyxl -S --no-substitutes 
> The following derivation will be built:
>    /gnu/store/71xjlgdqw50a76ycqrmjrqh7b4mdi5s7-python-openpyxl-3.0.5-checkout.drv
> building /gnu/store/71xjlgdqw50a76ycqrmjrqh7b4mdi5s7-python-openpyxl-3.0.5-checkout.drv...
> guile: warning: failed to install locale
> environment variable `PATH' set to `/gnu/store/378zjf2kgajcfd7mfr98jn5xyc5wa3qv-gzip-1.10/bin:/gnu/store/sf3rbvb6iqcphgm1afbplcs72hsywg25-tar-1.32/bin'
> Trying to download from Software Heritage...
> SWH: found revision 4d4e19e58314de481b98f4c239c3b3e9fd080613 with directory at 'https://archive.softwareheritage.org/api/1/directory/fb5bbba156e6b8e2bb94ea1dd5807ef2c8931864/'
> SWH vault: requested bundle cooking, waiting for completion...
> SWH vault: requested bundle cooking, waiting for completion...
> SWH vault: requested bundle cooking, waiting for completion...
> SWH vault: requested bundle cooking, waiting for completion...
> SWH vault: requested bundle cooking, waiting for completion...
> SWH vault: requested bundle cooking, waiting for completion...
> SWH vault: requested bundle cooking, waiting for completion...
> --8<---------------cut here---------------end--------------->8---
>
> It’s still waiting for the bundle…

Hmm, I tried it on a package in local config and it seemed to work fine
with commit IDs, I guess this was because I ran ‘guix hash -r’ on the
extracted tarball I downloaded from SWH, so the source was already in
the store when I tried to build it...

> Regarding #2, there are some good news:
> <https://forge.softwareheritage.org/T2849> is about implementing a
> mapping from “VCS IDs” (hg commit IDs in this case) to SWHIDs.  Thus,
> eventually, ‘lookup-origin-revision’ should be able to take an hg commit
> ID and to map it to an SWHID.  That JSON API isn’t exposed yet.

Oh, that’s great!

[signature.asc (application/pgp-signature, inline)]

Message #22 received at 48971 <at> debbugs.gnu.org (full text, mbox):

From: zimoun <zimon.toutoune <at> gmail.com>
To: Xinglu Chen <public <at> yoctocell.xyz>, 48971 <at> debbugs.gnu.org
Subject: Re: [bug#48971] [PATCH 0/2] Add SWH support for Hg repositories
Date: Wed, 16 Jun 2021 07:23:03 +0200

Hi,

On Sat, 12 Jun 2021 at 13:52, Xinglu Chen <public <at> yoctocell.xyz> wrote:
> This series adds SWH support for Hg repositories, meaning that if the
> upstream Hg repository is missing, Guix will try to fetch the source
> from SWH.  SWH has archived more than 250 000 Hg repositories that were
> previously hosted od BitBucket, so I think will be a useful feature.  :)
>
>   https://www.softwareheritage.org/2020/09/17/250-000-endangered-mercurial-repositories-rescued/

Cool!  Thanks for working on this.


> Xinglu Chen (2):
>   swh: Support lookup of Hg tags.
>   hg-download: Support falling back to SWH.
>
>  guix/hg-download.scm | 31 ++++++++++++++++++++++++++++---
>  guix/swh.scm         | 10 ++++++++--
>  2 files changed, 36 insertions(+), 5 deletions(-)

LGTM.

All the best,
simon

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.