GNU bug report logs - #39547
[PATCH] website: Provide JSON sources list used by Software Heritage.

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 39547 in the body.
You can then email your comments to 39547 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: zimoun <zimon.toutoune <at> gmail.com>
To: guix-patches <at> gnu.org
Cc: zimoun <zimon.toutoune <at> gmail.com>
Subject: [PATCH] website: Provide JSON sources list used by Software Heritage.
Date: Mon, 10 Feb 2020 18:04:18 +0100

Format discussed here <https://forge.softwareheritage.org/D2025#51269>.

* website/apps/packages/builder.scm (sources-json-builder): New procedure.
---
 website/apps/packages/builder.scm | 62 +++++++++++++++++++++++++++++++
 1 file changed, 62 insertions(+)

diff --git a/website/apps/packages/builder.scm b/website/apps/packages/builder.scm
index 9dc44c9..5279096 100644
--- a/website/apps/packages/builder.scm
+++ b/website/apps/packages/builder.scm
@@ -2,6 +2,7 @@
 ;;; Copyright © 2017 Ludovic Courtès <ludo <at> gnu.org>
 ;;; Copyright © 2019 Ricardo Wurmus <rekado <at> elephly.net>
 ;;; Copyright © 2019 Nicolò Balzarotti <nicolo <at> nixo.xyz>
+;;; Copyright © 2020 Simon Tournier <zimon.toutoune <at> gmail.com>
 ;;;
 ;;; Initially written by sirgazil
 ;;; who waives all copyright interest on this file.
@@ -37,6 +38,8 @@
   #:use-module (haunt page)
   #:use-module (haunt utils)
   #:use-module (srfi srfi-1)
+  #:use-module ((web uri) #:select (string->uri uri->string uri-scheme))
+  #:use-module ((guix build download) #:select (maybe-expand-mirrors))
   #:use-module (guix packages)
   #:use-module (guix download)
   #:use-module (guix git-download)
@@ -70,6 +73,7 @@
   (flatten
    (list
     (index-builder)
+    (sources-json-builder)
     (packages-json-builder)
     (packages-builder)
     (package-list-builder))))
@@ -84,6 +88,64 @@
   ;; Maximum number of packages shown on /packages.
   30)
 
+(define (sources-json-builder)
+  "Return a JSON page listing all the sources."
+  (define (origin->json origin)
+    (define method
+      (origin-method origin))
+
+    (define uri                         ;represented as string
+      (origin-uri origin))
+
+    (define (mirror->url uri)
+      (uri->string (car (maybe-expand-mirrors uri %mirrors))))
+
+    (define (resolve urls)
+      (let* ((url (car urls))
+             (uri (string->uri url))
+             (rest (cdr urls)))
+        (case (uri-scheme uri)
+          ((mirror) (mirror->url uri))
+          ((http) url)
+          ((https) url)
+          (else
+           (if (null? rest)
+               url
+               (resolve rest))))))
+
+    `((type . ,(cond ((eq? url-fetch method) 'url)
+                     ((eq? git-fetch method) 'git)
+                     ((eq? svn-fetch method) 'svn)
+                     (else                   #nil)))
+      ,@(cond ((eq? url-fetch method)
+               `(("url" . ,(match uri
+				  ((? string? url) (mirror->url (string->uri url)))
+				  ((urls ...) (resolve urls))))))
+              ((eq? git-fetch method)
+               `(("git_url" . ,(git-reference-url uri))))
+              ((eq? svn-fetch method)
+               `(("svn_url" . ,(svn-reference-url uri))))
+              (else '()))
+      ,@(if (eq? method git-fetch)
+            `(("git_ref" . ,(git-reference-commit uri)))
+            '())
+      ,@(if (eq? method svn-fetch)
+            `(("svn_revision" . ,(svn-reference-revision
+                                  uri)))
+            '())))
+
+  (define (package->json package)
+    `(,@(if (origin? (package-source package))
+            (origin->json (package-source package))
+            `(("type" . "no-origin")
+              ("name" . ,(package-name package))))))
+
+  (make-page "sources.json"
+             `(("sources" . ,(list->vector (map package->json (all-packages))))
+               ("version" . "1"))
+             scm->json))
+
+
 (define (packages-json-builder)
   "Return a JSON page listing all packages."
   (define (origin->json origin)
-- 
2.23.0

Message #8 received at 39547 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludovic.courtes <at> inria.fr>
To: zimoun <zimon.toutoune <at> gmail.com>
Cc: 39547 <at> debbugs.gnu.org
Subject: Re: [bug#39547] [PATCH] website: Provide JSON sources list used by
 Software Heritage.
Date: Fri, 14 Feb 2020 09:40:45 +0100

Hello,

zimoun <zimon.toutoune <at> gmail.com> skribis:

> Format discussed here <https://forge.softwareheritage.org/D2025#51269>.
>
> * website/apps/packages/builder.scm (sources-json-builder): New procedure.

[...]

> +(define (sources-json-builder)
> +  "Return a JSON page listing all the sources."

Please add a reference to
<https://forge.softwareheritage.org/D2025#51269> here.

> +  (define (origin->json origin)
> +    (define method
> +      (origin-method origin))
> +
> +    (define uri                         ;represented as string
> +      (origin-uri origin))
> +
> +    (define (mirror->url uri)
> +      (uri->string (car (maybe-expand-mirrors uri %mirrors))))
> +
> +    (define (resolve urls)
> +      (let* ((url (car urls))
> +             (uri (string->uri url))
> +             (rest (cdr urls)))
> +        (case (uri-scheme uri)
> +          ((mirror) (mirror->url uri))
> +          ((http) url)
> +          ((https) url)
> +          (else
> +           (if (null? rest)
> +               url
> +               (resolve rest))))))
> +
> +    `((type . ,(cond ((eq? url-fetch method) 'url)
> +                     ((eq? git-fetch method) 'git)
> +                     ((eq? svn-fetch method) 'svn)
> +                     (else                   #nil)))
> +      ,@(cond ((eq? url-fetch method)
> +               `(("url" . ,(match uri
> +				  ((? string? url) (mirror->url (string->uri url)))
> +				  ((urls ...) (resolve urls))))))
> +              ((eq? git-fetch method)
> +               `(("git_url" . ,(git-reference-url uri))))
> +              ((eq? svn-fetch method)
> +               `(("svn_url" . ,(svn-reference-url uri))))
> +              (else '()))
> +      ,@(if (eq? method git-fetch)
> +            `(("git_ref" . ,(git-reference-commit uri)))
> +            '())
> +      ,@(if (eq? method svn-fetch)
> +            `(("svn_revision" . ,(svn-reference-revision
> +                                  uri)))
> +            '())))

Could you, in a first patch, move ‘origin->json’ out of
‘packages-json-builder’, and in a second patch, add mirror-expansion
feature?

For mirror:// expansion, I think you can just write something like:

  (define uris
    (append-map (cut maybe-expand-mirrors <> %mirrors)
                (match url
                  ((_ ...) (map string->uri url))
                  (_       (list (string->uri url))))))

and then pick the first element of the list.

In parallel, I’d recommend suggesting a format change or addition that
would allow us to provide all the URLs.  :-)

With those changes in place, I think we’ll be ready to go!

Thanks for working on it, Simon!

Ludo’.

Message #11 received at 39547 <at> debbugs.gnu.org (full text, mbox):

From: zimoun <zimon.toutoune <at> gmail.com>
To: Ludovic Courtès <ludovic.courtes <at> inria.fr>
Cc: 39547 <at> debbugs.gnu.org
Subject: Re: [bug#39547] [PATCH] website: Provide JSON sources list used by
 Software Heritage.
Date: Fri, 14 Feb 2020 10:04:01 +0100

Hi Ludo,

Thank you for the review.


On Fri, 14 Feb 2020 at 09:40, Ludovic Courtès <ludovic.courtes <at> inria.fr> wrote:
> zimoun <zimon.toutoune <at> gmail.com> skribis:

> > Format discussed here <https://forge.softwareheritage.org/D2025#51269>.
> >
> > * website/apps/packages/builder.scm (sources-json-builder): New procedure.
>
> [...]
>
> > +(define (sources-json-builder)
> > +  "Return a JSON page listing all the sources."
>
> Please add a reference to
> <https://forge.softwareheritage.org/D2025#51269> here.

ok


> > +  (define (origin->json origin)
> > +    (define method
> > +      (origin-method origin))
> > +
> > +    (define uri                         ;represented as string
> > +      (origin-uri origin))
> > +
> > +    (define (mirror->url uri)
> > +      (uri->string (car (maybe-expand-mirrors uri %mirrors))))
> > +
> > +    (define (resolve urls)
> > +      (let* ((url (car urls))
> > +             (uri (string->uri url))
> > +             (rest (cdr urls)))
> > +        (case (uri-scheme uri)
> > +          ((mirror) (mirror->url uri))
> > +          ((http) url)
> > +          ((https) url)
> > +          (else
> > +           (if (null? rest)
> > +               url
> > +               (resolve rest))))))
> > +
> > +    `((type . ,(cond ((eq? url-fetch method) 'url)
> > +                     ((eq? git-fetch method) 'git)
> > +                     ((eq? svn-fetch method) 'svn)
> > +                     (else                   #nil)))
> > +      ,@(cond ((eq? url-fetch method)
> > +               `(("url" . ,(match uri
> > +                               ((? string? url) (mirror->url (string->uri url)))
> > +                               ((urls ...) (resolve urls))))))
> > +              ((eq? git-fetch method)
> > +               `(("git_url" . ,(git-reference-url uri))))
> > +              ((eq? svn-fetch method)
> > +               `(("svn_url" . ,(svn-reference-url uri))))
> > +              (else '()))
> > +      ,@(if (eq? method git-fetch)
> > +            `(("git_ref" . ,(git-reference-commit uri)))
> > +            '())
> > +      ,@(if (eq? method svn-fetch)
> > +            `(("svn_revision" . ,(svn-reference-revision
> > +                                  uri)))
> > +            '())))
>
> Could you, in a first patch, move ‘origin->json’ out of
> ‘packages-json-builder’, and in a second patch, add mirror-expansion
> feature?

Yes, I will try but I am a bit lost. There is:
 - packages-json-builder that I did not modified
 - sources-json-builder which the "adaptation" of the former to output
to the sources.json format.

Well, do you want a refactor of 'origin->json' shared by the 2
"{sources,packages}-json-builder"?


> For mirror:// expansion, I think you can just write something like:
>
>   (define uris
>     (append-map (cut maybe-expand-mirrors <> %mirrors)
>                 (match url
>                   ((_ ...) (map string->uri url))
>                   (_       (list (string->uri url))))))
>
> and then pick the first element of the list.

Yes, it is better. :-)


> In parallel, I’d recommend suggesting a format change or addition that
> would allow us to provide all the URLs.  :-)

Ok :-)


> With those changes in place, I think we’ll be ready to go!

Working on updating the package Julia, I have seen that some patches
are 'origin' and live for example upstream
(https://blablab/project.git/patches/fancy-name.patch) and the current
patch will not list them. Therefore, if upstream disappears and/or
change in-place the patches, Guix would not be able to re-build in the
future (time-machine).

I am trying to implement a recursive exporter.
Well, I will try to make that the v2 contains your suggestions and all
the patches.



Cheers,
simon

Message #14 received at 39547 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludovic.courtes <at> inria.fr>
To: zimoun <zimon.toutoune <at> gmail.com>
Cc: 39547 <at> debbugs.gnu.org
Subject: Re: [bug#39547] [PATCH] website: Provide JSON sources list used by
 Software Heritage.
Date: Fri, 14 Feb 2020 11:20:38 +0100

Hi,

zimoun <zimon.toutoune <at> gmail.com> skribis:

>> Could you, in a first patch, move ‘origin->json’ out of
>> ‘packages-json-builder’, and in a second patch, add mirror-expansion
>> feature?
>
> Yes, I will try but I am a bit lost. There is:
>  - packages-json-builder that I did not modified
>  - sources-json-builder which the "adaptation" of the former to output
> to the sources.json format.
>
> Well, do you want a refactor of 'origin->json' shared by the 2
> "{sources,packages}-json-builder"?

Yes, exactly.  No reason to have two copies of that code.

> Working on updating the package Julia, I have seen that some patches
> are 'origin' and live for example upstream
> (https://blablab/project.git/patches/fancy-name.patch) and the current
> patch will not list them. Therefore, if upstream disappears and/or
> change in-place the patches, Guix would not be able to re-build in the
> future (time-machine).

Right.  I guess you can implement that in the next version.  What you
propose is already nice to have.

Ludo’.

Message #17 received at 39547 <at> debbugs.gnu.org (full text, mbox):

From: zimoun <zimon.toutoune <at> gmail.com>
To: Ludovic Courtès <ludovic.courtes <at> inria.fr>
Cc: 39547 <at> debbugs.gnu.org
Subject: Re: [bug#39547] [PATCH] website: Provide JSON sources list used by
 Software Heritage.
Date: Mon, 17 Feb 2020 18:59:24 +0100

Hi Ludo,

On Fri, 14 Feb 2020 at 11:20, Ludovic Courtès <ludovic.courtes <at> inria.fr> wrote:
> zimoun <zimon.toutoune <at> gmail.com> skribis:

> >> Could you, in a first patch, move ‘origin->json’ out of
> >> ‘packages-json-builder’, and in a second patch, add mirror-expansion
> >> feature?
> >
> > Yes, I will try but I am a bit lost. There is:
> >  - packages-json-builder that I did not modified
> >  - sources-json-builder which the "adaptation" of the former to output
> > to the sources.json format.
> >
> > Well, do you want a refactor of 'origin->json' shared by the 2
> > "{sources,packages}-json-builder"?
>
> Yes, exactly.  No reason to have two copies of that code.

The minor "issue" is the 'match' piece of 'origin->json':

--8<---------------cut here---------------start------------->8---
               `(("url" . ,(match uri
                             ((? string? url) (vector url))
                             ((urls ...) (list->vector urls))))))
--8<---------------cut here---------------end--------------->8---

Because in the 'sources.json' it is always one unique string.
However, in 'packages.json' it is always a vector of one or several
elements. And in the previous patch, the job of 'resolve' was to
uniquify these elements for 'sources.json'.

For example, see the package 'adns' in 'packages.json'

--8<---------------cut here---------------start------------->8---
{
    "name": "adns",
    "version": "1.5.1",
    "source": {
      "type": "url",
      "url": [
        "mirror://gnu/adns/adns-1.5.1.tar.gz",
        "http://www.chiark.greenend.org.uk/~ian/adns/ftp/adns-1.5.1.tar.gz"
      ]
    },
    "synopsis": "Asynchronous DNS client library and utilities",
    "homepage": "https://www.gnu.org/software/adns/",
    "location": "gnu/packages/adns.scm:32"
  },
--8<---------------cut here---------------end--------------->8---



Just to be on the same wavelength, does 'packages.json' stay as it is
(vector for the source url field)?
Or is 'packages.json' modified to record only one url for the source
url field with mirror:// expanded?


Cheers,
simon

Message #20 received at 39547 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludovic.courtes <at> inria.fr>
To: zimoun <zimon.toutoune <at> gmail.com>
Cc: 39547 <at> debbugs.gnu.org
Subject: Re: [bug#39547] [PATCH] website: Provide JSON sources list used by
 Software Heritage.
Date: Tue, 18 Feb 2020 09:24:49 +0100

Hi!

zimoun <zimon.toutoune <at> gmail.com> skribis:

> On Fri, 14 Feb 2020 at 11:20, Ludovic Courtès <ludovic.courtes <at> inria.fr> wrote:
>> zimoun <zimon.toutoune <at> gmail.com> skribis:
>
>> >> Could you, in a first patch, move ‘origin->json’ out of
>> >> ‘packages-json-builder’, and in a second patch, add mirror-expansion
>> >> feature?
>> >
>> > Yes, I will try but I am a bit lost. There is:
>> >  - packages-json-builder that I did not modified
>> >  - sources-json-builder which the "adaptation" of the former to output
>> > to the sources.json format.
>> >
>> > Well, do you want a refactor of 'origin->json' shared by the 2
>> > "{sources,packages}-json-builder"?
>>
>> Yes, exactly.  No reason to have two copies of that code.
>
> The minor "issue" is the 'match' piece of 'origin->json':
>
>                `(("url" . ,(match uri
>                              ((? string? url) (vector url))
>                              ((urls ...) (list->vector urls))))))
>
>
> Because in the 'sources.json' it is always one unique string.
> However, in 'packages.json' it is always a vector of one or several
> elements. And in the previous patch, the job of 'resolve' was to
> uniquify these elements for 'sources.json'.

Ah, good point.  What about adding a keyword parameter to ‘origin->json’
that would tell whether to return a single URL or a list thereof?

> For example, see the package 'adns' in 'packages.json'
>
> {
>     "name": "adns",
>     "version": "1.5.1",
>     "source": {
>       "type": "url",
>       "url": [
>         "mirror://gnu/adns/adns-1.5.1.tar.gz",
>         "http://www.chiark.greenend.org.uk/~ian/adns/ftp/adns-1.5.1.tar.gz"
>       ]
>     },
>     "synopsis": "Asynchronous DNS client library and utilities",
>     "homepage": "https://www.gnu.org/software/adns/",
>     "location": "gnu/packages/adns.scm:32"
>   },
>
>
>
> Just to be on the same wavelength, does 'packages.json' stay as it is
> (vector for the source url field)?

I think ‘packages.json’ should still return a list of URLs.

I’ve also added a comment asking whether ‘sources.json’ could be changed
to contain a list of URLs:

  https://forge.softwareheritage.org/D2025#64037

> Or is 'packages.json' modified to record only one url for the source
> url field with mirror:// expanded?

Yeah, I think we can expand mirror URLs.

Thanks,
Ludo’.

Message #23 received at 39547 <at> debbugs.gnu.org (full text, mbox):

From: zimoun <zimon.toutoune <at> gmail.com>
To: Ludovic Courtès <ludovic.courtes <at> inria.fr>
Cc: 39547 <at> debbugs.gnu.org
Subject: Re: [bug#39547] [PATCH] website: Provide JSON sources list used by
 Software Heritage.
Date: Tue, 18 Feb 2020 09:38:58 +0100

Hi Ludo,

On Tue, 18 Feb 2020 at 09:24, Ludovic Courtès <ludovic.courtes <at> inria.fr> wrote:
> zimoun <zimon.toutoune <at> gmail.com> skribis:
> > On Fri, 14 Feb 2020 at 11:20, Ludovic Courtès <ludovic.courtes <at> inria.fr> wrote:
> >> zimoun <zimon.toutoune <at> gmail.com> skribis:

> >> >> Could you, in a first patch, move ‘origin->json’ out of
> >> >> ‘packages-json-builder’, and in a second patch, add mirror-expansion
> >> >> feature?
> >> >
> >> > Yes, I will try but I am a bit lost. There is:
> >> >  - packages-json-builder that I did not modified
> >> >  - sources-json-builder which the "adaptation" of the former to output
> >> > to the sources.json format.
> >> >
> >> > Well, do you want a refactor of 'origin->json' shared by the 2
> >> > "{sources,packages}-json-builder"?
> >>
> >> Yes, exactly.  No reason to have two copies of that code.
> >
> > The minor "issue" is the 'match' piece of 'origin->json':
> >
> >                `(("url" . ,(match uri
> >                              ((? string? url) (vector url))
> >                              ((urls ...) (list->vector urls))))))
> >
> >
> > Because in the 'sources.json' it is always one unique string.
> > However, in 'packages.json' it is always a vector of one or several
> > elements. And in the previous patch, the job of 'resolve' was to
> > uniquify these elements for 'sources.json'.
>
> Ah, good point.  What about adding a keyword parameter to ‘origin->json’
> that would tell whether to return a single URL or a list thereof?

Sharing the same code.

 - packages.json: as it is now.
 - sources.json: expand mirror and resolve to display only URL.

I am going for that.


> > Just to be on the same wavelength, does 'packages.json' stay as it is
> > (vector for the source url field)?
>
> I think ‘packages.json’ should still return a list of URLs.

Ok.


> I’ve also added a comment asking whether ‘sources.json’ could be changed
> to contain a list of URLs:
>
>   https://forge.softwareheritage.org/D2025#64037

I have seen. :-)
Even, all the mirrors list should be expanded, not only the first one.


> > Or is 'packages.json' modified to record only one url for the source
> > url field with mirror:// expanded?
>
> Yeah, I think we can expand mirror URLs.

So, 'mirror://' will become a list (from '%mirrors') of URLs, right?


Cheers,
simon

Message #26 received at 39547 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludovic.courtes <at> inria.fr>
To: zimoun <zimon.toutoune <at> gmail.com>
Cc: 39547 <at> debbugs.gnu.org
Subject: Re: [bug#39547] [PATCH] website: Provide JSON sources list used by
 Software Heritage.
Date: Tue, 18 Feb 2020 09:43:01 +0100

Hi,

zimoun <zimon.toutoune <at> gmail.com> skribis:

> On Tue, 18 Feb 2020 at 09:24, Ludovic Courtès <ludovic.courtes <at> inria.fr> wrote:

[...]

>> > Or is 'packages.json' modified to record only one url for the source
>> > url field with mirror:// expanded?
>>
>> Yeah, I think we can expand mirror URLs.
>
> So, 'mirror://' will become a list (from '%mirrors') of URLs, right?

Yup!

Thanks,
Ludo’.

Message #29 received at 39547 <at> debbugs.gnu.org (full text, mbox):

From: zimoun <zimon.toutoune <at> gmail.com>
To: 39547 <at> debbugs.gnu.org
Cc: zimoun <zimon.toutoune <at> gmail.com>
Subject: [PATCH v2 1/2] website: Refactor and resolve mirror:// of JSON
 package list.
Date: Tue, 18 Feb 2020 13:32:45 +0100

* website/apps/packages/builder.scm (origin->json): New procedure.
---
 website/apps/packages/builder.scm | 34 ++++++++++++++++++++++---------
 1 file changed, 24 insertions(+), 10 deletions(-)

diff --git a/website/apps/packages/builder.scm b/website/apps/packages/builder.scm
index 9dc44c9..d3a777e 100644
--- a/website/apps/packages/builder.scm
+++ b/website/apps/packages/builder.scm
@@ -2,6 +2,7 @@
 ;;; Copyright © 2017 Ludovic Courtès <ludo <at> gnu.org>
 ;;; Copyright © 2019 Ricardo Wurmus <rekado <at> elephly.net>
 ;;; Copyright © 2019 Nicolò Balzarotti <nicolo <at> nixo.xyz>
+;;; Copyright © 2020 Simon Tournier <zimon.toutoune <at> gmail.com>
 ;;;
 ;;; Initially written by sirgazil
 ;;; who waives all copyright interest on this file.
@@ -37,13 +38,16 @@
   #:use-module (haunt page)
   #:use-module (haunt utils)
   #:use-module (srfi srfi-1)
+  #:use-module (srfi srfi-26)
   #:use-module (guix packages)
   #:use-module (guix download)
   #:use-module (guix git-download)
   #:use-module (guix svn-download)
   #:use-module (guix utils)                       ;location
+  #:use-module ((guix build download) #:select (maybe-expand-mirrors))
   #:use-module (json)
   #:use-module (ice-9 match)
+  #:use-module ((web uri) #:select (string->uri uri->string))
   #:export (builder))
 
 
@@ -84,33 +88,43 @@
   ;; Maximum number of packages shown on /packages.
   30)
 
-(define (packages-json-builder)
-  "Return a JSON page listing all packages."
-  (define (origin->json origin)
+(define (origin->json origin)
     (define method
       (origin-method origin))
 
+    (define uri                         ;represented as string
+      (origin-uri origin))
+
+    (define (resolve urls)
+      (map uri->string
+           (append-map (cut maybe-expand-mirrors <> %mirrors)
+                       (map string->uri urls))))
+
     `((type . ,(cond ((eq? url-fetch method) 'url)
                      ((eq? git-fetch method) 'git)
                      ((eq? svn-fetch method) 'svn)
                      (else                   #nil)))
       ,@(cond ((eq? url-fetch method)
-               `(("url" . ,(match (origin-uri origin)
-                             ((? string? url) (vector url))
-                             ((urls ...) (list->vector urls))))))
+               `(("url" . ,(list->vector
+                            (resolve
+                             (match uri
+                               ((? string? url) (list url))
+                               ((urls ...) urls)))))))
               ((eq? git-fetch method)
-               `(("git_url" . ,(git-reference-url (origin-uri origin)))))
+               `(("git_url" . ,(git-reference-url uri))))
               ((eq? svn-fetch method)
-               `(("svn_url" . ,(svn-reference-url (origin-uri origin)))))
+               `(("svn_url" . ,(svn-reference-url uri))))
               (else '()))
       ,@(if (eq? method git-fetch)
-            `(("git_ref" . ,(git-reference-commit (origin-uri origin))))
+            `(("git_ref" . ,(git-reference-commit uri)))
             '())
       ,@(if (eq? method svn-fetch)
             `(("svn_revision" . ,(svn-reference-revision
-                                  (origin-uri origin))))
+                                  uri)))
             '())))
 
+(define (packages-json-builder)
+  "Return a JSON page listing all packages."
   (define (package->json package)
     (define cpe-name
       (assoc-ref (package-properties package) 'cpe-name))
-- 
2.25.0

Message #32 received at 39547 <at> debbugs.gnu.org (full text, mbox):

From: zimoun <zimon.toutoune <at> gmail.com>
To: 39547 <at> debbugs.gnu.org
Cc: zimoun <zimon.toutoune <at> gmail.com>
Subject: [PATCH v2 2/2] website: Provide JSON sources list used by Software
 Heritage.
Date: Tue, 18 Feb 2020 13:32:46 +0100

Format discussed here <https://forge.softwareheritage.org/D2025#51269>.

* website/apps/packages/builder.scm (origin->json): Add list modifier.
* website/apps/packages/builder.scm (sources-json-builder): New procedure.
---
 website/apps/packages/builder.scm | 22 +++++++++++++++++++---
 1 file changed, 19 insertions(+), 3 deletions(-)

diff --git a/website/apps/packages/builder.scm b/website/apps/packages/builder.scm
index d3a777e..49721f6 100644
--- a/website/apps/packages/builder.scm
+++ b/website/apps/packages/builder.scm
@@ -74,6 +74,7 @@
   (flatten
    (list
     (index-builder)
+    (sources-json-builder)
     (packages-json-builder)
     (packages-builder)
     (package-list-builder))))
@@ -88,7 +89,7 @@
   ;; Maximum number of packages shown on /packages.
   30)
 
-(define (origin->json origin)
+(define (origin->json origin transformer)
     (define method
       (origin-method origin))
 
@@ -105,7 +106,7 @@
                      ((eq? svn-fetch method) 'svn)
                      (else                   #nil)))
       ,@(cond ((eq? url-fetch method)
-               `(("url" . ,(list->vector
+               `(("url" . ,(transformer
                             (resolve
                              (match uri
                                ((? string? url) (list url))
@@ -136,7 +137,7 @@
       ,@(if cpe-name `(("cpe_name" . ,cpe-name)) '())
       ,@(if cpe-version `(("cpe_version" . ,cpe-version)) '())
       ,@(if (origin? (package-source package))
-            `(("source" . ,(origin->json (package-source package))))
+            `(("source" . ,(origin->json (package-source package) list->vector)))
             '())
       ("synopsis" . ,(package-synopsis package))
       ,@(if (package-home-page package)
@@ -155,6 +156,21 @@
 	     (list->vector (map package->json (all-packages)))
              scm->json))
 
+(define (sources-json-builder)
+  "Return a JSON page listing all the sources.
+
+See <https://forge.softwareheritage.org/D2025#51269>."
+  (define (package->json package)
+    `(,@(if (origin? (package-source package))
+            (origin->json (package-source package) car)
+            `(("type" . "no-origin")
+              ("name" . ,(package-name package))))))
+
+  (make-page "sources.json"
+             `(("sources" . ,(list->vector (map package->json (all-packages))))
+               ("version" . "1"))
+             scm->json))
+
 (define (index-builder)
   "Return a Haunt page listing some random packages."
   (define (sample n from)
-- 
2.25.0

Message #35 received at 39547 <at> debbugs.gnu.org (full text, mbox):

From: zimoun <zimon.toutoune <at> gmail.com>
To: 39547 <at> debbugs.gnu.org, Ludovic Courtès <ludo <at> gnu.org>
Subject: [PATCH v3] sources.json: array instead of list
Date: Mon, 2 Mar 2020 18:24:49 +0100

[Message part 1 (text/plain, inline)]

Hi Ludo,

Attached, the tiny modification to output the list (array) of URLs
instead of the first one.
The version number is still '1' because I do not know yet if lewo would bump it.
Note also that "transformer" is not useful any more because
'packages-json-builder' and 'sources-json-builder' returns both
vectors now.


Cheers,
simon

[v3-0001-website-Refactor-and-resolve-mirror-of-JSON-packa.patch (text/x-patch, attachment)]

[v3-0002-website-Provide-JSON-sources-list-used-by-Softwar.patch (text/x-patch, attachment)]

Message #38 received at 39547 <at> debbugs.gnu.org (full text, mbox):

From: zimoun <zimon.toutoune <at> gmail.com>
To: 39547 <at> debbugs.gnu.org, Ludovic Courtès <ludo <at> gnu.org>
Subject: Addition of %content-addressed-mirrors to sources.json (SWH)?
Date: Tue, 3 Mar 2020 19:00:23 +0100

Hi,

Some upstream sources could disappear between the moment when the
package enters in Guix and the moment when SWH ingests it. (As an edge
case, let consider the recent harfbuzz one.)

Now the format 'sources.json' should accept an array. Does it make
sense to put in this array the content addressed mirrors
(ci.guix.gnu.org and tarball.nixos.org)?

What do you think?


Cheers,
simon

Message #41 received at 39547 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: zimoun <zimon.toutoune <at> gmail.com>
Cc: 39547 <at> debbugs.gnu.org
Subject: Re: Addition of %content-addressed-mirrors to sources.json (SWH)?
Date: Thu, 05 Mar 2020 17:19:11 +0100

Hello!

zimoun <zimon.toutoune <at> gmail.com> skribis:

> Some upstream sources could disappear between the moment when the
> package enters in Guix and the moment when SWH ingests it. (As an edge
> case, let consider the recent harfbuzz one.)
>
> Now the format 'sources.json' should accept an array. Does it make
> sense to put in this array the content addressed mirrors
> (ci.guix.gnu.org and tarball.nixos.org)?

Yes, I think ‘sources.json’ should contain, for each item, a list of
URLs rather than a single URL, and mirror:// URLs should be expanded.

However, ‘sources.json’ is currently specified to contain a single URL,
so we’re stuck with a single URL for now.

Is that what you meant?

Thanks,
Ludo’.

Message #44 received at 39547 <at> debbugs.gnu.org (full text, mbox):

From: zimoun <zimon.toutoune <at> gmail.com>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 39547 <at> debbugs.gnu.org
Subject: Re: Addition of %content-addressed-mirrors to sources.json (SWH)?
Date: Fri, 6 Mar 2020 10:26:36 +0100

Hi Ludo,

On Thu, 5 Mar 2020 at 17:19, Ludovic Courtès <ludo <at> gnu.org> wrote:
> zimoun <zimon.toutoune <at> gmail.com> skribis:
>
> > Some upstream sources could disappear between the moment when the
> > package enters in Guix and the moment when SWH ingests it. (As an edge
> > case, let consider the recent harfbuzz one.)
> >
> > Now the format 'sources.json' should accept an array. Does it make
> > sense to put in this array the content addressed mirrors
> > (ci.guix.gnu.org and tarball.nixos.org)?
>
> Yes, I think ‘sources.json’ should contain, for each item, a list of
> URLs rather than a single URL, and mirror:// URLs should be expanded.

Yes, now it is accepted [1] and it becomes the format of the
'sources.json' file.

[1] https://forge.softwareheritage.org/D2025#65063


The patch set v2 supports the first discussion of the format (single
url); which is not relevant anymore.
The patch set attached in v3 [2] supports the array of URLs.

[2] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=39547#35



> However, ‘sources.json’ is currently specified to contain a single URL,
> so we’re stuck with a single URL for now.
>
> Is that what you meant?

What I mean is: append %content-addressed-mirrors for each package in
the list of URLs.



Cheers,
simon

Message #47 received at 39547 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: zimoun <zimon.toutoune <at> gmail.com>
Cc: 39547 <at> debbugs.gnu.org
Subject: Re: Addition of %content-addressed-mirrors to sources.json (SWH)?
Date: Fri, 06 Mar 2020 11:57:01 +0100

Hi Simon!

zimoun <zimon.toutoune <at> gmail.com> skribis:

> On Thu, 5 Mar 2020 at 17:19, Ludovic Courtès <ludo <at> gnu.org> wrote:

[...]

>> Yes, I think ‘sources.json’ should contain, for each item, a list of
>> URLs rather than a single URL, and mirror:// URLs should be expanded.
>
> Yes, now it is accepted [1] and it becomes the format of the
> 'sources.json' file.
>
> [1] https://forge.softwareheritage.org/D2025#65063

Excellent!

> The patch set v2 supports the first discussion of the format (single
> url); which is not relevant anymore.
> The patch set attached in v3 [2] supports the array of URLs.
>
> [2] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=39547#35

Oh sorry, I hadn’t seen it.

>> However, ‘sources.json’ is currently specified to contain a single URL,
>> so we’re stuck with a single URL for now.
>>
>> Is that what you meant?
>
> What I mean is: append %content-addressed-mirrors for each package in
> the list of URLs.

Now I understand.  :-)  So yes, sure, the array of URLs in
‘sources.json’ could include a ci.guix.gnu.org URL and maybe the other
ones in there too.  For a subsequent patch I guess?

Thanks!

Ludo’.

Message #50 received at 39547 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludovic.courtes <at> inria.fr>
To: zimoun <zimon.toutoune <at> gmail.com>
Cc: 39547 <at> debbugs.gnu.org
Subject: Re: [bug#39547] [PATCH v3] sources.json: array instead of list
Date: Fri, 06 Mar 2020 12:01:36 +0100

Hi!

zimoun <zimon.toutoune <at> gmail.com> skribis:

> Attached, the tiny modification to output the list (array) of URLs
> instead of the first one.
> The version number is still '1' because I do not know yet if lewo would bump it.
> Note also that "transformer" is not useful any more because
> 'packages-json-builder' and 'sources-json-builder' returns both
> vectors now.

Cool.

> From 57a444f6f215fb6327719161a6e6ad4ad229273f Mon Sep 17 00:00:00 2001
> From: zimoun <zimon.toutoune <at> gmail.com>
> Date: Mon, 10 Feb 2020 17:52:13 +0100
> Subject: [PATCH v3 1/2] website: Refactor and resolve mirror:// of JSON
>  package list.
>
> * website/apps/packages/builder.scm (origin->json): New procedure.

LGTM!

> From 73557bc00760b4404bfe17ecb3aca983c6dcc11e Mon Sep 17 00:00:00 2001
> From: zimoun <zimon.toutoune <at> gmail.com>
> Date: Tue, 18 Feb 2020 13:25:14 +0100
> Subject: [PATCH v3 2/2] website: Provide JSON sources list used by Software
>  Heritage.
>
> Format discussed here <https://forge.softwareheritage.org/D2025#51269>.
>
> * website/apps/packages/builder.scm (origin->json): Add list modifier.
> * website/apps/packages/builder.scm (sources-json-builder): New procedure.

Nitpick: no need to repeat the file name here.

Also: please mention the addition to the ‘builder’ procedure.

> +(define* (origin->json origin #:optional (transformer list->vector))

You can remove this parameter, as you wrote.

I can make these changes and push it on your behalf if you want, let me
know!

Thanks,
Ludo’.

Message #53 received at 39547 <at> debbugs.gnu.org (full text, mbox):

From: zimoun <zimon.toutoune <at> gmail.com>
To: Ludovic Courtès <ludovic.courtes <at> inria.fr>
Cc: "39547 <at> debbugs.gnu.org" <39547 <at> debbugs.gnu.org>
Subject: Re: [PATCH v3] sources.json: array instead of list
Date: Sat, 7 Mar 2020 23:17:48 +0100

[Message part 1 (text/plain, inline)]

Hi Ludo,

Yes please push.

Cheers,
simon

PS: i am typing with my smartphone because i am sleeping in a hospital
(Purpan, Toulouse). I lose part of sensitivity and mobility of my right arm
after a shock to my neck during a basket ball party with some friends.
Everything is on the process but I cannot say when i will have access to my
laptop next ; maybe Tomorrow, maybe one week later. So please go ahead (-:

On Friday, 6 March 2020, Ludovic Courtès <ludovic.courtes <at> inria.fr> wrote:

> Hi!
>
> zimoun <zimon.toutoune <at> gmail.com> skribis:
>
> > Attached, the tiny modification to output the list (array) of URLs
> > instead of the first one.
> > The version number is still '1' because I do not know yet if lewo would
> bump it.
> > Note also that "transformer" is not useful any more because
> > 'packages-json-builder' and 'sources-json-builder' returns both
> > vectors now.
>
> Cool.
>
> > From 57a444f6f215fb6327719161a6e6ad4ad229273f Mon Sep 17 00:00:00 2001
> > From: zimoun <zimon.toutoune <at> gmail.com>
> > Date: Mon, 10 Feb 2020 17:52:13 +0100
> > Subject: [PATCH v3 1/2] website: Refactor and resolve mirror:// of JSON
> >  package list.
> >
> > * website/apps/packages/builder.scm (origin->json): New procedure.
>
> LGTM!
>
> > From 73557bc00760b4404bfe17ecb3aca983c6dcc11e Mon Sep 17 00:00:00 2001
> > From: zimoun <zimon.toutoune <at> gmail.com>
> > Date: Tue, 18 Feb 2020 13:25:14 +0100
> > Subject: [PATCH v3 2/2] website: Provide JSON sources list used by
> Software
> >  Heritage.
> >
> > Format discussed here <https://forge.softwareheritage.org/D2025#51269>.
> >
> > * website/apps/packages/builder.scm (origin->json): Add list modifier.
> > * website/apps/packages/builder.scm (sources-json-builder): New
> procedure.
>
> Nitpick: no need to repeat the file name here.
>
> Also: please mention the addition to the ‘builder’ procedure.
>
> > +(define* (origin->json origin #:optional (transformer list->vector))
>
> You can remove this parameter, as you wrote.
>
> I can make these changes and push it on your behalf if you want, let me
> know!
>
> Thanks,
> Ludo’.
>

[Message part 2 (text/html, inline)]

Message #58 received at 39547-done <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludovic.courtes <at> inria.fr>
To: zimoun <zimon.toutoune <at> gmail.com>
Cc: "39547 <at> debbugs.gnu.org" <39547-done <at> debbugs.gnu.org>
Subject: Re: [PATCH v3] sources.json: array instead of list
Date: Mon, 09 Mar 2020 10:53:09 +0100

Hi,

zimoun <zimon.toutoune <at> gmail.com> skribis:

> Yes please push.

Done!  <https://guix.gnu.org/sources.json> should show up within an hour.

> PS: i am typing with my smartphone because i am sleeping in a hospital
> (Purpan, Toulouse). I lose part of sensitivity and mobility of my right arm
> after a shock to my neck during a basket ball party with some friends.
> Everything is on the process but I cannot say when i will have access to my
> laptop next ; maybe Tomorrow, maybe one week later. So please go ahead (-:

Ouch, I wish you quick recovery!

Take care of yourself,
Ludo’.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.