GNU bug report logs - #28251
[PATCH 0/3] Add generic JSON importer

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 28251 in the body.
You can then email your comments to 28251 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Ricardo Wurmus <rekado <at> elephly.net>
To: guix-patches <at> gnu.org
Cc: Ricardo Wurmus <rekado <at> elephly.net>
Subject: [PATCH 0/3] Add generic JSON importer
Date: Sun, 27 Aug 2017 17:58:20 +0200

Hi Guix,

this patch set adds a somewhat unusual importer.  Assume we have a file
"package.json" with the following contents:

--8<---------------cut here---------------start------------->8---
{
  "name": "hello",
  "version": "2.10",
  "source": {
    "method": "url-fetch",
    "uri": "mirror://gnu/hello/hello-2.10.tar.gz",
    "sha256": {
      "base32": "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i"
    }
  }
  "build-system": "gnu",
  "home-page": "https://www.gnu.org/software/hello/",
  "synopsis": "Hello, GNU world: An example GNU package",
  "description": "It really works.",
  "license": "GPL-3.0+",
  "inputs": ["r-minimal <at> 3", "ghc-pandoc", "samtools <at> 0"]
}
--8<---------------cut here---------------end--------------->8---

Let’s run the new “json” importer on this file:

--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env guix import json package.json

(package
  (name "hello")
  (version "2.10")
  (source
    (origin
      (uri (string-append
             "mirror://gnu/hello/hello-2.10.tar.gz"))
      (method url-fetch)
      (sha256
        (base32
          "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i"))))
  (build-system r-build-system)
  (inputs
    `(("r-minimal"
       ,(@ (gnu packages statistics) r-minimal))
      ("ghc-pandoc"
       ,(@ (gnu packages haskell) ghc-pandoc))
      ("samtools"
       ,(@ (gnu packages bioinformatics) samtools-0.1))))
  (home-page "https://www.gnu.org/software/hello/")
  (synopsis
    "Hello, GNU world: An example GNU package")
  (description "It really works.")
  (license gpl3+))
--8<---------------cut here---------------end--------------->8---

What you don’t see here is that the JSON importer internally creates a
package object, which could already be built (e.g. from within the REPL)
— without having to write it to a file first and setting
GUIX_PACKAGE_PATH.

What is this good for?  Users could create simple Guix packages for
their own immature projects using a syntax that they may be more
familiar with and generate a proper Scheme package definition from it to
allow other people to install it.  For more complicated packages they
would, of course, be better served by using the usual Scheme syntax for
package definitions.

To make the importer behave like all other importers, we use the new
“package->code” procedure, which takes a package and generates the code,
which would create an equivalent package object when evaluated.

There are some minor problems with “package->code”, which are marked
with FIXME comments.  We probably shouldn’t print out “(@ (gnu packages
statistics) r-minimal)” and instead let “package->code” return two
values: the package code and a list of modules needed to evaluate it.

What do you think?  Terrible?  Exciting?  Both? *raises hand*

Documentation of this importer is missing because I’m not sure if this
is the best way of doing this.  Let’s discuss!

~~ Ricardo

Ricardo Wurmus (3):
  packages: Add package->code.
  import: Add generic data to package converter.
  import: Add JSON importer.

 guix/import/utils.scm        |  77 ++++++++++++++++++++++++-
 guix/packages.scm            | 131 +++++++++++++++++++++++++++++++++++++++++++
 guix/scripts/import.scm      |   2 +-
 guix/scripts/import/json.scm | 101 +++++++++++++++++++++++++++++++++
 4 files changed, 309 insertions(+), 2 deletions(-)
 create mode 100644 guix/scripts/import/json.scm

-- 
2.14.1

Message #8 received at 28251 <at> debbugs.gnu.org (full text, mbox):

From: Ricardo Wurmus <rekado <at> elephly.net>
To: 28251 <at> debbugs.gnu.org
Cc: Ricardo Wurmus <rekado <at> elephly.net>
Subject: [PATCH 1/3] packages: Add package->code.
Date: Sun, 27 Aug 2017 18:00:44 +0200

* guix/packages.scm (package->code): New procedure.
---
 guix/packages.scm | 131 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 131 insertions(+)

diff --git a/guix/packages.scm b/guix/packages.scm
index f619d9b37..d25920010 100644
--- a/guix/packages.scm
+++ b/guix/packages.scm
@@ -4,6 +4,7 @@
 ;;; Copyright © 2015 Eric Bavier <bavier <at> member.fsf.org>
 ;;; Copyright © 2016 Alex Kost <alezost <at> gmail.com>
 ;;; Copyright © 2017 Efraim Flashner <efraim <at> flashner.co.il>
+;;; Copyright © 2017 Ricardo Wurmus <rekado <at> elephly.net>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -31,6 +32,7 @@
   #:use-module (guix derivations)
   #:use-module (guix memoization)
   #:use-module (guix build-system)
+  #:use-module (guix licenses)
   #:use-module (guix search-paths)
   #:use-module (guix sets)
   #:use-module (ice-9 match)
@@ -84,6 +86,7 @@
             package-maintainers
             package-properties
             package-location
+            package->code
             hidden-package
             hidden-package?
             package-superseded
@@ -306,6 +309,134 @@ name of its URI."
                                                        package)
                                                       16)))))
 
+;; FIXME: the quasiquoted arguments field may contain embedded package
+;; objects, e.g. in #:disallowed-references; they will just be printed with
+;; their usual #<package ...> representation, not as variable names.
+(define (package->code package)
+  "Return an S-expression representing the source code that produces PACKAGE
+when evaluated."
+  ;; The module in which the package PKG is defined
+  (define (package-module-name pkg)
+    (map string->symbol
+         (string-split (string-drop-right
+                        (location-file (package-location pkg)) 4)
+                       #\/)))
+
+  ;; Return the first candidate variable name that is bound to VAL.
+  ;; TODO: avoid '%pkg-config
+  (define (variable-name val mod)
+    (let ((candidates (filter identity
+                              (module-map
+                               (lambda (sym var)
+                                 (if (equal? val (variable-ref var)) sym #f))
+                               (resolve-interface mod)))))
+      (if (null? candidates) #f (car candidates))))
+
+  ;; Print either license variable name or the code for a license object
+  (define (print-license lic)
+    (let ((var (variable-name lic '(guix licenses))))
+      (or var
+          `(license
+            (name ,(license-name lic))
+            (uri ,(license-uri lic))
+            (comment ,(license-comment lic))))))
+
+  (define (print-search-path-specification spec)
+    `(search-path-specification
+      (variable ,(search-path-specification-variable spec))
+      (files (list ,@(search-path-specification-files spec)))
+      (separator ,(search-path-specification-separator spec))
+      (file-type (quote ,(search-path-specification-file-type spec)))
+      (file-pattern ,(search-path-specification-file-pattern spec))))
+
+  (define (print-source source version)
+    ;; FIXME: we cannot use factorize-uri because (guix import utils)
+    ;; cannot be imported in this module.
+    (let ((factorize-uri (lambda (uri version)
+                           (list uri))))
+      (match source
+        (($ <origin> uri method sha256 file-name patches)
+         `(origin
+            (uri (string-append ,@(factorize-uri uri version)))
+            (method ,(procedure-name method))
+            (sha256
+             (base32
+              ,(format #f "~a" (bytevector->nix-base32-string sha256))))
+            ;; FIXME: in order to be able to throw away the directory prefix,
+            ;; we just assume that the patch files can be found with
+            ;; "search-patches".
+            ,@(let ((ps (force patches)))
+                (if (null? ps) '()
+                    `((patches (search-patches ,@(map basename ps)))))))))))
+
+  (define (print-package-lists lsts)
+    (list 'quasiquote
+          (map (match-lambda
+                 ((label pkg)
+                  (let ((mod (package-module-name pkg)))
+                    (list label
+                          ;; FIXME: using '@ certainly isn't pretty, but it
+                          ;; avoids having to import the individual package
+                          ;; modules.
+                          (list 'unquote
+                                (list '@ mod (variable-name pkg mod)))))))
+               lsts)))
+
+  (match package
+    (($ <package> name version source build-system
+                  arguments inputs propagated-inputs native-inputs
+                  self-native-input?
+                  outputs
+                  native-search-paths
+                  search-paths
+                  replacement
+                  synopsis description license
+                  home-page supported-systems maintainers
+                  properties location)
+     `(package
+        (name ,name)
+        (version ,version)
+        (source ,(print-source source version))
+        ,@(if (null? properties)  '()
+              `((properties ,properties)))
+        ,@(let ((rep (replacement)))
+            (if rep
+                `((replacement ,rep))
+                '()))
+        (build-system ,(symbol-append (build-system-name build-system)
+                                      '-build-system))
+        ,@(let ((args (arguments)))
+            (if (null? args) '()
+                `((arguments ,(list 'quasiquote (arguments))))))
+        ,@(if (equal? outputs '("out")) '()
+              `((outputs (list ,@outputs))))
+        ,@(let ((pkgs (native-inputs)))
+            (if (null? pkgs) '()
+                `((native-inputs ,(print-package-lists pkgs)))))
+        ,@(let ((pkgs (inputs)))
+            (if (null? pkgs) '()
+                `((inputs ,(print-package-lists pkgs)))))
+        ,@(let ((pkgs (propagated-inputs)))
+            (if (null? pkgs) '()
+                `((propagated-inputs ,(print-package-lists pkgs)))))
+        ,@(if (lset= string=? supported-systems %supported-systems)
+              '()
+              `((supported-systems (list ,@supported-systems))))
+        ,@(let ((paths (map print-search-path-specification native-search-paths)))
+            (if (null? paths) '()
+                `((native-search-paths
+                   (list ,@paths)))))
+        ,@(let ((paths (map print-search-path-specification search-paths)))
+            (if (null? paths) '()
+                `((search-paths
+                   (list ,@paths)))))
+        (home-page ,home-page)
+        (synopsis ,synopsis)
+        (description ,description)
+        (license ,(if (list? license)
+                      `(list ,@(map print-license license))
+                      (print-license license)))))))
+
 (define (package-upstream-name package)
   "Return the upstream name of PACKAGE, which could be different from the name
 it has in Guix."
-- 
2.14.1

Message #11 received at 28251 <at> debbugs.gnu.org (full text, mbox):

From: Ricardo Wurmus <rekado <at> elephly.net>
To: 28251 <at> debbugs.gnu.org
Cc: Ricardo Wurmus <rekado <at> elephly.net>
Subject: [PATCH 2/3] import: Add generic data to package converter.
Date: Sun, 27 Aug 2017 18:00:45 +0200

* guix/import/utils.scm (build-system-modules, guix-modules): New variables.
(lookup-build-system-by-name, specs->package-lists, convert-source,
data->guix-package): New procedures.
---
 guix/import/utils.scm | 77 ++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 76 insertions(+), 1 deletion(-)

diff --git a/guix/import/utils.scm b/guix/import/utils.scm
index be1980d08..edc6fda26 100644
--- a/guix/import/utils.scm
+++ b/guix/import/utils.scm
@@ -2,6 +2,7 @@
 ;;; Copyright © 2012, 2013 Ludovic Courtès <ludo <at> gnu.org>
 ;;; Copyright © 2016 Jelle Licht <jlicht <at> fsfe.org>
 ;;; Copyright © 2016 David Craven <david <at> craven.ch>
+;;; Copyright © 2017 Ricardo Wurmus <rekado <at> elephly.net>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -25,6 +26,10 @@
   #:use-module (guix http-client)
   #:use-module ((guix licenses) #:prefix license:)
   #:use-module (guix utils)
+  #:use-module (guix packages)
+  #:use-module (guix discovery)
+  #:use-module (guix build-system)
+  #:use-module (gnu packages)
   #:use-module (ice-9 match)
   #:use-module (ice-9 regex)
   #:use-module (srfi srfi-1)
@@ -45,7 +50,9 @@
             license->symbol
 
             snake-case
-            beautify-description))
+            beautify-description
+
+            data->guix-package))
 
 (define (factorize-uri uri version)
   "Factorize URI, a package tarball URI as a string, such that any occurrences
@@ -241,3 +248,71 @@ package definition."
     (('package ('name (? string? name)) _ ...)
      `(define-public ,(string->symbol name)
         ,guix-package))))
+
+(define build-system-modules
+  (all-modules (map (lambda (entry)
+                      `(,entry . "guix/build-system"))
+                    %load-path)))
+
+(define guix-modules
+  (all-modules (map (lambda (entry)
+                      `(,entry . "guix"))
+                    %load-path)))
+
+(define (lookup-build-system-by-name name)
+  (fold-module-public-variables (lambda (obj result)
+                                  (if (and (build-system? obj)
+                                           (eq? name (build-system-name obj)))
+                                      obj result))
+                                #f
+                                build-system-modules))
+
+(define (specs->package-lists specs)
+  (map (lambda (spec)
+         (let ((pkg (specification->package spec)))
+           (list (package-name pkg) pkg)))
+       specs))
+
+(define (convert-source source)
+  (match source
+    ((? string? file) (local-file file))
+    (#f #f)
+    (orig (let ((sha (match (car (assoc-ref orig "sha256"))
+                       (("base32" . value)
+                        (base32 value))
+                       (_ #f))))
+            (origin
+              (method (match (assoc-ref orig "method")
+                        ("url-fetch" (@ (guix download) url-fetch))
+                        ("git-fetch" (@ (guix git-download) git-fetch))
+                        ("svn-fetch" (@ (guix svn-download) svn-fetch))
+                        ("hg-fetch"  (@ (guix hg-download) hg-fetch))
+                        (_ #f)))
+              (uri (assoc-ref orig "uri"))
+              (sha256 sha))))))
+
+(define (data->guix-package meta)
+  (package
+    (name (assoc-ref meta "name"))
+    (version (assoc-ref meta "version"))
+    (source (convert-source (assoc-ref meta "source")))
+    (build-system
+      (lookup-build-system-by-name
+       (string->symbol (assoc-ref meta "build-system"))))
+    (native-inputs
+     (specs->package-lists (or (assoc-ref meta "native-inputs") '())))
+    (inputs
+     (specs->package-lists (or (assoc-ref meta "inputs") '())))
+    (propagated-inputs
+     (specs->package-lists (or (assoc-ref meta "propagated-inputs") '())))
+    (home-page
+     (assoc-ref meta "home-page"))
+    (synopsis
+     (assoc-ref meta "synopsis"))
+    (description
+     (assoc-ref meta "description"))
+    (license
+     (let ((l (assoc-ref meta "license")))
+       (or (module-ref (resolve-interface '(guix licenses) #:prefix 'license:)
+                       (spdx-string->license l))
+           (fsdg-compatible l))))))
-- 
2.14.1

Message #14 received at 28251 <at> debbugs.gnu.org (full text, mbox):

From: Ricardo Wurmus <rekado <at> elephly.net>
To: 28251 <at> debbugs.gnu.org
Cc: Ricardo Wurmus <rekado <at> elephly.net>
Subject: [PATCH 3/3] import: Add JSON importer.
Date: Sun, 27 Aug 2017 18:00:46 +0200

* guix/scripts/import/json.scm: New file.
* guix/scripts/import.scm (importers): Add json.
---
 guix/scripts/import.scm      |   2 +-
 guix/scripts/import/json.scm | 101 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 102 insertions(+), 1 deletion(-)
 create mode 100644 guix/scripts/import/json.scm

diff --git a/guix/scripts/import.scm b/guix/scripts/import.scm
index 9bba074e8..67bc7a755 100644
--- a/guix/scripts/import.scm
+++ b/guix/scripts/import.scm
@@ -74,7 +74,7 @@ rather than \\n."
 ;;;
 
 (define importers '("gnu" "nix" "pypi" "cpan" "hackage" "stackage" "elpa" "gem"
-                    "cran" "crate" "texlive"))
+                    "cran" "crate" "texlive" "json"))
 
 (define (resolve-importer name)
   (let ((module (resolve-interface
diff --git a/guix/scripts/import/json.scm b/guix/scripts/import/json.scm
new file mode 100644
index 000000000..b459ef819
--- /dev/null
+++ b/guix/scripts/import/json.scm
@@ -0,0 +1,101 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2014 Eric Bavier <bavier <at> member.fsf.org>
+;;; Copyright © 2015, 2017 Ricardo Wurmus <rekado <at> elephly.net>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix scripts import json)
+  #:use-module (json)
+  #:use-module (guix ui)
+  #:use-module (guix utils)
+  #:use-module (guix scripts)
+  #:use-module (guix import utils)
+  #:use-module (guix scripts import)
+  #:use-module (guix packages)
+  #:use-module (srfi srfi-1)
+  #:use-module (srfi srfi-9 gnu)
+  #:use-module (srfi srfi-11)
+  #:use-module (srfi srfi-37)
+  #:use-module (srfi srfi-41)
+  #:use-module (ice-9 match)
+  #:use-module (ice-9 rdelim)
+  #:use-module (ice-9 format)
+  #:export (guix-import-json))
+
+
+;;;
+;;; Command-line options.
+;;;
+
+(define %default-options
+  '())
+
+(define (show-help)
+  (display (G_ "Usage: guix import json PACKAGE-FILE
+Import and convert the JSON package definition in PACKAGE-FILE.\n"))
+  (display (G_ "
+  -h, --help             display this help and exit"))
+  (display (G_ "
+  -V, --version          display version information and exit"))
+  (newline)
+  (show-bug-report-information))
+
+(define %options
+  ;; Specification of the command-line options.
+  (cons* (option '(#\h "help") #f #f
+                 (lambda args
+                   (show-help)
+                   (exit 0)))
+         (option '(#\V "version") #f #f
+                 (lambda args
+                   (show-version-and-exit "guix import json")))
+         %standard-import-options))
+
+
+;;;
+;;; Entry point.
+;;;
+
+(define (guix-import-json . args)
+  (define (parse-options)
+    ;; Return the alist of option values.
+    (args-fold* args %options
+                (lambda (opt name arg result)
+                  (leave (G_ "~A: unrecognized option~%") name))
+                (lambda (arg result)
+                  (alist-cons 'argument arg result))
+                %default-options))
+
+  (let* ((opts (parse-options))
+         (args (filter-map (match-lambda
+                            (('argument . value)
+                             value)
+                            (_ #f))
+                           (reverse opts))))
+    (match args
+      ((file-name)
+       (catch 'json-invalid
+         (lambda ()
+           (let ((json (json-string->scm
+                        (with-input-from-file file-name read-string))))
+             ;; TODO: also print define-module boilerplate
+             (package->code (data->guix-package (hash-table->alist json)))))
+         (lambda ()
+           (leave (G_ "invalid JSON in file '~a'~%") file-name))))
+      (()
+       (leave (G_ "too few arguments~%")))
+      ((many ...)
+       (leave (G_ "too many arguments~%"))))))
-- 
2.14.1

Message #17 received at 28251 <at> debbugs.gnu.org (full text, mbox):

From: Ricardo Wurmus <rekado <at> elephly.net>
To: 28251 <at> debbugs.gnu.org
Subject: Re: [PATCH 0/3] Add generic JSON importer
Date: Mon, 28 Aug 2017 14:27:22 +0200

I have since made a couple of minor changes like adding a missing (guix
gexp) import, adding the new script to the MODULES in Makefile.am, and
adding a simple test for “data->guix-package”.

I also have a small patch to “guix build -f”; given a file ending on
“.json” it will parse the JSON and run “data->guix-package”.  I have
confirmed that this works for the “hello” package in JSON format.

If this looks like a good idea I’ll convert the recursive CRAN importer
to produce package objects; as a next step I’d extend “guix build” to
accept a “--via=IMPORTER” option, which would cause a package to be
imported and then built without requiring the manual work of writing or
generating a package definition in Scheme.  (Optionally, it could
generate the code that could be contributed to Guix.)

This is not meant to ever replace Scheme package definitions, but I
think it can make importers useful to users, not only developers.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Message #20 received at 28251 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Ricardo Wurmus <rekado <at> elephly.net>
Cc: 28251 <at> debbugs.gnu.org
Subject: Re: [bug#28251] [PATCH 0/3] Add generic JSON importer
Date: Fri, 01 Sep 2017 17:35:46 +0200

Hi Ricardo!

Ricardo Wurmus <rekado <at> elephly.net> skribis:

> this patch set adds a somewhat unusual importer.  Assume we have a file
> "package.json" with the following contents:
>
> {
>   "name": "hello",
>   "version": "2.10",
>   "source": {
>     "method": "url-fetch",
>     "uri": "mirror://gnu/hello/hello-2.10.tar.gz",
>     "sha256": {
>       "base32": "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i"
>     }
>   }
>   "build-system": "gnu",
>   "home-page": "https://www.gnu.org/software/hello/",
>   "synopsis": "Hello, GNU world: An example GNU package",
>   "description": "It really works.",
>   "license": "GPL-3.0+",
>   "inputs": ["r-minimal <at> 3", "ghc-pandoc", "samtools <at> 0"]
> }

Neat!

I wonder if we could further simplify the “source” part, like allowing
(maybe optionally) for:

  "source": "mirror://gnu…"

and then letting the importer download the thing and fill in the hash.
Likewise for a Git checkout:

  "git": { "commit": "cabba9e"; "url": "…" };

Thoughts?

> What do you think?  Terrible?  Exciting?  Both? *raises hand*

Both!  :-)

As discussed at the GHM, I don’t fully measure that, but it can probably
help us reach out to more people.

Thanks!

Ludo’, who goes look at the patches.

Message #23 received at 28251 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Ricardo Wurmus <rekado <at> elephly.net>
Cc: 28251 <at> debbugs.gnu.org
Subject: Re: [bug#28251] [PATCH 1/3] packages: Add package->code.
Date: Fri, 01 Sep 2017 17:55:27 +0200

Ricardo Wurmus <rekado <at> elephly.net> skribis:

> * guix/packages.scm (package->code): New procedure.

We’ll need tests for this.  :-)

I would move it to (guix import utils) or a new (guix import print)
module or similar (which will also allow us to use ‘factorize-uri’).  In
my mind, eventually all importers will produce a <package> object
directly, and so this will be a core part of the import machinery.
Since it’s not a crucial component, I would prefer to have it out of
(guix packages) though.

> +;; FIXME: the quasiquoted arguments field may contain embedded package
> +;; objects, e.g. in #:disallowed-references; they will just be printed with
> +;; their usual #<package ...> representation, not as variable names.

Not sure how to solve that; maybe we can ignore for now.

> +(define (package->code package)
> +  "Return an S-expression representing the source code that produces PACKAGE
> +when evaluated."

Like you wrote, it would be nice to also return a spec of modules in
scope, like:

  ((gnu packages r)
   ((guix licenses) #:prefix license:))

WDYT?

That way, we can eventually change ‘guix import’ to systematically print
both the ‘define-module’ clause and the package definition.

> +  ;; The module in which the package PKG is defined
> +  (define (package-module-name pkg)
> +    (map string->symbol
> +         (string-split (string-drop-right
> +                        (location-file (package-location pkg)) 4)
> +                       #\/)))
> +
> +  ;; Return the first candidate variable name that is bound to VAL.
> +  ;; TODO: avoid '%pkg-config
> +  (define (variable-name val mod)
> +    (let ((candidates (filter identity
> +                              (module-map
> +                               (lambda (sym var)
> +                                 (if (equal? val (variable-ref var)) sym #f))
> +                               (resolve-interface mod)))))
> +      (if (null? candidates) #f (car candidates))))
> +

I think we should compare values with ‘eq?’ (usually we’re concerned
with pointer identity of records), and also use ‘module-for-each’ +
‘let/ec’ to avoid building a list for nothing, like:

--8<---------------cut here---------------start------------->8---
scheme@(guile-user)> (match (let/ec return
			      (module-for-each (lambda (sym var)
						 (if (eq? + (variable-ref var))
						     (return sym)
						     #f))
					       the-scm-module))
		       ((? symbol? sym) sym)
		       (_ #f))
$17 = +
--8<---------------cut here---------------end--------------->8---

> +  ;; Print either license variable name or the code for a license object
> +  (define (print-license lic)

Nitpick: I’d rename all the ‘print-*’ procedures to ‘*->code’.

> +  (match package
> +    (($ <package> name version source build-system

If we move this to (guix import …), we can no longer match on the
record, but that’s not necessarily a bad thing anyway.  :-)

> +        ,@(let ((args (arguments)))
> +            (if (null? args) '()
> +                `((arguments ,(list 'quasiquote (arguments))))))
> +        ,@(if (equal? outputs '("out")) '()
> +              `((outputs (list ,@outputs))))
> +        ,@(let ((pkgs (native-inputs)))
> +            (if (null? pkgs) '()
> +                `((native-inputs ,(print-package-lists pkgs)))))
> +        ,@(let ((pkgs (inputs)))
> +            (if (null? pkgs) '()
> +                `((inputs ,(print-package-lists pkgs)))))
> +        ,@(let ((pkgs (propagated-inputs)))
> +            (if (null? pkgs) '()
> +                `((propagated-inputs ,(print-package-lists pkgs)))))

‘match’!  :-)

This looks pretty cool already!

Thanks,
Ludo’.

Message #26 received at 28251 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Ricardo Wurmus <rekado <at> elephly.net>
Cc: 28251 <at> debbugs.gnu.org
Subject: Re: [bug#28251] [PATCH 2/3] import: Add generic data to package
 converter.
Date: Mon, 04 Sep 2017 15:04:10 +0200

Hello!

Ricardo Wurmus <rekado <at> elephly.net> skribis:

> * guix/import/utils.scm (build-system-modules, guix-modules): New variables.
> (lookup-build-system-by-name, specs->package-lists, convert-source,
> data->guix-package): New procedures.

[...]

> +(define build-system-modules
> +  (all-modules (map (lambda (entry)
> +                      `(,entry . "guix/build-system"))
> +                    %load-path)))
> +
> +(define guix-modules
> +  (all-modules (map (lambda (entry)
> +                      `(,entry . "guix"))
> +                    %load-path)))

‘all-modules’ causes a directory traversal, so it should not be called
at the top level.  The solution is to turn these two things in a promise
or a thunk or probably an ‘mlambda’ thunk (depending on whether they are
expected to be called frequently.)

> +(define (lookup-build-system-by-name name)
> +  (fold-module-public-variables (lambda (obj result)

Docstring please.  :-)

> +(define (specs->package-lists specs)
> +  (map (lambda (spec)
> +         (let ((pkg (specification->package spec)))
> +           (list (package-name pkg) pkg)))
> +       specs))

This should probably use ‘specification->package+output’ so that one can
use specs like “hwloc:lib”.

> +(define (convert-source source)

Maybe ‘source-spec->object’?

> +  (match source
> +    ((? string? file) (local-file file))
> +    (#f #f)
> +    (orig (let ((sha (match (car (assoc-ref orig "sha256"))
> +                       (("base32" . value)
> +                        (base32 value))
> +                       (_ #f))))
> +            (origin
> +              (method (match (assoc-ref orig "method")
> +                        ("url-fetch" (@ (guix download) url-fetch))
> +                        ("git-fetch" (@ (guix git-download) git-fetch))
> +                        ("svn-fetch" (@ (guix svn-download) svn-fetch))
> +                        ("hg-fetch"  (@ (guix hg-download) hg-fetch))
> +                        (_ #f)))
> +              (uri (assoc-ref orig "uri"))
> +              (sha256 sha))))))

Though as discussed earlier, I’m unsure about exposing “url-fetch” and
co. in the spec that people write.

> +(define (data->guix-package meta)

Maybe ‘alist->package’?

Ludo’.

Message #29 received at 28251 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Ricardo Wurmus <rekado <at> elephly.net>
Cc: 28251 <at> debbugs.gnu.org
Subject: Re: [bug#28251] [PATCH 3/3] import: Add JSON importer.
Date: Mon, 04 Sep 2017 15:04:37 +0200

Ricardo Wurmus <rekado <at> elephly.net> skribis:

> * guix/scripts/import/json.scm: New file.
> * guix/scripts/import.scm (importers): Add json.

With a bit of doc, this looks all good!

Ludo'.

Message #34 received at 28251-done <at> debbugs.gnu.org (full text, mbox):

From: Ricardo Wurmus <rekado <at> elephly.net>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 28251-done <at> debbugs.gnu.org
Subject: Re: [bug#28251] [PATCH 1/3] packages: Add package->code.
Date: Thu, 28 Sep 2017 13:19:21 +0200

Ludovic Courtès <ludo <at> gnu.org> writes:

> Ricardo Wurmus <rekado <at> elephly.net> skribis:
>
>> * guix/packages.scm (package->code): New procedure.
>
> We’ll need tests for this.  :-)

I’ve added some simple tests to tests/print.scm and import-utils.scm.

> I would move it to (guix import utils) or a new (guix import print)
> module or similar (which will also allow us to use ‘factorize-uri’).  In
> my mind, eventually all importers will produce a <package> object
> directly, and so this will be a core part of the import machinery.
> Since it’s not a crucial component, I would prefer to have it out of
> (guix packages) though.

Okay, done.

>> +;; FIXME: the quasiquoted arguments field may contain embedded package
>> +;; objects, e.g. in #:disallowed-references; they will just be printed with
>> +;; their usual #<package ...> representation, not as variable names.
>
> Not sure how to solve that; maybe we can ignore for now.

That’s why I originally experimented with overriding the printer.  For
the purposes of a JSON importer, however, this really isn’t important.

>> +(define (package->code package)
>> +  "Return an S-expression representing the source code that produces PACKAGE
>> +when evaluated."
>
> Like you wrote, it would be nice to also return a spec of modules in
> scope, like:
>
>   ((gnu packages r)
>    ((guix licenses) #:prefix license:))
>
> WDYT?

I’ll leave this as a later improvement, but yes: I’ll add this at some
point.

> I think we should compare values with ‘eq?’ (usually we’re concerned
> with pointer identity of records), and also use ‘module-for-each’ +
> ‘let/ec’ to avoid building a list for nothing

That’s good!

> Nitpick: I’d rename all the ‘print-*’ procedures to ‘*->code’.

Done.

I’ve implemented the other suggested changes and added some
documentation.  I’ll push it to master in a few minutes.

Thanks for the comments!

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Message #37 received at 28251 <at> debbugs.gnu.org (full text, mbox):

From: Ricardo Wurmus <rekado <at> elephly.net>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 28251 <at> debbugs.gnu.org
Subject: Re: [bug#28251] [PATCH 0/3] Add generic JSON importer
Date: Thu, 28 Sep 2017 13:23:40 +0200

Ludovic Courtès <ludo <at> gnu.org> writes:

> I wonder if we could further simplify the “source” part, like allowing
> (maybe optionally) for:
>
>   "source": "mirror://gnu…"
>
> and then letting the importer download the thing and fill in the hash.
> Likewise for a Git checkout:
>
>   "git": { "commit": "cabba9e"; "url": "…" };
>
> Thoughts?

I have added support for a simple source string as in the first example.
I left the support for a direct translation of origin expressions
unchanged, because I’m not sure yet how to express this.

So, right now we can do either

  "source": "mirror://gnu/hello/hello-2.10.tar.gz",

or

  "source": {
    "method": "url-fetch",
    "uri": "mirror://gnu/hello/hello-2.10.tar.gz",
    "sha256": {
      "base32": "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i"
    }
  }

I’m open to simplifying this further.  What do other people think?

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Message #40 received at 28251 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Ricardo Wurmus <rekado <at> elephly.net>
Cc: 28251 <at> debbugs.gnu.org
Subject: Re: [bug#28251] [PATCH 0/3] Add generic JSON importer
Date: Thu, 28 Sep 2017 21:58:34 +0200

Ricardo Wurmus <rekado <at> elephly.net> skribis:

> Ludovic Courtès <ludo <at> gnu.org> writes:
>
>> I wonder if we could further simplify the “source” part, like allowing
>> (maybe optionally) for:
>>
>>   "source": "mirror://gnu…"
>>
>> and then letting the importer download the thing and fill in the hash.
>> Likewise for a Git checkout:
>>
>>   "git": { "commit": "cabba9e"; "url": "…" };
>>
>> Thoughts?
>
> I have added support for a simple source string as in the first example.
> I left the support for a direct translation of origin expressions
> unchanged, because I’m not sure yet how to express this.
>
> So, right now we can do either
>
>   "source": "mirror://gnu/hello/hello-2.10.tar.gz",
>
> or
>
>   "source": {
>     "method": "url-fetch",
>     "uri": "mirror://gnu/hello/hello-2.10.tar.gz",
>     "sha256": {
>       "base32": "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i"
>     }
>   }
>
> I’m open to simplifying this further.  What do other people think?

That sounds good to me.

One important consideration is to have a format that is future-proof,
which is probably the case here.  We could perhaps add a
“format-version” field in the JSON input to be on the safe side, though
it would defeat the whole simplification.

(In other news, I’ve been considering the implementation of an
“upstream” importer, where “guix import upstream URL” would return a
package template as complete as possible.)

Thanks,
Ludo’.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.