GNU bug report logs - #35627
[PATCH] Add golang-importer

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 35627 in the body.
You can then email your comments to 35627 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: nly <nly <at> disroot.org>
To: guix-patches <at> gnu.org
Subject: [PATCH] Add golang-importer
Date: Wed, 08 May 2019 11:40:55 +0530

[Message part 1 (text/plain, inline)]

Hi. I've attached some patches for a golang importer

To use:
1. (load "golang.scm")
2. (define gopkg (make-go-package go-name*))
3. (package-sexp gopkg)
;; go-name looks like "golang.org/auth/pkg"

;; TODO
1. reliant on github urls
2. extract license
3. Recursive?

Suggestions welcome

Thanks,
Amar

[0001-Import-Add-make-go-package.patch (text/x-patch, attachment)]

[0002-golang-Import-formatting.patch (text/x-patch, attachment)]

[0003-Golang-Import-Add-go-name-inputs.patch (text/x-patch, attachment)]

[0004-Golang-Import-Add-package-sexp-go-name-inputs.-forma.patch (text/x-patch, attachment)]

[0005-Removed-alist-sexp-Added-shell-command-go-name-guix-.patch (text/x-patch, attachment)]

[0006-Formatting.patch (text/x-patch, attachment)]

[0007-Formatting.patch (text/x-patch, attachment)]

[0008-Mark-DONE-go-name-guix-name.patch (text/x-patch, attachment)]

[0009-Move-todo-to-top-Add-usage-instructions.patch (text/x-patch, attachment)]

[0010-Bootstrap-don-t-use-gnu-packages-golang.patch (text/x-patch, attachment)]

Message #8 received at 35627 <at> debbugs.gnu.org (full text, mbox):

From: Amar Singh <nly <at> disroot.org>
To: 35627 <at> debbugs.gnu.org
Subject: [PATCH] Importer: Add golang
Date: Sun, 26 May 2019 22:14:39 +0530

This Go importer is flaky. It's a rebased patch.

From f54e0aa0ae3f553701b1253a6e8605a493e4d4ac Mon Sep 17 00:00:00 2001
From: Amar Singh <nly <at> disroot.org>
Date: Tue, 30 Apr 2019 23:17:51 +0530
Subject: [PATCH] Importer: Add golang

* guix/import/golang.scm.

Signed-off-by: Amar Singh <nly <at> disroot.org>
---
 guix/import/golang.scm | 217 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 217 insertions(+)
 create mode 100644 guix/import/golang.scm

diff --git a/guix/import/golang.scm b/guix/import/golang.scm
new file mode 100644
index 0000000000..b51d496602
--- /dev/null
+++ b/guix/import/golang.scm
@@ -0,0 +1,217 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2019 by Amar Singh <nly <at> disroot.org>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; This program is free software: you can redistribute it and/or modify
+;;; it under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation, either version 3 of the License, or
+;;; (at your option) any later version.
+;;;
+;;; This program is distributed in the hope that it will be useful,
+;;; but WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+(define-module (guix import golang))
+(use-modules
+ (srfi srfi-1) ;; fold
+ (ice-9 rdelim) ;; read-string
+ (guix import github)  ;; latest-release
+ (guix utils) ;; string-replace-substring
+ (guix memoization)        ;; memoize network operations
+ (guix download)       ;; download-to-store
+ ((guix import utils) #:prefix utils:)   ;; hash
+ (guix packages)       ;; packages
+ ((guix licenses) #:prefix license:) ;; licenses
+ (guix build-system)   ;; build-system printer
+ (guix build-system go)  ;; go-build-system
+ (guix store)          ;; with-store
+ ;; (gnu packages golang) ;; inherit (simple) go package
+ (ice-9 textual-ports) ;; to parse readme.md
+ (ice-9 popen) ;; open-input-ouput-pipe
+ (web uri) ;; uri->string
+ (srfi srfi-26) ;; cut
+ )
+
+;;; To use, simply:
+;;; 1. (load "golang.scm")
+;;; 2. (define go-package (make-package go-name*))
+;;; 3. (package-sexp go-package)
+
+;;; STATUS
+;;; 1. latest-release  DONE
+;;; 1.b latest-commit PENDING/STALLED
+;;; 2. go-name->guix-name DONE
+;;; 2.b style go-github-com-user-project DONE
+;;; 4. go-name->url DONE
+;;; 4.b go-name->tarball DONE
+;;; 5. go-name->sha256 (go-name version) DONE
+;;; 6. go-name->synopsis DONE
+;;; 7. go-name->description DONE
+;;; 4-7.b. Memoize, network procedures DONE
+;;; 6-7.b try to extract sentences. TODO
+;;; 8. go-name->license TODO
+;;; 9. go-name->inputs DONE
+;;; 9.b. inputs alist DONE
+;;; 10. package-sexp DONE
+;;; 10.a origin-sexp DONE
+;;; 11. Package Builds TODO
+
+(define-public go-name* "github.com/gohugoio/hugo") ;; for tests
+
+(define* (go-name->url go-name #:rest args)
+  (if (string-contains go-name ".")
+      (uri->string (string->uri (apply string-append
+                                       "https://" go-name args)))
+      #f))
+
+(define (go-name->tarball go-name version)
+  (go-name->url go-name "/archive/v"
+                version ".tar.gz"))
+
+(define* (string-replace-substrings string substrings
+                                    #:optional (replacement "-"))
+  (if (null-list? substrings)
+      string
+      ((cut string-replace-substring <> (car substrings) replacement)
+       (string-replace-substrings string (cdr substrings)))))
+
+;;; Possible remove @@ if upstream exports the symbols
+(define (go-name->guix-name go-name)
+  (string-append "go-"
+                 (string-replace-substrings go-name '("." "/") "-")))
+
+;;; Slow; accesses the network; memoized
+(define latest-release
+  (memoize
+   (lambda (go-name)
+     ((@@ (guix import github) latest-released-version)
+      (go-name->url go-name)
+      (go-name->guix-name go-name)))))
+
+;;; Slow; downloads the url from network; memoized
+(define url->store
+  (@@ (guix import cran) download))
+
+;;; Slow; download src tarball from network, returns base32 nix-hash;
+;;; memoized
+(define (go-name->sha256 go-name version)
+  (utils:guix-hash-url (url->store (go-name->tarball go-name version))))
+
+;;; Slow; network access; memoized
+(define go-name->readme-string
+  (memoize
+   (lambda (go-name)
+     (define (go-name->readme go-name)
+       (go-name->url "raw.githubusercontent.com"
+                     ;; TODO, detect the domain
+                     (substring go-name
+                                (string-length "github.com"))
+                     "/master/"
+                     "README.md"))
+     (call-with-input-file (url->store (go-name->readme go-name))
+       read-string))))
+
+;;; TODO: try to match the first sentence.
+(define (go-name->synopsis go-name)
+  (substring (go-name->readme-string go-name) 0 100))
+
+;;; TODO: try to match the the next two sentences.
+(define (go-name->description go-name)
+  (substring (go-name->readme-string go-name) 100 300))
+
+(define shell-command
+  (lambda* (command #:rest args)
+    (let* ((cmd (string-join (cons command (delete #f (delete '() args))) " "))
+           (port (open-input-output-pipe cmd))
+           (result (read-string port))
+           (exit-code (close-pipe port)))
+      (and (zero? exit-code)
+           (string-split (string-trim-right result) #\newline)))))
+
+(define go-name->inputs
+  (lambda (go-name)
+    (let ((recursive-depends "-f '{{ join .Deps \"\\n\" }}'")
+          (direct-depends "-f '{{ join .Imports \"\\n\" }}'")
+          (go-command (car (shell-command "which go"))))
+      (shell-command go-command "list" direct-depends go-name))))
+
+;;; License
+(define (string->license license-string)
+  ((@@ (guix import cran) string->license) (string-upcase license-string)))
+
+;;; For inputs
+(define format-inputs
+  (@@ (guix import cran) format-inputs))
+
+(define-public (make-go-package go-name)
+  ;; Do the expensive operations only once; query network for latest
+  ;; version
+  (let* ((version (latest-release go-name))
+         (sha256 (go-name->sha256 go-name version))
+         (readme-string (go-name->readme-string go-name)))
+    (package
+      ;; (inherit go-github-com-alsm-ioprogress)
+      (name
+       (string-append "go-" go-name))
+      (version version)
+      (source
+       (origin (method url-fetch)
+               (uri (go-name->tarball go-name version))
+               (sha256 (base32 sha256))))
+      (home-page
+       (go-name->url go-name))
+      (build-system
+        go-build-system)
+      (arguments
+       `(#:import-path ,go-name))
+      ;; TODO: make inputs into (unquote ..) form
+      (inputs
+       (format-inputs (map go-name->guix-name (go-name->inputs go-name))))
+      (synopsis (go-name->synopsis go-name))
+      (description (go-name->description go-name))
+      ;; TODO: license
+      (license license:expat)
+      )))
+
+(define (filter-newlines string)
+  (string-filter (lambda (x) (not (equal? x #\newline))) string))
+
+(define bv->nix-base32 (@@ (guix packages)
+                           bytevector->nix-base32-string))
+
+(define (origin-sexp origin)
+  `(origin
+     (method url-fetch)
+     (uri ,(origin-uri origin))
+     (sha256 (base32 ,(bv->nix-base32 (origin-sha256 origin))))
+     (file-name ,(origin-file-name origin))
+     (patches ,(origin-patches origin))
+     (snippet ,(origin-snippet origin))
+     (patch-flags ,(origin-patch-flags origin))
+     (patch-inputs ,(origin-patch-inputs origin))
+     (modules ,(origin-modules origin))
+     (patch-guile ,(origin-patch-guile origin))))
+
+(define (build-system-sexp build-system)
+  (symbol-append (build-system-name build-system) '-build-system))
+
+(define-public (package-sexp package)
+  `(package
+     (name ,(package-name package))
+     (version ,(package-version package))
+     (source ,(origin-sexp (package-source package)))
+     (home-page ,(package-home-page package))
+     (build-system ,(build-system-sexp (package-build-system package)))
+     (arguments ,(package-arguments package))
+     (synopsis ,(filter-newlines (package-synopsis package)))
+     (description ,(filter-newlines (package-description package)))
+     (inputs ,(format-inputs (map car (package-inputs package))))
+     (native-inputs ,(format-inputs (map car (package-native-inputs package))))
+     (propagated-inputs ,(format-inputs (map car (package-propagated-inputs package))))))
+
+;;; golang.scm ends here
-- 
2.21.0

Message #13 received at 35627-done <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: nly <nly <at> disroot.org>
Cc: 35627-done <at> debbugs.gnu.org
Subject: Re: bug#35627: [PATCH] Add golang-importer
Date: Thu, 05 Aug 2021 23:40:20 -0400

Hello Amar,

nly <nly <at> disroot.org> writes:

> Hi. I've attached some patches for a golang importer
>
> To use:
> 1. (load "golang.scm")
> 2. (define gopkg (make-go-package go-name*))
> 3. (package-sexp gopkg)
> ;; go-name looks like "golang.org/auth/pkg"
>
> ;; TODO
> 1. reliant on github urls
> 2. extract license
> 3. Recursive?
>
> Suggestions welcome
>
> Thanks,
> Amar

Thank you for your efforts.  Perhaps others have built on it, as we now
have a relatively featureful Go importer in Guix, as you probably saw!

Closing,

Maxim

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.