GNU bug report logs - #45972
[PATCH] Add julia-json with dependencies

Previous Next

Package: guix-patches;

Reported by: Nicolò Balzarotti <anothersms <at> gmail.com>

Date: Mon, 18 Jan 2021 23:49:02 UTC

Severity: normal

Tags: patch

Done: Ludovic Courtès <ludo <at> gnu.org>

Bug is archived. No further changes may be made.

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 45972 in the body.
You can then email your comments to 45972 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to guix-patches <at> gnu.org:
bug#45972; Package guix-patches. (Mon, 18 Jan 2021 23:49:02 GMT) Full text and rfc822 format available.

Acknowledgement sent to Nicolò Balzarotti <anothersms <at> gmail.com>:
New bug report received and forwarded. Copy sent to guix-patches <at> gnu.org. (Mon, 18 Jan 2021 23:49:02 GMT) Full text and rfc822 format available.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Nicolò Balzarotti <anothersms <at> gmail.com>
To: guix-patches <at> gnu.org
Subject: [PATCH] Add julia-json with dependencies
Date: Tue, 19 Jan 2021 00:47:55 +0100
[Message part 1 (text/plain, inline)]
Hi guix!

This patch series adds julia-json together with its dependencies.

It also enable tests for julia packages.

Thanks, Nicolò

[0001-guix-build-system-julia-Enable-tests.patch (text/x-patch, attachment)]
[0002-gnu-julia-compat-Update-to-3.25.0.patch (text/x-patch, attachment)]
[0003-gnu-Add-julia-orderedcollections.patch (text/x-patch, attachment)]
[0004-gnu-Add-julia-datastructures.patch (text/x-patch, attachment)]
[0005-gnu-Add-julia-fixedpointnumbers.patch (text/x-patch, attachment)]
[0006-gnu-Add-julia-parsers.patch (text/x-patch, attachment)]
[0007-gnu-Add-julia-adapt.patch (text/x-patch, attachment)]
[0008-gnu-Add-julia-offsetarrays.patch (text/x-patch, attachment)]
[0009-gnu-Add-julia-json.patch (text/x-patch, attachment)]

Information forwarded to guix-patches <at> gnu.org:
bug#45972; Package guix-patches. (Tue, 26 Jan 2021 21:34:02 GMT) Full text and rfc822 format available.

Message #8 received at 45972 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Nicolò Balzarotti <anothersms <at> gmail.com>
Cc: 45972 <at> debbugs.gnu.org
Subject: Re: bug#45972: [PATCH] Add julia-json with dependencies
Date: Tue, 26 Jan 2021 22:33:37 +0100
Hi Nicolò,

Nicolò Balzarotti <anothersms <at> gmail.com> skribis:

> This patch series adds julia-json together with its dependencies.

Neat.  I looked at it quickly and it LGTM.  Nitpicks:

> +    (synopsis "Julia implementation of associative containers that preserve
> +insertion order")

Maybe remove “Julia implementation of”.

> +    (description "This package implements OrderedDicts and OrderedSets, which

Ideally use @code for identifiers.

> +    (synopsis "Fixed point types for julia")

s/julia/Julia/

> +    (synopsis "JSON parsing and printing")
> +    (description "Parsing and printing JSON in pure Julia.")

Please make that a full sentence.

If you’re confident with the licenses and ‘guix lint’ is happy,
please push!  Nice to see more Julia packages.

Ludo’.




Information forwarded to guix-patches <at> gnu.org:
bug#45972; Package guix-patches. (Tue, 26 Jan 2021 23:25:01 GMT) Full text and rfc822 format available.

Message #11 received at 45972 <at> debbugs.gnu.org (full text, mbox):

From: Nicolò Balzarotti <anothersms <at> gmail.com>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 45972 <at> debbugs.gnu.org
Subject: Re: [bug#45972] [PATCH] Add julia-json with dependencies
Date: Wed, 27 Jan 2021 00:23:44 +0100
[Message part 1 (text/plain, inline)]
I applied all your suggestions :)

> If you’re confident with the licenses and ‘guix lint’ is happy,
> please push!

Almost all of Julia packages are under MIT (expat), I double checked and
they are fine.  Also, 3 packages (out of 8) have been updated since my
submission, so I updated them now and guix lint does not complain anymore.

> Nice to see more Julia packages.

This is the first batch, I finally decided to submit in small batches
the 100+ packages I have; you have been warned :D

Here's the updated patchset (if somebody can push).

Thanks Ludo for the review

[v2-0001-guix-build-system-julia-Enable-tests.patch (text/x-patch, attachment)]
[v2-0002-gnu-julia-compat-Update-to-3.25.0.patch (text/x-patch, attachment)]
[v2-0003-gnu-Add-julia-orderedcollections.patch (text/x-patch, attachment)]
[v2-0004-gnu-Add-julia-datastructures.patch (text/x-patch, attachment)]
[v2-0005-gnu-Add-julia-fixedpointnumbers.patch (text/x-patch, attachment)]
[v2-0006-gnu-Add-julia-parsers.patch (text/x-patch, attachment)]
[v2-0007-gnu-Add-julia-adapt.patch (text/x-patch, attachment)]
[v2-0008-gnu-Add-julia-offsetarrays.patch (text/x-patch, attachment)]
[v2-0009-gnu-Add-julia-json.patch (text/x-patch, attachment)]

Information forwarded to guix-patches <at> gnu.org:
bug#45972; Package guix-patches. (Wed, 27 Jan 2021 22:41:02 GMT) Full text and rfc822 format available.

Message #14 received at 45972 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Nicolò Balzarotti <anothersms <at> gmail.com>
Cc: 45972 <at> debbugs.gnu.org
Subject: Re: [bug#45972] [PATCH] Add julia-json with dependencies
Date: Wed, 27 Jan 2021 23:40:28 +0100
Hi,

Nicolò Balzarotti <anothersms <at> gmail.com> skribis:

> Almost all of Julia packages are under MIT (expat), I double checked and
> they are fine.  Also, 3 packages (out of 8) have been updated since my
> submission, so I updated them now and guix lint does not complain anymore.

Great.

> This is the first batch, I finally decided to submit in small batches
> the 100+ packages I have; you have been warned :D

Oh, fun! :-)

So, actually, we have a problem:

> +(define-public julia-orderedcollections
> +  (package
> +    (name "julia-orderedcollections")
> +    (version "1.3.2")
> +    (source
> +     (origin
> +       (method git-fetch)
> +       (uri (git-reference
> +             (url "https://github.com/JuliaCollections/OrderedCollections.jl")
> +             (commit (string-append "v" version))))
> +       (file-name "OrderedCollections")

I was surprised that ‘guix lint’ doesn’t complain about this file name,
I thought it required the file name to match the package name and at
least that’s the spirit (I’ll take a look).

Anyway, I went ahead and replaced all these by (git-file-name name
version) as is done elsewhere.  But now the tests would fail like so:

--8<---------------cut here---------------start------------->8---
starting phase `check'
ERROR: LoadError: ArgumentError: Package Adapt not found in current path:
- Run `import Pkg; Pkg.add("Adapt")` to install the Adapt package.
--8<---------------cut here---------------end--------------->8---

My understanding is that the first patch expects the source file name to
match the Julia package name.  IMO, that shouldn’t be the case.  Can we
either extract the Julia package name from metadata that’s in the
package itself (?), or otherwise pass it to all the phases via
‘julia-build-system’?

Sorry for not noticing earlier!

Thanks,
Ludo’.




Information forwarded to guix-patches <at> gnu.org:
bug#45972; Package guix-patches. (Thu, 28 Jan 2021 00:31:02 GMT) Full text and rfc822 format available.

Message #17 received at 45972 <at> debbugs.gnu.org (full text, mbox):

From: Nicolò Balzarotti <anothersms <at> gmail.com>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 45972 <at> debbugs.gnu.org
Subject: Re: [bug#45972] [PATCH] Add julia-json with dependencies
Date: Thu, 28 Jan 2021 01:30:24 +0100
> My understanding is that the first patch expects the source file name to
> match the Julia package name.

That's correct, we use it in the build system.

> IMO, that shouldn’t be the case.

At first I wasn't sure it was ok, but it went on through the revision
process when I submitted the first package (Compat) so I tought it was fine.

> Can we either extract the Julia package name from metadata that’s in
> the package itself (?), or otherwise pass it to all the phases via
> ‘julia-build-system’?

Sure, I'd just read it from Package.toml (nowadays almost all the
packages have this file, and for when it's missing we also have the
julia-create-package-toml procedure that creates it).

The file is toml, but I don't see a toml parser in guix.  So, I'd use a
function like the following:

#+begin_src scheme
(define (package.toml->name file)
  (call-with-input-file file
    (lambda (in)
      (let loop ((line (read-line in 'concat)))
        (if (eof-object? line)
            #f                          ;What to do?
            (let ((m (string-match "name\\s*=\\s*\"(.*)\"" line)))
              (if m (match:substring m 1)
                  (loop (read-line in 'concat)))))))))
#+end_src

As you can see this is very minimal/naive (takes the first name = ""
occurrence, does not even consider comments, for which I'd add a
negative lookahead "^(?!#)" which I can't get to work with ice-9 regex),
but tested on a few packages it working.  I don't know what to do when
the match is not found (since it's something that might happen only
during development, the #f fallback should not be too bad, btw, as the
build will fail).

The other way I think this is easily solvable is by asking julia
directly, by reading the output of:

(invoke-julia "using Pkg; Pkg.TOML.parsefile("Project.toml")["name"] |> println")

doing something like cargo's manifest-target procedure does.  But it'd
go the other way if it's ok.

Let me know!  Once decided, I'll submit the updated patches

>
> Sorry for not noticing earlier!
>
np and thanks again!




Information forwarded to guix-patches <at> gnu.org:
bug#45972; Package guix-patches. (Thu, 28 Jan 2021 13:11:02 GMT) Full text and rfc822 format available.

Message #20 received at 45972 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Nicolò Balzarotti <anothersms <at> gmail.com>
Cc: 45972 <at> debbugs.gnu.org
Subject: Re: [bug#45972] [PATCH] Add julia-json with dependencies
Date: Thu, 28 Jan 2021 14:10:15 +0100
Buon giorno!

Nicolò Balzarotti <anothersms <at> gmail.com> skribis:

>> My understanding is that the first patch expects the source file name to
>> match the Julia package name.
>
> That's correct, we use it in the build system.
>
>> IMO, that shouldn’t be the case.
>
> At first I wasn't sure it was ok, but it went on through the revision
> process when I submitted the first package (Compat) so I tought it was fine.

If it was me it was probably an oversight, I’m sorry about that.

>> Can we either extract the Julia package name from metadata that’s in
>> the package itself (?), or otherwise pass it to all the phases via
>> ‘julia-build-system’?
>
> Sure, I'd just read it from Package.toml (nowadays almost all the
> packages have this file, and for when it's missing we also have the
> julia-create-package-toml procedure that creates it).
>
> The file is toml, but I don't see a toml parser in guix.  So, I'd use a
> function like the following:
>
> #+begin_src scheme
> (define (package.toml->name file)
>   (call-with-input-file file
>     (lambda (in)
>       (let loop ((line (read-line in 'concat)))
>         (if (eof-object? line)
>             #f                          ;What to do?
>             (let ((m (string-match "name\\s*=\\s*\"(.*)\"" line)))
>               (if m (match:substring m 1)
>                   (loop (read-line in 'concat)))))))))
> #+end_src

Sounds reasonable to me.  If I understand the toml format correctly,
“name=.*” is guaranteed to be on a line on its own, so that looks safe.

> As you can see this is very minimal/naive (takes the first name = ""
> occurrence, does not even consider comments, for which I'd add a
> negative lookahead "^(?!#)" which I can't get to work with ice-9 regex),
> but tested on a few packages it working.  I don't know what to do when
> the match is not found (since it's something that might happen only
> during development, the #f fallback should not be too bad, btw, as the
> build will fail).

Yes, sounds good.

To be on the safe side, we can add a #:julia-package-name parameter to
the build system; it would default to #f, in which case the name is
extracted from the toml file.

> The other way I think this is easily solvable is by asking julia
> directly, by reading the output of:
>
> (invoke-julia "using Pkg; Pkg.TOML.parsefile("Project.toml")["name"] |> println")
>
> doing something like cargo's manifest-target procedure does.  But it'd
> go the other way if it's ok.

Yeah, the basic “parser” seems to be good enough.

Thanks!

Ludo’.




Information forwarded to guix-patches <at> gnu.org:
bug#45972; Package guix-patches. (Thu, 28 Jan 2021 15:38:01 GMT) Full text and rfc822 format available.

Message #23 received at 45972 <at> debbugs.gnu.org (full text, mbox):

From: Nicolò Balzarotti <anothersms <at> gmail.com>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 45972 <at> debbugs.gnu.org
Subject: Re: [bug#45972] [PATCH] Add julia-json with dependencies
Date: Thu, 28 Jan 2021 16:37:32 +0100
[Message part 1 (text/plain, inline)]
Ludovic Courtès <ludo <at> gnu.org> writes:

> Buon giorno!
>
>>> IMO, that shouldn’t be the case.
> If it was me it was probably an oversight, I’m sorry about that.

No problem! It appeared also in the manual so I fixed it

>
>>> Can we either extract the Julia package name from metadata that’s in
>>> the package itself (?), or otherwise pass it to all the phases via
>>> ‘julia-build-system’?
>
> Sounds reasonable to me.  If I understand the toml format correctly,
> “name=.*” is guaranteed to be on a line on its own, so that looks
> safe.

exactly

>
> To be on the safe side, we can add a #:julia-package-name parameter to
> the build system; it would default to #f, in which case the name is
> extracted from the toml file.
>
Done, I attached the updated patch set!

[v4-0001-guix-build-system-julia-Enable-tests.patch (text/x-patch, attachment)]
[v4-0002-guix-julia-build-system-Don-t-rely-on-file-name-t.patch (text/x-patch, attachment)]
[v4-0003-gnu-julia-xyz-julia-compat-Set-file-name-accordin.patch (text/x-patch, attachment)]
[v4-0004-gnu-julia-compat-Update-to-3.25.0.patch (text/x-patch, attachment)]
[v4-0005-gnu-Add-julia-orderedcollections.patch (text/x-patch, attachment)]
[v4-0006-gnu-Add-julia-datastructures.patch (text/x-patch, attachment)]
[v4-0007-gnu-Add-julia-fixedpointnumbers.patch (text/x-patch, attachment)]
[v4-0008-gnu-Add-julia-parsers.patch (text/x-patch, attachment)]
[v4-0009-gnu-Add-julia-adapt.patch (text/x-patch, attachment)]
[v4-0010-gnu-Add-julia-offsetarrays.patch (text/x-patch, attachment)]
[v4-0011-gnu-Add-julia-json.patch (text/x-patch, attachment)]

Reply sent to Ludovic Courtès <ludo <at> gnu.org>:
You have taken responsibility. (Sat, 30 Jan 2021 14:11:01 GMT) Full text and rfc822 format available.

Notification sent to Nicolò Balzarotti <anothersms <at> gmail.com>:
bug acknowledged by developer. (Sat, 30 Jan 2021 14:11:01 GMT) Full text and rfc822 format available.

Message #28 received at 45972-done <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Nicolò Balzarotti <anothersms <at> gmail.com>
Cc: 45972-done <at> debbugs.gnu.org
Subject: Re: bug#45972: [PATCH] Add julia-json with dependencies
Date: Sat, 30 Jan 2021 15:10:15 +0100
Hi,

Nicolò Balzarotti <anothersms <at> gmail.com> skribis:

> Done, I attached the updated patch set!

Applied it all, thanks a lot!

Ludo’.




Information forwarded to guix-patches <at> gnu.org:
bug#45972; Package guix-patches. (Sat, 30 Jan 2021 14:50:02 GMT) Full text and rfc822 format available.

Message #31 received at 45972 <at> debbugs.gnu.org (full text, mbox):

From: zimoun <zimon.toutoune <at> gmail.com>
To: Nicolò Balzarotti <anothersms <at> gmail.com>, Ludovic
 Courtès <ludo <at> gnu.org>
Cc: 45972 <at> debbugs.gnu.org
Subject: Re: [bug#45972] Julia importer?
Date: Sat, 30 Jan 2021 15:46:02 +0100
Hi,

On Thu, 28 Jan 2021 at 01:30, Nicolò Balzarotti <anothersms <at> gmail.com> wrote:

> Sure, I'd just read it from Package.toml (nowadays almost all the
> packages have this file, and for when it's missing we also have the
> julia-create-package-toml procedure that creates it).
>
> The file is toml, but I don't see a toml parser in guix.  So, I'd use a
> function like the following:
>
> #+begin_src scheme
> (define (package.toml->name file)
>   (call-with-input-file file
>     (lambda (in)
>       (let loop ((line (read-line in 'concat)))
>         (if (eof-object? line)
>             #f                          ;What to do?
>             (let ((m (string-match "name\\s*=\\s*\"(.*)\"" line)))
>               (if m (match:substring m 1)
>                   (loop (read-line in 'concat)))))))))
> #+end_src

[...]

> The other way I think this is easily solvable is by asking julia
> directly, by reading the output of:
>
> (invoke-julia "using Pkg; Pkg.TOML.parsefile("Project.toml")["name"] |> println")

With a bit more glue, could this be transformed into something like
“julia->guix-package”?  And so have a Julia package importer, even if it
fails for some cases.

WDYT?


All the best,
simon




Information forwarded to guix-patches <at> gnu.org:
bug#45972; Package guix-patches. (Sat, 30 Jan 2021 20:14:02 GMT) Full text and rfc822 format available.

Message #34 received at 45972 <at> debbugs.gnu.org (full text, mbox):

From: Nicolò Balzarotti <anothersms <at> gmail.com>
To: zimoun <zimon.toutoune <at> gmail.com>, Ludovic Courtès
 <ludo <at> gnu.org>
Cc: 45972 <at> debbugs.gnu.org
Subject: Re: [bug#45972] Julia importer?
Date: Sat, 30 Jan 2021 21:13:14 +0100
[Message part 1 (text/plain, inline)]
zimoun <zimon.toutoune <at> gmail.com> writes:

> Hi,

Hi Simon!

>>
>> (invoke-julia "using Pkg; Pkg.TOML.parsefile("Project.toml")["name"] |> println")
>
> With a bit more glue, could this be transformed into something like
> “julia->guix-package”?  And so have a Julia package importer, even if it
> fails for some cases.

Well, if you mean "Can we use Pkg.jl to generate package definitions for
us?" the answer is "probably yes, but I never investigated this".  That
line uses just Julia Base TOML.jl, which for some reason is defined
inside module Pkg.jl (and it's not top-level).

If you instead meant "Can we have a Julia importer?" some time ago I
wrote one in Julia, I've not used it in the last year, but it did work
quite well back then.  I attach it here for reference.

But before digging into Pkg3, I need to push a new set of packages which
contains julia-jllwrappers, which is needed for Julia packages which
require binary distributions to work.  After that, it will be possible
to create an importer that work "ok" even with packages depending on
.so libraries.

[import.jl (text/plain, inline)]
using Pkg
using Pkg.TOML
using LibGit2

const base = ["Base64",
              "CRC32c",
              "Dates",
              "DelimitedFiles",
              "Distributed",
              "FileWatching",
              "Future",
              "InteractiveUtils",
              "Libdl",
              "LibGit2",
              "LinearAlgebra",
              "Logging",
              "Markdown",
              "Mmap",
              "Pkg",
              "Printf",
              "Profile",
              "Random",
              "REPL",
              "Serialization",
              "SHA",
              "SharedArrays",
              "Sockets",
              "SparseArrays",
              "Statistics",
              "SuiteSparse",
              "Test",
              "Unicode",
              "UUIDs"]
const disabled = ["WinRPM", "Homebrew", "CMake",
                  "Docile", "Color", 
                  "HTTPClient", "ICU", "Calendar", "LegacyStrings",
                  "Nulls"]

registrypath = expanduser("~/.julia-old/registries/General/")
registry = joinpath(registrypath, "Registry.toml")
const register = TOML.parse(join(readlines(registry), "\n"))

jlpkgname(info) = "julia-$(lowercase(info.name))"

function getrev(path)
    versions = TOML.parse(join(readlines(path), "\n"))
    versionv = findmax(map(x -> VersionNumber(x), keys(versions) |> collect))
    (rev = versions[string(versionv[1])]["git-tree-sha1"],
     ver = versionv[1])
end

function getpackagebyuuid(uuid)
    uuid in keys(register["packages"]) || return nothing
    path = register["packages"][uuid]["path"]
    getpath(x) =  joinpath(registrypath, joinpath(path, x))
    package = TOML.parse(join(readlines(getpath("Package.toml")), "\n"))
    deppath = getpath("Deps.toml")
    isfile(deppath) || return nothing
    deps = TOML.parse(join(readlines(deppath), "\n"))
    (name = package["name"],
     uuid = package["uuid"],
     repo = package["repo"],
     deps = deps,
     vers = getrev(getpath("Versions.toml")))
end

function getpackage(wanted)
    uuid = findfirst(p -> lowercase(p["name"]) == lowercase(wanted),
                     register["packages"])
    uuid == nothing && return nothing
    return getpackagebyuuid(uuid)
end

function getdeps!(deps, vers, recursive, out)
    flat(arr::Array) = mapreduce(x -> isa(x, Array) ? flat(x) : x, append!,
                                 arr, init=[])
    v = map(p -> VersionNumber.(split(p, '-')), keys(deps) |> collect)
    valid = findall(x -> length(x) == 1 ||
                    (x[2] == v"0" && x[1] <= vers) ||
                    x[1] <= vers <= x[2], v)
    f = flat(map(x -> values(x), values(collect(values(deps))[valid])))
    push!.(Ref(out), f)
    # if recursive
    #     push!
    # end
    nothing
end

function have(info)
    file = "/home/nixo/git/guix/gnu/packages/julia-xyz.scm"
    return "(name \"" * jlpkgname(info) * "\")" in strip.(readlines(file))
end

function gethash(info)
    wd = mktempdir()
    if info.name in base ||
        info.name in disabled ||
        have(info)
        return ""
    end
    println(stderr, "Cloning $(info.name) in $wd")
    repo = LibGit2.clone(info.repo, wd)
    hash = cd(wd) do
        out = Pipe()
        try
            LibGit2.checkout!(repo,
                              string(LibGit2.GitHash(LibGit2.peel(LibGit2.GitCommit,
                                                                  LibGit2.GitTag(repo, "v" * string(info.vers.ver))))))
        catch e
            try
                LibGit2.checkout!(repo,
                                  string(LibGit2.GitHash(LibGit2.peel(LibGit2.GitCommit,
                                                                      LibGit2.GitCommit(repo, "v" * string(info.vers.ver))))))
            catch e
                # FIXME: if this happens, return the commit too and use it in the package
                println(stderr, "Failed to checkout $(e), continuing")
            end
        end
        run(pipeline(`guix hash -rx .`, stdout=out))
        readline(out)
    end
    rm(wd, recursive = true)
    hash
end

function makepackage(info; done = [])
    if info === nothing
        @warn "Could not find package (have you cloned the registry?)"
        return
    elseif info in done
        return ""
    end
    push!(done, info)
    deps = String[]
    getdeps!(info.deps, info.vers.ver, true, deps)
    # TODO: remove deps that are in base
    deps = filter(x -> x !== nothing, getpackagebyuuid.(deps))
    deplist = join(map(name -> "(\"$name\" ,$name)",
                       jlpkgname.(deps)), '\n')
    packagedeps = join(makepackage.(deps, done = done), "")
    hash = gethash(info)
    hash == "" && return ""
    """
    $packagedeps
    (define-public $(jlpkgname(info))
      (package
       (name "$(jlpkgname(info))")
       (version "$(info.vers.ver)")
       (source
          (origin
            (method git-fetch)
            (uri (git-reference
              (url "$(info.repo)")
              (commit (string-append "v" version))))
            (file-name "$(info.name)")
            (sha256
                 (base32 "$hash"))))
       (propagated-inputs `($(deplist)))
       (build-system julia-build-system)
       (home-page "$(info.repo)")
       (synopsis "")
       (description "")
       (license license:expat)))
    """
end

println.(makepackage.(getpackage.(ARGS)))

Information forwarded to guix-patches <at> gnu.org:
bug#45972; Package guix-patches. (Sat, 30 Jan 2021 21:43:02 GMT) Full text and rfc822 format available.

Message #37 received at 45972 <at> debbugs.gnu.org (full text, mbox):

From: Nicolò Balzarotti <anothersms <at> gmail.com>
To: zimoun <zimon.toutoune <at> gmail.com>, Ludovic Courtès
 <ludo <at> gnu.org>
Cc: 45972 <at> debbugs.gnu.org
Subject: Re: [bug#45972] Julia importer?
Date: Sat, 30 Jan 2021 22:42:46 +0100
As a self reminder, I found this repo
https://github.com/cdluminate/DistroHelper.jl
which might contain something useful for our purposes




Information forwarded to guix-patches <at> gnu.org:
bug#45972; Package guix-patches. (Sun, 31 Jan 2021 19:37:02 GMT) Full text and rfc822 format available.

Message #40 received at 45972 <at> debbugs.gnu.org (full text, mbox):

From: zimoun <zimon.toutoune <at> gmail.com>
To: Nicolò Balzarotti <anothersms <at> gmail.com>, Ludovic
 Courtès <ludo <at> gnu.org>
Cc: 45972 <at> debbugs.gnu.org
Subject: Re: [bug#45972] Julia importer?
Date: Sun, 31 Jan 2021 20:35:07 +0100
Hi,

On Sat, 30 Jan 2021 at 21:13, Nicolò Balzarotti <anothersms <at> gmail.com> wrote:
> zimoun <zimon.toutoune <at> gmail.com> writes:

>>> (invoke-julia "using Pkg; Pkg.TOML.parsefile("Project.toml")["name"] |> println")
>>
>> With a bit more glue, could this be transformed into something like
>> “julia->guix-package”?  And so have a Julia package importer, even if it
>> fails for some cases.
>
> Well, if you mean "Can we use Pkg.jl to generate package definitions for
> us?" the answer is "probably yes, but I never investigated this".  That
> line uses just Julia Base TOML.jl, which for some reason is defined
> inside module Pkg.jl (and it's not top-level).

I mean the correct way is to write a TOML parser, probably using PEG.
As it is done with the other importers.

It is a piece of work.  Not especially hard but it could take some time.

So instead, the question is, using a bit of Julia glue and Guile glue,
is it possible to extract the necessary information to have a Guix
package?

Well, maybe these glue code is the same as writing a TOML parser. :-)


> If you instead meant "Can we have a Julia importer?" some time ago I
> wrote one in Julia, I've not used it in the last year, but it did work
> quite well back then.  I attach it here for reference.

Fun!

One thing is how to deal with ’Manifest.toml’.  And pre-compiled
substitutes do not make too much sense in the Julia world.  Well, I do
not know how these Manifest.toml overlap with how Guix works.


Cheers,
simon




Information forwarded to guix-patches <at> gnu.org:
bug#45972; Package guix-patches. (Sun, 31 Jan 2021 20:01:02 GMT) Full text and rfc822 format available.

Message #43 received at 45972 <at> debbugs.gnu.org (full text, mbox):

From: Nicolò Balzarotti <anothersms <at> gmail.com>
To: zimoun <zimon.toutoune <at> gmail.com>, Ludovic Courtès
 <ludo <at> gnu.org>
Cc: 45972 <at> debbugs.gnu.org
Subject: Re: [bug#45972] Julia importer?
Date: Sun, 31 Jan 2021 21:00:21 +0100
> I mean the correct way is to write a TOML parser, probably using PEG.
> As it is done with the other importers.
> It is a piece of work.  Not especially hard but it could take some time.

That wouldn't be too hard, but also it would not be enough.  As you can
see in [fn:1], there's just the list of dependencies, but no info on how
to get them, so parsing the Julia General Registry [fn:2] is still required.

> So instead, the question is, using a bit of Julia glue and Guile glue,
> is it possible to extract the necessary information to have a Guix
> package?
>
> Well, maybe these glue code is the same as writing a TOML parser. :-)

This might be feasible, but as I said above, unfortunately I don't think
that a TOML parser would be enough.  But asking "Pkg3" to resolve
dependencies for us, yes, this makes sense.

> One thing is how to deal with ’Manifest.toml’.  And pre-compiled
> substitutes do not make too much sense in the Julia world.  Well, I do
> not know how these Manifest.toml overlap with how Guix works.
WDYM? Julia uses .ji files (which are the analogoues of .go for guile
and .pyc for python), if this is what you are referring to.

I'm just ignoring Manifest.toml, which should be the same as venv in the
python world.

[fn:1] https://github.com/JuliaLang/TOML.jl/blob/master/Project.toml
[fn:2] https://github.com/JuliaRegistries/General




Information forwarded to guix-patches <at> gnu.org:
bug#45972; Package guix-patches. (Mon, 01 Feb 2021 07:32:02 GMT) Full text and rfc822 format available.

Message #46 received at 45972 <at> debbugs.gnu.org (full text, mbox):

From: zimoun <zimon.toutoune <at> gmail.com>
To: Nicolò Balzarotti <anothersms <at> gmail.com>, Ludovic
 Courtès <ludo <at> gnu.org>
Cc: 45972 <at> debbugs.gnu.org
Subject: Re: [bug#45972] Julia importer?
Date: Mon, 01 Feb 2021 08:22:33 +0100
Hi,

On Sun, 31 Jan 2021 at 21:00, Nicolò Balzarotti <anothersms <at> gmail.com> wrote:

> That wouldn't be too hard, but also it would not be enough.  As you can
> see in [fn:1], there's just the list of dependencies, but no info on how
> to get them, so parsing the Julia General Registry [fn:2] is still required.

[...]

> I'm just ignoring Manifest.toml, which should be the same as venv in the
> python world.

Sorry, you have right.  I have misremembered an old presentation about
Pkg3 by Stefan Karpinski.

Well, thanks for the pointers.  And I gave a look at the Pkg doc. ;-)

Once a PEG parser for TOML files is written, all the information is in
the Registry (by default this General repository), recursively.

However, the “resolve” part is not straightforward, IMHO.  It is what
confused me in my previous email and I thought was “Manifest.toml”.
Anyway.

Instead of reinventing the wheel and reimplement Pkg.jl in Guix, maybe
it is worth to have Julia code as you did that extracts the relevant
information and then generates the relevant Guix packages.  However, it
should mean that “guix import julia” requires to have the package julia
implicitly used.  Why not. :-)


Cheers,
simon







bug archived. Request was from Debbugs Internal Request <help-debbugs <at> gnu.org> to internal_control <at> debbugs.gnu.org. (Mon, 01 Mar 2021 12:24:05 GMT) Full text and rfc822 format available.

This bug report was last modified 3 years and 29 days ago.

Previous Next


GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.