GNU bug report logs -
#63001
Go importer doesn't know MODULE/vX.Y version tags
Previous Next
Reported by: "Timo Wilken" <guix <at> twilken.net>
Date: Fri, 21 Apr 2023 17:41:01 UTC
Severity: normal
Done: Sharlatan Hellseher <sharlatanus <at> gmail.com>
Bug is archived. No further changes may be made.
To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 63001 in the body.
You can then email your comments to 63001 AT debbugs.gnu.org in the normal way.
Toggle the display of automated, internal messages from the tracker.
Report forwarded
to
bug-guix <at> gnu.org
:
bug#63001
; Package
guix
.
(Fri, 21 Apr 2023 17:41:01 GMT)
Full text and
rfc822 format available.
Acknowledgement sent
to
"Timo Wilken" <guix <at> twilken.net>
:
New bug report received and forwarded. Copy sent to
bug-guix <at> gnu.org
.
(Fri, 21 Apr 2023 17:41:01 GMT)
Full text and
rfc822 format available.
Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):
Some Golang repositories apparently use a tagging scheme where each
submodule in the repo gets its own tags.
In this case, the example is cloud.google.com/go and its submodules
(hosted at https://github.com/googleapis/google-cloud-go).
For example, for the cloud.google.com/go/longrunning submodule at
version 0.4.1, the Go importer should look for a tag called
"refs/tags/longrunning/v0.4.1" in the Git repository.
#+begin_example
$ guix import go -r cloud.google.com/go/longrunning
Backtrace:
19 (primitive-load "/home/timo/.config/guix/current/bin/gu…")
In guix/ui.scm:
2300:7 18 (run-guix . _)
2263:10 17 (run-guix-command _ . _)
In guix/scripts/import.scm:
89:11 16 (guix-import . _)
In ice-9/boot-9.scm:
1752:10 15 (with-exception-handler _ _ #:unwind? _ # _)
In guix/scripts/import/go.scm:
114:22 14 (_)
In guix/import/utils.scm:
638:27 13 (recursive-import _ #:repo->guix-package _ #:guix-name . #)
630:33 12 (lookup-node "cloud.google.com/go/longrunning" #f)
In guix/memoization.scm:
98:0 11 (mproc "cloud.google.com/go/longrunning" # #<procedure…> …)
In unknown file:
10 (_ #<procedure 7fb3e173c820 at guix/memoization.scm:17…> …)
In guix/import/go.scm:
685:10 9 (_ _ #:version _ #:repo _)
In ice-9/exceptions.scm:
406:15 8 (go-module->guix-package* . _)
In ice-9/boot-9.scm:
1752:10 7 (with-exception-handler _ _ #:unwind? _ # _)
In guix/import/go.scm:
511:19 6 (go-module->guix-package "cloud.google.com/go/longrunn…" …)
In guix/git.scm:
291:4 5 (update-cached-checkout _ #:ref _ #:recursive? _ # _ # _ …)
277:19 4 (resolve _)
In git/reference.scm:
60:8 3 (_ _ _)
In git/bindings.scm:
77:2 2 (raise-git-error _)
In ice-9/boot-9.scm:
1685:16 1 (raise-exception _ #:continuable? _)
1683:16 0 (raise-exception _ #:continuable? _)
ice-9/boot-9.scm:1683:16: In procedure raise-exception:
Git error: reference 'refs/tags/v0.4.1' not found
#+end_example
The same is true for other modules, e.g.
github.com/Azure/azure-sdk-for-go/sdk/internal, where the tag is
called "refs/tags/sdk/internal/v1.3.0" instead:
#+begin_example
$ guix import go -r github.com/Azure/azure-sdk-for-go/sdk/internal
Backtrace:
19 (primitive-load "/home/timo/.config/guix/current/bin/gu…")
In guix/ui.scm:
2300:7 18 (run-guix . _)
2263:10 17 (run-guix-command _ . _)
In guix/scripts/import.scm:
89:11 16 (guix-import . _)
In ice-9/boot-9.scm:
1752:10 15 (with-exception-handler _ _ #:unwind? _ # _)
In guix/scripts/import/go.scm:
114:22 14 (_)
In guix/import/utils.scm:
638:27 13 (recursive-import _ #:repo->guix-package _ #:guix-name . #)
630:33 12 (lookup-node "github.com/Azure/azure-sdk-for-go/sdk/in…" …)
In guix/memoization.scm:
98:0 11 (mproc "github.com/Azure/azure-sdk-for-go/sdk/internal" …)
In unknown file:
10 (_ #<procedure 7fa449ec0300 at guix/memoization.scm:17…> …)
In guix/import/go.scm:
685:10 9 (_ _ #:version _ #:repo _)
In ice-9/exceptions.scm:
406:15 8 (go-module->guix-package* . _)
In ice-9/boot-9.scm:
1752:10 7 (with-exception-handler _ _ #:unwind? _ # _)
In guix/import/go.scm:
511:19 6 (go-module->guix-package "github.com/Azure/azure-sdk-f…" …)
In guix/git.scm:
291:4 5 (update-cached-checkout _ #:ref _ #:recursive? _ # _ # _ …)
277:19 4 (resolve _)
In git/reference.scm:
60:8 3 (_ _ _)
In git/bindings.scm:
77:2 2 (raise-git-error _)
In ice-9/boot-9.scm:
1685:16 1 (raise-exception _ #:continuable? _)
1683:16 0 (raise-exception _ #:continuable? _)
ice-9/boot-9.scm:1683:16: In procedure raise-exception:
Git error: reference 'refs/tags/v1.3.0' not found
#+end_example
Information forwarded
to
bug-guix <at> gnu.org
:
bug#63001
; Package
guix
.
(Sat, 17 Jun 2023 15:14:02 GMT)
Full text and
rfc822 format available.
Message #8 received at 63001 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
Hi Ludo', (hi everyone,)
On Wed Jun 14, 2023 at 11:09 PM CEST, Ludovic Courtès wrote:
> Timo Wilken <guix <at> twilken.net> skribis:
> > Here's a patch that fixes the reported issue (bug#54097) for me. I've only
> > tested this on the github.com/googleapis/google-cloud-go/compute package so
> > far, though it seems to work there. Perhaps others have more testcases?
> >
> > I don't know enough about Go tooling to use it, so I've just patched the Guile
> > logic of the importer. (I don't write Go, I just want to package stuff written
> > in it.) In terms of performance, at least the repo contents are apparently
> > cached by the first `git-checkout-hash' call, even if it fails, so the second
> > call doesn't have to redownload them.
I've been testing my patch further this weekend, and I have a couple more
patches in the pipeline; I suppose I ought to clean those up and submit them.
In particular, I've got fixes for the following queued up locally:
1. Finding the `module-path-subdir' needs another case for e.g.
cloud.google.com/go/*.
2. My patch sometimes generates an unnecessary `go-version->git-ref' call.
3. Go versions need to be parsed from go.mod, since some packages require a
newer Go compiler than our default. This I've got a patch for, but this Go
version also ought to propagate up the dependency tree. I haven't found an
easy way to do that, since the importer seems to generate top-level
packages first, before descending the dep tree...
4. `fetch-module-meta-data' ought to ignore 4xx HTTP errors to follow the
spec; gonum.org/v1/gonum specifically depends on this behaviour.
I've been trying to recursively import github.com/matrix-org/dendrite, which
has a particularly large and hairy dependency tree. While I can now import it
without crashes, I can't build it from the imported package definitions yet --
mainly because of lots of dependency cycles in the generated packages, but
there may be more issues hidden beneath that.
Still, I can recommend it as a test of everyone's importer patches, since
it'll find a lot of edge cases in importing alone!
> What you propose looks similar to part of the work Simon Tournier
> submitted at <https://issues.guix.gnu.org/63647>.
It seems lots of people have been working on the same problem -- in addition
to Simon's patches, I found a patch submitted by Elbek (issues 64035 & 64036;
Cc'd). I also forgot about the issue I submitted months ago (63001)...
> What would you suggest? Simon?
Here's a brief comparison between Simon's patches and mine -- Simon's seem to
contain fixes for a couple more things than mine currently does:
1. Simon sorts available versions in an error message; this can presumably be
merged independently since it doesn't conflict with other patches.
2. Simon always prepends a "SUBDIR/" prefix to the tag if found, whereas I try
to find the plain "vX" tag first, then fall back to "SUBDIR/vX". Judging by
https://go.dev/ref/mod#vcs-version, Simon's approach seems more correct.
I'll change my implementation to match and try it out.
3. For detecting the `module-path-subdirectory' in Simon's patches: that's the
same approach I used initially, but I found I have to try `(substring
module-path (string-length import-prefix))' first (to handle e.g.
cloud.google.com/go/*). This is one of the things I haven't submitted
yet...
> Thanks for the patch, Timo!
Thanks for your work in sorting through all of this, Ludo'!
Cheers,
Timo
[signature.asc (application/pgp-signature, inline)]
Information forwarded
to
bug-guix <at> gnu.org
:
bug#63001
; Package
guix
.
(Wed, 16 Aug 2023 17:12:03 GMT)
Full text and
rfc822 format available.
Message #11 received at 63001 <at> debbugs.gnu.org (full text, mbox):
Hi Timo,
On Sat, 17 Jun 2023 at 17:12, "Timo Wilken" <guix <at> twilken.net> wrote:
>> What would you suggest? Simon?
>
> Here's a brief comparison between Simon's patches and mine -- Simon's seem to
> contain fixes for a couple more things than mine currently does:
>
> 1. Simon sorts available versions in an error message; this can presumably be
> merged independently since it doesn't conflict with other patches.
>
> 2. Simon always prepends a "SUBDIR/" prefix to the tag if found, whereas I try
> to find the plain "vX" tag first, then fall back to "SUBDIR/vX". Judging by
> https://go.dev/ref/mod#vcs-version, Simon's approach seems more correct.
> I'll change my implementation to match and try it out.
>
> 3. For detecting the `module-path-subdirectory' in Simon's patches: that's the
> same approach I used initially, but I found I have to try `(substring
> module-path (string-length import-prefix))' first (to handle e.g.
> cloud.google.com/go/*). This is one of the things I haven't submitted
> yet...
Sorry if I have missed some patches or overlooked something. Do you
plan to send another patch series handling all?
Cheers,
simon
Reply sent
to
Sharlatan Hellseher <sharlatanus <at> gmail.com>
:
You have taken responsibility.
(Sat, 25 Jan 2025 13:32:01 GMT)
Full text and
rfc822 format available.
Notification sent
to
"Timo Wilken" <guix <at> twilken.net>
:
bug acknowledged by developer.
(Sat, 25 Jan 2025 13:32:01 GMT)
Full text and
rfc822 format available.
Message #16 received at 63001-done <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
Hi,
The latest changes in go-importer have resolved it:
--8<---------------cut here---------------start------------->8---
~$ guix describe
Generation 72 Jan 24 2025 22:18:29 (current)
guix ea7a6fa
repository URL: https://git.savannah.gnu.org/git/guix.git
branch: master
commit: ea7a6fa8c66ef3cfd0ccdec6c0a7a3e7ab3dfc9c
~$ guix import go cloud.google.com/go/longrunning
guix import: Importing package "cloud.google.com/go/longrunning"...
(define-public go-cloud-google-com-go-longrunning
(package
(name "go-cloud-google-com-go-longrunning")
(version "0.6.4")
(source
(origin
(method git-fetch)
(uri (git-reference
(url "https://github.com/googleapis/google-cloud-go")
(commit (go-version->git-ref version
#:subdir "longrunning"))))
(file-name (git-file-name name version))
(sha256
(base32 "0w04dgxk0lx5k4s255ladf5w8n2nivvhp5vlyb2va96j5aa8j2q5"))))
(build-system go-build-system)
(arguments
(list
#:import-path "cloud.google.com/go/longrunning"
#:unpack-path "cloud.google.com/go"))
(propagated-inputs (list go-google-golang-org-protobuf
go-google-golang-org-grpc
go-google-golang-org-genproto-googleapis-rpc
go-google-golang-org-genproto-googleapis-api
go-google-golang-org-genproto
go-google-golang-org-api
go-github-com-googleapis-gax-go-v2
go-cloud-google-com-go))
(home-page "https://cloud.google.com/go")
(synopsis "longrunning")
(description
"Package longrunning supports Long Running Operations for the Google Cloud
Libraries. See google.golang.org/genproto/googleapis/longrunning for its
service definition.")
(license license:asl2.0)))
--8<---------------cut here---------------end--------------->8---
The recursive operation is tests against restic and it imported 400 new
packages after 1h.
Closing as resolved.
--
Thanks,
Oleg
[signature.asc (application/pgp-signature, inline)]
bug archived.
Request was from
Debbugs Internal Request <help-debbugs <at> gnu.org>
to
internal_control <at> debbugs.gnu.org
.
(Sun, 23 Feb 2025 12:24:05 GMT)
Full text and
rfc822 format available.
This bug report was last modified 18 days ago.
Previous Next
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997,2003 nCipher Corporation Ltd,
1994-97 Ian Jackson.