GNU bug report logs - #35588
guix package --search does not search output names

Previous Next

Package: guix;

Reported by: Chris Marusich <cmmarusich <at> gmail.com>

Date: Sun, 5 May 2019 19:40:01 UTC

Severity: normal

Tags: fixed

Done: Chris Marusich <cmmarusich <at> gmail.com>

Bug is archived. No further changes may be made.

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 35588 in the body.
You can then email your comments to 35588 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to bug-guix <at> gnu.org:
bug#35588; Package guix. (Sun, 05 May 2019 19:40:02 GMT) Full text and rfc822 format available.

Acknowledgement sent to Chris Marusich <cmmarusich <at> gmail.com>:
New bug report received and forwarded. Copy sent to bug-guix <at> gnu.org. (Sun, 05 May 2019 19:40:02 GMT) Full text and rfc822 format available.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Chris Marusich <cmmarusich <at> gmail.com>
To: bug-guix <at> gnu.org
Subject: guix package --search does not search output names
Date: Sun, 05 May 2019 12:39:05 -0700
[Message part 1 (text/plain, inline)]
Hi,

mikadoZero reported a bug here:

https://lists.gnu.org/archive/html/help-guix/2019-04/msg00206.html

The bug is reproducible.  On Guix commit
aa7cdc57dc28673dedfc6ec210974aaa0099a419, a search for keyword "cargo"
does not find the "cargo" output of the "rust" package:

--8<---------------cut here---------------start------------->8---
$ guix package --search=cargo
name: chromium-bsu
version: 0.9.16.1
outputs: out
systems: x86_64-linux i686-linux armhf-linux aarch64-linux mips64el-linux
dependencies: gettext-minimal <at> 0.19.8.1 glu <at> 9.0.0 pkg-config <at> 0.29.2
+ quesoglc <at> 0.7.2 sdl-union <at> 1.2.15
location: gnu/packages/games.scm:3295:2
homepage: http://chromium-bsu.sourceforge.net/
license: Clarified Artistic, Expat
synopsis: Fast-paced, arcade-style, top-scrolling space shooter  
description: In this game you are the captain of the cargo ship Chromium
+ B.S.U.  and are responsible for delivering supplies to the troops on the front
+ line.  Your ship has a small fleet of robotic fighters which you control from
+ the relative safety of the Chromium vessel.
relevance: 2
--8<---------------cut here---------------end--------------->8---

It should have printed all the "rust" packages, and the "mrustc"
package, since each of them has a "cargo" output.  Example:

--8<---------------cut here---------------start------------->8---
name: rust
version: 1.34.0
outputs: out doc cargo
systems: x86_64-linux i686-linux armhf-linux aarch64-linux mips64el-linux
dependencies: bison <at> 3.0.5 cmake <at> 3.13.1 curl <at> 7.63.0 flex <at> 2.6.4 gdb <at> 8.2.1
+ jemalloc <at> 5.1.0 libssh2 <at> 1.8.2 llvm <at> 6.0.1 openssl <at> 1.0.2p pkg-config <at> 0.29.2
+ procps <at> 3.3.15 python2 <at> 2.7.15 rust <at> 1.33.0 which <at> 2.21
location: gnu/packages/rust.scm:1026:4
homepage: https://www.rust-lang.org
license: ASL 2.0, Expat
synopsis: Compiler for the Rust progamming language  
description: Rust is a systems programming language that provides memory
+ safety and thread safety guarantees.
--8<---------------cut here---------------end--------------->8---

The general problem here is that Guix does not include outputs whose
name matches the given regex.  One might argue that we should simply add
the output's name to a searchable field, like the description, in cases
like this, but if it's easy to fix, we should just fix it.

-- 
Chris
[signature.asc (application/pgp-signature, inline)]

Information forwarded to bug-guix <at> gnu.org:
bug#35588; Package guix. (Sun, 05 May 2019 21:43:02 GMT) Full text and rfc822 format available.

Message #8 received at 35588 <at> debbugs.gnu.org (full text, mbox):

From: Tobias Geerinckx-Rice <me <at> tobias.gr>
To: 35588 <at> debbugs.gnu.org
Subject: [PATCH] ui: Search matches additional package outputs.
Date: Sun,  5 May 2019 23:41:53 +0200
* guix/ui.scm (%package-metrics): Add a PACKAGE-OUTPUTS metric with a
relevance of 1.
* guix/scripts/package.scm (process-query)<search>: Add the
REGEXP/NEWLINE flag.
---

mikadoZero, Guix,

Here's a patch to match package outputs (except ‘out’, since it can't affect the relative score) in ‘guix search’.

Before:

  ~ λ guix search ernel-patch
  # nothing

After:

  ~ λ guix search ernel-patch
  name: wireguard
  version: 0.0.20190406
  outputs: out kernel-patch
  …

  ~ λ guix search ^ernel-patch
  # nothing

While the new REGEXP/NEWLINE flag affects how all fields are matched, I don't think it actually changes anything in practice without the second hunk.

If there's a possibility that it might, I'd split this into two commits.

Kind regards,

T G-R

 guix/scripts/package.scm | 3 ++-
 guix/ui.scm              | 6 ++++++
 2 files changed, 8 insertions(+), 1 deletion(-)

diff --git a/guix/scripts/package.scm b/guix/scripts/package.scm
index aa27984ea2..a31e78484e 100644
--- a/guix/scripts/package.scm
+++ b/guix/scripts/package.scm
@@ -751,7 +751,8 @@ processed, #f otherwise."
                                       (('query 'search rx) rx)
                                       (_                   #f))
                                     opts))
-              (regexps  (map (cut make-regexp* <> regexp/icase) patterns)))
+              (regexps  (map (cut make-regexp* <> regexp/icase regexp/newline)
+                             patterns)))
          (leave-on-EPIPE
           (let-values (((packages scores)
                         (find-packages-by-description regexps)))
diff --git a/guix/ui.scm b/guix/ui.scm
index 92c845e944..f2466b605b 100644
--- a/guix/ui.scm
+++ b/guix/ui.scm
@@ -1404,6 +1404,12 @@ score, the more relevant OBJ is to REGEXPS."
   ;; of regexps.
   `((,package-name . 4)
 
+    ;; Separate package outputs by newlines to match regexps like "^tools$".
+    ;; Hard-codedly ignore ‘out’ since it presumably exists for every package.
+    (,(lambda (package)
+        (string-join (delete "out" (package-outputs package))
+                     "\n")) . 1)
+
     ;; Match regexps on the raw Texinfo since formatting it is quite expensive
     ;; and doesn't have much of an effect on search results.
     (,(lambda (package)
-- 
2.21.0





Information forwarded to bug-guix <at> gnu.org:
bug#35588; Package guix. (Mon, 06 May 2019 09:10:02 GMT) Full text and rfc822 format available.

Message #11 received at 35588 <at> debbugs.gnu.org (full text, mbox):

From: Chris Marusich <cmmarusich <at> gmail.com>
To: Tobias Geerinckx-Rice <me <at> tobias.gr>
Cc: 35588 <at> debbugs.gnu.org
Subject: Re: bug#35588: [PATCH] ui: Search matches additional package outputs.
Date: Mon, 06 May 2019 02:08:58 -0700
[Message part 1 (text/plain, inline)]
Hi Tobias!

Tobias Geerinckx-Rice <me <at> tobias.gr> writes:

> Here's a patch to match package outputs (except ‘out’, since it can't affect the relative score) in ‘guix search’.

Wow, thank you for the quick patch!  Indeed, it works as advertised.

That said, I see that your patch only omits the "out" output, and it
also changes the way the regular expression matching works for all
fields.  Previously, a regex like "foo$" would have matched against
"foo" at the end of a package's description string, but now it matches
"foo" right before any newline in the description.  In practice, it
seems we have many newlines in the descriptions, since people wrap the
lines in the source code with non-escaped newlines.  I doubt anyone is
relying on the use of $ or ^ to match at the start of end of the
description, so this probably isn't a big deal, but it made me wonder
how else we might be able to accomplish the same task without changing
the way the regexes already work.

I've attached a patch that attempts to do that.  Basically, this patch
is like yours in spirit, but it excludes a few more "common" outputs,
and it splits the outputs into a list, rather than separating them by
newlines in a single string.  The downside is that I had to update all
the other "metrics" procedures so they would always return a list.  I
kind of feel like returning the same type of object is an upside, not a
downside, but I guess the pattern of returning #f for special cases is
pretty common in Guile, so maybe my change isn't very idiomatic.

What do you think of this solution?  I think it's a little more drastic,
but it feels cleaner to me.  If I'm bike shedding, feel free to call me
out on that!  ;-)

-- 
Chris
[0001-ui-Make-package-outputs-searchable.patch (text/x-patch, attachment)]
[signature.asc (application/pgp-signature, inline)]

Information forwarded to bug-guix <at> gnu.org:
bug#35588; Package guix. (Mon, 06 May 2019 09:33:01 GMT) Full text and rfc822 format available.

Message #14 received at 35588 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Chris Marusich <cmmarusich <at> gmail.com>
Cc: Tobias Geerinckx-Rice <me <at> tobias.gr>, 35588 <at> debbugs.gnu.org
Subject: Re: bug#35588: [PATCH] ui: Search matches additional package outputs.
Date: Mon, 06 May 2019 11:32:12 +0200
Hello,

Chris Marusich <cmmarusich <at> gmail.com> skribis:

> Tobias Geerinckx-Rice <me <at> tobias.gr> writes:
>
>> Here's a patch to match package outputs (except ‘out’, since it can't affect the relative score) in ‘guix search’.
>
> Wow, thank you for the quick patch!  Indeed, it works as advertised.
>
> That said, I see that your patch only omits the "out" output, and it
> also changes the way the regular expression matching works for all
> fields.

Yeah, how does this regexp/newline related to matching outputs, Tobias?

> What do you think of this solution?  I think it's a little more drastic,
> but it feels cleaner to me.  If I'm bike shedding, feel free to call me
> out on that!  ;-)

FWIW I find it a bit too drastic.  :-)  It leads to much verbosity and
I’m not sure this is warranted.

>  (define %package-metrics
>    ;; Metrics used to compute the "relevance score" of a package against a set
>    ;; of regexps.
> -  `((,package-name . 4)
> -
> +  `((,(lambda (package)
> +        (list (package-name package)))
> +     . 4)
> +    ;; Match against uncommon outputs.
> +    (,(lambda (package)
> +        (filter (lambda (output)
> +                  (not (member output
> +                               ;; Some common outpus shared by many packages.
> +                               '("out" "debug" "doc" "static"))))
> +                (package-outputs package)))
> +     . 1)

Could we have just this hunk or is there something I’m missing that
would make it insufficient?

Thank you,
Ludo’.




Information forwarded to bug-guix <at> gnu.org:
bug#35588; Package guix. (Tue, 07 May 2019 00:59:02 GMT) Full text and rfc822 format available.

Message #17 received at 35588 <at> debbugs.gnu.org (full text, mbox):

From: Chris Marusich <cmmarusich <at> gmail.com>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: Tobias Geerinckx-Rice <me <at> tobias.gr>, 35588 <at> debbugs.gnu.org
Subject: Re: bug#35588: [PATCH] ui: Search matches additional package outputs.
Date: Mon, 06 May 2019 17:57:46 -0700
[Message part 1 (text/plain, inline)]
Ludovic Courtès <ludo <at> gnu.org> writes:

> Could we have just this hunk or is there something I’m missing that
> would make it insufficient?

That hunk alone is not enough, but I think the attached patch would do
the trick.  We just need to allow for the new possibility that the
"field" procedure returns a list of strings.

What do you think?

-- 
Chris
[0001-ui-Make-package-outputs-searchable.patch (text/x-patch, attachment)]
[signature.asc (application/pgp-signature, inline)]

Information forwarded to bug-guix <at> gnu.org:
bug#35588; Package guix. (Tue, 07 May 2019 08:26:02 GMT) Full text and rfc822 format available.

Message #20 received at 35588 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Chris Marusich <cmmarusich <at> gmail.com>
Cc: Tobias Geerinckx-Rice <me <at> tobias.gr>, 35588 <at> debbugs.gnu.org
Subject: Re: bug#35588: [PATCH] ui: Search matches additional package outputs.
Date: Tue, 07 May 2019 10:25:39 +0200
Hello,

Chris Marusich <cmmarusich <at> gmail.com> skribis:

> From c1150a217a416ef4ceccf87c56e36e8e921f873a Mon Sep 17 00:00:00 2001
> From: Chris Marusich <cmmarusich <at> gmail.com>
> Date: Mon, 6 May 2019 01:51:30 -0700
> Subject: [PATCH] ui: Make package outputs searchable.
>
> * guix/ui.scm (relevance): Allow the "field" procedure of a metric to
> return a list, and handle that case appropriately.  Update docstring.
> (%package-metrics): Add a metric for package outputs.
> * guix/scripts/package.scm (find-packages-by-description): Update
> docstring.
>
> Co-authored-by: Tobias Geerinckx-Rice <me <at> tobias.gr>

[...]

>               (match (field obj)
>                 (#f  relevance)
> -               (str (+ relevance
> -                       (* (score str) weight)))))))
> +               ((? string? str) (+ relevance
> +                                   (* (score str) weight)))
> +               ((? list? lst) (+ relevance
> +                                 (* weight
> +                                    (apply + (map score lst)))))))))

Nitpick: it’s a bit subjective, but I think this clause might be
slightly nicer like this:

  ((lst ...)
   (+ relevance (* weight (reduce + 0 (map score lst)))))

Anyway, LGTM!

Thanks,
Ludo’.




Information forwarded to bug-guix <at> gnu.org:
bug#35588; Package guix. (Tue, 07 May 2019 22:25:01 GMT) Full text and rfc822 format available.

Message #23 received at 35588 <at> debbugs.gnu.org (full text, mbox):

From: swedebugia <swedebugia <at> riseup.net>
Cc: 35588 <at> debbugs.gnu.org
Subject: Re: bug#35588: [PATCH] ui: Search matches additional package outputs.
Date: Wed, 8 May 2019 00:24:39 +0200
On 2019-05-07 02:57, Chris Marusich wrote:
> +    ;; Match against uncommon outputs.
> +    (,(lambda (package)
> +        (filter (lambda (output)
> +                  (not (member output
> +                               ;; Some common outpus shared by many packages.
> +                               '("out" "debug" "doc" "static"))))

I suggest we add "gui" and "lib" to this list.

-- 
Cheers
Swedebugia




Information forwarded to bug-guix <at> gnu.org:
bug#35588; Package guix. (Wed, 08 May 2019 06:51:02 GMT) Full text and rfc822 format available.

Message #26 received at 35588 <at> debbugs.gnu.org (full text, mbox):

From: Chris Marusich <cmmarusich <at> gmail.com>
To: Ludovic Courtès <ludo <at> gnu.org>, swedebugia
 <swedebugia <at> riseup.net>
Cc: Tobias Geerinckx-Rice <me <at> tobias.gr>, 35588 <at> debbugs.gnu.org
Subject: Re: bug#35588: [PATCH] ui: Search matches additional package outputs.,
 bug#35588: [PATCH] ui: Search matches additional package outputs.
Date: Tue, 07 May 2019 23:50:29 -0700
[Message part 1 (text/plain, inline)]
Ludovic Courtès <ludo <at> gnu.org> writes:

>>               (match (field obj)
>>                 (#f  relevance)
>> -               (str (+ relevance
>> -                       (* (score str) weight)))))))
>> +               ((? string? str) (+ relevance
>> +                                   (* (score str) weight)))
>> +               ((? list? lst) (+ relevance
>> +                                 (* weight
>> +                                    (apply + (map score lst)))))))))
>
> Nitpick: it’s a bit subjective, but I think this clause might be
> slightly nicer like this:
>
>   ((lst ...)
>    (+ relevance (* weight (reduce + 0 (map score lst)))))

Works for me!  I've changed the match clause accordingly in the attached
patch.

swedebugia <swedebugia <at> riseup.net> writes:

> On 2019-05-07 02:57, Chris Marusich wrote:
>> +    ;; Match against uncommon outputs.
>> +    (,(lambda (package)
>> +        (filter (lambda (output)
>> +                  (not (member output
>> +                               ;; Some common outpus shared by many packages.
>> +                               '("out" "debug" "doc" "static"))))
>
> I suggest we add "gui" and "lib" to this list.

Actually, I was curious about this, so I checked how many outputs are
being used by all our packages today.  Here are the results:

--8<---------------cut here---------------start------------->8---
scheme@(guix-user)> ,use (gnu packages) (guix packages)
scheme@(guix-user)> (define (increment table key) (hash-set! table key (+ 1 (hash-ref table key 0))))
scheme@(guix-user)> (define (increment-outputs package table) (for-each (lambda (output) (increment table output)) (package-outputs package)) table)
scheme@(guix-user)> (define outputs-to-count (fold-packages increment-outputs (make-hash-table)))
scheme@(guix-user)> ,pp (sort (hash-map->list cons outputs-to-count) (lambda (a b) (< (cdr a) (cdr b))))
$1 = (("kernel-patch" . 1)
 ("pcf-8bit" . 1)
 ("python2" . 1)
 ("schema" . 1)
 ("octave" . 1)
 ("jp" . 1)
 ("db" . 1)
 ("gtk3" . 1)
 ("kr" . 1)
 ("subtree" . 1)
 ("pulseaudio" . 1)
 ("cn" . 1)
 ("ruby" . 1)
 ("fbgrab" . 1)
 ("credential-netrc" . 1)
 ("front-end" . 1)
 ("psf" . 1)
 ("tw" . 1)
 ("libedataserverui" . 1)
 ("gtk2" . 1)
 ("pcf" . 1)
 ("send-email" . 1)
 ("jack" . 1)
 ("python3" . 1)
 ("ndiff" . 1)
 ("installer" . 1)
 ("svn" . 1)
 ("image" . 1)
 ("tiles" . 2)
 ("fortran" . 3)
 ("include" . 3)
 ("utils" . 3)
 ("tests" . 3)
 ("python" . 4)
 ("tk" . 4)
 ("metis" . 4)
 ("gui" . 4)
 ("examples" . 5)
 ("jdk" . 6)
 ("bin" . 8)
 ("cargo" . 16)
 ("static" . 33)
 ("lib" . 38)
 ("debug" . 87)
 ("doc" . 134)
 ("out" . 9811))
scheme@(guix-user)> 
--8<---------------cut here---------------end--------------->8---

In light of that, I've chosen to exclude all of the following outputs:

  '("out" "doc" "debug" "lib" "static" "bin" "examples" "gui" "tests"
    "utils" "include")

I've also added a test to verify that package outputs are included in
search results.

What do you all think of this latest version?

-- 
Chris
[0001-ui-Make-package-outputs-searchable.patch (text/x-patch, attachment)]
[signature.asc (application/pgp-signature, inline)]

Information forwarded to bug-guix <at> gnu.org:
bug#35588; Package guix. (Wed, 08 May 2019 10:40:02 GMT) Full text and rfc822 format available.

Message #29 received at 35588 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Chris Marusich <cmmarusich <at> gmail.com>
Cc: swedebugia <swedebugia <at> riseup.net>, Tobias Geerinckx-Rice <me <at> tobias.gr>,
 35588 <at> debbugs.gnu.org
Subject: Re: bug#35588: [PATCH] ui: Search matches additional package outputs.,
 bug#35588: [PATCH] ui: Search matches additional package outputs.
Date: Wed, 08 May 2019 12:39:44 +0200
Hi!

Chris Marusich <cmmarusich <at> gmail.com> skribis:

> In light of that, I've chosen to exclude all of the following outputs:
>
>   '("out" "doc" "debug" "lib" "static" "bin" "examples" "gui" "tests"
>     "utils" "include")
>
> I've also added a test to verify that package outputs are included in
> search results.

I would exclude only the standard output names used by (guix build
gnu-build-system), so:

  ("out" "doc" "debug" "lib" "include" "bin")

That way one can still do ‘guix search git gui’ or ‘guix search bind utils’.

> What do you all think of this latest version?

Thanks for the new test!  LGTM.

Ludo’.




Reply sent to Chris Marusich <cmmarusich <at> gmail.com>:
You have taken responsibility. (Thu, 09 May 2019 07:23:02 GMT) Full text and rfc822 format available.

Notification sent to Chris Marusich <cmmarusich <at> gmail.com>:
bug acknowledged by developer. (Thu, 09 May 2019 07:23:02 GMT) Full text and rfc822 format available.

Message #34 received at 35588-close <at> debbugs.gnu.org (full text, mbox):

From: Chris Marusich <cmmarusich <at> gmail.com>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: swedebugia <swedebugia <at> riseup.net>, 35588-close <at> debbugs.gnu.org,
 Tobias Geerinckx-Rice <me <at> tobias.gr>
Subject: Re: bug#35588: [PATCH] ui: Search matches additional package outputs.
Date: Thu, 09 May 2019 00:22:03 -0700
[Message part 1 (text/plain, inline)]
Ludovic Courtès <ludo <at> gnu.org> writes:

> I would exclude only the standard output names used by (guix build
> gnu-build-system), so:
>
>   ("out" "doc" "debug" "lib" "include" "bin")
>
> That way one can still do ‘guix search git gui’ or ‘guix search bind utils’.

I didn't know you could do that!  Your suggestion makes sense.  I've
done as you suggest in commit 387e6b9e340ce4b401f220f72881415623a466f7.

Thanks everyone!  I'm closing this bug report.

-- 
Chris
[signature.asc (application/pgp-signature, inline)]

Added tag(s) fixed. Request was from Ludovic Courtès <ludo <at> gnu.org> to control <at> debbugs.gnu.org. (Thu, 09 May 2019 10:19:02 GMT) Full text and rfc822 format available.

bug archived. Request was from Debbugs Internal Request <help-debbugs <at> gnu.org> to internal_control <at> debbugs.gnu.org. (Thu, 06 Jun 2019 11:24:07 GMT) Full text and rfc822 format available.

This bug report was last modified 4 years and 297 days ago.

Previous Next


GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.