GNU bug report logs - #77833
Xapian cache/search proof of concept

Previous Next

To reply to this bug, email your comments to 77833 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Noé Lopez <noe <at> xn--no-cja.eu>
To: guix-patches <at> gnu.org
Cc: Arun Isaac <arunisaac <at> systemreboot.net>,
 Ludovic Courtès <ludo <at> gnu.org>
Subject: Xapian cache/search proof of concept
Date: Tue, 15 Apr 2025 23:14:41 +0200

[Message part 1 (text/plain, inline)]

Hi Arun, Ludo, and everyone,

I Just stumbled upon Arun’s suggestion of doing a guix xsearch extension
to use a xapian search in #39258 so I gave it a try tonight, here’s the
proof of concept.

Search is less than a second (loading guile modules, actual search is
instantaneous) and building the cache takes ~20 seconds.

The whole thing was very easy to make following the guile-xapian
example, and it shows as it is only 71 loc.

So, what do you think? Is it an idea worth pursuing?

[guix-xsearch.scm (text/plain, inline)]

(define-module (guix-xsearch)
  #:use-module ((gnu packages) #:select (fold-packages
					 find-packages-by-name))
  #:use-module ((guix build utils) #:select (package-name->name+version))
  #:use-module (guix packages)
  #:use-module (ice-9 match)
  #:use-module (srfi srfi-11)
  #:use-module (xapian xapian)
  #:use-module (statprof))

(define %database-path "guix-xsearch.xapian")

(define (index)
  (call-with-writable-database
   %database-path
   (lambda (database)
     (fold-packages
      (lambda (package _)
	(let* ((name (package-name package))
	       (version (package-version package))
	       (description (package-description package))
	       (synopsis (package-synopsis package))
	       (name+version (string-append name "@" version))
	       (idterm (string-append "Q" name+version))
	       (document (make-document #:data name+version
					#:terms `((,idterm . 0))))
	       (term-generator (make-term-generator #:stem (make-stem "en")
						    #:document document)))
          ;; Index title and description with a suitable
          ;; prefix. This is used to allow for searching separate
          ;; fields as in name:openttd, description:leather,
          ;; etc.
	  ;; Disabled for performance.
	  (index-text! term-generator name #:prefix "S")
	  (index-text! term-generator synopsis #:prefix "B")
          (index-text! term-generator description #:prefix "XD")
          ;; Index title and description without prefixes for
          ;; general search.
	  (index-text! term-generator name)
          (increase-termpos! term-generator)
	  (index-text! term-generator synopsis)
          (index-text! term-generator description)
          ;; Add the document to the database. The unique idterm
          ;; ensures each object ends up in the database only once
          ;; no matter how many times we run the indexer.
          (replace-document! database idterm document)
	  #nil))
      #nil))))

(define (search query-string)
  (call-with-database
   %database-path
   (lambda (database)
     (let* ((query (parse-query query-string
			       #:stemmer (make-stem "en")
			       #:prefixes '(("name" . "S")
					    ("synopsis" . "B")
					    ("description" . "XD"))))
	    (mset (enquire-mset (enquire database query)
				#:maximum-items 10)))
       (mset-fold
	(lambda (item _)
	  (let* ((name+version (string-split (document-data (mset-item-document item))
					     #\@))
		 ;; FIXME: Use a more precise way to restore the
		 ;; package, like the package cache does.
		 (packages (find-packages-by-name
			    (car name+version)
			    (cadr name+version))))
	    (for-each
	     (lambda (package)
	       (format #t "~a: #~3,'0d ~a~%"
		       (mset-item-rank item)
		       (mset-item-docid item)
		       package))
	     packages))
	  #nil)
	#nil
	mset)))))

(match (command-line)
  ((_ "index")
   (index))
  ((_ "search" query)
   (search query))
  ((program . _)
   (format (current-error-port) "Usage: ~a index | search <query>~%" program)))

[signature.asc (application/pgp-signature, inline)]

Message #8 received at 77833 <at> debbugs.gnu.org (full text, mbox):

From: Simon Tournier <zimon.toutoune <at> gmail.com>
To: Noé Lopez <noe <at> xn--no-cja.eu>, 77833 <at> debbugs.gnu.org
Cc: Arun Isaac <arunisaac <at> systemreboot.net>,
 Ludovic Courtès <ludo <at> gnu.org>
Subject: Re: [bug#77833] Xapian cache/search proof of concept
Date: Wed, 16 Apr 2025 17:42:50 +0200

Hi Noé,

On Tue, 15 Apr 2025 at 23:14, Noé Lopez via Guix-patches via <guix-patches <at> gnu.org> wrote:

> So, what do you think? Is it an idea worth pursuing?

Yes.  Here [1] the description of what appears to me worth to
continue. :-)

Well, I’ve started something in that direction… but it’s much faster and
easier to look for an email indexed with notmuch (Xapian) than to browse
my files on various machines. ;-)

I think the best is to open a repository for your extension; maybe on
Codeberg.  Then, we could iterate on that.  WDYT?

Cheers,
simon

1: Extension for improving Guix search?
Simon Tournier <zimon.toutoune <at> gmail.com>
Wed, 22 May 2024 12:15:09 +0200
id:87msoi40gi.fsf <at> gmail.com
https://lists.gnu.org/archive/html/guix-devel/2024-05
https://yhetil.org/guix/87msoi40gi.fsf <at> gmail.com

Message #11 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Arun Isaac <arunisaac <at> systemreboot.net>
To: Noé Lopez <noe <at> xn--no-cja.eu>, guix-patches <at> gnu.org
Cc: Ludovic Courtès <ludo <at> gnu.org>
Subject: Re: Xapian cache/search proof of concept
Date: Wed, 16 Apr 2025 18:12:40 +0100

Hi Noé,

That's fabulous! Thank you for taking this up. :-) Please consider
making this its own project with its own repo, and advertising on
guix-devel and elsewhere.

> (match (command-line)
>   ((_ "index")
>    (index))
>   ((_ "search" query)
>    (search query))

If there's some way to hide the indexing step from the user, that would
be good. For example, one way could be to build the index the first time
search is run. You'll have to figure out some way to make this a smooth
user experience.

Remember that the user may jump back and forth between multiple guix
profiles. So, multiple indices may be necessary.

Regards,
Arun

Message #14 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Noé Lopez <noe <at> xn--no-cja.eu>
Cc: Arun Isaac <arunisaac <at> systemreboot.net>, guix-patches <at> gnu.org
Subject: Re: Xapian cache/search proof of concept
Date: Wed, 16 Apr 2025 22:01:37 +0200

Hello,

Noé Lopez <noe <at> noé.eu> writes:

> Search is less than a second (loading guile modules, actual search is
> instantaneous) and building the cache takes ~20 seconds.
>
> The whole thing was very easy to make following the guile-xapian
> example, and it shows as it is only 71 loc.

I forgot the outcome of discussions with Simon back when they worked on
it, but now I wonder: would it be reasonable to have it in ‘guix search’
proper, with indexing happening on first ‘guix search’ for a given
channel set?

I suppose 20s is on an SSD with a warm cache; would be nice to check on
spinning disks (on an SSD current ‘guix search’ is fast enough IMO).

One thing is that I’m not confident about the use of SWIG in
Guile-Xapian (I used SWIG back when it was fashionable and it didn’t do
a great job), I’d rather not have Guix depend on it.

Ludo’.

Message #17 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Arun Isaac <arunisaac <at> systemreboot.net>
To: Ludovic Courtès <ludo <at> gnu.org>, Noé
 Lopez <noe <at> xn--no-cja.eu>
Cc: guix-patches <at> gnu.org
Subject: Re: Xapian cache/search proof of concept
Date: Wed, 16 Apr 2025 21:28:09 +0100

Hi Ludo,

> I forgot the outcome of discussions with Simon back when they worked on
> it, but now I wonder: would it be reasonable to have it in ‘guix search’
> proper, with indexing happening on first ‘guix search’ for a given
> channel set?

We might also consider building the xapian index along with the package
cache. But, that does come with the cost of slowing down profile builds
slightly.

> One thing is that I’m not confident about the use of SWIG in
> Guile-Xapian (I used SWIG back when it was fashionable and it didn’t do
> a great job), I’d rather not have Guix depend on it.

I'm not a big fan of SWIG either. But, the main reason guile-xapian uses
SWIG is because that's what upstream
https://github.com/xapian/xapian/tree/master/xapian-bindings provides us
with. I'm not saying it is the best situation, but it is what it is, I
guess.

Regards,
Arun

Message #20 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Arun Isaac <arunisaac <at> systemreboot.net>
Cc: Noé Lopez <noe <at> xn--no-cja.eu>, guix-patches <at> gnu.org
Subject: Re: Xapian cache/search proof of concept
Date: Wed, 16 Apr 2025 23:00:00 +0200

Arun Isaac <arunisaac <at> systemreboot.net> writes:

>> I forgot the outcome of discussions with Simon back when they worked on
>> it, but now I wonder: would it be reasonable to have it in ‘guix search’
>> proper, with indexing happening on first ‘guix search’ for a given
>> channel set?
>
> We might also consider building the xapian index along with the package
> cache. But, that does come with the cost of slowing down profile builds
> slightly.

More than slightly I’m afraid, even if it’s “just” 20s.  :-)

Ludo’.

Message #23 received at 77833 <at> debbugs.gnu.org (full text, mbox):

From: Noé Lopez <noe <at> xn--no-cja.eu>
To: Simon Tournier <zimon.toutoune <at> gmail.com>, 77833 <at> debbugs.gnu.org
Cc: Arun Isaac <arunisaac <at> systemreboot.net>,
 Ludovic Courtès <ludo <at> gnu.org>
Subject: Re: [bug#77833] Xapian cache/search proof of concept
Date: Wed, 16 Apr 2025 23:38:23 +0200

[Message part 1 (text/plain, inline)]

Simon Tournier <zimon.toutoune <at> gmail.com> writes:

> Hi Noé,
>
> On Tue, 15 Apr 2025 at 23:14, Noé Lopez via Guix-patches via <guix-patches <at> gnu.org> wrote:
>
>> So, what do you think? Is it an idea worth pursuing?
>
> Yes.  Here [1] the description of what appears to me worth to
> continue. :-)
>
> Well, I’ve started something in that direction… but it’s much faster and
> easier to look for an email indexed with notmuch (Xapian) than to browse
> my files on various machines. ;-)
>
> I think the best is to open a repository for your extension; maybe on
> Codeberg.  Then, we could iterate on that.  WDYT?
>

Will do, thanks!

> Cheers,
> simon
>
> 1: Extension for improving Guix search?
> Simon Tournier <zimon.toutoune <at> gmail.com>
> Wed, 22 May 2024 12:15:09 +0200
> id:87msoi40gi.fsf <at> gmail.com
> https://lists.gnu.org/archive/html/guix-devel/2024-05
> https://yhetil.org/guix/87msoi40gi.fsf <at> gmail.com

There seems to have been a lot of attempts at this, do you know why they
did not work? What is there to learn from these previous attempts?

Good day,
Noé

[signature.asc (application/pgp-signature, inline)]

Message #26 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Noé Lopez <noe <at> xn--no-cja.eu>
To: Arun Isaac <arunisaac <at> systemreboot.net>, guix-patches <at> gnu.org
Cc: Ludovic Courtès <ludo <at> gnu.org>
Subject: Re: Xapian cache/search proof of concept
Date: Wed, 16 Apr 2025 23:46:07 +0200

[Message part 1 (text/plain, inline)]

Arun Isaac <arunisaac <at> systemreboot.net> writes:

> Hi Noé,
>
> That's fabulous! Thank you for taking this up. :-) Please consider
> making this its own project with its own repo, and advertising on
> guix-devel and elsewhere.
>

I’m amazed by the interest :) Will do when I have a working extension.

>> (match (command-line)
>>   ((_ "index")
>>    (index))
>>   ((_ "search" query)
>>    (search query))
>
> If there's some way to hide the indexing step from the user, that would
> be good. For example, one way could be to build the index the first time
> search is run. You'll have to figure out some way to make this a smooth
> user experience.
>

I could schedule an index in the background and run the normal search 🤔

> Remember that the user may jump back and forth between multiple guix
> profiles. So, multiple indices may be necessary.

Right, I’ll think about that.

>
> Regards,
> Arun

Good day,
Noé

[signature.asc (application/pgp-signature, inline)]

Message #29 received at 77833 <at> debbugs.gnu.org (full text, mbox):

From: Simon Tournier <zimon.toutoune <at> gmail.com>
To: Noé Lopez <noe <at> xn--no-cja.eu>, 77833 <at> debbugs.gnu.org
Cc: Arun Isaac <arunisaac <at> systemreboot.net>,
 Ludovic Courtès <ludo <at> gnu.org>
Subject: Re: [bug#77833] Xapian cache/search proof of concept
Date: Thu, 17 Apr 2025 11:58:47 +0200

Hi Noé,

On Wed, 16 Apr 2025 at 23:38, Noé Lopez via Guix-patches via <guix-patches <at> gnu.org> wrote:

> There seems to have been a lot of attempts at this, do you know why they
> did not work?

Why? On my side, procrastination coupled to other fish to fry.  Or maybe
we were just waiting you. :-)

Somehow, I wanted to clean how to write Guix extensions before.  Recent
discussions with Nicolas (Graves) seems a very good to resume all that!

Thank you for pushing forward.

Cheers,
simon

Message #32 received at 77833 <at> debbugs.gnu.org (full text, mbox):

From: Simon Tournier <zimon.toutoune <at> gmail.com>
To: Arun Isaac <arunisaac <at> systemreboot.net>, noe <at> xn--no-cja.eu,
 77833 <at> debbugs.gnu.org
Cc: Ludovic Courtès <ludo <at> gnu.org>
Subject: Re: [bug#77833] Xapian cache/search proof of concept
Date: Thu, 17 Apr 2025 12:37:34 +0200

Hi,

On Wed, 16 Apr 2025 at 18:12, Arun Isaac <arunisaac <at> systemreboot.net> wrote:

> If there's some way to hide the indexing step from the user, that would
> be good. For example, one way could be to build the index the first time
> search is run. You'll have to figure out some way to make this a smooth
> user experience.

Well, the design isn’t obvious to me: that’s a typical case of cache
invalidation problem and we all know, there are two hard problems:
naming thing and cache invalidation. ;-)

Somehow, the best would to have the ability to hook “guix pull” (or
“guix time-machine”) with this indexing step.  Currently, I do not think
it is doable, is it?   But that could be helpful: trigger some actions
(hook) registered by some Guix extensions.  Well, it seems out of the
scope at first. :-)


> Remember that the user may jump back and forth between multiple guix
> profiles. So, multiple indices may be necessary.

Yes.  Today, I’m more interested in being able to search in all the
history than in having faster search. :-) I mean, I see faster search as
a collateral “damage” of searching in all the Guix history
revisions. ;-)

Somehow, what I started^W failed – because I’m a procrastinator ;-) –
long ago was to be able to substitute some Xapian database.

For example, the extension was named “guix chase”, IIRC, and I wanted to
have “guix chase pull” which updates some local Xapian database.  Then,
I could run “guix chase search hello” and find various versions of
’hello’ with some associated Guix revisions.  Bah in the middle I
entered in a blackhole. Whatever.

Today, searching across the Guix history is really annoying.  Somehow, I
do:

    $ git -C ~/src/guix/guix log --format="%h %s" | grep 'gnu: bowtie:'
    a47a90b900 gnu: bowtie: Remove reference to %outputs.
    f336cc4fe7 gnu: bowtie: Replace invalid characters.
    e5a26a1f02 gnu: bowtie: Remove trailing #T.
    2ec601580b gnu: bowtie: Use TBB 2020.
    21c837405a gnu: bowtie: Update to 2.3.4.3.
    06e372360e gnu: bowtie: Use 'modify-phases'.
    d6e63cf31c gnu: bowtie: Update to 2.3.2.
    2642231b39 gnu: bowtie: Update to 2.2.9.
    0047d26a22 gnu: bowtie: Update to 2.2.6.
    241e122193 gnu: bowtie: fix build errors

which is not super handy.  Well, it was somehow an idea behind the
Magali’s Outreachy internship implementing some “guix git log”: make it
a bit more handy.

Anyway.

Thanks Noé for working on that.  Let me know where is the Git
repository. :-)

Cheers,
simon

Message #35 received at 77833 <at> debbugs.gnu.org (full text, mbox):

From: Simon Tournier <zimon.toutoune <at> gmail.com>
To: Ludovic Courtès <ludo <at> gnu.org>, Noé
 Lopez <noe <at> xn--no-cja.eu>
Cc: arunisaac <at> systemreboot.net, 77833 <at> debbugs.gnu.org
Subject: Re: [bug#77833] Xapian cache/search proof of concept
Date: Thu, 17 Apr 2025 12:04:15 +0200

Hi Ludo,

On Wed, 16 Apr 2025 at 22:01, Ludovic Courtès <ludo <at> gnu.org> wrote:

> I forgot the outcome of discussions with Simon back when they worked on
> it, but now I wonder: would it be reasonable to have it in ‘guix search’
> proper, with indexing happening on first ‘guix search’ for a given
> channel set?

Adding Xapian as a dependency of Guix would be unwise, IMHO.

I would prefer we take the other direction: having a light core Guix and
add around some extensions – at least as light as possible :-) –;
instead of adding more and more to what comes by default with “guix
pull”.

Cheers,
simon

Message #38 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Noé Lopez <noe <at> xn--no-cja.eu>
To: guix-patches <at> gnu.org
Cc: Arun Isaac <arunisaac <at> systemreboot.net>,
 Ludovic Courtès <ludo <at> gnu.org>,
 Simon Tournier <zimon.toutoune <at> gmail.com>
Subject: Re: Xapian cache/search proof of concept
Date: Sat, 19 Apr 2025 00:07:26 +0200

[Message part 1 (text/plain, inline)]

Hi everyone!

I have finished cooking, so here is Guile Xsearch 0.1!

It is available at https://codeberg.org/Baleine/guix-xsearch.  After
doing a local clone, you can use it like this:

- make env
- guix xsearch --index
- guix xsearch <search>

Please try it out and tell me what you think! Comments on the code are
also welcome :)

Good day,
Noé

[signature.asc (application/pgp-signature, inline)]

Message #41 received at 77833 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Noé Lopez <noe <at> xn--no-cja.eu>
Cc: Arun Isaac <arunisaac <at> systemreboot.net>,
 Simon Tournier <zimon.toutoune <at> gmail.com>, 77833 <at> debbugs.gnu.org
Subject: Re: Xapian cache/search proof of concept
Date: Sat, 19 Apr 2025 11:09:17 +0200

Noé Lopez <noe <at> noé.eu> writes:

> I have finished cooking, so here is Guile Xsearch 0.1!

Wo0t!

> It is available at https://codeberg.org/Baleine/guix-xsearch.  After
> doing a local clone, you can use it like this:

If you make it a channel, people should be able to add it to their
channel list and then ‘guix xsearch’ will work out of the box (see
commit 60c41183d9c47fb25270fe810d03c0785406faad).

Ludo’.

Message #44 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Arun Isaac <arunisaac <at> systemreboot.net>
To: Noé Lopez <noe <at> xn--no-cja.eu>, guix-patches <at> gnu.org
Cc: Ludovic Courtès <ludo <at> gnu.org>,
 Simon Tournier <zimon.toutoune <at> gmail.com>
Subject: Re: Xapian cache/search proof of concept
Date: Wed, 23 Apr 2025 17:02:45 +0100

Hi Noé,

> - make env
> - guix xsearch --index
> - guix xsearch <search>
>
> Please try it out and tell me what you think! Comments on the code are
> also welcome :)

Lovely!

I think your comment at
https://codeberg.org/Baleine/guix-xsearch/src/commit/1a246259d9241a3ca27adf807169187d941810d6/src/guix-xsearch/search.scm#L35
is a good idea. It will make the search even more blazing fast.

Regards,
Arun

Message #47 received at 77833 <at> debbugs.gnu.org (full text, mbox):

From: Arun Isaac <arunisaac <at> systemreboot.net>
To: Ludovic Courtès <ludo <at> gnu.org>, Noé
 Lopez <noe <at> xn--no-cja.eu>
Cc: Simon Tournier <zimon.toutoune <at> gmail.com>, 77833 <at> debbugs.gnu.org
Subject: Re: Xapian cache/search proof of concept
Date: Wed, 23 Apr 2025 17:03:32 +0100

> If you make it a channel, people should be able to add it to their
> channel list and then ‘guix xsearch’ will work out of the box (see
> commit 60c41183d9c47fb25270fe810d03c0785406faad).

Good idea. +1 to that.

Message #50 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Noé Lopez <noe <at> xn--no-cja.eu>
To: guix-patches <at> gnu.org
Cc: Arun Isaac <arunisaac <at> systemreboot.net>,
 Ludovic Courtès <ludo <at> gnu.org>,
 Simon Tournier <zimon.toutoune <at> gmail.com>
Subject: Re: Xapian cache/search proof of concept
Date: Mon, 28 Apr 2025 00:05:59 +0200

[Message part 1 (text/plain, inline)]

Hi everyone!

I’ve implemented everything I wanted, so here is Guix Xapian v1.1!

As proposed by Ludo, you can now use it as a channel, detailed usage
instructions are in the README at
<https://codeberg.org/Baleine/guix-xsearch>.

After installing, you can just run guix xsearch --index and then guix
xsearch <search> to get <0.2 seconds search!

New features compared to 0.1:
- Use as channel.
- Separate caches per profile.
- Faster search by not loading modules.

I’d appreciate if you could try it out and see if it works for you
before I announce it on guix-devel.

Have a nice day,
Noé

[signature.asc (application/pgp-signature, inline)]

Message #53 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Gabriel Santos <gabrielsantosdesouza <at> disroot.org>
To: Noé Lopez <noe <at> xn--no-cja.eu>,
 Noé Lopez via Guix-patches via <guix-patches <at> gnu.org>,
 77833 <at> debbugs.gnu.org
Cc: Arun Isaac <arunisaac <at> systemreboot.net>,
 Ludovic Courtès <ludo <at> gnu.org>,
 Simon Tournier <zimon.toutoune <at> gmail.com>
Subject: Re: [bug#77833] Xapian cache/search proof of concept
Date: Sun, 27 Apr 2025 20:12:39 -0300

Em 27 de abril de 2025 19:05:59 BRT, "Noé Lopez via Guix-patches via" <guix-patches <at> gnu.org> escreveu:
>Hi everyone!
>
>I’ve implemented everything I wanted, so here is Guix Xapian v1.1!
>
>As proposed by Ludo, you can now use it as a channel, detailed usage
>instructions are in the README at
><https://codeberg.org/Baleine/guix-xsearch>.
>
>After installing, you can just run guix xsearch --index and then guix
>xsearch <search> to get <0.2 seconds search!
>
>New features compared to 0.1:
>- Use as channel.
>- Separate caches per profile.
>- Faster search by not loading modules.
>
>I’d appreciate if you could try it out and see if it works for you
>before I announce it on guix-devel.
>
>Have a nice day,
>Noé

Greetings,

I'll try it out, and do an additional test by adding it as a television[1]
backend.

First issue I can point out though, your channel introduction has a
mistake:

>/home/noe/src/guix-xsearch

Should be:

>https://codeberg.org/Baleine/guix-xsearch

Regards,

-- 
Gabriel Santos

Message #59 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Gabriel Santos <gabrielsantosdesouza <at> disroot.org>
To: Noé Lopez <noe <at> xn--no-cja.eu>,
 Noé Lopez via Guix-patches via <guix-patches <at> gnu.org>,
 77833 <at> debbugs.gnu.org
Cc: Arun Isaac <arunisaac <at> systemreboot.net>,
 Ludovic Courtès <ludo <at> gnu.org>,
 Simon Tournier <zimon.toutoune <at> gmail.com>
Subject: Re: [bug#77833] Xapian cache/search proof of concept
Date: Sun, 27 Apr 2025 20:14:19 -0300

>I'll try it out, and do an additional test by adding it as a television[1]
>backend.

[1] <https://github.com/alexpasmantier/television>

-- 
Gabriel Santos

Message #65 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Gabriel Santos <gabrielsantosdesouza <at> disroot.org>
To: noe <at> xn--no-cja.eu, 77833 <at> debbugs.gnu.org,
 Gabriel Santos via Guix-patches via <guix-patches <at> gnu.org>
Cc: Arun Isaac <arunisaac <at> systemreboot.net>,
 Ludovic Courtès <ludo <at> gnu.org>,
 Simon Tournier <zimon.toutoune <at> gmail.com>
Subject: Re: [bug#77833] Xapian cache/search proof of concept
Date: Sun, 27 Apr 2025 20:17:16 -0300

>First issue I can point out though, your channel introduction has a
>mistake:
>
>>/home/noe/src/guix-xsearch
>
>Should be:
>
>>https://codeberg.org/Baleine/guix-xsearch

Just tried to pull, you're missing the "keyring" branch[1].

You just have to export the keys used there.

[1] <https://guix.gnu.org/manual/en/html_node/Specifying-Channel-Authorizations.html>

-- 
Gabriel Santos

Message #71 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Noé Lopez <noe <at> xn--no-cja.eu>
To: Gabriel Santos <gabrielsantosdesouza <at> disroot.org>,
 77833 <at> debbugs.gnu.org, Gabriel Santos via Guix-patches via
 <guix-patches <at> gnu.org>
Cc: Arun Isaac <arunisaac <at> systemreboot.net>,
 Ludovic Courtès <ludo <at> gnu.org>,
 Simon Tournier <zimon.toutoune <at> gmail.com>
Subject: Re: [bug#77833] Xapian cache/search proof of concept
Date: Mon, 28 Apr 2025 10:43:53 +0200

[Message part 1 (text/plain, inline)]

Gabriel Santos <gabrielsantosdesouza <at> disroot.org> writes:

>>First issue I can point out though, your channel introduction has a
>>mistake:
>>
>>>/home/noe/src/guix-xsearch
>>
>>Should be:
>>
>>>https://codeberg.org/Baleine/guix-xsearch
>
> Just tried to pull, you're missing the "keyring" branch[1].
>
> You just have to export the keys used there.
>
> [1] <https://guix.gnu.org/manual/en/html_node/Specifying-Channel-Authorizations.html>
>
> -- 
> Gabriel Santos

Woops! I forgot to push that branch, well couldn’t notice directly since
I was mistakenly using my local repository.

It’s fixed now, here’s the new channel spec for reference:
(channel
 (name 'guix-xsearch)
 (url "https://codeberg.org/baleine/guix-xsearch")
 (introduction
  (make-channel-introduction
   "8035a2599cb94fca558e518c8e592cae646f5966"
   (openpgp-fingerprint
    "5D54 CF25 57B2 38E8 8DC1 80A2 2D22 3241 0AB7 4043"))))

Thanks for the report,
Noé

[signature.asc (application/pgp-signature, inline)]

Message #77 received at 77833 <at> debbugs.gnu.org (full text, mbox):

From: Gabriel Santos <gabrielsantosdesouza <at> disroot.org>
To: 77833 <at> debbugs.gnu.org
Cc: Noé Lopez <noe <at> xn--no-cja.eu>,
 Arun Isaac <arunisaac <at> systemreboot.net>,
 Gabriel Santos <gabrielsantosdesouza <at> disroot.org>,
 Ludovic Courtès <ludo <at> gnu.org>,
 Simon Tournier <zimon.toutoune <at> gmail.com>
Subject: Re: Xapian cache/search proof of concept
Date: Mon, 28 Apr 2025 08:42:56 -0300

Greetings,

I subscribed to the channel[1] and installed the package[2], but
couldn't get it working on my system. I then tried to install guile
(which I didn't do because I already had it in my profile from some
other package), but that was to no avail:

>$ guix package -i guix-xsearch guile
>The following packages will be installed:
>   guile        3.0.9
>   guix-xsearch 1.1
>$ guix xsearch
>guix: xsearch: command not found
>hint: Did you mean `search'?
>
>Try `guix --help' for more information.

I then tried the "make env" shell from the channel and that worked:

>$ make env
>guix shell -m env/manifest.scm -- || exit 0	#ignore the shell’s return value
>$ guix xsearch guix-search
>name: guix-xsearch
>[...]

Additionally, trying to install it as package by adding the channel to the load
path also doesn't work:

>$ guix package -i guix-xsearch guile -L src/channel
>The following packages will be installed:
>   guix-xsearch 1.1
>$ guix xsearch
>guix: xsearch: command not found
>hint: Did you mean `search'?
>
>Try `guix --help' for more information.

Because of this, I couldn't do the test with television[3-4] that I wanted.

I tried 10 different searches, and this seems to have "issues"—more like
warnings, as there's no crashes—with longer results:

>$ make env
>guix shell -m env/manifest.scm -- || exit 0	#ignore the shell’s return value
>$$ guix xsearch kde
>name: kde-games
>[...]

I then scroll to the third result, which in my case is kde-cli-tools, by
pressing "PageDown" + "Up Arrow" to keep the "name" line at the top:

>name: kde-cli-tools
>[...]

And press "q", to quit (I think that search uses the default pager,
which in my case is still less):

>Backtrace:
>           9 (primitive-load "/home/gabriel/.config/guix/current/bin…")
>In guix/ui.scm:
>   2352:7  8 (run-guix . _)
>  2315:10  7 (run-guix-command _ . _)
>  1807:16  6 (call-with-paginated-output-port _ #:less-options _)
>  1850:11  5 (_ #<output: #{write pipe}# 20>)
>   1655:2  4 (package->recutils _ #<output: #{write pipe}# 20> _ # _ …)
>In ice-9/format.scm:
>  1548:18  3 (format #<output: #{write pipe}# 20> "~a~%" "descriptio…")
>   216:11  2 (format:format-work "~a~%" ("description: `trayer' is …"))
>     65:8  1 (format:out-obj-padded _ _ _ _)
>In unknown file:
>           0 (display "description: `trayer' is small program desig…" …)
>
>ERROR: In procedure display:
>In procedure fport_write: Broken pipe

This is only a issue if you stop at the middle of a search result. It
doesn't happen when you press "END" for examplle to get to the very last
one.

With those issues aside, it seems realy good, the search seems to be almost
instant.

[1] <https://codeberg.org/gs-101/.files/commit/d8405ce168b67323e0de03e51c111ec75b127c5d>
[2] <https://codeberg.org/gs-101/.files/commit/9d8a9d81a5aa5b5304997e714537df771d7725df>
[3] <https://github.com/alexpasmantier/television>
[4] <https://github.com/alexpasmantier/television/wiki/Cable-channels#%EF%B8%8F-guix>

Regards,

-- 
Gabriel Santos

Message #80 received at 77833 <at> debbugs.gnu.org (full text, mbox):

From: Noé Lopez <noe <at> xn--no-cja.eu>
To: Gabriel Santos <gabrielsantosdesouza <at> disroot.org>, 77833 <at> debbugs.gnu.org
Cc: Gabriel Santos <gabrielsantosdesouza <at> disroot.org>,
 Arun Isaac <arunisaac <at> systemreboot.net>,
 Ludovic Courtès <ludo <at> gnu.org>,
 Simon Tournier <zimon.toutoune <at> gmail.com>
Subject: Re: Xapian cache/search proof of concept
Date: Mon, 28 Apr 2025 21:38:23 +0200

[Message part 1 (text/plain, inline)]

Gabriel Santos <gabrielsantosdesouza <at> disroot.org> writes:

> Greetings,
>
> I subscribed to the channel[1] and installed the package[2], but
> couldn't get it working on my system. I then tried to install guile
> (which I didn't do because I already had it in my profile from some
> other package), but that was to no avail:
>
>>$ guix package -i guix-xsearch guile
>>The following packages will be installed:
>>   guile        3.0.9
>>   guix-xsearch 1.1
>>$ guix xsearch
>>guix: xsearch: command not found
>>hint: Did you mean `search'?
>>
>>Try `guix --help' for more information.

Did you try loading the package profile?

For me it works only after loading
".guix-profile/etc/profile". Sometimes it just feels like environment
variables never set themselves when you want them. I made sure that the
guix-xsearch package has the native-search-path! What else can I
possibly do?

>
> I then tried the "make env" shell from the channel and that worked:
>
>>$ make env
>>guix shell -m env/manifest.scm -- || exit 0	#ignore the shell’s return value
>>$ guix xsearch guix-search
>>name: guix-xsearch
>>[...]
>
> Additionally, trying to install it as package by adding the channel to the load
> path also doesn't work:
>
>>$ guix package -i guix-xsearch guile -L src/channel
>>The following packages will be installed:
>>   guix-xsearch 1.1
>>$ guix xsearch
>>guix: xsearch: command not found
>>hint: Did you mean `search'?
>>
>>Try `guix --help' for more information.
>
> Because of this, I couldn't do the test with television[3-4] that I wanted.
>

Right, it seems guix package might not be setting the search paths, but
it should have worked when adding to your home config.

> I tried 10 different searches, and this seems to have "issues"—more like
> warnings, as there's no crashes—with longer results:
>
>>$ make env
>>guix shell -m env/manifest.scm -- || exit 0	#ignore the shell’s return value
>>$$ guix xsearch kde
>>name: kde-games
>>[...]
>
> I then scroll to the third result, which in my case is kde-cli-tools, by
> pressing "PageDown" + "Up Arrow" to keep the "name" line at the top:
>
>>name: kde-cli-tools
>>[...]
>
> And press "q", to quit (I think that search uses the default pager,
> which in my case is still less):
>
>>Backtrace:
>>           9 (primitive-load "/home/gabriel/.config/guix/current/bin…")
>>In guix/ui.scm:
>>   2352:7  8 (run-guix . _)
>>  2315:10  7 (run-guix-command _ . _)
>>  1807:16  6 (call-with-paginated-output-port _ #:less-options _)
>>  1850:11  5 (_ #<output: #{write pipe}# 20>)
>>   1655:2  4 (package->recutils _ #<output: #{write pipe}# 20> _ # _ …)
>>In ice-9/format.scm:
>>  1548:18  3 (format #<output: #{write pipe}# 20> "~a~%" "descriptio…")
>>   216:11  2 (format:format-work "~a~%" ("description: `trayer' is …"))
>>     65:8  1 (format:out-obj-padded _ _ _ _)
>>In unknown file:
>>           0 (display "description: `trayer' is small program desig…" …)
>>
>>ERROR: In procedure display:
>>In procedure fport_write: Broken pipe
>
> This is only a issue if you stop at the middle of a search result. It
> doesn't happen when you press "END" for examplle to get to the very last
> one.
>

Good catch, I will fix this.

> With those issues aside, it seems realy good, the search seems to be almost
> instant.
>

Nice! I like your idea of trying it with television, although I wonder
what you plan to do?

The guix package command is not covered by guix-xsearch so you will have
to adapt the commands.  That said, you can probably make a script to go
through the database and list packages (could be slow or fast I don’t
know) and use guix xsearch title:<package> to find candidates.

Good luck!
Noé

> [1] <https://codeberg.org/gs-101/.files/commit/d8405ce168b67323e0de03e51c111ec75b127c5d>
> [2] <https://codeberg.org/gs-101/.files/commit/9d8a9d81a5aa5b5304997e714537df771d7725df>
> [3] <https://github.com/alexpasmantier/television>
> [4] <https://github.com/alexpasmantier/television/wiki/Cable-channels#%EF%B8%8F-guix>
>
> Regards,
>
> -- 
> Gabriel Santos

[signature.asc (application/pgp-signature, inline)]

Message #83 received at 77833 <at> debbugs.gnu.org (full text, mbox):

From: gabrielsantosdesouza <gabrielsantosdesouza <at> disroot.org>
To: Noé Lopez <noe <at> xn--no-cja.eu>
Cc: Arun Isaac <arunisaac <at> systemreboot.net>,
 Simon Tournier <zimon.toutoune <at> gmail.com>,
 Ludovic Courtès <ludo <at> gnu.org>, 77833 <at> debbugs.gnu.org
Subject: Re: Xapian cache/search proof of concept
Date: Mon, 28 Apr 2025 20:53:10 +0000

On 2025-04-28 19:38, Noé Lopez wrote:

> Did you try loading the package profile?

Oh, I didn't, because it's often not necessary for other packages, but I 
guess it
would make sense for it to not work with an extension to the package 
manager's
command.

I'll try messing around with the profiles and paths later to see if I 
can get it
work.

> Nice! I like your idea of trying it with television, although I wonder
> what you plan to do?

I plan to use it and maybe suggest it on the wiki as a faster search 
alternative.
I often use the Guix (television) "channel" configuration I linked to 
complete
packages when spawning shells for example—the calls to "cut" and "tr" 
are there
to grab just the package's name as a result—so it would be nice to have 
that
optimized.

> Good luck!

Thanks!

Regards,

-- 
Gabriel Santos

Message #86 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Arun Isaac <arunisaac <at> systemreboot.net>
To: Noé Lopez <noe <at> xn--no-cja.eu>, Gabriel Santos
 <gabrielsantosdesouza <at> disroot.org>, 77833 <at> debbugs.gnu.org, Gabriel Santos
 via Guix-patches via <guix-patches <at> gnu.org>
Cc: Ludovic Courtès <ludo <at> gnu.org>,
 Simon Tournier <zimon.toutoune <at> gmail.com>
Subject: Re: [bug#77833] Xapian cache/search proof of concept
Date: Tue, 29 Apr 2025 00:09:55 +0100

Hi Noé,

Works beautifully, thanks for hacking on this!

How does the multiple profile support work? Do I have to build the index
separately for each profile? What happens when I switch back and forth
between profiles—do I need to rebuild the index each time I switch? And,
where is the index stored?

Cheers!
Arun

Message #92 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Noé Lopez <noe <at> xn--no-cja.eu>
To: Arun Isaac <arunisaac <at> systemreboot.net>, Gabriel Santos
 <gabrielsantosdesouza <at> disroot.org>, 77833 <at> debbugs.gnu.org, Gabriel Santos
 via Guix-patches via <guix-patches <at> gnu.org>
Cc: Ludovic Courtès <ludo <at> gnu.org>,
 Simon Tournier <zimon.toutoune <at> gmail.com>
Subject: Re: [bug#77833] Xapian cache/search proof of concept
Date: Tue, 29 Apr 2025 21:27:49 +0200

[Message part 1 (text/plain, inline)]

Arun Isaac <arunisaac <at> systemreboot.net> writes:

> Hi Noé,
>
> Works beautifully, thanks for hacking on this!
>
> How does the multiple profile support work? Do I have to build the index
> separately for each profile? What happens when I switch back and forth
> between profiles—do I need to rebuild the index each time I switch? And,
> where is the index stored?
>
> Cheers!
> Arun

We take the profile at /var/guix/profiles/per-user/<user>/current-guix/
and build an index for it. Yes. No. In
~/.cache/guix-xsearch/<profile-hash>.

What I originaly wanted to do was a derivation that computes the index
based on the current guix, but I couldn’t quite get it to work. It would
have been best since it would be in the store and benefit from guix gc.

Now that I think about it, this is a bad solution. It might not work
with guix time-machine.

I would like to know what you mean by switching profile, is that guix
pull generations or guix shell guix or something else?

All the best,
Noé

[signature.asc (application/pgp-signature, inline)]

Message #98 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Arun Isaac <arunisaac <at> systemreboot.net>
To: Noé Lopez <noe <at> xn--no-cja.eu>, Gabriel Santos
 <gabrielsantosdesouza <at> disroot.org>, 77833 <at> debbugs.gnu.org, Gabriel Santos
 via Guix-patches via <guix-patches <at> gnu.org>
Cc: Ludovic Courtès <ludo <at> gnu.org>,
 Simon Tournier <zimon.toutoune <at> gmail.com>
Subject: Re: [bug#77833] Xapian cache/search proof of concept
Date: Thu, 01 May 2025 21:46:05 +0100

Hi Noé,

> We take the profile at /var/guix/profiles/per-user/<user>/current-guix/
> and build an index for it. Yes. No. In
> ~/.cache/guix-xsearch/<profile-hash>.

Ok, sounds good.

> I would like to know what you mean by switching profile, is that guix
> pull generations or guix shell guix or something else?

I have additional guix profiles, for example ~/.guix-extra-profiles/foo
that I activate with `source ~/.guix-extra-profiles/foo/etc/profile'. I
sometimes also have this happen automatically using direnv.

Regards,
Arun

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.