Received: (at 39258) by debbugs.gnu.org; 4 Apr 2025 19:17:19 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Fri Apr 04 15:17:19 2025 Received: from localhost ([127.0.0.1]:39972 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1u0mXX-0007Cf-Db for submit <at> debbugs.gnu.org; Fri, 04 Apr 2025 15:17:19 -0400 Received: from mugam.systemreboot.net ([139.59.75.54]:34112) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from <arunisaac@HIDDEN>) id 1u0mXT-0007CL-GJ for 39258 <at> debbugs.gnu.org; Fri, 04 Apr 2025 15:17:17 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=systemreboot.net; s=default; t=1743794229; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to; bh=B8/4ahSyLEQyPfs7TxmiQ98OYSLX7pK+vLlVOHrOZcc=; b=ltLOQjPlahThwzvdvgJyxEPL5CFwqKlrfRMWl4yukXelVFQeGQxOHq4dGIUw5z7TvDIDVI Eukv1aEd8NQf7d8v/Q8WPrj6tePOQABZIrysXAykvTOU5uRo2Ml2PWCIQryzgJTjqGRpLS P/ii00yr3o6TIE4duwCPnbZolLJQpLJQkKrOXnUFjqcPB21sBCtoDXsXi2GN8Jzf8aTutC pRdCCx7IC/M33p3jmPG4plJ8+90ohHEHOVTWgGmSA2x0yaWZECaK9KwBDamSUgnXIXf43K uNULQL4XefnfXo4DI4OeczKO0N+AR/cibmI2dL7tl9Tl26kxTJr28jLLrB9Mvg== Received: from localhost (<unknown> [192.168.2.1]) by mugam.systemreboot.net (OpenSMTPD) with ESMTPSA id d077a1d9 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Fri, 4 Apr 2025 19:17:08 +0000 (UTC) From: Arun Isaac <arunisaac@HIDDEN> To: 39258 <at> debbugs.gnu.org Subject: Re: Faster guix search using an sqlite cache In-Reply-To: <CAJ3okZ0Vv9mFbeRV9+b4T-M7ekBGid_VyWEzbMKCMbC-t3HdaA@HIDDEN> Date: Fri, 04 Apr 2025 20:17:00 +0100 Message-ID: <87bjtbka8j.fsf@HIDDEN> MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 39258 Cc: Arun Isaac <arunisaac@HIDDEN>, Jonathan Brielmaier <jonathan.brielmaier@HIDDEN>, zimoun <zimon.toutoune@HIDDEN>, Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>, Leo Famulari <leo@HIDDEN>, Pierre Neidhardt <mail@HIDDEN> X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) Hi all, This issue has gone quite stale, and it's not clear if we want to have a faster guix search based on xapian or on a sqlite cache. I still think a xapian search is a good idea, but perhaps we don't want Guix to depend on xapian. So, perhaps, the right thing to do is to have a "guix xsearch" extension that implements this and is separate from Guix. guile-xapian is quite mature now, and this should be a relatively easy project. Perhaps it could be something for a GSoC student to work on. I am happy to help mentor such a project if others see value in that. May I close this issue now? We can open new issues, if there is something specific to discuss. Thanks, Arun
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.
Received: (at 39258) by debbugs.gnu.org; 11 Oct 2021 08:10:01 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Oct 11 04:10:01 2021
Received: from localhost ([127.0.0.1]:56348 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1mZqNl-0005LB-5R
for submit <at> debbugs.gnu.org; Mon, 11 Oct 2021 04:10:01 -0400
Received: from mail-qv1-f49.google.com ([209.85.219.49]:44598)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <zimon.toutoune@HIDDEN>)
id 1mZqNj-0005Ko-4v; Mon, 11 Oct 2021 04:09:59 -0400
Received: by mail-qv1-f49.google.com with SMTP id v2so1035025qve.11;
Mon, 11 Oct 2021 01:09:59 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112;
h=mime-version:references:in-reply-to:from:date:message-id:subject:to
:cc:content-transfer-encoding;
bh=tF0jofKwjLZe58b2PzbjtdbloibShajG9H42X+OcoVI=;
b=JLdoJF5p3sKj7xLnIEpZ7awq697EwYlVuaItltteoani0UGi/wZfM7lBeoPX0gtmuL
SwXGnT7RQDoWDYMuDBDsCq4n4iWUDzmwMiI45jVRdo2EgQp6k7QBaFdRvx6/fjq1EK3n
OoQvl0bMFjiUMeTAVxRdf4nudwMihP1CfIYkr/ROEjAcCodo/nKv2uaAZO6jx/YWFA8R
pe3F/XxXCveg9dOrU0a/tXFGCHQobzABPHBeN15+ztzJigmjVm+8WEAUXIZtjdX/nwL3
1fBtCoA/uMgOPGpW66qWG//U6aMadB7VNY4UaL1Ed5KQ5jDUs8coaApVqaOdJ2F2gl61
d4Rw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20210112;
h=x-gm-message-state:mime-version:references:in-reply-to:from:date
:message-id:subject:to:cc:content-transfer-encoding;
bh=tF0jofKwjLZe58b2PzbjtdbloibShajG9H42X+OcoVI=;
b=eJ0O1xO2LXlsSLbqNQkonKRHQ8cQEGbVvuy0IKijkbzy3gJ+POK0a5mQu6UMrhYdwr
ojRl85z+aSUhuKtdsRulBUNYSYKrHTOyJ30uzdyXVcxeblKY15pPfaEN8kejHg9i8Ush
VcXNV8mrdYCNpq8bt+2Ggwm+c63S6y1c54kd6SS0iCNsXaPe4SlORQOZouv3esUONpFw
Edlw6jXyE1QVnlHX1AFZRjAMBIg+xYSL0JynWSCueJOG0/2pSlLZjB2UKdRplPhx+7ar
nocZZs5x4XGTFnlUyHPBsxF+R+2uXg1U4aEHnNJPTUgfyHfpluAISkd66+iGidPGOgBh
Zc3g==
X-Gm-Message-State: AOAM530fM3MfZPlzNRuazfYIJ5SNtM5SxKq7o13iMd54T9LRxyTPFuPd
3JiHasKM27kUZFJKXU9daoEP0tnI6YSJKVWJYxk=
X-Google-Smtp-Source: ABdhPJzUqTwpNhmhHATMzSqH9yWvbPMnfHdw0LKR5xrWUm+XA38JkwlZW/ESBhcqJjYOlQJwcy3YX8DaJAL6CZ/KH/0=
X-Received: by 2002:ad4:5f48:: with SMTP id p8mr21781530qvg.39.1633939793687;
Mon, 11 Oct 2021 01:09:53 -0700 (PDT)
MIME-Version: 1.0
References: <8900fa8c8eef7f72fc97adc2408be26c88de7803.camel@HIDDEN>
<87tuj0xaja.fsf@HIDDEN>
<0ec7f0270fcccec730808f9210f074cd5339961f.camel@HIDDEN>
<87ee9xerac.fsf_-_@HIDDEN> <87r1dhj2bk.fsf_-_@HIDDEN>
<3563a99b37c8ef3b226b7fef25a5afa273c08b6e.camel@HIDDEN>
<87czouj8m7.fsf@HIDDEN> <87h7dwviu7.fsf_-_@HIDDEN>
<87sfxc55gi.fsf@HIDDEN>
In-Reply-To: <87sfxc55gi.fsf@HIDDEN>
From: zimoun <zimon.toutoune@HIDDEN>
Date: Mon, 11 Oct 2021 10:09:42 +0200
Message-ID: <CAJ3okZ0Vv9mFbeRV9+b4T-M7ekBGid_VyWEzbMKCMbC-t3HdaA@HIDDEN>
Subject: Re: bug#50384: [PATCH] Optimise search-patch (reducing I/O)
To: =?UTF-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>, 39258 <at> debbugs.gnu.org
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 39258
Cc: Maxime Devos <maximedevos@HIDDEN>, 50384 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)
Hi Ludo,
On Fri, 8 Oct 2021 at 09:41, Ludovic Court=C3=A8s <ludo@HIDDEN> wrote:
> Yes, that=E2=80=99s what I thought: <proto-package> could be used in
> =E2=80=98fold-available-packages=E2=80=99 & co., so that would kind of un=
ify that.
Therefore, do we agree that the cache /lib/guix/package.cache should
be extended? Because the v3 adding an extra cache [1] shows
improvements of "guix search" and v4 moving the extra fields to the
regular cache [2] shows the same improvements for searching and in
addition improves this cache generation. The issue with v4 is the
field 'license'. And v6 is an attempt to by-pass that (keep the
regular cache and only read on-the-fly the required information); but
because of this, I guess:
(module-ref (resolve-interface module) symbol)
it is slower!
1: <http://issues.guix.gnu.org/39258#68>
2: <http://issues.guix.gnu.org/39258#89>
3: <http://issues.guix.gnu.org/39258#119>
All the bests,
simon
PS: For cross-reference, discussion originally happened on
<https://issues.guix.gnu.org/50384>.
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 20 Aug 2021 15:43:04 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Fri Aug 20 11:43:04 2021 Received: from localhost ([127.0.0.1]:35388 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1mH6ff-0005g4-Ju for submit <at> debbugs.gnu.org; Fri, 20 Aug 2021 11:43:04 -0400 Received: from mail-qk1-f176.google.com ([209.85.222.176]:36836) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <zimon.toutoune@HIDDEN>) id 1mH6fd-0005fY-ST for 39258 <at> debbugs.gnu.org; Fri, 20 Aug 2021 11:43:02 -0400 Received: by mail-qk1-f176.google.com with SMTP id e14so11284814qkg.3 for <39258 <at> debbugs.gnu.org>; Fri, 20 Aug 2021 08:43:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=mf25MnZ6G5H5etVNqqabJ6qILKwrSctYOO8R1azgZ4Y=; b=Wtfn3lSG3yR7228TBMRujvAf2MUDWzzQxRbidXZfWZxKp5Bs9Maf7D4vd/3q6ysUMb t/lfh6n8WN6MZmLQfgERZmCHMqlGY1gxcWJQxrwgUxzEkgXoHUehhM2/qX3HU/5ynjhQ n0+b+9DRniJxIUigsxxpgv14K2gF1eWphxuZ5wPj9oF/b3Fb2Wi6m982ALDxgAAhE9CP peJI4gXGAm1gq6wGQj7yUY3QybmR5EEMTnKwQLPmZ6TjqiNJkyo10kEXgGomRENgS3WO 5sxBmnAgRjNFBdwef99DVkJUmdbhi2Ue9h3rOUA1p10S0zMMk4I0K54El/8UpbcNNjvy /Vyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=mf25MnZ6G5H5etVNqqabJ6qILKwrSctYOO8R1azgZ4Y=; b=hYcGbxI1gRp3LiFjaffawXemZnLRx2Sd9XubDgNOeTuQN63o+Baoe9O7VMpk3R6HoU SzaCb0lKhFra9GUbuTZ5dG6ZMXsR1PHJNGz1KzYpSqDXHxDbshoybsw6PHGfwGMOZOJe o222NYY2zOr5uMfN8+HiO+zy6wWWIs1RXQG0BmFZTcumIeTdE7KwAunRHbPF6SYgbP+n KVEnHzH1ClvYWgyFm3wsH0vTUiqeaaSx/LApC6DuMe65MaCVECFXS5vmVq5FNX9I5DOU NthHw/1QDx9rSfXlDnbgSlPy9F4VbxuWPeZqVb72ZKgB7dhyFPnsVuvfIfkz84/uv+uC Zh0A== X-Gm-Message-State: AOAM5330942DtUlJMfNQKmRB+wKYCXKJ8CeP/4aEKjrybsghosdWDzyU nmYeXBzUbkszJGIEHjpTBW9MsH0M0CqrrJW52qs= X-Google-Smtp-Source: ABdhPJzRJuwkwfXdFNLC3hrfi3f3BuWOoVNN/Fe3CUhTS6GTkW/o+QDwabxHoI1ny3+WqhEXoJCOxjh+VFojbFoqu6w= X-Received: by 2002:a05:620a:56e:: with SMTP id p14mr3078103qkp.126.1629474176118; Fri, 20 Aug 2021 08:42:56 -0700 (PDT) MIME-Version: 1.0 References: <20210715073328.212123-1-zimon.toutoune@HIDDEN> <875yx1ave7.fsf@HIDDEN> In-Reply-To: <875yx1ave7.fsf@HIDDEN> From: zimoun <zimon.toutoune@HIDDEN> Date: Fri, 20 Aug 2021 17:42:44 +0200 Message-ID: <CAJ3okZ3ijny997BMMZ2yqwHTQJG7C2YZo+QMHOE+z0xhspsSug@HIDDEN> Subject: Re: [PATCH v6 0/2] DRAFT "guix search" performances To: =?UTF-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 39258 Cc: Arun Isaac <arunisaac@HIDDEN>, 39258 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) Hi Ludo, On Fri, 23 Jul 2021 at 17:43, Ludovic Court=C3=A8s <ludo@HIDDEN> wrote: > > From my understanding, the issue that 'package-relevance' accepts a 'pa= ckage' > > (and then all the chain until displaying) and 'fold-avaibale-packages' = does > > not return a package. Well, I do not know; especially where to put som= ething > > similiar to 'read-package-from'. > > Yeah that=E2=80=99s annoying. Perhaps we need <proto-package> or > <package-metadata>. With some trickery we could have record type > inheritance or something, maybe. Dunno. > > It would be good if we could arrange so that =E2=80=98fold-available-pack= ages=E2=80=99 > doesn=E2=80=99t allocate anything though. It does not allocate, the allocation is done by 'find-packages-by-descripti= on'. Well, I think v4/v3 [0] is the direction to follow. Therefore, I would revisit this version and try to address two of Ludo's comments [1] and the other ones in v3. BTW, I have not investigated from where the slowness comes: - allocation - garbage collection - '(module-ref (resolve-interface module) symbol)' because I have been a bit disappointed by the performance of this v6. 0: <http://issues.guix.gnu.org/39258#89> 1: <http://issues.guix.gnu.org/39258#93> > > Let compare only for cold cache and time this cache building (Guix 7db8= fd6): > > > > sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' > > time guix build --check $(guix gc --derivers $(readlink -f ~/.config/= guix/current/lib/guix/package.cache)) > > > > real 0m28,848s > > user 0m1,481s > > sys 0m0,252s > > > > sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' > > time guix build --check $(guix gc --derivers $(readlink -f /tmp/new/l= ib/guix/package.cache)) > > > > real 0m40,279s > > user 0m1,582s > > sys 0m0,232s > > > > It seems longer but compared to the time of "guix pull" completion, it = seems > > acceptable. > > Both the initial timing and the target are waaay too much. :-/ Yes, but that's another story. :-) We cannot fix all in the same time, IMH= O. Here the point is to speedup "guix search" and to accept to pay a little more at "guix pull" time; then we could optimize the cache generation. Considering the overall time of "guix pull", this extra time appears to me acceptable---if "guix search" is faster. :-) > On my i7 laptop I have: > > --8<---------------cut here---------------start------------->8--- > $ time ./pre-inst-env guile -c '(use-modules (gnu packages)) (generate-p= ackage-cache "/tmp/t.cache")' > > real 0m20.738s > user 0m44.413s > sys 0m0.341s > --8<---------------cut here---------------end--------------->8--- > > It=E2=80=99s CPU-bound; we should probably start by optimizing that. > > In Guile 3.0.7 there was a change that improved this noticeably: > > https://git.savannah.gnu.org/cgit/guile.git/commit/?id=3D05614f792bfabb= c33798863edd0bb67c744e9299 > > We should prolly look for similar optimization opportunities in the > assembler=E2=80=A6 Yes, but this kind of optimization will not provide a faster "guix search" but a faster "guix pull". --8<---------------cut here---------------start------------->8--- $ sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' $ time guile -c '(use-modules (gnu packages)) (generate-package-cache "/tmp/t1.cache")' real 0m15,728s user 0m23,940s sys 0m0,826s $ sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' $ time guile -c '(load-compiled "/tmp/t1.cache/lib/guix/package.cache")' real 0m1,026s user 0m0,258s sys 0m0,051s $ sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' $ time ./pre-inst-env guile -c '(use-modules (gnu packages)) (generate-package-cache "/tmp/t2.cache")' real 0m35,570s user 3m12,951s sys 0m3,807s $ sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' $ time guile -c '(load-compiled "/tmp/t2.cache/lib/guix/package.cache")' real 0m1,055s user 0m0,283s sys 0m0,055s --8<---------------cut here---------------end--------------->8--- (And if we speak about performance, raw loading the cache takes ~1s but "guix package -A >/dev/null" takes ~8s. It is a big gap for parsing the CLI and sorting. Worse, "guix --version >/dev/null" takes ~2s on cold cache. We should probably start by reduce this overhead.) > > Let compare for some queries: > > > > sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' > > time guix search game | recsel -C -P name | wc -l > > 371 > > > > real 0m7,561s > > user 0m3,525s > > sys 0m0,391s > > I think you should run: > > time guix search game > /dev/null > > otherwise Bash=E2=80=99s built-in =E2=80=98time=E2=80=99 shows the wall-c= lock time of the whole > pipeline, and the processing time of =E2=80=98recsel=E2=80=99 is probably= not negligible > here. First, I am confused... If the formatter (recsel) is not negligible, then it should be dropped and replaced by an internal (fast) formatter. Well, I mean that as an end-user I am interested by the time of the whole pipeline. Second, on my machine the time is somehow negligible*. ;-) On cold cache, 10 runs using the pipe or using the redirection, keeping the max and the min for each: --8<---------------cut here---------------start------------->8--- real 0m9,598s user 0m3,961s sys 0m0,415s real 0m8,744s user 0m3,772s sys 0m0,431s --8<---------------cut here---------------end--------------->8--- --8<---------------cut here---------------start------------->8--- real 0m8,755s user 0m3,869s sys 0m0,540s real 0m8,390s user 0m3,767s sys 0m0,416s --8<---------------cut here---------------end--------------->8--- *negligible: better said, it does not give the bad impression. Even if it is roughly 5% of difference. Cheers, simon
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.
Received: (at 39258) by debbugs.gnu.org; 17 Aug 2021 19:21:03 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Aug 17 15:21:03 2021
Received: from localhost ([127.0.0.1]:54151 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1mG4dy-0008EH-Ix
for submit <at> debbugs.gnu.org; Tue, 17 Aug 2021 15:21:03 -0400
Received: from mail-wr1-f45.google.com ([209.85.221.45]:39881)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <zimon.toutoune@HIDDEN>) id 1mG4dt-0008DR-2T
for 39258 <at> debbugs.gnu.org; Tue, 17 Aug 2021 15:21:01 -0400
Received: by mail-wr1-f45.google.com with SMTP id q6so18697627wrv.6
for <39258 <at> debbugs.gnu.org>; Tue, 17 Aug 2021 12:20:57 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
h=from:to:cc:subject:in-reply-to:references:date:message-id
:mime-version:content-transfer-encoding;
bh=NIYxTF9U5NbY2OrvQvqDpRT0W1LQEUqBfPBTP80BwtY=;
b=isRdKSWUG/JP+sivjVLfzqFF4K7/ecfHVMCzZxtE9vw+VQTBC1WqZGTdtJfHodB+R+
4lET0764HBUht0X1c6hLqeg0qkyJvfkeGe/IZBoRfm612C/YLZwKJGHLPIoTdMNYLmtR
FlOrzVbmIQhHYJW4TWgMy97cn54YmZPPthxJ9HbmUtjEyNLHfSRdWqAw1qMNKPthwWWJ
P46I9nI9ilsXHQLwIkf/tLAcu9cdiOxPzWP4T7IZdKeBvXJdAuaePPJNJb+G8txlY/pR
tCcltqr7WdW9AceBKYCytoJOGk3g0BYO1wt6jUeHVjbppRNUuaIjJ4W6FJyS8pUlZUHU
srqA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20161025;
h=x-gm-message-state:from:to:cc:subject:in-reply-to:references:date
:message-id:mime-version:content-transfer-encoding;
bh=NIYxTF9U5NbY2OrvQvqDpRT0W1LQEUqBfPBTP80BwtY=;
b=MIs3aDhLLPXNOiWV6+DYZkXtV5V+uZWW1wQ9gM9fMm9TBugnsaZ2lLsYKl89TVOJmI
MKYo00MQZROoJrp7NmpgqVHg+lpgC8a0+88CzsCXHddi4wMRZinjRoJPSOgyLAmza1Xq
eNCI0b6fhtiDYXu6VNdxav8wXVwHPrJ5sL2gC8Bju9R53cSb6CGlMZFKFAxMLfT8eCHV
NK+ztxvu7raTa4IkUyd7eAfUfZICUX7G+hDGFsPhC14oQvbu7XIh36MUPl6A9rZhlh3F
ObHgxT/uBAHuc8H0skCGpotVCpurfOmf2toWEVeCEyqs+/42JrI0fre7vPVxteZBtlRx
QT4w==
X-Gm-Message-State: AOAM530H4HSeHNSKYleuoDCCknmT+ZQlnxk1cO8VwomypKQcKYV3q6L5
l5vb9oZ2kv54s2PNCyCgaJoxEq+6EFw=
X-Google-Smtp-Source: ABdhPJwH+uj67RxrQ8nYPTfm4uCYla5jDpIPOKPyG1BrAyzsw27Ff0QBezti+WXxwOyBwRCXaWchdQ==
X-Received: by 2002:adf:f288:: with SMTP id k8mr6105361wro.350.1629228051220;
Tue, 17 Aug 2021 12:20:51 -0700 (PDT)
Received: from lili (roam-nat-fw-prg-194-254-61-44.net.univ-paris-diderot.fr.
[194.254.61.44])
by smtp.gmail.com with ESMTPSA id e11sm3336106wrm.80.2021.08.17.12.20.50
(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
Tue, 17 Aug 2021 12:20:50 -0700 (PDT)
From: zimoun <zimon.toutoune@HIDDEN>
To: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>, Arun Isaac
<arunisaac@HIDDEN>
Subject: Re: [PATCH v6 1/2] DRAFT packages: Add fields to packages cache.
In-Reply-To: <87czr9avzc.fsf@HIDDEN>
References: <20210715073328.212123-1-zimon.toutoune@HIDDEN>
<20210715073328.212123-2-zimon.toutoune@HIDDEN>
<877dhpmjda.fsf@HIDDEN> <87czr9avzc.fsf@HIDDEN>
Date: Tue, 17 Aug 2021 16:03:17 +0200
Message-ID: <86pmucb23e.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: 1.1 (+)
X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org",
has NOT identified this incoming email as spam. The original
message has been attached to this so you can view it or label
similar future email. If you have any questions, see
the administrator of that system for details.
Content preview: Hi Arun and Ludo, Thanks for the review. On Sat, 17 Jul 2021
at 14:01, Arun Isaac wrote:
Content analysis details: (1.1 points, 10.0 required)
pts rule name description
---- ---------------------- --------------------------------------------------
-0.0 SPF_PASS SPF: sender matches SPF record
0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail
provider (zimon.toutoune[at]gmail.com)
0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record
1.1 DATE_IN_PAST_03_06 Date: is 3 to 6 hours before Received: date
-0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/,
no trust [209.85.221.45 listed in list.dnswl.org]
-0.0 RCVD_IN_MSPIKE_H2 RBL: Average reputation (+2)
[209.85.221.45 listed in wl.mailspike.net]
X-Debbugs-Envelope-To: 39258
Cc: 39258 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: 0.1 (/)
Hi Arun and Ludo,
Thanks for the review.
On Sat, 17 Jul 2021 at 14:01, Arun Isaac <arunisaac@HIDDEN> wrote:
> I understand that one of the things you are trying to do is to have a
> common interface for the cache and no-cache cases. To achieve this, I
> think fold-available-packages and fold-packages should have the same
> function signature. They should both pass a <package> object to
> PROC. Currently, fold-packages is passing a <package> object whereas
> fold-available-packages is passing the fields of the <package> object as
> individual parameters. If fold-packages and fold-available-packages have
> the same function signature, then the changes in your [PATCH v6 2/2]
> would be way simpler.
I agree. Previously [1], I created =E2=80=99fold-packages*=E2=80=99 which =
was a cached
=E2=80=99fold-packages=E2=80=99 and Ludo answered [2]:
Did you see =E2=80=98fold-available-packages=E2=80=99? It seems yo=
u could
extend it instead of introducing =E2=80=98fold-packages*=E2=80=99, =
no?
therefore, it is somehow another attempt on the other side. :-)
1: <http://issues.guix.gnu.org/39258#91>
2: <http://issues.guix.gnu.org/39258#93>
> Also, why do we need two separate functions---fold-available-packages
> and fold-packages? Can't fold-available-packages do everything
> fold-packages can and thus totally replace it?
To be honest, I have been lazy because unifying
=E2=80=99fold-available-packages=E2=80=99 and =E2=80=99fold-packages=E2=80=
=99 means to change the
signature and so a bit of refactoring. And as Ludo explained,
=E2=80=99fold-available-packages=E2=80=99 has to be as light as possible be=
cause it is
used by Emacs-Guix and maybe Nyxt for completion. :-)
>> * gnu/packages.scm (generate-package-cache)[expand-cache]: Add synopsis =
and
>> description.
>> (load-package-cache, find-packages-by-names, find-packages-locations): A=
dapt
>> accordingly.
>
> A couple of typos here:
>
> find-packages-by-names -> find-packages-by-name
> find-packages-locations -> find-package-locations
Thanks for the spot.
On Fri, 23 Jul 2021 at 17:30, Ludovic Court=C3=A8s <ludo@HIDDEN> wrote:
> One benchmark is =E2=80=9Cguix package -A > /dev/null=E2=80=9D. This sho=
uld take
> ideally 0.5s at most because that=E2=80=99s what=E2=80=99s used by shell =
completion (the
> first time); currently it takes 0.82s on my laptop, though.
On cold cache, on my laptop:
--8<---------------cut here---------------start------------->8---
$ time guix package -A > /dev/null
real 0m1.717s
user 0m2.526s
sys 0m0.083s
--8<---------------cut here---------------end--------------->8---
and on my (slow) desktop:
real 0m6.196s
user 0m2.008s
sys 0m0.093s
Warn cache:
laptop desktop
real 0m1.425s 0m1.217s
user 0m2.505s 0m1.901s
sys 0m0.033s 0m0.051s
Well, another story I guess. :-)
Cheers,
simon
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 23 Jul 2021 15:43:24 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Fri Jul 23 11:43:24 2021 Received: from localhost ([127.0.0.1]:44354 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1m6xKd-0004fo-Ma for submit <at> debbugs.gnu.org; Fri, 23 Jul 2021 11:43:24 -0400 Received: from eggs.gnu.org ([209.51.188.92]:42858) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <ludo@HIDDEN>) id 1m6xKb-0004fb-TH for 39258 <at> debbugs.gnu.org; Fri, 23 Jul 2021 11:43:22 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:59328) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from <ludo@HIDDEN>) id 1m6xKV-0004wY-95; Fri, 23 Jul 2021 11:43:15 -0400 Received: from [2001:660:6102:320:e120:2c8f:8909:cdfe] (port=42914 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from <ludo@HIDDEN>) id 1m6xKV-0003pT-0s; Fri, 23 Jul 2021 11:43:15 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN> To: zimoun <zimon.toutoune@HIDDEN> Subject: Re: [PATCH v6 0/2] DRAFT "guix search" performances References: <20210715073328.212123-1-zimon.toutoune@HIDDEN> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 5 Thermidor an 229 de la =?utf-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Fri, 23 Jul 2021 17:43:12 +0200 In-Reply-To: <20210715073328.212123-1-zimon.toutoune@HIDDEN> (zimoun's message of "Thu, 15 Jul 2021 09:33:26 +0200") Message-ID: <875yx1ave7.fsf@HIDDEN> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 39258 Cc: arunisaac@HIDDEN, 39258 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -3.3 (---) Hi! zimoun <zimon.toutoune@HIDDEN> skribis: > This is an attempt to improve the performance of "guix search". It is st= ill > half baked but it allows to discuss further the idea about expanding the > current '/lib/guix/package.cache' and avoids to forget an IRL discussion.= ;-) Thanks for resuming this discussion. :-) > From my understanding, the issue that 'package-relevance' accepts a 'pack= age' > (and then all the chain until displaying) and 'fold-avaibale-packages' do= es > not return a package. Well, I do not know; especially where to put somet= hing > similiar to 'read-package-from'. Yeah that=E2=80=99s annoying. Perhaps we need <proto-package> or <package-metadata>. With some trickery we could have record type inheritance or something, maybe. Dunno. It would be good if we could arrange so that =E2=80=98fold-available-packag= es=E2=80=99 doesn=E2=80=99t allocate anything though. > Let compare only for cold cache and time this cache building (Guix 7db8fd= 6): > > sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' > time guix build --check $(guix gc --derivers $(readlink -f ~/.config/gu= ix/current/lib/guix/package.cache)) > > real 0m28,848s > user 0m1,481s > sys 0m0,252s > > sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' > time guix build --check $(guix gc --derivers $(readlink -f /tmp/new/lib= /guix/package.cache)) > > real 0m40,279s > user 0m1,582s > sys 0m0,232s > > It seems longer but compared to the time of "guix pull" completion, it se= ems > acceptable. Both the initial timing and the target are waaay too much. :-/ On my i7 laptop I have: --8<---------------cut here---------------start------------->8--- $ time ./pre-inst-env guile -c '(use-modules (gnu packages)) (generate-pac= kage-cache "/tmp/t.cache")' real 0m20.738s user 0m44.413s sys 0m0.341s --8<---------------cut here---------------end--------------->8--- It=E2=80=99s CPU-bound; we should probably start by optimizing that. In Guile 3.0.7 there was a change that improved this noticeably: https://git.savannah.gnu.org/cgit/guile.git/commit/?id=3D05614f792bfabbc3= 3798863edd0bb67c744e9299 We should prolly look for similar optimization opportunities in the assembler=E2=80=A6 > Let compare for some queries: > > sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' > time guix search game | recsel -C -P name | wc -l > 371 > > real 0m7,561s > user 0m3,525s > sys 0m0,391s I think you should run: time guix search game > /dev/null otherwise Bash=E2=80=99s built-in =E2=80=98time=E2=80=99 shows the wall-clo= ck time of the whole pipeline, and the processing time of =E2=80=98recsel=E2=80=99 is probably n= ot negligible here. [...] > Last, two minor remarks about previous comments. > > 1. Ludo commented: > > > Therefore the cache '/lib/guix/package.cache' contains more > > information. > > This breaks the binary interface, so we=E2=80=99ll have to analyz= e the impact of > such a change and devise a strategy. > > <http://issues.guix.gnu.org/39258#93> > > and after some checking, this should be fine, IIUC. The '--news' is ok > because of '#:allow-other-keys'. And other parts are modified accordingl= y. > Guix revision N creates a cache that Guix revision N+1 will read but it s= hould > not be an issue; see 'inferior-available-packages'. > > 2. And Ludo wrote: > > I realize the other cache also has that problem, but it would be = nice to add a > version tag to the cache. Basically emit something like: > > (package-metadata-cache (version 0) VECTOR =E2=80=A6) > > instead of just: > > (VECTOR =E2=80=A6) > > <http://issues.guix.gnu.org/39258#93> > > which is, after discussions, not necessary. Versioning does not make sen= se > here because the cache is read by the Guix which generates it. Therefore, > specify a version is extraneous here. I confirm! :-) Thanks, Ludo=E2=80=99.
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 23 Jul 2021 15:30:48 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Fri Jul 23 11:30:48 2021 Received: from localhost ([127.0.0.1]:44333 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1m6x8S-0004NO-5R for submit <at> debbugs.gnu.org; Fri, 23 Jul 2021 11:30:48 -0400 Received: from eggs.gnu.org ([209.51.188.92]:39468) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <ludo@HIDDEN>) id 1m6x8P-0004N9-5j for 39258 <at> debbugs.gnu.org; Fri, 23 Jul 2021 11:30:46 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:58728) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from <ludo@HIDDEN>) id 1m6x8G-0004lU-TG; Fri, 23 Jul 2021 11:30:37 -0400 Received: from [2001:660:6102:320:e120:2c8f:8909:cdfe] (port=42912 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from <ludo@HIDDEN>) id 1m6x8E-0003kh-Ln; Fri, 23 Jul 2021 11:30:36 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN> To: Arun Isaac <arunisaac@HIDDEN> Subject: Re: [PATCH v6 1/2] DRAFT packages: Add fields to packages cache. References: <20210715073328.212123-1-zimon.toutoune@HIDDEN> <20210715073328.212123-2-zimon.toutoune@HIDDEN> <877dhpmjda.fsf@HIDDEN> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 5 Thermidor an 229 de la =?utf-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Fri, 23 Jul 2021 17:30:31 +0200 In-Reply-To: <877dhpmjda.fsf@HIDDEN> (Arun Isaac's message of "Sat, 17 Jul 2021 14:01:45 +0530") Message-ID: <87czr9avzc.fsf@HIDDEN> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 39258 Cc: 39258 <at> debbugs.gnu.org, zimoun <zimon.toutoune@HIDDEN> X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -3.3 (---) Hi Arun! Arun Isaac <arunisaac@HIDDEN> skribis: > Also, why do we need two separate functions---fold-available-packages > and fold-packages? Can't fold-available-packages do everything > fold-packages can and thus totally replace it? The initial goal was for =E2=80=98fold-available-packages=E2=80=99 to be li= ghtweight. Currently, it doesn=E2=80=99t allocate anything; instead, it passes info as keyword parameters, which the callee is free to ignore. That=E2=80=99s why these two procedures have different signatures. One benchmark is =E2=80=9Cguix package -A > /dev/null=E2=80=9D. This shoul= d take ideally 0.5s at most because that=E2=80=99s what=E2=80=99s used by shell co= mpletion (the first time); currently it takes 0.82s on my laptop, though. Thanks, Ludo=E2=80=99.
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 17 Jul 2021 08:32:26 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Sat Jul 17 04:32:26 2021 Received: from localhost ([127.0.0.1]:52833 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1m4fkI-000603-0P for submit <at> debbugs.gnu.org; Sat, 17 Jul 2021 04:32:26 -0400 Received: from mugam.systemreboot.net ([139.59.75.54]:37690) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <arunisaac@HIDDEN>) id 1m4fkF-0005zt-2P for 39258 <at> debbugs.gnu.org; Sat, 17 Jul 2021 04:32:25 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=systemreboot.net; s=default; h=Content-Type:MIME-Version:Message-ID:Date: References:In-Reply-To:Subject:Cc:To:From:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=9XHhE3oRkVKbYEVFBv116tBS1PpoUsm4abu55KSjdFA=; b=bLnmOToenaHnjudm+XNDAK7YgL Kwhz7T/H0UCU1qVSDsZZxrR5qVRQydKaa3xX2T17eWgG87rAZVqdDqxWHw+1Qq2wY75ZnCrbYd56X n/hauZQWyc8BXFcMPech70e0fN+jBAUXRxiYrs/J8vNCi3p0q4UeyRrtt+f7JBN1eQaQAINj5i0D7 ih/Bpwil+U53TGwtrExni2oBnjbRP+/0G44Lc3ZcqLEVf0ygRbFybLsLYipPhE3JxUwWduXjD85Yc qikSmnaHxe6WOvWFm+lKRnrnjcZhQ5bKYKlSEVGmWTp66n6OtDvnqownZiAZ2s3FwcMPAvE+XktVG D368bsDg==; Received: from [192.168.2.1] (helo=steel) by systemreboot.net with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from <arunisaac@HIDDEN>) id 1m4fk0-000Bej-Po; Sat, 17 Jul 2021 14:02:08 +0530 From: Arun Isaac <arunisaac@HIDDEN> To: zimoun <zimon.toutoune@HIDDEN>, 39258 <at> debbugs.gnu.org Subject: Re: [PATCH v6 1/2] DRAFT packages: Add fields to packages cache. In-Reply-To: <20210715073328.212123-2-zimon.toutoune@HIDDEN> References: <20210715073328.212123-1-zimon.toutoune@HIDDEN> <20210715073328.212123-2-zimon.toutoune@HIDDEN> Date: Sat, 17 Jul 2021 14:01:45 +0530 Message-ID: <877dhpmjda.fsf@HIDDEN> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 39258 Cc: ludo@HIDDEN, zimoun <zimon.toutoune@HIDDEN> X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain Hi Simon, I understand that one of the things you are trying to do is to have a common interface for the cache and no-cache cases. To achieve this, I think fold-available-packages and fold-packages should have the same function signature. They should both pass a <package> object to PROC. Currently, fold-packages is passing a <package> object whereas fold-available-packages is passing the fields of the <package> object as individual parameters. If fold-packages and fold-available-packages have the same function signature, then the changes in your [PATCH v6 2/2] would be way simpler. Also, why do we need two separate functions---fold-available-packages and fold-packages? Can't fold-available-packages do everything fold-packages can and thus totally replace it? > * gnu/packages.scm (generate-package-cache)[expand-cache]: Add synopsis and > description. > (load-package-cache, find-packages-by-names, find-packages-locations): Adapt > accordingly. A couple of typos here: find-packages-by-names -> find-packages-by-name find-packages-locations -> find-package-locations Regards, Arun --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQFPBAEBCAA5FiEEf3MDQ/Lwnzx3v3nTLiXui2GAK7MFAmDylXEbHGFydW5pc2Fh Y0BzeXN0ZW1yZWJvb3QubmV0AAoJEC4l7othgCuze+0H/RN+fBuGpE+4lRyvgxS5 uGLsQ3Y4jmg3pW/SzlFY2yBVg2AehMr06S/sxmT9HK81tKGkkxwYXD/bJz8/aWwu cv51Y0aKRBtVOtv849fByqacQ0aEhETCgF3xw/6cwH8mjfX0YHZmL8dx4flYuYrS fdpkUuRBfYVl1zxDr37IHK6eUtifsBUuRkh/4mDTHJJ2I78rfX58qOITgrRinMZX oqkhOUpJ6MXMMK1YDzwUN4ljgvvUt3FzJM/sz3zBNEKUbRGObi/qv0OCtz40/077 Bqm5hJBUNUKZt4qDZ+ANzk0rpAVx5N5ioQLiX42Yd0HeWrUMnT4NTnGuAHRP9Deg RcA= =+QtZ -----END PGP SIGNATURE----- --=-=-=--
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.
Received: (at 39258) by debbugs.gnu.org; 15 Jul 2021 07:33:48 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Jul 15 03:33:48 2021
Received: from localhost ([127.0.0.1]:46667 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1m3vsS-0006vu-6y
for submit <at> debbugs.gnu.org; Thu, 15 Jul 2021 03:33:48 -0400
Received: from mail-wr1-f41.google.com ([209.85.221.41]:43748)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <zimon.toutoune@HIDDEN>) id 1m3vsN-0006vI-M8
for 39258 <at> debbugs.gnu.org; Thu, 15 Jul 2021 03:33:44 -0400
Received: by mail-wr1-f41.google.com with SMTP id a13so6388669wrf.10
for <39258 <at> debbugs.gnu.org>; Thu, 15 Jul 2021 00:33:43 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
h=from:to:cc:subject:date:message-id:in-reply-to:references
:mime-version:content-transfer-encoding;
bh=Q96Oqdu7AN+kHj5/yhfWJ5qZNWVSRHSqyOco2OpaOS0=;
b=o4MUinOnfDyJOuYrV8XNTr2wgth12z0/2DOIUhFi5mM/z7+ksPWn4JvFZwAoGwDBU+
DPWbVFfw3F6jZWcbIotJv+ALbSZUijFG3iu7PEZPWes2gsBuSKTaHQxrgyp3kDpkMcgd
4+TY7zFWE6f2v8W8tbjwpe+NsokOK/7m893nc2DjrTZHreNK4KV4T9/Y8g+BspDeQO1E
CwXAXi6uMyt//XGddKn3TUnwG9MHrN7xRj8EEetJyJd+IYHH8fEck8tdue/gVkmulBbk
zDS1RThQP+m6VqKnTRPcTOCB95KmA8DfUh5gw5Y5qK725T+SYUiTm3BKoNL9h5gpnHSY
kcdw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20161025;
h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to
:references:mime-version:content-transfer-encoding;
bh=Q96Oqdu7AN+kHj5/yhfWJ5qZNWVSRHSqyOco2OpaOS0=;
b=LL+NCX0Ppw0jbIWo+z8QNq/jbtYslKyxFk+AOmM7rAFmAEk9vMneEBiudxTcGSvUfl
O9qgOON727gBO0mQoer4SJxYR9nxrCLdkC9x19QjZZGnC7KzaHx44E2ITUjWiNs9ngMA
Y9eCS7kCqsSjdaw5UlUJc+LM4A7UA3oxHcEDtc7Fk5M/R5Jpm2ycoiPKlca4wT7Dkwr9
i15n6qdw095De2JzpOPOUB3JqKPSJCRoQiYYB0IN9EzQGtCrXCRPyDk5n3OocyX/Gijy
gYy7sGFLfBkQ+qJDHvbKQ996lxckpv0iT9ZzkvZwxek5CHe5bXKGqph2eh0g/wFzl0ir
DEAQ==
X-Gm-Message-State: AOAM530hcQ6bZ8htDfI9VdGiRFMG+2w+PqO3Ythf7eVNfu4/D3BmSOUm
V4JL5wvYb/pPUUH9UAa//wLSfYs/FSM=
X-Google-Smtp-Source: ABdhPJzbzCalx+tl+6TfRqV6nVv1R4h5gUVcklKnl+8PyXB71NjMQJ3lZqReLZS3z/Tr7NthXBRzxQ==
X-Received: by 2002:adf:ed07:: with SMTP id a7mr3822135wro.70.1626334418108;
Thu, 15 Jul 2021 00:33:38 -0700 (PDT)
Received: from localhost.localdomain ([193.48.40.117])
by smtp.gmail.com with ESMTPSA id y3sm5376205wrh.16.2021.07.15.00.33.37
(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
Thu, 15 Jul 2021 00:33:37 -0700 (PDT)
From: zimoun <zimon.toutoune@HIDDEN>
To: 39258 <at> debbugs.gnu.org
Subject: [PATCH v6 2/2] DRAFT scripts: package: Use cache in
'find-packages-by-description'.
Date: Thu, 15 Jul 2021 09:33:28 +0200
Message-Id: <20210715073328.212123-3-zimon.toutoune@HIDDEN>
X-Mailer: git-send-email 2.32.0
In-Reply-To: <20210715073328.212123-1-zimon.toutoune@HIDDEN>
References: <20210715073328.212123-1-zimon.toutoune@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 39258
Cc: ludo@HIDDEN, arunisaac@HIDDEN, zimoun <zimon.toutoune@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)
* guix/scripts/package.scm (find-packages-by-description): Replace
'fold-packages' by 'fold-available-packages'.
---
guix/scripts/package.scm | 46 +++++++++++++++++++++++++++++-----------
1 file changed, 34 insertions(+), 12 deletions(-)
diff --git a/guix/scripts/package.scm b/guix/scripts/package.scm
index 694959d326..ff2aed6905 100644
--- a/guix/scripts/package.scm
+++ b/guix/scripts/package.scm
@@ -8,7 +8,7 @@
;;; Copyright © 2016 Chris Marusich <cmmarusich@HIDDEN>
;;; Copyright © 2019 Tobias Geerinckx-Rice <me@HIDDEN>
;;; Copyright © 2020 Ricardo Wurmus <rekado@HIDDEN>
-;;; Copyright © 2020 Simon Tournier <zimon.toutoune@HIDDEN>
+;;; Copyright © 2020, 2021 Simon Tournier <zimon.toutoune@HIDDEN>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -181,17 +181,39 @@ hooks\" run when building the profile."
"Return a list of pairs: packages whose name, synopsis, description,
or output matches at least one of REGEXPS sorted by relevance, and its
non-zero relevance score."
- (let ((matches (fold-packages (lambda (package result)
- (if (package-superseded package)
- result
- (match (package-relevance package
- regexps)
- ((? zero?)
- result)
- (score
- (cons (cons package score)
- result)))))
- '())))
+ (define (read-package-from module symbol)
+ (module-ref (resolve-interface module) symbol))
+
+ (let ((matches (fold-available-packages
+ (lambda* (name version result
+ #:key outputs description synopsis location
+ module symbol
+ deprecated?
+ #:allow-other-keys)
+ (if deprecated?
+ result
+ (let* ((package
+ (package
+ (name name)
+ (version version)
+ (source #f)
+ (build-system #f)
+ (outputs outputs)
+ (synopsis synopsis)
+ (description description)
+ (home-page #f)
+ (license #f)
+ (location location))))
+ (match (package-relevance package
+ regexps)
+ ((? zero?)
+ result)
+ (score
+ (cons (cons (read-package-from module symbol)
+ score)
+ result))))))
+ '())))
+
(sort matches
(lambda (m1 m2)
(match m1
--
2.32.0
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.
Received: (at 39258) by debbugs.gnu.org; 15 Jul 2021 07:33:46 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Jul 15 03:33:46 2021
Received: from localhost ([127.0.0.1]:46665 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1m3vsP-0006vg-OJ
for submit <at> debbugs.gnu.org; Thu, 15 Jul 2021 03:33:46 -0400
Received: from mail-wr1-f47.google.com ([209.85.221.47]:40785)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <zimon.toutoune@HIDDEN>) id 1m3vsN-0006vF-8O
for 39258 <at> debbugs.gnu.org; Thu, 15 Jul 2021 03:33:43 -0400
Received: by mail-wr1-f47.google.com with SMTP id l7so6414785wrv.7
for <39258 <at> debbugs.gnu.org>; Thu, 15 Jul 2021 00:33:43 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
h=from:to:cc:subject:date:message-id:in-reply-to:references
:mime-version:content-transfer-encoding;
bh=/ETrfqt6DlChIkn+STgAdUgzArDXN/30X5H0lAwQnU4=;
b=O+jNMlbCW5xz2OPfbZh027qckUJO0bRN+XZ32So/mbYdG0s6PdYzVsQXYSx/sQR9Qh
mpGbLlnrI9waYMP61gAZv3Nfe0zCKb6gpawnDLy6H/Vzk4aNY9hVB8CLFLzdtHBLqbuw
g2VJsmEomG/kXmaYzEkGNCDSRm5avAUJ2tcQL3w7rS1GqZZDmpAU2vfdZEJ59zlTKWj1
7nwmIhYMMByto+lw4Z2r29b/7pcMqj9+lzLTnGHFBlAaziryjaLc2iO4qFD6NiHVOIvf
a/Zo7UJc7vG7Ekl4XCDOLvoeg7+C+0rVWiHOjn1/4qYmsnzLen5kGS058sSBLWMdSve4
x9kA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20161025;
h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to
:references:mime-version:content-transfer-encoding;
bh=/ETrfqt6DlChIkn+STgAdUgzArDXN/30X5H0lAwQnU4=;
b=sSY0c5555FOAkCoHZCvhqnFAH8eAsyMvWJJPU+THEbLq/atfLcFSuud1uUK/ERSUKq
3bH6g4qMlGOMOTM7D1XJHHIE/HSBeRbSQNGyEO+XFX5ll2Dn6RCEmzmb5RXTk7HkpV46
hsLWMDpYRLoCnhkFKa9yDZl6C/j5ros2RPU8/AEm2JO7JLu97G13FA/P+yxew/kqHlID
aIRG5lTAdrGOLZLKqKDJpPhpMFQme9yUIgbXo7jV11Qe0vV4YH7BU3kDHqxKfdHzw/ZA
N388nbtc5QM+qawalTy/PY5MmA3ht80yNwcB2B/2GdZpqgtn79UNzqACCxKpCyIwUadg
jcBw==
X-Gm-Message-State: AOAM530r+HpijDJM4qM+v+iPZ4KJ2RoX5oaoYCUigUqox3JLEyciUNwh
JuUP1grLogYVzoz1/nOmPZvohYpZ5i4=
X-Google-Smtp-Source: ABdhPJykfFrSsZdXN7lqBguFHEd4FzSOna+Nl1Sxim6YOYkeHn+rwYvDvQouieAOoLjp3hgkn3R19A==
X-Received: by 2002:adf:a1c2:: with SMTP id v2mr3657932wrv.155.1626334417589;
Thu, 15 Jul 2021 00:33:37 -0700 (PDT)
Received: from localhost.localdomain ([193.48.40.117])
by smtp.gmail.com with ESMTPSA id y3sm5376205wrh.16.2021.07.15.00.33.37
(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
Thu, 15 Jul 2021 00:33:37 -0700 (PDT)
From: zimoun <zimon.toutoune@HIDDEN>
To: 39258 <at> debbugs.gnu.org
Subject: [PATCH v6 1/2] DRAFT packages: Add fields to packages cache.
Date: Thu, 15 Jul 2021 09:33:27 +0200
Message-Id: <20210715073328.212123-2-zimon.toutoune@HIDDEN>
X-Mailer: git-send-email 2.32.0
In-Reply-To: <20210715073328.212123-1-zimon.toutoune@HIDDEN>
References: <20210715073328.212123-1-zimon.toutoune@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 39258
Cc: ludo@HIDDEN, arunisaac@HIDDEN, zimoun <zimon.toutoune@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)
* gnu/packages.scm (generate-package-cache)[expand-cache]: Add synopsis and
description.
(load-package-cache, find-packages-by-names, find-packages-locations): Adapt
accordingly.
(fold-available-packages): Add synopsis, description, module and symbol when
cache is authoritative. Replace 'fold-packages' by
'fold-module-public-variables*' when cache is not authoritative.
---
gnu/packages.scm | 52 +++++++++++++++++++++++++++++++++---------------
1 file changed, 36 insertions(+), 16 deletions(-)
diff --git a/gnu/packages.scm b/gnu/packages.scm
index ccfc83dd11..34c6d73b86 100644
--- a/gnu/packages.scm
+++ b/gnu/packages.scm
@@ -4,6 +4,7 @@
;;; Copyright © 2014 Eric Bavier <bavier@HIDDEN>
;;; Copyright © 2016, 2017 Alex Kost <alezost@HIDDEN>
;;; Copyright © 2016 Mathieu Lirzin <mthl@HIDDEN>
+;;; Copyright © 2021 Simon Tournier <zimon.toutoune@HIDDEN>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -211,28 +212,45 @@ package module."
(vhash-fold (lambda (name vector result)
(match vector
(#(name version module symbol outputs
+ synopsis description
supported? deprecated?
file line column)
(proc name version result
#:outputs outputs
+ #:synopsis synopsis
+ #:description description
#:location (and file
(location file line column))
+ #:module module
+ #:symbol symbol
#:supported? supported?
#:deprecated? deprecated?))))
init
cache)
- (fold-packages (lambda (package result)
- (proc (package-name package)
- (package-version package)
- result
- #:outputs (package-outputs package)
- #:location (package-location package)
- #:supported?
- (->bool (supported-package? package))
- #:deprecated?
- (->bool
- (package-superseded package))))
- init)))
+ (fold-module-public-variables*
+ (lambda (module symbol variable result)
+ (let ((package (false-if-exception
+ (variable-ref variable))))
+ (if (package? package)
+ (proc (package-name package)
+ (package-version package)
+ result
+ #:outputs (package-outputs package)
+ #:synopsis (package-synopsis package)
+ #:description (package-description package)
+ #:location (package-location package)
+ #:module (module-name module)
+ #:symbol symbol
+ #:supported?
+ (->bool (supported-package? package))
+ #:deprecated?
+ (->bool
+ (package-superseded package)))
+ result)))
+ init
+ (all-modules (%package-module-path)
+ #:warn
+ warn-about-load-error))))
(define* (fold-packages proc init
#:optional
@@ -268,6 +286,7 @@ package names. Return #f on failure."
(fold (lambda (item vhash)
(match item
(#(name version module symbol outputs
+ synopsis description
supported? deprecated?
file line column)
(vhash-cons name item vhash))))
@@ -316,7 +335,7 @@ decreasing version order."
(if (and (cache-is-authoritative?) cache)
(match (cache-lookup cache name)
(#f #f)
- ((#(_ versions modules symbols _ _ _ _ _ _) ...)
+ ((#(_ versions modules symbols _ _ _ _ _ _ _ _) ...)
(fold (lambda (version* module symbol result)
(if (or (not version)
(version-prefix? version version*))
@@ -337,9 +356,8 @@ matching NAME and VERSION."
(if (and cache (cache-is-authoritative?))
(match (cache-lookup cache name)
(#f '())
- ((#(name versions modules symbols outputs
- supported? deprecated?
- files lines columns) ...)
+ ((#(_ versions _ _ _ _ _ _ _
+ files lines columns) ...)
(fold (lambda (version* file line column result)
(if (and file
(or (not version)
@@ -393,6 +411,8 @@ reducing the memory footprint."
,(module-name module)
,symbol
,(package-outputs package)
+ ,(package-synopsis package)
+ ,(package-description package)
,(->bool (supported-package? package))
,(->bool (package-superseded package))
,@(let ((loc (package-location package)))
--
2.32.0
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.
Received: (at 39258) by debbugs.gnu.org; 15 Jul 2021 07:33:45 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Jul 15 03:33:45 2021
Received: from localhost ([127.0.0.1]:46663 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1m3vsP-0006ve-76
for submit <at> debbugs.gnu.org; Thu, 15 Jul 2021 03:33:45 -0400
Received: from mail-wr1-f45.google.com ([209.85.221.45]:34658)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <zimon.toutoune@HIDDEN>) id 1m3vsM-0006vE-T0
for 39258 <at> debbugs.gnu.org; Thu, 15 Jul 2021 03:33:43 -0400
Received: by mail-wr1-f45.google.com with SMTP id u1so6464491wrs.1
for <39258 <at> debbugs.gnu.org>; Thu, 15 Jul 2021 00:33:42 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
h=from:to:cc:subject:date:message-id:mime-version
:content-transfer-encoding;
bh=SxulHMwv2ho0c/VIBIpkJ+f2dRMjwqwllobobw5/ykY=;
b=q8NHSwUSYRMexyfN28eeHqFGVPswoIrKK4Lmd2/U69ZMnw9c0C7rAQkUpPwT+8Dz+3
K8ykKL0tkz32E4eI3RadbhR76WbKT1KQ56h4DaMFo6Bwrp1EdYpTgxmQMblLVTabqkg8
sV1ud6rNAjQssxHIkB9AXijzPu0Pp0N0exTIkOlGvjAXvPrFKB/2Ff4aGzWpplNJtUmV
pi9I8iW/GyIOKsYsBvuu0j86kIwW213rEWYBVMJgcu32QLI9DWi1nhw1mWIZwYfxjvm9
OcKVbFKG/xBQj7bzjKJ1hSOXpPAAXZtk9rNFCkV6PPGVeBd/qEIMlvE8//suDDl1fQYq
S3Tg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20161025;
h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version
:content-transfer-encoding;
bh=SxulHMwv2ho0c/VIBIpkJ+f2dRMjwqwllobobw5/ykY=;
b=PM2LZNhpgT+008Wmc2hzjHSUf/SRL/I33qcE7oaWThRmgAW/SybzeoCOTIrqso7s1Q
GzDOTyay0EKn6k21oH8AEuqbYK1C6b1aFaKP4wGUBD597huwgFvElxRr+Ovw/axGN3Z/
BB0u5/fR0ui2OE+uw3N/sO9hv3hNW/nqsBiGb9udMfNDbk2rlEn68y74BSGTpZZkiVaT
T/y8Q0dDBMgvyZ5TTQijms3ups3gufO4VYflNYf1fNW/0Pc+lSY0EI+oSqdes7v9XgZb
n36mNypsNtElYMgkxSraz9zUysL2lcJmv/6d29LV3VMLwjjF8duv5JRc3Jw43tX8YkLZ
lKcA==
X-Gm-Message-State: AOAM533nkQYykn7n7IQTz0aR0Vk+rNsmdoqgzaTfmyUEvrZgjGXZis3c
MA3fEmp/Wq9meHIc491Lr8TUrZaBMlg=
X-Google-Smtp-Source: ABdhPJw4Q2xPor1BtBZX6x+gzqeqK691ET7saEFmkVH3dSlzxQgUow16apstSIeenNb51QJ+M1D4zw==
X-Received: by 2002:a05:6000:1248:: with SMTP id
j8mr3655805wrx.391.1626334417042;
Thu, 15 Jul 2021 00:33:37 -0700 (PDT)
Received: from localhost.localdomain ([193.48.40.117])
by smtp.gmail.com with ESMTPSA id y3sm5376205wrh.16.2021.07.15.00.33.36
(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
Thu, 15 Jul 2021 00:33:36 -0700 (PDT)
From: zimoun <zimon.toutoune@HIDDEN>
To: 39258 <at> debbugs.gnu.org
Subject: [PATCH v6 0/2] DRAFT "guix search" performances
Date: Thu, 15 Jul 2021 09:33:26 +0200
Message-Id: <20210715073328.212123-1-zimon.toutoune@HIDDEN>
X-Mailer: git-send-email 2.32.0
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 39258
Cc: ludo@HIDDEN, arunisaac@HIDDEN, zimoun <zimon.toutoune@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)
Hi,
This is an attempt to improve the performance of "guix search". It is still
half baked but it allows to discuss further the idea about expanding the
current '/lib/guix/package.cache' and avoids to forget an IRL discussion. ;-)
Let start by what needs to be improved: the part when cache is not
authoritative. It is slower than the current approach because the package is
read twice, i.e., the module is indeed loaded twice, once by
'fold-available-packages' via 'fold-module-public-variables*' and then again
by 'find-packages-by-description' via 'read-package-from'. The issue is to
have a common interface for both cases (cache and no-cache). More thoughts
are required. ;-)
Then, using the cache is slower than expected. Therefore, something is maybe
twisted -- quick implementation before holidays ;-) -- with the use of
'fold-avaibale-packages' as proposed by Ludo [1]. Note that instead another
'fold-packages' (say 'fold-packages*') using the new cache should be used. As
it is done with v4 and the performances were as expected:
<http://issues.guix.gnu.org/39258#89>
1: <http://issues.guix.gnu.org/39258#93>
From my understanding, the issue that 'package-relevance' accepts a 'package'
(and then all the chain until displaying) and 'fold-avaibale-packages' does
not return a package. Well, I do not know; especially where to put something
similiar to 'read-package-from'.
To test, after applying the patches, the command is:
./pre-inst-env guix pull --allow-downgrades --disable-authentication \
--url=$(pwd) --branch=search-v6 -p /tmp/new
Let compare only for cold cache and time this cache building (Guix 7db8fd6):
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time guix build --check $(guix gc --derivers $(readlink -f ~/.config/guix/current/lib/guix/package.cache))
real 0m28,848s
user 0m1,481s
sys 0m0,252s
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time guix build --check $(guix gc --derivers $(readlink -f /tmp/new/lib/guix/package.cache))
real 0m40,279s
user 0m1,582s
sys 0m0,232s
It seems longer but compared to the time of "guix pull" completion, it seems
acceptable. However, maybe it could become an issue when running a lot of
"guix time-machine"... Well, hard trade-off. ;-)
Let compare for some queries:
--8<---------------cut here---------------start------------->8---
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time guix search game | recsel -C -P name | wc -l
371
real 0m7,561s
user 0m3,525s
sys 0m0,391s
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time /tmp/new/bin/guix search game | recsel -C -P name | wc -l
371
real 0m9,814s
user 0m3,240s
sys 0m0,363s
--8<---------------cut here---------------end--------------->8---
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time guix search strategy game | recsel -C -P name | wc -l
16
real 0m8,565s
user 0m2,803s
sys 0m0,430s
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time /tmp/new/bin/guix search strategy game | recsel -C -P name | wc -l
16
real 0m9,679s
user 0m2,370s
sys 0m0,334s
--8<---------------cut here---------------start------------->8---
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time guix search strategy game caesar | recsel -C -P name | wc -l
0
real 0m8,307s
user 0m2,388s
sys 0m0,366s
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time /tmp/new/bin/guix search strategy game caesar | recsel -C -P name | wc -l
0
real 0m3,626s
user 0m0,948s
sys 0m0,101s
--8<---------------cut here---------------end--------------->8---
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time guix search game strategy the | recsel -C -P name | wc -l
4
real 0m8,776s
user 0m2,903s
sys 0m0,454s
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time /tmp/new/bin/guix search game strategy the | recsel -C -P name | wc -l
4
real 0m9,495s
user 0m2,546s
sys 0m0,313s
--8<---------------cut here---------------start------------->8---
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time guix search the game strategy | recsel -C -P name | wc -l
4
real 0m8,502s
user 0m2,534s
sys 0m0,388s
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time /tmp/new/bin/guix search the game strategy | recsel -C -P name | wc -l
4
real 0m9,508s
user 0m2,254s
sys 0m0,363s
--8<---------------cut here---------------end--------------->8---
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time guix search crypto library | recsel -C -P name | grep libb2
libb2
real 0m8,744s
user 0m2,875s
sys 0m0,374s
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time /tmp/new/bin/guix search crypto library | recsel -C -P name | grep libb2
libb2
real 0m9,229s
user 0m2,448s
sys 0m0,397s
--8<---------------cut here---------------start------------->8---
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time guix search cuirass integration | recsel -C -P name | wc -l
1
real 0m8,132s
user 0m2,343s
sys 0m0,407s
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time /tmp/new/bin/guix search cuirass integration | recsel -C -P name | wc -l
1
real 0m8,940s
user 0m2,036s
sys 0m0,369s
--8<---------------cut here---------------end--------------->8---
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time guix search cuirass | recsel -C -P name | wc -l
2
real 0m8,240s
user 0m2,461s
sys 0m0,367s
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time /tmp/new/bin/guix search cuirass | recsel -C -P name | wc -l
2
real 0m8,863s
user 0m2,019s
sys 0m0,377s
--8<---------------cut here---------------start------------->8---
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time guix search cuirass integration foo | recsel -C -P name | wc -l
0
real 0m8,258s
user 0m2,418s
sys 0m0,521s
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time /tmp/new/bin/guix search cuirass integration foo | recsel -C -P name | wc -l
0
real 0m3,358s
user 0m0,867s
sys 0m0,139s
--8<---------------cut here---------------end--------------->8---
This last example suggests that 'read-package-from' is the slowdown.
(On a side note, maybe I am doing wrong, but there is no improvement by the
recent introduction of 'cut' for multi-terms as the query "the game strategy"
and "game strategy the". Another story. :-))
When cache is not authoritative, it is worse, as expected:
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time /tmp/new/bin/guix search -L /tmp/my-pkgs cuirass integration foo | recsel -C -P name | wc -l
0
real 0m12,503s
user 0m7,807s
sys 0m0,529s
and note that currently the performances of "guix search" is the same for both
cases (authoritative and not authoritative); i.e., see previous timing.
Last, two minor remarks about previous comments.
1. Ludo commented:
> Therefore the cache '/lib/guix/package.cache' contains more
> information.
This breaks the binary interface, so we’ll have to analyze the impact of
such a change and devise a strategy.
<http://issues.guix.gnu.org/39258#93>
and after some checking, this should be fine, IIUC. The '--news' is ok
because of '#:allow-other-keys'. And other parts are modified accordingly.
Guix revision N creates a cache that Guix revision N+1 will read but it should
not be an issue; see 'inferior-available-packages'.
2. And Ludo wrote:
I realize the other cache also has that problem, but it would be nice to add a
version tag to the cache. Basically emit something like:
(package-metadata-cache (version 0) VECTOR …)
instead of just:
(VECTOR …)
<http://issues.guix.gnu.org/39258#93>
which is, after discussions, not necessary. Versioning does not make sense
here because the cache is read by the Guix which generates it. Therefore,
specify a version is extraneous here.
Comments are welcome for this work-in-progress. ;-)
Cheers,
simon
zimoun (2): DRAFT packages: Add fields to packages cache. DRAFT scripts:
package: Use cache in 'find-packages-by-description'.
gnu/packages.scm | 52 +++++++++++++++++++++++++++-------------
guix/scripts/package.scm | 46 +++++++++++++++++++++++++----------
2 files changed, 70 insertions(+), 28 deletions(-)
base-commit: 4196087f3d6fc254db5b4c47658e5679c835516f
--
2.32.0
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 15 Jun 2020 20:18:46 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Mon Jun 15 16:18:46 2020 Received: from localhost ([127.0.0.1]:47560 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1jkvZ8-0000NP-Az for submit <at> debbugs.gnu.org; Mon, 15 Jun 2020 16:18:46 -0400 Received: from mugam.systemreboot.net ([139.59.75.54]:47076) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <arunisaac@HIDDEN>) id 1jkvZ4-0000ND-Ke for 39258 <at> debbugs.gnu.org; Mon, 15 Jun 2020 16:18:44 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=systemreboot.net; s=default; h=Content-Type:MIME-Version:Message-ID:Date: References:In-Reply-To:Subject:Cc:To:From:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=XTjt+hL6kTgRtgDdRGuFat1EXlW6tKcxy5xWcfA8ysE=; b=f5Odcm9OWeOjxnPoMgE8ILZXW /47MNizWrCMuL3jqkHOi+ZUU2vRoj+Dcq+0Ps8Sz/n8tEl2WHBtVzhPN1MI+s8UlUAiWK6Qo6Dfn1 009N+KfykymAAebw3vjVoQh6nzAJi5cmnBUwdonF6Y6A+yvXkN23O5I+xfuAIFwXbk68U=; Received: from [192.168.2.1] (helo=steel) by systemreboot.net with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.93) (envelope-from <arunisaac@HIDDEN>) id 1jkvYz-001RD6-1b; Tue, 16 Jun 2020 01:48:37 +0530 From: Arun Isaac <arunisaac@HIDDEN> To: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN> Subject: Re: [bug#39258] [PATCH 2/4] ui: Use string matching with literal search strings. In-Reply-To: <87ftayzrqi.fsf@HIDDEN> References: <20200601000030.7443-1-arunisaac@HIDDEN> <20200601000030.7443-3-arunisaac@HIDDEN> <87y2owmyf3.fsf@HIDDEN> <cu7bllnnnty.fsf@HIDDEN> <87ftayzrqi.fsf@HIDDEN> Date: Tue, 16 Jun 2020 01:48:31 +0530 Message-ID: <87a714nkvc.fsf@HIDDEN> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 39258 Cc: 39258 <at> debbugs.gnu.org, zimoun <zimon.toutoune@HIDDEN> X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable >> So, shall I push the remaining patches (patches 1, 3, 4) after applying >> the change you suggested for patch 1 (use of if versus cond)? > > Yes, definitely! Done! >>>> * guix/scripts/package.scm (process-query): Make search query a regexp= only if >>>> it is not a literal search string. >>>> * guix/ui.scm (relevance): Use string matching with literal search str= ings and >>>> regexp matching with regexp search strings. >>> >>> How does this affect performance? > > (To be clear, I=E2=80=99m referring specifically to this patch.) Oh, I misunderstood. Here are the results specifically comparing patch 2 against the latest master (that includes the patches 1, 3 and 4 I just pushed). All readings are on a warm cache. |----------------------------------+--------+-------| | query | before | after | |----------------------------------+--------+-------| | guix search strategy game | 2.1 | 1.7 | | guix search strategy game caesar | 1.8 | 1.5 | |----------------------------------+--------+-------| --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEf3MDQ/Lwnzx3v3nTLiXui2GAK7MFAl7n15cACgkQLiXui2GA K7NgJgf/YrTf2gVKPTV6D930QUmTCTHK7+62b4y5WuOU5UxZWcgizs6iZD3AoVq8 E39md5kiA7qECfA+YFfBIEA9LgRLdlRYyKm4+WWdDIdbEaFiYAyiVw3hA5seu4/n 7wPgK7tQTU0vv5yi7ovG2aIezkkw/9TOATEJRerKIlzK/TsThpBBCftnxiIHJ9z9 2WD5wfxh7UHGBbRMQ5mkPIfmx1UVLoRBAdeFU+vCZ/CWOwh6uU8VwFYWEV9tmYM/ 0hVqS8LowgmzytBPn+G5sbI6+hffMAvHJXkJoesazrQLxnQWYW3ew1Y/ogYbeOBa LqkYT9uXy3+EqfqoPETsYwCus/DWfg== =92VK -----END PGP SIGNATURE----- --=-=-=--
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.
Received: (at 39258) by debbugs.gnu.org; 14 Jun 2020 19:14:59 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Jun 14 15:14:58 2020
Received: from localhost ([127.0.0.1]:44523 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1jkY5q-0006bS-GB
for submit <at> debbugs.gnu.org; Sun, 14 Jun 2020 15:14:58 -0400
Received: from mail-wr1-f53.google.com ([209.85.221.53]:37304)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <zimon.toutoune@HIDDEN>) id 1jkY5n-0006bF-T2
for 39258 <at> debbugs.gnu.org; Sun, 14 Jun 2020 15:14:56 -0400
Received: by mail-wr1-f53.google.com with SMTP id x13so14909098wrv.4
for <39258 <at> debbugs.gnu.org>; Sun, 14 Jun 2020 12:14:55 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
h=from:to:cc:subject:in-reply-to:date:message-id:mime-version;
bh=t0IjIRjNaXuyy/TBt+Ok+vhyTVH35/BEWnKzF+RTR+E=;
b=cTJosOqmduwecaI1GJmzevUqWBUpDTpi9RRIbJUcn7CQSCrt9hacOi8+G1e9NCaWXg
wx7c3uhpZeZXn4VBEdRe9W9ycvXNd8sFpLIdgW4wWsyNV6eaJYft49oBwp+1W54qqFgJ
CRPtT2UQhfJ1pYDl8Yc56xDVuL8sEY4x6oKObleXHS1mO3Ny6oXNvvrUn5H2WaFH/KeV
rEhpaohpYiZs/SUHcnKFCfPCczqiUrksIcrTlgrDGYuNKYIK4aQABryWZT5vv9ymHIbv
ZPd52vHkZZUHZXpI8c9K3UIDQG6z1EAIoImCQuLUH66sBW4JFHb9+Cmo6LIr7XEwgBiN
JuQQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20161025;
h=x-gm-message-state:from:to:cc:subject:in-reply-to:date:message-id
:mime-version;
bh=t0IjIRjNaXuyy/TBt+Ok+vhyTVH35/BEWnKzF+RTR+E=;
b=bmhlLkM0GPezMnTK6k5OwBd2F5YQxuOGrb4TsKyqtZZXoxbPtF/GTMr3cQCpk6eGd+
d2GigsG1xTyZNAM+fIrfcb3XBVs1abm1Pd2Uu48D09ktN1O6Myr9pd7m6VwV7Rc0EecW
SAR4Z63Ln4kYLOVzHYbd76GYx4EkwDwLR2sQ/fl4rH8yFTzk7gat9acU/+t9FsCwgO7N
1yvLARCYqB8HY0peHWj9dXD336uoKvYIJN+1+uN/B22fdREFbTFkG2sbO7DfMKCLkO5z
nzdrmWkhLe3gHRThRF9hhDwM3yDfnDL6ZDh+3BPnCABJ3IIRExBYjAU2qnyCbHV/lz8M
mV5g==
X-Gm-Message-State: AOAM530jgn9HB+95AohcPIjT/LUfPxxhft5JqnZe1YcI+ZrKzwYnua+P
e2MYubP9Ed2d3crcQ+OFr7IfCqWdn7I=
X-Google-Smtp-Source: ABdhPJxVxrQMwTtR9+nzq+pmBww2vVjo+VoR6azb3L21v2h++wMxi77OWXiU+toDpcJY1tRIVSfyMw==
X-Received: by 2002:a05:6000:1292:: with SMTP id
f18mr25643027wrx.208.1592162089528;
Sun, 14 Jun 2020 12:14:49 -0700 (PDT)
Received: from lili (57.246.195.77.rev.sfr.net. [77.195.246.57])
by smtp.gmail.com with ESMTPSA id a1sm19032136wmj.29.2020.06.14.12.14.48
(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
Sun, 14 Jun 2020 12:14:48 -0700 (PDT)
From: zimoun <zimon.toutoune@HIDDEN>
To: Arun Isaac <arunisaac@HIDDEN>, Ludovic =?utf-8?Q?Court=C3=A8?=
=?utf-8?Q?s?= <ludo@HIDDEN>
Subject: Re: [bug#39258] [PATCH 2/4] ui: Use string matching with literal
search strings.
In-Reply-To: <cu7zh96dgql.fsf@HIDDEN>
Date: Sun, 14 Jun 2020 21:14:34 +0200
Message-ID: <86sgex8nol.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 39258
Cc: 39258 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)
Dear Arun,
Here, I am speaking about only the first patch: the cut-off.
TL;DR:
1. I was wrong about the bottleneck.
2. The queries were not the good ones to see a clear effect
-- on my machine.
On Sat, 13 Jun 2020 at 22:51, Arun Isaac <arunisaac@HIDDEN> wrote:
> Yes, I did read your earlier mail. And, I tried again, this time with
> patch 1 alone. It certainly makes a difference on my machine. It is
> clear from the code logic that it should make a difference on your
> machine as well, at least for longer queries. But, somehow it isn't and
> I do not understand why. :-(
Well, I spent some hours* to do some stats (Student's t-test). Roughly
speaking, on my machine, the standard deviation error (stddev) hides the
point -- depending on the query -- and that's why I am not always seeing
the improvement, I guess.
*ah all my Sunday in fact. ;-)
I compared different conditions for the query "game strategy":
- cold vs warm
- xterm vs shell in Emacs (my config vs -q)
- no pipe vs pipe
And I run 10 times in a row each experiment. The conclusion is: in
average -- on my machine -- the cut-off improves. But sometimes
considering only 3 repeats in a row, the improvement is not obvious (on
the mean); because the both tails of distribution overlap a bit on my
machine and so it is kind of bad luck. And it is ``worse'' depending
against which commit your patch is rebased: a357849 (old) vs e782756.
The t-test captures this variation, even with only 3 repeats, but I have
not done in my previous email and only compared the visible mean. Sorry
about that.
Moreover, printing increases the stddev, so the results are more
fluctuating inside Emacs vs xterm and piping helps in this case.
Piping does not change the final result -- hopefully. :-) It adds an
extra time but in average it is the same.
About cold vs warm cache, I notice that the improvement is not the same
(in average). Considering the raw time, there is a difference about 10%
(with "good" confidence); it could be worth to understand why.
Well, considering that, I did other stats with other queries and the
conclusion for my machine is that *the patch improves* on average by
reducing the timing for typical usages. Which is really cool! :-)
I definitively have wrong about the bottleneck and this one could be
one. One way to have an idea is to use "statprof" but it is hard for me
to read the results (I believe Guile master have a fix improving the
'anon #addr', but do not really know more).
--8<---------------cut here---------------start------------->8---
$ /tmp/v5-1/bin/guix repl
scheme@(guix-user)> ,use(guix scripts search)
scheme@(guix-user)> ,pr (guix-search "game" "strategy")
% cumulative self
time seconds seconds procedure
17.81 0.29 0.27 anon #xe40178
12.33 0.20 0.18 ice-9/boot-9.scm:2201:0:%load-announce
12.33 0.18 0.18 anon #xe3c770
5.48 0.08 0.08 ice-9/boot-9.scm:1396:0:symbol-append
4.11 1.57 0.06 guix/memoization.scm:100:0
4.11 0.06 0.06 ice-9/popen.scm:145:0:reap-pipes
2.74 0.55 0.04 guix/ui.scm:1511:12
2.74 0.33 0.04 ice-9/regex.scm:170:0:fold-matches
2.74 0.04 0.04 ice-9/boot-9.scm:3540:0:autoload-done-or-in-progress?
2.74 0.04 0.04 texinfo/string-utils.scm:98:5
2.74 0.04 0.04 ice-9/vlist.scm:539:0:vhash-assq
1.37 69.81 0.02 ice-9/threads.scm:388:4
[...]
---
Sample count: 73
Total time: 1.490955132 seconds (0.387756476 seconds in GC)
--8<---------------cut here---------------end--------------->8---
To compare with the default:
--8<---------------cut here---------------start------------->8---
time seconds seconds procedure
24.47 0.49 0.46 anon #x1d89178
21.28 0.40 0.40 anon #x1d85770
9.57 0.20 0.18 ice-9/boot-9.scm:2201:0:%load-announce
3.19 4.71 0.06 ice-9/boot-9.scm:1673:4:with-exception-handler
3.19 1.64 0.06 guix/memoization.scm:100:0
3.19 0.06 0.06 ice-9/boot-9.scm:3540:0:autoload-done-or-in-progress?
3.19 0.06 0.06 anon #x1d84c78
3.19 0.06 0.06 ice-9/popen.scm:145:0:reap-pipes
2.13 1.01 0.04 guix/ui.scm:1511:12
2.13 0.08 0.04 ice-9/boot-9.scm:1396:0:symbol-append
2.13 0.04 0.04 anon #x1d83248
1.06 0.30 0.02 anon #x7f057e6c90e8
[...]
--8<---------------cut here---------------end--------------->8---
So clearly the patch has an effect! If someone knows what is:
- ice-9/boot-9.scm:2201:0:%load-announce
- ice-9/boot-9.scm:1396:0:symbol-append
and from where they could come from, it could help. :-)
Well, I am interested to know which part is the Regex Engine and the
string search. :-) Linking to the discussion about KMP and others.
> Here are more fresh results. Could you try for longer queries like
> "strategy game caesar" and without the output being piped to recsel,
> grep, etc.? For simplicity, let's talk only about warm cache results.
>
> |----------------------------------+--------+-------|
> | query | before | after |
> |----------------------------------+--------+-------|
> | guix search strategy game | 2.58 | 1.96 |
> | guix search strategy game caesar | 2.95 | 1.76 |
> |----------------------------------+--------+-------|
At first, I was confused why one more terms returns faster. This is
because the query "caesar" returns only one package so the query
"strategy game caesar" cuts off all the packages when searching the
terms "game" and then "strategy". I mean
guix search julius
should be as long as
guix search strategy game caesar
It is; in average on my machine.
And secondly, I was confused because the timing of the query "caesar
strategy game" is almost the same (2.8% +/- 2.5% with 99.0% of
confidence; 10 repeats). Well, it is because in one case the term
"caesar" is applied to 15 packages and in another case the terms
"strategy" and "game" are applied to 1 package. Adding some stddev
error and not enough repeats (nor good stats), the confusion is complete
and my conclusion is wrong.
That's said, the effect of the cut-off is clear (on my machine even with
on shot) with the queries:
- game strategy the
- the game strategy
Thank you,
simon
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 13 Jun 2020 19:32:52 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Sat Jun 13 15:32:52 2020 Received: from localhost ([127.0.0.1]:42519 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1jkBtb-0001aM-V5 for submit <at> debbugs.gnu.org; Sat, 13 Jun 2020 15:32:52 -0400 Received: from eggs.gnu.org ([209.51.188.92]:52730) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <ludo@HIDDEN>) id 1jkBta-0001a9-20 for 39258 <at> debbugs.gnu.org; Sat, 13 Jun 2020 15:32:50 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:34110) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from <ludo@HIDDEN>) id 1jkBtT-0005MU-F5; Sat, 13 Jun 2020 15:32:43 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=45556 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from <ludo@HIDDEN>) id 1jkBtR-0006QA-7W; Sat, 13 Jun 2020 15:32:42 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN> To: Arun Isaac <arunisaac@HIDDEN> Subject: Re: [bug#39258] [PATCH 2/4] ui: Use string matching with literal search strings. References: <20200601000030.7443-1-arunisaac@HIDDEN> <20200601000030.7443-3-arunisaac@HIDDEN> <87y2owmyf3.fsf@HIDDEN> <cu7bllnnnty.fsf@HIDDEN> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 26 Prairial an 228 de la =?utf-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Sat, 13 Jun 2020 21:32:37 +0200 In-Reply-To: <cu7bllnnnty.fsf@HIDDEN> (Arun Isaac's message of "Sat, 13 Jun 2020 18:07:45 +0530") Message-ID: <87ftayzrqi.fsf@HIDDEN> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 39258 Cc: 39258 <at> debbugs.gnu.org, zimoun <zimon.toutoune@HIDDEN> X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -3.3 (---) Hi, Arun Isaac <arunisaac@HIDDEN> skribis: >>> * guix/scripts/package.scm (process-query): Make search query a regexp = only if >>> it is not a literal search string. >>> * guix/ui.scm (relevance): Use string matching with literal search stri= ngs and >>> regexp matching with regexp search strings. >> >> How does this affect performance? (To be clear, I=E2=80=99m referring specifically to this patch.) > See my results from earlier. > > time ./pre-inst-env guix search game > > real 0m2.261s > user 0m2.351s > sys 0m0.104s > > time guix search game > > real 0m2.661s > user 0m2.843s > sys 0m0.080s > >> I would expect the regexp engine in libc to do something similar >> internally, so I wonder if the extra work in Scheme pays off. > > I agree it would better to do this optimization at the regexp engine, if > it doesn't do it already. Yeah. I feel like we shouldn=E2=80=99t have to do this, so I=E2=80=99d lea= n towards excluding this patch from the series. It=E2=80=99s too early to be confide= nt about it, but it might be something as discussed in <https://lists.gnu.org/archive/html/guile-user/2020-06/msg00038.html>, i.e., a problem to solve at the Guile level. > So, shall I push the remaining patches (patches 1, 3, 4) after applying > the change you suggested for patch 1 (use of if versus cond)? Yes, definitely! Thank you, Ludo=E2=80=99.
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 13 Jun 2020 17:21:21 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Sat Jun 13 13:21:21 2020 Received: from localhost ([127.0.0.1]:42477 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1jk9qL-0006st-Fz for submit <at> debbugs.gnu.org; Sat, 13 Jun 2020 13:21:21 -0400 Received: from mugam.systemreboot.net ([139.59.75.54]:45734) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <arunisaac@HIDDEN>) id 1jk9qI-0006si-FH for 39258 <at> debbugs.gnu.org; Sat, 13 Jun 2020 13:21:20 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=systemreboot.net; s=default; h=Content-Type:MIME-Version:Message-ID:Date: References:In-Reply-To:Subject:Cc:To:From:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=7ymjbnZlBDnuGlzJJ5Cuhc/q8TpMzKJROJQSwXLx9iE=; b=bHyQz9Llr230FR5GJRfjOpW8+ rSsILRWkMsfL0poX+Z7Kstn2GRU8o6nrbzU402+kJ5GD97x3Vt4J93FH/qYDdQGlC1qY6wXnId6+y tU/BhgY+OYTv+jngQ3lLJMQHCIEWEwV9cDqPucaod7qEntsLLd6BZynqX7fGTL4f+9D2k=; Received: from [192.168.2.1] (helo=steel) by systemreboot.net with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.93) (envelope-from <arunisaac@HIDDEN>) id 1jk9qD-001LwO-A4; Sat, 13 Jun 2020 22:51:13 +0530 From: Arun Isaac <arunisaac@HIDDEN> To: zimoun <zimon.toutoune@HIDDEN>, Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN> Subject: Re: [bug#39258] [PATCH 2/4] ui: Use string matching with literal search strings. In-Reply-To: <865zbvjddy.fsf@HIDDEN> References: <20200601000030.7443-1-arunisaac@HIDDEN> <20200601000030.7443-3-arunisaac@HIDDEN> <87y2owmyf3.fsf@HIDDEN> <cu7bllnnnty.fsf@HIDDEN> <865zbvjddy.fsf@HIDDEN> Date: Sat, 13 Jun 2020 22:51:06 +0530 Message-ID: <cu7zh96dgql.fsf@HIDDEN> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 39258 Cc: 39258 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain > I confirm that it changes nothing. See [1]. > > 1: http://issues.guix.gnu.org/39258#112 Yes, I did read your earlier mail. And, I tried again, this time with patch 1 alone. It certainly makes a difference on my machine. It is clear from the code logic that it should make a difference on your machine as well, at least for longer queries. But, somehow it isn't and I do not understand why. :-( Here are more fresh results. Could you try for longer queries like "strategy game caesar" and without the output being piped to recsel, grep, etc.? For simplicity, let's talk only about warm cache results. |----------------------------------+--------+-------| | query | before | after | |----------------------------------+--------+-------| | guix search strategy game | 2.58 | 1.96 | | guix search strategy game caesar | 2.95 | 1.76 | |----------------------------------+--------+-------| --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEf3MDQ/Lwnzx3v3nTLiXui2GAK7MFAl7lCwIACgkQLiXui2GA K7Prngf/V/Aw0jm45wsXT7y4WBkXsgG4noCOmagQck0gvadqDPYAvDomKVYgbatr I5YIy7IdpSfILpHKMQzaPHbUX6J2DxZ/F+yaO/nQDtybWujl/2VsxgF0mkc0r3d+ 1IY4jka8vxUleSgNzbfJBSMEvj5evSEnnlpBiN0M/ko0RY87HrrGKmZd9MTAzDCO WuUjlqIYFSCwndL6WfAXZeNLVNn3VogP98FSJWbYNL4D0B3hx87apvg4P6qq2j+l W5l601178ojQA2MaQ4qE+LuvXHzghknooWaAW5bUXKDMetxy/qytW/4SeLLHnelv 5Hjh6muvTy8M2f9mJLBbA82qoynr+w== =CSq0 -----END PGP SIGNATURE----- --=-=-=--
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 13 Jun 2020 13:37:09 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Sat Jun 13 09:37:09 2020 Received: from localhost ([127.0.0.1]:41089 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1jk6LN-0001AP-Cd for submit <at> debbugs.gnu.org; Sat, 13 Jun 2020 09:37:09 -0400 Received: from mail-wr1-f47.google.com ([209.85.221.47]:37874) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <zimon.toutoune@HIDDEN>) id 1jk6LK-0001A2-R5 for 39258 <at> debbugs.gnu.org; Sat, 13 Jun 2020 09:37:07 -0400 Received: by mail-wr1-f47.google.com with SMTP id x13so12626397wrv.4 for <39258 <at> debbugs.gnu.org>; Sat, 13 Jun 2020 06:37:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:in-reply-to:references:date:message-id :mime-version; bh=7ttlD2/BQV3EpWrj8Few+mcd6bPps0SOY3FUt9eU1Eo=; b=Hk9cOQRbGefb8YrDE/hIacU5dT6B8kwuM1yDhr4+ZQc64B9O92GBqY9atMKy+UwSm3 z4G+E+AXfUqXAehW1oZVgpzwXZZ+Yac07gz3rmM1KZJBD0vqS+gYarOH1w/wg1yEZiwh MEf17Ezy+648yDSzJDvJhyL9s8KOfZcnKZ2GHXQ1d/v4nCzX38shEP6a8wYd4lqw4qSc usIBtx4NgHW6nfku/2nJmCvpI+arVX0L5K1/TiAM38bya80HTfocDu3ogd1EL7APp6oj b4/OrZX1QSOM54jvIqXQOOMAZDAfLCWSE7JQOf2whOr0Sm4rQh04fM8KL/uol7f59r6E /A5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:in-reply-to:references:date :message-id:mime-version; bh=7ttlD2/BQV3EpWrj8Few+mcd6bPps0SOY3FUt9eU1Eo=; b=ISg+qUKhQpCqtQTZOl96lpF/dmYSyZpxrFYnMCNQg46BMdKI7KWdowE4WH3J57qlEF ubUF05vB1G9jepnW8dVd4zjFzyheWlHZQ+PEbjeUXqtT7aoILb+IR55LHI5KjQD5PHON fKP+7t28IyV64l51Fvs7zg2ZlGvGjPO9YYgKr7DCUAUrAs4T7/2prls3BPBR+QFfNPLa 5p1wnidDq7GjY7EvYHxs12O3K0G799YU7GbkluYckIzhxrZzMrfSbTlwI6oWD4VFwasI HCcDhaSnklF/E3E5U37bHzw6bUehoqWezsZBJKDqwRZHs7KlplGubbtaVdaYQDfDSVm5 1zKA== X-Gm-Message-State: AOAM532zI7oEacrIfD96NhKBwdfxMQZX+JBT0EJ44JwdGp97pUUvC9yL 2cOXxf14tKg/ox8aR7Qk1jrchCCqHGs= X-Google-Smtp-Source: ABdhPJyeJHrqzvEbN13ejwjU7a7sKCNu+LvKn44uypRZ07WQDXKBG641zixI8PT0MGKm+yTn1Ns1Ng== X-Received: by 2002:adf:e883:: with SMTP id d3mr19646300wrm.274.1592055420700; Sat, 13 Jun 2020 06:37:00 -0700 (PDT) Received: from lili (57.246.195.77.rev.sfr.net. [77.195.246.57]) by smtp.gmail.com with ESMTPSA id h27sm17416598wrb.18.2020.06.13.06.36.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 13 Jun 2020 06:37:00 -0700 (PDT) From: zimoun <zimon.toutoune@HIDDEN> To: Arun Isaac <arunisaac@HIDDEN>, Ludovic =?utf-8?Q?Court=C3=A8?= =?utf-8?Q?s?= <ludo@HIDDEN> Subject: Re: [bug#39258] [PATCH 2/4] ui: Use string matching with literal search strings. In-Reply-To: <cu7bllnnnty.fsf@HIDDEN> References: <20200601000030.7443-1-arunisaac@HIDDEN> <20200601000030.7443-3-arunisaac@HIDDEN> <87y2owmyf3.fsf@HIDDEN> <cu7bllnnnty.fsf@HIDDEN> Date: Sat, 13 Jun 2020 15:36:57 +0200 Message-ID: <865zbvjddy.fsf@HIDDEN> MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 39258 Cc: 39258 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) Dear Arun, On Sat, 13 Jun 2020 at 18:07, Arun Isaac <arunisaac@HIDDEN> wrote: >> How does this affect performance? > > See my results from earlier. > > --8<---------------cut here---------------start------------->8--- > time ./pre-inst-env guix search game > > real 0m2.261s > user 0m2.351s > sys 0m0.104s > --8<---------------cut here---------------end--------------->8--- > > --8<---------------cut here---------------start------------->8--- > time guix search game > > real 0m2.661s > user 0m2.843s > sys 0m0.080s > --8<---------------cut here---------------end--------------->8--- I confirm that it changes nothing. See [1]. 1: http://issues.guix.gnu.org/39258#112 On the other hand, on my machine I do not see any timing improvement with this patch set. Could you check that you are comparing apple to apple? All the best, simon
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 13 Jun 2020 12:37:53 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Sat Jun 13 08:37:53 2020 Received: from localhost ([127.0.0.1]:41026 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1jk5Q1-0008Bs-Mp for submit <at> debbugs.gnu.org; Sat, 13 Jun 2020 08:37:53 -0400 Received: from mugam.systemreboot.net ([139.59.75.54]:45602) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <arunisaac@HIDDEN>) id 1jk5Pz-0008Bj-PU for 39258 <at> debbugs.gnu.org; Sat, 13 Jun 2020 08:37:53 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=systemreboot.net; s=default; h=Content-Type:MIME-Version:Message-ID:Date: References:In-Reply-To:Subject:Cc:To:From:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=Jbx0SkXo5juyhA0p2Y83NIpbe/dmAZZtSF666i/IlDI=; b=EGiD9b4o8ufklGEMbvzfKgXPb qtJMw6q0ByXsDO6fPdatZ2U0vAJQzWZAIIX+mi+LKc9TtTtTPVMXjZWATZi2c7ENJ0YNmCFr+k5TR Pa/E6CCYIviXeauV0Qhyxg9pzI6VWIlu2WDyym8554GhXXf0uYsUIN9D2KhGwd4/Dw8I4=; Received: from [192.168.2.1] (helo=steel) by systemreboot.net with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.93) (envelope-from <arunisaac@HIDDEN>) id 1jk5Pu-001LTR-23; Sat, 13 Jun 2020 18:07:46 +0530 From: Arun Isaac <arunisaac@HIDDEN> To: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN> Subject: Re: [bug#39258] [PATCH 2/4] ui: Use string matching with literal search strings. In-Reply-To: <87y2owmyf3.fsf@HIDDEN> References: <20200601000030.7443-1-arunisaac@HIDDEN> <20200601000030.7443-3-arunisaac@HIDDEN> <87y2owmyf3.fsf@HIDDEN> Date: Sat, 13 Jun 2020 18:07:45 +0530 Message-ID: <cu7bllnnnty.fsf@HIDDEN> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 39258 Cc: 39258 <at> debbugs.gnu.org, zimoun <zimon.toutoune@HIDDEN> X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain >> * guix/scripts/package.scm (process-query): Make search query a regexp only if >> it is not a literal search string. >> * guix/ui.scm (relevance): Use string matching with literal search strings and >> regexp matching with regexp search strings. > > How does this affect performance? See my results from earlier. --8<---------------cut here---------------start------------->8--- time ./pre-inst-env guix search game real 0m2.261s user 0m2.351s sys 0m0.104s --8<---------------cut here---------------end--------------->8--- --8<---------------cut here---------------start------------->8--- time guix search game real 0m2.661s user 0m2.843s sys 0m0.080s --8<---------------cut here---------------end--------------->8--- > I would expect the regexp engine in libc to do something similar > internally, so I wonder if the extra work in Scheme pays off. I agree it would better to do this optimization at the regexp engine, if it doesn't do it already. So, shall I push the remaining patches (patches 1, 3, 4) after applying the change you suggested for patch 1 (use of if versus cond)? --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEf3MDQ/Lwnzx3v3nTLiXui2GAK7MFAl7kyJkACgkQLiXui2GA K7NJfwf/TPGvR5TUzosNFh3QFfh1xG4+FXINh28h/djH2E4fQPqsTJs9ohGyD0Re uA9TszCd/BBAKUNU4sMUQv+ucw6/37jVqAqTaP9XPM4UmyhcUwyMmoJ6GJJB+Y2f ndG7NkrF3bcQfXm9ZzGtj69zaJ+G/hQWmczO2C3us0odx8t+FHcUp79nWLpPweCX QY3Qa2e7vjAMWvQSqWGl1qc6Z089LY6GGy/KSYceFhrDV6gwrMyAEpvZrgzqpeLb CI97JXG/o7q32YKgYmxPiyjEqG2hMirnCc9Cudoqak218LG7++yR60Xy1K6cEDQ8 6QR82b2KpwbwSaZtgYsAQzfc6dSafA== =brgh -----END PGP SIGNATURE----- --=-=-=--
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 9 Jun 2020 09:56:17 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Tue Jun 09 05:56:16 2020 Received: from localhost ([127.0.0.1]:58976 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1jiazQ-0001MM-Kn for submit <at> debbugs.gnu.org; Tue, 09 Jun 2020 05:56:16 -0400 Received: from mail-qk1-f195.google.com ([209.85.222.195]:44139) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <zimon.toutoune@HIDDEN>) id 1jiazO-0001M9-S4 for 39258 <at> debbugs.gnu.org; Tue, 09 Jun 2020 05:56:15 -0400 Received: by mail-qk1-f195.google.com with SMTP id c14so20141454qka.11 for <39258 <at> debbugs.gnu.org>; Tue, 09 Jun 2020 02:56:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=+F4FQojwOqaYCLYgf/1f+cC6vScgmEvK0R/7Zm/a8KM=; b=WfahGFvkIE8b7X2xzKfUu4hM9V5rPB0QqvkMHgF8UoHlFNypePf8rYHEz0/vpl2tQc MJynu6Y0hbkcLG7SkjUSyhasV6YP5sLC8YaPWJja7JidJC7B2uPt2BObdQD7qMi40W27 4ITulrdOQkUxJIK3kmUlkOlqv1vlD4t7P/ACAd19K5E0TuLJv5IWH3c0Psnl7FEThLbo RlBL0JcZytQkFWkH3BN5wp9aIJaRUAq3dH4qAA/M2IN4ZDcQwX96I53HtVIEh5O3LDHu U+Bv3XW24DwkUYKgfUjzbY4oQkpd8DmOrpSQnumaVgZm3SABF7MTHEKxONtPNaMh82QB gO8A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=+F4FQojwOqaYCLYgf/1f+cC6vScgmEvK0R/7Zm/a8KM=; b=kMVnb3lHW2EQZ1iqhkaD3kWulxbEn6c31xfTuRY+yz2UiIeYjQxXO71Eovb2sOaNrq RdO10rjzYV0HLqfJgwCEugN1IUVf4JAw3zZG9L7CDAwlnEOnkiw3upvoUjWLGmpXD+uH Cl42aMio1qUwlb4VGBvqHuCNrHQO8+NMZf/8OP5uoiRq/RT8UkKIiLCyWvURJJRXQ6t2 tPjJOBrSfwEYtFV4uP11YnaBi8H7HaGaCCPPvGZO0KhZpLRcc2XnVnNOoCCO9sZgn86V +aDV5lYLZlMOX/MkXw/MLuwgqdQFsVNDKeFsbVsicSNDys4ImF6cFVWBnSFPZhMs62pv Rh8A== X-Gm-Message-State: AOAM5326BhZ7kdqMKRixQ3orhYyENlH4GUhM+fdLANV9xbSW7jEUMMBa bjXY9ArXC3HukH+myMtjqLD0gqAihdoR1LBWrNDTWeMS X-Google-Smtp-Source: ABdhPJycU0Du+2YzmO31QNjOK9UYsKlA2YO1XatjjLJ+83I0GzLbOKyVQUysbphiRewpIJogYTsCZauimjFeVOU3IGc= X-Received: by 2002:a37:4656:: with SMTP id t83mr26625709qka.126.1591696568243; Tue, 09 Jun 2020 02:56:08 -0700 (PDT) MIME-Version: 1.0 References: <20200601000030.7443-1-arunisaac@HIDDEN> <20200601000030.7443-3-arunisaac@HIDDEN> <87y2owmyf3.fsf@HIDDEN> In-Reply-To: <87y2owmyf3.fsf@HIDDEN> From: zimoun <zimon.toutoune@HIDDEN> Date: Tue, 9 Jun 2020 11:55:56 +0200 Message-ID: <CAJ3okZ2anpq-YdTuRgxyu=yF97VgsEcCqxs=Ur0ZExdLw07auQ@HIDDEN> Subject: Re: [bug#39258] [PATCH 2/4] ui: Use string matching with literal search strings. To: =?UTF-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 39258 Cc: Arun Isaac <arunisaac@HIDDEN>, 39258 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) On Tue, 9 Jun 2020 at 10:34, Ludovic Court=C3=A8s <ludo@HIDDEN> wrote: > Arun Isaac <arunisaac@HIDDEN> skribis: > > > * guix/scripts/package.scm (process-query): Make search query a regexp = only if > > it is not a literal search string. > > * guix/ui.scm (relevance): Use string matching with literal search stri= ngs and > > regexp matching with regexp search strings. > > How does this affect performance? On my machine, it changes nothing. Even, I have applied the patches of the serie one by one to see the effect on timing and I do not see an improvement. Below an email that I started but never completed. :-) However, it seems to be The Right Thing to do. :-) All the best, simon -- Here a quick benchmark. Because once reading the code, I was not convinced by the improvement. :-) About the cut-off, the optimization should be hard to see because the bottleneck is elsewhere. And I was doubtful about the string literal but who knows. :-) And to compare apple to apple, the patch set is rebased onto a357849f5b as all the others. Warm the cache is done by "guix search foo". * Cut-off [PATCH 1/4] The first patch: cut off i.e., finer implementation of '(map regexp->score regexps)'. ** Query: crypto library The query used is: guix search crypto library | recsel -P name | grep libb2 | cache | default | v5 | |-------+----------+----------| | cold | 0m2.083s | 0m2.292s | | warm | 0m1.404s | 0m1.470s | And for another data point on the same query, see [1]: | time | default | |------+----------| | real | 0m2.216s | cold |------+----------| | real | 0m1.197s | warm [1] http://issues.guix.gnu.org/issue/39258#78 ** Query: strategy game Using the query: guix search strategy game | recsel -P name | grep julius | cache | default | v5 | |-------+----------+----------| | cold | 0m2.006s | 0m2.165s | | warm | 0m1.253s | 0m1.081s | * String literal [PATCH 2/4] (+cut-off) | cache | strategy game | crypto library | |-------+---------------+----------------| | cold | 0m2.110s | 0m2.246s | | warm | 0m1.058s | 0m1.217s |
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 9 Jun 2020 08:34:48 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Tue Jun 09 04:34:48 2020 Received: from localhost ([127.0.0.1]:58872 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1jiZia-0005h4-7J for submit <at> debbugs.gnu.org; Tue, 09 Jun 2020 04:34:48 -0400 Received: from eggs.gnu.org ([209.51.188.92]:39394) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <ludo@HIDDEN>) id 1jiZiY-0005gs-L7 for 39258 <at> debbugs.gnu.org; Tue, 09 Jun 2020 04:34:47 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:37344) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from <ludo@HIDDEN>) id 1jiZiS-00017w-R8; Tue, 09 Jun 2020 04:34:40 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=59248 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from <ludo@HIDDEN>) id 1jiZiR-00063x-Om; Tue, 09 Jun 2020 04:34:40 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN> To: Arun Isaac <arunisaac@HIDDEN> Subject: Re: [bug#39258] [PATCH 4/4] ui: Use package-description-string. References: <20200601000030.7443-1-arunisaac@HIDDEN> <20200601000030.7443-5-arunisaac@HIDDEN> Date: Tue, 09 Jun 2020 10:34:36 +0200 In-Reply-To: <20200601000030.7443-5-arunisaac@HIDDEN> (Arun Isaac's message of "Mon, 1 Jun 2020 05:30:30 +0530") Message-ID: <87pna8mycj.fsf@HIDDEN> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 39258 Cc: 39258 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -3.3 (---) Arun Isaac <arunisaac@HIDDEN> skribis: > * guix/ui.scm (package->recutils): Use package-description-string instead of > package-description and P_. LGTM, thank you!
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 9 Jun 2020 08:34:08 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Tue Jun 09 04:34:08 2020 Received: from localhost ([127.0.0.1]:58869 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1jiZhv-0005gB-Vv for submit <at> debbugs.gnu.org; Tue, 09 Jun 2020 04:34:08 -0400 Received: from eggs.gnu.org ([209.51.188.92]:39312) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <ludo@HIDDEN>) id 1jiZhv-0005fr-9u for 39258 <at> debbugs.gnu.org; Tue, 09 Jun 2020 04:34:07 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:37331) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from <ludo@HIDDEN>) id 1jiZhp-000144-6u; Tue, 09 Jun 2020 04:34:01 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=59246 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from <ludo@HIDDEN>) id 1jiZhm-00029l-VM; Tue, 09 Jun 2020 04:34:00 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN> To: Arun Isaac <arunisaac@HIDDEN> Subject: Re: [bug#39258] [PATCH 3/4] ui: Do not translate package synopsis a second time. References: <20200601000030.7443-1-arunisaac@HIDDEN> <20200601000030.7443-4-arunisaac@HIDDEN> Date: Tue, 09 Jun 2020 10:33:57 +0200 In-Reply-To: <20200601000030.7443-4-arunisaac@HIDDEN> (Arun Isaac's message of "Mon, 1 Jun 2020 05:30:29 +0530") Message-ID: <87tuzkmydm.fsf@HIDDEN> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 39258 Cc: 39258 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -3.3 (---) Arun Isaac <arunisaac@HIDDEN> skribis: > * guix/ui.scm (package->recutils): package-synopsis-string already returns a > translated string. Do not attempt to translate it again. LGTM!
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 9 Jun 2020 08:33:16 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Tue Jun 09 04:33:15 2020 Received: from localhost ([127.0.0.1]:58865 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1jiZh5-0005eY-O9 for submit <at> debbugs.gnu.org; Tue, 09 Jun 2020 04:33:15 -0400 Received: from eggs.gnu.org ([209.51.188.92]:39204) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <ludo@HIDDEN>) id 1jiZh4-0005eM-I7 for 39258 <at> debbugs.gnu.org; Tue, 09 Jun 2020 04:33:15 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:37317) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from <ludo@HIDDEN>) id 1jiZgy-0000ye-Hn; Tue, 09 Jun 2020 04:33:08 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=59236 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from <ludo@HIDDEN>) id 1jiZgv-0006es-Je; Tue, 09 Jun 2020 04:33:06 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN> To: Arun Isaac <arunisaac@HIDDEN> Subject: Re: [bug#39258] [PATCH 2/4] ui: Use string matching with literal search strings. References: <20200601000030.7443-1-arunisaac@HIDDEN> <20200601000030.7443-3-arunisaac@HIDDEN> Date: Tue, 09 Jun 2020 10:33:04 +0200 In-Reply-To: <20200601000030.7443-3-arunisaac@HIDDEN> (Arun Isaac's message of "Mon, 1 Jun 2020 05:30:28 +0530") Message-ID: <87y2owmyf3.fsf@HIDDEN> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 39258 Cc: 39258 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -3.3 (---) Arun Isaac <arunisaac@HIDDEN> skribis: > * guix/scripts/package.scm (process-query): Make search query a regexp on= ly if > it is not a literal search string. > * guix/ui.scm (relevance): Use string matching with literal search string= s and > regexp matching with regexp search strings. How does this affect performance? I would expect the regexp engine in libc to do something similar internally, so I wonder if the extra work in Scheme pays off. Ludo=E2=80=99.
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 9 Jun 2020 08:29:25 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Tue Jun 09 04:29:25 2020 Received: from localhost ([127.0.0.1]:58861 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1jiZdN-0005XT-6v for submit <at> debbugs.gnu.org; Tue, 09 Jun 2020 04:29:25 -0400 Received: from eggs.gnu.org ([209.51.188.92]:38846) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <ludo@HIDDEN>) id 1jiZdL-0005XF-JB for 39258 <at> debbugs.gnu.org; Tue, 09 Jun 2020 04:29:24 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:37266) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from <ludo@HIDDEN>) id 1jiZdF-0000J4-HC; Tue, 09 Jun 2020 04:29:17 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=59226 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from <ludo@HIDDEN>) id 1jiZdE-0001rU-MN; Tue, 09 Jun 2020 04:29:17 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN> To: Arun Isaac <arunisaac@HIDDEN> Subject: Re: [bug#39258] [PATCH 1/4] ui: Cut off search early if any regexp does not match. References: <20200601000030.7443-1-arunisaac@HIDDEN> <20200601000030.7443-2-arunisaac@HIDDEN> Date: Tue, 09 Jun 2020 10:29:14 +0200 In-Reply-To: <20200601000030.7443-2-arunisaac@HIDDEN> (Arun Isaac's message of "Mon, 1 Jun 2020 05:30:27 +0530") Message-ID: <874krkod5x.fsf@HIDDEN> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 39258 Cc: 39258 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -3.3 (---) Hi Arun, Arun Isaac <arunisaac@HIDDEN> skribis: > * guix/ui.scm (relevance): When one of the regexps does not match, cut off > early and return 0. Do not try to match the remaining regexps. Good catch, LGTM! > diff --git a/guix/ui.scm b/guix/ui.scm > index ea5f460865..4a22358963 100644 > --- a/guix/ui.scm > +++ b/guix/ui.scm > @@ -1519,11 +1519,16 @@ score, the more relevant OBJ is to REGEXPS." > (+ relevance (* weight (apply + (map score-regexp ls= t))))))))) > 0 metrics))) >=20=20 > - (let ((scores (map regexp->score regexps))) > - ;; Return zero if one of REGEXPS doesn't match. > - (if (any zero? scores) > - 0 > - (reduce + 0 scores)))) > + (let loop ((regexps regexps) > + (total-score 0)) > + (match regexps > + ((head . tail) > + (let ((score (regexp->score head))) > + ;; Return zero if one of PATTERNS doesn't match. > + (cond > + ((zero? score) 0) > + (else (loop tail (+ total-score score)))))) You can use =E2=80=98if=E2=80=99 since there are only two arms. Thanks, Ludo=E2=80=99.
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 2 Jun 2020 08:49:21 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Tue Jun 02 04:49:21 2020 Received: from localhost ([127.0.0.1]:37851 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1jg2bp-0001kv-5c for submit <at> debbugs.gnu.org; Tue, 02 Jun 2020 04:49:21 -0400 Received: from eggs.gnu.org ([209.51.188.92]:57592) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <ludo@HIDDEN>) id 1jg2bn-0001ki-Et for 39258 <at> debbugs.gnu.org; Tue, 02 Jun 2020 04:49:19 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:40675) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from <ludo@HIDDEN>) id 1jg2bh-0004Yc-Cm; Tue, 02 Jun 2020 04:49:13 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=33286 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from <ludo@HIDDEN>) id 1jg2bg-0001JT-UB; Tue, 02 Jun 2020 04:49:13 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN> To: Arun Isaac <arunisaac@HIDDEN> Subject: Re: [bug#39258] KMP string search algorithm? References: <cu7pnfaar36.fsf@HIDDEN> <CAJ3okZ2ZNYaH=_OjP2Axvq-anY6Q9CkVmjUVLPx-Z6+kZTVsdg@HIDDEN> <20200601222414.GA30829@HIDDEN> <cu7v9kaxs7w.fsf@HIDDEN> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 15 Prairial an 228 de la =?utf-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Tue, 02 Jun 2020 10:49:10 +0200 In-Reply-To: <cu7v9kaxs7w.fsf@HIDDEN> (Arun Isaac's message of "Tue, 02 Jun 2020 05:18:51 +0530") Message-ID: <87tuztvomx.fsf@HIDDEN> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 39258 Cc: zimoun <zimon.toutoune@HIDDEN>, 39258 <at> debbugs.gnu.org, Leo Famulari <leo@HIDDEN> X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -3.3 (---) Arun Isaac <arunisaac@HIDDEN> skribis: >> I also recommend taking a look at the Boyer Moore string search >> implementation in (guix build grafts). > > Nice, I didn't know Guix had an implementation of Boyer Moore. I'll take > a look at it. At the very least, I need something similar for > guile-email. > > But, the current implementation of guile's string-contains is in C. So, > I assume a KMP or Boyer Moore implementation of string-contains should > also be in C. Not necessarily. But it=E2=80=99d be great to have it in Guile proper, for= sure! Ludo=E2=80=99.
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 1 Jun 2020 23:49:08 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Mon Jun 01 19:49:08 2020 Received: from localhost ([127.0.0.1]:37323 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1jfuB2-0005L1-7n for submit <at> debbugs.gnu.org; Mon, 01 Jun 2020 19:49:08 -0400 Received: from mugam.systemreboot.net ([139.59.75.54]:37964) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <arunisaac@HIDDEN>) id 1jfuAy-0005Kr-Jg for 39258 <at> debbugs.gnu.org; Mon, 01 Jun 2020 19:49:06 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=systemreboot.net; s=default; h=Content-Type:MIME-Version:Message-ID:Date: References:In-Reply-To:Subject:Cc:To:From:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=5EWaSsbvyl6wfysRyzD7qfnslHeIEAihke5LVM7WA3Q=; b=Tj/GKMFIp5rgrhXpo6GqX/R79 Eda9KGuYooK/4XuqRhTiUQKHsNNeRnAxvX6OgmgRYrwBUqRa078M6xBRop8E3GVH0uDGklSx7LUPG 1908OYAdn/QJII+GRSJ9VeWtH3FWNe4rfXti9LFA7H5+GxGI2BwSxyKtdiuTIJtTe9x18=; Received: from [192.168.2.1] (helo=steel) by systemreboot.net with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.93) (envelope-from <arunisaac@HIDDEN>) id 1jfuAs-000fUF-Ai; Tue, 02 Jun 2020 05:18:58 +0530 From: Arun Isaac <arunisaac@HIDDEN> To: Leo Famulari <leo@HIDDEN>, zimoun <zimon.toutoune@HIDDEN> Subject: Re: [bug#39258] KMP string search algorithm? In-Reply-To: <20200601222414.GA30829@HIDDEN> References: <cu7pnfaar36.fsf@HIDDEN> <CAJ3okZ2ZNYaH=_OjP2Axvq-anY6Q9CkVmjUVLPx-Z6+kZTVsdg@HIDDEN> <20200601222414.GA30829@HIDDEN> Date: Tue, 02 Jun 2020 05:18:51 +0530 Message-ID: <cu7v9kaxs7w.fsf@HIDDEN> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 39258 Cc: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>, 39258 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain > I also recommend taking a look at the Boyer Moore string search > implementation in (guix build grafts). Nice, I didn't know Guix had an implementation of Boyer Moore. I'll take a look at it. At the very least, I need something similar for guile-email. But, the current implementation of guile's string-contains is in C. So, I assume a KMP or Boyer Moore implementation of string-contains should also be in C. --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEf3MDQ/Lwnzx3v3nTLiXui2GAK7MFAl7Vk+MACgkQLiXui2GA K7NyVAgAuwZNN+btTCt/EC/v/Eu+NsMC06cKQ634M9x6f599DjCyKDnKFLJ0cDxX l6BKGon5o4W/X6ZYltXakviRXNrmVjsDdTDZ0zFcrs0aS/XSAU2aaybGEmKhrhC1 PsDIjCThgDkIiDcdx/fLg2hMXtw882KpfPK4dBQWiTtdhJmpnioyk/tHXwLCR9v3 6k7shW5KxyM8NrLhpLPkULZbv98Dams/JnDJ16NqNN5yH+mYGRxL2RPsmngrOCPz 7a3yUsOpBwyA+ZoaviYLv7/cgfyqjmn4CwbDlyeZoSj/CyhNHGcK4pnqm2DsHF5D 8yWVzzRdBXSaVbVHbPWADMp3UuI9NQ== =tpfE -----END PGP SIGNATURE----- --=-=-=--
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 1 Jun 2020 22:24:25 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Mon Jun 01 18:24:25 2020 Received: from localhost ([127.0.0.1]:37268 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1jfsr2-0003Nf-UX for submit <at> debbugs.gnu.org; Mon, 01 Jun 2020 18:24:25 -0400 Received: from wout2-smtp.messagingengine.com ([64.147.123.25]:54225) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <leo@HIDDEN>) id 1jfsr1-0003NT-6y for 39258 <at> debbugs.gnu.org; Mon, 01 Jun 2020 18:24:23 -0400 Received: from compute2.internal (compute2.nyi.internal [10.202.2.42]) by mailout.west.internal (Postfix) with ESMTP id 55B4E9CD; Mon, 1 Jun 2020 18:24:17 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute2.internal (MEProxy); Mon, 01 Jun 2020 18:24:17 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=famulari.name; h=date:from:to:cc:subject:message-id:references:mime-version :content-type:in-reply-to; s=mesmtp; bh=rBTGbzaqNCLwuyZUt/axayMN HtAcCyX93g9xiJ3WoI4=; b=M701//WVMnOPSw/SxpKkPiXc9kawGh2u13zFz2Ci Zks3puqNz++L3YQCiUtIG479gD9sjpeHKCGk3CXppuUx3I418mug1Zp2XxfqM/ll RDV7Yh17qLgSjlS21MKRoTvEFu7loy9MCOd2mX3LjvnVgPW2JT6rrrYYdPztwwXc vhQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=rBTGbz aqNCLwuyZUt/axayMNHtAcCyX93g9xiJ3WoI4=; b=NxKDv9XSSkWWgOhlhBAK31 Zkwa0WNeUpIj0XB3XCUtRQKNjiFEWQntqiwwV32PmRHh7Bqn2x/AgOXz5yqGbYHD YKYdDm6qBQkK/g20s/CZNyBAkQb9gK7+3q14GcJm/WWfF4ygOf8HCXkujYuYWztJ OigzcDJx3fvGLkuXX+88KGyoKo5W/VypPhlD7c0U49fi36L3eu5qsiqJToOuw/u+ F2XzY+Res265f9nBtGvQXxct5jacM8Zpd520uvfOB8A4q8ScgTsPCfHPni1Y/2NB p+VzBggOwgm5NBr00Gg72nNdIcbYAYwq711KrMGWn/RKQzTTg5OcztdUGA7/b6XQ == X-ME-Sender: <xms:EIDVXmN4yHi1WW0rHlyKJtt1muPQ8KYSmMCTk9UKccmcPXc0XU_rRQ> X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduhedrudefiedgtdelucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvffukfhfgggtuggjsehttdertddttddvnecuhfhrohhmpefnvghoucfh rghmuhhlrghrihcuoehlvghosehfrghmuhhlrghrihdrnhgrmhgvqeenucggtffrrghtth gvrhhnpeektdfhtdeigeejleeiffeuuedtiedttddvhfdvhfejgfdujeeiveeifefhtdej veenucffohhmrghinhepfihikhhiphgvughirgdrohhrghenucfkphepjeeirdduvdegrd dufeekrdeifeenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhr ohhmpehlvghosehfrghmuhhlrghrihdrnhgrmhgv X-ME-Proxy: <xmx:EIDVXk9QvqYtZLnzavmVJ4_ogujDQ3iiOsAb2jmZpxis08Re9wOkDg> <xmx:EIDVXtSDY33Zxzb59nRqJJA5QFrqhFFCbsoWZQrszhWCvCU_f5bCcw> <xmx:EIDVXmsFI3Bt7Y6DibTksGrJYtTeJwxjFbaX5vbsDvtz9Su54ZS7QA> <xmx:EIDVXmnHpdpffaFGGpjGykviWFB4uYxPEGRYzYIiscogGcuy7dxHyg> Received: from localhost (c-76-124-138-63.hsd1.pa.comcast.net [76.124.138.63]) by mail.messagingengine.com (Postfix) with ESMTPA id 64B2B30624CC; Mon, 1 Jun 2020 18:24:16 -0400 (EDT) Date: Mon, 1 Jun 2020 18:24:14 -0400 From: Leo Famulari <leo@HIDDEN> To: zimoun <zimon.toutoune@HIDDEN> Subject: Re: [bug#39258] KMP string search algorithm? Message-ID: <20200601222414.GA30829@HIDDEN> References: <cu7pnfaar36.fsf@HIDDEN> <CAJ3okZ2ZNYaH=_OjP2Axvq-anY6Q9CkVmjUVLPx-Z6+kZTVsdg@HIDDEN> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <CAJ3okZ2ZNYaH=_OjP2Axvq-anY6Q9CkVmjUVLPx-Z6+kZTVsdg@HIDDEN> X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 39258 Cc: Arun Isaac <arunisaac@HIDDEN>, Ludovic =?iso-8859-1?Q?Court=E8s?= <ludo@HIDDEN>, 39258 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.7 (-) On Mon, Jun 01, 2020 at 12:11:52PM +0200, zimoun wrote: > Dear, > > > > Often our search strings are only literal strings. So, we can save some time > > > by using string-contains instead of invoking the regexp engine. Patch 2 does > > > this. In addition, guile's string-contains uses a naive O(n^2) string search > > > algorithm. We should perhaps use the O(n) Knuth-Morris-Pratt algorithm[1]. In > > > fact, a comment on line 2006 of libguile/srfi-13.c in the guile source code > > > mentions this. If implemented, the KMP algorithm could speed up guix search > > > further. > > > > > > [1]: https://en.wikipedia.org/wiki/Knuth%E2%80%93Morris%E2%80%93Pratt_algorithm > > It could improve. > Well, I will try to do some back-to-envelop computations because I am > not convinced that the mean value of 'n' (length of description, > isn't) is large enough to really see an improvement for the end-user; > the visible bottleneck is I/O. > > All the best, > simon > > ps; > To be honest, I thought this kind of algorithm was the default. :-) I also recommend taking a look at the Boyer Moore string search implementation in (guix build grafts). It would be great to generalize it and make it accessible to other parts of Guix.
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 1 Jun 2020 10:12:10 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Mon Jun 01 06:12:10 2020 Received: from localhost ([127.0.0.1]:34570 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1jfhQQ-0005rF-IW for submit <at> debbugs.gnu.org; Mon, 01 Jun 2020 06:12:10 -0400 Received: from mail-qk1-f172.google.com ([209.85.222.172]:46378) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <zimon.toutoune@HIDDEN>) id 1jfhQP-0005r2-5T for 39258 <at> debbugs.gnu.org; Mon, 01 Jun 2020 06:12:09 -0400 Received: by mail-qk1-f172.google.com with SMTP id c12so8418191qkk.13 for <39258 <at> debbugs.gnu.org>; Mon, 01 Jun 2020 03:12:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=ohMequnGklGeHnzbOaO2DgtPJJxLJ/i5TQ861Q5o4rw=; b=sYZC8hSuoA/IQj1T2gj6WDf2xFBImPabh9+WSiVJDtBWdce1so2eidqsnIuqBKJLkH qrgCLtn2NS30j8w+PnVipnTXJoNaBNPM/MWn2fOzLhQpbH0X9WRbArfeMFoPv740KhGc SaKMbxNvG5llytHf6IlGqmBTPlm27MB9vdYyWzH23rpRkKi8jZXcRkOMSj7O13JE7TNZ V3LfzbkkWqFzZocfzXtajL1+2d0NGwTCqfTHMASQpnd8pPHRe/WZTaEPQi3zImwmc9i3 /syC5CkZAKX3TO/u5bptEuyhX9NstoDxdzBAjiXYMMqcwWo5S1jKduqb0cm/5I1OYhpK llgA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=ohMequnGklGeHnzbOaO2DgtPJJxLJ/i5TQ861Q5o4rw=; b=DIDQkEHtxBo2ShTd0ti0OMfs7wsd9FjCt00H/X4EizEA2yWqgvhcyr/nvYqO0S6gPV Lb3ds2Tk5E+CcO/rvMt45MJoqspXR6XJpY8S762kRPgCYT4cKOlJ5dJIsMgbFUN/Ehg3 Gfu7ye3KmEKtabSesaMSArEmIxM7TX3rzAHou+Mm5oq/9sdMzWOo5P91dpY8PVm6SQc9 Me6dTf5+MwEOUFMcd+joSDaKlH0NJD75gA+MAntsoIVX3k3B9znNR4wH1u+UMWBcx4bI uc5ALgi6oPnmVJFy1CVIbsdnddeFkjLEmDCev7JA6XXUA5tCQkUawgH/u+4Wlg4h0TRp 93hQ== X-Gm-Message-State: AOAM530GHEOZCXOK78D8MqXiNgJ+H3sd0goL6Kff1OtkqcbswYTnLrLV ee267dN0zxC9hkGx0KdScp6ydvWBABxEpT0JuA8= X-Google-Smtp-Source: ABdhPJzhZmhhKZY/wfP4TrxB8xe9aT0UER/7zRJ7FPXsUKBGtdD3C7S3za1n4vLKogKD7L1kADZczVfiBxMrxJGGmak= X-Received: by 2002:a05:620a:148a:: with SMTP id w10mr17287713qkj.201.1591006323659; Mon, 01 Jun 2020 03:12:03 -0700 (PDT) MIME-Version: 1.0 From: zimoun <zimon.toutoune@HIDDEN> Date: Mon, 1 Jun 2020 12:11:52 +0200 Message-ID: <CAJ3okZ2ZNYaH=_OjP2Axvq-anY6Q9CkVmjUVLPx-Z6+kZTVsdg@HIDDEN> Subject: KMP string search algorithm? To: Arun Isaac <arunisaac@HIDDEN>, 39258 <at> debbugs.gnu.org, =?UTF-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN> Content-Type: text/plain; charset="UTF-8" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 39258 X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) Dear, > > Often our search strings are only literal strings. So, we can save some time > > by using string-contains instead of invoking the regexp engine. Patch 2 does > > this. In addition, guile's string-contains uses a naive O(n^2) string search > > algorithm. We should perhaps use the O(n) Knuth-Morris-Pratt algorithm[1]. In > > fact, a comment on line 2006 of libguile/srfi-13.c in the guile source code > > mentions this. If implemented, the KMP algorithm could speed up guix search > > further. > > > > [1]: https://en.wikipedia.org/wiki/Knuth%E2%80%93Morris%E2%80%93Pratt_algorithm It could improve. Well, I will try to do some back-to-envelop computations because I am not convinced that the mean value of 'n' (length of description, isn't) is large enough to really see an improvement for the end-user; the visible bottleneck is I/O. All the best, simon ps; To be honest, I thought this kind of algorithm was the default. :-)
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 1 Jun 2020 10:01:30 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Mon Jun 01 06:01:30 2020 Received: from localhost ([127.0.0.1]:34544 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1jfhG5-0005bA-NK for submit <at> debbugs.gnu.org; Mon, 01 Jun 2020 06:01:29 -0400 Received: from mail-qk1-f196.google.com ([209.85.222.196]:43292) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <zimon.toutoune@HIDDEN>) id 1jfhG4-0005aw-Nk for 39258 <at> debbugs.gnu.org; Mon, 01 Jun 2020 06:01:28 -0400 Received: by mail-qk1-f196.google.com with SMTP id v79so8403918qkb.10 for <39258 <at> debbugs.gnu.org>; Mon, 01 Jun 2020 03:01:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=XfF3ze6jQOwTrgFRCjGAsHsIwUs+NmlAaQaSz/jtGyo=; b=NbORiZHjZ9izx2ui8wILGRYQucJMxMShqZ7q5Nmmzui8ZIrBHVlfPPeLubngl0e3Sg AjLA7GRfDLsZxbJLjIFsVB3YWWaNZB1IkKhSZ3AnITdCv6m47h6SlfnQPQGXcmHu75LG Q25kPamH8CYXfbUgtoKnOBRloFVk6inOOsPpxfLB+zjgnFKWTha8hikuceEa6ogU/9bK UI4z0DtLp0fDLv9imXyeLV2XEInggsZSWNyWTl4Hjzwh6JnbADH10BWZwhI3eYmBSl4C lPcwDKePYLrJHxxNk+YaZ7JJYs5m3/+Dw5t+Ac9OQPlckt47qOMRJqSU8HeVSd3gC1gh VqOw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=XfF3ze6jQOwTrgFRCjGAsHsIwUs+NmlAaQaSz/jtGyo=; b=dzNIUtVMbIYQWhnfcjqTlwLyOZQMNrYDhABt+743kvLop1JqnKLZoviydzdQvHrHtj PnebpD+udW0n+dGhr1TUtnnEdG7G6nxfKTeNBNfQiIscuiYEZMtSbF3ALVBpuMnI/0EZ 1aNXFAqiqUOOVMRgnbSlAyGHV5ix6SEyW/RWiQN0S/8giSbPjGLZlgW5UcsaHZcN8S8v vKkIPhBMJZfhlqEAxtAUhoZ9vYwpxgfwZUNbJWBlI3ActCD5bIqBv6Tnw81ef+iQTum6 0GzimXZzyBNbbn3MmY21Foy5b8YbK8uzv1BMbmDBGST/3AR8s0Kl6kONCwKVz/a/M2Ef TDfw== X-Gm-Message-State: AOAM531mh1jzUXYUvsXb6i9g4zskwwxJdh3sjtUGr3b4bRI3q0vLNOzY MWQbYNOalmJcw2MMgDfrSIsreq+LFQ592mm8FXY= X-Google-Smtp-Source: ABdhPJxe9ghLuL/Bx5nTbn9JWdWncCzwP12d7a4b1carRNuSD3KI5Nt2BZrI1Gybaiz7pJR4v9cEHZoK8TYQOoe+Nq0= X-Received: by 2002:a37:4b88:: with SMTP id y130mr18916741qka.80.1591005682486; Mon, 01 Jun 2020 03:01:22 -0700 (PDT) MIME-Version: 1.0 References: <20200601000030.7443-1-arunisaac@HIDDEN> <CAJ3okZ1aJWz9k2EwYNfPdBmKNzx-OTC27J4bg7as48qxcoLG-w@HIDDEN> In-Reply-To: <CAJ3okZ1aJWz9k2EwYNfPdBmKNzx-OTC27J4bg7as48qxcoLG-w@HIDDEN> From: zimoun <zimon.toutoune@HIDDEN> Date: Mon, 1 Jun 2020 12:01:11 +0200 Message-ID: <CAJ3okZ0A56ZciESqyFZBaonjuf1obU8GhubEGswTHZ2wAUGniQ@HIDDEN> Subject: Re: [PATCH v5 0/4] Optimize guix search To: Arun Isaac <arunisaac@HIDDEN> Content-Type: text/plain; charset="UTF-8" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 39258 Cc: =?UTF-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>, 39258 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) > > --8<---------------cut here---------------start------------->8--- > > time ./pre-inst-env guix search strategy game > > > > real 0m1.613s > > user 0m1.635s > > sys 0m0.096s > > --8<---------------cut here---------------end--------------->8--- > > > > --8<---------------cut here---------------start------------->8--- > > time guix search strategy game > > > > real 0m2.520s > > user 0m2.583s > > sys 0m0.112s > > --8<---------------cut here---------------end--------------->8--- I do not see any improvement on my machine. Well, I am double-checking because I should have screwed up something...
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 1 Jun 2020 02:24:24 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Sun May 31 22:24:24 2020 Received: from localhost ([127.0.0.1]:34219 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1jfa7k-00072d-IH for submit <at> debbugs.gnu.org; Sun, 31 May 2020 22:24:24 -0400 Received: from mugam.systemreboot.net ([139.59.75.54]:37244) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <arunisaac@HIDDEN>) id 1jfa7i-00072U-6n for 39258 <at> debbugs.gnu.org; Sun, 31 May 2020 22:24:23 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=systemreboot.net; s=default; h=Content-Type:MIME-Version:Message-ID:Date: References:In-Reply-To:Subject:Cc:To:From:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=1ha3jJgzoSHBpjspjOQU0K51VBAgR/czrJ+U+ypaXm0=; b=huoYEJ1gljE6k0mtfvGEUQwo3 FJftBEX6vkMwsugnqTVGQEIsqkPRkVVtimBYTuDtuaCb/VLaWtI6SUBIVAvC+wL5S9QnzNKE8j9dB 6TpJaGZmcIX15jRyncInG1VcSGKC/SppH6Bult85S/3rO5zl8RQsKjf1NSNbSZdMU6flI=; Received: from [192.168.2.1] (helo=steel) by systemreboot.net with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.93) (envelope-from <arunisaac@HIDDEN>) id 1jfa7c-000aFg-27; Mon, 01 Jun 2020 07:54:16 +0530 From: Arun Isaac <arunisaac@HIDDEN> To: zimoun <zimon.toutoune@HIDDEN> Subject: Re: [PATCH v5 0/4] Optimize guix search In-Reply-To: <CAJ3okZ1aJWz9k2EwYNfPdBmKNzx-OTC27J4bg7as48qxcoLG-w@HIDDEN> References: <20200601000030.7443-1-arunisaac@HIDDEN> <CAJ3okZ1aJWz9k2EwYNfPdBmKNzx-OTC27J4bg7as48qxcoLG-w@HIDDEN> Date: Mon, 01 Jun 2020 07:54:06 +0530 Message-ID: <cu71rmzh6bd.fsf@HIDDEN> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 39258 Cc: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>, 39258 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain > Based on the Ludo's comments [1] on v4 which is a simple re-write of > your v3, I am finishing a vN+1.. but time flies and I am late on the > topic too. :-) > > Well, this still unsent vN+1 series has the same performance of v4 on > "guix pull" which is a key point compared to v3. Obviously, the > performance on "guix search" are equivalent on both version. This > vN+1 builds two caches -- to avoid binary breakage -- in only one go; > the consuming 'fold-modules-public-variables*' is applied only once. Interesting, I'll be waiting for your patchset. :-) > [1] http://issues.guix.gnu.org/39258#93 >> Here's a rough performance comparison. > > On cold or warm cache? On a warm cache. > So in the best case, you have the ratio old/new is 1.5; this new > version is 1.5 faster. > > Well, in the extra cache approach (v3 or v4) the ration old/new is > really higher: 3.1 faster on cold cache (which is the one I am > interested in) and 2.4 faster on warm cache. We could always have both my optimizations and your improved cache. So, that's a win on both fronts. > I will give a look to this new series and report what happens on my > laptop. But basically, I would like "guix search" under the 1.0 > second on my machine. ;-) Indeed, I would love that too! :-) --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEf3MDQ/Lwnzx3v3nTLiXui2GAK7MFAl7UZscACgkQLiXui2GA K7NfHAgArXRLVjefvVH9W2bDlquAKl4xoeYClxmrU7wSF/tvo+XjrLDmI7yiXmwG kIQTPtbivZDVWoyr/ReL2MoaZSoXLdf1rLu/ta7AXlQsnWQCblBJApZ6xoh7wPv6 7SecBmC3MA540tAo5Zxy1q8e9UkzqSADevVlklW+dEyEH/BgwdNOsThUDA7DPnuN pGkz+Ef7p9gzITu9Q9j6yRz4s1EpFLMs4c5VfmnudvFVvhXgbwrxBhY2cyvOnOi5 bz0wjKb0nzezOrj838NHrSPRyunEzv1OuD1YwfG5XFC/OCttuirBZmQc015HxFH+ Mqgj1F2hxaOJ/PK2mrXq8VcSTT0W7Q== =J1FC -----END PGP SIGNATURE----- --=-=-=--
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 1 Jun 2020 01:25:28 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Sun May 31 21:25:28 2020 Received: from localhost ([127.0.0.1]:34190 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1jfZCh-0005eP-Tj for submit <at> debbugs.gnu.org; Sun, 31 May 2020 21:25:28 -0400 Received: from mail-qt1-f193.google.com ([209.85.160.193]:39444) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <zimon.toutoune@HIDDEN>) id 1jfZCg-0005e9-MH for 39258 <at> debbugs.gnu.org; Sun, 31 May 2020 21:25:27 -0400 Received: by mail-qt1-f193.google.com with SMTP id k22so6551607qtm.6 for <39258 <at> debbugs.gnu.org>; Sun, 31 May 2020 18:25:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=/gbfEJHcOpkZHNmmRKEOTH6rOPp5GZXuXF3+L3Pe7W4=; b=RJVpNU9buXibDcBj67narNogTWh01MzOur+jAvsaj4KC/NNGcSl9grVoc6y5RRRb7e fPlff1++tSGa79Yos+N0ViBFaSGLvps9twGpqG+8mdBkTKKQ/rn1Qu+uvEjlKMp71zNs dxzaURhDOpVjkPRgojzlNK5T1Kl3n5v+lMCKKtbv4Mt01z8dkEha+TA5hUpwj6a5PqYq EO0LKzNsUhgK8kcSwFgzW5t4ieVUkT9kDh06A9gBXo/Lq+A96EmCO7copG+vzrzM0OXn E5VILhL+Xk/ffd8BiBy82fUOyhT3UyY+E9jfnhpM5zOSALrzrKqEIMABZlQpATapYtBa sdaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=/gbfEJHcOpkZHNmmRKEOTH6rOPp5GZXuXF3+L3Pe7W4=; b=Acw21+JJ9HCUfRptyDhuz2T2Oxw8YNJyfWIFdbtbPB+y4+L0yuxCS1xVjd0FJ3W1+D FxCFslVm2O4oFmgOzTsnqQM9MMBWiwXIAmQxca6UCgHfu/ufoj8mcDT3WN8dSycFtlwK 5UFcHyfOrIFYsggH3gQU577r8jwuOWLzHHgze3H5nfxI40P9m6BkG5TXK9XcPjXEJd5R H/KDs0DGwAKHm0Tk2LEwBLDJvUI9BQBMVgsmWIVmxjT/+kQFwCz1IkOVVhX0f20UtzN1 lP73kjKbkkD36Wq3U1smim++dsqLCtAfZZ2peZgRLt7hb+/YAW/WBVbXMd+5ZwJIFFwp cGyA== X-Gm-Message-State: AOAM530IFsp6097852P9mPEz6MKRKaYktKP5RmQPcHM0paAMc/gElxBy ZRXfB5oq1diwehQ9Rx1uBO/RMIW3IYtPDdJrf0I= X-Google-Smtp-Source: ABdhPJx8NosTpnZl78ZOtvdCZ9rAxvg+5jRMkVvoQL5iAPDN7bnCmA6VMBsEXBJfTEJaXOGWbSwUIl1SfZf+TimgxjQ= X-Received: by 2002:aed:3169:: with SMTP id 96mr18719772qtg.211.1590974720770; Sun, 31 May 2020 18:25:20 -0700 (PDT) MIME-Version: 1.0 References: <20200601000030.7443-1-arunisaac@HIDDEN> In-Reply-To: <20200601000030.7443-1-arunisaac@HIDDEN> From: zimoun <zimon.toutoune@HIDDEN> Date: Mon, 1 Jun 2020 03:25:09 +0200 Message-ID: <CAJ3okZ1aJWz9k2EwYNfPdBmKNzx-OTC27J4bg7as48qxcoLG-w@HIDDEN> Subject: Re: [PATCH v5 0/4] Optimize guix search To: Arun Isaac <arunisaac@HIDDEN> Content-Type: text/plain; charset="UTF-8" X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 39258 Cc: =?UTF-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>, 39258 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) Hi Arun, On Mon, 1 Jun 2020 at 02:00, Arun Isaac <arunisaac@HIDDEN> wrote: > Sorry for the long delay in replying to this thread. Based on the Ludo's comments [1] on v4 which is a simple re-write of your v3, I am finishing a vN+1.. but time flies and I am late on the topic too. :-) Well, this still unsent vN+1 series has the same performance of v4 on "guix pull" which is a key point compared to v3. Obviously, the performance on "guix search" are equivalent on both version. This vN+1 builds two caches -- to avoid binary breakage -- in only one go; the consuming 'fold-modules-public-variables*' is applied only once. [1] http://issues.guix.gnu.org/39258#93 > I think Ludo is right in that we can improve guix search performance with only > simple code improvements rather than including xapian or improving our > existing cache. Here are a few patches on those lines. Well, improving the cache is easy; at least as you did in v3 by adding another one. The most annoying part is the arguments rewrite of 'package->recutils' to be compliant. However after some comparisons, I am not convinced that BM25 will be worth to implement... > In `relevance`, we set our score to 0 if any of the regexps don't match. Then, > we might as well not match the remaining regexps. Patch 1 does this early cut > off optimization. Interesting. > Often our search strings are only literal strings. So, we can save some time > by using string-contains instead of invoking the regexp engine. Patch 2 does > this. In addition, guile's string-contains uses a naive O(n^2) string search > algorithm. We should perhaps use the O(n) Knuth-Morris-Pratt algorithm[1]. In > fact, a comment on line 2006 of libguile/srfi-13.c in the guile source code > mentions this. If implemented, the KMP algorithm could speed up guix search > further. > > [1]: https://en.wikipedia.org/wiki/Knuth%E2%80%93Morris%E2%80%93Pratt_algorithm Really interesting idea, > Patch 3 and 4 are minor improvements. > > Here's a rough performance comparison. On cold or warm cache? > --8<---------------cut here---------------start------------->8--- > time ./pre-inst-env guix search game > > real 0m2.261s > user 0m2.351s > sys 0m0.104s > --8<---------------cut here---------------end--------------->8--- > > --8<---------------cut here---------------start------------->8--- > time guix search game > > real 0m2.661s > user 0m2.843s > sys 0m0.080s > --8<---------------cut here---------------end--------------->8--- > > --8<---------------cut here---------------start------------->8--- > time ./pre-inst-env guix search strategy game > > real 0m1.613s > user 0m1.635s > sys 0m0.096s > --8<---------------cut here---------------end--------------->8--- > > --8<---------------cut here---------------start------------->8--- > time guix search strategy game > > real 0m2.520s > user 0m2.583s > sys 0m0.112s > --8<---------------cut here---------------end--------------->8--- So in the best case, you have the ratio old/new is 1.5; this new version is 1.5 faster. Well, in the extra cache approach (v3 or v4) the ration old/new is really higher: 3.1 faster on cold cache (which is the one I am interested in) and 2.4 faster on warm cache. I will give a look to this new series and report what happens on my laptop. But basically, I would like "guix search" under the 1.0 second on my machine. ;-) Thank you for this new input. Cheers, simon
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.
Received: (at 39258) by debbugs.gnu.org; 1 Jun 2020 00:00:41 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun May 31 20:00:41 2020
Received: from localhost ([127.0.0.1]:34108 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1jfXsf-0003Zf-85
for submit <at> debbugs.gnu.org; Sun, 31 May 2020 20:00:41 -0400
Received: from mugam.systemreboot.net ([139.59.75.54]:37186)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <arunisaac@HIDDEN>) id 1jfXsa-0003Yg-GY
for 39258 <at> debbugs.gnu.org; Sun, 31 May 2020 20:00:39 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed;
d=systemreboot.net; s=default; h=Content-Transfer-Encoding:MIME-Version:
References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To:
Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From:
Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:
List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive;
bh=qZBdE0ZfKP/ze9QPewx5RIbs99IBZQjftJB+yWVWKGg=; b=C0W7eZewjmVG5gFSq8utnD0Zt1
5YPNbzIrscLxkKSLkqx1jRj4IMPBio7Cobpp+059lffHTWocwOJWytadxjU71IJ7xxZ1RMMAZ9w8S
a0gRyX5EQ89skwtgm8MR5oXIyHxNDiT5Uppq2CV/ECvu1K/yEXa+hGK68oqM/1089otY=;
Received: from [192.168.2.1] (helo=steel.lan)
by systemreboot.net with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384
(Exim 4.93) (envelope-from <arunisaac@HIDDEN>)
id 1jfXsX-000Zk2-5b; Mon, 01 Jun 2020 05:30:33 +0530
From: Arun Isaac <arunisaac@HIDDEN>
To: 39258 <at> debbugs.gnu.org
Subject: [PATCH 2/4] ui: Use string matching with literal search strings.
Date: Mon, 1 Jun 2020 05:30:28 +0530
Message-Id: <20200601000030.7443-3-arunisaac@HIDDEN>
X-Mailer: git-send-email 2.26.2
In-Reply-To: <20200601000030.7443-1-arunisaac@HIDDEN>
References: <20200601000030.7443-1-arunisaac@HIDDEN>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 39258
Cc: Arun Isaac <arunisaac@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)
* guix/scripts/package.scm (process-query): Make search query a regexp only if
it is not a literal search string.
* guix/ui.scm (relevance): Use string matching with literal search strings and
regexp matching with regexp search strings.
---
guix/scripts/package.scm | 12 +++++++---
guix/ui.scm | 50 +++++++++++++++++++++++++---------------
2 files changed, 40 insertions(+), 22 deletions(-)
diff --git a/guix/scripts/package.scm b/guix/scripts/package.scm
index 1246147798..1b637f7802 100644
--- a/guix/scripts/package.scm
+++ b/guix/scripts/package.scm
@@ -675,6 +675,11 @@ doesn't need it."
(define (process-query opts)
"Process any query specified by OPTS. Return #t when a query was actually
processed, #f otherwise."
+ (define (regexp-pattern? str)
+ (string-any
+ (char-set #\. #\[ #\{ #\} #\( #\) #\\ #\* #\+ #\? #\| #\^ #\$)
+ str))
+
(let* ((profiles (delete-duplicates
(match (filter-map (match-lambda
(('profile . p) p)
@@ -781,11 +786,12 @@ processed, #f otherwise."
(('search _)
(let* ((patterns (filter-map (match-lambda
- (('query 'search rx) rx)
+ (('query 'search (? regexp-pattern? rx))
+ (make-regexp* rx regexp/icase))
+ (('query 'search pattern) pattern)
(_ #f))
opts))
- (regexps (map (cut make-regexp* <> regexp/icase) patterns))
- (matches (find-packages-by-description regexps)))
+ (matches (find-packages-by-description patterns)))
(leave-on-EPIPE
(display-search-results matches (current-output-port)))
#t))
diff --git a/guix/ui.scm b/guix/ui.scm
index 4a22358963..56754dba83 100644
--- a/guix/ui.scm
+++ b/guix/ui.scm
@@ -1489,41 +1489,53 @@ HYPERLINKS? is true, emit hyperlink escape sequences when appropriate."
;;; Searching.
;;;
-(define (relevance obj regexps metrics)
+(define (relevance obj patterns metrics)
"Compute a \"relevance score\" for OBJ as a function of its number of
-matches of REGEXPS and accordingly to METRICS. METRICS is list of
+matches of PATTERNS and accordingly to METRICS. METRICS is list of
field/weight pairs, where FIELD is a procedure that returns a string or list
of strings describing OBJ, and WEIGHT is a positive integer denoting the
weight of this field in the final score.
-A score of zero means that OBJ does not match any of REGEXPS. The higher the
-score, the more relevant OBJ is to REGEXPS."
- (define (score regexp str)
- (fold-matches regexp str 0
- (lambda (m score)
- (+ score
- (if (string=? (match:substring m) str)
- 5 ;exact match
- 1)))))
-
- (define (regexp->score regexp)
- (let ((score-regexp (lambda (str) (score regexp str))))
+A score of zero means that OBJ does not match any of PATTERNS. The higher the
+score, the more relevant OBJ is to PATTERNS."
+ (define (score pattern str)
+ (match pattern
+ ((? string? pattern)
+ (cond
+ ((string=? str pattern) 5)
+ (else
+ (let loop ((score 0) (start 0))
+ (cond
+ ((string-contains-ci str pattern start)
+ => (lambda (index)
+ (loop (+ score 1) (+ index (string-length pattern)))))
+ (else score))))))
+ ((? regexp? regexp)
+ (fold-matches regexp str 0
+ (lambda (m score)
+ (+ score
+ (if (string=? (match:substring m) str)
+ 5 ;exact match
+ 1)))))))
+
+ (define (pattern->score pattern)
+ (let ((score-pattern (lambda (str) (score pattern str))))
(fold (lambda (metric relevance)
(match metric
((field . weight)
(match (field obj)
(#f relevance)
((? string? str)
- (+ relevance (* (score-regexp str) weight)))
+ (+ relevance (* (score-pattern str) weight)))
((lst ...)
- (+ relevance (* weight (apply + (map score-regexp lst)))))))))
+ (+ relevance (* weight (apply + (map score-pattern lst)))))))))
0 metrics)))
- (let loop ((regexps regexps)
+ (let loop ((patterns patterns)
(total-score 0))
- (match regexps
+ (match patterns
((head . tail)
- (let ((score (regexp->score head)))
+ (let ((score (pattern->score head)))
;; Return zero if one of PATTERNS doesn't match.
(cond
((zero? score) 0)
--
2.26.2
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.
Received: (at 39258) by debbugs.gnu.org; 1 Jun 2020 00:00:41 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun May 31 20:00:41 2020
Received: from localhost ([127.0.0.1]:34106 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1jfXsf-0003ZY-05
for submit <at> debbugs.gnu.org; Sun, 31 May 2020 20:00:41 -0400
Received: from mugam.systemreboot.net ([139.59.75.54]:37188)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <arunisaac@HIDDEN>) id 1jfXsa-0003Yi-Gi
for 39258 <at> debbugs.gnu.org; Sun, 31 May 2020 20:00:38 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed;
d=systemreboot.net; s=default; h=Content-Transfer-Encoding:MIME-Version:
References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To:
Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From:
Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:
List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive;
bh=koRPHMk+1jW4Poi+61aDL6+TguQVOsEzO6WPFTcu4P8=; b=EbHbbb293KW4oVed2AEp4tTkR2
qyDC9IqAffBoWvsG7ux/6rWhNFZ/9r1PSlJk32j8miVFH2UX/WzC+Qtw0j2PFUS1nnEPIWBJHJlG+
AJDU7BYqaf6vF9AswhHFxzRn1dTpXii9cNWxBFWvMbyRdUieVLnCSp2yptyFzH9ItMyg=;
Received: from [192.168.2.1] (helo=steel.lan)
by systemreboot.net with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384
(Exim 4.93) (envelope-from <arunisaac@HIDDEN>)
id 1jfXsX-000Zk2-Af; Mon, 01 Jun 2020 05:30:33 +0530
From: Arun Isaac <arunisaac@HIDDEN>
To: 39258 <at> debbugs.gnu.org
Subject: [PATCH 3/4] ui: Do not translate package synopsis a second time.
Date: Mon, 1 Jun 2020 05:30:29 +0530
Message-Id: <20200601000030.7443-4-arunisaac@HIDDEN>
X-Mailer: git-send-email 2.26.2
In-Reply-To: <20200601000030.7443-1-arunisaac@HIDDEN>
References: <20200601000030.7443-1-arunisaac@HIDDEN>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 39258
Cc: Arun Isaac <arunisaac@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)
* guix/ui.scm (package->recutils): package-synopsis-string already returns a
translated string. Do not attempt to translate it again.
---
guix/ui.scm | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/guix/ui.scm b/guix/ui.scm
index 56754dba83..744758d1f3 100644
--- a/guix/ui.scm
+++ b/guix/ui.scm
@@ -1463,8 +1463,7 @@ HYPERLINKS? is true, emit hyperlink escape sequences when appropriate."
(string-map (match-lambda
(#\newline #\space)
(chr chr))
- (or (and=> (package-synopsis-string p) P_)
- "")))
+ (or (package-synopsis-string p) "")))
(format port "~a~%"
(string->recutils
(string-trim-right
--
2.26.2
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.
Received: (at 39258) by debbugs.gnu.org; 1 Jun 2020 00:00:41 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun May 31 20:00:41 2020
Received: from localhost ([127.0.0.1]:34104 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1jfXse-0003ZR-O4
for submit <at> debbugs.gnu.org; Sun, 31 May 2020 20:00:40 -0400
Received: from mugam.systemreboot.net ([139.59.75.54]:37190)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <arunisaac@HIDDEN>) id 1jfXsa-0003Yj-HC
for 39258 <at> debbugs.gnu.org; Sun, 31 May 2020 20:00:38 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed;
d=systemreboot.net; s=default; h=Content-Transfer-Encoding:MIME-Version:
References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To:
Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From:
Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:
List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive;
bh=NTHVfTd+ffFR2Zw8V8SaT5CvLkoPUwm2ZBFts1vBTuk=; b=l/yaCqC1EcyvTiCsQi1MRpgYlw
wVj+BVNQwc83y8u8Iuo7e499BvKE/y48zU00MIxAlHgeVulO33+kKrM8YCfHqirACS9ygDHSNdgzY
hogQ4XnJtfAOaBLh77YKwtMPc8dUQmeRCKQid1xi7nUfLCMyX859VvCQmZtvKC3RNPt0=;
Received: from [192.168.2.1] (helo=steel.lan)
by systemreboot.net with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384
(Exim 4.93) (envelope-from <arunisaac@HIDDEN>)
id 1jfXsX-000Zk2-Fl; Mon, 01 Jun 2020 05:30:33 +0530
From: Arun Isaac <arunisaac@HIDDEN>
To: 39258 <at> debbugs.gnu.org
Subject: [PATCH 4/4] ui: Use package-description-string.
Date: Mon, 1 Jun 2020 05:30:30 +0530
Message-Id: <20200601000030.7443-5-arunisaac@HIDDEN>
X-Mailer: git-send-email 2.26.2
In-Reply-To: <20200601000030.7443-1-arunisaac@HIDDEN>
References: <20200601000030.7443-1-arunisaac@HIDDEN>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 39258
Cc: Arun Isaac <arunisaac@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)
* guix/ui.scm (package->recutils): Use package-description-string instead of
package-description and P_.
---
guix/ui.scm | 6 ++----
1 file changed, 2 insertions(+), 4 deletions(-)
diff --git a/guix/ui.scm b/guix/ui.scm
index 744758d1f3..caa506a645 100644
--- a/guix/ui.scm
+++ b/guix/ui.scm
@@ -1468,10 +1468,8 @@ HYPERLINKS? is true, emit hyperlink escape sequences when appropriate."
(string->recutils
(string-trim-right
(parameterize ((%text-width width*))
- (texi->plain-text
- (string-append "description: "
- (or (and=> (package-description p) P_)
- ""))))
+ (string-append "description: "
+ (or (package-description-string p) "")))
#\newline)))
(for-each (match-lambda
((field . value)
--
2.26.2
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.
Received: (at 39258) by debbugs.gnu.org; 1 Jun 2020 00:00:40 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun May 31 20:00:40 2020
Received: from localhost ([127.0.0.1]:34102 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1jfXse-0003ZK-Fg
for submit <at> debbugs.gnu.org; Sun, 31 May 2020 20:00:40 -0400
Received: from mugam.systemreboot.net ([139.59.75.54]:37184)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <arunisaac@HIDDEN>) id 1jfXsa-0003Yh-Hb
for 39258 <at> debbugs.gnu.org; Sun, 31 May 2020 20:00:38 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed;
d=systemreboot.net; s=default; h=Content-Transfer-Encoding:MIME-Version:
References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To:
Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From:
Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:
List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive;
bh=dT1VBs9ZDJ00QXb7pSgilowGUSaU3mCd3cia1N2k7HA=; b=XrsJtCMK3hw2aaL/QkeMxSnEXo
L6rq1sKILtGc90Rcs6m8gQQ6627iM+Gdkoq1U57cdaNI6bOy+uQA3elRQ5OOWRh4m4u9bVwktkbDc
Z+acADudoG+a6gH6EAPd3B3irJWlwRj5res+ybvWQAyHGhPanMTebS5TBsMYmcWCO+i8=;
Received: from [192.168.2.1] (helo=steel.lan)
by systemreboot.net with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384
(Exim 4.93) (envelope-from <arunisaac@HIDDEN>)
id 1jfXsW-000Zk2-MW; Mon, 01 Jun 2020 05:30:33 +0530
From: Arun Isaac <arunisaac@HIDDEN>
To: 39258 <at> debbugs.gnu.org
Subject: [PATCH 1/4] ui: Cut off search early if any regexp does not match.
Date: Mon, 1 Jun 2020 05:30:27 +0530
Message-Id: <20200601000030.7443-2-arunisaac@HIDDEN>
X-Mailer: git-send-email 2.26.2
In-Reply-To: <20200601000030.7443-1-arunisaac@HIDDEN>
References: <20200601000030.7443-1-arunisaac@HIDDEN>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 39258
Cc: Arun Isaac <arunisaac@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)
* guix/ui.scm (relevance): When one of the regexps does not match, cut off
early and return 0. Do not try to match the remaining regexps.
---
guix/ui.scm | 15 ++++++++++-----
1 file changed, 10 insertions(+), 5 deletions(-)
diff --git a/guix/ui.scm b/guix/ui.scm
index ea5f460865..4a22358963 100644
--- a/guix/ui.scm
+++ b/guix/ui.scm
@@ -1519,11 +1519,16 @@ score, the more relevant OBJ is to REGEXPS."
(+ relevance (* weight (apply + (map score-regexp lst)))))))))
0 metrics)))
- (let ((scores (map regexp->score regexps)))
- ;; Return zero if one of REGEXPS doesn't match.
- (if (any zero? scores)
- 0
- (reduce + 0 scores))))
+ (let loop ((regexps regexps)
+ (total-score 0))
+ (match regexps
+ ((head . tail)
+ (let ((score (regexp->score head)))
+ ;; Return zero if one of PATTERNS doesn't match.
+ (cond
+ ((zero? score) 0)
+ (else (loop tail (+ total-score score))))))
+ (() total-score))))
(define %package-metrics
;; Metrics used to compute the "relevance score" of a package against a set
--
2.26.2
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 1 Jun 2020 00:00:40 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Sun May 31 20:00:40 2020 Received: from localhost ([127.0.0.1]:34100 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1jfXse-0003ZH-6k for submit <at> debbugs.gnu.org; Sun, 31 May 2020 20:00:40 -0400 Received: from mugam.systemreboot.net ([139.59.75.54]:37192) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <arunisaac@HIDDEN>) id 1jfXsb-0003Yl-57 for 39258 <at> debbugs.gnu.org; Sun, 31 May 2020 20:00:38 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=systemreboot.net; s=default; h=Content-Transfer-Encoding:MIME-Version: Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To:Content-Type:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:In-Reply-To:References:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=2jTYtU4y5MC0/bYrF3fsZzG9NRjCcmUfkno2BNH95o0=; b=ZU0i6PzLrAeER8sMT+UCURsQZo SFgC3aR6rlb2qMkWVnBSqIfWboZiwyrDwJoWECL1ajwZ1pX+tkqyT9USzhlw9iD5dGqrvTrykILds k5S9KDboT3tVUR8XGb31Lh84EggCag0uAXT/WjP1Z43JmuzlcrCWcr5av0V/caVi8DPQ=; Received: from [192.168.2.1] (helo=steel.lan) by systemreboot.net with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.93) (envelope-from <arunisaac@HIDDEN>) id 1jfXsW-000Zk2-EI; Mon, 01 Jun 2020 05:30:32 +0530 From: Arun Isaac <arunisaac@HIDDEN> To: 39258 <at> debbugs.gnu.org Subject: [PATCH 0/4] Optimize guix search Date: Mon, 1 Jun 2020 05:30:26 +0530 Message-Id: <20200601000030.7443-1-arunisaac@HIDDEN> X-Mailer: git-send-email 2.26.2 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 39258 Cc: Arun Isaac <arunisaac@HIDDEN>, ludo@HIDDEN, zimon.toutoune@HIDDEN X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) Hi, Sorry for the long delay in replying to this thread. I think Ludo is right in that we can improve guix search performance with only simple code improvements rather than including xapian or improving our existing cache. Here are a few patches on those lines. In `relevance`, we set our score to 0 if any of the regexps don't match. Then, we might as well not match the remaining regexps. Patch 1 does this early cut off optimization. Often our search strings are only literal strings. So, we can save some time by using string-contains instead of invoking the regexp engine. Patch 2 does this. In addition, guile's string-contains uses a naive O(n^2) string search algorithm. We should perhaps use the O(n) Knuth-Morris-Pratt algorithm[1]. In fact, a comment on line 2006 of libguile/srfi-13.c in the guile source code mentions this. If implemented, the KMP algorithm could speed up guix search further. [1]: https://en.wikipedia.org/wiki/Knuth%E2%80%93Morris%E2%80%93Pratt_algorithm Patch 3 and 4 are minor improvements. Here's a rough performance comparison. --8<---------------cut here---------------start------------->8--- time ./pre-inst-env guix search game real 0m2.261s user 0m2.351s sys 0m0.104s --8<---------------cut here---------------end--------------->8--- --8<---------------cut here---------------start------------->8--- time guix search game real 0m2.661s user 0m2.843s sys 0m0.080s --8<---------------cut here---------------end--------------->8--- --8<---------------cut here---------------start------------->8--- time ./pre-inst-env guix search strategy game real 0m1.613s user 0m1.635s sys 0m0.096s --8<---------------cut here---------------end--------------->8--- --8<---------------cut here---------------start------------->8--- time guix search strategy game real 0m2.520s user 0m2.583s sys 0m0.112s --8<---------------cut here---------------end--------------->8--- Arun Isaac (4): ui: Cut off search early if any regexp does not match. ui: Use string matching with literal search strings. ui: Do not translate package synopsis a second time. ui: Use package-description-string. guix/scripts/package.scm | 12 +++++-- guix/ui.scm | 68 ++++++++++++++++++++++++---------------- 2 files changed, 50 insertions(+), 30 deletions(-) -- 2.26.2
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 3 May 2020 19:49:19 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Sun May 03 15:49:19 2020 Received: from localhost ([127.0.0.1]:58061 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1jVKc3-0006FK-Fe for submit <at> debbugs.gnu.org; Sun, 03 May 2020 15:49:19 -0400 Received: from eggs.gnu.org ([209.51.188.92]:34622) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <ludo@HIDDEN>) id 1jVKc2-0006F7-1r for 39258 <at> debbugs.gnu.org; Sun, 03 May 2020 15:49:18 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:58785) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from <ludo@HIDDEN>) id 1jVKbu-00054M-6N; Sun, 03 May 2020 15:49:10 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=49782 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from <ludo@HIDDEN>) id 1jVKbt-0004Up-Cg; Sun, 03 May 2020 15:49:09 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN> To: zimoun <zimon.toutoune@HIDDEN> Subject: Re: [PATCH v4 0/3] Faster cache generation (similar as v3) References: <20200503150154.26532-1-zimon.toutoune@HIDDEN> <87r1w1ynnm.fsf@HIDDEN> <CAJ3okZ1GS3aMjX3kGBYOkJi03MzGe2qgfAznWE5aGNn+zKonrw@HIDDEN> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 15 =?utf-8?Q?Flor=C3=A9al?= an 228 de la =?utf-8?Q?R?= =?utf-8?Q?=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Sun, 03 May 2020 21:49:06 +0200 In-Reply-To: <CAJ3okZ1GS3aMjX3kGBYOkJi03MzGe2qgfAznWE5aGNn+zKonrw@HIDDEN> (zimoun's message of "Sun, 3 May 2020 20:10:10 +0200") Message-ID: <87tv0wyf2l.fsf@HIDDEN> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 39258 Cc: Arun Isaac <arunisaac@HIDDEN>, Pierre Neidhardt <mail@HIDDEN>, 39258 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -3.3 (---) Hi, zimoun <zimon.toutoune@HIDDEN> skribis: > On Sun, 3 May 2020 at 18:43, Ludovic Court=C3=A8s <ludo@HIDDEN> wrote: > >> > Therefore the cache '/lib/guix/package.cache' contains more >> > information. >> >> This breaks the binary interface, so we=E2=80=99ll have to analyze the i= mpact of >> such a change and devise a strategy. > > Interface between what and what? Guix revision N creates a cache that will be read by revision N+1, upon =E2=80=98guix pull=E2=80=99 completion. > Note that I have read your comment in v3 2/3 but I did not understand it.= Sorry. > > I realize the other cache also has that problem, but it would be nice to > add a version tag to the cache. Basically emit something like: > > (package-metadata-cache (version 0) VECTOR =E2=80=A6) > > instead of just: > > (VECTOR =E2=80=A6) Yes, it would be better. > For example, 'supported-systems' is saved as a list of strings, > whereas 'license' is expanded as 3 strings without be packed in a list > of strings. From my point of view, it is inconsistent and I do not > know what is the best (readibility, startup time, etc.). I guess both =E2=80=98license=E2=80=99 and =E2=80=98supported-systems=E2=80= =99 should be list of strings. It doesn=E2=80=99t really have an impact on startup time (I thoug= ht you were suggesting storing the cache as an sexp instead of an object file.) >> Something that must be done explicitly doesn=E2=80=99t seem great to me.= As a >> user, I=E2=80=99d rather not think about search indexes and all. But I = don=E2=80=99t >> know, maybe if it happened automatically on the first =E2=80=98guix sear= ch=E2=80=99 >> invocation that=E2=80=99d be fine. > > I do not think it is an option to build the BM25 the first time "guix > search" is called. Back-to-envelop estimation, it needs ~25 seconds > to Xapian* to do so. > > From my point of view, two options: > a) "guix pull" does this extra ~25 seconds (compared to 10 seconds to > build the v4 cache) > b) the user manually build the index (I agree it is awkward!) > > Well, the first question is to evaluate if it is worth -- I am using > the v2 version based on Xapian to have an idea. Please if you have > suggestions about query (terms an user could type) and results > (packages an user could expect), there are welcome. Yeah, dunno. Maybe an option would be to create the index in such a way that it is substitutable. [...] > Yes, it was already your suggestion in v3 3/3. Do you suggest to > refactor 'package->recutils'? For example, > > (define* (package->recutils name version > ... all-the-other-fields ... > port #:optional (width (%text-width)) > #:key > (hyperlinks? (supports-hyperlinks? port)) > (extra-fields '())) Yes. >> > 4. Impolite '@@' is used to access the private license construction. >> >> (guix licenses) could provide a =E2=80=98string->license=E2=80=99 proced= ure. > > Well, do you suggest: > > (define (string->license name) (license name #f #f)) No; rather, it would look up the license in a dictionary and return the corresponding object or #f if it=E2=80=99s not a known license. Thanks, Ludo=E2=80=99.
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.
Received: (at 39258) by debbugs.gnu.org; 3 May 2020 18:10:29 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun May 03 14:10:29 2020
Received: from localhost ([127.0.0.1]:57994 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1jVJ4P-0003sE-AS
for submit <at> debbugs.gnu.org; Sun, 03 May 2020 14:10:29 -0400
Received: from mail-qt1-f196.google.com ([209.85.160.196]:45766)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <zimon.toutoune@HIDDEN>) id 1jVJ4N-0003s2-Cy
for 39258 <at> debbugs.gnu.org; Sun, 03 May 2020 14:10:27 -0400
Received: by mail-qt1-f196.google.com with SMTP id 71so12145831qtc.12
for <39258 <at> debbugs.gnu.org>; Sun, 03 May 2020 11:10:27 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
h=mime-version:references:in-reply-to:from:date:message-id:subject:to
:cc:content-transfer-encoding;
bh=7kYCtT1FhcggnkrKSTbjZGekDA8sRbduJ/AL7R0h/zE=;
b=KTOwmFgQUVmgaEuBqP5KdVN6+FM1WF07h5BMPGJPJVa9pqo82v+5EXx5HxqifBEqeA
lmWr94eiDfielYzsShW5ufY2VlixRGoIyZLUNS1AEdgIyFUia6dOdusHM3MbTAl820r6
O5YAMivzHF5hgKIwImUoXK8EZSX/MrqOMhIP3DpCU4ZGD7ElWeJ7ttZlt/YJxrZQaBaU
PyBiPPJK6b1fz+vp9JQndVI1pwhSaPrnABB3VG4APHwzYi29iZKLmju/8eD95t+9fOzP
i9nFNuQ67O1tCpl0xLn8JavQB3ZrhFVqAPCija6lA+Mm7ZEww/XyXQNdt00wwx0wt6RU
5fbA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20161025;
h=x-gm-message-state:mime-version:references:in-reply-to:from:date
:message-id:subject:to:cc:content-transfer-encoding;
bh=7kYCtT1FhcggnkrKSTbjZGekDA8sRbduJ/AL7R0h/zE=;
b=c1UiP3vRXYpAMHdK6f0TPJuSnsYjVwACasmMvLHQlAo6FJGvUw7cnsTQ9ceWpd+b+i
6rwSH1wzUx71A6bJ66sWo46luiQckLBJes+Jq9ez8N4fF7P1qpR5mRMDmpGnLR2whehy
9nPyPVIC7ODwotvGYZXJHlAfkkCpo14zsCapRsLmgpdWkHXW1aPfVOeIlR1pGXG017J+
Xz3OU+oMLPwn0QqvfV4ylWl3NmV1sPx+Zgjsfm3JAgflhMFCd974NQdYAHM4g+fclrA0
MffqP+kG4gLLa8FIOkZzbGNO9AyUuXFSMt4RlEJBssyH3cXmtx6PFh5yHJgAogK6YrKG
iEwg==
X-Gm-Message-State: AGi0PuYD9HSDoyUhkKmj+UC+ONpDLB0K05LW0heuraoH+j02LPqO2oph
8K4WByDCo1k645zPcuCQnf2qMAbxaZXrUSufwyQ=
X-Google-Smtp-Source: APiQypJ0OEWSOv2csOgkkS35VqviNxFQjjfvPJjdc/utUeOm6v76m6Xoax3dZGu7AH6yPGB1Mom5LTliIEhU8uSRTco=
X-Received: by 2002:ac8:19f5:: with SMTP id s50mr13931499qtk.186.1588529421693;
Sun, 03 May 2020 11:10:21 -0700 (PDT)
MIME-Version: 1.0
References: <20200503150154.26532-1-zimon.toutoune@HIDDEN>
<87r1w1ynnm.fsf@HIDDEN>
In-Reply-To: <87r1w1ynnm.fsf@HIDDEN>
From: zimoun <zimon.toutoune@HIDDEN>
Date: Sun, 3 May 2020 20:10:10 +0200
Message-ID: <CAJ3okZ1GS3aMjX3kGBYOkJi03MzGe2qgfAznWE5aGNn+zKonrw@HIDDEN>
Subject: Re: [PATCH v4 0/3] Faster cache generation (similar as v3)
To: =?UTF-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 39258
Cc: Arun Isaac <arunisaac@HIDDEN>,
Pierre Neidhardt <mail@HIDDEN>, 39258 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)
Hi Ludo,
On Sun, 3 May 2020 at 18:43, Ludovic Court=C3=A8s <ludo@HIDDEN> wrote:
> > Therefore the cache '/lib/guix/package.cache' contains more
> > information.
>
> This breaks the binary interface, so we=E2=80=99ll have to analyze the im=
pact of
> such a change and devise a strategy.
Interface between what and what?
Because from my understanding, this file is only used by only one
guix. What do I miss?
Note that I have read your comment in v3 2/3 but I did not understand it. S=
orry.
--8<---------------cut here---------------start------------->8---
I realize the other cache also has that problem, but it would be nice to
add a version tag to the cache. Basically emit something like:
(package-metadata-cache (version 0) VECTOR =E2=80=A6)
instead of just:
(VECTOR =E2=80=A6)
--8<---------------cut here---------------end--------------->8---
> > (The v4 structure of 'package.cache' is a quick draft, so details
> > should be discussed and an interesting move should to have a
> > structured (binary and all strings) S-exp; because it should become an
> > entry point to export the packages list to JSON. WDYT?)
>
> It=E2=80=99s on purpose that this cache is an object file: it just needs =
to be
> mmap=E2=80=99d, and that=E2=80=99s it. It=E2=80=99s the cheapest possibl=
e way to do it.
> Parsing sexps would be more costly, and since we=E2=80=99re talking about
> startup time, this is sensitive.
I agree and I have badly worded or I misunderstand something.
For example, 'supported-systems' is saved as a list of strings,
whereas 'license' is expanded as 3 strings without be packed in a list
of strings. From my point of view, it is inconsistent and I do not
know what is the best (readibility, startup time, etc.).
> > To be clear about BM25 and caching, what I have in mind is:
> > 1. "guix search --build-index" optionally done by the user if they wa=
nts for example the BM25 ranking.
>
> Something that must be done explicitly doesn=E2=80=99t seem great to me. =
As a
> user, I=E2=80=99d rather not think about search indexes and all. But I d=
on=E2=80=99t
> know, maybe if it happened automatically on the first =E2=80=98guix searc=
h=E2=80=99
> invocation that=E2=80=99d be fine.
I do not think it is an option to build the BM25 the first time "guix
search" is called. Back-to-envelop estimation, it needs ~25 seconds
to Xapian* to do so.
From my point of view, two options:
a) "guix pull" does this extra ~25 seconds (compared to 10 seconds to
build the v4 cache)
b) the user manually build the index (I agree it is awkward!)
Well, the first question is to evaluate if it is worth -- I am using
the v2 version based on Xapian to have an idea. Please if you have
suggestions about query (terms an user could type) and results
(packages an user could expect), there are welcome.
*Xapian: I do not think we could do better but I have not checked yet
if there is a bottleneck Guix, Guile-Xapian and Xapian.
> > 1. The name of 'fold-packages*' should be misleading since it does not=
return "true" packages.
>
> Did you see =E2=80=98fold-available-packages=E2=80=99? It seems you coul=
d extend it
> instead of introducing =E2=80=98fold-packages*=E2=80=99, no?
Yes and no.
a) 'fold-available-packages' requires to modify the 'lambda' in
'find-package-by-description',
b) 'fold-package*' returning a 'package' is less tweaks, IMHO.
Well, I agree that on the long term, what 'fold-package*' does could
be done by 'fold-available-packages' with the adequate 'proc'.
Thank you for the suggestion; even if once re-read correctly v3 2/3
you already mentioned it. :-)
> > 2. The function 'package->recutils' in 'guix/ui.scm' is modified but i=
t is not the better.
> >
> > (match (package-supported-systems p)
> > (('cache supported-systems)
> > (string-join supported-systems))
> > (_
> > (string-join (package-transitive-supported-systems p)))))
> >
> > However it avoids to duplicate code; as it is done in version v3.
>
> I made suggestions to Arun=E2=80=99s v3 about the API here. Essentially,=
I
> think I proposed having a procedure that takes the list of fields as
> keyword parameters, and =E2=80=98package->recutils=E2=80=99 would just de=
legate to that.
Yes, it was already your suggestion in v3 3/3. Do you suggest to
refactor 'package->recutils'? For example,
--8<---------------cut here---------------start------------->8---
(define* (package->recutils name version
... all-the-other-fields ...
port #:optional (width (%text-width))
#:key
(hyperlinks? (supports-hyperlinks? port))
(extra-fields '()))
--8<---------------cut here---------------end--------------->8---
> > 4. Impolite '@@' is used to access the private license construction.
>
> (guix licenses) could provide a =E2=80=98string->license=E2=80=99 procedu=
re.
Well, do you suggest:
(define (string->license name) (license name #f #f))
? Skipping 'uri' and 'comment'? Naive question: what is the purpose
of these 2 fields? Because there are not exposed at the CLI level,
AFAIK, and I do not think an user evaluate '(license-uri pkg)' in a
script.
Well, I think that the hyperlink feature could be used to display the
license URI too. WDYT?
> Stopping here for now because I=E2=80=99m sorta drowning in patch review.=
:-)
Thank you for all the comments.
> Thanks for exploring this design space, we=E2=80=99re making progress!
My pleasure. Scheme is designed to explore. ;-)
Cheers,
simon
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.
Received: (at 39258) by debbugs.gnu.org; 3 May 2020 16:43:53 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun May 03 12:43:53 2020
Received: from localhost ([127.0.0.1]:57765 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1jVHia-0001U2-95
for submit <at> debbugs.gnu.org; Sun, 03 May 2020 12:43:53 -0400
Received: from eggs.gnu.org ([209.51.188.92]:38276)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <ludo@HIDDEN>) id 1jVHiZ-0001Tq-1f
for 39258 <at> debbugs.gnu.org; Sun, 03 May 2020 12:43:51 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:55928)
by eggs.gnu.org with esmtp (Exim 4.90_1)
(envelope-from <ludo@HIDDEN>)
id 1jVHiS-0005fb-Aw; Sun, 03 May 2020 12:43:44 -0400
Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=49660 helo=ribbon)
by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
(Exim 4.82) (envelope-from <ludo@HIDDEN>)
id 1jVHiR-0001GQ-HC; Sun, 03 May 2020 12:43:44 -0400
From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>
To: zimoun <zimon.toutoune@HIDDEN>
Subject: Re: [PATCH v4 0/3] Faster cache generation (similar as v3)
References: <20200503150154.26532-1-zimon.toutoune@HIDDEN>
X-URL: http://www.fdn.fr/~lcourtes/
X-Revolutionary-Date: 15 =?utf-8?Q?Flor=C3=A9al?= an 228 de la =?utf-8?Q?R?=
=?utf-8?Q?=C3=A9volution?=
X-PGP-Key-ID: 0x090B11993D9AEBB5
X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc
X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5
X-OS: x86_64-pc-linux-gnu
Date: Sun, 03 May 2020 18:43:41 +0200
In-Reply-To: <20200503150154.26532-1-zimon.toutoune@HIDDEN> (zimoun's
message of "Sun, 3 May 2020 17:01:51 +0200")
Message-ID: <87r1w1ynnm.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 39258
Cc: arunisaac@HIDDEN, mail@HIDDEN, 39258 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)
Hello!
zimoun <zimon.toutoune@HIDDEN> skribis:
> The aim of this version v4 is to keep the same searching performances as =
the previous version v3 but to drastically reduce the generation of the cac=
he. On my laptop, the overhead is now 4 seconds; compared to more than 20 =
seconds for v2 and v3.
>
> # default
> time guix build /gnu/store/0nfpp82mqglpwvl1nbfpaphw5db2ivcp-guix-package-=
cache.drv --check
> # v4
> time guix build /gnu/store/y78gfh1n7m3kyrj8wsqj25qc2cbc1a4d-guix-package-=
cache.drv --check
>
> | | default | v4 |
> |------+----------+-----------|
> | real | 0m6.012s | 0m10.244s |
> | user | 0m0.541s | 0m0.542s |
> | sys | 0m0.033s | 0m0.032s |
Not bad!
> In the version v3, the cache is built using 'cons' and 'fold-packages' (w=
rapper to 'fold-module-public-variables'). The version v4 modifies -- by a=
dding other information -- the function 'generate-package-cache' which uses=
'vhash' and 'fold-module-public-variables*'.
>
> Therefore the cache '/lib/guix/package.cache' contains more
> information.
This breaks the binary interface, so we=E2=80=99ll have to analyze the impa=
ct of
such a change and devise a strategy.
> (The v4 structure of 'package.cache' is a quick draft, so details
> should be discussed and an interesting move should to have a
> structured (binary and all strings) S-exp; because it should become an
> entry point to export the packages list to JSON. WDYT?)
It=E2=80=99s on purpose that this cache is an object file: it just needs to=
be
mmap=E2=80=99d, and that=E2=80=99s it. It=E2=80=99s the cheapest possible =
way to do it.
Parsing sexps would be more costly, and since we=E2=80=99re talking about
startup time, this is sensitive.
> Now, we are comparing apples to apples and the cost to compute BM25 (v2) =
is not free at all. Remember that BM25 is the state-of-the-art of informat=
ion retrieval (relevance ranking) and it is delegated to Xapian (v2). I do=
not know if there is perfomance bottleneck between Guix, Guile-Xapian and =
Xapian itself but for sure the computation of BM25 is not free. More about=
that soon.
>
> To be clear about BM25 and caching, what I have in mind is:
> 1. "guix search --build-index" optionally done by the user if they want=
s for example the BM25 ranking.
Something that must be done explicitly doesn=E2=80=99t seem great to me. A=
s a
user, I=E2=80=99d rather not think about search indexes and all. But I don=
=E2=80=99t
know, maybe if it happened automatically on the first =E2=80=98guix search=
=E2=80=99
invocation that=E2=80=99d be fine.
> 2. Use BM25 metrics to detect poor package meta-data (synopsis and desc=
ription); if it worth why not add another checker to "guix lint".
That=E2=80=99d be interesting!
> 1. The name of 'fold-packages*' should be misleading since it does not r=
eturn "true" packages.
Did you see =E2=80=98fold-available-packages=E2=80=99? It seems you could =
extend it
instead of introducing =E2=80=98fold-packages*=E2=80=99, no?
> 2. The function 'package->recutils' in 'guix/ui.scm' is modified but it =
is not the better.
>
> (match (package-supported-systems p)
> (('cache supported-systems)
> (string-join supported-systems))
> (_
> (string-join (package-transitive-supported-systems p)))))
>
> However it avoids to duplicate code; as it is done in version v3.
I made suggestions to Arun=E2=80=99s v3 about the API here. Essentially, I
think I proposed having a procedure that takes the list of fields as
keyword parameters, and =E2=80=98package->recutils=E2=80=99 would just dele=
gate to that.
> 3. Deprecated packages are displayed (bug in v3 too).
>
> 4. Impolite '@@' is used to access the private license construction.
(guix licenses) could provide a =E2=80=98string->license=E2=80=99 procedure.
Stopping here for now because I=E2=80=99m sorta drowning in patch review. =
:-)
Thanks for exploring this design space, we=E2=80=99re making progress!
Ludo=E2=80=99.
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.
Received: (at 39258) by debbugs.gnu.org; 3 May 2020 15:02:29 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun May 03 11:02:29 2020
Received: from localhost ([127.0.0.1]:57644 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1jVG8T-0005B8-0N
for submit <at> debbugs.gnu.org; Sun, 03 May 2020 11:02:29 -0400
Received: from mail-wr1-f67.google.com ([209.85.221.67]:46571)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <zimon.toutoune@HIDDEN>) id 1jVG8R-0005AW-3K
for 39258 <at> debbugs.gnu.org; Sun, 03 May 2020 11:02:27 -0400
Received: by mail-wr1-f67.google.com with SMTP id f13so17774925wrm.13
for <39258 <at> debbugs.gnu.org>; Sun, 03 May 2020 08:02:27 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
h=from:to:cc:subject:date:message-id:in-reply-to:references
:mime-version:content-transfer-encoding;
bh=iUk5aRBWe+0sYmv5/HX8bzBOaoivPCZk8Goly8Oin4A=;
b=kXrcXCCe7XIBPjLQgMIyH2n2sUT5eM7cCLF3JKEPEQFq7FJPXaOLU0DYTM4v1R/XX/
HHTZxgoo6CWDhzpSmzSrFJmfU+wOngIY48XnFB6GZE8x+PuRnALU3HlBILJcWzZxhMhf
HasSQmgDS6gUCdjedF6Z2WFoCKKCNdDNch3vWgpJV9M+MzyH3yYLTMbB9o6quOBqeFHx
W8dHuvcB80215JeAHJ7dz9ZWGKzOqs2Lpr827Ilm+Wsk2l5+PgE2+NtzoskGAHQUvOQv
n5XLVtl70ASvXiKAQSbu8UCg1EZBNQ9TBjgtrU50mtMfw/VXHeld8jazBWK5dRep/Q6H
zeIQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20161025;
h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to
:references:mime-version:content-transfer-encoding;
bh=iUk5aRBWe+0sYmv5/HX8bzBOaoivPCZk8Goly8Oin4A=;
b=YLoJEEaFjmEgFE+33mRONwSDspcOE8FmHWWG7HQJ66Xn4khB6OUWZzHXnk2rcfR5iM
MeXdugDjDpoR+sOzI39ps42aish0XiSyME+XFJ+k4JFY2+bK3V1zSpa1VuhU4aVjV3Pv
RjkhI0ezAZ5ueH347upkS8n7MSYbUF3jV9mo3kTFT5GJZ2rgcjXf/pkOQ4HiYPJr6W3E
/HYMvJAv3ADgPUvzOWQQWR/tWEDwFt4QQUVuNk1kgtZU8ShjP0xCn4ViBlOUAizhNCXy
2+WrS6/s/jR6NxTfCZT8WKOacj7pfCMLftyaAKLPa5HszBfC4ZYsVT0nJKpde4QWGNoM
4Ndg==
X-Gm-Message-State: AGi0Pua/lFzib4rTRNGdBI4Dc/+6dbrfUoOMt4mdCVsmwFWfiaPdXvX8
yU2eu+pt4xsJdydiYePtkojz7WD8
X-Google-Smtp-Source: APiQypI/fDORsUp1o7/lXlNckqXRq+1eqXZp7lUQCbp3F3iPATsw08yflmm2IUvJg8Nvldc+NRfNaA==
X-Received: by 2002:a5d:6b85:: with SMTP id n5mr14583110wrx.370.1588518141213;
Sun, 03 May 2020 08:02:21 -0700 (PDT)
Received: from localhost.localdomain (57.246.195.77.rev.sfr.net.
[77.195.246.57])
by smtp.gmail.com with ESMTPSA id x13sm9787829wmc.5.2020.05.03.08.02.20
(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
Sun, 03 May 2020 08:02:20 -0700 (PDT)
From: zimoun <zimon.toutoune@HIDDEN>
To: 39258 <at> debbugs.gnu.org
Subject: [PATCH v4 3/3] DRAFT guix package: Use cache in
'find-packages-by-description'.
Date: Sun, 3 May 2020 17:01:54 +0200
Message-Id: <20200503150154.26532-4-zimon.toutoune@HIDDEN>
X-Mailer: git-send-email 2.26.1
In-Reply-To: <20200503150154.26532-1-zimon.toutoune@HIDDEN>
References: <20200503150154.26532-1-zimon.toutoune@HIDDEN>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 39258
Cc: arunisaac@HIDDEN, mail@HIDDEN, ludo@HIDDEN,
zimoun <zimon.toutoune@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)
---
guix/scripts/package.scm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/guix/scripts/package.scm b/guix/scripts/package.scm
index badb1dcd38..6b982eb172 100644
--- a/guix/scripts/package.scm
+++ b/guix/scripts/package.scm
@@ -174,7 +174,7 @@ hooks\" run when building the profile."
"Return a list of pairs: packages whose name, synopsis, description,
or output matches at least one of REGEXPS sorted by relevance, and its
non-zero relevance score."
- (let ((matches (fold-packages (lambda (package result)
+ (let ((matches (fold-packages* (lambda (package result)
(if (package-superseded package)
result
(match (package-relevance package
--
2.26.1
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.
Received: (at 39258) by debbugs.gnu.org; 3 May 2020 15:02:28 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun May 03 11:02:28 2020
Received: from localhost ([127.0.0.1]:57640 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1jVG8R-0005An-8T
for submit <at> debbugs.gnu.org; Sun, 03 May 2020 11:02:27 -0400
Received: from mail-wr1-f67.google.com ([209.85.221.67]:34924)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <zimon.toutoune@HIDDEN>) id 1jVG8P-0005AN-DX
for 39258 <at> debbugs.gnu.org; Sun, 03 May 2020 11:02:25 -0400
Received: by mail-wr1-f67.google.com with SMTP id x18so17833251wrq.2
for <39258 <at> debbugs.gnu.org>; Sun, 03 May 2020 08:02:25 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
h=from:to:cc:subject:date:message-id:in-reply-to:references
:mime-version:content-transfer-encoding;
bh=g/Ap+DsFXX88hbi+Yvm1efNAcOgiB+00QdKk0X55xTc=;
b=h8CE3txtoW2DMHQVxDWhhGVIKkbsKKPieGNdtY4LGffYuTGeRMJZ8nQqIO5QAc8WX9
eNHfn0F+VY3qD6BDUM1nYno5isLWW7nnk1NqzL5edC2yDspUR5PcW7xADD3w301CHS+3
EeuuoLEwWa7scP/1Eb1mVS3o2bPzCPwuvjS2CniWBBsEzx9cikBdb4n5/t1DhWYsovfT
5TVLROpFiVQK3MS3AJqCeJT1b9CPdPkBCatjAsNCOe6HgGihtJ5ExuXHZ0BQNEJJ7nMo
xKUe30c0hH6hlR9Vd4wRVGA9qE0eSB1nP7xvDXd9bG36OR3kHPxduJzbY48iiUiKkiz2
9ShQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20161025;
h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to
:references:mime-version:content-transfer-encoding;
bh=g/Ap+DsFXX88hbi+Yvm1efNAcOgiB+00QdKk0X55xTc=;
b=Q/fbeGgqtvtX6oqH67Fc5tj/eR6e7xrzcZKTHHRa8HaKTUhj00cueX4LlIAvXKt3iX
iBa1AmG5x9B6h6anU1OTU5pQqMw17BgkQxMX0wbl5tHkPFDcS3T2w99n1DiflTSpzHhW
2MNr2PDx4gBEDDqaulfgcS63uGRPa4qE0JhHnMwdZnnC1q7yzwWxYloZPZ+VaxmeV+kQ
uPs5JYbn/6s6p0Xak+eArtFIhSUyaRZpIBBrTg+k6Nc7Xaz6vNMEY1/CyZ5U76Mtny5q
5tH3nwPJnnS7o7HsXPmSpT6TzKwBA0WjxEBJQPAPa9npvjrfiZZ4EWrtVVXRBZTM81DW
6w+w==
X-Gm-Message-State: AGi0PuZNvOzZbOh4xz1DSZIh920u3OE6hM2nIFEXegCa9mHdtUDAN+eG
DZ9/L2k2wQu4C2vzq6XwSA+1AAqq
X-Google-Smtp-Source: APiQypI+C5VHCpwASn0z9sOBkb905oUanZs6V+nM8elFaB1cKtUSw6gI/znRc2Ce3she9ZtYcZCcbA==
X-Received: by 2002:adf:fcc8:: with SMTP id f8mr14213653wrs.230.1588518139268;
Sun, 03 May 2020 08:02:19 -0700 (PDT)
Received: from localhost.localdomain (57.246.195.77.rev.sfr.net.
[77.195.246.57])
by smtp.gmail.com with ESMTPSA id x13sm9787829wmc.5.2020.05.03.08.02.18
(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
Sun, 03 May 2020 08:02:18 -0700 (PDT)
From: zimoun <zimon.toutoune@HIDDEN>
To: 39258 <at> debbugs.gnu.org
Subject: [PATCH v4 2/3] DRAFT packages: Add new procedure 'fold-packages*'.
Date: Sun, 3 May 2020 17:01:53 +0200
Message-Id: <20200503150154.26532-3-zimon.toutoune@HIDDEN>
X-Mailer: git-send-email 2.26.1
In-Reply-To: <20200503150154.26532-1-zimon.toutoune@HIDDEN>
References: <20200503150154.26532-1-zimon.toutoune@HIDDEN>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 39258
Cc: arunisaac@HIDDEN, mail@HIDDEN, ludo@HIDDEN,
zimoun <zimon.toutoune@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)
---
gnu/packages.scm | 47 ++++++++++++++++++++++++++++++++++++++++++++++
guix/ui.scm | 29 +++++++++++++++++-----------
tests/packages.scm | 31 ++++++++++++++++++++++++++++++
3 files changed, 96 insertions(+), 11 deletions(-)
diff --git a/gnu/packages.scm b/gnu/packages.scm
index fa18f81487..a0c5835b8b 100644
--- a/gnu/packages.scm
+++ b/gnu/packages.scm
@@ -55,6 +55,7 @@
fold-packages
fold-available-packages
+ fold-packages*
find-newest-available-packages
find-packages-by-name
@@ -253,6 +254,52 @@ is guaranteed to never traverse the same package twice."
init
modules))
+(define (fold-packages* proc init)
+ "Fold (PROC PACKAGE RESULT) over the list of available packages. When a
+package cache is available, this procedure does not actually load any package
+module. Moreover when package cache is available, this procedure
+re-constructs a new package skipping some package record field. The usage of
+this procedure is User Interface (ui) only."
+ (define cache
+ (load-package-cache (current-profile)))
+
+ (define license (@@ (guix licenses) license))
+
+ (if (and cache (cache-is-authoritative?))
+ (vhash-fold (lambda (name vector result)
+ (match vector
+ (#(name version module symbol outputs
+ supported? deprecated?
+ file line column
+ synopsis description home-page
+ build-system-name build-system-description
+ supported-systems direct-inputs
+ license-name license-uri license-comment)
+ (proc (package
+ (name name)
+ (version version)
+ (source #f) ;TODO: ?
+ (build-system
+ (build-system
+ (name (string->symbol build-system-name))
+ (description build-system-description)
+ (lower #f))) ; never used by ui
+ (inputs ; list of "full-name@version"
+ (list 'cache direct-inputs))
+ (outputs outputs)
+ (synopsis synopsis)
+ (description description)
+ (license (license
+ license-name license-uri license-comment))
+ (home-page home-page)
+ (supported-systems (list 'cache supported-systems))
+ (location (location
+ file line column)))
+ result))))
+ init
+ cache)
+ (fold-packages proc init)))
+
(define %package-cache-file
;; Location of the package cache.
"/lib/guix/package.cache")
diff --git a/guix/ui.scm b/guix/ui.scm
index 1e24fe5dca..257d119798 100644
--- a/guix/ui.scm
+++ b/guix/ui.scm
@@ -1416,13 +1416,10 @@ HYPERLINKS? is true, emit hyperlink escape sequences when appropriate."
;; the initial "+ " prefix.
(if (> width 2) (- width 2) width))
- (define (dependencies->recutils packages)
- (let ((list (string-join (delete-duplicates
- (map package-full-name
- (sort packages package<?))) " ")))
- (string->recutils
- (fill-paragraph list width*
- (string-length "dependencies: ")))))
+ (define (dependencies->string packages)
+ (string-join (delete-duplicates
+ (map package-full-name
+ (sort packages package<?))) " "))
(define (package<? p1 p2)
(string<? (package-full-name p1) (package-full-name p2)))
@@ -1432,11 +1429,21 @@ HYPERLINKS? is true, emit hyperlink escape sequences when appropriate."
(format port "version: ~a~%" (package-version p))
(format port "outputs: ~a~%" (string-join (package-outputs p)))
(format port "systems: ~a~%"
- (string-join (package-transitive-supported-systems p)))
+ (match (package-supported-systems p)
+ (('cache supported-systems)
+ (string-join supported-systems))
+ (_
+ (string-join (package-transitive-supported-systems p)))))
(format port "dependencies: ~a~%"
- (match (package-direct-inputs p)
- (((labels inputs . _) ...)
- (dependencies->recutils (filter package? inputs)))))
+ (let ((dependencies
+ (match (package-direct-inputs p)
+ (('cache inputs)
+ (string-join inputs))
+ (((labels inputs . _) ...)
+ (dependencies->string (filter package? inputs))))))
+ (string->recutils
+ (fill-paragraph dependencies width*
+ (string-length "dependencies: ")))))
(format port "location: ~a~%"
(or (and=> (package-location p)
(if hyperlinks? location->hyperlink location->string))
diff --git a/tests/packages.scm b/tests/packages.scm
index 7a8b5e4a2d..4504f6cf33 100644
--- a/tests/packages.scm
+++ b/tests/packages.scm
@@ -1169,6 +1169,37 @@
((one)
(eq? one guile-2.0))))
+(test-assert "fold-packages* hello with/without cache"
+ (let ()
+ (define (equal-package? p1 p2)
+ ;; fold-package* re-constructs a new package skipping 'source' and 'lower'
+ ;; so equal? does not apply
+ (and (equal? (package-full-name p1) (package-full-name p2))
+ (equal? (package-description p1) (package-description p2))))
+
+ (define no-cache
+ (fold-packages* (lambda (p r)
+ (if (string=? (package-name p) "hello")
+ p
+ r))
+ #f))
+
+ (define from-cache
+ (call-with-temporary-directory
+ (lambda (cache)
+ (generate-package-cache cache)
+ (mock ((guix describe) current-profile (const cache))
+ (mock ((gnu packages) cache-is-authoritative? (const #t))
+ (fold-packages* (lambda (p r)
+ (if (string=? (package-name p) "hello")
+ p
+ r))
+ #f))))))
+
+ (and (equal? no-cache hello)
+ (equal-package? from-cache hello)
+ (equal-package? no-cache from-cache))))
+
(test-assert "fold-available-packages with/without cache"
(let ()
(define no-cache
--
2.26.1
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.
Received: (at 39258) by debbugs.gnu.org; 3 May 2020 15:02:22 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun May 03 11:02:22 2020
Received: from localhost ([127.0.0.1]:57636 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1jVG8L-0005AX-UI
for submit <at> debbugs.gnu.org; Sun, 03 May 2020 11:02:22 -0400
Received: from mail-wm1-f51.google.com ([209.85.128.51]:32904)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <zimon.toutoune@HIDDEN>) id 1jVG8K-0005A5-K1
for 39258 <at> debbugs.gnu.org; Sun, 03 May 2020 11:02:20 -0400
Received: by mail-wm1-f51.google.com with SMTP id v8so13605639wma.0
for <39258 <at> debbugs.gnu.org>; Sun, 03 May 2020 08:02:20 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
h=from:to:cc:subject:date:message-id:in-reply-to:references
:mime-version:content-transfer-encoding;
bh=y04cqSlE39HhVd4m/qFcpwd9f0fVV7i91JQx8qrUMzw=;
b=iY8ptmiXpAF652gZ6YEcpqqPhwG6//eBSIgM17SO0LVy6U4+0YCBEcQdJgyOYotVXl
nKm/Eg1lpUGKNc0fsq+yW1RKdx3l5f9WGdU/GQ7X0tU0qXBwP+pxDDaeY1CfURc0ws/6
+Ied4NsLnPqGcitldVLl4FxgOS/nUkeGytw3YoDs1u+h7ffR+kPH48V8ntAOlXZouM24
Lh/EF0bC9GsGOE+f5SJL91DWlBqjZ/zA9YUv0kzyuqR1ffw+vTD6pwY7tp1kW7DL5xdi
Y5qpfB6fgXGM5RyJbTPK3WLLP/J5l98U5L6x/hDa52l420ltWsZzFCgccoArXMrvnzGn
7ADg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20161025;
h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to
:references:mime-version:content-transfer-encoding;
bh=y04cqSlE39HhVd4m/qFcpwd9f0fVV7i91JQx8qrUMzw=;
b=CFh9bbvHqcwlvgDhaf2+hemC2FMfPQebkmCD1rJ/O97XEmBiw0C284YxEaazFpKCLY
vwqNvsZRxuL/jsCSpwy2ZeoVAxu0wlpb7vXPi3d8Skle1Au5W9wQCUSt5x3XNgBmGaox
qM/l2KQF/V96Ky2XB4V59M4s9nMBTYou9z22g4ESMWDI68sKoIJwKeU3EollkIN5QTee
8cz8y70nrjGjoBKAucqTKV9Bcn531iMfXelDmEHFtbmklzPVJCsZyqOvYkgIs6FM3ZvT
yTLjnA1XD3saFHXkVnj28anpu7bgOPvuDXwq5BhiSbc+CrH6JKd/E4fo9C5OjHmrXfGd
wRPg==
X-Gm-Message-State: AGi0Pubt5C+XHK11zW7PLhdWHLBjo5tgHCyDPNkokYkVBUHttr9lRcvx
BEbekYrIXaT6hwlbfKu9hMjMKxSk
X-Google-Smtp-Source: APiQypI7s7YEhwiDdvCYaY67HeKN5HTLiuK/MeDcBvi7CARVNcjedPgpTuG6SxDuQZEDVY7ATFcFGw==
X-Received: by 2002:a1c:24d4:: with SMTP id k203mr9857854wmk.49.1588518134442;
Sun, 03 May 2020 08:02:14 -0700 (PDT)
Received: from localhost.localdomain (57.246.195.77.rev.sfr.net.
[77.195.246.57])
by smtp.gmail.com with ESMTPSA id x13sm9787829wmc.5.2020.05.03.08.02.13
(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
Sun, 03 May 2020 08:02:14 -0700 (PDT)
From: zimoun <zimon.toutoune@HIDDEN>
To: 39258 <at> debbugs.gnu.org
Subject: [PATCH v4 1/3] DRAFT packages: Add fields to packages cache.
Date: Sun, 3 May 2020 17:01:52 +0200
Message-Id: <20200503150154.26532-2-zimon.toutoune@HIDDEN>
X-Mailer: git-send-email 2.26.1
In-Reply-To: <20200503150154.26532-1-zimon.toutoune@HIDDEN>
References: <20200503150154.26532-1-zimon.toutoune@HIDDEN>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 39258
Cc: arunisaac@HIDDEN, mail@HIDDEN, ludo@HIDDEN,
zimoun <zimon.toutoune@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)
---
gnu/packages.scm | 51 +++++++++++++++++++++++++++++++++++++++++++-----
1 file changed, 46 insertions(+), 5 deletions(-)
diff --git a/gnu/packages.scm b/gnu/packages.scm
index d22c992bb1..fa18f81487 100644
--- a/gnu/packages.scm
+++ b/gnu/packages.scm
@@ -33,6 +33,8 @@
#:use-module (guix profiles)
#:use-module (guix describe)
#:use-module (guix deprecation)
+ #:use-module (guix build-system)
+ #:use-module (guix licenses)
#:use-module (ice-9 vlist)
#:use-module (ice-9 match)
#:use-module (ice-9 binary-ports)
@@ -212,7 +214,8 @@ package module."
(match vector
(#(name version module symbol outputs
supported? deprecated?
- file line column)
+ file line column
+ _ _ _ _ _ _ _ _ _ _)
(proc name version result
#:outputs outputs
#:location (and file
@@ -269,7 +272,11 @@ package names. Return #f on failure."
(match item
(#(name version module symbol outputs
supported? deprecated?
- file line column)
+ file line column
+ synopsis description home-page
+ build-system-name build-system-description
+ supported-systems direct-inputs
+ license-name license-uri license-comment)
(vhash-cons name item vhash))))
vlist-null
lst))
@@ -316,7 +323,8 @@ decreasing version order."
(if (and (cache-is-authoritative?) cache)
(match (cache-lookup cache name)
(#f #f)
- ((#(_ versions modules symbols _ _ _ _ _ _) ...)
+ ((#(_ versions modules symbols _ _ _ _ _ _
+ _ _ _ _ _ _ _ _ _ _) ...)
(fold (lambda (version* module symbol result)
(if (or (not version)
(version-prefix? version version*))
@@ -339,7 +347,8 @@ matching NAME and VERSION."
(#f '())
((#(name versions modules symbols outputs
supported? deprecated?
- files lines columns) ...)
+ files lines columns
+ _ _ _ _ _ _ _ _ _ _) ...)
(fold (lambda (version* file line column result)
(if (and file
(or (not version)
@@ -401,7 +410,39 @@ reducing the memory footprint."
`(,(location-file loc)
,(location-line loc)
,(location-column loc))
- '(#f #f #f))))
+ '(#f #f #f)))
+
+ ,(package-synopsis package)
+ ,(package-description package)
+ ,(package-home-page package)
+
+ ,@(let ((build-system
+ (package-build-system package)))
+ `(,(symbol->string
+ (build-system-name build-system))
+ ,(build-system-description build-system)))
+
+ ,(package-transitive-supported-systems package)
+
+ ,(delete-duplicates
+ (sort (map package-full-name
+ (match (package-direct-inputs package)
+ (((labels inputs . _) ...)
+ (filter package? inputs))))
+ string<?))
+
+ ,@(match (package-license package)
+ (((? license? licenses) ...) ; multilicenses
+ `(,(string-join (map license-name licenses)
+ ", ")
+ ,(license-uri (car licenses)) ;TODO: names>uris?
+ ;; see gpl1+ comment #f
+ ,(license-comment (car licenses))))
+ ((? license? license)
+ `(,(license-name license)
+ ,(license-uri license)
+ ,(license-comment license)))
+ (_ '(#f #f #f))))
result)
(vhash-consq package #t seen))))))
(_
--
2.26.1
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.
Received: (at 39258) by debbugs.gnu.org; 3 May 2020 15:02:19 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun May 03 11:02:18 2020
Received: from localhost ([127.0.0.1]:57633 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1jVG8I-0005AE-Gt
for submit <at> debbugs.gnu.org; Sun, 03 May 2020 11:02:18 -0400
Received: from mail-wr1-f49.google.com ([209.85.221.49]:38465)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <zimon.toutoune@HIDDEN>) id 1jVG8G-0005A0-Rp
for 39258 <at> debbugs.gnu.org; Sun, 03 May 2020 11:02:17 -0400
Received: by mail-wr1-f49.google.com with SMTP id x17so17811311wrt.5
for <39258 <at> debbugs.gnu.org>; Sun, 03 May 2020 08:02:16 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
h=from:to:cc:subject:date:message-id:mime-version
:content-transfer-encoding;
bh=XpDBkCV1DffKgjJoHS+zSKi+yRynZG1skgetGV1GmE8=;
b=sNCxwcYe9mZD152kYS+gaBWjO8yxXYXhLP/IU5XiqgN2FGodHdyt8+nmuyY5x7XVrR
LABRsXXTvnEx4lb40d5p1PreOpVjEaHkx4yONpi4yWU/FHdjuQbM83CBpBvvxDh41Z02
X3rwY2NtYFhsMG8vCn7aXw45esgeO/sfaPZds3Y8yVnipxAWKrUaFN+o8v8yaXtE8NAo
QsROiwJljO+wDpppLM+OmW+PBM3FDnf3U7gFQBXKVeE+t/x2OJliOKkSj4GCkhaOK0QD
15WO6P2SbBEfF4CZ4XkoidGa6JN2GcnVKs5a3IIgcbeuRbfspX4XRq40a2IuaixQernM
18eA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20161025;
h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version
:content-transfer-encoding;
bh=XpDBkCV1DffKgjJoHS+zSKi+yRynZG1skgetGV1GmE8=;
b=X4A0XfTcJ4E+vcnd/9SEpitajfeG1lMEzyzSAbDHO//eADCWBhWHPv5G+NfpXzCwtd
wusA3WSdG+SoiRoZZ0TKXG2w3/ug6I6ulhd/s0ncW7arv0JvejurB7FePi4BTa2z1Sit
Dy3+PIGllY/RNDv+9BivgBz5zRyuq0dYPT63T7nZ6Z54MJKekz7mfrX1xi4X9WaZ0io/
aIKxtNMCFjd8se3wu7ZIMbV06CEIgjz7tWTQe4Q7QrZcruwVFy9+QfU3kfXO9J7/1Cj4
hTOGuEXDWJtRNM+E/QPt/p5/ziBqRxZ7i1IIvR7jULjJ/Zg0L8bnJCKYT/Yx8zmCErG5
5WyQ==
X-Gm-Message-State: AGi0PuZRz/xczQIgAV5VVEKewAc7cSIBSPPmTJVj12iPzDDpwZEG28g/
dGKURhLY10Ohn68Y1tErBGHjBGXN
X-Google-Smtp-Source: APiQypL0vX7OC3rAki25pkau40GUKhiwicXoCdr4XaIdqEhuz+OuTbDDw72MjlNspao31qKHR2A/8w==
X-Received: by 2002:adf:dd8a:: with SMTP id x10mr14711946wrl.308.1588518130518;
Sun, 03 May 2020 08:02:10 -0700 (PDT)
Received: from localhost.localdomain (57.246.195.77.rev.sfr.net.
[77.195.246.57])
by smtp.gmail.com with ESMTPSA id x13sm9787829wmc.5.2020.05.03.08.02.09
(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
Sun, 03 May 2020 08:02:09 -0700 (PDT)
From: zimoun <zimon.toutoune@HIDDEN>
To: 39258 <at> debbugs.gnu.org
Subject: [PATCH v4 0/3] Faster cache generation (similar as v3)
Date: Sun, 3 May 2020 17:01:51 +0200
Message-Id: <20200503150154.26532-1-zimon.toutoune@HIDDEN>
X-Mailer: git-send-email 2.26.1
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 39258
Cc: arunisaac@HIDDEN, mail@HIDDEN, ludo@HIDDEN,
zimoun <zimon.toutoune@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)
Dear,
The aim of this version v4 is to keep the same searching performances as the previous version v3 but to drastically reduce the generation of the cache. On my laptop, the overhead is now 4 seconds; compared to more than 20 seconds for v2 and v3.
--8<---------------cut here---------------start------------->8---
# default
time guix build /gnu/store/0nfpp82mqglpwvl1nbfpaphw5db2ivcp-guix-package-cache.drv --check
# v4
time guix build /gnu/store/y78gfh1n7m3kyrj8wsqj25qc2cbc1a4d-guix-package-cache.drv --check
--8<---------------cut here---------------end--------------->8---
| | default | v4 |
|------+----------+-----------|
| real | 0m6.012s | 0m10.244s |
| user | 0m0.541s | 0m0.542s |
| sys | 0m0.033s | 0m0.032s |
In the version v3, the cache is built using 'cons' and 'fold-packages' (wrapper to 'fold-module-public-variables'). The version v4 modifies -- by adding other information -- the function 'generate-package-cache' which uses 'vhash' and 'fold-module-public-variables*'.
Therefore the cache '/lib/guix/package.cache' contains more information. (The v4 structure of 'package.cache' is a quick draft, so details should be discussed and an interesting move should to have a structured (binary and all strings) S-exp; because it should become an entry point to export the packages list to JSON. WDYT?)
Now, we are comparing apples to apples and the cost to compute BM25 (v2) is not free at all. Remember that BM25 is the state-of-the-art of information retrieval (relevance ranking) and it is delegated to Xapian (v2). I do not know if there is perfomance bottleneck between Guix, Guile-Xapian and Xapian itself but for sure the computation of BM25 is not free. More about that soon.
To be clear about BM25 and caching, what I have in mind is:
1. "guix search --build-index" optionally done by the user if they wants for example the BM25 ranking.
2. Use BM25 metrics to detect poor package meta-data (synopsis and description); if it worth why not add another checker to "guix lint".
However, ranking is another story and I am not convinced yet if BM25 fits Guix needs or not.
* Details
~~~~~~~~~
The pacthes applies against the commit a357849f5b (and it is not yet rebased).
--8<---------------cut here---------------start------------->8---
time ./pre-env-inst guix pull --branch=search-v4 --url=$PWD -p /tmp/v4
--8<---------------cut here---------------end--------------->8---
Similar test than the previous benchmark (cold cache).
--8<---------------cut here---------------start------------->8---
time ./pre-env-inst /tmp/v4/bin/guix search crypto library \
| recsel -P name | grep libb2
name: libb2
real 0m0.784s
user 0m0.810s
sys 0m0.037s
--8<---------------cut here---------------end--------------->8---
And the option '--load-path' turns off the cache and it fallbacks to the usual 'fold-package'.
--8<---------------cut here---------------start------------->8---
time ./pre-inst-env /tmp/v4/bin/guix search -L /tmp/my-pkgs crypto library \
| recsel -C -p name | grep libb2
name: libb2
real 0m2.446s
user 0m1.872s
sys 0m0.187s
--8<---------------cut here---------------end--------------->8---
* Still draft
~~~~~~~~~~~~~
1. The name of 'fold-packages*' should be misleading since it does not return "true" packages.
--8<---------------cut here---------------start------------->8---
(define get-hello (p r)
(if (string=? (package-name p) "hello")
p
r))
(define no-cache (fold-packages get-hello '()))
(define from-cache (fold-packages* get-hello '()))
(equal? no-cache from-cache)
;;; #f
--8<---------------cut here---------------end--------------->8---
Another name for the procedure is welcome if it is an issue.
2. The function 'package->recutils' in 'guix/ui.scm' is modified but it is not the better.
--8<---------------cut here---------------start------------->8---
(match (package-supported-systems p)
(('cache supported-systems)
(string-join supported-systems))
(_
(string-join (package-transitive-supported-systems p)))))
--8<---------------cut here---------------end--------------->8---
However it avoids to duplicate code; as it is done in version v3.
3. Deprecated packages are displayed (bug in v3 too).
4. Impolite '@@' is used to access the private license construction.
5. Commit messages are incomplete, copyright header too, etc..
* Next?
~~~~~~~
IMHO, simply caching improves the current situation:
- a bit of extra time at pull time (less than 5s on my machine)
+ speed up at search time (2x faster)
* maintainable code?
Is it in the right direction?
Could you advise for a more compliant code?
Could you test on your machines to have another point of comparison?
Best regards,
simon
zimoun (3):
DRAFT packages: Add fields to packages cache.
DRAFT packages: Add new procedure 'fold-packages*'.
DRAFT guix package: Use cache in 'find-packages-by-description'.
gnu/packages.scm | 98 ++++++++++++++++++++++++++++++++++++++--
guix/scripts/package.scm | 2 +-
guix/ui.scm | 29 +++++++-----
tests/packages.scm | 31 +++++++++++++
4 files changed, 143 insertions(+), 17 deletions(-)
--
2.26.1
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.
Received: (at 39258) by debbugs.gnu.org; 30 Apr 2020 13:10:59 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Apr 30 09:10:59 2020
Received: from localhost ([127.0.0.1]:45486 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1jU8xu-0001C3-L2
for submit <at> debbugs.gnu.org; Thu, 30 Apr 2020 09:10:59 -0400
Received: from mail-qk1-f194.google.com ([209.85.222.194]:41883)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <zimon.toutoune@HIDDEN>) id 1jU8xt-0001Br-Mn
for 39258 <at> debbugs.gnu.org; Thu, 30 Apr 2020 09:10:58 -0400
Received: by mail-qk1-f194.google.com with SMTP id n143so5548603qkn.8
for <39258 <at> debbugs.gnu.org>; Thu, 30 Apr 2020 06:10:57 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
h=mime-version:references:in-reply-to:from:date:message-id:subject:to
:cc:content-transfer-encoding;
bh=E5wfgat59i9DKwr5SUTbHiaFHWJ0ZNazTDKYHD/EA58=;
b=YgJKZAr+ToRI1yVi7gPo2/zsJ6CrLciFRJJv5dbOlckCOwQmSNm/HNwTLyvbBopaQH
LWuv5VERyx+WH+HClzWyVAdj2mnxBof800tyfkywzNlWB2Xm3b4K9N6SsGcJZ69tYGS7
nlDbs6+6o78/9UTTYB6KqlejsVf/WrIGVQ1Td4sypf7xv6ApEZ4zv6ax13NqwU2exvJl
vOkw0ESkoR04K5ZaIY4UHY9aNgnGGlZF3CMks9q0a6VkrFfSgkSD9usPuvreMSvUdZkA
ktQlG4rfn8PY6gL0sluZXBKnpeZyWfc3Lgrg4DKI8JK+oP3LQeYi1ssi6sfW6GnmkhBu
DXHA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20161025;
h=x-gm-message-state:mime-version:references:in-reply-to:from:date
:message-id:subject:to:cc:content-transfer-encoding;
bh=E5wfgat59i9DKwr5SUTbHiaFHWJ0ZNazTDKYHD/EA58=;
b=nLrRxKB9Lj6yCTlSPIGJecpNal+uG3dFqVxI3wZCz8xOX6P88RcqiDCEJlXNErDJth
/ieVnbRpIatfQyMm2/Kxdu8qV/ASzTc89RcVOffIPIHHZP2/hE545qdIH0BuXMc11RlO
fUoUUu4CNaFwZ7IV4iu0EMYFVhKv1Y4IigtQuRdC/JI4QXPpf2r9nA4OUT2+zD0ZnwQP
/FD0oSep4wVNvw10B45U/C9ZQeDJI9k/bTyQ4dltynsJjTqwIMC3ONNwsACbKshS1Pue
V4W0WM2P3HNtcdSAqi0A8mD+avI7IULyAaWZMC7/800FdYYr8+uEEDlqXpL53j/uTbTK
6jYw==
X-Gm-Message-State: AGi0PuaXZjgfrPdOS/aYGh9FPCl5rNi73Rx2fpgWjmqHUYCrzQ1bhrcs
nPNNizJewbx/ixxbkgEsucOVaBYorqrBS8/uxpk=
X-Google-Smtp-Source: APiQypLQNFXr2Exbjf+yr8jxXdnmhQkX7IXfdpOnvgMOj7/K/35cf9VeRjZXKK19flw3kHNlMb+QAvd4xNwihKBiO0c=
X-Received: by 2002:a05:620a:239:: with SMTP id
u25mr3157286qkm.304.1588252252100;
Thu, 30 Apr 2020 06:10:52 -0700 (PDT)
MIME-Version: 1.0
References: <CAJ3okZ0KP0bYuwShZoNkkFWrhGNPsUGOOfiL5azj_x8Q+GNqLA@HIDDEN>
<87lfmi2qod.fsf@HIDDEN>
In-Reply-To: <87lfmi2qod.fsf@HIDDEN>
From: zimoun <zimon.toutoune@HIDDEN>
Date: Thu, 30 Apr 2020 15:10:40 +0200
Message-ID: <CAJ3okZ04SYqisveQf5QFNx57s1bZ5iCGu7Szmx1drF5+-S_5Ng@HIDDEN>
Subject: Re: benchmark search: default vs v2 vs v3
To: =?UTF-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 39258
Cc: Arun Isaac <arunisaac@HIDDEN>,
Pierre Neidhardt <mail@HIDDEN>, 39258 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)
Hi Ludo,
On Sun, 26 Apr 2020 at 17:49, Ludovic Court=C3=A8s <ludo@HIDDEN> wrote:
> That=E2=80=99s only one criterion among others. I hear the argument that=
25s is
> =E2=80=9Cnothing=E2=80=9D compared to the rest, but it=E2=80=99s really a=
tradeoff. Like, if I
> spent a day optimizing =E2=80=98guix pull=E2=80=99 and managed to save 25=
s, I would find
> it nice. :-)
I am not sure to understand all what "guix pull" does.
Does "guix pull" compile all the scheme files under 'gnu/'? Probably
only recompiles the "new" files?
I do not know if it makes sense, but I just note this difference:
1. Search without compiling of all files under 'gnu/packages/'
2. Compile all the files under 'gnu/packages/' then search
3. Search with only the file gnu/packages/emacs-xyz.scm not compiled
(all the other files are compiled)
4. Compile the file above and then search
3b and 4b with gnu/packages/cobol.scm which is smaller than emacs-xyz.scm.
Results:
1) 1m43.312s
2) 0m1.301s (but 9m51.801s compiling)
3) 0m6.526s
4) 0m1.389s (1m8.670s compiling)
3b) 0m0.921s
4b) 0m0.924s (0m1.884s compiling)
Therefore, an option to reduce the time when pulling should to relax
the "compilation" for 'gnu/packages/' and 'gnu/services'; something
less optimized since the packages and services "just" need to be
transformed into bytecode to improve IO when reading them. Perhaps I
miss a point...
And maybe, it is similar than what Andy Wingo is proposing in [1].
[1] https://lists.gnu.org/archive/html/guix-devel/2020-04/msg00444.html
Cheers,
simon
--8<---------------cut here---------------start------------->8---
find gnu/packages -name "*.scm" -type f -exec touch {} \;
time ./pre-inst-env guix search gmsh | recsel -C -p name
;;; note: source file /home/simon/src/guix/wk/tmp/gnu/packages/abduco.scm
;;; newer than compiled /home/simon/src/guix/wk/tmp/gnu/packages/abdu=
co.go
[...]
;;; note: source file /home/simon/src/guix/wk/tmp/gnu/packages/zwave.scm
;;; newer than compiled /home/simon/src/guix/wk/tmp/gnu/packages/zwav=
e.go
name: gmsh
real 1m43.312s
user 2m19.318s
--8<---------------cut here---------------end--------------->8---
--8<---------------cut here---------------start------------->8---
find gnu/packages -name "*.scm" -type f -exec touch {} \;
time make -j4 && time ./pre-inst-env guix search gmsh | recsel -C -p name
make all-recursive
make[1]: Entering directory '/home/simon/src/guix/wk/tmp'
Making all in po/guix
make[2]: Entering directory '/home/simon/src/guix/wk/tmp/po/guix'
make[2]: Leaving directory '/home/simon/src/guix/wk/tmp/po/guix'
Making all in po/packages
make[2]: Entering directory '/home/simon/src/guix/wk/tmp/po/packages'
make[2]: Leaving directory '/home/simon/src/guix/wk/tmp/po/packages'
make[2]: Entering directory '/home/simon/src/guix/wk/tmp'
Compiling Scheme modules...
[ 0%] LOAD gnu/packages/abduco.scm
;;; note: source file ./gnu/packages/abduco.scm
;;; newer than compiled /home/simon/src/guix/wk/tmp/gnu/packages/abdu=
co.go
[...]
[100%] GUILEC gnu/packages/zwave.go
make[2]: Leaving directory '/home/simon/src/guix/wk/tmp'
make[1]: Leaving directory '/home/simon/src/guix/wk/tmp'
real 9m51.801s
user 29m18.938s
sys 0m5.822s
name: gmsh
real 0m1.301s
user 0m1.266s
sys 0m0.101s
--8<---------------cut here---------------end--------------->8---
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 26 Apr 2020 20:23:08 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Sun Apr 26 16:23:08 2020 Received: from localhost ([127.0.0.1]:35037 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1jSnnv-0005Ny-Qk for submit <at> debbugs.gnu.org; Sun, 26 Apr 2020 16:23:08 -0400 Received: from eggs.gnu.org ([209.51.188.92]:51936) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <ludo@HIDDEN>) id 1jSnnu-0005N4-Uz for 39258 <at> debbugs.gnu.org; Sun, 26 Apr 2020 16:23:07 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:51232) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from <ludo@HIDDEN>) id 1jSnno-0005cB-C5; Sun, 26 Apr 2020 16:23:00 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=51642 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from <ludo@HIDDEN>) id 1jSnnn-000429-VG; Sun, 26 Apr 2020 16:23:00 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN> To: zimoun <zimon.toutoune@HIDDEN> Subject: Re: benchmark search: default vs v2 vs v3 References: <CAJ3okZ0KP0bYuwShZoNkkFWrhGNPsUGOOfiL5azj_x8Q+GNqLA@HIDDEN> <87lfmi2qod.fsf@HIDDEN> <CAJ3okZ1AeYdprjmntvSYOq1-NNyvibJOatwPm20WVNfkm-RxkA@HIDDEN> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 8 =?utf-8?Q?Flor=C3=A9al?= an 228 de la =?utf-8?Q?R?= =?utf-8?Q?=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Sun, 26 Apr 2020 22:22:58 +0200 In-Reply-To: <CAJ3okZ1AeYdprjmntvSYOq1-NNyvibJOatwPm20WVNfkm-RxkA@HIDDEN> (zimoun's message of "Sun, 26 Apr 2020 19:01:41 +0200") Message-ID: <87pnbuyp25.fsf@HIDDEN> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 39258 Cc: Arun Isaac <arunisaac@HIDDEN>, Pierre Neidhardt <mail@HIDDEN>, 39258 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -3.3 (---) zimoun <zimon.toutoune@HIDDEN> skribis: >> Could you do, for v2 and v3: >> >> time guix build /gnu/store/=E2=80=A6-package-metadata-cache.drv --check > > Newbie me! :-) > > Two points: > > 1. It may not be reproducible... I am checking. > 2. The time seems similar (v2=3D26s and v3=3D29s) considering the time > to start Guile and so on. Good, so it means that=E2=80=99s not Xapian taking time here. Thanks again! Ludo=E2=80=99.
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.
Received: (at 39258) by debbugs.gnu.org; 26 Apr 2020 17:02:01 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Apr 26 13:02:01 2020
Received: from localhost ([127.0.0.1]:34794 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1jSkfJ-0005nE-6l
for submit <at> debbugs.gnu.org; Sun, 26 Apr 2020 13:02:01 -0400
Received: from mail-qt1-f174.google.com ([209.85.160.174]:45675)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <zimon.toutoune@HIDDEN>) id 1jSkfH-0005n1-Bg
for 39258 <at> debbugs.gnu.org; Sun, 26 Apr 2020 13:01:59 -0400
Received: by mail-qt1-f174.google.com with SMTP id 71so12333801qtc.12
for <39258 <at> debbugs.gnu.org>; Sun, 26 Apr 2020 10:01:59 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
h=mime-version:references:in-reply-to:from:date:message-id:subject:to
:cc:content-transfer-encoding;
bh=T46kNsd90cn0Z34pQMLDrYi8twHT3V0xsHpDUhklB0Q=;
b=uBmtYAFFOp8amJ+J+99mVQwyKVxZ27J76bi9Gie9tgSYUG3zgw7DB/Ncxj2jKrdNvi
gpi2RzKrxbwEP6u+kKVejkFjVg4F80nT7Dq8piSP685WpK2/n+bZ0yAL7jBbC2giW5qU
n+U5ALW+LrmiaC7ZVXvS9Mu/yZsFIQBgFM489j0tQCWNmwwKkCo5Ii2ud0aWIrUdlP1z
qVeP6lpT8po5jTszLXLlD8F+ANSeacK7sF8dxZ85uW5qS3gw2I4CRQf0Ues4rjodzS92
TQHzH9PkEnUhUzfgsAaEKKdOgONKLvbLXtKCVR2FHSNU8BsSmBG6pLrxpdw+BjJavnIF
AQAg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20161025;
h=x-gm-message-state:mime-version:references:in-reply-to:from:date
:message-id:subject:to:cc:content-transfer-encoding;
bh=T46kNsd90cn0Z34pQMLDrYi8twHT3V0xsHpDUhklB0Q=;
b=SJ+3dJ7e2bV0Y6cJYFofP4ZeuSYjjeMjZJ/m03oEJdd5ttPiDRDKIQgjCILCmjzJmb
RNT9RVZDGW7duJbfLrAgSDAAC6VWDetvEmhjesKfDXGugPGqTlaSROqhaekBfExrs7Uc
KbwxqfBM8+Hk28r1hoXXHFFj39Z0PMVnrakM/A/OpqPSGX/W67g4v9RdlYO8V67rIBoG
Wqn2K++W0L66PNAhq+/QVVIMrlstFiAarNDzWsNJRZO7oOs8T0/fo2aXqLT3aRZj6NB5
xchgbSruQXOAc+2G92uznmu76YI5j7DGtrZmNnqCAJUZ1at7C0mYmQW5lrIRGGATwn3o
rjDQ==
X-Gm-Message-State: AGi0PubXt5gzXKcxKWJGcni1Bb6A2JrmiJ+avVcd/n/NW1v/SqVbytqZ
YG/CIofn8hLSJErJfToIeJUuBFDgAWoZt5OQLIKfRQkn
X-Google-Smtp-Source: APiQypIX+jD7Tl9wPaDslxIZbSRAsH8BZOQVnrflsK2GWCPifw0z0yprzTklh+ktE3lOY/VTLRR6r3lM9OewoU7doGc=
X-Received: by 2002:ac8:19fd:: with SMTP id s58mr19572563qtk.354.1587920512567;
Sun, 26 Apr 2020 10:01:52 -0700 (PDT)
MIME-Version: 1.0
References: <CAJ3okZ0KP0bYuwShZoNkkFWrhGNPsUGOOfiL5azj_x8Q+GNqLA@HIDDEN>
<87lfmi2qod.fsf@HIDDEN>
In-Reply-To: <87lfmi2qod.fsf@HIDDEN>
From: zimoun <zimon.toutoune@HIDDEN>
Date: Sun, 26 Apr 2020 19:01:41 +0200
Message-ID: <CAJ3okZ1AeYdprjmntvSYOq1-NNyvibJOatwPm20WVNfkm-RxkA@HIDDEN>
Subject: Re: benchmark search: default vs v2 vs v3
To: =?UTF-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 39258
Cc: Arun Isaac <arunisaac@HIDDEN>,
Pierre Neidhardt <mail@HIDDEN>, 39258 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)
Hi Ludo,
On Sun, 26 Apr 2020 at 17:49, Ludovic Court=C3=A8s <ludo@HIDDEN> wrote:
> It does seem like Arun=E2=80=99s v3 (or maybe even v2) would work nicely.
The v3 is more interesting because it does not change the relevance
scoring and does not add other dependency.
However v2 is interesting to easily test BM25 which is another
relevance scoring... work in progress. :-)
> > The question is the tradeoff between: the slowdown of pull vs the
> > speedup of search. What is acceptable?
>
> That=E2=80=99s only one criterion among others. I hear the argument that=
25s is
> =E2=80=9Cnothing=E2=80=9D compared to the rest, but it=E2=80=99s really a=
tradeoff. Like, if I
> spent a day optimizing =E2=80=98guix pull=E2=80=99 and managed to save 25=
s, I would find
> it nice. :-)
And I expect that the middle-term roadmap would even decrease more the
computations of derivations. ;-)
> > $ time guix pull -C ~/.config/guix/default-channels.scm
>
> It also depends on what=E2=80=99s in that file, of course.
Contains only one line: %default-channels
See my wishlist ;-)
https://lists.gnu.org/archive/html/guix-devel/2020-04/msg00393.html
me: 2m13.693s
you: 0m57.916s
As we already discussed elsewhere, it is hard to "test" 'guix pull'.
Does it make sense to measure "guix pull"? As Chris (Marusich) did for
CDN.
> > Well, let remove the profiles and garbage collect the index files:
> >
> > rm /tmp/default /tmp/v{2,3}*
> > guix gc -D \
> > /gnu/store/g5c08vqsv31nkn2r0hr32dbrkhf3cvd8-guix-package-cache \
> > /gnu/store/8xbzhn81hmshagbgazmnr7xfps1cdsa3-guix-package-search-inde=
x \
> > /gnu/store/8j78b5c4ddic21gcx7wpbq2akjn7x7mr-guix-package-metadata-ca=
che
>
> Could you do, for v2 and v3:
>
> time guix build /gnu/store/=E2=80=A6-package-metadata-cache.drv --check
Newbie me! :-)
Two points:
1. It may not be reproducible... I am checking.
2. The time seems similar (v2=3D26s and v3=3D29s) considering the time
to start Guile and so on.
--8<---------------cut here---------------start------------->8---
guix gc --list-live | grep metadata
time /tmp/v3/bin/guix build
/gnu/store/jxs0abica8kjz1ppym95df97jk0qa9by-guix-package-metadata-cache.drv
--check
The following profile hook will be built:
/gnu/store/jxs0abica8kjz1ppym95df97jk0qa9by-guix-package-metadata-cache.=
drv
building package cache...
(repl-version 0 1 1)
Generating package metadata cache for
'/gnu/store/95mi525syinh08jmcd3q7a7a8mr1sykb-profile'...
(values (value "/gnu/store/zhp7wv87vr6iis0fa3ff925i5r04i08q-guix-package-me=
tadata-cache/lib/guix/package-metadata.cache"))
guix build: error: derivation
`/gnu/store/jxs0abica8kjz1ppym95df97jk0qa9by-guix-package-metadata-cache.dr=
v'
may not be deterministic: output
`/gnu/store/zhp7wv87vr6iis0fa3ff925i5r04i08q-guix-package-metadata-cache'
differs
real 0m29.788s
user 0m0.535s
sys 0m0.025s
--8<---------------cut here---------------end--------------->8---
> That we=E2=80=99ll give us the exact cost of that part. It=E2=80=99ll be=
interesting
> especially in the Xapian case, which we expected to be higher.
--8<---------------cut here---------------start------------->8---
time /tmp/v2/bin/guix build
/gnu/store/w0dhl2n3ngi4v2ld8lprkqjl1g1q2m4p-guix-package-search-index.drv
--check
The following profile hook will be built:
/gnu/store/w0dhl2n3ngi4v2ld8lprkqjl1g1q2m4p-guix-package-search-index.dr=
v
running profile hook of type 'package-search-index'...
(repl-version 0 1 1)
Generating package search index for
'/gnu/store/wiinj9nrb45wlf2cgbgkjl9chxz9cb9b-profile'...
(values (value "/gnu/store/8xbzhn81hmshagbgazmnr7xfps1cdsa3-guix-package-se=
arch-index/lib/guix/package-search.index"))
guix build: error: derivation
`/gnu/store/w0dhl2n3ngi4v2ld8lprkqjl1g1q2m4p-guix-package-search-index.drv'
may not be deterministic: output
`/gnu/store/8xbzhn81hmshagbgazmnr7xfps1cdsa3-guix-package-search-index'
differs
real 0m26.552s
user 0m0.626s
sys 0m0.046s
--8<---------------cut here---------------end--------------->8---
It is not higher. Why should it be?
Considering aside the issue of reproducibility -- which should be one!
-- well, should be possible to download the index file as any other
substitute?
Cheers,
simon
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.
Received: (at 39258) by debbugs.gnu.org; 26 Apr 2020 15:49:30 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Apr 26 11:49:30 2020
Received: from localhost ([127.0.0.1]:33960 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1jSjX7-00030f-Q8
for submit <at> debbugs.gnu.org; Sun, 26 Apr 2020 11:49:30 -0400
Received: from eggs.gnu.org ([209.51.188.92]:36370)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <ludo@HIDDEN>) id 1jSjX6-00030T-2V
for 39258 <at> debbugs.gnu.org; Sun, 26 Apr 2020 11:49:28 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:44907)
by eggs.gnu.org with esmtp (Exim 4.90_1)
(envelope-from <ludo@HIDDEN>)
id 1jSjWz-0002xl-3h; Sun, 26 Apr 2020 11:49:21 -0400
Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=49140 helo=ribbon)
by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
(Exim 4.82) (envelope-from <ludo@HIDDEN>)
id 1jSjWn-0002cf-9X; Sun, 26 Apr 2020 11:49:18 -0400
From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>
To: zimoun <zimon.toutoune@HIDDEN>
Subject: Re: benchmark search: default vs v2 vs v3
References: <CAJ3okZ0KP0bYuwShZoNkkFWrhGNPsUGOOfiL5azj_x8Q+GNqLA@HIDDEN>
X-URL: http://www.fdn.fr/~lcourtes/
X-Revolutionary-Date: 8 =?utf-8?Q?Flor=C3=A9al?= an 228 de la =?utf-8?Q?R?=
=?utf-8?Q?=C3=A9volution?=
X-PGP-Key-ID: 0x090B11993D9AEBB5
X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc
X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5
X-OS: x86_64-pc-linux-gnu
Date: Sun, 26 Apr 2020 17:49:06 +0200
In-Reply-To: <CAJ3okZ0KP0bYuwShZoNkkFWrhGNPsUGOOfiL5azj_x8Q+GNqLA@HIDDEN>
(zimoun's message of "Sun, 26 Apr 2020 05:54:21 +0200")
Message-ID: <87lfmi2qod.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 39258
Cc: Arun Isaac <arunisaac@HIDDEN>,
Pierre Neidhardt <mail@HIDDEN>, 39258 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)
Hi,
zimoun <zimon.toutoune@HIDDEN> skribis:
> 1) around 25 seconds added to "guix pull"... but I am more than often
> waiting around 10 minutes when pulling.
> 2) the speedup is clear: more than 2x.
Nice!
It does seem like Arun=E2=80=99s v3 (or maybe even v2) would work nicely.
> The question is the tradeoff between: the slowdown of pull vs the
> speedup of search. What is acceptable?
That=E2=80=99s only one criterion among others. I hear the argument that 2=
5s is
=E2=80=9Cnothing=E2=80=9D compared to the rest, but it=E2=80=99s really a t=
radeoff. Like, if I
spent a day optimizing =E2=80=98guix pull=E2=80=99 and managed to save 25s,=
I would find
it nice. :-)
> $ time guix pull -C ~/.config/guix/default-channels.scm
It also depends on what=E2=80=99s in that file, of course.
> Then, if I pull back from 8cf6d15 to '--commit=3Da357849f5b' then it take=
s:
>
> real 2m13.693s
> user 1m37.418s
> sys 0m0.666s
For me:
--8<---------------cut here---------------start------------->8---
$ guix describe
Generacio 139 Apr 13 2020 21:50:08 (nuna)
guix bad368b
repository URL: https://git.savannah.gnu.org/git/guix.git
branch: master
commit: bad368b0d794689f3a8a11b58f1ea4987938682e
$ time guix pull -p /tmp/test --commit=3Dbad368b0d794689f3a8a11b58f1ea49879=
38682e
Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.or=
g/git/guix.git'...
Building from this channel:
guix https://git.savannah.gnu.org/git/guix.git bad368b
[...]
real 0m57.916s
user 1m1.017s
sys 0m0.609s
--8<---------------cut here---------------end--------------->8---
(On a 2.6=C2=A0GHz i7 though.)
> Well, let remove the profiles and garbage collect the index files:
>
> rm /tmp/default /tmp/v{2,3}*
> guix gc -D \
> /gnu/store/g5c08vqsv31nkn2r0hr32dbrkhf3cvd8-guix-package-cache \
> /gnu/store/8xbzhn81hmshagbgazmnr7xfps1cdsa3-guix-package-search-index \
> /gnu/store/8j78b5c4ddic21gcx7wpbq2akjn7x7mr-guix-package-metadata-cache
Could you do, for v2 and v3:
time guix build /gnu/store/=E2=80=A6-package-metadata-cache.drv --check
?
That we=E2=80=99ll give us the exact cost of that part. It=E2=80=99ll be i=
nteresting
especially in the Xapian case, which we expected to be higher.
Thanks for the insightful benchmarks!
Ludo=E2=80=99.
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 26 Apr 2020 15:33:50 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Sun Apr 26 11:33:50 2020 Received: from localhost ([127.0.0.1]:33936 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1jSjHy-0002cn-HL for submit <at> debbugs.gnu.org; Sun, 26 Apr 2020 11:33:50 -0400 Received: from eggs.gnu.org ([209.51.188.92]:34398) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <ludo@HIDDEN>) id 1jSjHw-0002cZ-Pf for 39258 <at> debbugs.gnu.org; Sun, 26 Apr 2020 11:33:49 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:44677) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from <ludo@HIDDEN>) id 1jSjHo-00024n-Qt; Sun, 26 Apr 2020 11:33:40 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=49132 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from <ludo@HIDDEN>) id 1jSjHo-0001HT-Dw; Sun, 26 Apr 2020 11:33:40 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN> To: Pierre Neidhardt <mail@HIDDEN> Subject: Re: [bug#39258] [PATCH v3 1/3] guix: Generate package metadata cache. References: <20200327162654.18785-1-arunisaac@HIDDEN> <20200327162654.18785-2-arunisaac@HIDDEN> <87h7x8haor.fsf@HIDDEN> <CAJ3okZ1M=xvMS7ZQpTV5PgPKn83skLx7Vn6ty7ezY1MwsA=qaw@HIDDEN> <87r1wa48n0.fsf@HIDDEN> <87r1wa70wj.fsf@HIDDEN> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 8 =?utf-8?Q?Flor=C3=A9al?= an 228 de la =?utf-8?Q?R?= =?utf-8?Q?=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Sun, 26 Apr 2020 17:33:37 +0200 In-Reply-To: <87r1wa70wj.fsf@HIDDEN> (Pierre Neidhardt's message of "Sun, 26 Apr 2020 16:54:36 +0200") Message-ID: <87368q45ym.fsf@HIDDEN> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.4 (/) X-Debbugs-Envelope-To: 39258 Cc: Arun Isaac <arunisaac@HIDDEN>, 39258 <at> debbugs.gnu.org, zimoun <zimon.toutoune@HIDDEN> X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -0.6 (/) Hey! Pierre Neidhardt <mail@HIDDEN> skribis: > Ludovic Court=C3=A8s <ludo@HIDDEN> writes: > >> It=E2=80=99s complicated. As it stands, I=E2=80=99d rather not add over= head to =E2=80=98guix >> pull=E2=80=99, especially since current =E2=80=98guix search=E2=80=99 on= my SSD is fast enough >> and can hardly be made any faster. > > The question is, what is fast enough? I have an NVMe here that has a > throughput of some 2GB/s, and yet > > time guix search emacs > /dev/null > real 0m1.545s > user 0m1.938s > sys 0m0.080s That accounts for the time to render 864 entries: $ guix search emacs| grep ^name| wc -l 864 Compare with: $ time guix search emacs | head -100 > /dev/null real 0m0.674s user 0m0.802s sys 0m0.048s Again, this is not to say it cannot be improved, but it=E2=80=99s quite a challenge to do better on such hardware. Though as discussed with Arun, there may be low-hanging optimization fruits in Texinfo parsing and rendering. I guess we need to go ahead fire up statprof now. :-) Ludo=E2=80=99.
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 26 Apr 2020 15:06:07 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Sun Apr 26 11:06:07 2020 Received: from localhost ([127.0.0.1]:33909 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1jSir9-0001wh-BD for submit <at> debbugs.gnu.org; Sun, 26 Apr 2020 11:06:07 -0400 Received: from mail-qv1-f65.google.com ([209.85.219.65]:38876) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <zimon.toutoune@HIDDEN>) id 1jSir7-0001w6-6N for 39258 <at> debbugs.gnu.org; Sun, 26 Apr 2020 11:06:05 -0400 Received: by mail-qv1-f65.google.com with SMTP id t8so7206808qvw.5 for <39258 <at> debbugs.gnu.org>; Sun, 26 Apr 2020 08:06:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=4Z3aagJ+zj4Y0QN34WKwFH7cIwoOacmM9VPvrz9ApsA=; b=u50I1VVmVqIrGNNb+7GRusqrTTayyLKJpkBRiQMh6Ty6w6Lhy4b19mmqWIJG2M2f2d 3gdo+axTUkPpPDVyF4w/8p8PLDNiD/Kp+tx3jrY93q2+uHFA1tZmJTkGTsx36WNId5lf xk62VhwTuKMJlvdEVPCbU5sg7hEKBdVnxzhuuH8Vq5CCWRdKHc5npdn03uz41NN/T264 Y4/n6BNRpU4qEaeXmsxa2NShms749ubcfjBy734avjy4VJpnRU9qaun1HQTzpfvm0HH/ Z4b830tbdmenruL9RjswH1nnHm+GgOzJJCHxKPh3J9VBVw/VBO62HS9+W1V5jRMxwuVy Z0ZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=4Z3aagJ+zj4Y0QN34WKwFH7cIwoOacmM9VPvrz9ApsA=; b=Z8rmCpzV7lOjQ1t2lfKeBp2394t3A+mL1zFojq9A1WYKpLn96krakcK8KkuxO8B30z Zbktq6FS3N4POSf+uIm7b6k+sU0QpThNV2hG0nfB4HToPky3QP9Hdp2UAzlwQm9QBbuz 0ocLqdiYZ4nrpHQMaFOejbUIOHuZbO+RmTskP94RJ0BFU5oxfeO9tifZnz2HE64qbW94 8r8ViEK6MfLaNS6AZTGCUN945CCVvPLOHbXo9ZkSA4czOycS1GlhKmM8jDaXcyGpaq8m T9MD7XKw28VvhbCotPa5i7C5sQLQWPJoxjM6rhfSIRSRu+Xx7KzCzM8D+utDANWeXSbZ W8Cg== X-Gm-Message-State: AGi0PuZ7OpQuySt6dvGhGnAfp2zfkkx6BVUnQOiEpYpWyNxzTXQsUjmu Rhzv9gWpxN4BCBn/GrJ/Qdc7V9Hu72gGwW7llukJBU4+ X-Google-Smtp-Source: APiQypKiP47td7abqgE6NPGVNeO8z9cLB4YFKmNXYHoRHHgMv69ia9jUQrx3tJVb0SDvAAat7UYBmYIOx1vAKbkKrHM= X-Received: by 2002:ad4:5604:: with SMTP id ca4mr18428357qvb.6.1587913559722; Sun, 26 Apr 2020 08:05:59 -0700 (PDT) MIME-Version: 1.0 References: <20200327162654.18785-1-arunisaac@HIDDEN> <20200327162654.18785-2-arunisaac@HIDDEN> <87h7x8haor.fsf@HIDDEN> <CAJ3okZ1M=xvMS7ZQpTV5PgPKn83skLx7Vn6ty7ezY1MwsA=qaw@HIDDEN> <87r1wa48n0.fsf@HIDDEN> In-Reply-To: <87r1wa48n0.fsf@HIDDEN> From: zimoun <zimon.toutoune@HIDDEN> Date: Sun, 26 Apr 2020 17:05:48 +0200 Message-ID: <CAJ3okZ1p5OE6451Ze+qSn2q7Xh-U5b7oqMbbuBkGxWM+tQ-GYQ@HIDDEN> Subject: Re: [bug#39258] [PATCH v3 1/3] guix: Generate package metadata cache. To: =?UTF-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 39258 Cc: Arun Isaac <arunisaac@HIDDEN>, Pierre Neidhardt <mail@HIDDEN>, 39258 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) Hi Ludo, On Sun, 26 Apr 2020 at 16:35, Ludovic Court=C3=A8s <ludo@HIDDEN> wrote: > Realistically though, I understand that things are different on slower > machines and/or spinning disks. That=E2=80=99s why I=E2=80=99m intereste= d in seeing how > Arun=E2=80=99s proposed changes can affect such machines. I understand. I have done a small benchmark [1] of the 3 ways: the current, the v2 using Xapian (which is not an option on the long term) and the v3. My "slower" machine is at my office... but it provides already interesting numbers, IMHO. [1] http://issues.guix.gnu.org/39258#78 > If, as a bonus, it allows us to have an inverted index and thus improve > the quality of search results, that=E2=80=99s great! This "issue" is: any improvement on both sides performance and accuracy would add an somehow extra cost. The question is what is the maximum users would accept to pay for? Well, it is complicated as you said. :-) A trade off between extra cost, maintenance, complexity, etc is not easy to draw, as you said too elsewhere. I am seeing all that as experimental: explore ideas to see if they are worth or not. And what should be concluded now could change in the (near) future; for example if the computations of derivations are faster, resulting on "guix pull" faster, etc.. Cheer, simon
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.
Received: (at 39258) by debbugs.gnu.org; 26 Apr 2020 14:54:47 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Apr 26 10:54:47 2020
Received: from localhost ([127.0.0.1]:33895 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1jSigA-0001e2-OB
for submit <at> debbugs.gnu.org; Sun, 26 Apr 2020 10:54:46 -0400
Received: from relay4-d.mail.gandi.net ([217.70.183.196]:42639)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <mail@HIDDEN>) id 1jSig9-0001do-15
for 39258 <at> debbugs.gnu.org; Sun, 26 Apr 2020 10:54:45 -0400
X-Originating-IP: 78.199.129.170
Received: from mimimi (moi44-1-78-199-129-170.fbx.proxad.net [78.199.129.170])
(Authenticated sender: mail@HIDDEN)
by relay4-d.mail.gandi.net (Postfix) with ESMTPSA id 28DC4E0004;
Sun, 26 Apr 2020 14:54:36 +0000 (UTC)
From: Pierre Neidhardt <mail@HIDDEN>
To: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>, zimoun
<zimon.toutoune@HIDDEN>
Subject: Re: [bug#39258] [PATCH v3 1/3] guix: Generate package metadata cache.
In-Reply-To: <87r1wa48n0.fsf@HIDDEN>
References: <20200327162654.18785-1-arunisaac@HIDDEN>
<20200327162654.18785-2-arunisaac@HIDDEN> <87h7x8haor.fsf@HIDDEN>
<CAJ3okZ1M=xvMS7ZQpTV5PgPKn83skLx7Vn6ty7ezY1MwsA=qaw@HIDDEN>
<87r1wa48n0.fsf@HIDDEN>
Date: Sun, 26 Apr 2020 16:54:36 +0200
Message-ID: <87r1wa70wj.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="=-=-=";
micalg=pgp-sha256; protocol="application/pgp-signature"
X-Spam-Score: 2.5 (++)
X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org",
has NOT identified this incoming email as spam. The original
message has been attached to this so you can view it or label
similar future email. If you have any questions, see
the administrator of that system for details.
Content preview: Hi Ludo! Ludovic Courtès <ludo@HIDDEN> writes: > It’s
complicated. As it stands, I’d rather not add overhead to ‘guix > pull’,
especially since current ‘guix search’ on my SSD is fast enough > and
can hardly be made any faster.
Content analysis details: (2.5 points, 10.0 required)
pts rule name description
---- ---------------------- --------------------------------------------------
0.0 URIBL_BLOCKED ADMINISTRATOR NOTICE: The query to URIBL was
blocked. See
http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block
for more information.
[URIs: gnu.org]
-0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at https://www.dnswl.org/,
low trust
[217.70.183.196 listed in list.dnswl.org]
-0.0 RCVD_IN_MSPIKE_H3 RBL: Good reputation (+3)
[217.70.183.196 listed in wl.mailspike.net]
0.1 URIBL_SBL_A Contains URL's A record listed in the Spamhaus SBL
blocklist
[URIs: ambrevar.xyz]
0.6 URIBL_SBL Contains an URL's NS IP listed in the Spamhaus SBL
blocklist
[URIs: ambrevar.xyz]
-0.0 SPF_PASS SPF: sender matches SPF record
0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record
2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs
[URI: ambrevar.xyz (xyz)]
0.5 FROM_SUSPICIOUS_NTLD From abused NTLD
-0.0 RCVD_IN_MSPIKE_WL Mailspike good senders
X-Debbugs-Envelope-To: 39258
Cc: Arun Isaac <arunisaac@HIDDEN>, 39258 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: 2.5 (++)
X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org",
has NOT identified this incoming email as spam. The original
message has been attached to this so you can view it or label
similar future email. If you have any questions, see
the administrator of that system for details.
Content preview: Hi Ludo! Ludovic Courtès <ludo@HIDDEN> writes: > It’s
complicated. As it stands, I’d rather not add overhead to ‘guix > pull’,
especially since current ‘guix search’ on my SSD is fast enough > and
can hardly be made any faster.
Content analysis details: (2.5 points, 10.0 required)
pts rule name description
---- ---------------------- --------------------------------------------------
0.0 URIBL_BLOCKED ADMINISTRATOR NOTICE: The query to URIBL was
blocked. See
http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block
for more information.
[URIs: gnu.org]
-0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at https://www.dnswl.org/,
low trust
[217.70.183.196 listed in list.dnswl.org]
-0.0 RCVD_IN_MSPIKE_H3 RBL: Good reputation (+3)
[217.70.183.196 listed in wl.mailspike.net]
0.1 URIBL_SBL_A Contains URL's A record listed in the Spamhaus SBL
blocklist
[URIs: ambrevar.xyz]
0.6 URIBL_SBL Contains an URL's NS IP listed in the Spamhaus SBL
blocklist
[URIs: ambrevar.xyz]
-0.0 SPF_PASS SPF: sender matches SPF record
0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record
2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs
[URI: ambrevar.xyz (xyz)]
0.5 FROM_SUSPICIOUS_NTLD From abused NTLD
1.0 BULK_RE_SUSP_NTLD Precedence bulk and RE: from a suspicious TLD
-0.0 RCVD_IN_MSPIKE_WL Mailspike good senders
-1.0 MAILING_LIST_MULTI Multiple indicators imply a widely-seen list
manager
--=-=-=
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Hi Ludo!
Ludovic Court=C3=A8s <ludo@HIDDEN> writes:
> It=E2=80=99s complicated. As it stands, I=E2=80=99d rather not add overh=
ead to =E2=80=98guix
> pull=E2=80=99, especially since current =E2=80=98guix search=E2=80=99 on =
my SSD is fast enough
> and can hardly be made any faster.
The question is, what is fast enough? I have an NVMe here that has a
throughput of some 2GB/s, and yet
=2D-8<---------------cut here---------------start------------->8---
time guix search emacs > /dev/null
real 0m1.545s
user 0m1.938s
sys 0m0.080s
=2D-8<---------------cut here---------------end--------------->8---
on a hot cache, which is too slow in my opinion :p
Mildly impatient users might be slightly discouraged from iterating
search queries.
It also makes `guix search` very impractical to use in (non-guile)
script. Which is too bad considering that the recsel-formatting makes
`guix search` a very good candidate for scripting.
Cheers!
=2D-=20
Pierre Neidhardt
https://ambrevar.xyz/
--=-=-=
Content-Type: application/pgp-signature; name="signature.asc"
-----BEGIN PGP SIGNATURE-----
iQEzBAEBCAAdFiEEUPM+LlsMPZAEJKvom9z0l6S7zH8FAl6loKwACgkQm9z0l6S7
zH8XxAf+N5eNs9ZFVdD/bZ6EjRN8lm4vH3L8FobMWtGptX86qlvgayEwferhobfE
ogycDjCFVp0pXmYw2O60T9mY5lQn6cghfJS2zZqjFrgLMo16PtGh7sQVu28AMQOq
y2j1plUYkXlRPyw0qQRONjKPaLxZ9p0fBtnQubFvjTAXMTf28TSHX+ZJfE5/ESvd
cgZ7y4nMkYrBXqdBKyWLJ3RNqyeLCgY0TeeE3blYHu4iMv+2xzUIDLy1rsg5Xlb0
ttsIXAnmVBrx4nSCG4Kj7irGW6YTlg0PC+73Dt2VU4yuT7CFUEpApA7f88D1U4vs
pQuJDRuwnQrC4toal1EYLedup5beLw==
=B9Hq
-----END PGP SIGNATURE-----
--=-=-=--
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 26 Apr 2020 14:35:59 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Sun Apr 26 10:35:59 2020 Received: from localhost ([127.0.0.1]:33882 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1jSiNz-0007Zm-LJ for submit <at> debbugs.gnu.org; Sun, 26 Apr 2020 10:35:59 -0400 Received: from eggs.gnu.org ([209.51.188.92]:55972) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <ludo@HIDDEN>) id 1jSiNy-0007Za-ES for 39258 <at> debbugs.gnu.org; Sun, 26 Apr 2020 10:35:58 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:43941) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from <ludo@HIDDEN>) id 1jSiNs-0000RB-8W; Sun, 26 Apr 2020 10:35:52 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=49108 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from <ludo@HIDDEN>) id 1jSiNr-0000k6-MO; Sun, 26 Apr 2020 10:35:52 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN> To: zimoun <zimon.toutoune@HIDDEN> Subject: Re: [bug#39258] [PATCH v3 1/3] guix: Generate package metadata cache. References: <20200327162654.18785-1-arunisaac@HIDDEN> <20200327162654.18785-2-arunisaac@HIDDEN> <87h7x8haor.fsf@HIDDEN> <CAJ3okZ1M=xvMS7ZQpTV5PgPKn83skLx7Vn6ty7ezY1MwsA=qaw@HIDDEN> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 8 =?utf-8?Q?Flor=C3=A9al?= an 228 de la =?utf-8?Q?R?= =?utf-8?Q?=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Sun, 26 Apr 2020 16:35:47 +0200 In-Reply-To: <CAJ3okZ1M=xvMS7ZQpTV5PgPKn83skLx7Vn6ty7ezY1MwsA=qaw@HIDDEN> (zimoun's message of "Sun, 26 Apr 2020 11:48:36 +0200") Message-ID: <87r1wa48n0.fsf@HIDDEN> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 39258 Cc: Arun Isaac <arunisaac@HIDDEN>, Pierre Neidhardt <mail@HIDDEN>, 39258 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -3.3 (---) Hi Simon, zimoun <zimon.toutoune@HIDDEN> skribis: > On Fri, 24 Apr 2020 at 22:48, Ludovic Court=C3=A8s <ludo@HIDDEN> wrote: [...] >> Note that this is probably the place where we could eventually add the >> computation of an inverted index like zimoun suggested in >> <https://lists.gnu.org/archive/html/guix-devel/2020-01/msg00243.html>. > > We should first agree on the extra cost (time) we are ready to pay to > build improvements. It=E2=80=99s complicated. As it stands, I=E2=80=99d rather not add overhea= d to =E2=80=98guix pull=E2=80=99, especially since current =E2=80=98guix search=E2=80=99 on my= SSD is fast enough and can hardly be made any faster. Realistically though, I understand that things are different on slower machines and/or spinning disks. That=E2=80=99s why I=E2=80=99m interested = in seeing how Arun=E2=80=99s proposed changes can affect such machines. If, as a bonus, it allows us to have an inverted index and thus improve the quality of search results, that=E2=80=99s great! Thanks, Ludo=E2=80=99.
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 26 Apr 2020 09:48:54 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Sun Apr 26 05:48:54 2020 Received: from localhost ([127.0.0.1]:60870 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1jSdu9-0005cD-PX for submit <at> debbugs.gnu.org; Sun, 26 Apr 2020 05:48:53 -0400 Received: from mail-qk1-f195.google.com ([209.85.222.195]:43753) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <zimon.toutoune@HIDDEN>) id 1jSdu8-0005bx-O4 for 39258 <at> debbugs.gnu.org; Sun, 26 Apr 2020 05:48:53 -0400 Received: by mail-qk1-f195.google.com with SMTP id 20so14973273qkl.10 for <39258 <at> debbugs.gnu.org>; Sun, 26 Apr 2020 02:48:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=5Qf4nF7aA35MeErDf7R7dQ6u1MK8CUD+4TOA79pMooY=; b=RJHM0H84XyKYhBcrvXeTBlEC6lfPTvMht6YfNaiV7AU8iQAq/x+GNrEs7WC/QWBdGu A/1zOcbo5oWuxPKFblQgFU4Q+6AkZCQGfBAZEh6diKRTofCqKxvYmqSChTc0rEBa/Wc+ ff5/mpfaSUSxi3+EAxPNByUAc0QcZNRedzCA5F/YG+YBelKWVYtmJUVM/n9ZI1JpY7GD 45yag50gyhMDRLghBS7472jctH7jsD/MW2YcvqVv/wIiNAR0vk0E2yBqVE9ErTi5o0Tr 2Ij9MXBuuHDuv3EqQFIWZHxW0JdcINsezxgCKzGKrJP9jHtJs+iRZV5HE+DuhbYO3Ilh Jo0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=5Qf4nF7aA35MeErDf7R7dQ6u1MK8CUD+4TOA79pMooY=; b=j1legbDKTPyguzcLihiSKa5LXCIU/Pyokk1l3k+mjVhMOUC22qc4/ExQaR3ciZNHVj kdPCKBaW8nU0Qu50LtXtMM0jJU/+SF77vtDzdZPWBZpCwjSHtkvnoNpGM81njjqR+9J5 CmNBnmkSb8ym39GD+vZbw3bSSZ32WduDOaO6CUcMb7Gv5sxFizacY+m+YiiaeFVZgxBW 3oyvUJhwQ9rx2M0OFYbDygFgod56jZ2YFNVYANWiIzkxBcLbRvmkcQgXy79OV1UtVjQP 0SEnkdBBltqWC/OnD5Nu9hMzfUF881elH2VppuyKHJGssXIz67R56ByCLyDUXG9k/Ghk Z6Jg== X-Gm-Message-State: AGi0PuZW3/YGbxSnAA/KMZGvCgYXdkfoOxhVpTZGKBQR9cLtV9SD9Y4T 2O4gUyqqC4hgVTHWWxrI6uwW3TmKygocpArW+RM= X-Google-Smtp-Source: APiQypLr18mUlfERbTYIwexcMQwqgCSDJKljdE7BEUgacp+qgU+GvpEq+Mx6qbmKnCxlA7Q9xi9qUyRmtHjA+jBP68o= X-Received: by 2002:a37:4e05:: with SMTP id c5mr17198157qkb.232.1587894527191; Sun, 26 Apr 2020 02:48:47 -0700 (PDT) MIME-Version: 1.0 References: <20200327162654.18785-1-arunisaac@HIDDEN> <20200327162654.18785-2-arunisaac@HIDDEN> <87h7x8haor.fsf@HIDDEN> In-Reply-To: <87h7x8haor.fsf@HIDDEN> From: zimoun <zimon.toutoune@HIDDEN> Date: Sun, 26 Apr 2020 11:48:36 +0200 Message-ID: <CAJ3okZ1M=xvMS7ZQpTV5PgPKn83skLx7Vn6ty7ezY1MwsA=qaw@HIDDEN> Subject: Re: [bug#39258] [PATCH v3 1/3] guix: Generate package metadata cache. To: =?UTF-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 39258 Cc: Arun Isaac <arunisaac@HIDDEN>, Pierre Neidhardt <mail@HIDDEN>, 39258 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) On Fri, 24 Apr 2020 at 22:48, Ludovic Court=C3=A8s <ludo@HIDDEN> wrote: > > + (define (expand-cache package result) > > + (cons `#(,(package-name package) > > + ,(package-version package) > > + ,(delete-duplicates > > + (map package-full-name > > + (sort (filter package? (package-direct-inputs pack= age)) > > + package<?))) > > + ,(package-outputs package) > > + ,(package-supported-systems package) > > + ,(package-synopsis package) > > + ,(package-description package) > > + ,(package-home-page package) > > + ,(let ((location (package-location package))) > > + (list (location-file location) > > + (location-line location) > > + (location-column location)))) > > I was wondering if we could omit inputs, which are not that useful. Agree. > Note that this is probably the place where we could eventually add the > computation of an inverted index like zimoun suggested in > <https://lists.gnu.org/archive/html/guix-devel/2020-01/msg00243.html>. We should first agree on the extra cost (time) we are ready to pay to build improvements. See the lengthy message [1] about only the caching "inverted index" using the current 'relevance' scoring function. [1] http://issues.guix.gnu.org/39258#78 Cheers, simon
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.
Received: (at 39258) by debbugs.gnu.org; 26 Apr 2020 07:29:48 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Apr 26 03:29:48 2020
Received: from localhost ([127.0.0.1]:60800 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1jSbjX-0008QQ-W5
for submit <at> debbugs.gnu.org; Sun, 26 Apr 2020 03:29:48 -0400
Received: from relay5-d.mail.gandi.net ([217.70.183.197]:53717)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <mail@HIDDEN>) id 1jSbjW-0008QA-Bh
for 39258 <at> debbugs.gnu.org; Sun, 26 Apr 2020 03:29:46 -0400
X-Originating-IP: 78.199.129.170
Received: from mimimi (moi44-1-78-199-129-170.fbx.proxad.net [78.199.129.170])
(Authenticated sender: mail@HIDDEN)
by relay5-d.mail.gandi.net (Postfix) with ESMTPSA id 265A81C0008;
Sun, 26 Apr 2020 07:29:38 +0000 (UTC)
From: Pierre Neidhardt <mail@HIDDEN>
To: zimoun <zimon.toutoune@HIDDEN>, 39258 <at> debbugs.gnu.org, Arun Isaac
<arunisaac@HIDDEN>, Ludovic =?utf-8?Q?Court=C3=A8s?=
<ludo@HIDDEN>
Subject: Re: benchmark search: default vs v2 vs v3
In-Reply-To: <CAJ3okZ0KP0bYuwShZoNkkFWrhGNPsUGOOfiL5azj_x8Q+GNqLA@HIDDEN>
References: <CAJ3okZ0KP0bYuwShZoNkkFWrhGNPsUGOOfiL5azj_x8Q+GNqLA@HIDDEN>
Date: Sun, 26 Apr 2020 09:29:37 +0200
Message-ID: <877dy2902m.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="=-=-=";
micalg=pgp-sha256; protocol="application/pgp-signature"
X-Spam-Score: 2.5 (++)
X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org",
has NOT identified this incoming email as spam. The original
message has been attached to this so you can view it or label
similar future email. If you have any questions, see
the administrator of that system for details.
Content preview: Hi Simon, Thanks for taking the time to benchmark this, this
is very insightful! > Usually, my feeling about "guix pull" is... I am waiting!
Therefore, > I will not see this extra 25s because it is masked by all the
other > work "guix pull" is doing.
Content analysis details: (2.5 points, 10.0 required)
pts rule name description
---- ---------------------- --------------------------------------------------
0.0 URIBL_BLOCKED ADMINISTRATOR NOTICE: The query to URIBL was
blocked. See
http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block
for more information. [URIs: ambrevar.xyz]
-0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at https://www.dnswl.org/,
low trust [217.70.183.197 listed in list.dnswl.org]
-0.0 RCVD_IN_MSPIKE_H2 RBL: Average reputation (+2)
[217.70.183.197 listed in wl.mailspike.net]
0.1 URIBL_SBL_A Contains URL's A record listed in the Spamhaus SBL
blocklist [URIs: ambrevar.xyz]
0.6 URIBL_SBL Contains an URL's NS IP listed in the Spamhaus SBL
blocklist [URIs: ambrevar.xyz]
-0.0 SPF_PASS SPF: sender matches SPF record
0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record
2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs
[URI: ambrevar.xyz (xyz)]
0.5 FROM_SUSPICIOUS_NTLD From abused NTLD
X-Debbugs-Envelope-To: 39258
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: 2.5 (++)
X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org",
has NOT identified this incoming email as spam. The original
message has been attached to this so you can view it or label
similar future email. If you have any questions, see
the administrator of that system for details.
Content preview: Hi Simon, Thanks for taking the time to benchmark this, this
is very insightful! > Usually, my feeling about "guix pull" is... I am waiting!
Therefore, > I will not see this extra 25s because it is masked by all the
other > work "guix pull" is doing.
Content analysis details: (2.5 points, 10.0 required)
pts rule name description
---- ---------------------- --------------------------------------------------
0.0 URIBL_BLOCKED ADMINISTRATOR NOTICE: The query to URIBL was
blocked. See
http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block
for more information.
[URIs: ambrevar.xyz]
-0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at https://www.dnswl.org/,
low trust
[217.70.183.197 listed in list.dnswl.org]
-0.0 RCVD_IN_MSPIKE_H2 RBL: Average reputation (+2)
[217.70.183.197 listed in wl.mailspike.net]
0.1 URIBL_SBL_A Contains URL's A record listed in the Spamhaus SBL
blocklist
[URIs: ambrevar.xyz]
0.6 URIBL_SBL Contains an URL's NS IP listed in the Spamhaus SBL
blocklist
[URIs: ambrevar.xyz]
-0.0 SPF_PASS SPF: sender matches SPF record
0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record
2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs
[URI: ambrevar.xyz (xyz)]
-1.0 MAILING_LIST_MULTI Multiple indicators imply a widely-seen list
manager
0.5 FROM_SUSPICIOUS_NTLD From abused NTLD
1.0 BULK_RE_SUSP_NTLD Precedence bulk and RE: from a suspicious TLD
--=-=-=
Content-Type: text/plain
Content-Transfer-Encoding: quoted-printable
Hi Simon,
Thanks for taking the time to benchmark this, this is very insightful!
> Usually, my feeling about "guix pull" is... I am waiting! Therefore,
> I will not see this extra 25s because it is masked by all the other
> work "guix pull" is doing.
I agree and this is a very good point in my opinion.
While I don't expect nor do I need "guix pull" to complete immediately,
this is not true of "guix search".
As Simon suggested, maybe we can wrap a benchmark script together, post
it on the mailing list and ask member to report their results. Maybe
a few dozen results would give us a better idea of the numbers we are
dealing with.
Cheers!
=2D-=20
Pierre Neidhardt
https://ambrevar.xyz/
--=-=-=
Content-Type: application/pgp-signature; name="signature.asc"
-----BEGIN PGP SIGNATURE-----
iQEzBAEBCAAdFiEEUPM+LlsMPZAEJKvom9z0l6S7zH8FAl6lOGEACgkQm9z0l6S7
zH/HUwf/YH8o+NEQOqbycpcKL4sJe3GXB1HFmwVALSGRggRdguaqBcabj8JygHjx
/Oi0n47C69K27bJWsdu1BvfmHHdmmXtHIWIvMl4t1o5NT1Z11sDw7cUL4FDnfJ9Y
qV1gS0qVIfx5ccZAbtSSeAhg31IrrzmgXbfBIPksWIWzCF8US6s2ftSh3/QkbJnL
2AodteVHlm9rKaAQPJVQGcRVka7VdaDaH2zUXs1w6e5Q6cp/71oeoeok+QtAnqg5
7jwZceCUu3OA3sMCzabMAadnzC9w/xPJI5knR6Lb5HqtQla4slHgYBTVLtNdXxol
1itksg9vevWgqqwzHdJFr+jADoe1Ew==
=prUD
-----END PGP SIGNATURE-----
--=-=-=--
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.
Received: (at 39258) by debbugs.gnu.org; 26 Apr 2020 03:54:41 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Apr 25 23:54:41 2020
Received: from localhost ([127.0.0.1]:60717 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1jSYNN-0002tN-AU
for submit <at> debbugs.gnu.org; Sat, 25 Apr 2020 23:54:41 -0400
Received: from mail-qk1-f173.google.com ([209.85.222.173]:37747)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <zimon.toutoune@HIDDEN>) id 1jSYNK-0002t9-Uk
for 39258 <at> debbugs.gnu.org; Sat, 25 Apr 2020 23:54:39 -0400
Received: by mail-qk1-f173.google.com with SMTP id s63so14651806qke.4
for <39258 <at> debbugs.gnu.org>; Sat, 25 Apr 2020 20:54:38 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
h=mime-version:from:date:message-id:subject:to;
bh=5jMR+PGs8JLCip2nlAvpwgFMDJsoZIpX0NM6gfceTe4=;
b=U4nArzCB84qSZAwUAFGwzogrC1+b6M6ZbzZIQ+9duy+0oULCMIg3Cxa83cSUI4AVqn
inXA/F7ZJTAa3peEzag+GFhDKDztZPOY+un36LW5wenISAp1BfexBj6VMk+uaWbUF6w6
IKdli4MuWQsoTzsPEtawKVoUF6T9Nqf7nhtZych2253HV8Mj4COLxdNZvw6pZTDVs/2n
DqpkK4blVCMo/lf6xVi1tEexYvxDkT+HZlHP7iTbO4z/t44kIKp0zaLXT1cbti4v+Fwx
tr/oSafC1KlV+Evm79x6AdjcYGDnAMI34JaYH7UIUw3FsLjKZQ6o3rVqrzgH1LX3GI4/
jdFQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20161025;
h=x-gm-message-state:mime-version:from:date:message-id:subject:to;
bh=5jMR+PGs8JLCip2nlAvpwgFMDJsoZIpX0NM6gfceTe4=;
b=XztUfyYKHoriWwiPmhq5qgCncVehxUpCv7n/In8TqGva07PsUM/vwIDHfOgJ63VxmF
ZEDcwtw2CLoFhG6mQL7+13DH+7/b6yJg2/Bb/t368GvVMC4qJl0f85Cj8St4KpG2q3Lm
zDhSVOZ92/95aTQR8GowB6aXnZMU/EjOdKSAbmHIaCcQ7fj2R3YcbCK9HNFWImfC5XCH
WEt4RzOjGfLicWKBpMf5NIMcXZLPY0joSMmhvTaBkdmZZQI7rFSNc6UD7X6ExTdaRG6y
5/gXJcWdNH7EuCmL9WxEg2dBblJjN6wOfyB7LV9URozaAcA2KOq6rHatBEDS9IdtElaC
+fRA==
X-Gm-Message-State: AGi0PuaQ9uQ9UspdvdWJOjrIC4EZTfxoztaxQRef7qqMFAxPpkolIf+3
lvhcvnG89yzfbW0dKJ1K5ZIp95Pglp5H1cvbvcb1GgHY
X-Google-Smtp-Source: APiQypL6LNEH/iLgJtABjU0YOKqP13YqizWqdQJzYcumU4CV80RF9dBolOzQL5HT4c4iTdFawfdk99rwB7F4WHBZvPc=
X-Received: by 2002:a37:a6d4:: with SMTP id
p203mr15081998qke.201.1587873272735;
Sat, 25 Apr 2020 20:54:32 -0700 (PDT)
MIME-Version: 1.0
From: zimoun <zimon.toutoune@HIDDEN>
Date: Sun, 26 Apr 2020 05:54:21 +0200
Message-ID: <CAJ3okZ0KP0bYuwShZoNkkFWrhGNPsUGOOfiL5azj_x8Q+GNqLA@HIDDEN>
Subject: benchmark search: default vs v2 vs v3
To: 39258 <at> debbugs.gnu.org, Arun Isaac <arunisaac@HIDDEN>,
=?UTF-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>,
Pierre Neidhardt <mail@HIDDEN>
Content-Type: text/plain; charset="UTF-8"
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 39258
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)
Hi,
Thank you Arun for the patches and all the work. Sorryfor the delay.
TLDR:
1) around 25 seconds added to "guix pull"... but I am more than often
waiting around 10 minutes when pulling.
2) the speedup is clear: more than 2x.
The question is the tradeoff between: the slowdown of pull vs the
speedup of search. What is acceptable?
Here let benchmark 3 versions of Guix:
- default is a357849f5b
- v2 rebased on default and based on Xapian
- v3 rebased on default too and based on "custom" index
and let compare the time of "guix pull" and then "guix search".
Because v2 uses Xapian, the accuracy is different and so the list of
outputs is different depending on the query; the impact on the
performance seems minimal. Let discuss elsewhere about accuracy and
BM25 and let focus on performance for now.
* guix pull
-----------
The idea is: measure if computing the new index is expensive or not,
compared to all of what "guix pull" computes.
** Reference
------------
Maybe, I should have misconfigured something or my laptop is really
not powerful at all, but here some numbers.
(Note: /proc/cpuinfo says 4 times Intel(R) Core(TM) i5-5200U CPU @ 2.20GHz
and /sys/block/sda/queue/rotational says 0 which is SSD.)
--8<---------------cut here---------------start------------->8---
$ guix describe
Generation 8 Apr 25 2020 09:00:01 (current)
guix f84b036
repository URL: https://git.savannah.gnu.org/git/guix.git
branch: master
commit: f84b0363053e5479464f6ce6ded45f80360d90fc
--8<---------------cut here---------------end--------------->8---
--8<---------------cut here---------------start------------->8---
$ time guix pull -C ~/.config/guix/default-channels.scm
Updating channel 'guix' from Git repository at
'https://git.savannah.gnu.org/git/guix.git'...
Building from this channel:
guix https://git.savannah.gnu.org/git/guix.git 8cf6d15
downloading from
https://ci.guix.gnu.org/nar/gzip/xgakzpfs3rz57m666hsk1v3d3zcy7wgn-config.scm
...
config.scm
[...]
building fonts directory...
building directory of Info manuals...
building database for manual pages...
building profile with 1 package...
building /gnu/store/kq1zlj5rxz8wrxc3ha8vck2wv2iakfnb-inferior-script.scm.drv...
building package cache...
building profile with 1 package...
New in this revision:
2 new packages: cl-osicat, sbcl-osicat
real 13m37.997s
user 1m38.129s
sys 0m0.856s
--8<---------------cut here---------------end--------------->8---
And because "guix search" is used say 10 times more than "guix pull",
an increase of 10% of "guix pull" will ease the experience of the user
if "guix search" is faster, IMHO.
Therefore, because "guix pull" takes around 13 minutes, the extra cost
to index all the packages can be roughly 1min30s (at most).
Then, if I pull back from 8cf6d15 to '--commit=a357849f5b' then it takes:
real 2m13.693s
user 1m37.418s
sys 0m0.666s
so in this case 10% means around 7s. But after 1 minute waiting, the
command feels too long to me and personally I am already waiting so I
do not mind much if it would take 2m13s or 3m00s.
Well, it is hard to draw a clear line about what could be accepted as
the time of indexing because the time of pulling is already highly
variable.
What is the average of "guix pull"?
It could be really interresting to probe the users. They could report:
- guix describe
- time guix pull
whatever which channels are up.
Just to have an idea about what should be the acceptable extra time
added by indexing. For sure it depends on the hardware but it would
provide an idea and help to see if the extra time is worth or not.
WDYT?
** Let's compare the index time
-------------------------------
Let pull for the 3 cases and populate the store by all the necessary
items. Could be looooonng! (20minutes) For example, for the version
2 of patches -- living in my branch 'search-v2' using a worktree.
--8<---------------cut here---------------start------------->8---
time ./pre-inst-env guix pull -p /tmp/v2 \
--url=$PWD --branch=search-v2 \
-C ~/.config/guix/default-channels.scm
--8<---------------cut here---------------end--------------->8---
and then let spot the index file for each version:
--8<---------------cut here---------------start------------->8---
# ls -l /tmp/default/lib/guix
/gnu/store/g5c08vqsv31nkn2r0hr32dbrkhf3cvd8-guix-package-cache
readlink /tmp/v2/lib/guix/package-search.index
/gnu/store/8xbzhn81hmshagbgazmnr7xfps1cdsa3-guix-package-search-index/lib/guix/package-search.index
readlink /tmp/v3/lib/guix/package-metadata.cache
/gnu/store/8j78b5c4ddic21gcx7wpbq2akjn7x7mr-guix-package-metadata-cache/lib/guix/package-metadata.cache
--8<---------------cut here---------------end--------------->8---
Well, let remove the profiles and garbage collect the index files:
--8<---------------cut here---------------start------------->8---
rm /tmp/default /tmp/v{2,3}*
guix gc -D \
/gnu/store/g5c08vqsv31nkn2r0hr32dbrkhf3cvd8-guix-package-cache \
/gnu/store/8xbzhn81hmshagbgazmnr7xfps1cdsa3-guix-package-search-index \
/gnu/store/8j78b5c4ddic21gcx7wpbq2akjn7x7mr-guix-package-metadata-cache
--8<---------------cut here---------------end--------------->8---
And then re-run "guix pull". We are now comparing apple to apple, I guess.
| time | default | v2 | v3 |
|------+-----------+-----------+-----------|
| real | 1m11.899s | 1m30.806s | 1m34.341s |
| user | 1m23.845s | 1m24.160s | 1m24.233s |
| sys | 0m0.570s | 0m0.563s | 0m0.529s |
Therefore less than extra 20s and 25s for v2 and v3.
All the question is an extra 25s compared to which time of "guix pull":
- more than 13m: adding 25s is acceptable
- less than 2m: adding 25s is questionable
Usually, my feeling about "guix pull" is... I am waiting! Therefore,
I will not see this extra 25s because it is masked by all the other
work "guix pull" is doing.
* guix search
-------------
Let compare cold (sudo echo 3 > /proc/sys/vm/drop_caches) and warm
cache. For example for the query 'inkscape'.
| time | default | v2 | v3 |
|------+----------+----------+----------|
| real | 0m1.842s | 0m0.331s | 0m0.437s |
| user | 0m1.270s | 0m0.179s | 0m0.336s |
| sys | 0m0.142s | 0m0.047s | 0m0.052s |
|------+----------+----------+----------|
| real | 0m0.898s | 0m0.132s | 0m0.292s |
| user | 0m1.069s | 0m0.168s | 0m0.353s |
| sys | 0m0.072s | 0m0.008s | 0m0.019s |
Therefore the speedup is at least 3.
| cache | default-vs-v2 | default-vs-v3 |
|-------+---------------+---------------|
| cold | 5.6 | 4.2 |
| warm | 6.8 | 3.1 |
Another query:
--8<---------------cut here---------------start------------->8---
time guix search crypto library | recsel -P name | grep libb2
--8<---------------cut here---------------end--------------->8---
| time | default | v2 | v3 |
|------+----------+----------+----------|
| real | 0m2.216s | 0m1.109s | 0m0.689s |
| user | 0m1.655s | 0m1.309s | 0m0.683s |
| sys | 0m0.193s | 0m0.073s | 0m0.035s |
|------+----------+----------+----------|
| real | 0m1.197s | 0m0.490s | 0m0.491s |
| user | 0m1.448s | 0m0.819s | 0m0.625s |
| sys | 0m0.089s | 0m0.034s | 0m0.039s |
| cache | default-vs-v2 | default-vs-v3 |
|-------+---------------+---------------|
| cold | 2.0 | 3.2 |
| warm | 2.4 | 2.4 |
Before going further, especially about any other more sophisticated
inverted index (BM25), it appears to me important to fix what is
"cost" on "guix pull" that the users are ready to pay. Because
somehow the inverted index has to be computed. And without an
inverted index, it seems difficult to improve the accurary.
One solution should be: let compute the inverted index in the
background with a low priority. If the index is not done yet when
"guix search" is called, then fallback to the current default
behaviour.
WDYT?
Cheers,
simon
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 24 Apr 2020 21:05:54 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Fri Apr 24 17:05:54 2020 Received: from localhost ([127.0.0.1]:58090 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1jS5WD-0001Hu-8v for submit <at> debbugs.gnu.org; Fri, 24 Apr 2020 17:05:53 -0400 Received: from eggs.gnu.org ([209.51.188.92]:41022) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <ludo@HIDDEN>) id 1jS5Vv-0001HO-Ux for 39258 <at> debbugs.gnu.org; Fri, 24 Apr 2020 17:05:50 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:40450) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from <ludo@HIDDEN>) id 1jS5Vp-0003gX-1O; Fri, 24 Apr 2020 17:05:29 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=45876 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from <ludo@HIDDEN>) id 1jS5Vo-0000Hs-Kz; Fri, 24 Apr 2020 17:05:28 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN> To: Arun Isaac <arunisaac@HIDDEN> Subject: Re: [bug#39258] [PATCH v3 0/3] Package metadata cache for guix search References: <cu7pnfaar36.fsf@HIDDEN> <20200327162654.18785-1-arunisaac@HIDDEN> Date: Fri, 24 Apr 2020 23:05:26 +0200 In-Reply-To: <20200327162654.18785-1-arunisaac@HIDDEN> (Arun Isaac's message of "Fri, 27 Mar 2020 21:56:51 +0530") Message-ID: <87pnbwfvc9.fsf@HIDDEN> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 39258 Cc: mail@HIDDEN, 39258 <at> debbugs.gnu.org, zimon.toutoune@HIDDEN X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.7 (-) Hi, Arun Isaac <arunisaac@HIDDEN> skribis: > Speedup is around 2x. Both measurements below are with a warm cache. > > $ time guix search inkscape > > real 0m1.722s > user 0m1.776s > sys 0m0.097s > > $ time /tmp/test/bin/guix search inkscape > > real 0m0.749s > user 0m0.770s > sys 0m0.020s > > This patchset does not affect the search API nor does it improve the rele= vance > of search results. If there is interest in this approach, I'll complete t= his > patchset properly. But, in the long run, I do think we should aim to get > xapian or the like for guix search. WDYT? > > Unfortunately, generate-package-metadata-cache takes 43 seconds to build = the > cache on my relatively slow computer. Performance should be better on oth= er > people's machines. 43 seconds is a lot. How do these 43 seconds compare to current =E2=80=98g= uix search=E2=80=99 on your computer? (Both do roughly the same thing.) Thanks, Ludo=E2=80=99.
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 24 Apr 2020 21:03:21 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Fri Apr 24 17:03:21 2020 Received: from localhost ([127.0.0.1]:58086 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1jS5Tk-0001EA-Kv for submit <at> debbugs.gnu.org; Fri, 24 Apr 2020 17:03:21 -0400 Received: from eggs.gnu.org ([209.51.188.92]:40708) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <ludo@HIDDEN>) id 1jS5Tj-0001Dy-Bh for 39258 <at> debbugs.gnu.org; Fri, 24 Apr 2020 17:03:19 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:40313) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from <ludo@HIDDEN>) id 1jS5Tc-0002SD-NZ; Fri, 24 Apr 2020 17:03:12 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=45874 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from <ludo@HIDDEN>) id 1jS5Tb-0007TQ-Bs; Fri, 24 Apr 2020 17:03:12 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN> To: Arun Isaac <arunisaac@HIDDEN> Subject: Re: [bug#39258] [PATCH v3 3/3] guix: Use package metadata cache for package search. References: <20200327162654.18785-1-arunisaac@HIDDEN> <20200327162654.18785-4-arunisaac@HIDDEN> Date: Fri, 24 Apr 2020 23:03:10 +0200 In-Reply-To: <20200327162654.18785-4-arunisaac@HIDDEN> (Arun Isaac's message of "Fri, 27 Mar 2020 21:56:54 +0530") Message-ID: <87v9lofvg1.fsf@HIDDEN> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 39258 Cc: mail@HIDDEN, 39258 <at> debbugs.gnu.org, zimon.toutoune@HIDDEN X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.7 (-) Hello, Arun Isaac <arunisaac@HIDDEN> skribis: > * guix/scripts/package.scm (process-query): Call search-packages and > display-package-search-results instead of find-packages-by-description and > display-search-results respectively. > * guix/ui.scm (package-metadata->recutils): New function. > (%package-metrics): Use package-metadata record field accessors. > (package-relevance): Rename argument package to package-metadata. > (display-package-search-results): New function. [...] > +(define* (package-metadata->recutils p port #:optional (width (%text-wid= th)) > + #:key > + (hyperlinks? (supports-hyperlinks? = port)) > + (extra-fields '())) > + "Write to PORT a `recutils' record of <package-metadata> object P, arr= anging > +to fit within WIDTH columns. EXTRA-FIELDS is a list of symbol/value pai= rs to > +emit. When HYPERLINKS? is true, emit hyperlink escape sequences when > +appropriate." I think we should avoid copy/paste of =E2=80=98package->recutils=E2=80=99. How about factorizing by having a common procedure that takes the fields as keyword arguments instead of taking a record? > (define %package-metrics > ;; Metrics used to compute the "relevance score" of a package against = a set > ;; of regexps. > - `((,package-name . 4) > + `((,package-metadata-name . 4) Here we would also need to arrange so that this can apply to both a <package> and <package-metadata> (or whatever), perhaps by defining the two sets of metrics at once, or defining the second one by mapping over the first one. Thanks, Ludo=E2=80=99.
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 24 Apr 2020 20:58:43 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Fri Apr 24 16:58:43 2020 Received: from localhost ([127.0.0.1]:58082 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1jS5PG-000165-Vx for submit <at> debbugs.gnu.org; Fri, 24 Apr 2020 16:58:43 -0400 Received: from eggs.gnu.org ([209.51.188.92]:40070) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <ludo@HIDDEN>) id 1jS5PE-00015p-Ky for 39258 <at> debbugs.gnu.org; Fri, 24 Apr 2020 16:58:41 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:40042) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from <ludo@HIDDEN>) id 1jS5P7-0001Iz-Ew; Fri, 24 Apr 2020 16:58:33 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=45872 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from <ludo@HIDDEN>) id 1jS5P6-0001LA-0B; Fri, 24 Apr 2020 16:58:32 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN> To: Arun Isaac <arunisaac@HIDDEN> Subject: Re: [bug#39258] [PATCH v3 2/3] guix: Search package metadata cache. References: <20200327162654.18785-1-arunisaac@HIDDEN> <20200327162654.18785-3-arunisaac@HIDDEN> Date: Fri, 24 Apr 2020 22:58:30 +0200 In-Reply-To: <20200327162654.18785-3-arunisaac@HIDDEN> (Arun Isaac's message of "Fri, 27 Mar 2020 21:56:53 +0530") Message-ID: <874kt8ha89.fsf@HIDDEN> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 39258 Cc: mail@HIDDEN, 39258 <at> debbugs.gnu.org, zimon.toutoune@HIDDEN X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.7 (-) Arun Isaac <arunisaac@HIDDEN> skribis: > * gnu/packages.scm (search-packages): New function. > * guix/packages.scm (<package-metadata>): New record type. [...] > +(define (search-packages profile regexps) > + "Return a list of pairs: <package-metadata> objects corresponding to > +packages whose name, synopsis, description, or output matches at least o= ne of > +REGEXPS sorted by relevance, and its non-zero relevance score." > + (define cache-file > + (string-append profile %package-metadata-cache-file)) Here we=E2=80=99re missing something that checks if the cache is authoritat= ive and falls back to the old method if it=E2=80=99s not, akin to what =E2=80=98fold-available-packages=E2=80=99 does. > + (define cache > + (catch 'system-error > + (lambda () > + (map (match-lambda > + (#(name version dependencies outputs systems > + synopsis description home-page (file line column)) > + (make-package-metadata > + name version dependencies outputs systems > + synopsis description home-page > + (location file line column)))) > + (load-compiled cache-file))) I realize the other cache also has that problem, but it would be nice to add a version tag to the cache. Basically emit something like: (package-metadata-cache (version 0) VECTOR =E2=80=A6) instead of just: (VECTOR =E2=80=A6) > +(define-record-type* <package-metadata> > + package-metadata make-package-metadata > + package-metadata? > + this-package-metadata > + (name package-metadata-name) > + (version package-metadata-version) > + (dependencies package-metadata-dependencies) > + (outputs package-metadata-outputs) > + (supported-systems package-metadata-supported-systems) > + (synopsis package-metadata-synopsis) > + (description package-metadata-description) > + ;; TODO: Add license > + ;; (license package-metadata-license) > + (home-page package-metadata-home-page) > + (location package-metadata-location)) I=E2=80=99m not comfortable with this data structure duplication, especially right in (guix packages, but I=E2=80=99m not sure it=E2=80=99s avoidable. =E2=80=98fold-available-packages=E2=80=99 avoids it by passing all the fiel= ds as arguments to the fold procedure, I=E2=80=99m not sure if it=E2=80=99s appli= cable here. Thanks, Ludo=E2=80=99.
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 24 Apr 2020 20:48:48 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Fri Apr 24 16:48:48 2020 Received: from localhost ([127.0.0.1]:58076 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1jS5Fg-0000s2-0L for submit <at> debbugs.gnu.org; Fri, 24 Apr 2020 16:48:48 -0400 Received: from eggs.gnu.org ([209.51.188.92]:38604) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <ludo@HIDDEN>) id 1jS5Fe-0000ro-FZ for 39258 <at> debbugs.gnu.org; Fri, 24 Apr 2020 16:48:47 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:39424) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from <ludo@HIDDEN>) id 1jS5FX-000744-Eg; Fri, 24 Apr 2020 16:48:39 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=45870 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from <ludo@HIDDEN>) id 1jS5FX-0005P6-2u; Fri, 24 Apr 2020 16:48:39 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN> To: Arun Isaac <arunisaac@HIDDEN> Subject: Re: [bug#39258] [PATCH v3 1/3] guix: Generate package metadata cache. References: <20200327162654.18785-1-arunisaac@HIDDEN> <20200327162654.18785-2-arunisaac@HIDDEN> Date: Fri, 24 Apr 2020 22:48:36 +0200 In-Reply-To: <20200327162654.18785-2-arunisaac@HIDDEN> (Arun Isaac's message of "Fri, 27 Mar 2020 21:56:52 +0530") Message-ID: <87h7x8haor.fsf@HIDDEN> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 39258 Cc: mail@HIDDEN, 39258 <at> debbugs.gnu.org, zimon.toutoune@HIDDEN X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.7 (-) Hi Arun, Arun Isaac <arunisaac@HIDDEN> skribis: > * gnu/packages.scm (%package-metadata-cache-file): New variable. > (generate-package-metadata-cache): New function. > * guix/channels.scm (package-metadata-cache-file): New function. > (%channel-profile-hooks): Add package-metadata-cache-file. This is short and sweet, nice! > + (define (expand-cache package result) > + (cons `#(,(package-name package) > + ,(package-version package) > + ,(delete-duplicates > + (map package-full-name > + (sort (filter package? (package-direct-inputs packag= e)) > + package<?))) > + ,(package-outputs package) > + ,(package-supported-systems package) > + ,(package-synopsis package) > + ,(package-description package) > + ,(package-home-page package) > + ,(let ((location (package-location package))) > + (list (location-file location) > + (location-line location) > + (location-column location)))) I was wondering if we could omit inputs, which are not that useful. Apart from that it LGTM. Note that this is probably the place where we could eventually add the computation of an inverted index like zimoun suggested in <https://lists.gnu.org/archive/html/guix-devel/2020-01/msg00243.html>. > + #:properties '((type . profile-hook) > + (hook . package-cache)) =E2=80=98package-metadata-cache=E2=80=99, even (it=E2=80=99s for UI purpose= s). Nitpick: I=E2=80=99d use =E2=80=9Cpackages:=E2=80=9D as the prefix in the s= ubject line. Thanks, Ludo=E2=80=99.
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Ludovic Courtès <ludo@HIDDEN>
to control <at> debbugs.gnu.org.
Full text available.Received: (at 39258) by debbugs.gnu.org; 5 Apr 2020 14:08:39 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Sun Apr 05 10:08:38 2020 Received: from localhost ([127.0.0.1]:46440 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1jL5x0-0006dc-MP for submit <at> debbugs.gnu.org; Sun, 05 Apr 2020 10:08:38 -0400 Received: from eggs.gnu.org ([209.51.188.92]:44037) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <ludo@HIDDEN>) id 1jL5wz-0006dA-0V for 39258 <at> debbugs.gnu.org; Sun, 05 Apr 2020 10:08:37 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:37298) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <ludo@HIDDEN>) id 1jL5ws-0006mN-JT; Sun, 05 Apr 2020 10:08:30 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=39454 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from <ludo@HIDDEN>) id 1jL5ws-0007xJ-58; Sun, 05 Apr 2020 10:08:30 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN> To: Arun Isaac <arunisaac@HIDDEN> Subject: Re: [PATCH v3 0/3] Package metadata cache for guix search References: <20200327162654.18785-1-arunisaac@HIDDEN> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 17 Germinal an 228 de la =?utf-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Sun, 05 Apr 2020 16:08:28 +0200 In-Reply-To: <20200327162654.18785-1-arunisaac@HIDDEN> (Arun Isaac's message of "Fri, 27 Mar 2020 21:56:51 +0530") Message-ID: <87blo6hvoj.fsf@HIDDEN> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 39258 Cc: mail@HIDDEN, 39258 <at> debbugs.gnu.org, zimon.toutoune@HIDDEN X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.7 (-) Hi Arun, Arun Isaac <arunisaac@HIDDEN> skribis: > This is v3 of my attempt to make guix search faster. In this version, I h= ave > abandoned use of xapian. Instead I build a cache of the metadata of all > packages in a profile hook. Then, I use that cache to search and display > search results. This way, package guile modules are not loaded during guix > search. > > Speedup is around 2x. Both measurements below are with a warm cache. Sorry for the delay! Just to say that I like the approach, and I=E2=80=99ll take a closer look once the release is out=E2=80=A6 Thank you! Ludo=E2=80=99.
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.
Received: (at 39258) by debbugs.gnu.org; 27 Mar 2020 16:27:26 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Mar 27 12:27:26 2020
Received: from localhost ([127.0.0.1]:33288 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1jHrpN-0000Vd-O3
for submit <at> debbugs.gnu.org; Fri, 27 Mar 2020 12:27:26 -0400
Received: from mugam.systemreboot.net ([139.59.75.54]:36920)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <arunisaac@HIDDEN>) id 1jHrpH-0000V4-9p
for 39258 <at> debbugs.gnu.org; Fri, 27 Mar 2020 12:27:20 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed;
d=systemreboot.net; s=default; h=Content-Transfer-Encoding:Content-Type:
MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender
:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From:
Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:
List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive;
bh=+xeMBbi0lnhx5F/Pbxsj5gXUpnAYL2wwHS+Oi9X11KI=; b=Mn/IRFNQScuBTd594sqeuVyXxL
Nd4HWtEpXJppFEHuWW6qEiVZ4ILFNmh2UB/9/jqHp4irzu2wYt1pdMEWr/ZJcBYqu+N/bimzcWang
7ZJGrxvjZ/ExSuDiucJjkljp3ywl/GK+vBArd2+Vgp0yxrJZKzQwt05UDEj4bLrDs+qY=;
Received: from [192.168.2.1] (helo=steel.lan)
by systemreboot.net with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384
(Exim 4.93) (envelope-from <arunisaac@HIDDEN>)
id 1jHrp6-0044U6-9U; Fri, 27 Mar 2020 21:57:08 +0530
From: Arun Isaac <arunisaac@HIDDEN>
To: 39258 <at> debbugs.gnu.org
Subject: [PATCH v3 3/3] guix: Use package metadata cache for package search.
Date: Fri, 27 Mar 2020 21:56:54 +0530
Message-Id: <20200327162654.18785-4-arunisaac@HIDDEN>
X-Mailer: git-send-email 2.25.1
In-Reply-To: <20200327162654.18785-1-arunisaac@HIDDEN>
References: <20200327162654.18785-1-arunisaac@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 39258
Cc: Arun Isaac <arunisaac@HIDDEN>, mail@HIDDEN, ludo@HIDDEN,
zimon.toutoune@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)
* guix/scripts/package.scm (process-query): Call search-packages and
display-package-search-results instead of find-packages-by-description and
display-search-results respectively.
* guix/ui.scm (package-metadata->recutils): New function.
(%package-metrics): Use package-metadata record field accessors.
(package-relevance): Rename argument package to package-metadata.
(display-package-search-results): New function.
---
guix/scripts/package.scm | 5 +-
guix/ui.scm | 132 ++++++++++++++++++++++++++++++++++++---
2 files changed, 125 insertions(+), 12 deletions(-)
diff --git a/guix/scripts/package.scm b/guix/scripts/package.scm
index 110d4f2977..c11f92f5a2 100644
--- a/guix/scripts/package.scm
+++ b/guix/scripts/package.scm
@@ -7,6 +7,7 @@
;;; Copyright © 2016 Benz Schenk <benz.schenk@HIDDEN>
;;; Copyright © 2016 Chris Marusich <cmmarusich@HIDDEN>
;;; Copyright © 2019 Tobias Geerinckx-Rice <me@HIDDEN>
+;;; Copyright © 2020 Arun Isaac <arunisaac@HIDDEN>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -770,9 +771,9 @@ processed, #f otherwise."
(_ #f))
opts))
(regexps (map (cut make-regexp* <> regexp/icase) patterns))
- (matches (find-packages-by-description regexps)))
+ (matches (search-packages (current-profile) regexps)))
(leave-on-EPIPE
- (display-search-results matches (current-output-port)))
+ (display-package-search-results matches (current-output-port)))
#t))
(('show requested-name)
diff --git a/guix/ui.scm b/guix/ui.scm
index 1e24fe5dca..934699f065 100644
--- a/guix/ui.scm
+++ b/guix/ui.scm
@@ -14,6 +14,7 @@
;;; Copyright © 2019 Chris Marusich <cmmarusich@HIDDEN>
;;; Copyright © 2019 Tobias Geerinckx-Rice <me@HIDDEN>
;;; Copyright © 2019 Simon Tournier <zimon.toutoune@HIDDEN>
+;;; Copyright © 2020 Arun Isaac <arunisaac@HIDDEN>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -112,6 +113,7 @@
package-synopsis-string
string->recutils
package->recutils
+ package-metadata->recutils
package-specification->name+version+output
supports-hyperlinks?
@@ -122,6 +124,7 @@
relevance
package-relevance
display-search-results
+ display-package-search-results
with-profile-lock
string->generations
@@ -1484,6 +1487,75 @@ HYPERLINKS? is true, emit hyperlink escape sequences when appropriate."
extra-fields)
(newline port))
+(define* (package-metadata->recutils p port #:optional (width (%text-width))
+ #:key
+ (hyperlinks? (supports-hyperlinks? port))
+ (extra-fields '()))
+ "Write to PORT a `recutils' record of <package-metadata> object P, arranging
+to fit within WIDTH columns. EXTRA-FIELDS is a list of symbol/value pairs to
+emit. When HYPERLINKS? is true, emit hyperlink escape sequences when
+appropriate."
+ (define width*
+ ;; The available number of columns once we've taken into account space for
+ ;; the initial "+ " prefix.
+ (if (> width 2) (- width 2) width))
+
+ ;; Note: Don't i18n field names so that people can post-process it.
+ (format port "name: ~a~%" (package-metadata-name p))
+ (format port "version: ~a~%" (package-metadata-version p))
+ (format port "outputs: ~a~%" (string-join (package-metadata-outputs p)))
+ (format port "systems: ~a~%"
+ (string-join (package-metadata-supported-systems p)))
+ (format port "dependencies: ~a~%"
+ (string-join (package-metadata-dependencies p) " "))
+ (format port "location: ~a~%"
+ (or (and=> (package-metadata-location p)
+ (if hyperlinks? location->hyperlink location->string))
+ (G_ "unknown")))
+
+ ;; Note: Starting from version 1.6 or recutils, hyphens are not allowed in
+ ;; field identifiers.
+ (format port "homepage: ~a~%" (package-metadata-home-page p))
+
+ ;; TODO: Print license
+ ;; (format port "license: ~a~%"
+ ;; (match (package-metadata-license p)
+ ;; (((? license? licenses) ...)
+ ;; (string-join (map license-name licenses)
+ ;; ", "))
+ ;; ((? license? license)
+ ;; (let ((text (license-name license))
+ ;; (uri (license-uri license)))
+ ;; (if (and hyperlinks? uri (string-prefix? "http" uri))
+ ;; (hyperlink uri text)
+ ;; text)))
+ ;; (x
+ ;; (G_ "unknown"))))
+ (format port "synopsis: ~a~%"
+ (string-map (match-lambda
+ (#\newline #\space)
+ (chr chr))
+ (or (and=> (package-metadata-synopsis p) P_)
+ "")))
+ (format port "~a~%"
+ (string->recutils
+ (string-trim-right
+ (parameterize ((%text-width width*))
+ (texi->plain-text
+ (string-append "description: "
+ (or (and=> (package-metadata-description p) P_)
+ ""))))
+ #\newline)))
+ (for-each (match-lambda
+ ((field . value)
+ (let ((field (symbol->string field)))
+ (format port "~a: ~a~%"
+ field
+ (fill-paragraph (object->string value) width*
+ (string-length field))))))
+ extra-fields)
+ (newline port))
+
;;;
;;; Searching.
@@ -1528,34 +1600,74 @@ score, the more relevant OBJ is to REGEXPS."
(define %package-metrics
;; Metrics used to compute the "relevance score" of a package against a set
;; of regexps.
- `((,package-name . 4)
+ `((,package-metadata-name . 4)
;; Match against uncommon outputs.
- (,(lambda (package)
+ (,(lambda (package-metadata)
(filter (lambda (output)
(not (member output
;; Some common outpus shared by many packages.
'("out" "doc" "debug" "lib" "include" "bin"))))
- (package-outputs package)))
+ (package-metadata-outputs package-metadata)))
. 1)
;; Match regexps on the raw Texinfo since formatting it is quite expensive
;; and doesn't have much of an effect on search results.
- (,(lambda (package)
- (and=> (package-synopsis package) P_)) . 3)
- (,(lambda (package)
- (and=> (package-description package) P_)) . 2)
+ (,(lambda (package-metadata)
+ (and=> (package-metadata-synopsis package-metadata) P_)) . 3)
+ (,(lambda (package-metadata)
+ (and=> (package-metadata-description package-metadata) P_)) . 2)
(,(lambda (type)
- (match (and=> (package-location type) location-file)
+ (match (and=> (package-metadata-location type) location-file)
((? string? file) (basename file ".scm"))
(#f "")))
. 1)))
-(define (package-relevance package regexps)
+(define (package-relevance package-metadata regexps)
"Return a score denoting the relevance of PACKAGE for REGEXPS. A score of
zero means that PACKAGE does not match any of REGEXPS."
- (relevance package regexps %package-metrics))
+ (relevance package-metadata regexps %package-metrics))
+
+(define* (display-package-search-results matches port
+ #:key
+ (command "guix search"))
+ "Display MATCHES, a list of <package-metadata>/score pairs. If PORT is a
+terminal, print at most a full screen of results."
+ (define first-line
+ (port-line port))
+
+ (define max-rows
+ (and first-line (isatty? port)
+ (terminal-rows port)))
+
+ (define (line-count str)
+ (string-count str #\newline))
+
+ (let loop ((matches matches))
+ (match matches
+ (((package-metadata . score) rest ...)
+ (let* ((links? (supports-hyperlinks? port))
+ (text (call-with-output-string
+ (lambda (port)
+ (package-metadata->recutils package-metadata port
+ #:hyperlinks? links?
+ #:extra-fields
+ `((relevance . ,score)))))))
+ (if (and (not (getenv "INSIDE_EMACS"))
+ max-rows
+ (> (port-line port) first-line) ;print at least one result
+ (> (+ 4 (line-count text) (port-line port))
+ max-rows))
+ (unless (null? rest)
+ (display-hint (format #f (G_ "Run @code{~a ... | less} \
+to view all the results.")
+ command)))
+ (begin
+ (display text port)
+ (loop rest)))))
+ (()
+ #t))))
(define* (display-search-results matches port
#:key
--
2.25.1
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.
Received: (at 39258) by debbugs.gnu.org; 27 Mar 2020 16:27:25 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Mar 27 12:27:25 2020
Received: from localhost ([127.0.0.1]:33286 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1jHrpN-0000Va-C0
for submit <at> debbugs.gnu.org; Fri, 27 Mar 2020 12:27:25 -0400
Received: from mugam.systemreboot.net ([139.59.75.54]:36906)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <arunisaac@HIDDEN>) id 1jHrpE-0000Uo-Nv
for 39258 <at> debbugs.gnu.org; Fri, 27 Mar 2020 12:27:19 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed;
d=systemreboot.net; s=default; h=Content-Transfer-Encoding:Content-Type:
MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender
:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From:
Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:
List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive;
bh=8r87CbgSCGqiElc44RBwtu0UE2I72qor0+QbDZom44U=; b=HLobDPTt9fY+UkyrARSke3D0dL
dfxG+XkbwTtAu9ugqttjdgWVcHDzBXGWNsjoKCeALbvfnUOif/Sa/fmKCgthAAo217qLNSarIABmK
7g2Kpi8jFnEG/uG8cgzs96FmNoW4Sp8oRLKEcUIzvnDJqaa74TZjTbBRj2Asq6u95GtI=;
Received: from [192.168.2.1] (helo=steel.lan)
by systemreboot.net with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384
(Exim 4.93) (envelope-from <arunisaac@HIDDEN>)
id 1jHrp4-0044U6-LV; Fri, 27 Mar 2020 21:57:06 +0530
From: Arun Isaac <arunisaac@HIDDEN>
To: 39258 <at> debbugs.gnu.org
Subject: [PATCH v3 1/3] guix: Generate package metadata cache.
Date: Fri, 27 Mar 2020 21:56:52 +0530
Message-Id: <20200327162654.18785-2-arunisaac@HIDDEN>
X-Mailer: git-send-email 2.25.1
In-Reply-To: <20200327162654.18785-1-arunisaac@HIDDEN>
References: <20200327162654.18785-1-arunisaac@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 39258
Cc: Arun Isaac <arunisaac@HIDDEN>, mail@HIDDEN, ludo@HIDDEN,
zimon.toutoune@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)
* gnu/packages.scm (%package-metadata-cache-file): New variable.
(generate-package-metadata-cache): New function.
* guix/channels.scm (package-metadata-cache-file): New function.
(%channel-profile-hooks): Add package-metadata-cache-file.
---
gnu/packages.scm | 50 ++++++++++++++++++++++++++++++++++++++++++++++-
guix/channels.scm | 34 +++++++++++++++++++++++++++++++-
2 files changed, 82 insertions(+), 2 deletions(-)
diff --git a/gnu/packages.scm b/gnu/packages.scm
index d22c992bb1..c0b527acf0 100644
--- a/gnu/packages.scm
+++ b/gnu/packages.scm
@@ -4,6 +4,7 @@
;;; Copyright © 2014 Eric Bavier <bavier@HIDDEN>
;;; Copyright © 2016, 2017 Alex Kost <alezost@HIDDEN>
;;; Copyright © 2016 Mathieu Lirzin <mthl@HIDDEN>
+;;; Copyright © 2020 Arun Isaac <arunisaac@HIDDEN>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -64,7 +65,8 @@
specification->location
specifications->manifest
- generate-package-cache))
+ generate-package-cache
+ generate-package-metadata-cache))
;;; Commentary:
;;;
@@ -426,6 +428,52 @@ reducing the memory footprint."
#:opts '(#:to-file? #t)))))
cache-file)
+(define %package-metadata-cache-file
+ ;; Location of the package metadata cache.
+ "/lib/guix/package-metadata.cache")
+
+(define (generate-package-metadata-cache directory)
+ "Generate under DIRECTORY a cache of the metadata of all available packages.
+
+The primary purpose of this cache is to speed up package metadata lookup
+during package search so that we don't have to traverse and load all the
+package modules."
+ (define cache-file
+ (string-append directory %package-metadata-cache-file))
+
+ (define (package<? p1 p2)
+ (string<? (package-full-name p1) (package-full-name p2)))
+
+ (define (expand-cache package result)
+ (cons `#(,(package-name package)
+ ,(package-version package)
+ ,(delete-duplicates
+ (map package-full-name
+ (sort (filter package? (package-direct-inputs package))
+ package<?)))
+ ,(package-outputs package)
+ ,(package-supported-systems package)
+ ,(package-synopsis package)
+ ,(package-description package)
+ ,(package-home-page package)
+ ,(let ((location (package-location package)))
+ (list (location-file location)
+ (location-line location)
+ (location-column location))))
+ result))
+
+ (define exp
+ (fold-packages expand-cache '()))
+
+ (mkdir-p (dirname cache-file))
+ (call-with-output-file cache-file
+ (lambda (port)
+ (put-bytevector port
+ (compile `'(,@exp)
+ #:to 'bytecode
+ #:opts '(#:to-file? #t)))))
+ cache-file)
+
(define %sigint-prompt
;; The prompt to jump to upon SIGINT.
diff --git a/guix/channels.scm b/guix/channels.scm
index f0261dc2da..c4efaa7300 100644
--- a/guix/channels.scm
+++ b/guix/channels.scm
@@ -2,6 +2,7 @@
;;; Copyright © 2018, 2019, 2020 Ludovic Courtès <ludo@HIDDEN>
;;; Copyright © 2018 Ricardo Wurmus <rekado@HIDDEN>
;;; Copyright © 2019 Jan (janneke) Nieuwenhuizen <janneke@HIDDEN>
+;;; Copyright © 2020 Arun Isaac <arunisaac@HIDDEN>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -581,9 +582,40 @@ be used as a profile hook."
(hook . package-cache))
#:local-build? #t)))
+(define (package-metadata-cache-file manifest)
+ "Build a package metadata cache file for the instance in MANIFEST. This is
+meant to be used as a profile hook."
+ (mlet %store-monad ((profile (profile-derivation manifest
+ #:hooks '())))
+
+ (define build
+ #~(begin
+ (use-modules (gnu packages))
+
+ (if (defined? 'generate-package-metadata-cache)
+ (begin
+ ;; Delegate package cache generation to the inferior.
+ (format (current-error-port)
+ "Generating package metadata cache for '~a'...~%"
+ #$profile)
+ (generate-package-metadata-cache #$output))
+ (mkdir #$output))))
+
+ (gexp->derivation-in-inferior "guix-package-metadata-cache" build
+ profile
+
+ ;; If the Guix in PROFILE is too old and
+ ;; lacks 'guix repl', don't build the cache
+ ;; instead of failing.
+ #:silent-failure? #t
+
+ #:properties '((type . profile-hook)
+ (hook . package-cache))
+ #:local-build? #t)))
+
(define %channel-profile-hooks
;; The default channel profile hooks.
- (cons package-cache-file %default-profile-hooks))
+ (cons* package-cache-file package-metadata-cache-file %default-profile-hooks))
(define (channel-instances->derivation instances)
"Return the derivation of the profile containing INSTANCES, a list of
--
2.25.1
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.
Received: (at 39258) by debbugs.gnu.org; 27 Mar 2020 16:27:20 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Mar 27 12:27:20 2020
Received: from localhost ([127.0.0.1]:33284 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1jHrpH-0000V8-T2
for submit <at> debbugs.gnu.org; Fri, 27 Mar 2020 12:27:20 -0400
Received: from mugam.systemreboot.net ([139.59.75.54]:36900)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <arunisaac@HIDDEN>) id 1jHrpD-0000Uk-4S
for 39258 <at> debbugs.gnu.org; Fri, 27 Mar 2020 12:27:17 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed;
d=systemreboot.net; s=default; h=Content-Transfer-Encoding:Content-Type:
MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender
:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From:
Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:
List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive;
bh=8009glw+epzSdw5fvr0/RvcHVylaG/W/J8vu/6Gx8iQ=; b=TArhqYsNVyfjNE+iMhCyJ6+Ldf
cHHBOZQNPRkdwq4GhNmuazzVUSQzgFlioFNcUpS5JicSer6p0hoADGBPVi893FFqKMwtHGlHKbYeu
UBoPMkK0jy0ydl4sxFv57ekWh9ObiunWgWPXm8E4HJB6UDrvWSux0WCb0OlzBBXyi/dI=;
Received: from [192.168.2.1] (helo=steel.lan)
by systemreboot.net with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384
(Exim 4.93) (envelope-from <arunisaac@HIDDEN>)
id 1jHrp5-0044U6-08; Fri, 27 Mar 2020 21:57:07 +0530
From: Arun Isaac <arunisaac@HIDDEN>
To: 39258 <at> debbugs.gnu.org
Subject: [PATCH v3 2/3] guix: Search package metadata cache.
Date: Fri, 27 Mar 2020 21:56:53 +0530
Message-Id: <20200327162654.18785-3-arunisaac@HIDDEN>
X-Mailer: git-send-email 2.25.1
In-Reply-To: <20200327162654.18785-1-arunisaac@HIDDEN>
References: <20200327162654.18785-1-arunisaac@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 39258
Cc: Arun Isaac <arunisaac@HIDDEN>, mail@HIDDEN, ludo@HIDDEN,
zimon.toutoune@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)
* gnu/packages.scm (search-packages): New function.
* guix/packages.scm (<package-metadata>): New record type.
---
gnu/packages.scm | 38 ++++++++++++++++++++++++++++++++++++++
guix/packages.scm | 32 ++++++++++++++++++++++++++++++++
2 files changed, 70 insertions(+)
diff --git a/gnu/packages.scm b/gnu/packages.scm
index c0b527acf0..2510b1fe49 100644
--- a/gnu/packages.scm
+++ b/gnu/packages.scm
@@ -59,6 +59,7 @@
find-packages-by-name
find-package-locations
find-best-packages-by-name
+ search-packages
specification->package
specification->package+output
@@ -474,6 +475,43 @@ package modules."
#:opts '(#:to-file? #t)))))
cache-file)
+(define (search-packages profile regexps)
+ "Return a list of pairs: <package-metadata> objects corresponding to
+packages whose name, synopsis, description, or output matches at least one of
+REGEXPS sorted by relevance, and its non-zero relevance score."
+ (define cache-file
+ (string-append profile %package-metadata-cache-file))
+
+ (define cache
+ (catch 'system-error
+ (lambda ()
+ (map (match-lambda
+ (#(name version dependencies outputs systems
+ synopsis description home-page (file line column))
+ (make-package-metadata
+ name version dependencies outputs systems
+ synopsis description home-page
+ (location file line column))))
+ (load-compiled cache-file)))
+ (lambda args
+ (if (= ENOENT (system-error-errno args))
+ #f
+ (apply throw args)))))
+
+ (let ((matches
+ (filter-map (lambda (package-metadata)
+ (let ((score (package-relevance package-metadata regexps)))
+ (and (positive? score)
+ (cons package-metadata score))))
+ cache)))
+ (sort matches
+ (lambda (m1 m2)
+ (match m1
+ ((package1 . score1)
+ (match m2
+ ((package2 . score2)
+ (> score1 score2)))))))))
+
(define %sigint-prompt
;; The prompt to jump to upon SIGINT.
diff --git a/guix/packages.scm b/guix/packages.scm
index 70b1478c91..bb06baa1ee 100644
--- a/guix/packages.scm
+++ b/guix/packages.scm
@@ -5,6 +5,7 @@
;;; Copyright © 2016 Alex Kost <alezost@HIDDEN>
;;; Copyright © 2017, 2019 Efraim Flashner <efraim@HIDDEN>
;;; Copyright © 2019 Marius Bakke <mbakke@HIDDEN>
+;;; Copyright © 2020 Arun Isaac <arunisaac@HIDDEN>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -115,6 +116,21 @@
transitive-input-references
+ package-metadata
+ make-package-metadata
+ package-metadata?
+ this-package-metadata
+ package-metadata-name
+ package-metadata-version
+ package-metadata-dependencies
+ package-metadata-outputs
+ package-metadata-synopsis
+ package-metadata-description
+ package-metadata-license
+ package-metadata-home-page
+ package-metadata-supported-systems
+ package-metadata-location
+
%supported-systems
%hurd-systems
%hydra-supported-systems
@@ -310,6 +326,22 @@ name of its URI."
package)
16)))))
+(define-record-type* <package-metadata>
+ package-metadata make-package-metadata
+ package-metadata?
+ this-package-metadata
+ (name package-metadata-name)
+ (version package-metadata-version)
+ (dependencies package-metadata-dependencies)
+ (outputs package-metadata-outputs)
+ (supported-systems package-metadata-supported-systems)
+ (synopsis package-metadata-synopsis)
+ (description package-metadata-description)
+ ;; TODO: Add license
+ ;; (license package-metadata-license)
+ (home-page package-metadata-home-page)
+ (location package-metadata-location))
+
(define (package-upstream-name package)
"Return the upstream name of PACKAGE, which could be different from the name
it has in Guix."
--
2.25.1
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 27 Mar 2020 16:27:20 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Fri Mar 27 12:27:19 2020 Received: from localhost ([127.0.0.1]:33281 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1jHrpH-0000V6-JY for submit <at> debbugs.gnu.org; Fri, 27 Mar 2020 12:27:19 -0400 Received: from mugam.systemreboot.net ([139.59.75.54]:36902) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <arunisaac@HIDDEN>) id 1jHrpD-0000Uj-4D for 39258 <at> debbugs.gnu.org; Fri, 27 Mar 2020 12:27:17 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=systemreboot.net; s=default; h=Content-Transfer-Encoding:MIME-Version: Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To:Content-Type:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:In-Reply-To:References:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=B64pW/Bb79FmNyJmie0BJLwHNVHZT3dbkR8Ek8IwJWw=; b=KcTZRQ0sRRvJ/tKV6hy86zQCuQ SNseCuBp11SoFCSoIj32BkMYBLHauf92+MtrxuBq/BOmKoyGSkImDHB8iXyaNss9JnWsQIxDxCHEQ +1eDJW6ZXKl23pO8bF5J3IjD9hw6NrWyoS+ja3n6YshaxRLR1n1EklLC1vrDwfrUE+60=; Received: from [192.168.2.1] (helo=steel.lan) by systemreboot.net with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.93) (envelope-from <arunisaac@HIDDEN>) id 1jHrp4-0044U6-FN; Fri, 27 Mar 2020 21:57:06 +0530 From: Arun Isaac <arunisaac@HIDDEN> To: 39258 <at> debbugs.gnu.org Subject: [PATCH v3 0/3] Package metadata cache for guix search Date: Fri, 27 Mar 2020 21:56:51 +0530 Message-Id: <20200327162654.18785-1-arunisaac@HIDDEN> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 39258 Cc: Arun Isaac <arunisaac@HIDDEN>, mail@HIDDEN, ludo@HIDDEN, zimon.toutoune@HIDDEN X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) Hi everyone, This is v3 of my attempt to make guix search faster. In this version, I have abandoned use of xapian. Instead I build a cache of the metadata of all packages in a profile hook. Then, I use that cache to search and display search results. This way, package guile modules are not loaded during guix search. Speedup is around 2x. Both measurements below are with a warm cache. --8<---------------cut here---------------start------------->8--- $ time guix search inkscape real 0m1.722s user 0m1.776s sys 0m0.097s --8<---------------cut here---------------end--------------->8--- --8<---------------cut here---------------start------------->8--- $ time /tmp/test/bin/guix search inkscape real 0m0.749s user 0m0.770s sys 0m0.020s --8<---------------cut here---------------end--------------->8--- This patchset does not affect the search API nor does it improve the relevance of search results. If there is interest in this approach, I'll complete this patchset properly. But, in the long run, I do think we should aim to get xapian or the like for guix search. WDYT? Unfortunately, generate-package-metadata-cache takes 43 seconds to build the cache on my relatively slow computer. Performance should be better on other people's machines. Meanwhile, it would still be useful if someone built patchset v2 on their machine and reported the time it took to build the xapian index. * How to test this patchset Apply patches and build as usual. Do a guix pull into a temporary profile. $ ./pre-inst-env guix pull --url=$PWD --branch=the-name-of-the-branch-you-applied-patches-to -p /tmp/test Then, run guix search from the built profile $ /tmp/test/bin/guix search inkscape Thanks! Arun Isaac (3): guix: Generate package metadata cache. guix: Search package metadata cache. guix: Use package metadata cache for package search. gnu/packages.scm | 88 +++++++++++++++++++++++++- guix/channels.scm | 34 +++++++++- guix/packages.scm | 32 ++++++++++ guix/scripts/package.scm | 5 +- guix/ui.scm | 132 ++++++++++++++++++++++++++++++++++++--- 5 files changed, 277 insertions(+), 14 deletions(-) -- 2.25.1
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 15 Mar 2020 20:40:22 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Sun Mar 15 16:40:22 2020 Received: from localhost ([127.0.0.1]:35664 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1jDa3a-0007qw-3R for submit <at> debbugs.gnu.org; Sun, 15 Mar 2020 16:40:22 -0400 Received: from eggs.gnu.org ([209.51.188.92]:47366) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <ludo@HIDDEN>) id 1jDa3X-0007qe-L7 for 39258 <at> debbugs.gnu.org; Sun, 15 Mar 2020 16:40:20 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:59319) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <ludo@HIDDEN>) id 1jDa3R-0004dP-AM; Sun, 15 Mar 2020 16:40:13 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=41578 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from <ludo@HIDDEN>) id 1jDa3Q-0002cH-P4; Sun, 15 Mar 2020 16:40:13 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN> To: Arun Isaac <arunisaac@HIDDEN> Subject: Re: [PATCH v2 0/3] Xapian for Guix package search References: <20200307133116.11443-1-arunisaac@HIDDEN> <87sgijgb1v.fsf@HIDDEN> <cu7tv2zcj9l.fsf@HIDDEN> <875zffcc87.fsf@HIDDEN> <cu7imjed22r.fsf@HIDDEN> <87r1y13jew.fsf@HIDDEN> <cu736agcmzu.fsf@HIDDEN> <87sgifj8zo.fsf@HIDDEN> <cu7o8t0byt6.fsf@HIDDEN> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 26 =?utf-8?Q?Vent=C3=B4se?= an 228 de la =?utf-8?Q?R?= =?utf-8?Q?=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Sun, 15 Mar 2020 21:40:10 +0100 In-Reply-To: <cu7o8t0byt6.fsf@HIDDEN> (Arun Isaac's message of "Fri, 13 Mar 2020 11:07:09 +0530") Message-ID: <87imj5ic7p.fsf@HIDDEN> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 39258 Cc: mail@HIDDEN, 39258 <at> debbugs.gnu.org, zimon.toutoune@HIDDEN X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.7 (-) Hi Arun, Arun Isaac <arunisaac@HIDDEN> skribis: >> Currently, profile hooks such as =E2=80=98%channel-profile-hooks=E2=80= =99, receive a >> complete profile=E2=80=94in this case, the composition of all the channe= ls the >> user chose. >> >> So if we want to achieve what you propose, we=E2=80=99d need to find ano= ther way >> to hook database generation. > > Hmmm. Tough luck, I suppose. Do you have suggestions for anywhere else > to hook database generation? For the core database (packages that come with Guix), (guix self) could take care of it. >> BTW, there=E2=80=99s also the problem of modules added dynamically with >> $GUIX_PACKAGE_PATH or =E2=80=98-L=E2=80=99. With the proposed scheme, i= t seems that >> they could no longer be searched. Is that correct? > > Unfortunately, that is correct. To address this, we discussed retaining > the current search implementation along with the new xapian > implementation. But, that changes the search query behaviour and > adds a lot of complexity. I'll think of some other way out. Yeah, I think we=E2=80=99d want to have roughly a single implementation. I wonder if the relevant metrics that Xapian implements, like zimoun mentioned, could be available directly in Scheme in a way that allows us to compute them at run time when the pre-built cache is unavailable. Or would that be necessarily too slow? If so, perhaps a slightly less fancy metric could work with better performance? Thanks, Ludo=E2=80=99.
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 13 Mar 2020 05:37:44 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Fri Mar 13 01:37:44 2020 Received: from localhost ([127.0.0.1]:58035 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1jCd0y-0005FL-A2 for submit <at> debbugs.gnu.org; Fri, 13 Mar 2020 01:37:44 -0400 Received: from mugam.systemreboot.net ([139.59.75.54]:52230) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <arunisaac@HIDDEN>) id 1jCd0u-0005F8-FI for 39258 <at> debbugs.gnu.org; Fri, 13 Mar 2020 01:37:42 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=systemreboot.net; s=default; h=Content-Type:MIME-Version:Message-ID:Date: References:In-Reply-To:Subject:Cc:To:From:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=SWBLUH+JX/nrfye7oWt+l9ogHfUZ7F9mgDvXboQuAM8=; b=Puzi9jFjd+Oz2gNmOuq4X7hhH EuA868lUbA3bLqWeqE4zBlqx9MQ5cb9rFmXSA6O4y+DvOfElKqWijdLdgm8FtHzVtuMN4ekecPG2/ xyEBdnNZjZv18DgaFcCLJ1eseDYuA7ZafVKcbsXfXMT4f7pKFy4+is6P6OhC528VGxy1Y=; Received: from [192.168.2.1] (helo=steel) by systemreboot.net with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.93) (envelope-from <arunisaac@HIDDEN>) id 1jCd0a-002yur-41; Fri, 13 Mar 2020 11:07:20 +0530 From: Arun Isaac <arunisaac@HIDDEN> To: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN> Subject: Re: [PATCH v2 0/3] Xapian for Guix package search In-Reply-To: <87sgifj8zo.fsf@HIDDEN> References: <20200307133116.11443-1-arunisaac@HIDDEN> <87sgijgb1v.fsf@HIDDEN> <cu7tv2zcj9l.fsf@HIDDEN> <875zffcc87.fsf@HIDDEN> <cu7imjed22r.fsf@HIDDEN> <87r1y13jew.fsf@HIDDEN> <cu736agcmzu.fsf@HIDDEN> <87sgifj8zo.fsf@HIDDEN> Date: Fri, 13 Mar 2020 11:07:09 +0530 Message-ID: <cu7o8t0byt6.fsf@HIDDEN> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 39258 Cc: mail@HIDDEN, 39258 <at> debbugs.gnu.org, zimon.toutoune@HIDDEN X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable > Currently, profile hooks such as =E2=80=98%channel-profile-hooks=E2=80=99= , receive a > complete profile=E2=80=94in this case, the composition of all the channel= s the > user chose. > > So if we want to achieve what you propose, we=E2=80=99d need to find anot= her way > to hook database generation. Hmmm. Tough luck, I suppose. Do you have suggestions for anywhere else to hook database generation? > BTW, there=E2=80=99s also the problem of modules added dynamically with > $GUIX_PACKAGE_PATH or =E2=80=98-L=E2=80=99. With the proposed scheme, it= seems that > they could no longer be searched. Is that correct? Unfortunately, that is correct. To address this, we discussed retaining the current search implementation along with the new xapian implementation. But, that changes the search query behaviour and adds a lot of complexity. I'll think of some other way out. > (Conversely the package cache is optional: it=E2=80=99s only used when it= =E2=80=99s > considered authoritative, see (gnu packages). The API and behavior are > exactly the same whether or not the package cache is used.) Thanks, Arun --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEf3MDQ/Lwnzx3v3nTLiXui2GAK7MFAl5rHAYACgkQLiXui2GA K7OAdgf/cYuLpIA4tY7koTlO4E5nwmfyUxAS8J1lpM6H2xaCCwKjyEWXg8SCxMk2 nSoj57D1jJIZoRQpqXyxu2tVFkyF1S/L1YuBSS4MX1w9dcjsyCKHiNkr5qdV6WT/ 5z1t7rxdzVWBLfHUYJexfjTxepyxApu9jwAoXIpRDSHdDKhSC7s4N/vtxW8TSwae kpH+ferfN5TBRPBkIvVlrm28Q9M7IcJxVktePtMgX3/R0PNhO5p/8l4p/pKkKf2W wuwPjKuRJhYwDjyNBFqX8dBv6TGSeFNX6Lmp34wa009ucshYpvrkaXHKfEAkn9UG /FhrvGeHSotyjWXJ0bSYHg1zkowQfA== =yBNx -----END PGP SIGNATURE----- --=-=-=--
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 11 Mar 2020 13:51:02 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Wed Mar 11 09:51:02 2020 Received: from localhost ([127.0.0.1]:53799 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1jC1lG-0005CI-EP for submit <at> debbugs.gnu.org; Wed, 11 Mar 2020 09:51:02 -0400 Received: from eggs.gnu.org ([209.51.188.92]:48400) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <ludo@HIDDEN>) id 1jC1lF-0005Bp-AH for 39258 <at> debbugs.gnu.org; Wed, 11 Mar 2020 09:51:01 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:60359) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <ludo@HIDDEN>) id 1jC1l8-0008MN-Ek; Wed, 11 Mar 2020 09:50:54 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=37924 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from <ludo@HIDDEN>) id 1jC1l7-0002rc-Fq; Wed, 11 Mar 2020 09:50:53 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN> To: Arun Isaac <arunisaac@HIDDEN> Subject: Re: [PATCH v2 0/3] Xapian for Guix package search References: <20200307133116.11443-1-arunisaac@HIDDEN> <87sgijgb1v.fsf@HIDDEN> <cu7tv2zcj9l.fsf@HIDDEN> <875zffcc87.fsf@HIDDEN> <cu7imjed22r.fsf@HIDDEN> <87r1y13jew.fsf@HIDDEN> <cu736agcmzu.fsf@HIDDEN> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 22 =?utf-8?Q?Vent=C3=B4se?= an 228 de la =?utf-8?Q?R?= =?utf-8?Q?=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Wed, 11 Mar 2020 14:50:51 +0100 In-Reply-To: <cu736agcmzu.fsf@HIDDEN> (Arun Isaac's message of "Tue, 10 Mar 2020 19:47:57 +0530") Message-ID: <87sgifj8zo.fsf@HIDDEN> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 39258 Cc: mail@HIDDEN, 39258 <at> debbugs.gnu.org, zimon.toutoune@HIDDEN X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.7 (-) Hello! Arun Isaac <arunisaac@HIDDEN> skribis: >> We could provide a substitute for users who use only the official 'guix >> channel. However, as soon as users combine multiple channels, they=E2= =80=99ll >> have to build the index locally. > > We could build a separate Xapian database for each channel. Xapian does > support searching across multiple databases at once and will handle > merging the results together appropriately. Nice! > If I understand correctly, this means we can provide substitutes for > at least the official guix channel and let the user build the index > locally for other channels. Is that correct? I=E2=80=99m afraid not, or at least not trivially. Currently, profile hooks such as =E2=80=98%channel-profile-hooks=E2=80=99, = receive a complete profile=E2=80=94in this case, the composition of all the channels = the user chose. So if we want to achieve what you propose, we=E2=80=99d need to find anothe= r way to hook database generation. BTW, there=E2=80=99s also the problem of modules added dynamically with $GUIX_PACKAGE_PATH or =E2=80=98-L=E2=80=99. With the proposed scheme, it s= eems that they could no longer be searched. Is that correct? (Conversely the package cache is optional: it=E2=80=99s only used when it= =E2=80=99s considered authoritative, see (gnu packages). The API and behavior are exactly the same whether or not the package cache is used.) Thanks, Ludo=E2=80=99.
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 10 Mar 2020 14:34:16 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Tue Mar 10 10:34:16 2020 Received: from localhost ([127.0.0.1]:53042 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1jBfxX-0007lZ-Uy for submit <at> debbugs.gnu.org; Tue, 10 Mar 2020 10:34:16 -0400 Received: from mail-qv1-f67.google.com ([209.85.219.67]:45344) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <zimon.toutoune@HIDDEN>) id 1jBfxW-0007lM-DH for 39258 <at> debbugs.gnu.org; Tue, 10 Mar 2020 10:34:14 -0400 Received: by mail-qv1-f67.google.com with SMTP id du17so5576885qvb.12 for <39258 <at> debbugs.gnu.org>; Tue, 10 Mar 2020 07:34:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=QZrj5y6d77vjdLBwljwkJOouuD3jy6Ajojjc13eaFz0=; b=ml5G1w0zwuEOOjwYwI9TouKgMwcCmUSdmSxNzgaDf0PQl6DD9duDvBW5f3m14mTj1A PJ5RHFDthHJrC7ROYoPhNJjQBAYncb0IuOKXcT5DhAy/jviiIw1uU0wNRNpKoCZMtwip RaGHPjFDBCvu95TZp1lQwbvC/gX+t8282BmLz4wuKA3pnmtvdx9Vjswj6VPSm3bVRsFd 49CsZzcjX4SyqcKr9BU+DfXy7oUHkfvCRnJqwonawjfJh+gwj6M3MqJJmq13L8dXlv9n JJmBj558YAUNHyWj0H1LiNbD67ZCWjdF6ubc4IdImH/k4Kwl0ijxHUTkWnek2VLBzNAa P+1g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=QZrj5y6d77vjdLBwljwkJOouuD3jy6Ajojjc13eaFz0=; b=Ocv6TqV5pZ1QAJUkHXwAabtIbKEb25nZ9rVfA45vGsjvjZZDU4Xpzceml7QE5C/PTq /hbGtk4HJPbbmm2KfpaIyvbhdvIJjDJo/d5DeL6ILyE1WE62nlNwJFegfTq5yW+rJ79c 8iBXtbDkwq1bReQJQajQo5Sp+A9/sYpn3kMQPlsw/6aAfy7Uy75P4TJxDaa5Ddx+u28P ZCzglhQWydsLohmZMIheN51cYNkfToBF44+5X1VDJNm2b3wIRYl3FTduNh82QQFV+pUm e5zD3XbKKe5iEDv/RB3vtsvrrW2GQIRgkKucFpWLJY/qKLaNLMKxaxzo9ALffVWtzDhE oD4g== X-Gm-Message-State: ANhLgQ1aNSC1Ss071QiW/5Z6QLTnyyroYW6SCu+Rnoc/ZYWWA7JMr8rI fGAwXiG911FYA+tpCTSJzlvev4FdZls4uMDWnZM= X-Google-Smtp-Source: ADFU+vuWsM2OOlUsL84dErjMAfEDzqI4ZKfDb2dLmstMrdid9f80ux6bxapxzCneZPBPoB1UlgmY4scQh6mBBmI1cuc= X-Received: by 2002:ad4:4862:: with SMTP id u2mr15516523qvy.67.1583850847575; Tue, 10 Mar 2020 07:34:07 -0700 (PDT) MIME-Version: 1.0 References: <20200307133116.11443-1-arunisaac@HIDDEN> <87sgijgb1v.fsf@HIDDEN> <cu7tv2zcj9l.fsf@HIDDEN> <875zffcc87.fsf@HIDDEN> <cu7imjed22r.fsf@HIDDEN> <87r1y13jew.fsf@HIDDEN> <cu736agcmzu.fsf@HIDDEN> In-Reply-To: <cu736agcmzu.fsf@HIDDEN> From: zimoun <zimon.toutoune@HIDDEN> Date: Tue, 10 Mar 2020 15:33:55 +0100 Message-ID: <CAJ3okZ0aK7+oCaOEdXReC0Br=A0LSab1yttp8VGOyn_FkDcr_g@HIDDEN> Subject: Re: [PATCH v2 0/3] Xapian for Guix package search To: Arun Isaac <arunisaac@HIDDEN> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 39258 Cc: =?UTF-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>, Pierre Neidhardt <mail@HIDDEN>, 39258 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) Hi Arun, On Tue, 10 Mar 2020 at 15:18, Arun Isaac <arunisaac@HIDDEN> wrote= : > > We could provide a substitute for users who use only the official 'guix > > channel. However, as soon as users combine multiple channels, they=E2= =80=99ll > > have to build the index locally. > > We could build a separate Xapian database for each channel. Xapian does > support searching across multiple databases at once and will handle > merging the results together appropriately. If I understand correctly, > this means we can provide substitutes for at least the official guix > channel and let the user build the index locally for other channels. Is > that correct? To complement your words, you could also imagine index all the history as any other channels. It needs some thoughts but it seems a path that I would to go. > Also, could someone please build the patchset v2 on their machine and > measure the time taken by generate-package-search-index? My laptop, > particularly my HDD is slow even as far as HDDs go. So, my figure of > 20-50 seconds may not be representative. I will do when I will be fully back. :-) All the best, simon
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 10 Mar 2020 14:18:29 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Tue Mar 10 10:18:29 2020 Received: from localhost ([127.0.0.1]:53010 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1jBfiH-0007Lg-KB for submit <at> debbugs.gnu.org; Tue, 10 Mar 2020 10:18:29 -0400 Received: from mugam.systemreboot.net ([139.59.75.54]:41162) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <arunisaac@HIDDEN>) id 1jBfiE-0007LV-3Q for 39258 <at> debbugs.gnu.org; Tue, 10 Mar 2020 10:18:28 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=systemreboot.net; s=default; h=Content-Type:MIME-Version:Message-ID:Date: References:In-Reply-To:Subject:Cc:To:From:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=WMtM/wjYgy6qtYVmpecIEETMIIutD88+lVbHvIPSMZU=; b=jXPwiAoTzATz5GlCUtc5/dSCG Dzfm3Lzkps7A9wLS48NCS3u6QIUdSazo0Yb0ZozT2NfnbJnb5WfPLyk2lay2+lx05URKrTolrFh7o zXN/sAh+haei+hz55yk/bIReRtqd17ZF6KUmBRWpys4w/JPNpoEHwHq+FK+1+zXZUNwSg=; Received: from [192.168.2.1] (helo=steel) by systemreboot.net with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.93) (envelope-from <arunisaac@HIDDEN>) id 1jBfhy-002oCc-Mh; Tue, 10 Mar 2020 19:48:10 +0530 From: Arun Isaac <arunisaac@HIDDEN> To: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN> Subject: Re: [PATCH v2 0/3] Xapian for Guix package search In-Reply-To: <87r1y13jew.fsf@HIDDEN> References: <20200307133116.11443-1-arunisaac@HIDDEN> <87sgijgb1v.fsf@HIDDEN> <cu7tv2zcj9l.fsf@HIDDEN> <875zffcc87.fsf@HIDDEN> <cu7imjed22r.fsf@HIDDEN> <87r1y13jew.fsf@HIDDEN> Date: Tue, 10 Mar 2020 19:47:57 +0530 Message-ID: <cu736agcmzu.fsf@HIDDEN> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 39258 Cc: mail@HIDDEN, 39258 <at> debbugs.gnu.org, zimon.toutoune@HIDDEN X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable > Note that Texinfo rendering doesn=E2=80=99t use (@ (guix ui) fill-paragra= ph). > It has its own paragraph-filling code. We cannot use =E2=80=98fill-parag= raph=E2=80=99 > after Texinfo rendering anyway, since Texinfo knows where things can be > filled and where they cannot=E2=80=94e.g., @example. True, I did not think of this. > We could provide a substitute for users who use only the official 'guix > channel. However, as soon as users combine multiple channels, they=E2=80= =99ll > have to build the index locally. We could build a separate Xapian database for each channel. Xapian does support searching across multiple databases at once and will handle merging the results together appropriately. If I understand correctly, this means we can provide substitutes for at least the official guix channel and let the user build the index locally for other channels. Is that correct? Also, could someone please build the patchset v2 on their machine and measure the time taken by generate-package-search-index? My laptop, particularly my HDD is slow even as far as HDDs go. So, my figure of 20-50 seconds may not be representative. Thanks, Arun. --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEf3MDQ/Lwnzx3v3nTLiXui2GAK7MFAl5noZUACgkQLiXui2GA K7Nw9gf/b/wKWyQ4b6s7K0/oUeiQOOY54Mg45SpV58vTQs3/n97iLOmhEDsRUoJL QjSpl/BwbpeyjQTeAToBom6UT2QW0rWQ/x4DOlCeFnBlL6LFhGvkNEptHFb2ag7G OpgdO+ljsgO5BkFlU31HvHbe7wLZ7KWSmREXWOw3enGymQD6e/VdDUZz71AwMx/c 95SCGtEY2wKUc0ZceBsgPr7+LyHCQ3h3ga+SX/G9HFyJ5/rU2+7ytzjri6oXaDXJ vWycwBmvutrNZASSuZ/XZaLy7nnK9HWpYABPiiKUTsYei62WtVyiTvzDkFpI0kkY G8Qn7VU6ZJ95cZb4lY60ETYbgdsnnw== =fOJU -----END PGP SIGNATURE----- --=-=-=--
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 10 Mar 2020 05:24:43 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Tue Mar 10 01:24:43 2020 Received: from localhost ([127.0.0.1]:51737 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1jBXNj-0005nu-9k for submit <at> debbugs.gnu.org; Tue, 10 Mar 2020 01:24:43 -0400 Received: from mugam.systemreboot.net ([139.59.75.54]:37980) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <arunisaac@HIDDEN>) id 1jBXNg-0005nj-Fo for 39258 <at> debbugs.gnu.org; Tue, 10 Mar 2020 01:24:42 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=systemreboot.net; s=default; h=Content-Type:MIME-Version:Message-ID:Date: References:In-Reply-To:Subject:Cc:To:From:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=h966rtfLRplkFsWmSOiwzwA0uOJI0DLUqSsYEYmt7js=; b=EJF5x7ZEKFOeMetkbxFRfORS/ LsicLtcbuoCG/iF1fEzb4eQ9GwyI0t2TrXyEOWQWq7G7p4L7KykRmyrIXx6fwudxCb4xJMHAMQMnT XN7y4YRRI2m67Gxpyu31acWFHEB9jf0rohj/XYLQBt+fCerQLbAHLHYUXhct5jT6vwku8=; Received: from [192.168.2.1] (helo=steel) by systemreboot.net with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.93) (envelope-from <arunisaac@HIDDEN>) id 1jBXNc-002mtb-6p; Tue, 10 Mar 2020 10:54:36 +0530 From: Arun Isaac <arunisaac@HIDDEN> To: Jonathan Brielmaier <jonathan.brielmaier@HIDDEN> Subject: Re: [bug#39258] [PATCH v2 1/3] build-self: Add guile-xapian to Guix dependencies. In-Reply-To: <67edd6fa-b17f-7a86-2e43-a8d4e9397e3c@HIDDEN> References: <20200307133116.11443-1-arunisaac@HIDDEN> <20200307133116.11443-2-arunisaac@HIDDEN> <67edd6fa-b17f-7a86-2e43-a8d4e9397e3c@HIDDEN> Date: Tue, 10 Mar 2020 10:54:27 +0530 Message-ID: <cu78sk8dbp0.fsf@HIDDEN> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 39258 Cc: 39258 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain Jonathan Brielmaier <jonathan.brielmaier@HIDDEN> writes: > Could you please trigger a release (e.g. 0.0.1) for guile-xapian? This > would make it more easy for distributions like openSUSE to package it. Done! :-) See https://git.systemreboot.net/guile-xapian/snapshot/guile-xapian-0.1.0.tar.xz But I must warn you that guile-xapian is terribly incomplete and terribly unstable. The API may change at any time without notice. --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEf3MDQ/Lwnzx3v3nTLiXui2GAK7MFAl5nJIsACgkQLiXui2GA K7OwOAf+PiKPV9ZajPUC1nGaA2wvaUOaBlpFNk6vS9HGspnmX2kY7eVaf6mIjW0X MffQwA8lZalTl8psqVzC4991aQvyQVVDIH+5AeyFZZX5MeiOCcYwzu7r6LA8oHOj NCtXX2fEtUPlFn++k0+IoEpivxFHyOk8dSDe4hT8ztyD6hz2ZkBg6Ib5HE0Tl3DG B+73ENuvAJ+GIqdfbh/BwrzwNZQrtvK3pj2gOvt/APsLgz3CvwGY/SVv3YOVZueE S1cpLdPkO2+9KcCgICgSZClTI2SfSe+yVN2HCEu5923FvyiVdc8A0B4gDmfNp+yL uaZm0UiuvrC/2Ch7Ehr1MIBe/leXLQ== =uSoh -----END PGP SIGNATURE----- --=-=-=--
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 9 Mar 2020 23:40:46 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Mon Mar 09 19:40:46 2020 Received: from localhost ([127.0.0.1]:51631 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1jBS0s-0003wR-4W for submit <at> debbugs.gnu.org; Mon, 09 Mar 2020 19:40:46 -0400 Received: from mout.web.de ([212.227.15.4]:60773) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <jonathan.brielmaier@HIDDEN>) id 1jBS0q-0003wE-MA for 39258 <at> debbugs.gnu.org; Mon, 09 Mar 2020 19:40:45 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=web.de; s=dbaedf251592; t=1583797234; bh=5mgmrbByvPOxC8JfxwS1vNQCBjISk03YUbyyP6+nMS0=; h=X-UI-Sender-Class:Subject:To:Cc:References:From:Date:In-Reply-To; b=eQl3UiU3yQ1WCgIYbiRZObHikyFcYtH8k/TiLu40VC7xWZ2gIm2pm/2o9Tzkr0g07 OYGEPdXSm8dHF8t5PR+nTl4cT27EEG3/wmGGVfh3G+v05lbgIMih88LArPTU20nrSk WxLTdLID0wDPIMun0I9aEIWC92iKInDuQQLFoy6s= X-UI-Sender-Class: c548c8c5-30a9-4db5-a2e7-cb6cb037b8f9 Received: from [192.168.178.119] ([5.146.193.216]) by smtp.web.de (mrweb003 [213.165.67.108]) with ESMTPSA (Nemesis) id 0M3T28-1jSqIs3F2W-00qwYn; Tue, 10 Mar 2020 00:40:34 +0100 Subject: Re: [bug#39258] [PATCH v2 1/3] build-self: Add guile-xapian to Guix dependencies. To: Arun Isaac <arunisaac@HIDDEN>, 39258 <at> debbugs.gnu.org References: <20200307133116.11443-1-arunisaac@HIDDEN> <20200307133116.11443-2-arunisaac@HIDDEN> From: Jonathan Brielmaier <jonathan.brielmaier@HIDDEN> Message-ID: <67edd6fa-b17f-7a86-2e43-a8d4e9397e3c@HIDDEN> Date: Tue, 10 Mar 2020 00:40:33 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.5.0 MIME-Version: 1.0 In-Reply-To: <20200307133116.11443-2-arunisaac@HIDDEN> Content-Type: text/plain; charset=utf-8 Content-Language: en-GB Content-Transfer-Encoding: 7bit X-Provags-ID: V03:K1:VTbBCQYb74h8tXOkqhh6VXdqAI090svGK8pWUIAHX+F1MqWqBqR 9r/q8wXs19smV2T2LxiwJXHF+xetM2r75ggFopbt6Mb6Sho6TXu2F6fkdmz+PyUzSZ2IH1D y6Y/sCiHAizHNRSb2By1rzfDSPGoARcrW1GV1MRrsu+N9Kz7/WKmyiJHY4YJFmng9EsCgBy roVLxhzPdOz4xdL3UbppQ== X-Spam-Flag: NO X-UI-Out-Filterresults: notjunk:1;V03:K0:t8eE1sSpeWc=:o1kdwSNYjRCttWVGqMrHQz 3znh15YbJVxjTMIuCAhhhJk2mQPQD+oRiWJ4lAVujTTsyCAvmnf11sI5VzGUBW2Py26mvA8ph Crc9i9yllVv2FcgUwAhV/r4cYa3Yp41+Ie7hRfWHCdsefcvfyAPvx1qhBzC22baPDOVCN75M6 PNFazkkVDLJB/ORGGssIt6h8F9YDW59jRZFooiyE7D52xnaVY4J7Wpn3yXs2u3htAs7fjoEsg iVsNiPAQxH3CtErgLywp0zFNTAgSpXpL+KiLo7JaxvbXeWvsyCL0zcbRaWSg5EgHOMTzonCYZ b8h06qoOBZRIpNNwcn16OkGgeuyYnwPRcOv0wGaw/mOJIG170GLoFz7d7cWujDEFrR/VflGPa vhoBLwzkaXmoNRFuoM6sj6p3AMolmHbtFiP8a2r4sgfKRk4YQVgm2W5c3ZQdAK1ZFmM2V02D9 e+pUZhZO4cAld2dTZbYxVEQg3cuxcqAY2fwU2TNKJ1w0QmW0A4N56ofWOYokcwkC38o/haZ9J xXrp4Ktf0uOf2hbTYkE0cCs+P6gWTCJFpe5k8rbl2c3fjigs8rcccFTPMd/IPrBoRtflIj32J ikU0lR3U040v1yuKLMPnRTN4qChjMDjU/cBl4jnG8Kc5I5Kn+znPbwDSW4pIyNzoSapwEsGyO QL672L5Qmi01qA2LasVpZDQxG3juDrTyoG7TrpM8Z1SXVVH8tJdVqX0tdlbjU/nGbuEFVUWU2 mvOr3KOuLkrQJd1mWDiyWHLMGb2oJIHsWlLhFlDNcc/rklnRuS6z3DQTVIXfsv2XgMNpL9PdG fs/+foJQtAxHqaqq22Wu5YPSiu5TaYYhkN1YHGlvX2OK+PePLOx52BSzLJsaYIAXr1F1mPxk+ nM5Nau/1MBnE56lfNfoxNm+NcpNHvG0s/n8d9Uz3p35V5crNSFdmdDt7KSC2jZ5EpK2a+D4O5 XQcL3S363POH0iFShdAzwv90UJ0cpEO5WUp6G7k0YqbgWdt1/t4HMoR5Li/TcORjMFBPgEcol xbT96BnQmjZaVfKy6+IjfUuWqyhik/7BY7LLUtGg5yCtWuTogN20TvNDYQlwlhh3w5VhFkL3k xMHAQ3LsPAvlPc9MhHGWUUMQx1jY5TzX+P+2PRGUkCMhZyCOnP5HcqWUtKF627ulsHp6HgvVM wqJPwPgfsL1U0thV7edCrvZvS3YiURdyw0f+MQG2BjcEM4qW8YahkziOsMg0Tam5nkIw7DokZ vRMioHkJvHrXy5h4I X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 39258 Cc: ludo@HIDDEN, mail@HIDDEN, zimon.toutoune@HIDDEN X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.7 (-) On 07.03.20 14:31, Arun Isaac wrote: > * build-aux/build-self.scm (build-program): Import fake guile-xapian module. > * guix/self.scm (compiled-guix): Add guile-xapian to Guix dependencies. Could you please trigger a release (e.g. 0.0.1) for guile-xapian? This would make it more easy for distributions like openSUSE to package it.
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 9 Mar 2020 18:19:47 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Mon Mar 09 14:19:47 2020 Received: from localhost ([127.0.0.1]:51447 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1jBN0E-0004Om-Uw for submit <at> debbugs.gnu.org; Mon, 09 Mar 2020 14:19:47 -0400 Received: from mail-qk1-f194.google.com ([209.85.222.194]:32845) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <zimon.toutoune@HIDDEN>) id 1jBN0D-0004OY-J9 for 39258 <at> debbugs.gnu.org; Mon, 09 Mar 2020 14:19:45 -0400 Received: by mail-qk1-f194.google.com with SMTP id p62so10236556qkb.0 for <39258 <at> debbugs.gnu.org>; Mon, 09 Mar 2020 11:19:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=fEHQ+yrhEnUirq0+TLk20ZxNEdxLbSGmklvpWQBJ/Eg=; b=GnYm9Xev5HstMaD2of9pyCGXtR2UKehOl1SVpqcWljCuACJsUmEdAl/4ZYGU9EeUQT LeOlS83xqVik/dVNkbaiEEd36gKo7QuNSzKM7ACO8NVxr3d1d6Gm/0Aou3L+ARSoH26X F1/2qZUhLYqPx3gBt52CwZzO+gxWSbDZNEq97TWJ1zB6KYQ97jL4II1kZBGO+6u0gZGb IEqgSUre+udqunCSYKkRHzWn67zGtMUO4R36w8mFrNP2WNvYrtoeZrwNBBJOFd288+6n QSsHFnBCzdurktw5/mB8ZKIfo+T5lQKoyxsARW5BWMJaRjbvbm6rwNrZ4altBV305ICH pMoA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=fEHQ+yrhEnUirq0+TLk20ZxNEdxLbSGmklvpWQBJ/Eg=; b=i/eaZ8SA8E/yxYZ9zl2liNsiD8jmCep1MYFAULNT99/rjPaBpP5Blab6UDJCcVQ7/q 9nbDyK1WjpstF8xCkgNWx+xOFyXkxJ9Xxc5wxfinDxyCMpL4oJevfMsWyrOrwdvWAZOm S/UBOPuT3GhgXL+eRjkouGny9nnbFQ559MPJT7/nnp2jjy0kl6UDoE4Mja2O3Hz17bsM qprUGsSVPJ9IgYBxVjZhWJIIXk0I6tdxig9zc54oBlw5SGHoViuKCNCmiqPd2ydToXpO MLP7u+MnSFviI/kOGC5b/w5uWwk4D1MnQYVQ8VjOsxBitML2dKdz+/w6JUuG/2blY0Gb QCSA== X-Gm-Message-State: ANhLgQ1I2UE8B/4VLL3IXKKtHjepNrKjHfnCkZ5cInF+FAessdA3yPt7 3YlSj38F7IPOnY+xEAH4PEFS0NxYdfIt3j9NSEA= X-Google-Smtp-Source: ADFU+vt9x9cBOS64PTIyIQ2j/njxY4n6lsUW2HQMrhYWt/hhaJIkjWNZ2Zj4H+Wa/ik6B2eVukvEMM+fGjNgqhgWyJY= X-Received: by 2002:a37:6852:: with SMTP id d79mr8196802qkc.304.1583777980149; Mon, 09 Mar 2020 11:19:40 -0700 (PDT) MIME-Version: 1.0 References: <20200307133116.11443-1-arunisaac@HIDDEN> <20200307133116.11443-3-arunisaac@HIDDEN> In-Reply-To: <20200307133116.11443-3-arunisaac@HIDDEN> From: zimoun <zimon.toutoune@HIDDEN> Date: Mon, 9 Mar 2020 19:19:27 +0100 Message-ID: <CAJ3okZ1n=RTexsb5vCCsXdnBO0KEy6Hm0BuSXh1yKsL2GnHTJA@HIDDEN> Subject: Re: [PATCH v2 2/3] gnu: Generate Xapian package search index. To: Arun Isaac <arunisaac@HIDDEN> Content-Type: text/plain; charset="UTF-8" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 39258 Cc: =?UTF-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>, Pierre Neidhardt <mail@HIDDEN>, 39258 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) On Sat, 7 Mar 2020 at 14:31, Arun Isaac <arunisaac@HIDDEN> wrote: > diff --git a/gnu/packages.scm b/gnu/packages.scm > index d22c992bb1..c8e221de68 100644 > --- a/gnu/packages.scm > +++ b/gnu/packages.scm [...] > @@ -426,6 +429,43 @@ reducing the memory footprint." > #:opts '(#:to-file? #t))))) > cache-file) > > +(define %package-search-index > + ;; Location of the package search-index > + "/lib/guix/package-search.index") > + > +(define (generate-package-search-index directory) > + "Generate under DIRECTORY a Xapian index of all the available packages." > + (define db-path > + (string-append directory %package-search-index)) > + > + (mkdir-p (dirname db-path)) > + (call-with-writable-database db-path > + (lambda (db) > + (fold-packages (lambda (package _) > + (let* ((idterm (string-append "Q" (package-name package))) > + (doc (make-document #:data (string-trim-right > + (call-with-output-string > + (cut package->recutils package <>)) > + #\newline) > + #:terms `((,idterm . 0)))) > + (term-generator (make-term-generator #:stem (make-stem "en") > + #:document doc))) > + (for-each (match-lambda > + ((field . weight) > + (match (field package) > + ((? string? str) > + (index-text! term-generator str > + #:wdf-increment weight)) > + ((lst ...) > + (for-each (cut index-text! term-generator <> > + #:wdf-increment weight) > + lst))) > + (replace-document! db idterm doc))) > + %package-metrics))) > + #f))) > + > + db-path) If I understand correctly, the index is stored with a weight coming from '%package-metrics', right? Well, I am not convinced it is the correct way but I have not tried by myself yet. :-)
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 9 Mar 2020 18:14:32 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Mon Mar 09 14:14:32 2020 Received: from localhost ([127.0.0.1]:51443 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1jBMvA-0004Gs-Bd for submit <at> debbugs.gnu.org; Mon, 09 Mar 2020 14:14:32 -0400 Received: from mail-qk1-f193.google.com ([209.85.222.193]:47024) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <zimon.toutoune@HIDDEN>) id 1jBMv9-0004Gg-9i for 39258 <at> debbugs.gnu.org; Mon, 09 Mar 2020 14:14:31 -0400 Received: by mail-qk1-f193.google.com with SMTP id f28so3886023qkk.13 for <39258 <at> debbugs.gnu.org>; Mon, 09 Mar 2020 11:14:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=u5x6DTCA0Y/AkWuFqbf2NKsPyOsiFMoWzTRu8sn+atc=; b=mjeu6WEwTX3R6LGrR8naEQVvK3yDD9fCZPVFta/acm7qlI2nF6Tvho+CZveBq7QR8/ AySqsUpVbpina/8smjL+sr4SWPb/ZcBTsuRRP8ek2iF62xvNDzX0sSFa1nNSONisId1u wTvk10/o2acWdMK3RkRP65gYYMa57gSpSFxNZKELLTBV5JubKTyyDhriYNsqLIMiGEiA MiTKgaH5Gat7/mcNXRGAq4dBcLOdgbd4DVqfn78GooB+TznCY4oKA+gmvUKRQV65Sso7 pC7pRIGivC9rxjXKI+Uhl6eEmS8NcT0JlxWNl6EotvZlgwxuogJ+Yjiwprvoc619bGmB /G2g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=u5x6DTCA0Y/AkWuFqbf2NKsPyOsiFMoWzTRu8sn+atc=; b=mJCVpEvUGyZMk2CHCp4fdU0k5L89l7BWmSAwbLctb9TBaGMZAsQrT4M3zOAcbO20k/ 9FuVIkTX3tiGqll+lMwrqovzjxEU47VJ0qxhpSA+4VH2RHuOLGpq8yNVvIRlsTbunYD4 NlYEgHa/MP6LrxI6m4joxWXEwsJuvZ2T4pfwliT8Bj+iMc8qL7LCY6KEU08b3E1PT9Fz NQeSWPvw8fSu5R5101EhHb/KVXcvXQIN4L4reZS9ijHUE9+TrBIc0lrA+3gfLaP0ctAc bSEMygsS1DgQ6woevFH583SIDsGrmTkqS2pZEgEkXlGKJip5SeCJG9lww9wyF1NLMmuI h23Q== X-Gm-Message-State: ANhLgQ11Z3miqC8VV2nALifkIxVfE9nFm0LC3iwPLCu/68i4n2FrNRVr d/H33uawVw1Vv207iMDcbUCcyoaX7X9+LAuTIp0= X-Google-Smtp-Source: ADFU+vvT/dLm+KIy0NQUc1lzga8fnjHYgVBetXlAj4kqeHv9p9ks6PiDjlOXtOkklEFQot1NGsfNQ6UXhfu3IGfrozQ= X-Received: by 2002:a37:6852:: with SMTP id d79mr8176717qkc.304.1583777665727; Mon, 09 Mar 2020 11:14:25 -0700 (PDT) MIME-Version: 1.0 References: <20200307133116.11443-1-arunisaac@HIDDEN> <20200307133116.11443-2-arunisaac@HIDDEN> In-Reply-To: <20200307133116.11443-2-arunisaac@HIDDEN> From: zimoun <zimon.toutoune@HIDDEN> Date: Mon, 9 Mar 2020 19:14:13 +0100 Message-ID: <CAJ3okZ05nva16fAAjhUeY=-Fd+RT-F118UXG-PmiuFTEDHNmGw@HIDDEN> Subject: Re: [PATCH v2 1/3] build-self: Add guile-xapian to Guix dependencies. To: Arun Isaac <arunisaac@HIDDEN> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 39258 Cc: =?UTF-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>, Pierre Neidhardt <mail@HIDDEN>, 39258 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) On Sat, 7 Mar 2020 at 14:31, Arun Isaac <arunisaac@HIDDEN> wrote: > diff --git a/build-aux/build-self.scm b/build-aux/build-self.scm > index f2e785b7f1..05d0353ccf 100644 > --- a/build-aux/build-self.scm > +++ b/build-aux/build-self.scm > @@ -1,5 +1,6 @@ > ;;; GNU Guix --- Functional package management for GNU > ;;; Copyright =C2=A9 2014, 2016, 2017, 2018, 2019, 2020 Ludovic Court=C3= =A8s <ludo@HIDDEN> > +;;; Copyright =C2=A9 2020 Arun Isaac <arunisaac@HIDDEN> > ;;; > ;;; This file is part of GNU Guix. > ;;; > @@ -261,6 +262,10 @@ interface (FFI) of Guile.") > #~(define-module (gcrypt hash) > #:export (sha1 sha256)))) > > + (define fake-xapian-hash > + ;; Fake (xapian xapian) module; see below. > + (scheme-file "xapian.scm" #~(define-module (xapian xapian)))) > + Why 'fake-xapian-hash' and not simply 'fake-xapian'?
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 9 Mar 2020 13:03:26 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Mon Mar 09 09:03:26 2020 Received: from localhost ([127.0.0.1]:49975 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1jBI45-0002mx-U0 for submit <at> debbugs.gnu.org; Mon, 09 Mar 2020 09:03:26 -0400 Received: from mail-qt1-f193.google.com ([209.85.160.193]:34444) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <zimon.toutoune@HIDDEN>) id 1jBI44-0002mk-JB for 39258 <at> debbugs.gnu.org; Mon, 09 Mar 2020 09:03:24 -0400 Received: by mail-qt1-f193.google.com with SMTP id 59so6882978qtb.1 for <39258 <at> debbugs.gnu.org>; Mon, 09 Mar 2020 06:03:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=waE7k756z7mml/RIzvy+SVEPtD/IMknUdqHdApZvdpM=; b=dGHgvz69a1pOFlT4gO4i7VS2oJtITAu+5T5vfcrYdPVcPuRgWDfyMWR0xH58LLBsHA wAJCYib6PCUAtQtzV6mqEkG58Jn2qaY2/UWVvoRCreAL0Y8OK9MO8W3gVPdJkNc2Djrs vvAZ0t8I6cWlMb9HYnoqxJfXoOhA4HWqvQOabg9F37YHhQGJUT+VQ5+FdyXwQm28jUew OuLil0iZVE287XCyZCIqH+C9ehC/FsDwh0VzwcFyzcr4XF4K2nE3mFZ+Koo1Z0SNCeOZ 1MhLayySqFGrFpXEU52AzL5F9ZzgwvZFU1nAWXJ8hSkXnbMraZ1Yn/QVld+bz8yekOFL L/Fg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=waE7k756z7mml/RIzvy+SVEPtD/IMknUdqHdApZvdpM=; b=djdqUi/4O5K3BGaBYi07gtGDmmO9AK+3jdiei6SUdxvo4r3Ew9XcL9dgb1/zU6DrdL gdrXwvwAdhrWfV84Jyr2qUmmzcWMamySlKHvXHrm6M6O1ZvVT1j454GKowhOCR26q9Qu WGjL63zrho9bAHRP1EzV1Q2jp61FRQDPtXRclIlHlNp+WPx8YzIoHHUiovadTy4vsmon 077+ndhxqi6HOyH65gaNDd4FPOOAlkTaFbW6z/7tM5y5BWA3AdJBDecH6jXHKEN5r27h Yqo/V3bWeJtFAlS6cEYTeej7033XRCym5i9zPmOjD1d4wd4pBmDANgGmM4i9ZLvFb4XJ y/UA== X-Gm-Message-State: ANhLgQ34MrnnAEl0oq975rd3CJa4OWB2Eqa8jb9MH1uSGWO06lUwL/Z2 zwruMEWGaJP2rfwmAV7IIY6oJDzOqRMNnwS0T5s= X-Google-Smtp-Source: ADFU+vuku6BKlNml22hB13H5ROYSxEjb2c6FLggCYgigwAJM9Dme8QQ9lne/pBMZtOMXTXmF+Nh5GNgraG5ovUOmrew= X-Received: by 2002:ac8:6b44:: with SMTP id x4mr3143322qts.186.1583758999045; Mon, 09 Mar 2020 06:03:19 -0700 (PDT) MIME-Version: 1.0 References: <20200307133116.11443-1-arunisaac@HIDDEN> <87sgijgb1v.fsf@HIDDEN> <cu7tv2zcj9l.fsf@HIDDEN> <875zffcc87.fsf@HIDDEN> <877dzuvues.fsf@HIDDEN> <87blp54yag.fsf@HIDDEN> In-Reply-To: <87blp54yag.fsf@HIDDEN> From: zimoun <zimon.toutoune@HIDDEN> Date: Mon, 9 Mar 2020 14:03:06 +0100 Message-ID: <CAJ3okZ2r5WDEF7jnaFSbg4Lznt_kjo6XtUA-2oEe74RztmN1ZQ@HIDDEN> Subject: Re: [PATCH v2 0/3] Xapian for Guix package search To: =?UTF-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 39258 Cc: Arun Isaac <arunisaac@HIDDEN>, Pierre Neidhardt <mail@HIDDEN>, 39258 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) On Mon, 9 Mar 2020 at 11:29, Ludovic Court=C3=A8s <ludo@HIDDEN> wrote: > > Back to the topic: I believe that Xapian is a huge win both for the > > shell and the future GUI :) > > It could be, but we need to consider all the aspects of the story, > including the maintenance cost and overhead moved to =E2=80=98guix pull= =E2=80=99. So > it=E2=80=99s not so much about =E2=80=9Cbeliefs=E2=80=9D at this point, b= ut rather about > demonstrating what can be done, and I=E2=80=99m glad Arun is exploring th= at > space! I agree. What is currently tested with Xapian is: 1- speeding up (or not) using an inverted index 2- the accuracy using the state-of-art of information retrieval (BM25) About 1- I do not have a strong opinion; even if I find "guix search" terribly slow as I mentioned earlier (one year ago ;-)). About 2- as I mentioned earlier, the 'relevance' function could be improved. Currently, the score is computed only considering the package itself and not the other packages (the words they use, their number etc.). BM25 is the state-of-art using what I tried to explained some time ago when I showed for example TF-IDF. The question is so what the best move to improve the accuracy. And the improvement necessarily uses a global index (of terms, at least). But on the other hand, the improvement should not pay off because it would add complexity and burden, more than the improvement itself. Without testing, we cannot say. Thank you Arun for pushing forward. All the best, simon
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 9 Mar 2020 12:53:31 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Mon Mar 09 08:53:31 2020 Received: from localhost ([127.0.0.1]:49967 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1jBHuV-0002Xt-Ek for submit <at> debbugs.gnu.org; Mon, 09 Mar 2020 08:53:31 -0400 Received: from mail-qt1-f194.google.com ([209.85.160.194]:44633) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <zimon.toutoune@HIDDEN>) id 1jBHuT-0002Xf-46 for 39258 <at> debbugs.gnu.org; Mon, 09 Mar 2020 08:53:29 -0400 Received: by mail-qt1-f194.google.com with SMTP id h16so6807355qtr.11 for <39258 <at> debbugs.gnu.org>; Mon, 09 Mar 2020 05:53:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=x6ipvJqeA/MyReGRGXoahljb7CX4va5mAQShPXMVKNM=; b=Ed3+r/nBU90/76i41Pt4/DIqVLZtIhYYIalWqUlumGsxOmHu6yif2XITJWcMdlUNaE ZVH98TBchhZxjfWDRcXYOlCPdnBg6fffYP9Sk/GmiuUo80JNZegX6a0ZhF6Ar5cYce9y 3qjTUnJCQopbt4VrKecX6Bo6VmKMKooL2/ntE4m7WNe96s7kCyyC0YeV4gZz921Wod7E 50o6KyApnK9EhInxI5H7t6uNYzsjmqNvYdpQoeQGyywwngmJkwyfnCMlTEPGOuqnFvs7 8HTeltGBXIK4Lyfd8NugOfkcwHU/EWPGOvpVr2RLHvWXCOMnkEHplVqqrnP520GsBhTT iJpw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=x6ipvJqeA/MyReGRGXoahljb7CX4va5mAQShPXMVKNM=; b=G6eLvSTSMQq6FRPkF0eS4FZ6Eh5218ETTNz0zJXmQcDCJNP6bKGYQMIOsIprPAu2Wn oGzN6f7SbRWa+nGwmkIFwWzoW4a0ZEMaSo3jCBmsW6BPS4/soWeGTF+e2nyeA2lDosXy 6CfzyzUg75niUMZd9HP+sftJnNd6BSOKhvB1yRuAPocc/kr2lSJz0xE+0ndKw/5gjnlW EyqosY3mV78tcPu7CkrTQScsYEPIl3fznokWhysekZSn1YZvoaDe90zaGAQnj6/6sb6o 3typDRpHwA/FajKimE14m/rg5EWo8i4lfVLk6Nxyn5U/2wGRc+68JshX16NSGL6SDd+X X4EQ== X-Gm-Message-State: ANhLgQ0jO/Z8e/pDKEhBDpecfWqLN36JmjmSnCS6eeMyOcee+oYeNOGK lv1sUXICxDilrLWZvfMxV7HNQOHqkyTcsGBIayE= X-Google-Smtp-Source: ADFU+vtnPvBsbLdP1T2csXK267HFS9ScTkveoX2PnWSeLsIEziKSij6FHOOGN08vH6uZwpufGRayCUFc8GlQAkRu/UU= X-Received: by 2002:aed:2050:: with SMTP id 74mr14212334qta.217.1583758403665; Mon, 09 Mar 2020 05:53:23 -0700 (PDT) MIME-Version: 1.0 References: <20200307133116.11443-1-arunisaac@HIDDEN> <87sgijgb1v.fsf@HIDDEN> <cu7tv2zcj9l.fsf@HIDDEN> <875zffcc87.fsf@HIDDEN> <877dzuvues.fsf@HIDDEN> In-Reply-To: <877dzuvues.fsf@HIDDEN> From: zimoun <zimon.toutoune@HIDDEN> Date: Mon, 9 Mar 2020 13:53:10 +0100 Message-ID: <CAJ3okZ2a=+04ZoEnG=1DrzpQfVJsLkXyd4XqCKPaaeY1kbpoKA@HIDDEN> Subject: Re: [PATCH v2 0/3] Xapian for Guix package search To: Pierre Neidhardt <mail@HIDDEN> Content-Type: text/plain; charset="UTF-8" X-Spam-Score: 2.0 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: On Mon, 9 Mar 2020 at 08:50, Pierre Neidhardt wrote: > Back to the topic: I believe that Xapian is a huge win both for the > shell and the future GUI :) I agree. The big win is to test the strategy of the inverted index strategy and in the same time the state-of-art of scoring (relevance). Content analysis details: (2.0 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- 0.0 URIBL_BLOCKED ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [URIs: ambrevar.xyz] -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [209.85.160.194 listed in list.dnswl.org] 2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: ambrevar.xyz (xyz)] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider (zimon.toutoune[at]gmail.com) -0.0 SPF_PASS SPF: sender matches SPF record 0.0 RCVD_IN_MSPIKE_H3 RBL: Good reputation (+3) [209.85.160.194 listed in wl.mailspike.net] 0.0 RCVD_IN_MSPIKE_WL Mailspike good senders X-Debbugs-Envelope-To: 39258 Cc: =?UTF-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>, Arun Isaac <arunisaac@HIDDEN>, 39258 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: 1.0 (+) On Mon, 9 Mar 2020 at 08:50, Pierre Neidhardt <mail@HIDDEN> wrote: > Back to the topic: I believe that Xapian is a huge win both for the > shell and the future GUI :) I agree. The big win is to test the strategy of the inverted index strategy and in the same time the state-of-art of scoring (relevance).
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.
Received: (at 39258) by debbugs.gnu.org; 9 Mar 2020 12:51:11 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Mar 09 08:51:11 2020
Received: from localhost ([127.0.0.1]:49963 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1jBHsF-0002Um-0X
for submit <at> debbugs.gnu.org; Mon, 09 Mar 2020 08:51:11 -0400
Received: from mail-qv1-f67.google.com ([209.85.219.67]:39700)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <zimon.toutoune@HIDDEN>) id 1jBHsD-0002UY-EB
for 39258 <at> debbugs.gnu.org; Mon, 09 Mar 2020 08:51:09 -0400
Received: by mail-qv1-f67.google.com with SMTP id fc12so4261612qvb.6
for <39258 <at> debbugs.gnu.org>; Mon, 09 Mar 2020 05:51:09 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
h=mime-version:references:in-reply-to:from:date:message-id:subject:to
:cc:content-transfer-encoding;
bh=T/y9xWTArtB3dcEXAycn9B0yVtiarPFvh2nUT3TpsMY=;
b=qj5nwdfZJZ8NgyRxtWpTF1ERt4cW67DonhgYSlMLE4G+U2HuPayGIdWGtbtZWcDW9n
kp8XFeqm+dwiiBw+XExoTujTWzicPnYV0iFgilcBZXrlD3vl01bAn8gDA1rCf5C4nt+n
zNZ0G+mO7N70fIl1LsxP8PTslQGCFvDdiJSwiByQfo0M3yh+pR1K1/+2KCK7ssYtO6o6
7CIedj9lz+eY3zVyCMmDADbdhGEG5q+ve3rOWuL16K2Q86zViR565NDbi4adZzmSVQBX
FxSOFsaMJaW/reEW/FVd9CAKQYPpDMV9LL8VIE3ObY3INMfWkvdoXov20s/6fRmH9bkB
Pxrg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20161025;
h=x-gm-message-state:mime-version:references:in-reply-to:from:date
:message-id:subject:to:cc:content-transfer-encoding;
bh=T/y9xWTArtB3dcEXAycn9B0yVtiarPFvh2nUT3TpsMY=;
b=fc1EAlYU4a3381/sP7aGk09Nf6Vl4ImVz62IaNT/EwvpkRfVwD4mal/B5CGw1t1Fbx
939pQzLh4d3sBlzil+5PJdA6By0EbmVSUIfncyL8D7sE86CjXfMewAXXd41WulfCFU7S
KBceI+CpZwgHRs2iyEE6qQzHns47S/KfmxTOhKHhbFuIIyOyAWUwFXKRKGM0w1yfqueV
12kYLLb6qatbqeeEWwKh6u6jlcFcp2fB65OrzYuJ+JCk6hIOGizuoHKdA9tin74wlX5L
Y5+BpTKY89D8o+0d+0ZL/xwAoNPM1qarR+DjSe9xs/sL257DGMAWwFZmPtPg+BKXmlZP
UvxA==
X-Gm-Message-State: ANhLgQ0JblkgsLzN9oJHW9RmDDShfvXWwoBZNIX/BuECkgHeibPpmw7x
a4sgKlrJNNPgtLfpKJtk120Q7RgV56uXzsM/Dugkrw==
X-Google-Smtp-Source: ADFU+vuoHRbzAgE+nu2OS8wVyc8Ntsaji1Icj5asF6IViUFiRyJBKTVAo0U8mWFFldaICf07eto5uneWcQD72hukkys=
X-Received: by 2002:a0c:c70b:: with SMTP id w11mr13910967qvi.246.1583758263615;
Mon, 09 Mar 2020 05:51:03 -0700 (PDT)
MIME-Version: 1.0
References: <20200307133116.11443-1-arunisaac@HIDDEN>
<87sgijgb1v.fsf@HIDDEN> <cu7tv2zcj9l.fsf@HIDDEN>
<875zffcc87.fsf@HIDDEN> <cu7imjed22r.fsf@HIDDEN>
<87a74qvusm.fsf@HIDDEN>
In-Reply-To: <87a74qvusm.fsf@HIDDEN>
From: zimoun <zimon.toutoune@HIDDEN>
Date: Mon, 9 Mar 2020 13:50:49 +0100
Message-ID: <CAJ3okZ3ozYEhajKb3=CKpVu5AUsFTd-BiD3+zHf28yjy6LUhBw@HIDDEN>
Subject: Re: [PATCH v2 0/3] Xapian for Guix package search
To: Pierre Neidhardt <mail@HIDDEN>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: 2.0 (++)
X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org",
has NOT identified this incoming email as spam. The original
message has been attached to this so you can view it or label
similar future email. If you have any questions, see
the administrator of that system for details.
Content preview: On Mon, 9 Mar 2020 at 08:42, Pierre Neidhardt wrote: > > Arun
Isaac writes: > > >> I’m not sufficiently familiar with Xapian’s query
language. The > >> examples I had in mind were: > >> It’s [...]
Content analysis details: (2.0 points, 10.0 required)
pts rule name description
---- ---------------------- --------------------------------------------------
0.0 URIBL_BLOCKED ADMINISTRATOR NOTICE: The query to URIBL was
blocked. See
http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block
for more information.
[URIs: systemreboot.net]
2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs
[URI: ambrevar.xyz (xyz)]
0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record
0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail
provider (zimon.toutoune[at]gmail.com)
-0.0 SPF_PASS SPF: sender matches SPF record
-0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/,
no trust
[209.85.219.67 listed in list.dnswl.org]
-0.0 RCVD_IN_MSPIKE_H2 RBL: Average reputation (+2)
[209.85.219.67 listed in wl.mailspike.net]
X-Debbugs-Envelope-To: 39258
Cc: Arun Isaac <arunisaac@HIDDEN>,
=?UTF-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>, 39258 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: 1.0 (+)
On Mon, 9 Mar 2020 at 08:42, Pierre Neidhardt <mail@HIDDEN> wrote:
>
> Arun Isaac <arunisaac@HIDDEN> writes:
>
> >> I=E2=80=99m not sufficiently familiar with Xapian=E2=80=99s query lang=
uage. The
> >> examples I had in mind were:
> >> It=E2=80=99s not so much about regexps than it is about selecting indi=
vidual
> >> fields.
> >
> > I have totally not tested this, but I imagine that equivalent Xapian
> > queries might look something like:
> >
> >> guix search | recsel -p name -e 'license ~ "LGPL 3"'
> >
> > guix search license:LGPL3
> >
> >> guix search crypto library | \
> >> recsel -e '! (name ~ "^(ghc|perl|python|ruby)")' -p name,synopsis
> >
> > guix search crypto library AND (NOT ghc) AND (NOT perl) AND (NOT python=
)
> > AND (NOT ruby)
>
> Indeed, if you look at the notmuch-search-terms man page, you'll see
> that you can select fields.
> In my opinion, the recsel format is fully superseded by Xapian.
No!
Because implementing the "fields" using Xapian is not done and it is
not as straightforward as it seems.
For sure, Xapian could do a lot of thing. But we should move one step
after one step.
Let first focus on speed and accuracy. For example, the fact that
"guix search emacs" does not returns first the package 'emacs' using
Xapian is really an issue.
Cheers,
simon
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 9 Mar 2020 12:47:23 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Mon Mar 09 08:47:23 2020 Received: from localhost ([127.0.0.1]:49951 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1jBHoZ-0002Oi-5G for submit <at> debbugs.gnu.org; Mon, 09 Mar 2020 08:47:23 -0400 Received: from mail-qt1-f194.google.com ([209.85.160.194]:38227) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <zimon.toutoune@HIDDEN>) id 1jBHoX-0002OV-4j for 39258 <at> debbugs.gnu.org; Mon, 09 Mar 2020 08:47:22 -0400 Received: by mail-qt1-f194.google.com with SMTP id e20so6830866qto.5 for <39258 <at> debbugs.gnu.org>; Mon, 09 Mar 2020 05:47:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=6mNp3kpk1T+VE1Or+ukp62eiTscFBS06eQ5kQy+gbjA=; b=iRqqytY6j08ksq2iL3jAimCSUK6DWvlkGcSFkJ886i8vHb1+YmKWhoUeJoiT37iYUF T/LfzK5Fepb8iteOXbn+e3I3nA18suf50hHtgk9slpA18O04krK9XugDCNOaYRslAw2I jrOPiGwNXfckE36aIz5y+imdM8mbHkAj0owruz/xRuvt2T30P0zU/hHryBKTAL7fZgB4 jIZvOnEoGqNyAj7uOgFBVTGTkXgMsg8az8nhC0oCLCTdvjFopK3hlc563gV9Q4E6O6M2 UVjKh87qO4FHiWFO3ZErfZo0wljP0Ls2h4AKUIH0D7+AV9i9wS/m4Gf2+7G8VegHJkIY ZOnw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=6mNp3kpk1T+VE1Or+ukp62eiTscFBS06eQ5kQy+gbjA=; b=cnGd6kYaQW1SyyF9/m5Y8pUIgpvzcqrS1fyMNHjlFdJk/ezn71CaEdYGA/Ojai7nQu 7Pfb6c85/6rC6ReXlqn23AEcvi4i2VIs5VHQF+3Ej2okMMsj5qrhku6MvYiU8WmAG/xJ ecLeuoCVk+6NFsTeoWMdzF4zK6p7iqESHQGiFJxSGfaOWJNokTQg/UJiaggUHFXdYUFa LMLilMEJcmrOoms/aJfWKHp7LW20PWukjo/QarRKekDdhRdVE97FA9rTpmfCTaSeGvyC jK5+hP9rKrNdtfVqDi9UoLbybOBDJeeE4+46xQILyqv5E5s4AenSu/rvqMsHzUWrrpZd CQiA== X-Gm-Message-State: ANhLgQ2tv6UApTyY76HvPefFU2+Oz1Y+GrXnZsyLME9+64SyM51MV/bW hDQm5ngY5I0bVV6kKHlwqHZY/0YSVPLuwKjsCdo= X-Google-Smtp-Source: ADFU+vs8UpwL5RdAe2ORXekdW6Vml1ukL0qZHtJPEpq0F7FN+JrnJU3mnRTeepa/BuhqZpZUv2u7Tznwjd/675nDpjs= X-Received: by 2002:aed:2050:: with SMTP id 74mr14190980qta.217.1583758035535; Mon, 09 Mar 2020 05:47:15 -0700 (PDT) MIME-Version: 1.0 References: <20200307133116.11443-1-arunisaac@HIDDEN> <87sgijgb1v.fsf@HIDDEN> <cu7tv2zcj9l.fsf@HIDDEN> <875zffcc87.fsf@HIDDEN> In-Reply-To: <875zffcc87.fsf@HIDDEN> From: zimoun <zimon.toutoune@HIDDEN> Date: Mon, 9 Mar 2020 13:47:01 +0100 Message-ID: <CAJ3okZ3ebRtuZa=zRRQvKbwESvdwcAJ+B-QTNMF=WwAwi3Z8fA@HIDDEN> Subject: Re: [PATCH v2 0/3] Xapian for Guix package search To: =?UTF-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 39258 Cc: Arun Isaac <arunisaac@HIDDEN>, Pierre Neidhardt <mail@HIDDEN>, 39258 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) On Sun, 8 Mar 2020 at 12:33, Ludovic Court=C3=A8s <ludo@HIDDEN> wrote: > Arun Isaac <arunisaac@HIDDEN> skribis: > > This is a problem, but I would see it as a necessary "compilation" > > step. :-P In fact, this whole patchset speeds up `guix search` by doing > > part of the work of `guix search` ahead of time. So, some such cost is > > unavoidable. > > Yeah. I think we need to take the whole user experience into account, > not just =E2=80=98guix search=E2=80=99. =E2=80=98guix pull=E2=80=99 alre= ady feels very slow, and it=E2=80=99s a > fairly common operation. Conversely, =E2=80=98guix search=E2=80=99 takes= roughly > between 0.5 and 2 seconds and is an uncommon operation on a =E2=80=9Cslow= path=E2=80=9D > (in the sense that when you=E2=80=99re searching for software, you=E2=80= =99ll probably > have to spend more than a couple of seconds to find what you=E2=80=99re l= ooking > for.) We could imagine something doing the job of indexing in the background; using the daemon or whatever. > >> What I like about the recutils format in this context is that it=E2=80= =99s both > >> human- and machine-readable. The examples in the manual show how it c= an > >> be useful to select the information displayed or to refine the search > >> (info "(guix) Invoking guix package"). > > > > Xapian's query language is much more natural (as in natural language) > > than the regexp based techniques we need to use with recutils. I have > > hardly ever used the regexp based search and I suspect many others > > haven't either. Also, refining the search query should be easier to do > > with Xapian. We could even use Xapian's query expansion feature to > > suggest improved queries to the user. > > I=E2=80=99m not sufficiently familiar with Xapian=E2=80=99s query languag= e. The > examples I had in mind were: > > guix search malloc | recsel -p name,version,relevance > guix search | recsel -p name -e 'license ~ "LGPL 3"' > guix search crypto library | \ > recsel -e '! (name ~ "^(ghc|perl|python|ruby)")' -p name,synopsis I think these examples are good ones to benchmark the different approaches. Because the speed is one thing, the accuracy is another one. Let cut the "slow path" by providing a better experience when searching. ;-= ) > It=E2=80=99s not so much about regexps than it is about selecting individ= ual > fields. The regexp should be provided directly to "guix search" actually and 'recsel' is only a "filter" allowing to deal differently with the fields. > To me, adding 20=E2=80=9350 seconds on =E2=80=98guix pull=E2=80=99 would = be undesirable. :-/ Ok, at least it is clear. :-) And computing in the background? All the best, simon
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 9 Mar 2020 12:41:00 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Mon Mar 09 08:41:00 2020 Received: from localhost ([127.0.0.1]:49940 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1jBHiO-0002EH-0P for submit <at> debbugs.gnu.org; Mon, 09 Mar 2020 08:41:00 -0400 Received: from mail-qt1-f196.google.com ([209.85.160.196]:44394) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <zimon.toutoune@HIDDEN>) id 1jBHiM-0002E4-4O for 39258 <at> debbugs.gnu.org; Mon, 09 Mar 2020 08:40:58 -0400 Received: by mail-qt1-f196.google.com with SMTP id h16so6777559qtr.11 for <39258 <at> debbugs.gnu.org>; Mon, 09 Mar 2020 05:40:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=lyEK1xGVq9wbyxBsvLKiCTo7/3Cxi26+1BTnqsidQO8=; b=POqiy9wXHNQd7Nw3CFLvuKN0zsNuQ6Yod+YZ0h472zhsV/qaafRKvV7kFm6xuXSSRD sZ3P48r2NX9WoYmkjCVF7GReQoU/vVG7fCyeq/vskNcBZJyN9S5TOP765IRvNROTYcPL DZx4oxNRG5oT1H299bTaGuItaA5oQjrYNVuh2oBJACL+4BL8ODXiYO9LUrBQqNrVBNI4 buP4o+28G0wgKtXBlLfqfiJ2UEG6TpOXeXPMwLQdWi0Ev9+YSt1BWBSdgTTg/l08AHAH 8NKe8yk+LExO+5lGAvCtPK4CNjYA89E/eg5Wn2v8OERUk2hh6rRgxuTyEFOuI45Ka5UD mzgA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=lyEK1xGVq9wbyxBsvLKiCTo7/3Cxi26+1BTnqsidQO8=; b=DBSSvL0dNQ8vyvMIm10SqBCehpB0BVW8a0aL9RCOARswmWnNZ+BrTKwrIMFGKlGs/6 CEooaghNacPe5r8ioxZ4jAgpglJ1V0XB/CS3NcnlCj36I6ha3pHbUJJnmkUoyoNLzefZ a6fvjjFTuUCHxKb1jsiuhWWmNI6Tk3jiP9xgjESu1sdE25ZihTixS/hK3XRpF3nsRS3p RZlKPMCJuqZZQE+HMJlApyKI7tT0BE4LWJpqrrzQkWacQUrD+wn7hM97/3LKs+FwOjau 80++R89Ll9CfEeOTbtGPR0JVNNSMXQsl7f3vehAsFrfR1oCHZD74OtLnRFR+/A2NbvIO pfOQ== X-Gm-Message-State: ANhLgQ36+QQ4kXMwNfcWbZ5e7fplG3J9RhogkmzNEplJVXcAqzPMXnV0 56zbk7Rhd97DqzDqEs1FBinbARe/2c7nmrbSTI4= X-Google-Smtp-Source: ADFU+vtzPZ/0TKTXFnXvZF15UtqOR8RHPOCYtfDoQHDUyWFtubLoCbBAq3rZwDtGBBZPNOtNjRfvqroTIxrCZ4X4rgA= X-Received: by 2002:ac8:6b44:: with SMTP id x4mr3058646qts.186.1583757652303; Mon, 09 Mar 2020 05:40:52 -0700 (PDT) MIME-Version: 1.0 References: <20200307133116.11443-1-arunisaac@HIDDEN> <87sgijgb1v.fsf@HIDDEN> <cu7tv2zcj9l.fsf@HIDDEN> In-Reply-To: <cu7tv2zcj9l.fsf@HIDDEN> From: zimoun <zimon.toutoune@HIDDEN> Date: Mon, 9 Mar 2020 13:40:39 +0100 Message-ID: <CAJ3okZ2pxP6ALANTxtEpp9Gnaejj69D5ULficdmmH0CfROSLiQ@HIDDEN> Subject: Re: [PATCH v2 0/3] Xapian for Guix package search To: Arun Isaac <arunisaac@HIDDEN> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 39258 Cc: =?UTF-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>, Pierre Neidhardt <mail@HIDDEN>, 39258 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) On Sun, 8 Mar 2020 at 10:02, Arun Isaac <arunisaac@HIDDEN> wrote: > >> It turns out that most of the time is spent in printing and texinfo > >> rendering of the search results. > > Also, when we put all package metadata into the Xapian index, we don't > have to look up any of the package variables in (gnu packages *) during > `guix search` time. This also contributes substantially to the speedup. Yes, magic power of inverted index. ;-) > > Also, if the 12K+ descriptions need to be rendered at the time the user > > runs =E2=80=98guix pull=E2=80=99, the experience may not be great, beca= use it could take > > a bit of time. > > This is a problem, but I would see it as a necessary "compilation" > step. :-P In fact, this whole patchset speeds up `guix search` by doing > part of the work of `guix search` ahead of time. So, some such cost is > unavoidable. Currently "guix pull" is rather long on my machine. I would accept a couple of seconds more (even minutes). So this compilation step could be done at the "guix pull" time. Or even we could imagine something indexing in the background. > > What I like about the recutils format in this context is that it=E2=80= =99s both > > human- and machine-readable. The examples in the manual show how it ca= n > > be useful to select the information displayed or to refine the search > > (info "(guix) Invoking guix package"). > > Xapian's query language is much more natural (as in natural language) > than the regexp based techniques we need to use with recutils. I have > hardly ever used the regexp based search and I suspect many others > haven't either. Also, refining the search query should be easier to do > with Xapian. We could even use Xapian's query expansion feature to > suggest improved queries to the user. > > That said, if we want the recutils format, we can still keep it in a > simplified form like so. > > name: inkscape > version: 0.92.4 > synopsis: Vector graphics editor > > name: inklingreader > version: 0.8 > synopsis: Wacom Inkling skecth format conversion and manipulation > > > Also: I=E2=80=99d recommend tackling one thing at a time. :-) > > I totally agree, but I'm tempted to say that pre-rendering would be a > lot cheaper with the simplified form of search results. :-) IMHO, we "just" need to propose different outputs mimicking "git log --format". Soemthing like "guix search --format=3D". What do you think? All the best, simon
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 9 Mar 2020 12:35:01 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Mon Mar 09 08:35:01 2020 Received: from localhost ([127.0.0.1]:49936 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1jBHcb-00025E-4w for submit <at> debbugs.gnu.org; Mon, 09 Mar 2020 08:35:01 -0400 Received: from mail-qt1-f194.google.com ([209.85.160.194]:37962) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <zimon.toutoune@HIDDEN>) id 1jBHcZ-000250-9I for 39258 <at> debbugs.gnu.org; Mon, 09 Mar 2020 08:35:00 -0400 Received: by mail-qt1-f194.google.com with SMTP id e20so6801346qto.5 for <39258 <at> debbugs.gnu.org>; Mon, 09 Mar 2020 05:34:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=hXNbi+sZS/8CstIh8f5lAfkkg89LaVQ/ULwY4A9dBGo=; b=DHsjdAEbYlZoFz69QE4evkHlOiYJTh7LQvWd3jIXWPWts4b07XSwrh5yksGu9ksm3P c7cOxmxWxLhJVbgEgy/FCHDH6bhrVHeRp34lV1wP6V3eHiekn/I+AtgrKKJs0Srnhbam 8n+Dhx9XcUY/e6WsYRUXtTiAXpwfoumYk5zwCZ79SvsZb7mz0sah0fSz29+K+0Fk1rAV W0koGW3gEG1Itejmy5jTD1SzNTdTQPD4S+kkn/yY5itYE39X3pRcGGtKrWPHv5uXyOF6 0r3PlY36rKG/+aP7FY4L7AVKIgHuDeXo6QmK6I/VM+BWLJK05+hKo2QFh+aRzwwEEkyV R17Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=hXNbi+sZS/8CstIh8f5lAfkkg89LaVQ/ULwY4A9dBGo=; b=fJaT8A2LDJYfFgPpTp4tx2BN1y9rAMh/feku5Kjzn8+xfA6c5s6xRY1M4/hpObfoip 8x3Ur5GL0gSmFYvZMB0GiDM6eNJ+P8CNJpX7fQWd04MDxj8YZ3Wdf11OoCeQgR4gtenM Iqe+HPS2c2/XJTk9m8/mYX1WMDm7c6ZeRMZ1cYR1R/y7ClDSN0jtih8/+8GrGqqQCdmA 9NzwcHMBP54VG9judWAha2VHaV1dwxJFJGr+jIuyPG7k7CbiCZIB5PC6WxxArKnimOPL K9adVLLHnbn+ePttVCQfFlt6DRqGJCnNS29lAzQ/Y+cZz2ExuViVghpZ1I9jk9Eh/yZp QNew== X-Gm-Message-State: ANhLgQ1Jl8HSfNBHDMpCyHwSAU4wP6a4gj9LuU5L20gLkf9iFKV/19hS sWTBf+iNcZfgQMT6Wc5CToyjAhcW/Qphey7i2/E= X-Google-Smtp-Source: ADFU+vtX1+nedWy0pUqCoNX8ldYMpTxLwq3pGKUjaOtZqdPrmDjt1L7UaIFtqCnliqZu4hKluVlPWOIF3kZjZbzKLmw= X-Received: by 2002:ac8:2fd4:: with SMTP id m20mr13812947qta.313.1583757293658; Mon, 09 Mar 2020 05:34:53 -0700 (PDT) MIME-Version: 1.0 References: <20200307133116.11443-1-arunisaac@HIDDEN> <87sgijgb1v.fsf@HIDDEN> In-Reply-To: <87sgijgb1v.fsf@HIDDEN> From: zimoun <zimon.toutoune@HIDDEN> Date: Mon, 9 Mar 2020 13:34:40 +0100 Message-ID: <CAJ3okZ3aHikno_vxt=Ox8yzQs4T7Dmx815HQiPX1_0Xt7TBZhA@HIDDEN> Subject: Re: [PATCH v2 0/3] Xapian for Guix package search To: =?UTF-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 39258 Cc: Arun Isaac <arunisaac@HIDDEN>, Pierre Neidhardt <mail@HIDDEN>, 39258 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) Hi, On Sat, 7 Mar 2020 at 21:33, Ludovic Court=C3=A8s <ludo@HIDDEN> wrote: > Arun Isaac <arunisaac@HIDDEN> skribis: > > Why not use a simpler package search results format like Arch Linux or = Debian > > does? We could just display the package name, version and synopsis like= so. > > > > inkscape 0.92.4 > > Vector graphics editor > > inklingreader 0.8 > > Wacom Inkling sketch format conversion and manipulation > > > > Why do we need the entire recutils format? If the user is interested, t= hey can > > always use `guix package --show` to get the full recutils formatted > > info. Having shorter search results will make everything even faster an= d much > > more readable. WDYT? > > What I like about the recutils format in this context is that it=E2=80=99= s both > human- and machine-readable. The examples in the manual show how it can > be useful to select the information displayed or to refine the search > (info "(guix) Invoking guix package"). > > Also: I=E2=80=99d recommend tackling one thing at a time. :-) I agree with Ludo. And IMHO, we should add "guix search --format=3D<options>" mimicking how "git log" works. By default, displays as Arun proposes. Using '--format=3Dfull" as it is done now by default. And we could imagine "--format=3D%name \t %version \n %description" etc. > I think we should look at a profile of =E2=80=98package->recutils=E2=80= =99, there=E2=80=99s > probably room for improvement there. Interesting. Note that speed was the initial motivation but accuracy is another important one. As we discussed earlier when I showed an example with TF-IDF. And Xapian implemets the state-of-art (BM25) for scoring. All the best, simon
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 9 Mar 2020 12:28:36 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Mon Mar 09 08:28:36 2020 Received: from localhost ([127.0.0.1]:49925 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1jBHWN-0001uE-Vi for submit <at> debbugs.gnu.org; Mon, 09 Mar 2020 08:28:36 -0400 Received: from mail-qt1-f193.google.com ([209.85.160.193]:37096) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <zimon.toutoune@HIDDEN>) id 1jBHWK-0001ty-GG for 39258 <at> debbugs.gnu.org; Mon, 09 Mar 2020 08:28:33 -0400 Received: by mail-qt1-f193.google.com with SMTP id l20so5131750qtp.4 for <39258 <at> debbugs.gnu.org>; Mon, 09 Mar 2020 05:28:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=No12Ae9jpp1K4RC70LbWxeSMbSmRS6/GpY3BfUS2+j4=; b=Bf2rA/mOW19nqXTojWr6qLwBxzX3qM1b5mCu9SGIyXy6hv9bQXrjLLcaekk1vdOKD0 d3gv4UXf/JduZVZpU8mf6A4PgDTI2ojaa03a/Z+PTX1MRe+jvFdWlft/KB51SkRLwJVz xZTXTT1a+oZhlD35iJi7PfA3XomhMgGfYwKyM94nYmGrdtmuj2X7ffm0FkasngkwUUxa 3cKLwzy0aR7rhdSD4A2pUI63qVPY/9Y/AdZGWJz6/ih1aPeg9PCUVGz1oXxv0H9PbJF8 9mykeSG8wSg2qi/TiJeTPwJxZtVCVnzjODzBoZzdtL2NYgaVp5cZgPeVxLSIMBVPk2Xr zHLg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=No12Ae9jpp1K4RC70LbWxeSMbSmRS6/GpY3BfUS2+j4=; b=SyaqdeLxoIyW42DEikbkMz8nvM7D8zi6SRhQ2GldlxaYd+9WZA7IGec08m+EAZByFS kzRWpdNlasJEqSu7T7K/iUR2oYRZ3KhYuF2KxkSbRJDjOIU7dyIdSw2FSYiARm36FmmA tfO/OtUeJ7lmdKEfMLN4VF9GmZ+q7fXOGx6HrXfnXLsJ9E0s7N7NYfTPH6vt0fwN3oqE maZ1y3HLmXfatdFicUdxJdu3vG0suqaIEsOwDYVp8ntT0KHtKR3ZJ+Nzw1DqQnahVu9r XFG0u4Yt8svt7GPAbP6GV+C2iNTeW512XJ5+KMPpL510ZiwcEqQz+Bbq8tfQp+CDWMe2 H2Ug== X-Gm-Message-State: ANhLgQ3T3Z7f24iLZtcL59/fBQHxXeqLXKrDglAqXVzyeDeWeTVWZhdt tm0dwOd2Ln4xusThw6DaRa6M8aP9Z//DRMor+CQ= X-Google-Smtp-Source: ADFU+vtEm58t6CU8k58cDT5/+TrrGebfqqu8IHiw0c+DMBEZA2M3RR/PenDsdKctZar2uRdpBp48msC7o8uHf1Iz+4U= X-Received: by 2002:ac8:6b44:: with SMTP id x4mr3010736qts.186.1583756906844; Mon, 09 Mar 2020 05:28:26 -0700 (PDT) MIME-Version: 1.0 References: <20200307133116.11443-1-arunisaac@HIDDEN> In-Reply-To: <20200307133116.11443-1-arunisaac@HIDDEN> From: zimoun <zimon.toutoune@HIDDEN> Date: Mon, 9 Mar 2020 13:28:08 +0100 Message-ID: <CAJ3okZ1C-9EKjgX=2B5t9wz1_wnXwuLeHAXBQocEEsHeXMv8GA@HIDDEN> Subject: Re: [PATCH v2 0/3] Xapian for Guix package search To: Arun Isaac <arunisaac@HIDDEN> Content-Type: text/plain; charset="UTF-8" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 39258 Cc: =?UTF-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>, Pierre Neidhardt <mail@HIDDEN>, 39258 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) Hi, On Sat, 7 Mar 2020 at 14:31, Arun Isaac <arunisaac@HIDDEN> wrote: > --8<---------------cut here---------------start------------->8--- > With a warm cache, > $ time guix search inkscape > > real 0m1.787s > user 0m1.745s > sys 0m0.111s > --8<---------------cut here---------------end--------------->8--- > > --8<---------------cut here---------------start------------->8--- > $ time /tmp/test/bin/guix search inkscape > > real 0m0.199s > user 0m0.182s > sys 0m0.024s > --8<---------------cut here---------------end--------------->8--- IMHO, it is interesting to compare the list of results and the order of the both query; as i did with Emacs. Speed is one thing, the initial motivation. But accuracy is maybe more important. > - The package cache would grow in size, and lookup would be slowed down > because we need to load the entire cache into memory. Xapian, on the other > hand, need only look up the specific packages that match the search query. I agree that 'fold-packages' could become soon a bottleneck. IMHO, 'mset-fold' should be a drop-in replacement of 'fold-package' in the search function. > - Xapian can provide superior search results due to it stemming and language > models. > - Xapian can provide spelling correction and query expansion -- that is, > suggest search terms to improve search results. Note that I haven't > implemented this yet and is out of scope in this patchset. I agree too that Xapian should improve the user experience when searching. > * Simplify our package search results > > Why not use a simpler package search results format like Arch Linux or Debian > does? We could just display the package name, version and synopsis like so. > > inkscape 0.92.4 > Vector graphics editor > inklingreader 0.8 > Wacom Inkling sketch format conversion and manipulation2 > > Why do we need the entire recutils format? If the user is interested, they can > always use `guix package --show` to get the full recutils formatted > info. Having shorter search results will make everything even faster and much > more readable. WDYT? I disagree. What I proposed some time ago was to have different flavour of the ouput of search; as e.g., 'git log --pretty=oneline' etc.. For example by default, it should be what you suggest. Then "guix search --format=full" should output the current. And we could imagine mimick the Git log strategy: "guix search --format="%name %version\n%license" etc. WDYT? > > Is (make-stem "en") for the locale? > > I still have English hard-coded. I haven't yet figured out how to detect the > locale and stem accordingly. But, there is a larger problem. Since we cannot > anticipate what locale the user will run guix search with, should we build the > Xapian index for all locales? That is, should we index not only the English > versions of the packages but also all other translations as well? I understand. Let consider that for the next round. > > package-search-index and package-cache-file could be refactored > > because they share all the same code. > > Yes, they could be. However, I'll postpone to the next iteration of the > patchset. Ok. > > I do not know what is the convention for the bindings. > > But there is 'fold-packages' so I would be inclined to 'fold-msets' or > > something in this flavour. > > Well, everywhere else in guile we have such things as vhash-fold, string-fold, > hash-fold, stream-fold, etc. That's why I went with mset-fold. Also, we are > folding over a single mset (match-set). So, mset should be in the singular. I understand. > > And more importantly, 'make as-derivations' to avoid a "guix pull" breakage, > > Ah do not forget to adapt some tests. > > Will do this once we have consensus about the other features of this patchset. And we should test that on different machines and states. > > Xapian does not return the package 'emacs' itself as the first. And worse, > > it is not returned at all. > > In this patchset, since we're indexing the package name as well, emacs is > returned but it is still far from the beginning. This is an issue. IMHO, it is because of the BM25 score. It is too rough and some weight should be applied. But that another story. The fix is: a- provide a scoring function to Xapian as the doc explains b- adapt 'fold-package' to 'mset-fold' in 'find-packages-by-description' and implement our version of BM25 then use it in 'relevance' > > I propose the value of 4294967295 for pagesize. > > In this patchset, I pass (database-document-count db) as the #:maximum-items > keyword argument to enquire-mset. This is the upstream recommended way to get > all search results. I hadn't done this earlier since I hadn't yet wrapped > database-document-count in guile-xapian. Cool! > My laptop is quite old with a particularly slow HDD. Hence my motivation to > improve guix search performance! I agree. But performance is not all. Accuracy counts more! :-) > > I think we should weigh the pros and cons on all these aspects: speed, > > complexity and maintenance cost, search result quality, search features, > > etc. > > I agree. I agree too. We should write a benchmark. For example, using Emacs as query or more complex we could think of. All the best, simon
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 9 Mar 2020 10:35:47 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Mon Mar 09 06:35:47 2020 Received: from localhost ([127.0.0.1]:49844 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1jBFlD-0003cV-7L for submit <at> debbugs.gnu.org; Mon, 09 Mar 2020 06:35:47 -0400 Received: from eggs.gnu.org ([209.51.188.92]:53072) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <ludo@HIDDEN>) id 1jBFlB-0003cH-Ga for 39258 <at> debbugs.gnu.org; Mon, 09 Mar 2020 06:35:45 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:39688) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <ludo@HIDDEN>) id 1jBFl4-0001Sw-4s; Mon, 09 Mar 2020 06:35:38 -0400 Received: from [2001:660:6102:320:e120:2c8f:8909:cdfe] (port=40414 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from <ludo@HIDDEN>) id 1jBFl3-0002tB-0L; Mon, 09 Mar 2020 06:35:37 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN> To: Arun Isaac <arunisaac@HIDDEN> Subject: Re: [PATCH v2 0/3] Xapian for Guix package search References: <20200307133116.11443-1-arunisaac@HIDDEN> <87sgijgb1v.fsf@HIDDEN> <cu7tv2zcj9l.fsf@HIDDEN> <875zffcc87.fsf@HIDDEN> <cu7imjed22r.fsf@HIDDEN> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 20 =?utf-8?Q?Vent=C3=B4se?= an 228 de la =?utf-8?Q?R?= =?utf-8?Q?=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Mon, 09 Mar 2020 11:35:35 +0100 In-Reply-To: <cu7imjed22r.fsf@HIDDEN> (Arun Isaac's message of "Mon, 09 Mar 2020 01:57:40 +0530") Message-ID: <87r1y13jew.fsf@HIDDEN> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 39258 Cc: mail@HIDDEN, 39258 <at> debbugs.gnu.org, zimon.toutoune@HIDDEN X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.7 (-) Hello! Arun Isaac <arunisaac@HIDDEN> skribis: >>> This could be accomplished even with pre-rendering. Xapian provides >>> "slots" to store arbitrary strings with a document. Instead of storing >>> the pre-rendered document as a whole, we could store pre-rendered fields >>> in separate slots. Then, during `guix search` time, we can assemble the >>> result from these pre-rendered fields. >> >> I=E2=80=99m not sure I understand. The index wouldn=E2=80=99t store pre= -rendered >> strings for every possible terminal width, right? > > No, it wouldn't. It would store a partially pre-rendered string, that is > without fill-paragraph. We run fill-paragraph at `guix search` time to > complete the rendering. Note that Texinfo rendering doesn=E2=80=99t use (@ (guix ui) fill-paragraph= ). It has its own paragraph-filling code. We cannot use =E2=80=98fill-paragra= ph=E2=80=99 after Texinfo rendering anyway, since Texinfo knows where things can be filled and where they cannot=E2=80=94e.g., @example. >> I think we need to take the whole user experience into account, not >> just =E2=80=98guix search=E2=80=99. =E2=80=98guix pull=E2=80=99 already= feels very slow, and it=E2=80=99s a >> fairly common operation. Conversely, =E2=80=98guix search=E2=80=99 take= s roughly >> between 0.5 and 2 seconds and is an uncommon operation on a =E2=80=9Cslow >> path=E2=80=9D (in the sense that when you=E2=80=99re searching for softw= are, you=E2=80=99ll >> probably have to spend more than a couple of seconds to find what >> you=E2=80=99re looking for.) > > I agree we can't compromise too much on `guix pull` performance. > >> To me, adding 20=E2=80=9350 seconds on =E2=80=98guix pull=E2=80=99 would= be undesirable. :-/ > > Maybe I'm missing something here. guix pull takes around 40 minutes on > my machine. In comparison to that, is another 20-50 seconds (roughly 1 > minute) a big deal? How much time would it be acceptable to spend on > building the Xapian index? On my laptop, in the best case, when all the substitutes are available (not uncommon), it takes 2 minutes. Sometimes, when some substitutes are missing, it takes 15 minutes. So of course, the 20=E2=80=9350 seconds matter only in the best case. But = they matter primarily because that index build may not be substitutable: it=E2= =80=99s possibly unique to each profile (see below). That means we know we=E2=80= =99re often going to pay for it. > Also, is it possible to somehow provide substitutes for the Xapian index > so that the user does not have to actually build it locally during `guix > pull` time? We could provide a substitute for users who use only the official 'guix channel. However, as soon as users combine multiple channels, they=E2=80= =99ll have to build the index locally. >> I=E2=80=99m not sufficiently familiar with Xapian=E2=80=99s query langua= ge. The >> examples I had in mind were: >> It=E2=80=99s not so much about regexps than it is about selecting indivi= dual >> fields. > > I have totally not tested this, but I imagine that equivalent Xapian > queries might look something like: > >> guix search | recsel -p name -e 'license ~ "LGPL 3"' > > guix search license:LGPL3 Nice. >> guix search crypto library | \ >> recsel -e '! (name ~ "^(ghc|perl|python|ruby)")' -p name,synopsis > > guix search crypto library AND (NOT ghc) AND (NOT perl) AND (NOT python) > AND (NOT ruby) This one is not quite equivalent I guess, but yeah. :-) >> What I meant was that we could use (statprof) to see whether/how Texinfo >> rendering/parsing can be optimized. > > Oh, ok. I'll try this if we decide not to pre-render. It=E2=80=99d be beneficial anyways. Thank you! Ludo=E2=80=99.
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.
Received: (at 39258) by debbugs.gnu.org; 9 Mar 2020 10:29:10 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Mar 09 06:29:10 2020
Received: from localhost ([127.0.0.1]:49840 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1jBFeo-0003S7-EM
for submit <at> debbugs.gnu.org; Mon, 09 Mar 2020 06:29:10 -0400
Received: from eggs.gnu.org ([209.51.188.92]:51765)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <ludo@HIDDEN>) id 1jBFel-0003Rv-4V
for 39258 <at> debbugs.gnu.org; Mon, 09 Mar 2020 06:29:07 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:39591)
by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <ludo@HIDDEN>)
id 1jBFee-0004jT-A6; Mon, 09 Mar 2020 06:29:00 -0400
Received: from [2001:660:6102:320:e120:2c8f:8909:cdfe] (port=40412 helo=ribbon)
by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
(Exim 4.82) (envelope-from <ludo@HIDDEN>)
id 1jBFec-000323-H6; Mon, 09 Mar 2020 06:28:59 -0400
From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>
To: Pierre Neidhardt <mail@HIDDEN>
Subject: Re: [PATCH v2 0/3] Xapian for Guix package search
References: <20200307133116.11443-1-arunisaac@HIDDEN>
<87sgijgb1v.fsf@HIDDEN> <cu7tv2zcj9l.fsf@HIDDEN>
<875zffcc87.fsf@HIDDEN> <877dzuvues.fsf@HIDDEN>
X-URL: http://www.fdn.fr/~lcourtes/
X-Revolutionary-Date: 20 =?utf-8?Q?Vent=C3=B4se?= an 228 de la =?utf-8?Q?R?=
=?utf-8?Q?=C3=A9volution?=
X-PGP-Key-ID: 0x090B11993D9AEBB5
X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc
X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5
X-OS: x86_64-pc-linux-gnu
Date: Mon, 09 Mar 2020 11:28:55 +0100
In-Reply-To: <877dzuvues.fsf@HIDDEN> (Pierre Neidhardt's message of
"Mon, 09 Mar 2020 08:50:35 +0100")
Message-ID: <87blp54yag.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Spam-Score: 1.3 (+)
X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org",
has NOT identified this incoming email as spam. The original
message has been attached to this so you can view it or label
similar future email. If you have any questions, see
the administrator of that system for details.
Content preview: Hello, Pierre Neidhardt <mail@HIDDEN> skribis: > Ludovic
Courtès <ludo@HIDDEN> writes: > >> Yeah. I think we need to take the whole
user experience into account, >> not just ‘guix search’. ‘guix pull’
already feels very slow, and it’s [...]
Content analysis details: (1.3 points, 10.0 required)
pts rule name description
---- ---------------------- --------------------------------------------------
0.0 URIBL_BLOCKED ADMINISTRATOR NOTICE: The query to URIBL was
blocked. See
http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block
for more information.
[URIs: gnu.org]
-0.0 SPF_PASS SPF: sender matches SPF record
0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record
2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs
[URI: ambrevar.xyz (xyz)]
-0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at https://www.dnswl.org/,
low trust
[209.51.188.92 listed in list.dnswl.org]
X-Debbugs-Envelope-To: 39258
Cc: Arun Isaac <arunisaac@HIDDEN>, 39258 <at> debbugs.gnu.org,
zimon.toutoune@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: 0.3 (/)
Hello,
Pierre Neidhardt <mail@HIDDEN> skribis:
> Ludovic Court=C3=A8s <ludo@HIDDEN> writes:
>
>> Yeah. I think we need to take the whole user experience into account,
>> not just =E2=80=98guix search=E2=80=99. =E2=80=98guix pull=E2=80=99 alr=
eady feels very slow, and it=E2=80=99s a
>> fairly common operation. Conversely, =E2=80=98guix search=E2=80=99 take=
s roughly
>> between 0.5 and 2 seconds and is an uncommon operation on a =E2=80=9Cslo=
w path=E2=80=9D
>> (in the sense that when you=E2=80=99re searching for software, you=E2=80=
=99ll probably
>> have to spend more than a couple of seconds to find what you=E2=80=99re =
looking
>> for.)
>
> I think I disagree with "guix search" being an uncommon operation and a
> slow path.
(Not =E2=80=9Cand=E2=80=9D but =E2=80=9Con=E2=80=9D a slow path.)
> - The slowness of `guix search' (and the awkwardness of recutils) is
> maybe what makes it uncommon: users refrain from using it because it's
> too impractical.
I think =E2=80=9Cslowness=E2=80=9D and =E2=80=9Cawkwardness=E2=80=9D are ov=
erstatements. I=E2=80=99m not saying
this is perfect, but to me it=E2=80=99s not bad. (Of course I=E2=80=99m bi=
ased :-), but
I=E2=80=99ve used other similar tools and this one looks rather good compar=
ed to
what I=E2=80=99ve used.)
> - Searches are typically refined, i.e. you run a search multiple times
> by precising the terms, so in that sense I believe `guix search` is a
> very common operation. Or should be.
>
> Anyways, one of the key issues here is the inherent limitation of the
> shell interface that does not allow us to directly and contextually
> process the output of a command (at least not without rerunning it).
I agree, but =E2=80=98guix search=E2=80=99 is a shell command, so we have t=
o adapt to
that context.
> Concretely, the GUI search would only return the package name, version
> and synopses. No need for the Texinfo / recutils juggling.
>
> Then the user would select the packages of interest to display more
> details. This allows us to query the full details just-in-time.
Note that Emacs-Guix does that, although it doesn=E2=80=99t use the search
facility of (guix ui) with relevance metrics.
> Back to the topic: I believe that Xapian is a huge win both for the
> shell and the future GUI :)
It could be, but we need to consider all the aspects of the story,
including the maintenance cost and overhead moved to =E2=80=98guix pull=E2=
=80=99. So
it=E2=80=99s not so much about =E2=80=9Cbeliefs=E2=80=9D at this point, but=
rather about
demonstrating what can be done, and I=E2=80=99m glad Arun is exploring that
space!
Ludo=E2=80=99.
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.
Received: (at 39258) by debbugs.gnu.org; 9 Mar 2020 07:50:46 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Mar 09 03:50:46 2020
Received: from localhost ([127.0.0.1]:49684 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1jBDBW-0003zJ-DB
for submit <at> debbugs.gnu.org; Mon, 09 Mar 2020 03:50:46 -0400
Received: from relay4-d.mail.gandi.net ([217.70.183.196]:55333)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <mail@HIDDEN>) id 1jBDBU-0003z5-2W
for 39258 <at> debbugs.gnu.org; Mon, 09 Mar 2020 03:50:44 -0400
X-Originating-IP: 92.169.129.147
Received: from mimimi (lfbn-idf2-1-1315-147.w92-169.abo.wanadoo.fr
[92.169.129.147]) (Authenticated sender: mail@HIDDEN)
by relay4-d.mail.gandi.net (Postfix) with ESMTPSA id A7279E0004;
Mon, 9 Mar 2020 07:50:36 +0000 (UTC)
From: Pierre Neidhardt <mail@HIDDEN>
To: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>, Arun Isaac
<arunisaac@HIDDEN>
Subject: Re: [PATCH v2 0/3] Xapian for Guix package search
In-Reply-To: <875zffcc87.fsf@HIDDEN>
References: <20200307133116.11443-1-arunisaac@HIDDEN>
<87sgijgb1v.fsf@HIDDEN> <cu7tv2zcj9l.fsf@HIDDEN>
<875zffcc87.fsf@HIDDEN>
Date: Mon, 09 Mar 2020 08:50:35 +0100
Message-ID: <877dzuvues.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="=-=-=";
micalg=pgp-sha256; protocol="application/pgp-signature"
X-Spam-Score: 1.8 (+)
X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org",
has NOT identified this incoming email as spam. The original
message has been attached to this so you can view it or label
similar future email. If you have any questions, see
the administrator of that system for details.
Content preview: Ludovic Courtès <ludo@HIDDEN> writes: > Yeah. I think we
need to take the whole user experience into account, > not just ‘guix search’.
‘guix pull’ already feels very slow, and it’s a > fairly common operation.
Conversely, ‘gu [...]
Content analysis details: (1.8 points, 10.0 required)
pts rule name description
---- ---------------------- --------------------------------------------------
0.0 URIBL_BLOCKED ADMINISTRATOR NOTICE: The query to URIBL was
blocked. See
http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block
for more information.
[URIs: gnu.org]
-0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at https://www.dnswl.org/,
low trust
[217.70.183.196 listed in list.dnswl.org]
-0.0 SPF_PASS SPF: sender matches SPF record
-0.0 RCVD_IN_MSPIKE_H3 RBL: Good reputation (+3)
[217.70.183.196 listed in wl.mailspike.net]
0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record
2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs
[URI: ambrevar.xyz (xyz)]
0.5 FROM_SUSPICIOUS_NTLD From abused NTLD
-0.0 RCVD_IN_MSPIKE_WL Mailspike good senders
X-Debbugs-Envelope-To: 39258
Cc: 39258 <at> debbugs.gnu.org, zimon.toutoune@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: 1.8 (+)
X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org",
has NOT identified this incoming email as spam. The original
message has been attached to this so you can view it or label
similar future email. If you have any questions, see
the administrator of that system for details.
Content preview: Ludovic Courtès <ludo@HIDDEN> writes: > Yeah. I think we
need to take the whole user experience into account, > not just ‘guix search’.
‘guix pull’ already feels very slow, and it’s a > fairly common operation.
Conversely, ‘gu [...]
Content analysis details: (1.8 points, 10.0 required)
pts rule name description
---- ---------------------- --------------------------------------------------
0.0 URIBL_BLOCKED ADMINISTRATOR NOTICE: The query to URIBL was
blocked. See
http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block
for more information.
[URIs: ambrevar.xyz]
-0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at https://www.dnswl.org/,
low trust
[217.70.183.196 listed in list.dnswl.org]
-0.0 RCVD_IN_MSPIKE_H3 RBL: Good reputation (+3)
[217.70.183.196 listed in wl.mailspike.net]
-0.0 SPF_PASS SPF: sender matches SPF record
0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record
2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs
[URI: ambrevar.xyz (xyz)]
-1.0 MAILING_LIST_MULTI Multiple indicators imply a widely-seen list
manager
0.5 FROM_SUSPICIOUS_NTLD From abused NTLD
1.0 BULK_RE_SUSP_NTLD Precedence bulk and RE: from a suspicious TLD
-0.0 RCVD_IN_MSPIKE_WL Mailspike good senders
--=-=-=
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Ludovic Court=C3=A8s <ludo@HIDDEN> writes:
> Yeah. I think we need to take the whole user experience into account,
> not just =E2=80=98guix search=E2=80=99. =E2=80=98guix pull=E2=80=99 alre=
ady feels very slow, and it=E2=80=99s a
> fairly common operation. Conversely, =E2=80=98guix search=E2=80=99 takes=
roughly
> between 0.5 and 2 seconds and is an uncommon operation on a =E2=80=9Cslow=
path=E2=80=9D
> (in the sense that when you=E2=80=99re searching for software, you=E2=80=
=99ll probably
> have to spend more than a couple of seconds to find what you=E2=80=99re l=
ooking
> for.)
I think I disagree with "guix search" being an uncommon operation and a
slow path.
=2D The slowness of `guix search' (and the awkwardness of recutils) is
maybe what makes it uncommon: users refrain from using it because it's
too impractical.
=2D Searches are typically refined, i.e. you run a search multiple times
by precising the terms, so in that sense I believe `guix search` is a
very common operation. Or should be.
Anyways, one of the key issues here is the inherent limitation of the
shell interface that does not allow us to directly and contextually
process the output of a command (at least not without rerunning it).
This issue can only be tackled with a GUI: there the user would be able
to interactively act with the result of the search, without having to
re-run the search.
Concretely, the GUI search would only return the package name, version
and synopses. No need for the Texinfo / recutils juggling.
Then the user would select the packages of interest to display more
details. This allows us to query the full details just-in-time.
Back to the topic: I believe that Xapian is a huge win both for the
shell and the future GUI :)
=2D-=20
Pierre Neidhardt
https://ambrevar.xyz/
--=-=-=
Content-Type: application/pgp-signature; name="signature.asc"
-----BEGIN PGP SIGNATURE-----
iQEzBAEBCAAdFiEEUPM+LlsMPZAEJKvom9z0l6S7zH8FAl5l9UsACgkQm9z0l6S7
zH8LzQf/W/eTpJMUphwkd4KFmmhNeUQeXk4K07i+1b2zm2BipqYVN2DOGMdFfAzO
MiVg/G0fNJQ/AMAr7mSjJnoSP0OPQ3o25/FBu2O5wLcw/8CNRUOL85tfEq/4iBlS
vm+J33eiJZBGXDzRTxXV2wY/tE0pL7Jc98vy/2d5yEBP2yc87gr2WgEUO9AwqyV9
xSCmu+/ZNI381zT3hWGfR2FkctLpXHnWt6roHcIiIbuFiVxhLkQ8/VYcxAkB+jNK
KJEfohyQoXPuefeWCHQLqTuk5XakAtXpVV7NChq2jf3Yl6676A53VBddGSj7ctoo
cyUL78LbDWheHbHvT/8n4aXynYt1bQ==
=M5Cf
-----END PGP SIGNATURE-----
--=-=-=--
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.
Received: (at 39258) by debbugs.gnu.org; 9 Mar 2020 07:42:27 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Mar 09 03:42:27 2020
Received: from localhost ([127.0.0.1]:49680 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1jBD3T-0003nR-GB
for submit <at> debbugs.gnu.org; Mon, 09 Mar 2020 03:42:27 -0400
Received: from relay1-d.mail.gandi.net ([217.70.183.193]:22953)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <mail@HIDDEN>) id 1jBD3R-0003nE-FS
for 39258 <at> debbugs.gnu.org; Mon, 09 Mar 2020 03:42:26 -0400
X-Originating-IP: 92.169.129.147
Received: from mimimi (lfbn-idf2-1-1315-147.w92-169.abo.wanadoo.fr
[92.169.129.147]) (Authenticated sender: mail@HIDDEN)
by relay1-d.mail.gandi.net (Postfix) with ESMTPSA id CD14E240007;
Mon, 9 Mar 2020 07:42:17 +0000 (UTC)
From: Pierre Neidhardt <mail@HIDDEN>
To: Arun Isaac <arunisaac@HIDDEN>, Ludovic =?utf-8?Q?Court=C3=A8?=
=?utf-8?Q?s?= <ludo@HIDDEN>
Subject: Re: [PATCH v2 0/3] Xapian for Guix package search
In-Reply-To: <cu7imjed22r.fsf@HIDDEN>
References: <20200307133116.11443-1-arunisaac@HIDDEN>
<87sgijgb1v.fsf@HIDDEN> <cu7tv2zcj9l.fsf@HIDDEN>
<875zffcc87.fsf@HIDDEN> <cu7imjed22r.fsf@HIDDEN>
Date: Mon, 09 Mar 2020 08:42:17 +0100
Message-ID: <87a74qvusm.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="=-=-=";
micalg=pgp-sha256; protocol="application/pgp-signature"
X-Spam-Score: 1.8 (+)
X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org",
has NOT identified this incoming email as spam. The original
message has been attached to this so you can view it or label
similar future email. If you have any questions, see
the administrator of that system for details.
Content preview: Arun Isaac <arunisaac@HIDDEN> writes: >> I’m not
sufficiently familiar with Xapian’s query language. The >> examples I had
in mind were: >> It’s not so much about regexps than it is about selecting
individual >> fields. > > I have t [...]
Content analysis details: (1.8 points, 10.0 required)
pts rule name description
---- ---------------------- --------------------------------------------------
0.0 URIBL_BLOCKED ADMINISTRATOR NOTICE: The query to URIBL was
blocked. See
http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block
for more information.
[URIs: systemreboot.net]
-0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at https://www.dnswl.org/,
low trust
[217.70.183.193 listed in list.dnswl.org]
-0.0 SPF_PASS SPF: sender matches SPF record
-0.0 RCVD_IN_MSPIKE_H3 RBL: Good reputation (+3)
[217.70.183.193 listed in wl.mailspike.net]
0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record
2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs
[URI: ambrevar.xyz (xyz)]
0.5 FROM_SUSPICIOUS_NTLD From abused NTLD
-0.0 RCVD_IN_MSPIKE_WL Mailspike good senders
X-Debbugs-Envelope-To: 39258
Cc: 39258 <at> debbugs.gnu.org, zimon.toutoune@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: 1.8 (+)
X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org",
has NOT identified this incoming email as spam. The original
message has been attached to this so you can view it or label
similar future email. If you have any questions, see
the administrator of that system for details.
Content preview: Arun Isaac <arunisaac@HIDDEN> writes: >> I’m not
sufficiently familiar with Xapian’s query language. The >> examples I had
in mind were: >> It’s not so much about regexps than it is about selecting
individual >> fields. > > I have t [...]
Content analysis details: (1.8 points, 10.0 required)
pts rule name description
---- ---------------------- --------------------------------------------------
0.0 URIBL_BLOCKED ADMINISTRATOR NOTICE: The query to URIBL was
blocked. See
http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block
for more information.
[URIs: ambrevar.xyz]
-0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at https://www.dnswl.org/,
low trust
[217.70.183.193 listed in list.dnswl.org]
-0.0 RCVD_IN_MSPIKE_H3 RBL: Good reputation (+3)
[217.70.183.193 listed in wl.mailspike.net]
-0.0 SPF_PASS SPF: sender matches SPF record
0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record
2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs
[URI: ambrevar.xyz (xyz)]
-1.0 MAILING_LIST_MULTI Multiple indicators imply a widely-seen list
manager
0.5 FROM_SUSPICIOUS_NTLD From abused NTLD
1.0 BULK_RE_SUSP_NTLD Precedence bulk and RE: from a suspicious TLD
-0.0 RCVD_IN_MSPIKE_WL Mailspike good senders
--=-=-=
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Arun Isaac <arunisaac@HIDDEN> writes:
>> I=E2=80=99m not sufficiently familiar with Xapian=E2=80=99s query langua=
ge. The
>> examples I had in mind were:
>> It=E2=80=99s not so much about regexps than it is about selecting indivi=
dual
>> fields.
>
> I have totally not tested this, but I imagine that equivalent Xapian
> queries might look something like:
>
>> guix search | recsel -p name -e 'license ~ "LGPL 3"'
>
> guix search license:LGPL3
>
>> guix search crypto library | \
>> recsel -e '! (name ~ "^(ghc|perl|python|ruby)")' -p name,synopsis
>
> guix search crypto library AND (NOT ghc) AND (NOT perl) AND (NOT python)
> AND (NOT ruby)
Indeed, if you look at the notmuch-search-terms man page, you'll see
that you can select fields.
In my opinion, the recsel format is fully superseded by Xapian.
=2D-=20
Pierre Neidhardt
https://ambrevar.xyz/
--=-=-=
Content-Type: application/pgp-signature; name="signature.asc"
-----BEGIN PGP SIGNATURE-----
iQEzBAEBCAAdFiEEUPM+LlsMPZAEJKvom9z0l6S7zH8FAl5l81kACgkQm9z0l6S7
zH8T8Af9HItQX5qR3TAF/dqk4XxKqjZFl+hq17yIpC6ZTIa12RA8GnKhDhLgCxi9
IyCbRxbCmeqLSRyBictizH/pnnRwKVNoZdgDtLvxiL6AO750AWy7I/Ly4j4KJ+/Y
9LLK9U6eOXiCCtggfutIxyYApeQYUvLjzvcjtyNooUfXzEa6Od962/BZaOAfveti
07W/IGCi1PIPIZJTOTSF0kbiiqvs91Phty+a1f1Me4ZY5PuJxhETDJFUtkZ8j+zj
0oym8CKCbWLAgIsb0v/8Vv/pWavTWJqxxjALLa3mqVwn1JHeufbLN6YRf1bIOrO5
6AWkufRWzP+KxdkpwDUMq8zklWkSpg==
=z+V4
-----END PGP SIGNATURE-----
--=-=-=--
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 8 Mar 2020 20:41:19 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Sun Mar 08 16:41:19 2020 Received: from localhost ([127.0.0.1]:49349 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1jB2jf-00050u-FA for submit <at> debbugs.gnu.org; Sun, 08 Mar 2020 16:41:19 -0400 Received: from mugam.systemreboot.net ([139.59.75.54]:57882) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <arunisaac@HIDDEN>) id 1jB2jd-00050m-I4 for 39258 <at> debbugs.gnu.org; Sun, 08 Mar 2020 16:41:18 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=systemreboot.net; s=default; h=Content-Type:MIME-Version:Message-ID:Date: References:In-Reply-To:Subject:Cc:To:From:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=us+jEC1u5B3RtJ9TGP6slO243/zUVp2TImIjAgm/zCg=; b=qpdOU7pJ+x4K6HPvMU6xI6zuN q2j75rZaKUIv1BQXP8/R5Imc6KXySWsyo5/NA+c+DHfRV4rAUsGqEqrJkSzHNMSCaIVmb3prMp0Zo LPObj/wtE2ys/WXe5A1Tz6TcW3nZdL4zeqNPqjfArUuvO8CVWSRW2c7NiMWxeieOgm394=; Received: from [192.168.2.1] (helo=steel) by systemreboot.net with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.93) (envelope-from <arunisaac@HIDDEN>) id 1jB2jU-002hSw-3k; Mon, 09 Mar 2020 02:11:08 +0530 From: Arun Isaac <arunisaac@HIDDEN> To: zimoun <zimon.toutoune@HIDDEN> Subject: Re: [PATCH v2 0/3] Xapian for Guix package search In-Reply-To: <CAJ3okZ0tDs9s2Ju+0oKX2uf-JuRdFHiSXSpQonfF1wrmJYKvaQ@HIDDEN> References: <20200307133116.11443-1-arunisaac@HIDDEN> <CAJ3okZ0tDs9s2Ju+0oKX2uf-JuRdFHiSXSpQonfF1wrmJYKvaQ@HIDDEN> Date: Mon, 09 Mar 2020 02:10:57 +0530 Message-ID: <cu7fteid1gm.fsf@HIDDEN> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 39258 Cc: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>, Pierre Neidhardt <mail@HIDDEN>, 39258 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain zimoun <zimon.toutoune@HIDDEN> writes: > I will be probably far from keyboard for a couple of days for medical > reasons* and I will not be able to look into this second set patches > soon. No problem, take care! :-) > *nothing to worry, even if I am currently typing that in an hospital, > just the bad consequence of sports. ;-) --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEf3MDQ/Lwnzx3v3nTLiXui2GAK7MFAl5lWFkACgkQLiXui2GA K7MigQgAk+jg+2ta+gYnG+Iubpd5GFdoOT0ndCBqtHn3gULZTOypW8qqc5HHz5K6 3WXzL5IAdW0bhpuvGQIsmqUXMKAKqiIPzOsjFfexRllAvXGmqUhORnCd7RG5w+TA dVOPhrB2/xOIi3Hs9RJwGWruf3715QOZ7qUCSYpuf/gBj0/htTIiMBDqjxk96RoS /GnYTB8IpLXM5gFU1JONvlAtEMUZQ0UHjHrEqTdiGNvg7PwygIB9nCu4A0ey7eoY y6a6s8eXcEyUdZ6M/WOFfc74HQBTrvqlr7FBSSyiR+LZQPrD6L4XNv8ohyRuVIG0 yMzQnApc4RckPIK5HYlkQlPsYZmhRg== =k6ET -----END PGP SIGNATURE----- --=-=-=--
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 8 Mar 2020 20:28:17 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Sun Mar 08 16:28:17 2020 Received: from localhost ([127.0.0.1]:49334 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1jB2X3-0004hA-2Z for submit <at> debbugs.gnu.org; Sun, 08 Mar 2020 16:28:17 -0400 Received: from mugam.systemreboot.net ([139.59.75.54]:57846) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <arunisaac@HIDDEN>) id 1jB2X0-0004gy-CW for 39258 <at> debbugs.gnu.org; Sun, 08 Mar 2020 16:28:16 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=systemreboot.net; s=default; h=Content-Type:MIME-Version:Message-ID:Date: References:In-Reply-To:Subject:Cc:To:From:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=ym4XHLmZ1wsNI3faKASmYZDvhUZCdqYfMdhIUoCxxVU=; b=lX0BoRE1EMjtuuOjkzoQy0HUR wi8Q9HW2V0bOZ3YGOMYcUCeS630N0uVhb4GdtQy0ENGnOqp/kjUgMc6IN60nyGKk/bdnS0XHaRwwJ S1kxVBNhtMgEFzhDQ5CTtNA6vokckQ6ND4DgOogyVtfl4AUWT6hI0BGbS1W5DCjtkmUzA=; Received: from [192.168.2.1] (helo=steel) by systemreboot.net with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.93) (envelope-from <arunisaac@HIDDEN>) id 1jB2Wg-002hQk-1f; Mon, 09 Mar 2020 01:57:54 +0530 From: Arun Isaac <arunisaac@HIDDEN> To: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN> Subject: Re: [PATCH v2 0/3] Xapian for Guix package search In-Reply-To: <875zffcc87.fsf@HIDDEN> References: <20200307133116.11443-1-arunisaac@HIDDEN> <87sgijgb1v.fsf@HIDDEN> <cu7tv2zcj9l.fsf@HIDDEN> <875zffcc87.fsf@HIDDEN> Date: Mon, 09 Mar 2020 01:57:40 +0530 Message-ID: <cu7imjed22r.fsf@HIDDEN> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 39258 Cc: mail@HIDDEN, 39258 <at> debbugs.gnu.org, zimon.toutoune@HIDDEN X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable >> This could be accomplished even with pre-rendering. Xapian provides >> "slots" to store arbitrary strings with a document. Instead of storing >> the pre-rendered document as a whole, we could store pre-rendered fields >> in separate slots. Then, during `guix search` time, we can assemble the >> result from these pre-rendered fields. > > I=E2=80=99m not sure I understand. The index wouldn=E2=80=99t store pre-= rendered > strings for every possible terminal width, right? No, it wouldn't. It would store a partially pre-rendered string, that is without fill-paragraph. We run fill-paragraph at `guix search` time to complete the rendering. > I think we need to take the whole user experience into account, not > just =E2=80=98guix search=E2=80=99. =E2=80=98guix pull=E2=80=99 already = feels very slow, and it=E2=80=99s a > fairly common operation. Conversely, =E2=80=98guix search=E2=80=99 takes= roughly > between 0.5 and 2 seconds and is an uncommon operation on a =E2=80=9Cslow > path=E2=80=9D (in the sense that when you=E2=80=99re searching for softwa= re, you=E2=80=99ll > probably have to spend more than a couple of seconds to find what > you=E2=80=99re looking for.) I agree we can't compromise too much on `guix pull` performance. > To me, adding 20=E2=80=9350 seconds on =E2=80=98guix pull=E2=80=99 would = be undesirable. :-/ Maybe I'm missing something here. guix pull takes around 40 minutes on my machine. In comparison to that, is another 20-50 seconds (roughly 1 minute) a big deal? How much time would it be acceptable to spend on building the Xapian index? Also, is it possible to somehow provide substitutes for the Xapian index so that the user does not have to actually build it locally during `guix pull` time? > I=E2=80=99m not sufficiently familiar with Xapian=E2=80=99s query languag= e. The > examples I had in mind were: > It=E2=80=99s not so much about regexps than it is about selecting individ= ual > fields. I have totally not tested this, but I imagine that equivalent Xapian queries might look something like: > guix search | recsel -p name -e 'license ~ "LGPL 3"' guix search license:LGPL3 > guix search crypto library | \ > recsel -e '! (name ~ "^(ghc|perl|python|ruby)")' -p name,synopsis guix search crypto library AND (NOT ghc) AND (NOT perl) AND (NOT python) AND (NOT ruby) > What I meant was that we could use (statprof) to see whether/how Texinfo > rendering/parsing can be optimized. Oh, ok. I'll try this if we decide not to pre-render. --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEf3MDQ/Lwnzx3v3nTLiXui2GAK7MFAl5lVTwACgkQLiXui2GA K7MufAf+PuuYkEHruEk5UrGI/8ofRfpKs0DL5GUdhc09xLrYI7GnBu3IkOB/2ock UfNDd1Vkgs0UKPs6rbkxer/IJ8Vf/Eat5B7r+RlDoUxByICgj/IRhgBYz42IUrfS oZuM5XxeBS158WvNvQcFfYgrXrkUl/WpxyoWZP64xEZBhgSEJYbV783Mf6trYiOY TLwuRi4vzZQru1RAHu8L+GoAUE5SGTf6jLb5xPZ5cYdNXwlRq2zSZcm8+UWv7RCR v9MVlydIsOtb++TSI/n/xwCL9eeSl8Sz5VTphVpHrZ74VKKakI0DEiubg3bTzxR9 WD9peO5+8BTwuI3kI/hmmKDx3vmQQw== =XhPo -----END PGP SIGNATURE----- --=-=-=--
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 8 Mar 2020 20:27:50 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Sun Mar 08 16:27:50 2020 Received: from localhost ([127.0.0.1]:49330 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1jB2Wb-0004g8-Ro for submit <at> debbugs.gnu.org; Sun, 08 Mar 2020 16:27:49 -0400 Received: from mail-qt1-f196.google.com ([209.85.160.196]:37772) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <zimon.toutoune@HIDDEN>) id 1jB2WZ-0004fv-DO for 39258 <at> debbugs.gnu.org; Sun, 08 Mar 2020 16:27:47 -0400 Received: by mail-qt1-f196.google.com with SMTP id l20so3965233qtp.4 for <39258 <at> debbugs.gnu.org>; Sun, 08 Mar 2020 13:27:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=9QboejUljpxUEVgd31eq8U3k/6rrjoe6RyAS4AcBxys=; b=ipnD8xVU4hK94uCWc62diuIlAqaL+xkI56uMn4AtSqzy0kxSp+G+QwWp86Hq2uVU2H gc3sxlee6YWZ6Yph8bET/mBXwu63nH0gflBm9r0yHdrSm2USfQOkPzHgGxP2MR5GBkv5 N5LIoRpyoEhsY4tytcATvRkpnp1wyo7BXp2RdC8nv+YVQImZ3KdeWpSXIXCJl58Mvm0R OKXPDZw8yep0i7eSmYTPbFfMpDY2mn1WyGUYGWyFgOny23BQBo2EXq9XYBNC8k/IY9K0 8ZtGZTmLzoaJZjZiIJgvu5PrvD1a+7CxpYd4QPUVVvgjyxQxmzK3Qvtrm1qaJ9CvmbfJ 7Pmg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=9QboejUljpxUEVgd31eq8U3k/6rrjoe6RyAS4AcBxys=; b=LugU/qxnvN21+E0t5crxcOb5sXNnjLXoPYej2gPq6h08CzygnQyYP77wUzEfUt0q2k gHGrgluuTT9vo25Tk/XoThMXgt7ovbqEmjfIp0z6mgOwp9k/audSCU3MAVuGINbcdpeZ puJR0zbtm7q5vzKLvHF2m6QiWCwgegTcs/kffK+bq63oZS6MMzS6u2ITb72JZE9LYi3y J9DEjP92TzhY+9LtMjsaBBXJ/ynDzbC0c/JSSyQzLlwe2mIcG6/M387xcahOoVvk7rIH 90EkT1jZyoeSbLZwgSdbjbl7ywke1WJlFt4YzJLSK5k8TLVwb9jQ6xhNdV6w6TbGmGg4 zjXQ== X-Gm-Message-State: ANhLgQ1TDyBgndLQmFNphlvmgAOy6TQUSIiNqL/PKppUSWnAsl7EWwoO 9WqLV9GuLkGsk9EqBvhfZ2C5KQf6urRqBTFF1SFx4Yso X-Google-Smtp-Source: ADFU+vv+HtcBH0TqLISMWWBHSH9yGu5Blq68bQ/+i03kB4iijs5IMMhWkZpZAoqHH6qnMhTDVsK7jf1HSpVZdAp5tPc= X-Received: by 2002:ac8:2fd4:: with SMTP id m20mr11610125qta.313.1583699261898; Sun, 08 Mar 2020 13:27:41 -0700 (PDT) MIME-Version: 1.0 References: <20200307133116.11443-1-arunisaac@HIDDEN> In-Reply-To: <20200307133116.11443-1-arunisaac@HIDDEN> From: zimoun <zimon.toutoune@HIDDEN> Date: Sun, 8 Mar 2020 21:27:28 +0100 Message-ID: <CAJ3okZ0tDs9s2Ju+0oKX2uf-JuRdFHiSXSpQonfF1wrmJYKvaQ@HIDDEN> Subject: Re: [PATCH v2 0/3] Xapian for Guix package search To: Arun Isaac <arunisaac@HIDDEN> Content-Type: text/plain; charset="UTF-8" X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 39258 Cc: =?UTF-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>, Pierre Neidhardt <mail@HIDDEN>, 39258 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) Hi Arun, Thank you for that. I will be probably far from keyboard for a couple of days for medical reasons* and I will not be able to look into this second set patches soon. Cheers, simon *nothing to worry, even if I am currently typing that in an hospital, just the bad consequence of sports. ;-)
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.
Received: (at 39258) by debbugs.gnu.org; 8 Mar 2020 11:33:55 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Mar 08 07:33:55 2020
Received: from localhost ([127.0.0.1]:47973 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1jAuBv-0006Vd-2W
for submit <at> debbugs.gnu.org; Sun, 08 Mar 2020 07:33:55 -0400
Received: from eggs.gnu.org ([209.51.188.92]:54547)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <ludo@HIDDEN>) id 1jAuBt-0006VP-8m
for 39258 <at> debbugs.gnu.org; Sun, 08 Mar 2020 07:33:53 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:52245)
by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <ludo@HIDDEN>)
id 1jAuBm-0007i4-QB; Sun, 08 Mar 2020 07:33:46 -0400
Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=35386 helo=ribbon)
by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
(Exim 4.82) (envelope-from <ludo@HIDDEN>)
id 1jAuBm-0000uw-4B; Sun, 08 Mar 2020 07:33:46 -0400
From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>
To: Arun Isaac <arunisaac@HIDDEN>
Subject: Re: [PATCH v2 0/3] Xapian for Guix package search
References: <20200307133116.11443-1-arunisaac@HIDDEN>
<87sgijgb1v.fsf@HIDDEN> <cu7tv2zcj9l.fsf@HIDDEN>
X-URL: http://www.fdn.fr/~lcourtes/
X-Revolutionary-Date: 19 =?utf-8?Q?Vent=C3=B4se?= an 228 de la =?utf-8?Q?R?=
=?utf-8?Q?=C3=A9volution?=
X-PGP-Key-ID: 0x090B11993D9AEBB5
X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc
X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5
X-OS: x86_64-pc-linux-gnu
Date: Sun, 08 Mar 2020 12:33:44 +0100
In-Reply-To: <cu7tv2zcj9l.fsf@HIDDEN> (Arun Isaac's message of "Sun,
08 Mar 2020 14:31:42 +0530")
Message-ID: <875zffcc87.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Spam-Score: -0.7 (/)
X-Debbugs-Envelope-To: 39258
Cc: mail@HIDDEN, 39258 <at> debbugs.gnu.org, zimon.toutoune@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)
Hi,
Arun Isaac <arunisaac@HIDDEN> skribis:
>>> It turns out that most of the time is spent in printing and texinfo
>>> rendering of the search results.
>
> Also, when we put all package metadata into the Xapian index, we don't
> have to look up any of the package variables in (gnu packages *) during
> `guix search` time. This also contributes substantially to the speedup.
Yup.
>> In general, pre-rendering doesn=E2=80=99t seem practical to me: the outp=
ut of
>> =E2=80=98guix search=E2=80=99 is locale-dependent (it speaks the user=E2=
=80=99s language) and
>
> Note that we already need to index package synopses and descriptions in
> all languages. I still haven't implemented this, though.
Oh, right. Tricky!
>> adjusts to the terminal width (well, this is temporarily broken on
>> Guile 3.0.0, but see =E2=80=98%text-width=E2=80=99 in (guix ui)).
>
> This could be accomplished even with pre-rendering. Xapian provides
> "slots" to store arbitrary strings with a document. Instead of storing
> the pre-rendered document as a whole, we could store pre-rendered fields
> in separate slots. Then, during `guix search` time, we can assemble the
> result from these pre-rendered fields.
I=E2=80=99m not sure I understand. The index wouldn=E2=80=99t store pre-re=
ndered
strings for every possible terminal width, right?
>> Also, if the 12K+ descriptions need to be rendered at the time the user
>> runs =E2=80=98guix pull=E2=80=99, the experience may not be great, becau=
se it could take
>> a bit of time.
>
> This is a problem, but I would see it as a necessary "compilation"
> step. :-P In fact, this whole patchset speeds up `guix search` by doing
> part of the work of `guix search` ahead of time. So, some such cost is
> unavoidable.
Yeah. I think we need to take the whole user experience into account,
not just =E2=80=98guix search=E2=80=99. =E2=80=98guix pull=E2=80=99 alread=
y feels very slow, and it=E2=80=99s a
fairly common operation. Conversely, =E2=80=98guix search=E2=80=99 takes r=
oughly
between 0.5 and 2 seconds and is an uncommon operation on a =E2=80=9Cslow p=
ath=E2=80=9D
(in the sense that when you=E2=80=99re searching for software, you=E2=80=99=
ll probably
have to spend more than a couple of seconds to find what you=E2=80=99re loo=
king
for.)
>> What I like about the recutils format in this context is that it=E2=80=
=99s both
>> human- and machine-readable. The examples in the manual show how it can
>> be useful to select the information displayed or to refine the search
>> (info "(guix) Invoking guix package").
>
> Xapian's query language is much more natural (as in natural language)
> than the regexp based techniques we need to use with recutils. I have
> hardly ever used the regexp based search and I suspect many others
> haven't either. Also, refining the search query should be easier to do
> with Xapian. We could even use Xapian's query expansion feature to
> suggest improved queries to the user.
I=E2=80=99m not sufficiently familiar with Xapian=E2=80=99s query language.=
The
examples I had in mind were:
guix search malloc | recsel -p name,version,relevance
guix search | recsel -p name -e 'license ~ "LGPL 3"'
guix search crypto library | \
recsel -e '! (name ~ "^(ghc|perl|python|ruby)")' -p name,synopsis
It=E2=80=99s not so much about regexps than it is about selecting individual
fields.
>> Were you able to measure the cost of rendering specifically?
>
> generate-package-search-index takes around 50 seconds. If I modify
> generate-package-search-index to not pre-render but simply store the
> package description alone, it takes around 20 seconds. That gives us a
> rough idea of the cost of pre-rendering.
To me, adding 20=E2=80=9350 seconds on =E2=80=98guix pull=E2=80=99 would be=
undesirable. :-/
>> I think we should look at a profile of =E2=80=98package->recutils=E2=80=
=99, there=E2=80=99s
>> probably room for improvement there.
>
> On quick inspection, most of the time in package->recutils is spent in
> texinfo rendering the description. Unless we use the simplified search
> results format as discussed above, we cannot avoid it.
What I meant was that we could use (statprof) to see whether/how Texinfo
rendering/parsing can be optimized.
Thanks,
Ludo=E2=80=99.
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 8 Mar 2020 09:02:08 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Sun Mar 08 05:02:08 2020 Received: from localhost ([127.0.0.1]:47883 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1jArp1-0000Gy-Um for submit <at> debbugs.gnu.org; Sun, 08 Mar 2020 05:02:08 -0400 Received: from mugam.systemreboot.net ([139.59.75.54]:55482) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <arunisaac@HIDDEN>) id 1jAroy-0000D9-Mw for 39258 <at> debbugs.gnu.org; Sun, 08 Mar 2020 05:02:07 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=systemreboot.net; s=default; h=Content-Type:MIME-Version:Message-ID:Date: References:In-Reply-To:Subject:Cc:To:From:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=4/wT/fafeenlTNLNNGegj//mrkuKxN6QDSDIWcy+GE8=; b=hZfB10G7R6YUgpAVczEoAsuVj Ec26IdseUmbFztjkIeVN4hlzDUi8j/s48KCFL72M1eds10h2DChT14Tw5a8x9+9unQWot5fWXdm+i PKT/s23nyX/pF5ipa3Op5jfSsbtAhb/CiR1qzXxGocSLjPmRGb/mhxirwYFaWSd6vTwWk=; Received: from [192.168.2.1] (helo=steel) by systemreboot.net with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.93) (envelope-from <arunisaac@HIDDEN>) id 1jAroo-002fZg-Rs; Sun, 08 Mar 2020 14:31:55 +0530 From: Arun Isaac <arunisaac@HIDDEN> To: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN> Subject: Re: [PATCH v2 0/3] Xapian for Guix package search In-Reply-To: <87sgijgb1v.fsf@HIDDEN> References: <20200307133116.11443-1-arunisaac@HIDDEN> <87sgijgb1v.fsf@HIDDEN> Date: Sun, 08 Mar 2020 14:31:42 +0530 Message-ID: <cu7tv2zcj9l.fsf@HIDDEN> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 39258 Cc: mail@HIDDEN, 39258 <at> debbugs.gnu.org, zimon.toutoune@HIDDEN X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable >> It turns out that most of the time is spent in printing and texinfo >> rendering of the search results. Also, when we put all package metadata into the Xapian index, we don't have to look up any of the package variables in (gnu packages *) during `guix search` time. This also contributes substantially to the speedup. > In general, pre-rendering doesn=E2=80=99t seem practical to me: the outpu= t of > =E2=80=98guix search=E2=80=99 is locale-dependent (it speaks the user=E2= =80=99s language) and Note that we already need to index package synopses and descriptions in all languages. I still haven't implemented this, though. > adjusts to the terminal width (well, this is temporarily broken on > Guile 3.0.0, but see =E2=80=98%text-width=E2=80=99 in (guix ui)). This could be accomplished even with pre-rendering. Xapian provides "slots" to store arbitrary strings with a document. Instead of storing the pre-rendered document as a whole, we could store pre-rendered fields in separate slots. Then, during `guix search` time, we can assemble the result from these pre-rendered fields. > Also, if the 12K+ descriptions need to be rendered at the time the user > runs =E2=80=98guix pull=E2=80=99, the experience may not be great, becaus= e it could take > a bit of time. This is a problem, but I would see it as a necessary "compilation" step. :-P In fact, this whole patchset speeds up `guix search` by doing part of the work of `guix search` ahead of time. So, some such cost is unavoidable. > What I like about the recutils format in this context is that it=E2=80=99= s both > human- and machine-readable. The examples in the manual show how it can > be useful to select the information displayed or to refine the search > (info "(guix) Invoking guix package"). Xapian's query language is much more natural (as in natural language) than the regexp based techniques we need to use with recutils. I have hardly ever used the regexp based search and I suspect many others haven't either. Also, refining the search query should be easier to do with Xapian. We could even use Xapian's query expansion feature to suggest improved queries to the user. That said, if we want the recutils format, we can still keep it in a simplified form like so. name: inkscape version: 0.92.4 synopsis: Vector graphics editor name: inklingreader version: 0.8 synopsis: Wacom Inkling skecth format conversion and manipulation > Also: I=E2=80=99d recommend tackling one thing at a time. :-) I totally agree, but I'm tempted to say that pre-rendering would be a lot cheaper with the simplified form of search results. :-) > Were you able to measure the cost of rendering specifically? generate-package-search-index takes around 50 seconds. If I modify generate-package-search-index to not pre-render but simply store the package description alone, it takes around 20 seconds. That gives us a rough idea of the cost of pre-rendering. > I think we should look at a profile of =E2=80=98package->recutils=E2=80= =99, there=E2=80=99s > probably room for improvement there. On quick inspection, most of the time in package->recutils is spent in texinfo rendering the description. Unless we use the simplified search results format as discussed above, we cannot avoid it. --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEf3MDQ/Lwnzx3v3nTLiXui2GAK7MFAl5ktHYACgkQLiXui2GA K7NqqQgAnPZx1ilBTQfl8kAoHTyQrdcdhL//PI4LHFMtfYqi+s+AnYvu10jNVM9K lPX2AL3hDRcDYSJL1r2KQ+uicc/FZwY9qWVPg5axflohUnwa6pzn7LDnJ3U5ClU/ ON5uY7Vh6nixBjsIMwMqklRSWb2Lk/VEYTM9+HYUXCPgpvTzUz1XuJN5IP6YeXMo EIIMpOUVFz3Qxg3bZqPXSZ29gxAg+KdS3DhtK4zgmeFZTBeT+81Lld6rDy1q68Y3 kwyFKmiLSsg0XWRStK5tai7J0h7snN+/EM4SkORjOGBlEnsUPQSYfeRnAUXy2f5Y kxw+OnaWgsrF+UtGYrWAg4+SLo0ocw== =rzzL -----END PGP SIGNATURE----- --=-=-=--
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 7 Mar 2020 20:33:27 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Sat Mar 07 15:33:27 2020 Received: from localhost ([127.0.0.1]:47619 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1jAg8V-0000Be-DN for submit <at> debbugs.gnu.org; Sat, 07 Mar 2020 15:33:27 -0500 Received: from eggs.gnu.org ([209.51.188.92]:33435) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <ludo@HIDDEN>) id 1jAg8T-0000BR-Ne for 39258 <at> debbugs.gnu.org; Sat, 07 Mar 2020 15:33:26 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:43337) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <ludo@HIDDEN>) id 1jAg8N-0004oG-7p; Sat, 07 Mar 2020 15:33:19 -0500 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=34126 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from <ludo@HIDDEN>) id 1jAg8M-0007Si-MR; Sat, 07 Mar 2020 15:33:19 -0500 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN> To: Arun Isaac <arunisaac@HIDDEN> Subject: Re: [PATCH v2 0/3] Xapian for Guix package search References: <20200307133116.11443-1-arunisaac@HIDDEN> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 18 =?utf-8?Q?Vent=C3=B4se?= an 228 de la =?utf-8?Q?R?= =?utf-8?Q?=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Sat, 07 Mar 2020 21:33:16 +0100 In-Reply-To: <20200307133116.11443-1-arunisaac@HIDDEN> (Arun Isaac's message of "Sat, 7 Mar 2020 19:01:13 +0530") Message-ID: <87sgijgb1v.fsf@HIDDEN> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 39258 Cc: mail@HIDDEN, 39258 <at> debbugs.gnu.org, zimon.toutoune@HIDDEN X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.7 (-) Hello, Arun Isaac <arunisaac@HIDDEN> skribis: > Here is the second iteration of my Xapian Guix package search patchset. I= have > found the reason the earlier patchset did not show significant speedup. It > turns out that most of the time is spent in printing and texinfo renderin= g of > the search results. So, in this patchset, I pre-render the search results > while building the Xapian index and stuff them into the Xapian database > itself. Therefore, during `guix search`, I just pull out the pre-rendered > search results and print it on the screen. This is much faster. See compa= rison > below. > > With a warm cache, > $ time guix search inkscape > > real 0m1.787s > user 0m1.745s > sys 0m0.111s > > $ time /tmp/test/bin/guix search inkscape > > real 0m0.199s > user 0m0.182s > sys 0m0.024s Nice! In general, pre-rendering doesn=E2=80=99t seem practical to me: the output = of =E2=80=98guix search=E2=80=99 is locale-dependent (it speaks the user=E2=80= =99s language) and adjusts to the terminal width (well, this is temporarily broken on Guile 3.0.0, but see =E2=80=98%text-width=E2=80=99 in (guix ui)). Also, if the 12K+ descriptions need to be rendered at the time the user runs =E2=80=98guix pull=E2=80=99, the experience may not be great, because = it could take a bit of time. WDYT? > Why not use a simpler package search results format like Arch Linux or De= bian > does? We could just display the package name, version and synopsis like s= o. > > inkscape 0.92.4 > Vector graphics editor > inklingreader 0.8 > Wacom Inkling sketch format conversion and manipulation > > Why do we need the entire recutils format? If the user is interested, the= y can > always use `guix package --show` to get the full recutils formatted > info. Having shorter search results will make everything even faster and = much > more readable. WDYT? What I like about the recutils format in this context is that it=E2=80=99s = both human- and machine-readable. The examples in the manual show how it can be useful to select the information displayed or to refine the search (info "(guix) Invoking guix package"). Also: I=E2=80=99d recommend tackling one thing at a time. :-) > Ludovic Court=C3=A8s <ludo@HIDDEN> writes: > >> Note that =E2=80=98guix search=E2=80=99 time is largely dominated by I/O. > > Yes, `guix search` is I/O intensive. That is why I expect Xapian to do be= tter > since it only needs to access matching packages not all packages. Also, t= he > Xapian index is fast at all times. It is not very dependent on a warm > filesystem cache. Yes, indeed. >> On my laptop, >> I get (first measurement is cold cache, second one is warm cache): >> >> --8<---------------cut here---------------start------------->8--- >> $ sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' >> $ time guix search foo >/dev/null >> >> real 0m2.631s >> user 0m1.134s >> sys 0m0.124s >> $ time guix search foo >/dev/null >> >> real 0m0.836s >> user 0m1.027s >> sys 0m0.053s >> --8<---------------cut here---------------end--------------->8--- >> >> It=E2=80=99s hard to do better on the warm cache case because at this le= vel, >> there may be other things to optimize having little to do with searching >> itself. >> >> Note that this is on an SSD; the cold-cache case must be worse on NFS or >> on a spinning disk, and there we could gain a lot. > > My laptop is quite old with a particularly slow HDD. Hence my motivation = to > improve guix search performance! Were you able to measure the cost of rendering specifically? Here=E2=80=99s what I see when I turn =E2=80=98package->recutils=E2=80=99 i= nto a no-op: --8<---------------cut here---------------start------------->8--- $ sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' $ time ./pre-inst-env guix search foo=20 real 0m1.617s user 0m0.812s sys 0m0.094s $ time ./pre-inst-env guix search foo=20 real 0m0.595s user 0m0.747s sys 0m0.043s --8<---------------cut here---------------end--------------->8--- To compare with: --8<---------------cut here---------------start------------->8--- $ time ./pre-inst-env guix search foo >/dev/null real 0m0.829s user 0m1.026s sys 0m0.046s --8<---------------cut here---------------end--------------->8--- I think we should look at a profile of =E2=80=98package->recutils=E2=80=99,= there=E2=80=99s probably room for improvement there. Thoughts? Ludo=E2=80=99.
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.
Received: (at 39258) by debbugs.gnu.org; 7 Mar 2020 13:31:43 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Mar 07 08:31:43 2020
Received: from localhost ([127.0.0.1]:46408 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1jAZYM-0002rX-Jx
for submit <at> debbugs.gnu.org; Sat, 07 Mar 2020 08:31:43 -0500
Received: from mugam.systemreboot.net ([139.59.75.54]:51818)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <arunisaac@HIDDEN>) id 1jAZYI-0002qp-EU
for 39258 <at> debbugs.gnu.org; Sat, 07 Mar 2020 08:31:39 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed;
d=systemreboot.net; s=default; h=Content-Transfer-Encoding:Content-Type:
MIME-Version:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To:Content-ID:
Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc
:Resent-Message-ID:In-Reply-To:References:List-Id:List-Help:List-Unsubscribe:
List-Subscribe:List-Post:List-Owner:List-Archive;
bh=rXDwP1qkIzy8aLmjFVgOLQDhdvzvRhHYevj0cfTSqAA=; b=VSdMuAa1HQnSgBIFhc9jeuSOLz
mZrodOc8CPh+4paiJoOlJEMjwiMY5ZswTCvcBU+BfrSNaaQYVGlmwkAtKLkMg1QqT33Llp97Xbmcc
04LdQHPqdFeaLD4MEtxKg+V5tRFsJpSjCn8EY8CFVRZVJJCf9ZbhcZm+m9s6redY9Uhw=;
Received: from [192.168.2.1] (helo=steel.lan)
by systemreboot.net with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384
(Exim 4.93) (envelope-from <arunisaac@HIDDEN>)
id 1jAZY7-002bX3-8r; Sat, 07 Mar 2020 19:01:27 +0530
From: Arun Isaac <arunisaac@HIDDEN>
To: 39258 <at> debbugs.gnu.org
Subject: [PATCH v2 0/3] Xapian for Guix package search
Date: Sat, 7 Mar 2020 19:01:13 +0530
Message-Id: <20200307133116.11443-1-arunisaac@HIDDEN>
X-Mailer: git-send-email 2.25.1
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-Spam-Score: 2.0 (++)
X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org",
has NOT identified this incoming email as spam. The original
message has been attached to this so you can view it or label
similar future email. If you have any questions, see
the administrator of that system for details.
Content preview: Hi, Here is the second iteration of my Xapian Guix package
search patchset. I have found the reason the earlier patchset did not show
significant speedup. It turns out that most of the time is spent in pr [...]
Content analysis details: (2.0 points, 10.0 required)
pts rule name description
---- ---------------------- --------------------------------------------------
0.0 URIBL_BLOCKED ADMINISTRATOR NOTICE: The query to URIBL was
blocked. See
http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block
for more information. [URIs: gnu.org]
2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs
[URI: ambrevar.xyz (xyz)]
0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record
-0.0 SPF_PASS SPF: sender matches SPF record
X-Debbugs-Envelope-To: 39258
Cc: Arun Isaac <arunisaac@HIDDEN>, mail@HIDDEN, ludo@HIDDEN,
zimon.toutoune@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: 1.0 (+)
Hi,
Here is the second iteration of my Xapian Guix package search patchset. I have
found the reason the earlier patchset did not show significant speedup. It
turns out that most of the time is spent in printing and texinfo rendering of
the search results. So, in this patchset, I pre-render the search results
while building the Xapian index and stuff them into the Xapian database
itself. Therefore, during `guix search`, I just pull out the pre-rendered
search results and print it on the screen. This is much faster. See comparison
below.
--8<---------------cut here---------------start------------->8---
With a warm cache,
$ time guix search inkscape
real 0m1.787s
user 0m1.745s
sys 0m0.111s
--8<---------------cut here---------------end--------------->8---
--8<---------------cut here---------------start------------->8---
$ time /tmp/test/bin/guix search inkscape
real 0m0.199s
user 0m0.182s
sys 0m0.024s
--8<---------------cut here---------------end--------------->8---
If most of the speedup comes from pre-rendering the results, it might seem
that the Xapian search is not so useful. We might as well have stuffed the
pre-rendered search results into the existing package cache generated by
generate-package-cache, or so it might seem. But, there are the following
arguments in favor of Xapian.
- The package cache would grow in size, and lookup would be slowed down
because we need to load the entire cache into memory. Xapian, on the other
hand, need only look up the specific packages that match the search query.
- Xapian can provide superior search results due to it stemming and language
models.
- Xapian can provide spelling correction and query expansion -- that is,
suggest search terms to improve search results. Note that I haven't
implemented this yet and is out of scope in this patchset.
* Simplify our package search results
Why not use a simpler package search results format like Arch Linux or Debian
does? We could just display the package name, version and synopsis like so.
inkscape 0.92.4
Vector graphics editor
inklingreader 0.8
Wacom Inkling sketch format conversion and manipulation
Why do we need the entire recutils format? If the user is interested, they can
always use `guix package --show` to get the full recutils formatted
info. Having shorter search results will make everything even faster and much
more readable. WDYT?
* How to test this patchset
To get guile-xapian, run a `guix pull`, if you haven't already. Then in your
Guix source directory, drop into an environment with guix dependencies and
guile-xapian.
$ guix environment guix --ad-hoc guile-xapian
Apply patches and build.
$ git am v2-0000-cover-letter.patch v2-0002-gnu-Generate-Xapian-package-search-index.patch v2-0001-build-self-Add-guile-xapian-to-Guix-dependencies.patch v2-0003-gnu-Use-Xapian-index-for-package-search.patch
$ make
Run a test guix pull.
$ ./pre-inst-env guix pull --url=$PWD --branch=xapian -p /tmp/test
where xapian is the name of the branch you committed the patches to.
Then, run the guix search in /tmp/test.
$ /tmp/test/bin/guix search game
* Comments
Pierre Neidhardt <mail@HIDDEN> writes:
>> +(define (search-package-index profile querystring)
>
> Maybe `query-string'?
Done in this patchset.
>> + (define (regexp? str)
>> + (string-any
>> + (char-set #\. #\[ #\{ #\} #\( #\) #\\ #\* #\+ #\? #\| #\^ #\$)
>> + str))
>> +
>> + (if (and (current-profile)
>> + (not (any regexp? patterns)))
>
> I would not put characters like ".", "$", or "+" here, lest we mistake a
> Xapian pattern for a regexp.
>
> As you said, I don't think both are compatible without ambiguity
> anyways, so we should probably drop regexp (or at least toggle them with
> a command line argument).
I agree.
zimoun <zimon.toutoune@HIDDEN> writes:
> In the commit message, I would capitalize Xapian.
Done in this patchset.
>> +(define (generate-package-search-index directory)
>> + "Generate under DIRECTORY a xapian index of all the available packages."
>
> Xapian with capital.
Done in this patchset.
> Is (make-stem "en") for the locale?
I still have English hard-coded. I haven't yet figured out how to detect the
locale and stem accordingly. But, there is a larger problem. Since we cannot
anticipate what locale the user will run guix search with, should we build the
Xapian index for all locales? That is, should we index not only the English
versions of the packages but also all other translations as well?
> package-search-index and package-cache-file could be refactored
> because they share all the same code.
Yes, they could be. However, I'll postpone to the next iteration of the
patchset.
> I do not know what is the convention for the bindings.
> But there is 'fold-packages' so I would be inclined to 'fold-msets' or
> something in this flavour.
Well, everywhere else in guile we have such things as vhash-fold, string-fold,
hash-fold, stream-fold, etc. That's why I went with mset-fold. Also, we are
folding over a single mset (match-set). So, mset should be in the singular.
> And more importantly, 'make as-derivations' to avoid a "guix pull" breakage,
> Ah do not forget to adapt some tests.
Will do this once we have consensus about the other features of this patchset.
> b. The xapian relevance should truncated
Done in this patchset.
> Xapian does not return the package 'emacs' itself as the first. And worse,
> it is not returned at all.
In this patchset, since we're indexing the package name as well, emacs is
returned but it is still far from the beginning.
> I propose the value of 4294967295 for pagesize.
In this patchset, I pass (database-document-count db) as the #:maximum-items
keyword argument to enquire-mset. This is the upstream recommended way to get
all search results. I hadn't done this earlier since I hadn't yet wrapped
database-document-count in guile-xapian.
>> In this patchset, I have only indexed the package descriptions. In the next
>> version of this patchset, I will index all other terms as specified in
>> %package-metrics of guix/ui.scm.
>
> Yes, it appears to me a detail that should be easy to fix. I mean, it
> does not seems blocking.
Done in this patchset.
Ludovic Courtès <ludo@HIDDEN> writes:
> Note that ‘guix search’ time is largely dominated by I/O.
Yes, `guix search` is I/O intensive. That is why I expect Xapian to do better
since it only needs to access matching packages not all packages. Also, the
Xapian index is fast at all times. It is not very dependent on a warm
filesystem cache.
> On my laptop,
> I get (first measurement is cold cache, second one is warm cache):
>
> --8<---------------cut here---------------start------------->8---
> $ sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
> $ time guix search foo >/dev/null
>
> real 0m2.631s
> user 0m1.134s
> sys 0m0.124s
> $ time guix search foo >/dev/null
>
> real 0m0.836s
> user 0m1.027s
> sys 0m0.053s
> --8<---------------cut here---------------end--------------->8---
>
> It’s hard to do better on the warm cache case because at this level,
> there may be other things to optimize having little to do with searching
> itself.
>
> Note that this is on an SSD; the cold-cache case must be worse on NFS or
> on a spinning disk, and there we could gain a lot.
My laptop is quite old with a particularly slow HDD. Hence my motivation to
improve guix search performance!
> I think we should weigh the pros and cons on all these aspects: speed,
> complexity and maintenance cost, search result quality, search features,
> etc.
I agree.
> PS: I have not yet looked at the whole series as I’m just coming back to
> the keyboard. :-)
Welcome back! :-)
Arun Isaac (3):
build-self: Add guile-xapian to Guix dependencies.
gnu: Generate Xapian package search index.
gnu: Use Xapian index for package search.
build-aux/build-self.scm | 11 +++++++
gnu/packages.scm | 62 +++++++++++++++++++++++++++++++++++++++-
guix/channels.scm | 34 +++++++++++++++++++++-
guix/scripts/package.scm | 7 +++--
guix/self.scm | 7 ++++-
guix/ui.scm | 37 ++++++++++++++++++++++++
6 files changed, 153 insertions(+), 5 deletions(-)
--
2.25.1
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.
Received: (at 39258) by debbugs.gnu.org; 7 Mar 2020 13:31:41 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Mar 07 08:31:41 2020
Received: from localhost ([127.0.0.1]:46406 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1jAZYL-0002rQ-68
for submit <at> debbugs.gnu.org; Sat, 07 Mar 2020 08:31:41 -0500
Received: from mugam.systemreboot.net ([139.59.75.54]:51828)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <arunisaac@HIDDEN>) id 1jAZYI-0002qw-Dh
for 39258 <at> debbugs.gnu.org; Sat, 07 Mar 2020 08:31:39 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed;
d=systemreboot.net; s=default; h=Content-Transfer-Encoding:Content-Type:
MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender
:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From:
Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:
List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive;
bh=KfB6VaHDc8ym5DA2S5eEW2qMotN8Qo5p82aaen+WX80=; b=m+HoWXLQ5NxJ6zyb8LojJ9A48C
uKIVm/VZUYfDwgOcq0wUXVAu3SE9Dl3CJiF+K+r1ez3cSqmOl3oV/maRZc0pyAraYAc8LbE87RKY8
AeAAIpGeI+M1jGJdNltf56QsQ7SIpcIKlUgCJ1f4Ov+78wiK/5ExOGJggNrjiLfDlEdY=;
Received: from [192.168.2.1] (helo=steel.lan)
by systemreboot.net with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384
(Exim 4.93) (envelope-from <arunisaac@HIDDEN>)
id 1jAZY7-002bX3-VK; Sat, 07 Mar 2020 19:01:28 +0530
From: Arun Isaac <arunisaac@HIDDEN>
To: 39258 <at> debbugs.gnu.org
Subject: [PATCH v2 3/3] gnu: Use Xapian index for package search.
Date: Sat, 7 Mar 2020 19:01:16 +0530
Message-Id: <20200307133116.11443-4-arunisaac@HIDDEN>
X-Mailer: git-send-email 2.25.1
In-Reply-To: <20200307133116.11443-1-arunisaac@HIDDEN>
References: <20200307133116.11443-1-arunisaac@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 39258
Cc: Arun Isaac <arunisaac@HIDDEN>, mail@HIDDEN, ludo@HIDDEN,
zimon.toutoune@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)
* gnu/packages.scm (search-package-index): New function.
* guix/ui.scm (display-package-search-results): New function.
* guix/scripts/package.scm (process-query): Search using the Xapian package
index if current profile is available. Else, search using regexps.
---
gnu/packages.scm | 22 +++++++++++++++++++++-
guix/scripts/package.scm | 7 +++++--
guix/ui.scm | 35 +++++++++++++++++++++++++++++++++++
3 files changed, 61 insertions(+), 3 deletions(-)
diff --git a/gnu/packages.scm b/gnu/packages.scm
index c8e221de68..3cbd7c63e3 100644
--- a/gnu/packages.scm
+++ b/gnu/packages.scm
@@ -67,7 +67,8 @@
specifications->manifest
generate-package-cache
- generate-package-search-index))
+ generate-package-search-index
+ search-package-index))
;;; Commentary:
;;;
@@ -466,6 +467,25 @@ reducing the memory footprint."
db-path)
+(define (search-package-index profile query-string)
+ "Search Xapian index in PROFILE for packages matching the Xapian query
+QUERY-STRING. Return a list of search result texts each corresponding to one
+matching package."
+ (call-with-database (string-append profile %package-search-index)
+ (lambda (db)
+ (let ((query (parse-query query-string #:stemmer (make-stem "en"))))
+ (mset-fold (lambda (item result)
+ (let ((search-result-text
+ (call-with-output-string
+ (cut format <> "~a~%relevance: ~a~%~%"
+ (document-data (mset-item-document item))
+ ;; Round score to one decimal place.
+ (/ (round (* 10 (mset-item-weight item))) 10)))))
+ (append result (list search-result-text))))
+ '()
+ (enquire-mset (enquire db query)
+ #:maximum-items (database-document-count db)))))))
+
(define %sigint-prompt
;; The prompt to jump to upon SIGINT.
diff --git a/guix/scripts/package.scm b/guix/scripts/package.scm
index d2f4f1ccd3..91c975b168 100644
--- a/guix/scripts/package.scm
+++ b/guix/scripts/package.scm
@@ -7,6 +7,7 @@
;;; Copyright © 2016 Benz Schenk <benz.schenk@HIDDEN>
;;; Copyright © 2016 Chris Marusich <cmmarusich@HIDDEN>
;;; Copyright © 2019 Tobias Geerinckx-Rice <me@HIDDEN>
+;;; Copyright © 2020 Arun Isaac <arunisaac@HIDDEN>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -781,9 +782,11 @@ processed, #f otherwise."
(_ #f))
opts))
(regexps (map (cut make-regexp* <> regexp/icase) patterns))
- (matches (find-packages-by-description regexps)))
+ (matches (if (current-profile)
+ (search-package-index (current-profile) (string-join patterns " "))
+ (find-packages-by-description regexps))))
(leave-on-EPIPE
- (display-search-results matches (current-output-port)))
+ (display-package-search-results matches (current-output-port)))
#t))
(('show requested-name)
diff --git a/guix/ui.scm b/guix/ui.scm
index 3bc82111a5..163042054c 100644
--- a/guix/ui.scm
+++ b/guix/ui.scm
@@ -121,6 +121,7 @@
relevance
package-relevance
display-search-results
+ display-package-search-results
%package-metrics
with-profile-lock
@@ -1490,6 +1491,40 @@ to view all the results.")
(()
#t))))
+(define* (display-package-search-results search-results port
+ #:key
+ (command "guix search"))
+ "Display SEARCH-RESULTS, a list of search result texts each corresponding to
+one matching package. If PORT is a terminal, print at most a full screen of
+results."
+ (define first-line
+ (port-line port))
+
+ (define max-rows
+ (and first-line (isatty? port)
+ (terminal-rows port)))
+
+ (define (line-count str)
+ (string-count str #\newline))
+
+ (let loop ((search-results search-results))
+ (match search-results
+ ((text rest ...)
+ (if (and (not (getenv "INSIDE_EMACS"))
+ max-rows
+ (> (port-line port) first-line) ;print at least one result
+ (> (+ 4 (line-count text) (port-line port))
+ max-rows))
+ (unless (null? rest)
+ (display-hint (format #f (G_ "Run @code{~a ... | less} \
+to view all the results.")
+ command)))
+ (begin
+ (display text port)
+ (loop rest))))
+ (()
+ #t))))
+
(define (string->generations str)
"Return the list of generations matching a pattern in STR. This function
--
2.25.1
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.
Received: (at 39258) by debbugs.gnu.org; 7 Mar 2020 13:31:41 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Mar 07 08:31:41 2020
Received: from localhost ([127.0.0.1]:46404 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1jAZYK-0002rN-OO
for submit <at> debbugs.gnu.org; Sat, 07 Mar 2020 08:31:41 -0500
Received: from mugam.systemreboot.net ([139.59.75.54]:51824)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <arunisaac@HIDDEN>) id 1jAZYH-0002qq-OF
for 39258 <at> debbugs.gnu.org; Sat, 07 Mar 2020 08:31:38 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed;
d=systemreboot.net; s=default; h=Content-Transfer-Encoding:Content-Type:
MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender
:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From:
Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:
List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive;
bh=BO+LbygYA4iKXddmWm8457vAuodCLPfBfiwe5W17c6M=; b=MtA/wa/NxZFTULsqh2b0RUArsg
eWJ8ayXVAleRdtv5YcwqDpKcHGZV8bzODSDyObaCalmWNXCnVkMFecRhWdzHoL8Pxrwi2BdySGBKd
8sqvmZYuDRD9FLz56Eaat2gI5t3cZH9VoZo1GsZWOfWqlf6TXtnp6Km/j0CklKgQuark=;
Received: from [192.168.2.1] (helo=steel.lan)
by systemreboot.net with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384
(Exim 4.93) (envelope-from <arunisaac@HIDDEN>)
id 1jAZY7-002bX3-Om; Sat, 07 Mar 2020 19:01:27 +0530
From: Arun Isaac <arunisaac@HIDDEN>
To: 39258 <at> debbugs.gnu.org
Subject: [PATCH v2 2/3] gnu: Generate Xapian package search index.
Date: Sat, 7 Mar 2020 19:01:15 +0530
Message-Id: <20200307133116.11443-3-arunisaac@HIDDEN>
X-Mailer: git-send-email 2.25.1
In-Reply-To: <20200307133116.11443-1-arunisaac@HIDDEN>
References: <20200307133116.11443-1-arunisaac@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 39258
Cc: Arun Isaac <arunisaac@HIDDEN>, mail@HIDDEN, ludo@HIDDEN,
zimon.toutoune@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)
* guix/ui.scm: Export %package-metrics.
* gnu/packages.scm (%package-search-index): New variable.
(generate-package-search-index): New function.
* guix/channels.scm (package-search-index): New function.
(%channel-profile-hooks): Add package-search-index.
---
gnu/packages.scm | 42 +++++++++++++++++++++++++++++++++++++++++-
guix/channels.scm | 34 +++++++++++++++++++++++++++++++++-
guix/ui.scm | 2 ++
3 files changed, 76 insertions(+), 2 deletions(-)
diff --git a/gnu/packages.scm b/gnu/packages.scm
index d22c992bb1..c8e221de68 100644
--- a/gnu/packages.scm
+++ b/gnu/packages.scm
@@ -4,6 +4,7 @@
;;; Copyright © 2014 Eric Bavier <bavier@HIDDEN>
;;; Copyright © 2016, 2017 Alex Kost <alezost@HIDDEN>
;;; Copyright © 2016 Mathieu Lirzin <mthl@HIDDEN>
+;;; Copyright © 2020 Arun Isaac <arunisaac@HIDDEN>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -43,6 +44,7 @@
#:use-module (srfi srfi-34)
#:use-module (srfi srfi-35)
#:use-module (srfi srfi-39)
+ #:use-module (xapian xapian)
#:export (search-patch
search-patches
search-auxiliary-file
@@ -64,7 +66,8 @@
specification->location
specifications->manifest
- generate-package-cache))
+ generate-package-cache
+ generate-package-search-index))
;;; Commentary:
;;;
@@ -426,6 +429,43 @@ reducing the memory footprint."
#:opts '(#:to-file? #t)))))
cache-file)
+(define %package-search-index
+ ;; Location of the package search-index
+ "/lib/guix/package-search.index")
+
+(define (generate-package-search-index directory)
+ "Generate under DIRECTORY a Xapian index of all the available packages."
+ (define db-path
+ (string-append directory %package-search-index))
+
+ (mkdir-p (dirname db-path))
+ (call-with-writable-database db-path
+ (lambda (db)
+ (fold-packages (lambda (package _)
+ (let* ((idterm (string-append "Q" (package-name package)))
+ (doc (make-document #:data (string-trim-right
+ (call-with-output-string
+ (cut package->recutils package <>))
+ #\newline)
+ #:terms `((,idterm . 0))))
+ (term-generator (make-term-generator #:stem (make-stem "en")
+ #:document doc)))
+ (for-each (match-lambda
+ ((field . weight)
+ (match (field package)
+ ((? string? str)
+ (index-text! term-generator str
+ #:wdf-increment weight))
+ ((lst ...)
+ (for-each (cut index-text! term-generator <>
+ #:wdf-increment weight)
+ lst)))
+ (replace-document! db idterm doc)))
+ %package-metrics)))
+ #f)))
+
+ db-path)
+
(define %sigint-prompt
;; The prompt to jump to upon SIGINT.
diff --git a/guix/channels.scm b/guix/channels.scm
index f0261dc2da..c70c70938c 100644
--- a/guix/channels.scm
+++ b/guix/channels.scm
@@ -2,6 +2,7 @@
;;; Copyright © 2018, 2019, 2020 Ludovic Courtès <ludo@HIDDEN>
;;; Copyright © 2018 Ricardo Wurmus <rekado@HIDDEN>
;;; Copyright © 2019 Jan (janneke) Nieuwenhuizen <janneke@HIDDEN>
+;;; Copyright © 2020 Arun Isaac <arunisaac@HIDDEN>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -581,9 +582,40 @@ be used as a profile hook."
(hook . package-cache))
#:local-build? #t)))
+(define (package-search-index manifest)
+ "Build a package search index for the instance in MANIFEST. This is meant
+to be used as a profile hook."
+ (mlet %store-monad ((profile (profile-derivation manifest
+ #:hooks '())))
+
+ (define build
+ #~(begin
+ (use-modules (gnu packages))
+
+ (if (defined? 'generate-package-search-index)
+ (begin
+ ;; Delegate package search index generation to the inferior.
+ (format (current-error-port)
+ "Generating package search index for '~a'...~%"
+ #$profile)
+ (generate-package-search-index #$output))
+ (mkdir #$output))))
+
+ (gexp->derivation-in-inferior "guix-package-search-index" build
+ profile
+
+ ;; If the Guix in PROFILE is too old and
+ ;; lacks 'guix repl', don't build the cache
+ ;; instead of failing.
+ #:silent-failure? #t
+
+ #:properties '((type . profile-hook)
+ (hook . package-search-index))
+ #:local-build? #t)))
+
(define %channel-profile-hooks
;; The default channel profile hooks.
- (cons package-cache-file %default-profile-hooks))
+ (cons* package-cache-file package-search-index %default-profile-hooks))
(define (channel-instances->derivation instances)
"Return the derivation of the profile containing INSTANCES, a list of
diff --git a/guix/ui.scm b/guix/ui.scm
index fbe2b70485..3bc82111a5 100644
--- a/guix/ui.scm
+++ b/guix/ui.scm
@@ -14,6 +14,7 @@
;;; Copyright © 2019 Chris Marusich <cmmarusich@HIDDEN>
;;; Copyright © 2019 Tobias Geerinckx-Rice <me@HIDDEN>
;;; Copyright © 2019 Simon Tournier <zimon.toutoune@HIDDEN>
+;;; Copyright © 2020 Arun Isaac <arunisaac@HIDDEN>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -120,6 +121,7 @@
relevance
package-relevance
display-search-results
+ %package-metrics
with-profile-lock
string->generations
--
2.25.1
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.
Received: (at 39258) by debbugs.gnu.org; 7 Mar 2020 13:31:39 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Mar 07 08:31:39 2020
Received: from localhost ([127.0.0.1]:46402 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1jAZYJ-0002rA-D1
for submit <at> debbugs.gnu.org; Sat, 07 Mar 2020 08:31:39 -0500
Received: from mugam.systemreboot.net ([139.59.75.54]:51814)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <arunisaac@HIDDEN>) id 1jAZYF-0002ql-1D
for 39258 <at> debbugs.gnu.org; Sat, 07 Mar 2020 08:31:37 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed;
d=systemreboot.net; s=default; h=Content-Transfer-Encoding:Content-Type:
MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender
:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From:
Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:
List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive;
bh=zoaipT+rHGiH/JjrvRyY6G1OW4AJNxxgAqR1qpbWqoU=; b=S4ZjSUcaRrynOK21fHLL8HOi78
eWcU7+KyrUAk2vQQlnyQiacbpzStQf2nu6Yss1e5f/8jzTKHHc+4FcbAgCHf4Zzh38NuZ6W57HEhF
DU/pipBNn0JM4YxvdspB41b7Vk1+iDA3Ri9elgnf0j/VlSGVhhTfDlttYybMIl8ZtF9M=;
Received: from [192.168.2.1] (helo=steel.lan)
by systemreboot.net with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384
(Exim 4.93) (envelope-from <arunisaac@HIDDEN>)
id 1jAZY7-002bX3-If; Sat, 07 Mar 2020 19:01:27 +0530
From: Arun Isaac <arunisaac@HIDDEN>
To: 39258 <at> debbugs.gnu.org
Subject: [PATCH v2 1/3] build-self: Add guile-xapian to Guix dependencies.
Date: Sat, 7 Mar 2020 19:01:14 +0530
Message-Id: <20200307133116.11443-2-arunisaac@HIDDEN>
X-Mailer: git-send-email 2.25.1
In-Reply-To: <20200307133116.11443-1-arunisaac@HIDDEN>
References: <20200307133116.11443-1-arunisaac@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 39258
Cc: Arun Isaac <arunisaac@HIDDEN>, mail@HIDDEN, ludo@HIDDEN,
zimon.toutoune@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)
* build-aux/build-self.scm (build-program): Import fake guile-xapian module.
* guix/self.scm (compiled-guix): Add guile-xapian to Guix dependencies.
---
build-aux/build-self.scm | 11 +++++++++++
guix/self.scm | 7 ++++++-
2 files changed, 17 insertions(+), 1 deletion(-)
diff --git a/build-aux/build-self.scm b/build-aux/build-self.scm
index f2e785b7f1..05d0353ccf 100644
--- a/build-aux/build-self.scm
+++ b/build-aux/build-self.scm
@@ -1,5 +1,6 @@
;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2014, 2016, 2017, 2018, 2019, 2020 Ludovic Courtès <ludo@HIDDEN>
+;;; Copyright © 2020 Arun Isaac <arunisaac@HIDDEN>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -261,6 +262,10 @@ interface (FFI) of Guile.")
#~(define-module (gcrypt hash)
#:export (sha1 sha256))))
+ (define fake-xapian-hash
+ ;; Fake (xapian xapian) module; see below.
+ (scheme-file "xapian.scm" #~(define-module (xapian xapian))))
+
(define fake-git
(scheme-file "git.scm" #~(define-module (git))))
@@ -273,6 +278,12 @@ interface (FFI) of Guile.")
;; adjust %LOAD-PATH later on.
((gcrypt hash) => ,fake-gcrypt-hash)
+ ;; To avoid relying on 'with-extensions', which was
+ ;; introduced in 0.15.0, provide a fake (xapian
+ ;; xapian) just so that we can build modules, and
+ ;; adjust %LOAD-PATH later on.
+ ((xapian xapian) => ,fake-xapian-hash)
+
;; (guix git-download) depends on (git) but only
;; for peripheral functionality. Provide a dummy
;; (git) to placate it.
diff --git a/guix/self.scm b/guix/self.scm
index 6b633f9bc0..a4f40574d1 100644
--- a/guix/self.scm
+++ b/guix/self.scm
@@ -1,5 +1,6 @@
;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2017, 2018, 2019, 2020 Ludovic Courtès <ludo@HIDDEN>
+;;; Copyright © 2020 Arun Isaac <arunisaac@HIDDEN>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -54,6 +55,7 @@
("guile-git" (ref '(gnu packages guile) 'guile3.0-git))
("guile-sqlite3" (ref '(gnu packages guile) 'guile3.0-sqlite3))
("guile-gcrypt" (ref '(gnu packages gnupg) 'guile3.0-gcrypt))
+ ("guile-xapian" (ref '(gnu packages guile-xyz) 'guile3.0-xapian))
("gnutls" (ref '(gnu packages tls) 'guile3.0-gnutls))
("zlib" (ref '(gnu packages compression) 'zlib))
("lzlib" (ref '(gnu packages compression) 'lzlib))
@@ -682,6 +684,9 @@ Info manual."
(define guile-gcrypt
(specification->package "guile-gcrypt"))
+ (define guile-xapian
+ (specification->package "guile-xapian"))
+
(define gnutls
(specification->package "gnutls"))
@@ -690,7 +695,7 @@ Info manual."
(cons (list "x" package)
(package-transitive-propagated-inputs package)))
(list guile-gcrypt gnutls guile-git guile-json
- guile-ssh guile-sqlite3))
+ guile-ssh guile-sqlite3 guile-xapian))
(((labels packages _ ...) ...)
packages)))
--
2.25.1
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 5 Mar 2020 20:26:51 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Thu Mar 05 15:26:51 2020 Received: from localhost ([127.0.0.1]:43764 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1j9x50-000678-PI for submit <at> debbugs.gnu.org; Thu, 05 Mar 2020 15:26:50 -0500 Received: from mugam.systemreboot.net ([139.59.75.54]:44490) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <arunisaac@HIDDEN>) id 1j9x4y-00066x-3c for 39258 <at> debbugs.gnu.org; Thu, 05 Mar 2020 15:26:50 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=systemreboot.net; s=default; h=Content-Type:MIME-Version:Message-ID:Date: References:In-Reply-To:Subject:Cc:To:From:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=1iuD6K3+/mbiu4rwopVYkvihyTg9FrMylaJlzoXvCA0=; b=PxtYjdEk27Xh56NPRjD/OTH9u IWQR/0vZb+bdCqyV3zx2lBHLQDM/jprixIAdm5AOvs1a+rwuO296BOZr8WGAIthLPSnOL4ltx+zh+ GxXN7zWxiCDoGrRqK4kpbK4nUS7OBeAVSMdc7RKsXAEWxAxw3YOi0KelBQDJkqHvy7aRk=; Received: from [192.168.2.1] (helo=steel) by systemreboot.net with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.93) (envelope-from <arunisaac@HIDDEN>) id 1j9x4m-002RRd-2P; Fri, 06 Mar 2020 01:56:36 +0530 From: Arun Isaac <arunisaac@HIDDEN> To: Pierre Neidhardt <mail@HIDDEN> Subject: Re: [bug#39258] [PATCH 3/4] gnu: Generate xapian package search index. In-Reply-To: <87k1475e94.fsf@HIDDEN> References: <20200227204150.30985-1-arunisaac@HIDDEN> <20200227204150.30985-4-arunisaac@HIDDEN> <87k1475e94.fsf@HIDDEN> Date: Fri, 06 Mar 2020 01:56:22 +0530 Message-ID: <cu7k13yeefl.fsf@HIDDEN> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 39258 Cc: ludo@HIDDEN, 39258 <at> debbugs.gnu.org, zimon.toutoune@HIDDEN X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain >> + (fold-packages (lambda (package _) >> + (let* ((idterm (string-append "Q" (package-name package))) >> + (doc (make-document #:data (package-name package) >> + #:terms `((,idterm . 0)))) >> + (term-generator (make-term-generator #:stem (make-stem "en") >> + #:document doc))) >> + (index-text! term-generator (package-description package)) >> + (replace-document! db idterm doc))) > > I guess these non-functional functions (index-text!, replace-document!) > represent how Xapian works at the C++ level. Would it be possible to > make more functional bindings nonetheless? I somehow overlooked this particular email and am reading it just now. Yes, the non-functional bindings are a bit ugly. But, I'm not able to think of a clean way to make functional bindings without supporting all features offered by xapian. Any suggestions you have in this regard would be useful. Look through xapian/termgenerator.h for more details. In particular, look at functions increase_termpos, index_text_without_positions. --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEf3MDQ/Lwnzx3v3nTLiXui2GAK7MFAl5hYG8ACgkQLiXui2GA K7PkRwf+NvdP6lMsIbZH4CZRl0J/rfHZv4wQD6P4VA+JejO0wuiT9L7lYezPz06U VdRGo6pt/GGt/XIA0qKA3+HumBtaWJ5YLCRFbgJKa/iqHaMUEfl1LU/rWo4peUDk DRrSP4tGFhYJtxGiHuPU/ia7KyWaCzoERhFRfQgKUM69/WSkpxKjQpo6tzciBBuv LVvRTcy0CcPyUww210W2qwL50fzQ27dgbFaX4WXLHmzpQGPDMXbmRQCXPsbCsTFX iLPJzdPHiu1BCl15nevLCcfVCdmHKelyIU23LmhD8LtrtuLnIaYlb4UE4fMPlE9U 5wubfuszWe90QibFjZmjuKHZRoLTeA== =WJVN -----END PGP SIGNATURE----- --=-=-=--
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.
Received: (at 39258) by debbugs.gnu.org; 5 Mar 2020 16:46:16 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Mar 05 11:46:16 2020
Received: from localhost ([127.0.0.1]:43640 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1j9tdX-00006S-UT
for submit <at> debbugs.gnu.org; Thu, 05 Mar 2020 11:46:16 -0500
Received: from eggs.gnu.org ([209.51.188.92]:46659)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <ludo@HIDDEN>) id 1j9tdV-00006F-DJ
for 39258 <at> debbugs.gnu.org; Thu, 05 Mar 2020 11:46:13 -0500
Received: from fencepost.gnu.org ([2001:470:142:3::e]:46920)
by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <ludo@HIDDEN>)
id 1j9tdO-0000fh-HM; Thu, 05 Mar 2020 11:46:06 -0500
Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=33932 helo=ribbon)
by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
(Exim 4.82) (envelope-from <ludo@HIDDEN>)
id 1j9tdO-0007jf-0Q; Thu, 05 Mar 2020 11:46:06 -0500
From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>
To: Arun Isaac <arunisaac@HIDDEN>
Subject: Re: [PATCH 0/4] Xapian for Guix package search
References: <20200227204150.30985-1-arunisaac@HIDDEN>
X-URL: http://www.fdn.fr/~lcourtes/
X-Revolutionary-Date: 16 =?utf-8?Q?Vent=C3=B4se?= an 228 de la =?utf-8?Q?R?=
=?utf-8?Q?=C3=A9volution?=
X-PGP-Key-ID: 0x090B11993D9AEBB5
X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc
X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5
X-OS: x86_64-pc-linux-gnu
Date: Thu, 05 Mar 2020 17:46:01 +0100
In-Reply-To: <20200227204150.30985-1-arunisaac@HIDDEN> (Arun Isaac's
message of "Fri, 28 Feb 2020 02:11:46 +0530")
Message-ID: <87h7z292d2.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Spam-Score: -0.7 (/)
X-Debbugs-Envelope-To: 39258
Cc: mail@HIDDEN, 39258 <at> debbugs.gnu.org, zimon.toutoune@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)
Hello Arun,
Arun Isaac <arunisaac@HIDDEN> skribis:
> * Speed improvement
>
> Despite search-package-index in gnu/packages.scm taking only around 1.5ms=
, I
> see an overall speedup in `guix search` of only a factor of 2 -- from aro=
und
> 2s to around 1s. I wonder what else in `guix search` is taking up so much
> time.
Note that =E2=80=98guix search=E2=80=99 time is largely dominated by I/O. =
On my laptop,
I get (first measurement is cold cache, second one is warm cache):
--8<---------------cut here---------------start------------->8---
$ sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
$ time guix search foo >/dev/null
real 0m2.631s
user 0m1.134s
sys 0m0.124s
$ time guix search foo >/dev/null
real 0m0.836s
user 0m1.027s
sys 0m0.053s
--8<---------------cut here---------------end--------------->8---
It=E2=80=99s hard to do better on the warm cache case because at this level,
there may be other things to optimize having little to do with searching
itself.
Note that this is on an SSD; the cold-cache case must be worse on NFS or
on a spinning disk, and there we could gain a lot.
I think we should weigh the pros and cons on all these aspects: speed,
complexity and maintenance cost, search result quality, search features,
etc.
Thanks,
Ludo=E2=80=99.
PS: I have not yet looked at the whole series as I=E2=80=99m just coming ba=
ck to
the keyboard. :-)
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 3 Mar 2020 20:04:56 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Tue Mar 03 15:04:56 2020 Received: from localhost ([127.0.0.1]:39995 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1j9Dmi-00088a-C9 for submit <at> debbugs.gnu.org; Tue, 03 Mar 2020 15:04:56 -0500 Received: from mail-qt1-f196.google.com ([209.85.160.196]:46987) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <zimon.toutoune@HIDDEN>) id 1j9Dmg-00088N-Nb for 39258 <at> debbugs.gnu.org; Tue, 03 Mar 2020 15:04:54 -0500 Received: by mail-qt1-f196.google.com with SMTP id i14so3797932qtv.13 for <39258 <at> debbugs.gnu.org>; Tue, 03 Mar 2020 12:04:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=OL++D0jUcXb39SoPufciQnKNtRQ29JxMZxGV1aFQ7bE=; b=CkMtvFZj3wNI6N7pa8etjEa0wC4qURIniX9dgVEerc0W8lgIOvYO36ocfPYSe+35SC W4280yisyIlsMph1tUkTw6KXVZgnVCTkjWNiRPgWum3p4GjcpePh4kuPEv+kxdR9KOic IgbmJCzt/9xyT91vKSBKvHhnmYQGqQorQl/cA7TmGFdhr7jwqEVoTwHDNGpy0I+kqStC KKhB/5e7mFovSlUe+s1nCAtzigAHQjUI7VVu2CFu3SAUP43ubmcwwBqNLfIQWIxWN6fh IpkmOt72FX0uQFxCTUerJSDg7T4bHDKfF8duZ4B4QaCPpU7rDQfRrz24Jj/5e7jKHIIo T++A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=OL++D0jUcXb39SoPufciQnKNtRQ29JxMZxGV1aFQ7bE=; b=Kx8BQfKQmv4NFbO7vuM7DFH5zDG33uXT+oLaQAX32+Zt7hl8v4+sEtHCGXZP0DlRyb aAmLtuWG0C3hwaLqyaSkZoKbOFUgt/N029ujS5qDNgRc7Q+tOEw2c34BSiXJ6dX5E+1V Z2s91Ps5tJBBlRGzuMW+VusIRmKUDnLvHWPzigbrLBsJtIZX5+QKubvSGgqI22nN/ETc H4jDfvEeclBzdD8sge4e5uYKdFY7Tj2U5/YmE/O6XekMqvMXTMhXPF7+cEUldVdQTHtJ /4YxbbdMn6kEDMUYxU0QvxtD64PxWed+hCD5RxN9aaQqFrINivwvonzyLGiqi7iY9QaZ +eVQ== X-Gm-Message-State: ANhLgQ0db6PnhIXXbtYJbCCVFc74gHCmP8rZwTy2YG450wIg2vrlQ2QI fRVFrghenPuSAJhgvNyP5892Q/apvEx9Uc8Ypwg= X-Google-Smtp-Source: ADFU+vvCncqER+uwMO5Sg186Q94TfXpg3QamlzxE8lqQB9DYe2WiCyx5aHpLgcGDBikD4n8uuYNhYo785qjoWXK8Mys= X-Received: by 2002:ac8:39c2:: with SMTP id v60mr5757523qte.211.1583265889100; Tue, 03 Mar 2020 12:04:49 -0800 (PST) MIME-Version: 1.0 References: <20200227204150.30985-1-arunisaac@HIDDEN> <87eeuf5dty.fsf@HIDDEN> <cu7d09ypvu1.fsf@HIDDEN> <cu74kvapuja.fsf@HIDDEN> <CAJ3okZ2g8iDRoAt40MXzPD8vP58ntiGyZsPqG6pZ+ZXBPaquEA@HIDDEN> <CAJ3okZ0=jbtutBFFYTxa8S=d3MDNvJRuS_19WKp-Nvqpmx7eVQ@HIDDEN> In-Reply-To: <CAJ3okZ0=jbtutBFFYTxa8S=d3MDNvJRuS_19WKp-Nvqpmx7eVQ@HIDDEN> From: zimoun <zimon.toutoune@HIDDEN> Date: Tue, 3 Mar 2020 21:04:38 +0100 Message-ID: <CAJ3okZ2ugJ9B6-pwcjezCfL2F+KpgmTt=mr1LH3ugMdnWYkwYg@HIDDEN> Subject: Re: [PATCH 0/4] Xapian for Guix package search To: Arun Isaac <arunisaac@HIDDEN> Content-Type: text/plain; charset="UTF-8" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 39258 Cc: =?UTF-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>, Pierre Neidhardt <mail@HIDDEN>, 39258 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) Hi, On Mon, 2 Mar 2020 at 20:13, zimoun <zimon.toutoune@HIDDEN> wrote: > --8<---------------cut here---------------start------------->8--- > /tmp/test/bin/guix search emacs | recsel -C -P name | wc -l > 10 > --8<---------------cut here---------------end--------------->8--- > > Maybe I am doing a mistake. I think this issue is fixed when changing the 'pagesize' value. Well, with '(pagesize 4294967295)' and using the same commit (c1febbbf94), I get: --8<---------------cut here---------------start------------->8--- guix time-machine --commit=c1febbbf94 -- guix search games | recsel -C -p name | wc -l 247 ./pre-inst-env guix search games | recsel -C -p name | wc -l 236 --8<---------------cut here---------------end--------------->8--- (I modified the patches in order to pull once to generate the index at commit c1febbbf94 and then do some stuff.) Note that the old "guix search" does not output blender and Xapian does even if the term 'games' is not in the description but 'game' is. Well, I am comparing the different list, i.e., "guix search games | recsel -C -P name | sort" to see which one is in one list and not the other one. But before going more ahead, let polish a bit the patches to more easily test without the double environment etc. And because I am using good old HDD and some SSD comparison should be welcome. All the best, simon
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 3 Mar 2020 19:51:36 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Tue Mar 03 14:51:36 2020 Received: from localhost ([127.0.0.1]:39977 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1j9DZn-0007nx-Ry for submit <at> debbugs.gnu.org; Tue, 03 Mar 2020 14:51:36 -0500 Received: from mail-qk1-f194.google.com ([209.85.222.194]:45886) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <zimon.toutoune@HIDDEN>) id 1j9DZm-0007nl-GO for 39258 <at> debbugs.gnu.org; Tue, 03 Mar 2020 14:51:35 -0500 Received: by mail-qk1-f194.google.com with SMTP id z12so4643219qkg.12 for <39258 <at> debbugs.gnu.org>; Tue, 03 Mar 2020 11:51:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=zTUzrJ5VddGPSa7CUQB6IAV2+2Ewj4ZfOQImEfV1R5k=; b=QcyW1cuxVtqtDPcfJImvVRLifYeiZy5QiyAv/QDLVixYSo90WSHg4+zuwZKSVd35n7 ZzCDc/WSV0DN+b8sGYwAduGYdEatF6EjfuqPObr3oYL5MWGlxFd5ww6Uq4gp6yLa5g16 DKb0Ah137+W4raGEajaqva/oibnNEXdZ3AIEGPtuUawE5bJP2uLoAUAb2lnhmlD7yl5r PuMR4xMMPOG60ivcozQEBFqvU8hrdeaqM0pPPmuM05Mvh+exJ2AB03D/EvAEkH1jwj5R DnKEmddrJ3MpCe0G+pU+aYLeJDwZEryhfVvtVWprrltsKdpkRHXfdJd+cpx6umXMBIpa iETg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=zTUzrJ5VddGPSa7CUQB6IAV2+2Ewj4ZfOQImEfV1R5k=; b=HqJUTP3qoNCzbKNjVy+4G2ytPS9fZNGBGDgCybU6ZA6xkPViLYO4PjMufJ7QmL1rPa MSL3EQrPqMB9ljeTnDZaaGagv5CkbogPwf7EhyTkDDkOA3UlIgW78nuffzfxHO4LcmGx BN7HUNmelQHYPfie/xlpho+HlljKCF9eD+9Rjzw7cBOfynLeCxpXHpOQfTc+rkxRYKGy 1ZB2S1ZAvNfhYEjC6oSUS6UeBBRVQzvNyAsTqPI3IFW0I4/0PBSoQfjeCeR4Wxr+k1jx BdLu8H7HY758UM4EVio3n0EKZIRFnAAmVowHZHDBlancTmFnOMGvpx2WvHzmo7I4UfRB 76QQ== X-Gm-Message-State: ANhLgQ1tr7130eA4kLoaJBskvMJThxHhwYcuHcBh2n6R6mr203lyFHKv WY8aeYDn3vNdNQWM6ruB46QRtBiuOU3IUECCOacHQV5O X-Google-Smtp-Source: ADFU+vu18GyGoy9FVlrHzih/QsCLKbbEEj1xgSzRAGiZgEi/1OormWELY6loy1h6JTmxzJkP4RcHcK4A6xZetUMCzUU= X-Received: by 2002:ae9:ef4c:: with SMTP id d73mr5215297qkg.201.1583265088834; Tue, 03 Mar 2020 11:51:28 -0800 (PST) MIME-Version: 1.0 References: <20200227204150.30985-1-arunisaac@HIDDEN> <20200227204150.30985-5-arunisaac@HIDDEN> <CAJ3okZ0XiJcAwq8bqOWL7bPWMXrZ6zo9D0JMx86x0nJa4apBpQ@HIDDEN> In-Reply-To: <CAJ3okZ0XiJcAwq8bqOWL7bPWMXrZ6zo9D0JMx86x0nJa4apBpQ@HIDDEN> From: zimoun <zimon.toutoune@HIDDEN> Date: Tue, 3 Mar 2020 20:51:17 +0100 Message-ID: <CAJ3okZ0zoD6N74Wrx29Thp-Hx+kMWVKoPQ-eZSt84R4UiBa_Gw@HIDDEN> Subject: Re: [PATCH 4/4] gnu: Use xapian index for package search. To: Arun Isaac <arunisaac@HIDDEN> Content-Type: text/plain; charset="UTF-8" X-Spam-Score: -1.5 (-) X-Debbugs-Envelope-To: 39258 Cc: =?UTF-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>, 39258 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -2.5 (--) On Tue, 3 Mar 2020 at 20:21, zimoun <zimon.toutoune@HIDDEN> wrote: > On Thu, 27 Feb 2020 at 21:42, Arun Isaac <arunisaac@HIDDEN> wrote: > > +(define (search-package-index profile querystring) > > + (let ((offset 0) > > + (pagesize 10)) > > Why this value of 10? > This fix the number of packages returned. Hum? > I have tried to replace by 100 and I got 100 packages. :-) I propose the value of 4294967295 for pagesize.
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.
Received: (at 39258) by debbugs.gnu.org; 3 Mar 2020 19:22:06 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Mar 03 14:22:06 2020
Received: from localhost ([127.0.0.1]:39964 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1j9D7F-000556-Uj
for submit <at> debbugs.gnu.org; Tue, 03 Mar 2020 14:22:06 -0500
Received: from mail-qk1-f196.google.com ([209.85.222.196]:34309)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <zimon.toutoune@HIDDEN>) id 1j9D7D-00054a-BE
for 39258 <at> debbugs.gnu.org; Tue, 03 Mar 2020 14:22:04 -0500
Received: by mail-qk1-f196.google.com with SMTP id 11so4635986qkd.1
for <39258 <at> debbugs.gnu.org>; Tue, 03 Mar 2020 11:22:03 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
h=mime-version:references:in-reply-to:from:date:message-id:subject:to
:cc:content-transfer-encoding;
bh=stLyBu/NQkc+ySqkSnQV1vi0adZuLk32p1ShQSSK00k=;
b=gWxvI/F+VR6uSDibOSDnq6n6xgQ3IltY79Yvar+O+RcfP1DQTdjcOpX3HqlJXPD/wN
uNe6kXQiDRZWuG2Ve4Elnx2fYTEMreN/C1pL8TLva7j694YOqhDytWcg1CW4dUbJbxLw
CAlPPri/iuELIsWi6DqxGHwRxoM1xlgmFa3bTfNmYUj1lwf2ZymSoGCR1u8+YRrMTr86
YHN0xDlcPaqmLTCBvCg6X8PKHEiyxRB65EfHP/SE5hfVALITo5cF/jjKQGR4YG9QceeW
Q5N1rvFHeasYzefVzXLry/XNU/Pwcdoj5Ex2XMMelUST66J9znS2vQ68lm2tQHOVMcpR
Ymjg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20161025;
h=x-gm-message-state:mime-version:references:in-reply-to:from:date
:message-id:subject:to:cc:content-transfer-encoding;
bh=stLyBu/NQkc+ySqkSnQV1vi0adZuLk32p1ShQSSK00k=;
b=ZX+hwXbWRtSuc35SDaRQEylgfoeUk5D7EcYdwYLYXCgo3sf9GvtEEfsANT5sw6daOQ
0rkhQJsglLpw6VKP6FO0zENgo+yj0weUbV8Sj+iTrL9X0yrU9/iYvXB6ytR/sgwLVqZV
hK4RwnjVQP30loRk6UtVHjX56UzaPZtfxsMMCpIcvHPOwj09MZlA8EdJbuyoFxpZGhpM
NJhiQi25cBA+xeFn3MAeF2MTYnJNVz50HZAVo8SpKQhVzCelw+ieX3znftDYRhKk4D6d
o8p55J6s+mA4gltqVTXVdFt4cN2Svdh00iwYlZr0enhHv1o7F84faeeo1tDbReUqde8V
uf0A==
X-Gm-Message-State: ANhLgQ0PpWjXg9KNsq9iKnzCGVjoKhR7XQpsHChk/xADjavP2sIasygs
Y6dDRjedlD1mu01ru0wDw4Roc7HJNyHxVumVw94=
X-Google-Smtp-Source: ADFU+vuBDaW/Y0RQnI0JW4lINJ16kohl6sCSw13F7Br2g4Xb9kz4688xfAqgmqqLHpF4T3/taFw+pAtFWt/PMN/t3DA=
X-Received: by 2002:a05:620a:2282:: with SMTP id
o2mr5134427qkh.304.1583263317489;
Tue, 03 Mar 2020 11:21:57 -0800 (PST)
MIME-Version: 1.0
References: <20200227204150.30985-1-arunisaac@HIDDEN>
<20200227204150.30985-5-arunisaac@HIDDEN>
In-Reply-To: <20200227204150.30985-5-arunisaac@HIDDEN>
From: zimoun <zimon.toutoune@HIDDEN>
Date: Tue, 3 Mar 2020 20:21:46 +0100
Message-ID: <CAJ3okZ0XiJcAwq8bqOWL7bPWMXrZ6zo9D0JMx86x0nJa4apBpQ@HIDDEN>
Subject: Re: [PATCH 4/4] gnu: Use xapian index for package search.
To: Arun Isaac <arunisaac@HIDDEN>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -1.5 (-)
X-Debbugs-Envelope-To: 39258
Cc: =?UTF-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>, 39258 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -2.5 (--)
Hi Arun,
On Thu, 27 Feb 2020 at 21:42, Arun Isaac <arunisaac@HIDDEN> wrote=
:
>
> * gnu/packages.scm (search-package-index): New function.
> * guix/scripts/package.scm (find-packages-by-description): Search using t=
he
> xapian package index if search patterns are literal strings. Else, search
> using fold-packages.
> ---
> gnu/packages.scm | 17 +++++++++++-
> guix/scripts/package.scm | 57 +++++++++++++++++++++++-----------------
> 2 files changed, 49 insertions(+), 25 deletions(-)
>
> diff --git a/gnu/packages.scm b/gnu/packages.scm
> index e91753e2a8..5b5b29bf84 100644
> --- a/gnu/packages.scm
> +++ b/gnu/packages.scm
> @@ -67,7 +67,8 @@
> specifications->manifest
>
> generate-package-cache
> - generate-package-search-index))
> + generate-package-search-index
> + search-package-index))
>
> ;;; Commentary:
> ;;;
> @@ -453,6 +454,20 @@ reducing the memory footprint."
>
> db-path)
>
> +(define (search-package-index profile querystring)
> + (let ((offset 0)
> + (pagesize 10))
Why this value of 10?
This fix the number of packages returned. Hum?
I have tried to replace by 100 and I got 100 packages. :-)
> + (call-with-database (string-append profile %package-search-index)
> + (lambda (db)
> + (let ((query (parse-query querystring #:stemmer (make-stem "en")=
)))
> + (mset-fold (lambda (item result)
I do not know what is the convention for the bindings.
But there is 'fold-packages' so I would be inclined to 'fold-msets' or
something in this flavour.
> + (match (find-packages-by-name
> + (document-data (mset-item-document item))=
)
> + ((package _ ...)
> + (append result `((,package . ,(mset-item-weigh=
t item)))))))
> + '()
> + (enquire-mset (enquire db query) offset pagesize)))=
))))
> +
>
> (define %sigint-prompt
> ;; The prompt to jump to upon SIGINT.
> diff --git a/guix/scripts/package.scm b/guix/scripts/package.scm
> index 1cb0d382bf..6a3b9002dd 100644
> --- a/guix/scripts/package.scm
> +++ b/guix/scripts/package.scm
> @@ -7,6 +7,7 @@
> ;;; Copyright =C2=A9 2016 Benz Schenk <benz.schenk@HIDDEN>
> ;;; Copyright =C2=A9 2016 Chris Marusich <cmmarusich@HIDDEN>
> ;;; Copyright =C2=A9 2019 Tobias Geerinckx-Rice <me@HIDDEN>
> +;;; Copyright =C2=A9 2020 Arun Isaac <arunisaac@HIDDEN>
> ;;;
> ;;; This file is part of GNU Guix.
> ;;;
> @@ -178,31 +179,40 @@ hooks\" run when building the profile."
> ;;; Package specifications.
> ;;;
>
> -(define (find-packages-by-description regexps)
> +(define (find-packages-by-description patterns)
> "Return a list of pairs: packages whose name, synopsis, description,
> or output matches at least one of REGEXPS sorted by relevance, and its
> non-zero relevance score."
> - (let ((matches (fold-packages (lambda (package result)
> - (if (package-superseded package)
> - result
> - (match (package-relevance package
> - regexps)
> - ((? zero?)
> - result)
> - (score
> - (cons (cons package score)
> - result)))))
> - '())))
> - (sort matches
> - (lambda (m1 m2)
> - (match m1
> - ((package1 . score1)
> - (match m2
> - ((package2 . score2)
> - (if (=3D score1 score2)
> - (string>? (package-full-name package1)
> - (package-full-name package2))
> - (> score1 score2))))))))))
> + (define (regexp? str)
> + (string-any
> + (char-set #\. #\[ #\{ #\} #\( #\) #\\ #\* #\+ #\? #\| #\^ #\$)
> + str))
Instead of reverting this, I would let the current
'find-packages-by-description' and would add
'find-packages-by-description-indexed' doing just
'(search-package-index (current-profile) (string-join patterns " "))'.
And maybe refactoring the sort of scores. Then I would put the test
branch in 'guix/scripts/packages.scm'...
> + (if (and (current-profile)
> + (not (any regexp? patterns)))
> + (search-package-index (current-profile) (string-join patterns " ")=
)
> + (let* ((regexps (map (cut make-regexp* <> regexp/icase) patterns))
> + (matches (fold-packages (lambda (package result)
> + (if (package-superseded package)
> + result
> + (match (package-relevance pac=
kage
Note that I am in the process of implementing the BM25 weights as
'package-relevance'; at least really thinking about it! :-)
I have already talked about TF-IDF as relevance, for example here [1].
And reading the Xapian documentation [2], it seems affordable. Or not
;-) because of the regexp... Need some thoughts... I mean "in the
process". ;-)
And in this case, it is almost a drop-in replacement of
'fold-packages' by 'mset-fold'; well it should add some flexibility
and a more unified code.
(Aside the searching, IMHO 'package-relevance' should help too in the
linting process of bad written descriptions, another story. ;-)
[1] https://lists.gnu.org/archive/html/guix-devel/2019-07/msg00252.html
[2] https://xapian.org/docs/bm25.html
> + reg=
exps)
> + ((? zero?)
> + result)
> + (score
> + (cons (cons package score)
> + result)))))
> + '())))
> + (sort matches
> + (lambda (m1 m2)
> + (match m1
> + ((package1 . score1)
> + (match m2
> + ((package2 . score2)
> + (if (=3D score1 score2)
> + (string>? (package-full-name package1)
> + (package-full-name package2))
> + (> score1 score2)))))))))))
>
> (define (transaction-upgrade-entry store entry transaction)
> "Return a variant of TRANSACTION that accounts for the upgrade of ENTR=
Y, a
> @@ -777,8 +787,7 @@ processed, #f otherwise."
...here.
+ (define (regexp? str)
+ (string-any
+ (char-set #\. #\[ #\{ #\} #\( #\) #\\ #\* #\+ #\? #\| #\^ #\$)
+ str))
> (('query 'search rx) rx)
> (_ #f))
> opts))
>
> - (regexps (map (cut make-regexp* <> regexp/icase) patterns=
))
> - (matches (find-packages-by-description regexps)))
+ (if (any regexp? patterns)
+ (matches (find-packages-by-description regexps))
+ (matches (find-packages-by-description-indexed patterns))
I mean something like that.
> (leave-on-EPIPE
> (display-search-results matches (current-output-port)))
> #t))
> --
> 2.23.0
All the best,
simon
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 3 Mar 2020 18:30:17 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Tue Mar 03 13:30:17 2020 Received: from localhost ([127.0.0.1]:39943 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1j9CJ6-0003pl-Ty for submit <at> debbugs.gnu.org; Tue, 03 Mar 2020 13:30:17 -0500 Received: from mail-qv1-f68.google.com ([209.85.219.68]:43875) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <zimon.toutoune@HIDDEN>) id 1j9CJ4-0003pW-Gg for 39258 <at> debbugs.gnu.org; Tue, 03 Mar 2020 13:30:14 -0500 Received: by mail-qv1-f68.google.com with SMTP id eb12so2125498qvb.10 for <39258 <at> debbugs.gnu.org>; Tue, 03 Mar 2020 10:30:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=fnaoObAJrdN/Q8kNIVxp3iZrEeI38DHp6t/2y05c17E=; b=R/9ZO0fKoUrmBPBSGU1DtFUMBtGZPJ1Hih/W4HWBfJO5rU4WENW8PgdCQdIKYOKuIM oK5tpibNRUjsJYM+nsUNynECRy3FT9gj8MNnTO23//AwkITViM5/hmXsr9FqZA8nJ3Es 8xIL9zgrhq20zIiOz59mHaP/xy8W5CbZGjjZP6Tcka3dstqWgBtIvLM3Pfj3K9Frzs0H V3D/vG25KuDmv4nGFIsHUKJ03znBgF95WGDHGjUxMY3TfVmju9aD2k1pUcrNH9OwTnOQ lRWbJ9vlupG/RiXDygEF0HfoLx92fBeYxkRfyz9YpmfcK4Xsu1iBBs6844lIGefCiaqF ITfw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=fnaoObAJrdN/Q8kNIVxp3iZrEeI38DHp6t/2y05c17E=; b=k2xHxSq6lYh1LL/6sgxEsrrHhOvLrA1Hv+kn3pqUDDxh+wg58EM+QMdgMCElgNoBuM sNiiVFn6iRZxVlBvCW0TDy8dWtVLmhSiYq5m7BIVrWZDZ2HgBJwylAZiYVeUbausvBAw +Dti7xAPlXPIDMzuxQij3uWoW3NhawSwtZCRtKn7e9xpdj5MZ8Al47LjqY2AetkQhzLP egeHeko+WXdB/B/YwfC3lVUyT+pNUQt4fYYU8N03WMYczhqv+oxVnIv503jR+DmIiVGS 0PPNuU9t93JhkhYa6ERSDvX8Lu+xsEMa/Wv226uVlPDySL4J/cEzeuMh/TCDjuDTK05r Cyvg== X-Gm-Message-State: ANhLgQ07gKadlfsYuUzjuiQDMCQzpqcurnU6I46hgIhbqi3KZsN/ok47 CtZ+4pPMRL9KSWjJW8lqmgGBBkhVUSupJC3JP4Y= X-Google-Smtp-Source: ADFU+vuDjEX6N6wlUQh4Z2Pxmsyfnh/M8O2qVJwDhgpbJbvACbkc4Io47undnSfL7PfuEkiHzskr6iHKEP7tsTmdPYI= X-Received: by 2002:a0c:c70b:: with SMTP id w11mr5013805qvi.246.1583260208719; Tue, 03 Mar 2020 10:30:08 -0800 (PST) MIME-Version: 1.0 References: <20200227204150.30985-1-arunisaac@HIDDEN> <20200227204150.30985-4-arunisaac@HIDDEN> In-Reply-To: <20200227204150.30985-4-arunisaac@HIDDEN> From: zimoun <zimon.toutoune@HIDDEN> Date: Tue, 3 Mar 2020 19:29:57 +0100 Message-ID: <CAJ3okZ2TxN=mxQKhP7S7-yeAmBhYFiBkGHDYbzjD2kO_X7igFQ@HIDDEN> Subject: Re: [PATCH 3/4] gnu: Generate xapian package search index. To: Arun Isaac <arunisaac@HIDDEN> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spam-Score: -1.5 (-) X-Debbugs-Envelope-To: 39258 Cc: =?UTF-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>, 39258 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -2.5 (--) Hi Arun, In the commit message, I would capitalize Xapian. On Thu, 27 Feb 2020 at 21:42, Arun Isaac <arunisaac@HIDDEN> wrote= : > > * gnu/packages.scm (%package-search-index): New variable. > (generate-package-search-index): New function. > * guix/channels.scm (package-search-index): New function. > (%channel-profile-hooks): Add package-search-index. > --- > gnu/packages.scm | 29 ++++++++++++++++++++++++++++- > guix/channels.scm | 34 +++++++++++++++++++++++++++++++++- > 2 files changed, 61 insertions(+), 2 deletions(-) > > diff --git a/gnu/packages.scm b/gnu/packages.scm > index d22c992bb1..e91753e2a8 100644 > --- a/gnu/packages.scm > +++ b/gnu/packages.scm > @@ -4,6 +4,7 @@ > ;;; Copyright =C2=A9 2014 Eric Bavier <bavier@HIDDEN> > ;;; Copyright =C2=A9 2016, 2017 Alex Kost <alezost@HIDDEN> > ;;; Copyright =C2=A9 2016 Mathieu Lirzin <mthl@HIDDEN> > +;;; Copyright =C2=A9 2020 Arun Isaac <arunisaac@HIDDEN> > ;;; > ;;; This file is part of GNU Guix. > ;;; > @@ -43,6 +44,7 @@ > #:use-module (srfi srfi-34) > #:use-module (srfi srfi-35) > #:use-module (srfi srfi-39) > + #:use-module (xapian xapian) > #:export (search-patch > search-patches > search-auxiliary-file > @@ -64,7 +66,8 @@ > specification->location > specifications->manifest > > - generate-package-cache)) > + generate-package-cache > + generate-package-search-index)) > > ;;; Commentary: > ;;; > @@ -426,6 +429,30 @@ reducing the memory footprint." > #:opts '(#:to-file? #t))))) > cache-file) > > +(define %package-search-index > + ;; Location of the package search-index > + "/lib/guix/package-search.index") > + > +(define (generate-package-search-index directory) > + "Generate under DIRECTORY a xapian index of all the available packages= ." Xapian with capital. > + (define db-path > + (string-append directory %package-search-index)) > + > + (mkdir-p (dirname db-path)) > + (call-with-writable-database db-path > + (lambda (db) > + (fold-packages (lambda (package _) > + (let* ((idterm (string-append "Q" (package-name p= ackage))) > + (doc (make-document #:data (package-name p= ackage) > + #:terms `((,idterm . 0= )))) > + (term-generator (make-term-generator #:ste= m (make-stem "en") > + #:doc= ument doc))) > + (index-text! term-generator (package-descriptio= n package)) Instead, this: (index-term! term-generator (string-append (package-synopsis package) (package-description package))) should index both 'synopsis' and 'description'. Is (make-stem "en") for the locale? > + (replace-document! db idterm doc))) > + #f))) > + > + db-path) > + > > (define %sigint-prompt > ;; The prompt to jump to upon SIGINT. > diff --git a/guix/channels.scm b/guix/channels.scm > index f0261dc2da..c70c70938c 100644 > --- a/guix/channels.scm > +++ b/guix/channels.scm > @@ -2,6 +2,7 @@ > ;;; Copyright =C2=A9 2018, 2019, 2020 Ludovic Court=C3=A8s <ludo@HIDDEN= > > ;;; Copyright =C2=A9 2018 Ricardo Wurmus <rekado@HIDDEN> > ;;; Copyright =C2=A9 2019 Jan (janneke) Nieuwenhuizen <janneke@HIDDEN> > +;;; Copyright =C2=A9 2020 Arun Isaac <arunisaac@HIDDEN> > ;;; > ;;; This file is part of GNU Guix. > ;;; > @@ -581,9 +582,40 @@ be used as a profile hook." > (hook . package-cache)) > #:local-build? #t))) > > +(define (package-search-index manifest) > + "Build a package search index for the instance in MANIFEST. This is m= eant > +to be used as a profile hook." > + (mlet %store-monad ((profile (profile-derivation manifest > + #:hooks '()))) > + > + (define build > + #~(begin > + (use-modules (gnu packages)) > + > + (if (defined? 'generate-package-search-index) > + (begin > + ;; Delegate package search index generation to the infer= ior. > + (format (current-error-port) > + "Generating package search index for '~a'...~%" > + #$profile) > + (generate-package-search-index #$output)) > + (mkdir #$output)))) > + > + (gexp->derivation-in-inferior "guix-package-search-index" build > + profile > + > + ;; If the Guix in PROFILE is too old a= nd > + ;; lacks 'guix repl', don't build the = cache > + ;; instead of failing. > + #:silent-failure? #t > + > + #:properties '((type . profile-hook) > + (hook . package-search-= index)) > + #:local-build? #t))) > + package-search-index and package-cache-file could be refactored because they share all the same code. > (define %channel-profile-hooks > ;; The default channel profile hooks. > - (cons package-cache-file %default-profile-hooks)) > + (cons* package-cache-file package-search-index %default-profile-hooks)= ) > > (define (channel-instances->derivation instances) > "Return the derivation of the profile containing INSTANCES, a list of > -- > 2.23.0 >
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 3 Mar 2020 16:29:43 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Tue Mar 03 11:29:42 2020 Received: from localhost ([127.0.0.1]:39899 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1j9AQQ-0000zj-PN for submit <at> debbugs.gnu.org; Tue, 03 Mar 2020 11:29:42 -0500 Received: from mail-qk1-f195.google.com ([209.85.222.195]:41251) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <zimon.toutoune@HIDDEN>) id 1j9AQP-0000zW-Dy for 39258 <at> debbugs.gnu.org; Tue, 03 Mar 2020 11:29:42 -0500 Received: by mail-qk1-f195.google.com with SMTP id b5so3959099qkh.8 for <39258 <at> debbugs.gnu.org>; Tue, 03 Mar 2020 08:29:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=fNX7tuO/YG795g/dJKQOSFzby49xCySe3nr40HStT8Q=; b=GYZ8COjwBXKO8qRrvH14sbLQ8TJY7CFLYrZpr2Q8aSs1DtNku/xgQ4mhad9xQsuEf1 tw24ADEMG8ttGHsTFM2c6sfWWbhS+lpUGKO+JyW8lLLYiOi8tLom4dUffOJkc2m7qjPh 83cBwyGuLtq3AGfASomcjQu2jOFVBt8GSpyaeTUo9DT8Cj+PbbZpG24VvSr7HMXTO3E7 EwxRxFkN05XZdhxPXoE/4xwS7J2j1BKmfMF7+mqR02MaN0/1RDMFFYEQBzz/bdJPJR6e X234209avRVPm8EqwOpFJNyecxS4TpSF/pl+ck4m0vnd0AICsajPbvJXKtdp1VXW/aBL 6f8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=fNX7tuO/YG795g/dJKQOSFzby49xCySe3nr40HStT8Q=; b=p8f8/vRdJOxqEdGx2jdlwXsxhKE4Cym6jYczwsiH1c9Uf8zzuKp76YMQkPqXOiha/e Z4bnBeyHzQ0ZNh8QSUMRVb81zn221OOLwgJNRg0yJ+C4TfqRc86CVoGUV1sp6Uo3hlGJ qZeuYbo1rDDXokHnKfWaoIme3EtfQNi6PlowjVB2RwKJEeCbeDzDZ79xTweUtSK4NAZf F+uNu0yux0E9cSzE3LDR//q28oi187X/heQy8bpbhnb1YYX8uOXhPQsN1TubCJ14itsY HgniwMq+HhaJb8HUKzwF/EWyOtWRUuRMEe1+R3c63LWjHOe/q13fPnthGz3if+3OXOxk AkTA== X-Gm-Message-State: ANhLgQ3UOFQUxGNtJg6RMnkctEtjW9O3T1oZpkY1QX00erytmUan707d nGeiL6mKZoaIq0hogLmKe8HNUYLOtl99q6hCPdg= X-Google-Smtp-Source: ADFU+vsX2h9Meat5sE8uNLTGKUsNYV0x9ROAW25MJHtl8+5OdReL3mEOVehz2MzBqFgywLtn2oTKvuuo61vN+5SHM5I= X-Received: by 2002:ae9:ef4c:: with SMTP id d73mr4499131qkg.201.1583252975788; Tue, 03 Mar 2020 08:29:35 -0800 (PST) MIME-Version: 1.0 References: <20200227204150.30985-1-arunisaac@HIDDEN> <20200227204150.30985-2-arunisaac@HIDDEN> In-Reply-To: <20200227204150.30985-2-arunisaac@HIDDEN> From: zimoun <zimon.toutoune@HIDDEN> Date: Tue, 3 Mar 2020 17:29:24 +0100 Message-ID: <CAJ3okZ1XAysirS_w1Z_3FUnwZYUyfUqTEUuHQaWTyEp40DNFLw@HIDDEN> Subject: Re: [PATCH 1/4] gnu: Add guile-xapian. To: Arun Isaac <arunisaac@HIDDEN> Content-Type: text/plain; charset="UTF-8" X-Spam-Score: -1.5 (-) X-Debbugs-Envelope-To: 39258 Cc: =?UTF-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>, 39258 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -2.5 (--) Hi Arun, On Thu, 27 Feb 2020 at 21:42, Arun Isaac <arunisaac@HIDDEN> wrote: > * gnu/packages/guile-xyz.scm (guile-xapian, guile3.0-xapian): New variables. I am a bit lost with the Guile update. Now the convention should not be the opposite: guile-xapian using 3.0 and guile2.2-xapian using 2.2 (or simply 2.2 since 2.0 seems not really used). Otherwise, feel free to push it. :-) (It will ease to reach a large audience of testers for "guix search" ;-)) All the best, simon
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 2 Mar 2020 19:13:51 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Mon Mar 02 14:13:51 2020 Received: from localhost ([127.0.0.1]:38367 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1j8qVj-0005cp-KG for submit <at> debbugs.gnu.org; Mon, 02 Mar 2020 14:13:51 -0500 Received: from mail-qv1-f65.google.com ([209.85.219.65]:45497) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <zimon.toutoune@HIDDEN>) id 1j8qVi-0005cc-9m for 39258 <at> debbugs.gnu.org; Mon, 02 Mar 2020 14:13:50 -0500 Received: by mail-qv1-f65.google.com with SMTP id r8so409224qvs.12 for <39258 <at> debbugs.gnu.org>; Mon, 02 Mar 2020 11:13:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=3dNy6xvPlSYWj3ep0Yce7D5MwEW1Loy6IVw4MITxO1s=; b=E6Lh/5BeH7gKictrcysKB17GmdWPqgIqaIKEB/xwMFj35ceOf2y3hzdQDtLH33u8ZP HSpVrXyZ/XGZdf1qLwEA/a5SrQRMVQAqn1brBiHEo3pdvF0/ThjLnMTpmkpmcoQ8rdzX XPnF6wtJhymvsa1lj/HGCxgIT2GXFgzmKUbe/wh4sKbvWywg/naJJnaq/wMFCHm2D/Ti dVRi1ARuUdj1ZF7H/sUvuVDR48lHu3ssp09StgSpF7eWvtrALGoe0dQATBi9W5ElQABZ vp/X18sc0yhdpmmMJMSas1PwxsbzSIseYoM/mmWhxQCNkRAFXsRlL5w7FZS76+WYq7mT paNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=3dNy6xvPlSYWj3ep0Yce7D5MwEW1Loy6IVw4MITxO1s=; b=jI8Zi8X0ny7B7Au0aAfFgTU5AYC5cFvz/Nbau5EnMuw5VAavbPc+sSwhkUvIGz//hV 6G9V9C5HEhgEdpmxo5tvsEdh/AOiQC1GAk6QB1/2IeO3TkTBmCScn8YkBoIrv6kkEYy4 4aE8OJ4mpt15PK5D//cEVvEHV5j6lqnshSPdN3OKhHuSQSBMP5oWvPm0og29K1TLtprs f9pDenU2GdURrQvlkTdp3w6uReSlcwDKera+RYXnn84zzI5K2qd7KmeM/sb3NVQm4LZH ChY2hRKVUxfHQ5qwkUdPptrqYbF7w5jnK2J+Nqk94Pq1k9o+mHj1GnckT36gHAbu6x3a LNiQ== X-Gm-Message-State: ANhLgQ2JRtjnMXURAn33xI2/tQjHDzNWsUhjZ271YZZ24+5OG0r7Vab0 ipqhlww1OKCfIxknKaKw2YX2Go9jmiXUFuMzj3k= X-Google-Smtp-Source: ADFU+vvTpbbrzmxXepwz7XyuNM52rUXPDmxKEaAgxivwYWW/MbgVYq6QUEVadvJgVjzHRXewxERE1+DVC22NiOgyc8I= X-Received: by 2002:a0c:f98e:: with SMTP id t14mr847559qvn.74.1583176424775; Mon, 02 Mar 2020 11:13:44 -0800 (PST) MIME-Version: 1.0 References: <20200227204150.30985-1-arunisaac@HIDDEN> <87eeuf5dty.fsf@HIDDEN> <cu7d09ypvu1.fsf@HIDDEN> <cu74kvapuja.fsf@HIDDEN> <CAJ3okZ2g8iDRoAt40MXzPD8vP58ntiGyZsPqG6pZ+ZXBPaquEA@HIDDEN> In-Reply-To: <CAJ3okZ2g8iDRoAt40MXzPD8vP58ntiGyZsPqG6pZ+ZXBPaquEA@HIDDEN> From: zimoun <zimon.toutoune@HIDDEN> Date: Mon, 2 Mar 2020 20:13:33 +0100 Message-ID: <CAJ3okZ0=jbtutBFFYTxa8S=d3MDNvJRuS_19WKp-Nvqpmx7eVQ@HIDDEN> Subject: Re: [PATCH 0/4] Xapian for Guix package search To: Arun Isaac <arunisaac@HIDDEN> Content-Type: text/plain; charset="UTF-8" X-Spam-Score: -1.5 (-) X-Debbugs-Envelope-To: 39258 Cc: =?UTF-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>, Pierre Neidhardt <mail@HIDDEN>, 39258 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -2.5 (--) Hi, After a quick benchmark: a. It is faster. Between x2 and x3. Really? b. The xapian relevance should truncated and examined in more details. --8<---------------cut here---------------start------------->8--- time guix search emacs | recsel -p name,relevance | head -n18 name: emacs relevance: 33 name: emacs-with-editor relevance: 19 name: emacs-restart-emacs relevance: 19 name: emacs-epkg relevance: 18 name: guile-emacs relevance: 17 name: emacs-xwidgets relevance: 17 real 0m1.530s user 0m1.827s sys 0m0.074s --8<---------------cut here---------------end--------------->8--- --8<---------------cut here---------------start------------->8--- time /tmp/test/bin/guix search emacs | recsel -p name,relevance | head -n18 name: emacs-helm-pass relevance: 5.0774748262821685 name: emacs-spark relevance: 4.898640632723127 name: emacs-evil-smartparens relevance: 4.898640632723127 name: emacs-howm relevance: 4.8638448958830685 name: emacs-el-mock relevance: 4.8638448958830685 name: emacs-strace-mode relevance: 4.693676055650271 real 0m0.440s user 0m0.482s sys 0m0.058s --8<---------------cut here---------------end--------------->8--- Here for example, Xapian does not return the package 'emacs' itself as the first. And worse, it is not returned at all. That's said, I do not know if it is really faster since: --8<---------------cut here---------------start------------->8--- guix search emacs | recsel -C -P name | wc -l 829 --8<---------------cut here---------------end--------------->8--- and --8<---------------cut here---------------start------------->8--- /tmp/test/bin/guix search emacs | recsel -C -P name | wc -l 10 --8<---------------cut here---------------end--------------->8--- Maybe I am doing a mistake. Well, thank you Arun for the Xapian bindings which will improve the searching experience. :-) And now it needs some polishing. All the best simo
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 2 Mar 2020 18:38:09 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Mon Mar 02 13:38:09 2020 Received: from localhost ([127.0.0.1]:38350 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1j8pxB-0004gW-7X for submit <at> debbugs.gnu.org; Mon, 02 Mar 2020 13:38:09 -0500 Received: from mail-qt1-f195.google.com ([209.85.160.195]:33998) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <zimon.toutoune@HIDDEN>) id 1j8px9-0004gG-7q for 39258 <at> debbugs.gnu.org; Mon, 02 Mar 2020 13:38:07 -0500 Received: by mail-qt1-f195.google.com with SMTP id n1so775594qtv.1 for <39258 <at> debbugs.gnu.org>; Mon, 02 Mar 2020 10:38:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=iAsSaQBsFI/nD+AYuRtv+a6RvRw3cGJVFyso9y+9HLI=; b=EKqA/KXI1lNzdQynwobSK+fVDKEY6xrAxXVR1rY29qXOYdDLiYAjxjBBYSiwqXhStP qRmU9eQTn1BAtie02ZgPL9A7jYtKpessR67UJkaORd9Y/e3eu8LVoaHQ3ECjcb8JyT9a n81XYyI4yqrQoNe9Ano+W2Y7h4jKjn5pyZLD90gIS3WTkYV3OUvLb4SkDTnQrSMr4gum hM9qksNakUG9bjoTITMrkoDXNYvTX4/rhy35nfFlg/St/J/o2CgdjP7w/oRORAEqmGnP 5Oxvemt6JPT7ibGnhs8rW7379HULc//t8FQye2qQb3u7iErmUW9csXbrRrMOFJ+Dey0Z woLw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=iAsSaQBsFI/nD+AYuRtv+a6RvRw3cGJVFyso9y+9HLI=; b=ZuxW6FRhui1KOPRRBq7K595a4vGhnRBfKKXZjLpixJcvrjtBNrueD3sHKtgMWg4b6u uCYKLgkIvXRTSQXxM6Zk94khn5Xuvm8tjM/7VQeeOb58jj5cDQ9aoxUc2GhK5ZXtdRdh HgSCYGVHn7jeCXqdOe5oxWRaGV9wvA5ICfZg8jqgVS6cqsSZmZZbPS70GjMEwwL+cMj4 2JycsWQlzMhGLO+miFoaE9J9I3rHucZ91Fevw592hyyX8D14Fn2JNJ6EG+24m+hBpKBT zzaInRGedp7oAM6v24CPkVXyjaB028hiZ3MiQkRbLeDUo1qPQ5JxRvdsQcDGlL/ndcnQ QYPw== X-Gm-Message-State: ANhLgQ0YSgG6xcWoR5zLfkZmOQozI4AbsJAD9B1ARR+xo9T2iRAWHial VHHDOw6uykFPIxNu6Fp4cvDh17iNh1GJ8k0cqBw= X-Google-Smtp-Source: ADFU+vuLPRMAzw6RMaW5TpY3TUbXOBy/WykRCWDTLtpPq5Kd7nBmYwOnC5yzPaUIWBBErtzql8wm5359QiBs9oFJRKY= X-Received: by 2002:ac8:4f43:: with SMTP id i3mr1053030qtw.186.1583174281604; Mon, 02 Mar 2020 10:38:01 -0800 (PST) MIME-Version: 1.0 References: <20200227204150.30985-1-arunisaac@HIDDEN> <87eeuf5dty.fsf@HIDDEN> <cu7d09ypvu1.fsf@HIDDEN> <cu74kvapuja.fsf@HIDDEN> In-Reply-To: <cu74kvapuja.fsf@HIDDEN> From: zimoun <zimon.toutoune@HIDDEN> Date: Mon, 2 Mar 2020 19:37:50 +0100 Message-ID: <CAJ3okZ2g8iDRoAt40MXzPD8vP58ntiGyZsPqG6pZ+ZXBPaquEA@HIDDEN> Subject: Re: [PATCH 0/4] Xapian for Guix package search To: Arun Isaac <arunisaac@HIDDEN> Content-Type: text/plain; charset="UTF-8" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 39258 Cc: =?UTF-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>, Pierre Neidhardt <mail@HIDDEN>, 39258 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) Hi Arun, Do you have some benchmark in mind? On Fri, 28 Feb 2020 at 17:05, Arun Isaac <arunisaac@HIDDEN> wrote: > ./pre-inst-env guix pull --url=$PWD --branch=xapian -p /tmp/test We need to benchmark on different machines the new "guix pull". Well, it is nothing compared to the derivation computations. :-) And more importantly, 'make as-derivations' to avoid a "guix pull" breakage, Then on cold caches, the new "guix search" for a couple of query. There is no so much inspiration in tests/. :-) Ah do not forget to adapt some tests. All the best, simon
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 2 Mar 2020 18:27:22 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Mon Mar 02 13:27:22 2020 Received: from localhost ([127.0.0.1]:38345 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1j8pmk-0004QJ-6i for submit <at> debbugs.gnu.org; Mon, 02 Mar 2020 13:27:22 -0500 Received: from mail-qk1-f193.google.com ([209.85.222.193]:45135) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <zimon.toutoune@HIDDEN>) id 1j8pmi-0004Q6-1e for 39258 <at> debbugs.gnu.org; Mon, 02 Mar 2020 13:27:20 -0500 Received: by mail-qk1-f193.google.com with SMTP id z12so601801qkg.12 for <39258 <at> debbugs.gnu.org>; Mon, 02 Mar 2020 10:27:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=Z3Yjy+hXVNEaIOPUwZt/m6NmTEXMciR9iUJkjF9zisw=; b=JOJyjSGRSkkTFWCSmCuy15zBgTQwoHbIIygvpel9UMNUSgTgh56Yv3Lh0Y/dbdRnDN /GvyhVejhI/JvGE/yVPoq5YtfsLx3iJqsb5OABTLQJl+MEQYrYIWUq5OMDhiDEgkiJOa /F+6Vk0oujM247Zxynv7mo/+2Bb/+oc94ESX3vXcrpwhdb+D07hMutAMeSCCxO03zlX6 Lcg1iKqHybuyI8L8Z/AU2yWkZu8Nav2NwvdvNNRhq0xHTU/IU03nCBr2UXa1T24ZcBtF VOZ/tvaGn8SerzLxtWmiN3YrI5pttQBRwpBgyBad+P7pMzGHzs5e6n73h4JHSHJViZYb LESA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=Z3Yjy+hXVNEaIOPUwZt/m6NmTEXMciR9iUJkjF9zisw=; b=GacZMnWteTuyqLafWJoaGV135PSSpeC/uWw89YnVl5MpPs3xXs83iXjqDHh94ELYWD Hio86HgzZ7iYHn6SbHOhW58lUP2z/cznrD3B4ZiDYwUG99RfTmj/mYhdWNW09XG0FdLA znyBbqc4uDEsR4nONTyhZDR48EctASmZC6PC9KgNHwTQ5T7sRJYm2t4nQypetEksEVYi mOsDKXoXxW3e6g9p2HUR4MfnTp4U58jAa7Jm1849l3HekpNIBTryGi4o4V0bcYJdCSJD L0QHm///fIaE/egMNTpuPm7wnF8HqPrk7iAfM4r43PuoywjZ9a+0UJxw+MRUxqyxIBBW Cc+g== X-Gm-Message-State: ANhLgQ1bAHN2kpW4YvfolE8tqHjNI8j/2f2Nd9IUeRrGcX3h64s2et/U 59X7B1xQFt1QUHhx8yvLiBQ9eYpn1P2my+HWW7IPlg== X-Google-Smtp-Source: ADFU+vu9vs9WSpdqkuKEBOacJk2VZXONJk42/Q/a6D9gQPbOvWEXBFRh2FNfPqNsGNRoso8X8ccaE64x6F9SPC45hq4= X-Received: by 2002:a37:a496:: with SMTP id n144mr558619qke.126.1583173634598; Mon, 02 Mar 2020 10:27:14 -0800 (PST) MIME-Version: 1.0 References: <20200227204150.30985-1-arunisaac@HIDDEN> <87eeuf5dty.fsf@HIDDEN> <cu7d09ypvu1.fsf@HIDDEN> <cu7k14594wi.fsf@HIDDEN> In-Reply-To: <cu7k14594wi.fsf@HIDDEN> From: zimoun <zimon.toutoune@HIDDEN> Date: Mon, 2 Mar 2020 19:27:03 +0100 Message-ID: <CAJ3okZ3sD9kPkvDNOgTzns-uh4CQWANenWQ372HSY3=2=D5PKQ@HIDDEN> Subject: Re: [bug#39258] [PATCH 0/4] Xapian for Guix package search To: Arun Isaac <arunisaac@HIDDEN> Content-Type: text/plain; charset="UTF-8" X-Spam-Score: -1.5 (-) X-Debbugs-Envelope-To: 39258 Cc: Pierre Neidhardt <mail@HIDDEN>, 39258 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -2.5 (--) Hi Arun, On Sat, 29 Feb 2020 at 09:25, Arun Isaac <arunisaac@HIDDEN> wrote: > Shall I push patch 1 (add guile-xapian) alone to master? Yes, it seems a good idea and it will ease the process for building and then benchmarking the "guix search" via Xapian. All the best, simon
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 29 Feb 2020 08:25:33 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Sat Feb 29 03:25:33 2020 Received: from localhost ([127.0.0.1]:34206 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1j7xRE-00055A-Qb for submit <at> debbugs.gnu.org; Sat, 29 Feb 2020 03:25:33 -0500 Received: from mugam.systemreboot.net ([139.59.75.54]:44420) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <arunisaac@HIDDEN>) id 1j7xRB-00054z-3f for 39258 <at> debbugs.gnu.org; Sat, 29 Feb 2020 03:25:31 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=systemreboot.net; s=default; h=Content-Type:MIME-Version:Message-ID:Date: References:In-Reply-To:Subject:Cc:To:From:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=BFwWYaOI5EFknJ6ipUIJhawJCd8UHhRSmkabZk/bM24=; b=FZ00ZOnuqEZpepgLC5XA4Vmkt WaX21TzqntdCOb1m5ClhqkaZoJHIbSAjRLhEuRqhebGktRAnWQLePVc/FgKNFebm8ou8zpR4xaR4s tlY1+FbdLZ+NFu3afdpBM0fdhcSV8ze06HSCi50Q4zCR13zDzJxM4v5ofFQhIJxTh70tI=; Received: from [192.168.2.1] (helo=steel) by systemreboot.net with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.93) (envelope-from <arunisaac@HIDDEN>) id 1j7xR4-001p43-Nw; Sat, 29 Feb 2020 13:55:23 +0530 From: Arun Isaac <arunisaac@HIDDEN> To: Pierre Neidhardt <mail@HIDDEN>, 39258 <at> debbugs.gnu.org Subject: Re: [bug#39258] [PATCH 0/4] Xapian for Guix package search In-Reply-To: <cu7d09ypvu1.fsf@HIDDEN> References: <20200227204150.30985-1-arunisaac@HIDDEN> <87eeuf5dty.fsf@HIDDEN> <cu7d09ypvu1.fsf@HIDDEN> Date: Sat, 29 Feb 2020 13:55:17 +0530 Message-ID: <cu7k14594wi.fsf@HIDDEN> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 39258 Cc: zimon.toutoune@HIDDEN X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain > This whole process will be simpler if the guile-xapian package is pushed > to master and guile-xapian added as an input to the guix package in > gnu/packages/package-management.scm. But, for now... Shall I push patch 1 (add guile-xapian) alone to master? --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEf3MDQ/Lwnzx3v3nTLiXui2GAK7MFAl5aH+0ACgkQLiXui2GA K7Ms1wf/fH0DfN6qH0mqwpS0RNHgrPznYh16rcDMqniHkeX7Hy+Lnlx5c9nzC6Tk 8sfBrywZfIaSRl1DVwolbfPKJdlJnY1bleW3xAoVFfgidm8zgiO4QKuU0GNE/w7y wm/c5HgtFp7FRwp3sihWhqv3gZBQQR5dBWZmE430XnGqJphNH+w72KG1SRHRnMWN jGkvdXsU7IqnW2vj9gDaPpBLobOP0AVKKJcT/cj2tn7lqAOvSn6Xs38ZcPtX0QxG acBuf+KHo5sDTj6Okjgm2gMARJ0Jjr0Rc7juKasWrFNVdOkEcUfgL539xJ7q7S7s I05AvH0KgOGd9YIggzgYfxbZwwTrWg== =am5M -----END PGP SIGNATURE----- --=-=-=--
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 28 Feb 2020 16:05:17 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Fri Feb 28 11:05:17 2020 Received: from localhost ([127.0.0.1]:33692 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1j7i8b-0000jO-5Y for submit <at> debbugs.gnu.org; Fri, 28 Feb 2020 11:05:17 -0500 Received: from mugam.systemreboot.net ([139.59.75.54]:41580) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <arunisaac@HIDDEN>) id 1j7i8X-0000j7-Na for 39258 <at> debbugs.gnu.org; Fri, 28 Feb 2020 11:05:15 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=systemreboot.net; s=default; h=Content-Type:MIME-Version:Message-ID:Date: References:In-Reply-To:Subject:Cc:To:From:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=x01wJckLZe4K3sDJjGmM8K6IdowdzJy7pJGCv1Lx8Ug=; b=nlsgehnHd8eBLWhIp35stEaEj 2mqiat/XIZwqLUyPJVbeEe8WI+uCGIntlKrLb8qbzBY1JRvMcyuNwrrZqCRxjW5fcUTZKLL3FXRWE uDqEx1lrSmAW3QDYuzgaG7tRe1YdI39at/N+J4r+BdSL/qvmeoG4uxsSriDctHVQXgrDA=; Received: from [192.168.2.1] (helo=steel) by systemreboot.net with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.93) (envelope-from <arunisaac@HIDDEN>) id 1j7i8N-001j6S-Qi; Fri, 28 Feb 2020 21:35:03 +0530 From: Arun Isaac <arunisaac@HIDDEN> To: Pierre Neidhardt <mail@HIDDEN>, 39258 <at> debbugs.gnu.org Subject: Re: [PATCH 0/4] Xapian for Guix package search In-Reply-To: <cu7d09ypvu1.fsf@HIDDEN> References: <20200227204150.30985-1-arunisaac@HIDDEN> <87eeuf5dty.fsf@HIDDEN> <cu7d09ypvu1.fsf@HIDDEN> Date: Fri, 28 Feb 2020 21:34:57 +0530 Message-ID: <cu74kvapuja.fsf@HIDDEN> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 39258 Cc: ludo@HIDDEN, zimon.toutoune@HIDDEN X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain > $ ./pre-inst-env guix pull -p /tmp/test One mistake. This command should be ./pre-inst-env guix pull --url=$PWD --branch=xapian -p /tmp/test where xapian is the name of the branch you committed the patches to. Also, I acknowledge the corrections you both suggested. I will incorporate them in v2 of the patchset. --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEf3MDQ/Lwnzx3v3nTLiXui2GAK7MFAl5ZOikACgkQLiXui2GA K7PEnAgAkIWWspGpqT7hMeD88YDFkXWEX4qJb5Z8C9Gk90AsAMlA/GCZYWVOM/FK HOuw/KhM1FEmnKX1TX3O6BV1EZgY+gjtgQKdy0Yf++kdH9kQPb+jBFCWk2mFJ72S /RY5SZ6SFCfCSEJg9elcEtCK9edAcysHNz5oY1SL0oGPSxsM+0+CVi2C+476gOJl jB9u7DPpjpnrOsmyhMd5/H6ojJ6MkndLtl3srAHokkbyrhG7KpqOOfM/UEnFZb3S yxgQY5g9IHEHUU+sVgFUE2XAA3hVcxThFjYrtgPpnru6Cd3JKxHoediJJKM6Lmdl xoM2UBchjEHAf6OnIBQ74jjUALPyNA== =d4UJ -----END PGP SIGNATURE----- --=-=-=--
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 28 Feb 2020 15:37:11 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Fri Feb 28 10:37:11 2020 Received: from localhost ([127.0.0.1]:33664 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1j7hhP-0008Mf-KR for submit <at> debbugs.gnu.org; Fri, 28 Feb 2020 10:37:11 -0500 Received: from mugam.systemreboot.net ([139.59.75.54]:41494) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <arunisaac@HIDDEN>) id 1j7hhM-0008MU-RV for 39258 <at> debbugs.gnu.org; Fri, 28 Feb 2020 10:37:10 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=systemreboot.net; s=default; h=Content-Type:MIME-Version:Message-ID:Date: References:In-Reply-To:Subject:Cc:To:From:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=81EbGKfW6K41ZVd6p90G+qVKXf30O7pu/B6c80fF3Sw=; b=VsrjF4RwphOTqdZo+9bq60HmQ lrQ32e9tyWqv7btEEdmZrfLYi7EGY9EgkZ34vyjQhHfeXr7sjigMmnnjmOPJtWibLAgbQeh9vipiS urqZ39PQdGlIz0D9oFjnH/cb89emnnbMKtbGU9xr1PNmyZpJEekS7dPiStQTvRiGBAm8s=; Received: from [192.168.2.1] (helo=steel) by systemreboot.net with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.93) (envelope-from <arunisaac@HIDDEN>) id 1j7hhC-001ixc-Ew; Fri, 28 Feb 2020 21:06:58 +0530 From: Arun Isaac <arunisaac@HIDDEN> To: Pierre Neidhardt <mail@HIDDEN>, 39258 <at> debbugs.gnu.org Subject: Re: [PATCH 0/4] Xapian for Guix package search In-Reply-To: <87eeuf5dty.fsf@HIDDEN> References: <20200227204150.30985-1-arunisaac@HIDDEN> <87eeuf5dty.fsf@HIDDEN> Date: Fri, 28 Feb 2020 21:06:54 +0530 Message-ID: <cu7d09ypvu1.fsf@HIDDEN> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 39258 Cc: ludo@HIDDEN, zimon.toutoune@HIDDEN X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain > I can't build your patch though: > > ice-9/eval.scm:293:34: no code for module (xapian xapian) Sorry, I forgot to mention this in my patch cover letter. The above error is happening because of the new guile-xapian dependency. It's a little tricky to get right at the moment. Here goes. Drop into a guix development environment. $ guix environment guix Commit patch 1 (the patch that adds guile-xapian) alone, and build. $ git am 0001-gnu-Add-guile-xapian.patch $ make Then, drop into an environment where guile-xapian is available. $ ./pre-inst-env guix environment guix --ad-hoc guile-xapian Apply the other 3 patches and build. $ git am 0002-build-self-Add-guile-xapian-to-Guix-dependencies.patch 0003-gnu-Generate-xapian-package-search-index.patch 0004-gnu-Use-xapian-index-for-package-search.patch $ make Now, the build should have completed successfully. Let's do a test guix pull to actually test the new guix search. $ ./pre-inst-env guix pull -p /tmp/test Then, run the guix search in /tmp/test. $ /tmp/test/bin/guix search game That's it! :-) This whole process will be simpler if the guile-xapian package is pushed to master and guile-xapian added as an input to the guix package in gnu/packages/package-management.scm. But, for now... --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEf3MDQ/Lwnzx3v3nTLiXui2GAK7MFAl5ZM5YACgkQLiXui2GA K7OmQwf+M9OBdWM5FdfL+jv9P6BB+KsTqEHEM/UhQGhjgXQVj0TBP9F0L/dIKLlR 1fSe8k4ZUgXNJVeMzkt9qUt+Q7TfDzu9WRkQ3lh+dh7d4fPIvYUchmg3eqJ5lvw0 jvYhiwCu/PNguvbACpV0FRRHspV6gZLtICfxPTg+3LYVRgIkEknxmekbCcrYQWa1 0x7FJ0vUkWylzn35Sim0sEeInCaTtJFwlBqRwxSd1+0r8BoYwsj8g8f3YKRlV9jT 9rRPNLdrqP88RBXMu19u4uAzHag61irD/k8cSQSl4o1hBLNDX7OLxDdhmN6MON1n eRhv5MrzYMVdcSvSlW4VZivTyBrT1Q== =GMio -----END PGP SIGNATURE----- --=-=-=--
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.
Received: (at 39258) by debbugs.gnu.org; 28 Feb 2020 12:49:42 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Feb 28 07:49:42 2020
Received: from localhost ([127.0.0.1]:60752 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1j7f5K-0002qr-7l
for submit <at> debbugs.gnu.org; Fri, 28 Feb 2020 07:49:42 -0500
Received: from relay1-d.mail.gandi.net ([217.70.183.193]:34267)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <mail@HIDDEN>) id 1j7f5I-0002qb-2n
for 39258 <at> debbugs.gnu.org; Fri, 28 Feb 2020 07:49:40 -0500
X-Originating-IP: 92.169.129.147
Received: from mimimi (lfbn-idf2-1-1315-147.w92-169.abo.wanadoo.fr
[92.169.129.147]) (Authenticated sender: mail@HIDDEN)
by relay1-d.mail.gandi.net (Postfix) with ESMTPSA id 94DF9240004;
Fri, 28 Feb 2020 12:49:32 +0000 (UTC)
From: Pierre Neidhardt <mail@HIDDEN>
To: zimoun <zimon.toutoune@HIDDEN>
Subject: Re: [PATCH 0/4] Xapian for Guix package search
In-Reply-To: <CAJ3okZ1twVzgfxRKv8MyNYY17Davd7QdkVHSp9+jb_9FV35HhQ@HIDDEN>
References: <20200227204150.30985-1-arunisaac@HIDDEN>
<87eeuf5dty.fsf@HIDDEN>
<CAJ3okZ1twVzgfxRKv8MyNYY17Davd7QdkVHSp9+jb_9FV35HhQ@HIDDEN>
Date: Fri, 28 Feb 2020 13:49:31 +0100
Message-ID: <87zhd2ho6c.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="=-=-=";
micalg=pgp-sha256; protocol="application/pgp-signature"
X-Spam-Score: 1.8 (+)
X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org",
has NOT identified this incoming email as spam. The original
message has been attached to this so you can view it or label
similar future email. If you have any questions, see
the administrator of that system for details.
Content preview: zimoun <zimon.toutoune@HIDDEN> writes: > Hi Pierre, > >
On Fri, 28 Feb 2020 at 09:13, Pierre Neidhardt <mail@HIDDEN> wrote:
> >> Beside this issue, how do you test it? I guess we first need to install
>> a bunch of package with `pre- [...]
Content analysis details: (1.8 points, 10.0 required)
pts rule name description
---- ---------------------- --------------------------------------------------
0.0 URIBL_BLOCKED ADMINISTRATOR NOTICE: The query to URIBL was
blocked. See
http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block
for more information. [URIs: ambrevar.xyz]
-0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at https://www.dnswl.org/,
low trust [217.70.183.193 listed in list.dnswl.org]
0.0 RCVD_IN_MSPIKE_H3 RBL: Good reputation (+3)
[217.70.183.193 listed in wl.mailspike.net]
-0.0 SPF_PASS SPF: sender matches SPF record
0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record
2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs
[URI: ambrevar.xyz (xyz)]
0.0 RCVD_IN_MSPIKE_WL Mailspike good senders
0.5 FROM_SUSPICIOUS_NTLD From abused NTLD
X-Debbugs-Envelope-To: 39258
Cc: Arun Isaac <arunisaac@HIDDEN>,
Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>, 39258 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: 1.8 (+)
X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org",
has NOT identified this incoming email as spam. The original
message has been attached to this so you can view it or label
similar future email. If you have any questions, see
the administrator of that system for details.
Content preview: zimoun <zimon.toutoune@HIDDEN> writes: > Hi Pierre, > >
On Fri, 28 Feb 2020 at 09:13, Pierre Neidhardt <mail@HIDDEN> wrote:
> >> Beside this issue, how do you test it? I guess we first need to install
>> a bunch of package with `pre- [...]
Content analysis details: (1.8 points, 10.0 required)
pts rule name description
---- ---------------------- --------------------------------------------------
0.0 URIBL_BLOCKED ADMINISTRATOR NOTICE: The query to URIBL was
blocked. See
http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block
for more information.
[URIs: ambrevar.xyz]
-0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at https://www.dnswl.org/,
low trust
[217.70.183.193 listed in list.dnswl.org]
0.0 RCVD_IN_MSPIKE_H3 RBL: Good reputation (+3)
[217.70.183.193 listed in wl.mailspike.net]
-0.0 SPF_PASS SPF: sender matches SPF record
0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record
2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs
[URI: ambrevar.xyz (xyz)]
1.0 BULK_RE_SUSP_NTLD Precedence bulk and RE: from a suspicious TLD
-1.0 MAILING_LIST_MULTI Multiple indicators imply a widely-seen list
manager
0.0 RCVD_IN_MSPIKE_WL Mailspike good senders
0.5 FROM_SUSPICIOUS_NTLD From abused NTLD
--=-=-=
Content-Type: text/plain
Content-Transfer-Encoding: quoted-printable
zimoun <zimon.toutoune@HIDDEN> writes:
> Hi Pierre,
>
> On Fri, 28 Feb 2020 at 09:13, Pierre Neidhardt <mail@HIDDEN> wrote:
>
>> Beside this issue, how do you test it? I guess we first need to install
>> a bunch of package with `pre-inst-env guix ...` then to a `pre-inst-env =
search`?
>
> It is not searching in the installed packages but in all the packages.
> So, to test it, you need to "./pre-inst-env guix pull -p" or something
> like that to populate the Xapian index database. Then "./pre-inst-env
> guix search" will lookup into.
> I mean, it is how I understand it should work. I have not yet looked
> into the code.
What I meant with "install a bunch of packages" is "guix pull -p", is
you said. Xapian cache
is populated as a hook of guix pull if I got it correctly.
=2D-=20
Pierre Neidhardt
https://ambrevar.xyz/
--=-=-=
Content-Type: application/pgp-signature; name="signature.asc"
-----BEGIN PGP SIGNATURE-----
iQEzBAEBCAAdFiEEUPM+LlsMPZAEJKvom9z0l6S7zH8FAl5ZDFsACgkQm9z0l6S7
zH/zOgf8CNjylikzLFmq6wV4BSdcmPP4sAdqlQSPsea2Gl6pwybKHC6nAWWWC/Pn
X79HMHiQk9lcywIsTcyW+hqHMQzWIQfFY24zndSAfAfQoRkyfEA8Icukwe0BVqmn
bpVVZvaCyUKJ8ZUpoTGJitanqbeSSQHMpKMbg9AvYq9o1ccFTwPar4f696Hm0hAM
Tgaf2p1XKf3z5XxMStF6ov+nXtM/KjWhBR3FrWvg4tRczmzkRUZItaH9ds6yT6wk
uk3kDGscvz2mzfvJaERb2jEaRpkbCnkHJC2HN7O7k/8jvuS+gLUhMw+mQs+ZuK6V
xtUrCqzg0KviCBoYNgmEPItz6sRknA==
=eYIZ
-----END PGP SIGNATURE-----
--=-=-=--
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 28 Feb 2020 12:39:27 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Fri Feb 28 07:39:27 2020 Received: from localhost ([127.0.0.1]:60746 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1j7evP-0002WP-73 for submit <at> debbugs.gnu.org; Fri, 28 Feb 2020 07:39:27 -0500 Received: from mail-qt1-f195.google.com ([209.85.160.195]:43408) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <zimon.toutoune@HIDDEN>) id 1j7evN-0002W9-MD for 39258 <at> debbugs.gnu.org; Fri, 28 Feb 2020 07:39:25 -0500 Received: by mail-qt1-f195.google.com with SMTP id g21so1844727qtq.10 for <39258 <at> debbugs.gnu.org>; Fri, 28 Feb 2020 04:39:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=D4rH6smFnaAwY/6ORlOFG01dtPv1lp7TmW85M9JETTE=; b=eM4cwAu64PWDHgn1r8fxrTGFN2qnstv/huwaOWFKXuA8QdcYlRgfJl6Ue2VDZ16W27 ZQWrvhE6Si4vmmurCfuApnYyfAPpY0IteamGz8ds9AdM/Q35KYNJ6IooTmie6ItzfmJn 7T+b95GWwXqrHQA/l0iMK6ADJ2ZVdy0FQGGj1AA4eXxNxVN77xhcd+xDpKaGK77qnHGd DEAjIOWp0XIorxLxUNy8EQumZpSDdpCyh0zwiFSwHpG5ew+95GzFumpXpocIvdpOgEg+ PFPbuCDHdiCbROpJztGlxG2RbdaNBMw9UMjgQRDj2BddtpaTAmAwSUkzjmImvjAZIWyr pFYw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=D4rH6smFnaAwY/6ORlOFG01dtPv1lp7TmW85M9JETTE=; b=Q1jF8Q21HADofe1mzDXbq/qwlLez3lfU3ygNNcqRfyZ2q8SFhMuc6XGQ+XUXkTbFJc 2v/IJFqi1SqpkkfacqS/FKDeSM3Bra7hxyufCuDU90hfneXzes6qbngB4xxYVelXjM/E yCDahwT1LQDt31IUMTCVLQXqa7aewFfnJ4phk0rAVLm+JnFoImUe/uaDxEC5tNRRjdmT JxU+Cw0u4cyPan8hynAQMaNLf41G+4uqkq1akcv1ZWkCd8MRHRQm8qQX27otQJ591rly LVLQU+QrB2db95dbahW8daNyUzIjWqkBR6kgPW8XQsRs+YHssfMfdlICqlFvBcpS0j63 ZrAw== X-Gm-Message-State: APjAAAXeyNmO72V/XlR6IYFzSYkW2+Tz0Z51i3W//JPWu1zkdlzNVUn+ 4oIPyJiLYg05b1Q4qAPxbl7dCUZJymmZiJ7ac/8= X-Google-Smtp-Source: APXvYqyEr9vXvHwSv85wOpwzs9WnBN52OGdzq8fHwLn+JA+9DQ/y/kTq7KbuPpzLIOanHqd9+L1qsx1bzmahCg0P2dU= X-Received: by 2002:ac8:2fd4:: with SMTP id m20mr3833780qta.313.1582893560155; Fri, 28 Feb 2020 04:39:20 -0800 (PST) MIME-Version: 1.0 References: <20200227204150.30985-1-arunisaac@HIDDEN> <87eeuf5dty.fsf@HIDDEN> In-Reply-To: <87eeuf5dty.fsf@HIDDEN> From: zimoun <zimon.toutoune@HIDDEN> Date: Fri, 28 Feb 2020 13:39:08 +0100 Message-ID: <CAJ3okZ1twVzgfxRKv8MyNYY17Davd7QdkVHSp9+jb_9FV35HhQ@HIDDEN> Subject: Re: [PATCH 0/4] Xapian for Guix package search To: Pierre Neidhardt <mail@HIDDEN> Content-Type: text/plain; charset="UTF-8" X-Spam-Score: 2.0 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Hi Pierre, On Fri, 28 Feb 2020 at 09:13, Pierre Neidhardt wrote: > Beside this issue, how do you test it? I guess we first need to install > a bunch of package with `pre-inst-env guix ...` then to a `pre-inst-env search`? Content analysis details: (2.0 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- 0.0 URIBL_BLOCKED ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [URIs: ambrevar.xyz] 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider (zimon.toutoune[at]gmail.com) -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record 2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: ambrevar.xyz (xyz)] 0.0 RCVD_IN_MSPIKE_H3 RBL: Good reputation (+3) [209.85.160.195 listed in wl.mailspike.net] -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [209.85.160.195 listed in list.dnswl.org] 0.0 RCVD_IN_MSPIKE_WL Mailspike good senders X-Debbugs-Envelope-To: 39258 Cc: Arun Isaac <arunisaac@HIDDEN>, =?UTF-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>, 39258 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: 1.0 (+) Hi Pierre, On Fri, 28 Feb 2020 at 09:13, Pierre Neidhardt <mail@HIDDEN> wrote: > Beside this issue, how do you test it? I guess we first need to install > a bunch of package with `pre-inst-env guix ...` then to a `pre-inst-env search`? It is not searching in the installed packages but in all the packages. So, to test it, you need to "./pre-inst-env guix pull -p" or something like that to populate the Xapian index database. Then "./pre-inst-env guix search" will lookup into. I mean, it is how I understand it should work. I have not yet looked into the code. Cheers, simon
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 28 Feb 2020 12:36:25 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Fri Feb 28 07:36:25 2020 Received: from localhost ([127.0.0.1]:60742 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1j7esS-0002QM-O5 for submit <at> debbugs.gnu.org; Fri, 28 Feb 2020 07:36:24 -0500 Received: from mail-qk1-f196.google.com ([209.85.222.196]:38375) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <zimon.toutoune@HIDDEN>) id 1j7esR-0002Q3-81 for 39258 <at> debbugs.gnu.org; Fri, 28 Feb 2020 07:36:23 -0500 Received: by mail-qk1-f196.google.com with SMTP id z19so2747084qkj.5 for <39258 <at> debbugs.gnu.org>; Fri, 28 Feb 2020 04:36:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=ql2er9hVSfXLsVg7SCcUz2vhrbkIG/mO0k+XJT6hGfY=; b=ND/PEar5AF11ptlSDxx4urVpmUTf3yz06oMpUdYsGRuPrZhkRsiUfmisWnpBy91XK6 6cY/+5wgMJDy5K/mGa6yCsgqpAYIzaJ0mRAkomXtbpjd5jrRS04ua/bI+8uLM+HXEDWY dYeBZArdF561gfkPQQG/DQ5kEksHYqhvST85/Cpb9IEKwbrKbeZC8RaAl6dN7OsgKkDE LbvotAW+w3on+k63YaY6OmrtCXc1+nEO43MEjOfUVG8HFi5KeRabMnq8Z4ltrYS1FHSa eVo/mb/rg8oQVsySxEjQTjZpqmQ1HKIDiWOdhxg4YlnSKbYQF3X5pp63VLq3vrFDYY4e 0asQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=ql2er9hVSfXLsVg7SCcUz2vhrbkIG/mO0k+XJT6hGfY=; b=M8CrjwjzP0ogvzMuQqGjUX9EaBvaELImAcGanWr5tFNOt0PFO3gsQQou7NwpGyU4s5 Dnl1HxK/Yv/xf9lIU//KNMARfpJ2Wz1RCRN8pzbd3ilDatTny0yEX/rGicdkM6Asre18 DXk3Y3qHJhg/Ju1uUW0kS8KboxwY5o4nFrUdVxt/5wWZgsJCmYAdxobV/r/DZgE0XxBd UIpoxIwKgr15F7kRINC3E9NNxai6Ngx/Csq8wKTWeYJ6GyD5AbmboJOJm5xUaK7YUJyv 8O3WiWRm6vx/dLEJzdehfHGWC/YpciX4xLhD5POfQNgtslWfzOP/3D1G3t91RJoyEE1a rBKA== X-Gm-Message-State: APjAAAVrw1H4sZ5b4b7uQgo/PmiG31IteFdrrULGYWANEjCbUm6klt6G BsF2zwq5pNKldCMoARkfmY/XIWiqXoJ+pYGiZZDitg== X-Google-Smtp-Source: APXvYqwajaO8P+myNqSuUgKjfMI8P9Ah2ShKWY8GFKYITMKKQuN4I8BQMd7PVGXY5uqfviqwHW6eFbc2qJcw7W9izvM= X-Received: by 2002:a05:620a:a45:: with SMTP id j5mr4310682qka.232.1582893377553; Fri, 28 Feb 2020 04:36:17 -0800 (PST) MIME-Version: 1.0 References: <20200227204150.30985-1-arunisaac@HIDDEN> In-Reply-To: <20200227204150.30985-1-arunisaac@HIDDEN> From: zimoun <zimon.toutoune@HIDDEN> Date: Fri, 28 Feb 2020 13:36:06 +0100 Message-ID: <CAJ3okZ1cda2ypFfo550C=DvvsqagYsweKi1xH-8kbQSBGYXVFA@HIDDEN> Subject: Re: [PATCH 0/4] Xapian for Guix package search To: Arun Isaac <arunisaac@HIDDEN> Content-Type: text/plain; charset="UTF-8" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 39258 Cc: =?UTF-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>, Pierre Neidhardt <mail@HIDDEN>, 39258 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) Hi Arun, Really cool! Thank you! On Thu, 27 Feb 2020 at 21:42, Arun Isaac <arunisaac@HIDDEN> wrote: > * Speed improvement > > Despite search-package-index in gnu/packages.scm taking only around 1.5ms, I > see an overall speedup in `guix search` of only a factor of 2 -- from around > 2s to around 1s. I wonder what else in `guix search` is taking up so much > time. Interesting... maybe an hidden 'fold-packages'? Well, I have not yet looked into your code. > * Currently indexing only the package descriptions > > In this patchset, I have only indexed the package descriptions. In the next > version of this patchset, I will index all other terms as specified in > %package-metrics of guix/ui.scm. Yes, it appears to me a detail that should be easy to fix. I mean, it does not seems blocking. > * Should I add guile-xapian as a propagated input to guix in > gnu/packages/package-management.scm? IMHO, yes. I mean, I guess. :-) > * Drop regexp search support > > In this patchset, I have retained the older regexp search support. But, I > think we should drop it and only have xapian search. In cases where the search > index is not authoritative, we can build an in-memory xapian search index on > the fly and use it to search. This will slow down the search, but will ensure > our search results are consistent and do not depend on the authoritativeness > of the search index. I understand why you have turned off the regexp support. It is not necessary at the first experimentation to see if it is worth the addition or not. So, before investigating how some better regexp could be used with Xapian, let start to benchmark Xapian vs plain 'fold-packages'. > * Commit messages > > Except for patch 1, I am not sure what prefixes (build-self, gnu, etc.) to use > in the first line of the commit message. Some advice there would be helpful. I cannot help. )-: All the best, simon
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.
Received: (at 39258) by debbugs.gnu.org; 28 Feb 2020 08:13:55 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Feb 28 03:13:55 2020
Received: from localhost ([127.0.0.1]:60612 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1j7amR-0005ko-6V
for submit <at> debbugs.gnu.org; Fri, 28 Feb 2020 03:13:55 -0500
Received: from relay1-d.mail.gandi.net ([217.70.183.193]:54317)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <mail@HIDDEN>) id 1j7amP-0005ka-1V
for 39258 <at> debbugs.gnu.org; Fri, 28 Feb 2020 03:13:53 -0500
X-Originating-IP: 92.169.129.147
Received: from bababa (lfbn-idf2-1-1315-147.w92-169.abo.wanadoo.fr
[92.169.129.147]) (Authenticated sender: mail@HIDDEN)
by relay1-d.mail.gandi.net (Postfix) with ESMTPSA id EFE6124000C;
Fri, 28 Feb 2020 08:13:45 +0000 (UTC)
From: Pierre Neidhardt <mail@HIDDEN>
To: Arun Isaac <arunisaac@HIDDEN>, 39258 <at> debbugs.gnu.org
Subject: Re: [PATCH 0/4] Xapian for Guix package search
In-Reply-To: <20200227204150.30985-1-arunisaac@HIDDEN>
References: <20200227204150.30985-1-arunisaac@HIDDEN>
Date: Fri, 28 Feb 2020 09:13:45 +0100
Message-ID: <87eeuf5dty.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="=-=-=";
micalg=pgp-sha256; protocol="application/pgp-signature"
X-Spam-Score: 1.8 (+)
X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org",
has NOT identified this incoming email as spam. The original
message has been attached to this so you can view it or label
similar future email. If you have any questions, see
the administrator of that system for details.
Content preview: Fantastic, thank you so much for this great feature! I can't
build your patch though: --8<cut herestart>8--- [ 30%] LOAD
guix/scripts/package.scm
;;; note: source file ./guix/scripts/package.scm ;;; newer than compiled
/home/ambrevar/projects/ [...]
Content analysis details: (1.8 points, 10.0 required)
pts rule name description
---- ---------------------- --------------------------------------------------
0.0 URIBL_BLOCKED ADMINISTRATOR NOTICE: The query to URIBL was
blocked. See
http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block
for more information. [URIs: ambrevar.xyz]
-0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at https://www.dnswl.org/,
low trust [217.70.183.193 listed in list.dnswl.org]
0.0 RCVD_IN_MSPIKE_H3 RBL: Good reputation (+3)
[217.70.183.193 listed in wl.mailspike.net]
-0.0 SPF_PASS SPF: sender matches SPF record
0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record
2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs
[URI: ambrevar.xyz (xyz)]
0.0 RCVD_IN_MSPIKE_WL Mailspike good senders
0.5 FROM_SUSPICIOUS_NTLD From abused NTLD
X-Debbugs-Envelope-To: 39258
Cc: Arun Isaac <arunisaac@HIDDEN>, ludo@HIDDEN,
zimon.toutoune@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: 1.8 (+)
X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org",
has NOT identified this incoming email as spam. The original
message has been attached to this so you can view it or label
similar future email. If you have any questions, see
the administrator of that system for details.
Content preview: Fantastic, thank you so much for this great feature! I can't
build your patch though: --8<cut herestart>8--- [ 30%] LOAD guix/scripts/package.scm
;;; note: source file ./guix/scripts/package.scm ;;; newer than compiled
/home/ambrevar/projects/ [...]
Content analysis details: (1.8 points, 10.0 required)
pts rule name description
---- ---------------------- --------------------------------------------------
0.0 URIBL_BLOCKED ADMINISTRATOR NOTICE: The query to URIBL was
blocked. See
http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block
for more information.
[URIs: ambrevar.xyz]
-0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at https://www.dnswl.org/,
low trust
[217.70.183.193 listed in list.dnswl.org]
0.0 RCVD_IN_MSPIKE_H3 RBL: Good reputation (+3)
[217.70.183.193 listed in wl.mailspike.net]
-0.0 SPF_PASS SPF: sender matches SPF record
0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record
2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs
[URI: ambrevar.xyz (xyz)]
1.0 BULK_RE_SUSP_NTLD Precedence bulk and RE: from a suspicious TLD
-1.0 MAILING_LIST_MULTI Multiple indicators imply a widely-seen list
manager
0.0 RCVD_IN_MSPIKE_WL Mailspike good senders
0.5 FROM_SUSPICIOUS_NTLD From abused NTLD
--=-=-=
Content-Type: text/plain
Content-Transfer-Encoding: quoted-printable
Fantastic, thank you so much for this great feature!
I can't build your patch though:
=2D-8<---------------cut here---------------start------------->8---
[ 30%] LOAD guix/scripts/package.scm
;;; note: source file ./guix/scripts/package.scm
;;; newer than compiled /home/ambrevar/projects/guix/guix/scripts/pac=
kage.go
;;; note: source file ./guix/scripts/package.scm
;;; newer than compiled /home/ambrevar/projects/guix/guix/scripts/pac=
kage.go
;;; note: source file ./gnu/packages.scm
;;; newer than compiled /home/ambrevar/projects/guix/gnu/packages.go
;;; note: source file ./gnu/packages.scm
;;; newer than compiled /home/ambrevar/projects/guix/gnu/packages.go
error: failed to load 'gnu/packages.scm':
ice-9/eval.scm:293:34: no code for module (xapian xapian)
=2D-8<---------------cut here---------------end--------------->8---
Beside this issue, how do you test it? I guess we first need to install
a bunch of package with `pre-inst-env guix ...` then to a `pre-inst-env sea=
rch`?
=2D-=20
Pierre Neidhardt
https://ambrevar.xyz/
--=-=-=
Content-Type: application/pgp-signature; name="signature.asc"
-----BEGIN PGP SIGNATURE-----
iQEzBAEBCAAdFiEEUPM+LlsMPZAEJKvom9z0l6S7zH8FAl5Yy7kACgkQm9z0l6S7
zH8QWwf+IGDJgU0kdoe0i5VpNxWdDDL7J+MOttmJhSsJEDS1jMG6XTEPC8UPoTtM
ftuf5t5LB+KEispRXIsg5mIUwd2qGvP9N5dbEmKqRrnpLyD6T8qalH9neYAN+++F
Jed8csUTyrMD7BlseFOEFzUrQCiPabHK/+hOWfP5rAsecwr07NYt5yZPLgqgqLQU
HMMJGjGzG8Zd2FEZGiRFhINlRh+WD8rMCyLmIsRNjXQDY153LPzC1KqCNHE2fIIs
5Naxy2dejwD0hos5Nn7EF9F8GiecnKKnludcQmFa1oCE7mZrGL0JPqrJZa+ZhpZE
N9kbwFiswZVu+avqGKi/FTxdZdF2KA==
=XIu9
-----END PGP SIGNATURE-----
--=-=-=--
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.
Received: (at 39258) by debbugs.gnu.org; 28 Feb 2020 08:11:40 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Feb 28 03:11:40 2020
Received: from localhost ([127.0.0.1]:60608 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1j7akF-0005hH-Nt
for submit <at> debbugs.gnu.org; Fri, 28 Feb 2020 03:11:39 -0500
Received: from relay7-d.mail.gandi.net ([217.70.183.200]:35817)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <mail@HIDDEN>) id 1j7akE-0005h2-2X
for 39258 <at> debbugs.gnu.org; Fri, 28 Feb 2020 03:11:38 -0500
X-Originating-IP: 92.169.129.147
Received: from bababa (lfbn-idf2-1-1315-147.w92-169.abo.wanadoo.fr
[92.169.129.147]) (Authenticated sender: mail@HIDDEN)
by relay7-d.mail.gandi.net (Postfix) with ESMTPSA id 9A4A120004;
Fri, 28 Feb 2020 08:11:31 +0000 (UTC)
From: Pierre Neidhardt <mail@HIDDEN>
To: Arun Isaac <arunisaac@HIDDEN>
Subject: Re: [bug#39258] [PATCH 4/4] gnu: Use xapian index for package search.
References: <20200227204150.30985-1-arunisaac@HIDDEN>
<20200227204150.30985-5-arunisaac@HIDDEN>
Date: Fri, 28 Feb 2020 09:11:31 +0100
In-Reply-To: <20200227204150.30985-5-arunisaac@HIDDEN> (Arun Isaac's
message of "Fri, 28 Feb 2020 02:11:50 +0530")
Message-ID: <87h7zb5dxo.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux)
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="=-=-=";
micalg=pgp-sha256; protocol="application/pgp-signature"
X-Spam-Score: 1.8 (+)
X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org",
has NOT identified this incoming email as spam. The original
message has been attached to this so you can view it or label
similar future email. If you have any questions, see
the administrator of that system for details.
Content preview: Arun Isaac <arunisaac@HIDDEN> writes: > @@ -453, 6
+454,
20 @@ reducing the memory footprint." > > db-path) > > +(define
(search-package-index
profile querystring) Maybe `query-string'?
Content analysis details: (1.8 points, 10.0 required)
pts rule name description
---- ---------------------- --------------------------------------------------
0.0 URIBL_BLOCKED ADMINISTRATOR NOTICE: The query to URIBL was
blocked. See
http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block
for more information. [URIs: tobias.gr]
-0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at https://www.dnswl.org/,
low trust [217.70.183.200 listed in list.dnswl.org]
-0.0 SPF_PASS SPF: sender matches SPF record
0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record
2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs
[URI: ambrevar.xyz (xyz)]
-0.0 RCVD_IN_MSPIKE_H2 RBL: Average reputation (+2)
[217.70.183.200 listed in wl.mailspike.net]
0.5 FROM_SUSPICIOUS_NTLD From abused NTLD
X-Debbugs-Envelope-To: 39258
Cc: ludo@HIDDEN, 39258 <at> debbugs.gnu.org, zimon.toutoune@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: 1.8 (+)
X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org",
has NOT identified this incoming email as spam. The original
message has been attached to this so you can view it or label
similar future email. If you have any questions, see
the administrator of that system for details.
Content preview: Arun Isaac <arunisaac@HIDDEN> writes: > @@ -453,6
+454,20 @@ reducing the memory footprint." > > db-path) > > +(define (search-package-index
profile querystring) Maybe `query-string'?
Content analysis details: (1.8 points, 10.0 required)
pts rule name description
---- ---------------------- --------------------------------------------------
0.0 URIBL_BLOCKED ADMINISTRATOR NOTICE: The query to URIBL was
blocked. See
http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block
for more information.
[URIs: uzh.ch]
-0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at https://www.dnswl.org/,
low trust
[217.70.183.200 listed in list.dnswl.org]
-0.0 RCVD_IN_MSPIKE_H2 RBL: Average reputation (+2)
[217.70.183.200 listed in wl.mailspike.net]
-0.0 SPF_PASS SPF: sender matches SPF record
0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record
2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs
[URI: ambrevar.xyz (xyz)]
1.0 BULK_RE_SUSP_NTLD Precedence bulk and RE: from a suspicious TLD
-1.0 MAILING_LIST_MULTI Multiple indicators imply a widely-seen list
manager
0.5 FROM_SUSPICIOUS_NTLD From abused NTLD
--=-=-=
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Arun Isaac <arunisaac@HIDDEN> writes:
> @@ -453,6 +454,20 @@ reducing the memory footprint."
>=20=20
> db-path)
>=20=20
> +(define (search-package-index profile querystring)
Maybe `query-string'?
>
> --- a/guix/scripts/package.scm
> +++ b/guix/scripts/package.scm
> @@ -7,6 +7,7 @@
> ;;; Copyright =C2=A9 2016 Benz Schenk <benz.schenk@HIDDEN>
> ;;; Copyright =C2=A9 2016 Chris Marusich <cmmarusich@HIDDEN>
> ;;; Copyright =C2=A9 2019 Tobias Geerinckx-Rice <me@HIDDEN>
> +;;; Copyright =C2=A9 2020 Arun Isaac <arunisaac@HIDDEN>
> ;;;
> ;;; This file is part of GNU Guix.
> ;;;
> @@ -178,31 +179,40 @@ hooks\" run when building the profile."
> ;;; Package specifications.
> ;;;
>=20=20
> -(define (find-packages-by-description regexps)
> +(define (find-packages-by-description patterns)
> "Return a list of pairs: packages whose name, synopsis, description,
> or output matches at least one of REGEXPS sorted by relevance, and its
> non-zero relevance score."
Need to update the docstring.
> - (let ((matches (fold-packages (lambda (package result)
> - (if (package-superseded package)
> - result
> - (match (package-relevance package
> - regexps)
> - ((? zero?)
> - result)
> - (score
> - (cons (cons package score)
> - result)))))
> - '())))
> - (sort matches
> - (lambda (m1 m2)
> - (match m1
> - ((package1 . score1)
> - (match m2
> - ((package2 . score2)
> - (if (=3D score1 score2)
> - (string>? (package-full-name package1)
> - (package-full-name package2))
> - (> score1 score2))))))))))
> + (define (regexp? str)
> + (string-any
> + (char-set #\. #\[ #\{ #\} #\( #\) #\\ #\* #\+ #\? #\| #\^ #\$)
> + str))
> +
> + (if (and (current-profile)
> + (not (any regexp? patterns)))
I would not put characters like ".", "$", or "+" here, lest we mistake a
Xapian pattern for a regexp.
As you said, I don't think both are compatible without ambiguity
anyways, so we should probably drop regexp (or at least toggle them with
a command line argument).
=2D-=20
Pierre Neidhardt
https://ambrevar.xyz/
--=-=-=
Content-Type: application/pgp-signature; name="signature.asc"
-----BEGIN PGP SIGNATURE-----
iQEzBAEBCAAdFiEEUPM+LlsMPZAEJKvom9z0l6S7zH8FAl5YyzMACgkQm9z0l6S7
zH8myAf/dd2HZVdNaDmPZ/rEzf9eveD0wrhxS4pEwO8lJyBnobVHXfzcpDQ5ZjRx
u/ZWqMIjbmjz8VFAPvQGMDIxQubxoDXii5ps94ZQNgitlhJfb4qq8REHC5rhuZHY
bGxq4qTGKQYCXC3Yakg/uhRlQH4PhYvhVZDgWreJ2ay19JQV4fnfDeshdCq/oAUW
IGti/XiBt50KWOBmRIctI3hYhEdA1mISQqh4RoPA9xKEQvnWSwS5hs1OZwcRCzGi
mL3tgY7OKOmlojVVDCVo/r8Q9oXKlNiy+/sSilGCvO5AbsyFMsE3DqFYbQdFKpZq
wFyI6XkXFHHmiKSVgYLct89Qc5Lgiw==
=tOlQ
-----END PGP SIGNATURE-----
--=-=-=--
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.
Received: (at 39258) by debbugs.gnu.org; 28 Feb 2020 08:04:49 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Feb 28 03:04:49 2020
Received: from localhost ([127.0.0.1]:60596 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1j7add-0005Vd-Fx
for submit <at> debbugs.gnu.org; Fri, 28 Feb 2020 03:04:49 -0500
Received: from relay7-d.mail.gandi.net ([217.70.183.200]:41951)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <mail@HIDDEN>) id 1j7adb-0005VO-LG
for 39258 <at> debbugs.gnu.org; Fri, 28 Feb 2020 03:04:48 -0500
X-Originating-IP: 92.169.129.147
Received: from bababa (lfbn-idf2-1-1315-147.w92-169.abo.wanadoo.fr
[92.169.129.147]) (Authenticated sender: mail@HIDDEN)
by relay7-d.mail.gandi.net (Postfix) with ESMTPSA id 92A9A20007;
Fri, 28 Feb 2020 08:04:40 +0000 (UTC)
From: Pierre Neidhardt <mail@HIDDEN>
To: Arun Isaac <arunisaac@HIDDEN>
Subject: Re: [bug#39258] [PATCH 3/4] gnu: Generate xapian package search index.
References: <20200227204150.30985-1-arunisaac@HIDDEN>
<20200227204150.30985-4-arunisaac@HIDDEN>
Date: Fri, 28 Feb 2020 09:04:39 +0100
In-Reply-To: <20200227204150.30985-4-arunisaac@HIDDEN> (Arun Isaac's
message of "Fri, 28 Feb 2020 02:11:49 +0530")
Message-ID: <87k1475e94.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux)
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="=-=-=";
micalg=pgp-sha256; protocol="application/pgp-signature"
X-Spam-Score: 1.8 (+)
X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org",
has NOT identified this incoming email as spam. The original
message has been attached to this so you can view it or label
similar future email. If you have any questions, see
the administrator of that system for details.
Content preview: Arun Isaac <arunisaac@HIDDEN> writes: > +(define
(generate-package-search-index directory) > + "Generate under DIRECTORY a
xapian index of all the available packages." > + (define db-path > +
(string-append directory %package-search-index [...]
Content analysis details: (1.8 points, 10.0 required)
pts rule name description
---- ---------------------- --------------------------------------------------
0.0 URIBL_BLOCKED ADMINISTRATOR NOTICE: The query to URIBL was
blocked. See
http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block
for more information. [URIs: systemreboot.net]
-0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at https://www.dnswl.org/,
low trust [217.70.183.200 listed in list.dnswl.org]
-0.0 RCVD_IN_MSPIKE_H2 RBL: Average reputation (+2)
[217.70.183.200 listed in wl.mailspike.net]
-0.0 SPF_PASS SPF: sender matches SPF record
0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record
2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs
[URI: ambrevar.xyz (xyz)]
0.5 FROM_SUSPICIOUS_NTLD From abused NTLD
X-Debbugs-Envelope-To: 39258
Cc: ludo@HIDDEN, 39258 <at> debbugs.gnu.org, zimon.toutoune@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: 1.8 (+)
X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org",
has NOT identified this incoming email as spam. The original
message has been attached to this so you can view it or label
similar future email. If you have any questions, see
the administrator of that system for details.
Content preview: Arun Isaac <arunisaac@HIDDEN> writes: > +(define
(generate-package-search-index directory) > + "Generate under DIRECTORY a
xapian index of all the available packages." > + (define db-path > + (string-append
directory %package-search-index [...]
Content analysis details: (1.8 points, 10.0 required)
pts rule name description
---- ---------------------- --------------------------------------------------
0.0 URIBL_BLOCKED ADMINISTRATOR NOTICE: The query to URIBL was
blocked. See
http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block
for more information.
[URIs: systemreboot.net]
-0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at https://www.dnswl.org/,
low trust
[217.70.183.200 listed in list.dnswl.org]
-0.0 RCVD_IN_MSPIKE_H2 RBL: Average reputation (+2)
[217.70.183.200 listed in wl.mailspike.net]
-0.0 SPF_PASS SPF: sender matches SPF record
0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record
2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs
[URI: ambrevar.xyz (xyz)]
1.0 BULK_RE_SUSP_NTLD Precedence bulk and RE: from a suspicious TLD
-1.0 MAILING_LIST_MULTI Multiple indicators imply a widely-seen list
manager
0.5 FROM_SUSPICIOUS_NTLD From abused NTLD
--=-=-=
Content-Type: text/plain
Content-Transfer-Encoding: quoted-printable
Arun Isaac <arunisaac@HIDDEN> writes:
> +(define (generate-package-search-index directory)
> + "Generate under DIRECTORY a xapian index of all the available packages=
."
> + (define db-path
> + (string-append directory %package-search-index))
> +
> + (mkdir-p (dirname db-path))
> + (call-with-writable-database db-path
> + (lambda (db)
> + (fold-packages (lambda (package _)
> + (let* ((idterm (string-append "Q" (package-name p=
ackage)))
> + (doc (make-document #:data (package-name p=
ackage)
> + #:terms `((,idterm . 0=
))))
> + (term-generator (make-term-generator #:ste=
m (make-stem "en")
> + #:doc=
ument doc)))
> + (index-text! term-generator (package-descriptio=
n package))
> + (replace-document! db idterm doc)))
I guess these non-functional functions (index-text!, replace-document!)
represent how Xapian works at the C++ level. Would it be possible to
make more functional bindings nonetheless?
=2D-=20
Pierre Neidhardt
https://ambrevar.xyz/
--=-=-=
Content-Type: application/pgp-signature; name="signature.asc"
-----BEGIN PGP SIGNATURE-----
iQEzBAEBCAAdFiEEUPM+LlsMPZAEJKvom9z0l6S7zH8FAl5YyZcACgkQm9z0l6S7
zH9EkAf/eCddrqNRJXNrLpUYJ2kOAuqshemvMAF4v0WSF3enTf7Y59qDF56FS9V7
SsvxrUL9CF1LzJAquLLSNUnTEDLcmLb1p6O/ksSq0ZMQ9Yr4MGBeLJU36m1+WGv3
n+03tUCFvSLDupAlyBxNxCX4QBmzGd79shu3X5NZLVp06G842chKFB6GXlQyFRhh
iJLDq1kynhsW6eZIPONvHqQxTzEfb8Wh5wbNbzPvvN8fIWKPoBpP5yxXHLDGC2vM
VcodopQOoXebt1XAh4WIz4Tr3KaF3/B5m9OunrdLmLj3ETZQwTbifkObhnPG7tOp
NI1rOx5CP47ckZtbdK9HnfvSkvO0Tw==
=UV3T
-----END PGP SIGNATURE-----
--=-=-=--
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.
Received: (at 39258) by debbugs.gnu.org; 27 Feb 2020 20:42:26 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Feb 27 15:42:26 2020
Received: from localhost ([127.0.0.1]:60356 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1j7PzG-0008JW-76
for submit <at> debbugs.gnu.org; Thu, 27 Feb 2020 15:42:26 -0500
Received: from mugam.systemreboot.net ([139.59.75.54]:37312)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <arunisaac@HIDDEN>) id 1j7Pz9-0008Iq-Qk
for 39258 <at> debbugs.gnu.org; Thu, 27 Feb 2020 15:42:20 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed;
d=systemreboot.net; s=default; h=Content-Transfer-Encoding:Content-Type:
MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender
:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From:
Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:
List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive;
bh=+qfmxXIuB58cypMqiYHpA/+3NJ3qUzwieaNRxXHkquQ=; b=HLAxgPtntu2rhEJ63A3jYxzkKN
ku6985TvNdc4Ql4xBCP7pLL18vHpXzsThc+7XQMLNcVzgH2VM1FzJcq9hWIZEJaY6WLhBmbpbyngF
qhGe+MNTvKvqMnAkKe21VZDHBiusvwYB1TwREWxV9KEE27tO+W4RZS6CohO4LRjawJB4=;
Received: from [192.168.2.1] (helo=steel.lan)
by systemreboot.net with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384
(Exim 4.93) (envelope-from <arunisaac@HIDDEN>)
id 1j7Pyz-001ckB-KX; Fri, 28 Feb 2020 02:12:09 +0530
From: Arun Isaac <arunisaac@HIDDEN>
To: 39258 <at> debbugs.gnu.org
Subject: [PATCH 2/4] build-self: Add guile-xapian to Guix dependencies.
Date: Fri, 28 Feb 2020 02:11:48 +0530
Message-Id: <20200227204150.30985-3-arunisaac@HIDDEN>
X-Mailer: git-send-email 2.23.0
In-Reply-To: <20200227204150.30985-1-arunisaac@HIDDEN>
References: <20200227204150.30985-1-arunisaac@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 39258
Cc: Arun Isaac <arunisaac@HIDDEN>, ludo@HIDDEN,
zimon.toutoune@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)
* build-aux/build-self.scm (build-program): Import fake guile-xapian module.
* guix/self.scm (compiled-guix): Add guile-xapian to Guix dependencies.
---
build-aux/build-self.scm | 11 +++++++++++
guix/self.scm | 7 ++++++-
2 files changed, 17 insertions(+), 1 deletion(-)
diff --git a/build-aux/build-self.scm b/build-aux/build-self.scm
index f2e785b7f1..05d0353ccf 100644
--- a/build-aux/build-self.scm
+++ b/build-aux/build-self.scm
@@ -1,5 +1,6 @@
;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2014, 2016, 2017, 2018, 2019, 2020 Ludovic Courtès <ludo@HIDDEN>
+;;; Copyright © 2020 Arun Isaac <arunisaac@HIDDEN>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -261,6 +262,10 @@ interface (FFI) of Guile.")
#~(define-module (gcrypt hash)
#:export (sha1 sha256))))
+ (define fake-xapian-hash
+ ;; Fake (xapian xapian) module; see below.
+ (scheme-file "xapian.scm" #~(define-module (xapian xapian))))
+
(define fake-git
(scheme-file "git.scm" #~(define-module (git))))
@@ -273,6 +278,12 @@ interface (FFI) of Guile.")
;; adjust %LOAD-PATH later on.
((gcrypt hash) => ,fake-gcrypt-hash)
+ ;; To avoid relying on 'with-extensions', which was
+ ;; introduced in 0.15.0, provide a fake (xapian
+ ;; xapian) just so that we can build modules, and
+ ;; adjust %LOAD-PATH later on.
+ ((xapian xapian) => ,fake-xapian-hash)
+
;; (guix git-download) depends on (git) but only
;; for peripheral functionality. Provide a dummy
;; (git) to placate it.
diff --git a/guix/self.scm b/guix/self.scm
index 6b633f9bc0..a4f40574d1 100644
--- a/guix/self.scm
+++ b/guix/self.scm
@@ -1,5 +1,6 @@
;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2017, 2018, 2019, 2020 Ludovic Courtès <ludo@HIDDEN>
+;;; Copyright © 2020 Arun Isaac <arunisaac@HIDDEN>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -54,6 +55,7 @@
("guile-git" (ref '(gnu packages guile) 'guile3.0-git))
("guile-sqlite3" (ref '(gnu packages guile) 'guile3.0-sqlite3))
("guile-gcrypt" (ref '(gnu packages gnupg) 'guile3.0-gcrypt))
+ ("guile-xapian" (ref '(gnu packages guile-xyz) 'guile3.0-xapian))
("gnutls" (ref '(gnu packages tls) 'guile3.0-gnutls))
("zlib" (ref '(gnu packages compression) 'zlib))
("lzlib" (ref '(gnu packages compression) 'lzlib))
@@ -682,6 +684,9 @@ Info manual."
(define guile-gcrypt
(specification->package "guile-gcrypt"))
+ (define guile-xapian
+ (specification->package "guile-xapian"))
+
(define gnutls
(specification->package "gnutls"))
@@ -690,7 +695,7 @@ Info manual."
(cons (list "x" package)
(package-transitive-propagated-inputs package)))
(list guile-gcrypt gnutls guile-git guile-json
- guile-ssh guile-sqlite3))
+ guile-ssh guile-sqlite3 guile-xapian))
(((labels packages _ ...) ...)
packages)))
--
2.23.0
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.
Received: (at 39258) by debbugs.gnu.org; 27 Feb 2020 20:42:25 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Feb 27 15:42:25 2020
Received: from localhost ([127.0.0.1]:60354 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1j7PzA-0008JD-J5
for submit <at> debbugs.gnu.org; Thu, 27 Feb 2020 15:42:25 -0500
Received: from mugam.systemreboot.net ([139.59.75.54]:37304)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <arunisaac@HIDDEN>) id 1j7Pz6-0008IN-LY
for 39258 <at> debbugs.gnu.org; Thu, 27 Feb 2020 15:42:18 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed;
d=systemreboot.net; s=default; h=Content-Transfer-Encoding:Content-Type:
MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender
:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From:
Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:
List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive;
bh=PvLHNWGiLSPRr+w84SBaVoddHcDGgiKIDnzT9hWFMVA=; b=C9dtcu9OhaIzJuaJbl+WCAdkfB
I2c0TImaQT9nBRXd2Wy1H8+1horlIoXo6c1yq13GJ/YokGc0Fj2ZDD5DL5nCrXhQrW1f2SQILiyUo
Owtd5nH0kq39SKPltOU3XmBGCH6I8CzRa3Gysjm9aqthHKkdXJdLsWE7qqg/CyxkEzR8=;
Received: from [192.168.2.1] (helo=steel.lan)
by systemreboot.net with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384
(Exim 4.93) (envelope-from <arunisaac@HIDDEN>)
id 1j7Pyz-001ckB-EN; Fri, 28 Feb 2020 02:12:09 +0530
From: Arun Isaac <arunisaac@HIDDEN>
To: 39258 <at> debbugs.gnu.org
Subject: [PATCH 1/4] gnu: Add guile-xapian.
Date: Fri, 28 Feb 2020 02:11:47 +0530
Message-Id: <20200227204150.30985-2-arunisaac@HIDDEN>
X-Mailer: git-send-email 2.23.0
In-Reply-To: <20200227204150.30985-1-arunisaac@HIDDEN>
References: <20200227204150.30985-1-arunisaac@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 39258
Cc: Arun Isaac <arunisaac@HIDDEN>, ludo@HIDDEN,
zimon.toutoune@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)
* gnu/packages/guile-xyz.scm (guile-xapian, guile3.0-xapian): New variables.
---
gnu/packages/guile-xyz.scm | 50 +++++++++++++++++++++++++++++++++++++-
1 file changed, 49 insertions(+), 1 deletion(-)
diff --git a/gnu/packages/guile-xyz.scm b/gnu/packages/guile-xyz.scm
index 37a5198e4e..75aba83593 100644
--- a/gnu/packages/guile-xyz.scm
+++ b/gnu/packages/guile-xyz.scm
@@ -17,7 +17,7 @@
;;; Copyright © 2017 ng0 <ng0@HIDDEN>
;;; Copyright © 2017, 2018 Tobias Geerinckx-Rice <me@HIDDEN>
;;; Copyright © 2018 Maxim Cournoyer <maxim.cournoyer@HIDDEN>
-;;; Copyright © 2018, 2019 Arun Isaac <arunisaac@HIDDEN>
+;;; Copyright © 2018, 2019, 2020 Arun Isaac <arunisaac@HIDDEN>
;;; Copyright © 2018 Pierre-Antoine Rouby <pierre-antoine.rouby@HIDDEN>
;;; Copyright © 2018 Eric Bavier <bavier@HIDDEN>
;;; Copyright © 2019 swedebugia <swedebugia@HIDDEN>
@@ -80,8 +80,10 @@
#:use-module (gnu packages python)
#:use-module (gnu packages readline)
#:use-module (gnu packages sdl)
+ #:use-module (gnu packages search)
#:use-module (gnu packages slang)
#:use-module (gnu packages sqlite)
+ #:use-module (gnu packages swig)
#:use-module (gnu packages tex)
#:use-module (gnu packages texinfo)
#:use-module (gnu packages tls)
@@ -3109,3 +3111,49 @@ currently a re-implementation of the lentes library for Clojure. Lenses
provide composable procedures, which can be used to focus, apply functions
over, or update a value in arbitrary data structures.")
(license license:gpl3+))))
+
+(define-public guile-xapian
+ (let ((commit "bfad1b0e2a88bfe1d4c100046da0d585b96d2a73")
+ (revision "1"))
+ (package
+ (name "guile-xapian")
+ (version (git-version "0.1.0" revision commit))
+ (home-page "https://git.systemreboot.net/guile-xapian")
+ (source
+ (origin
+ (method git-fetch)
+ (uri (git-reference (url home-page)
+ (commit commit)))
+ (file-name (git-file-name name version))
+ (sha256
+ (base32
+ "1nrs23abb0lxx7gw14jw5k8jgbma0gi21gzahw0jgv6b25d9jdwp"))))
+ (build-system gnu-build-system)
+ (arguments
+ '(#:make-flags '("GUILE_AUTO_COMPILE=0"))) ; to prevent guild warnings
+ (inputs
+ `(("guile" ,guile-2.2)
+ ("xapian" ,xapian)
+ ("zlib" ,zlib)))
+ (native-inputs
+ `(("autoconf" ,autoconf)
+ ("autoconf-archive" ,autoconf-archive)
+ ("automake" ,automake)
+ ("libtool" ,libtool)
+ ("pkg-config" ,pkg-config)
+ ("swig" ,swig)))
+ (synopsis "Guile bindings for Xapian")
+ (description "@code{guile-xapian} provides Guile bindings for Xapian, a
+search engine library. Xapian is a highly adaptable toolkit which allows
+developers to easily add advanced indexing and search facilities to their own
+applications. It has built-in support for several families of weighting
+models and also supports a rich set of boolean query operators.")
+ (license license:gpl2+))))
+
+(define-public guile3.0-xapian
+ (package
+ (inherit guile-xapian)
+ (name "guile3.0-xapian")
+ (inputs
+ `(("guile" ,guile-next)
+ ,@(alist-delete "guile" (package-inputs guile-xapian))))))
--
2.23.0
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.
Received: (at 39258) by debbugs.gnu.org; 27 Feb 2020 20:42:20 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Feb 27 15:42:20 2020
Received: from localhost ([127.0.0.1]:60351 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1j7PzA-0008J1-0f
for submit <at> debbugs.gnu.org; Thu, 27 Feb 2020 15:42:20 -0500
Received: from mugam.systemreboot.net ([139.59.75.54]:37294)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <arunisaac@HIDDEN>) id 1j7Pz5-0008II-Gz
for 39258 <at> debbugs.gnu.org; Thu, 27 Feb 2020 15:42:18 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed;
d=systemreboot.net; s=default; h=Content-Transfer-Encoding:Content-Type:
MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender
:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From:
Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:
List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive;
bh=sQEq9FYeMCTpeCbSa0BT6wsOvHihjAdebVbxE4b6Xv0=; b=bKhUzRcGGgmRLO/tm9x6FZnZcW
bThUjGRDJfczKHWi5PQTz+OMXPgjEoQz6nd8m0hlmkhQmv24gRbDHZAM7YWZW/oeMGiQxw7YsFMx0
Nxi30o2OIP32Xu1mksoUUGmUhblw11KvXPZh8tDxgt0tXEjlNFRtLJjS8EcF8siBXvW0=;
Received: from [192.168.2.1] (helo=steel.lan)
by systemreboot.net with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384
(Exim 4.93) (envelope-from <arunisaac@HIDDEN>)
id 1j7Pyz-001ckB-W4; Fri, 28 Feb 2020 02:12:10 +0530
From: Arun Isaac <arunisaac@HIDDEN>
To: 39258 <at> debbugs.gnu.org
Subject: [PATCH 4/4] gnu: Use xapian index for package search.
Date: Fri, 28 Feb 2020 02:11:50 +0530
Message-Id: <20200227204150.30985-5-arunisaac@HIDDEN>
X-Mailer: git-send-email 2.23.0
In-Reply-To: <20200227204150.30985-1-arunisaac@HIDDEN>
References: <20200227204150.30985-1-arunisaac@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 39258
Cc: Arun Isaac <arunisaac@HIDDEN>, ludo@HIDDEN,
zimon.toutoune@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)
* gnu/packages.scm (search-package-index): New function.
* guix/scripts/package.scm (find-packages-by-description): Search using the
xapian package index if search patterns are literal strings. Else, search
using fold-packages.
---
gnu/packages.scm | 17 +++++++++++-
guix/scripts/package.scm | 57 +++++++++++++++++++++++-----------------
2 files changed, 49 insertions(+), 25 deletions(-)
diff --git a/gnu/packages.scm b/gnu/packages.scm
index e91753e2a8..5b5b29bf84 100644
--- a/gnu/packages.scm
+++ b/gnu/packages.scm
@@ -67,7 +67,8 @@
specifications->manifest
generate-package-cache
- generate-package-search-index))
+ generate-package-search-index
+ search-package-index))
;;; Commentary:
;;;
@@ -453,6 +454,20 @@ reducing the memory footprint."
db-path)
+(define (search-package-index profile querystring)
+ (let ((offset 0)
+ (pagesize 10))
+ (call-with-database (string-append profile %package-search-index)
+ (lambda (db)
+ (let ((query (parse-query querystring #:stemmer (make-stem "en"))))
+ (mset-fold (lambda (item result)
+ (match (find-packages-by-name
+ (document-data (mset-item-document item)))
+ ((package _ ...)
+ (append result `((,package . ,(mset-item-weight item)))))))
+ '()
+ (enquire-mset (enquire db query) offset pagesize)))))))
+
(define %sigint-prompt
;; The prompt to jump to upon SIGINT.
diff --git a/guix/scripts/package.scm b/guix/scripts/package.scm
index 1cb0d382bf..6a3b9002dd 100644
--- a/guix/scripts/package.scm
+++ b/guix/scripts/package.scm
@@ -7,6 +7,7 @@
;;; Copyright © 2016 Benz Schenk <benz.schenk@HIDDEN>
;;; Copyright © 2016 Chris Marusich <cmmarusich@HIDDEN>
;;; Copyright © 2019 Tobias Geerinckx-Rice <me@HIDDEN>
+;;; Copyright © 2020 Arun Isaac <arunisaac@HIDDEN>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -178,31 +179,40 @@ hooks\" run when building the profile."
;;; Package specifications.
;;;
-(define (find-packages-by-description regexps)
+(define (find-packages-by-description patterns)
"Return a list of pairs: packages whose name, synopsis, description,
or output matches at least one of REGEXPS sorted by relevance, and its
non-zero relevance score."
- (let ((matches (fold-packages (lambda (package result)
- (if (package-superseded package)
- result
- (match (package-relevance package
- regexps)
- ((? zero?)
- result)
- (score
- (cons (cons package score)
- result)))))
- '())))
- (sort matches
- (lambda (m1 m2)
- (match m1
- ((package1 . score1)
- (match m2
- ((package2 . score2)
- (if (= score1 score2)
- (string>? (package-full-name package1)
- (package-full-name package2))
- (> score1 score2))))))))))
+ (define (regexp? str)
+ (string-any
+ (char-set #\. #\[ #\{ #\} #\( #\) #\\ #\* #\+ #\? #\| #\^ #\$)
+ str))
+
+ (if (and (current-profile)
+ (not (any regexp? patterns)))
+ (search-package-index (current-profile) (string-join patterns " "))
+ (let* ((regexps (map (cut make-regexp* <> regexp/icase) patterns))
+ (matches (fold-packages (lambda (package result)
+ (if (package-superseded package)
+ result
+ (match (package-relevance package
+ regexps)
+ ((? zero?)
+ result)
+ (score
+ (cons (cons package score)
+ result)))))
+ '())))
+ (sort matches
+ (lambda (m1 m2)
+ (match m1
+ ((package1 . score1)
+ (match m2
+ ((package2 . score2)
+ (if (= score1 score2)
+ (string>? (package-full-name package1)
+ (package-full-name package2))
+ (> score1 score2)))))))))))
(define (transaction-upgrade-entry store entry transaction)
"Return a variant of TRANSACTION that accounts for the upgrade of ENTRY, a
@@ -777,8 +787,7 @@ processed, #f otherwise."
(('query 'search rx) rx)
(_ #f))
opts))
- (regexps (map (cut make-regexp* <> regexp/icase) patterns))
- (matches (find-packages-by-description regexps)))
+ (matches (find-packages-by-description patterns)))
(leave-on-EPIPE
(display-search-results matches (current-output-port)))
#t))
--
2.23.0
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.
Received: (at 39258) by debbugs.gnu.org; 27 Feb 2020 20:42:20 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Feb 27 15:42:20 2020
Received: from localhost ([127.0.0.1]:60349 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1j7Pz9-0008It-J1
for submit <at> debbugs.gnu.org; Thu, 27 Feb 2020 15:42:19 -0500
Received: from mugam.systemreboot.net ([139.59.75.54]:37300)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <arunisaac@HIDDEN>) id 1j7Pz5-0008IH-HA
for 39258 <at> debbugs.gnu.org; Thu, 27 Feb 2020 15:42:17 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed;
d=systemreboot.net; s=default; h=Content-Transfer-Encoding:Content-Type:
MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender
:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From:
Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:
List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive;
bh=SX6el7Fgocm4MXmlCLaq3jNgIZifmGn86XxWLvpUS/E=; b=YpPvLc1uI7RCn0XyQdwSVLArSP
wM2dvmmUKLal+6JVkofvThIgJu56cahVD4rsXbFb7zePWFPr8M26wadyb+8Rw+NfVNSOnkzJphQI9
ZZoZl6w6sj/n13AXvD1lfCp4WE5wO1BYJqmTNszqrd3NRnY3lqQl6jystDVl9Km/ymsM=;
Received: from [192.168.2.1] (helo=steel.lan)
by systemreboot.net with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384
(Exim 4.93) (envelope-from <arunisaac@HIDDEN>)
id 1j7Pyz-001ckB-QI; Fri, 28 Feb 2020 02:12:09 +0530
From: Arun Isaac <arunisaac@HIDDEN>
To: 39258 <at> debbugs.gnu.org
Subject: [PATCH 3/4] gnu: Generate xapian package search index.
Date: Fri, 28 Feb 2020 02:11:49 +0530
Message-Id: <20200227204150.30985-4-arunisaac@HIDDEN>
X-Mailer: git-send-email 2.23.0
In-Reply-To: <20200227204150.30985-1-arunisaac@HIDDEN>
References: <20200227204150.30985-1-arunisaac@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 39258
Cc: Arun Isaac <arunisaac@HIDDEN>, ludo@HIDDEN,
zimon.toutoune@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)
* gnu/packages.scm (%package-search-index): New variable.
(generate-package-search-index): New function.
* guix/channels.scm (package-search-index): New function.
(%channel-profile-hooks): Add package-search-index.
---
gnu/packages.scm | 29 ++++++++++++++++++++++++++++-
guix/channels.scm | 34 +++++++++++++++++++++++++++++++++-
2 files changed, 61 insertions(+), 2 deletions(-)
diff --git a/gnu/packages.scm b/gnu/packages.scm
index d22c992bb1..e91753e2a8 100644
--- a/gnu/packages.scm
+++ b/gnu/packages.scm
@@ -4,6 +4,7 @@
;;; Copyright © 2014 Eric Bavier <bavier@HIDDEN>
;;; Copyright © 2016, 2017 Alex Kost <alezost@HIDDEN>
;;; Copyright © 2016 Mathieu Lirzin <mthl@HIDDEN>
+;;; Copyright © 2020 Arun Isaac <arunisaac@HIDDEN>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -43,6 +44,7 @@
#:use-module (srfi srfi-34)
#:use-module (srfi srfi-35)
#:use-module (srfi srfi-39)
+ #:use-module (xapian xapian)
#:export (search-patch
search-patches
search-auxiliary-file
@@ -64,7 +66,8 @@
specification->location
specifications->manifest
- generate-package-cache))
+ generate-package-cache
+ generate-package-search-index))
;;; Commentary:
;;;
@@ -426,6 +429,30 @@ reducing the memory footprint."
#:opts '(#:to-file? #t)))))
cache-file)
+(define %package-search-index
+ ;; Location of the package search-index
+ "/lib/guix/package-search.index")
+
+(define (generate-package-search-index directory)
+ "Generate under DIRECTORY a xapian index of all the available packages."
+ (define db-path
+ (string-append directory %package-search-index))
+
+ (mkdir-p (dirname db-path))
+ (call-with-writable-database db-path
+ (lambda (db)
+ (fold-packages (lambda (package _)
+ (let* ((idterm (string-append "Q" (package-name package)))
+ (doc (make-document #:data (package-name package)
+ #:terms `((,idterm . 0))))
+ (term-generator (make-term-generator #:stem (make-stem "en")
+ #:document doc)))
+ (index-text! term-generator (package-description package))
+ (replace-document! db idterm doc)))
+ #f)))
+
+ db-path)
+
(define %sigint-prompt
;; The prompt to jump to upon SIGINT.
diff --git a/guix/channels.scm b/guix/channels.scm
index f0261dc2da..c70c70938c 100644
--- a/guix/channels.scm
+++ b/guix/channels.scm
@@ -2,6 +2,7 @@
;;; Copyright © 2018, 2019, 2020 Ludovic Courtès <ludo@HIDDEN>
;;; Copyright © 2018 Ricardo Wurmus <rekado@HIDDEN>
;;; Copyright © 2019 Jan (janneke) Nieuwenhuizen <janneke@HIDDEN>
+;;; Copyright © 2020 Arun Isaac <arunisaac@HIDDEN>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -581,9 +582,40 @@ be used as a profile hook."
(hook . package-cache))
#:local-build? #t)))
+(define (package-search-index manifest)
+ "Build a package search index for the instance in MANIFEST. This is meant
+to be used as a profile hook."
+ (mlet %store-monad ((profile (profile-derivation manifest
+ #:hooks '())))
+
+ (define build
+ #~(begin
+ (use-modules (gnu packages))
+
+ (if (defined? 'generate-package-search-index)
+ (begin
+ ;; Delegate package search index generation to the inferior.
+ (format (current-error-port)
+ "Generating package search index for '~a'...~%"
+ #$profile)
+ (generate-package-search-index #$output))
+ (mkdir #$output))))
+
+ (gexp->derivation-in-inferior "guix-package-search-index" build
+ profile
+
+ ;; If the Guix in PROFILE is too old and
+ ;; lacks 'guix repl', don't build the cache
+ ;; instead of failing.
+ #:silent-failure? #t
+
+ #:properties '((type . profile-hook)
+ (hook . package-search-index))
+ #:local-build? #t)))
+
(define %channel-profile-hooks
;; The default channel profile hooks.
- (cons package-cache-file %default-profile-hooks))
+ (cons* package-cache-file package-search-index %default-profile-hooks))
(define (channel-instances->derivation instances)
"Return the derivation of the profile containing INSTANCES, a list of
--
2.23.0
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 27 Feb 2020 20:42:19 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Thu Feb 27 15:42:19 2020 Received: from localhost ([127.0.0.1]:60347 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1j7Pz9-0008Ir-9E for submit <at> debbugs.gnu.org; Thu, 27 Feb 2020 15:42:19 -0500 Received: from mugam.systemreboot.net ([139.59.75.54]:37292) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <arunisaac@HIDDEN>) id 1j7Pz5-0008IJ-ID for 39258 <at> debbugs.gnu.org; Thu, 27 Feb 2020 15:42:17 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=systemreboot.net; s=default; h=Content-Transfer-Encoding:MIME-Version: Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To:Content-Type:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:In-Reply-To:References:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=KPrBpAstwRR8GKo437fM9sJZbczJYvO3b2sEglK3y98=; b=RYts7wfsaPjOlUx6vfUcCbiDRs vk8HYHoGWTlQYeaqqB9OlssSvTP6LfcsgANtkQLGDiuhXLY2Svt/p0K1M/ozRmr1/OKihN7nQsjq9 MARvcc5TCq755Vv4QqxrDmZ61vvGuyyaKmxg7Yh+ZHWXx8na5bEm4tiY/136NVuxwz/o=; Received: from [192.168.2.1] (helo=steel.lan) by systemreboot.net with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.93) (envelope-from <arunisaac@HIDDEN>) id 1j7Pyz-001ckB-3W; Fri, 28 Feb 2020 02:12:09 +0530 From: Arun Isaac <arunisaac@HIDDEN> To: 39258 <at> debbugs.gnu.org Subject: [PATCH 0/4] Xapian for Guix package search Date: Fri, 28 Feb 2020 02:11:46 +0530 Message-Id: <20200227204150.30985-1-arunisaac@HIDDEN> X-Mailer: git-send-email 2.23.0 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 39258 Cc: Arun Isaac <arunisaac@HIDDEN>, mail@HIDDEN, ludo@HIDDEN, zimon.toutoune@HIDDEN X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) Hi, I have finally got xapian working for package search. Some comments follow. * Speed improvement Despite search-package-index in gnu/packages.scm taking only around 1.5ms, I see an overall speedup in `guix search` of only a factor of 2 -- from around 2s to around 1s. I wonder what else in `guix search` is taking up so much time. * Currently indexing only the package descriptions In this patchset, I have only indexed the package descriptions. In the next version of this patchset, I will index all other terms as specified in %package-metrics of guix/ui.scm. * Should I add guile-xapian as a propagated input to guix in gnu/packages/package-management.scm? * Drop regexp search support In this patchset, I have retained the older regexp search support. But, I think we should drop it and only have xapian search. In cases where the search index is not authoritative, we can build an in-memory xapian search index on the fly and use it to search. This will slow down the search, but will ensure our search results are consistent and do not depend on the authoritativeness of the search index. * Commit messages Except for patch 1, I am not sure what prefixes (build-self, gnu, etc.) to use in the first line of the commit message. Some advice there would be helpful. Regards, Arun. Arun Isaac (4): gnu: Add guile-xapian. build-self: Add guile-xapian to Guix dependencies. gnu: Generate xapian package search index. gnu: Use xapian index for package search. build-aux/build-self.scm | 11 ++++++++ gnu/packages.scm | 44 ++++++++++++++++++++++++++++- gnu/packages/guile-xyz.scm | 50 ++++++++++++++++++++++++++++++++- guix/channels.scm | 34 ++++++++++++++++++++++- guix/scripts/package.scm | 57 ++++++++++++++++++++++---------------- guix/self.scm | 7 ++++- 6 files changed, 175 insertions(+), 28 deletions(-) -- 2.23.0
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 15 Feb 2020 14:50:47 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Sat Feb 15 09:50:47 2020 Received: from localhost ([127.0.0.1]:35128 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1j2ymN-0002aS-0W for submit <at> debbugs.gnu.org; Sat, 15 Feb 2020 09:50:47 -0500 Received: from mugam.systemreboot.net ([139.59.75.54]:36306) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <arunisaac@HIDDEN>) id 1j2ymJ-0002aG-Ku for 39258 <at> debbugs.gnu.org; Sat, 15 Feb 2020 09:50:45 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=systemreboot.net; s=default; h=Content-Type:MIME-Version:Message-ID:Date: References:In-Reply-To:Subject:Cc:To:From:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=cxAbkQo0H225gAuJTrAhPoO8Cpml86uU+2ZaHrN9oxc=; b=I6x/bTuC/o30gnC7wE/Nbwci0 1F2YJcyGGmHEH5cPqmAjjE5RLm67QDexPn3uq2i04NVpR4l7gLPDBFU5dj/QfuqpfXmqGSXi5ThXo zhGZl68lrAMARL8MRBIOM9oBp2CsSHJuW/rl9BQXYS47s21t+XMOnfKLbmxZU1ZDlLzWg=; Received: from [192.168.2.1] (helo=steel) by systemreboot.net with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.93) (envelope-from <arunisaac@HIDDEN>) id 1j2yly-000Q3t-0T; Sat, 15 Feb 2020 20:20:22 +0530 From: Arun Isaac <arunisaac@HIDDEN> To: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>, zimoun <zimon.toutoune@HIDDEN> Subject: Re: [bug#39258] Faster guix search using an sqlite cache In-Reply-To: <cu7zhdplz84.fsf@HIDDEN> References: <cu7pnfaar36.fsf@HIDDEN> <CAJ3okZ05JXVgbKJd_WeJKERJd6rc4SwxLRrRR0AyW48QDyX8eQ@HIDDEN> <cu74kwdawc7.fsf@HIDDEN> <CAJ3okZ0GXS9ibh__1DFsuCCjO82R6-YXtPAOF3GoquN=sSsaWA@HIDDEN> <cu7y2tkadwi.fsf@HIDDEN> <CAJ3okZ37rFh=SB-f6nqc0kMpCai=pw2D0-GFiakXY1p4FHHftA@HIDDEN> <cu7v9oka341.fsf@HIDDEN> <8736bhytn9.fsf@HIDDEN> <CAJ3okZ2EF5244EN3XcO-U9Uz8me42byuPMeUGeGQ_mWi6r864A@HIDDEN> <87sgjhx92g.fsf@HIDDEN> <cu7zhdplz84.fsf@HIDDEN> Date: Sat, 15 Feb 2020 20:20:21 +0530 Message-ID: <cu7h7zrucpe.fsf@HIDDEN> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 39258 Cc: Pierre Neidhardt <mail@HIDDEN>, 39258 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain > Also, I've started working on guile-xapian bindings. With that, it seems > simpler to keep the current package cache and add a xapian index only to > speed up package search. I have published the first version of guile-xapian! Feedback is welcome. :-) https://git.systemreboot.net/guile-xapian/about/ I will now move on to building a Xapian index for Guix's package search. --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEf3MDQ/Lwnzx3v3nTLiXui2GAK7MFAl5IBS0ACgkQLiXui2GA K7NmlQf/YWcix4M3tpDfOfHAyQiMGkgM++FG9f9tLdaDd3NJxcwnSzmiLPbKXwzJ c1UuAIq8LVyFqDvzp23ZWI2+61HrIFwpEO0ClZhh5/l1n1gHQaGNjxMthgD0GzMV zNenJISebJW2tlbV2+RTpaVKd6DqcN/Ww7hxZbPdETl72Y9gxuIx6Z5xUsqLrtTc QiSQo7My6aIG3dPJdnhMQUESga+2Sr2Cns8Z3j+NTKJQmpxipG4JDTgbyJ98tIH1 lT8V8opTbJ4kncQUG8FfheTWnoemWymc76uHLuR1XSQqNTvrlGB4goNKs1nBK0y7 T8ST01zuEQdDQcDR0z7OdCpoqA4fdw== =i8Qu -----END PGP SIGNATURE----- --=-=-=--
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 11 Feb 2020 20:20:56 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Tue Feb 11 15:20:56 2020 Received: from localhost ([127.0.0.1]:57623 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1j1c1g-0005dF-HE for submit <at> debbugs.gnu.org; Tue, 11 Feb 2020 15:20:56 -0500 Received: from mail-qk1-f195.google.com ([209.85.222.195]:39360) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <zimon.toutoune@HIDDEN>) id 1j1c1e-0005d2-D2 for 39258 <at> debbugs.gnu.org; Tue, 11 Feb 2020 15:20:54 -0500 Received: by mail-qk1-f195.google.com with SMTP id w15so11491345qkf.6 for <39258 <at> debbugs.gnu.org>; Tue, 11 Feb 2020 12:20:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=o6e6xjO5pBKI6J43h1ASM3POuFGUcPM015oYlxJw9xI=; b=KnTSeZs8OwpPZzrgjksQOEgXfS3bhJRoQoinCnRaq6v6R9LDrwXJfieA23Yi2Om5AR yUfD3hf3yDHhV3CtFWX8t9pGwQ6qvvpv/b6Lmqs1Ffq6RRlyKAX+FHTDLC+l7kQsTKuD tBJpeCtu+d05ytW17fx53uqo/80pTfNrkEDkPxs5E2/6jFAn6IoMt90WnEKwfE7xS3y3 ZSfqJOzHWOmqk7XKVJODgEroK6tN0b0r6ZmfoYeWgxxxogfRMsSdXbdCRg9IFZR+iVul yStz6qNZ+KZtehFR/jKh35wM21VxPmkkB+meKyF2ZWqryfCrywWU4Uwp4EWdCrwn35z8 mCgg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=o6e6xjO5pBKI6J43h1ASM3POuFGUcPM015oYlxJw9xI=; b=lo5q7R2wki1yDgsRl7rU/VF1GVAr8w/Vlq+G6UvOaPcGKqb6w8lfsb0PH9IJB/HKgj zrrYWJA4GFRbQTHFKfBtKlcXTCawfMd+uwaIU/zax+rAowxuoXsztv9BWUZEmTR5h4zH hpuISjzTq5i2fGaoIDbjZ0NTSlJ9YmovoWE38QChlPI6eaGDJlqultzwADr2iakT5u56 rb5QNrg0SBS5M5Ijoqamczl5uRZUTYjudqFLZ+oTuU9oSCySExKCI6zDbeRyXU0nbZJc J6t0Vpg3qEuUKtouTfsUWTRomM97S0UZjVlR6HoXvrl530q9VMDCXvisQ4Z1AKc4+MuJ uHuQ== X-Gm-Message-State: APjAAAW29F4KqyXmJXsMQ+rS6Cb8gTSM582o2/oH006/EtVrSCgMRAxD cmLCW+NZv++y7B/8P6IXK1n36RcbaXCivyxYzIEv2djH X-Google-Smtp-Source: APXvYqxZNbub2wRlSXX/06gF7KIyA9RwNOfpHV/hP/vOeiXwDKk/Zxs7jonEpRDBN/JI3rcbMRpVk9eDpTSH3lCKIBs= X-Received: by 2002:a05:620a:124f:: with SMTP id a15mr7211654qkl.201.1581452448834; Tue, 11 Feb 2020 12:20:48 -0800 (PST) MIME-Version: 1.0 References: <cu7pnfaar36.fsf@HIDDEN> <CAJ3okZ05JXVgbKJd_WeJKERJd6rc4SwxLRrRR0AyW48QDyX8eQ@HIDDEN> <cu74kwdawc7.fsf@HIDDEN> <CAJ3okZ0GXS9ibh__1DFsuCCjO82R6-YXtPAOF3GoquN=sSsaWA@HIDDEN> <cu7y2tkadwi.fsf@HIDDEN> <CAJ3okZ37rFh=SB-f6nqc0kMpCai=pw2D0-GFiakXY1p4FHHftA@HIDDEN> <cu7v9oka341.fsf@HIDDEN> <8736bhytn9.fsf@HIDDEN> <CAJ3okZ2EF5244EN3XcO-U9Uz8me42byuPMeUGeGQ_mWi6r864A@HIDDEN> <87sgjhx92g.fsf@HIDDEN> <cu7zhdplz84.fsf@HIDDEN> In-Reply-To: <cu7zhdplz84.fsf@HIDDEN> From: zimoun <zimon.toutoune@HIDDEN> Date: Tue, 11 Feb 2020 21:20:37 +0100 Message-ID: <CAJ3okZ3nTmv+dD_2m7mtcHXQRsoVcxcqW1bJyRzOZfKWaq0Opw@HIDDEN> Subject: Re: [bug#39258] Faster guix search using an sqlite cache To: Arun Isaac <arunisaac@HIDDEN> Content-Type: text/plain; charset="UTF-8" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 39258 Cc: =?UTF-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>, 39258 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) Hi Arun, On Tue, 11 Feb 2020 at 20:07, Arun Isaac <arunisaac@HIDDEN> wrote: > > Arun started the discussion on guix-devel with the idea of an inverted > > index, and I thought this would become a second index (possibly > > implemented using SQLite). Perhaps I misunderstood the discussion all > > along though, let me know! :-) > > No, you didn't misunderstand. That's where it began. But, while > implementing it, I thought I might as well replace the existing cache. An inverted index backed by Guile as you did on guix-devel or backed by SQLite seems a good improvement for "guix search". > Also, I've started working on guile-xapian bindings. With that, it seems > simpler to keep the current package cache and add a xapian index only to > speed up package search. Xapian would be cool! And an SQLite based index seems easier to index locally the packages and their history. The Guix Data Service is already doing such thing but AFAIK using PostgreSQL and some magic. :-) http://data.guix.gnu.org/repository/1/branch/master/package/git All the best, simon
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 11 Feb 2020 20:13:54 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Tue Feb 11 15:13:54 2020 Received: from localhost ([127.0.0.1]:57613 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1j1bus-0003TN-Ij for submit <at> debbugs.gnu.org; Tue, 11 Feb 2020 15:13:54 -0500 Received: from mail-qt1-f172.google.com ([209.85.160.172]:44009) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <zimon.toutoune@HIDDEN>) id 1j1buq-0003TA-LG for 39258 <at> debbugs.gnu.org; Tue, 11 Feb 2020 15:13:53 -0500 Received: by mail-qt1-f172.google.com with SMTP id d18so8978066qtj.10 for <39258 <at> debbugs.gnu.org>; Tue, 11 Feb 2020 12:13:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=Si1Km8WMwVjV002T5tmwksWftaiHa+fzxlF8ZpTj1+4=; b=umbl1g03luGy9yUh9mZxdJAOFaSHPcb0ZI+FxcS3EmA6yYG353CQhwWsQoTJ1H4zCQ RKFWvblFtn6OFvcnAoI9tYoSKzDseWOe+ER06yhB0RqCaY3P70VRoSoJQLDUODyceCSm YUh/eiKwjOqxQ4N5krQKqKLN6UH2F+oKzEG1eN4su0ZULCNR7oySVu96mNEm9X+x4sDK bMvWXykcGb2gjtgQCt8o3SEU9h0o1VLJBvZZPMoS4CME1pypTEo0Upl3e1ZSHWpybygt OaAP4xtXGn9/u1DIz5QvpKSwxXg7izjdGGa6oftrqTNRFv126ccJLrIalMrAj2UUWs+h UvNA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=Si1Km8WMwVjV002T5tmwksWftaiHa+fzxlF8ZpTj1+4=; b=aNoAxEQLpvEwv1SGWE1MYt/FUf22V6gShssC3nRnPTeBH4Sm4/iyMF4B9tzM3wZtto xRnZKanLRfOyH2jKmFMlC/W68l396jW7qdUcYwDVs8hUl4BkTbmxfcqKEs/3/T5m62LQ qIPNngzEKy9bApaK5003YCnFUV314AYPDPBBnWfHMepkBigBJhqCqxZEuBAbbFskGJwc O6iuMGs9hkFg5O0OY3NeGD6VR4513j0raO+x3hCRPZWj9sWaR+wkJ6EQ2dBY/gSaB8Ju 3AJYKbAxBUHwP6uwicjx0He77y9EOkuH6nk/Zjk96hkt8TrsmLwVfk1fS734c0X3eTZY WINw== X-Gm-Message-State: APjAAAVaLTpha4P5xBkalgGM1d+Rms32Sw2vPUq+xquk5bz8SdL4N/DZ Z/IR36bf9NAjViXSty8PhSjIfXbCLaeH7dQ2adU= X-Google-Smtp-Source: APXvYqxiBNnEvthzkE7C0NA95KeLh5JU77B+/hfSKnbYcbn4XheYphHgeYYwdFqqrS9UZxo8vKugV02PGFUHfZGWxXE= X-Received: by 2002:ac8:740f:: with SMTP id p15mr3938408qtq.211.1581452027124; Tue, 11 Feb 2020 12:13:47 -0800 (PST) MIME-Version: 1.0 References: <cu7pnfaar36.fsf@HIDDEN> <CAJ3okZ05JXVgbKJd_WeJKERJd6rc4SwxLRrRR0AyW48QDyX8eQ@HIDDEN> <cu74kwdawc7.fsf@HIDDEN> <CAJ3okZ0GXS9ibh__1DFsuCCjO82R6-YXtPAOF3GoquN=sSsaWA@HIDDEN> <cu7y2tkadwi.fsf@HIDDEN> <CAJ3okZ37rFh=SB-f6nqc0kMpCai=pw2D0-GFiakXY1p4FHHftA@HIDDEN> <cu7v9oka341.fsf@HIDDEN> <8736bhytn9.fsf@HIDDEN> <CAJ3okZ2EF5244EN3XcO-U9Uz8me42byuPMeUGeGQ_mWi6r864A@HIDDEN> <87sgjhx92g.fsf@HIDDEN> In-Reply-To: <87sgjhx92g.fsf@HIDDEN> From: zimoun <zimon.toutoune@HIDDEN> Date: Tue, 11 Feb 2020 21:13:35 +0100 Message-ID: <CAJ3okZ0sXg7xqFLahk+Ej_aE8bR8GmUMX9DFaRjgNV417xvAsw@HIDDEN> Subject: Re: [bug#39258] Faster guix search using an sqlite cache To: =?UTF-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 39258 Cc: Arun Isaac <arunisaac@HIDDEN>, 39258 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) Hi Ludo, On Tue, 11 Feb 2020 at 19:39, Ludovic Court=C3=A8s <ludo@HIDDEN> wrote: > > About performance, the idea was to first implement something with > > sqlite and then see if it makes the difference. I mean I have > > understood that. > > Yes. But keep in mind that this package cache is used exclusively for > package lookups by name. Namely, the goal is to speed package lookup in I agree that some confusion happens here. And this cache cannot be improved= . > >> However, using sqlite for keyword search as you initially proposed on > >> guix-devel does sound like a great idea to me. > > > > If I understand correctly, you are proposing 2 caches, right? > > Or are you proposing an inverted index (VHash/VList table) based on > > trigrams to speed up the lookup? > > Arun started the discussion on guix-devel with the idea of an inverted > index, and I thought this would become a second index (possibly > implemented using SQLite). Perhaps I misunderstood the discussion all > along though, let me know! :-) Well, your suggestion is very welcome and 2 caches are required: one for the lookup by name, as it is already (and does a good job); another one for "guix search" speeds up, SQLite or whatever (based on inverted index or whatever). Thanks, simon
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 11 Feb 2020 19:07:49 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Tue Feb 11 14:07:49 2020 Received: from localhost ([127.0.0.1]:57553 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1j1asu-0001tB-Vq for submit <at> debbugs.gnu.org; Tue, 11 Feb 2020 14:07:49 -0500 Received: from mugam.systemreboot.net ([139.59.75.54]:39568) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <arunisaac@HIDDEN>) id 1j1asr-0001t0-G8 for 39258 <at> debbugs.gnu.org; Tue, 11 Feb 2020 14:07:47 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=systemreboot.net; s=default; h=Content-Type:MIME-Version:Message-ID:Date: References:In-Reply-To:Subject:Cc:To:From:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=dWwvXq2h0nPLqZkCN248aPTXqYXwC6j1nkR+IzH3O5w=; b=LPRG7y9LDnNhbAefgUW5f6ivK 1hPKfV7y7CG15l5GnJVMFIxyQ2IswOyCaCPWhXob9btazOnXUnI+PtyS91MiS1ZjkuXFBcMlWqlU5 mfUbHU2Ioi45k+w36M6zQ3tCq+ag6CsW6I0ehaSDP+4wVUT8ZqbLwzmRZ8s85i+DZ9Juw=; Received: from [192.168.2.1] (helo=steel) by systemreboot.net with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.93) (envelope-from <arunisaac@HIDDEN>) id 1j1asb-001VB3-VN; Wed, 12 Feb 2020 00:37:30 +0530 From: Arun Isaac <arunisaac@HIDDEN> To: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>, zimoun <zimon.toutoune@HIDDEN> Subject: Re: [bug#39258] Faster guix search using an sqlite cache In-Reply-To: <87sgjhx92g.fsf@HIDDEN> References: <cu7pnfaar36.fsf@HIDDEN> <CAJ3okZ05JXVgbKJd_WeJKERJd6rc4SwxLRrRR0AyW48QDyX8eQ@HIDDEN> <cu74kwdawc7.fsf@HIDDEN> <CAJ3okZ0GXS9ibh__1DFsuCCjO82R6-YXtPAOF3GoquN=sSsaWA@HIDDEN> <cu7y2tkadwi.fsf@HIDDEN> <CAJ3okZ37rFh=SB-f6nqc0kMpCai=pw2D0-GFiakXY1p4FHHftA@HIDDEN> <cu7v9oka341.fsf@HIDDEN> <8736bhytn9.fsf@HIDDEN> <CAJ3okZ2EF5244EN3XcO-U9Uz8me42byuPMeUGeGQ_mWi6r864A@HIDDEN> <87sgjhx92g.fsf@HIDDEN> Date: Wed, 12 Feb 2020 00:37:23 +0530 Message-ID: <cu7zhdplz84.fsf@HIDDEN> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 39258 Cc: 39258 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable >>> I would rather keep the current package cache as-is instead of inserting >>> sqlite in here. I don=E2=80=99t expect it to bring much compared >>> performance-wise to the current simple cache (especially if we look at >>> load time), and it does increase complexity quite a bit. >> >> Complexity is about taste. ;-) > > It=E2=80=99s measurable to some extent (lines of code, cyclomatic complex= ity, > etc.) I agree with Ludo here. I think it does increase the complexity, and probably unnecessarily so. > Arun started the discussion on guix-devel with the idea of an inverted > index, and I thought this would become a second index (possibly > implemented using SQLite). Perhaps I misunderstood the discussion all > along though, let me know! :-) No, you didn't misunderstand. That's where it began. But, while implementing it, I thought I might as well replace the existing cache. Also, I've started working on guile-xapian bindings. With that, it seems simpler to keep the current package cache and add a xapian index only to speed up package search. --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEf3MDQ/Lwnzx3v3nTLiXui2GAK7MFAl5C+2sACgkQLiXui2GA K7O3DQgAnz3GCW1FVXQowVB/RqkTb4meDytN8r/LXnA06VVUHhjPuMMSV6d2xPj8 B9p0lagCeCYN8ivFvBq0X5qFwT4gBtmdXWkn2VmU4nBvUAw+uFd9d5S8bFjNeXe4 IGnmWOBuokF6ttYRkGH9Z4wStz49a/9CJcVW4jpCKyjEyodM/PJ9bVOiOTFgqpFA TTQhMSQM1GuJO8OKhctsyZ8xpNejTW/LFPe2CgoIEQavdlMnzgL7Nl3d5NOa2rS8 VyIamqbofxI9Uj9QBXjn8HxIV5FaA2On/XmWXF/uCr44Fw1xGLyy/AFlReE9KgJE jP/TUwhXMr/s9/e8k0bEDQMESnEfFw== =L3bY -----END PGP SIGNATURE----- --=-=-=--
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 11 Feb 2020 18:39:31 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Tue Feb 11 13:39:30 2020 Received: from localhost ([127.0.0.1]:57542 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1j1aRW-0001BU-K8 for submit <at> debbugs.gnu.org; Tue, 11 Feb 2020 13:39:30 -0500 Received: from eggs.gnu.org ([209.51.188.92]:51554) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <ludo@HIDDEN>) id 1j1aRV-0001BI-0c for 39258 <at> debbugs.gnu.org; Tue, 11 Feb 2020 13:39:29 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:49111) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <ludo@HIDDEN>) id 1j1aRO-0008Cx-Vw; Tue, 11 Feb 2020 13:39:23 -0500 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=34260 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from <ludo@HIDDEN>) id 1j1aRO-0003u6-G6; Tue, 11 Feb 2020 13:39:22 -0500 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN> To: zimoun <zimon.toutoune@HIDDEN> Subject: Re: [bug#39258] Faster guix search using an sqlite cache References: <cu7pnfaar36.fsf@HIDDEN> <CAJ3okZ05JXVgbKJd_WeJKERJd6rc4SwxLRrRR0AyW48QDyX8eQ@HIDDEN> <cu74kwdawc7.fsf@HIDDEN> <CAJ3okZ0GXS9ibh__1DFsuCCjO82R6-YXtPAOF3GoquN=sSsaWA@HIDDEN> <cu7y2tkadwi.fsf@HIDDEN> <CAJ3okZ37rFh=SB-f6nqc0kMpCai=pw2D0-GFiakXY1p4FHHftA@HIDDEN> <cu7v9oka341.fsf@HIDDEN> <8736bhytn9.fsf@HIDDEN> <CAJ3okZ2EF5244EN3XcO-U9Uz8me42byuPMeUGeGQ_mWi6r864A@HIDDEN> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 23 =?utf-8?Q?Pluvi=C3=B4se?= an 228 de la =?utf-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Tue, 11 Feb 2020 19:39:19 +0100 In-Reply-To: <CAJ3okZ2EF5244EN3XcO-U9Uz8me42byuPMeUGeGQ_mWi6r864A@HIDDEN> (zimoun's message of "Tue, 11 Feb 2020 19:21:55 +0100") Message-ID: <87sgjhx92g.fsf@HIDDEN> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 39258 Cc: Arun Isaac <arunisaac@HIDDEN>, 39258 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.7 (-) zimoun <zimon.toutoune@HIDDEN> skribis: > On Tue, 11 Feb 2020 at 17:29, Ludovic Court=C3=A8s <ludo@HIDDEN> wrote: > >> I would rather keep the current package cache as-is instead of inserting >> sqlite in here. I don=E2=80=99t expect it to bring much compared >> performance-wise to the current simple cache (especially if we look at >> load time), and it does increase complexity quite a bit. > > Complexity is about taste. ;-) It=E2=80=99s measurable to some extent (lines of code, cyclomatic complexit= y, etc.) > About performance, the idea was to first implement something with > sqlite and then see if it makes the difference. I mean I have > understood that. Yes. But keep in mind that this package cache is used exclusively for package lookups by name. Namely, the goal is to speed package lookup in operations like =E2=80=9Cguix install foo=E2=80=9D (mapping =E2=80=9Cfoo=E2= =80=9D to the right <package> in the right module without walking through all the modules) and =E2=80=9Cg= uix package -A=E2=80=9D (which is what the shell completion hooks use). Currently =E2=80=9Cguix package -A=E2=80=9D runs in .5s on my laptop, and I= suspect it=E2=80=99s going to be hard to do better just by touching the cache. >> However, using sqlite for keyword search as you initially proposed on >> guix-devel does sound like a great idea to me. > > If I understand correctly, you are proposing 2 caches, right? > Or are you proposing an inverted index (VHash/VList table) based on > trigrams to speed up the lookup? Arun started the discussion on guix-devel with the idea of an inverted index, and I thought this would become a second index (possibly implemented using SQLite). Perhaps I misunderstood the discussion all along though, let me know! :-) Thanks, Ludo=E2=80=99.
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 11 Feb 2020 18:22:14 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Tue Feb 11 13:22:14 2020 Received: from localhost ([127.0.0.1]:57506 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1j1aAo-0000iG-1s for submit <at> debbugs.gnu.org; Tue, 11 Feb 2020 13:22:14 -0500 Received: from mail-qt1-f171.google.com ([209.85.160.171]:35008) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <zimon.toutoune@HIDDEN>) id 1j1aAl-0000hz-Gv for 39258 <at> debbugs.gnu.org; Tue, 11 Feb 2020 13:22:11 -0500 Received: by mail-qt1-f171.google.com with SMTP id n17so8713605qtv.2 for <39258 <at> debbugs.gnu.org>; Tue, 11 Feb 2020 10:22:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=BWhBwj73c9JxCfGkWaSjk77vwHXygt/320xHkfIIULE=; b=u9WWgDmNL2RVToEicjOHinUXd/FNdOUuhBz05RsoO5VWwSQAuRI2q9E0BUHOAw8Md/ 09jxgalfDJI6DJ+uoCUIXHVNoc300SecKkggSh/2iFEHgyTNeBaVzLVP9VGmweClnfJf dVe6gjaX9kQR+O03gZhNGpEOc8tan9KRtTr0ym3D6NyQymJrntXriBcaTWj8i2RGRxMA xPKSZx5Qd/FckJ5yJsQBYqI0cz2wnOl+hmwcQDmJ/fa65FD/HJeOL0M0SEwVb6kKqsZM BuBRs11R2XJqz8REb55OawKz90FxH+GB9OWpyC3svGpWoWeAx99MVldBfVGKU5ojOe88 /kxA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=BWhBwj73c9JxCfGkWaSjk77vwHXygt/320xHkfIIULE=; b=eaiE2WJnDFbgQnZmk+av5elqpsCChFDyn3XpbbfE3bdGiYjagI4MB4etz35SFeQGdv C1o0JWysxfBaTpzGAuZlc3QUicykJhtJpag1qBWLYmIF/8Eg+s0buGdv97bsinoj0mAn fh4aR+HLciXaQKmTP/bFEufG/YZifpcsGUUxBCQZ8ky5WWVFIIw3Iatu0fn73hIojEVd nqOg1leNT8OpPXPhx95ut9Z2gqDa0In/7q3H8d/0rIKisRBe5LNcOod5QpHgdvhT46k3 LDczSH+VAkROVFYL1Yjx9zMHywdobkABW3Kta77WYYU4N1Vx8ihrKEpzVEnJ5dEkREeq KdEw== X-Gm-Message-State: APjAAAXDJgd5bT7GvQVb6O30W8KjxR+zVJNPJrH6qT5H91vZ6haMY+LI /D3TtkJbCdbcHBEOwpnNnO6SsInwGY1KUdVZJmnVjQ== X-Google-Smtp-Source: APXvYqxBTFgcfBi0JAIqY4Fo9W3y/JGyDyUm3jkNaohC04J8F08dZh/9IhZIdKZCCDB4fqS3VfDen8FW8xkNpuy8J3s= X-Received: by 2002:ac8:319c:: with SMTP id h28mr16315257qte.186.1581445326078; Tue, 11 Feb 2020 10:22:06 -0800 (PST) MIME-Version: 1.0 References: <cu7pnfaar36.fsf@HIDDEN> <CAJ3okZ05JXVgbKJd_WeJKERJd6rc4SwxLRrRR0AyW48QDyX8eQ@HIDDEN> <cu74kwdawc7.fsf@HIDDEN> <CAJ3okZ0GXS9ibh__1DFsuCCjO82R6-YXtPAOF3GoquN=sSsaWA@HIDDEN> <cu7y2tkadwi.fsf@HIDDEN> <CAJ3okZ37rFh=SB-f6nqc0kMpCai=pw2D0-GFiakXY1p4FHHftA@HIDDEN> <cu7v9oka341.fsf@HIDDEN> <8736bhytn9.fsf@HIDDEN> In-Reply-To: <8736bhytn9.fsf@HIDDEN> From: zimoun <zimon.toutoune@HIDDEN> Date: Tue, 11 Feb 2020 19:21:55 +0100 Message-ID: <CAJ3okZ2EF5244EN3XcO-U9Uz8me42byuPMeUGeGQ_mWi6r864A@HIDDEN> Subject: Re: [bug#39258] Faster guix search using an sqlite cache To: =?UTF-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 39258 Cc: Arun Isaac <arunisaac@HIDDEN>, 39258 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) Hi Ludo, On Tue, 11 Feb 2020 at 17:29, Ludovic Court=C3=A8s <ludo@HIDDEN> wrote: > I would rather keep the current package cache as-is instead of inserting > sqlite in here. I don=E2=80=99t expect it to bring much compared > performance-wise to the current simple cache (especially if we look at > load time), and it does increase complexity quite a bit. Complexity is about taste. ;-) About performance, the idea was to first implement something with sqlite and then see if it makes the difference. I mean I have understood that. > However, using sqlite for keyword search as you initially proposed on > guix-devel does sound like a great idea to me. If I understand correctly, you are proposing 2 caches, right? Or are you proposing an inverted index (VHash/VList table) based on trigrams to speed up the lookup? Cheers, simon
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 11 Feb 2020 16:29:42 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Tue Feb 11 11:29:42 2020 Received: from localhost ([127.0.0.1]:57411 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1j1YPu-0004RI-Ij for submit <at> debbugs.gnu.org; Tue, 11 Feb 2020 11:29:42 -0500 Received: from eggs.gnu.org ([209.51.188.92]:40322) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <ludo@HIDDEN>) id 1j1YPs-0004R6-FH for 39258 <at> debbugs.gnu.org; Tue, 11 Feb 2020 11:29:41 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:46770) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <ludo@HIDDEN>) id 1j1YPm-0002am-Gq; Tue, 11 Feb 2020 11:29:34 -0500 Received: from [2001:660:6102:320:e120:2c8f:8909:cdfe] (port=34126 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from <ludo@HIDDEN>) id 1j1YPl-000741-De; Tue, 11 Feb 2020 11:29:34 -0500 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN> To: Arun Isaac <arunisaac@HIDDEN> Subject: Re: [bug#39258] Faster guix search using an sqlite cache References: <cu7pnfaar36.fsf@HIDDEN> <CAJ3okZ05JXVgbKJd_WeJKERJd6rc4SwxLRrRR0AyW48QDyX8eQ@HIDDEN> <cu74kwdawc7.fsf@HIDDEN> <CAJ3okZ0GXS9ibh__1DFsuCCjO82R6-YXtPAOF3GoquN=sSsaWA@HIDDEN> <cu7y2tkadwi.fsf@HIDDEN> <CAJ3okZ37rFh=SB-f6nqc0kMpCai=pw2D0-GFiakXY1p4FHHftA@HIDDEN> <cu7v9oka341.fsf@HIDDEN> Date: Tue, 11 Feb 2020 17:29:30 +0100 In-Reply-To: <cu7v9oka341.fsf@HIDDEN> (Arun Isaac's message of "Thu, 06 Feb 2020 07:28:22 +0530") Message-ID: <8736bhytn9.fsf@HIDDEN> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 39258 Cc: 39258 <at> debbugs.gnu.org, zimoun <zimon.toutoune@HIDDEN> X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.7 (-) Hello Arun! Arun Isaac <arunisaac@HIDDEN> skribis: > From 4c883fcff1f44339b28df6ccdb2b10c906439e3d Mon Sep 17 00:00:00 2001 > From: Arun Isaac <arunisaac@HIDDEN> > Date: Tue, 21 Jan 2020 20:45:43 +0530 > Subject: [PATCH] fast search [...] > --- a/gnu/packages.scm > +++ b/gnu/packages.scm > @@ -43,6 +43,7 @@ > #:use-module (srfi srfi-34) > #:use-module (srfi srfi-35) > #:use-module (srfi srfi-39) > + #:use-module (sqlite3) > #:export (search-patch > search-patches > search-auxiliary-file > @@ -204,10 +205,8 @@ PROC is called along these lines: > PROC can use #:allow-other-keys to ignore the bits it's not interested i= n. > When a package cache is available, this procedure does not actually load= any > package module." > - (define cache > - (load-package-cache (current-profile))) > - > - (if (and cache (cache-is-authoritative?)) > + (if (and (cache-is-authoritative?) > + (current-profile)) > (vhash-fold (lambda (name vector result) > (match vector > (#(name version module symbol outputs > @@ -220,7 +219,7 @@ package module." > #:supported? supported? > #:deprecated? deprecated?)))) > init > - cache) > + (cache-lookup (current-profile))) > (fold-packages (lambda (package result) > (proc (package-name package) > (package-version package) > @@ -252,31 +251,7 @@ is guaranteed to never traverse the same package twi= ce." >=20=20 > (define %package-cache-file > ;; Location of the package cache. > - "/lib/guix/package.cache") > - > -(define load-package-cache [...] > +(define* (cache-lookup profile #:optional name) > "Lookup package NAME in CACHE. Return a list sorted in increasing ver= sion > order." > (define (package-version<? v1 v2) > (version>? (vector-ref v2 1) (vector-ref v1 1))) >=20=20 > - (sort (vhash-fold* cons '() name cache) > - package-version<?)) > + (define (int->boolean n) > + (case n > + ((0) #f) > + ((1) #t))) > + > + (define (string->list str) > + (call-with-input-string str read)) > + > + (define select-statement > + (string-append > + "SELECT name, version, module, symbol, outputs, supported, supersed= ed, locationFile, locationLine, locationColumn from packages" > + (if name " WHERE name =3D :name" ""))) I would rather keep the current package cache as-is instead of inserting sqlite in here. I don=E2=80=99t expect it to bring much compared performance-wise to the current simple cache (especially if we look at load time), and it does increase complexity quite a bit. However, using sqlite for keyword search as you initially proposed on guix-devel does sound like a great idea to me. WDYT? Thanks, Ludo=E2=80=99.
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.
Received: (at 39258) by debbugs.gnu.org; 6 Feb 2020 01:58:49 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Wed Feb 05 20:58:49 2020
Received: from localhost ([127.0.0.1]:46966 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1izWRN-0006k1-89
for submit <at> debbugs.gnu.org; Wed, 05 Feb 2020 20:58:49 -0500
Received: from mugam.systemreboot.net ([139.59.75.54]:35458)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <arunisaac@HIDDEN>) id 1izWRI-0006jp-O3
for 39258 <at> debbugs.gnu.org; Wed, 05 Feb 2020 20:58:47 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed;
d=systemreboot.net; s=default; h=Content-Type:MIME-Version:Message-ID:Date:
References:In-Reply-To:Subject:Cc:To:From:Sender:Reply-To:
Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date:
Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:
List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive;
bh=x5WRND8ZKSjnqBSnMgncM63QAmvrxCWp/A8/EJmx3OM=; b=LH/Q70zv8XIRoRMAowYy+i66e
D3QS04ZxdKQuP+Hi59m1BKIwKbBTIutoMVolctvUBS9yWuPG3uMb9+m6yeQL3YZOOEKU15kKZDOm0
r/o1z+Q07axTi7eSIz2O3YncvURqqOL6tvtBP6xrYgAB5voAH7eNO0Q5apyIylXp7f90M=;
Received: from [192.168.2.1] (helo=steel)
by systemreboot.net with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384
(Exim 4.93) (envelope-from <arunisaac@HIDDEN>)
id 1izWR8-000zOQ-NP; Thu, 06 Feb 2020 07:28:34 +0530
From: Arun Isaac <arunisaac@HIDDEN>
To: zimoun <zimon.toutoune@HIDDEN>
Subject: Re: [bug#39258] Faster guix search using an sqlite cache
In-Reply-To: <CAJ3okZ37rFh=SB-f6nqc0kMpCai=pw2D0-GFiakXY1p4FHHftA@HIDDEN>
References: <cu7pnfaar36.fsf@HIDDEN>
<CAJ3okZ05JXVgbKJd_WeJKERJd6rc4SwxLRrRR0AyW48QDyX8eQ@HIDDEN>
<cu74kwdawc7.fsf@HIDDEN>
<CAJ3okZ0GXS9ibh__1DFsuCCjO82R6-YXtPAOF3GoquN=sSsaWA@HIDDEN>
<cu7y2tkadwi.fsf@HIDDEN>
<CAJ3okZ37rFh=SB-f6nqc0kMpCai=pw2D0-GFiakXY1p4FHHftA@HIDDEN>
Date: Thu, 06 Feb 2020 07:28:22 +0530
Message-ID: <cu7v9oka341.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="==-=-=";
micalg=pgp-sha256; protocol="application/pgp-signature"
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 39258
Cc: 39258 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)
--==-=-=
Content-Type: multipart/mixed; boundary="=-=-="
--=-=-=
Content-Type: text/plain
>> Thank you, this was useful. I was able to catch and report the error. I
>
> Where have you reported the error?
I reported the error to the derivation log. For example, if the
derivation for the guix-package-cache derivation is
/gnu/store/cyf2h3frcjxm147dii5qic8d6kpm39nq-guix-package-cache.drv, the
log file will be at
/var/log/guix/drvs/cy/f2h3frcjxm147dii5qic8d6kpm39nq-guix-package-cache.drv.bz2. Notice
that the directory name under drvs is the first two letters of the hash,
and the file name under that directory is the remaining letters.
Also please find attached a dump of my code so far.
>> This could be a permission error, or something to do with the existence
>> or lack thereof of certain directories (such as /var) in the chroot of
>> the build daemon. I'm still figuring it out.
>
> Hum? And this should explain why it is working with the REPL and not
> with the CLI, right?
This could expalin it, but I am not sure if this is the correct
explanation.
>> I'm also in half a mind to get some guile xapian bindings ready so we
>> can just do that instead of messing with sqlite here. But, let's
>> see. :-P
>
> Cool!
> Let me know if you push something somewhere.
Sure, will let you know.
--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment; filename=0001-fast-search.patch
Content-Transfer-Encoding: quoted-printable
From=204c883fcff1f44339b28df6ccdb2b10c906439e3d Mon Sep 17 00:00:00 2001
From: Arun Isaac <arunisaac@HIDDEN>
Date: Tue, 21 Jan 2020 20:45:43 +0530
Subject: [PATCH] fast search
=2D--
build-aux/build-self.scm | 5 +
gnu/packages.scm | 234 +++++++++++++++++++++++++--------------
2 files changed, 155 insertions(+), 84 deletions(-)
diff --git a/build-aux/build-self.scm b/build-aux/build-self.scm
index fc13032b73..c123ad3b11 100644
=2D-- a/build-aux/build-self.scm
+++ b/build-aux/build-self.scm
@@ -264,6 +264,9 @@ interface (FFI) of Guile.")
(define fake-git
(scheme-file "git.scm" #~(define-module (git))))
=20
+ (define fake-sqlite3
+ (scheme-file "sqlite3.scm" #~(define-module (sqlite3))))
+
(with-imported-modules `(((guix config)
=3D> ,(make-config.scm))
=20
@@ -278,6 +281,8 @@ interface (FFI) of Guile.")
;; (git) to placate it.
((git) =3D> ,fake-git)
=20
+ ((sqlite3) =3D> ,fake-sqlite3)
+
,@(source-module-closure `((guix store)
(guix self)
(guix derivations)
diff --git a/gnu/packages.scm b/gnu/packages.scm
index d22c992bb1..0ae5b84284 100644
=2D-- a/gnu/packages.scm
+++ b/gnu/packages.scm
@@ -43,6 +43,7 @@
#:use-module (srfi srfi-34)
#:use-module (srfi srfi-35)
#:use-module (srfi srfi-39)
+ #:use-module (sqlite3)
#:export (search-patch
search-patches
search-auxiliary-file
@@ -204,10 +205,8 @@ PROC is called along these lines:
PROC can use #:allow-other-keys to ignore the bits it's not interested in.
When a package cache is available, this procedure does not actually load a=
ny
package module."
=2D (define cache
=2D (load-package-cache (current-profile)))
=2D
=2D (if (and cache (cache-is-authoritative?))
+ (if (and (cache-is-authoritative?)
+ (current-profile))
(vhash-fold (lambda (name vector result)
(match vector
(#(name version module symbol outputs
@@ -220,7 +219,7 @@ package module."
#:supported? supported?
#:deprecated? deprecated?))))
init
=2D cache)
+ (cache-lookup (current-profile)))
(fold-packages (lambda (package result)
(proc (package-name package)
(package-version package)
@@ -252,31 +251,7 @@ is guaranteed to never traverse the same package twice=
."
=20
(define %package-cache-file
;; Location of the package cache.
=2D "/lib/guix/package.cache")
=2D
=2D(define load-package-cache
=2D (mlambda (profile)
=2D "Attempt to load the package cache. On success return a vhash keyed=
by
=2Dpackage names. Return #f on failure."
=2D (match profile
=2D (#f #f)
=2D (profile
=2D (catch 'system-error
=2D (lambda ()
=2D (define lst
=2D (load-compiled (string-append profile %package-cache-file)))
=2D (fold (lambda (item vhash)
=2D (match item
=2D (#(name version module symbol outputs
=2D supported? deprecated?
=2D file line column)
=2D (vhash-cons name item vhash))))
=2D vlist-null
=2D lst))
=2D (lambda args
=2D (if (=3D ENOENT (system-error-errno args))
=2D #f
=2D (apply throw args))))))))
+ "/lib/guix/package-cache.sqlite")
=20
(define find-packages-by-name/direct ;bypass the cache
(let ((packages (delay
@@ -297,25 +272,57 @@ decreasing version order."
matching)
matching)))))
=20
=2D(define (cache-lookup cache name)
+(define* (cache-lookup profile #:optional name)
"Lookup package NAME in CACHE. Return a list sorted in increasing versi=
on
order."
(define (package-version<? v1 v2)
(version>? (vector-ref v2 1) (vector-ref v1 1)))
=20
=2D (sort (vhash-fold* cons '() name cache)
=2D package-version<?))
+ (define (int->boolean n)
+ (case n
+ ((0) #f)
+ ((1) #t)))
+
+ (define (string->list str)
+ (call-with-input-string str read))
+
+ (define select-statement
+ (string-append
+ "SELECT name, version, module, symbol, outputs, supported, superseded=
, locationFile, locationLine, locationColumn from packages"
+ (if name " WHERE name =3D :name" "")))
+
+ (define cache-file
+ (string-append profile %package-cache-file))
+
+ (let* ((db (sqlite-open cache-file SQLITE_OPEN_READONLY))
+ (statement (sqlite-prepare db select-statement)))
+ (when name
+ (sqlite-bind-arguments statement #:name name))
+ (let ((result (sqlite-fold (lambda (v result)
+ (match v
+ (#(name version module symbol outputs s=
upported superseded file line column)
+ (cons
+ (vector name
+ version
+ (string->list module)
+ (string->symbol symbol)
+ (string->list outputs)
+ (int->boolean supported)
+ (int->boolean superseded)
+ (list file line column))
+ result))))
+ '() statement)))
+ (sqlite-finalize statement)
+ (sqlite-close db)
+ (sort result package-version<?))))
=20
(define* (find-packages-by-name name #:optional version)
"Return the list of packages with the given NAME. If VERSION is not #f,
then only return packages whose version is prefixed by VERSION, sorted in
decreasing version order."
=2D (define cache
=2D (load-package-cache (current-profile)))
=2D
=2D (if (and (cache-is-authoritative?) cache)
=2D (match (cache-lookup cache name)
=2D (#f #f)
+ (if (and (cache-is-authoritative?)
+ (current-profile))
+ (match (cache-lookup (current-profile) name)
((#(_ versions modules symbols _ _ _ _ _ _) ...)
(fold (lambda (version* module symbol result)
(if (or (not version)
@@ -331,12 +338,9 @@ decreasing version order."
(define* (find-package-locations name #:optional version)
"Return a list of version/location pairs corresponding to each package
matching NAME and VERSION."
=2D (define cache
=2D (load-package-cache (current-profile)))
=2D
=2D (if (and cache (cache-is-authoritative?))
=2D (match (cache-lookup cache name)
=2D (#f '())
+ (if (and (cache-is-authoritative?)
+ (current-profile))
+ (match (cache-lookup (current-profile) name)
((#(name versions modules symbols outputs
supported? deprecated?
files lines columns) ...)
@@ -372,6 +376,33 @@ VERSION."
;; Prevent Guile 3 from inlining this procedure so we can mock it in tests.
(set! find-best-packages-by-name find-best-packages-by-name)
=20
+;; (generate-package-cache "/tmp/test")
+
+;; XXX: missing in guile-sqlite3@HIDDEN
+(define SQLITE_BUSY 5)
+
+(define (call-with-transaction db proc)
+ "Start a transaction with DB (make as many attempts as necessary) and run
+PROC. If PROC exits abnormally, abort the transaction, otherwise commit t=
he
+transaction after it finishes."
+ (catch 'sqlite-error
+ (lambda ()
+ ;; We use begin immediate here so that if we need to retry, we
+ ;; figure that out immediately rather than because some SQLITE_BUSY
+ ;; exception gets thrown partway through PROC - in which case the
+ ;; part already executed (which may contain side-effects!) would be
+ ;; executed again for every retry.
+ (sqlite-exec db "begin immediate;")
+ (let ((result (proc)))
+ (sqlite-exec db "commit;")
+ result))
+ (lambda (key who error description)
+ (if (=3D error SQLITE_BUSY)
+ (call-with-transaction db proc)
+ (begin
+ (sqlite-exec db "rollback;")
+ (throw 'sqlite-error who error description))))))
+
(define (generate-package-cache directory)
"Generate under DIRECTORY a cache of all the available packages.
=20
@@ -381,49 +412,84 @@ reducing the memory footprint."
(define cache-file
(string-append directory %package-cache-file))
=20
=2D (define (expand-cache module symbol variable result+seen)
+ (define schema
+ "CREATE TABLE packages (name text,
+version text,
+module text,
+symbol text,
+outputs text,
+supported int,
+superseded int,
+locationFile text,
+locationLine int,
+locationColumn int);
+CREATE VIRTUAL TABLE packageSearch USING fts5(name, searchText);")
+
+ (define insert-statement
+ "INSERT INTO packages(name, version, module, symbol, outputs, supporte=
d, superseded, locationFile, locationLine, locationColumn)
+VALUES(:name, :version, :module, :symbol, :outputs, :supported, :supersede=
d, :locationfile, :locationline, :locationcolumn)")
+
+ (define insert-package-search-statement
+ "INSERT INTO packageSearch(name, searchText) VALUES(:name, :searchtext=
)")
+
+ (define (boolean->int x)
+ (if x 1 0))
+
+ (define (list->string x)
+ (call-with-output-string (cut write x <>)))
+
+ (define (insert-package db module symbol variable seen)
(match (false-if-exception (variable-ref variable))
((? package? package)
=2D (match result+seen
=2D ((result . seen)
=2D (if (or (vhash-assq package seen)
=2D (hidden-package? package))
=2D (cons result seen)
=2D (cons (cons `#(,(package-name package)
=2D ,(package-version package)
=2D ,(module-name module)
=2D ,symbol
=2D ,(package-outputs package)
=2D ,(->bool (supported-package? package))
=2D ,(->bool (package-superseded package))
=2D ,@(let ((loc (package-location package)))
=2D (if loc
=2D `(,(location-file loc)
=2D ,(location-line loc)
=2D ,(location-column loc))
=2D '(#f #f #f))))
=2D result)
=2D (vhash-consq package #t seen))))))
=2D (_
=2D result+seen)))
=2D
=2D (define exp
=2D (first
=2D (fold-module-public-variables* expand-cache
=2D (cons '() vlist-null)
=2D (all-modules (%package-module-path)
=2D #:warn
=2D warn-about-load-error))=
))
+ (cond
+ ((or (vhash-assq package seen)
+ (hidden-package? package))
+ seen)
+ (else
+ (let ((statement (sqlite-prepare db insert-statement)))
+ (sqlite-bind-arguments statement
+ #:name (package-name package)
+ #:version (package-version package)
+ #:module (list->string (module-name modu=
le))
+ #:symbol (symbol->string symbol)
+ #:outputs (list->string (package-outputs=
package))
+ #:supported (boolean->int (supported-pac=
kage? package))
+ #:superseded (boolean->int (package-supe=
rseded package))
+ #:locationfile (cond
+ ((package-location packa=
ge) =3D> location-file)
+ (else #f))
+ #:locationline (cond
+ ((package-location packa=
ge) =3D> location-line)
+ (else #f))
+ #:locationcolumn (cond
+ ((package-location pac=
kage) =3D> location-column)
+ (else #f)))
+ (sqlite-fold cons '() statement)
+ (sqlite-finalize statement))
+ (let ((statement (sqlite-prepare db insert-package-search-stateme=
nt)))
+ (sqlite-bind-arguments statement
+ #:name (package-name package)
+ #:searchtext (package-description packag=
e))
+ (sqlite-fold cons '() statement)
+ (sqlite-finalize statement))
+ (vhash-consq package #t seen))))
+ (_ seen)))
=20
(mkdir-p (dirname cache-file))
=2D (call-with-output-file cache-file
=2D (lambda (port)
=2D ;; Store the cache as a '.go' file. This makes loading fast and r=
educes
=2D ;; heap usage since some of the static data is directly mmapped.
=2D (put-bytevector port
=2D (compile `'(,@exp)
=2D #:to 'bytecode
=2D #:opts '(#:to-file? #t)))))
+ (let ((tmp (string-append (dirname cache-file) "/tmp")))
+ (mkdir-p tmp)
+ (setenv "SQLITE_TMPDIR" tmp))
+ (let ((db (sqlite-open cache-file)))
+ (sqlite-exec db schema)
+ (call-with-transaction db
+ (lambda ()
+ (fold-module-public-variables* (cut insert-package db <> <> <> <=
>)
+ vlist-null
+ (all-modules (%package-module-pat=
h)
+ #:warn
+ warn-about-load-erro=
r))))
+ (sqlite-close db))
+
cache-file)
=20
=2D-=20
2.23.0
--=-=-=--
--==-=-=
Content-Type: application/pgp-signature; name="signature.asc"
-----BEGIN PGP SIGNATURE-----
iQEzBAEBCAAdFiEEf3MDQ/Lwnzx3v3nTLiXui2GAK7MFAl47cr4ACgkQLiXui2GA
K7OLPwf9FgQ/N8AazOSXRwuXn8N+1QZ0b0G1bq2ZpQYppHpBl0e8SgmTyYXpE4za
U2tBfJEMTUXW7YUZD/36DKNpN5JUD1c4Hnsn/BDqRQFFXt4agzTE3PhtOV2gFLjC
wYH/i9xtLoRM+7g36/ZYAmTfAZoiFro4A3kq1ndy8nQCcf++v/v3tQvFVWn3ywKT
Fig63/wm8hXz7kP0+fi6MVKlhKe05VDBXxZCNfM7tSjmEw2qFNmwALUncWUyN9Sz
uF0aU+/JzPUF6kvXMZR9cQxBI5ISZ+gxsdg7rikmwso9jvyxRbEDFAck4Etv9j32
drcNfNH4ZSUaZY/NcxIpzu/yI9dKKA==
=M4NZ
-----END PGP SIGNATURE-----
--==-=-=--
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 4 Feb 2020 10:19:31 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Tue Feb 04 05:19:31 2020 Received: from localhost ([127.0.0.1]:42635 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1iyvIp-0000tR-JW for submit <at> debbugs.gnu.org; Tue, 04 Feb 2020 05:19:31 -0500 Received: from mail-qk1-f169.google.com ([209.85.222.169]:44536) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <zimon.toutoune@HIDDEN>) id 1iyvIo-0000t9-5R for 39258 <at> debbugs.gnu.org; Tue, 04 Feb 2020 05:19:30 -0500 Received: by mail-qk1-f169.google.com with SMTP id v195so17284398qkb.11 for <39258 <at> debbugs.gnu.org>; Tue, 04 Feb 2020 02:19:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=NuzaTIrcWuV1Td2S53i/wfjU4uvdID+cSbqGU8HW9Ls=; b=eoX040qBxEr23eWFOQm6KYCq45W1h0wPf+MhW5jyIWfyP0EXdnyD9mVLYr7tqKvttJ RguoAwYWA9QjOakSA7smbNZix2kmkertldHvVGXDELjI3N3UIEDD1QcfOGpcp4YBMxBt ClCZLosxMxtIGpsxzOCkOidlyB32emJeVtrQVamUsrHDmezJp5hb5boP8um3eJIwCKgk QY756DTJpCBFyhNgJ9Le9GFdPp+tKwCmeSoqtF+/Nw/6vg8UTKIyujhNh76bE7KhJIYO UDjY0dwkrCwrFQifjiwd5m5FjH17yAZS2f016RtNHK9bNsESQQh9t1HS/58oWEQ8/wUz FD2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=NuzaTIrcWuV1Td2S53i/wfjU4uvdID+cSbqGU8HW9Ls=; b=sStLv7LkVy4fN53+VnRLdF7RvuqP0N9xLf/LIeWRgqHv8SZuyesb/IqqXaj4BlKxDi vjANOx9rgeBrckgFxPRX3fFqOb3tUVTGBPLADZNQYkCjdo5vdjiEIZrOQgLPpbTUEILz 4CjpTYPhhnB/P0a71yHdW2s1RyNVQ9Mph0O23zkZ1NihHHwQAHsvmkPSbSclfJXbmp7f JYUuXNLoUnIELinjUj3xPm56bJxh2D8Abli3Svu96XZcFhJaxUDePjEnPKdlw/kiytwf LbjUMi9eZ45UpwRXObwV36ONyIPDmzZvDiFRtSBwSq540J33AP/VvR+U6hsfym/EGaHy LZbA== X-Gm-Message-State: APjAAAWWG78SjCeebKU8tYEkvsVCHJQC/3Qh+Cj787q348jiOwt6G8i2 aBHbtX12NIoakPZ/huFomDMGA7qLDJabpgj0gx89lQ== X-Google-Smtp-Source: APXvYqzqdiAP3XECLyP2fPtX9EjgBUFbpvIaWuPlEzwuwC6FAYfaC+f+8FZy8vHJlReEghxcNjIZ4qmfAMv0sjCmliE= X-Received: by 2002:a37:63c7:: with SMTP id x190mr27760819qkb.232.1580811564425; Tue, 04 Feb 2020 02:19:24 -0800 (PST) MIME-Version: 1.0 References: <cu7pnfaar36.fsf@HIDDEN> <CAJ3okZ05JXVgbKJd_WeJKERJd6rc4SwxLRrRR0AyW48QDyX8eQ@HIDDEN> <cu74kwdawc7.fsf@HIDDEN> <CAJ3okZ0GXS9ibh__1DFsuCCjO82R6-YXtPAOF3GoquN=sSsaWA@HIDDEN> <cu7y2tkadwi.fsf@HIDDEN> In-Reply-To: <cu7y2tkadwi.fsf@HIDDEN> From: zimoun <zimon.toutoune@HIDDEN> Date: Tue, 4 Feb 2020 11:19:13 +0100 Message-ID: <CAJ3okZ37rFh=SB-f6nqc0kMpCai=pw2D0-GFiakXY1p4FHHftA@HIDDEN> Subject: Re: [bug#39258] Faster guix search using an sqlite cache To: Arun Isaac <arunisaac@HIDDEN> Content-Type: text/plain; charset="UTF-8" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 39258 Cc: 39258 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) Hi, On Sun, 2 Feb 2020 at 22:16, Arun Isaac <arunisaac@HIDDEN> wrote: > Thank you, this was useful. I was able to catch and report the error. I Where have you reported the error? > also found the log file for the guix-package-cache profile hook. It says > > (repl-version 0 0) > Generating package cache for '/gnu/store/b6f9b5qbcn4r932whrr6m15rdimbgrhs-profile'... > (exception sqlite-error (value sqlite-open) (value 14) (value "Unable to open the database file")) > > This could be a permission error, or something to do with the existence > or lack thereof of certain directories (such as /var) in the chroot of > the build daemon. I'm still figuring it out. Hum? And this should explain why it is working with the REPL and not with the CLI, right? > I'm also in half a mind to get some guile xapian bindings ready so we > can just do that instead of messing with sqlite here. But, let's > see. :-P Cool! Let me know if you push something somewhere. Cheers, simon
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 2 Feb 2020 21:16:40 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Sun Feb 02 16:16:40 2020 Received: from localhost ([127.0.0.1]:40646 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1iyMbf-0002zK-Vo for submit <at> debbugs.gnu.org; Sun, 02 Feb 2020 16:16:40 -0500 Received: from mugam.systemreboot.net ([139.59.75.54]:47868) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <arunisaac@HIDDEN>) id 1iyMbd-0002z9-Cg for 39258 <at> debbugs.gnu.org; Sun, 02 Feb 2020 16:16:39 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=systemreboot.net; s=default; h=Content-Type:MIME-Version:Message-ID:Date: References:In-Reply-To:Subject:Cc:To:From:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=fTtfIfuOCVGXh2uLN340Z4DGMr/Nno1Tk/gevwJV00E=; b=mXRVK0ouzN6JWvA7W3DClT0tI Ka9AHo6YT4e+9xGOfFQgvBh5GMpBT3RZVY76/ZkMH+/cmUcnhtrZoRvPy5KKkQfMYP9D4zoeWra6P dSrfYF7MCRJKY0BgwV8qJ43ZLGqIWwe1rbnTGZk29NlhFOgeIRiNhijEYai57EOT+uzbY=; Received: from [192.168.2.1] (helo=steel) by systemreboot.net with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.93) (envelope-from <arunisaac@HIDDEN>) id 1iyMbT-000lGl-8p; Mon, 03 Feb 2020 02:46:27 +0530 From: Arun Isaac <arunisaac@HIDDEN> To: zimoun <zimon.toutoune@HIDDEN> Subject: Re: [bug#39258] Faster guix search using an sqlite cache In-Reply-To: <CAJ3okZ0GXS9ibh__1DFsuCCjO82R6-YXtPAOF3GoquN=sSsaWA@HIDDEN> References: <cu7pnfaar36.fsf@HIDDEN> <CAJ3okZ05JXVgbKJd_WeJKERJd6rc4SwxLRrRR0AyW48QDyX8eQ@HIDDEN> <cu74kwdawc7.fsf@HIDDEN> <CAJ3okZ0GXS9ibh__1DFsuCCjO82R6-YXtPAOF3GoquN=sSsaWA@HIDDEN> Date: Mon, 03 Feb 2020 02:46:13 +0530 Message-ID: <cu7y2tkadwi.fsf@HIDDEN> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 39258 Cc: 39258 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain > Danny told me something like: > > (catch (sqlite-error > > I have not tried yet. Thank you, this was useful. I was able to catch and report the error. I also found the log file for the guix-package-cache profile hook. It says (repl-version 0 0) Generating package cache for '/gnu/store/b6f9b5qbcn4r932whrr6m15rdimbgrhs-profile'... (exception sqlite-error (value sqlite-open) (value 14) (value "Unable to open the database file")) This could be a permission error, or something to do with the existence or lack thereof of certain directories (such as /var) in the chroot of the build daemon. I'm still figuring it out. I'm also in half a mind to get some guile xapian bindings ready so we can just do that instead of messing with sqlite here. But, let's see. :-P --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEf3MDQ/Lwnzx3v3nTLiXui2GAK7MFAl43PB4ACgkQLiXui2GA K7O0Hwf/bpqqdDy9FynrZNWUqne4w/IQ5nqJLIxZJ30IcbBu8TxWZp31+UjrBE5M lJI9GQglb+K1atySjm1ZPhQ/uzsiheXF+ENULTP6HeC1bHgZaWQpPhzmknMEfeoG d1NU3ePQHLiE5IjPm6PptdYnbzCRmcmGupD23VHCIPbr1LVFSS7ajsFiKofehDyt LgpANYLVDokhe0w99Oa6Htf4C3WwkICMPCVJA01LborzdVs65Kso+HpQkBh3egjw +m8sdF+xoPBYKSR+GlhZjXTWFKv8l/btMu6UCJmjRFflN+jqHT01HCiWmeXtESV1 zowEz7TtWQOjPhfJeW7igm73ECKkGA== =62li -----END PGP SIGNATURE----- --=-=-=--
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 31 Jan 2020 15:15:48 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Fri Jan 31 10:15:48 2020 Received: from localhost ([127.0.0.1]:37856 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1ixY1M-0003Rx-Hg for submit <at> debbugs.gnu.org; Fri, 31 Jan 2020 10:15:48 -0500 Received: from mail-qk1-f177.google.com ([209.85.222.177]:43958) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <zimon.toutoune@HIDDEN>) id 1ixY1K-0003Rh-RT for 39258 <at> debbugs.gnu.org; Fri, 31 Jan 2020 10:15:47 -0500 Received: by mail-qk1-f177.google.com with SMTP id j20so6823389qka.10 for <39258 <at> debbugs.gnu.org>; Fri, 31 Jan 2020 07:15:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=uLjKv6khMsD8lK7a1k/F7GVdxZN0Kl+Rk/DrW/HFIuE=; b=l9egGbJoTXnY3x2FpjS3uT960Y/To8ZM0rNnuz8NWdLFTmv8p49OY5C3zzRjCeS0wt nTtO+hi18b/XivDU6k1XE2nc2QUtiIcotOIr3+z1/Idi2F0o3ZTmQe5sXC17OOzwV0nM VVLwsggka4dWfplUdkR3uvrSZmvfprTiVYm5JmVDPbX5KVhgv29Y7aKTNIwfr8Rxnc61 sxklW0N90BtpB3/Dzj6aAZtqaL64s7mCsoVQAkGK39lsvqLWHc1TRslPpHPg7QO/NVgJ UDq8Yo5jTbhX3pnDZvob/HWpWjQbMTTh7CQfBS9Oj1f+45p7odWefbV7NPUP7i/T5izo GN+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=uLjKv6khMsD8lK7a1k/F7GVdxZN0Kl+Rk/DrW/HFIuE=; b=YrhNembiAhbYO2GDu5C7Kpu4FUSm9h3lH73vt9sqSYP39SeM3jqHs0wr6la9IPPW69 9uMPzR2nd4+Y/RLkfn9IyoRvNCj6v+EIaEs5Sx99qv2HUdz0ttQTH2gmMXf533DIPzq/ yb8ZA80i4C2F/7IAAQrox6XaoqP1xyJo+WCncCf0nXPE4EuqQ/r9Yh/5YFuohR5U67G9 73JH0AJT6+o0iABn8UQuSOL2WhBMnENBeuTPnReINqQsoS8/6KL0IsSIdpSS+D2vMe7M NcZQg/mDQSOvn3SOGafGjOXhan7KrbZscUQnSMZK7673MI+vQwTwPdioQQUA1n8AM0Ug /e6g== X-Gm-Message-State: APjAAAW0QDF2IkkG0JKLYpT3hslmOaIX8Gi3UwXHMD2QSDphWhB3a8yM 8i/0fcpj4Ln8orvwE3MqkHn63ZBL8AWkA52tnty5sQ== X-Google-Smtp-Source: APXvYqyehJXpMxoJQ7/aFAj6F0UOgcZtfkAmkZ4wFW/9w+fuEoEXRCkfVxFbyRqYLHLhLfV+kzbx13CxuEmt3r4CXVk= X-Received: by 2002:a37:710:: with SMTP id 16mr10250950qkh.304.1580483741286; Fri, 31 Jan 2020 07:15:41 -0800 (PST) MIME-Version: 1.0 References: <cu7pnfaar36.fsf@HIDDEN> <CAJ3okZ05JXVgbKJd_WeJKERJd6rc4SwxLRrRR0AyW48QDyX8eQ@HIDDEN> <cu74kwdawc7.fsf@HIDDEN> In-Reply-To: <cu74kwdawc7.fsf@HIDDEN> From: zimoun <zimon.toutoune@HIDDEN> Date: Fri, 31 Jan 2020 13:48:16 +0100 Message-ID: <CAJ3okZ0GXS9ibh__1DFsuCCjO82R6-YXtPAOF3GoquN=sSsaWA@HIDDEN> Subject: Re: [bug#39258] Faster guix search using an sqlite cache To: Arun Isaac <arunisaac@HIDDEN> Content-Type: text/plain; charset="UTF-8" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 39258 Cc: 39258 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) Hi, On Thu, 30 Jan 2020 at 14:49, Arun Isaac <arunisaac@HIDDEN> wrote: > >> Any ideas what's going on? [...] > Do you know of any way sqlite can create an error log to report what's > going on? That might really help debug this issue. Danny told me something like: (catch (sqlite-error I have not tried yet. > > The docstring of 'cache-lookup' is not coherent anymore. :-) > > Yes, I haven't gotten around to fixing up all those yet. I thought I'll > get the code working first. Yes, I imagine. Just to notice. :-) All the best, simon
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 30 Jan 2020 13:49:26 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Thu Jan 30 08:49:26 2020 Received: from localhost ([127.0.0.1]:35060 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1ixACE-0004jZ-2k for submit <at> debbugs.gnu.org; Thu, 30 Jan 2020 08:49:26 -0500 Received: from mugam.systemreboot.net ([139.59.75.54]:55156) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <arunisaac@HIDDEN>) id 1ixACA-0004jL-Fw for 39258 <at> debbugs.gnu.org; Thu, 30 Jan 2020 08:49:24 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=systemreboot.net; s=default; h=Content-Type:MIME-Version:Message-ID:Date: References:In-Reply-To:Subject:Cc:To:From:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=po6WSlvm76W+cyGEAyCt0xLGxVCdJ4XqVkHzxhbvwms=; b=b6/ogKEfMeZiLTZlTcqX8goIO e7LR9+siJ21xW/bpjup5wdEI6vG7m9IqLXeDg3XSYfmqaY6cq/AfyDAgIMD36EfZRS6CK8b9asRci 2dDEymoSSndhamPODvbTDPt8N19e9a0hKt83HH5Ntodvr78+1FpMGLO2bGV7RriraLaXA=; Received: from [192.168.2.1] (helo=steel) by systemreboot.net with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.93) (envelope-from <arunisaac@HIDDEN>) id 1ixABz-000WUq-Mj; Thu, 30 Jan 2020 19:19:11 +0530 From: Arun Isaac <arunisaac@HIDDEN> To: zimoun <zimon.toutoune@HIDDEN> Subject: Re: [bug#39258] Faster guix search using an sqlite cache In-Reply-To: <CAJ3okZ05JXVgbKJd_WeJKERJd6rc4SwxLRrRR0AyW48QDyX8eQ@HIDDEN> References: <cu7pnfaar36.fsf@HIDDEN> <CAJ3okZ05JXVgbKJd_WeJKERJd6rc4SwxLRrRR0AyW48QDyX8eQ@HIDDEN> Date: Thu, 30 Jan 2020 19:18:56 +0530 Message-ID: <cu74kwdawc7.fsf@HIDDEN> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 39258 Cc: 39258 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain > Hum? weird... > Is it possible that a module is loaded when Guile repl is used and not > with Guix pull? It could be. But I don't know how to confirm this theory. > What about "guix repl"? I just tried 'guix repl'. It worked correctly, just like guile repl. >> If I try to populate the database with each package record being >> inserted in its own transaction, at least some of the insertions > > You mean 'commit' the database after each insertion, right? Yes, that is what I mean. >> work. But the journal file still lingers. My unverified guess is that >> everything except the last transaction was successful. > > And this does not happen with the repl, right? No, this does not happen with the repl. >> Any ideas what's going on? > > I have no idea. > Weird. Do you know of any way sqlite can create an error log to report what's going on? That might really help debug this issue. > What about adding 'last-insert-row-id' from 'guix/store/database.scm'? > I mean without really understanding and just grepping > 'sqlite-finalize' spots that 'last-insert-row-id' seems often around. > :-) I tried this just now, but still the journal lingers. > Otherwise, 'list->string' is defined twice. And the first one is not > necessary, I guess. Ah, thanks for catching this! > The docstring of 'cache-lookup' is not coherent anymore. :-) Yes, I haven't gotten around to fixing up all those yet. I thought I'll get the code working first. --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEf3MDQ/Lwnzx3v3nTLiXui2GAK7MFAl4y3sgACgkQLiXui2GA K7O5mQf/QRSZdYPR4DAVXlfg04OEoCKgTi4m9SLL3ByJjMazxfuR9gXS6o97Sr/p 5IH2tchH2WwXM/ZN1R8a5utlrWR49azy9Z9uW+b2Bk48aQkp2l5uLz+uQDekBZ+z o7gIYA+dnuGtJMlLG90kB2P6TP1LoaOm0rZyOCRPb4GoNnIjv3v9yxEU2Z7vMcbQ 6AOd/lj2Owr4Ur3BrXn5+XNfZO34uXMHHleZq/R7cwxPqmK8xj7xsArPG0/5o2XN vJYVe9AC8J2DMEcyU5wafxs9DebxnuZZAB3umSQeUuFiLIT34fkBSQmW2vEIvaC2 R5zEZ9E6PqcpRz8K7rp18z8hcX+jVw== =Akh8 -----END PGP SIGNATURE----- --=-=-=--
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.Received: (at 39258) by debbugs.gnu.org; 29 Jan 2020 23:34:15 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Wed Jan 29 18:34:15 2020 Received: from localhost ([127.0.0.1]:34709 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1iwwqc-0003Yo-So for submit <at> debbugs.gnu.org; Wed, 29 Jan 2020 18:34:15 -0500 Received: from mail-qt1-f172.google.com ([209.85.160.172]:46296) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <zimon.toutoune@HIDDEN>) id 1iwwqb-0003Yb-IF for 39258 <at> debbugs.gnu.org; Wed, 29 Jan 2020 18:34:14 -0500 Received: by mail-qt1-f172.google.com with SMTP id e25so944644qtr.13 for <39258 <at> debbugs.gnu.org>; Wed, 29 Jan 2020 15:34:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=LoCPANSQcRYKL7WvsA1UPP8c/yFpkPntg+GlDHw5LfQ=; b=nOUNoINq2LgFQVkvghW+wmN65dXoTfG1RDsbw8g7PjauqVkLS7p9IWwrzalMmYRar7 OtNPHgQXtBJtLcSovSsPJH099NyJUegPvYCMqqjFDCm89XO0WcFHtDNONMgGwNDtRFle uqE9G0S7u2TnQ/x8eu8O/9JF384W5pPMlpLdqXJiGsPtxO+MGJloZWtIeJYo+tEQNmtk Fp4ka7Dx9+XjA560MD7DSdc3SZmtTeJ7a3uLaqSBF2Yz8DYlwelxMm7v5KC9gUguAdi8 y9IPzjedL2nDLgiU14/ojUNsBaEpJon1ESUV6IKUna3Uu/2P7Phy22bV/57QKPzN+YUM oL/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=LoCPANSQcRYKL7WvsA1UPP8c/yFpkPntg+GlDHw5LfQ=; b=DP2QMenHuANFHPaLyyYUFhBJ6GezBr7LyW/pLw0d6RhnaU8WRafcTfBufS6CTw05Gk PoQ4olKqYZo2Fo6PR97s6SbOnewE8gfjASNjAsxHCale4jlC2GRaf/sJ9BlKV0Kgbfii wFOLbd8OTEP0fP/DBxPv8nJulafFhdEqQTjmtD0ho2xOY5RLjGGq8JrpHeUYZeyfH0fA xfcDPUJgxVisimDAPCRU6muua4RHJHA/TOwEi5/XWGQ6WIxVshkyohek+/XYQuLMDFJl js9tmxT4VRLumNVJS9dvUFashX08tHH39whk4j/mdi5O2KAVB85H7Fo+1Lbrg2bsQHq4 1LMw== X-Gm-Message-State: APjAAAUbFmYBaK9yF2sWO1W3E8DUMLZL0fToJWM9m3aha4D2HhSHl7Wu rNIZ3JieuYD4lcnO/HlprbtTryWyP7umMXtT6McRLfF8eSk= X-Google-Smtp-Source: APXvYqxvZjie7g6Bak2iifeJhhT+gOSKffcNZJUzUXfyzaz3AIMKTDA+MP1eofrQA9rGFDBNoVWY1XlIIVUcIV43QkU= X-Received: by 2002:ac8:2e53:: with SMTP id s19mr1967828qta.354.1580340847857; Wed, 29 Jan 2020 15:34:07 -0800 (PST) MIME-Version: 1.0 References: <cu7pnfaar36.fsf@HIDDEN> In-Reply-To: <cu7pnfaar36.fsf@HIDDEN> From: zimoun <zimon.toutoune@HIDDEN> Date: Thu, 30 Jan 2020 00:33:56 +0100 Message-ID: <CAJ3okZ05JXVgbKJd_WeJKERJd6rc4SwxLRrRR0AyW48QDyX8eQ@HIDDEN> Subject: Re: [bug#39258] Faster guix search using an sqlite cache To: Arun Isaac <arunisaac@HIDDEN> Content-Type: text/plain; charset="UTF-8" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 39258 Cc: 39258 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) Hi Arun, Thank you for the patch! Cool! :-) I have not tested it yet. Sorry. On Thu, 23 Jan 2020 at 20:53, Arun Isaac <arunisaac@HIDDEN> wrote: > At the moment, I am having some difficulty populating the sqlite > database. generate-package-cache populates the database correctly when > invoked from a normal guile REPL using geiser, but fails to do so when > run by the guix daemon during guix pull. [...] > On examining package-cache.sqlite, I find that no records have been > written. And, there is a lingering journal file that shouldn't be > there. For some reason, populating the sqlite database does not work > with guix pull. sqlite probably crashes and leaves the journal file. Hum? weird... Is it possible that a module is loaded when Guile repl is used and not with Guix pull? What about "guix repl"? > If I try to populate the database with each package record being > inserted in its own transaction, at least some of the insertions You mean 'commit' the database after each insertion, right? > work. But the journal file still lingers. My unverified guess is that > everything except the last transaction was successful. And this does not happen with the repl, right? > Any ideas what's going on? I have no idea. Weird. What about adding 'last-insert-row-id' from 'guix/store/database.scm'? I mean without really understanding and just grepping 'sqlite-finalize' spots that 'last-insert-row-id' seems often around. :-) > Also, inserting each package in its own transaction is ridiculously slow > and so that is out of the question. See https://www.sqlite.org/faq.html#q19 Agree that it is not an option. :-) Otherwise, 'list->string' is defined twice. And the first one is not necessary, I guess. The docstring of 'cache-lookup' is not coherent anymore. :-) Cheers, simon
guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.
Received: (at submit) by debbugs.gnu.org; 23 Jan 2020 19:52:33 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Jan 23 14:52:33 2020
Received: from localhost ([127.0.0.1]:53208 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1iuiWn-00019K-8s
for submit <at> debbugs.gnu.org; Thu, 23 Jan 2020 14:52:33 -0500
Received: from lists.gnu.org ([209.51.188.17]:55538)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <arunisaac@HIDDEN>) id 1iuiWk-00019A-Ju
for submit <at> debbugs.gnu.org; Thu, 23 Jan 2020 14:52:31 -0500
Received: from eggs.gnu.org ([2001:470:142:3::10]:38921)
by lists.gnu.org with esmtp (Exim 4.90_1)
(envelope-from <arunisaac@HIDDEN>) id 1iuiWh-0008EY-Ap
for guix-patches@HIDDEN; Thu, 23 Jan 2020 14:52:30 -0500
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level:
X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50,URIBL_BLOCKED
autolearn=disabled version=3.3.2
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
(envelope-from <arunisaac@HIDDEN>) id 1iuiWe-00087M-FS
for guix-patches@HIDDEN; Thu, 23 Jan 2020 14:52:26 -0500
Received: from mugam.systemreboot.net ([139.59.75.54]:60578)
by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)
(Exim 4.71) (envelope-from <arunisaac@HIDDEN>)
id 1iuiWd-00082X-4H
for guix-patches@HIDDEN; Thu, 23 Jan 2020 14:52:24 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed;
d=systemreboot.net; s=default; h=Content-Type:MIME-Version:Message-ID:Date:
Subject:To:From:Sender:Reply-To:Cc:Content-Transfer-Encoding:Content-ID:
Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc
:Resent-Message-ID:In-Reply-To:References:List-Id:List-Help:List-Unsubscribe:
List-Subscribe:List-Post:List-Owner:List-Archive;
bh=m58VqhXMlYNbWwgkqnBfSm2SHXEBcN2pNnULcRktOt0=; b=lxKPiLfEtbVbt5FKBuTu0mWTDC
vVyWwl+Ps0w4MfGSzWCpqt36Yvby5jcnx5o4BkhoKEii3opfHxnin65h/IynHQxTP3DjU8DMP72y9
tsh9x6HEU/Y7q3hJHhg882OczgmkYenh6z5R/zKtLzShLalFSabpDgQoHLBIpqA3DUFY=;
Received: from [192.168.2.1] (helo=steel)
by systemreboot.net with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384
(Exim 4.93) (envelope-from <arunisaac@HIDDEN>)
id 1iuiWX-0009z3-Pm
for guix-patches@HIDDEN; Fri, 24 Jan 2020 01:22:17 +0530
From: Arun Isaac <arunisaac@HIDDEN>
To: guix-patches@HIDDEN
Subject: Faster guix search using an sqlite cache
Date: Fri, 24 Jan 2020 01:21:57 +0530
Message-ID: <cu7pnfaar36.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="==-=-=";
micalg=pgp-sha256; protocol="application/pgp-signature"
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
recognized.
X-Received-From: 139.59.75.54
X-Spam-Score: 0.2 (/)
X-Debbugs-Envelope-To: submit
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.8 (/)
--==-=-=
Content-Type: multipart/mixed; boundary="=-=-="
--=-=-=
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Hi,
As discussed on guix-devel at
https://lists.gnu.org/archive/html/guix-devel/2020-01/msg00310.html , I
am working on an sqlite cache to improve guix search performance. I have
attached a highly incomplete WIP patch. The patch attempts to
reimplement the package-cache-file hook in guix/channels.scm using a
sqlite database. To this end, it rewrites most of the
generate-package-cache and cache-lookup functions in gnu/packages.scm. I
am yet to hook this up to guix search.
At the moment, I am having some difficulty populating the sqlite
database. generate-package-cache populates the database correctly when
invoked from a normal guile REPL using geiser, but fails to do so when
run by the guix daemon during guix pull.
I ran guix pull using
$ ./pre-inst-env guix pull --url=3D$PWD --branch=3Dsearch -p /tmp/test
where search is the branch I am working on.
Running
$ ls /tmp/test/lib/guix -lh
shows
total 2.1M
=2Dr--r--r-- 2 root root 2.1M =E0=AE=9C=E0=AE=A9. 1 1970 package-cache.s=
qlite
=2Dr--r--r-- 2 root root 26K =E0=AE=9C=E0=AE=A9. 1 1970 package-cache.s=
qlite-journal
On examining package-cache.sqlite, I find that no records have been
written. And, there is a lingering journal file that shouldn't be
there. For some reason, populating the sqlite database does not work
with guix pull. sqlite probably crashes and leaves the journal file.
If I try to populate the database with each package record being
inserted in its own transaction, at least some of the insertions
work. But the journal file still lingers. My unverified guess is that
everything except the last transaction was successful.
Any ideas what's going on?
Also, inserting each package in its own transaction is ridiculously slow
and so that is out of the question. See https://www.sqlite.org/faq.html#q19
--=-=-=
Content-Type: text/x-patch
Content-Disposition: inline; filename=0001-fast-search.patch
Content-Transfer-Encoding: quoted-printable
From=20d1305351a90a84eb75e4769284d5e06927eade3e Mon Sep 17 00:00:00 2001
From: Arun Isaac <arunisaac@HIDDEN>
Date: Tue, 21 Jan 2020 20:45:43 +0530
Subject: [PATCH] fast search
=2D--
build-aux/build-self.scm | 5 +
gnu/packages.scm | 207 +++++++++++++++++++++++----------------
2 files changed, 128 insertions(+), 84 deletions(-)
diff --git a/build-aux/build-self.scm b/build-aux/build-self.scm
index fc13032b73..c123ad3b11 100644
=2D-- a/build-aux/build-self.scm
+++ b/build-aux/build-self.scm
@@ -264,6 +264,9 @@ interface (FFI) of Guile.")
(define fake-git
(scheme-file "git.scm" #~(define-module (git))))
=20
+ (define fake-sqlite3
+ (scheme-file "sqlite3.scm" #~(define-module (sqlite3))))
+
(with-imported-modules `(((guix config)
=3D> ,(make-config.scm))
=20
@@ -278,6 +281,8 @@ interface (FFI) of Guile.")
;; (git) to placate it.
((git) =3D> ,fake-git)
=20
+ ((sqlite3) =3D> ,fake-sqlite3)
+
,@(source-module-closure `((guix store)
(guix self)
(guix derivations)
diff --git a/gnu/packages.scm b/gnu/packages.scm
index d22c992bb1..4e2c52e62d 100644
=2D-- a/gnu/packages.scm
+++ b/gnu/packages.scm
@@ -43,6 +43,7 @@
#:use-module (srfi srfi-34)
#:use-module (srfi srfi-35)
#:use-module (srfi srfi-39)
+ #:use-module (sqlite3)
#:export (search-patch
search-patches
search-auxiliary-file
@@ -204,10 +205,8 @@ PROC is called along these lines:
PROC can use #:allow-other-keys to ignore the bits it's not interested in.
When a package cache is available, this procedure does not actually load a=
ny
package module."
=2D (define cache
=2D (load-package-cache (current-profile)))
=2D
=2D (if (and cache (cache-is-authoritative?))
+ (if (and (cache-is-authoritative?)
+ (current-profile))
(vhash-fold (lambda (name vector result)
(match vector
(#(name version module symbol outputs
@@ -220,7 +219,7 @@ package module."
#:supported? supported?
#:deprecated? deprecated?))))
init
=2D cache)
+ (cache-lookup (current-profile)))
(fold-packages (lambda (package result)
(proc (package-name package)
(package-version package)
@@ -252,31 +251,7 @@ is guaranteed to never traverse the same package twice=
."
=20
(define %package-cache-file
;; Location of the package cache.
=2D "/lib/guix/package.cache")
=2D
=2D(define load-package-cache
=2D (mlambda (profile)
=2D "Attempt to load the package cache. On success return a vhash keyed=
by
=2Dpackage names. Return #f on failure."
=2D (match profile
=2D (#f #f)
=2D (profile
=2D (catch 'system-error
=2D (lambda ()
=2D (define lst
=2D (load-compiled (string-append profile %package-cache-file)))
=2D (fold (lambda (item vhash)
=2D (match item
=2D (#(name version module symbol outputs
=2D supported? deprecated?
=2D file line column)
=2D (vhash-cons name item vhash))))
=2D vlist-null
=2D lst))
=2D (lambda args
=2D (if (=3D ENOENT (system-error-errno args))
=2D #f
=2D (apply throw args))))))))
+ "/lib/guix/package-cache.sqlite")
=20
(define find-packages-by-name/direct ;bypass the cache
(let ((packages (delay
@@ -297,25 +272,57 @@ decreasing version order."
matching)
matching)))))
=20
=2D(define (cache-lookup cache name)
+(define* (cache-lookup profile #:optional name)
"Lookup package NAME in CACHE. Return a list sorted in increasing versi=
on
order."
(define (package-version<? v1 v2)
(version>? (vector-ref v2 1) (vector-ref v1 1)))
=20
=2D (sort (vhash-fold* cons '() name cache)
=2D package-version<?))
+ (define (int->boolean n)
+ (case n
+ ((0) #f)
+ ((1) #t)))
+
+ (define (string->list str)
+ (call-with-input-string str read))
+
+ (define select-statement
+ (string-append
+ "SELECT name, version, module, symbol, outputs, supported, superseded=
, locationFile, locationLine, locationColumn from packages"
+ (if name " WHERE name =3D :name" "")))
+
+ (define cache-file
+ (string-append profile %package-cache-file))
+
+ (let* ((db (sqlite-open cache-file SQLITE_OPEN_READONLY))
+ (statement (sqlite-prepare db select-statement)))
+ (when name
+ (sqlite-bind-arguments statement #:name name))
+ (let ((result (sqlite-fold (lambda (v result)
+ (match v
+ (#(name version module symbol outputs s=
upported superseded file line column)
+ (cons
+ (vector name
+ version
+ (string->list module)
+ (string->symbol symbol)
+ (string->list outputs)
+ (int->boolean supported)
+ (int->boolean superseded)
+ (list file line column))
+ result))))
+ '() statement)))
+ (sqlite-finalize statement)
+ (sqlite-close db)
+ (sort result package-version<?))))
=20
(define* (find-packages-by-name name #:optional version)
"Return the list of packages with the given NAME. If VERSION is not #f,
then only return packages whose version is prefixed by VERSION, sorted in
decreasing version order."
=2D (define cache
=2D (load-package-cache (current-profile)))
=2D
=2D (if (and (cache-is-authoritative?) cache)
=2D (match (cache-lookup cache name)
=2D (#f #f)
+ (if (and (cache-is-authoritative?)
+ (current-profile))
+ (match (cache-lookup (current-profile) name)
((#(_ versions modules symbols _ _ _ _ _ _) ...)
(fold (lambda (version* module symbol result)
(if (or (not version)
@@ -331,12 +338,9 @@ decreasing version order."
(define* (find-package-locations name #:optional version)
"Return a list of version/location pairs corresponding to each package
matching NAME and VERSION."
=2D (define cache
=2D (load-package-cache (current-profile)))
=2D
=2D (if (and cache (cache-is-authoritative?))
=2D (match (cache-lookup cache name)
=2D (#f '())
+ (if (and (cache-is-authoritative?)
+ (current-profile))
+ (match (cache-lookup (current-profile) name)
((#(name versions modules symbols outputs
supported? deprecated?
files lines columns) ...)
@@ -372,6 +376,9 @@ VERSION."
;; Prevent Guile 3 from inlining this procedure so we can mock it in tests.
(set! find-best-packages-by-name find-best-packages-by-name)
=20
+(define (list->string x)
+ (call-with-output-string (cut write x <>)))
+
(define (generate-package-cache directory)
"Generate under DIRECTORY a cache of all the available packages.
=20
@@ -381,49 +388,81 @@ reducing the memory footprint."
(define cache-file
(string-append directory %package-cache-file))
=20
=2D (define (expand-cache module symbol variable result+seen)
+ (define schema
+ "CREATE TABLE packages (name text,
+version text,
+module text,
+symbol text,
+outputs text,
+supported int,
+superseded int,
+locationFile text,
+locationLine int,
+locationColumn int);
+CREATE VIRTUAL TABLE packageSearch USING fts5(name, searchText);")
+
+ (define insert-statement
+ "INSERT INTO packages(name, version, module, symbol, outputs, supporte=
d, superseded, locationFile, locationLine, locationColumn)
+VALUES(:name, :version, :module, :symbol, :outputs, :supported, :supersede=
d, :locationfile, :locationline, :locationcolumn)")
+
+ (define insert-package-search-statement
+ "INSERT INTO packageSearch(name, searchText) VALUES(:name, :searchtext=
)")
+
+ (define (boolean->int x)
+ (if x 1 0))
+
+ (define (list->string x)
+ (call-with-output-string (cut write x <>)))
+
+ (define (insert-package db module symbol variable seen)
(match (false-if-exception (variable-ref variable))
((? package? package)
=2D (match result+seen
=2D ((result . seen)
=2D (if (or (vhash-assq package seen)
=2D (hidden-package? package))
=2D (cons result seen)
=2D (cons (cons `#(,(package-name package)
=2D ,(package-version package)
=2D ,(module-name module)
=2D ,symbol
=2D ,(package-outputs package)
=2D ,(->bool (supported-package? package))
=2D ,(->bool (package-superseded package))
=2D ,@(let ((loc (package-location package)))
=2D (if loc
=2D `(,(location-file loc)
=2D ,(location-line loc)
=2D ,(location-column loc))
=2D '(#f #f #f))))
=2D result)
=2D (vhash-consq package #t seen))))))
=2D (_
=2D result+seen)))
=2D
=2D (define exp
=2D (first
=2D (fold-module-public-variables* expand-cache
=2D (cons '() vlist-null)
=2D (all-modules (%package-module-path)
=2D #:warn
=2D warn-about-load-error))=
))
+ (cond
+ ((or (vhash-assq package seen)
+ (hidden-package? package))
+ seen)
+ (else
+ (let ((statement (sqlite-prepare db insert-statement)))
+ (sqlite-bind-arguments statement
+ #:name (package-name package)
+ #:version (package-version package)
+ #:module (list->string (module-name modu=
le))
+ #:symbol (symbol->string symbol)
+ #:outputs (list->string (package-outputs=
package))
+ #:supported (boolean->int (supported-pac=
kage? package))
+ #:superseded (boolean->int (package-supe=
rseded package))
+ #:locationfile (cond
+ ((package-location packa=
ge) =3D> location-file)
+ (else #f))
+ #:locationline (cond
+ ((package-location packa=
ge) =3D> location-line)
+ (else #f))
+ #:locationcolumn (cond
+ ((package-location pac=
kage) =3D> location-column)
+ (else #f)))
+ (sqlite-fold cons '() statement)
+ (sqlite-finalize statement))
+ (let ((statement (sqlite-prepare db insert-package-search-stateme=
nt)))
+ (sqlite-bind-arguments statement
+ #:name (package-name package)
+ #:searchtext (package-description packag=
e))
+ (sqlite-fold cons '() statement)
+ (sqlite-finalize statement))
+ (vhash-consq package #t seen))))
+ (_ seen)))
=20
(mkdir-p (dirname cache-file))
=2D (call-with-output-file cache-file
=2D (lambda (port)
=2D ;; Store the cache as a '.go' file. This makes loading fast and r=
educes
=2D ;; heap usage since some of the static data is directly mmapped.
=2D (put-bytevector port
=2D (compile `'(,@exp)
=2D #:to 'bytecode
=2D #:opts '(#:to-file? #t)))))
+ (let ((db (sqlite-open cache-file)))
+ (sqlite-exec db schema)
+ (sqlite-exec db "BEGIN")
+ (fold-module-public-variables* (cut insert-package db <> <> <> <>)
+ vlist-null
+ (all-modules (%package-module-path)
+ #:warn
+ warn-about-load-error))
+ (sqlite-exec db "COMMIT;")
+ (sqlite-close db))
+
cache-file)
=20
=2D-=20
2.23.0
--=-=-=--
--==-=-=
Content-Type: application/pgp-signature; name="signature.asc"
-----BEGIN PGP SIGNATURE-----
iQEzBAEBCAAdFiEEf3MDQ/Lwnzx3v3nTLiXui2GAK7MFAl4p+V0ACgkQLiXui2GA
K7Oe/wf/VUd/Gcd+KFyJxXTfML9vGIxR7xUXl6M92mevuADXCxI4JECqMmjIuj9f
ZbC6o/D+XcnJz7XyttQZi/iyjbwZIA0DwUbdAg5BRP8cK6ZkCflPfjamWNQ2RVYu
2S+oITgatidZTLDTFGP6RYeXN27I+fkK5P28XSJHa69aE34bVor0R3bb7Ki57OVS
+cEYu6nlGbADqpFpLT6VjB7ewgr9wt0tQyq721JevZzi3PNb+WVq6Pi2N69nHDdO
kslYfUVi2kWnN9i3gtBnEwVo2cj2uaD2eSYd5YoA2c5kxHVomo5/CiuDUqdCyxd+
/Xbjbug+6C7SeCcBbfVW0RgRo2Dvng==
=HLpI
-----END PGP SIGNATURE-----
--==-=-=--
Arun Isaac <arunisaac@HIDDEN>:guix-patches@HIDDEN.
Full text available.guix-patches@HIDDEN:bug#39258; Package guix-patches.
Full text available.
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997 nCipher Corporation Ltd,
1994-97 Ian Jackson.