GNU bug report logs - #37871
27.0.50; Merge feature/gnus-select2

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 37871 in the body.
You can then email your comments to 37871 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Eric Abrahamsen <eric <at> ericabrahamsen.net>
To: bug-gnu-emacs <at> gnu.org
Cc: Andrew Cohen <cohen <at> bu.edu>, Lars Ingebrigtsen <larsi <at> gnus.org>
Subject: 27.0.50; Merge feature/gnus-select2
Date: Tue, 22 Oct 2019 09:14:43 -0700

Hi all, Andy Cohen's nnselect Gnus backend looks ready to merge -- he
wrote to emacs.devel but it occurs to me it might be better to start
here. His original message is here, and I've pasted the full text below
the link:

https://lists.gnu.org/archive/html/emacs-devel/2019-10/msg00689.html

------

After many months I have updated the gnus-select2 feature branch. This
incorporates a number of bug fixes as well as updating to the latest
master. I have been using this version (or something close to it) for
nearly a year with no problems. For those who use gnus please check it
out and if no obvious bugs appear I will figure out how to push this
into master.

This branch introduces a new "virtual" backend, nnselect, that allows an
arbitrary list of messages to be treated as a normal gnus group. The
messages may come from multiple servers and multiple backends. The
resulting group behaves in (almost) all respects like any real group:
messages may be threaded, marked, moved, deleted, copied, etc. nnselect
groups may be ephemeral or persistent.

An obvious use case for nnselect is to create a functional group out of
the messages produced by some kind of search. In this feature branch
searching (performed by nnir) simply returns a list of articles that is
passed to nnselect, which handles the group creation and message
manipulation. (This should be transparent to the user---that is,
searching should appear to function as before, except that the group
created from the search has more functionality).

Below is a first pass at some documentation that hasn't yet been pushed
into the gnus manual on the branch.

=nnselect= is a select method for virtual groups composed of messages
from other groups that may span multiple servers and backends. For the
most part these groups behave like any other group: messages may be
threaded, marked, moved, deleted, copied, etc.; groups may be ephemeral
or persistent; groups may be created via gnus-group-make-group or
browsed as foreign via gnus-group-browse-foreign-server.

The key to using an nnselect group is specifying the messages to
include. Each nnselect group has a group parameter =nnselect-specs=
which is an alist with two elements: a function =nnselect-function=; and
arguments =nnselect-args= to be passed to the function, if any.

The function =nnselect-function= must return a vector. Each element of
this vector is in turn a 3-element vector corresponding to one
message. The 3 elements are: the fully-qualified group name; the message
number; and a "score" that can be used for additional sorting. The
values for the score are arbitrary, and are not used directly by the
nnselect method---they may, for example, all be set to 100.

As an example we might have a group whose nnselect-specs are

 (nnselect-specs
  (nnselect-function . identity)
  (nnselect-args .
                 [["nnimap+work:mail" 595 100]
                  ["nnimap+home:sent" 223 100]
                ["nntp+news.gmane.org:gmane.emacs.gnus.general" 23666 100])))

Here the function is the identity and the argument is just a list of
messages.

Or the function might be a search query (using nnir):

 (nnselect-specs
  (nnselect-function . nnir-run-query)
  (nnselect-args
   (nnir-query-spec
    (query . "FLAGGED")
    (criteria . ""))
   (nnir-group-spec
    ("nnimap:home")
    ("nnimap:work"))))

This creates a group composed of all flagged messages on two imap
servers, "home" and "work".

And one last example. Here is a function that runs a search query to
find all message that have been received over the past =days=

(defun my-recent-email (days)
      (let ((query-spec
             (list
              (cons 'query
                    (format-time-string "SENTSINCE %d-%b-%Y"
                                        (time-subtract (current-time)
                                                       (days-to-time days))))
              (cons 'criteria "")))
            (group-spec  (list (list "nnimap:home") (list "nnimap:work"))))
        (nnir-run-query   (cons 'nnir-specs
                                (list (cons 'nnir-query-spec query-spec)
                                      (cons 'nnir-group-spec group-spec))))))

Then an nnselect-specs 

 (nnselect-specs
  (nnselect-function . my-recent-email)
  (nnselect-args . 7))

will provide a group composed of all messages on the home and work
servers received in the last 7 days.

** scanning

Refreshing the selection of an nnselect group by running the
nnselect-function may take a long time to complete. Consequently
nnselect groups are not refreshed by default when
=gnus-group-get-new-news= is invoked. In those cases where running the
function is not too time-consuming, a group parameter of nnselect-rescan
can be set to allow automatic refreshing. A refresh can always be
invoked manually through =gnus-group-get-new-news-this-group=.

Message #8 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Eric Abrahamsen <eric <at> ericabrahamsen.net>
Cc: Andrew Cohen <cohen <at> bu.edu>, bug-gnu-emacs <at> gnu.org
Subject: Re: 27.0.50; Merge feature/gnus-select2
Date: Tue, 22 Oct 2019 20:13:06 +0200

Eric Abrahamsen <eric <at> ericabrahamsen.net> writes:

> Hi all, Andy Cohen's nnselect Gnus backend looks ready to merge -- he
> wrote to emacs.devel but it occurs to me it might be better to start
> here. His original message is here, and I've pasted the full text below
> the link:
>
> https://lists.gnu.org/archive/html/emacs-devel/2019-10/msg00689.html

Sounds nice.  Is there any impact on Gnus itself, or is it all
constrained to the new backend?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #11 received at 37871 <at> debbugs.gnu.org (full text, mbox):

From: Eric Abrahamsen <eric <at> ericabrahamsen.net>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: cohen <at> bu.edu, 37871 <at> debbugs.gnu.org
Subject: Re: bug#37871: 27.0.50; Merge feature/gnus-select2
Date: Tue, 22 Oct 2019 11:26:59 -0700

Lars Ingebrigtsen <larsi <at> gnus.org> writes:

> Eric Abrahamsen <eric <at> ericabrahamsen.net> writes:
>
>> Hi all, Andy Cohen's nnselect Gnus backend looks ready to merge -- he
>> wrote to emacs.devel but it occurs to me it might be better to start
>> here. His original message is here, and I've pasted the full text below
>> the link:
>>
>> https://lists.gnu.org/archive/html/emacs-devel/2019-10/msg00689.html
>
> Sounds nice.  Is there any impact on Gnus itself, or is it all
> constrained to the new backend?

It shifts nnir over to use nnselect for holding search results, I think
that's all. It makes nnir much easier to use (in the hacking sense,
should be transparent for users). It also let me write the unified
gnus-search interface on top of it, which wasn't possible with the
current implementation of nnir.

Message #14 received at 37871 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Eric Abrahamsen <eric <at> ericabrahamsen.net>
Cc: cohen <at> bu.edu, 37871 <at> debbugs.gnu.org
Subject: Re: bug#37871: 27.0.50; Merge feature/gnus-select2
Date: Tue, 22 Oct 2019 20:31:07 +0200

Eric Abrahamsen <eric <at> ericabrahamsen.net> writes:

> It shifts nnir over to use nnselect for holding search results, I think
> that's all. It makes nnir much easier to use (in the hacking sense,
> should be transparent for users). It also let me write the unified
> gnus-search interface on top of it, which wasn't possible with the
> current implementation of nnir.

Sounds great.  Merge away.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #17 received at 37871 <at> debbugs.gnu.org (full text, mbox):

From: Eric Abrahamsen <eric <at> ericabrahamsen.net>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: cohen <at> bu.edu, 37871 <at> debbugs.gnu.org
Subject: Re: bug#37871: 27.0.50; Merge feature/gnus-select2
Date: Tue, 22 Oct 2019 12:39:00 -0700

Lars Ingebrigtsen <larsi <at> gnus.org> writes:

> Eric Abrahamsen <eric <at> ericabrahamsen.net> writes:
>
>> It shifts nnir over to use nnselect for holding search results, I think
>> that's all. It makes nnir much easier to use (in the hacking sense,
>> should be transparent for users). It also let me write the unified
>> gnus-search interface on top of it, which wasn't possible with the
>> current implementation of nnir.
>
> Sounds great.  Merge away.  :-)

Andy, do you want to do the honors?

Message #20 received at 37871 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Eric Abrahamsen <eric <at> ericabrahamsen.net>
Cc: cohen <at> bu.edu, 37871 <at> debbugs.gnu.org
Subject: Re: bug#37871: 27.0.50; Merge feature/gnus-select2
Date: Sun, 19 Jul 2020 17:35:09 +0200

Eric Abrahamsen <eric <at> ericabrahamsen.net> writes:

>> Sounds great.  Merge away.  :-)
>
> Andy, do you want to do the honors?

This was in October 2019, and nnselect.el doesn't seem to exist yet.
:-)  Somebody should get mergin'.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #23 received at 37871 <at> debbugs.gnu.org (full text, mbox):

From: Eric Abrahamsen <eric <at> ericabrahamsen.net>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: cohen <at> bu.edu, 37871 <at> debbugs.gnu.org
Subject: Re: bug#37871: 27.0.50; Merge feature/gnus-select2
Date: Sun, 19 Jul 2020 22:26:24 -0700

Lars Ingebrigtsen <larsi <at> gnus.org> writes:

> Eric Abrahamsen <eric <at> ericabrahamsen.net> writes:
>
>>> Sounds great.  Merge away.  :-)
>>
>> Andy, do you want to do the honors?
>
> This was in October 2019, and nnselect.el doesn't seem to exist yet.
> :-)  Somebody should get mergin'.

It's possible Andy was waiting for me to finish my *-retrieve-headers
patch, allowing backends to return headers directly, before he merged
nnselect, as that patch affects how that backend works. Either way, I
really need to take half a day and finish that patch (it's delicate, and
needs some consideration). And Andy should be the one to do the nnselect
merge.

Message #26 received at 37871 <at> debbugs.gnu.org (full text, mbox):

From: Eric Abrahamsen <eric <at> ericabrahamsen.net>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: cohen <at> bu.edu, 37871 <at> debbugs.gnu.org
Subject: Re: bug#37871: 27.0.50; Merge feature/gnus-select2
Date: Fri, 04 Sep 2020 18:31:21 -0700

Eric Abrahamsen <eric <at> ericabrahamsen.net> writes:

> Lars Ingebrigtsen <larsi <at> gnus.org> writes:
>
>> Eric Abrahamsen <eric <at> ericabrahamsen.net> writes:
>>
>>>> Sounds great.  Merge away.  :-)
>>>
>>> Andy, do you want to do the honors?
>>
>> This was in October 2019, and nnselect.el doesn't seem to exist yet.
>> :-)  Somebody should get mergin'.
>
> It's possible Andy was waiting for me to finish my *-retrieve-headers
> patch, allowing backends to return headers directly, before he merged
> nnselect, as that patch affects how that backend works. Either way, I
> really need to take half a day and finish that patch (it's delicate, and
> needs some consideration). And Andy should be the one to do the nnselect
> merge.

And it's in!

Lars, what do you think about bumping the Gnus version number for big
changes like this? If only to make bug triage easier. In retrospect I
wish I'd done this for the obarray->hashtable change.

Message #29 received at 37871 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Eric Abrahamsen <eric <at> ericabrahamsen.net>
Cc: cohen <at> bu.edu, 37871 <at> debbugs.gnu.org
Subject: Re: bug#37871: 27.0.50; Merge feature/gnus-select2
Date: Sat, 05 Sep 2020 14:58:35 +0200

Eric Abrahamsen <eric <at> ericabrahamsen.net> writes:

> Lars, what do you think about bumping the Gnus version number for big
> changes like this? If only to make bug triage easier. In retrospect I
> wish I'd done this for the obarray->hashtable change.

Hm...  I'm not sure this is a big enough reason to bump the version
number in itself, but it's sure been a while since it's been increased.
The main reason is, of course, that I haven't done the t-shirts for Ma
Gnus yet.  :-/

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #32 received at 37871 <at> debbugs.gnu.org (full text, mbox):

From: Eric Abrahamsen <eric <at> ericabrahamsen.net>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: cohen <at> bu.edu, 37871 <at> debbugs.gnu.org
Subject: Re: bug#37871: 27.0.50; Merge feature/gnus-select2
Date: Sat, 05 Sep 2020 08:44:37 -0700

Lars Ingebrigtsen <larsi <at> gnus.org> writes:

> Eric Abrahamsen <eric <at> ericabrahamsen.net> writes:
>
>> Lars, what do you think about bumping the Gnus version number for big
>> changes like this? If only to make bug triage easier. In retrospect I
>> wish I'd done this for the obarray->hashtable change.
>
> Hm...  I'm not sure this is a big enough reason to bump the version
> number in itself, but it's sure been a while since it's been increased.
> The main reason is, of course, that I haven't done the t-shirts for Ma
> Gnus yet.  :-/

Well I've got this gnus-search package I'd like to put in on top of
nnselect, which is much more user-visible, so maybe then.

T-shirts should be the priority, though :)

Message #35 received at 37871 <at> debbugs.gnu.org (full text, mbox):

From: "Basil L. Contovounesios" <contovob <at> tcd.ie>
To: cohen <at> bu.edu, Eric Abrahamsen <eric <at> ericabrahamsen.net>
Cc: Lars Ingebrigtsen <larsi <at> gnus.org>, 37871 <at> debbugs.gnu.org
Subject: Re: bug#37871: 27.0.50; Merge feature/gnus-select2
Date: Sun, 06 Sep 2020 16:33:40 +0100

Eric Abrahamsen <eric <at> ericabrahamsen.net> writes:

> And it's in!

Thanks to everyone who worked on this!

I already pushed some minor code/doc cleanups to more closely follow
Emacs conventions in:

Fix formatting of recent Gnus nnselect changes
d7197f9d99 2020-09-06 16:06:57 +0100
https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=d7197f9d99ca6aa326d38e64ca9eb1d13e18d664

But I have some remaining comments.

First, the merge introduced the following build warning:

  In end of data:
  org/ol-gnus.el:271:1: Warning: the function ‘nnir-article-group’ is not known
      to be defined.

Org should probably be updated to use the new names of things.

Second, several changes in gnus-msg.el have resulted in misindented code
of the form:

  (unwind-protect
      (progn
        (one-single-form))
    ;; No unwind forms!
    )

If the unwind-protect+progn wrappers are no longer needed, they should
be removed and the code reindented.

More worryingly, some of the changes that let-bind gnus-newsgroup-name
are preceded by the following commentary:

  ;; We can't `let' gnus-newsgroup-name here, since that leads
  ;; to local variables leaking.

If the leaking is still an issue, then the nnselect changes should be
reconsidered; otherwise these stale comments should be removed.

Third, the bodies of gnus-fetch-headers and nnheader-parse-nov are
misindented.  (BTW, please respect the setting of indent-tabs-mode in
Emacs' top-level dir-locals-file, and especially don't use tab
characters for aligning comments to the right of code.)

Finally, the merge makes several backward-incompatible changes to user
options which should be rectified, lest we break existing user
configurations.  The renamed user options gnus-refer-thread-use-nnir and
nnir-retrieve-headers-override-function should be defined as obsolete
varaliases for their new names gnus-refer-thread-use-search and
nnselect-retrieve-headers-override-function, respectively.  Also, their
:version tags should be bumped to 28.1.  The removed user option
nnir-summary-line-format should still be defined but made obsolete.

Thanks,

-- 
Basil

Message #38 received at 37871 <at> debbugs.gnu.org (full text, mbox):

From: Andrew Cohen <cohen <at> bu.edu>
To: "Basil L. Contovounesios" <contovob <at> tcd.ie>
Cc: cohen <at> bu.edu, Eric Abrahamsen <eric <at> ericabrahamsen.net>,
 Lars Ingebrigtsen <larsi <at> gnus.org>, 37871 <at> debbugs.gnu.org
Subject: Re: bug#37871: 27.0.50; Merge feature/gnus-select2
Date: Mon, 07 Sep 2020 06:53:30 +0800

Dear Basil

>>>>> "BLC" == Basil L Contovounesios <contovob <at> tcd.ie> writes:

[...]

    BLC> I already pushed some minor code/doc cleanups to more closely
    BLC> follow Emacs conventions in:

    BLC> Fix formatting of recent Gnus nnselect changes d7197f9d99
    BLC> 2020-09-06 16:06:57 +0100
    BLC> https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=d7197f9d99ca6aa326d38e64ca9eb1d13e18d664

Thanks for all this!

    BLC> But I have some remaining comments.

    BLC> First, the merge introduced the following build warning:

    BLC>   In end of data: org/ol-gnus.el:271:1: Warning: the function
    BLC> ‘nnir-article-group’ is not known to be defined.

    BLC> Org should probably be updated to use the new names of things.

Clearly this needs to be changed. I wasn't expecting this internal
function to be used outside of gnus (and it probably shouldn't). 

    BLC> Second, several changes in gnus-msg.el have resulted in
    BLC> misindented code of the form:

Ugh. I spent hours trying to make sure the code was indented properly. I
must be doing something wrong.

    BLC>   (unwind-protect (progn (one-single-form)) ;; No unwind forms!
    BLC> )

I'm sorry I don't see exactly which unwind-protect you are referring to?
I don't think I changed any of the several unwind protects in
gnus-msg.el. Maybe someone else remembers why they are this way?

    BLC> If the unwind-protect+progn wrappers are no longer needed, they
    BLC> should be removed and the code reindented.

    BLC> More worryingly, some of the changes that let-bind
    BLC> gnus-newsgroup-name are preceded by the following commentary:

    BLC>   ;; We can't `let' gnus-newsgroup-name here, since that leads
    BLC> ;; to local variables leaking.

    BLC> If the leaking is still an issue, then the nnselect changes
    BLC> should be reconsidered; otherwise these stale comments should
    BLC> be removed.

I believe the leaking was a bug that has been dealt with, so the
comments should be removed. However I might have failed to fix it (I
have had a variety of people test it, but you never know).

    BLC> Third, the bodies of gnus-fetch-headers and nnheader-parse-nov
    BLC> are misindented.  (BTW, please respect the setting of
    BLC> indent-tabs-mode in Emacs' top-level dir-locals-file, and
    BLC> especially don't use tab characters for aligning comments to
    BLC> the right of code.)

See my above comment :(  All settings are at default, so I must be
making some other stupid mistake.

    BLC> Finally, the merge makes several backward-incompatible changes
    BLC> to user options which should be rectified, lest we break
    BLC> existing user configurations.  The renamed user options
    BLC> gnus-refer-thread-use-nnir and
    BLC> nnir-retrieve-headers-override-function should be defined as
    BLC> obsolete varaliases for their new names
    BLC> gnus-refer-thread-use-search and
    BLC> nnselect-retrieve-headers-override-function, respectively.
    BLC> Also, their :version tags should be bumped to 28.1.  The
    BLC> removed user option nnir-summary-line-format should still be
    BLC> defined but made obsolete.

Right. Thanks for noting these.

    BLC> Thanks,

    BLC> -- Basil

Message #41 received at 37871 <at> debbugs.gnu.org (full text, mbox):

From: Eric Abrahamsen <eric <at> ericabrahamsen.net>
To: Andrew Cohen <cohen <at> bu.edu>
Cc: "Basil L. Contovounesios" <contovob <at> tcd.ie>,
 Lars Ingebrigtsen <larsi <at> gnus.org>, 37871 <at> debbugs.gnu.org,
 37871-done <at> debbugs.gnu.org
Subject: Re: bug#37871: 27.0.50; Merge feature/gnus-select2
Date: Thu, 15 Oct 2020 09:04:30 -0700

This feature is now in and fixed up, so I'm closing this report.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.