GNU bug report logs - #50877
28.0.50; Gnus: nnimap backend is extremely slow to initialise new groups

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 50877 in the body.
You can then email your comments to 50877 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Morgan Willcock <mwillcock <at> precedence.co.uk>
To: bug-gnu-emacs <at> gnu.org
Subject: 28.0.50; Gnus: nnimap backend is extremely slow to initialise new
 groups
Date: Tue, 28 Sep 2021 22:27:06 +0100

It seems that the nnimap backend in Emacs 28 is now extremely slow to
initialise.

The initialisation first downloads 10MB of data from the IMAP server:

  nnimap read 10219k from server (initial sync of 16 groups; please wait)

Once all data is downloaded Emacs 27 takes 2 or 3 seconds to process the
data and display the group buffer whereas Emacs 28 takes 5 minutes 30
seconds for the same data. CPU usage is at 100% for this time. The time
taken is long enough that I've repeatedly thought the nnimap backend was
broken in Emacs 28.

To recreate with emacs -Q all I am doing is removing all local news
files:

  rm ~/.newsrc*

...and then configuring the bare minimum for Gnus to use nnimap:

  ;; No primary server
  (setq gnus-select-method '(nnnil ""))

  ;; IMAP as a secondary
  (setq gnus-secondary-select-methods
        '((nnimap "company"
                  (nnimap-address "server")
                  (nnimap-server-port "imap")
                  (nnimap-stream plain))))

I imagine this problem is only going to show where mailbox sizes are
fairly large (this example is 10MB of headers not 10MB of e-mail). Using
the option to debug-on-quit it seems that I always interupt somewhere
inside `seq-difference' inside of `nnimap-update-info':

Debugger entered--Lisp error: (quit)
  #f(compiled-function (e) #<bytecode -0x1e4a229aa5fcebcc>)(4942)
  mapc(#f(compiled-function (e) #<bytecode -0x1e4a229aa5fcebcc>) (1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 ...))
  seq-do(#f(compiled-function (e) #<bytecode -0x1e4a229aa5fcebcc>) (1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 ...))
  seq-contains-p((1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 ...) 8819 eq)
  #f(compiled-function (acc elt) #<bytecode -0x972f2b967b847d4>)(nil 8819)
  #f(compiled-function (elt) #<bytecode 0x13bb653afa8e7fc>)(8819)
  mapc(#f(compiled-function (elt) #<bytecode 0x13bb653afa8e7fc>) (23684 23683 23682 23679 23678 23677 23676 23675 23674 23673 23672 23671 23670 23669 23668 23667 23666 23665 23664 23663 23662 23661 23660 23659 23658 23657 23656 23655 23654 23653 23652 23651 23650 23649 23648 23647 23646 23645 23644 23643 23642 23641 23640 23639 23638 23637 23636 23635 23634 23633 ...))
  seq-do(#f(compiled-function (elt) #<bytecode 0x13bb653afa8e7fc>) (23684 23683 23682 23679 23678 23677 23676 23675 23674 23673 23672 23671 23670 23669 23668 23667 23666 23665 23664 23663 23662 23661 23660 23659 23658 23657 23656 23655 23654 23653 23652 23651 23650 23649 23648 23647 23646 23645 23644 23643 23642 23641 23640 23639 23638 23637 23636 23635 23634 23633 ...))
  seq-reduce(#f(compiled-function (acc elt) #<bytecode -0x972f2b967b847d4>) (23684 23683 23682 23679 23678 23677 23676 23675 23674 23673 23672 23671 23670 23669 23668 23667 23666 23665 23664 23663 23662 23661 23660 23659 23658 23657 23656 23655 23654 23653 23652 23651 23650 23649 23648 23647 23646 23645 23644 23643 23642 23641 23640 23639 23638 23637 23636 23635 23634 23633 ...) nil)
  seq-difference((1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 ...) (1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 ...) eq)
  nnimap-update-info(("nnimap+company:2012" 3 nil nil "nnimap:company" ((permanent-flags %Answered %Flagged %Deleted %Seen %Draft receipt-handled $Forwarded %*))) ((1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 ...) (($Forwarded 14440 18790 18797) (receipt-handled 19651 20355 23684) (%Seen 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 ...) (%Answered 1 37 45 53 69 80 103 113 117 128 133 136 148 149 166 169 170 179 184 190 205 233 246 281 303 312 314 316 317 321 323 324 331 334 337 339 340 342 359 364 366 371 375 382 383 385 388 391 392 ...)) 23684 1 23685 1 (%Answered %Flagged %Deleted %Seen %Draft receipt-handled $Forwarded %*) "1356127665" nil "6"))
  nnimap-update-infos((("SentMail" ... ... 24763 1 24764 1 ... "1592949197" nil "10270") ("2011" ... ... 4281 1 4282 1 ... "1354209749" nil "3") ("2012" ... ... 23684 1 23685 1 ... "1356127665" nil "6") ("2013" ... ... 24194 1 24195 1 ... "1378389574" nil "11") ("2014" ... ... 23234 1 23235 1 ... "1407968232" nil "11") ("2015" ... ... 34462 1 34463 1 ... "1433152228" nil "22") ("2016" ... ... 26795 1 26796 1 ... "1465460341" nil "9") ("2017" ... ... 21412 1 21414 1 ... "1501180732" nil "13") ("Drafts" nil nil nil nil 54521 1 ... "1318590639" nil "52419") ("Queue" nil nil nil nil 9259 1 ... "1407957310" nil "12991") ("saved-messages" nil nil nil nil 1 1 ... "1318335793" nil "1") ("Trash" ... ... 83838 83553 83839 1 ... "1500397966" nil "30157") ("2020" ... ... 26229 1 26230 1 ... "1519415818" nil "88") ("2018" ... ... 23098 1 23099 1 ... "1519415810" nil "81") ("2019" ... ... 22988 1 22989 1 ... "1519415816" nil "72") ("INBOX" ... ... 377585 349408 377586 1 ... "1461657110" nil "194264")) (("nnimap+company:Se..." 3 ... ... "nnimap:company" ...) ("nnimap+company:20..." 3 ... ... "nnimap:company" ...) ("nnimap+company:20..." 3 nil nil "nnimap:company" ...) ("nnimap+company:20..." 3 nil nil "nnimap:company" nil) ("nnimap+company:20..." 3 nil nil "nnimap:company" nil) ("nnimap+company:20..." 3 nil nil "nnimap:company" nil) ("nnimap+company:20..." 3 nil nil "nnimap:company" nil) ("nnimap+company:20..." 3 nil nil "nnimap:company" nil) ("nnimap+company:Dr..." 3 nil nil "nnimap:company" nil) ("nnimap+company:Qu..." 3 nil nil "nnimap:company" nil) ("nnimap+company:sa..." 3 nil nil "nnimap:company" nil) ("nnimap+company:Tr..." 3 nil nil "nnimap:company" nil) ("nnimap+company:20..." 3 nil nil "nnimap:company" nil) ("nnimap+company:20..." 3 nil nil "nnimap:company" nil) ("nnimap+company:20..." 3 nil nil "nnimap:company" nil) ("nnimap+company:IN..." 3 nil nil "nnimap:company" nil)))
  nnimap-finish-retrieve-group-infos("company" (("nnimap+company:SentMail" 3 ((1 . 24763)) ((unexist 1325 4995 5028 5055 5059 5070 5846 6170 6663 (6930 . 6931) 7057 7468 7668 7690 9550 10500 11077 11522 11559 12039 12381 12614 13536 13773 13845 14904 15141 15637 16029 16032 16034 16037 16041 16051 16625 16634 20860 21771 21821 22119 (22220 . 22355) (22581 . 22582) (22661 . 22664) (22765 . 22773) 23517 (23519 . 23520) 23562 (23568 . 23569) (23580 . 23581) ...) (forward 23684 24126) (reply 36 229 232 235 240 247 269 271 276 283 317 355 402 436 455 464 524 558 565 581 607 616 620 625 627 665 670 675 678 686 689 693 695 697 709 716 811 813 879 885 888 890 897 935 967 979 981 991 (995 . 996) ...)) "nnimap:company" ((modseq . "10270") (uidvalidity . "1592949197") (active 1 . 24763) (permanent-flags %Answered %Flagged %Deleted %Seen %Draft receipt-handled $has_cal $NotJunk $SENT $Forwarded gnus-expire gnus-forward %*))) ("nnimap+company:2011" 3 ((1 . 4281)) ((unexist) (reply 71 86 143 209 219 320 379 632 640 647 652 657 (684 . 685) 690 703 713 (716 . 717) 722 735 759 787 810 816 841 903 910 1022 1044 1063 1065 1068 1164 1173 1200 1202 1249 1262 1267 1287 1300 1303 1327 1333 1337 1344 1394 1415 1420 1424 ...)) "nnimap:company" ((modseq . "3") (uidvalidity . "1354209749") (active 1 . 4281) (permanent-flags %Answered %Flagged %Deleted %Seen %Draft $NotJunk %*))) ("nnimap+company:2012" 3 nil nil "nnimap:company" ((permanent-flags %Answered %Flagged %Deleted %Seen %Draft receipt-handled $Forwarded %*))) ("nnimap+company:2013" 3 nil nil "nnimap:company" nil) ("nnimap+company:2014" 3 nil nil "nnimap:company" nil) ("nnimap+company:2015" 3 nil nil "nnimap:company" nil) ("nnimap+company:2016" 3 nil nil "nnimap:company" nil) ("nnimap+company:2017" 3 nil nil "nnimap:company" nil) ("nnimap+company:Drafts" 3 nil nil "nnimap:company" nil) ("nnimap+company:Queue" 3 nil nil "nnimap:company" nil) ("nnimap+company:saved-messages" 3 nil nil "nnimap:company" nil) ("nnimap+company:Trash" 3 nil nil "nnimap:company" nil) ("nnimap+company:2020" 3 nil nil "nnimap:company" nil) ("nnimap+company:2018" 3 nil nil "nnimap:company" nil) ("nnimap+company:2019" 3 nil nil "nnimap:company" nil) ("nnimap+company:INBOX" 3 nil nil "nnimap:company" nil)) ((108 109 1 "INBOX" "SELECT")))
  gnus-finish-retrieve-group-infos((nnimap "company" (nnimap-address "server") (nnimap-server-port "imap") (nnimap-stream plain)) (("nnimap+company:SentMail" 3 ((1 . 24763)) ((unexist 1325 4995 5028 5055 5059 5070 5846 6170 6663 (6930 . 6931) 7057 7468 7668 7690 9550 10500 11077 11522 11559 12039 12381 12614 13536 13773 13845 14904 15141 15637 16029 16032 16034 16037 16041 16051 16625 16634 20860 21771 21821 22119 (22220 . 22355) (22581 . 22582) (22661 . 22664) (22765 . 22773) 23517 (23519 . 23520) 23562 (23568 . 23569) (23580 . 23581) ...) (forward 23684 24126) (reply 36 229 232 235 240 247 269 271 276 283 317 355 402 436 455 464 524 558 565 581 607 616 620 625 627 665 670 675 678 686 689 693 695 697 709 716 811 813 879 885 888 890 897 935 967 979 981 991 (995 . 996) ...)) "nnimap:company" ((modseq . "10270") (uidvalidity . "1592949197") (active 1 . 24763) (permanent-flags %Answered %Flagged %Deleted %Seen %Draft receipt-handled $has_cal $NotJunk $SENT $Forwarded gnus-expire gnus-forward %*))) ("nnimap+company:2011" 3 ((1 . 4281)) ((unexist) (reply 71 86 143 209 219 320 379 632 640 647 652 657 (684 . 685) 690 703 713 (716 . 717) 722 735 759 787 810 816 841 903 910 1022 1044 1063 1065 1068 1164 1173 1200 1202 1249 1262 1267 1287 1300 1303 1327 1333 1337 1344 1394 1415 1420 1424 ...)) "nnimap:company" ((modseq . "3") (uidvalidity . "1354209749") (active 1 . 4281) (permanent-flags %Answered %Flagged %Deleted %Seen %Draft $NotJunk %*))) ("nnimap+company:2012" 3 nil nil "nnimap:company" ((permanent-flags %Answered %Flagged %Deleted %Seen %Draft receipt-handled $Forwarded %*))) ("nnimap+company:2013" 3 nil nil "nnimap:company" nil) ("nnimap+company:2014" 3 nil nil "nnimap:company" nil) ("nnimap+company:2015" 3 nil nil "nnimap:company" nil) ("nnimap+company:2016" 3 nil nil "nnimap:company" nil) ("nnimap+company:2017" 3 nil nil "nnimap:company" nil) ("nnimap+company:Drafts" 3 nil nil "nnimap:company" nil) ("nnimap+company:Queue" 3 nil nil "nnimap:company" nil) ("nnimap+company:saved-messages" 3 nil nil "nnimap:company" nil) ("nnimap+company:Trash" 3 nil nil "nnimap:company" nil) ("nnimap+company:2020" 3 nil nil "nnimap:company" nil) ("nnimap+company:2018" 3 nil nil "nnimap:company" nil) ("nnimap+company:2019" 3 nil nil "nnimap:company" nil) ("nnimap+company:INBOX" 3 nil nil "nnimap:company" nil)) ((108 109 1 "INBOX" "SELECT")))
  gnus-read-active-for-groups((nnimap "company" (nnimap-address "server") (nnimap-server-port "imap") (nnimap-stream plain)) (("nnimap+company:SentMail" 3 ((1 . 24763)) ((unexist 1325 4995 5028 5055 5059 5070 5846 6170 6663 (6930 . 6931) 7057 7468 7668 7690 9550 10500 11077 11522 11559 12039 12381 12614 13536 13773 13845 14904 15141 15637 16029 16032 16034 16037 16041 16051 16625 16634 20860 21771 21821 22119 (22220 . 22355) (22581 . 22582) (22661 . 22664) (22765 . 22773) 23517 (23519 . 23520) 23562 (23568 . 23569) (23580 . 23581) ...) (forward 23684 24126) (reply 36 229 232 235 240 247 269 271 276 283 317 355 402 436 455 464 524 558 565 581 607 616 620 625 627 665 670 675 678 686 689 693 695 697 709 716 811 813 879 885 888 890 897 935 967 979 981 991 (995 . 996) ...)) "nnimap:company" ((modseq . "10270") (uidvalidity . "1592949197") (active 1 . 24763) (permanent-flags %Answered %Flagged %Deleted %Seen %Draft receipt-handled $has_cal $NotJunk $SENT $Forwarded gnus-expire gnus-forward %*))) ("nnimap+company:2011" 3 ((1 . 4281)) ((unexist) (reply 71 86 143 209 219 320 379 632 640 647 652 657 (684 . 685) 690 703 713 (716 . 717) 722 735 759 787 810 816 841 903 910 1022 1044 1063 1065 1068 1164 1173 1200 1202 1249 1262 1267 1287 1300 1303 1327 1333 1337 1344 1394 1415 1420 1424 ...)) "nnimap:company" ((modseq . "3") (uidvalidity . "1354209749") (active 1 . 4281) (permanent-flags %Answered %Flagged %Deleted %Seen %Draft $NotJunk %*))) ("nnimap+company:2012" 3 nil nil "nnimap:company" ((permanent-flags %Answered %Flagged %Deleted %Seen %Draft receipt-handled $Forwarded %*))) ("nnimap+company:2013" 3 nil nil "nnimap:company" nil) ("nnimap+company:2014" 3 nil nil "nnimap:company" nil) ("nnimap+company:2015" 3 nil nil "nnimap:company" nil) ("nnimap+company:2016" 3 nil nil "nnimap:company" nil) ("nnimap+company:2017" 3 nil nil "nnimap:company" nil) ("nnimap+company:Drafts" 3 nil nil "nnimap:company" nil) ("nnimap+company:Queue" 3 nil nil "nnimap:company" nil) ("nnimap+company:saved-messages" 3 nil nil "nnimap:company" nil) ("nnimap+company:Trash" 3 nil nil "nnimap:company" nil) ("nnimap+company:2020" 3 nil nil "nnimap:company" nil) ("nnimap+company:2018" 3 nil nil "nnimap:company" nil) ("nnimap+company:2019" 3 nil nil "nnimap:company" nil) ("nnimap+company:INBOX" 3 nil nil "nnimap:company" nil)) ((108 109 1 "INBOX" "SELECT")))
  gnus-get-unread-articles(nil nil)
  gnus-setup-news(nil nil nil)
  #f(compiled-function () #<bytecode 0x99da026c58a97ad>)()
  gnus-1(nil nil nil)
  gnus(nil)
  funcall-interactively(gnus nil)
  call-interactively(gnus record nil)
  command-execute(gnus record)
  execute-extended-command(nil "gnus" "gnus")
  funcall-interactively(execute-extended-command nil "gnus" "gnus")
  call-interactively(execute-extended-command nil nil)
  command-execute(execute-extended-command)


In GNU Emacs 28.0.50 (build 1, x86_64-pc-linux-gnu, X toolkit, cairo version 1.16.0, Xaw3d scroll bars)
 of 2021-09-28 built on inspiron
Repository revision: ccb35fb8fb9de5f069fd0103f24e3048d716febc
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12011000
System Description: Debian GNU/Linux 11 (bullseye)

Configured using:
 'configure --with-cairo --with-json --with-xml2 --with-x-toolkit=lucid'

Configured features:
CAIRO FREETYPE GIF GLIB GMP GNUTLS GSETTINGS HARFBUZZ JPEG JSON
LIBSELINUX LIBXML2 MODULES NOTIFY INOTIFY PDUMPER PNG SECCOMP SOUND
THREADS TIFF TOOLKIT_SCROLL_BARS X11 XAW3D XDBE XIM XPM LUCID ZLIB

Important settings:
  value of $LANG: en_GB.UTF-8
  locale-coding-system: utf-8-unix

Major mode: ELisp/d

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rmc puny dired dired-loaddefs
rfc822 mml mml-sec epa derived epg rfc6068 epg-config gnus-util rmail
rmail-loaddefs auth-source cl-seq eieio eieio-core cl-macs
eieio-loaddefs password-cache json map text-property-search time-date
subr-x seq byte-opt gv bytecomp byte-compile cconv mm-decode mm-bodies
mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail
rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils misearch
multi-isearch cl-loaddefs cl-lib iso-transl tooltip eldoc electric
uniquify ediff-hook vc-hooks lisp-float-type elisp-mode mwheel
term/x-win x-win term/common-win x-dnd tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch easymenu
timer select scroll-bar mouse jit-lock font-lock syntax font-core
term/tty-colors frame minibuffer cl-generic cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese composite emoji-zwj charscript charprop case-table
epa-hook jka-cmpr-hook help simple abbrev obarray cl-preloaded nadvice
button loaddefs faces cus-face macroexp files window text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote threads inotify dynamic-setting
system-font-setting font-render-setting cairo x-toolkit x multi-tty
make-network-process emacs)

Memory information:
((conses 16 53711 11564)
 (symbols 48 6651 4)
 (strings 32 19116 1933)
 (string-bytes 1 617579)
 (vectors 16 13830)
 (vector-slots 8 185399 17936)
 (floats 8 25 78)
 (intervals 56 624 17)
 (buffers 992 15))

Message #8 received at 50877 <at> debbugs.gnu.org (full text, mbox):

From: Eric Abrahamsen <eric <at> ericabrahamsen.net>
To: Morgan Willcock <mwillcock <at> precedence.co.uk>
Cc: 50877 <at> debbugs.gnu.org
Subject: Re: bug#50877: 28.0.50; Gnus: nnimap backend is extremely slow to
 initialise new groups
Date: Tue, 28 Sep 2021 14:34:05 -0700

Morgan Willcock <mwillcock <at> precedence.co.uk> writes:

> It seems that the nnimap backend in Emacs 28 is now extremely slow to
> initialise.
>
> The initialisation first downloads 10MB of data from the IMAP server:
>
>   nnimap read 10219k from server (initial sync of 16 groups; please wait)
>
> Once all data is downloaded Emacs 27 takes 2 or 3 seconds to process the
> data and display the group buffer whereas Emacs 28 takes 5 minutes 30
> seconds for the same data. CPU usage is at 100% for this time. The time
> taken is long enough that I've repeatedly thought the nnimap backend was
> broken in Emacs 28.
>
> To recreate with emacs -Q all I am doing is removing all local news
> files:
>
>   rm ~/.newsrc*
>
> ...and then configuring the bare minimum for Gnus to use nnimap:
>
>   ;; No primary server
>   (setq gnus-select-method '(nnnil ""))
>
>   ;; IMAP as a secondary
>   (setq gnus-secondary-select-methods
>         '((nnimap "company"
>                   (nnimap-address "server")
>                   (nnimap-server-port "imap")
>                   (nnimap-stream plain))))
>
> I imagine this problem is only going to show where mailbox sizes are
> fairly large (this example is 10MB of headers not 10MB of e-mail). Using
> the option to debug-on-quit it seems that I always interupt somewhere
> inside `seq-difference' inside of `nnimap-update-info':

You could try reverting 20f7fa691b7c2859b96550d9ccb326bf394e160d and see
if that fixes it. That change went in in April, though, so unless you
haven't updated for a while (or you've been seeing this problem for a
long time) it might not be likely.

Message #11 received at 50877 <at> debbugs.gnu.org (full text, mbox):

From: Morgan Willcock <mwillcock <at> precedence.co.uk>
To: Eric Abrahamsen <eric <at> ericabrahamsen.net>
Cc: 50877 <at> debbugs.gnu.org
Subject: Re: bug#50877: 28.0.50; Gnus: nnimap backend is extremely slow to
 initialise new groups
Date: Tue, 28 Sep 2021 22:53:45 +0100

Eric Abrahamsen <eric <at> ericabrahamsen.net> writes:

> You could try reverting 20f7fa691b7c2859b96550d9ccb326bf394e160d and see
> if that fixes it. That change went in in April, though, so unless you
> haven't updated for a while (or you've been seeing this problem for a
> long time) it might not be likely.

Reverting that commit fixes it for me.

Message #14 received at 50877 <at> debbugs.gnu.org (full text, mbox):

From: Eric Abrahamsen <eric <at> ericabrahamsen.net>
To: Morgan Willcock <mwillcock <at> precedence.co.uk>
Cc: 50877 <at> debbugs.gnu.org, Lars Ingebrigtsen <larsi <at> gnus.org>,
 Stefan Kangas <stefankangas <at> gmail.com>
Subject: Re: bug#50877: 28.0.50; Gnus: nnimap backend is extremely slow to
 initialise new groups
Date: Tue, 28 Sep 2021 21:25:37 -0700

Morgan Willcock <mwillcock <at> precedence.co.uk> writes:

> Eric Abrahamsen <eric <at> ericabrahamsen.net> writes:
>
>> You could try reverting 20f7fa691b7c2859b96550d9ccb326bf394e160d and see
>> if that fixes it. That change went in in April, though, so unless you
>> haven't updated for a while (or you've been seeing this problem for a
>> long time) it might not be likely.
>
> Reverting that commit fixes it for me.

I'm cc'ing Lars and Stefan Kangas, who are the most "involved" here. The
benchmarks are gruesome:

(defun old-gnus-set-difference (list1 list2)
  "Return a list of elements of LIST1 that do not appear in LIST2."
  (let ((hash2 (make-hash-table :test 'eq))
        (result nil))
    (dolist (elt list2) (puthash elt t hash2))
    (dolist (elt list1)
      (unless (gethash elt hash2)
        (setq result (cons elt result))))
    (nreverse result)))

(let* ((common1 (number-sequence 0 200))
       (common2 (number-sequence 501 700))
       (l1 (append common1
		   (number-sequence 200 500 2)
		   common2))
       (l2 (append common1
		   (number-sequence 201 500 2)
		   common2))
       (results (list
		 (benchmark-run 1000 (old-gnus-set-difference l1 l2))
		 (benchmark-run 1000 (seq-difference l1 l2 #'eq)))))
  results)->
  ((1.842791826 7 1.4057783080000021)
   (14.284496493999999 4 0.7823414249999985))

This is a particularly bad example -- when there was no common2
`seq-difference' performed better, but I think it still around 6 seconds
to the old function's 2.

Probably still worth a dedicated function for Gnus? (Or a rewrite of
`seq-difference'...)

Eric

Message #17 received at 50877 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Eric Abrahamsen <eric <at> ericabrahamsen.net>
Cc: 50877 <at> debbugs.gnu.org, Morgan Willcock <mwillcock <at> precedence.co.uk>,
 Stefan Kangas <stefankangas <at> gmail.com>
Subject: Re: bug#50877: 28.0.50; Gnus: nnimap backend is extremely slow to
 initialise new groups
Date: Wed, 29 Sep 2021 08:18:06 +0200

Eric Abrahamsen <eric <at> ericabrahamsen.net> writes:

> I'm cc'ing Lars and Stefan Kangas, who are the most "involved" here. The
> benchmarks are gruesome:

Yeah, sounds like that commit has to be reverted.

> Probably still worth a dedicated function for Gnus? (Or a rewrite of
> `seq-difference'...)

Rewriting it using the same trick that the Gnus version does (i.e., a
hash table) might be nice, but it does use more memory, so I'm not sure
that's a general solution.  Perhaps there should be a separate version
of seq-difference.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #20 received at 50877 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Kangas <stefan <at> marxist.se>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: Eric Abrahamsen <eric <at> ericabrahamsen.net>, 50877 <at> debbugs.gnu.org,
 Morgan Willcock <mwillcock <at> precedence.co.uk>
Subject: Re: bug#50877: 28.0.50; Gnus: nnimap backend is extremely slow to
 initialise new groups
Date: Wed, 29 Sep 2021 02:11:14 -0700

Lars Ingebrigtsen <larsi <at> gnus.org> writes:

>> I'm cc'ing Lars and Stefan Kangas, who are the most "involved" here. The
>> benchmarks are gruesome:
>
> Yeah, sounds like that commit has to be reverted.

Agreed.  We should also add a comment about why we shouldn't use
`seq-difference' here.

Message #23 received at 50877 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Stefan Kangas <stefan <at> marxist.se>
Cc: Eric Abrahamsen <eric <at> ericabrahamsen.net>, 50877 <at> debbugs.gnu.org,
 Morgan Willcock <mwillcock <at> precedence.co.uk>
Subject: Re: bug#50877: 28.0.50; Gnus: nnimap backend is extremely slow to
 initialise new groups
Date: Wed, 29 Sep 2021 17:27:34 +0200

Stefan Kangas <stefan <at> marxist.se> writes:

>> Yeah, sounds like that commit has to be reverted.
>
> Agreed.  We should also add a comment about why we shouldn't use
> `seq-difference' here.

Now done.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.