GNU logs - #27194, boring messages

Message sent to bug-gnu-emacs@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: bug#27194: 25.2; gnus: nnmaildir: article numbers keep growing -> nnmaildir--new-number slow
Resent-From: Justus-dev@HIDDEN
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 02 Jun 2017 07:06:02 +0000
Resent-Message-ID: <handler.27194.B.14963871408776 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: report 27194
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: 27194 <at> debbugs.gnu.org
X-Debbugs-Original-To: bug-gnu-emacs@HIDDEN
Received: via spool by submit <at> debbugs.gnu.org id=B.14963871408776
          (code B ref -1); Fri, 02 Jun 2017 07:06:02 +0000
Received: (at submit) by debbugs.gnu.org; 2 Jun 2017 07:05:40 +0000
Received: from localhost ([127.0.0.1]:50358 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1dGgeS-0002HT-2P
	for submit <at> debbugs.gnu.org; Fri, 02 Jun 2017 03:05:40 -0400
Received: from eggs.gnu.org ([208.118.235.92]:42077)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <Justus-dev@HIDDEN>) id 1dGgES-0001eE-M8
 for submit <at> debbugs.gnu.org; Fri, 02 Jun 2017 02:38:49 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <Justus-dev@HIDDEN>) id 1dGgEL-0006Ox-Uj
 for submit <at> debbugs.gnu.org; Fri, 02 Jun 2017 02:38:43 -0400
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50 autolearn=disabled
 version=3.3.2
Received: from lists.gnu.org ([2001:4830:134:3::11]:56730)
 by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32)
 (Exim 4.71) (envelope-from <Justus-dev@HIDDEN>)
 id 1dGgEL-0006Oa-S0
 for submit <at> debbugs.gnu.org; Fri, 02 Jun 2017 02:38:41 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:33032)
 by lists.gnu.org with esmtp (Exim 4.71)
 (envelope-from <Justus-dev@HIDDEN>) id 1dGgEJ-0004Aw-R3
 for bug-gnu-emacs@HIDDEN; Fri, 02 Jun 2017 02:38:41 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <Justus-dev@HIDDEN>) id 1dGgEG-0006J2-LP
 for bug-gnu-emacs@HIDDEN; Fri, 02 Jun 2017 02:38:39 -0400
Received: from smtp.uibk.ac.at ([138.232.1.140]:58670)
 by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)
 (Exim 4.71) (envelope-from <Justus-dev@HIDDEN>)
 id 1dGgEG-0006GJ-9m
 for bug-gnu-emacs@HIDDEN; Fri, 02 Jun 2017 02:38:36 -0400
Received: from jp (213-47-186-175.cable.dynamic.surfer.at [213.47.186.175])
 (authenticated bits=0)
 by smtp.uibk.ac.at (8.14.4/8.14.4/F1) with ESMTP id v526cWcl016643
 (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256 verify=NO)
 for <bug-gnu-emacs@HIDDEN>; Fri, 2 Jun 2017 08:38:33 +0200
From: Justus-dev@HIDDEN
Date: Fri, 02 Jun 2017 08:38:32 +0200
Message-ID: <87lgparj13.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Scanned-By: MIMEDefang 2.75 at uibk.ac.at
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [fuzzy]
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x
X-Received-From: 2001:4830:134:3::11
X-Spam-Score: -5.0 (-----)
X-Mailman-Approved-At: Fri, 02 Jun 2017 03:05:39 -0400
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -5.0 (-----)


In Gnus' nnmaildir backend, article numbers grow without bounds.
Getting new news and moving articles into and out of those of my inbox
groups (that see a lot of churn) took really long lately (up to several
minutes).  Profiling revealed that 99% of this time was spent inside
nnmaildir--new-number, and indeed, the affected .nnmaildir/num/
directories contained thousands of hardlinks to article numbers.

For each affected group, I got rid of these exessive article number
hardlinks as follows:

1. move all messages from group to group.new
2. edit the parameters of group.new to match those of group
3. delete group
4. rename group.new to group

(I'm not sure this procedure is guaranteed to be free of unintended side
effects, but I have not observed any.)

Since then, Gnus is snappy again.

This is apparently a poorly-designed data structure / algorithm.  If
nnmaildir requires gapless article number sequences, why doesn't it just
store the highest allocated number?  Why does it waste time on
sequential search (as I suspect)?  If it does not rely on gapless
article numbers, why doesn't it remove hardlinks corresponding to
articles that no longer exist in this group? (After moving all messages
out of an affected group, all 15000+ hardlinks were still present in
.nnmaildir/num/.)



In GNU Emacs 25.2.1 (x86_64-unknown-linux-gnu, GTK+ Version 3.22.10)
 of 2017-04-22 built on juergen
Windowing system distributor 'The X.Org Foundation', version 11.0.11903000
System Description:	Arch Linux

Configured using:
 'configure --prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib
 --localstatedir=/var --with-x-toolkit=gtk3 --with-xft --with-modules
 'CFLAGS=-march=x86-64 -mtune=generic -O2 -pipe
 -fstack-protector-strong' CPPFLAGS=-D_FORTIFY_SOURCE=2
 LDFLAGS=-Wl,-O1,--sort-common,--as-needed,-z,relro'

Configured features:
XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS GCONF GSETTINGS
NOTIFY ACL GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB
TOOLKIT_SCROLL_BARS GTK3 X11 MODULES

Important settings:
  value of $LC_COLLATE: de_AT.UTF-8
  value of $LC_MONETARY: de_AT.UTF-8
  value of $LC_TIME: de_AT.UTF-8
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Org

Minor modes in effect:
  diff-auto-refine-mode: t
  magit-auto-revert-mode: t
  global-git-commit-mode: t
  async-bytecomp-package-mode: t
  org-indent-mode: t
  show-paren-mode: t
  shell-dirtrack-mode: t
  pdf-occur-global-minor-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent messages:
Expiring articles...done
No more unseen articles
No more unread articles
Expiring articles...done
Saving /home/piater/.newsrc.eld...
Saving file /home/piater/.newsrc.eld...
Wrote /home/piater/.newsrc.eld
Saving /home/piater/.newsrc.eld...done
(No files need saving)
Scanning for dabbrevs...done

Load-path shadows:
None found.

Features:
(shadow emacsbug latexenc org-colview pcmpl-unix pcmpl-gnu gnus-eform
gnus-fun kmacro gnus-uu yenc profiler eimp ffap locate pdf-sync
pdf-annot pdf-outline pdf-links pdf-history tabify image-file
org-capture term ehelp ispell gnus-draft grep sh-script smie tramp-cache
view woman man vc-git magit-blame magit-stash magit-bisect magit-remote
magit-commit magit-sequence magit magit-apply magit-wip magit-log
magit-diff smerge-mode diff-mode magit-core magit-autorevert autorevert
filenotify magit-process magit-mode magit-popup magit-git crm
magit-section magit-utils git-commit log-edit pcvs-util add-log
with-editor async-bytecomp async tramp-sh dash cal-move diary-lib
diary-loaddefs cal-iso org-indent org-rmail org-mhe org-irc org-info
org-gnus org-docview doc-view org-bibtex bibtex org-bbdb org-w3m
org-agenda mm-extern mm-url nxml-uchnm rng-xsd xsd-regexp rng-cmpct
rng-nxml rng-valid rng-loc rng-uri rng-parse nxml-parse rng-match rng-dt
rng-util rng-pttrn nxml-ns nxml-mode nxml-outln nxml-rap nxml-util
nxml-glyph nxml-enc xmltok dabbrev mailalias smtpmail sendmail dired-aux
flow-fill gnus-dup misearch multi-isearch gnus-cite smiley shr-color
color url url-proxy url-privacy url-expand url-methods url-history
url-cookie url-domsuf url-util url-parse url-vars qp shr dom subr-x
mm-archive mail-extr gnus-async gnus-bcklg sort gnus-ml disp-table
gnus-topic utf-7 network-stream nsm starttls imap epa-file epa derived
nndraft nnmh nnmaildir gnus-agent gnus-srvr gnus-score score-mode
nnvirtual gnus-msg gnus-art mm-uu mml2015 mm-view mml-smime smime dig
mailcap nntp gnus-cache nnir gnus-sum gnus-group gnus-undo gnus-start
gnus-cloud nnimap nnmail mail-source tls gnutls utf7 netrc nnoo
parse-time gnus-spec gnus-int gnus-range message idna rfc822 mml mml-sec
epg mm-decode mm-bodies mm-encode mail-parse rfc2231 rfc2047 rfc2045
ietf-drums mailabbrev gmm-utils mailheader gnus-win mmm-mode mmm-univ
mmm-class mmm-region mmm-auto mmm-vars mmm-utils mmm-compat paren gnus
gnus-ems nnheader mail-utils gnus-dired ox-latex ox-icalendar ox-html
ox-ascii ox-publish ox org-element org-id org org-macro org-footnote
org-pcomplete org-list org-faces org-entities org-version ob-emacs-lisp
ob ob-tangle ob-ref ob-lob ob-table ob-exp org-src ob-keys ob-comint
ob-core ob-eval org-compat org-macs org-loaddefs cal-menu calendar
cal-loaddefs ido ess-toolbar ess-mouse mouseme thingatpt browse-url
ess-menu ess-swv ess-noweb ess-noweb-font-lock-mode ess-bugs-l essd-els
ess-sas-d ess-sas-l ess-sas-a ess-sta-d ess-sta-l cc-vars cc-defs
make-regexp ess-sp6-d ess-dde ess-sp3-d ess-julia julia-mode ess-r-d
ess-r-syntax ess-r-completion ess-roxy essddr noutline outline
easy-mmode hideshow ess-help ess-r-package ess-s-l ess ess-inf
ess-tracebug tramp tramp-compat auth-source cl-seq gnus-util mm-util
help-fns mail-prsvr password-cache tramp-loaddefs trampver ucs-normalize
shell pcomplete ess-mode ess-noweb-mode ess-utils ess-generics cl
ess-custom executable ess-compat ess-site pdf-occur ibuf-ext ibuffer
tablist tablist-filter semantic/wisent/comp semantic/wisent
semantic/wisent/wisent semantic/util-modes semantic/util semantic
semantic/tag semantic/lex semantic/fw eieio eieio-core cl-macs
mode-local find-func cedet dired pdf-isearch let-alist pdf-misc imenu
pdf-tools compile comint ansi-color ring cus-edit cus-start cus-load
wid-edit pdf-view bookmark pp jka-compr pdf-cache pdf-info tq pdf-util
advice format-spec image-mode mozmail server finder-inf info package
epg-config seq byte-opt gv bytecomp byte-compile cl-extra help-mode
easymenu cconv cl-loaddefs pcase cl-lib time-date mule-util tooltip
eldoc electric uniquify ediff-hook vc-hooks lisp-float-type mwheel x-win
term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list newcomment elisp-mode lisp-mode prog-mode register page
menu-bar rfn-eshadow timer select scroll-bar mouse jit-lock font-lock
syntax facemenu font-core frame cl-generic cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese charscript case-table epa-hook jka-cmpr-hook help
simple abbrev minibuffer cl-preloaded nadvice loaddefs button faces
cus-face macroexp files text-properties overlay sha1 md5 base64 format
env code-pages mule custom widget hashtable-print-readable backquote
dbusbind inotify dynamic-setting system-font-setting font-render-setting
move-toolbar gtk x-toolkit x multi-tty make-network-process emacs)

Memory information:
((conses 16 4492676 448493)
 (symbols 48 279807 1477)
 (miscs 40 5257 10111)
 (strings 32 755088 73077)
 (string-bytes 1 46054567)
 (vectors 16 304729)
 (vector-slots 8 4348424 117230)
 (floats 8 9603 2491)
 (intervals 56 295380 4749)
 (buffers 976 313))

Message sent:


Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Mailer: MIME-tools 5.505 (Entity 5.505)
Content-Type: text/plain; charset=utf-8
X-Loop: help-debbugs@HIDDEN
From: help-debbugs@HIDDEN (GNU bug Tracking System)
To: Justus-dev@HIDDEN
Subject: bug#27194: Acknowledgement (25.2; gnus: nnmaildir: article
 numbers keep growing -> nnmaildir--new-number slow)
Message-ID: <handler.27194.B.14963871408776.ack <at> debbugs.gnu.org>
References: <87lgparj13.fsf@HIDDEN>
X-Gnu-PR-Message: ack 27194
X-Gnu-PR-Package: emacs
Reply-To: 27194 <at> debbugs.gnu.org
Date: Fri, 02 Jun 2017 07:06:02 +0000

Thank you for filing a new bug report with debbugs.gnu.org.

This is an automatically generated reply to let you know your message
has been received.

Your message is being forwarded to the package maintainers and other
interested parties for their attention; they will reply in due course.

Your message has been sent to the package maintainer(s):
 bug-gnu-emacs@HIDDEN

If you wish to submit further information on this problem, please
send it to 27194 <at> debbugs.gnu.org.

Please do not send mail to help-debbugs@HIDDEN unless you wish
to report a problem with the Bug-tracking system.

--=20
27194: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D27194
GNU Bug Tracking System
Contact help-debbugs@HIDDEN with problems

Message received at control <at> debbugs.gnu.org:


Received: (at control) by debbugs.gnu.org; 19 Sep 2017 08:47:51 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Sep 19 04:47:51 2017
Received: from localhost ([127.0.0.1]:47242 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1duEC7-00008g-9P
	for submit <at> debbugs.gnu.org; Tue, 19 Sep 2017 04:47:51 -0400
Received: from mezzogiorno.tim-landscheidt.de ([78.47.87.37]:49255)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <tim@HIDDEN>) id 1duEC6-00008Y-0F
 for control <at> debbugs.gnu.org; Tue, 19 Sep 2017 04:47:50 -0400
Received: from x4e369dc7.dyn.telefonica.de ([78.54.157.199]:34174
 helo=passepartout.tim-landscheidt.de)
 by mezzogiorno.tim-landscheidt.de with esmtpsa
 (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2)
 (envelope-from <tim@HIDDEN>) id 1duEC4-00055f-HY
 for control <at> debbugs.gnu.org; Tue, 19 Sep 2017 08:47:48 +0000
From: Tim Landscheidt <tim@HIDDEN>
To: control <at> debbugs.gnu.org
Subject: none
Organization: http://www.tim-landscheidt.de/
Date: Tue, 19 Sep 2017 08:47:43 +0000
Message-ID: <87ingfnkgg.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.3 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: control
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.0 (/)

reassign 28477 emacs,gnus
reassign 28234 emacs,gnus
reassign 27194 emacs,gnus
reassign 26298 emacs,gnus
reassign 25892 emacs,gnus
reassign 25383 emacs,gnus

Message received at control <at> debbugs.gnu.org:


Received: (at control) by debbugs.gnu.org; 19 Sep 2017 08:47:51 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Sep 19 04:47:51 2017
Received: from localhost ([127.0.0.1]:47242 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1duEC7-00008g-9P
	for submit <at> debbugs.gnu.org; Tue, 19 Sep 2017 04:47:51 -0400
Received: from mezzogiorno.tim-landscheidt.de ([78.47.87.37]:49255)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <tim@HIDDEN>) id 1duEC6-00008Y-0F
 for control <at> debbugs.gnu.org; Tue, 19 Sep 2017 04:47:50 -0400
Received: from x4e369dc7.dyn.telefonica.de ([78.54.157.199]:34174
 helo=passepartout.tim-landscheidt.de)
 by mezzogiorno.tim-landscheidt.de with esmtpsa
 (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2)
 (envelope-from <tim@HIDDEN>) id 1duEC4-00055f-HY
 for control <at> debbugs.gnu.org; Tue, 19 Sep 2017 08:47:48 +0000
From: Tim Landscheidt <tim@HIDDEN>
To: control <at> debbugs.gnu.org
Subject: none
Organization: http://www.tim-landscheidt.de/
Date: Tue, 19 Sep 2017 08:47:43 +0000
Message-ID: <87ingfnkgg.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.3 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: control
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.0 (/)

reassign 28477 emacs,gnus
reassign 28234 emacs,gnus
reassign 27194 emacs,gnus
reassign 26298 emacs,gnus
reassign 25892 emacs,gnus
reassign 25383 emacs,gnus
Last modified: Mon, 25 Nov 2019 12:00:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.