GNU bug report logs - #43395
28.0.50; memory leak

Previous Next

Package: emacs;

Reported by: Madhu <enometh <at> meer.net>

Date: Mon, 14 Sep 2020 05:37:01 UTC

Severity: normal

Merged with 43389, 43876, 44666

Found in version 28.0.50

Done: Stefan Monnier <monnier <at> iro.umontreal.ca>

Bug is archived. No further changes may be made.

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 43395 in the body.
You can then email your comments to 43395 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to bug-gnu-emacs <at> gnu.org:
bug#43395; Package emacs. (Mon, 14 Sep 2020 05:37:02 GMT) Full text and rfc822 format available.

Acknowledgement sent to Madhu <enometh <at> meer.net>:
New bug report received and forwarded. Copy sent to bug-gnu-emacs <at> gnu.org. (Mon, 14 Sep 2020 05:37:02 GMT) Full text and rfc822 format available.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Madhu <enometh <at> meer.net>
To: bug-gnu-emacs <at> gnu.org
Subject: 28.0.50; memory leak
Date: Sat, 12 Sep 2020 07:42:42 +0530
Following up on the thread:
https://lists.gnu.org/archive/html/help-gnu-emacs/2020-09/msg00147.html

There appears to be a memory leak with emacs RSS growing inordinately in
size.

$ ps o pid,rss,drs,sz,share,start_time,vsize,cmd 26285
  PID   RSS   DRS  SIZE - START    VSZ CMD
26285 2643236 2996379 2664940 - Sep09 2998948 /7/gtk/emacs/build-xt-xft/src/emacs --debug-init --daemon

I usually only notice the leak when it has gone beyond 2G - when linux
refuses to suspend because I have limited swap.  In most cases emacs
would be running for a few days.

The values reported by garbage-collect amount do not reflect the 2GB
allocation being used by emacs.

Advice on tooling is called for to instrument emacs and monitor the
system for memory changes and flag the point when the leak occurs.


In GNU Emacs 28.0.50 (build 1, x86_64-pc-linux-gnu, X toolkit, Xaw3d scroll bars)
 of 2020-09-06 built on maher
Emacs Repository revision: 6fc502c1ef327ab357c971b9bffbbd7cb6a436f1
Repository branch: madhu-tip
Windowing system distributor 'The X.Org Foundation', version 11.0.12008000
System Description: Gentoo/Linux

Configured using:
 'configure -C --with-harfbuzz --without-cairo --with-x-toolkit=athena
 --with-xft'

Configured features:
XAW3D XPM JPEG TIFF GIF PNG RSVG SOUND GPM DBUS GSETTINGS GLIB NOTIFY
INOTIFY ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE HARFBUZZ M17N_FLT LIBOTF
XFT ZLIB TOOLKIT_SCROLL_BARS LUCID X11 XDBE XIM MODULES THREADS JSON
PDUMPER LCMS2

Important settings:
  value of $LC_COLLATE: C
  value of $LANG: en_US.utf8
  locale-coding-system: utf-8-unix

Major mode: Fundamental

Minor modes in effect:
  global-log4sly-mode: t
  global-magit-file-mode: t
  global-git-commit-mode: t
  async-bytecomp-package-mode: t
  other-frame-window-mode: t
  savehist-mode: t
  xclip-mode: t
  dired-single-mode: t
  save-place-mode: t
  recentf-mode: t
  show-paren-mode: t
  shell-dirtrack-mode: t
  minibuffer-depth-indicate-mode: t
  display-time-mode: t
  which-function-mode: t
  foomadhu-clear-output-mode: t
  foomadhu-translate-kbd-paren-mode: t
  new-shell-activate-mode: t
  foomadhu-mode: t
  ivy-prescient-mode: t
  prescient-persist-mode: t
  ivy-mode: t
  tooltip-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:

Features:
(shadow emacsbug sendmail proced meson-mode yaml-mode idlwave
idlwave-help idlw-help desktop frameset mhtml-mode tex-mode latexenc
net-utils url-file url-dired vc-filewise cal-china lunar cal-bahai
cal-islam cal-hebrew holidays hol-loaddefs markdown-mode eieio-opt
speedbar ezimage dframe nndir tabify man wdired log-view log4sly nnagent
nnml mule-util ibuf-ext ibuffer ibuffer-loaddefs cal-julian solar
cal-dst conf-mode cl-indent dabbrev ielm html5-schema css-mode eww
url-queue mm-url scroll-lock rng-xsd xsd-regexp rng-cmpct python js
cc-mode cc-fonts cc-guess cc-menus cc-cmds cc-styles cc-align cc-engine
cc-vars cc-defs vc-rcs vc vc-dispatcher bug-reference make-mode
magit-bookmark magit-imenu git-rebase magit-extras magit-gitignore
magit-ediff ediff ediff-merg ediff-mult ediff-wind ediff-diff ediff-help
ediff-init ediff-util magit-subtree magit-patch magit-submodule
magit-obsolete magit-popup magit-blame magit-stash magit-reflog
magit-bisect magit-push magit-pull magit-fetch magit-clone magit-remote
magit-commit magit-sequence magit-notes magit-worktree magit-tag
magit-merge magit-branch magit-reset magit-files magit-refs magit-status
magit magit-repos magit-apply magit-wip magit-log magit-diff magit-core
magit-autorevert autorevert filenotify magit-margin magit-transient
magit-process magit-mode git-commit transient magit-git magit-section
magit-utils crm log-edit pcvs-util with-editor async-bytecomp async dash
vc-git sly-mk-defsystem grep sly-undefmethod sly-fancy sly-tramp
sly-stickers pulse hi-lock sly-trace-dialog sly-fontifying-fu
sly-package-fu sly-scratch sly-fancy-trace sly-fancy-inspector sly-mrepl
sly-autodoc sly-parse warnings sly-c-p-c sly-retro sly gud
sly-completion sly-buttons sly-messages sly-common apropos arc-mode
archive-mode hyperspec ebuild-mode skeleton sh-script smie executable
two-column iso-transl smerge-mode diff-mode nnfolder canlock org-element
avl-tree ol-eww ol-rmail ol-mhe ol-irc ol-info ol-gnus nnir ol-docview
doc-view jka-compr image-mode exif ol-bibtex bibtex ol-bbdb ol-w3m
org-capture flow-fill mm-archive qp view help-fns radix-tree cl-print
debug backtrace sort gnus-cite mail-extr gnus-bcklg gnus-async gnus-kill
gnus-ml epa-file gnutls nndraft nnmh nnnil gnus-agent gnus-srvr
gnus-score score-mode nnvirtual gnus-msg gnus-cache gnus-art mm-uu
mml2015 mm-view mml-smime smime dig gnus-sum url url-proxy url-privacy
url-expand url-methods url-history mailcap shr kinsoku url-cookie
url-domsuf url-util svg nntp gnus-group gnus-undo gnus-start gnus-dbus
gnus-cloud nnimap nnmail mail-source utf7 netrc nnoo gnus-spec gnus-int
gnus-range message rfc822 mml mml-sec epa epg epg-config mm-decode
mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader
gnus-win misearch multi-isearch network-stream puny nsm rmc bookmark
time-stamp mew-varsx dired-aux term/xterm xterm add-log pinentry
other-frame-window lw-manual lw-manual-data-7-1-0-0 savehist xclip
elisp-slime-nav gnus nnheader gnus-util rmail rmail-loaddefs rfc2047
rfc2045 ietf-drums text-property-search mail-utils mm-util mail-prsvr
company pcase cus-start cus-load ggtags etags fileloop generator ewoc
zenicb-color zenicb-whereis zenicb-complete zenicb-stamp zenicb-history
zenicb-away zenicb zenirc-sasl erc-goodies erc erc-backend pp
erc-loaddefs zenirc-color zenirc-stamp zenirc-trigger zenirc-notify
zenirc-netsplit zenirc-ignore zenirc-history zenirc-format zenirc-dcc
zenirc-complete zenirc-command-queue zenirc-away zenirc sly-autoloads
org-mew mew-auth mew-config mew-imap2 mew-imap mew-nntp2 mew-nntp
mew-pop mew-smtp mew-ssl mew-ssh mew-net mew-highlight mew-sort mew-fib
mew-ext mew-refile mew-demo mew-attach mew-draft mew-message mew-thread
mew-virtual mew-summary4 mew-summary3 mew-summary2 mew-summary
mew-search mew-pick mew-passwd mew-scan mew-syntax mew-bq mew-smime
mew-pgp mew-header mew-exec mew-mark mew-mime mew-unix mew-edit
mew-decode mew-encode mew-cache mew-minibuf mew-complete mew-addrbook
mew-local mew-vars3 mew-vars2 mew-vars mew-env mew-mule3 mew-mule
mew-gemacs mew-key mew-func mew-blvs mew-const mew winner windmove
whitespace tramp-sh tramp tramp-loaddefs trampver tramp-integration
files-x tramp-compat ls-lisp ange-ftp term disp-table ehelp saveplace
recentf tree-widget wid-edit paren ob-lisp ob-shell shell org ob
ob-tangle ob-ref ob-lob ob-table ob-exp org-macro org-footnote org-src
ob-comint org-pcomplete pcomplete org-list org-faces org-entities
noutline outline org-version ob-emacs-lisp ob-core ob-eval org-table ol
org-keys org-compat org-macs org-loaddefs format-spec find-func cal-menu
calendar cal-loaddefs rng-nxml rng-valid rng-loc rng-uri rng-parse
nxml-parse rng-match rng-dt rng-util rng-pttrn nxml-ns nxml-mode
nxml-outln nxml-rap sgml-mode dom nxml-util nxml-enc xmltok mb-depth
ffap thingatpt battery dbus xml time which-func imenu parse-time iso8601
time-date cookie1 server diff generic derived easy-mmode dired-x
gh-common marshal eieio-compat info rx finder-inf package browse-url
url-handlers url-parse auth-source password-cache json url-vars cl
ivy-prescient prescient subr-x map edmacro kmacro counsel xdg advice
xref project eieio eieio-core cl-macs eieio-loaddefs dired
dired-loaddefs compile comint ansi-color swiper cl-seq cl-extra
help-mode easymenu seq byte-opt gv bytecomp byte-compile cconv ivy
delsel ring ivy-faces ivy-overlay colir color cl-loaddefs cl-lib tooltip
eldoc electric uniquify ediff-hook vc-hooks lisp-float-type mwheel
term/x-win x-win term/common-win x-dnd tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode elisp-mode
lisp-mode prog-mode register page tab-bar menu-bar rfn-eshadow isearch
timer select scroll-bar mouse jit-lock font-lock syntax facemenu
font-core term/tty-colors frame minibuffer cl-generic cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese composite charscript charprop
case-table epa-hook jka-cmpr-hook help simple abbrev obarray
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote threads dbusbind
inotify lcms2 dynamic-setting system-font-setting font-render-setting
x-toolkit x multi-tty make-network-process emacs)

Memory information:
((conses 16 3591911 2541685)
 (symbols 48 87049 452)
 (strings 32 528119 452566)
 (string-bytes 1 30189681)
 (vectors 16 217149)
 (vector-slots 8 3232842 6057920)
 (floats 8 1637 5252)
 (intervals 56 501483 50429)
 (buffers 992 581))




Merged 43389 43395. Request was from Eli Zaretskii <eliz <at> gnu.org> to control <at> debbugs.gnu.org. (Mon, 14 Sep 2020 15:00:05 GMT) Full text and rfc822 format available.

Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#43395; Package emacs. (Mon, 14 Sep 2020 15:09:02 GMT) Full text and rfc822 format available.

Message #10 received at 43395 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Madhu <enometh <at> meer.net>
Cc: 43395 <at> debbugs.gnu.org
Subject: Re: bug#43395: 28.0.50; memory leak
Date: Mon, 14 Sep 2020 18:08:26 +0300
> From: Madhu <enometh <at> meer.net>
> Date: Sat, 12 Sep 2020 07:42:42 +0530
> 
> There appears to be a memory leak with emacs RSS growing inordinately in
> size.
> 
> $ ps o pid,rss,drs,sz,share,start_time,vsize,cmd 26285
>   PID   RSS   DRS  SIZE - START    VSZ CMD
> 26285 2643236 2996379 2664940 - Sep09 2998948 /7/gtk/emacs/build-xt-xft/src/emacs --debug-init --daemon
> 
> I usually only notice the leak when it has gone beyond 2G - when linux
> refuses to suspend because I have limited swap.  In most cases emacs
> would be running for a few days.
> 
> The values reported by garbage-collect amount do not reflect the 2GB
> allocation being used by emacs.

Is the GC report below, collected by report-emacs-bug, from the
session whose RSS has grown up to 2GB?  If not, can you post the
output from garbage-collect in that session?

Thanks.




Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#43395; Package emacs. (Tue, 15 Sep 2020 01:24:02 GMT) Full text and rfc822 format available.

Message #13 received at 43395 <at> debbugs.gnu.org (full text, mbox):

From: Madhu <enometh <at> meer.net>
To: eliz <at> gnu.org
Cc: 43395 <at> debbugs.gnu.org
Subject: Re: bug#43395: 28.0.50; memory leak
Date: Tue, 15 Sep 2020 06:53:00 +0530 (IST)
*  Eli Zaretskii <eliz <at> gnu.org> <83y2lc9z11.fsf <at> gnu.org>
Wrote on Mon, 14 Sep 2020 18:08:26 +0300
> Is the GC report below, collected by report-emacs-bug, from the
> session whose RSS has grown up to 2GB?  If not, can you post the
> output from garbage-collect in that session?

Yes it is from the same offending session and was collected by
report-emacs-bug. (that session is now long gone, "pining for the
fjords")





Merged 43389 43395 43876. Request was from Eli Zaretskii <eliz <at> gnu.org> to control <at> debbugs.gnu.org. (Fri, 09 Oct 2020 06:59:03 GMT) Full text and rfc822 format available.

Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#43395; Package emacs. (Tue, 10 Nov 2020 14:25:02 GMT) Full text and rfc822 format available.

Message #18 received at 43395 <at> debbugs.gnu.org (full text, mbox):

From: Jean Louis <bugs <at> gnu.support>
To: Madhu <enometh <at> meer.net>
Cc: 43395 <at> debbugs.gnu.org
Subject: Re: memory leaks
Date: Tue, 10 Nov 2020 17:22:46 +0300
* Madhu <enometh <at> meer.net> [2020-11-10 04:27]:
> [Test case follows. Sorry Drew for trying to post to help-gnu and
> debbugs simultaneously, but this is my third attempt at sending this
> test case - I've tried sending it to debbugs.gnu.org and to
> gmane.emacs.help newsgroup - and my messages have been dropped]
> 
> 
> * Madhu <m35z8gise6.fsf <at> leonis4.robolove.meer.net> :
> Wrote on Mon, 14 Sep 2020 15:36:57 +0530:
> > The numbers I reported by garbage-collect do not account for the memory
> > usage. FWIW I've posted that info again at
> > https://debbugs.gnu.org/cgi/bugreport.cgi?bug=43395
> 
> I found test case for this bug and I sent mail to 43395 <at> debbugs.gnu.org,
> and qmail told me that the mail was accepted:
> delivery 1: success:
> 209.51.188.43_accepted_message./Remote_host_said:_250_OK_id=1kc8Bf-00061T-3u/
> But the bugreport didn't show up on gnu.emacs.bug.
> 
> Here is the test case:
> 
> #+BEGIN_SRC
> $ cat > f.c
> #include <stdio.h>
> int
> main()
> {
>   char c = ' ';
>   while (c != 'q' && c != 'Q')
>     {
>       fprintf(stdout, "Press q then enter to quit: ");
>       c = fgetc(stdin);
>     }
>   return 0;
> }
> ^D
> 
> $ gcc f.c
> $ emacs -Q
> #+END_SRC
> 
> M-x shell-command ./a.out
> 
> Then the process hangs. But Emacs' memory grows unbounded.
> 
> WARNING. Be careful to interrupt it with ^G before the OOM killer
> kicks in.  If you have swap you may lose control of your machine
> indefinitely.
> 
> After you interrupt the sub process, Emacs is left with unreclaimed
> gigabytes of RSS
> 
> Please let me know if you can reproduce this

Me I have been testing now. It all blocks. Then I had to switch to
console from X to kill process.

I can see memory is not reclaimed graphically. Not practically. I have
graphical indicator.

This same things happens from time to time in Emacs. Exactly like you
say when I have swap I may lose control of machine. Then I had to
power off with the hard key. Bad.

I hope this can be resolved.





Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#43395; Package emacs. (Tue, 10 Nov 2020 15:29:02 GMT) Full text and rfc822 format available.

Message #21 received at 43395 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Jean Louis <bugs <at> gnu.support>
Cc: enometh <at> meer.net, 43395 <at> debbugs.gnu.org
Subject: Re: bug#43395: memory leaks
Date: Tue, 10 Nov 2020 17:28:13 +0200
> Date: Tue, 10 Nov 2020 17:22:46 +0300
> From: Jean Louis <bugs <at> gnu.support>
> Cc: 43395 <at> debbugs.gnu.org
> 
> > #+BEGIN_SRC
> > $ cat > f.c
> > #include <stdio.h>
> > int
> > main()
> > {
> >   char c = ' ';
> >   while (c != 'q' && c != 'Q')
> >     {
> >       fprintf(stdout, "Press q then enter to quit: ");
> >       c = fgetc(stdin);
> >     }
> >   return 0;
> > }
> > ^D
> > 
> > $ gcc f.c
> > $ emacs -Q
> > #+END_SRC
> > 
> > M-x shell-command ./a.out
> > 
> > Then the process hangs. But Emacs' memory grows unbounded.

This is expected, and is not a bug.

'shell-command' is intended for running non-interactive programs,
those which produce output, but don't expect any input.  So it runs
the sub-process with its standard input connected to the null device.
Your program has a bug in that case: it doesn't detect the EOF
condition on its standard input (to see that, invoke it as
"./a.out < /dev/null").  So it loops indefinitely, with each iteration
pumping the prompts into Emacs, which obediently collects that in a
buffer, that grows and grows and grows, until it eats up all the
available memory.  And evidently, you didn't configure your system to
have resident size limitation on user processes, so instead of
reporting "memory full", Emacs is allowed to eat up all your memory
and swap.

In short: don't do that.

I see no bug here: Emacs cannot know in advance how much stuff will be
emitted by the sub-process, and it cannot know how much memory it cqan
swallow before OOMK will sporing into action.

And, of course, this is unrelated to the problems being discussed in
this bug report.




Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#43395; Package emacs. (Tue, 10 Nov 2020 19:41:01 GMT) Full text and rfc822 format available.

Message #24 received at 43395 <at> debbugs.gnu.org (full text, mbox):

From: Jean Louis <bugs <at> gnu.support>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: enometh <at> meer.net, 43395 <at> debbugs.gnu.org
Subject: Re: bug#43395: memory leaks
Date: Tue, 10 Nov 2020 22:40:23 +0300
* Eli Zaretskii <eliz <at> gnu.org> [2020-11-10 18:28]:
> > > M-x shell-command ./a.out
> > > 
> > > Then the process hangs. But Emacs' memory grows unbounded.
> 
> This is expected, and is not a bug.
> 
> 'shell-command' is intended for running non-interactive programs,
> those which produce output, but don't expect any input.  So it runs
> the sub-process with its standard input connected to the null device.
> Your program has a bug in that case: it doesn't detect the EOF
> condition on its standard input (to see that, invoke it as
> "./a.out < /dev/null").  So it loops indefinitely, with each iteration
> pumping the prompts into Emacs, which obediently collects that in a
> buffer, that grows and grows and grows, until it eats up all the
> available memory.  And evidently, you didn't configure your system to
> have resident size limitation on user processes, so instead of
> reporting "memory full", Emacs is allowed to eat up all your memory
> and swap.

Thank you, it helps!

Do I need to use `ulimit -m'?

I guess following should be for 1 GiB:
ulimit -m 1048576

and this for 2 GiB:
ulimit -m 2097152

Now I can execute ./a.out without getting blocked in Emacs. I will see
how to tweak that more.





Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#43395; Package emacs. (Tue, 10 Nov 2020 20:30:03 GMT) Full text and rfc822 format available.

Message #27 received at 43395 <at> debbugs.gnu.org (full text, mbox):

From: Madhu <enometh <at> meer.net>
To: eliz <at> gnu.org
Cc: 43395 <at> debbugs.gnu.org, bugs <at> gnu.support
Subject: Re: bug#43395: memory leaks
Date: Wed, 11 Nov 2020 01:59:10 +0530 (IST)
*  Eli Zaretskii <eliz <at> gnu.org> <83k0ut2pv6.fsf <at> gnu.org>
Wrote on Tue, 10 Nov 2020 17:28:13 +0200

> This is expected, and is not a bug.
>
> 'shell-command' is intended for running non-interactive programs,
> those which produce output, but don't expect any input.  So it runs
> the sub-process with its standard input connected to the null device.
> Your program has a bug in that case: it doesn't detect the EOF
> condition on its standard input (to see that, invoke it as
> "./a.out < /dev/null").  So it loops indefinitely, with each iteration
> pumping the prompts into Emacs, which obediently collects that in a
> buffer, that grows and grows and grows, until it eats up all the
> available memory.  And evidently, you didn't configure your system to
> have resident size limitation on user processes, so instead of
> reporting "memory full", Emacs is allowed to eat up all your memory
> and swap.
>
> In short: don't do that.
>
> I see no bug here: Emacs cannot know in advance how much stuff will be
> emitted by the sub-process, and it cannot know how much memory it cqan
> swallow before OOMK will sporing into action.
>
> And, of course, this is unrelated to the problems being discussed in
> this bug report.

Sorry, yes - the memory is indeed freed up when *Shell Command Output*
is killed.  I was mistaken in thinking that the memory was not freed
up when all the buffers had been deleted.

Apologies.




Merged 43389 43395 43876 44666. Request was from Eli Zaretskii <eliz <at> gnu.org> to control <at> debbugs.gnu.org. (Mon, 16 Nov 2020 20:24:01 GMT) Full text and rfc822 format available.

bug archived. Request was from Debbugs Internal Request <help-debbugs <at> gnu.org> to internal_control <at> debbugs.gnu.org. (Sun, 07 Mar 2021 12:24:07 GMT) Full text and rfc822 format available.

bug unarchived. Request was from Madhu <enometh <at> meer.net> to control <at> debbugs.gnu.org. (Sun, 21 Mar 2021 15:53:02 GMT) Full text and rfc822 format available.

bug archived. Request was from Debbugs Internal Request <help-debbugs <at> gnu.org> to internal_control <at> debbugs.gnu.org. (Mon, 19 Apr 2021 11:24:06 GMT) Full text and rfc822 format available.

This bug report was last modified 2 years and 344 days ago.

Previous Next


GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.