GNU bug report logs - #28329
25.2; Eshell: 40M+ output: Stack overflow in regexp matcher

Previous Next

To reply to this bug, email your comments to 28329 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Pierre Neidhardt <ambrevar <at> gmail.com>
To: bug-gnu-emacs <at> gnu.org
Subject: 25.2; Eshell: 40M+ output: Stack overflow in regexp matcher
Date: Sat, 02 Sep 2017 10:34:55 +0100

Today I update my Gentoo:

	> emerge --update --newuse --deep @world

in Eshell.  After a very long output (40M+), the shell started stuttering and hanging, messaging the following errors on every new line of output:

	error in process filter: eshell-watch-for-password-prompt: Stack overflow in regexp matcher
	error in process filter: Stack overflow in regexp matcher

The process became so slow it was unlikely it would ever complete, so I terminated it and had to re-run it in a *term*.



In GNU Emacs 25.2.1 (x86_64-pc-linux-gnu, GTK+ Version 3.22.15)
 of 2017-08-28 built on dhiov23k
Windowing system distributor 'The X.Org Foundation', version 11.0.11903000
System Description:	Gentoo Base System release 2.3

Configured using:
 'configure --prefix=/usr --build=x86_64-pc-linux-gnu
 --host=x86_64-pc-linux-gnu --mandir=/usr/share/man
 --infodir=/usr/share/info --datadir=/usr/share --sysconfdir=/etc
 --localstatedir=/var/lib --disable-dependency-tracking
 --disable-silent-rules --docdir=/usr/share/doc/emacs-25.2
 --htmldir=/usr/share/doc/emacs-25.2/html --libdir=/usr/lib64
 --program-suffix=-emacs-25 --infodir=/usr/share/info/emacs-25
 --localstatedir=/var
 --enable-locallisppath=/etc/emacs:/usr/share/emacs/site-lisp
 --with-gameuser=:gamestat --without-compress-install
 --with-file-notification=inotify --enable-acl --without-dbus
 --without-modules --without-gpm --without-hesiod --without-kerberos
 --without-kerberos5 --with-xml2 --without-selinux --with-gnutls
 --without-wide-int --with-zlib --with-sound=alsa --with-x --without-ns
 --without-gconf --without-gsettings --without-toolkit-scroll-bars
 --with-gif --with-jpeg --with-png --with-rsvg --with-tiff --with-xpm
 --with-imagemagick --with-xft --without-cairo --without-libotf
 --without-m17n-flt --with-x-toolkit=gtk3 --without-xwidgets
 GENTOO_PACKAGE=app-editors/emacs-25.2 'CFLAGS=-march=ivybridge -O2
 -pipe' CPPFLAGS= 'LDFLAGS=-Wl,-O1 -Wl,--as-needed''

Configured features:
XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND NOTIFY ACL GNUTLS LIBXML2
FREETYPE XFT ZLIB GTK3 X11

Important settings:
  value of $LANG: en_US.utf8
  locale-coding-system: utf-8-unix

Message #8 received at 28329 <at> debbugs.gnu.org (full text, mbox):

From: npostavs <at> users.sourceforge.net
To: Pierre Neidhardt <ambrevar <at> gmail.com>
Cc: 28329 <at> debbugs.gnu.org
Subject: Re: bug#28329: 25.2;
 Eshell: 40M+ output: Stack overflow in regexp matcher
Date: Sat, 02 Sep 2017 08:30:01 -0400

Pierre Neidhardt <ambrevar <at> gmail.com> writes:

> Today I update my Gentoo:
>
> 	> emerge --update --newuse --deep @world
>
> in Eshell.  After a very long output (40M+), the shell started stuttering and hanging, messaging the following errors on every new line of output:
>
> 	error in process filter: eshell-watch-for-password-prompt: Stack overflow in regexp matcher
> 	error in process filter: Stack overflow in regexp matcher
>
> The process became so slow it was unlikely it would ever complete, so I terminated it and had to re-run it in a *term*.

Can you get a backtrace if you M-x toggle-debug-on-error?  It would be
helpful to get a sample of output which triggers this.

Does changing the password regexp help?

    (setq eshell-password-prompt-regexp
          (format "^.\\{,80\\}%s.\\{,80\\}:\\s-*\\'"
                  (regexp-opt password-word-equivalents)))

Message #11 received at 28329 <at> debbugs.gnu.org (full text, mbox):

From: Pierre Neidhardt <ambrevar <at> gmail.com>
To: npostavs <at> users.sourceforge.net
Cc: 28329 <at> debbugs.gnu.org
Subject: Re: bug#28329: 25.2;
 Eshell: 40M+ output: Stack overflow in regexp matcher
Date: Sat, 02 Sep 2017 14:27:25 +0100

Sadly, I had to kill Emacs before I could do it (it was almost
completely hanging).

Thinking of a way of reproduicing this... Maybe create a 40M+ file of
random clang/g++ output and dump it to Eshell's stdout.

-- 
Pierre Neidhardt

Message #14 received at 28329 <at> debbugs.gnu.org (full text, mbox):

From: npostavs <at> users.sourceforge.net
To: 28329 <at> debbugs.gnu.org, Pierre Neidhardt <ambrevar <at> gmail.com>
Subject: Re: bug#28329: 25.2;
 Eshell: 40M+ output: Stack overflow in regexp matcher
Date: Sat, 02 Sep 2017 22:19:20 -0400

[Message part 1 (text/plain, inline)]

[forwarding to bug list]

[Message part 2 (message/rfc822, inline)]

From: Pierre Neidhardt <pe.neidhardt <at> googlemail.com>
To: npostavs <at> users.sourceforge.net
Subject: Re: bug#28329: 25.2; Eshell: 40M+ output: Stack overflow in regexp matcher
Date: Sat, 02 Sep 2017 19:02:21 +0100

Alright, I managed to get a stack trace:

Debugger entered--Lisp error: (error "Stack overflow in regexp matcher")
  re-search-forward("\\(\\(?:adgangskode\\|contrase\\(?:\\(?:ny\\|ñ\\)a\\)\\|geslo\\|h\\(?:\\(?:asł\\|esl\\)o\\)\\|iphasiwedi\\|jelszó\\|l\\(?:ozinka\\|ösenord\\)\\|m\\(?:ot de passe\\|ật khẩu\\)\\|pa\\(?:rola\\|s\\(?:ahitza\\|s\\(?: phrase\\|code\\|ord\\|phrase\\|wor[dt]\\)\\|vorto\\)\\)\\|s\\(?:alasana\\|enha\\|laptažodis\\)\\|wachtwoord\\|лозинка\\|пароль\\|ססמה\\|كلمة السر\\|गुप्तशब्द\\|शब्दकूट\\|গুপ্তশব্দ\\|পাসওয়ার্ড\\|ਪਾਸਵਰਡ\\|પાસવર્ડ\\|ପ୍ରବେଶ ସଙ୍କେତ\\|கடவுச்சொல்\\|సంకేతపదము\\|ಗುಪ್ತಪದ\\|അടയാളവാക്ക്\\|රහස්පදය\\|ពាក្យសម្ងាត់\\|パスワード\\|密[码碼]\\|암호\\)\\).*:\\s *\\'" #<marker at 20572261 in *eshell*> t)
  eshell-watch-for-password-prompt()
  run-hooks(eshell-output-filter-functions)
  eshell-run-output-filters()
  eshell-output-filter(#<process emerge> "ium/third_party/WebKit/Source/platform/network/blink_platform.WebSocketHandshakeRequest.o obj/src/3rdparty/chromium/third_party/WebKit/Source/platform/network/blink_platform.WebSocketHandshakeResponse.o obj/src/3rdparty/chromium/third_party/WebKit/Source/platform/plugins/blink_platform.PluginData.o obj/src/3rdparty/chromium/third_party/WebKit/Source/platform/plugins/blink_platform.PluginListBuilder.o obj/src/3rdparty/chromium/third_party/WebKit/Source/platform/scheduler/blink_platform.CancellableTaskFactory.o obj/src/3rdparty/chromium/third_party/WebKit/Source/platform/scroll/blink_platform.ProgrammaticScrollAnimator.o obj/src/3rdparty/chromium/third_party/WebKit/Source/platform/scroll/blink_platform.ScrollAnimatorBase.o obj/src/3rdparty/chromium/third_party/WebKit/Source/platform/scroll/blink_platform.ScrollAnimatorCompositorCoordinator.o obj/src/3rdparty/chromium/third_party/WebKit/Source/platform/scroll/blink_platform.ScrollAnimator.o obj/src/3rdparty/chromium/third_party/WebKit/Source/platform/scroll/blink_platform.ScrollableArea.o obj/src/3rdparty/chromium/third_party/WebKit/Source/platform/scroll/blink_platform.Scrollbar.o obj/src/3rdparty/chromium/third_party/WebKit/Source/platform/scroll/blink_platform.ScrollbarTheme.o obj/src/3rdparty/chromium/third_party/WebKit/Source/platform/scroll/blink_platform.ScrollbarThemeAura.o obj/src/3rdparty/chromium/third_party/WebKit/Source/platform/scroll/blink_platform.ScrollbarThemeMock.o obj/src/3rdparty/chromium/third_party/WebKit/Source/platform/scroll/blink_platform.ScrollbarThemeNonMacCommon.o obj/src/3rdparty/chromium/third_party/WebKit/Source/platform/scroll/blink_platform.ScrollbarThemeOverlay.o obj/src/3rdparty/chromium/third_party/WebKit/Source/platform/speech/blink_platform.PlatformSpeechSynthesisUtterance.o obj/src/3rdparty/chromium/third_party/WebKit/Source/platform/speech/blink_platform.PlatformSpeechSynthesisVoice.o obj/src/3rdparty/chromium/third_party/WebKit/Source/platform/speech/blink_platform.PlatformSpeechSynthesizer.o obj/src/3rdparty/chromium/third_party/WebKit/Source/platform/text/blink_platform.BidiCharacterRun.o obj/src/3rdparty/chromium/third_party/WebKit/Source/platform/text/blink_platform.BidiContext.o obj/src/3rdparty/chromium/third_party/WebKit/Source/platform/text/blink_platform.BidiTextRun.o obj/src/3rdparty/chromium/third_party/WebKit/Source/platform/text/blink_platform.DateTimeFormat.o obj/src/3rdparty/chromium/third_party/WebKit/Source/platform/text/blink_platform.LineEnding.o obj/src/3rdparty/chromium/third_party/WebKit/Source/platform/text/blink_platform.LocaleICU.o obj/src/3rdparty/chromium/third_party/WebKit/Source/platform/text/blink_platform.LocaleToScriptMapping.o obj/src/3rdparty/chromium/third_party/WebKit/Source/platform/text/blink_platform.PlatformLocale.o obj/src/3rdparty/chromium/third_party/WebKit/Source/platform/text/blink_platform.QuotedPrintable.o obj/src/3rdparty/chromium/third_party/WebKit/Source/platform/text/blink_platform.SegmentedString.o obj/src/3rdparty/chromium/third_party/WebKit/Source/platform/text/blink_platform.StringTruncator.o obj/src/3rdparty/chromium/third_party/WebKit/Source/platform/text/blink_platform.TextBoundaries.o obj/src/3rdparty/chromium/third_party/WebKit/Source/platform/text/blink_platform.TextBreakIterator.o obj/src/3rdparty/chromium/third_party/WebKit/Source/platform/text/blink_platform.TextBreakIteratorICU.o obj/src/3rdparty/chromium/third_party/WebKit/Source/platform/text/blink_platform.TextBreakIteratorInternalICU.o obj/src/3rdparty/chromium/third_party/WebKit/Source/platform/text/blink_platform.TextCheckerClient.o obj/src/3rdparty/chromium/third_party/WebKit/Source/platform/text/blink_platform.TextEncodingDetector.o obj/src/3rdparty/chromium/third_party/WebKit/Source/platform/text/blink_platform.TextRun.o obj/src/3rdparty/chromium/third_party/WebKit/Source/platform/text/blink_platform.TextStream.o obj/src/3rdparty/chromium/third_party/WebKit/Source/platform/text/blink_platform.UnicodeRange.o obj/src/3rdparty/chromium/third_party/WebKit/Source/platform/text/blink_platform.UnicodeUtilities.o ob")

Apparently, the line is just plain too long...
I guess limiting the regexp matcher over, say, the first 256 characters
would fix the issue.

Message #17 received at 28329 <at> debbugs.gnu.org (full text, mbox):

From: Noam Postavsky <npostavs <at> users.sourceforge.net>
To: 28329 <at> debbugs.gnu.org
Cc: Pierre Neidhardt <ambrevar <at> gmail.com>
Subject: Re: bug#28329: 25.2;
 Eshell: 40M+ output: Stack overflow in regexp matcher
Date: Sun, 24 Sep 2017 08:55:30 -0400

> From: Pierre Neidhardt <pe.neidhardt <at> googlemail.com>
> Subject: Re: bug#28329: 25.2; Eshell: 40M+ output: Stack overflow in regexp matcher
> To: npostavs <at> users.sourceforge.net
> Date: Sat, 02 Sep 2017 19:02:21 +0100 (3 weeks, 18 hours, 25 minutes ago)
>
> Alright, I managed to get a stack trace:
>
> Debugger entered--Lisp error: (error "Stack overflow in regexp matcher")
>   re-search-forward("\\(\\(?:adgangskode\\|contrase\\(?:\\(?:ny\\|ñ\\)a\\)\\|geslo\\|h\\(?:\\(?:asł\\|esl\\)o\\)\\|iphasiwedi\\|jelszó\\|l\\(?:ozinka\\|ösenord\\)\\|m\\(?:ot
> de passe\\|ật khẩu\\)\\|pa\\(?:rola\\|s\\(?:ahitza\\|s\\(?:
> phrase\\|code\\|ord\\|phrase\\|wor[dt]\\)\\|vorto\\)\\)\\|s\\(?:alasana\\|enha\\|laptažodis\\)\\|wachtwoord\\|лозинка\\|пароль\\|ססמה\\|كلمة
> السر\\|गुप्तशब्द\\|शब्दकूट\\|গুপ্তশব্দ\\|পাসওয়ার্ড\\|ਪਾਸਵਰਡ\\|પાસવર્ડ\\|ପ୍ରବେଶ
> ସଙ୍କେତ\\|கடவுச்சொல்\\|సంకేతపదము\\|ಗುಪ್ತಪದ\\|അടയാളവാക്ക്\\|රහස්පදය\\|ពាក្យសម្ងាត់\\|
> パスワード\\|密[码碼]\\|암호\\)\\).*:\\s *\\'" #<marker at 20572261 in
> *eshell*> t)
>   eshell-watch-for-password-prompt()
>   run-hooks(eshell-output-filter-functions)
>   eshell-run-output-filters()
>   eshell-output-filter(#<process emerge>
> "ium/third_party/WebKit/Source/platform/network/blink_platform.WebSocketHandshakeRequest.o
> obj/src/3rdparty/chromium/third_party/WebKit/Source/platform/network/blink_platform.WebSocketHandshakeResponse.o
[...]
> ob")
>
> Apparently, the line is just plain too long...
> I guess limiting the regexp matcher over, say, the first 256 characters
> would fix the issue.
> ----------

Can you catch that build output in a file, so it could be reproduced by
running 'cat build.log' in eshell?  I tried copying the fragment you
have in your backtrace a couple of times to make it longer, but I was
unable to trigger the error.  Perhaps the particular contents of the
string matters too (some values might let the matcher short-circuit the
backtracking).

Message #20 received at 28329 <at> debbugs.gnu.org (full text, mbox):

From: Noam Postavsky <npostavs <at> users.sourceforge.net>
To: 28329 <at> debbugs.gnu.org
Cc: Pierre Neidhardt <ambrevar <at> gmail.com>
Subject: Re: bug#28329: 25.2;
 Eshell: 40M+ output: Stack overflow in regexp matcher
Date: Fri, 20 Oct 2017 21:20:06 -0400

tags 28329 + moreinfo unreproducible
quit

Noam Postavsky <npostavs <at> users.sourceforge.net> writes:

>> Apparently, the line is just plain too long...
>> I guess limiting the regexp matcher over, say, the first 256 characters
>> would fix the issue.
>> ----------
>
> Can you catch that build output in a file, so it could be reproduced by
> running 'cat build.log' in eshell?  I tried copying the fragment you
> have in your backtrace a couple of times to make it longer, but I was
> unable to trigger the error.  Perhaps the particular contents of the
> string matters too (some values might let the matcher short-circuit the
> backtracking).

ping?

Message #25 received at 28329 <at> debbugs.gnu.org (full text, mbox):

From: Pierre Neidhardt <ambrevar <at> gmail.com>
To: Noam Postavsky <npostavs <at> users.sourceforge.net>
Cc: 28329 <at> debbugs.gnu.org
Subject: Re: bug#28329: 25.2;
 Eshell: 40M+ output: Stack overflow in regexp matcher
Date: Sun, 05 Nov 2017 14:58:47 +0100

Alright, I run into this again while emerging qtwebengine on Gentoo.
Now I think about it, this is the only and systematic cause of the issue
for me.
I suspect that emerging / compiling Chrome or affiliated projects would
also trigger the issue.

This time I managed to save the output, 60M+ big.

I tried to following

- emacs -Q
- M-x eshell
- cat portage.output.log

It took about 2 hours to complete!  While it takes half a second to
paste to the buffer.
Damn! Eshell's filters are slow! :/

That being said, no issue doing that.  So the output "saved after the
fact" does not seem to produce the issue.

-- 
Pierre Neidhardt

Finagle's Seventh Law:
	The perversity of the universe tends toward a maximum.

Message #28 received at 28329 <at> debbugs.gnu.org (full text, mbox):

From: Noam Postavsky <npostavs <at> users.sourceforge.net>
To: Pierre Neidhardt <ambrevar <at> gmail.com>
Cc: 28329 <at> debbugs.gnu.org
Subject: Re: bug#28329: 25.2;
 Eshell: 40M+ output: Stack overflow in regexp matcher
Date: Sun, 05 Nov 2017 09:44:42 -0500

Pierre Neidhardt <ambrevar <at> gmail.com> writes:

> That being said, no issue doing that.  So the output "saved after the
> fact" does not seem to produce the issue.

Damn.  I guess the chunking is significant too.  Could you try saving
the output chunks, with this:

    (defvar eshell-chunk-number 0)
    (defconst eshell-output-chunk-dir "eshell-output")
    (make-directory eshell-output-chunk-dir t)

    (defun catch-eshell-output-chunk ()
      (write-region eshell-last-output-block-begin
                    eshell-last-output-end
                    (format "%s/chunk.%d"
                            eshell-output-chunk-dir
                            eshell-chunk-number)
                    nil :quiet)
      (setq eshell-chunk-number (1+ eshell-chunk-number)))

    (add-hook 'eshell-output-filter-functions
              'catch-eshell-output-chunk)

And then afterwards 'cat eshell-output/chunk.*' should hopefully
reproduce it?

Message #31 received at 28329 <at> debbugs.gnu.org (full text, mbox):

From: Pierre Neidhardt <ambrevar <at> gmail.com>
To: Noam Postavsky <npostavs <at> users.sourceforge.net>
Cc: 28329 <at> debbugs.gnu.org
Subject: Re: bug#28329: 25.2;
 Eshell: 40M+ output: Stack overflow in regexp matcher
Date: Sun, 05 Nov 2017 15:54:17 +0100

[Message part 1 (text/plain, inline)]

Will do.  It's a tough one though because for now reproducing involves 1
hour+ of compiling...

--
Pierre Neidhardt

Pascal:
	A programming language named after a man who would turn over
	in his grave if he knew about it.
		-- Datamation, January 15, 1984

[signature.asc (application/pgp-signature, inline)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.