GNU bug report logs - #34910
27.0.50; Too much matching reentrancy

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 34910 in the body.
You can then email your comments to 34910 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Mattias Engdegård <mattiase <at> acm.org>
To: bug-gnu-emacs <at> gnu.org, dancol <at> dancol.org
Subject: 27.0.50; Too much matching reentrancy
Date: Mon, 18 Mar 2019 22:34:07 +0100

Run, in *scratch*, with emacs -Q:

(with-temp-buffer
  (dotimes (i 10000)
    (insert "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"))
  (insert "b")
  (goto-char (point-min))
  (looking-at ".*b"))

Result: "Too much matching reentrancy" in the minibuffer. No traceback.
Expected: The usual "Stack overflow in regexp matcher" error and
traceback.

The culprit seems to be

commit 938d252d1c6c5e2027aa250c649deb024154f936
Author: Daniel Colascione <dancol <at> dancol.org>
Date:   Sat Jun 16 13:46:10 2018 -0700

    Make regex matching reentrant; update syntax during match



In GNU Emacs 27.0.50 (build 9, x86_64-pc-linux-gnu, GTK+ Version
3.24.1)
 of 2019-03-18 built on dhcppc1
Repository revision: 799eb257970b1f6aeb5833417b6dd943d6479977
Repository branch: master
Windowing system distributor 'Fedora Project', version 11.0.12004000
System Description: Fedora 29 (Workstation Edition)

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.

Configured using:
 'configure --prefix=/home/mattias/eminst'

Configured features:
XPM JPEG TIFF GIF PNG SOUND DBUS GSETTINGS GLIB NOTIFY INOTIFY
LIBSELINUX GNUTLS FREETYPE XFT ZLIB TOOLKIT_SCROLL_BARS GTK3 X11 XDBE
XIM THREADS PDUMPER GMP

Important settings:
  value of $LC_MONETARY: sv_SE.UTF-8
  value of $LC_NUMERIC: sv_SE.UTF-8
  value of $LC_TIME: sv_SE.UTF-8
  value of $LANG: sv_SE.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rmc puny seq byte-opt gv
bytecomp byte-compile cconv dired dired-loaddefs format-spec rfc822 mml
easymenu mml-sec password-cache epa derived epg epg-config gnus-util
rmail rmail-loaddefs time-date mm-decode mm-bodies mm-encode mail-parse
rfc2231 mailabbrev gmm-utils mailheader cl-loaddefs cl-lib sendmail
rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils elec-pair
mule-util tooltip eldoc electric uniquify ediff-hook vc-hooks
lisp-float-type mwheel term/x-win x-win term/common-win x-dnd tool-bar
dnd fontset image regexp-opt fringe tabulated-list replace newcomment
text-mode elisp-mode lisp-mode prog-mode register page menu-bar
rfn-eshadow isearch timer select scroll-bar mouse jit-lock font-lock
syntax facemenu font-core term/tty-colors frame cl-generic cham
georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese composite charscript charprop
case-table epa-hook jka-cmpr-hook help simple abbrev obarray minibuffer
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote threads dbusbind
inotify dynamic-setting system-font-setting font-render-setting
move-toolbar gtk x-toolkit x multi-tty make-network-process emacs)

Memory information:
((conses 16 44594 10186)
 (symbols 48 5845 2)
 (strings 32 14860 1818)
 (string-bytes 1 490684)
 (vectors 16 9450)
 (vector-slots 8 116818 14096)
 (floats 8 17 58)
 (intervals 56 186 0)
 (buffers 992 11))

Message #8 received at 34910 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Mattias Engdegård <mattiase <at> acm.org>, dancol <at> dancol.org
Cc: dancol <at> dancol.org, 34910 <at> debbugs.gnu.org
Subject: Re: bug#34910: 27.0.50; Too much matching reentrancy
Date: Tue, 19 Mar 2019 10:05:34 +0200

> From: Mattias Engdegård <mattiase <at> acm.org>
> Date: Mon, 18 Mar 2019 22:34:07 +0100
> 
> Run, in *scratch*, with emacs -Q:
> 
> (with-temp-buffer
>   (dotimes (i 10000)
>     (insert "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"))
>   (insert "b")
>   (goto-char (point-min))
>   (looking-at ".*b"))
> 
> Result: "Too much matching reentrancy" in the minibuffer. No traceback.
> Expected: The usual "Stack overflow in regexp matcher" error and
> traceback.

Are you complaining against a different error message, or against the
lack of backtrace?

If the former, please explain why you expected that particular
error, and no other.

Thanks.

Message #11 received at 34910 <at> debbugs.gnu.org (full text, mbox):

From: Mattias Engdegård <mattiase <at> acm.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: dancol <at> dancol.org, 34910 <at> debbugs.gnu.org
Subject: Re: bug#34910: 27.0.50; Too much matching reentrancy
Date: Tue, 19 Mar 2019 10:27:09 +0100

19 mars 2019 kl. 09.05 skrev Eli Zaretskii <eliz <at> gnu.org>:
> 
>> Result: "Too much matching reentrancy" in the minibuffer. No traceback.
>> Expected: The usual "Stack overflow in regexp matcher" error and
>> traceback.
> 
> Are you complaining against a different error message, or against the
> lack of backtrace?

Both. In fact, a gdb-level stack trace indicates that the "too much reentrancy" error was signalled when trying to deal with the stack overflow error somehow. I didn't get further then, and thought that Daniel Colascione would know more, since bisecting found his particular change.

The should be no question that "stack overflow" + debugger is the correct and intended behaviour, should it?

Message #14 received at 34910 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Mattias Engdegård <mattiase <at> acm.org>
Cc: dancol <at> dancol.org, 34910 <at> debbugs.gnu.org
Subject: Re: bug#34910: 27.0.50; Too much matching reentrancy
Date: Tue, 19 Mar 2019 12:01:42 +0200

> From: Mattias Engdegård <mattiase <at> acm.org>
> Date: Tue, 19 Mar 2019 10:27:09 +0100
> Cc: dancol <at> dancol.org, 34910 <at> debbugs.gnu.org
> 
> The should be no question that "stack overflow" + debugger is the correct and intended behaviour, should it?

I don't know yet.  I asked you to elaborate on the reasons for your
expectations, and I still hope you will humor me.  Because just
taking the problem's manifestations at face value, one could argue
that the precise error message is immaterial, as long as it points to
the right direction, and "stack overflow in regexp matcher" is as
obscure as the other message.  But maybe I'm missing something, which
is why I asked you to elaborate on your reasoning.

Thanks.

Message #17 received at 34910 <at> debbugs.gnu.org (full text, mbox):

From: Mattias Engdegård <mattiase <at> acm.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: dancol <at> dancol.org, 34910 <at> debbugs.gnu.org
Subject: Re: bug#34910: 27.0.50; Too much matching reentrancy
Date: Tue, 19 Mar 2019 13:25:13 +0100

[Message part 1 (text/plain, inline)]

tis 2019-03-19 klockan 12:01 +0200 skrev Eli Zaretskii:
> 
> I don't know yet.  I asked you to elaborate on the reasons for your
> expectations, and I still hope you will humor me.  Because just
> taking the problem's manifestations at face value, one could argue
> that the precise error message is immaterial, as long as it points to
> the right direction, and "stack overflow in regexp matcher" is as
> obscure as the other message.  But maybe I'm missing something, which
> is why I asked you to elaborate on your reasoning.

Sorry, I didn't mean to sound touchy! Being the bug reporter, I deserve
every probing question. I'm most grateful for your quick reply!

It turns out the regexp cache logic is broken: it always uses the last
(LRU) entry, and fails if that entry is busy. It should really use the
last free entry.

What happened was that a regexp failed because of stack overflow, which
caused lots of code to run and many regexps to be compiled, but the
original (failing) regexp was still marked busy in the cache.
Eventually, it had been pushed down to the last position in the cache,
where it caused the next request for a free cache entry to fail with
the reentrancy error.

This patch fixes it. Please review.

[0001-Fix-spurious-regexp-reentrancy-error.patch (text/x-patch, attachment)]

Message #20 received at 34910 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Mattias Engdegård <mattiase <at> acm.org>
Cc: dancol <at> dancol.org, 34910 <at> debbugs.gnu.org
Subject: Re: bug#34910: 27.0.50; Too much matching reentrancy
Date: Tue, 19 Mar 2019 14:31:39 +0200

> From: Mattias Engdegård <mattiase <at> acm.org>
> Cc: dancol <at> dancol.org, 34910 <at> debbugs.gnu.org
> Date: Tue, 19 Mar 2019 13:25:13 +0100
> 
> What happened was that a regexp failed because of stack overflow, which
> caused lots of code to run and many regexps to be compiled, but the
> original (failing) regexp was still marked busy in the cache.
> Eventually, it had been pushed down to the last position in the cache,
> where it caused the next request for a free cache entry to fail with
> the reentrancy error.
> 
> This patch fixes it. Please review.

Thanks, this looked reasonable.  Let's wait for Daniel to comment.

Message #23 received at 34910 <at> debbugs.gnu.org (full text, mbox):

From: Mattias Engdegård <mattiase <at> acm.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: dancol <at> dancol.org, 34910 <at> debbugs.gnu.org
Subject: Re: bug#34910: 27.0.50; Too much matching reentrancy
Date: Tue, 19 Mar 2019 13:50:00 +0100

[Message part 1 (text/plain, inline)]

Here is another improvement: there should be no reason to keep the
regexp busy during the overflow error processing, so release it first.

By the way, is there any reason why only calls to re_search_2 are made
with a busy-marked regexp buffer, not calls to re_search? Are the
latter guaranteed not to be reentrant?

[0001-Release-regexp-before-signalling-overflow-error.patch (text/x-patch, attachment)]

Message #28 received at 34910 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Mattias Engdegård <mattiase <at> acm.org>
Cc: dancol <at> dancol.org, 34910 <at> debbugs.gnu.org
Subject: Re: bug#34910: 27.0.50; Too much matching reentrancy
Date: Sat, 30 Mar 2019 12:29:45 +0300

> From: Mattias Engdegård <mattiase <at> acm.org>
> Cc: dancol <at> dancol.org, 34910 <at> debbugs.gnu.org
> Date: Tue, 19 Mar 2019 13:25:13 +0100
> 
> It turns out the regexp cache logic is broken: it always uses the last
> (LRU) entry, and fails if that entry is busy. It should really use the
> last free entry.
> 
> What happened was that a regexp failed because of stack overflow, which
> caused lots of code to run and many regexps to be compiled, but the
> original (failing) regexp was still marked busy in the cache.
> Eventually, it had been pushed down to the last position in the cache,
> where it caused the next request for a free cache entry to fail with
> the reentrancy error.
> 
> This patch fixes it. Please review.

Since there were no further comments, I think you should push this.

Thanks.

Message #31 received at 34910 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Mattias Engdegård <mattiase <at> acm.org>,
 Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: dancol <at> dancol.org, 34910 <at> debbugs.gnu.org
Subject: Re: bug#34910: 27.0.50; Too much matching reentrancy
Date: Sat, 30 Mar 2019 12:30:54 +0300

> From: Mattias Engdegård <mattiase <at> acm.org>
> Cc: dancol <at> dancol.org, 34910 <at> debbugs.gnu.org
> Date: Tue, 19 Mar 2019 13:50:00 +0100
> 
> Here is another improvement: there should be no reason to keep the
> regexp busy during the overflow error processing, so release it first.

Looks OK to me.

> By the way, is there any reason why only calls to re_search_2 are made
> with a busy-marked regexp buffer, not calls to re_search? Are the
> latter guaranteed not to be reentrant?

Not sure.  Stefan, any thoughts?

Message #34 received at 34910 <at> debbugs.gnu.org (full text, mbox):

From: Mattias Engdegård <mattiase <at> acm.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: dancol <at> dancol.org, Stefan Monnier <monnier <at> iro.umontreal.ca>,
 34910 <at> debbugs.gnu.org
Subject: Re: bug#34910: 27.0.50; Too much matching reentrancy
Date: Sat, 30 Mar 2019 12:18:31 +0100

30 mars 2019 kl. 10.30 skrev Eli Zaretskii <eliz <at> gnu.org>:
> 
>> Here is another improvement: there should be no reason to keep the
>> regexp busy during the overflow error processing, so release it first.
> 
> Looks OK to me.

Thank you, both pushed.

>> By the way, is there any reason why only calls to re_search_2 are made
>> with a busy-marked regexp buffer, not calls to re_search? Are the
>> latter guaranteed not to be reentrant?
> 
> Not sure.  Stefan, any thoughts?

Maybe Daniel only had reentrancy concerns about regexp matching on buffers, not on strings?

Message #37 received at 34910 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> IRO.UMontreal.CA>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: Mattias Engdegård <mattiase <at> acm.org>,
 dancol <at> dancol.org, 34910 <at> debbugs.gnu.org
Subject: Re: bug#34910: 27.0.50; Too much matching reentrancy
Date: Sat, 30 Mar 2019 08:55:31 -0400

>> By the way, is there any reason why only calls to re_search_2 are made
>> with a busy-marked regexp buffer, not calls to re_search? Are the
>> latter guaranteed not to be reentrant?
> Not sure.  Stefan, any thoughts?

It was probably just an oversight, because whether we're searching
within strings or within buffers the same problems appear with \{N\}
repetitions (where the counter is stored directly inside the
regexp-byte-code) and with on-the-fly strength-reduction (where we
replace some byte-code with cheaper ones (where applicable) on the first
pass through the regexp).


        Stefan

Message #40 received at 34910 <at> debbugs.gnu.org (full text, mbox):

From: Mattias Engdegård <mattiase <at> acm.org>
To: Stefan Monnier <monnier <at> IRO.UMontreal.CA>
Cc: Eli Zaretskii <eliz <at> gnu.org>, dancol <at> dancol.org, 34910 <at> debbugs.gnu.org
Subject: Re: bug#34910: 27.0.50; Too much matching reentrancy
Date: Sat, 30 Mar 2019 14:05:50 +0100

30 mars 2019 kl. 13.55 skrev Stefan Monnier <monnier <at> IRO.UMontreal.CA>:
> 
> It was probably just an oversight, because whether we're searching
> within strings or within buffers the same problems appear with \{N\}
> repetitions (where the counter is stored directly inside the
> regexp-byte-code) and with on-the-fly strength-reduction (where we
> replace some byte-code with cheaper ones (where applicable) on the first
> pass through the regexp).

Is the reentrancy just in preparation for future needs, or is it possible to re-enter the regexp code right now?

If the latter, we may just as well fix the string case here and now; otherwise, I'd like to close the bug.

Message #43 received at 34910 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Mattias Engdegård <mattiase <at> acm.org>
Cc: dancol <at> dancol.org, monnier <at> IRO.UMontreal.CA, 34910 <at> debbugs.gnu.org
Subject: Re: bug#34910: 27.0.50; Too much matching reentrancy
Date: Sat, 30 Mar 2019 16:37:03 +0300

> From: Mattias Engdegård <mattiase <at> acm.org>
> Date: Sat, 30 Mar 2019 14:05:50 +0100
> Cc: Eli Zaretskii <eliz <at> gnu.org>, dancol <at> dancol.org, 34910 <at> debbugs.gnu.org
> 
> 30 mars 2019 kl. 13.55 skrev Stefan Monnier <monnier <at> IRO.UMontreal.CA>:
> > 
> > It was probably just an oversight, because whether we're searching
> > within strings or within buffers the same problems appear with \{N\}
> > repetitions (where the counter is stored directly inside the
> > regexp-byte-code) and with on-the-fly strength-reduction (where we
> > replace some byte-code with cheaper ones (where applicable) on the first
> > pass through the regexp).
> 
> Is the reentrancy just in preparation for future needs, or is it possible to re-enter the regexp code right now?

I think the former, since reentrance could only happen if a Lisp
thread yields while inside a regexp function, which I think is not
possible, since we never call 'select' or any Lisp from there, do we?

Message #46 received at 34910 <at> debbugs.gnu.org (full text, mbox):

From: Daniel Colascione <dancol <at> dancol.org>
To: Mattias Engdegård <mattiase <at> acm.org>,
 Eli Zaretskii <eliz <at> gnu.org>
Cc: Stefan Monnier <monnier <at> iro.umontreal.ca>, 34910 <at> debbugs.gnu.org
Subject: Re: bug#34910: 27.0.50; Too much matching reentrancy
Date: Sat, 30 Mar 2019 07:52:12 -0700

On 3/30/19 4:18 AM, Mattias Engdegård wrote:
> 30 mars 2019 kl. 10.30 skrev Eli Zaretskii <eliz <at> gnu.org>:
>>
>>> Here is another improvement: there should be no reason to keep the
>>> regexp busy during the overflow error processing, so release it first.
>>
>> Looks OK to me.
> 
> Thank you, both pushed.

Thanks!

> 
>>> By the way, is there any reason why only calls to re_search_2 are made
>>> with a busy-marked regexp buffer, not calls to re_search? Are the
>>> latter guaranteed not to be reentrant?
>>
>> Not sure.  Stefan, any thoughts?
> 
> Maybe Daniel only had reentrancy concerns about regexp matching on buffers, not on strings?

We do automatic syntax discovery on buffers, not strings.

Message #49 received at 34910 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Mattias Engdegård <mattiase <at> acm.org>
Cc: Eli Zaretskii <eliz <at> gnu.org>, dancol <at> dancol.org, 34910 <at> debbugs.gnu.org
Subject: Re: bug#34910: 27.0.50; Too much matching reentrancy
Date: Sat, 30 Mar 2019 12:22:09 -0400

> Is the reentrancy just in preparation for future needs, or is it possible to
> re-enter the regexp code right now?

IIRC we do when the regexp matcher needs to call syntax-propertize.


        Stefan

Message #52 received at 34910 <at> debbugs.gnu.org (full text, mbox):

From: "Daniel Colascione" <dancol <at> dancol.org>
To: "Stefan Monnier" <monnier <at> iro.umontreal.ca>
Cc: "Mattias Engdegård" <mattiase <at> acm.org>,
 Eli Zaretskii <eliz <at> gnu.org>, dancol <at> dancol.org, 34910 <at> debbugs.gnu.org
Subject: Re: bug#34910: 27.0.50; Too much matching reentrancy
Date: Sat, 30 Mar 2019 10:10:37 -0700

>> Is the reentrancy just in preparation for future needs, or is it
>> possible to
>> re-enter the regexp code right now?
>
> IIRC we do when the regexp matcher needs to call syntax-propertize.

Right. That was the original motivation for the patch.

Message #57 received at 34910-done <at> debbugs.gnu.org (full text, mbox):

From: Mattias Engdegård <mattiase <at> acm.org>
To: Daniel Colascione <dancol <at> dancol.org>
Cc: Eli Zaretskii <eliz <at> gnu.org>, Stefan Monnier <monnier <at> iro.umontreal.ca>,
 34910-done <at> debbugs.gnu.org
Subject: Re: bug#34910: 27.0.50; Too much matching reentrancy
Date: Sat, 30 Mar 2019 20:06:39 +0100

30 mars 2019 kl. 18.10 skrev Daniel Colascione <dancol <at> dancol.org>:
> 
>>> Is the reentrancy just in preparation for future needs, or is it
>>> possible to
>>> re-enter the regexp code right now?
>> 
>> IIRC we do when the regexp matcher needs to call syntax-propertize.
> 
> Right. That was the original motivation for the patch.

Thank you for the explanation.
There is no shortage of work to be done in the regexp code but it looks like this bug can be closed.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.