GNU bug report logs - #36328
26.2; Args out of range on search-and-replace of *.cc file

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 36328 in the body.
You can then email your comments to 36328 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Jayden Navarro <jayden <at> yugabyte.com>
To: bug-gnu-emacs <at> gnu.org
Subject: 26.2; Args out of range on search-and-replace of *.cc file
Date: Fri, 21 Jun 2019 16:03:43 -0700

[Message part 1 (text/plain, inline)]

This bug report will be sent to the Bug-GNU-Emacs mailing list
and the GNU bug tracker at debbugs.gnu.org.  Please check that
the From: line contains a valid email address.  After a delay of up
to one day, you should receive an acknowledgment at that address.

Please write in English if possible, as the Emacs maintainers
usually do not have translators for other languages.

Please describe exactly what actions triggered the bug, and
the precise symptoms of the bug.  If you can, give a recipe
starting from 'emacs -Q':

Fill a *.cc file with 100 lines of any string (e.g. "bar"). At line 101
write a unique string (e.g. "foo"). Close the file and reopen (emacs -Q)
and perform search-and-replace on "foo" (replacing with any other string).
At this point you should see "Args out of range: #<buffer test.cc>, 0, 1".

If Emacs crashed, and you have the Emacs process in the gdb debugger,
please include the output from the following gdb commands:
    'bt full' and 'xbacktrace'.
For information about debugging Emacs, please read the file
/usr/local/Cellar/emacs/26.2/share/emacs/26.2/etc/DEBUG.

In GNU Emacs 26.2 (build 1, x86_64-apple-darwin18.5.0)
 of 2019-04-13 built on Mojave-2.local
Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.

Configured using:
 'configure --disable-dependency-tracking --disable-silent-rules
 --enable-locallisppath=/usr/local/share/emacs/site-lisp
 --infodir=/usr/local/Cellar/emacs/26.2/share/info/emacs
 --prefix=/usr/local/Cellar/emacs/26.2 --with-gnutls --without-x
 --with-xml2 --without-dbus --with-modules --without-ns
 --without-imagemagick'

Configured features:
NOTIFY ACL GNUTLS LIBXML2 ZLIB MODULES THREADS

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: C++//l

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t
  abbrev-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rmc puny seq byte-opt gv
bytecomp byte-compile cconv dired dired-loaddefs format-spec rfc822 mml
mml-sec password-cache epa derived epg epg-config gnus-util rmail
tool-bar rmail-loaddefs mm-decode mm-bodies mm-encode mail-parse rfc2231
mailabbrev gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums
mm-util mail-prsvr mail-utils cc-mode cc-fonts easymenu cc-guess
cc-menus cc-cmds cc-styles cc-align cc-engine cc-vars cc-defs regexp-opt
cl-loaddefs cl-lib term/xterm xterm time-date elec-pair mule-util
tooltip eldoc electric uniquify ediff-hook vc-hooks lisp-float-type
tabulated-list replace newcomment text-mode elisp-mode lisp-mode
prog-mode register page menu-bar rfn-eshadow isearch timer select mouse
jit-lock font-lock syntax facemenu font-core term/tty-colors frame
cl-generic cham georgian utf-8-lang misc-lang vietnamese tibetan thai
tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek romanian
slovak czech european ethiopic indian cyrillic chinese composite
charscript charprop case-table epa-hook jka-cmpr-hook help simple abbrev
obarray minibuffer cl-preloaded nadvice loaddefs button faces cus-face
macroexp files text-properties overlay sha1 md5 base64 format env
code-pages mule custom widget hashtable-print-readable backquote threads
kqueue multi-tty make-network-process emacs)

Memory information:
((conses 16 114832 6835)
 (symbols 48 21718 1)
 (miscs 40 36 157)
 (strings 32 33556 1381)
 (string-bytes 1 1027563)
 (vectors 16 14526)
 (vector-slots 8 479822 7914)
 (floats 8 49 415)
 (intervals 56 205 0)
 (buffers 992 12))

[Message part 2 (text/html, inline)]

Message #8 received at 36328 <at> debbugs.gnu.org (full text, mbox):

From: Alan Mackenzie <acm <at> muc.de>
To: Jayden Navarro <jayden <at> yugabyte.com>
Cc: 36328 <at> debbugs.gnu.org
Subject: Re: bug#36328: 26.2;
 Args out of range on search-and-replace of *.cc file
Date: 22 Jun 2019 13:25:49 -0000

Hello, Jayden.

In article <mailman.612.1561158667.10840.bug-gnu-emacs <at> gnu.org> you wrote:
> [-- text/plain, encoding 7bit, charset: UTF-8, 97 lines --]

[ .... ]

> Fill a *.cc file with 100 lines of any string (e.g. "bar"). At line 101
> write a unique string (e.g. "foo").

Does your file look like:

bar
bar
bar
...
...
bar
foo

?  Or does it look like:

"bar"
"bar"
"bar"
...
...
"bar"
"foo"

?

> Close the file and reopen (emacs -Q).

OK.

> and perform search-and-replace on "foo" (replacing with any other string).

What, precisely, did you do to attempt this "search-and-replace"?  With
emacs-26.2, after emacs -Q, I did

    C-x C-f test.cc

, followed by

    M-% foo <CR> FOO

, in both possibilities for the file (see above), yet could not
reproduce the error.  The replacement command simply worked for me.

> At this point you should see "Args out of range: #<buffer test.cc>, 0, 1".

This is what I don't see.  :-(  At least, not yet.

[ .... ]

-- 
Alan Mackenzie (Nuremberg, Germany).

Message #11 received at 36328 <at> debbugs.gnu.org (full text, mbox):

From: Jayden Navarro <jayden <at> yugabyte.com>
To: Alan Mackenzie <acm <at> muc.de>
Cc: 36328 <at> debbugs.gnu.org
Subject: Re: bug#36328: 26.2;
 Args out of range on search-and-replace of *.cc file
Date: Sat, 22 Jun 2019 07:25:30 -0700

[Message part 1 (text/plain, inline)]

Hello Alan,

Thank you for your response. Apologies for the ambiguous steps. Please find
more detailed information below:

Here are the steps:

1. Open a file in c++-mode (e.g. emacs -Q test.cc).

2. Add 100 lines of some string (e.g. the word "bar" on every line for 100
lines, no quotes in the actual file):

bar
bar
bar
bar
...
bar

3. Add a unique string to line 101 (e.g. the word "foo", no quotes in the
actual file).

bar
bar
bar
bar
...
bar
foo
<INCLUDE NEWLINE AT END OF FILE>

4. Close Emacs

5. Open up the file again: emacs -Q test.cc

6. Replace the unique string with some other string: M-x query-replace
<RET> foo <RET> bar <RET>

7. You should hit: Args out of range: #<buffer test.cc>, 0, 1

Here's the backtrace when using debug-on-error:

Debugger entered--Lisp error: (args-out-of-range #<buffer test.cc> 0 1)
  buffer-substring-no-properties(0 1)
  perform-replace("foo" "a" t nil nil nil nil nil nil nil nil)
  query-replace("foo" "a" nil nil nil nil nil)
  funcall-interactively(query-replace "foo" "a" nil nil nil nil nil)
  call-interactively(query-replace nil nil)
  command-execute(query-replace)

I also wanted to add that I tried reproducing with a file that looked like
the following, but did NOT see the issue then,
either with replacing "foo" (with quotes) or just foo (no quotes):

"bar"
"bar"
"bar"
...
"bar"
"foo"

Even though I'm using "emacs -Q", could it still be interacting with some
of the packages I have installed?

Here's the list of packages I have installed under $HOME/.emacs.d/elpa:

avy-0.3.0
company-20181105.2312
company-lean-20171102.1454
dash-20180910.1856
dash-functional-20180107.1618
epl-20180205.2049
f-20180106.922
flycheck-20181127.1510
gnupg
go-mode-1.3.1
haskell-mode-13.16
lean-mode-20180906.1645
pkg-info-20150517.1143
rust-mode-20181008.1628
s-20180406.808

Best,
Jayden


On Sat, Jun 22, 2019 at 6:25 AM Alan Mackenzie <acm <at> muc.de> wrote:

> Hello, Jayden.
>
> In article <mailman.612.1561158667.10840.bug-gnu-emacs <at> gnu.org> you wrote:
> > [-- text/plain, encoding 7bit, charset: UTF-8, 97 lines --]
>
> [ .... ]
>
> > Fill a *.cc file with 100 lines of any string (e.g. "bar"). At line 101
> > write a unique string (e.g. "foo").
>
> Does your file look like:
>
> bar
> bar
> bar
> ...
> ...
> bar
> foo
>
> ?  Or does it look like:
>
> "bar"
> "bar"
> "bar"
> ...
> ...
> "bar"
> "foo"
>
> ?
>
> > Close the file and reopen (emacs -Q).
>
> OK.
>
> > and perform search-and-replace on "foo" (replacing with any other
> string).
>
> What, precisely, did you do to attempt this "search-and-replace"?  With
> emacs-26.2, after emacs -Q, I did
>
>     C-x C-f test.cc
>
> , followed by
>
>     M-% foo <CR> FOO
>
> , in both possibilities for the file (see above), yet could not
> reproduce the error.  The replacement command simply worked for me.
>
> > At this point you should see "Args out of range: #<buffer test.cc>, 0,
> 1".
>
> This is what I don't see.  :-(  At least, not yet.
>
> [ .... ]
>
> --
> Alan Mackenzie (Nuremberg, Germany).
>
>

[Message part 2 (text/html, inline)]

Message #14 received at 36328 <at> debbugs.gnu.org (full text, mbox):

From: Juanma Barranquero <lekktu <at> gmail.com>
To: Jayden Navarro <jayden <at> yugabyte.com>
Cc: Alan Mackenzie <acm <at> muc.de>, 36328 <at> debbugs.gnu.org
Subject: Re: bug#36328: 26.2;
 Args out of range on search-and-replace of *.cc file
Date: Sat, 22 Jun 2019 16:51:41 +0200

On Sat, Jun 22, 2019 at 4:26 PM Jayden Navarro <jayden <at> yugabyte.com> wrote:

> 7. You should hit: Args out of range: #<buffer test.cc>, 0, 1

I cannot reproduce it with 26.2.90 on Windows.

Message #17 received at 36328 <at> debbugs.gnu.org (full text, mbox):

From: Jayden Navarro <jayden <at> yugabyte.com>
To: Juanma Barranquero <lekktu <at> gmail.com>
Cc: Alan Mackenzie <acm <at> muc.de>, 36328 <at> debbugs.gnu.org
Subject: Re: bug#36328: 26.2;
 Args out of range on search-and-replace of *.cc file
Date: Sat, 22 Jun 2019 09:09:48 -0700

[Message part 1 (text/plain, inline)]

I've been able to reproduce this on MacOS 10.14 Mojave running Emacs 26.2
and Emacs 27 (forget exact version, installed it yesterday using "brew
install emacs --HEAD").

I've also reproduced it on Windows in Cygwin running Emacs 26.2. On an
Ubuntu server I use running Emacs 24.5.1 I cannot reproduce the issue.

"emacs -Q" should ignore all init files and packages correct? Is there
anything else on my system that could be affecting this?

Note that for the same file contents this issue does not occur if I name
the file test.py, so I believe it's related to c++-mode.

Best,
Jayden


On Sat, Jun 22, 2019 at 7:52 AM Juanma Barranquero <lekktu <at> gmail.com> wrote:

> On Sat, Jun 22, 2019 at 4:26 PM Jayden Navarro <jayden <at> yugabyte.com>
> wrote:
>
> > 7. You should hit: Args out of range: #<buffer test.cc>, 0, 1
>
> I cannot reproduce it with 26.2.90 on Windows.
>

[Message part 2 (text/html, inline)]

Message #20 received at 36328 <at> debbugs.gnu.org (full text, mbox):

From: Alan Mackenzie <acm <at> muc.de>
To: Jayden Navarro <jayden <at> yugabyte.com>
Cc: 36328 <at> debbugs.gnu.org
Subject: Re: bug#36328: 26.2; Args out of range on search-and-replace of *.cc
 file
Date: Sat, 22 Jun 2019 20:50:33 +0000

Hello again, Jayden.

On Sat, Jun 22, 2019 at 07:25:30 -0700, Jayden Navarro wrote:
> Hello Alan,

> Thank you for your response. Apologies for the ambiguous steps. Please find
> more detailed information below:

Thanks!

> Here are the steps:

> 1. Open a file in c++-mode (e.g. emacs -Q test.cc).

> 2. Add 100 lines of some string (e.g. the word "bar" on every line for 100
> lines, no quotes in the actual file):

> bar
> bar
> bar
> bar
> ...
> bar

> 3. Add a unique string to line 101 (e.g. the word "foo", no quotes in the
> actual file).

> bar
> bar
> bar
> bar
> ...
> bar
> foo
> <INCLUDE NEWLINE AT END OF FILE>

> 4. Close Emacs

> 5. Open up the file again: emacs -Q test.cc

> 6. Replace the unique string with some other string: M-x query-replace
> <RET> foo <RET> bar <RET>

Are you _sure_ that's what you typed?  ;-)

> 7. You should hit: Args out of range: #<buffer test.cc>, 0, 1

> Here's the backtrace when using debug-on-error:

> Debugger entered--Lisp error: (args-out-of-range #<buffer test.cc> 0 1)
>   buffer-substring-no-properties(0 1)  <==============================
>   perform-replace("foo" "a" t nil nil nil nil nil nil nil nil)
>   query-replace("foo" "a" nil nil nil nil nil)
>   funcall-interactively(query-replace "foo" "a" nil nil nil nil nil)
>   call-interactively(query-replace nil nil)
>   command-execute(query-replace)

There, it looks like you are trying to replace "foo" by "a".  I'm
interested in the (invalid) arguments 0, 1 passed to
buffer-substring-no-properties.  I suspect that these are derived from
the "match-data" for a string, in particular for the string "a".

Could you please repeat the bug scenario, but this time try to replace
"foo" by "bar".  I predict you will then get the error message

    (args-out-of-range #<buffer test.cc> 0 3)

since the replacement string will then be 3 characters long.

If that does indeed happen, it would be a very strong clue as to the
underlying bug.  Please try it as above, and post the backtrace here.
Thanks!

[ .... ]

> Here's the list of packages I have installed under $HOME/.emacs.d/elpa:

> avy-0.3.0
> company-20181105.2312
> company-lean-20171102.1454
> dash-20180910.1856
> dash-functional-20180107.1618
> epl-20180205.2049
> f-20180106.922
> flycheck-20181127.1510
> gnupg
> go-mode-1.3.1
> haskell-mode-13.16
> lean-mode-20180906.1645
> pkg-info-20150517.1143
> rust-mode-20181008.1628
> s-20180406.808

I think, I hope very strongly, that the -Q in emacs -Q will prevent any
packages being loaded.  Otherwise we have a problem in the Emacs core.

> Best,
> Jayden

-- 
Alan Mackenzie (Nuremberg, Germany).

Message #23 received at 36328 <at> debbugs.gnu.org (full text, mbox):

From: Jayden Navarro <jayden <at> yugabyte.com>
To: Alan Mackenzie <acm <at> muc.de>
Cc: 36328 <at> debbugs.gnu.org
Subject: Re: bug#36328: 26.2;
 Args out of range on search-and-replace of *.cc file
Date: Sat, 22 Jun 2019 14:27:00 -0700

[Message part 1 (text/plain, inline)]

Hi Alan,

Yes, you are completely correct :-) Posted one thing but unintentionally
took a "shortcut" when producing the backtrace (not a smart thing to do
when trying to get people to repro...).

Here's the backtrace when doing doing M-% foo <RET> bar <RET> (note that
it's still 0, 1 not 0, 3!):

Debugger entered--Lisp error: (args-out-of-range #<buffer test.cc> 0 1)
  buffer-substring-no-properties(0 1)
  perform-replace("foo" "bar" t nil nil nil nil nil nil nil nil)
  query-replace("foo" "bar" nil nil nil nil nil)
  funcall-interactively(query-replace "foo" "bar" nil nil nil nil nil)
  call-interactively(query-replace nil nil)
  command-execute(query-replace)

Thank you for your help!

Best,
Jayden


On Sat, Jun 22, 2019 at 1:50 PM Alan Mackenzie <acm <at> muc.de> wrote:

> Hello again, Jayden.
>
> On Sat, Jun 22, 2019 at 07:25:30 -0700, Jayden Navarro wrote:
> > Hello Alan,
>
> > Thank you for your response. Apologies for the ambiguous steps. Please
> find
> > more detailed information below:
>
> Thanks!
>
> > Here are the steps:
>
> > 1. Open a file in c++-mode (e.g. emacs -Q test.cc).
>
> > 2. Add 100 lines of some string (e.g. the word "bar" on every line for
> 100
> > lines, no quotes in the actual file):
>
> > bar
> > bar
> > bar
> > bar
> > ...
> > bar
>
> > 3. Add a unique string to line 101 (e.g. the word "foo", no quotes in the
> > actual file).
>
> > bar
> > bar
> > bar
> > bar
> > ...
> > bar
> > foo
> > <INCLUDE NEWLINE AT END OF FILE>
>
> > 4. Close Emacs
>
> > 5. Open up the file again: emacs -Q test.cc
>
> > 6. Replace the unique string with some other string: M-x query-replace
> > <RET> foo <RET> bar <RET>
>
> Are you _sure_ that's what you typed?  ;-)
>
> > 7. You should hit: Args out of range: #<buffer test.cc>, 0, 1
>
> > Here's the backtrace when using debug-on-error:
>
> > Debugger entered--Lisp error: (args-out-of-range #<buffer test.cc> 0 1)
> >   buffer-substring-no-properties(0 1)  <==============================
> >   perform-replace("foo" "a" t nil nil nil nil nil nil nil nil)
> >   query-replace("foo" "a" nil nil nil nil nil)
> >   funcall-interactively(query-replace "foo" "a" nil nil nil nil nil)
> >   call-interactively(query-replace nil nil)
> >   command-execute(query-replace)
>
> There, it looks like you are trying to replace "foo" by "a".  I'm
> interested in the (invalid) arguments 0, 1 passed to
> buffer-substring-no-properties.  I suspect that these are derived from
> the "match-data" for a string, in particular for the string "a".
>
> Could you please repeat the bug scenario, but this time try to replace
> "foo" by "bar".  I predict you will then get the error message
>
>     (args-out-of-range #<buffer test.cc> 0 3)
>
> since the replacement string will then be 3 characters long.
>
> If that does indeed happen, it would be a very strong clue as to the
> underlying bug.  Please try it as above, and post the backtrace here.
> Thanks!
>
> [ .... ]
>
> > Here's the list of packages I have installed under $HOME/.emacs.d/elpa:
>
> > avy-0.3.0
> > company-20181105.2312
> > company-lean-20171102.1454
> > dash-20180910.1856
> > dash-functional-20180107.1618
> > epl-20180205.2049
> > f-20180106.922
> > flycheck-20181127.1510
> > gnupg
> > go-mode-1.3.1
> > haskell-mode-13.16
> > lean-mode-20180906.1645
> > pkg-info-20150517.1143
> > rust-mode-20181008.1628
> > s-20180406.808
>
> I think, I hope very strongly, that the -Q in emacs -Q will prevent any
> packages being loaded.  Otherwise we have a problem in the Emacs core.
>
> > Best,
> > Jayden
>
> --
> Alan Mackenzie (Nuremberg, Germany).
>

[Message part 2 (text/html, inline)]

Message #26 received at 36328 <at> debbugs.gnu.org (full text, mbox):

From: Jayden Navarro <jayden <at> yugabyte.com>
To: Alan Mackenzie <acm <at> muc.de>
Cc: 36328 <at> debbugs.gnu.org
Subject: Re: bug#36328: 26.2;
 Args out of range on search-and-replace of *.cc file
Date: Sat, 22 Jun 2019 15:38:25 -0700

[Message part 1 (text/plain, inline)]

I've figured out the trigger! I setup a CentOS 7 VM and installed Emacs
26.2.90 and couldn't repro on it. So I compared all the environment
variables between Cygwin and the VM and performed some tests, and
discovered the issue is triggered by my "TERM" environment variable being
set to "xterm-256color".

I could reproduce on the clean VM using: TERM="xterm-256color" emacs -Q
test.cc

Launching Emacs in that way, and following the other steps given above,
should allow you to reproduce.

Best,
Jayden


On Sat, Jun 22, 2019 at 2:27 PM Jayden Navarro <jayden <at> yugabyte.com> wrote:

> Hi Alan,
>
> Yes, you are completely correct :-) Posted one thing but unintentionally
> took a "shortcut" when producing the backtrace (not a smart thing to do
> when trying to get people to repro...).
>
> Here's the backtrace when doing doing M-% foo <RET> bar <RET> (note that
> it's still 0, 1 not 0, 3!):
>
> Debugger entered--Lisp error: (args-out-of-range #<buffer test.cc> 0 1)
>   buffer-substring-no-properties(0 1)
>   perform-replace("foo" "bar" t nil nil nil nil nil nil nil nil)
>   query-replace("foo" "bar" nil nil nil nil nil)
>   funcall-interactively(query-replace "foo" "bar" nil nil nil nil nil)
>   call-interactively(query-replace nil nil)
>   command-execute(query-replace)
>
> Thank you for your help!
>
> Best,
> Jayden
>
>
> On Sat, Jun 22, 2019 at 1:50 PM Alan Mackenzie <acm <at> muc.de> wrote:
>
>> Hello again, Jayden.
>>
>> On Sat, Jun 22, 2019 at 07:25:30 -0700, Jayden Navarro wrote:
>> > Hello Alan,
>>
>> > Thank you for your response. Apologies for the ambiguous steps. Please
>> find
>> > more detailed information below:
>>
>> Thanks!
>>
>> > Here are the steps:
>>
>> > 1. Open a file in c++-mode (e.g. emacs -Q test.cc).
>>
>> > 2. Add 100 lines of some string (e.g. the word "bar" on every line for
>> 100
>> > lines, no quotes in the actual file):
>>
>> > bar
>> > bar
>> > bar
>> > bar
>> > ...
>> > bar
>>
>> > 3. Add a unique string to line 101 (e.g. the word "foo", no quotes in
>> the
>> > actual file).
>>
>> > bar
>> > bar
>> > bar
>> > bar
>> > ...
>> > bar
>> > foo
>> > <INCLUDE NEWLINE AT END OF FILE>
>>
>> > 4. Close Emacs
>>
>> > 5. Open up the file again: emacs -Q test.cc
>>
>> > 6. Replace the unique string with some other string: M-x query-replace
>> > <RET> foo <RET> bar <RET>
>>
>> Are you _sure_ that's what you typed?  ;-)
>>
>> > 7. You should hit: Args out of range: #<buffer test.cc>, 0, 1
>>
>> > Here's the backtrace when using debug-on-error:
>>
>> > Debugger entered--Lisp error: (args-out-of-range #<buffer test.cc> 0 1)
>> >   buffer-substring-no-properties(0 1)  <==============================
>> >   perform-replace("foo" "a" t nil nil nil nil nil nil nil nil)
>> >   query-replace("foo" "a" nil nil nil nil nil)
>> >   funcall-interactively(query-replace "foo" "a" nil nil nil nil nil)
>> >   call-interactively(query-replace nil nil)
>> >   command-execute(query-replace)
>>
>> There, it looks like you are trying to replace "foo" by "a".  I'm
>> interested in the (invalid) arguments 0, 1 passed to
>> buffer-substring-no-properties.  I suspect that these are derived from
>> the "match-data" for a string, in particular for the string "a".
>>
>> Could you please repeat the bug scenario, but this time try to replace
>> "foo" by "bar".  I predict you will then get the error message
>>
>>     (args-out-of-range #<buffer test.cc> 0 3)
>>
>> since the replacement string will then be 3 characters long.
>>
>> If that does indeed happen, it would be a very strong clue as to the
>> underlying bug.  Please try it as above, and post the backtrace here.
>> Thanks!
>>
>> [ .... ]
>>
>> > Here's the list of packages I have installed under $HOME/.emacs.d/elpa:
>>
>> > avy-0.3.0
>> > company-20181105.2312
>> > company-lean-20171102.1454
>> > dash-20180910.1856
>> > dash-functional-20180107.1618
>> > epl-20180205.2049
>> > f-20180106.922
>> > flycheck-20181127.1510
>> > gnupg
>> > go-mode-1.3.1
>> > haskell-mode-13.16
>> > lean-mode-20180906.1645
>> > pkg-info-20150517.1143
>> > rust-mode-20181008.1628
>> > s-20180406.808
>>
>> I think, I hope very strongly, that the -Q in emacs -Q will prevent any
>> packages being loaded.  Otherwise we have a problem in the Emacs core.
>>
>> > Best,
>> > Jayden
>>
>> --
>> Alan Mackenzie (Nuremberg, Germany).
>>
>

[Message part 2 (text/html, inline)]

Message #29 received at 36328 <at> debbugs.gnu.org (full text, mbox):

From: Jayden Navarro <jayden <at> yugabyte.com>
To: Alan Mackenzie <acm <at> muc.de>
Cc: 36328 <at> debbugs.gnu.org
Subject: Re: bug#36328: 26.2;
 Args out of range on search-and-replace of *.cc file
Date: Sat, 22 Jun 2019 16:02:45 -0700

[Message part 1 (text/plain, inline)]

Just also wanted to add that I tried this with test.java (java-mode) and
reproduced the issue as well (whereas no issue when using test.py), so it
definitely seems CcMode related.

Also, I reproduced with "xterm-16color" and all the other
"xterm-[0-9]+color" terminal types, but no other terminal types I tried
reproduced the problem, including regular "xterm-color".

Best,
Jayden

[Message part 2 (text/html, inline)]

Message #32 received at 36328 <at> debbugs.gnu.org (full text, mbox):

From: Alan Mackenzie <acm <at> muc.de>
To: Jayden Navarro <jayden <at> yugabyte.com>
Cc: 36328 <at> debbugs.gnu.org
Subject: Re: bug#36328: 26.2; Args out of range on search-and-replace of *.cc
 file
Date: Sun, 23 Jun 2019 12:22:07 +0000

Hello, Jayden.

On Sat, Jun 22, 2019 at 15:38:25 -0700, Jayden Navarro wrote:
> I've figured out the trigger! I setup a CentOS 7 VM and installed Emacs
> 26.2.90 and couldn't repro on it. So I compared all the environment
> variables between Cygwin and the VM and performed some tests, and
> discovered the issue is triggered by my "TERM" environment variable being
> set to "xterm-256color".

Well done!

> I could reproduce on the clean VM using: TERM="xterm-256color" emacs -Q
> test.cc

In my X-Windows setup, an xterm has the setting of TERM anyway.  So I was
able to reproduce the failure with emacs -Q -nw.  -nw means "no windows",
i.e. run in the terminal, not as a GUI application.  (Normally, I run
Emacs in a linux virtual terminal.)

> Launching Emacs in that way, and following the other steps given above,
> should allow you to reproduce.

Yes.  It didn't take me too long to track down where it's going wrong.
`perform-replace' calls `replace-highlight' to highlight the "foo", and
this calls `isearch-lazy-highlight-new-loop'.  This in its turn performs
a redisplay (which is probably where CC Mode affects things).  It seems
too much to expect that the "match-data" will be preserved over all this
activity, but `perform-replace' does expect this.

To fix this will involve putting a `save-match-data' around some form
somewhere.  Please allow us some time to decide where this would be best.

> On Sat, Jun 22, 2019 at 2:27 PM Jayden Navarro <jayden <at> yugabyte.com> wrote:

> > Hi Alan,

> > Yes, you are completely correct :-) Posted one thing but unintentionally
> > took a "shortcut" when producing the backtrace (not a smart thing to do
> > when trying to get people to repro...).

> > Here's the backtrace when doing doing M-% foo <RET> bar <RET> (note that
> > it's still 0, 1 not 0, 3!):

I still have no specific idea what's generating that 0 and 1.  Will
probably never find out.

> > Debugger entered--Lisp error: (args-out-of-range #<buffer test.cc> 0 1)
> >   buffer-substring-no-properties(0 1)
> >   perform-replace("foo" "bar" t nil nil nil nil nil nil nil nil)
> >   query-replace("foo" "bar" nil nil nil nil nil)
> >   funcall-interactively(query-replace "foo" "bar" nil nil nil nil nil)
> >   call-interactively(query-replace nil nil)
> >   command-execute(query-replace)

> Best,
> Jayden

-- 
Alan Mackenzie (Nuremberg, Germany).

Message #35 received at 36328 <at> debbugs.gnu.org (full text, mbox):

From: Jayden Navarro <jayden <at> yugabyte.com>
To: Alan Mackenzie <acm <at> muc.de>
Cc: 36328 <at> debbugs.gnu.org
Subject: Re: bug#36328: 26.2;
 Args out of range on search-and-replace of *.cc file
Date: Sun, 23 Jun 2019 09:14:19 -0700

[Message part 1 (text/plain, inline)]

Hi Alan,

Thank you for looking into this!

Until this is officially fixed I've come up with the following workaround,
going off of the details you provided:

I created a "replace_fixed.el" file in "~/.emacs.d/lisp/" that is
replace.el taken from
https://raw.githubusercontent.com/emacs-mirror/emacs/emacs-26/lisp/replace.el
with
the following diff:

diff --git a/replace.el b/replace_fixed.el
index 08feb8e..8280fdd 100644
--- a/replace.el
+++ b/replace_fixed.el
@@ -2227,7 +2227,7 @@ It is called with three arguments, as if it were
            (isearch-forward (not backward))
            (isearch-other-end match-beg)
            (isearch-error nil))
-       (isearch-lazy-highlight-new-loop range-beg range-end))))
+       (save-match-data (isearch-lazy-highlight-new-loop range-beg
range-end)))))

 (defun replace-dehighlight ()
   (when replace-overlay

Then I added the following to my "~/.emacs", restarted my emacs server, and
the issue was gone!:

(load-library "~/.emacs.d/lisp/replace_fixed.el")

This probably isn't the proper fix, but just thought I'd share in case
anyone else is experiencing this and wanted a temporary workaround.

Best,
Jayden

[Message part 2 (text/html, inline)]

Message #38 received at 36328 <at> debbugs.gnu.org (full text, mbox):

From: Alan Mackenzie <acm <at> muc.de>
To: Jayden Navarro <jayden <at> yugabyte.com>
Cc: 36328 <at> debbugs.gnu.org
Subject: Re: bug#36328: 26.2; Args out of range on search-and-replace of *.cc
 file
Date: Sun, 23 Jun 2019 19:32:58 +0000

Hello, Jayden.

On Sun, Jun 23, 2019 at 09:14:19 -0700, Jayden Navarro wrote:
> Hi Alan,

> Thank you for looking into this!

> Until this is officially fixed I've come up with the following workaround,
> going off of the details you provided:

> I created a "replace_fixed.el" file in "~/.emacs.d/lisp/" that is
> replace.el taken from
> https://raw.githubusercontent.com/emacs-mirror/emacs/emacs-26/lisp/replace.el
> with
> the following diff:

> diff --git a/replace.el b/replace_fixed.el
> index 08feb8e..8280fdd 100644
> --- a/replace.el
> +++ b/replace_fixed.el
> @@ -2227,7 +2227,7 @@ It is called with three arguments, as if it were
>             (isearch-forward (not backward))
>             (isearch-other-end match-beg)
>             (isearch-error nil))
> -       (isearch-lazy-highlight-new-loop range-beg range-end))))
> +       (save-match-data (isearch-lazy-highlight-new-loop range-beg
> range-end)))))

>  (defun replace-dehighlight ()
>    (when replace-overlay

> Then I added the following to my "~/.emacs", restarted my emacs server, and
> the issue was gone!:

> (load-library "~/.emacs.d/lisp/replace_fixed.el")

> This probably isn't the proper fix, but just thought I'd share in case
> anyone else is experiencing this and wanted a temporary workaround.

Excellent!  To be honest, I was thinking of sending just that workaround
to you as a temporary patch, but it seems I didn't need to.  :-)

That might well be the fix we end up putting into Emacs, but it might
not be.  Sorry we're so slow, here.

Have a great afternoon!  I'll be off to bed, soon.

Thanks for taking the trouble to report this bug.

> Best,
> Jayden

-- 
Alan Mackenzie (Nuremberg, Germany).

Message #41 received at 36328 <at> debbugs.gnu.org (full text, mbox):

From: Alan Mackenzie <acm <at> muc.de>
To: Juri Linkov <juri <at> linkov.net>
Cc: Jayden Navarro <jayden <at> yugabyte.com>, 36328 <at> debbugs.gnu.org
Subject: [jayden <at> yugabyte.com: Re: bug#36328: 26.2; Args out of range on
 search-and-replace of *.cc file]
Date: Sun, 23 Jun 2019 20:10:48 +0000

Hello, Juri.

As the replace.el expert here, could you please have a look at bug
#36328, and in particular, comment on Jayden's patch (below), which I
think would be a good fix for the bug.

Is there anything we've missed, such as unforeseen and unwanted effects
somewhere else?

Thanks!

-- 
Alan Mackenzie (Nuremberg, Germany).


----- Forwarded message from Jayden Navarro <jayden <at> yugabyte.com> -----

X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on f9b3f715-3f29-11e8-b508-002264fbbacc
X-Spam-Level: 
X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,HTML_MESSAGE, RCVD_IN_DNSWL_NONE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.2
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;        d=yugabyte-com.20150623.gappssmtp.com; s=20150623;        h=mime-version:references:in-reply-to:from:date:message-id:subject:to         :cc;
	bh=eZM1FmYPfQWqIxfpYI+vG/ojcQEm248LbjE6PiQr8CE=;        b=2ELl2xbqAuUFiUbRfNinSQfbRbxhWgpTsLbHINR3oKe37wJPYbt4LoKqFc2wo7uEjo         79POKqduJ+PerwW+epBOAGP8zeqwAUNnKG7F/TMvPYixI8qh+oz6qQ7MuJfhtfc7ZVuw
	h1jiPQYSnJXkTMtZJZk3n+2RrKo4rRKbQQ4wrT+fq1ihZB8F3xVdbz3pGmLLmD0wwOzm         Qr+tYUtR9ix6yDSKq+Jr5JS/Rmh3kH8HDq67OpFwWYHSClPjGxzGTJQWEn5JqEoR6bL6         agHOiamb4rXFNvDdxyd70EaOQ5cNNHfRlpODk1suZdZpoCfX8FpYcdcepmixs/qx/PAF
	Fbhw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;        d=1e100.net; s=20161025;        h=x-gm-message-state:mime-version:references:in-reply-to:from:date         :message-id:subject:to:cc;
	bh=eZM1FmYPfQWqIxfpYI+vG/ojcQEm248LbjE6PiQr8CE=;        b=I3nt5jWknsKit8BXYashILtTd3kbV+DPx6pTHDgPZW6ZtzX87SzXiuwGZoczrJXsD3         cNss+BZVBJtZLFTsIC74LkWhtYh2OCdU3o5TG7TQ0SQqyQJ+FwWOfAMyBOVi4GvTJCLA
	Wg9rokIiwKUR6O+DvD0M+gpepHhc0NlW4iAeOrGaIb/jerwNticHF0MRdRSL2DKQ5nrH         fbMnO+5+nHEmIU2tUpd8EYqqHZQByXJfaKu7UKUJ/g1uCwbE5/SX401kslfM5ckl8xAC         5TJi08H4w3MKAGDzHJQrLdau7YoowFGZHvy1b0QiO7KjLDtRJ7HWVEu9TH3UUnILKg1r
	r90g==
X-Gm-Message-State: APjAAAW9vckX+zTLPd5fU5sesgvGsywasM/voob85fK5EyMnJsiEqvlv +R4mtzsBALaEP8quUhVW6AZS2FgQuzeq+SsPVGjXoAsw9sg=
X-Google-Smtp-Source: APXvYqxMxLxnGbgDBDwO7TQTe8IgNSSsQVMZ9S4j4kjCIGq7xdMYxG9J7kbLnxz35VR1RawX6tIm+0sJqHRqhftmU/c=
X-Received: by 2002:a2e:3913:: with SMTP id g19mr15782122lja.212.1561306471058; Sun, 23 Jun 2019 09:14:31 -0700 (PDT)
From: Jayden Navarro <jayden <at> yugabyte.com>
Date: Sun, 23 Jun 2019 09:14:19 -0700
Subject: Re: bug#36328: 26.2; Args out of range on search-and-replace of *.cc file
To: Alan Mackenzie <acm <at> muc.de>
Cc: 36328 <at> debbugs.gnu.org

Hi Alan,

Thank you for looking into this!

Until this is officially fixed I've come up with the following workaround,
going off of the details you provided:

I created a "replace_fixed.el" file in "~/.emacs.d/lisp/" that is
replace.el taken from
https://raw.githubusercontent.com/emacs-mirror/emacs/emacs-26/lisp/replace.el
with
the following diff:

diff --git a/replace.el b/replace_fixed.el
index 08feb8e..8280fdd 100644
--- a/replace.el
+++ b/replace_fixed.el
@@ -2227,7 +2227,7 @@ It is called with three arguments, as if it were
            (isearch-forward (not backward))
            (isearch-other-end match-beg)
            (isearch-error nil))
-       (isearch-lazy-highlight-new-loop range-beg range-end))))
+       (save-match-data (isearch-lazy-highlight-new-loop range-beg
range-end)))))

 (defun replace-dehighlight ()
   (when replace-overlay

Then I added the following to my "~/.emacs", restarted my emacs server, and
the issue was gone!:

(load-library "~/.emacs.d/lisp/replace_fixed.el")

This probably isn't the proper fix, but just thought I'd share in case
anyone else is experiencing this and wanted a temporary workaround.

Best,
Jayden

----- End forwarded message -----

Message #44 received at 36328 <at> debbugs.gnu.org (full text, mbox):

From: Juri Linkov <juri <at> linkov.net>
To: Alan Mackenzie <acm <at> muc.de>
Cc: Jayden Navarro <jayden <at> yugabyte.com>, 36328 <at> debbugs.gnu.org
Subject: Re: bug#36328: 26.2;
 Args out of range on search-and-replace of *.cc file
Date: Mon, 24 Jun 2019 00:19:22 +0300

>> Until this is officially fixed I've come up with the following workaround,
>> going off of the details you provided:
>
>> I created a "replace_fixed.el" file in "~/.emacs.d/lisp/" that is
>> replace.el taken from
>> https://raw.githubusercontent.com/emacs-mirror/emacs/emacs-26/lisp/replace.el
>> with
>> the following diff:
>
>> diff --git a/replace.el b/replace_fixed.el
>> index 08feb8e..8280fdd 100644
>> --- a/replace.el
>> +++ b/replace_fixed.el
>> @@ -2227,7 +2227,7 @@ It is called with three arguments, as if it were
>>             (isearch-forward (not backward))
>>             (isearch-other-end match-beg)
>>             (isearch-error nil))
>> -       (isearch-lazy-highlight-new-loop range-beg range-end))))
>> +       (save-match-data (isearch-lazy-highlight-new-loop range-beg
>> range-end)))))
>
>>  (defun replace-dehighlight ()
>>    (when replace-overlay
>
>> Then I added the following to my "~/.emacs", restarted my emacs server, and
>> the issue was gone!:
>
>> (load-library "~/.emacs.d/lisp/replace_fixed.el")
>
>> This probably isn't the proper fix, but just thought I'd share in case
>> anyone else is experiencing this and wanted a temporary workaround.
>
> Excellent!  To be honest, I was thinking of sending just that workaround
> to you as a temporary patch, but it seems I didn't need to.  :-)
>
> That might well be the fix we end up putting into Emacs, but it might
> not be.  Sorry we're so slow, here.

I think first we should try to narrow down the source of this match data leak.
Then we could decide what is the best solution.  Currently I see no such place
in isearch-lazy-highlight-new-loop that calls external code.  OTOH, while
looking at CC-Mode I noticed that it does some matches on before-change hooks.
I could debug it but can't reproduce neither in 26.1 nor in 27, even with -Q -nw.

Message #47 received at 36328 <at> debbugs.gnu.org (full text, mbox):

From: Jayden Navarro <jayden <at> yugabyte.com>
To: Juri Linkov <juri <at> linkov.net>
Cc: Alan Mackenzie <acm <at> muc.de>, 36328 <at> debbugs.gnu.org
Subject: Re: bug#36328: 26.2;
 Args out of range on search-and-replace of *.cc file
Date: Sun, 23 Jun 2019 14:42:48 -0700

[Message part 1 (text/plain, inline)]

Hi Juri,

Did you open it with TERM="xterm-256color"?

Best,
Jayden


On Sun, Jun 23, 2019 at 2:41 PM Juri Linkov <juri <at> linkov.net> wrote:

> >> Until this is officially fixed I've come up with the following
> workaround,
> >> going off of the details you provided:
> >
> >> I created a "replace_fixed.el" file in "~/.emacs.d/lisp/" that is
> >> replace.el taken from
> >>
> https://raw.githubusercontent.com/emacs-mirror/emacs/emacs-26/lisp/replace.el
> >> with
> >> the following diff:
> >
> >> diff --git a/replace.el b/replace_fixed.el
> >> index 08feb8e..8280fdd 100644
> >> --- a/replace.el
> >> +++ b/replace_fixed.el
> >> @@ -2227,7 +2227,7 @@ It is called with three arguments, as if it were
> >>             (isearch-forward (not backward))
> >>             (isearch-other-end match-beg)
> >>             (isearch-error nil))
> >> -       (isearch-lazy-highlight-new-loop range-beg range-end))))
> >> +       (save-match-data (isearch-lazy-highlight-new-loop range-beg
> >> range-end)))))
> >
> >>  (defun replace-dehighlight ()
> >>    (when replace-overlay
> >
> >> Then I added the following to my "~/.emacs", restarted my emacs server,
> and
> >> the issue was gone!:
> >
> >> (load-library "~/.emacs.d/lisp/replace_fixed.el")
> >
> >> This probably isn't the proper fix, but just thought I'd share in case
> >> anyone else is experiencing this and wanted a temporary workaround.
> >
> > Excellent!  To be honest, I was thinking of sending just that workaround
> > to you as a temporary patch, but it seems I didn't need to.  :-)
> >
> > That might well be the fix we end up putting into Emacs, but it might
> > not be.  Sorry we're so slow, here.
>
> I think first we should try to narrow down the source of this match data
> leak.
> Then we could decide what is the best solution.  Currently I see no such
> place
> in isearch-lazy-highlight-new-loop that calls external code.  OTOH, while
> looking at CC-Mode I noticed that it does some matches on before-change
> hooks.
> I could debug it but can't reproduce neither in 26.1 nor in 27, even with
> -Q -nw.
>

[Message part 2 (text/html, inline)]

Message #50 received at 36328 <at> debbugs.gnu.org (full text, mbox):

From: Alan Mackenzie <acm <at> muc.de>
To: Juri Linkov <juri <at> linkov.net>
Cc: Jayden Navarro <jayden <at> yugabyte.com>, 36328 <at> debbugs.gnu.org
Subject: Re: bug#36328: 26.2; Args out of range on search-and-replace of *.cc
 file
Date: Mon, 24 Jun 2019 07:52:18 +0000

Hello, Juri.

On Mon, Jun 24, 2019 at 00:19:22 +0300, Juri Linkov wrote:
> >> Until this is officially fixed I've come up with the following workaround,
> >> going off of the details you provided:

> >> I created a "replace_fixed.el" file in "~/.emacs.d/lisp/" that is
> >> replace.el taken from
> >> https://raw.githubusercontent.com/emacs-mirror/emacs/emacs-26/lisp/replace.el
> >> with
> >> the following diff:

> >> diff --git a/replace.el b/replace_fixed.el
> >> index 08feb8e..8280fdd 100644
> >> --- a/replace.el
> >> +++ b/replace_fixed.el
> >> @@ -2227,7 +2227,7 @@ It is called with three arguments, as if it were
> >>             (isearch-forward (not backward))
> >>             (isearch-other-end match-beg)
> >>             (isearch-error nil))
> >> -       (isearch-lazy-highlight-new-loop range-beg range-end))))
> >> +       (save-match-data (isearch-lazy-highlight-new-loop range-beg
> >> range-end)))))

> >>  (defun replace-dehighlight ()
> >>    (when replace-overlay

> >> Then I added the following to my "~/.emacs", restarted my emacs server, and
> >> the issue was gone!:

> >> (load-library "~/.emacs.d/lisp/replace_fixed.el")

> >> This probably isn't the proper fix, but just thought I'd share in case
> >> anyone else is experiencing this and wanted a temporary workaround.

> > Excellent!  To be honest, I was thinking of sending just that workaround
> > to you as a temporary patch, but it seems I didn't need to.  :-)

> > That might well be the fix we end up putting into Emacs, but it might
> > not be.  Sorry we're so slow, here.

> I think first we should try to narrow down the source of this match
> data leak.

Is there really such a thing as a match data leak?  I don't think there's
any convention that the match data are preserved over large bits of code,
particularly when different libraries are involved.  There is nothing
documented in the Elisp manual that I can see.

> Then we could decide what is the best solution.  Currently I see no
> such place in isearch-lazy-highlight-new-loop that calls external code.

isearch-lazy-highlight-new-loop calls (sit-for 0), which calls redisplay,
which calls font locking.

> OTOH, while looking at CC-Mode I noticed that it does some matches on
> before-change hooks.

The bulk of c-before-change is inside a save-match-data, as is the bulk
of c-after-change.  Other entry points (such as
c-font-lock-fontify-region) aren't enclosed in save-match-data.

> I could debug it but can't reproduce neither in 26.1 nor in 27, even
> with -Q -nw.

For what it's worth, I only saw the bug in X Windows, immediately after
starting emacs with emacs -Q -nw.  A second try with M-% (after the first
one has failed) just works.

-- 
Alan Mackenzie (Nuremberg, Germany).

Message #53 received at 36328 <at> debbugs.gnu.org (full text, mbox):

From: Juri Linkov <juri <at> linkov.net>
To: Jayden Navarro <jayden <at> yugabyte.com>
Cc: Alan Mackenzie <acm <at> muc.de>, 36328 <at> debbugs.gnu.org
Subject: Re: bug#36328: 26.2;
 Args out of range on search-and-replace of *.cc file
Date: Mon, 24 Jun 2019 22:05:48 +0300

Hi Jayden,

Thank you very much for the detailed test case.
Previously I tried in the MATE terminal emulator,
but xterm-256color was an essential detail indeed.
Now I was able to reproduce with TERM="xterm-256color"
and to track down the source of this problem.

It happens while the color "ForestGreen" is loaded for
the face font-lock-type-face that has this definition:

  (((class color) (min-colors 16) (background light))
   :foreground "ForestGreen")

by tty-color-canonicalize.

Could you please try this patch to see if it fixes the problem:

diff --git a/lisp/term/tty-colors.el b/lisp/term/tty-colors.el
index 307586f221..5af8170203 100644
--- a/lisp/term/tty-colors.el
+++ b/lisp/term/tty-colors.el
@@ -820,7 +820,7 @@ tty-color-canonicalize
   "Return COLOR in canonical form.
 A canonicalized color name is all-lower case, with any blanks removed."
   (let ((case-fold-search nil))
-    (if (string-match "[A-Z ]" color)
+    (if (string-match-p "[A-Z ]" color)
 	(replace-regexp-in-string " +" "" (downcase color))
       color)))
 

> Hi Juri,
>
> Did you open it with TERM="xterm-256color"?
>
> Best,
> Jayden
>
> On Sun, Jun 23, 2019 at 2:41 PM Juri Linkov <juri <at> linkov.net> wrote:
>
>> I think first we should try to narrow down the source of this match data
>> leak.
>> Then we could decide what is the best solution.  Currently I see no such
>> place
>> in isearch-lazy-highlight-new-loop that calls external code.  OTOH, while
>> looking at CC-Mode I noticed that it does some matches on before-change
>> hooks.
>> I could debug it but can't reproduce neither in 26.1 nor in 27, even with
>> -Q -nw.
>>

Message #56 received at 36328 <at> debbugs.gnu.org (full text, mbox):

From: Juri Linkov <juri <at> linkov.net>
To: Alan Mackenzie <acm <at> muc.de>
Cc: Jayden Navarro <jayden <at> yugabyte.com>, 36328 <at> debbugs.gnu.org
Subject: Re: bug#36328: 26.2;
 Args out of range on search-and-replace of *.cc file
Date: Mon, 24 Jun 2019 22:18:53 +0300

Hello, Alan.

>> I think first we should try to narrow down the source of this match
>> data leak.
>
> Is there really such a thing as a match data leak?  I don't think there's
> any convention that the match data are preserved over large bits of code,
> particularly when different libraries are involved.  There is nothing
> documented in the Elisp manual that I can see.

Yes, it seems such match-data leak is considered at least undesirable.
I remember efforts to replace string-match with string-match-p in
potentially unsafe places and to wrap more code in save-match-data.
But I guess such efforts are futile since this task is endless.

Usually it's enough for a function that cares about preserving match-data
to protect it from mutation.

>> Then we could decide what is the best solution.  Currently I see no
>> such place in isearch-lazy-highlight-new-loop that calls external code.
>
> isearch-lazy-highlight-new-loop calls (sit-for 0), which calls redisplay,
> which calls font locking.

You are right that it's too much to expect that the match-data will be
preserved after redisplay, and we can't find and fix all places that
change match-data, so save-match-data needs be added to perform-replace
somewhere to protect match-data.

Since (sit-for 0) is unsafe for match-data, the first candidate to be
wrapped in save-match-data is (sit-for 0) itself in isearch-lazy-highlight-new-loop.

But perhaps more correct would be to use save-match-data in the same
function that cares about preserving its match-data, so the second
candidate to use save-match-data is perform-replace.  Then the need
of using save-match-data will be self-evident for everyone who will
look at the code in perform-replace: here we use match-data, and here
we protect it in the same function.

Message #59 received at 36328 <at> debbugs.gnu.org (full text, mbox):

From: Jayden Navarro <jayden <at> yugabyte.com>
To: Juri Linkov <juri <at> linkov.net>
Cc: Alan Mackenzie <acm <at> muc.de>, 36328 <at> debbugs.gnu.org
Subject: Re: bug#36328: 26.2;
 Args out of range on search-and-replace of *.cc file
Date: Mon, 24 Jun 2019 13:03:31 -0700

[Message part 1 (text/plain, inline)]

Hi Juri,

Thank you very much for the detailed test case.
> Previously I tried in the MATE terminal emulator,
> but xterm-256color was an essential detail indeed.
> Now I was able to reproduce with TERM="xterm-256color"
> and to track down the source of this problem.
>
> It happens while the color "ForestGreen" is loaded for
> the face font-lock-type-face that has this definition:
>
>   (((class color) (min-colors 16) (background light))
>    :foreground "ForestGreen")
>
> by tty-color-canonicalize.
>
> Could you please try this patch to see if it fixes the problem:
>
> diff --git a/lisp/term/tty-colors.el b/lisp/term/tty-colors.el
> index 307586f221..5af8170203 100644
> --- a/lisp/term/tty-colors.el
> +++ b/lisp/term/tty-colors.el
> @@ -820,7 +820,7 @@ tty-color-canonicalize
>    "Return COLOR in canonical form.
>  A canonicalized color name is all-lower case, with any blanks removed."
>    (let ((case-fold-search nil))
> -    (if (string-match "[A-Z ]" color)
> +    (if (string-match-p "[A-Z ]" color)
>         (replace-regexp-in-string " +" "" (downcase color))
>        color)))
>

Thank you for root-causing this! The patch you provided does indeed fix the
issue for me.

Best,
Jayden

[Message part 2 (text/html, inline)]

Message #62 received at 36328 <at> debbugs.gnu.org (full text, mbox):

From: Alan Mackenzie <acm <at> muc.de>
To: Juri Linkov <juri <at> linkov.net>
Cc: Jayden Navarro <jayden <at> yugabyte.com>, 36328 <at> debbugs.gnu.org
Subject: Re: bug#36328: 26.2; Args out of range on search-and-replace of *.cc
 file
Date: Tue, 25 Jun 2019 09:47:08 +0000

Hello, Juri.

On Mon, Jun 24, 2019 at 22:18:53 +0300, Juri Linkov wrote:
> Hello, Alan.

> >> I think first we should try to narrow down the source of this match
> >> data leak.

> > Is there really such a thing as a match data leak?  I don't think
> > there's any convention that the match data are preserved over large
> > bits of code, particularly when different libraries are involved.
> > There is nothing documented in the Elisp manual that I can see.

> Yes, it seems such match-data leak is considered at least undesirable.
> I remember efforts to replace string-match with string-match-p in
> potentially unsafe places and to wrap more code in save-match-data.
> But I guess such efforts are futile since this task is endless.

I think so, too.  I remembered being puzzled in my early Emacs days,
wondering whether the save-match-data should go in the function which
messes it up, or the function which cares about it.

> Usually it's enough for a function that cares about preserving
> match-data to protect it from mutation.

I now think this is the best place to put save-match-data.

> >> Then we could decide what is the best solution.  Currently I see no
> >> such place in isearch-lazy-highlight-new-loop that calls external
> >> code.

> > isearch-lazy-highlight-new-loop calls (sit-for 0), which calls
> > redisplay, which calls font locking.

> You are right that it's too much to expect that the match-data will be
> preserved after redisplay, and we can't find and fix all places that
> change match-data, so save-match-data needs be added to perform-replace
> somewhere to protect match-data.

Yes, I think so.

> Since (sit-for 0) is unsafe for match-data, the first candidate to be
> wrapped in save-match-data is (sit-for 0) itself in
> isearch-lazy-highlight-new-loop.

> But perhaps more correct would be to use save-match-data in the same
> function that cares about preserving its match-data, so the second
> candidate to use save-match-data is perform-replace.  Then the need
> of using save-match-data will be self-evident for everyone who will
> look at the code in perform-replace: here we use match-data, and here
> we protect it in the same function.

My feeling is that perform-replace (or, possibly, replace-highlight) is
the best place to put a save-match-data.

-- 
Alan Mackenzie (Nuremberg, Germany).

Message #65 received at 36328 <at> debbugs.gnu.org (full text, mbox):

From: Juri Linkov <juri <at> linkov.net>
To: Alan Mackenzie <acm <at> muc.de>
Cc: Jayden Navarro <jayden <at> yugabyte.com>, 36328 <at> debbugs.gnu.org
Subject: Re: bug#36328: 26.2;
 Args out of range on search-and-replace of *.cc file
Date: Tue, 25 Jun 2019 22:58:56 +0300

[Message part 1 (text/plain, inline)]

Hello, Alan.

> My feeling is that perform-replace (or, possibly, replace-highlight) is
> the best place to put a save-match-data.

Doing this in perform-replace means adding save-match-data in all places
where replace-highlight is called, so maybe better to add it only once
in replace-highlight like Jayden proposed initially.

But then it's important to add an explanatory comment since
the need of using of save-match-data is not self-evident here.

Do you think this comment is clear enough?

[replace-save-match-data.patch (text/x-diff, inline)]

diff --git a/lisp/replace.el b/lisp/replace.el
index 9d1b7bf747..50c74be8f6 100644
--- a/lisp/replace.el
+++ b/lisp/replace.el
@@ -2316,7 +2316,11 @@ replace-highlight
 	    (isearch-forward (not backward))
 	    (isearch-other-end match-beg)
 	    (isearch-error nil))
-	(isearch-lazy-highlight-new-loop range-beg range-end))))
+	(save-match-data
+	  ;; Preserve match-data for perform-replace since
+	  ;; isearch-lazy-highlight-new-loop calls `sit-for' that
+	  ;; does redisplay that might clobber match data (bug#36328).
+	  (isearch-lazy-highlight-new-loop range-beg range-end)))))
 
 (defun replace-dehighlight ()
   (when replace-overlay
diff --git a/lisp/term/tty-colors.el b/lisp/term/tty-colors.el
index 307586f221..5af8170203 100644
--- a/lisp/term/tty-colors.el
+++ b/lisp/term/tty-colors.el
@@ -820,7 +820,7 @@ tty-color-canonicalize
   "Return COLOR in canonical form.
 A canonicalized color name is all-lower case, with any blanks removed."
   (let ((case-fold-search nil))
-    (if (string-match "[A-Z ]" color)
+    (if (string-match-p "[A-Z ]" color)
 	(replace-regexp-in-string " +" "" (downcase color))
       color)))
 

Message #68 received at 36328 <at> debbugs.gnu.org (full text, mbox):

From: Juri Linkov <juri <at> linkov.net>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: Alan Mackenzie <acm <at> muc.de>, 36328 <at> debbugs.gnu.org,
 Jayden Navarro <jayden <at> yugabyte.com>
Subject: Re: bug#36328: 26.2;
 Args out of range on search-and-replace of *.cc file
Date: Fri, 05 Jul 2019 00:09:47 +0300

Eli, do you think this fix should be pushed to emacs-26?

>> My feeling is that perform-replace (or, possibly, replace-highlight) is
>> the best place to put a save-match-data.
>
> Doing this in perform-replace means adding save-match-data in all places
> where replace-highlight is called, so maybe better to add it only once
> in replace-highlight like Jayden proposed initially.
>
> But then it's important to add an explanatory comment since
> the need of using of save-match-data is not self-evident here.
>
> Do you think this comment is clear enough?
>
> diff --git a/lisp/replace.el b/lisp/replace.el
> index 9d1b7bf747..50c74be8f6 100644
> --- a/lisp/replace.el
> +++ b/lisp/replace.el
> @@ -2316,7 +2316,11 @@ replace-highlight
>  	    (isearch-forward (not backward))
>  	    (isearch-other-end match-beg)
>  	    (isearch-error nil))
> -	(isearch-lazy-highlight-new-loop range-beg range-end))))
> +	(save-match-data
> +	  ;; Preserve match-data for perform-replace since
> +	  ;; isearch-lazy-highlight-new-loop calls `sit-for' that
> +	  ;; does redisplay that might clobber match data (bug#36328).
> +	  (isearch-lazy-highlight-new-loop range-beg range-end)))))
>  
>  (defun replace-dehighlight ()
>    (when replace-overlay
> diff --git a/lisp/term/tty-colors.el b/lisp/term/tty-colors.el
> index 307586f221..5af8170203 100644
> --- a/lisp/term/tty-colors.el
> +++ b/lisp/term/tty-colors.el
> @@ -820,7 +820,7 @@ tty-color-canonicalize
>    "Return COLOR in canonical form.
>  A canonicalized color name is all-lower case, with any blanks removed."
>    (let ((case-fold-search nil))
> -    (if (string-match "[A-Z ]" color)
> +    (if (string-match-p "[A-Z ]" color)
>  	(replace-regexp-in-string " +" "" (downcase color))
>        color)))

Message #71 received at 36328 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Juri Linkov <juri <at> linkov.net>
Cc: acm <at> muc.de, 36328 <at> debbugs.gnu.org, jayden <at> yugabyte.com
Subject: Re: bug#36328: 26.2;
 Args out of range on search-and-replace of *.cc file
Date: Fri, 05 Jul 2019 09:11:37 +0300

> From: Juri Linkov <juri <at> linkov.net>
> Cc: Alan Mackenzie <acm <at> muc.de>,  Jayden Navarro <jayden <at> yugabyte.com>,  36328 <at> debbugs.gnu.org
> Date: Fri, 05 Jul 2019 00:09:47 +0300
> 
> Eli, do you think this fix should be pushed to emacs-26?

I'd like to avoid any code changes on the branch that aren't 110%
necessary.  We should release 26.3 ASAP, otherwise Emacs 27 will wait
for too long.

Sorry.

Message #74 received at 36328 <at> debbugs.gnu.org (full text, mbox):

From: Juri Linkov <juri <at> linkov.net>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: acm <at> muc.de, 36328 <at> debbugs.gnu.org, jayden <at> yugabyte.com
Subject: Re: bug#36328: 26.2;
 Args out of range on search-and-replace of *.cc file
Date: Fri, 05 Jul 2019 22:12:16 +0300

tags 36328 + patch
fixed 36328 27.0.50
thanks

>> Eli, do you think this fix should be pushed to emacs-26?
>
> I'd like to avoid any code changes on the branch that aren't 110%
> necessary.  We should release 26.3 ASAP, otherwise Emacs 27 will wait
> for too long.

OK, pushed to master in dde0320020.

Message #83 received at 36328-done <at> debbugs.gnu.org (full text, mbox):

From: Stefan Kangas <stefan <at> marxist.se>
To: Juri Linkov <juri <at> linkov.net>
Cc: Alan Mackenzie <acm <at> muc.de>, Eli Zaretskii <eliz <at> gnu.org>,
 36328-done <at> debbugs.gnu.org, jayden <at> yugabyte.com
Subject: Re: bug#36328: 26.2;
 Args out of range on search-and-replace of *.cc file
Date: Thu, 3 Oct 2019 01:53:48 +0200

Juri Linkov <juri <at> linkov.net> writes:

> tags 36328 + patch
> fixed 36328 27.0.50
> thanks
>
> >> Eli, do you think this fix should be pushed to emacs-26?
> >
> > I'd like to avoid any code changes on the branch that aren't 110%
> > necessary.  We should release 26.3 ASAP, otherwise Emacs 27 will wait
> > for too long.
>
> OK, pushed to master in dde0320020.

I'll assume that fixed it and close this bug.  If this is still an
issue, please reopen.

Best regards,
Stefan Kangas

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.