GNU bug report logs - #35264
"Match data clobbered by buffer modification hooks" when hooks only shifted match-data's markers

Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.

Package: emacs; Reported by: Noam Postavsky <npostavs@HIDDEN>; dated Sun, 14 Apr 2019 04:41:01 UTC; Maintainer for emacs is bug-gnu-emacs@HIDDEN.

Message received at 35264 <at> debbugs.gnu.org:


Received: (at 35264) by debbugs.gnu.org; 30 May 2019 23:04:05 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu May 30 19:04:05 2019
Received: from localhost ([127.0.0.1]:34318 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1hWU5c-0000X5-QM
	for submit <at> debbugs.gnu.org; Thu, 30 May 2019 19:04:05 -0400
Received: from mail-it1-f182.google.com ([209.85.166.182]:39899)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <npostavs@HIDDEN>) id 1hWU5a-0000WX-0a
 for 35264 <at> debbugs.gnu.org; Thu, 30 May 2019 19:04:03 -0400
Received: by mail-it1-f182.google.com with SMTP id j204so6893880ite.4
 for <35264 <at> debbugs.gnu.org>; Thu, 30 May 2019 16:04:01 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=from:to:cc:subject:references:date:in-reply-to:message-id
 :user-agent:mime-version;
 bh=uPI4KOQe9rq+2EMxvVflJUtWVIHqOAa0RqZWYYjnu7A=;
 b=o4Af/+OYqabOf33LYZGYKCyH3L4J+UyiY+X3UXBYdAiHfqhozErz0eNdTrhjKPYTMe
 NKpAf0Oj4gASFMBI7qnngjhfoIkO7ifzxnSLADVkN+j86YV9aMIXtXUyJ7X04Gz1Macy
 oHFsZHpOCtmTE9q2k9SbDhPpcwbVglSxKtsFw0R7+jsfXEEgUtg86iJxSwhGimU+dJ5j
 sOBWHHAXRB+B2YaiHtpFXk/QFnTyXKKLmWgOdyhvs24ne/BvSQo3rQ5Tdo9WkBreiuED
 8jnCXHcu+F+tQ38T+CzPoSYsIbJGcU8omdC4VpLed5GoRpflncppCPviFNKqpM5K183B
 SlPQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to
 :message-id:user-agent:mime-version;
 bh=uPI4KOQe9rq+2EMxvVflJUtWVIHqOAa0RqZWYYjnu7A=;
 b=KuzdZJsOhWwFxqnny+I0a8ProiGHHluvCBjok1VFVptLt9pENG2WZYXdE1+p6qdnbn
 GRJyAh0GT33y9S522LlmD03UfkTOglk0ZLV/5VzUXGzc7yvFEDLNj95uThUrS9UEnIP2
 hjupXQVkjK/VF0sR//bh5+Js8BSgv8n0JjQVKJpj4F+kev/ufsoQXoD/Dw16pnGspSZP
 M/R7DMmPwHY6EHoKAiakP6MgHVvxbsVtnwd5OHk3/uAlALDRs+FTHrregAVwlMyEZWTd
 IUg0B9uiXYEyybfbTyZSPqEx8yS+qg99KQyWqyFtkoPldFvqKQb+I9Qn8mszbx8Pa3uu
 fa2Q==
X-Gm-Message-State: APjAAAUNMdLD5f0+b9OmDg9mqWx3UJLQp5z8289asTLuv5vVGiBNGX+d
 2pkYFoRctNGWJ4u2ob5EAAo=
X-Google-Smtp-Source: APXvYqwlA5xggJ8h4djSsnldTAJQr+xfPXt5RE3RBtKKR7DjSHZQNC9svppZDr0iRMfwn6eQh9a8HQ==
X-Received: by 2002:a24:64c6:: with SMTP id t189mr5012622itc.161.1559257436427; 
 Thu, 30 May 2019 16:03:56 -0700 (PDT)
Received: from minid (cbl-45-2-119-34.yyz.frontiernetworks.ca. [45.2.119.34])
 by smtp.gmail.com with ESMTPSA id
 u134sm1777255itb.32.2019.05.30.16.03.55
 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256);
 Thu, 30 May 2019 16:03:55 -0700 (PDT)
From: Noam Postavsky <npostavs@HIDDEN>
To: 35264 <at> debbugs.gnu.org
Subject: Re: bug#35264: "Match data clobbered by buffer modification hooks"
 when hooks only shifted match-data's markers
References: <87h8b119oy.fsf@HIDDEN>
Date: Thu, 30 May 2019 19:03:54 -0400
In-Reply-To: <87h8b119oy.fsf@HIDDEN> (Noam Postavsky's message of "Sun, 14
 Apr 2019 00:40:13 -0400")
Message-ID: <871s0f7det.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="=-=-="
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 35264
Cc: stefan monnier <monnier@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

--=-=-=
Content-Type: text/plain


> I think this is a variant of Bug#23917, there was some talk there about
> removing the check, perhaps that is the right solution.

So, this, I guess.


--=-=-=
Content-Type: text/x-diff
Content-Disposition: inline;
 filename=0001-Remove-unreliable-test-for-match-data-clobbering-Bug.patch
Content-Description: patch

From 6fd6605c63ecc031a3fd6ba8b8e2e754c183b3f2 Mon Sep 17 00:00:00 2001
From: Noam Postavsky <npostavs@HIDDEN>
Date: Thu, 30 May 2019 19:00:31 -0400
Subject: [PATCH] Remove unreliable test for match data clobbering (Bug#35264)

* src/search.c (Freplace_match): Don't test for change in search_regs
start and end, this is unreliable if change hooks modify text earlier
in the buffer.
---
 src/search.c | 18 +++++-------------
 1 file changed, 5 insertions(+), 13 deletions(-)

diff --git a/src/search.c b/src/search.c
index db7fecd9ba..1d4550849e 100644
--- a/src/search.c
+++ b/src/search.c
@@ -2730,22 +2730,16 @@ DEFUN ("replace-match", Freplace_match, Sreplace_match, 1, 5, 0,
   /* The functions below modify the buffer, so they could trigger
      various modification hooks (see signal_before_change and
      signal_after_change).  If these hooks clobber the match data we
-     error out since otherwise this will result in confusing bugs.  */
-  ptrdiff_t sub_start = search_regs.start[sub];
-  ptrdiff_t sub_end = search_regs.end[sub];
+     error out since otherwise this will result in confusing bugs.  We
+     used to check for changes in search_regs start and end, but that
+     fails if modification hooks remove or add text earlier in the
+     buffer, so just check num_regs now.  */
   unsigned  num_regs = search_regs.num_regs;
   newpoint = search_regs.start[sub] + SCHARS (newtext);
 
   /* Replace the old text with the new in the cleanest possible way.  */
   replace_range (search_regs.start[sub], search_regs.end[sub],
                  newtext, 1, 0, 1, 1);
-  /* Update saved data to match adjustment made by replace_range.  */
-  {
-    ptrdiff_t change = newpoint - sub_end;
-    if (sub_start >= sub_end)
-      sub_start += change;
-    sub_end += change;
-  }
 
   if (case_action == all_caps)
     Fupcase_region (make_number (search_regs.start[sub]),
@@ -2755,9 +2749,7 @@ DEFUN ("replace-match", Freplace_match, Sreplace_match, 1, 5, 0,
     Fupcase_initials_region (make_number (search_regs.start[sub]),
 			     make_number (newpoint));
 
-  if (search_regs.start[sub] != sub_start
-      || search_regs.end[sub] != sub_end
-      || search_regs.num_regs != num_regs)
+  if (search_regs.num_regs != num_regs)
     error ("Match data clobbered by buffer modification hooks");
 
   /* Put point back where it was in the text.  */
-- 
2.11.0


--=-=-=--




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#35264; Package emacs. Full text available.

Message received at submit <at> debbugs.gnu.org:


Received: (at submit) by debbugs.gnu.org; 14 Apr 2019 04:40:27 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Apr 14 00:40:27 2019
Received: from localhost ([127.0.0.1]:59647 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1hFWwN-0002q6-4v
	for submit <at> debbugs.gnu.org; Sun, 14 Apr 2019 00:40:27 -0400
Received: from eggs.gnu.org ([209.51.188.92]:44433)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <npostavs@HIDDEN>) id 1hFWwL-0002pu-Lr
 for submit <at> debbugs.gnu.org; Sun, 14 Apr 2019 00:40:26 -0400
Received: from lists.gnu.org ([209.51.188.17]:37392)
 by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32)
 (Exim 4.71) (envelope-from <npostavs@HIDDEN>) id 1hFWwG-000822-Fm
 for submit <at> debbugs.gnu.org; Sun, 14 Apr 2019 00:40:20 -0400
Received: from eggs.gnu.org ([209.51.188.92]:58814)
 by lists.gnu.org with esmtp (Exim 4.71)
 (envelope-from <npostavs@HIDDEN>) id 1hFWwF-0003SS-8N
 for bug-gnu-emacs@HIDDEN; Sun, 14 Apr 2019 00:40:20 -0400
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=BAYES_40,FREEMAIL_FROM,
 URIBL_BLOCKED autolearn=disabled version=3.3.2
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <npostavs@HIDDEN>) id 1hFWwE-00080C-7U
 for bug-gnu-emacs@HIDDEN; Sun, 14 Apr 2019 00:40:19 -0400
Received: from mail-qk1-x72e.google.com ([2607:f8b0:4864:20::72e]:34464)
 by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16)
 (Exim 4.71) (envelope-from <npostavs@HIDDEN>) id 1hFWwC-0007yT-UT
 for bug-gnu-emacs@HIDDEN; Sun, 14 Apr 2019 00:40:17 -0400
Received: by mail-qk1-x72e.google.com with SMTP id n68so8042246qka.1
 for <bug-gnu-emacs@HIDDEN>; Sat, 13 Apr 2019 21:40:16 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=from:to:subject:date:message-id:mime-version;
 bh=t1/blLmDk392Fbv6JE4t/cjh/+eVx5p5RKY4GjxfWW4=;
 b=EBWU5ddtpH1dV+MYqE6j4S1eBa4Q1JrmzPJmJjFIqMcaTxH3zVdWXMjty5eWgCpayP
 ewNHArj6TjPo4u65dr3cnELkaBUKWR6h+X1TYublk9+7Jtej5yudw554bG5TeaicqNhk
 c28J4DQGi+OqfcER+ejTdmixTVaLuojexfthWHwkCI56pF/AbF6EKbBql5qxsnhBH1kK
 xSRLLU6swKhIqlK3dzcQ5xMnfsLVfvoc6AHQ3QuRm3u7wJRybpqDSlbUfLFh4NTAsIHN
 Ii3LVNrvAe0jrjfPdNVWmjDqCiZyshlPlaNE5ix/OdS2twlJP7u9DJWXViK6fFG70XpC
 SyLg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:from:to:subject:date:message-id:mime-version;
 bh=t1/blLmDk392Fbv6JE4t/cjh/+eVx5p5RKY4GjxfWW4=;
 b=kveMi1aBjEpUpXqjPq3GNEucECbImtr+Y9tEIhU1n3LKKB7OSk05Tp26LU8Fqfu7bh
 qfMoiYnjGEKNewVbksL3U9T4NTslkMfB5ZhvZTTbPvHOoqWSESI9420DOoZqhEBCYJVy
 NjNYvu+dlzjAzk5LPiBYo9sdKmSAP2b2MogWaPVakfQ8NX9LJQHnledj+9IlhNEK7OAK
 v8+lVOs/h/UOaRZHvlBUVQZ1MKj+C34XDg2PfgbXx52ammK8g7FUOjpoGpZewVpjC0aF
 uxxZdjRpC/MsirSnQcyJP5x9Z/axnOJgJlQJqM8sGjNlTv4TpcdpnJMjVjRS3hVQGEt4
 WKOg==
X-Gm-Message-State: APjAAAWCPa+f8yXtALHVMF9K7ECnrT7cnPz5aeaIEZcTjSII3EAqAn2o
 gvtOK4ByaINoShaIphbsfgiYPrQN
X-Google-Smtp-Source: APXvYqw6SkCiu13wKAQ2kXfka514OQq+kyRnzatEo+khEL2thCWA3YGI3PeWMj/8tjefukwzU+L/kA==
X-Received: by 2002:a37:e507:: with SMTP id e7mr52311435qkg.322.1555216816161; 
 Sat, 13 Apr 2019 21:40:16 -0700 (PDT)
Received: from minid (cbl-45-2-119-34.yyz.frontiernetworks.ca. [45.2.119.34])
 by smtp.googlemail.com with ESMTPSA id
 m189sm27646469qkf.2.2019.04.13.21.40.14 for <bug-gnu-emacs@HIDDEN>
 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256);
 Sat, 13 Apr 2019 21:40:14 -0700 (PDT)
From: Noam Postavsky <npostavs@HIDDEN>
To: bug-gnu-emacs@HIDDEN
Subject: "Match data clobbered by buffer modification hooks" when hooks only
 shifted match-data's markers
Date: Sun, 14 Apr 2019 00:40:13 -0400
Message-ID: <87h8b119oy.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="=-=-="
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
 recognized.
X-Received-From: 2607:f8b0:4864:20::72e
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x
X-Spam-Score: -1.3 (-)
X-Debbugs-Envelope-To: submit
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -2.3 (--)

--=-=-=
Content-Type: text/plain

Version: 27.0.50 26.2 25.3
X-Debbugs-CC: Stefan Monnier <monnier@HIDDEN>

emacs -Q -l bug-xxxx-match-data-marker-clobber.el

Hit <f12>, gives

Debugger entered--Lisp error: (error "Match data clobbered by buffer modification hooks")
  replace-match("ABCDEF" t t)
  (let* ((after-change-functions (list (function (lambda (&rest _) (let (... ...) (save-excursion ...))))))) (search-backward "abcdef") (replace-match "ABCDEF" t t))
  (save-current-buffer (set-buffer (get-buffer-create "*test*")) (display-buffer (current-buffer)) (erase-buffer) (insert "1234567890\n") (insert "abcdefghilk\n") (make-local-variable (quote after-change-functions)) (let* ((after-change-functions (list (function (lambda (&rest _) (let ... ...)))))) (search-backward "abcdef") (replace-match "ABCDEF" t t)))
  bug-match-data-marker-clobber()
  funcall-interactively(bug-match-data-marker-clobber)
  call-interactively(bug-match-data-marker-clobber nil nil)
  command-execute(bug-match-data-marker-clobber)

But the modification hook in question did call save-match-data.  As far
as I can tell, the problem is that the match-data consists of markers,
whose position gets shifted by deletion of characters.  The check for
this error uses simple integers, so there's no way it can account for
this.

I think this is a variant of Bug#23917, there was some talk there about
removing the check, perhaps that is the right solution.

DEFUN ("replace-match", Freplace_match, Sreplace_match, 1, 5, 0,
  [...]
  /* The functions below modify the buffer, so they could trigger
     various modification hooks (see signal_before_change and
     signal_after_change).  If these hooks clobber the match data we
     error out since otherwise this will result in confusing bugs.  */
  ptrdiff_t sub_start = search_regs.start[sub];
  ptrdiff_t sub_end = search_regs.end[sub];
  unsigned  num_regs = search_regs.num_regs;
  newpoint = search_regs.start[sub] + SCHARS (newtext);

  /* Replace the old text with the new in the cleanest possible way.  */
  replace_range (search_regs.start[sub], search_regs.end[sub],
                 newtext, 1, 0, 1, 1);
  [...]
  if (search_regs.start[sub] != sub_start
      || search_regs.end[sub] != sub_end
      || search_regs.num_regs != num_regs)
    error ("Match data clobbered by buffer modification hooks");


bug-xxxx-match-data-marker-clobber.el:


--=-=-=
Content-Type: text/plain
Content-Disposition: inline; filename=bug-xxxx-match-data-marker-clobber.el
Content-Description: bug reproducer

(defun bug-match-data-marker-clobber ()
  (interactive)
  (with-current-buffer (get-buffer-create "*test*")
    (display-buffer (current-buffer))
    (erase-buffer)
    (insert "1234567890\n")
    (insert "abcdefghilk\n")
    (make-local-variable 'after-change-functions)
    (let* ((after-change-functions `
            (,(lambda (&rest _)
                (let ((inhibit-modification-hooks nil)
                      (after-change-functions nil))
                  (save-excursion
                    (save-match-data
                      (goto-char (point-min))
                      (looking-at "[0-9]")
                      (delete-char 1))
                    ;; match-data is restored, but markers have a
                    ;; different position now, because of the
                    ;; deletion.
                    ;;
                    ;; (match-data) ;=> (#<marker@11> #<marker@17>)
                    ))))))
      (search-backward "abcdef")
      ;; (match-data) ;=> (#<marker@12> #<marker@18>)
      (replace-match "ABCDEF" t t) ;; Triggers `after-change-functions'.
      )))

(setq debug-on-error t)
(define-key global-map [f12] 'bug-match-data-marker-clobber)


--=-=-=--




Acknowledgement sent to Noam Postavsky <npostavs@HIDDEN>:
New bug report received and forwarded. Copy sent to monnier@HIDDEN, bug-gnu-emacs@HIDDEN.

Your message had a Version: pseudo-header with an invalid package version:

27.0.50 26.2 25.3

please either use found or fixed to the control server with a correct version, or reply to this report indicating the correct version so the maintainer (or someone else) can correct it for you.

Full text available.


Report forwarded to monnier@HIDDEN, bug-gnu-emacs@HIDDEN:
bug#35264; Package emacs. Full text available.
Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.
Last modified: Mon, 25 Nov 2019 12:00:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.