GNU bug report logs - #65451
30.0.50; `after-change-functions' are not triggered in the same order the changes are made

Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.
Package: emacs; Reported by: Ihor Radchenko <yantar92@HIDDEN>; dated Tue, 22 Aug 2023 09:31:01 UTC; Maintainer for emacs is bug-gnu-emacs@HIDDEN.

Message received at 65451 <at> debbugs.gnu.org:


Received: (at 65451) by debbugs.gnu.org; 24 Aug 2023 11:24:21 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Aug 24 07:24:21 2023
Received: from localhost ([127.0.0.1]:36269 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qZ8Rp-0008GD-FL
	for submit <at> debbugs.gnu.org; Thu, 24 Aug 2023 07:24:21 -0400
Received: from mout02.posteo.de ([185.67.36.66]:40865)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <yantar92@HIDDEN>) id 1qZ8Rn-0008Fx-1w
 for 65451 <at> debbugs.gnu.org; Thu, 24 Aug 2023 07:24:19 -0400
Received: from submission (posteo.de [185.67.36.169]) 
 by mout02.posteo.de (Postfix) with ESMTPS id 1AE3E240103
 for <65451 <at> debbugs.gnu.org>; Thu, 24 Aug 2023 13:24:09 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;
 t=1692876249; bh=m5Nea0jKwXcoXWX6r4XyFZ4lWiJHTB2vrUXEX7V8k7E=;
 h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:From;
 b=elAMqPvtUXG48Sg2swVdzqO4GGi/hLlkIPp3tf8ahG2gOqabWHYug7rDlR8+f/j/M
 QetSwSU1kzMClSHPRmFHcFnrOzdpLiDmuVZ+IhdRZrGqWfSYT+cvxMmv88SyXMMdk6
 ZOxmcSFtwc+3j+WmnKdCOAuMUpiXGqod0h/6RVrNvVxPwoUSUjwSjHPFBviPChA5qW
 ehzq+lHmlxCBbLvl3esz7nMVqZdTWVVckhnOmq6tNXYjDzZ8Q2AVIQLgKuM33sW4yI
 O/bqqeiCB7hHVhdeg7A5j0BZ+i6mON/F19ugS4focr4FvN3kqECnukewHaCfOnpv8l
 Esf4UXlbbpy9w==
Received: from customer (localhost [127.0.0.1])
 by submission (posteo.de) with ESMTPSA id 4RWghS2Fz5z6tvw;
 Thu, 24 Aug 2023 13:24:08 +0200 (CEST)
From: Ihor Radchenko <yantar92@HIDDEN>
To: Eli Zaretskii <eliz@HIDDEN>
Subject: Re: bug#65451: 30.0.50; `after-change-functions' are not triggered
 in the same order the changes are made
In-Reply-To: <83zg2gq2vj.fsf@HIDDEN>
References: <871qfv2zlk.fsf@localhost> <83a5ujtgfo.fsf@HIDDEN>
 <87jztn1c5x.fsf@localhost> <834jkrters.fsf@HIDDEN>
 <87v8d7i48y.fsf@localhost> <83ttsrrroo.fsf@HIDDEN>
 <874jkq87jl.fsf@localhost> <83y1i1r689.fsf@HIDDEN>
 <87fs487uip.fsf@localhost> <83zg2gq2vj.fsf@HIDDEN>
Date: Thu, 24 Aug 2023 11:24:37 +0000
Message-ID: <871qfsel8q.fsf@localhost>
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 65451
Cc: casouri@HIDDEN, 65451 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Eli Zaretskii <eliz@HIDDEN> writes:

> Exposing buffer text changes to Lisp is inherently dangerous, because
> Lisp code can do anything and everything in these hooks, and many
> packages already do.  The fact that we allow this via those two hooks
> is unfortunate as it is, but adding more opportunities for Lisp to do
> potentially dangerous stuff, and doing that on a lower level, where
> the buffer object is sometimes in a state that is not 100% consistent,
> is unwise, to say the least.  It will make Emacs much less stable.
> No, thanks.

I can see the danger running lisp code while buffer state is transient.

Would it be acceptable to accumulate treesit_record_change transactions
into a queue, combine them later, and run the Elisp hook only when it is
safe (around the same place `after-change-functions' is called, but only
when actual edit is made to the buffer text)?

That way, we can have a hook that will run strictly less frequently
compared to `after-change-functions'.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>
Information forwarded to bug-gnu-emacs@HIDDEN:
bug#65451; Package emacs. Full text available.

Message received at 65451 <at> debbugs.gnu.org:


Received: (at 65451) by debbugs.gnu.org; 24 Aug 2023 08:08:05 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Aug 24 04:08:05 2023
Received: from localhost ([127.0.0.1]:36067 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qZ5Ns-0002fK-Sb
	for submit <at> debbugs.gnu.org; Thu, 24 Aug 2023 04:08:05 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:37744)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qZ5Nq-0002el-Fv
 for 65451 <at> debbugs.gnu.org; Thu, 24 Aug 2023 04:08:03 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qZ5Ng-0002rD-UN; Thu, 24 Aug 2023 04:07:52 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=lBJw0q221Airh9YICpczvSgbTpqsbdGw/jenVhkNDdQ=; b=X/cjkLZ5lZ6Q
 VyQ6m4n8s1S5brgs5JXCFunFiYKshaa8Dl5eKXplQNJGhjkPCDHZrWLlt32PS1pNruH9R/PvEO6Pn
 Gl/wtGtJSGnuExH9FjhnerMhGsQMpVOiPv8XOKcK+pv0o9u4kmcyYopsrG4eGIapejgSyhUjf2w1g
 k2eISv8R77S7AEngZx5TTEp4tWKiEtqIHm7iRWVlnpAxKjMXBvDbt5lJMCu3i8YFROgyMzCyjjEq7
 YmO1YlPRC9wFy8T4CyVQyLeRNtiGvpQAvtShSudokDo9arHmAZjQEB92ATa1DOGEQTOIRbvA1pqzi
 MMqev//oiTX+28sgEfqP1A==;
Date: Thu, 24 Aug 2023 11:08:16 +0300
Message-Id: <83zg2gq2vj.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Ihor Radchenko <yantar92@HIDDEN>
In-Reply-To: <87fs487uip.fsf@localhost> (message from Ihor Radchenko on Thu,
 24 Aug 2023 07:46:06 +0000)
Subject: Re: bug#65451: 30.0.50; `after-change-functions' are not triggered
 in the same order the changes are made
References: <871qfv2zlk.fsf@localhost> <83a5ujtgfo.fsf@HIDDEN>
 <87jztn1c5x.fsf@localhost> <834jkrters.fsf@HIDDEN>
 <87v8d7i48y.fsf@localhost> <83ttsrrroo.fsf@HIDDEN>
 <874jkq87jl.fsf@localhost> <83y1i1r689.fsf@HIDDEN> <87fs487uip.fsf@localhost>
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 65451
Cc: casouri@HIDDEN, 65451 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: Ihor Radchenko <yantar92@HIDDEN>
> Cc: casouri@HIDDEN, 65451 <at> debbugs.gnu.org
> Date: Thu, 24 Aug 2023 07:46:06 +0000
> 
> Eli Zaretskii <eliz@HIDDEN> writes:
> 
> > If these measures still don't help you enough, perhaps the conclusion
> > is that it isn't feasible to implement text parsers in Lisp, at least
> > as long as you want all those micro-optimizations of knowing exactly
> > which parts of the buffer text were modified (as opposed to only know
> > how many characters at the beginning and at the end of the buffer
> > remain unchanged, and reparse the rest).  Maybe it must be done in C,
> > if we want Emacs to remain a relatively safe environment.
> 
> Do I understand correctly that `treesit_record_change' is called
> __less frequently__ compared to before-change-functions and
> after-change-functions?

No, treesit_record_change is called at a lower level than
buffer-change hooks are, and therefore in some cases the hooks will
not be called, but treesit_record_change will be.

The frequency might be lower, but only because treesit_record_change
is called once per change; there's no separate "before" and "after"
calls.  In any case, the correspondence is not 1:1, because they are
called on different levels.

> If yes, I do not see how exposing it to Elisp will make things any
> worse than already available
> `before-change-functions'/`after-change-functions'.

Exposing buffer text changes to Lisp is inherently dangerous, because
Lisp code can do anything and everything in these hooks, and many
packages already do.  The fact that we allow this via those two hooks
is unfortunate as it is, but adding more opportunities for Lisp to do
potentially dangerous stuff, and doing that on a lower level, where
the buffer object is sometimes in a state that is not 100% consistent,
is unwise, to say the least.  It will make Emacs much less stable.
No, thanks.
Information forwarded to bug-gnu-emacs@HIDDEN:
bug#65451; Package emacs. Full text available.

Message received at 65451 <at> debbugs.gnu.org:


Received: (at 65451) by debbugs.gnu.org; 24 Aug 2023 07:45:50 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Aug 24 03:45:50 2023
Received: from localhost ([127.0.0.1]:36056 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qZ52M-000238-ES
	for submit <at> debbugs.gnu.org; Thu, 24 Aug 2023 03:45:50 -0400
Received: from mout01.posteo.de ([185.67.36.65]:49247)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <yantar92@HIDDEN>) id 1qZ52J-00022t-MM
 for 65451 <at> debbugs.gnu.org; Thu, 24 Aug 2023 03:45:49 -0400
Received: from submission (posteo.de [185.67.36.169]) 
 by mout01.posteo.de (Postfix) with ESMTPS id DDEFE240027
 for <65451 <at> debbugs.gnu.org>; Thu, 24 Aug 2023 09:45:37 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;
 t=1692863137; bh=v6ysih1UFiGqQ33TgWNHhsES9Xvp9DqaQXWtaArG2yU=;
 h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:From;
 b=k4aJn+W6fs4KJuOnJ05YAYo2EdkAtylyBKbiKDzb79b48jXi2twoRKC7e/63Ae2Ok
 nQz6RdF4QO83x1b/APqvvcD2dj1D15/B0yXkPXYUei7pAJkcbtVhJOc6WHujUjl0LY
 hl1nJP6St3FvnOLx396fI6UsUkR+jBXzVIE5lJXyQfO5ukTLCuRLH9rPseVSV2YV8C
 apLEpKUolAcADh+Vljc1zOAWxPoWyd1m54BImNzSUrrL/TOT2kZqFqLtuLalWWVkqn
 KU8Xhmg3EwTAEKcJR78fR9CsL8eJkb2zezuYZxwYWBl6h96Os10IbPuElCNglf9kzl
 PeIY9rDG5WRyA==
Received: from customer (localhost [127.0.0.1])
 by submission (posteo.de) with ESMTPSA id 4RWZrJ7298z9rxF;
 Thu, 24 Aug 2023 09:45:36 +0200 (CEST)
From: Ihor Radchenko <yantar92@HIDDEN>
To: Eli Zaretskii <eliz@HIDDEN>
Subject: Re: bug#65451: 30.0.50; `after-change-functions' are not triggered
 in the same order the changes are made
In-Reply-To: <83y1i1r689.fsf@HIDDEN>
References: <871qfv2zlk.fsf@localhost> <83a5ujtgfo.fsf@HIDDEN>
 <87jztn1c5x.fsf@localhost> <834jkrters.fsf@HIDDEN>
 <87v8d7i48y.fsf@localhost> <83ttsrrroo.fsf@HIDDEN>
 <874jkq87jl.fsf@localhost> <83y1i1r689.fsf@HIDDEN>
Date: Thu, 24 Aug 2023 07:46:06 +0000
Message-ID: <87fs487uip.fsf@localhost>
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 65451
Cc: casouri@HIDDEN, 65451 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Eli Zaretskii <eliz@HIDDEN> writes:

> If these measures still don't help you enough, perhaps the conclusion
> is that it isn't feasible to implement text parsers in Lisp, at least
> as long as you want all those micro-optimizations of knowing exactly
> which parts of the buffer text were modified (as opposed to only know
> how many characters at the beginning and at the end of the buffer
> remain unchanged, and reparse the rest).  Maybe it must be done in C,
> if we want Emacs to remain a relatively safe environment.

Do I understand correctly that `treesit_record_change' is called
__less frequently__ compared to before-change-functions and
after-change-functions? If yes, I do not see how exposing it to Elisp
will make things any worse than already available
`before-change-functions'/`after-change-functions'.

Elisp code that does not care about text property changes will not be
forced to use `before/after-change-functions' (because no other option)
and could be called less frequently.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>
Information forwarded to bug-gnu-emacs@HIDDEN:
bug#65451; Package emacs. Full text available.

Message received at 65451 <at> debbugs.gnu.org:


Received: (at 65451) by debbugs.gnu.org; 23 Aug 2023 17:58:07 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Wed Aug 23 13:58:07 2023
Received: from localhost ([127.0.0.1]:35127 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qYs7L-0003E3-Bw
	for submit <at> debbugs.gnu.org; Wed, 23 Aug 2023 13:58:07 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:49394)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qYs7F-0003DS-S6
 for 65451 <at> debbugs.gnu.org; Wed, 23 Aug 2023 13:58:05 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qYs76-0002UQ-NC; Wed, 23 Aug 2023 13:57:52 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=pyOmhdRnJqnDB0FrYgeN7nFgDGTd4/jGzZyms90+1y0=; b=WyZnMqrhqWsY
 TZYiCzPKnpmR0qOYZIpkXE3gR0EEy3ftymgCSzZIbO3nRLzoDosro5iGkRN1QI1Q1G6OSfxFLkwpR
 hLiKjb8L1RPd/u1tyQMxI/iiOY2x4OfVMHiqh5+tIZHh2HjwhuUPftDG3tcKTh/x8JZtC+IQ7kHeX
 KvKnkw225pLbX3XEeCeQbk1GgQSTWSwYmLz3O8NJko+x91ZEds5s6aAlkqempi/JuBFQVkBarbwOP
 LieZBfYpLCefjstcacLcE82GpdqFGTLddgwxyHLa3pfEwXb7SqVtv2rbFJDwFVeYzNMhk+L6kq3Ug
 7nI9wKdiD/20QSOFZ23f1w==;
Date: Wed, 23 Aug 2023 20:58:14 +0300
Message-Id: <83y1i1r689.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Ihor Radchenko <yantar92@HIDDEN>
In-Reply-To: <874jkq87jl.fsf@localhost> (message from Ihor Radchenko on Wed,
 23 Aug 2023 08:52:30 +0000)
Subject: Re: bug#65451: 30.0.50; `after-change-functions' are not triggered
 in the same order the changes are made
References: <871qfv2zlk.fsf@localhost> <83a5ujtgfo.fsf@HIDDEN>
 <87jztn1c5x.fsf@localhost> <834jkrters.fsf@HIDDEN>
 <87v8d7i48y.fsf@localhost> <83ttsrrroo.fsf@HIDDEN> <874jkq87jl.fsf@localhost>
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 65451
Cc: casouri@HIDDEN, 65451 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: Ihor Radchenko <yantar92@HIDDEN>
> Cc: casouri@HIDDEN, 65451 <at> debbugs.gnu.org
> Date: Wed, 23 Aug 2023 08:52:30 +0000
> 
> To update the old AST we need to know which regions in the old AST (and
> old buffer text) were edited and their change in length. Then, we (1)
> remove elements in AST that are structurally affected by the edit;
> (2) shift elements after the edit that are not structurally affected but
> whose boundaries must be adjusted to accommodate for the buffer length
> being changed; (3) re-parse buffer text (after the edit) and fill the
> gap in the AST.
> 
> For step (1), it is critical to have information about changed regions
> in the same order the edits happen. And, in theory, we might utilize
> before-change-functions to obtain this information (assuming that
> before-change-functions are always called in the same order the edits
> happen). But, unfortunately, before-change-functions are too noisy for
> this purpose because, for example, simply altering text properties also
> triggers before-change-functions, while no actual change in buffer text
> occurs in such scenario and re-parsing AST is unnecessary. So, we
> instead have to use after-change-functions and work out the original
> changed region boundaties using
> beg_changed end_changed pre-change_lentgh ->
> beg_before_change = beg_changed;
> end_before_change = end_changed - pre-change_length
> This calculation of the original changed region becomes incorrect when
> the order of after-change-functions is not the same as the editing order
> - this bug report.

If these measures still don't help you enough, perhaps the conclusion
is that it isn't feasible to implement text parsers in Lisp, at least
as long as you want all those micro-optimizations of knowing exactly
which parts of the buffer text were modified (as opposed to only know
how many characters at the beginning and at the end of the buffer
remain unchanged, and reparse the rest).  Maybe it must be done in C,
if we want Emacs to remain a relatively safe environment.

> Indeed. As I tried to explain, the problem for me is not the granularity
> of changes, but their ordering.

I still don't understand why the ordering matters, but I do know that
you cannot rely on it, and I hope I explained why.
Information forwarded to bug-gnu-emacs@HIDDEN:
bug#65451; Package emacs. Full text available.

Message received at 65451 <at> debbugs.gnu.org:


Received: (at 65451) by debbugs.gnu.org; 23 Aug 2023 08:52:15 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Wed Aug 23 04:52:15 2023
Received: from localhost ([127.0.0.1]:32791 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qYjb4-0001h2-Nx
	for submit <at> debbugs.gnu.org; Wed, 23 Aug 2023 04:52:15 -0400
Received: from mout01.posteo.de ([185.67.36.65]:39245)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <yantar92@HIDDEN>) id 1qYjb1-0001gk-Ow
 for 65451 <at> debbugs.gnu.org; Wed, 23 Aug 2023 04:52:13 -0400
Received: from submission (posteo.de [185.67.36.169]) 
 by mout01.posteo.de (Postfix) with ESMTPS id 75AC124002A
 for <65451 <at> debbugs.gnu.org>; Wed, 23 Aug 2023 10:52:02 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;
 t=1692780722; bh=SvN7BCt4RuFOItFbl44HxBrmYVsFJrcEP47qHxr4k94=;
 h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:From;
 b=BVviGfpdmNbUYKHSImIZoElf8kmmzi9iaVprI3dCUC12IGFvHfxIqIxwTT2VM2G7r
 WwQ00SNbLlylg2IwOy1gd23konJ2B1S/h25gJsbAipvSVz0Qxythveap0PlzKZ09zO
 aMo2TDGSYz5R2cndOLN92QLOgbtGmNQ9x69lc2gRLpcnQ9l+OJI01ssAfj1R4TMVx0
 NuB3DG++r/T38oPch/fTZ0XmNlxyq+p7iGJl6vlOYQmPsJyzJcNagRyOxW9KY347/F
 0ZxYson1Im4No51rBg9nsccsTtInQDU5qao+OF3eJ1Dz3ZrBwVxyLoPe63oLYi1sIR
 AVXWumvs/SxAA==
Received: from customer (localhost [127.0.0.1])
 by submission (posteo.de) with ESMTPSA id 4RW0MN2KxSz6tsj;
 Wed, 23 Aug 2023 10:52:00 +0200 (CEST)
From: Ihor Radchenko <yantar92@HIDDEN>
To: Eli Zaretskii <eliz@HIDDEN>
Subject: Re: bug#65451: 30.0.50; `after-change-functions' are not triggered
 in the same order the changes are made
In-Reply-To: <83ttsrrroo.fsf@HIDDEN>
References: <871qfv2zlk.fsf@localhost> <83a5ujtgfo.fsf@HIDDEN>
 <87jztn1c5x.fsf@localhost> <834jkrters.fsf@HIDDEN>
 <87v8d7i48y.fsf@localhost> <83ttsrrroo.fsf@HIDDEN>
Date: Wed, 23 Aug 2023 08:52:30 +0000
Message-ID: <874jkq87jl.fsf@localhost>
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 65451
Cc: casouri@HIDDEN, 65451 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Eli Zaretskii <eliz@HIDDEN> writes:

>> Org wants to do the same thing tree-sitter does - keep parsed AST in
>> sync with buffer modifications without having to re-parse the whole
>> buffer. So, we basically need the same information tree-sitter needs -
>> the sequence of buffer text changes, in their order.
>
> We don't expose the data you want to tree-sitter in Lisp.  What is
> exposed to Lisp are the parser and parse-tree objects that we build
> (in C) based on tree-sitter parsing results.

I understand. The difference is that Org mode tries to implement the
tree sitter algorithm (not exactly, but similar) in Elisp.

> ... When the buffer is
> modified, the information about the modifications is used internally
> by Emacs, in C code, to find and update the relevant parsers, and for
> that we call the tree-sitter functions involved in this process.  See
> the function treesit_record_change which does that, and which is
> called from C when buffer text changes in a way relevant to treesit.el
> functionalities.  (Note that some changes of buffer text are not
> visible even to tree-sitter, because we decided they are not relevant,
> for now.)

I understand. Org also does not need every single change. The changes
that, when combined, do not affect the buffer text are not important to
Org as well. In fact, Org even merge subsequent edits in the same, or
intersecting region of text. For example, if no queries to Org AST are
done between edits,

Foo bar<point> baz ->
Foo bar|new text|<point> baz ->
Foo bar|new text| and more|<point> baz

is simply treated as a single edit
Foo bar<point> baz -> Foo bar|new text| and more|<point> baz

> If tracking markers is not enough, then I wonder how the information
> from the lower levels, which is basically the same but noisier, will
> be able to help you.

We do not really need information from lower levels.

At every moment of time, Org parser maintains a linkage between buffer
text and its parsed AST. After a buffer edit, we need to know how to
update the AST corresponding to the last known buffer text state to the
new (current) text state.

To update the old AST we need to know which regions in the old AST (and
old buffer text) were edited and their change in length. Then, we (1)
remove elements in AST that are structurally affected by the edit;
(2) shift elements after the edit that are not structurally affected but
whose boundaries must be adjusted to accommodate for the buffer length
being changed; (3) re-parse buffer text (after the edit) and fill the
gap in the AST.

For step (1), it is critical to have information about changed regions
in the same order the edits happen. And, in theory, we might utilize
before-change-functions to obtain this information (assuming that
before-change-functions are always called in the same order the edits
happen). But, unfortunately, before-change-functions are too noisy for
this purpose because, for example, simply altering text properties also
triggers before-change-functions, while no actual change in buffer text
occurs in such scenario and re-parsing AST is unnecessary. So, we
instead have to use after-change-functions and work out the original
changed region boundaties using
beg_changed end_changed pre-change_lentgh ->
beg_before_change = beg_changed;
end_before_change = end_changed - pre-change_length
This calculation of the original changed region becomes incorrect when
the order of after-change-functions is not the same as the editing order
- this bug report.

Step (2) is technically not necessary if we use markers, but we cannot
do it yet for performance reasons. And without markers, we again need to
know the change in length of th edited region:
end_changed - beg_changed - pre-change_lentgh.
This calculation is also incorrect when after-change-function is not the
same as the editing order.

All the above information does not have to be granular. We are good as
long as nothing queries the AST between the edits. For example, internal
details of edit sequence done by replace-match are not important because
Org AST will not be queried from Emacs C internals.

>> I hope that we can solve this issue one way or another. This currently
>> breaks the very core functionality of Org. Every part of Org relies on
>> it to obtain reasonable performance. Prior to using cache, we had orders
>> of magnitude slowdowns.
>
> If you can arrange your design such that Lisp sees only AST-specific
> objects affected by the modifications in buffer text, then I believe
> we will have a good chance of finding a satisfactory solution.  If
> that requires to have some of your code in C (preferably, generalized
> to some extent), then so be it.

The problem is that whether an edit requires re-parsing an AST object or
not depends on the object type. For example,

:DRAWER:
Text inside
drawer.
:END:

Corresponds to the following simplified AST:

(drawer (:name "DRAWER") (paragraph))

Then, changing it to

:DR:
...
:END:

will require updating 'drawer' AST node:

(drawer (:name "DR") (paragraph))

while 

:DRAWER:
Text inside
drawer.
:ND:

will invalidate drawer and turn the whole thing into

(paragraph)

and 

:DRAWER:
Text inside
<more text>
drawer.
:END:

will not affect the drawer, just inner paragraph.

Further,

:DRAWER:
Text inside
* Heading
drawer.
:END:

will split the whole thing into

(paragraph) (heading (paragraph))

Basically, I do not see an obvious way to abstract AST away into C
without implementing Org parser in C as well.
With an exception of exposing treesit_record_change to Elisp, so that we
can get information about region before and after the edit when updating
the AST - that's what I had in mind in
https://yhetil.org/emacs-devel/83tu8jq2vl.fsf@HIDDEN/ We would need
something like after-change-text-functions that will only trigger after
actual text edits and provide information about region boundaries before
and after the edit.

> Moreover, I think the solution you think you want you actually _don't_
> want, because it will overwhelm you with changes that are not relevant
> to your purposes.  You can see a clear evidence to that in the fact
> that treesit_record_change is called only in several strategical
> places, not everywhere where we change buffer text, and not at the
> lowest level of such changes.  There's a reason to that.

Indeed. As I tried to explain, the problem for me is not the granularity
of changes, but their ordering.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>
Information forwarded to bug-gnu-emacs@HIDDEN:
bug#65451; Package emacs. Full text available.

Message received at 65451 <at> debbugs.gnu.org:


Received: (at 65451) by debbugs.gnu.org; 22 Aug 2023 16:02:22 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Aug 22 12:02:22 2023
Received: from localhost ([127.0.0.1]:60238 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qYTpm-0003dX-Cc
	for submit <at> debbugs.gnu.org; Tue, 22 Aug 2023 12:02:22 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:48366)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qYTpj-0003dK-Pj
 for 65451 <at> debbugs.gnu.org; Tue, 22 Aug 2023 12:02:21 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qYTpb-0005he-KP; Tue, 22 Aug 2023 12:02:11 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=2K/2TiaMWvdYUyvbioH9vx7EnJTgkxDBOaZR8PWhHLw=; b=Bd4yINZ5t9tf
 /DUO7L2Zoz9s/t0JwDMtWHQ3qmZE7K21JaxtJfc2NvJRRQnP/HsraRXadokrQJm2GVH7uzaJZ4A9K
 Z8gIc3tXueJQLTStVDJTsZVcbH1qKZ50YwFjFwyaUHDDtD3dnwuvBnAR2kCo6ZFlkN/eKWYn5vZWR
 3O4xf/cauHMobgMfzjuJDJxHN2qaVFuZto17irKZJ5Ge8yCOqkqL0CKUomldZeViqm+T7VAXUX/m4
 C1vasxZyAwal10XhWaGyn1J4YLHsZOM/mEEzbjzSi4cGV/NJm9gIbVuJKUHvC4u6llxacy4CEnxQT
 7XVPk7LFgaH22of28M+jyg==;
Date: Tue, 22 Aug 2023 19:02:31 +0300
Message-Id: <83ttsrrroo.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Ihor Radchenko <yantar92@HIDDEN>
In-Reply-To: <87v8d7i48y.fsf@localhost> (message from Ihor Radchenko on Tue,
 22 Aug 2023 13:41:17 +0000)
Subject: Re: bug#65451: 30.0.50; `after-change-functions' are not triggered
 in the same order the changes are made
References: <871qfv2zlk.fsf@localhost> <83a5ujtgfo.fsf@HIDDEN>
 <87jztn1c5x.fsf@localhost> <834jkrters.fsf@HIDDEN> <87v8d7i48y.fsf@localhost>
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 65451
Cc: casouri@HIDDEN, 65451 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: Ihor Radchenko <yantar92@HIDDEN>
> Cc: Yuan Fu <casouri@HIDDEN>, 65451 <at> debbugs.gnu.org
> Date: Tue, 22 Aug 2023 13:41:17 +0000
> 
> Eli Zaretskii <eliz@HIDDEN> writes:
> 
> >> Then, I'd like to point back to the previous discussion where I asked to
> >> expose to Elisp information about buffer changes available to
> >> tree-sitter.
> >> https://yhetil.org/emacs-devel/83tu8jq2vl.fsf@HIDDEN/
> >
> > I don't want to do that, sorry.  Not without a good understanding of
> > what exactly do you need from that and in what way.  If we will expose
> > anything, it will have to be the minimum possible exposure, not the
> > maximum, so I would like to understand this very well before I agree
> > to any change in this direction.
> 
> Org wants to do the same thing tree-sitter does - keep parsed AST in
> sync with buffer modifications without having to re-parse the whole
> buffer. So, we basically need the same information tree-sitter needs -
> the sequence of buffer text changes, in their order.

We don't expose the data you want to tree-sitter in Lisp.  What is
exposed to Lisp are the parser and parse-tree objects that we build
(in C) based on tree-sitter parsing results.  When the buffer is
modified, the information about the modifications is used internally
by Emacs, in C code, to find and update the relevant parsers, and for
that we call the tree-sitter functions involved in this process.  See
the function treesit_record_change which does that, and which is
called from C when buffer text changes in a way relevant to treesit.el
functionalities.  (Note that some changes of buffer text are not
visible even to tree-sitter, because we decided they are not relevant,
for now.)

> Note that the markers discussed in the thread I linked are not
> sufficient. When editing near AST node boundaries, even if the
> boundaries are represented by markers, we have to re-parse the AST
> around to account for the possible structural changes. So, information
> about buffer edits is still required.

If tracking markers is not enough, then I wonder how the information
from the lower levels, which is basically the same but noisier, will
be able to help you.

> >> In fact, I am not sure if tree-sitter will behave correctly if it is
> >> signaled changes in incorrect order.
> >
> > I will defer to Yuan, but tree-sitter doesn't use these hooks, we call
> > its functions directly from insdel.c where needed.  This makes sense
> > for a library to which we link and whose interface code we control,
> > but giving such access to Lisp (and Org on top of that) is out of the
> > question.  We don't even give such access to modules.
> 
> I hope that we can solve this issue one way or another. This currently
> breaks the very core functionality of Org. Every part of Org relies on
> it to obtain reasonable performance. Prior to using cache, we had orders
> of magnitude slowdowns.

If you can arrange your design such that Lisp sees only AST-specific
objects affected by the modifications in buffer text, then I believe
we will have a good chance of finding a satisfactory solution.  If
that requires to have some of your code in C (preferably, generalized
to some extent), then so be it.

You see, I think the buffer-change hooks we have are already too much:
Lisp programs abuse them all the time (you can see a good example in
the bug which I mentioned up-thread, and which led to the change you
are now complaining about).  Doing more of that is not very wise, to
say the least.

Moreover, I think the solution you think you want you actually _don't_
want, because it will overwhelm you with changes that are not relevant
to your purposes.  You can see a clear evidence to that in the fact
that treesit_record_change is called only in several strategical
places, not everywhere where we change buffer text, and not at the
lowest level of such changes.  There's a reason to that.
Information forwarded to bug-gnu-emacs@HIDDEN:
bug#65451; Package emacs. Full text available.

Message received at 65451 <at> debbugs.gnu.org:


Received: (at 65451) by debbugs.gnu.org; 22 Aug 2023 13:41:00 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Aug 22 09:41:00 2023
Received: from localhost ([127.0.0.1]:58771 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qYRcx-0007Z4-Ms
	for submit <at> debbugs.gnu.org; Tue, 22 Aug 2023 09:41:00 -0400
Received: from mout02.posteo.de ([185.67.36.66]:45021)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <yantar92@HIDDEN>) id 1qYRcu-0007Yq-NY
 for 65451 <at> debbugs.gnu.org; Tue, 22 Aug 2023 09:40:57 -0400
Received: from submission (posteo.de [185.67.36.169]) 
 by mout02.posteo.de (Postfix) with ESMTPS id EB658240101
 for <65451 <at> debbugs.gnu.org>; Tue, 22 Aug 2023 15:40:47 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;
 t=1692711648; bh=BURI7u/uy+X+xSSYTVB9f3QwLoyLrQcrWfLwsuVQNDc=;
 h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:From;
 b=WxInbUb1IX5UeYhI5F3YtU2lU+mYelxZmkj78DFf77n2hj+aG3i/OWGERycVsdgbO
 4YJgniFrDzatzLUb5Slp7WODOdNP5WHlq8SpAghbT/FriHZ1x2rwOi5/IPlVDcM56j
 rHsSmz5XgWy2QkhASH1nkfCegLVvvr3ZUb4wkOC1ztdDE/r3IcpJnTqUEnKPWhvGbq
 wHGiyeMZgdiBJUtek5zeWh8AZMGU3/oG2a6IA6v7vq0hMXVESErvZX67U9NLJ9NzZ1
 dVtW/GD5W5hXbbXqkX1/v2L8KEUdbrReXIK81bgoJK0SfunR/ao145QRM7beg2eTfq
 uYRJ5IJk+2qkw==
Received: from customer (localhost [127.0.0.1])
 by submission (posteo.de) with ESMTPSA id 4RVVq30l2zz6tm4;
 Tue, 22 Aug 2023 15:40:46 +0200 (CEST)
From: Ihor Radchenko <yantar92@HIDDEN>
To: Eli Zaretskii <eliz@HIDDEN>
Subject: Re: bug#65451: 30.0.50; `after-change-functions' are not triggered
 in the same order the changes are made
In-Reply-To: <834jkrters.fsf@HIDDEN>
References: <871qfv2zlk.fsf@localhost> <83a5ujtgfo.fsf@HIDDEN>
 <87jztn1c5x.fsf@localhost> <834jkrters.fsf@HIDDEN>
Date: Tue, 22 Aug 2023 13:41:17 +0000
Message-ID: <87v8d7i48y.fsf@localhost>
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 65451
Cc: Yuan Fu <casouri@HIDDEN>, 65451 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Eli Zaretskii <eliz@HIDDEN> writes:

>> Then, I'd like to point back to the previous discussion where I asked to
>> expose to Elisp information about buffer changes available to
>> tree-sitter.
>> https://yhetil.org/emacs-devel/83tu8jq2vl.fsf@HIDDEN/
>
> I don't want to do that, sorry.  Not without a good understanding of
> what exactly do you need from that and in what way.  If we will expose
> anything, it will have to be the minimum possible exposure, not the
> maximum, so I would like to understand this very well before I agree
> to any change in this direction.

Org wants to do the same thing tree-sitter does - keep parsed AST in
sync with buffer modifications without having to re-parse the whole
buffer. So, we basically need the same information tree-sitter needs -
the sequence of buffer text changes, in their order.

Note that the markers discussed in the thread I linked are not
sufficient. When editing near AST node boundaries, even if the
boundaries are represented by markers, we have to re-parse the AST
around to account for the possible structural changes. So, information
about buffer edits is still required.

>> In fact, I am not sure if tree-sitter will behave correctly if it is
>> signaled changes in incorrect order.
>
> I will defer to Yuan, but tree-sitter doesn't use these hooks, we call
> its functions directly from insdel.c where needed.  This makes sense
> for a library to which we link and whose interface code we control,
> but giving such access to Lisp (and Org on top of that) is out of the
> question.  We don't even give such access to modules.

I hope that we can solve this issue one way or another. This currently
breaks the very core functionality of Org. Every part of Org relies on
it to obtain reasonable performance. Prior to using cache, we had orders
of magnitude slowdowns.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>
Information forwarded to bug-gnu-emacs@HIDDEN:
bug#65451; Package emacs. Full text available.

Message received at 65451 <at> debbugs.gnu.org:


Received: (at 65451) by debbugs.gnu.org; 22 Aug 2023 12:58:30 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Aug 22 08:58:30 2023
Received: from localhost ([127.0.0.1]:58731 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qYQxp-0006Vt-Jm
	for submit <at> debbugs.gnu.org; Tue, 22 Aug 2023 08:58:30 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:51164)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qYQxo-0006Vg-8M
 for 65451 <at> debbugs.gnu.org; Tue, 22 Aug 2023 08:58:29 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qYQxg-0003Ub-4L; Tue, 22 Aug 2023 08:58:20 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=jtKuBnTiqmm/KuzD/SfB+7r+n+fZ8C8abeAWF37rcI8=; b=TcGPdCTRdegv
 wjEeUkcu3inx9zYn+Y/oMjQpWQCsC8G4LrkQozpRU9SkgxNxLpzJHfgPcr4t3NxicHLjrqm1ME1Gm
 BjWtqFmPl2hafHvP1/vT4CWpKqetv86xtR9zapqCYdrt69eHBgrS5GxO6k4VkBb0UwJ4pAl5lPy0Y
 xrEWMEuETHDfq9CubFxRXsjIdMN+f8fQTR8dG/ygJYeQ7V+TlxoanHAof6u6fuJ+VZ9/TaPgZCgyQ
 JTUGDJCUYKizrGsDoiOIreBR3pli7cnkwDeIb+fWiycGi2ExFSrBGn2h/J4SRLM2sdXpzEtG79OoK
 Azgq1U4Ij4JgqBqf1JKM7Q==;
Date: Tue, 22 Aug 2023 15:58:31 +0300
Message-Id: <834jkrters.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Ihor Radchenko <yantar92@HIDDEN>, Yuan Fu <casouri@HIDDEN>
In-Reply-To: <87jztn1c5x.fsf@localhost> (message from Ihor Radchenko on Tue,
 22 Aug 2023 12:42:18 +0000)
Subject: Re: bug#65451: 30.0.50; `after-change-functions' are not triggered
 in the same order the changes are made
References: <871qfv2zlk.fsf@localhost> <83a5ujtgfo.fsf@HIDDEN>
 <87jztn1c5x.fsf@localhost>
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 65451
Cc: 65451 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: Ihor Radchenko <yantar92@HIDDEN>
> Cc: 65451 <at> debbugs.gnu.org
> Date: Tue, 22 Aug 2023 12:42:18 +0000
> 
> Eli Zaretskii <eliz@HIDDEN> writes:
> 
> >> The order of after-change notifications thus do not correspond to the
> >> order of buffer changes. Org mode is relying upon the correct change
> >> order to update parser cache with buffer modifications.
> >
> > I think Org mode is relying on something it should not.
> 
> Then, I'd like to point back to the previous discussion where I asked to
> expose to Elisp information about buffer changes available to
> tree-sitter.
> https://yhetil.org/emacs-devel/83tu8jq2vl.fsf@HIDDEN/

I don't want to do that, sorry.  Not without a good understanding of
what exactly do you need from that and in what way.  If we will expose
anything, it will have to be the minimum possible exposure, not the
maximum, so I would like to understand this very well before I agree
to any change in this direction.

> If there is no guarantee that buffer modifications are not signaled in
> order, on-the-fly updates of the parsed AST will become impossible for
> Org, even if using markers (as you suggested in the linked thread).
> 
> In fact, I am not sure if tree-sitter will behave correctly if it is
> signaled changes in incorrect order.

I will defer to Yuan, but tree-sitter doesn't use these hooks, we call
its functions directly from insdel.c where needed.  This makes sense
for a library to which we link and whose interface code we control,
but giving such access to Lisp (and Org on top of that) is out of the
question.  We don't even give such access to modules.
Information forwarded to bug-gnu-emacs@HIDDEN:
bug#65451; Package emacs. Full text available.

Message received at 65451 <at> debbugs.gnu.org:


Received: (at 65451) by debbugs.gnu.org; 22 Aug 2023 12:42:05 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Aug 22 08:42:04 2023
Received: from localhost ([127.0.0.1]:58691 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qYQhw-00061p-DW
	for submit <at> debbugs.gnu.org; Tue, 22 Aug 2023 08:42:04 -0400
Received: from mout01.posteo.de ([185.67.36.65]:34675)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <yantar92@HIDDEN>) id 1qYQhr-00061B-Ev
 for 65451 <at> debbugs.gnu.org; Tue, 22 Aug 2023 08:42:02 -0400
Received: from submission (posteo.de [185.67.36.169]) 
 by mout01.posteo.de (Postfix) with ESMTPS id 2E67D240027
 for <65451 <at> debbugs.gnu.org>; Tue, 22 Aug 2023 14:41:49 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;
 t=1692708110; bh=Nc1erb2eNyMo8/JHNqiWJmHqKqddDM8TxFGvxkcAVAI=;
 h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:
 Content-Transfer-Encoding:From;
 b=AnE7qM5OWWGxn3pcAj4P+szehA5u9zZicbotXUj8Cs19GErefkaD3kEjO5GbGUkEZ
 iZRbeN2JoHbIg2LFzbFdM4jsiEdwwSbFtttzLyP1D8tFx3sAYDPU7DMVo8eA7DYCHp
 ZdFqQcuJIUH8iOGsyrfs4FulBV2csxLSQfrAMpP336CSRBfkqhn65eLyddGBPDRgQ+
 R+PrBjs8sFIEpkbZ7J2vqtWD4kibj1Xny5GhUiXEnSIoSaRP8VPPVMaSG7xsbgBIcG
 Qs7UbahjTft/EO1HX3vhnZaBH+Lg8ChKy8YblsQOlVpw2xzkK76pnwc9+R5jby4q6I
 DVDMN6rnxuUeg==
Received: from customer (localhost [127.0.0.1])
 by submission (posteo.de) with ESMTPSA id 4RVTW05TFPz9rxD;
 Tue, 22 Aug 2023 14:41:48 +0200 (CEST)
From: Ihor Radchenko <yantar92@HIDDEN>
To: Eli Zaretskii <eliz@HIDDEN>
Subject: Re: bug#65451: 30.0.50; `after-change-functions' are not triggered
 in the same order the changes are made
In-Reply-To: <83a5ujtgfo.fsf@HIDDEN>
References: <871qfv2zlk.fsf@localhost> <83a5ujtgfo.fsf@HIDDEN>
Date: Tue, 22 Aug 2023 12:42:18 +0000
Message-ID: <87jztn1c5x.fsf@localhost>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 65451
Cc: 65451 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Eli Zaretskii <eliz@HIDDEN> writes:

>> The order of after-change notifications thus do not correspond to the
>> order of buffer changes. Org mode is relying upon the correct change
>> order to update parser cache with buffer modifications.
>
> I think Org mode is relying on something it should not.

Then, I'd like to point back to the previous discussion where I asked to
expose to Elisp information about buffer changes available to
tree-sitter.
https://yhetil.org/emacs-devel/83tu8jq2vl.fsf@HIDDEN/

If there is no guarantee that buffer modifications are not signaled in
order, on-the-fly updates of the parsed AST will become impossible for
Org, even if using markers (as you suggested in the linked thread).

In fact, I am not sure if tree-sitter will behave correctly if it is
signaled changes in incorrect order.

https://tree-sitter.github.io/tree-sitter/using-parsers#advanced-parsing
says

    In applications like text editors, you often need to re-parse a file
    after its source code has changed. Tree-sitter is designed to support
    this use case efficiently. There are two steps required. First, you must
    edit the syntax tree, which adjusts the ranges of its nodes so that they
    stay in sync with the code.
=20=20=20=20
    typedef struct {
      uint32_t start_byte;
      uint32_t old_end_byte;
      uint32_t new_end_byte;
      TSPoint start_point;
      TSPoint old_end_point;
      TSPoint new_end_point;
    } TSInputEdit;
=20=20=20=20
    void ts_tree_edit(TSTree *, const TSInputEdit *);

Note how the API is similar to `after-change-functions' - it expects
edited region boundaries before/after the edit.

If Emacs feeds the following to tree-sitter
=E2=9B=94 Warning (emacs): After: 1278 1288 delta: 0
=E2=9B=94 Warning (emacs): After: 1278 1288 delta: 7
, the first "edit" query will apply to wrong range in the cached AST.

--=20
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>
Information forwarded to bug-gnu-emacs@HIDDEN:
bug#65451; Package emacs. Full text available.

Message received at 65451 <at> debbugs.gnu.org:


Received: (at 65451) by debbugs.gnu.org; 22 Aug 2023 12:22:30 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Aug 22 08:22:30 2023
Received: from localhost ([127.0.0.1]:58684 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qYQP0-0005XG-Bl
	for submit <at> debbugs.gnu.org; Tue, 22 Aug 2023 08:22:30 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:59382)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qYQOy-0005X1-DX
 for 65451 <at> debbugs.gnu.org; Tue, 22 Aug 2023 08:22:29 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qYQOq-0001Ue-0R; Tue, 22 Aug 2023 08:22:20 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=MIME-version:References:Subject:In-Reply-To:To:From:
 Date; bh=B/zX5hQzTxDPiXft1nUmQCxejOPyfMfN5vKUCXqDFbQ=; b=Bec3XbNDTm9cZdedZras
 fdjT04rXHzPJbSZgfkvlWALk1da09cfOdPfrgpIpyhBL82eWUns7Rim3VWkAzFktpT6LSIHlScigk
 ZyD516yE2y6s/QOq5pz/6Kmx0Kp3e2q6pCkJEzMrZ430A8OFK5hjrlu8bY+xOP9Fmy8htm05+AUX3
 ApS7ORRK1S+iHK/3/Mmnomxzl4rwY8MQYng65JRuZ6qCkuagEfXSJpa9ZXp1WcQ50kSmYqX+e4wzQ
 rAOO0+ZBOOMsW3fB1OHiPUT6AlDfSOLotQlUpZzdRFkhNjUk7sGXTj/OyyB+L6z0smNZejUWqFJoo
 EEws1gw92cldFQ==;
Date: Tue, 22 Aug 2023 15:22:35 +0300
Message-Id: <83a5ujtgfo.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Ihor Radchenko <yantar92@HIDDEN>
In-Reply-To: <871qfv2zlk.fsf@localhost> (message from Ihor Radchenko on Tue,
 22 Aug 2023 09:30:47 +0000)
Subject: Re: bug#65451: 30.0.50;
 `after-change-functions' are not triggered in the same order the
 changes are made
References: <871qfv2zlk.fsf@localhost>
MIME-version: 1.0
Content-type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 65451
Cc: 65451 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: Ihor Radchenko <yantar92@HIDDEN>
> Date: Tue, 22 Aug 2023 09:30:47 +0000
> 
> The last bit of the messages corresponds to dabbrev expansion of "UTF"
> to "utf-8-unix":
> 
> ⛔ Warning (emacs): Before: 1278 1281
> ⛔ Warning (emacs): Before: 1278 1288
> ⛔ Warning (emacs): After: 1278 1288 delta: 0
> ⛔ Warning (emacs): After: 1278 1288 delta: 7
> 
> Note how "After: 1278 1288 delta: 0" reports a change of "utf-8-unix"
> that did not alter the buffer text size. It is trigerred _before_
> "After: 1278 1288 delta: 7" that corresponds to replacing "UTF" with
> "utf-8-unix".

The inner Before+After is because replace-match also changes the
letter-case of the replaced text (to match the letter-case of the
original).

> The order of after-change notifications thus do not correspond to the
> order of buffer changes. Org mode is relying upon the correct change
> order to update parser cache with buffer modifications.

I think Org mode is relying on something it should not.  This
particular use case aside, Emacs is allowed to call a function that
changes the buffer from a function that itself changes the buffer, and
it is allowed to call that inner function _before_ it did all the
changes it intended to do.  In such cases you will see Before+After
pairs embedded inside an outer Before+After pair, because most
functions which end up calling these hooks try not to call them too
often, so they generally call the before-hook once at the beginning
and the after-hook once near the end.  IOW, we try to report all of
the changes in one go, instead of reporting them one small chunk
(perhaps even one character) at a time.  And that inevitably leads to
situations like this one.  Which is perfectly normal, from my POV, and
not a bug at all.  I'd hate to make our code more complex, let alone
slower, just to satisfy the reliance on the perfect pairing like you
do.

(We've been through this in other discussions: Org uses these hooks
for purposes they were not designed for.  The ELisp manual even warns
about such assumptions.)

> The same recipe using Emacs 27 yields the correct order

AFAIR, Emacs 27 had quite a few bugs in this area, which were fixed in
development since then.  See bug#42424, for example.
Information forwarded to bug-gnu-emacs@HIDDEN:
bug#65451; Package emacs. Full text available.

Message received at submit <at> debbugs.gnu.org:


Received: (at submit) by debbugs.gnu.org; 22 Aug 2023 09:30:35 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Aug 22 05:30:35 2023
Received: from localhost ([127.0.0.1]:58586 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qYNic-0006PN-TF
	for submit <at> debbugs.gnu.org; Tue, 22 Aug 2023 05:30:35 -0400
Received: from lists.gnu.org ([2001:470:142::17]:45354)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <yantar92@HIDDEN>) id 1qYNia-0006P8-Fs
 for submit <at> debbugs.gnu.org; Tue, 22 Aug 2023 05:30:33 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10])
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <yantar92@HIDDEN>)
 id 1qYNiS-0007Wr-62
 for bug-gnu-emacs@HIDDEN; Tue, 22 Aug 2023 05:30:24 -0400
Received: from mout02.posteo.de ([185.67.36.66])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <yantar92@HIDDEN>)
 id 1qYNiO-0008Pw-VB
 for bug-gnu-emacs@HIDDEN; Tue, 22 Aug 2023 05:30:23 -0400
Received: from submission (posteo.de [185.67.36.169]) 
 by mout02.posteo.de (Postfix) with ESMTPS id 4DF16240101
 for <bug-gnu-emacs@HIDDEN>; Tue, 22 Aug 2023 11:30:18 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;
 t=1692696618; bh=PK3Oda/wFPw5GWyEc+PUBvXRzmEm0++n+0AnyRAH5Wc=;
 h=From:To:Subject:Date:Message-ID:MIME-Version:
 Content-Transfer-Encoding:From;
 b=XJ7sRH4fIHqcPw64YAnFVly5smAxm3n++Zkxb8x0Kk795hx4IUIF2bFqQsJU+O94L
 M50k6uBJeOVOAODkv15ObMIJSPue/wuavxd4HBwNcj8OpP4LJlhY6Va0kaxbeYoPAt
 FWKCzqCpnH9hXV/2N4m6CH5m+ts61kH6u2bmMTo2WjQoMdI9SItYISfrCWlHfqMEEp
 Rggw2mhfofg7T7YzSxPJayzpTisJEsiI+ytf/EUZpjGSY8kfWYFtcII3dWZLdg6ymr
 ERI0DBby+tgT3GVCXRKOOYW4OLA3xTXQFbLiF/O01tNUfvrjYYdQcyRabdE+G6jFjm
 cc90GqJUiNIeA==
Received: from customer (localhost [127.0.0.1])
 by submission (posteo.de) with ESMTPSA id 4RVPG14vS7z6tw7
 for <bug-gnu-emacs@HIDDEN>; Tue, 22 Aug 2023 11:30:17 +0200 (CEST)
From: Ihor Radchenko <yantar92@HIDDEN>
To: bug-gnu-emacs@HIDDEN
Subject: 30.0.50; `after-change-functions' are not triggered in the same
 order the changes are made
Date: Tue, 22 Aug 2023 09:30:47 +0000
Message-ID: <871qfv2zlk.fsf@localhost>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Received-SPF: pass client-ip=185.67.36.66; envelope-from=yantar92@HIDDEN;
 helo=mout02.posteo.de
X-Spam_score_int: -43
X-Spam_score: -4.4
X-Spam_bar: ----
X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,
 DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1,
 RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001,
 SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-Spam-Score: 1.0 (+)
X-Debbugs-Envelope-To: submit
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.0 (/)


Consider the following reproducer:

1. Create /tmp/bug.el with the following contents

(defun my/setup ()
  (interactive)
  (defun my/before-change (beg end)
    (warn "Before: %d %d" beg end))
  (add-hook 'before-change-functions #'my/before-change nil 'local)

  (defun my/after-change (beg end pre)
    (warn "After: %d %d delta: %d" beg end (- end beg pre)))
  (add-hook 'after-change-functions #'my/after-change nil 'local))

2. Create /tmp/bug.org

https://www.gnu.org/software/emacs/manual/html_node/emacs/Mode-Line.html

-UUU: @**-   F2  UTF-8-demo.txt   Top   (1,0)     <N>   (Text s-/ WS Wrap)

| string | meaning                                  | note                 =
                                                |
|--------+------------------------------------------+----------------------=
------------------------------------------------|
| -      | input method                             |                      =
                                                |
| UUU    | coding system (keyboard terminal buffer) | (U utf-8-unix)       =
                                                |
| :      | end-of-line convention                   | (: LF) (/ CR) (\ CRLF=
)                                               |
| @      | emacsclient                              |                      =
                                                |
| **     | buffer status                            | (-- unmodified) (** m=
odified) (%% read-only) (%* read-only_modified) |
| -      | default-directory                        | (- local) (@ remote) =
                                                |
| F2     | frame name                               | (F2 the-2nd-frame)   =
                                                |
|        |                                          |                      =
                                                |

3. emacs -Q -l /tmp/bug.el /tmp/bug.org
4. M-x my/setup
5. Move point to

| F2     | frame name                               | (F2 the-2nd-frame)   =
                                                |
| <point>       |                                          |               =
                                                       |

6. Insert "UTF"
7. M-x dabbrev-expand
8. Observe the following in the *Warnings* buffer

=E2=9B=94 Warning (emacs): Before: 1278 1278
=E2=9B=94 Warning (emacs): After: 1278 1279 delta: 1
=E2=9B=94 Warning (emacs): Before: 1284 1285
=E2=9B=94 Warning (emacs): After: 1284 1284 delta: -1
=E2=9B=94 Warning (emacs): Before: 1279 1279
=E2=9B=94 Warning (emacs): After: 1279 1280 delta: 1
=E2=9B=94 Warning (emacs): Before: 1284 1285
=E2=9B=94 Warning (emacs): After: 1284 1284 delta: -1
=E2=9B=94 Warning (emacs): Before: 1280 1280
=E2=9B=94 Warning (emacs): After: 1280 1281 delta: 1
=E2=9B=94 Warning (emacs): Before: 1284 1285
=E2=9B=94 Warning (emacs): After: 1284 1284 delta: -1
=E2=9B=94 Warning (emacs): Before: 1278 1281
=E2=9B=94 Warning (emacs): Before: 1278 1288
=E2=9B=94 Warning (emacs): After: 1278 1288 delta: 0
=E2=9B=94 Warning (emacs): After: 1278 1288 delta: 7

The last bit of the messages corresponds to dabbrev expansion of "UTF"
to "utf-8-unix":

=E2=9B=94 Warning (emacs): Before: 1278 1281
=E2=9B=94 Warning (emacs): Before: 1278 1288
=E2=9B=94 Warning (emacs): After: 1278 1288 delta: 0
=E2=9B=94 Warning (emacs): After: 1278 1288 delta: 7

Note how "After: 1278 1288 delta: 0" reports a change of "utf-8-unix"
that did not alter the buffer text size. It is trigerred _before_
"After: 1278 1288 delta: 7" that corresponds to replacing "UTF" with
"utf-8-unix".

The order of after-change notifications thus do not correspond to the
order of buffer changes. Org mode is relying upon the correct change
order to update parser cache with buffer modifications.

The same recipe using Emacs 27 yields the correct order

<...>
Warning (emacs): Before: 1278 1281
Warning (emacs): After: 1278 1288 delta: 7
Warning (emacs): Before: 1278 1288
Warning (emacs): After: 1278 1288 delta: 0


In GNU Emacs 30.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version
 3.24.38, cairo version 1.17.8) of 2023-08-14 built on localhost
Repository revision: d483b38070120f17b1d00975081d27191d1deacc
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12101008
System Description: Gentoo Linux

--=20
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>
Acknowledgement sent to Ihor Radchenko <yantar92@HIDDEN>:
New bug report received and forwarded. Copy sent to bug-gnu-emacs@HIDDEN. Full text available.
Report forwarded to bug-gnu-emacs@HIDDEN:
bug#65451; Package emacs. Full text available.
Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.
Last modified: Thu, 24 Aug 2023 11:30:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.