Received: (at submit) by debbugs.gnu.org; 13 May 2018 07:32:12 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun May 13 03:32:12 2018
Received: from localhost ([127.0.0.1]:60116 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1fHlUK-0001F4-8N
for submit <at> debbugs.gnu.org; Sun, 13 May 2018 03:32:12 -0400
Received: from eggs.gnu.org ([208.118.235.92]:58068)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <andreas.roehler@HIDDEN>) id 1fHlUJ-0001Er-0r
for submit <at> debbugs.gnu.org; Sun, 13 May 2018 03:32:11 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
(envelope-from <andreas.roehler@HIDDEN>) id 1fHlUC-0004lJ-J7
for submit <at> debbugs.gnu.org; Sun, 13 May 2018 03:32:05 -0400
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level:
X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50 autolearn=disabled
version=3.3.2
Received: from lists.gnu.org ([2001:4830:134:3::11]:58346)
by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32)
(Exim 4.71) (envelope-from <andreas.roehler@HIDDEN>)
id 1fHlUC-0004lC-Fl
for submit <at> debbugs.gnu.org; Sun, 13 May 2018 03:32:04 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:49027)
by lists.gnu.org with esmtp (Exim 4.71)
(envelope-from <andreas.roehler@HIDDEN>) id 1fHlUB-0002Fc-1I
for bug-gnu-emacs@HIDDEN; Sun, 13 May 2018 03:32:04 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
(envelope-from <andreas.roehler@HIDDEN>) id 1fHlU7-0004jq-SG
for bug-gnu-emacs@HIDDEN; Sun, 13 May 2018 03:32:03 -0400
Received: from mout.kundenserver.de ([212.227.126.130]:38883)
by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_128_CBC_SHA1:16)
(Exim 4.71) (envelope-from <andreas.roehler@HIDDEN>)
id 1fHlU7-0004jL-I8
for bug-gnu-emacs@HIDDEN; Sun, 13 May 2018 03:31:59 -0400
Received: from [192.168.178.35] ([188.103.30.197]) by mrelayeu.kundenserver.de
(mreue006 [212.227.15.167]) with ESMTPSA (Nemesis) id
0MPeS5-1fD6ej3sUD-004kdl for <bug-gnu-emacs@HIDDEN>; Sun, 13 May 2018
09:31:57 +0200
Subject: Re: bug#31290: Fundamental bugs in syntax-propertize
To: bug-gnu-emacs@HIDDEN
References: <20180427210859.GA6023@ACM>
<e8658e46-da92-6310-7b1e-33c55f6255d2@HIDDEN> <20180512112612.GA4766@ACM>
From: =?UTF-8?Q?Andreas_R=c3=b6hler?= <andreas.roehler@HIDDEN>
Message-ID: <79bd61c9-1d9c-d2f4-9206-3623a0553ff1@HIDDEN>
Date: Sun, 13 May 2018 09:33:20 +0200
User-Agent: Mozilla/5.0 (X11; Linux i686; rv:52.0) Gecko/20100101
Thunderbird/52.7.0
MIME-Version: 1.0
In-Reply-To: <20180512112612.GA4766@ACM>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
X-Provags-ID: V03:K1:4xXONJZrL2d6kiMkNZaUsOONQr6go7o5lUXxB99/1i4sr/elsDP
EjXZ3/4Wi3N3/KZeg4fKcCcGoTP5MO0alvUZuIUwEDLSWrgQMad0cHk9iROW2+J+sq3/5n/
AHcYxPnfVqAi72F8E9++IUcX8webunvTT1zaMU7jPMks6ePqfnYJHsXtbTDrwLXyEkre3oJ
jrPcwTaRu9egEXyhJtmUA==
X-UI-Out-Filterresults: notjunk:1;V01:K0:vCBxJ2fpsPY=:HTM9dqgxmqPcnCQwbXQWif
+5TxWJGJ2dUjXGrE2Yx1BewEZIBSie67aIoK4F4IBqhmyKeoLeXbei7RbpMiHJVHVxRh4WG3N
LOuoiVHBhgab6khCAubVuvLBnd3+kZ4kh6/YkID0NWqf1uUB686EydgR4nZKYXrSDrjH3w7Tq
XW0/YoGi3kYCZdRx+63nqalSHjeoUdy8L68tTMeLMlmA9iH1bTzgwoA38RuqAcL/XlHdwJiaA
KQcWt1hkphNr7JfOI8IGkNJr5XzIb8UzSV0f7vPtZkaWRnCCT/t6geg4UceL+8Qe4smd3yJ//
pQQE3JsIwcw79qbHF3m9T6q2TGC3QWtMzwYnTstLQMBO55nBxOcrlu6Z6hkGtV7gzJwFlJmTr
WBQ2oxTSn4K3MPwyNuX52GkAFlalntq3dPPB1Qnn3QbKJRSD4JXEL3rqZZJgvYJBL34TPd0ki
xdN4g0xFIMpe2HZYsqWwnkDFlhPEsq+wYaTReZIRgQEDXjQZL/vBLJRlP7e6VYCn8hV5JJh5t
jGzUtJE894vyn/g67qKwVOaYmnIzROC0JwHtVLp4NEONnY6LRff87Wxv77Dj1/bX/9YSyooIa
SYZnGKgiRaFtmbGdXTQAxAWCoECIs2A2ZvJOJVpbKs/RUb01XJfqozaSv42sPwv+vhRE55ITh
pRCaDmxloiRiQmbuSb1JpF9as54uZeMnK2zWyxrm1Ch40wO3pm/gaiTqlURFIHxv0ScqwqpgM
lUuv38zXk8FRTNhL
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
[fuzzy]
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x
X-Received-From: 2001:4830:134:3::11
X-Spam-Score: -5.0 (-----)
X-Debbugs-Envelope-To: submit
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -6.0 (------)
On 12.05.2018 13:26, Alan Mackenzie wrote:
> Hello, Dmitry.
>
> On Tue, May 08, 2018 at 15:35:14 +0300, Dmitry Gutov wrote:
>> On 4/28/18 12:08 AM, Alan Mackenzie wrote:
>
>>> At least that would be true if syntax-propertize--done hadn't been
>>> prematurely and spuriously increased, crudely to prevent an infinite
>>> recursion, falsely indicating to the syntax-ppss infrastructure that the
>>> syntax-table properties have already been applied to the region (BEGIN
>>> END).
>
>>> .... but it should not call `syntax-ppss-flush-cache', ....
>
>>> Why not? Because syntax-ppss-flush-cache sets syntax-propertize--done
>>> back to its true value, allowing the wrongly allowed syntax-ppss calls at
>>> a later position to cause a recursive loop.
>
>> Maybe we should "allow" it to loop, in certain cases? Leaving it to be
>> the responsibility of the programmer, to make sure the result doesn't
>> infloop, even if these rules are violated.
>
> I'm not sure how this could work. We would need to formalise the rules
> very carefully, to avoid the need to read syntax.{c,el}'s source code.
>
>>> .... which means that it should not call `syntax-ppss' on some
>>> position and later modify the buffer on some earlier position.
>
>>> This is a bad restriction, because sometimes syntax-table properties can
>>> only be correctly determined by examining the syntax of later buffer
>>> positions. An example of this is giving the string-fence syntax-table
>>> text property to an unbalanced opening string quote, but not to correctly
>>> matched quotes.
>
>> I'm not exactly convinced by the given example (why would we use the
>> string-fence in that case?), but it might be better if something like
>> this was possible, indeed.
>
> String fence can be used to signal to font lock that the delimiter
> (together with the "mismatching" unescaped EOL) should be fontified in
> warning face.
>
> A better example might be C++ Mode's marking of a "< ... >" pair with
> paren syntax. This isn't done with syntax-propertize-function (as you
> know), but it would be nice if this were possible.
>
>>> 2. syntax-propertize-function's are banned from using syntax-ppss, the
>>> documentation instead directing them to use parse-partial-sexp directly.
>
>> The ones that currently call syntax-ppss, can't simply switch over to
>> parse-partial-sexp without becoming slower due to the lack of cache.
>
> The cache at the pertinent buffer position doesn't exist at the time:
> consistent syntax-table properties aren't on the preceding buffer
> positions.
>
>> Before tackling this bug, I'd rather we see a real-world problem that it
>> caused, and pick a particular approach based on it.
>
> My enhancements for bug#30393: "24.4; cperl-mode: indentation failure -
> Documentation enhancements", where (almost) any change which affects the
> syntactic state is programmed to call syntax-ppss-flush-cache from the C
> level, clashes with the mechanism in this bug report. Most of the time
> it's fine, but when a change affecting the syntactic state is made from
> inside a synax-propertize-function, Emacs goes into an infinite recursive
> loop.
>
> This isn't good.
>
>> But off the top of my head, we could introduce a "stricter but somewhat
>> slower" variation of syntax-ppss to be called inside
>> syntax-propertize-function's, which would treat the values in question
>> more carefully, somehow.
>
> That's an idea worth exploring.
>
Hi folks,
from what I've seen month ago just may stress the term fundamental.
Gave up to follow details WRT to check-ins made.
That part needs some person treating the gordic knot according to its
quality...
bug-gnu-emacs@HIDDEN:bug#31290; Package emacs.
Full text available.
Received: (at 31290) by debbugs.gnu.org; 12 May 2018 11:33:51 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat May 12 07:33:51 2018
Received: from localhost ([127.0.0.1]:58989 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1fHSmc-0003eV-UA
for submit <at> debbugs.gnu.org; Sat, 12 May 2018 07:33:51 -0400
Received: from colin.muc.de ([193.149.48.1]:38221 helo=mail.muc.de)
by debbugs.gnu.org with smtp (Exim 4.84_2)
(envelope-from <acm@HIDDEN>) id 1fHSma-0003eL-Fr
for 31290 <at> debbugs.gnu.org; Sat, 12 May 2018 07:33:49 -0400
Received: (qmail 81713 invoked by uid 3782); 12 May 2018 11:33:47 -0000
Received: from acm.muc.de (p5B146231.dip0.t-ipconnect.de [91.20.98.49]) by
colin.muc.de (tmda-ofmipd) with ESMTP;
Sat, 12 May 2018 13:33:45 +0200
Received: (qmail 6270 invoked by uid 1000); 12 May 2018 11:26:12 -0000
Date: Sat, 12 May 2018 11:26:12 +0000
To: Dmitry Gutov <dgutov@HIDDEN>
Subject: Re: bug#31290: Fundamental bugs in syntax-propertize
Message-ID: <20180512112612.GA4766@ACM>
References: <20180427210859.GA6023@ACM>
<e8658e46-da92-6310-7b1e-33c55f6255d2@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <e8658e46-da92-6310-7b1e-33c55f6255d2@HIDDEN>
User-Agent: Mutt/1.9.4 (2018-02-28)
X-Delivery-Agent: TMDA/1.1.12 (Macallan)
From: Alan Mackenzie <acm@HIDDEN>
X-Primary-Address: acm@HIDDEN
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 31290
Cc: 31290 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)
Hello, Dmitry.
On Tue, May 08, 2018 at 15:35:14 +0300, Dmitry Gutov wrote:
> On 4/28/18 12:08 AM, Alan Mackenzie wrote:
> > At least that would be true if syntax-propertize--done hadn't been
> > prematurely and spuriously increased, crudely to prevent an infinite
> > recursion, falsely indicating to the syntax-ppss infrastructure that the
> > syntax-table properties have already been applied to the region (BEGIN
> > END).
> > .... but it should not call `syntax-ppss-flush-cache', ....
> > Why not? Because syntax-ppss-flush-cache sets syntax-propertize--done
> > back to its true value, allowing the wrongly allowed syntax-ppss calls at
> > a later position to cause a recursive loop.
> Maybe we should "allow" it to loop, in certain cases? Leaving it to be
> the responsibility of the programmer, to make sure the result doesn't
> infloop, even if these rules are violated.
I'm not sure how this could work. We would need to formalise the rules
very carefully, to avoid the need to read syntax.{c,el}'s source code.
> > .... which means that it should not call `syntax-ppss' on some
> > position and later modify the buffer on some earlier position.
> > This is a bad restriction, because sometimes syntax-table properties can
> > only be correctly determined by examining the syntax of later buffer
> > positions. An example of this is giving the string-fence syntax-table
> > text property to an unbalanced opening string quote, but not to correctly
> > matched quotes.
> I'm not exactly convinced by the given example (why would we use the
> string-fence in that case?), but it might be better if something like
> this was possible, indeed.
String fence can be used to signal to font lock that the delimiter
(together with the "mismatching" unescaped EOL) should be fontified in
warning face.
A better example might be C++ Mode's marking of a "< ... >" pair with
paren syntax. This isn't done with syntax-propertize-function (as you
know), but it would be nice if this were possible.
> > 2. syntax-propertize-function's are banned from using syntax-ppss, the
> > documentation instead directing them to use parse-partial-sexp directly.
> The ones that currently call syntax-ppss, can't simply switch over to
> parse-partial-sexp without becoming slower due to the lack of cache.
The cache at the pertinent buffer position doesn't exist at the time:
consistent syntax-table properties aren't on the preceding buffer
positions.
> Before tackling this bug, I'd rather we see a real-world problem that it
> caused, and pick a particular approach based on it.
My enhancements for bug#30393: "24.4; cperl-mode: indentation failure -
Documentation enhancements", where (almost) any change which affects the
syntactic state is programmed to call syntax-ppss-flush-cache from the C
level, clashes with the mechanism in this bug report. Most of the time
it's fine, but when a change affecting the syntactic state is made from
inside a synax-propertize-function, Emacs goes into an infinite recursive
loop.
This isn't good.
> But off the top of my head, we could introduce a "stricter but somewhat
> slower" variation of syntax-ppss to be called inside
> syntax-propertize-function's, which would treat the values in question
> more carefully, somehow.
That's an idea worth exploring.
--
Alan Mackenzie (Nuremberg, Germany).
bug-gnu-emacs@HIDDEN:bug#31290; Package emacs.
Full text available.Received: (at 31290) by debbugs.gnu.org; 8 May 2018 12:35:28 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Tue May 08 08:35:28 2018 Received: from localhost ([127.0.0.1]:53540 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1fG1q3-0003oC-Sb for submit <at> debbugs.gnu.org; Tue, 08 May 2018 08:35:28 -0400 Received: from mail-wr0-f173.google.com ([209.85.128.173]:46771) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <raaahh@HIDDEN>) id 1fG1pz-0003nw-F7 for 31290 <at> debbugs.gnu.org; Tue, 08 May 2018 08:35:24 -0400 Received: by mail-wr0-f173.google.com with SMTP id a12-v6so1246492wrn.13 for <31290 <at> debbugs.gnu.org>; Tue, 08 May 2018 05:35:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:to:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=VYOIVEaIRrc4TFp82f2wDek7DlL8StKt4BkR4d74PTQ=; b=mBuczawT4iEmzk3SgRZNvOGXdAukcQaHOUxrT24lNbUlt96YjdhPsy4t+BrbrskPuN kF9nzgM8Kaz4yYP4CUy+KxcSVe3D9yQJu1mAFH4j0ruUsdMZdM/us/jYG1iLMQVUn4So /FBgaUuYwblAP8X+ATTnXJcFDbk+ihJQGfPVkJ+E+MNF5mDaxJu1t4HpMHaLKjsRpvgn mEN9gVjBKS8Qe2WfhRbcPnbdNLBpTR01xA/2g4Te8xwkzdrG8M8Hiigo5E3iSn7y39g/ wuh46xbIUrLCQ0+WQ9fzHFZmSiAb9a6URJLR6M353/E8lWjbvffv9blE9FMc/VpxPrFY ZacQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:to:references:from:message-id :date:user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=VYOIVEaIRrc4TFp82f2wDek7DlL8StKt4BkR4d74PTQ=; b=gJ8TsstQ9bA+oNJfNyIzn5rgvoyMIFqPyQt22x3NqfaGYd4ZrcWki6L6xUaVe4V556 6w3o+yYSidudHgSiz5HBHYj56HYJaf6+5dMWsuep3tL2JmJ5pkFF3FJDfECfV434OAGM 7TLDAI/wwkVnCQPXINIAN6YM4gIBwq5f8zy6BF57zcQG0pc8VkBD6nxfoknl1T0Ccz0F mIkb0t6CpD5xlG4oEcTw8LIPD3X7WimbAUteNZWNqpugeovX7uhPtRxxgVT6inKbpi1R eqqfuqBJwFW9IX8pyoXxc2piJyZThNDlGkNVkrTXEmJ+RlN80S93FvVyvQtuEx/YbKBz bFMA== X-Gm-Message-State: ALQs6tDxS8uqYvu8Zf69wHQUVyTu3Q9jLIHJUNb+ADf7ERHoPVW2nYQD hmXAIF8xwWudh/0gLIpfXhNR7RDz X-Google-Smtp-Source: AB8JxZoHbA1vlFJmZXO2KGHVnMZM2QfPcNZW5FPh2e/EflgVZ3VLKZE1Bf3mKTRf2Uo+y9fPzW247w== X-Received: by 2002:adf:b78b:: with SMTP id s11-v6mr26222145wre.247.1525782917366; Tue, 08 May 2018 05:35:17 -0700 (PDT) Received: from [192.168.0.200] ([212.50.99.193]) by smtp.googlemail.com with ESMTPSA id h133sm11716321wmf.47.2018.05.08.05.35.15 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 08 May 2018 05:35:16 -0700 (PDT) Subject: Re: bug#31290: Fundamental bugs in syntax-propertize To: Alan Mackenzie <acm@HIDDEN>, 31290 <at> debbugs.gnu.org References: <20180427210859.GA6023@ACM> From: Dmitry Gutov <dgutov@HIDDEN> Message-ID: <e8658e46-da92-6310-7b1e-33c55f6255d2@HIDDEN> Date: Tue, 8 May 2018 15:35:14 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.0 MIME-Version: 1.0 In-Reply-To: <20180427210859.GA6023@ACM> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Spam-Score: 0.5 (/) X-Debbugs-Envelope-To: 31290 X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -0.5 (/) On 4/28/18 12:08 AM, Alan Mackenzie wrote: > At least that would be true if syntax-propertize--done hadn't been > prematurely and spuriously increased, crudely to prevent an infinite > recursion, falsely indicating to the syntax-ppss infrastructure that the > syntax-table properties have already been applied to the region (BEGIN > END). > > .... but it should not call `syntax-ppss-flush-cache', .... > > Why not? Because syntax-ppss-flush-cache sets syntax-propertize--done > back to its true value, allowing the wrongly allowed syntax-ppss calls at > a later position to cause a recursive loop. Maybe we should "allow" it to loop, in certain cases? Leaving it to be the responsibility of the programmer, to make sure the result doesn't infloop, even if these rules are violated. > .... which means that it should not call `syntax-ppss' on some > position and later modify the buffer on some earlier position. > > This is a bad restriction, because sometimes syntax-table properties can > only be correctly determined by examining the syntax of later buffer > positions. An example of this is giving the string-fence syntax-table > text property to an unbalanced opening string quote, but not to correctly > matched quotes. I'm not exactly convinced by the given example (why would we use the string-fence in that case?), but it might be better if something like this was possible, indeed. > 2. syntax-propertize-function's are banned from using syntax-ppss, the > documentation instead directing them to use parse-partial-sexp directly. The ones that currently call syntax-ppss, can't simply switch over to parse-partial-sexp without becoming slower due to the lack of cache. Before tackling this bug, I'd rather we see a real-world problem that it caused, and pick a particular approach based on it. But off the top of my head, we could introduce a "stricter but somewhat slower" variation of syntax-ppss to be called inside syntax-propertize-function's, which would treat the values in question more carefully, somehow.
bug-gnu-emacs@HIDDEN:bug#31290; Package emacs.
Full text available.
Received: (at submit) by debbugs.gnu.org; 27 Apr 2018 21:14:07 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Apr 27 17:14:07 2018
Received: from localhost ([127.0.0.1]:41390 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1fCAgx-0005Lg-FH
for submit <at> debbugs.gnu.org; Fri, 27 Apr 2018 17:14:07 -0400
Received: from eggs.gnu.org ([208.118.235.92]:55182)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <acm@HIDDEN>) id 1fCAgv-0005L3-6Q
for submit <at> debbugs.gnu.org; Fri, 27 Apr 2018 17:14:06 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
(envelope-from <acm@HIDDEN>) id 1fCAgp-0003ka-2K
for submit <at> debbugs.gnu.org; Fri, 27 Apr 2018 17:13:59 -0400
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level:
X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50 autolearn=disabled
version=3.3.2
Received: from lists.gnu.org ([2001:4830:134:3::11]:48842)
by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32)
(Exim 4.71) (envelope-from <acm@HIDDEN>) id 1fCAgo-0003kQ-VJ
for submit <at> debbugs.gnu.org; Fri, 27 Apr 2018 17:13:59 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:46147)
by lists.gnu.org with esmtp (Exim 4.71) (envelope-from <acm@HIDDEN>)
id 1fCAgn-0002Jr-Ub
for bug-gnu-emacs@HIDDEN; Fri, 27 Apr 2018 17:13:58 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
(envelope-from <acm@HIDDEN>) id 1fCAgi-0003iQ-Vj
for bug-gnu-emacs@HIDDEN; Fri, 27 Apr 2018 17:13:57 -0400
Received: from colin.muc.de ([193.149.48.1]:31573 helo=mail.muc.de)
by eggs.gnu.org with smtp (Exim 4.71) (envelope-from <acm@HIDDEN>)
id 1fCAgi-0003gc-KK
for bug-gnu-emacs@HIDDEN; Fri, 27 Apr 2018 17:13:52 -0400
Received: (qmail 77206 invoked by uid 3782); 27 Apr 2018 21:13:47 -0000
Received: from acm.muc.de (p5B14720D.dip0.t-ipconnect.de [91.20.114.13]) by
colin.muc.de (tmda-ofmipd) with ESMTP;
Fri, 27 Apr 2018 23:13:46 +0200
Received: (qmail 6051 invoked by uid 1000); 27 Apr 2018 21:08:59 -0000
Date: Fri, 27 Apr 2018 21:08:59 +0000
To: bug-gnu-emacs@HIDDEN
Subject: Fundamental bugs in syntax-propertize
Message-ID: <20180427210859.GA6023@ACM>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.9.4 (2018-02-28)
X-Delivery-Agent: TMDA/1.1.12 (Macallan)
From: Alan Mackenzie <acm@HIDDEN>
X-Primary-Address: acm@HIDDEN
X-detected-operating-system: by eggs.gnu.org: FreeBSD 9.x [fuzzy]
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x
X-Received-From: 2001:4830:134:3::11
X-Spam-Score: -4.3 (----)
X-Debbugs-Envelope-To: submit
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -5.3 (-----)
Hello, Emacs.
There are fundamental bugs in syntax-propertize and
syntax-propertize-function. The doc string of the latter states:
The specified function may call `syntax-ppss' on any position before
END, ....
This is untrue. True is that syntax-ppss can be called on a position
only up to syntax-propertize--done. After this point, the syntax-table
properties haven't been applied, so calling syntax-ppss is, in general,
going to give a false result.
At least that would be true if syntax-propertize--done hadn't been
prematurely and spuriously increased, crudely to prevent an infinite
recursion, falsely indicating to the syntax-ppss infrastructure that the
syntax-table properties have already been applied to the region (BEGIN
END).
.... but it should not call `syntax-ppss-flush-cache', ....
Why not? Because syntax-ppss-flush-cache sets syntax-propertize--done
back to its true value, allowing the wrongly allowed syntax-ppss calls at
a later position to cause a recursive loop.
.... which means that it should not call `syntax-ppss' on some
position and later modify the buffer on some earlier position.
This is a bad restriction, because sometimes syntax-table properties can
only be correctly determined by examining the syntax of later buffer
positions. An example of this is giving the string-fence syntax-table
text property to an unbalanced opening string quote, but not to correctly
matched quotes.
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
The plain fact is that (syntax-ppss pos) calls (syntax-propertize pos),
so syntax-propertize cannot itself use syntax-ppss because of the
recursive loop thus created.
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
Proposed solutions:
1. Major modes' syntax-propertize-function's are somehow given read
access to syntax-propertize--done, and may call syntax-ppss up to that
point only. syntax-propertize--done is updated only after the
syntax-table properties have been applied. Or....
2. syntax-propertize-function's are banned from using syntax-ppss, the
documentation instead directing them to use parse-partial-sexp directly.
In either solution, the restriction on using syntax-ppss-flush-cache
would no longer be necessary, and there would be no restriction on
setting syntax-table text properties at an earlier position than the one
currently being analysed.
I think solution 2 is the better one.
--
Alan Mackenzie (Nuremberg, Germany).
Alan Mackenzie <acm@HIDDEN>:bug-gnu-emacs@HIDDEN.
Full text available.bug-gnu-emacs@HIDDEN:bug#31290; Package emacs.
Full text available.
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997 nCipher Corporation Ltd,
1994-97 Ian Jackson.