GNU bug report logs - #36432
26.2; SMIE does not request forward tokens when point is at point-max

Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.

Package: emacs; Reported by: Sam Halliday <sam.halliday@HIDDEN>; dated Sat, 29 Jun 2019 12:15:01 UTC; Maintainer for emacs is bug-gnu-emacs@HIDDEN.

Message received at 36432 <at> debbugs.gnu.org:


Received: (at 36432) by debbugs.gnu.org; 30 Jun 2019 08:50:46 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Jun 30 04:50:46 2019
Received: from localhost ([127.0.0.1]:45339 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1hhVXq-0002Y0-9H
	for submit <at> debbugs.gnu.org; Sun, 30 Jun 2019 04:50:46 -0400
Received: from mail-vs1-f51.google.com ([209.85.217.51]:37701)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <sam.halliday@HIDDEN>) id 1hhVXo-0002Xl-C9
 for 36432 <at> debbugs.gnu.org; Sun, 30 Jun 2019 04:50:44 -0400
Received: by mail-vs1-f51.google.com with SMTP id v6so6899324vsq.4
 for <36432 <at> debbugs.gnu.org>; Sun, 30 Jun 2019 01:50:44 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=mime-version:in-reply-to:references:from:date:message-id:subject:to
 :cc; bh=w0iupvud0wVSkVIo7mzmRzvIsNmn+anVPXHsNDoj0Mw=;
 b=AdNT4RQr52GtjdRe2EKEu9EzhTmVy5du/duE2ud/t5YkH7OxdXoNTIDKUzf6+0oWuj
 WUvh7d+mKKguH2WSMOqjw+DCfOUXlJfIyLnbv7sf3E7ggpA/b3uqUYBAdnUeLadV7fnD
 cCTR3doxXAnxAyjWyOc65+1O36YHychLwb2Subh5N/23xtryRGHj2VPL69c81yLSAqAi
 HDwZwZrYtlgo7eSpC002RxMgcUigysBcfGSySNuStQxbtLZ6sWkwuZ+/3A/a8YoFzrfV
 Cdfcb5A69fACW3bcDb30HGD0Ck9+t6APestcQoSIl5XEDqkLbowz0TOOjt6eUlaK8Evb
 wIfQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:mime-version:in-reply-to:references:from:date
 :message-id:subject:to:cc;
 bh=w0iupvud0wVSkVIo7mzmRzvIsNmn+anVPXHsNDoj0Mw=;
 b=gMWHmcHfgH549PwR/EExKB8qSOSXJpagEPwDDg+xOH/THUZNWpDZDnUfOmSxAMjf9w
 DU9/fcYlX4DYZ3kZzUoxgYCVdr/tPlIUlcnHNYqsgG4oINmNZA8fyFENuJ5ER5va9kMv
 5C2Dm0V7erndRvmUbNo6eiACnEMOwcGnZOt4+MJzvDKTpqc2OGmx8yVZsP0avwHP5/R4
 C9S8lt4G9VfHb/4F9hBH9A6Bmzuv733R9NUDO1aG+0mBeJXMcQEEMCbxNv3XIiEOlUsT
 H1Z9kK+3XJKhvrjYpruK8JaowOz8c50NZRSUEXg8WDU6+hTc4tGuehhMeqrL0k2cqGno
 4qfw==
X-Gm-Message-State: APjAAAVEeqQN9w4fJ1NYQGc6LoWxe80N2AD/q9syin/5a4PW00wfYIKx
 +TvQ2hmHKV8qOUt40y71j77yE80RvFckFqph1Wc=
X-Google-Smtp-Source: APXvYqyDIG9fVw4D/68mCLm5WDtbHkETRhL0JXyA5cXemrHM6HrCeExFQ5YN4D7p4UcjW6UY32U+5DQfViwdr5DBuMs=
X-Received: by 2002:a05:6102:3c5:: with SMTP id
 n5mr11057978vsq.56.1561884638761; 
 Sun, 30 Jun 2019 01:50:38 -0700 (PDT)
MIME-Version: 1.0
Received: by 2002:a9f:2427:0:0:0:0:0 with HTTP; Sun, 30 Jun 2019 01:50:38
 -0700 (PDT)
In-Reply-To: <jwvo92gkrbl.fsf-monnier+emacs@HIDDEN>
References: <CALR_T9CoA+xto771d_dWhHunByqB4EKO+ODBhXhzUimCfhCy_A@HIDDEN>
 <jwvo92gkrbl.fsf-monnier+emacs@HIDDEN>
From: Sam Halliday <sam.halliday@HIDDEN>
Date: Sun, 30 Jun 2019 09:50:38 +0100
Message-ID: <CALR_T9DyH-tWbHorL85zofMFh6cXVReMVwVLKbURyO2-2P5LxQ@HIDDEN>
Subject: Re: bug#36432: 26.2; SMIE does not request forward tokens when point
 is at point-max
To: Stefan Monnier <monnier@HIDDEN>
Content-Type: text/plain; charset="UTF-8"
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 36432
Cc: 36432 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

I'm seeing this when doing indentation.

e.g. in https://gitlab.com/tseenshe/haskell-tng.el/blob/tng/test/src/indentation.hs
move the point to the end of the last line and do a
`newline-and-indent'. The do it again when you have two newlines after
that last point. The results are different.

BTW, in addition to the edebug support you've added, I also have

(bind-key "C-M-<return>" 'haskell-tng-smie:debug-newline haskell-tng-mode-map)
(bind-key "C-M-<tab>" 'haskell-tng-smie:debug-tab haskell-tng-mode-map)

that are useful for seeing what's going on, with some haskell-tng
specific things.

On 29/06/2019, Stefan Monnier <monnier@HIDDEN> wrote:
>> SMIE (via a `indent-for-tab-command') does not request forward tokens
>> from the lexer when point is at `point-max'.
>
> After looking at the smie.el code I think this bug report is not
> sufficiently detailed: it definitely sometimes does, and I don't see any
> obvious place where it doesn't.  Can you clarify if it happens during
> something like smie-forward-sexp or rather within the smie-indent*
> code itself.
>
> Or do you mean when you trigger indent-according-to-mode with point at EOB?
>
>
>         Stefan
>
>




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#36432; Package emacs. Full text available.

Message received at 36432 <at> debbugs.gnu.org:


Received: (at 36432) by debbugs.gnu.org; 29 Jun 2019 21:39:34 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Jun 29 17:39:34 2019
Received: from localhost ([127.0.0.1]:45001 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1hhL4I-0005CR-9p
	for submit <at> debbugs.gnu.org; Sat, 29 Jun 2019 17:39:34 -0400
Received: from mailscanner.iro.umontreal.ca ([132.204.25.50]:40557)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <monnier@HIDDEN>) id 1hhL4G-0005CC-4n
 for 36432 <at> debbugs.gnu.org; Sat, 29 Jun 2019 17:39:32 -0400
Received: from pmg1.iro.umontreal.ca (localhost.localdomain [127.0.0.1])
 by pmg1.iro.umontreal.ca (Proxmox) with ESMTP id 645E1100B3B;
 Sat, 29 Jun 2019 17:39:24 -0400 (EDT)
Received: from mail01.iro.umontreal.ca (unknown [172.31.2.1])
 by pmg1.iro.umontreal.ca (Proxmox) with ESMTP id 426EF1009EC;
 Sat, 29 Jun 2019 17:39:23 -0400 (EDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=iro.umontreal.ca;
 s=mail; t=1561844363;
 bh=hPgyslU7pBpasa7S/sCEyDGayqWQT7P5MZQhB4+zET0=;
 h=From:To:Cc:Subject:References:Date:In-Reply-To:From;
 b=Ra9Y7ioFdeC45ys8JF5KDTllqs/FIUW8cT3goMTcE3PcDPQSv6w8HISpORHR+yL9P
 tUzlI2k6CYBQCbSiBVBeyZXLoSfJcX99uk3Uj+O1mBSP7f8wVKG7FNNuz+sC6CU1E4
 /85A2Lb71j4THJDf3f/okMgBi6KkrslAIfogF/ifyzjahI1F287mhn9izkPMaMIWCU
 rJpQk1SjlDkq14JJsXtvZ0nAU+AKDd7WS/ZHHD45sDQHYl7ANCDXzuB4pMeO82/Mzm
 pfUIo5uC/wz8uZlpJf3MD7OBME90UIqx3cNyNnOtLMpg5NYFJnvt8SItTMbpvXUHTO
 zmNgI3olVutJQ==
Received: from alfajor (76-10-151-214.dsl.teksavvy.com [76.10.151.214])
 by mail01.iro.umontreal.ca (Postfix) with ESMTPSA id E329512019B;
 Sat, 29 Jun 2019 17:39:22 -0400 (EDT)
From: Stefan Monnier <monnier@HIDDEN>
To: Sam Halliday <sam.halliday@HIDDEN>
Subject: Re: bug#36432: 26.2;
 SMIE does not request forward tokens when point is at point-max
Message-ID: <jwvo92gkrbl.fsf-monnier+emacs@HIDDEN>
References: <CALR_T9CoA+xto771d_dWhHunByqB4EKO+ODBhXhzUimCfhCy_A@HIDDEN>
Date: Sat, 29 Jun 2019 17:39:21 -0400
In-Reply-To: <CALR_T9CoA+xto771d_dWhHunByqB4EKO+ODBhXhzUimCfhCy_A@HIDDEN>
 (Sam Halliday's message of "Sat, 29 Jun 2019 13:14:01 +0100")
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-SPAM-INFO: Spam detection results:  0
 ALL_TRUSTED                -1 Passed through trusted hosts only via SMTP
 AWL 0.115 Adjusted score from AWL reputation of From: address
 BAYES_00                 -1.9 Bayes spam probability is 0 to 1%
 DKIM_SIGNED               0.1 Message has a DKIM or DK signature,
 not necessarily valid
 DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature
 DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's
 domain
X-SPAM-LEVEL: 
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 36432
Cc: 36432 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

> SMIE (via a `indent-for-tab-command') does not request forward tokens
> from the lexer when point is at `point-max'.

After looking at the smie.el code I think this bug report is not
sufficiently detailed: it definitely sometimes does, and I don't see any
obvious place where it doesn't.  Can you clarify if it happens during
something like smie-forward-sexp or rather within the smie-indent*
code itself.

Or do you mean when you trigger indent-according-to-mode with point at EOB?


        Stefan





Information forwarded to bug-gnu-emacs@HIDDEN:
bug#36432; Package emacs. Full text available.

Message received at 36432 <at> debbugs.gnu.org:


Received: (at 36432) by debbugs.gnu.org; 29 Jun 2019 13:43:46 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Jun 29 09:43:46 2019
Received: from localhost ([127.0.0.1]:43623 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1hhDdp-0007xG-Ph
	for submit <at> debbugs.gnu.org; Sat, 29 Jun 2019 09:43:46 -0400
Received: from eggs.gnu.org ([209.51.188.92]:42926)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1hhDdn-0007x1-Vv
 for 36432 <at> debbugs.gnu.org; Sat, 29 Jun 2019 09:43:44 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:48529)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>)
 id 1hhDdh-0001Bn-IV; Sat, 29 Jun 2019 09:43:37 -0400
Received: from [176.228.60.248] (port=2378 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1hhDdf-0001hc-AS; Sat, 29 Jun 2019 09:43:36 -0400
Date: Sat, 29 Jun 2019 16:43:25 +0300
Message-Id: <83r27cfqwy.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Sam Halliday <sam.halliday@HIDDEN>,
 Stefan Monnier <monnier@HIDDEN> 
In-reply-to: <CALR_T9BxO-t1HLU_jWLeRnULtuSBBwAFZyit6FdE-AhL+oZ2Xg@HIDDEN>
 (message from Sam Halliday on Sat, 29 Jun 2019 14:13:16 +0100)
Subject: Re: bug#36432: 26.2; SMIE does not request forward tokens when point
 is at point-max
References: <CALR_T9CoA+xto771d_dWhHunByqB4EKO+ODBhXhzUimCfhCy_A@HIDDEN>
 <83y31kfumo.fsf@HIDDEN>
 <CALR_T9CVNkFR9tH28Yowx1afEbitrWR8jyYp8icEBampxgDyBw@HIDDEN>
 <83woh4ftqk.fsf@HIDDEN>
 <CALR_T9Cwde_5CfPaATpdLZ3yottcqoopq0gKgy50-jKBvHLG0A@HIDDEN>
 <83tvc8fsmu.fsf@HIDDEN>
 <CALR_T9BxO-t1HLU_jWLeRnULtuSBBwAFZyit6FdE-AhL+oZ2Xg@HIDDEN>
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 36432
Cc: 36432 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: Sam Halliday <sam.halliday@HIDDEN>
> Date: Sat, 29 Jun 2019 14:13:16 +0100
> Cc: 36432 <at> debbugs.gnu.org
> 
> On 29/06/2019, Eli Zaretskii <eliz@HIDDEN> wrote:
> > It should be easier to modify SMIE to take characters from a string,
> > then you could put whatever you want into that string.  Or maybe SMIE
> > already supports reading from strings, I don't know.
> 
> I agree. SMIE should be operating on a list of tokens and a lookup
> from those tokens to the original buffer (and content). In most cases
> SMIE is written this way but I suspect there are a few lapses.

In any case, I'm CC'ing Stefan who knows much more about SMIE than I
do.  Apologies in advance if Stefan says my fears have no basis.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#36432; Package emacs. Full text available.

Message received at 36432 <at> debbugs.gnu.org:


Received: (at 36432) by debbugs.gnu.org; 29 Jun 2019 13:13:26 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Jun 29 09:13:25 2019
Received: from localhost ([127.0.0.1]:43607 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1hhDAT-0007H2-H1
	for submit <at> debbugs.gnu.org; Sat, 29 Jun 2019 09:13:25 -0400
Received: from mail-vs1-f53.google.com ([209.85.217.53]:35941)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <sam.halliday@HIDDEN>) id 1hhDAQ-0007Go-Ov
 for 36432 <at> debbugs.gnu.org; Sat, 29 Jun 2019 09:13:23 -0400
Received: by mail-vs1-f53.google.com with SMTP id s141so2887073vsc.3
 for <36432 <at> debbugs.gnu.org>; Sat, 29 Jun 2019 06:13:22 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=mime-version:in-reply-to:references:from:date:message-id:subject:to
 :cc; bh=Qqx6YtVEu81NspmwNiIZFo2eBu5L8RW0f9NKcYzxi+0=;
 b=u57DGrlXt32D3W8u5thM2swpnfTK5S88lOkLc0trFeKc8Iu8nRfgPpoEaZkLdVf3VZ
 NoREiAgi9bJKwhRQPRzT3AHZ0GDVORXK+nbW65Tj9NjnP3lBnhWmfOckQ+6nf67C7ZCJ
 rqtUbNYJZ0SoA5HlV1VRWkfs1aC0wLnX2aeQJECZKoYLNWqbrEC8Fm7CDltEpbg3zPCt
 zCdleSydoGfKN1Htd3w5+oySxnr7CVw+SVFrc6UVnq0UbTpS0Ne5B4oF0sDSesNLMn9H
 83wZCoEW5/HP36VZMyBqxsZ2HM1/qHc8RZPijIyOwS9sECnwI9j4XY2SlwoPz9wbqdED
 Ihag==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:mime-version:in-reply-to:references:from:date
 :message-id:subject:to:cc;
 bh=Qqx6YtVEu81NspmwNiIZFo2eBu5L8RW0f9NKcYzxi+0=;
 b=awIJus3H9lCAQ6baoDR+r1L69ZPx3hIWXTz+vKvYPSows6G6GRqzppXcECTsq/M5uz
 rMB+fNq6F3nuMPo8NPFRf5LsUi01tkkRLTU+fmusFLjDo5AntM1KE5StqrDgtQamyFgO
 QJ7K3HYkoSCByKy0G/B0/U2tahftbBSHGhdIgrqELvJU0z3/MZxSGzx7a3DwYxhW0BNR
 ct315NCieSjQcUA+Td+MTOwjcqtwsr+M2ZtggqSmusNbLgb/6BSf+8tT7TqYxw1RLb1B
 39k+qqS3xA5GcVloVp+yIaEwZ/QYxt2X0bDn8fRWjniyKqvJipVhTiAx6mzEzCli7EwC
 Uf/g==
X-Gm-Message-State: APjAAAU3ZFP8CWy55AHRXAHfzrSG3n6LsCA8xH7Tic18pQYPXvLq+Orf
 bKES3bN1sY8nOwrDUUdk3OU6uljj5SHKQEOTRQs=
X-Google-Smtp-Source: APXvYqyrt6+YPueJuSbhBR0F9p4OVgQhaHSABuER0/gcnsTCK7sxMzIGlOzqmBtgy41WPyaIxg3HIX/uKIAiWYZfmkA=
X-Received: by 2002:a05:6102:3c5:: with SMTP id
 n5mr9146379vsq.56.1561813997237; 
 Sat, 29 Jun 2019 06:13:17 -0700 (PDT)
MIME-Version: 1.0
Received: by 2002:a9f:2427:0:0:0:0:0 with HTTP; Sat, 29 Jun 2019 06:13:16
 -0700 (PDT)
In-Reply-To: <83tvc8fsmu.fsf@HIDDEN>
References: <CALR_T9CoA+xto771d_dWhHunByqB4EKO+ODBhXhzUimCfhCy_A@HIDDEN>
 <83y31kfumo.fsf@HIDDEN>
 <CALR_T9CVNkFR9tH28Yowx1afEbitrWR8jyYp8icEBampxgDyBw@HIDDEN>
 <83woh4ftqk.fsf@HIDDEN>
 <CALR_T9Cwde_5CfPaATpdLZ3yottcqoopq0gKgy50-jKBvHLG0A@HIDDEN>
 <83tvc8fsmu.fsf@HIDDEN>
From: Sam Halliday <sam.halliday@HIDDEN>
Date: Sat, 29 Jun 2019 14:13:16 +0100
Message-ID: <CALR_T9BxO-t1HLU_jWLeRnULtuSBBwAFZyit6FdE-AhL+oZ2Xg@HIDDEN>
Subject: Re: bug#36432: 26.2; SMIE does not request forward tokens when point
 is at point-max
To: Eli Zaretskii <eliz@HIDDEN>
Content-Type: text/plain; charset="UTF-8"
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 36432
Cc: 36432 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

On 29/06/2019, Eli Zaretskii <eliz@HIDDEN> wrote:
> It should be easier to modify SMIE to take characters from a string,
> then you could put whatever you want into that string.  Or maybe SMIE
> already supports reading from strings, I don't know.

I agree. SMIE should be operating on a list of tokens and a lookup
from those tokens to the original buffer (and content). In most cases
SMIE is written this way but I suspect there are a few lapses.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#36432; Package emacs. Full text available.

Message received at 36432 <at> debbugs.gnu.org:


Received: (at 36432) by debbugs.gnu.org; 29 Jun 2019 13:08:20 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Jun 29 09:08:20 2019
Received: from localhost ([127.0.0.1]:43601 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1hhD5X-0007AC-RU
	for submit <at> debbugs.gnu.org; Sat, 29 Jun 2019 09:08:20 -0400
Received: from eggs.gnu.org ([209.51.188.92]:36670)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1hhD5V-0007A0-PV
 for 36432 <at> debbugs.gnu.org; Sat, 29 Jun 2019 09:08:18 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:48050)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>)
 id 1hhD5Q-00012P-7z; Sat, 29 Jun 2019 09:08:12 -0400
Received: from [176.228.60.248] (port=4210 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1hhD5O-0006BU-96; Sat, 29 Jun 2019 09:08:12 -0400
Date: Sat, 29 Jun 2019 16:08:01 +0300
Message-Id: <83sgrsfsjy.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Sam Halliday <sam.halliday@HIDDEN>
In-reply-to: <CALR_T9A23zBOPn3NieGH4O51M8EBHnWkh5ABnjhWursfBxJvyQ@HIDDEN>
 (message from Sam Halliday on Sat, 29 Jun 2019 14:01:52 +0100)
Subject: Re: bug#36432: 26.2; SMIE does not request forward tokens when point
 is at point-max
References: <CALR_T9CoA+xto771d_dWhHunByqB4EKO+ODBhXhzUimCfhCy_A@HIDDEN>
 <83y31kfumo.fsf@HIDDEN>
 <CALR_T9CVNkFR9tH28Yowx1afEbitrWR8jyYp8icEBampxgDyBw@HIDDEN>
 <83woh4ftqk.fsf@HIDDEN> <83v9woftc4.fsf@HIDDEN>
 <CALR_T9A23zBOPn3NieGH4O51M8EBHnWkh5ABnjhWursfBxJvyQ@HIDDEN>
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 36432
Cc: 36432 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: Sam Halliday <sam.halliday@HIDDEN>
> Date: Sat, 29 Jun 2019 14:01:52 +0100
> Cc: 36432 <at> debbugs.gnu.org
> 
> On 29/06/2019, Eli Zaretskii <eliz@HIDDEN> wrote:
> >> Date: Sat, 29 Jun 2019 15:42:27 +0300
> >> From: Eli Zaretskii <eliz@HIDDEN>
> >> Cc: 36432 <at> debbugs.gnu.org
> >>
> >> How would Emacs know this is the case?  Why don't you also override
> >> point-max to make it consistent with those illusory characters?
> >
> > Or actually insert those characters at EOB, but make them invisible?
> 
> I think I'd like to avoid doing that unless I do it for all of the
> virtual tokens. I don't know how to do that without it impacting the
> underlying source code. Is there more documentation about doing it
> that way? It'd be a drastic change in how the lexer is written.

Why? does the lexer only pay attention to visible characters?

> BTW I had a look through smie.el and I can't see anywhere obvious
> where (eobp) or (point-max) are called that would lead to the bug I'm
> seeing.

It's likely in lower-level code.  Like I said: this assumption is
everywhere in Emacs.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#36432; Package emacs. Full text available.

Message received at 36432 <at> debbugs.gnu.org:


Received: (at 36432) by debbugs.gnu.org; 29 Jun 2019 13:06:43 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Jun 29 09:06:43 2019
Received: from localhost ([127.0.0.1]:43597 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1hhD3z-00077o-D3
	for submit <at> debbugs.gnu.org; Sat, 29 Jun 2019 09:06:43 -0400
Received: from eggs.gnu.org ([209.51.188.92]:36396)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1hhD3x-00077b-Gh
 for 36432 <at> debbugs.gnu.org; Sat, 29 Jun 2019 09:06:41 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:48025)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>)
 id 1hhD3m-0008KF-SR; Sat, 29 Jun 2019 09:06:31 -0400
Received: from [176.228.60.248] (port=4103 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1hhD3l-0004Pd-2T; Sat, 29 Jun 2019 09:06:30 -0400
Date: Sat, 29 Jun 2019 16:06:17 +0300
Message-Id: <83tvc8fsmu.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Sam Halliday <sam.halliday@HIDDEN>
In-reply-to: <CALR_T9Cwde_5CfPaATpdLZ3yottcqoopq0gKgy50-jKBvHLG0A@HIDDEN>
 (message from Sam Halliday on Sat, 29 Jun 2019 13:51:31 +0100)
Subject: Re: bug#36432: 26.2; SMIE does not request forward tokens when point
 is at point-max
References: <CALR_T9CoA+xto771d_dWhHunByqB4EKO+ODBhXhzUimCfhCy_A@HIDDEN>
 <83y31kfumo.fsf@HIDDEN>
 <CALR_T9CVNkFR9tH28Yowx1afEbitrWR8jyYp8icEBampxgDyBw@HIDDEN>
 <83woh4ftqk.fsf@HIDDEN>
 <CALR_T9Cwde_5CfPaATpdLZ3yottcqoopq0gKgy50-jKBvHLG0A@HIDDEN>
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 36432
Cc: 36432 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: Sam Halliday <sam.halliday@HIDDEN>
> Date: Sat, 29 Jun 2019 13:51:31 +0100
> Cc: 36432 <at> debbugs.gnu.org
> 
> > So you create an illusion of characters beyond the EOB?
> >
> > How would Emacs know this is the case?
> 
> When testing it is possible to keep polling the lexer until it returns
> nil when at point-max, rather than looking at `point-max` and giving
> up. I think that could work in general inside SMIE.
> https://gitlab.com/tseenshe/haskell-tng.el/blob/tng/test/haskell-tng-lexer-test.el

But SMIE is just an application on top of Emacs basic handling of
buffer positions.  The assumption that there can be nothing at EOB is
hardcoded into many Emacs primitives, into its display engine, and
into core Lisp infrastructure.  You are playing with fire trying to
force Emacs think there are some characters beyond EOB.  Just grep the
C sources for ZV, and you will see the enormous height of the hill you
will need to fight up.  I wouldn't recommend that to anyone.

It should be easier to modify SMIE to take characters from a string,
then you could put whatever you want into that string.  Or maybe SMIE
already supports reading from strings, I don't know.

> BTW, this also happens at the start of the buffer. SMIE doesn't ask
> for backwards tokens when at the beginning.

For the same basic reasons.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#36432; Package emacs. Full text available.

Message received at 36432 <at> debbugs.gnu.org:


Received: (at 36432) by debbugs.gnu.org; 29 Jun 2019 13:02:00 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Jun 29 09:02:00 2019
Received: from localhost ([127.0.0.1]:43593 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1hhCzP-00071D-OD
	for submit <at> debbugs.gnu.org; Sat, 29 Jun 2019 09:01:59 -0400
Received: from mail-vs1-f49.google.com ([209.85.217.49]:43827)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <sam.halliday@HIDDEN>) id 1hhCzO-000710-GQ
 for 36432 <at> debbugs.gnu.org; Sat, 29 Jun 2019 09:01:58 -0400
Received: by mail-vs1-f49.google.com with SMTP id j26so5906948vsn.10
 for <36432 <at> debbugs.gnu.org>; Sat, 29 Jun 2019 06:01:58 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=mime-version:in-reply-to:references:from:date:message-id:subject:to
 :cc; bh=8ltxzfejuc+/5CcOZSUIqt5vdbSsf62H0dxsK5JbG1U=;
 b=hCWjt0JeM7xnfBn0QJieLNPe3u/btrRxbqAEiqlAZxLc2ao+yXE9V9Qi7869Ke51ZD
 W58kzlgXj/v5iDpJxA50Tb8o+t0c6vSnUI0c6SrO9t2VdEwOiCQZ3XtAn7SpDjWuPl+U
 IfUfbNJUhZKGxsUxzTsgxaOIziq8dqH7CECf+/QvmYrFrxyLmnKNT5Gg5sewW0QXR0jx
 58YDaG7MHfu3TPcLZ1mub3FlrsGbCkByOYw117LADyePuUb/NdNH5vJrjhN2Daj/PDoF
 rIbIQc5Jn3pgEA8oS58IP/LsX3CmqOLurN9MC5zoqSydhUwJ6wKLO7Po+AGu21rX5jd3
 JoHw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:mime-version:in-reply-to:references:from:date
 :message-id:subject:to:cc;
 bh=8ltxzfejuc+/5CcOZSUIqt5vdbSsf62H0dxsK5JbG1U=;
 b=skact+t++bVgdmPnWJ/hhvj/pVpT7LZ5Zf6V1MqDij11thzeiuJSvOiARydBM3jSso
 5aToVZsOhACUoBdo/kQ3ctp9qOSmoNPZV53tmEOj46loUdjqjvNg/HmiFOOoJglzWFFn
 8Xm9TwX0B4CBP7yLz+h//ZMRrqBabSHccZySaePsfpYYwMpToj0/Hxzhsgz3TxZeD4L3
 80dFVr1mgbi+0hoTPs4KlXb9ysJ5X8rzTeOjSkfg2XLGW2G7NEvWz/1DDMDr8+VnFAPn
 /bRIblV8/l2RcJHaLSGYLqtf5CNuvAoziSTURnejZjt230Ro1lUG9+kixebUWQ7TMWeu
 bv2g==
X-Gm-Message-State: APjAAAUu5a2wdHwM3ubXx59d41Uukkq0u33mtvdge+uGqszJWeWUcF6U
 82bd4V4ggG3VzxhXrIXYpgGEGlqJgDdvlCEze54=
X-Google-Smtp-Source: APXvYqz/pKM/AGvj5B9xZyD/Lbbw5g1FZ8JM8QJi6gOTUcJKpP0G+ZSXQyGzH2t30LaVdqX0rgStZQeOUrJKiC31y/0=
X-Received: by 2002:a05:6102:3c5:: with SMTP id
 n5mr9118713vsq.56.1561813312615; 
 Sat, 29 Jun 2019 06:01:52 -0700 (PDT)
MIME-Version: 1.0
Received: by 2002:a9f:2427:0:0:0:0:0 with HTTP; Sat, 29 Jun 2019 06:01:52
 -0700 (PDT)
In-Reply-To: <83v9woftc4.fsf@HIDDEN>
References: <CALR_T9CoA+xto771d_dWhHunByqB4EKO+ODBhXhzUimCfhCy_A@HIDDEN>
 <83y31kfumo.fsf@HIDDEN>
 <CALR_T9CVNkFR9tH28Yowx1afEbitrWR8jyYp8icEBampxgDyBw@HIDDEN>
 <83woh4ftqk.fsf@HIDDEN> <83v9woftc4.fsf@HIDDEN>
From: Sam Halliday <sam.halliday@HIDDEN>
Date: Sat, 29 Jun 2019 14:01:52 +0100
Message-ID: <CALR_T9A23zBOPn3NieGH4O51M8EBHnWkh5ABnjhWursfBxJvyQ@HIDDEN>
Subject: Re: bug#36432: 26.2; SMIE does not request forward tokens when point
 is at point-max
To: Eli Zaretskii <eliz@HIDDEN>
Content-Type: text/plain; charset="UTF-8"
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 36432
Cc: 36432 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

On 29/06/2019, Eli Zaretskii <eliz@HIDDEN> wrote:
>> Date: Sat, 29 Jun 2019 15:42:27 +0300
>> From: Eli Zaretskii <eliz@HIDDEN>
>> Cc: 36432 <at> debbugs.gnu.org
>>
>> How would Emacs know this is the case?  Why don't you also override
>> point-max to make it consistent with those illusory characters?
>
> Or actually insert those characters at EOB, but make them invisible?

I think I'd like to avoid doing that unless I do it for all of the
virtual tokens. I don't know how to do that without it impacting the
underlying source code. Is there more documentation about doing it
that way? It'd be a drastic change in how the lexer is written.

BTW I had a look through smie.el and I can't see anywhere obvious
where (eobp) or (point-max) are called that would lead to the bug I'm
seeing. I'll most likely have to step debug at some point to get to
the bottom of this.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#36432; Package emacs. Full text available.

Message received at 36432 <at> debbugs.gnu.org:


Received: (at 36432) by debbugs.gnu.org; 29 Jun 2019 12:51:39 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Jun 29 08:51:39 2019
Received: from localhost ([127.0.0.1]:43582 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1hhCpP-0006lR-8w
	for submit <at> debbugs.gnu.org; Sat, 29 Jun 2019 08:51:39 -0400
Received: from mail-vs1-f54.google.com ([209.85.217.54]:41608)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <sam.halliday@HIDDEN>) id 1hhCpN-0006lF-8m
 for 36432 <at> debbugs.gnu.org; Sat, 29 Jun 2019 08:51:37 -0400
Received: by mail-vs1-f54.google.com with SMTP id 2so5899176vso.8
 for <36432 <at> debbugs.gnu.org>; Sat, 29 Jun 2019 05:51:37 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=mime-version:in-reply-to:references:from:date:message-id:subject:to
 :cc; bh=JEp6MgH5fxyQHQD/lxx5Fl2UxG7wBl/mo1lTwDnIkYE=;
 b=WXvnRpOo6sQl652ydavog+wFUqj2apmzRhqeD0b0im7oOsqZV8iupMikch1KCCUFY5
 CAJ4ATDcNkcqx52RG0bzIVZJXYu+ZPvdswq3uh3tKN3fyifhxLDflHBeXUHgr2x6QGTR
 dFOo1cQO8E1aEa3bYj5jYMdo808CB3wCRVK66r7CPTtkdem8oSyMbGQxk6lgF8jy42NK
 YXarNcz6qU2N6dNARhiiU5GB1Ua9bj1Mgxj0BXQqo/dIJgFVqKyAv8haXD34po0HwJqC
 Ka4EBnTwXsJJCiQWxN9iTXGllWaQc9VdqqfDRP9RyPzZ36wcMT1cd+0X6GRhbCWpPIpn
 EbSw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:mime-version:in-reply-to:references:from:date
 :message-id:subject:to:cc;
 bh=JEp6MgH5fxyQHQD/lxx5Fl2UxG7wBl/mo1lTwDnIkYE=;
 b=sAVxZ6At0hyhkvXqn5AAXm6JJu+fKvZIxIe7ZWZQ5CH7Ly/+PyK39MtYINsnxPp5Nn
 e3qPBmftJEM7Y7sgTEll0Ofco/y8TOkZnJPx+hB5IxXhJEMqM28ZO+E3kNv+F9F3NwMe
 grTG2j9Ci90qiN8+cGEQgvZyaKDEP734yuz4rjw5H5lwyCAyYeyXEUuTXN4hXRWG+rlc
 7OInZuLosEYlPxQ2MCOntNyzD+wigecLSkSTMIUuYDicYCU2B1rXeGOu3f7P/r0+kSBH
 /1u0mzS9CYrEu47JO1xLfbc/re8Bl5fEhx7n3Ys8dREOUlK/fdi+DpcPUfsAl2z8CEpT
 3rfg==
X-Gm-Message-State: APjAAAUT+7Kwao7T80uxkb+Bi5flBLSrCXpT0xAYH4/tWLVgNUXbaTmO
 fx25b3CI7CDxkuVTawj8EFDhnrPh+h7OTwxojGGbdg==
X-Google-Smtp-Source: APXvYqyBbzyxy/i56h7QO6Ilnq7faQ0lHAemD3PQ63dZHpLWGFTS9DclBoEz9YcBmYryS7tGn4QHQS4RpVpe5ZBq/kA=
X-Received: by 2002:a05:6102:3c5:: with SMTP id
 n5mr9099505vsq.56.1561812691677; 
 Sat, 29 Jun 2019 05:51:31 -0700 (PDT)
MIME-Version: 1.0
Received: by 2002:a9f:2427:0:0:0:0:0 with HTTP; Sat, 29 Jun 2019 05:51:31
 -0700 (PDT)
In-Reply-To: <83woh4ftqk.fsf@HIDDEN>
References: <CALR_T9CoA+xto771d_dWhHunByqB4EKO+ODBhXhzUimCfhCy_A@HIDDEN>
 <83y31kfumo.fsf@HIDDEN>
 <CALR_T9CVNkFR9tH28Yowx1afEbitrWR8jyYp8icEBampxgDyBw@HIDDEN>
 <83woh4ftqk.fsf@HIDDEN>
From: Sam Halliday <sam.halliday@HIDDEN>
Date: Sat, 29 Jun 2019 13:51:31 +0100
Message-ID: <CALR_T9Cwde_5CfPaATpdLZ3yottcqoopq0gKgy50-jKBvHLG0A@HIDDEN>
Subject: Re: bug#36432: 26.2; SMIE does not request forward tokens when point
 is at point-max
To: Eli Zaretskii <eliz@HIDDEN>
Content-Type: text/plain; charset="UTF-8"
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 36432
Cc: 36432 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

On 29/06/2019, Eli Zaretskii <eliz@HIDDEN> wrote:
>> From: Sam Halliday <sam.halliday@HIDDEN>
>> Date: Sat, 29 Jun 2019 13:34:06 +0100
>> Cc: 36432 <at> debbugs.gnu.org
>>
>> > ??? There can be noting at point-max, as that position is beyond the
>> > last buffer position.  Did you mean the position just before that?  Or
>> > am I missing something here?
>> >
>>
>> I mean at point-max.
>>
>> Consider this layout algorithm
>>
>> https://gitlab.com/tseenshe/haskell-tng.el/blob/tng/haskell-tng-layout.el
>>
>> and this lexer
>>
>> https://gitlab.com/tseenshe/haskell-tng.el/blob/tng/haskell-tng-lexer.el
>>
>> that can continue to produce tokens even when the point is at the very
>> end of the buffer.
>
> So you create an illusion of characters beyond the EOB?
>
> How would Emacs know this is the case?

When testing it is possible to keep polling the lexer until it returns
nil when at point-max, rather than looking at `point-max` and giving
up. I think that could work in general inside SMIE.
https://gitlab.com/tseenshe/haskell-tng.el/blob/tng/test/haskell-tng-lexer-test.el

I suspect the example forward lexer, from the documentation
https://www.gnu.org/software/emacs/manual/html_mono/elisp.html#SMIE-Lexer
would be ok in this situation. I'd be concerned that existing lexers
would throw an error if they were polled when at the beginning/end of
the buffer unexpectedly.

BTW, this also happens at the start of the buffer. SMIE doesn't ask
for backwards tokens when at the beginning.

> Why don't you also override
> point-max to make it consistent with those illusory characters?

Hmm, that is a workaround worth exploring. I'm not sure what the
consequences would be of changing something so fundamental. I think
changing SMIE would probably be easier, even with a monkey patch of
the relevant function or advice. I can have a go at trying to do that.
I just need to figure out exactly which function is doing the check.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#36432; Package emacs. Full text available.

Message received at 36432 <at> debbugs.gnu.org:


Received: (at 36432) by debbugs.gnu.org; 29 Jun 2019 12:51:31 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Jun 29 08:51:31 2019
Received: from localhost ([127.0.0.1]:43579 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1hhCpG-0006l7-SN
	for submit <at> debbugs.gnu.org; Sat, 29 Jun 2019 08:51:31 -0400
Received: from eggs.gnu.org ([209.51.188.92]:34102)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1hhCpF-0006ku-N3
 for 36432 <at> debbugs.gnu.org; Sat, 29 Jun 2019 08:51:30 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:47829)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>)
 id 1hhCp4-0004mN-VC; Sat, 29 Jun 2019 08:51:20 -0400
Received: from [176.228.60.248] (port=3156 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1hhCp4-0002P3-7p; Sat, 29 Jun 2019 08:51:18 -0400
Date: Sat, 29 Jun 2019 15:51:07 +0300
Message-Id: <83v9woftc4.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: sam.halliday@HIDDEN
In-reply-to: <83woh4ftqk.fsf@HIDDEN> (message from Eli Zaretskii on Sat, 29
 Jun 2019 15:42:27 +0300)
Subject: Re: bug#36432: 26.2;
 SMIE does not request forward tokens when point is at point-max
References: <CALR_T9CoA+xto771d_dWhHunByqB4EKO+ODBhXhzUimCfhCy_A@HIDDEN>
 <83y31kfumo.fsf@HIDDEN>
 <CALR_T9CVNkFR9tH28Yowx1afEbitrWR8jyYp8icEBampxgDyBw@HIDDEN>
 <83woh4ftqk.fsf@HIDDEN>
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 36432
Cc: 36432 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> Date: Sat, 29 Jun 2019 15:42:27 +0300
> From: Eli Zaretskii <eliz@HIDDEN>
> Cc: 36432 <at> debbugs.gnu.org
> 
> How would Emacs know this is the case?  Why don't you also override
> point-max to make it consistent with those illusory characters?

Or actually insert those characters at EOB, but make them invisible?




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#36432; Package emacs. Full text available.

Message received at 36432 <at> debbugs.gnu.org:


Received: (at 36432) by debbugs.gnu.org; 29 Jun 2019 12:42:45 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Jun 29 08:42:45 2019
Received: from localhost ([127.0.0.1]:43574 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1hhCgm-0006Yq-TI
	for submit <at> debbugs.gnu.org; Sat, 29 Jun 2019 08:42:45 -0400
Received: from eggs.gnu.org ([209.51.188.92]:32974)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1hhCgk-0006Ye-Ry
 for 36432 <at> debbugs.gnu.org; Sat, 29 Jun 2019 08:42:43 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:47767)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>)
 id 1hhCgf-000676-MY; Sat, 29 Jun 2019 08:42:37 -0400
Received: from [176.228.60.248] (port=2631 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1hhCgf-0001tX-6y; Sat, 29 Jun 2019 08:42:37 -0400
Date: Sat, 29 Jun 2019 15:42:27 +0300
Message-Id: <83woh4ftqk.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Sam Halliday <sam.halliday@HIDDEN>
In-reply-to: <CALR_T9CVNkFR9tH28Yowx1afEbitrWR8jyYp8icEBampxgDyBw@HIDDEN>
 (message from Sam Halliday on Sat, 29 Jun 2019 13:34:06 +0100)
Subject: Re: bug#36432: 26.2; SMIE does not request forward tokens when point
 is at point-max
References: <CALR_T9CoA+xto771d_dWhHunByqB4EKO+ODBhXhzUimCfhCy_A@HIDDEN>
 <83y31kfumo.fsf@HIDDEN>
 <CALR_T9CVNkFR9tH28Yowx1afEbitrWR8jyYp8icEBampxgDyBw@HIDDEN>
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 36432
Cc: 36432 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: Sam Halliday <sam.halliday@HIDDEN>
> Date: Sat, 29 Jun 2019 13:34:06 +0100
> Cc: 36432 <at> debbugs.gnu.org
> 
> > ??? There can be noting at point-max, as that position is beyond the
> > last buffer position.  Did you mean the position just before that?  Or
> > am I missing something here?
> >
> 
> I mean at point-max.
> 
> Consider this layout algorithm
> 
> https://gitlab.com/tseenshe/haskell-tng.el/blob/tng/haskell-tng-layout.el
> 
> and this lexer
> 
> https://gitlab.com/tseenshe/haskell-tng.el/blob/tng/haskell-tng-lexer.el
> 
> that can continue to produce tokens even when the point is at the very
> end of the buffer.

So you create an illusion of characters beyond the EOB?

How would Emacs know this is the case?  Why don't you also override
point-max to make it consistent with those illusory characters?




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#36432; Package emacs. Full text available.

Message received at 36432 <at> debbugs.gnu.org:


Received: (at 36432) by debbugs.gnu.org; 29 Jun 2019 12:34:16 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Jun 29 08:34:16 2019
Received: from localhost ([127.0.0.1]:43569 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1hhCYZ-0006N3-TG
	for submit <at> debbugs.gnu.org; Sat, 29 Jun 2019 08:34:16 -0400
Received: from mail-vs1-f46.google.com ([209.85.217.46]:33122)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <sam.halliday@HIDDEN>) id 1hhCYW-0006Mo-Mz
 for 36432 <at> debbugs.gnu.org; Sat, 29 Jun 2019 08:34:13 -0400
Received: by mail-vs1-f46.google.com with SMTP id m8so5902398vsj.0
 for <36432 <at> debbugs.gnu.org>; Sat, 29 Jun 2019 05:34:12 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=mime-version:in-reply-to:references:from:date:message-id:subject:to
 :cc; bh=Df3zBlmsm3MOYHPVi1Z4oGpJIt2YMAzandBQzspzCdk=;
 b=gosrBqpxqhmMFAqfw2/AYvtbzaxI+MLqxsloi6KAxPhnS16hQr7DI6KFAPQjQFwJLQ
 vWuBkDvuQt3fsa6OhOjXm0oljGEv0sI9XWuuMsv5mOi4jFoIRtbzMJtjpzik+EOtjt5p
 RlmhRIF+ml0phYuZXD3jnypPotIsR6rZ0aP8kFs6EPDjopHfFXc+ToPvhuwzX1yDUT74
 uQl5XFMuArDwjwUHhWDOeQ28w8ja6KRmH4I0YnkFD1xyhCaJn3uTFhaMIkxJ0bFrUvgZ
 Rlf91L59/jW7uK0/xf24JeFGXFjroDnbvzsQGV/3oq3AkgcsM193+yFwvaeFuq60KCHr
 YMmw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:mime-version:in-reply-to:references:from:date
 :message-id:subject:to:cc;
 bh=Df3zBlmsm3MOYHPVi1Z4oGpJIt2YMAzandBQzspzCdk=;
 b=r9VA8EdAJLOeaxi5NUYIEXOgdXEn8V0+SGX9H8hlWgC0wNgpyU3aUaNqoArVjvFjig
 lFMHdY5L7PB+owVVj1SrG72D+ORFzPhXF4LNtGk1J7SFa8BEy+9NGbWc7M48Vmwk/WOn
 J2ro3dOlewM4XDk4GOUIFPSTl//ymxu0Ffv6jKU0MVz35FyIWUZJbxhgVwc1TdhZNXXc
 qLPh+0uApLZwpBr/84K+4ih/6RITrU90EX9TORdGjP41VkLsny23RQ3dKHmjP2jbkx48
 Z4zEQYE6A4AohaxD4XJT/3IBUp8eN6TZeUk9Ip3rvboWOIcLB6oKIsml2lL9QJfbIS9v
 RMFg==
X-Gm-Message-State: APjAAAUxUeq5xxTt9Ooud6+tqO7DwyGLEZHPrt0wEuSVmDZyK27Y+6FN
 W68WjC2pJaaqvgIdkTDxILWmJAnJpdXDL1uYr40=
X-Google-Smtp-Source: APXvYqy+4l51zPwO1z1Pc/yfxscS65AZHeobFvOCdZ5GwE0Y1bASYIvSm/hO3KvUlEbJxp4iisnru3wXBK8CNV2DDds=
X-Received: by 2002:a67:d611:: with SMTP id n17mr9443328vsj.156.1561811647019; 
 Sat, 29 Jun 2019 05:34:07 -0700 (PDT)
MIME-Version: 1.0
Received: by 2002:a9f:2427:0:0:0:0:0 with HTTP; Sat, 29 Jun 2019 05:34:06
 -0700 (PDT)
In-Reply-To: <83y31kfumo.fsf@HIDDEN>
References: <CALR_T9CoA+xto771d_dWhHunByqB4EKO+ODBhXhzUimCfhCy_A@HIDDEN>
 <83y31kfumo.fsf@HIDDEN>
From: Sam Halliday <sam.halliday@HIDDEN>
Date: Sat, 29 Jun 2019 13:34:06 +0100
Message-ID: <CALR_T9CVNkFR9tH28Yowx1afEbitrWR8jyYp8icEBampxgDyBw@HIDDEN>
Subject: Re: bug#36432: 26.2; SMIE does not request forward tokens when point
 is at point-max
To: Eli Zaretskii <eliz@HIDDEN>
Content-Type: text/plain; charset="UTF-8"
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 36432
Cc: 36432 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

On 29/06/2019, Eli Zaretskii <eliz@HIDDEN> wrote:
>> From: Sam Halliday <sam.halliday@HIDDEN>
>> Date: Sat, 29 Jun 2019 13:14:01 +0100
>>
>> SMIE (via a `indent-for-tab-command') does not request forward tokens
>> from the lexer when point is at `point-max'.
>>
>> This might sound like a strange bug report: why should smie expect
>> there to be any tokens when it is already at point-max? The answer is:
>> virtual tokens. For example, Haskell may have many closing curly
>> brackets that live at the end of the buffer.
>
> ??? There can be noting at point-max, as that position is beyond the
> last buffer position.  Did you mean the position just before that?  Or
> am I missing something here?
>

I mean at point-max.

Consider this layout algorithm

https://gitlab.com/tseenshe/haskell-tng.el/blob/tng/haskell-tng-layout.el

and this lexer

https://gitlab.com/tseenshe/haskell-tng.el/blob/tng/haskell-tng-lexer.el

that can continue to produce tokens even when the point is at the very
end of the buffer.

e.g.

input https://gitlab.com/tseenshe/haskell-tng.el/blob/tng/test/src/layout.hs
with layout https://gitlab.com/tseenshe/haskell-tng.el/blob/tng/test/src/layout.hs.layout
just tokens https://gitlab.com/tseenshe/haskell-tng.el/blob/tng/test/src/layout.hs.lexer

note the trailing } that exists at the very end of the file. SMIE
always misses this.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#36432; Package emacs. Full text available.

Message received at 36432 <at> debbugs.gnu.org:


Received: (at 36432) by debbugs.gnu.org; 29 Jun 2019 12:23:30 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Jun 29 08:23:30 2019
Received: from localhost ([127.0.0.1]:43563 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1hhCO9-00067R-Od
	for submit <at> debbugs.gnu.org; Sat, 29 Jun 2019 08:23:30 -0400
Received: from eggs.gnu.org ([209.51.188.92]:58680)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1hhCO7-00067D-Lj
 for 36432 <at> debbugs.gnu.org; Sat, 29 Jun 2019 08:23:27 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:47610)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>)
 id 1hhCO1-00071X-TN; Sat, 29 Jun 2019 08:23:22 -0400
Received: from [176.228.60.248] (port=1371 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1hhCO0-0000S8-Dg; Sat, 29 Jun 2019 08:23:20 -0400
Date: Sat, 29 Jun 2019 15:23:11 +0300
Message-Id: <83y31kfumo.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Sam Halliday <sam.halliday@HIDDEN>
In-reply-to: <CALR_T9CoA+xto771d_dWhHunByqB4EKO+ODBhXhzUimCfhCy_A@HIDDEN>
 (message from Sam Halliday on Sat, 29 Jun 2019 13:14:01 +0100)
Subject: Re: bug#36432: 26.2;
 SMIE does not request forward tokens when point is at point-max
References: <CALR_T9CoA+xto771d_dWhHunByqB4EKO+ODBhXhzUimCfhCy_A@HIDDEN>
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 36432
Cc: 36432 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: Sam Halliday <sam.halliday@HIDDEN>
> Date: Sat, 29 Jun 2019 13:14:01 +0100
> 
> SMIE (via a `indent-for-tab-command') does not request forward tokens
> from the lexer when point is at `point-max'.
> 
> This might sound like a strange bug report: why should smie expect
> there to be any tokens when it is already at point-max? The answer is:
> virtual tokens. For example, Haskell may have many closing curly
> brackets that live at the end of the buffer.

??? There can be noting at point-max, as that position is beyond the
last buffer position.  Did you mean the position just before that?  Or
am I missing something here?




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#36432; Package emacs. Full text available.

Message received at submit <at> debbugs.gnu.org:


Received: (at submit) by debbugs.gnu.org; 29 Jun 2019 12:14:08 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Jun 29 08:14:08 2019
Received: from localhost ([127.0.0.1]:43552 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1hhCF6-0005tX-Ea
	for submit <at> debbugs.gnu.org; Sat, 29 Jun 2019 08:14:08 -0400
Received: from lists.gnu.org ([209.51.188.17]:34108)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <sam.halliday@HIDDEN>) id 1hhCF5-0005tP-2v
 for submit <at> debbugs.gnu.org; Sat, 29 Jun 2019 08:14:07 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:34921)
 by lists.gnu.org with esmtp (Exim 4.86_2)
 (envelope-from <sam.halliday@HIDDEN>) id 1hhCF3-0003PF-SV
 for bug-gnu-emacs@HIDDEN; Sat, 29 Jun 2019 08:14:06 -0400
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50,FREEMAIL_FROM
 autolearn=disabled version=3.3.2
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <sam.halliday@HIDDEN>) id 1hhCF3-0008FG-0K
 for bug-gnu-emacs@HIDDEN; Sat, 29 Jun 2019 08:14:05 -0400
Received: from mail-vs1-xe34.google.com ([2607:f8b0:4864:20::e34]:37068)
 by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16)
 (Exim 4.71) (envelope-from <sam.halliday@HIDDEN>)
 id 1hhCF2-0008E1-Sb
 for bug-gnu-emacs@HIDDEN; Sat, 29 Jun 2019 08:14:04 -0400
Received: by mail-vs1-xe34.google.com with SMTP id v6so5869492vsq.4
 for <bug-gnu-emacs@HIDDEN>; Sat, 29 Jun 2019 05:14:04 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=mime-version:from:date:message-id:subject:to;
 bh=vAuiF8FOsgo/aYQbT1VkYC1Z2PJ6tgSCJ74oqWbe2qo=;
 b=aj5uyk8PwjH2bwWMz4nlJIgky5RGstbg1RIhvMfo5o0uz0Qhd/UUWh242MHFMle8xM
 QM5xJjIm2BNhUF2H3ibY0j7pNDTK7EIwBu2ZRx45IscnXeidAkIBfr6IgNxjfZosPOON
 acJRRtfRDRQCATSdP1l2wUp45T0sE/EQjkfk0m954b9p6sKwX6Ur18LCAtcpWhJ1JVOJ
 bbSOMWvGL3EJzSlFXYU3QZ2kbnwaBfDWYV+6nFiwGkPeJIb2SJAVtTHkRNhR9H3+fDPb
 PKx53gpB9BXgQN/ha80DIZl4KnYhPkZytTLz9AEIvSgDvtPlEzI1DRh6Axlngy0dE8HC
 3pSA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:mime-version:from:date:message-id:subject:to;
 bh=vAuiF8FOsgo/aYQbT1VkYC1Z2PJ6tgSCJ74oqWbe2qo=;
 b=j2tQLuHCByjMTTK24zEWpM62cYW8Wk89evEqjy8Mp7T1pC0g1Jaoqesh2wMaDVTMq2
 uGrG+4JWn8dFHcQvO9BgSKXroRUyQi1hApfB2+sVmCSyI6c91nm7E9SJlo3YgToVGWuP
 pFyQH9MhjOMg1klH28/gPQ3mBaMgSylJ0Brc559i/W/rb7eeLU9njLvT8hI+e1+30RUY
 jlaNT6yEK38NJluxr5auP8B4Jfb+PQtSdqvZckVdaQVPI6hQFSK7dNY/65rxzgNQJOoQ
 KX56U/KmUU4fvIVmdR+GOHVaC2FpetdXh2qSk7Auv+jCW+KnqAHZwtefMoEwHNlqoyNy
 p9LQ==
X-Gm-Message-State: APjAAAXsI66GZ14Bwu+yR8HNPQwlzJ0+LXO7mQxq9AF+V8fQC2LrmHtO
 C/Pc9CffTqNKHnrlKH/o0KquIbGQokk9LTQl27UwiA==
X-Google-Smtp-Source: APXvYqzPJDAjQx3pfu0DYPeeGCG1Q3iujpq/CMcCL8FmKl80LEVmrVDyyeI+7mMISAGLDMNyAYmld25RuaalK1SA2GQ=
X-Received: by 2002:a67:d611:: with SMTP id n17mr9401702vsj.156.1561810443071; 
 Sat, 29 Jun 2019 05:14:03 -0700 (PDT)
MIME-Version: 1.0
Received: by 2002:a9f:2427:0:0:0:0:0 with HTTP; Sat, 29 Jun 2019 05:14:01
 -0700 (PDT)
From: Sam Halliday <sam.halliday@HIDDEN>
Date: Sat, 29 Jun 2019 13:14:01 +0100
Message-ID: <CALR_T9CoA+xto771d_dWhHunByqB4EKO+ODBhXhzUimCfhCy_A@HIDDEN>
Subject: 26.2; SMIE does not request forward tokens when point is at point-max
To: bug-gnu-emacs@HIDDEN
Content-Type: text/plain; charset="UTF-8"
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
 recognized.
X-Received-From: 2607:f8b0:4864:20::e34
X-Spam-Score: -1.3 (-)
X-Debbugs-Envelope-To: submit
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -2.3 (--)

SMIE (via a `indent-for-tab-command') does not request forward tokens
from the lexer when point is at `point-max'.

This might sound like a strange bug report: why should smie expect
there to be any tokens when it is already at point-max? The answer is:
virtual tokens. For example, Haskell may have many closing curly
brackets that live at the end of the buffer.

A workaround is to add a few stray newlines to the end of any buffer
that uses SMIE for indentation. Then SMIE will request the next tokens
(even thought there is only whitespace left until the end of the
buffer) and will receive at least one of those virtuals.




Acknowledgement sent to Sam Halliday <sam.halliday@HIDDEN>:
New bug report received and forwarded. Copy sent to bug-gnu-emacs@HIDDEN. Full text available.
Report forwarded to bug-gnu-emacs@HIDDEN:
bug#36432; Package emacs. Full text available.
Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.
Last modified: Mon, 25 Nov 2019 12:00:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.