GNU bug report logs - #39585
after-change-functions called with invalid positions in call-process

Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.

Package: emacs; Reported by: Clément Pit-Claudel <cpitclaudel@HIDDEN>; dated Thu, 13 Feb 2020 03:57:02 UTC; Maintainer for emacs is bug-gnu-emacs@HIDDEN.
bug Marked as fixed in versions 27.1. Request was from Noam Postavsky <npostavs@HIDDEN> to control <at> debbugs.gnu.org. Full text available.

Message received at 39585 <at> debbugs.gnu.org:


Received: (at 39585) by debbugs.gnu.org; 13 Feb 2020 19:15:25 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Feb 13 14:15:25 2020
Received: from localhost ([127.0.0.1]:60466 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1j2JxM-0007DK-Pd
	for submit <at> debbugs.gnu.org; Thu, 13 Feb 2020 14:15:25 -0500
Received: from colin.muc.de ([193.149.48.1]:13056 helo=mail.muc.de)
 by debbugs.gnu.org with smtp (Exim 4.84_2)
 (envelope-from <acm@HIDDEN>) id 1j2JxK-0007DA-LM
 for 39585 <at> debbugs.gnu.org; Thu, 13 Feb 2020 14:15:23 -0500
Received: (qmail 81497 invoked by uid 3782); 13 Feb 2020 19:15:20 -0000
Received: from acm.muc.de (p4FE15C42.dip0.t-ipconnect.de [79.225.92.66]) by
 colin.muc.de (tmda-ofmipd) with ESMTP;
 Thu, 13 Feb 2020 20:15:19 +0100
Received: (qmail 5220 invoked by uid 1000); 13 Feb 2020 19:15:19 -0000
Date: Thu, 13 Feb 2020 19:15:19 +0000
To: =?iso-8859-1?Q?Cl=E9ment?= Pit-Claudel <cpitclaudel@HIDDEN>
Subject: Re: bug#39585: after-change-functions called with invalid positions
 in call-process
Message-ID: <20200213191519.GA5215@ACM>
References: <0a3d3423-965f-4681-b6e2-04b8c1a36a6a@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <0a3d3423-965f-4681-b6e2-04b8c1a36a6a@HIDDEN>
User-Agent: Mutt/1.10.1 (2018-07-13)
X-Delivery-Agent: TMDA/1.1.12 (Macallan)
From: Alan Mackenzie <acm@HIDDEN>
X-Primary-Address: acm@HIDDEN
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 39585
Cc: 39585 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Hello, Clément.

On Wed, Feb 12, 2020 at 22:56:49 -0500, Clément Pit-Claudel wrote:
> Hi all,

> Recent changes in call-process have introduced surprising behavior in
> after-change-functions (and caused bugs in flycheck, e.g.
> https://github.com/flycheck/flycheck/issues/1677).  From the
> documentation, I expected that functions added to the
> after-change-functions hook would be called only with valid positions.
> However, the following snippets shows that it's not the case:

>     (defun ~/after-change (beg end len)
>       (message "(after-change %S %S %S); (buffer %S %S %S)"
>                beg end len (point-min) (point-max) (buffer-size)))

>     (with-current-buffer (get-buffer-create "*tmp*")
>       (make-variable-buffer-local 'after-change-functions)
>       (add-hook 'after-change-functions #'~/after-change)
>       (call-process "echo" nil t t "Hello"))

> Running it repeatedly, this is what I observe:

>     (after-change 7 13 0); (buffer 1 7 6)
>     (after-change 13 19 0); (buffer 1 13 12)
>     (after-change 19 25 0); (buffer 1 19 18)
>     (after-change 25 31 0); (buffer 1 25 24)

> Note how each time the after-change-functions hook is called with a
> region past the end of the buffer.  It's as if after-change-functions
> was in fact call right before the insertion, instead of after.

Well, something like that.  Sorry about it, and thanks for drawing my
attention to it.

> Previous versions of Emacs didn't call after-change-functions in this
> case; it seems that the new behavior was introduced by this commit:

>     commit 224e8d146485ce178086549d41fa8359dcc0e03e
>     Author: Alan Mackenzie <acm@HIDDEN>
>     Date:   Wed Jan 22 19:50:30 2020 +0000

>         Make call_process call signal_after_change.  This fixes bug #38691.

>         Now, functions such as call-proess-region invoke after-change-functions
>         correctly.

>         * src/callproc.c (call_process): Call prepare_to_modify_buffer in a single
>         place, no longer delegating the task to insert_1_both, etc.  Call
>         signal_after_change in each of two code branches, such that
>         before-change-functions and after-change-functions are always called in
>         balanced pairs.

> Alan, is this behavior expected?

No, it is not.  I've applied this corrective patch to the emacs-27
branch:



commit d1e8ce8bb6fadf3d034ae437ff1c1b81be7d5209
Author: Alan Mackenzie <acm@HIDDEN>
Date:   Thu Feb 13 19:00:36 2020 +0000

    Make after-change-functions called from call-process get the correct BEG
    
    This fixes bug #39585.
    
    * src/callproc.c (call_process): Supply the correct CHARPOS to
    signal_after_change (twice).

diff --git a/src/callproc.c b/src/callproc.c
index 07dcc4c3ae..8883415f3f 100644
--- a/src/callproc.c
+++ b/src/callproc.c
@@ -811,7 +811,7 @@ call_process (ptrdiff_t nargs, Lisp_Object *args, int filefd,
 		   && ! CODING_MAY_REQUIRE_DECODING (&process_coding))
             {
               insert_1_both (buf, nread, nread, 0, 0, 0);
-              signal_after_change (PT, 0, nread);
+              signal_after_change (PT - nread, 0, nread);
             }
 	  else
 	    {			/* We have to decode the input.  */
@@ -854,7 +854,8 @@ call_process (ptrdiff_t nargs, Lisp_Object *args, int filefd,
 
 	      TEMP_SET_PT_BOTH (PT + process_coding.produced_char,
 				PT_BYTE + process_coding.produced);
-              signal_after_change (PT, 0, process_coding.produced_char);
+              signal_after_change (PT - process_coding.produced_char,
+                                   0, process_coding.produced_char);
 	      carryover = process_coding.carryover_bytes;
 	      if (carryover > 0)
 		memcpy (buf, process_coding.carryover,



.  It should find its way into the master branch in the usual way.
Please let me know whether you agree with me that this bug can now be
closed.  Thanks!

> Thanks,
> Clément.

-- 
Alan Mackenzie (Nuremberg, Germany).




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#39585; Package emacs. Full text available.

Message received at submit <at> debbugs.gnu.org:


Received: (at submit) by debbugs.gnu.org; 13 Feb 2020 03:56:56 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Wed Feb 12 22:56:56 2020
Received: from localhost ([127.0.0.1]:59259 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1j25cW-0006Fw-HX
	for submit <at> debbugs.gnu.org; Wed, 12 Feb 2020 22:56:56 -0500
Received: from lists.gnu.org ([209.51.188.17]:35287)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <cpitclaudel@HIDDEN>) id 1j25cV-0006Fp-1Q
 for submit <at> debbugs.gnu.org; Wed, 12 Feb 2020 22:56:55 -0500
Received: from eggs.gnu.org ([2001:470:142:3::10]:34410)
 by lists.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <cpitclaudel@HIDDEN>) id 1j25cT-0006BN-OF
 for bug-gnu-emacs@HIDDEN; Wed, 12 Feb 2020 22:56:54 -0500
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50,FREEMAIL_FROM,
 URIBL_BLOCKED autolearn=disabled version=3.3.2
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <cpitclaudel@HIDDEN>) id 1j25cS-0004kt-J3
 for bug-gnu-emacs@HIDDEN; Wed, 12 Feb 2020 22:56:53 -0500
Received: from mail-qk1-x736.google.com ([2607:f8b0:4864:20::736]:32957)
 by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16)
 (Exim 4.71) (envelope-from <cpitclaudel@HIDDEN>)
 id 1j25cS-0004kM-Dl
 for bug-gnu-emacs@HIDDEN; Wed, 12 Feb 2020 22:56:52 -0500
Received: by mail-qk1-x736.google.com with SMTP id h4so4450887qkm.0
 for <bug-gnu-emacs@HIDDEN>; Wed, 12 Feb 2020 19:56:51 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=from:subject:to:message-id:date:user-agent:mime-version
 :content-language:content-transfer-encoding;
 bh=EDbRcgoieR85ItIjV7+TwnQGY7Ys49UriPigWjeZbb0=;
 b=c4Qx8eohJLMNPaDO0QhFo9J7yW9+SherktC6npkOx4X7EjbtjsyxEVg/Loj7A6mS+1
 Yn5gOds4Hu3h9hbTknhY2dtwiYRI5ezQrT6EEtwvs9i5mgRqXWXzdruyMh4JWtaR2a+A
 222+vOGjSiHVfrCEf4Yy3rI7tq+Iw0Gtr0wYKv7zjMF+dr6m9+coqpW60fe5pcjDHogv
 76BNNy/XuxSyXEeGYHfS3i1GxBwYqyasQ0GDDvUGE02uZhQLTIAFNR6f/z2PvDunjzNr
 YyXmg7TqgFCSLtYyIRkcnNZhk11Ry4NckecEBQxNvxmMPZHjMFNwXqc1uGQIPOWy4ApQ
 N72w==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:from:subject:to:message-id:date:user-agent
 :mime-version:content-language:content-transfer-encoding;
 bh=EDbRcgoieR85ItIjV7+TwnQGY7Ys49UriPigWjeZbb0=;
 b=Zn17o7KogNk8NaAEM96Fjhb1ydQcAYBdn+OMoRestsGy352JR0lGGit5FuXiK47YqH
 +u5CUnVpOfeR0W7qpg8ltDLmu6MCaGyAkxULneAWErhwEvUaTQ6RE7BxdL5Ppb0OWvl8
 /TTm5eQ4be7zza56/iHSPezg5rVHTGFjoC+ke+79AwQleGqkGg0hnOYaPHlWHCQ+7jSh
 7yHq6Cpdh56tts13r27wI+6t60MRlWe/zcmfRHiDre1wv8+ilpvCN05pRm6l2gqJrXs3
 AfNBTzZjvtMkXAYnsJxO1Eh7XAFEgAV9C7Xy1aaQ0SZmdK45tQxzNetM/dEfXT7m34zx
 tU7Q==
X-Gm-Message-State: APjAAAVoGC4MudMcSxOFs7Ealc/StaWOF7QRzUargMYtYsXzYMX4cT3+
 U3KinxrLBoa0+SU0dEqn8FOH7vVE
X-Google-Smtp-Source: APXvYqzOOapi9S7vU+8eS+WaahFzdB2yuwIvWMxfZQUjP1vUR/qZS1q3BtlsH1FeC+GGKZ+2H0UVrg==
X-Received: by 2002:a37:9c10:: with SMTP id f16mr9165753qke.275.1581566211053; 
 Wed, 12 Feb 2020 19:56:51 -0800 (PST)
Received: from ?IPv6:2601:184:4180:66e7:141d:589c:d2c9:6cf0?
 ([2601:184:4180:66e7:141d:589c:d2c9:6cf0])
 by smtp.googlemail.com with ESMTPSA id 89sm575163qth.3.2020.02.12.19.56.50
 for <bug-gnu-emacs@HIDDEN>
 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);
 Wed, 12 Feb 2020 19:56:50 -0800 (PST)
From: =?UTF-8?Q?Cl=c3=a9ment_Pit-Claudel?= <cpitclaudel@HIDDEN>
Subject: after-change-functions called with invalid positions in call-process
X-Debbugs-Cc: Alan Mackenzie <acm@HIDDEN>
To: bug-gnu-emacs <bug-gnu-emacs@HIDDEN>
Message-ID: <0a3d3423-965f-4681-b6e2-04b8c1a36a6a@HIDDEN>
Date: Wed, 12 Feb 2020 22:56:49 -0500
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101
 Thunderbird/68.4.1
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Language: en-GB
Content-Transfer-Encoding: 8bit
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
 recognized.
X-Received-From: 2607:f8b0:4864:20::736
X-Spam-Score: 2.3 (++)
X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org",
 has NOT identified this incoming email as spam.  The original
 message has been attached to this so you can view it or label
 similar future email.  If you have any questions, see
 the administrator of that system for details.
 Content preview:  Hi all,
 Recent changes in call-process have introduced surprising
 behavior in after-change-functions (and caused bugs in flycheck,
 e.g. https://github.com/flycheck/flycheck/issues/1677).
 From the documentatio [...] 
 Content analysis details:   (2.3 points, 10.0 required)
 pts rule name              description
 ---- ---------------------- --------------------------------------------------
 0.0 URIBL_BLOCKED          ADMINISTRATOR NOTICE: The query to URIBL was
 blocked.  See
 http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block
 for more information. [URIs: muc.de]
 0.0 SPF_HELO_NONE          SPF: HELO does not publish an SPF Record
 1.0 SPF_SOFTFAIL           SPF: sender does not match SPF record (softfail)
 0.0 FREEMAIL_FROM          Sender email is commonly abused enduser mail
 provider (cpitclaudel[at]gmail.com)
 -0.7 RCVD_IN_DNSWL_LOW      RBL: Sender listed at https://www.dnswl.org/,
 low trust [209.51.188.17 listed in list.dnswl.org]
 2.0 SPOOFED_FREEMAIL       No description available.
X-Debbugs-Envelope-To: submit
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.7 (/)

Hi all,

Recent changes in call-process have introduced surprising behavior in after-change-functions (and caused bugs in flycheck, e.g. https://github.com/flycheck/flycheck/issues/1677).  From the documentation, I expected that functions added to the after-change-functions hook would be called only with valid positions.  However, the following snippets shows that it's not the case:

    (defun ~/after-change (beg end len)
      (message "(after-change %S %S %S); (buffer %S %S %S)"
               beg end len (point-min) (point-max) (buffer-size)))

    (with-current-buffer (get-buffer-create "*tmp*")
      (make-variable-buffer-local 'after-change-functions)
      (add-hook 'after-change-functions #'~/after-change)
      (call-process "echo" nil t t "Hello"))

Running it repeatedly, this is what I observe:

    (after-change 7 13 0); (buffer 1 7 6)
    (after-change 13 19 0); (buffer 1 13 12)
    (after-change 19 25 0); (buffer 1 19 18)
    (after-change 25 31 0); (buffer 1 25 24)

Note how each time the after-change-functions hook is called with a region past the end of the buffer.  It's as if after-change-functions was in fact call right before the insertion, instead of after.

Previous versions of Emacs didn't call after-change-functions in this case; it seems that the new behavior was introduced by this commit:

    commit 224e8d146485ce178086549d41fa8359dcc0e03e
    Author: Alan Mackenzie <acm@HIDDEN>
    Date:   Wed Jan 22 19:50:30 2020 +0000

        Make call_process call signal_after_change.  This fixes bug #38691.

        Now, functions such as call-proess-region invoke after-change-functions
        correctly.

        * src/callproc.c (call_process): Call prepare_to_modify_buffer in a single
        place, no longer delegating the task to insert_1_both, etc.  Call
        signal_after_change in each of two code branches, such that
        before-change-functions and after-change-functions are always called in
        balanced pairs.

Alan, is this behavior expected?

Thanks,
Clément.




Acknowledgement sent to Clément Pit-Claudel <cpitclaudel@HIDDEN>:
New bug report received and forwarded. Copy sent to acm@HIDDEN, bug-gnu-emacs@HIDDEN. Full text available.
Report forwarded to acm@HIDDEN, bug-gnu-emacs@HIDDEN:
bug#39585; Package emacs. Full text available.
Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.
Last modified: Mon, 24 Feb 2020 18:30:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.