GNU bug report logs - #47711
27.1; Deferred highlighting support in `completion-all-completions', `vertico--all-completions`

Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.

Package: emacs; Reported by: Daniel Mendler <mail@HIDDEN>; dated Sun, 11 Apr 2021 20:52:01 UTC; Maintainer for emacs is bug-gnu-emacs@HIDDEN.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 16 Aug 2021 13:42:06 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Aug 16 09:42:06 2021
Received: from localhost ([127.0.0.1]:49027 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mFcsP-0001Z2-Qs
	for submit <at> debbugs.gnu.org; Mon, 16 Aug 2021 09:42:06 -0400
Received: from mail-pj1-f45.google.com ([209.85.216.45]:54162)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <joaotavora@HIDDEN>)
 id 1mFcsL-0001Xp-Kg; Mon, 16 Aug 2021 09:42:03 -0400
Received: by mail-pj1-f45.google.com with SMTP id j1so26563685pjv.3;
 Mon, 16 Aug 2021 06:42:01 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=mime-version:references:in-reply-to:from:date:message-id:subject:to
 :cc:content-transfer-encoding;
 bh=JB2/qZP1S4rAp75M/+Afkq6jb97NXO4w7kLkGR+djWA=;
 b=Yu2KcewzKG/qJlN0SihwCY7y+vysP54IOuKDS1OtJoTmyJ5QbHu8KwmmqbcSLaBH0Y
 vIpZacV4rnocvYuWOwF2pKRd5iCSmpD03ZiqKxTsDHx1p6fU7aN451kt2gDmFPmwRr/Q
 w07ERlwnkDVHODlxIwGoiwmAjVmM//RwSf/l+zvy6uI00hFAohDdC5VVKSGx2oKyZU+s
 hW01rux4d+a3yO3x0iLOu7Fozq9QdHkPJYpdI485/2Gt7DbRrHnxaRuTNpIBcfvrXSvL
 L5mqdU+Jir5x1GIh7VF7RZmwCg/74esiRNUl6IhERyfW7I0wF9Mnne3+7dWmAPwkCDo+
 zLqw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:mime-version:references:in-reply-to:from:date
 :message-id:subject:to:cc:content-transfer-encoding;
 bh=JB2/qZP1S4rAp75M/+Afkq6jb97NXO4w7kLkGR+djWA=;
 b=DiRTqQ3LsdNDDqeCJkEjliIjHe+iliARb7CpltUjTVZ2kN4ZMrknQl6yhq8/WgmjRc
 ctpRAGEDhTYhxoyTn6mzD9R8l172xopveBjJQUEzleX+8mZZf8+3sT98gIK1ShZdv+0t
 c1e3kTOn2skcJ/Zxt8MqRRascJvVRo9RnUwD94qrdPI7ufpIp7ywAaE35JAGdKDtsXmY
 nCxOlEG9Mvqb+CAvPwFVFbT82Py8zgOgCvAM6fPCEamNRuGQ3eaywl86mRohKfsJ3ag+
 lqi2bsVH6mnAD3t+rqFtF1a0y/ddLL9paCV+tMn27NQQWA3ac21Jt+L/Mt9/b5lbS3Bp
 y36w==
X-Gm-Message-State: AOAM532XS5BhQp3s9lv+EinSmR3b2KHUuNXoR3o6aaSJ1ldhOQBwAAmF
 HlzUh3KZFmAT3r0jbZdis/tZBpU8iUHR3jaFu/0=
X-Google-Smtp-Source: ABdhPJyePzhalqU929+kNUL5zmDJQwM7w+KfJDSztpA6kDzQ14RSTMdRFTJBW6cacsd8RedD0Smev4B6CV25zspzL34=
X-Received: by 2002:a05:6a00:164d:b0:3e1:ab35:f3e4 with SMTP id
 m13-20020a056a00164d00b003e1ab35f3e4mr7474897pfc.42.1629121315603; Mon, 16
 Aug 2021 06:41:55 -0700 (PDT)
MIME-Version: 1.0
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <f8c4419a-7e83-b8a9-347c-0221a2052376@HIDDEN>
 <fff6e291-bbcc-93e3-92e7-00af4664b8d4@HIDDEN>
 <3d3f894f-a6fa-53ae-5d50-c3aa9bffc73e@HIDDEN>
 <CALDnm53xh6boc6T9XSC4jZpkCPS3RB8JFNb12BpkoeGYg3Q9RQ@HIDDEN>
 <56ab18b1-4348-9b2c-85bb-af9b76cd429a@HIDDEN>
 <CALDnm52XM+S07OPXO3qtfLN4pup-kT2U=WRO9dUo3tv9uYJWXw@HIDDEN>
 <38a06f3c-4a7a-755c-c99b-708f91afabfa@HIDDEN>
 <CALDnm537=LHAoaGqPpy7=JOPsYn=z+=Q6QW=b9bjrAWOtxqRaQ@HIDDEN>
 <9f59f87c-2489-aaa0-5b3f-0e911b7ffa6c@HIDDEN>
 <CALDnm508EK5qM+AaOOJiNDm-=ORCuOieYpkjO=OSRas+_WTFwQ@HIDDEN>
 <8a36e61a-1c5b-bf3b-a454-077348589c4f@HIDDEN> <87y29471ov.fsf@HIDDEN>
 <837dgob6yo.fsf@HIDDEN> <31da8f43-02f7-e6dd-ab38-2160af138a4a@HIDDEN>
 <83tujp8vd9.fsf@HIDDEN> <769c0ee0-6d6f-6177-7929-1ac4ada30951@HIDDEN>
In-Reply-To: <769c0ee0-6d6f-6177-7929-1ac4ada30951@HIDDEN>
From: =?UTF-8?B?Sm/Do28gVMOhdm9yYQ==?= <joaotavora@HIDDEN>
Date: Mon, 16 Aug 2021 14:41:42 +0100
Message-ID: <CALDnm539fmNj=TOTA7C25tXC6VOoE91ucWh5UubeUGC+meL6QA@HIDDEN>
Subject: Re: bug#48841: bug#47711: [PATCH VERSION 2] Add new
 `completion-filter-completions` API and deferred highlighting
To: Dmitry Gutov <dgutov@HIDDEN>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 47711
Cc: Daniel Mendler <mail@HIDDEN>, Eli Zaretskii <eliz@HIDDEN>,
 Stefan Monnier <monnier@HIDDEN>, 48841 <at> debbugs.gnu.org,
 47711 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

On Mon, Aug 16, 2021 at 2:38 PM Dmitry Gutov <dgutov@HIDDEN> wrote:

> And speaking of "only private properties", the completion-score property
> can be used by downstream code, with all the associated potential for
> trouble.

That's true.  When I created it, I meant for it to be private, I think,
but indeed did forget to mark it as such.  It is not documented anywhere
but that hasn't stopped anyone in the past it, indeed.Can you point to
place(s) where it is indeed used other than the flex machinery of
`minibuffer.el`?  Thanks.

Jo=C3=A3o T=C3=A1vora




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 16 Aug 2021 13:38:30 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Aug 16 09:38:30 2021
Received: from localhost ([127.0.0.1]:49008 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mFcow-0001SP-7K
	for submit <at> debbugs.gnu.org; Mon, 16 Aug 2021 09:38:30 -0400
Received: from mail-wr1-f49.google.com ([209.85.221.49]:33578)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <raaahh@HIDDEN>)
 id 1mFcot-0001S9-SH; Mon, 16 Aug 2021 09:38:29 -0400
Received: by mail-wr1-f49.google.com with SMTP id r7so23782499wrs.0;
 Mon, 16 Aug 2021 06:38:27 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=sender:subject:to:cc:references:from:message-id:date:user-agent
 :mime-version:in-reply-to:content-language:content-transfer-encoding;
 bh=SqI0rHgABZha2WvRm9Y8w2oaSldHqcSN+3/YdatFStM=;
 b=P3asAjLAFMnMQalTG7bE/JtxvtRSVYQrO1SO8ibPC68FgWWLijkIgEzFGRkLUP2rWa
 tMbUCkJugsQ4qIarmfIjj1PWoLhlYNgvAYV9wNwTNdW6OVBRiSUycs4kUA0OFWlyPpXT
 sO0Pba0/zS2F9MfmmCRocbHU7XNfjnjMbkqC4ufJd330VYNR+u88qcW6LPHzjZh+5ENa
 2IfCnRxfnk03f4x03x9BMqOEEQZvbt2AZr2Oca2waE45jvhQOFcYLzoCbRagZn673Oqh
 VDfv/grSzZxbQIh9SrLVtdegvQ868Ok/dwJ01iXR47wNaW+Ft6oLaSqqgaG61KgifutI
 +Peg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:sender:subject:to:cc:references:from:message-id
 :date:user-agent:mime-version:in-reply-to:content-language
 :content-transfer-encoding;
 bh=SqI0rHgABZha2WvRm9Y8w2oaSldHqcSN+3/YdatFStM=;
 b=QB5A7bSEXh1zrowP8TpCIpUza44usXulBEwjSZxpOeybMnsKTY4i/b9TWfEhbMHpq/
 G1bnDU/wTJcUFFNkFmMT3yd2gUORdRC6+u7fk/2rycZlBR9uw05aSrL5cGD4SWm4BqTr
 UZL7qJWMCzlnXTnmxZoj8siFD/lHfKgBNP1TJnbSCAEYJIhwK2fsl8rYE1ILnCuyibIg
 z5GIQ8FQyBztPNmJuCQWzyZJY66Ir522fy8gQuMsbxRyNB5U1u/LHVfU87VJHEO3dS6J
 yTOknHBmqxgK5mzakd7wj6kGYrxdHXIFUgH2QPCwSf1qGovjjKp5jezJs+4bSjOX5lrn
 7a8A==
X-Gm-Message-State: AOAM5315ZW6diOufYdM5GevSsitT+N4503MMIfsia+3GVa2/FAs0sa8/
 YaeyMNmIJQY4+QusfTPoI3UU4DwA5tc=
X-Google-Smtp-Source: ABdhPJzZKIM/o/bTUBPGReFn6m2jVcVa2wAgTqynCgT+U8mlc53Mkju/lc+p6FTVvnJZe0381ucgzA==
X-Received: by 2002:a5d:534e:: with SMTP id t14mr18793923wrv.109.1629121101904; 
 Mon, 16 Aug 2021 06:38:21 -0700 (PDT)
Received: from [192.168.0.6] ([46.251.119.176])
 by smtp.googlemail.com with ESMTPSA id v17sm13960685wro.45.2021.08.16.06.38.20
 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);
 Mon, 16 Aug 2021 06:38:20 -0700 (PDT)
Subject: Re: bug#48841: bug#47711: [PATCH VERSION 2] Add new
 `completion-filter-completions` API and deferred highlighting
To: Eli Zaretskii <eliz@HIDDEN>
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <f8c4419a-7e83-b8a9-347c-0221a2052376@HIDDEN>
 <fff6e291-bbcc-93e3-92e7-00af4664b8d4@HIDDEN>
 <3d3f894f-a6fa-53ae-5d50-c3aa9bffc73e@HIDDEN>
 <CALDnm53xh6boc6T9XSC4jZpkCPS3RB8JFNb12BpkoeGYg3Q9RQ@HIDDEN>
 <56ab18b1-4348-9b2c-85bb-af9b76cd429a@HIDDEN>
 <CALDnm52XM+S07OPXO3qtfLN4pup-kT2U=WRO9dUo3tv9uYJWXw@HIDDEN>
 <38a06f3c-4a7a-755c-c99b-708f91afabfa@HIDDEN>
 <CALDnm537=LHAoaGqPpy7=JOPsYn=z+=Q6QW=b9bjrAWOtxqRaQ@HIDDEN>
 <9f59f87c-2489-aaa0-5b3f-0e911b7ffa6c@HIDDEN>
 <CALDnm508EK5qM+AaOOJiNDm-=ORCuOieYpkjO=OSRas+_WTFwQ@HIDDEN>
 <8a36e61a-1c5b-bf3b-a454-077348589c4f@HIDDEN> <87y29471ov.fsf@HIDDEN>
 <837dgob6yo.fsf@HIDDEN> <31da8f43-02f7-e6dd-ab38-2160af138a4a@HIDDEN>
 <83tujp8vd9.fsf@HIDDEN>
From: Dmitry Gutov <dgutov@HIDDEN>
Message-ID: <769c0ee0-6d6f-6177-7929-1ac4ada30951@HIDDEN>
Date: Mon, 16 Aug 2021 16:38:19 +0300
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.11.0
MIME-Version: 1.0
In-Reply-To: <83tujp8vd9.fsf@HIDDEN>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
X-Spam-Score: 0.4 (/)
X-Debbugs-Envelope-To: 47711
Cc: mail@HIDDEN, 47711 <at> debbugs.gnu.org, monnier@HIDDEN,
 48841 <at> debbugs.gnu.org, joaotavora@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.6 (/)

On 16.08.2021 14:46, Eli Zaretskii wrote:

>>> Text properties are stored separately from the string, so I don't
>>> think adding properties can in general be referred to as "change".
>>
>> Are you thinking of C strings?
> 
> No, about the implementation of Lisp strings in Emacs.

I was talking about their behavior.

>> Lisp strings carry text properties in addition to the array of
>> characters. It doesn't really matter where in the memory the properties
>> and the characters reside.
> 
> Well, it does, at least in some situations.  The string text is not
> affected, and so the code which processes the string will not notice
> that it has a property about which that code has no idea.  Only
> properties that are known to the processing code can affect it;
> non-standard properties private to some other code will generally pass
> unnoticed.

I don't think anybody was suggesting that changing text properties 
changes the character codes inside the "C string" part of the Lisp string.

>>> I'm not sure in the context of completion there's any reason to count
>>> as "change" adding properties that don't affect display.
>>
>> For the context in question, whether the properties affect display is
>> not particularly important. Properties affecting display just make it
>> easier to notice that something's wrong. Bug involving other properties
>> should be more difficult to investigate.
> 
> Once again, if some code invents its private property, not used
> anywhere else and not documented anywhere else, then putting such a
> property on a string has very high chances of going unnoticed.  I hope
> this consideration helps this discussion, because saying that
> properties change a string blurs the distinction between actually
> changing the string text or its properties that affect many parts in
> Emacs, and adding some obscure property that is not known to anyone.

What muddies the water is arguing against a solid engineering principle 
with statements like "those mutations are not mutations".

Yes, when the properties are prefixed, the damage is reduced. Even then, 
that increases the possibility of introducing bugs in the very code that 
sets those properties (like having different code paths where one branch 
sets them and another does not; forgetting to clear them in the other 
branch; having subsequent code use the property values set by some 
previous invocation of the code in question where it took another 
branch; not to mention the potential troubles with parallel execution, 
which is not a real concern these days, but we're designing for years 
ahead, and someday it can be). Memory leaks, too.

Our completion pipeline has multiple interchangeable/pluggable parts, so 
we have to be on the lookout even for problems which do not reproduce 
with stock Emacs, and that requires solid abstractions.

And speaking of "only private properties", the completion-score property 
can be used by downstream code, with all the associated potential for 
trouble.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 16 Aug 2021 13:21:37 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Aug 16 09:21:37 2021
Received: from localhost ([127.0.0.1]:48985 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mFcYX-0007Ll-BU
	for submit <at> debbugs.gnu.org; Mon, 16 Aug 2021 09:21:37 -0400
Received: from eggs.gnu.org ([209.51.188.92]:59452)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>)
 id 1mFcYS-0007LS-Lp; Mon, 16 Aug 2021 09:21:32 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:47768)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <eliz@HIDDEN>)
 id 1mFcYM-00022e-Bl; Mon, 16 Aug 2021 09:21:22 -0400
Received: from 84.94.185.95.cable.012.net.il ([84.94.185.95]:3998
 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1mFcYL-0004j1-Nk; Mon, 16 Aug 2021 09:21:22 -0400
Date: Mon, 16 Aug 2021 16:21:17 +0300
Message-Id: <83bl5x8r02.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Daniel Mendler <mail@HIDDEN>
In-Reply-To: <f53a5330-fe7c-269c-ce34-9f2870e24e61@HIDDEN> (message
 from Daniel Mendler on Mon, 16 Aug 2021 14:49:45 +0200)
Subject: Re: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new
 `completion-filter-completions` API and deferred highlighting
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <38a06f3c-4a7a-755c-c99b-708f91afabfa@HIDDEN>
 <CALDnm537=LHAoaGqPpy7=JOPsYn=z+=Q6QW=b9bjrAWOtxqRaQ@HIDDEN>
 <9f59f87c-2489-aaa0-5b3f-0e911b7ffa6c@HIDDEN>
 <CALDnm508EK5qM+AaOOJiNDm-=ORCuOieYpkjO=OSRas+_WTFwQ@HIDDEN>
 <8a36e61a-1c5b-bf3b-a454-077348589c4f@HIDDEN> <87y29471ov.fsf@HIDDEN>
 <837dgob6yo.fsf@HIDDEN> <87wnootec0.fsf@HIDDEN>
 <5d70b0ad-3838-ddb8-d341-9a5531d9da73@HIDDEN>
 <CALDnm51xYehwuGhMubovLTv_nqtdJYvOmH2e-bH3cCjf0PS2WQ@HIDDEN>
 <0cbf224b-b382-8a02-3f06-9f6d7e5e9741@HIDDEN>
 <CALDnm52x_S2UXE=YVokCWEZf9Gqc0O+g4ZVhi+R60SCeq6JtzA@HIDDEN>
 <eecaa682-496e-8173-caf7-2f9781cc7102@HIDDEN> <87a6lihv7b.fsf@HIDDEN>
 <63e03b18-db81-3b11-c692-0af9df20c506@HIDDEN>
 <CALDnm50W_=Y_zP7KLznqcqSx8KJp2-0u2NwpuEGzzAUzCNHvKQ@HIDDEN>
 <c95dea17-ca29-4dee-d53d-b5d46f7f2cf8@HIDDEN>
 <83k0kl8sxm.fsf@HIDDEN>
 <f53a5330-fe7c-269c-ce34-9f2870e24e61@HIDDEN>
MIME-version: 1.0
Content-type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 47711
Cc: monnier@HIDDEN, 48841 <at> debbugs.gnu.org, dgutov@HIDDEN,
 joaotavora@HIDDEN, larsi@HIDDEN, 47711 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

> Cc: joaotavora@HIDDEN, 48841 <at> debbugs.gnu.org, dgutov@HIDDEN,
>  larsi@HIDDEN, monnier@HIDDEN, 47711 <at> debbugs.gnu.org
> From: Daniel Mendler <mail@HIDDEN>
> Date: Mon, 16 Aug 2021 14:49:45 +0200
> 
> On 8/16/21 2:39 PM, Eli Zaretskii wrote:
> >> João, I am giving hard examples here. What is not an example about
> >> "memory leak" or making debugging output verbose thanks to the attached
> >> string properties?
> > 
> > FWIW, I also don't understand how adding properties could cause a
> > memory leak.  When a string is GCed, its properties get GCed as well,
> > all of them.  Am I missing something?
> 
> If you add string properties to all symbol names this memory will stay
> alive for much longer than necessary.

That's a very extreme example, something that I wouldn't expect a Lisp
program to do, without removing the properties shortly thereafter.
And even that isn't a leak.

Note that we already put all kind of properties (although not text
properties) on symbols.

> > As to more difficult debugging, I think adding a couple of properties
> > that have simple structure will not impair debugging too much.
> > Strings with many properties are not uncommon in Emacs, so we already
> > have to deal with that.
> 
> I disagree with that. We are talking about adding string properties to
> every symbol name. This is a global side effect and different from
> adding string properties to a set of isolated string in a controlled
> manner. I also don't understand why one would even want to take any
> chances here given that the feature can be implemented in a way which
> avoids this global side effect entirely as my patch shows.

I understand your aversion from such global effects, but I was talking
specifically about the debugging difficulties.

> > I would indeed suggest both to make sure there's no performance
> > regressions, and would like to see some data similar to what João
> > presented, which backs up your assessments about your proposal being
> > faster.  Since performance is the main motivation for these changes, I
> > think it's important for us to be on the same page wrt facts related
> > to performance, before we make the decision how to proceed.
> 
> I will prepare some benchmarks.

Thank you.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 16 Aug 2021 12:59:10 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Aug 16 08:59:10 2021
Received: from localhost ([127.0.0.1]:48923 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mFcCn-0002Pk-QH
	for submit <at> debbugs.gnu.org; Mon, 16 Aug 2021 08:59:10 -0400
Received: from eggs.gnu.org ([209.51.188.92]:54842)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>)
 id 1mFcCi-0002Ok-Hy; Mon, 16 Aug 2021 08:59:04 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:47060)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <eliz@HIDDEN>)
 id 1mFcCc-0004HU-LT; Mon, 16 Aug 2021 08:58:54 -0400
Received: from 84.94.185.95.cable.012.net.il ([84.94.185.95]:2559
 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1mFcCc-0006ac-7q; Mon, 16 Aug 2021 08:58:54 -0400
Date: Mon, 16 Aug 2021 15:58:47 +0300
Message-Id: <83h7fp8s1k.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Daniel Mendler <mail@HIDDEN>
In-Reply-To: <e2427908-17c5-ad18-a50a-2f2ebb61982c@HIDDEN> (message
 from Daniel Mendler on Mon, 16 Aug 2021 11:42:22 +0200)
Subject: Re: bug#48841: [PATCH] Add new `completion-filter-completions` API
 and deferred highlighting
References: <f94541d3-5f93-9a23-8256-3bbc453f127a@HIDDEN>
 <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <837dgrdrec.fsf@HIDDEN>
 <aff2bd3a-ea3e-1ac6-cf0d-2785c6591c84@HIDDEN>
 <83mtpkbky3.fsf@HIDDEN>
 <e2427908-17c5-ad18-a50a-2f2ebb61982c@HIDDEN>
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 47711
Cc: 48841 <at> debbugs.gnu.org, 47711 <at> debbugs.gnu.org, joaotavora@HIDDEN,
 monnier@HIDDEN, dgutov@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> Cc: joaotavora@HIDDEN, dgutov@HIDDEN, monnier@HIDDEN,
>  48841 <at> debbugs.gnu.org, 47711 <at> debbugs.gnu.org
> From: Daniel Mendler <mail@HIDDEN>
> Date: Mon, 16 Aug 2021 11:42:22 +0200
> 
> >> To address your technical comment - this variable is precisely what one
> >> of the technical difficulties mentioned in my other mail is about.  The
> >> question is how we can retain backward compatibility of the completion
> >> style 'all' functions, e.g., 'completion-basic-all-completions', while
> >> still allowing the function to return the newly introduced alist format
> >> with more data, which enables 'completion-filter-completions' to perform
> >> the efficient deferred highlighting.
> > 
> > I understand, but given that we provide this for other packages, it
> > shouldn't be an internal variable.
> 
> No, we explicitly don't provide this variable for other packages. It is
> explicitly only meant to be used for the existing completion styles
> emacs21, emacs22, basic, substring, partial-completion, initials and
> flex to opt-in in a backward-compatible/calling convention preserving
> way to the alist return format. The idea is to keep the existing APIs
> fully backward compatible.
> 
> Other packages should select the format returned from the completion
> styles differently. They should return the alist format on Emacs 28 or
> if the API 'completion-filter-completions' API is present. In the not so
> near future external packages which support only Emacs 28 and upwards
> will then only return the alist format and don't have to perform any
> detection anymore.

What if some package outside minibuffer.el will want to control the
format of the returned value, for some reason, like support for old
Emacs versions? are we going to disallow that?

> >> The new API 'completion-filter-completions' will substitute the existing
> >> API 'completion-all-completions'.
> > 
> > That's your hope, and I understand.  But we as a project didn't yet
> > decide to deprecate the original APIs, so talking about superseding is
> > premature.
> 
> It is not the hope - it is the explicit goal. The API has been designed
> to replace the existing API 'completion-all-completions'.

A goal is not a fact.  Until that goal is reached, we cannot in good
faith tell people an API is superseded.

> We can of
> course tone this down. However I, as a package author, would appreciate
> if Emacs tells me when a newer API aims to replace another API and when
> the documentation is explicit about it.

That's understood, and when we make that decision, we will of course
announce it.  But we didn't do so yet, and this discussion is not even
about that decision.  It could be, for example, that both APIs will
live side by side until we decide whether to deprecate the old one.

> Of course if you decide to have
> the doc strings written in a different tone, I will adapt my patch
> accordingly. Here I am just explaining why I chose the word "superseded"
> instead of a more neutral word.

I understand your motivation, I'm just saying that we cannot announce
deprecation before we actually decide to deprecate.

> > But the name says "filter completions".  Which would mean you take a
> > list of completions and filter out some of them.  A completion table
> > is much more general object than a list of strings.  Thus, I think
> > using "filter" here is sub-optimal.
> 
> Okay, you are right about this. But I think even if the name
> 'completion-filter-completions' is not 100% precise it still conveys
> what the API is about. 'completion-completions-alist' or
> 'completion-all-completions-as-alist' are valid names of course, but I
> dislike them for their verbosity. Already 'completion-all-completions'
> is quite verbose. A strong argument to use this long name is that the
> completion style functions are still called
> 'completion-basic-all-completions' etc. But if we accept that the new
> API 'completion-filter-completions' will actually supersede the existing
> API 'completion-all-completions' it makes sense to use a name which will
> not hurt our eyes in the long run. However this is of course just a
> personal preference of mine. I don't want to spent much time with name
> bikeshedding discussions. If you decide on a name, I will adapt my patch
> accordingly.

Emacs is frequently accused in having names that are hard to
discover.  The only time where we can improve that is when a symbol is
introduced, because later it's impossible for compatibility reasons.
So I'd like to come up with a good name before we install the changes.

That said, I'll let others chime in and agree or disagree with the
name you've chosen.

Thanks.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 16 Aug 2021 12:49:56 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Aug 16 08:49:56 2021
Received: from localhost ([127.0.0.1]:48897 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mFc3v-0002Ay-VF
	for submit <at> debbugs.gnu.org; Mon, 16 Aug 2021 08:49:56 -0400
Received: from server.qxqx.de ([178.63.65.180]:59823 helo=mail.qxqx.de)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mail@HIDDEN>)
 id 1mFc3u-0002Af-FE; Mon, 16 Aug 2021 08:49:55 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=qxqx.de;
 s=mail1392553390; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:
 MIME-Version:Date:Message-ID:From:References:Cc:To:Subject:Sender:Reply-To:
 Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender:
 Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:
 List-Subscribe:List-Post:List-Owner:List-Archive;
 bh=kZrT7LuVRK9XGgsyWGpuxLP8FMvP7eeoAoRJa1adfP8=; b=BH6TKB+fVaN/xvMTztdySZbl+R
 ICbPRP+GFV1yS4R2oqMVSOGN5kRp6rCsBT2bq3J/+7c9HG0cOZoCV5nGxnAbDenMkZDpZhnpX36Vg
 axzIfoOp3ByTbm/Tw1zZrSyLjmDosJbk1jC0bANXga0bAhGgMq341ZOM0He1EeQvUBmc=;
Subject: Re: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new
 `completion-filter-completions` API and deferred highlighting
To: Eli Zaretskii <eliz@HIDDEN>
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <38a06f3c-4a7a-755c-c99b-708f91afabfa@HIDDEN>
 <CALDnm537=LHAoaGqPpy7=JOPsYn=z+=Q6QW=b9bjrAWOtxqRaQ@HIDDEN>
 <9f59f87c-2489-aaa0-5b3f-0e911b7ffa6c@HIDDEN>
 <CALDnm508EK5qM+AaOOJiNDm-=ORCuOieYpkjO=OSRas+_WTFwQ@HIDDEN>
 <8a36e61a-1c5b-bf3b-a454-077348589c4f@HIDDEN> <87y29471ov.fsf@HIDDEN>
 <837dgob6yo.fsf@HIDDEN> <87wnootec0.fsf@HIDDEN>
 <5d70b0ad-3838-ddb8-d341-9a5531d9da73@HIDDEN>
 <CALDnm51xYehwuGhMubovLTv_nqtdJYvOmH2e-bH3cCjf0PS2WQ@HIDDEN>
 <0cbf224b-b382-8a02-3f06-9f6d7e5e9741@HIDDEN>
 <CALDnm52x_S2UXE=YVokCWEZf9Gqc0O+g4ZVhi+R60SCeq6JtzA@HIDDEN>
 <eecaa682-496e-8173-caf7-2f9781cc7102@HIDDEN> <87a6lihv7b.fsf@HIDDEN>
 <63e03b18-db81-3b11-c692-0af9df20c506@HIDDEN>
 <CALDnm50W_=Y_zP7KLznqcqSx8KJp2-0u2NwpuEGzzAUzCNHvKQ@HIDDEN>
 <c95dea17-ca29-4dee-d53d-b5d46f7f2cf8@HIDDEN>
 <83k0kl8sxm.fsf@HIDDEN>
From: Daniel Mendler <mail@HIDDEN>
Message-ID: <f53a5330-fe7c-269c-ce34-9f2870e24e61@HIDDEN>
Date: Mon, 16 Aug 2021 14:49:45 +0200
MIME-Version: 1.0
In-Reply-To: <83k0kl8sxm.fsf@HIDDEN>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 8bit
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 47711
Cc: monnier@HIDDEN, 48841 <at> debbugs.gnu.org, dgutov@HIDDEN,
 joaotavora@HIDDEN, larsi@HIDDEN, 47711 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

On 8/16/21 2:39 PM, Eli Zaretskii wrote:
>> João, I am giving hard examples here. What is not an example about
>> "memory leak" or making debugging output verbose thanks to the attached
>> string properties?
> 
> FWIW, I also don't understand how adding properties could cause a
> memory leak.  When a string is GCed, its properties get GCed as well,
> all of them.  Am I missing something?

If you add string properties to all symbol names this memory will stay
alive for much longer than necessary. It is not a memory leak in the
strongest sense. The memory is still reachable, but there is still no
need to keep the string properties allocated. This is comparable to
memory leaks in other GCed languages where memory is also kept alive for
longer than necessary.

> As to more difficult debugging, I think adding a couple of properties
> that have simple structure will not impair debugging too much.
> Strings with many properties are not uncommon in Emacs, so we already
> have to deal with that.

I disagree with that. We are talking about adding string properties to
every symbol name. This is a global side effect and different from
adding string properties to a set of isolated string in a controlled
manner. I also don't understand why one would even want to take any
chances here given that the feature can be implemented in a way which
avoids this global side effect entirely as my patch shows.

>> As I said, I will ensure that my API does not introduce performance
>> regressions. And since my new API performs strictly less work than your
>> proposal it will necessarily be faster if you consider only the
>> filtering, which is what matters for incrementally updating UIs.
> 
> I would indeed suggest both to make sure there's no performance
> regressions, and would like to see some data similar to what João
> presented, which backs up your assessments about your proposal being
> faster.  Since performance is the main motivation for these changes, I
> think it's important for us to be on the same page wrt facts related
> to performance, before we make the decision how to proceed.

I will prepare some benchmarks.

Daniel




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 16 Aug 2021 12:44:24 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Aug 16 08:44:24 2021
Received: from localhost ([127.0.0.1]:48885 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mFbyV-0008Ku-Pr
	for submit <at> debbugs.gnu.org; Mon, 16 Aug 2021 08:44:24 -0400
Received: from eggs.gnu.org ([209.51.188.92]:51464)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>)
 id 1mFbyL-0008KR-Sd; Mon, 16 Aug 2021 08:44:14 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:46722)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <eliz@HIDDEN>)
 id 1mFbyG-0001uW-Ct; Mon, 16 Aug 2021 08:44:04 -0400
Received: from 84.94.185.95.cable.012.net.il ([84.94.185.95]:1648
 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1mFbyG-0000ve-09; Mon, 16 Aug 2021 08:44:04 -0400
Date: Mon, 16 Aug 2021 15:43:59 +0300
Message-Id: <83im058sq8.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Daniel Mendler <mail@HIDDEN>
In-Reply-To: <d10119c3-aac5-89c4-e6a0-f6655e53a5a2@HIDDEN> (message
 from Daniel Mendler on Mon, 16 Aug 2021 14:05:54 +0200)
Subject: Re: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new
 `completion-filter-completions` API and deferred highlighting
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <CALDnm537=LHAoaGqPpy7=JOPsYn=z+=Q6QW=b9bjrAWOtxqRaQ@HIDDEN>
 <9f59f87c-2489-aaa0-5b3f-0e911b7ffa6c@HIDDEN>
 <CALDnm508EK5qM+AaOOJiNDm-=ORCuOieYpkjO=OSRas+_WTFwQ@HIDDEN>
 <8a36e61a-1c5b-bf3b-a454-077348589c4f@HIDDEN> <87y29471ov.fsf@HIDDEN>
 <837dgob6yo.fsf@HIDDEN> <87wnootec0.fsf@HIDDEN>
 <5d70b0ad-3838-ddb8-d341-9a5531d9da73@HIDDEN>
 <CALDnm51xYehwuGhMubovLTv_nqtdJYvOmH2e-bH3cCjf0PS2WQ@HIDDEN>
 <0cbf224b-b382-8a02-3f06-9f6d7e5e9741@HIDDEN>
 <CALDnm52x_S2UXE=YVokCWEZf9Gqc0O+g4ZVhi+R60SCeq6JtzA@HIDDEN>
 <eecaa682-496e-8173-caf7-2f9781cc7102@HIDDEN> <87a6lihv7b.fsf@HIDDEN>
 <63e03b18-db81-3b11-c692-0af9df20c506@HIDDEN>
 <CALDnm50W_=Y_zP7KLznqcqSx8KJp2-0u2NwpuEGzzAUzCNHvKQ@HIDDEN>
 <c95dea17-ca29-4dee-d53d-b5d46f7f2cf8@HIDDEN>
 <CALDnm53TD=FCf0davQC6q9esVxz687oBGN2LF=Egz3-kz6681Q@HIDDEN>
 <d10119c3-aac5-89c4-e6a0-f6655e53a5a2@HIDDEN>
MIME-version: 1.0
Content-type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 47711
Cc: monnier@HIDDEN, 48841 <at> debbugs.gnu.org, dgutov@HIDDEN,
 joaotavora@HIDDEN, larsi@HIDDEN, 47711 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> Cc: Dmitry Gutov <dgutov@HIDDEN>, Lars Ingebrigtsen <larsi@HIDDEN>,
>  47711 <at> debbugs.gnu.org, 48841 <at> debbugs.gnu.org,
>  Stefan Monnier <monnier@HIDDEN>, Eli Zaretskii <eliz@HIDDEN>
> From: Daniel Mendler <mail@HIDDEN>
> Date: Mon, 16 Aug 2021 14:05:54 +0200
> 
> João, the discussion is clearly not progressing. I propose that we both
> take a step back and let the Emacs maintainers, who participated in this
> discussion, decide on how to proceed. It seems to me that all arguments
> and data has been presented and there is no need for further
> reiterations in more and more colorful language.

As I wrote elsewhere, I'd like to see the performance aspects of this
to be presented from both sides, and agreed upon.  I don't think we
can make the decision before we have performance data we all agree
about.  The other pros and cons are all of qualitative nature, and
involve intuition, personal experience, and personal preferences, so
each one will have their own balance.  But performance is both basic
and qualitative, and we should have the facts and agree on them.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 16 Aug 2021 12:39:56 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Aug 16 08:39:56 2021
Received: from localhost ([127.0.0.1]:48871 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mFbuF-0008DK-On
	for submit <at> debbugs.gnu.org; Mon, 16 Aug 2021 08:39:56 -0400
Received: from eggs.gnu.org ([209.51.188.92]:50686)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>)
 id 1mFbu8-0008Cs-5T; Mon, 16 Aug 2021 08:39:50 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:46626)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <eliz@HIDDEN>)
 id 1mFbu1-00014t-Dy; Mon, 16 Aug 2021 08:39:41 -0400
Received: from 84.94.185.95.cable.012.net.il ([84.94.185.95]:1381
 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1mFbu0-0000Ui-Sm; Mon, 16 Aug 2021 08:39:41 -0400
Date: Mon, 16 Aug 2021 15:39:33 +0300
Message-Id: <83k0kl8sxm.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Daniel Mendler <mail@HIDDEN>
In-Reply-To: <c95dea17-ca29-4dee-d53d-b5d46f7f2cf8@HIDDEN> (message
 from Daniel Mendler on Mon, 16 Aug 2021 12:52:58 +0200)
Subject: Re: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new
 `completion-filter-completions` API and deferred highlighting
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <CALDnm52XM+S07OPXO3qtfLN4pup-kT2U=WRO9dUo3tv9uYJWXw@HIDDEN>
 <38a06f3c-4a7a-755c-c99b-708f91afabfa@HIDDEN>
 <CALDnm537=LHAoaGqPpy7=JOPsYn=z+=Q6QW=b9bjrAWOtxqRaQ@HIDDEN>
 <9f59f87c-2489-aaa0-5b3f-0e911b7ffa6c@HIDDEN>
 <CALDnm508EK5qM+AaOOJiNDm-=ORCuOieYpkjO=OSRas+_WTFwQ@HIDDEN>
 <8a36e61a-1c5b-bf3b-a454-077348589c4f@HIDDEN> <87y29471ov.fsf@HIDDEN>
 <837dgob6yo.fsf@HIDDEN> <87wnootec0.fsf@HIDDEN>
 <5d70b0ad-3838-ddb8-d341-9a5531d9da73@HIDDEN>
 <CALDnm51xYehwuGhMubovLTv_nqtdJYvOmH2e-bH3cCjf0PS2WQ@HIDDEN>
 <0cbf224b-b382-8a02-3f06-9f6d7e5e9741@HIDDEN>
 <CALDnm52x_S2UXE=YVokCWEZf9Gqc0O+g4ZVhi+R60SCeq6JtzA@HIDDEN>
 <eecaa682-496e-8173-caf7-2f9781cc7102@HIDDEN> <87a6lihv7b.fsf@HIDDEN>
 <63e03b18-db81-3b11-c692-0af9df20c506@HIDDEN>
 <CALDnm50W_=Y_zP7KLznqcqSx8KJp2-0u2NwpuEGzzAUzCNHvKQ@HIDDEN> 
 <c95dea17-ca29-4dee-d53d-b5d46f7f2cf8@HIDDEN>
MIME-version: 1.0
Content-type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 47711
Cc: monnier@HIDDEN, 48841 <at> debbugs.gnu.org, dgutov@HIDDEN,
 joaotavora@HIDDEN, larsi@HIDDEN, 47711 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> Cc: Dmitry Gutov <dgutov@HIDDEN>, Lars Ingebrigtsen <larsi@HIDDEN>,
>  47711 <at> debbugs.gnu.org, 48841 <at> debbugs.gnu.org,
>  Stefan Monnier <monnier@HIDDEN>, Eli Zaretskii <eliz@HIDDEN>
> From: Daniel Mendler <mail@HIDDEN>
> Date: Mon, 16 Aug 2021 12:52:58 +0200
> 
> On 8/16/21 12:15 PM, João Távora wrote:
> > On Mon, Aug 16, 2021 at 10:09 AM Daniel Mendler <mail@HIDDEN> wrote:
> > 
> >> There are serious drawbacks of attaching "private" string properties to
> >> arbitrary strings. For once it complicates debugging seriously if there
> >> are suddenly string properties attached to symbol names. It also leads
> >> to a potential memory leak.
> > 
> > Please, in the name of the sanity of this discussion, justify these two
> > statements with examples or follow them with a clause like "because...".
> 
> João, I am giving hard examples here. What is not an example about
> "memory leak" or making debugging output verbose thanks to the attached
> string properties?

FWIW, I also don't understand how adding properties could cause a
memory leak.  When a string is GCed, its properties get GCed as well,
all of them.  Am I missing something?

As to more difficult debugging, I think adding a couple of properties
that have simple structure will not impair debugging too much.
Strings with many properties are not uncommon in Emacs, so we already
have to deal with that.

> As I said, I will ensure that my API does not introduce performance
> regressions. And since my new API performs strictly less work than your
> proposal it will necessarily be faster if you consider only the
> filtering, which is what matters for incrementally updating UIs.

I would indeed suggest both to make sure there's no performance
regressions, and would like to see some data similar to what João
presented, which backs up your assessments about your proposal being
faster.  Since performance is the main motivation for these changes, I
think it's important for us to be on the same page wrt facts related
to performance, before we make the decision how to proceed.

Thanks.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 16 Aug 2021 12:19:29 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Aug 16 08:19:29 2021
Received: from localhost ([127.0.0.1]:48815 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mFbaT-0005RO-D9
	for submit <at> debbugs.gnu.org; Mon, 16 Aug 2021 08:19:29 -0400
Received: from eggs.gnu.org ([209.51.188.92]:44826)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>)
 id 1mFbaS-0005R8-MG; Mon, 16 Aug 2021 08:19:29 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:46066)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <eliz@HIDDEN>)
 id 1mFbaM-0002dP-36; Mon, 16 Aug 2021 08:19:22 -0400
Received: from 84.94.185.95.cable.012.net.il ([84.94.185.95]:4103
 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1mFbaL-00074k-Mw; Mon, 16 Aug 2021 08:19:22 -0400
Date: Mon, 16 Aug 2021 15:19:17 +0300
Message-Id: <83mtph8tve.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: =?utf-8?B?Sm/Do28gVMOhdm9yYQ==?= <joaotavora@HIDDEN>
In-Reply-To: <CALDnm5306P+FrKW12uRcFcLC0K=MCse_asGfAVrnKgVnvZNoSQ@HIDDEN>
 (message from =?utf-8?B?Sm/Do28gVMOhdm9yYQ==?= on Mon, 16 Aug 2021 13:02:04
 +0100)
Subject: Re: bug#48841: bug#47711: [PATCH VERSION 2] Add new
 `completion-filter-completions` API and deferred highlighting
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <f8c4419a-7e83-b8a9-347c-0221a2052376@HIDDEN>
 <fff6e291-bbcc-93e3-92e7-00af4664b8d4@HIDDEN>
 <3d3f894f-a6fa-53ae-5d50-c3aa9bffc73e@HIDDEN>
 <CALDnm53xh6boc6T9XSC4jZpkCPS3RB8JFNb12BpkoeGYg3Q9RQ@HIDDEN>
 <56ab18b1-4348-9b2c-85bb-af9b76cd429a@HIDDEN>
 <CALDnm52XM+S07OPXO3qtfLN4pup-kT2U=WRO9dUo3tv9uYJWXw@HIDDEN>
 <38a06f3c-4a7a-755c-c99b-708f91afabfa@HIDDEN>
 <CALDnm537=LHAoaGqPpy7=JOPsYn=z+=Q6QW=b9bjrAWOtxqRaQ@HIDDEN>
 <9f59f87c-2489-aaa0-5b3f-0e911b7ffa6c@HIDDEN>
 <CALDnm508EK5qM+AaOOJiNDm-=ORCuOieYpkjO=OSRas+_WTFwQ@HIDDEN>
 <8a36e61a-1c5b-bf3b-a454-077348589c4f@HIDDEN> <83h7fsbitl.fsf@HIDDEN>
 <75ce4b68-240f-2f0e-3953-c1f74962c3cd@HIDDEN>
 <83pmud8uwi.fsf@HIDDEN>
 <CALDnm5306P+FrKW12uRcFcLC0K=MCse_asGfAVrnKgVnvZNoSQ@HIDDEN>
MIME-version: 1.0
Content-type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 47711
Cc: mail@HIDDEN, 47711 <at> debbugs.gnu.org, monnier@HIDDEN,
 48841 <at> debbugs.gnu.org, dgutov@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: João Távora <joaotavora@HIDDEN>
> Date: Mon, 16 Aug 2021 13:02:04 +0100
> Cc: Daniel Mendler <mail@HIDDEN>, Dmitry Gutov <dgutov@HIDDEN>, 
> 	Stefan Monnier <monnier@HIDDEN>, 48841 <at> debbugs.gnu.org, 47711 <at> debbugs.gnu.org
> 
> On Mon, Aug 16, 2021 at 12:57 PM Eli Zaretskii <eliz@HIDDEN> wrote:
> 
> > I was talking about the infrastructure that produces the completion
> > candidates, not about the application that uses them.  My point is
> > that your approach requires the applications using the candidates to
> > copy them, whereas previously they could use them without copying.
> 
> If it helps, I think that that is true of all alternatives presented
> so far

Yes, I know.  I was comparing the proposed alternatives to what we
have now, and specifically because Dmitry mentioned this aspect.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 16 Aug 2021 12:18:16 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Aug 16 08:18:16 2021
Received: from localhost ([127.0.0.1]:48808 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mFbZI-0005Ou-BT
	for submit <at> debbugs.gnu.org; Mon, 16 Aug 2021 08:18:16 -0400
Received: from mail-pj1-f48.google.com ([209.85.216.48]:46021)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <joaotavora@HIDDEN>)
 id 1mFbZG-0005Oc-4H; Mon, 16 Aug 2021 08:18:15 -0400
Received: by mail-pj1-f48.google.com with SMTP id
 m24-20020a17090a7f98b0290178b1a81700so27283048pjl.4; 
 Mon, 16 Aug 2021 05:18:14 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=mime-version:references:in-reply-to:from:date:message-id:subject:to
 :cc:content-transfer-encoding;
 bh=vG3PtNb8GFe1kTTcPbNe6NfvS5vc+wIhZ/FWyFFtPIA=;
 b=aSH3HPLhgEj06AokK9atlnXYm/oOd2CjowEF2KvMYq4OxnlLlxnOcYegTgcjSZrUir
 /Gnqd2IF3JkQsi/P7PDf9sWksjDdJqDcCCOlCJwSJz6tfBtT42fLgRGpo8Bie73KkyEY
 3sMa17mBF5tUc4naAeibyvCC4I3vNffSI6++HNQ06W0LODRwswVIMbL/rNCRtMBs4QiP
 baMmFVNDsnxePgBeGwjgxaerbmFFBAMcb9rqwdLxml5NldIlSBq+0XMr7zorhL6OVPnM
 CwwKcY4cmbR5IHmKB0AVp2KVJ3ijYxFih/1F43g9ExDOg90vYdFXBYhMNmAr8mZZHUaz
 ZJBg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:mime-version:references:in-reply-to:from:date
 :message-id:subject:to:cc:content-transfer-encoding;
 bh=vG3PtNb8GFe1kTTcPbNe6NfvS5vc+wIhZ/FWyFFtPIA=;
 b=Qom4B1JVd2ZKFouUGQ25Szzp34t2JSAqbsBlhonSin5uEhbqc1uyLEuaiu1uiyTGQk
 sN4Oa+ujRE6TlkJbeChFQEEcarV2LlcTHVkRQ8oIUgh3eH/Sc+a7l5UnBXcAHd3cbLqN
 hoM25UtCRLfVHM5Lpb8TC7kH9G0DMLh/fCZJf3YLaqUm77Wb+NQ7Ikd88zGsnOZHKnv0
 tlcz2WZgiqa06Pbp4DHpPbXpk+lAdmYi64078Vt4JITHGNohBjLQHWJ+vMRwalfWVM7H
 Cwy8hCHbBsTnf7KKBq7NQAgDWAo6PdLI8rqP4I7BPQn0GvjtrQN+VOUNp7S8p2A77m4G
 ELJQ==
X-Gm-Message-State: AOAM530uHOE8XO7G+R8t3gJtzl2uHZn12DE1K2XH0w6nt4F7tKoRPqi0
 +O00AgqYEN4BLhgsPbqsZUPT5S+rqrlkfgWlrfw=
X-Google-Smtp-Source: ABdhPJywkcSc4K1vtg2RIk7Im7oLva1pYFsW7yZevcSSbs8sdCdSVuP1J3zmP4M1vGwMvMjrDIjDcCGF+zSjKJ/4/lQ=
X-Received: by 2002:a17:90a:3849:: with SMTP id l9mr6083033pjf.7.1629116288241; 
 Mon, 16 Aug 2021 05:18:08 -0700 (PDT)
MIME-Version: 1.0
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <CALDnm537=LHAoaGqPpy7=JOPsYn=z+=Q6QW=b9bjrAWOtxqRaQ@HIDDEN>
 <9f59f87c-2489-aaa0-5b3f-0e911b7ffa6c@HIDDEN>
 <CALDnm508EK5qM+AaOOJiNDm-=ORCuOieYpkjO=OSRas+_WTFwQ@HIDDEN>
 <8a36e61a-1c5b-bf3b-a454-077348589c4f@HIDDEN> <87y29471ov.fsf@HIDDEN>
 <837dgob6yo.fsf@HIDDEN> <87wnootec0.fsf@HIDDEN>
 <5d70b0ad-3838-ddb8-d341-9a5531d9da73@HIDDEN>
 <CALDnm51xYehwuGhMubovLTv_nqtdJYvOmH2e-bH3cCjf0PS2WQ@HIDDEN>
 <0cbf224b-b382-8a02-3f06-9f6d7e5e9741@HIDDEN>
 <CALDnm52x_S2UXE=YVokCWEZf9Gqc0O+g4ZVhi+R60SCeq6JtzA@HIDDEN>
 <eecaa682-496e-8173-caf7-2f9781cc7102@HIDDEN> <87a6lihv7b.fsf@HIDDEN>
 <63e03b18-db81-3b11-c692-0af9df20c506@HIDDEN>
 <CALDnm50W_=Y_zP7KLznqcqSx8KJp2-0u2NwpuEGzzAUzCNHvKQ@HIDDEN>
 <c95dea17-ca29-4dee-d53d-b5d46f7f2cf8@HIDDEN>
 <CALDnm53TD=FCf0davQC6q9esVxz687oBGN2LF=Egz3-kz6681Q@HIDDEN>
 <d10119c3-aac5-89c4-e6a0-f6655e53a5a2@HIDDEN>
In-Reply-To: <d10119c3-aac5-89c4-e6a0-f6655e53a5a2@HIDDEN>
From: =?UTF-8?B?Sm/Do28gVMOhdm9yYQ==?= <joaotavora@HIDDEN>
Date: Mon, 16 Aug 2021 13:17:56 +0100
Message-ID: <CALDnm51UMr2ZRPbuNtKZnMPmWPRcOgNNFs1-PA=29_1s=y9cDw@HIDDEN>
Subject: Re: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new
 `completion-filter-completions` API and deferred highlighting
To: Daniel Mendler <mail@HIDDEN>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 47711
Cc: Stefan Monnier <monnier@HIDDEN>, 48841 <at> debbugs.gnu.org,
 Dmitry Gutov <dgutov@HIDDEN>, Lars Ingebrigtsen <larsi@HIDDEN>,
 Eli Zaretskii <eliz@HIDDEN>, 47711 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

> Jo=C3=A3o, please feel free to also present your closing arguments here.

Sorry, I don't "close" arguments like this.

I hope you can provide:

* the fixes to the regression identified
* the benchmarks you say you have
* the patches to icomplete.el that show how it uses your new API
* the demonstrations of the bugs you accuse my patch to suffer from

Thanks.

On Mon, Aug 16, 2021 at 1:05 PM Daniel Mendler <mail@HIDDEN> wro=
te:
>
> Jo=C3=A3o, the discussion is clearly not progressing. I propose that we b=
oth
> take a step back and let the Emacs maintainers, who participated in this
> discussion, decide on how to proceed. It seems to me that all arguments
> and data has been presented and there is no need for further
> reiterations in more and more colorful language. I would also like to
> point out that implying that I don't understand your language is
> borderline acceptable. I understand the discussion very well, but I
> don't understand why you are using these unfair and invalid means of
> discussion.
>
> For example there could be these decision outcomes:
>
> 1. The information presented up to now does not allow the maintainers to
> make a decision. For example the maintainers may ask for further
> clarification from you, Jo=C3=A3o, or they may ask for benchmarks from me=
 or
> a prove that my patch does not lead to performance regressions.
>
> 2. The maintainers decide that no patch should be merged.
>
> 2. The maintainers decide that your patch will be merged. I will accept
> this decision.
>
> 3. The maintainers decide that my patch will be merged. You will accept
> this decision.
>
> 4. The maintainers decide that both patches will be merged such that
> both approaches will be supported. We both will accept this decision.
>
> I want to summarize the situation in the following:
>
> The patches in question address a performance issue in the current
> completion machinery which is caused by over-eager copying of the
> completion candidate strings and over-eager application of the
> highlighting to all candidate strings. For incrementally updating UIs it
> would be sufficient to only copy and highlight the strings which are
> actually going to be displayed.
>
> My patch takes the approach to expose the existing two-step completion
> process, which consists of filtering and highlighting. By returning the
> filtered completion strings and a highlighting function this two-step
> process is decomposed and the caller of the API has the ability to call
> the highlighting function on only the displayed subset of completion
> candidates. I argue that exposing the filtering and highlighting
> procession steps is the logical and natural conclusion of the existing
> machinery.
>
> My patch is fully backward compatible and aims to not introduce any
> regressions (also no performance regressions) to the existing API.
> Furthermore my patch adheres to the current guarantees given by the
> existing 'completion-all-completions' API. The completion strings
> provided by the completion backend are not mutated in any way, no string
> properties are attached. Since the API 'completion-filter-completions'
> proposed in my patch does the minimal amount of work necessary (only
> filtering), if no highlighting is requested, I argue that the new API is
> the most efficient possible API, given the current constraints.
>
> Furthermore since I am introducing a new API, outstanding issues can be
> solved which could not be solved until now given the constraints of the
> existing 'completion-all-completions' API. In particular the new API
> 'completion-filter-completions' API returns additional data like the end
> position of the completion boundaries. Until now the end position was
> not made available and 'completion-base-position' just used the length
> of the input to guess the end position. In a strict sense this guess is
> incorrect and there is a FIXME in minibuffer.el, mentioning this issue.
>
> The downside of my patch is that it is a large patch. While it adds only
> 196 lines of code, which is not much and expected given that it only
> reshuffles the existing machinery, it is still a large patch in total.
>
> On the other side, Jo=C3=A3o's patch avoids the complication of adhering =
to
> the existing guarantees of the APIs and takes the liberty of attaching
> "private" string properties to the completion strings of the completion
> table backend. I argue that attaching the string properties is a
> violation of the guarantees of the existing API and violates the
> expectations of the existing many completion tables. One very severe
> scenario is when obarray is used as completion table, since then each
> symbol name gets a private property attached. I argue that such global
> side effects like attaching string properties to all completion
> candidates should better be avoided. There is the issue that the
> attached string properties are a potential memory leak. When dumping the
> string representation of symbol names, the symbols will have additional
> properties which will complicate the debugging experience. Furthermore
> it will lead to confusion since the global side effect during completion
> will suddenly have an influence of symbols which don't have to do
> anything with completion. The big advantage of Jo=C3=A3o's patch is that =
it
> is very limited in scope and very simple. However I argue that this
> simplicity is hard-won and we will regret this approach later due to the
> global side effects.
>
> Therefore I conclude that the two-step process proposed in my patch,
> which does not introduce problematic global side effects is the better
> approach forward. Furthermore a new API is needed such that more
> completion data can be returned, e.g., the completion end position. One
> could even return additional match data in the future given that the new
> API 'completion-filter-completions' is extensible thanks to its alist
> return value.
>
> Jo=C3=A3o, please feel free to also present your closing arguments here.
>
> Daniel



--=20
Jo=C3=A3o T=C3=A1vora




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 16 Aug 2021 12:08:42 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Aug 16 08:08:42 2021
Received: from localhost ([127.0.0.1]:48737 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mFbQ2-0002sA-42
	for submit <at> debbugs.gnu.org; Mon, 16 Aug 2021 08:08:42 -0400
Received: from server.qxqx.de ([178.63.65.180]:45343 helo=mail.qxqx.de)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mail@HIDDEN>)
 id 1mFbQ0-0002rn-Ts; Mon, 16 Aug 2021 08:08:41 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=qxqx.de;
 s=mail1392553390; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:
 MIME-Version:Date:Message-ID:From:References:Cc:To:Subject:Sender:Reply-To:
 Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender:
 Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:
 List-Subscribe:List-Post:List-Owner:List-Archive;
 bh=R55hjCfJUe6SX5kZqUMWgP5U5R+Jn+Et7+fCwvgRegg=; b=xUpr01qQu0n2wvaRzOXBFioD6x
 WZOQKMXScwZKkhGaryKipNAaoHQY3Z34jY274hOKXT/Dg2nHk+3ngR56N0OHjYcFNE2TObaa+jnWo
 Fb5gH57N8WwLJjlZjMKWTuy9DJhSIxsAdn8YPaNZv0ICxhLSbprk+ZDPMKloA7Z7te/w=;
Subject: Re: bug#48841: bug#47711: [PATCH VERSION 2] Add new
 `completion-filter-completions` API and deferred highlighting
To: Eli Zaretskii <eliz@HIDDEN>
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <f8c4419a-7e83-b8a9-347c-0221a2052376@HIDDEN>
 <fff6e291-bbcc-93e3-92e7-00af4664b8d4@HIDDEN>
 <3d3f894f-a6fa-53ae-5d50-c3aa9bffc73e@HIDDEN>
 <CALDnm53xh6boc6T9XSC4jZpkCPS3RB8JFNb12BpkoeGYg3Q9RQ@HIDDEN>
 <56ab18b1-4348-9b2c-85bb-af9b76cd429a@HIDDEN>
 <CALDnm52XM+S07OPXO3qtfLN4pup-kT2U=WRO9dUo3tv9uYJWXw@HIDDEN>
 <38a06f3c-4a7a-755c-c99b-708f91afabfa@HIDDEN>
 <CALDnm537=LHAoaGqPpy7=JOPsYn=z+=Q6QW=b9bjrAWOtxqRaQ@HIDDEN>
 <9f59f87c-2489-aaa0-5b3f-0e911b7ffa6c@HIDDEN>
 <CALDnm508EK5qM+AaOOJiNDm-=ORCuOieYpkjO=OSRas+_WTFwQ@HIDDEN>
 <8a36e61a-1c5b-bf3b-a454-077348589c4f@HIDDEN> <83h7fsbitl.fsf@HIDDEN>
 <75ce4b68-240f-2f0e-3953-c1f74962c3cd@HIDDEN>
 <83pmud8uwi.fsf@HIDDEN>
From: Daniel Mendler <mail@HIDDEN>
Message-ID: <003d6015-8594-7623-16d9-167991a1ac16@HIDDEN>
Date: Mon, 16 Aug 2021 14:08:33 +0200
MIME-Version: 1.0
In-Reply-To: <83pmud8uwi.fsf@HIDDEN>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 7bit
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 47711
Cc: 48841 <at> debbugs.gnu.org, 47711 <at> debbugs.gnu.org, joaotavora@HIDDEN,
 monnier@HIDDEN, dgutov@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

On 8/16/21 1:57 PM, Eli Zaretskii wrote:
> I was talking about the infrastructure that produces the completion
> candidates, not about the application that uses them.  My point is
> that your approach requires the applications using the candidates to
> copy them, whereas previously they could use them without copying.

Okay, I understand. Yes, in my patch the strings returned by
'completion-filter-completions' must not be mutated by the API consumer
directly. This should be documented clearly, but it is not unexpected.
For example the API 'all-completions' which one can use to obtain the
strings from a completion table also requires the caller to not mutate
the returned strings.

Daniel




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 16 Aug 2021 12:06:06 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Aug 16 08:06:06 2021
Received: from localhost ([127.0.0.1]:48722 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mFbNV-0002mS-Tj
	for submit <at> debbugs.gnu.org; Mon, 16 Aug 2021 08:06:06 -0400
Received: from server.qxqx.de ([178.63.65.180]:59331 helo=mail.qxqx.de)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mail@HIDDEN>)
 id 1mFbNT-0002lo-8G; Mon, 16 Aug 2021 08:06:04 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=qxqx.de;
 s=mail1392553390; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:
 MIME-Version:Date:Message-ID:From:References:Cc:To:Subject:Sender:Reply-To:
 Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender:
 Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:
 List-Subscribe:List-Post:List-Owner:List-Archive;
 bh=Rqm3CwKvaY3irAWRkNIM3r7NyHLM9cGe9M3IfOA8XxM=; b=ktVOSLa8U2WRAXCOadinumlvJp
 FRn5yd3JtduX91ZpaoVdXSgrjRtx+cemkLkO/k7UphFRNq9cBPahXSsImb8uapcbODMEXeIhBRnGH
 oh+2hJ/SFUU7usXD9ubtxJrfKs4zQGlYeBC5I+w3mut2ZG26VRhpJIwkn95aORXMAX2s=;
Subject: Re: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new
 `completion-filter-completions` API and deferred highlighting
To: =?UTF-8?B?Sm/Do28gVMOhdm9yYQ==?= <joaotavora@HIDDEN>
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <CALDnm537=LHAoaGqPpy7=JOPsYn=z+=Q6QW=b9bjrAWOtxqRaQ@HIDDEN>
 <9f59f87c-2489-aaa0-5b3f-0e911b7ffa6c@HIDDEN>
 <CALDnm508EK5qM+AaOOJiNDm-=ORCuOieYpkjO=OSRas+_WTFwQ@HIDDEN>
 <8a36e61a-1c5b-bf3b-a454-077348589c4f@HIDDEN> <87y29471ov.fsf@HIDDEN>
 <837dgob6yo.fsf@HIDDEN> <87wnootec0.fsf@HIDDEN>
 <5d70b0ad-3838-ddb8-d341-9a5531d9da73@HIDDEN>
 <CALDnm51xYehwuGhMubovLTv_nqtdJYvOmH2e-bH3cCjf0PS2WQ@HIDDEN>
 <0cbf224b-b382-8a02-3f06-9f6d7e5e9741@HIDDEN>
 <CALDnm52x_S2UXE=YVokCWEZf9Gqc0O+g4ZVhi+R60SCeq6JtzA@HIDDEN>
 <eecaa682-496e-8173-caf7-2f9781cc7102@HIDDEN> <87a6lihv7b.fsf@HIDDEN>
 <63e03b18-db81-3b11-c692-0af9df20c506@HIDDEN>
 <CALDnm50W_=Y_zP7KLznqcqSx8KJp2-0u2NwpuEGzzAUzCNHvKQ@HIDDEN>
 <c95dea17-ca29-4dee-d53d-b5d46f7f2cf8@HIDDEN>
 <CALDnm53TD=FCf0davQC6q9esVxz687oBGN2LF=Egz3-kz6681Q@HIDDEN>
From: Daniel Mendler <mail@HIDDEN>
Message-ID: <d10119c3-aac5-89c4-e6a0-f6655e53a5a2@HIDDEN>
Date: Mon, 16 Aug 2021 14:05:54 +0200
MIME-Version: 1.0
In-Reply-To: <CALDnm53TD=FCf0davQC6q9esVxz687oBGN2LF=Egz3-kz6681Q@HIDDEN>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 8bit
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 47711
Cc: Stefan Monnier <monnier@HIDDEN>, 48841 <at> debbugs.gnu.org,
 Dmitry Gutov <dgutov@HIDDEN>, Lars Ingebrigtsen <larsi@HIDDEN>,
 Eli Zaretskii <eliz@HIDDEN>, 47711 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

João, the discussion is clearly not progressing. I propose that we both
take a step back and let the Emacs maintainers, who participated in this
discussion, decide on how to proceed. It seems to me that all arguments
and data has been presented and there is no need for further
reiterations in more and more colorful language. I would also like to
point out that implying that I don't understand your language is
borderline acceptable. I understand the discussion very well, but I
don't understand why you are using these unfair and invalid means of
discussion.

For example there could be these decision outcomes:

1. The information presented up to now does not allow the maintainers to
make a decision. For example the maintainers may ask for further
clarification from you, João, or they may ask for benchmarks from me or
a prove that my patch does not lead to performance regressions.

2. The maintainers decide that no patch should be merged.

2. The maintainers decide that your patch will be merged. I will accept
this decision.

3. The maintainers decide that my patch will be merged. You will accept
this decision.

4. The maintainers decide that both patches will be merged such that
both approaches will be supported. We both will accept this decision.

I want to summarize the situation in the following:

The patches in question address a performance issue in the current
completion machinery which is caused by over-eager copying of the
completion candidate strings and over-eager application of the
highlighting to all candidate strings. For incrementally updating UIs it
would be sufficient to only copy and highlight the strings which are
actually going to be displayed.

My patch takes the approach to expose the existing two-step completion
process, which consists of filtering and highlighting. By returning the
filtered completion strings and a highlighting function this two-step
process is decomposed and the caller of the API has the ability to call
the highlighting function on only the displayed subset of completion
candidates. I argue that exposing the filtering and highlighting
procession steps is the logical and natural conclusion of the existing
machinery.

My patch is fully backward compatible and aims to not introduce any
regressions (also no performance regressions) to the existing API.
Furthermore my patch adheres to the current guarantees given by the
existing 'completion-all-completions' API. The completion strings
provided by the completion backend are not mutated in any way, no string
properties are attached. Since the API 'completion-filter-completions'
proposed in my patch does the minimal amount of work necessary (only
filtering), if no highlighting is requested, I argue that the new API is
the most efficient possible API, given the current constraints.

Furthermore since I am introducing a new API, outstanding issues can be
solved which could not be solved until now given the constraints of the
existing 'completion-all-completions' API. In particular the new API
'completion-filter-completions' API returns additional data like the end
position of the completion boundaries. Until now the end position was
not made available and 'completion-base-position' just used the length
of the input to guess the end position. In a strict sense this guess is
incorrect and there is a FIXME in minibuffer.el, mentioning this issue.

The downside of my patch is that it is a large patch. While it adds only
196 lines of code, which is not much and expected given that it only
reshuffles the existing machinery, it is still a large patch in total.

On the other side, João's patch avoids the complication of adhering to
the existing guarantees of the APIs and takes the liberty of attaching
"private" string properties to the completion strings of the completion
table backend. I argue that attaching the string properties is a
violation of the guarantees of the existing API and violates the
expectations of the existing many completion tables. One very severe
scenario is when obarray is used as completion table, since then each
symbol name gets a private property attached. I argue that such global
side effects like attaching string properties to all completion
candidates should better be avoided. There is the issue that the
attached string properties are a potential memory leak. When dumping the
string representation of symbol names, the symbols will have additional
properties which will complicate the debugging experience. Furthermore
it will lead to confusion since the global side effect during completion
will suddenly have an influence of symbols which don't have to do
anything with completion. The big advantage of João's patch is that it
is very limited in scope and very simple. However I argue that this
simplicity is hard-won and we will regret this approach later due to the
global side effects.

Therefore I conclude that the two-step process proposed in my patch,
which does not introduce problematic global side effects is the better
approach forward. Furthermore a new API is needed such that more
completion data can be returned, e.g., the completion end position. One
could even return additional match data in the future given that the new
API 'completion-filter-completions' is extensible thanks to its alist
return value.

João, please feel free to also present your closing arguments here.

Daniel




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 16 Aug 2021 12:02:26 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Aug 16 08:02:26 2021
Received: from localhost ([127.0.0.1]:48715 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mFbJx-0002ft-VG
	for submit <at> debbugs.gnu.org; Mon, 16 Aug 2021 08:02:26 -0400
Received: from mail-pj1-f47.google.com ([209.85.216.47]:37852)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <joaotavora@HIDDEN>)
 id 1mFbJv-0002fV-8N; Mon, 16 Aug 2021 08:02:24 -0400
Received: by mail-pj1-f47.google.com with SMTP id
 cp15-20020a17090afb8fb029017891959dcbso31975574pjb.2; 
 Mon, 16 Aug 2021 05:02:23 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=mime-version:references:in-reply-to:from:date:message-id:subject:to
 :cc:content-transfer-encoding;
 bh=FgipmI8mrux/nK0E7ZMSHjfGrIdRCejKfhBwYRpSzms=;
 b=NL/i3b2zUrULUZHux5TA+MqbiNNWyPsLvWNIz288/2bHzqBhzF1PX1DAjP52KXueOq
 OqKJGwHo7W5uEcvkhTGqGNggg95bAF/5VDp8ElPqC608A6to5DwQfu9JwI1JGdhwOj3B
 iNi9JNWeoByf76QGKUOaunwbnyA7PdY7M/GjkwzJiX5tUHo/QmT/B2jYO9bu4EIicK+O
 9ddkvdZeU/29pwC0TDqpzqjuUQHe9qaHiSjMkEklDUjaxBOWdZrUTJy23ifbe9l03tb8
 KHBX/wPl9wHopXhQyM78V3pItqJ4ZCyU3xIQHlGWzHdAq+p9jNMFQYcoSHxXiN9ndvDk
 9BvQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:mime-version:references:in-reply-to:from:date
 :message-id:subject:to:cc:content-transfer-encoding;
 bh=FgipmI8mrux/nK0E7ZMSHjfGrIdRCejKfhBwYRpSzms=;
 b=qh+YkK4baWh29gu+mmNPQZmWuy3GonuzFBhVvqEwKaGAWzlIzMChcx9/rcXchHMdVo
 eTmNfBulLU9D96a99RZGo74CgPRNR2o/8RTGxhZ2BbUBcs+VzpAo2z0Neb7FSfRu2WVq
 He5+Z4A07QTKtJLGbAHrB94tIc1H3ok3MXMIXMUIvFK+3eVDjn50UkHmGBKBdxEtAbyD
 lGmo31dGjk+uq3Ri8uUFZUsgEQSfwx8Ch4uKwRsOiaTowR5sibxC54DmvXDZf5vVw9rj
 QfUN/+3XevQwY8zbzIj5XepdIJYBKvAwqbf9A9Ckt93vllfKTKfWUGVYnmYkEn7Z+V4t
 AZww==
X-Gm-Message-State: AOAM533OLx07BHDXGgeVlN1jsHlcr2KEp7DNf+TGazkhL47NP2gq1TaI
 Rh+3vWM7BVWJ99gQ0xn6YIoEgwtq9EnHBkr2G2Y=
X-Google-Smtp-Source: ABdhPJzjM5Qis+eLF+MA/lhmfwPXQa7dKbrodyXqHMG9z4pIxWVkYHQWf/UHH6u+FZiAOiv4JLPdPwnMZ1rVYV2C/uc=
X-Received: by 2002:a65:6653:: with SMTP id z19mr15605178pgv.394.1629115337363; 
 Mon, 16 Aug 2021 05:02:17 -0700 (PDT)
MIME-Version: 1.0
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <f8c4419a-7e83-b8a9-347c-0221a2052376@HIDDEN>
 <fff6e291-bbcc-93e3-92e7-00af4664b8d4@HIDDEN>
 <3d3f894f-a6fa-53ae-5d50-c3aa9bffc73e@HIDDEN>
 <CALDnm53xh6boc6T9XSC4jZpkCPS3RB8JFNb12BpkoeGYg3Q9RQ@HIDDEN>
 <56ab18b1-4348-9b2c-85bb-af9b76cd429a@HIDDEN>
 <CALDnm52XM+S07OPXO3qtfLN4pup-kT2U=WRO9dUo3tv9uYJWXw@HIDDEN>
 <38a06f3c-4a7a-755c-c99b-708f91afabfa@HIDDEN>
 <CALDnm537=LHAoaGqPpy7=JOPsYn=z+=Q6QW=b9bjrAWOtxqRaQ@HIDDEN>
 <9f59f87c-2489-aaa0-5b3f-0e911b7ffa6c@HIDDEN>
 <CALDnm508EK5qM+AaOOJiNDm-=ORCuOieYpkjO=OSRas+_WTFwQ@HIDDEN>
 <8a36e61a-1c5b-bf3b-a454-077348589c4f@HIDDEN> <83h7fsbitl.fsf@HIDDEN>
 <75ce4b68-240f-2f0e-3953-c1f74962c3cd@HIDDEN>
 <83pmud8uwi.fsf@HIDDEN>
In-Reply-To: <83pmud8uwi.fsf@HIDDEN>
From: =?UTF-8?B?Sm/Do28gVMOhdm9yYQ==?= <joaotavora@HIDDEN>
Date: Mon, 16 Aug 2021 13:02:04 +0100
Message-ID: <CALDnm5306P+FrKW12uRcFcLC0K=MCse_asGfAVrnKgVnvZNoSQ@HIDDEN>
Subject: Re: bug#48841: bug#47711: [PATCH VERSION 2] Add new
 `completion-filter-completions` API and deferred highlighting
To: Eli Zaretskii <eliz@HIDDEN>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 47711
Cc: Daniel Mendler <mail@HIDDEN>, 47711 <at> debbugs.gnu.org,
 Stefan Monnier <monnier@HIDDEN>, 48841 <at> debbugs.gnu.org,
 Dmitry Gutov <dgutov@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

On Mon, Aug 16, 2021 at 12:57 PM Eli Zaretskii <eliz@HIDDEN> wrote:

> I was talking about the infrastructure that produces the completion
> candidates, not about the application that uses them.  My point is
> that your approach requires the applications using the candidates to
> copy them, whereas previously they could use them without copying.

If it helps, I think that that is true of all alternatives presented
so far (though I haven't read the big patch fully yet). The difference is
that the consumers who copy the candidate strings will only copy a much
smaller number, typically only the ones that need to be displayed.
Whereas currently, all candidate strings are copied, displayed or not.

Jo=C3=A3o T=C3=A1vora




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 16 Aug 2021 11:57:16 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Aug 16 07:57:16 2021
Received: from localhost ([127.0.0.1]:48686 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mFbEy-0002T3-DN
	for submit <at> debbugs.gnu.org; Mon, 16 Aug 2021 07:57:16 -0400
Received: from eggs.gnu.org ([209.51.188.92]:38180)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>)
 id 1mFbEw-0002Se-Kt; Mon, 16 Aug 2021 07:57:15 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:44788)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <eliz@HIDDEN>)
 id 1mFbEq-0007if-P5; Mon, 16 Aug 2021 07:57:08 -0400
Received: from 84.94.185.95.cable.012.net.il ([84.94.185.95]:2716
 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1mFbEq-0004Nk-CS; Mon, 16 Aug 2021 07:57:08 -0400
Date: Mon, 16 Aug 2021 14:57:01 +0300
Message-Id: <83pmud8uwi.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Daniel Mendler <mail@HIDDEN>
In-Reply-To: <75ce4b68-240f-2f0e-3953-c1f74962c3cd@HIDDEN> (message
 from Daniel Mendler on Mon, 16 Aug 2021 10:48:19 +0200)
Subject: Re: bug#48841: bug#47711: [PATCH VERSION 2] Add new
 `completion-filter-completions` API and deferred highlighting
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <f8c4419a-7e83-b8a9-347c-0221a2052376@HIDDEN>
 <fff6e291-bbcc-93e3-92e7-00af4664b8d4@HIDDEN>
 <3d3f894f-a6fa-53ae-5d50-c3aa9bffc73e@HIDDEN>
 <CALDnm53xh6boc6T9XSC4jZpkCPS3RB8JFNb12BpkoeGYg3Q9RQ@HIDDEN>
 <56ab18b1-4348-9b2c-85bb-af9b76cd429a@HIDDEN>
 <CALDnm52XM+S07OPXO3qtfLN4pup-kT2U=WRO9dUo3tv9uYJWXw@HIDDEN>
 <38a06f3c-4a7a-755c-c99b-708f91afabfa@HIDDEN>
 <CALDnm537=LHAoaGqPpy7=JOPsYn=z+=Q6QW=b9bjrAWOtxqRaQ@HIDDEN>
 <9f59f87c-2489-aaa0-5b3f-0e911b7ffa6c@HIDDEN>
 <CALDnm508EK5qM+AaOOJiNDm-=ORCuOieYpkjO=OSRas+_WTFwQ@HIDDEN>
 <8a36e61a-1c5b-bf3b-a454-077348589c4f@HIDDEN> <83h7fsbitl.fsf@HIDDEN>
 <75ce4b68-240f-2f0e-3953-c1f74962c3cd@HIDDEN>
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 47711
Cc: 48841 <at> debbugs.gnu.org, 47711 <at> debbugs.gnu.org, joaotavora@HIDDEN,
 monnier@HIDDEN, dgutov@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> Cc: joaotavora@HIDDEN, monnier@HIDDEN, 48841 <at> debbugs.gnu.org,
>  47711 <at> debbugs.gnu.org
> From: Daniel Mendler <mail@HIDDEN>
> Date: Mon, 16 Aug 2021 10:48:19 +0200
> 
> On 8/14/21 9:12 AM, Eli Zaretskii wrote:
> >> Since up until now completion-pcm--hilit-commonality copied all strings 
> >> before modifying, completion tables such as described (with "shared" 
> >> strings) have all been "legal". Suddenly deciding to stop supporting 
> >> them would be a major API breakage with consequences that are hard to 
> >> predict. And while I perhaps agree that it's an inconvenience, I don't 
> >> think it's a choice we can simply make as this stage in c-a-p-f's 
> >> development.
> > 
> > This sounds like an argument against Daniel's approach as well, no?
> > Because if a completion API returns strings it "doesn't own", there
> > will be restrictions on Lisp programs that use those strings, because
> > those Lisp programs previously could do anything they liked with those
> > strings, and now they cannot.  Or am I missing something?
> 
> No, in my patch the displayed candidate strings are still copied before
> the text properties are attached. The strings are kept intact as they
> are now.

I was talking about the infrastructure that produces the completion
candidates, not about the application that uses them.  My point is
that your approach requires the applications using the candidates to
copy them, whereas previously they could use them without copying.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 16 Aug 2021 11:48:53 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Aug 16 07:48:53 2021
Received: from localhost ([127.0.0.1]:48677 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mFb6r-0002CF-Aw
	for submit <at> debbugs.gnu.org; Mon, 16 Aug 2021 07:48:53 -0400
Received: from eggs.gnu.org ([209.51.188.92]:36928)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>)
 id 1mFb6p-0002Bt-IX; Mon, 16 Aug 2021 07:48:51 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:44662)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <eliz@HIDDEN>)
 id 1mFb6j-00032d-4E; Mon, 16 Aug 2021 07:48:45 -0400
Received: from 84.94.185.95.cable.012.net.il ([84.94.185.95]:2200
 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1mFb6i-0003Lq-Hr; Mon, 16 Aug 2021 07:48:44 -0400
Date: Mon, 16 Aug 2021 14:48:39 +0300
Message-Id: <83sfz98vag.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Dmitry Gutov <dgutov@HIDDEN>
In-Reply-To: <42a56dff-197d-d223-a2d8-12db8577b505@HIDDEN> (message from
 Dmitry Gutov on Mon, 16 Aug 2021 06:26:58 +0300)
Subject: Re: bug#47711: [PATCH VERSION 2] Add new
 `completion-filter-completions` API and deferred highlighting
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <f8c4419a-7e83-b8a9-347c-0221a2052376@HIDDEN>
 <fff6e291-bbcc-93e3-92e7-00af4664b8d4@HIDDEN>
 <3d3f894f-a6fa-53ae-5d50-c3aa9bffc73e@HIDDEN>
 <CALDnm53xh6boc6T9XSC4jZpkCPS3RB8JFNb12BpkoeGYg3Q9RQ@HIDDEN>
 <56ab18b1-4348-9b2c-85bb-af9b76cd429a@HIDDEN>
 <CALDnm52XM+S07OPXO3qtfLN4pup-kT2U=WRO9dUo3tv9uYJWXw@HIDDEN>
 <38a06f3c-4a7a-755c-c99b-708f91afabfa@HIDDEN>
 <CALDnm537=LHAoaGqPpy7=JOPsYn=z+=Q6QW=b9bjrAWOtxqRaQ@HIDDEN>
 <9f59f87c-2489-aaa0-5b3f-0e911b7ffa6c@HIDDEN>
 <83im08bjc3.fsf@HIDDEN> <42a56dff-197d-d223-a2d8-12db8577b505@HIDDEN>
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 47711
Cc: mail@HIDDEN, joaotavora@HIDDEN, monnier@HIDDEN,
 48841 <at> debbugs.gnu.org, 47711 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> Cc: 47711 <at> debbugs.gnu.org, monnier@HIDDEN, joaotavora@HIDDEN,
>  48841 <at> debbugs.gnu.org
> From: Dmitry Gutov <dgutov@HIDDEN>
> Date: Mon, 16 Aug 2021 06:26:58 +0300
> 
> On 14.08.2021 10:01, Eli Zaretskii wrote:
> 
> > Just to make sure we are on the same page: adding a text property to a
> > string doesn't mutate a string.  Lisp programs that process these
> > strings will not necessarily see any difference, and displaying those
> > strings will also not show any difference if the property is not
> > related to display.  So the assumption that seems to be made here,
> > that adding a property is the same as mutating a string, is IMO
> > inaccurate if not incorrect.
> 
> This is nonsense.
> 
> A program won't necessarily see a difference in *any* changed value, as 
> long as some part of it stays the same.
> 
> I can zero out the tail of a string, and have a program that only looks 
> at its first few characters. It wouldn't mean that a string hasn't changed.

You are not making any sense.

Anyway, if what I wrote doesn't help, feel free to disregard it.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 16 Aug 2021 11:47:12 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Aug 16 07:47:12 2021
Received: from localhost ([127.0.0.1]:48668 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mFb5E-000298-7w
	for submit <at> debbugs.gnu.org; Mon, 16 Aug 2021 07:47:12 -0400
Received: from eggs.gnu.org ([209.51.188.92]:36658)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>)
 id 1mFb5D-00028o-G7; Mon, 16 Aug 2021 07:47:11 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:44640)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <eliz@HIDDEN>)
 id 1mFb57-00028Z-QC; Mon, 16 Aug 2021 07:47:05 -0400
Received: from 84.94.185.95.cable.012.net.il ([84.94.185.95]:2095
 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1mFb57-0003EM-Ac; Mon, 16 Aug 2021 07:47:05 -0400
Date: Mon, 16 Aug 2021 14:46:58 +0300
Message-Id: <83tujp8vd9.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Dmitry Gutov <dgutov@HIDDEN>
In-Reply-To: <31da8f43-02f7-e6dd-ab38-2160af138a4a@HIDDEN> (message from
 Dmitry Gutov on Mon, 16 Aug 2021 06:17:32 +0300)
Subject: Re: bug#48841: bug#47711: [PATCH VERSION 2] Add new
 `completion-filter-completions` API and deferred highlighting
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <f8c4419a-7e83-b8a9-347c-0221a2052376@HIDDEN>
 <fff6e291-bbcc-93e3-92e7-00af4664b8d4@HIDDEN>
 <3d3f894f-a6fa-53ae-5d50-c3aa9bffc73e@HIDDEN>
 <CALDnm53xh6boc6T9XSC4jZpkCPS3RB8JFNb12BpkoeGYg3Q9RQ@HIDDEN>
 <56ab18b1-4348-9b2c-85bb-af9b76cd429a@HIDDEN>
 <CALDnm52XM+S07OPXO3qtfLN4pup-kT2U=WRO9dUo3tv9uYJWXw@HIDDEN>
 <38a06f3c-4a7a-755c-c99b-708f91afabfa@HIDDEN>
 <CALDnm537=LHAoaGqPpy7=JOPsYn=z+=Q6QW=b9bjrAWOtxqRaQ@HIDDEN>
 <9f59f87c-2489-aaa0-5b3f-0e911b7ffa6c@HIDDEN>
 <CALDnm508EK5qM+AaOOJiNDm-=ORCuOieYpkjO=OSRas+_WTFwQ@HIDDEN>
 <8a36e61a-1c5b-bf3b-a454-077348589c4f@HIDDEN> <87y29471ov.fsf@HIDDEN>
 <837dgob6yo.fsf@HIDDEN> <31da8f43-02f7-e6dd-ab38-2160af138a4a@HIDDEN>
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 47711
Cc: mail@HIDDEN, 47711 <at> debbugs.gnu.org, joaotavora@HIDDEN,
 48841 <at> debbugs.gnu.org, monnier@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> Cc: mail@HIDDEN, monnier@HIDDEN, 48841 <at> debbugs.gnu.org,
>  47711 <at> debbugs.gnu.org
> From: Dmitry Gutov <dgutov@HIDDEN>
> Date: Mon, 16 Aug 2021 06:17:32 +0300
> 
> On 14.08.2021 14:29, Eli Zaretskii wrote:
> > Text properties are stored separately from the string, so I don't
> > think adding properties can in general be referred to as "change".
> 
> Are you thinking of C strings?

No, about the implementation of Lisp strings in Emacs.

> Lisp strings carry text properties in addition to the array of 
> characters. It doesn't really matter where in the memory the properties 
> and the characters reside.

Well, it does, at least in some situations.  The string text is not
affected, and so the code which processes the string will not notice
that it has a property about which that code has no idea.  Only
properties that are known to the processing code can affect it;
non-standard properties private to some other code will generally pass
unnoticed.

> > Whether in some particular situation that could count as a "change"
> > depends on that situation and on the particular property, of course.
> 
> I was talking in the general sense: modifying a value.
> 
> One can talk about whether a certain modification matters in certain 
> situations, but that's not the way to discount a general principle.

I didn't want to start a general philosophical discussion about string
mutability.  I hoped to provide input of specific practical use in the
context of this discussion.  If what I said is not useful, just
disregard it.

> > I'm not sure in the context of completion there's any reason to count
> > as "change" adding properties that don't affect display.
> 
> For the context in question, whether the properties affect display is 
> not particularly important. Properties affecting display just make it 
> easier to notice that something's wrong. Bug involving other properties 
> should be more difficult to investigate.

Once again, if some code invents its private property, not used
anywhere else and not documented anywhere else, then putting such a
property on a string has very high chances of going unnoticed.  I hope
this consideration helps this discussion, because saying that
properties change a string blurs the distinction between actually
changing the string text or its properties that affect many parts in
Emacs, and adding some obscure property that is not known to anyone.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 16 Aug 2021 11:37:37 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Aug 16 07:37:37 2021
Received: from localhost ([127.0.0.1]:48657 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mFavx-0001rh-HS
	for submit <at> debbugs.gnu.org; Mon, 16 Aug 2021 07:37:37 -0400
Received: from mail-pl1-f170.google.com ([209.85.214.170]:38758)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <joaotavora@HIDDEN>)
 id 1mFavv-0001rO-AX; Mon, 16 Aug 2021 07:37:36 -0400
Received: by mail-pl1-f170.google.com with SMTP id a5so20376640plh.5;
 Mon, 16 Aug 2021 04:37:35 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=mime-version:references:in-reply-to:from:date:message-id:subject:to
 :cc:content-transfer-encoding;
 bh=PP4sRFsXGoPRO3HMfAeZVAdQ65nsBuQPmVZTfft17hA=;
 b=GoSaAsmAvDaJ2uEnFUQL2xDzpkjCEOWINaIWLQb4j4gnIQT1XIeWP15QWTop+vtWrL
 dIqyXjGuNSjWTDDIRcm+rc+qvpNOFBTaUa50EZ0knZaSAyqLTEHe+0D7YeafD/auQ5X2
 RGKP9WaXJNwr3rFIgUo2eiYBkX/TPV9UyskbbMV9y9jkEXr/90eVe3j7sM1zqpV5/F2R
 sEAeJib5PBjWZ4elaJdCAyv7vRPZSPdvAn8JxOtmS8PjwAVgtnwE0mWXt99s7LZh3Xde
 JklCLkX8moxJ/d9NNSghPMprtnY3Ek7bHEBYuda8rUPe2EJWhiwj3b13giMQrvGSMq4H
 rfzA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:mime-version:references:in-reply-to:from:date
 :message-id:subject:to:cc:content-transfer-encoding;
 bh=PP4sRFsXGoPRO3HMfAeZVAdQ65nsBuQPmVZTfft17hA=;
 b=DQ/vdrTLsgX8WvR6REAhrMlk+Zc2S53hChzhYTrRE3NU3Q1OhK0INHm3avRoSSHrRK
 cKtUbaATb0UvCf04ttBCdspBczfm0tDC/b1uyCRmPNTYPHMxdZxCJf8I1vofSsXkFCMo
 KiBRZEryMrydzoDVzmqw7oBE5b4VQSc9TvPkd97nMFuQKzl6YRhBsRfb/e0mq5r0ymhT
 bJkkWSk+LZSfV2oD4B0d8wtCofyxXgSaYgcosPV6GhARXRZtUb2HkmT/g93HSvFd2sem
 uoFdDWUQuUEfDuwKwMxkej3afL7yeUYRDSWNR+CdwkPRcRoNOyaL9UZAGW3IbnC5lqfZ
 i16A==
X-Gm-Message-State: AOAM533DyepoAsd/XxbhrdehaJxKWlhYtH1ZnjfHNXiRhmemoTb1hQe0
 b/1O+8nTXOLEsSRlf97XeuSfiCOOJIKpj8KzcCg=
X-Google-Smtp-Source: ABdhPJw7BaRbhM71Xp9pUzVXJR+YVQTSu9Q0LemJBUf+41xxZ6Cjz3cWVrjUel5I3GCBaZtwjQzU2ns366mJJydGD10=
X-Received: by 2002:aa7:9050:0:b029:3af:7e99:f48f with SMTP id
 n16-20020aa790500000b02903af7e99f48fmr15986526pfo.2.1629113849377; Mon, 16
 Aug 2021 04:37:29 -0700 (PDT)
MIME-Version: 1.0
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <CALDnm52XM+S07OPXO3qtfLN4pup-kT2U=WRO9dUo3tv9uYJWXw@HIDDEN>
 <38a06f3c-4a7a-755c-c99b-708f91afabfa@HIDDEN>
 <CALDnm537=LHAoaGqPpy7=JOPsYn=z+=Q6QW=b9bjrAWOtxqRaQ@HIDDEN>
 <9f59f87c-2489-aaa0-5b3f-0e911b7ffa6c@HIDDEN>
 <CALDnm508EK5qM+AaOOJiNDm-=ORCuOieYpkjO=OSRas+_WTFwQ@HIDDEN>
 <8a36e61a-1c5b-bf3b-a454-077348589c4f@HIDDEN> <87y29471ov.fsf@HIDDEN>
 <837dgob6yo.fsf@HIDDEN> <87wnootec0.fsf@HIDDEN>
 <5d70b0ad-3838-ddb8-d341-9a5531d9da73@HIDDEN>
 <CALDnm51xYehwuGhMubovLTv_nqtdJYvOmH2e-bH3cCjf0PS2WQ@HIDDEN>
 <0cbf224b-b382-8a02-3f06-9f6d7e5e9741@HIDDEN>
 <CALDnm52x_S2UXE=YVokCWEZf9Gqc0O+g4ZVhi+R60SCeq6JtzA@HIDDEN>
 <eecaa682-496e-8173-caf7-2f9781cc7102@HIDDEN> <87a6lihv7b.fsf@HIDDEN>
 <63e03b18-db81-3b11-c692-0af9df20c506@HIDDEN>
 <CALDnm50W_=Y_zP7KLznqcqSx8KJp2-0u2NwpuEGzzAUzCNHvKQ@HIDDEN>
 <c95dea17-ca29-4dee-d53d-b5d46f7f2cf8@HIDDEN>
In-Reply-To: <c95dea17-ca29-4dee-d53d-b5d46f7f2cf8@HIDDEN>
From: =?UTF-8?B?Sm/Do28gVMOhdm9yYQ==?= <joaotavora@HIDDEN>
Date: Mon, 16 Aug 2021 12:37:17 +0100
Message-ID: <CALDnm53TD=FCf0davQC6q9esVxz687oBGN2LF=Egz3-kz6681Q@HIDDEN>
Subject: Re: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new
 `completion-filter-completions` API and deferred highlighting
To: Daniel Mendler <mail@HIDDEN>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 47711
Cc: Stefan Monnier <monnier@HIDDEN>, 48841 <at> debbugs.gnu.org,
 Dmitry Gutov <dgutov@HIDDEN>, Lars Ingebrigtsen <larsi@HIDDEN>,
 Eli Zaretskii <eliz@HIDDEN>, 47711 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

On Mon, Aug 16, 2021 at 11:53 AM Daniel Mendler <mail@HIDDEN> wr=
ote:

> >> There are serious drawbacks of attaching "private" string properties t=
o
> >> arbitrary strings. For once it complicates debugging seriously if ther=
e
> >> are suddenly string properties attached to symbol names. It also leads
> >> to a potential memory leak.
> > Please, in the name of the sanity of this discussion, justify these two
> > statements with examples or follow them with a clause like "because..."=
.
> Jo=C3=A3o, I am giving hard examples here.

If I say to you:  "It's quite obvious your patch breaks all Git repositorie=
s
in Kathmandu, Nepal"  I am expected to demonstrate how a patch to
Emacs, leads to a obscure security flaw in the Linux operating system,
that tickles a butterfly at a certain angle that causes an earthquake
in the Kathmandu data center, literally breaking their Git repositories.

This is the the kind of statement you are making to me and the kind
of logical reasoning I'm looking for.

Alternatively, you can provide an actual experiment I can run in
my computer that demonstrates the bug.

I am not a native English speaker, and maybe you don't understand
my language.  Another way to explain what I am talking  is to talk about
"bug reproduction".  You say there's a bug in my patch, I am asking you
for a "bug reproduction recipe" as defined by most,  if not all, the result=
s
you get by searching "bug reproduction recipe" in the Google search engine.

> goal is not to be right here just for the sake of being right. As Eli
> said, nobody has to prove anything.

This is clearly not what he said.

> >> 3. The `completion-filter-completions` API is the fastest possible API
> >
> > Again that's quite a statement that I cannot evaluate the veracity of
> > without hard proof.
>
> As I said, I will ensure that my API does not introduce performance
> regressions.

Not only that, to produce  veracity on that statement you would need
some much more demanding proof, which is somehow be able to
evaluate all possible APIs to compellingly demonstrate that yours
triumphs.

>  I have a patch for Vertico which shows
> this. I can provide patches for 'icomplete-mode' separately later on.

Yes, please do. The earlier, the better.

> No, we should not merge this problematic patch of yours. See the many
> arguments against this proposal.

I'm sorry to speak this child-like language, but a problem is a "bad thing"=
.
An undesirable thing that happens when presumably safe and good
action(s) is taken by some user.  Can you explain how, given my patch,
a user would take a sequence of innocent actions that would lead to a
"bad thing" that would _not_ happen if the same sequence of innocent
actions were taken  in a version of Emacs without the patch applied?
That, to me, is what constitutes "a bug/problem in the patch".

Let me give you an example: if I make a patch that deletes `/lisp` in
the Emacs source tree, the innocent action "make"  would probably
not work.  That would be the problem/"bad thing"/bug in that patch.

We cannot proceed this discussion without these explanations.  Mind
you, I'm not stating, because it is impossible to prove, that my
patch cannot possibly generate problems, subtle or obvious (that, by
the way, is my interpretation of what Eli meant). But since you so
confidently state that it does, it's quite reasonable that I ask you
for examples that demonstrate it.

Once you do demonstrate these bugs, it's reasonable I will go about
fixing them.  Exactly as you say you are going to do.  I demonstrated
with code and numbers a regression in _your_ patch, and you say are
going to fix it.  That's great, and that's the way it should be.  But
you possibly wouldn't go about fixing it if I hadn't demonstrated the
regression compellingly, just as I can't go about fixing a "memory leak"
or "debugging difficulties" if you don't explain what these things mean
to you or how my patch causes them.

Jo=C3=A3o




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 16 Aug 2021 10:53:10 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Aug 16 06:53:10 2021
Received: from localhost ([127.0.0.1]:48623 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mFaEv-000701-S4
	for submit <at> debbugs.gnu.org; Mon, 16 Aug 2021 06:53:10 -0400
Received: from server.qxqx.de ([178.63.65.180]:42099 helo=mail.qxqx.de)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mail@HIDDEN>)
 id 1mFaEt-0006zS-Mq; Mon, 16 Aug 2021 06:53:08 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=qxqx.de;
 s=mail1392553390; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:
 MIME-Version:Date:Message-ID:From:References:Cc:To:Subject:Sender:Reply-To:
 Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender:
 Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:
 List-Subscribe:List-Post:List-Owner:List-Archive;
 bh=/fVONndn1d2yojf57smO1XDuVRiSgpuk9VBMHyJfqo0=; b=IG5RSU61Y3C7Qtn3VtFY1CebtN
 +QPuxDouSXj1jJDLk3YuN8NerQGacWg826fbv27K1W4K4SkfLPHt6nBueSDh5KslUAx1/MigfCm+o
 USkrYrDdUGpOqnmNw6GjAa54BOEu+YC3aOsZg6Zvz6MbuekLzZ8IqlAsoY3ZQ21Bv2QQ=;
Subject: Re: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new
 `completion-filter-completions` API and deferred highlighting
To: =?UTF-8?B?Sm/Do28gVMOhdm9yYQ==?= <joaotavora@HIDDEN>
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <CALDnm52XM+S07OPXO3qtfLN4pup-kT2U=WRO9dUo3tv9uYJWXw@HIDDEN>
 <38a06f3c-4a7a-755c-c99b-708f91afabfa@HIDDEN>
 <CALDnm537=LHAoaGqPpy7=JOPsYn=z+=Q6QW=b9bjrAWOtxqRaQ@HIDDEN>
 <9f59f87c-2489-aaa0-5b3f-0e911b7ffa6c@HIDDEN>
 <CALDnm508EK5qM+AaOOJiNDm-=ORCuOieYpkjO=OSRas+_WTFwQ@HIDDEN>
 <8a36e61a-1c5b-bf3b-a454-077348589c4f@HIDDEN> <87y29471ov.fsf@HIDDEN>
 <837dgob6yo.fsf@HIDDEN> <87wnootec0.fsf@HIDDEN>
 <5d70b0ad-3838-ddb8-d341-9a5531d9da73@HIDDEN>
 <CALDnm51xYehwuGhMubovLTv_nqtdJYvOmH2e-bH3cCjf0PS2WQ@HIDDEN>
 <0cbf224b-b382-8a02-3f06-9f6d7e5e9741@HIDDEN>
 <CALDnm52x_S2UXE=YVokCWEZf9Gqc0O+g4ZVhi+R60SCeq6JtzA@HIDDEN>
 <eecaa682-496e-8173-caf7-2f9781cc7102@HIDDEN> <87a6lihv7b.fsf@HIDDEN>
 <63e03b18-db81-3b11-c692-0af9df20c506@HIDDEN>
 <CALDnm50W_=Y_zP7KLznqcqSx8KJp2-0u2NwpuEGzzAUzCNHvKQ@HIDDEN>
From: Daniel Mendler <mail@HIDDEN>
Message-ID: <c95dea17-ca29-4dee-d53d-b5d46f7f2cf8@HIDDEN>
Date: Mon, 16 Aug 2021 12:52:58 +0200
MIME-Version: 1.0
In-Reply-To: <CALDnm50W_=Y_zP7KLznqcqSx8KJp2-0u2NwpuEGzzAUzCNHvKQ@HIDDEN>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 8bit
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 47711
Cc: Stefan Monnier <monnier@HIDDEN>, 48841 <at> debbugs.gnu.org,
 Dmitry Gutov <dgutov@HIDDEN>, Lars Ingebrigtsen <larsi@HIDDEN>,
 Eli Zaretskii <eliz@HIDDEN>, 47711 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

On 8/16/21 12:15 PM, João Távora wrote:
> On Mon, Aug 16, 2021 at 10:09 AM Daniel Mendler <mail@HIDDEN> wrote:
> 
>> There are serious drawbacks of attaching "private" string properties to
>> arbitrary strings. For once it complicates debugging seriously if there
>> are suddenly string properties attached to symbol names. It also leads
>> to a potential memory leak.
> 
> Please, in the name of the sanity of this discussion, justify these two
> statements with examples or follow them with a clause like "because...".

João, I am giving hard examples here. What is not an example about
"memory leak" or making debugging output verbose thanks to the attached
string properties? You always repeat your demands but whatever I write
it is not satisfactory for you. Is it possible to convince you? Can you
try to interpret my arguments in a positive and constructive light
somehow and fill them with meaning such that it makes sense to you? My
goal is not to be right here just for the sake of being right. As Eli
said, nobody has to prove anything.

>> 3. The `completion-filter-completions` API is the fastest possible API
> 
> Again that's quite a statement that I cannot evaluate the veracity of
> without hard proof.

As I said, I will ensure that my API does not introduce performance
regressions. And since my new API performs strictly less work than your
proposal it will necessarily be faster if you consider only the
filtering, which is what matters for incrementally updating UIs.

>> 4. If 'completion-all-completions' does indeed get slower thanks to my
>> patch, it is a performance regression of my patch. I will fix this. And
>> I thank you João for bringing this to my attention. However one should
>> also consider that in the end, 'fido-mode' and 'icomplete-mode' should
>> move to the new API 'completion-filter-completions' such that they
>> benefit from the fast filtering and only copy and highlight the actually
>> displayed strings.
> 
> Maybe they will, maybe they will.  But it's still quite early to decide if
> we're going to move all frontends to that API.  There's no manual
> documentation for it. Conceivably, if you appreciate your API so, you
> could demonstrate in practice us how easy it is to use by providing
> a separate patch that adapts icomplete-mode and fido-mode to use it.

There is also no manual documentation of 'completion-all-completions'.
Of course you are right that it is early to make these decisions, but
the new API 'completion-filter-completions' is designed in a way to
allow adoption by 'icomplete-mode'. It is important to design the API
such that adoption is possible. I have a patch for Vertico which shows
this. I can provide patches for 'icomplete-mode' separately later on.

> In the meantime, there is a patch with a measured and documented
> performance boost where fido-mode and icomplete-mode move
> opt-in to a new `completion-lazy-hilit` feature in the "old" API with
> a total or four 1-line changes.  That patch lives in the branch
> scratch/icomplete-lazy-highlight-attempt-2.

As argued multiple times here now, the change you are proposing is
fragile. It will lead to problems later on. The goal is not to find the
smallest change which leads to a performance boost, all API violations
allowed. Attaching "private" string properties to arbitrary strings is
an API violation which we will regret later and which will make Emacs
harder to debug and harder to use.

> I think we should move to that, solving the bug#48841 while we
> do the evaluation of all aspects of your contributions.

No, we should not merge this problematic patch of yours. See the many
arguments against this proposal.

Daniel




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 16 Aug 2021 10:16:00 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Aug 16 06:16:00 2021
Received: from localhost ([127.0.0.1]:48599 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mFZeu-0005me-BP
	for submit <at> debbugs.gnu.org; Mon, 16 Aug 2021 06:16:00 -0400
Received: from mail-pj1-f49.google.com ([209.85.216.49]:50939)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <joaotavora@HIDDEN>)
 id 1mFZep-0005mL-7s; Mon, 16 Aug 2021 06:15:55 -0400
Received: by mail-pj1-f49.google.com with SMTP id bo18so25799769pjb.0;
 Mon, 16 Aug 2021 03:15:51 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=mime-version:references:in-reply-to:from:date:message-id:subject:to
 :cc:content-transfer-encoding;
 bh=oCu4cbI2ZHBlH+viXbn57tCqpQanGyFP2VKCGfxLh0g=;
 b=pOBU+OJWKVLpM+bgIOsSX7qDT6wD3fmisZzaVMYPZY7PDkcACW5A4vLnHqDhy/2MMB
 cKa8u/42PbX9UHUY5IPmRJN7xoGPyMJVXy7+ssfrBYza39Py4cN70Ltty0/DVuAelM17
 Eehl2glcdmfPy3QfEap7uc/uXKbDEcUTOjefFFcchyPcFGh/jpG5ubBq8cRFkkXIsDu+
 L2fpHWh1F6k/W8lpOiF0UfrvxZNuX42ApNTInrgGbuxY0kPrA3R2I+cNsmsP/ghLg8Dn
 bf3gRVr7KA/jsQbQGrRV2ecDJLP8NdjjrUx3a6pArAjDAeGhm3lgjU8fSsc6S4j20sHY
 WViQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:mime-version:references:in-reply-to:from:date
 :message-id:subject:to:cc:content-transfer-encoding;
 bh=oCu4cbI2ZHBlH+viXbn57tCqpQanGyFP2VKCGfxLh0g=;
 b=rhsHunsSCjvFv79B7foeYQKAK33wUzmRtqIkwHeKwcGA0vIBqBBQhEcdFPNyRGGpvy
 AarweVcTNTI4eF7fs2Hrf/C9fDOqJAWOGrNj2BYLOl5ywiVtDuqMOZ7QjwEE1aSHrJIZ
 OeE2ok2O9/3kVtu2V1rWp93Omyz87sjZ/sAGxnSfSC7sZCUUUizxMn2PTkIt1gtuvDij
 XTjYmDTLvcTnDalrduCKomn72KtbJGg2vE6LSCLFPfwRDhxf8oGRGjWTcvi6RBI9Id8K
 FYZjGLtFEZyoLxnXvtcWwyA0xwZf99NiJ+j7QwnkUcqMl5WdH48UBDn7U6J1mfOaj1OW
 R8Ew==
X-Gm-Message-State: AOAM530AAvV6A9459vlqXfrpFbwyQEfKBEcNBC6HESsdiu8ZXx1H0t+S
 qVUNecWQJwViMsgQdHcOuAXcWvBn4x5XTuxHwk0=
X-Google-Smtp-Source: ABdhPJywDmMGd3lrQvu/IRBaYjuAjSKHTu2VbM9Pbf95ULRtVPqyQviOnAxC5kaOj1E9zxtJfwHeTrP7lYUHnU5kk9E=
X-Received: by 2002:a17:90a:3849:: with SMTP id l9mr5591552pjf.7.1629108945143; 
 Mon, 16 Aug 2021 03:15:45 -0700 (PDT)
MIME-Version: 1.0
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <CALDnm53xh6boc6T9XSC4jZpkCPS3RB8JFNb12BpkoeGYg3Q9RQ@HIDDEN>
 <56ab18b1-4348-9b2c-85bb-af9b76cd429a@HIDDEN>
 <CALDnm52XM+S07OPXO3qtfLN4pup-kT2U=WRO9dUo3tv9uYJWXw@HIDDEN>
 <38a06f3c-4a7a-755c-c99b-708f91afabfa@HIDDEN>
 <CALDnm537=LHAoaGqPpy7=JOPsYn=z+=Q6QW=b9bjrAWOtxqRaQ@HIDDEN>
 <9f59f87c-2489-aaa0-5b3f-0e911b7ffa6c@HIDDEN>
 <CALDnm508EK5qM+AaOOJiNDm-=ORCuOieYpkjO=OSRas+_WTFwQ@HIDDEN>
 <8a36e61a-1c5b-bf3b-a454-077348589c4f@HIDDEN> <87y29471ov.fsf@HIDDEN>
 <837dgob6yo.fsf@HIDDEN> <87wnootec0.fsf@HIDDEN>
 <5d70b0ad-3838-ddb8-d341-9a5531d9da73@HIDDEN>
 <CALDnm51xYehwuGhMubovLTv_nqtdJYvOmH2e-bH3cCjf0PS2WQ@HIDDEN>
 <0cbf224b-b382-8a02-3f06-9f6d7e5e9741@HIDDEN>
 <CALDnm52x_S2UXE=YVokCWEZf9Gqc0O+g4ZVhi+R60SCeq6JtzA@HIDDEN>
 <eecaa682-496e-8173-caf7-2f9781cc7102@HIDDEN> <87a6lihv7b.fsf@HIDDEN>
 <63e03b18-db81-3b11-c692-0af9df20c506@HIDDEN>
In-Reply-To: <63e03b18-db81-3b11-c692-0af9df20c506@HIDDEN>
From: =?UTF-8?B?Sm/Do28gVMOhdm9yYQ==?= <joaotavora@HIDDEN>
Date: Mon, 16 Aug 2021 11:15:33 +0100
Message-ID: <CALDnm50W_=Y_zP7KLznqcqSx8KJp2-0u2NwpuEGzzAUzCNHvKQ@HIDDEN>
Subject: Re: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new
 `completion-filter-completions` API and deferred highlighting
To: Daniel Mendler <mail@HIDDEN>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 47711
Cc: Lars Ingebrigtsen <larsi@HIDDEN>, 47711 <at> debbugs.gnu.org,
 Stefan Monnier <monnier@HIDDEN>, 48841 <at> debbugs.gnu.org,
 Dmitry Gutov <dgutov@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

On Mon, Aug 16, 2021 at 10:09 AM Daniel Mendler <mail@HIDDEN> wr=
ote:

> There are serious drawbacks of attaching "private" string properties to
> arbitrary strings. For once it complicates debugging seriously if there
> are suddenly string properties attached to symbol names. It also leads
> to a potential memory leak.

Please, in the name of the sanity of this discussion, justify these two
statements with examples or follow them with a clause like "because...".

> 3. The `completion-filter-completions` API is the fastest possible API

Again that's quite a statement that I cannot evaluate the veracity of
without hard proof.

What I _can_ evaluate is the material you have put forward, and using
your patch and your Vertico completion software, version 0.14, the very
latest one. I try

   emacs -Q -f package-initialize benchmark.el

where benchmark.el is:

   (setq completion-styles '(flex))
   (defmacro heyhey ()
     `(progn
        ,@(cl-loop repeat 300000
           collect `(defun ,(intern (symbol-name (gensym "heyhey"))) () 42)=
)))
   (heyhey)

I then turn on vertico-mode and press C-h f.  It takes about 4-5 seconds.
It's *faster* than if I do the same with fido-vertical-mode and the current
master, but is noticeably *slower* than if I do the same with the patch
provided earlier and available at scratch/icomplete-lazy-highlight-attempt-=
2.

Unfortunately, I cannot measure quantitatively here, because I don't
know how to tell Vertico to wait until it gets the correct result.
In other words, take this form:

    (completing-read "bla" obarray)<cursor here>

if you type C-u C-x C-e C-m veeery s-l-o-w-l-y in Vertico, if prints
, correctly, the character "%".  But if you evaluate it quickly wrapped
in a benchmark-run, it returns immediately and prints "".

In fido-mode, it always waits blockinly until it prints the correct result
and the time it took it to achieve that result.  Not questioning if this is
a bug in Vertico, but it would help if it could do the same, or be
configured to do the same, so that we can measure.

Without that, we can't evaluate the speed of Vertico where,
presumably, the fastest API in the world is being used right now.

> 4. If 'completion-all-completions' does indeed get slower thanks to my
> patch, it is a performance regression of my patch. I will fix this. And
> I thank you Jo=C3=A3o for bringing this to my attention. However one shou=
ld
> also consider that in the end, 'fido-mode' and 'icomplete-mode' should
> move to the new API 'completion-filter-completions' such that they
> benefit from the fast filtering and only copy and highlight the actually
> displayed strings.

Maybe they will, maybe they will.  But it's still quite early to decide if
we're going to move all frontends to that API.  There's no manual
documentation for it. Conceivably, if you appreciate your API so, you
could demonstrate in practice us how easy it is to use by providing
a separate patch that adapts icomplete-mode and fido-mode to use it.

Then, I or other fido-mode users could test it for a while, evaluating
its speed and correctness.  If it performs well and the completion
API architects have a good outlook for it, I see no reason why it
shouldn't be merged and eventually supersede the new one.

In the meantime, there is a patch with a measured and documented
performance boost where fido-mode and icomplete-mode move
opt-in to a new `completion-lazy-hilit` feature in the "old" API with
a total or four 1-line changes.  That patch lives in the branch
scratch/icomplete-lazy-highlight-attempt-2.

I think we should move to that, solving the bug#48841 while we
do the evaluation of all aspects of your contributions.

Jo=C3=A3o T=C3=A1vora




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 16 Aug 2021 09:42:32 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Aug 16 05:42:32 2021
Received: from localhost ([127.0.0.1]:48557 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mFZ8Z-0004u9-TM
	for submit <at> debbugs.gnu.org; Mon, 16 Aug 2021 05:42:32 -0400
Received: from server.qxqx.de ([178.63.65.180]:43069 helo=mail.qxqx.de)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mail@HIDDEN>)
 id 1mFZ8Y-0004ts-8W; Mon, 16 Aug 2021 05:42:31 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=qxqx.de;
 s=mail1392553390; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:
 MIME-Version:Date:Message-ID:From:References:Cc:To:Subject:Sender:Reply-To:
 Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender:
 Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:
 List-Subscribe:List-Post:List-Owner:List-Archive;
 bh=C6FYAFTRxF50v0M2jWCYcMTMDZouKfxCvDwJMzz16sk=; b=FKUNVDxD41IMLfHD0mdoT3ASsD
 n3JSH3Cw1ZWQxsEb4xxFSb6GITypglkL0jp1mlqd/9S4aom7WFdGcGAWDwvHSV8H39mwsQ19mtL0i
 lUt0Wz1Rm2H835cMHlsoIZuWmt8e9ySFM5KF6+UFScatCxCtVJ3u0ijJPUX5qTV2hHcM=;
Subject: Re: bug#48841: [PATCH] Add new `completion-filter-completions` API
 and deferred highlighting
To: Eli Zaretskii <eliz@HIDDEN>
References: <f94541d3-5f93-9a23-8256-3bbc453f127a@HIDDEN>
 <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <837dgrdrec.fsf@HIDDEN>
 <aff2bd3a-ea3e-1ac6-cf0d-2785c6591c84@HIDDEN>
 <83mtpkbky3.fsf@HIDDEN>
From: Daniel Mendler <mail@HIDDEN>
Message-ID: <e2427908-17c5-ad18-a50a-2f2ebb61982c@HIDDEN>
Date: Mon, 16 Aug 2021 11:42:22 +0200
MIME-Version: 1.0
In-Reply-To: <83mtpkbky3.fsf@HIDDEN>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 8bit
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 47711
Cc: 48841 <at> debbugs.gnu.org, 47711 <at> debbugs.gnu.org, joaotavora@HIDDEN,
 monnier@HIDDEN, dgutov@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Hello Eli,

here I respond to the comments you sent out after I've already sent the
overhauled patch.

On 8/14/21 8:27 AM, Eli Zaretskii wrote:
>> The function 'completion-filter-completions' receives a completion table
>> as argument.  The strings produced by this table are returned
>> unmodified, but of course the completion table has to produce them.  For
>> a static completion table (e.g., in the simplest case a list of strings)
>> the completion table itself will not allocate strings.  In this scenario
>> 'completion-filter-completions' will not perform any string allocations,
>> only the list will be allocated.  This is what leads to major
>> performance gains.
> 
> My point was that at least some of this should be in the description,
> otherwise it will leave the reader wondering.

I agree with this. The documentation should be improved.

>>>> +(defvar completion--filter-completions nil
>>>> +  "Enable the new completions return value format.
>>>
>>> Btw, why is this an internal variable?  Shouldn't all completion
>>> styles ideally support it?  If so, it should be a public variable,
>>> documented in the ELisp manual.  And the name should also end with -p,
>>> since it's a boolean.  How about completion-filter-completions-format-p?
>>
>> (As I understood the style guide '-p' is not a good idea for boolean
>> variables, since a value is not a predicate in a strict sense.)
>>
>> To address your technical comment - this variable is precisely what one
>> of the technical difficulties mentioned in my other mail is about.  The
>> question is how we can retain backward compatibility of the completion
>> style 'all' functions, e.g., 'completion-basic-all-completions', while
>> still allowing the function to return the newly introduced alist format
>> with more data, which enables 'completion-filter-completions' to perform
>> the efficient deferred highlighting.
> 
> I understand, but given that we provide this for other packages, it
> shouldn't be an internal variable.

No, we explicitly don't provide this variable for other packages. It is
explicitly only meant to be used for the existing completion styles
emacs21, emacs22, basic, substring, partial-completion, initials and
flex to opt-in in a backward-compatible/calling convention preserving
way to the alist return format. The idea is to keep the existing APIs
fully backward compatible.

Other packages should select the format returned from the completion
styles differently. They should return the alist format on Emacs 28 or
if the API 'completion-filter-completions' API is present. In the not so
near future external packages which support only Emacs 28 and upwards
will then only return the alist format and don't have to perform any
detection anymore.

>>> Also, the "This function has been superseded..." part should be a new
>>> paragraph, so that it stands out.  (And I'm not yet sure we indeed
>>> want to say "superseded" here, but that's part of the on-going
>>> discussion.  maybe use a more neutral language here, like "See also".)
>>
>> The new API 'completion-filter-completions' will substitute the existing
>> API 'completion-all-completions'.
> 
> That's your hope, and I understand.  But we as a project didn't yet
> decide to deprecate the original APIs, so talking about superseding is
> premature.

It is not the hope - it is the explicit goal. The API has been designed
to replace the existing API 'completion-all-completions'. We can of
course tone this down. However I, as a package author, would appreciate
if Emacs tells me when a newer API aims to replace another API and when
the documentation is explicit about it. Of course if you decide to have
the doc strings written in a different tone, I will adapt my patch
accordingly. Here I am just explaining why I chose the word "superseded"
instead of a more neutral word.

>>> Is "filter" really the right word here (in the doc string)?  "Filer"
>>> means you take a sequence and produce another sequence with some
>>> members removed.  That's not what this API does, is it?  Suggest to
>>> use a different name, like completion-completions-alist or
>>> completion-all-completions-as-alist.
>>
>> "Filter" seems like exactly the right word to me.  The function takes a
>> list of strings (or a completion table) and returns a subset of matching
>> completion strings without further modifications to the strings. See
>> above what I wrote about allocations.
> 
> But the name says "filter completions".  Which would mean you take a
> list of completions and filter out some of them.  A completion table
> is much more general object than a list of strings.  Thus, I think
> using "filter" here is sub-optimal.

Okay, you are right about this. But I think even if the name
'completion-filter-completions' is not 100% precise it still conveys
what the API is about. 'completion-completions-alist' or
'completion-all-completions-as-alist' are valid names of course, but I
dislike them for their verbosity. Already 'completion-all-completions'
is quite verbose. A strong argument to use this long name is that the
completion style functions are still called
'completion-basic-all-completions' etc. But if we accept that the new
API 'completion-filter-completions' will actually supersede the existing
API 'completion-all-completions' it makes sense to use a name which will
not hurt our eyes in the long run. However this is of course just a
personal preference of mine. I don't want to spent much time with name
bikeshedding discussions. If you decide on a name, I will adapt my patch
accordingly.

>>>> +Only the elements of table that satisfy predicate PRED are considered.
>>>> +POINT is the position of point within STRING.  The METADATA may be
>>>> +modified by the completion style.  The return value is a alist with
>>>> +the keys:
>>>> +
>>>> +- base: Base position of the completion (from the start of STRING)
>>>
>>> "Base" here means the beginning?  If so, why not call it "beg" or
>>> somesuch?
>>
>> Base position is a fixed term which is already used in minibuffer.el for
>> completions.  See also 'completion-base-position' for example.
> 
> Well, we don't have to keep bad habits indefinitely.  It's okay to
> lose them and use better terminology.  Or at least to explain that
> terminology in parentheses the first time it is used in some context.

Okay, I agree. However I tried to avoid including superfluous changes
with my patch set. We should add more context and documentation and then
rename the variables in another patch if we decide that we want to go
through with it.

>>> Are we really losing the completion-score property here?  If so, why?
>>
>> Yes, the property is removed in the current patch.  It is not actually
>> used for anything in the new implementation.  But it is possible to
>> restore the property such that 'completion-all-completions' always
>> returns scored candidates as it does now.  See my other mail regarding
>> the caveats of the current patch.
> 
> I'd prefer not to lose existing features, because that'd potentially
> make the changes backward-incompatible.

The overhauled patch (version 2) ensures that no features are lost. The
patch is fully backward compatible. There is a potential performance
regression in 'completion-all-completions' as identified by João. I have
yet to confirm this regression. In any case, it should be fixable since
the refactored 'completion-all-completions' API does precisely the same
amount of work as it does now.

Furthermore more documentation should be added to my patch. As of now
'completion-all-completions' is not mentioned in the info manual and
'completion-filter-completions' is also not added there. We may want to
improve the documentation of that. But for now I would like to limit my
documentation improvements to expanding the doc strings of the functions
involved in my patch.

Daniel




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 16 Aug 2021 09:08:55 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Aug 16 05:08:55 2021
Received: from localhost ([127.0.0.1]:48488 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mFYbp-0003yu-Ms
	for submit <at> debbugs.gnu.org; Mon, 16 Aug 2021 05:08:55 -0400
Received: from server.qxqx.de ([178.63.65.180]:33691 helo=mail.qxqx.de)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mail@HIDDEN>)
 id 1mFYbn-0003ye-BM; Mon, 16 Aug 2021 05:08:40 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=qxqx.de;
 s=mail1392553390; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:
 MIME-Version:Date:Message-ID:From:References:Cc:To:Subject:Sender:Reply-To:
 Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender:
 Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:
 List-Subscribe:List-Post:List-Owner:List-Archive;
 bh=GJcT/T5DVTuHMs67rjtK1AoQVRPLwkXyW16s1GuaJfQ=; b=mcXl/177spo4DFSgMO5rICqYIR
 75/3YzAOxMWILnRCf6B7BUnCXi26il3qrnACsZKMwWM3/SAgDOzvgyBXxl68lpbg2q0raDMtzo6mp
 gQxuCbALUXXoCHXyqhjjJZKsttKSS9itzkbIzlbycWfYXP53dAxFWI7m7QnRnQHu90EU=;
Subject: Re: bug#48841: bug#47711: [PATCH VERSION 2] Add new
 `completion-filter-completions` API and deferred highlighting
To: =?UTF-8?B?Sm/Do28gVMOhdm9yYQ==?= <joaotavora@HIDDEN>,
 Dmitry Gutov <dgutov@HIDDEN>
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <CALDnm53xh6boc6T9XSC4jZpkCPS3RB8JFNb12BpkoeGYg3Q9RQ@HIDDEN>
 <56ab18b1-4348-9b2c-85bb-af9b76cd429a@HIDDEN>
 <CALDnm52XM+S07OPXO3qtfLN4pup-kT2U=WRO9dUo3tv9uYJWXw@HIDDEN>
 <38a06f3c-4a7a-755c-c99b-708f91afabfa@HIDDEN>
 <CALDnm537=LHAoaGqPpy7=JOPsYn=z+=Q6QW=b9bjrAWOtxqRaQ@HIDDEN>
 <9f59f87c-2489-aaa0-5b3f-0e911b7ffa6c@HIDDEN>
 <CALDnm508EK5qM+AaOOJiNDm-=ORCuOieYpkjO=OSRas+_WTFwQ@HIDDEN>
 <8a36e61a-1c5b-bf3b-a454-077348589c4f@HIDDEN> <87y29471ov.fsf@HIDDEN>
 <837dgob6yo.fsf@HIDDEN> <87wnootec0.fsf@HIDDEN>
 <5d70b0ad-3838-ddb8-d341-9a5531d9da73@HIDDEN>
 <CALDnm51xYehwuGhMubovLTv_nqtdJYvOmH2e-bH3cCjf0PS2WQ@HIDDEN>
 <0cbf224b-b382-8a02-3f06-9f6d7e5e9741@HIDDEN>
 <CALDnm52x_S2UXE=YVokCWEZf9Gqc0O+g4ZVhi+R60SCeq6JtzA@HIDDEN>
 <eecaa682-496e-8173-caf7-2f9781cc7102@HIDDEN> <87a6lihv7b.fsf@HIDDEN>
From: Daniel Mendler <mail@HIDDEN>
Message-ID: <63e03b18-db81-3b11-c692-0af9df20c506@HIDDEN>
Date: Mon, 16 Aug 2021 11:08:31 +0200
MIME-Version: 1.0
In-Reply-To: <87a6lihv7b.fsf@HIDDEN>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 8bit
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 47711
Cc: Lars Ingebrigtsen <larsi@HIDDEN>, Eli Zaretskii <eliz@HIDDEN>,
 Stefan Monnier <monnier@HIDDEN>, 48841 <at> debbugs.gnu.org,
 47711 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: 0.3 (/)

On 8/16/21 6:25 AM, João Távora wrote:
> Yes, I currently conclude that there are 0 drawbacks identified to
> creating, _via_ string properties, an association of, say, property
> 'joaot/answer' with value '42' to _any_ string in my current and future
> Emacs runtimes.
> 
> I conclude that because --- apart from your aesthetics considerations
> ("do we really want to see...") --- I have not seen identification of
> such drawbacks.  Have I missed any?

There are serious drawbacks of attaching "private" string properties to
arbitrary strings. For once it complicates debugging seriously if there
are suddenly string properties attached to symbol names. It also leads
to a potential memory leak. But even if we could chose this approach
with global side-effects I don't see a reason to do it given the
approach I am proposing in my patch, which avoids these problems entirely.

1. My approach decomposes the already existing completion machinery into
two steps: 1. filtering and 2. highlighting. It does not change the
fundamentals of the completion machinery. This decomposition of the
existing infrastructure is also what leads to the small number of added
lines to minibuffer.el, even if the patch itself is not as small as we
would like to have it.

2. Why take the chances of introducing potentially harmful global side
effects by attaching string properties to arbitrary strings if we can
avoid it easily?

3. The `completion-filter-completions` API is the fastest possible API
for filtering since it does not change the strings at all, it does not
attach string properties or modify the strings in any other way. By
construction, it is a pure function which only filters the list of
candidates. This makes the function easy to use and easy to reason
about. An API which adds private properties to the strings cannot be as
fast and non-intrusive.

4. If 'completion-all-completions' does indeed get slower thanks to my
patch, it is a performance regression of my patch. I will fix this. And
I thank you João for bringing this to my attention. However one should
also consider that in the end, 'fido-mode' and 'icomplete-mode' should
move to the new API 'completion-filter-completions' such that they
benefit from the fast filtering and only copy and highlight the actually
displayed strings. Given this a potential regression of
'completion-all-completions' would matter less for the incrementally
updating UIs. But of course I feel the same way as João - a performance
regression in the API 'completion-all-completions' is unacceptable. It
will be fixed.

Daniel




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 16 Aug 2021 08:53:40 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Aug 16 04:53:40 2021
Received: from localhost ([127.0.0.1]:48464 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mFYNI-0003ZM-Aq
	for submit <at> debbugs.gnu.org; Mon, 16 Aug 2021 04:53:40 -0400
Received: from server.qxqx.de ([178.63.65.180]:34305 helo=mail.qxqx.de)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mail@HIDDEN>)
 id 1mFYNH-0003Z0-16; Mon, 16 Aug 2021 04:53:39 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=qxqx.de;
 s=mail1392553390; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:
 MIME-Version:Date:Message-ID:From:References:Cc:To:Subject:Sender:Reply-To:
 Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender:
 Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:
 List-Subscribe:List-Post:List-Owner:List-Archive;
 bh=fpp4hAYM18zw+5pcKevICIykRc1pCDf5Y3PkSL0tb7Y=; b=OUfE89ov3IcQQzb6hGevkjrz8P
 irOW11NZG7k5ND6knED5g+woz2oU8VIk/25SMlkYTUj+A4ClRHUTENivqvtCtQYR7OMjQ6z54W1uz
 xZ3v5s46ROs3ffVlLWiC1fxMPMWTfBoIaAegq2FvWkOf9Bm07J/8O8F4YR+5R83t1EXE=;
Subject: Re: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new
 `completion-filter-completions` API and deferred highlighting
To: =?UTF-8?B?Sm/Do28gVMOhdm9yYQ==?= <joaotavora@HIDDEN>,
 Dmitry Gutov <dgutov@HIDDEN>
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <f8c4419a-7e83-b8a9-347c-0221a2052376@HIDDEN>
 <fff6e291-bbcc-93e3-92e7-00af4664b8d4@HIDDEN>
 <3d3f894f-a6fa-53ae-5d50-c3aa9bffc73e@HIDDEN>
 <CALDnm53xh6boc6T9XSC4jZpkCPS3RB8JFNb12BpkoeGYg3Q9RQ@HIDDEN>
 <56ab18b1-4348-9b2c-85bb-af9b76cd429a@HIDDEN>
 <CALDnm52XM+S07OPXO3qtfLN4pup-kT2U=WRO9dUo3tv9uYJWXw@HIDDEN>
 <328f87eb-6474-1442-e1ca-9ae8deb2a84a@HIDDEN> <87sfzca0zm.fsf@HIDDEN>
 <4cac92d2-056d-e7fb-0664-2dbccb5f980c@HIDDEN> <87eeauhvg6.fsf@HIDDEN>
From: Daniel Mendler <mail@HIDDEN>
Message-ID: <62d4ebdd-8d9c-13d3-77d2-df36352d4803@HIDDEN>
Date: Mon, 16 Aug 2021 10:53:31 +0200
MIME-Version: 1.0
In-Reply-To: <87eeauhvg6.fsf@HIDDEN>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 8bit
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 47711
Cc: Eli Zaretskii <eliz@HIDDEN>, Stefan Monnier <monnier@HIDDEN>,
 48841 <at> debbugs.gnu.org, 47711 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

On 8/16/21 6:20 AM, João Távora wrote:
> It's not me who is saying it, it's my Emacs :-) The real problem is that
> with Daniel's patch, the frontends using the current API (as in
> icomplete/fido) measurably become _slower_.  Though not by much (around
> 10%), it is still a shame.

I have to check this. I claim that 'completion-all-completions' should
not get slower with my patch. If it gets indeed slower as your benchmark
shows, this should be fixed and can be fixed since I am not doing
something else than decomposing the highlighting and filtering
processes, which are already present in the current machinery. The
amount of work stays the same. However in the case the new
'completion-filter-completions' API is used, the filtering will get much
faster since no highlighting and copying takes place.

Daniel




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 16 Aug 2021 08:48:35 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Aug 16 04:48:35 2021
Received: from localhost ([127.0.0.1]:48437 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mFYIJ-0003Ql-9P
	for submit <at> debbugs.gnu.org; Mon, 16 Aug 2021 04:48:35 -0400
Received: from server.qxqx.de ([178.63.65.180]:33607 helo=mail.qxqx.de)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mail@HIDDEN>)
 id 1mFYIE-0003QR-3v; Mon, 16 Aug 2021 04:48:30 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=qxqx.de;
 s=mail1392553390; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:
 MIME-Version:Date:Message-ID:From:References:Cc:To:Subject:Sender:Reply-To:
 Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender:
 Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:
 List-Subscribe:List-Post:List-Owner:List-Archive;
 bh=JPHtbETPUUl+hbFRN+vAcwY4jQIo9t7zr8cK40eYWmA=; b=lILtorBIeUZejadXdlsAZAftz8
 GwSlbBDxMmVJt4+9lA1546PwKiCUr1waQRx+VDMgjqBTQNwCo+X/f+bx1+8upDNzUPuRR5jbBnxTd
 dicn2cjyHJ1hNMTJJApMyqelX3Z8nI3ocxPk2/+29uxlHkpj/Oqr6YHgExJhoXIYGomM=;
Subject: Re: bug#48841: bug#47711: [PATCH VERSION 2] Add new
 `completion-filter-completions` API and deferred highlighting
To: Eli Zaretskii <eliz@HIDDEN>, Dmitry Gutov <dgutov@HIDDEN>
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <f8c4419a-7e83-b8a9-347c-0221a2052376@HIDDEN>
 <fff6e291-bbcc-93e3-92e7-00af4664b8d4@HIDDEN>
 <3d3f894f-a6fa-53ae-5d50-c3aa9bffc73e@HIDDEN>
 <CALDnm53xh6boc6T9XSC4jZpkCPS3RB8JFNb12BpkoeGYg3Q9RQ@HIDDEN>
 <56ab18b1-4348-9b2c-85bb-af9b76cd429a@HIDDEN>
 <CALDnm52XM+S07OPXO3qtfLN4pup-kT2U=WRO9dUo3tv9uYJWXw@HIDDEN>
 <38a06f3c-4a7a-755c-c99b-708f91afabfa@HIDDEN>
 <CALDnm537=LHAoaGqPpy7=JOPsYn=z+=Q6QW=b9bjrAWOtxqRaQ@HIDDEN>
 <9f59f87c-2489-aaa0-5b3f-0e911b7ffa6c@HIDDEN>
 <CALDnm508EK5qM+AaOOJiNDm-=ORCuOieYpkjO=OSRas+_WTFwQ@HIDDEN>
 <8a36e61a-1c5b-bf3b-a454-077348589c4f@HIDDEN> <83h7fsbitl.fsf@HIDDEN>
From: Daniel Mendler <mail@HIDDEN>
Message-ID: <75ce4b68-240f-2f0e-3953-c1f74962c3cd@HIDDEN>
Date: Mon, 16 Aug 2021 10:48:19 +0200
MIME-Version: 1.0
In-Reply-To: <83h7fsbitl.fsf@HIDDEN>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 7bit
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 47711
Cc: 47711 <at> debbugs.gnu.org, joaotavora@HIDDEN, 48841 <at> debbugs.gnu.org,
 monnier@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

On 8/14/21 9:12 AM, Eli Zaretskii wrote:
>> Since up until now completion-pcm--hilit-commonality copied all strings 
>> before modifying, completion tables such as described (with "shared" 
>> strings) have all been "legal". Suddenly deciding to stop supporting 
>> them would be a major API breakage with consequences that are hard to 
>> predict. And while I perhaps agree that it's an inconvenience, I don't 
>> think it's a choice we can simply make as this stage in c-a-p-f's 
>> development.
> 
> This sounds like an argument against Daniel's approach as well, no?
> Because if a completion API returns strings it "doesn't own", there
> will be restrictions on Lisp programs that use those strings, because
> those Lisp programs previously could do anything they liked with those
> strings, and now they cannot.  Or am I missing something?

No, in my patch the displayed candidate strings are still copied before
the text properties are attached. The strings are kept intact as they
are now.

Daniel




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 16 Aug 2021 08:47:30 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Aug 16 04:47:30 2021
Received: from localhost ([127.0.0.1]:48432 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mFYHG-0003On-88
	for submit <at> debbugs.gnu.org; Mon, 16 Aug 2021 04:47:30 -0400
Received: from server.qxqx.de ([178.63.65.180]:58513 helo=mail.qxqx.de)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mail@HIDDEN>)
 id 1mFYH6-0003OM-6C; Mon, 16 Aug 2021 04:47:21 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=qxqx.de;
 s=mail1392553390; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:
 MIME-Version:Date:Message-ID:From:References:Cc:To:Subject:Sender:Reply-To:
 Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender:
 Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:
 List-Subscribe:List-Post:List-Owner:List-Archive;
 bh=DXzv+Bnip6z/cdxMuTfEFoOWdakNTpWYQig4GgvIjnQ=; b=SSb5x9ZQ8ZsoUGdP+4TKoqrmPu
 ZYbOzyMx0c2frl89WsNnyvckcXnI8XtekCQ7ngPYb/Gy4u7jhZmLioX5utXVEh9JpCKi5V4Dlvama
 vU534OzP73bchDa85/UD87Ofk9BEUOkTgly/TVFZR7Q3P8TYwOPHHO3IvPry6oxzNsl8=;
Subject: Re: bug#47711: [PATCH VERSION 2] Add new
 `completion-filter-completions` API and deferred highlighting
To: Eli Zaretskii <eliz@HIDDEN>
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <f8c4419a-7e83-b8a9-347c-0221a2052376@HIDDEN>
 <fff6e291-bbcc-93e3-92e7-00af4664b8d4@HIDDEN>
 <3d3f894f-a6fa-53ae-5d50-c3aa9bffc73e@HIDDEN>
 <CALDnm53xh6boc6T9XSC4jZpkCPS3RB8JFNb12BpkoeGYg3Q9RQ@HIDDEN>
 <56ab18b1-4348-9b2c-85bb-af9b76cd429a@HIDDEN>
 <CALDnm52XM+S07OPXO3qtfLN4pup-kT2U=WRO9dUo3tv9uYJWXw@HIDDEN>
 <38a06f3c-4a7a-755c-c99b-708f91afabfa@HIDDEN>
 <CALDnm537=LHAoaGqPpy7=JOPsYn=z+=Q6QW=b9bjrAWOtxqRaQ@HIDDEN>
 <9f59f87c-2489-aaa0-5b3f-0e911b7ffa6c@HIDDEN>
 <83im08bjc3.fsf@HIDDEN>
From: Daniel Mendler <mail@HIDDEN>
Message-ID: <1d8ac48c-1627-cfa0-640c-9c1371612787@HIDDEN>
Date: Mon, 16 Aug 2021 10:47:06 +0200
MIME-Version: 1.0
In-Reply-To: <83im08bjc3.fsf@HIDDEN>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 7bit
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 47711
Cc: 48841 <at> debbugs.gnu.org, dgutov@HIDDEN, joaotavora@HIDDEN,
 monnier@HIDDEN, 47711 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

On 8/14/21 9:01 AM, Eli Zaretskii wrote:
> Just to make sure we are on the same page: adding a text property to a
> string doesn't mutate a string.  Lisp programs that process these
> strings will not necessarily see any difference, and displaying those
> strings will also not show any difference if the property is not
> related to display.  So the assumption that seems to be made here,
> that adding a property is the same as mutating a string, is IMO
> inaccurate if not incorrect.

While I agree that adding text properties is not mutation of the string
text itself it still mutates the string data structure. Dmitry made a
good point about this - if a completion table uses obarray as backend to
the completion table you suddenly attach text properties to symbol
names. This is not a good idea. I actually had such an issue once in a
package I developed, where I accidentally attached text properties (via
the mutating put-text-property API) to some strings I didn't own. This
lead to unexpected side effects at other places where I didn't expect
the strings to have properties attached.

Daniel




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 16 Aug 2021 04:25:54 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Aug 16 00:25:54 2021
Received: from localhost ([127.0.0.1]:48153 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mFUCA-0002k2-9M
	for submit <at> debbugs.gnu.org; Mon, 16 Aug 2021 00:25:54 -0400
Received: from mail-wm1-f50.google.com ([209.85.128.50]:41724)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <joaotavora@HIDDEN>)
 id 1mFUC9-0002jm-34; Mon, 16 Aug 2021 00:25:53 -0400
Received: by mail-wm1-f50.google.com with SMTP id
 c129-20020a1c35870000b02902e6b6135279so9318558wma.0; 
 Sun, 15 Aug 2021 21:25:53 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=from:to:cc:subject:references:date:in-reply-to:message-id
 :user-agent:mime-version:content-transfer-encoding;
 bh=xraikt2tOdI1hqt2PIK4lpMWEd383AhNGIeQUFzgCYk=;
 b=RB9s5ay9cJpj3JfZmyhyoSLYIFXMT5ydIoSYNM3u6zCL6QlpGmZP2oNQCpch40raog
 TOHiW70m6WVmD1IdrZCxKZvAscX+UzO9wJoo6ghAtyTj45PA1Jl5cvDG4tscWaKO7bJ2
 uGZPLmpv17sXGuYZBXWjxtzMqJjA9n/ww+/yYF9L11+dKO8YBO5X6FNDu9dM7rnBFDQE
 8I+yDwSiE1yPsJdptldyXf05QGVlN9T1nIC79gqv1bjkPHylki2GlSf2awZCMQogspE8
 SjmoZD3rlPMNfNvr2gyqsxtqCb8FiK0Ljy+5mEH25TYDBKppS4hi+YI7kUzX6RdbGpG6
 mq5g==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to
 :message-id:user-agent:mime-version:content-transfer-encoding;
 bh=xraikt2tOdI1hqt2PIK4lpMWEd383AhNGIeQUFzgCYk=;
 b=CvDHKtql6GmxrOse1+WhX89HKeZrb34/WTjTgbhKm1YPXvb1ojuVaJs2I9hIRJYOkV
 ZnDWQ2U2+hj+q6JkeqayGwPKvuz3dOmh2gxyx/gXqKhItaaQMZW7fNiKEEDHm519mNdK
 Sh9TKdowX/dqYMTokHDMk2gtNtGJP1yfB2eJg9jMtxQBkHRKr2S9Ob3EFNVutDW+bO+A
 qKxEI4TzsrvHYq844jcP8gPbixvXtAMvRiqVsMcS6Ypw+dzt0ija4EPQZ3ivR4pp61gi
 6NH8269JDqwZaiwXYLgXFNmpqCVa/3F3po5XyOBomdkqTlONKxxmQF7R9ygvOSuiNP9B
 lB1g==
X-Gm-Message-State: AOAM530kcnA4Y8lnn3jgc7WPsnLOC4ljT8TdBk6AxMlp4YY9VEneglkf
 r1Tv6KRyjY7CKd8bnnOqZFLyr9MU12M=
X-Google-Smtp-Source: ABdhPJwbYccrVXXTamaDRZJRM8ivS1HWK5Mr3M7d3sOwlkIhGYiqetskN2G/SF/Q/F6yYHq7PvDBLA==
X-Received: by 2002:a7b:cf0c:: with SMTP id l12mr13322338wmg.62.1629087946912; 
 Sun, 15 Aug 2021 21:25:46 -0700 (PDT)
Received: from krug (a94-133-27-132.cpe.netcabo.pt. [94.133.27.132])
 by smtp.gmail.com with ESMTPSA id h11sm9847108wrq.64.2021.08.15.21.25.45
 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
 Sun, 15 Aug 2021 21:25:46 -0700 (PDT)
From: =?utf-8?B?Sm/Do28gVMOhdm9yYQ==?= <joaotavora@HIDDEN>
To: Dmitry Gutov <dgutov@HIDDEN>
Subject: Re: bug#48841: bug#47711: [PATCH VERSION 2] Add new
 `completion-filter-completions` API and deferred highlighting
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <fff6e291-bbcc-93e3-92e7-00af4664b8d4@HIDDEN>
 <3d3f894f-a6fa-53ae-5d50-c3aa9bffc73e@HIDDEN>
 <CALDnm53xh6boc6T9XSC4jZpkCPS3RB8JFNb12BpkoeGYg3Q9RQ@HIDDEN>
 <56ab18b1-4348-9b2c-85bb-af9b76cd429a@HIDDEN>
 <CALDnm52XM+S07OPXO3qtfLN4pup-kT2U=WRO9dUo3tv9uYJWXw@HIDDEN>
 <38a06f3c-4a7a-755c-c99b-708f91afabfa@HIDDEN>
 <CALDnm537=LHAoaGqPpy7=JOPsYn=z+=Q6QW=b9bjrAWOtxqRaQ@HIDDEN>
 <9f59f87c-2489-aaa0-5b3f-0e911b7ffa6c@HIDDEN>
 <CALDnm508EK5qM+AaOOJiNDm-=ORCuOieYpkjO=OSRas+_WTFwQ@HIDDEN>
 <8a36e61a-1c5b-bf3b-a454-077348589c4f@HIDDEN>
 <87y29471ov.fsf@HIDDEN> <837dgob6yo.fsf@HIDDEN>
 <87wnootec0.fsf@HIDDEN>
 <5d70b0ad-3838-ddb8-d341-9a5531d9da73@HIDDEN>
 <CALDnm51xYehwuGhMubovLTv_nqtdJYvOmH2e-bH3cCjf0PS2WQ@HIDDEN>
 <0cbf224b-b382-8a02-3f06-9f6d7e5e9741@HIDDEN>
 <CALDnm52x_S2UXE=YVokCWEZf9Gqc0O+g4ZVhi+R60SCeq6JtzA@HIDDEN>
 <eecaa682-496e-8173-caf7-2f9781cc7102@HIDDEN>
Date: Mon, 16 Aug 2021 05:25:44 +0100
In-Reply-To: <eecaa682-496e-8173-caf7-2f9781cc7102@HIDDEN> (Dmitry Gutov's
 message of "Mon, 16 Aug 2021 06:59:52 +0300")
Message-ID: <87a6lihv7b.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 47711
Cc: Daniel Mendler <mail@HIDDEN>, Lars Ingebrigtsen <larsi@HIDDEN>,
 Stefan Monnier <monnier@HIDDEN>, 48841 <at> debbugs.gnu.org,
 47711 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Dmitry Gutov <dgutov@HIDDEN> writes:

> On 16.08.2021 06:53, Jo=C3=A3o T=C3=A1vora wrote:
>> But why do it, since a string plist is a such a nice place to do these
>> associations and there's -- apart from your aesthetics considerations
>> -- 0 drawbacks identified?
>
> You read all the explanations, and THAT'S your conclusion?

Yes, I currently conclude that there are 0 drawbacks identified to
creating, _via_ string properties, an association of, say, property
'joaot/answer' with value '42' to _any_ string in my current and future
Emacs runtimes.

I conclude that because --- apart from your aesthetics considerations
("do we really want to see...") --- I have not seen identification of
such drawbacks.  Have I missed any?

Jo=C3=A3o





Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 16 Aug 2021 04:20:37 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Aug 16 00:20:37 2021
Received: from localhost ([127.0.0.1]:48146 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mFU73-0002av-2z
	for submit <at> debbugs.gnu.org; Mon, 16 Aug 2021 00:20:37 -0400
Received: from mail-wr1-f47.google.com ([209.85.221.47]:43672)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <joaotavora@HIDDEN>)
 id 1mFU70-0002ae-HF; Mon, 16 Aug 2021 00:20:35 -0400
Received: by mail-wr1-f47.google.com with SMTP id z9so21578529wrh.10;
 Sun, 15 Aug 2021 21:20:34 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=from:to:cc:subject:references:date:in-reply-to:message-id
 :user-agent:mime-version:content-transfer-encoding;
 bh=6ORklZy2oh7+mIpgyzo/CZybnxRFNWUtbViQ+U+jU1o=;
 b=YL5LwTo17TvVYrpoNbfomTr0onwcObAQEtAFYSujsdtf3kybfR4RNg/AzbmC5d788V
 zc5R/0Y8+ufO/IID9VNIkX2GvdpcnyOgq5kjZseX1rU+wKLKZLqGJAHroIZC052u/FCj
 WDQ6B/rmVuklCfE0hq+U83kfr3WOby+DUsKNQ6lvCNh0bs5wfUUwZ4Bz+r1ZZuWjsDtX
 CUEErG7fzFY0hLaiTmP+zp1cZVqMsoNV7Hkp/ByhGeI1jTAjYDXwvEsEVjd0AOnX8rcA
 tERmS+PVbO+OSntEBAcOd4w9oN2v7WEe7KASbC3pLMK57/kdKLnNlyZXHWrgYcuDSRA7
 4vyQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to
 :message-id:user-agent:mime-version:content-transfer-encoding;
 bh=6ORklZy2oh7+mIpgyzo/CZybnxRFNWUtbViQ+U+jU1o=;
 b=KJ3WsdoN2n2fLipF6wF8qJEj4G+jfR82z2Zf5RsyVrnB6vhdqXiRlzOlnCQcmo5oFZ
 j2v/X4x3Venyn/nfV7QoTO8nmtlTEEHwIOuezJxDxEDvY/a4CIW0ACGTkKIOe09eUI2g
 D9kIkPLivLeqQKoKfYZc0TFpjLqqvsaRMbD8TIztxdiTcB9oJ9NPc7z6J4WbOhRTxfWl
 4ToFnMzPX0vLqViF6w9JrWOvAXpH9zIs0/Z5gRu1vdh5HFAnCk0tdPdvWXCp06QyHkQa
 ZW244N0vnmp8M9PxPsoz0W8S6IacO3yS82VYOqGTwbB23zmtg2wCuF3AXmV57jyGl5wJ
 8QzA==
X-Gm-Message-State: AOAM531ygnM6WIZS7Z2O3kRYBi7mtNGRCIjgGB63VxqfsNQdTpXMchOG
 U7iDVkWMWHMbo7lBlduqbhi6/NkN3cw=
X-Google-Smtp-Source: ABdhPJyqbcT14E1S1SsGdjfOZya8JhfeBXkq4rBHSUaGdB+Ob/fDPByYkpxDCFR/yApduR3DU7+9dg==
X-Received: by 2002:a5d:6da4:: with SMTP id u4mr16291268wrs.50.1629087628279; 
 Sun, 15 Aug 2021 21:20:28 -0700 (PDT)
Received: from krug (a94-133-27-132.cpe.netcabo.pt. [94.133.27.132])
 by smtp.gmail.com with ESMTPSA id k13sm1619589wms.33.2021.08.15.21.20.27
 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
 Sun, 15 Aug 2021 21:20:27 -0700 (PDT)
From: =?utf-8?B?Sm/Do28gVMOhdm9yYQ==?= <joaotavora@HIDDEN>
To: Dmitry Gutov <dgutov@HIDDEN>
Subject: Re: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new
 `completion-filter-completions` API and deferred highlighting
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <f8c4419a-7e83-b8a9-347c-0221a2052376@HIDDEN>
 <fff6e291-bbcc-93e3-92e7-00af4664b8d4@HIDDEN>
 <3d3f894f-a6fa-53ae-5d50-c3aa9bffc73e@HIDDEN>
 <CALDnm53xh6boc6T9XSC4jZpkCPS3RB8JFNb12BpkoeGYg3Q9RQ@HIDDEN>
 <56ab18b1-4348-9b2c-85bb-af9b76cd429a@HIDDEN>
 <CALDnm52XM+S07OPXO3qtfLN4pup-kT2U=WRO9dUo3tv9uYJWXw@HIDDEN>
 <328f87eb-6474-1442-e1ca-9ae8deb2a84a@HIDDEN>
 <87sfzca0zm.fsf@HIDDEN>
 <4cac92d2-056d-e7fb-0664-2dbccb5f980c@HIDDEN>
Date: Mon, 16 Aug 2021 05:20:25 +0100
In-Reply-To: <4cac92d2-056d-e7fb-0664-2dbccb5f980c@HIDDEN> (Dmitry Gutov's
 message of "Mon, 16 Aug 2021 06:48:32 +0300")
Message-ID: <87eeauhvg6.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 47711
Cc: Daniel Mendler <mail@HIDDEN>,
 Stefan Monnier <monnier@HIDDEN>, 48841 <at> debbugs.gnu.org,
 47711 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Dmitry Gutov <dgutov@HIDDEN> writes:

> As I pointed out later in the email you're replying to, copying won't
> happen N times.

_Currently_, as in origin/master,  it happens N times.

In my patch, when a frontend adheres to the thing, it happens D times
where D is the amount of strings you need to display.  I guess if you do
the work to adapt a frontend to work with the new API proposed in
Daniel's patch it will be about the same (though likely in many more
lines of Elisp).

> First of all, note that scoring is only essential to the 'flex'
> style. Whereas the improvements we're discussing should benefit all,
> and can be more pronounced if the scoring don't need to be performed.

Yep, I agree.  But I don't see why the same principles I espouse --
which really amount to: let the style know it doesn't need to
face-propertize -- can't be applied to other styles that don't need
scoring.  Although those don't seem to suffer from any performance
problems, at least I haven't seens any complaints/reports/mesurements
like you did for bug#48841.

> But ok, let's talk about flex in particular.

Yes, I think that is important since it is the style known to be least
performant by its very lax nature.

> I'm guessing you just skip the C step in your benchmarks? Which is
> something that breaks our current contract.

Right.  Skipping the 'C =3D Copying step' is the whole point.  It breaks
our contract because the completion styles currently promise to
"face"-propertize the string.  So this is why I propose to let the
completion-style know that it doesn't need to.  When it is told of that,
it is relieved of the necessity of copying the string.  It is the
frontend that will do that copy just before face-propertizing and
displaying the string.  As you note, and reality shows, that's much
faster.  There is no disagreement here.

> Still, Daniel's patch should provide a comparable performance

Kinf of, from what I've read, it _should_ open the way for that to
happen.  From what I understand, you must then change the frontend (in a
big way?) to stop using completion-all-completions and start using the
new thing.  That work has not been done, as far as I know.  Whereas in
my proposal (which is now a patch posted to bug#48841) you change the
frontend in a very minor way, and that work _has_ been done.

Icomplete was very easy to adapt.  I can try adapting company soon.

In practice, we can't kill off completion-all-completions and start
everyone on completion-filter-completions (if that's what it's called).
So if the latter does turn out to be a step in the right direction (I'm
mostly waiting on Stefan to chime in), then I also don't see why we
couldn't have, as Eli suggested, both strategies for lazy highlighting
at some point in the future.

> improvement. If you're saying it doesn't give numbers as good, I'll
> have to give it a more thorough read and testing tomorrow to comment
> on that.

It's not me who is saying it, it's my Emacs :-) The real problem is that
with Daniel's patch, the frontends using the current API (as in
icomplete/fido) measurably become _slower_.  Though not by much (around
10%), it is still a shame.

Yes, do your testing and please, as always, try to report as
quantitatively as possible, so that we can really compare apples to
apples.

Jo=C3=A3o




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 16 Aug 2021 04:00:03 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Aug 16 00:00:03 2021
Received: from localhost ([127.0.0.1]:48125 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mFTn9-00024L-7y
	for submit <at> debbugs.gnu.org; Mon, 16 Aug 2021 00:00:03 -0400
Received: from mail-wr1-f43.google.com ([209.85.221.43]:34705)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <raaahh@HIDDEN>)
 id 1mFTn6-00022i-P2; Mon, 16 Aug 2021 00:00:01 -0400
Received: by mail-wr1-f43.google.com with SMTP id h13so21678016wrp.1;
 Sun, 15 Aug 2021 21:00:00 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=sender:subject:to:cc:references:from:message-id:date:user-agent
 :mime-version:in-reply-to:content-language:content-transfer-encoding;
 bh=d2GuwYH/OdC8HDu+lSnSMT9xRK5kCYBVEDUP4oi0MZM=;
 b=uTwWVwq4ZwGlWV/QY+WWaC8zZdUtdrnMW7LZNN7A6oxjyGTz6i/Mi/EJFAMjZ3sCQy
 Y9HMXUX7mZLng+6YhcypmAt+lbIxItyAaMXmYoKljh6dsRiTlM2vysk0k+L/Y57kFpk1
 3VjdJvGo6TMcE8thEvKr0vhx24C7noEeEs8oLH1JhqWeYqIVxNqoy/iB+eSc59RztqzW
 m0pVPXwYeD79jyHok3Q2RBM/n//yKAMO6R+sHG44XT/Qkx6cM1L83WObolhoM9sOv6Bm
 bd1CFviTzdi1VfS4ubKpBSRei8k5wkFGmL29QLVeH95u5tjWlI+mQfvoyMjCgoh2Tidi
 Jo7Q==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:sender:subject:to:cc:references:from:message-id
 :date:user-agent:mime-version:in-reply-to:content-language
 :content-transfer-encoding;
 bh=d2GuwYH/OdC8HDu+lSnSMT9xRK5kCYBVEDUP4oi0MZM=;
 b=ZDiKvHD2WjjKr4AZcSAZyvIo7a0u9Ch0cEXAwiiVytZ7zKJGg+GBiP8Lh5L0QHwzC7
 0PmDlmzMW7aK6w5jPH1eouJ64wdY9+Fosp6EbPbrvRnO1ZrTM9hRBsK/s8+WlM3ZiXK5
 tn36pr900UI7kMXzHZXAEdoQuoK67qkxBax55KUnS2giTp9ehM1P6QVC694rp5hblA9b
 Kjv3t1i4q02IKtp+llpF0I0qabBOpWSPRlaOtMxej7juE3laHMjbwcgxkHN+KzYw2HE4
 nLNwVGC1Tnf0C8H1lUPTl6SAiy9OE/PEJ8lzLXw/90ddK24kEOFHj762j/AOjbVbcCrm
 rsrw==
X-Gm-Message-State: AOAM530maPgAi6saT69UVtan49cOEQX7LAWWQvz4sXyemPkSEex/azla
 LHi5XuSMmnLA38IQtzsD1jDxor0Sny8=
X-Google-Smtp-Source: ABdhPJzngxCUGZ9JhdYE4wBXFsHbXyxA7ML1nlcykdFMllRAkl5xfvw4blwUoYUviuVfwu/2ARNv8g==
X-Received: by 2002:a5d:64ef:: with SMTP id g15mr9422616wri.135.1629086394957; 
 Sun, 15 Aug 2021 20:59:54 -0700 (PDT)
Received: from [192.168.0.6] ([46.251.119.176])
 by smtp.googlemail.com with ESMTPSA id f17sm10074999wrt.49.2021.08.15.20.59.53
 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);
 Sun, 15 Aug 2021 20:59:54 -0700 (PDT)
Subject: Re: bug#48841: bug#47711: [PATCH VERSION 2] Add new
 `completion-filter-completions` API and deferred highlighting
To: =?UTF-8?B?Sm/Do28gVMOhdm9yYQ==?= <joaotavora@HIDDEN>
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <fff6e291-bbcc-93e3-92e7-00af4664b8d4@HIDDEN>
 <3d3f894f-a6fa-53ae-5d50-c3aa9bffc73e@HIDDEN>
 <CALDnm53xh6boc6T9XSC4jZpkCPS3RB8JFNb12BpkoeGYg3Q9RQ@HIDDEN>
 <56ab18b1-4348-9b2c-85bb-af9b76cd429a@HIDDEN>
 <CALDnm52XM+S07OPXO3qtfLN4pup-kT2U=WRO9dUo3tv9uYJWXw@HIDDEN>
 <38a06f3c-4a7a-755c-c99b-708f91afabfa@HIDDEN>
 <CALDnm537=LHAoaGqPpy7=JOPsYn=z+=Q6QW=b9bjrAWOtxqRaQ@HIDDEN>
 <9f59f87c-2489-aaa0-5b3f-0e911b7ffa6c@HIDDEN>
 <CALDnm508EK5qM+AaOOJiNDm-=ORCuOieYpkjO=OSRas+_WTFwQ@HIDDEN>
 <8a36e61a-1c5b-bf3b-a454-077348589c4f@HIDDEN> <87y29471ov.fsf@HIDDEN>
 <837dgob6yo.fsf@HIDDEN> <87wnootec0.fsf@HIDDEN>
 <5d70b0ad-3838-ddb8-d341-9a5531d9da73@HIDDEN>
 <CALDnm51xYehwuGhMubovLTv_nqtdJYvOmH2e-bH3cCjf0PS2WQ@HIDDEN>
 <0cbf224b-b382-8a02-3f06-9f6d7e5e9741@HIDDEN>
 <CALDnm52x_S2UXE=YVokCWEZf9Gqc0O+g4ZVhi+R60SCeq6JtzA@HIDDEN>
From: Dmitry Gutov <dgutov@HIDDEN>
Message-ID: <eecaa682-496e-8173-caf7-2f9781cc7102@HIDDEN>
Date: Mon, 16 Aug 2021 06:59:52 +0300
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.11.0
MIME-Version: 1.0
In-Reply-To: <CALDnm52x_S2UXE=YVokCWEZf9Gqc0O+g4ZVhi+R60SCeq6JtzA@HIDDEN>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 8bit
X-Spam-Score: 0.4 (/)
X-Debbugs-Envelope-To: 47711
Cc: Daniel Mendler <mail@HIDDEN>, Lars Ingebrigtsen <larsi@HIDDEN>,
 Stefan Monnier <monnier@HIDDEN>, 48841 <at> debbugs.gnu.org,
 47711 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.6 (/)

On 16.08.2021 06:53, João Távora wrote:
> But why do it, since a string plist is a such a nice place to do these
> associations and there's -- apart from your aesthetics considerations
> -- 0 drawbacks identified?

You read all the explanations, and THAT'S your conclusion?




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 16 Aug 2021 03:54:29 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Aug 15 23:54:29 2021
Received: from localhost ([127.0.0.1]:48112 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mFThh-0001ua-Pb
	for submit <at> debbugs.gnu.org; Sun, 15 Aug 2021 23:54:29 -0400
Received: from mail-pj1-f48.google.com ([209.85.216.48]:53780)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <joaotavora@HIDDEN>)
 id 1mFThY-0001uB-Q7; Sun, 15 Aug 2021 23:54:20 -0400
Received: by mail-pj1-f48.google.com with SMTP id j1so24399797pjv.3;
 Sun, 15 Aug 2021 20:54:16 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=mime-version:references:in-reply-to:from:date:message-id:subject:to
 :cc:content-transfer-encoding;
 bh=5U8epICC1itkFK7Ot8G6B3eKqRd2LOVPQEmirX1WEQI=;
 b=jxdnZS1FpAM8YS/mMD7kzFj0mtHzMu3zXrle8ijqhVn4K8vxhSEBK8XBDTWW5Ulfal
 1TfJ53Kp3XMldOSbitiN1Wg8G2YOSOcGxcnRVWaerr3OoosNzJnnf955hL4RUyKe0qnc
 pocnoLbXK8bt8JipFWmDnGkiseoDYsLDDgzFgawd/myMJ61DeVRPE05EZi0OU8p6iZbp
 FaAlPsllupTojXEf9QoNZqfHyMs19zhExBmB5bo6KorUm1or2z4LOwsb8vL9+a/GZjvX
 T+6TWRp6Z3Nmys42lvybHuoLEeYKHCsl2lks6DL8f8IyHUzvyRYJ/1tHc019p1kIEHTx
 IpPA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:mime-version:references:in-reply-to:from:date
 :message-id:subject:to:cc:content-transfer-encoding;
 bh=5U8epICC1itkFK7Ot8G6B3eKqRd2LOVPQEmirX1WEQI=;
 b=bF6ZSo/YG7xb1vO4uR3HlVYbBRgLpSHjvkM7xk9QUc0jMxZQMnQnMFj9qE2bBFESSA
 cAnV7A+sAqZ/JvtFxM2wTF2RjMzDl3qBHPfbvSz2edHNBH6lkhmLd1+2CtxyeNRKOrl0
 Lx06gyypEbtaho7INdSwX+vf+iIY0OsagTRA3It5apmrOIECO8yk6bYTV7YlVApp8tb/
 N4cx8A6mP/FY6PEyT7nOmXjl8JOPLZljrdQeGXYrscZlyAReewtWyiDGMrhc7OHlcMzD
 A8UNBKTBhmaV3f0PD6n4Ro6q6frFnfysi/o52/pwbTAeQt71rk2AYkTwmMZqs+aE5biV
 sGDA==
X-Gm-Message-State: AOAM533jrgNLT+djLmOQNZLGETco2HSzVqy8msbSi8ntCgP5HyurBjIC
 5Hm2rPp0A5bBhc+WVdDjLsOIPrWHBeUKfGLdGtg=
X-Google-Smtp-Source: ABdhPJw/eF2aVy16C6VGDXYzzJxISRy2gtTQ/5B/MqJg9pUwG0KUwr2/9Dw/q+u73fbyzqNfFi+1CZaXBjlEq2mMRkA=
X-Received: by 2002:a17:902:d487:b0:12d:89d3:a6b with SMTP id
 c7-20020a170902d48700b0012d89d30a6bmr11825580plg.52.1629086050783; Sun, 15
 Aug 2021 20:54:10 -0700 (PDT)
MIME-Version: 1.0
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <f8c4419a-7e83-b8a9-347c-0221a2052376@HIDDEN>
 <fff6e291-bbcc-93e3-92e7-00af4664b8d4@HIDDEN>
 <3d3f894f-a6fa-53ae-5d50-c3aa9bffc73e@HIDDEN>
 <CALDnm53xh6boc6T9XSC4jZpkCPS3RB8JFNb12BpkoeGYg3Q9RQ@HIDDEN>
 <56ab18b1-4348-9b2c-85bb-af9b76cd429a@HIDDEN>
 <CALDnm52XM+S07OPXO3qtfLN4pup-kT2U=WRO9dUo3tv9uYJWXw@HIDDEN>
 <38a06f3c-4a7a-755c-c99b-708f91afabfa@HIDDEN>
 <CALDnm537=LHAoaGqPpy7=JOPsYn=z+=Q6QW=b9bjrAWOtxqRaQ@HIDDEN>
 <9f59f87c-2489-aaa0-5b3f-0e911b7ffa6c@HIDDEN>
 <CALDnm508EK5qM+AaOOJiNDm-=ORCuOieYpkjO=OSRas+_WTFwQ@HIDDEN>
 <8a36e61a-1c5b-bf3b-a454-077348589c4f@HIDDEN> <87y29471ov.fsf@HIDDEN>
 <837dgob6yo.fsf@HIDDEN> <87wnootec0.fsf@HIDDEN>
 <5d70b0ad-3838-ddb8-d341-9a5531d9da73@HIDDEN>
 <CALDnm51xYehwuGhMubovLTv_nqtdJYvOmH2e-bH3cCjf0PS2WQ@HIDDEN>
 <0cbf224b-b382-8a02-3f06-9f6d7e5e9741@HIDDEN>
In-Reply-To: <0cbf224b-b382-8a02-3f06-9f6d7e5e9741@HIDDEN>
From: =?UTF-8?B?Sm/Do28gVMOhdm9yYQ==?= <joaotavora@HIDDEN>
Date: Mon, 16 Aug 2021 04:53:59 +0100
Message-ID: <CALDnm52x_S2UXE=YVokCWEZf9Gqc0O+g4ZVhi+R60SCeq6JtzA@HIDDEN>
Subject: Re: bug#48841: bug#47711: [PATCH VERSION 2] Add new
 `completion-filter-completions` API and deferred highlighting
To: Dmitry Gutov <dgutov@HIDDEN>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 47711
Cc: Daniel Mendler <mail@HIDDEN>, Lars Ingebrigtsen <larsi@HIDDEN>,
 Stefan Monnier <monnier@HIDDEN>, 48841 <at> debbugs.gnu.org,
 47711 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

On Mon, Aug 16, 2021 at 4:31 AM Dmitry Gutov <dgutov@HIDDEN> wrote:
>
> On 16.08.2021 06:27, Jo=C3=A3o T=C3=A1vora wrote:
> > I wouldn't mind that at all.  For me, it's quite the same as evaluating
> > (symbol-plist 'car) and seeing (is-vehicle t number-of-wheels 4) along =
with
> > all the other byte-compilation stuff already there.
>
> Those serve a real purpose, not just work as an accidental cache for
> some earlier computation.

Caches also serve "a real purpose".  the gv-expander there
would be the "cache of an earlier computation". And I'm not
sure what "accidental" means, but if it means "implementation
detail for something I don't care about", I agree `completion-score`
is "accidental".  Should it be called
`completion-score-internal-cache-dont-look-here`?
Maybe.

Bottom line here is that an outside observer has no clue, and
shouldn't need or want to have a clue, on what "foreign" properties
attached to strings or symbols are meant.  This is why Eli says, and
I agree, that if the property isn't display related, it's all good.  No-one
but the setter and reader of that particular property mind.  The CL
systems I've worked with use package qualifiers to fine-grain this
even further, but they use the same principle. That Elisp allows a
string property list doesn't really make a difference IMO.

And none of this really really matters to the discussion.  If we absolutely
had to store these associations away from the string plist, (for
some aesthetic reason, I guess), we could just use hash-tables.

Then we could return the string unchanged (and uncopied) and largely
keep performance intact.
But why do it, since a string plist is a such a nice place to do these
associations and there's -- apart from your aesthetics considerations
-- 0 drawbacks identified?

Jo=C3=A3o T=C3=A1vora




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 16 Aug 2021 03:48:43 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Aug 15 23:48:43 2021
Received: from localhost ([127.0.0.1]:48103 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mFTcA-0001mC-LO
	for submit <at> debbugs.gnu.org; Sun, 15 Aug 2021 23:48:43 -0400
Received: from mail-wr1-f52.google.com ([209.85.221.52]:38864)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <raaahh@HIDDEN>)
 id 1mFTc8-0001lu-BW; Sun, 15 Aug 2021 23:48:41 -0400
Received: by mail-wr1-f52.google.com with SMTP id u16so4479636wrn.5;
 Sun, 15 Aug 2021 20:48:40 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=sender:subject:to:cc:references:from:message-id:date:user-agent
 :mime-version:in-reply-to:content-language:content-transfer-encoding;
 bh=4BWckHAxffwgEJeyO1tLFsBqmwUJUtAVqNgNJVrxWAE=;
 b=jsbczeL+xRdtbkDfx2OONBFi8nnmGKmvDJrYBWgftiiAkmrTxXcmzpyzxiPp7SceT6
 wKzyI5XCQzD9OulT8irby/BG6oQ6ssjdKHpwYxos9Jmk2U2iyePILe7uJwHS6sSEt8B5
 M4nu3n4vTUbP6Spq8eVsQogrMgHMIPKHKFKNbU7+0Txx+qrdWvL7Qfs+VGv/n5MrLiYc
 ViuvDD15+KDI8dbQP/vlEAHUhzyXNOwhAShVausuAEr5aNe9n9pKCQolkU5w2eF8x+B4
 UqlqzstE31+koKHRbQ8OSxTIQ/1E0uveoh6k1GuV5D3gRMURlZJw491L9dgjjuYRBu8L
 64IQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:sender:subject:to:cc:references:from:message-id
 :date:user-agent:mime-version:in-reply-to:content-language
 :content-transfer-encoding;
 bh=4BWckHAxffwgEJeyO1tLFsBqmwUJUtAVqNgNJVrxWAE=;
 b=BRWu+dSCBHDXsCg86UsWqpo58D4BHIpDSaBhlIy6nXcxxHcym44MlPOhwoySf9ZYXN
 upMEF92CA8jGM0cmGoC8xwvJ2K+dH+FH/U3bSEtiC+2fCbcYceg29yUITJAQ1lKEh9hg
 ejnCKHnn8Hzx4CZ2Z88YEhNP3GA98KKgyi9/XK450pX0MqEDAiULsa3n/RJzOqT5kFA1
 9xXgwSuXCd3vemTG/VNpRFk+Sxq/l4aSPlqpy5ldXXwy2VEkGPQKCOkT4UpJcmOxdGUh
 R8SgMKM3M4zRIQtykkIaX8ooabSSlRki//Kp2iMzr6HtX8OAXqdvjcyUCelJJKCNu7Bk
 JDkw==
X-Gm-Message-State: AOAM531dtKQD07GdL5LFqcyyYitjJvFBHEeZpBuBFbXgPBss6sAwoEe1
 cwf5JrEVS8srZF4DRR5h8BAecTWyzJk=
X-Google-Smtp-Source: ABdhPJww1P+mqruluBF5QDFEvKw6xiYCIYpTpexCH/wDiRNgTNbouR7y2gWmehFel0dbkwksXZvfVA==
X-Received: by 2002:a5d:6386:: with SMTP id p6mr15956716wru.143.1629085714223; 
 Sun, 15 Aug 2021 20:48:34 -0700 (PDT)
Received: from [192.168.0.6] ([46.251.119.176])
 by smtp.googlemail.com with ESMTPSA id l7sm9468851wmj.9.2021.08.15.20.48.33
 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);
 Sun, 15 Aug 2021 20:48:33 -0700 (PDT)
Subject: Re: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new
 `completion-filter-completions` API and deferred highlighting
To: =?UTF-8?B?Sm/Do28gVMOhdm9yYQ==?= <joaotavora@HIDDEN>
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <f8c4419a-7e83-b8a9-347c-0221a2052376@HIDDEN>
 <fff6e291-bbcc-93e3-92e7-00af4664b8d4@HIDDEN>
 <3d3f894f-a6fa-53ae-5d50-c3aa9bffc73e@HIDDEN>
 <CALDnm53xh6boc6T9XSC4jZpkCPS3RB8JFNb12BpkoeGYg3Q9RQ@HIDDEN>
 <56ab18b1-4348-9b2c-85bb-af9b76cd429a@HIDDEN>
 <CALDnm52XM+S07OPXO3qtfLN4pup-kT2U=WRO9dUo3tv9uYJWXw@HIDDEN>
 <328f87eb-6474-1442-e1ca-9ae8deb2a84a@HIDDEN> <87sfzca0zm.fsf@HIDDEN>
From: Dmitry Gutov <dgutov@HIDDEN>
Message-ID: <4cac92d2-056d-e7fb-0664-2dbccb5f980c@HIDDEN>
Date: Mon, 16 Aug 2021 06:48:32 +0300
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.11.0
MIME-Version: 1.0
In-Reply-To: <87sfzca0zm.fsf@HIDDEN>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 8bit
X-Spam-Score: 0.4 (/)
X-Debbugs-Envelope-To: 47711
Cc: Daniel Mendler <mail@HIDDEN>,
 Stefan Monnier <monnier@HIDDEN>, 48841 <at> debbugs.gnu.org,
 47711 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.6 (/)

On 14.08.2021 11:23, João Távora wrote:
> Dmitry Gutov <dgutov@HIDDEN> writes:
> 
>> Aside from the mutability/ownership issue,
>>
>> On 13.08.2021 15:05, João Távora wrote:
>>> If one removes these lines, the process becomes much faster, but there is a
>>> problem with highlighting.  My idea is indeed to defer highlighting by not
>>> setting the 'face property directly on that shared string, but some
>>> other property
>>> that is read later from the shared string by compliant frontents.
>>
>> I haven't done any direct benchmarking, but I'm pretty sure that this
>> approach cannot, by definition, be as fast as the non-mutating one.
>>
>> Because you go through the whole list and mutate all of its elements,
>> you have to perform a certain bit of work W x N times, where N is the
>> size of the whole list.
> 
> Let's call W the work that you perform N times in this istuation.  In
> the non-mutation, let's call it Z.  So
> 
> W <= Z, because Z not only propertizes the string with a calculation of
> faces but _also copies its character contents_.

As I pointed out later in the email you're replying to, copying won't 
happen N times.

> Also I think it's better to start about copying rather than mutating.
> As Eli points out, putting a text property in a string (my idea) is not
> equivalent with "mutating" it.

In common industry terms, that's mutation. Lisp strings are not C 
strings, they are aggregate objects.

>> Whereas the deferred-mutation approach will only have to do its bit
>> (which is likely more work, like, WWW) only 20 times, or 100 times, or
>> however many completions are displayed. And this is usually
>> negligible.
> 
> I think you're going in the same fallacy you went briefly in the other
> bug report.  The flex and pcm styles (meaning
> completion-pcm--hilit-commonality) has to go through all the completions
> when deciding the score to atribute to each completion that we already
> know matches the pattern.  That's because this scoring is essential to
> sorting.  That's a given in both scenarios, copying and non-copying.

First of all, note that scoring is only essential to the 'flex' style. 
Whereas the improvements we're discussing should benefit all, and can be 
more pronounced if the scoring don't need to be performed.

But ok, let's talk about flex in particular.

> Then, it's true that only a very small set of those eventually have to
> be displayed to the user, depending on where wants she wants her
> scrolling page to be.  So that's when you have to apply 'face' to, say
> 20 strings, and that can indeed be pretty fast.  But where does the
> information come from?
> 
> - Currently, it comes from the string's 'face' itself, which was copied
>    entirely.
> 
> - In the non-copying approach, it must come from somewhere else.  One
>    idea is that it comes from a new "private" property 'lazy-face', also
>    in the string itselv, but which was _not_ copied.  Another idea is
>    just to remember the pattern and re-match it to those 20 strings.

Let's say that the cost to compute the score (on one completion) is S. 
And the cost to highlight it is H. The cost to copy a string is C (that 
would be amortized cost, including the subsequent GCs).

The current algorithm costs: N x (C + S + H)

C is unavoidable because of the existing API guarantees.

A non-mutating algorithm can cost:

   N x S (for sorting)

   +

   100 x (C + S + H) (let's say we didn't even cache the scoring info)

...where 100 is the number of displayed completions (the number is 
usually lower).

As we measured previously, copying is quite expensive. Even in the 
above, not-caching approach we win ((N - 100) x (C + H)), and, okay, 
lose 100 x S. For high values of N, it should be a clear win.

> I think the second alternative is always faster.
> 
>> However big the difference is going to be, I can't say in advance, of
>> course, or whether it's going to be shadowed by some other performance
>> costs. But the non-mutating approach should have the best optimization
>> potential when the list is long.
> 
> Don't think so.  I'm doing benchmarks, will post soon.

I'm guessing you just skip the C step in your benchmarks? Which is 
something that breaks our current contract.

Still, Daniel's patch should provide a comparable performance 
improvement. If you're saying it doesn't give numbers as good, I'll have 
to give it a more thorough read and testing tomorrow to comment on that.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 16 Aug 2021 03:32:03 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Aug 15 23:32:03 2021
Received: from localhost ([127.0.0.1]:48086 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mFTM2-0000qv-UK
	for submit <at> debbugs.gnu.org; Sun, 15 Aug 2021 23:32:03 -0400
Received: from mail-wm1-f44.google.com ([209.85.128.44]:40778)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <raaahh@HIDDEN>)
 id 1mFTM1-0000m2-Lm; Sun, 15 Aug 2021 23:32:01 -0400
Received: by mail-wm1-f44.google.com with SMTP id
 f12-20020a05600c4e8c00b002e6bdd6ffe2so8144551wmq.5; 
 Sun, 15 Aug 2021 20:32:01 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=sender:subject:to:cc:references:from:message-id:date:user-agent
 :mime-version:in-reply-to:content-language:content-transfer-encoding;
 bh=ikPRGbaH3/T2gGcoVYmunjrKkikyMmJTvMUt584VY4I=;
 b=iIfQZmfyfMVyTpotHKPgmDG7rIQ2SMt6UBKrylUWrdOvlrS2Hc6rSZ9Hak4TgEOA4r
 jccMDFFPaj0mQFKQZ52r+NbjrBaDS20QdtX3vftpWxv6LjepuzuVSHw/lmh/ghAaxaSL
 M8p39tboRwQSoAVtVLGMc4buz2hwFZJu6VfkzPdXnseY+vG08LjlDZvZ7s4AlCbUhKTz
 V9cfn+WxodEhVPThycfwtdZpc28iBGas/iUf8jE4rTWNdOY/gaLFyOUXeKTH8aI36FhE
 +BH1jiSfAHIxUf3YFIRDFAy6Oq4ZJVluaEbqbExMqyHP3bpVJD4WOxgk1T6BUyEI3s2y
 /28w==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:sender:subject:to:cc:references:from:message-id
 :date:user-agent:mime-version:in-reply-to:content-language
 :content-transfer-encoding;
 bh=ikPRGbaH3/T2gGcoVYmunjrKkikyMmJTvMUt584VY4I=;
 b=BjIq7Y+OO4TeMJl9+/8lOZGK+r7tVpycAniHCtYVYDNo1ymBhgnUIMr3vRgDqAWozQ
 kgQxWRr82gEkMLo/U6BKz2lEoz6IPbOYSjjY31RTQDldq3X3/CKXnyU6XlmJM/tRjzOl
 o4KbF5xUg8lffGrSQxxZ0ef3F/1TPaXqdQccwKLNb95taAEEGKfkmD7XDFRBo8XlobZR
 SRMTvbQTJrnSyctsTjOf3zKx+n1lrsRf093Brdg86vbLnQue7npns7RvJFnryu/CKbci
 BGgmfh7d/cwqLDvlbQtgo314OwErguZQauipJRiYR8Jw80HY6+bq0QDexqDWMdSWHmyG
 NMiQ==
X-Gm-Message-State: AOAM531TCQ1swcCXqFGVAkRuN0uLTZAWDATKMYE7Ic8HJ7aIv0cvY5tq
 GtKNZX6XlD0HI+BjaBp27JBfd8Kzz5A=
X-Google-Smtp-Source: ABdhPJxugN+DM7cE2KdKXkpqgIStJhRdl+Gc8od4Jjtp1AYYyonSZT49wfzzPkDubtYeDStakZRJcA==
X-Received: by 2002:a1c:f002:: with SMTP id a2mr13478861wmb.79.1629084715969; 
 Sun, 15 Aug 2021 20:31:55 -0700 (PDT)
Received: from [192.168.0.6] ([46.251.119.176])
 by smtp.googlemail.com with ESMTPSA id w1sm8830714wmc.19.2021.08.15.20.31.54
 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);
 Sun, 15 Aug 2021 20:31:55 -0700 (PDT)
Subject: Re: bug#48841: bug#47711: [PATCH VERSION 2] Add new
 `completion-filter-completions` API and deferred highlighting
To: =?UTF-8?B?Sm/Do28gVMOhdm9yYQ==?= <joaotavora@HIDDEN>
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <f8c4419a-7e83-b8a9-347c-0221a2052376@HIDDEN>
 <fff6e291-bbcc-93e3-92e7-00af4664b8d4@HIDDEN>
 <3d3f894f-a6fa-53ae-5d50-c3aa9bffc73e@HIDDEN>
 <CALDnm53xh6boc6T9XSC4jZpkCPS3RB8JFNb12BpkoeGYg3Q9RQ@HIDDEN>
 <56ab18b1-4348-9b2c-85bb-af9b76cd429a@HIDDEN>
 <CALDnm52XM+S07OPXO3qtfLN4pup-kT2U=WRO9dUo3tv9uYJWXw@HIDDEN>
 <38a06f3c-4a7a-755c-c99b-708f91afabfa@HIDDEN>
 <CALDnm537=LHAoaGqPpy7=JOPsYn=z+=Q6QW=b9bjrAWOtxqRaQ@HIDDEN>
 <9f59f87c-2489-aaa0-5b3f-0e911b7ffa6c@HIDDEN>
 <CALDnm508EK5qM+AaOOJiNDm-=ORCuOieYpkjO=OSRas+_WTFwQ@HIDDEN>
 <8a36e61a-1c5b-bf3b-a454-077348589c4f@HIDDEN> <87y29471ov.fsf@HIDDEN>
 <837dgob6yo.fsf@HIDDEN> <87wnootec0.fsf@HIDDEN>
 <5d70b0ad-3838-ddb8-d341-9a5531d9da73@HIDDEN>
 <CALDnm51xYehwuGhMubovLTv_nqtdJYvOmH2e-bH3cCjf0PS2WQ@HIDDEN>
From: Dmitry Gutov <dgutov@HIDDEN>
Message-ID: <0cbf224b-b382-8a02-3f06-9f6d7e5e9741@HIDDEN>
Date: Mon, 16 Aug 2021 06:31:54 +0300
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.11.0
MIME-Version: 1.0
In-Reply-To: <CALDnm51xYehwuGhMubovLTv_nqtdJYvOmH2e-bH3cCjf0PS2WQ@HIDDEN>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 8bit
X-Spam-Score: 0.4 (/)
X-Debbugs-Envelope-To: 47711
Cc: Daniel Mendler <mail@HIDDEN>, Lars Ingebrigtsen <larsi@HIDDEN>,
 Stefan Monnier <monnier@HIDDEN>, 48841 <at> debbugs.gnu.org,
 47711 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.6 (/)

On 16.08.2021 06:27, João Távora wrote:
> I wouldn't mind that at all.  For me, it's quite the same as evaluating
> (symbol-plist 'car) and seeing (is-vehicle t number-of-wheels 4) along with
> all the other byte-compilation stuff already there.

Those serve a real purpose, not just work as an accidental cache for 
some earlier computation.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 16 Aug 2021 03:27:43 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Aug 15 23:27:43 2021
Received: from localhost ([127.0.0.1]:48078 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mFTHr-0007ZC-C7
	for submit <at> debbugs.gnu.org; Sun, 15 Aug 2021 23:27:43 -0400
Received: from mail-pj1-f46.google.com ([209.85.216.46]:46777)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <joaotavora@HIDDEN>)
 id 1mFTHp-0007Yt-65; Sun, 15 Aug 2021 23:27:41 -0400
Received: by mail-pj1-f46.google.com with SMTP id
 w13-20020a17090aea0db029017897a5f7bcso25217295pjy.5; 
 Sun, 15 Aug 2021 20:27:41 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=mime-version:references:in-reply-to:from:date:message-id:subject:to
 :cc:content-transfer-encoding;
 bh=THe/EBE3o/Xs+vLnOgxL99jiwYDsHvA1ibY5VWm72UY=;
 b=fzHWbgWng70CfutZoXzUKkyXJxUAsZSl+l+ZNERBDBX384D7ydjcYu60ae7YI+H+j3
 aFM+gZCJoUQsgHkyySd5hk2bov0kD8JmOG4L/7ipBw5thIExJNbaa1gpd+2/B65jgSj6
 YX2xUf+vcktPbLMlvOYSMqBOyaHmpjHIoWMxO75oiykqvd3c/rScuk0WyIqwEOpx7l0q
 Bygx1Wz3ymfMf13dPJubTVOH3PsbqPI+XB+4zakkg+gyQK5FvXOOCMGWEvAhhP7+p0HT
 YtsIKduwKwquuS4lHOLY/3XLBe04R90OOCaqlT+6s+BdoIZpsN9xz2rE9hilebURUcDD
 VqSg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:mime-version:references:in-reply-to:from:date
 :message-id:subject:to:cc:content-transfer-encoding;
 bh=THe/EBE3o/Xs+vLnOgxL99jiwYDsHvA1ibY5VWm72UY=;
 b=fsg/YDRktpY+ph5JCTW/Y5FFP1RMoUjBr7LeFcsX/u92vj3vvRUqoeQgsPjBPpgfJ0
 xd5lHBvFYi2LKuxjsYITMXOzr2CJHbMCBxS5vL+eBTQlR7zfQgqyelrKFNbc0O7IjP0o
 QCWsVOiJJxpM8Ylh4L87w4lPStGFF/MbIKl09+49PgCKeXaTOPjLh4utf1YlsnPF3oXK
 epvQ0ZEj+xLGfK4rBhSZVac4hW04fPZrKs11SCZKRxIY+VYrkDrHOhWcPeGVN3R8t/i6
 ASLNASSEntuO4PFpJoFU0WfnkDfu7I0ZxF/DfmOkdC+53dnHYKQ2Y62BkMo5nlGaT4J3
 vHyQ==
X-Gm-Message-State: AOAM533Sa3sPbdxdq7a9t4IR7WUifyVTApJ6PAekQ5usD7PZgG5mo6JD
 vXSXCeU9Tadh8JU9h7laqx9YsJoLrb1ac8HZ14w=
X-Google-Smtp-Source: ABdhPJwXRDNZqcDvyWUJ5GYKbhiWoBmi1myOucrCSy0TmEivEx6WflHuw7taqOWFHuBAAKTnEVndnpFTM+iiaRliSsA=
X-Received: by 2002:a17:90a:a091:: with SMTP id
 r17mr15138587pjp.56.1629084455343; 
 Sun, 15 Aug 2021 20:27:35 -0700 (PDT)
MIME-Version: 1.0
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <f8c4419a-7e83-b8a9-347c-0221a2052376@HIDDEN>
 <fff6e291-bbcc-93e3-92e7-00af4664b8d4@HIDDEN>
 <3d3f894f-a6fa-53ae-5d50-c3aa9bffc73e@HIDDEN>
 <CALDnm53xh6boc6T9XSC4jZpkCPS3RB8JFNb12BpkoeGYg3Q9RQ@HIDDEN>
 <56ab18b1-4348-9b2c-85bb-af9b76cd429a@HIDDEN>
 <CALDnm52XM+S07OPXO3qtfLN4pup-kT2U=WRO9dUo3tv9uYJWXw@HIDDEN>
 <38a06f3c-4a7a-755c-c99b-708f91afabfa@HIDDEN>
 <CALDnm537=LHAoaGqPpy7=JOPsYn=z+=Q6QW=b9bjrAWOtxqRaQ@HIDDEN>
 <9f59f87c-2489-aaa0-5b3f-0e911b7ffa6c@HIDDEN>
 <CALDnm508EK5qM+AaOOJiNDm-=ORCuOieYpkjO=OSRas+_WTFwQ@HIDDEN>
 <8a36e61a-1c5b-bf3b-a454-077348589c4f@HIDDEN> <87y29471ov.fsf@HIDDEN>
 <837dgob6yo.fsf@HIDDEN> <87wnootec0.fsf@HIDDEN>
 <5d70b0ad-3838-ddb8-d341-9a5531d9da73@HIDDEN>
In-Reply-To: <5d70b0ad-3838-ddb8-d341-9a5531d9da73@HIDDEN>
From: =?UTF-8?B?Sm/Do28gVMOhdm9yYQ==?= <joaotavora@HIDDEN>
Date: Mon, 16 Aug 2021 04:27:24 +0100
Message-ID: <CALDnm51xYehwuGhMubovLTv_nqtdJYvOmH2e-bH3cCjf0PS2WQ@HIDDEN>
Subject: Re: bug#48841: bug#47711: [PATCH VERSION 2] Add new
 `completion-filter-completions` API and deferred highlighting
To: Dmitry Gutov <dgutov@HIDDEN>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 47711
Cc: Daniel Mendler <mail@HIDDEN>,
 Stefan Monnier <monnier@HIDDEN>, 48841 <at> debbugs.gnu.org,
 Lars Ingebrigtsen <larsi@HIDDEN>, Eli Zaretskii <eliz@HIDDEN>,
 47711 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

On Mon, Aug 16, 2021 at 4:21 AM Dmitry Gutov <dgutov@HIDDEN> wrote:

> And even if the effects are usually not serious, are we really okay with
> evaluating (symbol-name 'car) someday and seeing lots of properties
> attached to it?

I wouldn't mind that at all.  For me, it's quite the same as evaluating
(symbol-plist 'car) and seeing (is-vehicle t number-of-wheels 4) along with
all the other byte-compilation stuff already there.

Jo=C3=A3o T=C3=A1vora




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 16 Aug 2021 03:27:08 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Aug 15 23:27:08 2021
Received: from localhost ([127.0.0.1]:48073 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mFTHH-0007YD-Td
	for submit <at> debbugs.gnu.org; Sun, 15 Aug 2021 23:27:08 -0400
Received: from mail-wr1-f46.google.com ([209.85.221.46]:38856)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <raaahh@HIDDEN>)
 id 1mFTHG-0007Xe-IW; Sun, 15 Aug 2021 23:27:07 -0400
Received: by mail-wr1-f46.google.com with SMTP id u16so4439971wrn.5;
 Sun, 15 Aug 2021 20:27:06 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=sender:subject:to:cc:references:from:message-id:date:user-agent
 :mime-version:in-reply-to:content-language:content-transfer-encoding;
 bh=64L5cJPDoPRrS5Chh6NOZfcomFk5jztUAvnroYkjfPw=;
 b=fjigDKyawAO1shNmVuscU1AZyAArtiQcs3PVAS0Mjv7CNg3gjLc3yHODK92qHMvI58
 JF3oBPpCnX+GMDfDUAJ1DPsPjJqwI51Iduo+Yb6n9WlxLr/5VuR0sWUK9tuHJKCpCjG1
 Q6Ja3KSzknM09G57DfuwxJkapSfRgSrDnAV3VxQqn52bM8JLIfXliv4nvnecRO6myTKP
 SSKGYJ1gisG+ItQzxULAPnPQNo3KqZByKmpbeo/Y7qZvhSNnFhnDq60ErRdRXX6YLpc6
 GD01nHMdFi2KzvRIFrtD5BqJUXBRSAFzLlfwnwLZbjfKkjMkoVV5RfcWOUJhwqhbVezt
 v6iA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:sender:subject:to:cc:references:from:message-id
 :date:user-agent:mime-version:in-reply-to:content-language
 :content-transfer-encoding;
 bh=64L5cJPDoPRrS5Chh6NOZfcomFk5jztUAvnroYkjfPw=;
 b=lfY0aJqX17pgr/KrwQCcFcMWfz3tabLDvv3zzbArJInSdEmf0bKTz2JudhHfKWME7W
 OGGtlaqnkk9/a5l3c1brL0jnduDuyG1Qshl5KSszNTPciar7XoXqItV7eD9e9xKGyqw+
 YJYT8y/42KDgX5iJpmkdyrxbnuKVwKOwps3Y5K6mX2T6kPdAa5vg1dAx6d7SiXZp+dh4
 w73aFMSrb3RsmJTQW2wVEccBkomnU733IgkCTNKQjAnM4ZjT+cfrkK4rMlRhlTrhcZbd
 CzThntPdqSBA5lRLnKKT0JXpYw682Lp7Bq7ZEel+N8/T4uf8hMPa2njfbU9mj3cN8SWv
 HxBQ==
X-Gm-Message-State: AOAM531cbP5wQeWDiV2WLKFlqzSkeVTfh3xzMFnqJIM7nkmq5oVfVCPm
 5AJpsOgZaQgtEHMk+UMw/pV+MP+gk3I=
X-Google-Smtp-Source: ABdhPJyVliP1KOFR+r10ppj6J661h+p0H9YHW7C15qmTn8R30bpVXr/AWYkT/eZbqJK87JFYDpFk0w==
X-Received: by 2002:a5d:42c9:: with SMTP id t9mr15847021wrr.356.1629084420779; 
 Sun, 15 Aug 2021 20:27:00 -0700 (PDT)
Received: from [192.168.0.6] ([46.251.119.176])
 by smtp.googlemail.com with ESMTPSA id n10sm4316110wms.3.2021.08.15.20.26.59
 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);
 Sun, 15 Aug 2021 20:27:00 -0700 (PDT)
Subject: Re: bug#47711: [PATCH VERSION 2] Add new
 `completion-filter-completions` API and deferred highlighting
To: Eli Zaretskii <eliz@HIDDEN>, Daniel Mendler <mail@HIDDEN>
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <f8c4419a-7e83-b8a9-347c-0221a2052376@HIDDEN>
 <fff6e291-bbcc-93e3-92e7-00af4664b8d4@HIDDEN>
 <3d3f894f-a6fa-53ae-5d50-c3aa9bffc73e@HIDDEN>
 <CALDnm53xh6boc6T9XSC4jZpkCPS3RB8JFNb12BpkoeGYg3Q9RQ@HIDDEN>
 <56ab18b1-4348-9b2c-85bb-af9b76cd429a@HIDDEN>
 <CALDnm52XM+S07OPXO3qtfLN4pup-kT2U=WRO9dUo3tv9uYJWXw@HIDDEN>
 <38a06f3c-4a7a-755c-c99b-708f91afabfa@HIDDEN>
 <CALDnm537=LHAoaGqPpy7=JOPsYn=z+=Q6QW=b9bjrAWOtxqRaQ@HIDDEN>
 <9f59f87c-2489-aaa0-5b3f-0e911b7ffa6c@HIDDEN>
 <83im08bjc3.fsf@HIDDEN>
From: Dmitry Gutov <dgutov@HIDDEN>
Message-ID: <42a56dff-197d-d223-a2d8-12db8577b505@HIDDEN>
Date: Mon, 16 Aug 2021 06:26:58 +0300
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.11.0
MIME-Version: 1.0
In-Reply-To: <83im08bjc3.fsf@HIDDEN>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
X-Spam-Score: 0.4 (/)
X-Debbugs-Envelope-To: 47711
Cc: joaotavora@HIDDEN, monnier@HIDDEN, 48841 <at> debbugs.gnu.org,
 47711 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.6 (/)

On 14.08.2021 10:01, Eli Zaretskii wrote:

> Just to make sure we are on the same page: adding a text property to a
> string doesn't mutate a string.  Lisp programs that process these
> strings will not necessarily see any difference, and displaying those
> strings will also not show any difference if the property is not
> related to display.  So the assumption that seems to be made here,
> that adding a property is the same as mutating a string, is IMO
> inaccurate if not incorrect.

This is nonsense.

A program won't necessarily see a difference in *any* changed value, as 
long as some part of it stays the same.

I can zero out the tail of a string, and have a program that only looks 
at its first few characters. It wouldn't mean that a string hasn't changed.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 16 Aug 2021 03:21:27 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Aug 15 23:21:27 2021
Received: from localhost ([127.0.0.1]:48063 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mFTBm-0007Pt-Na
	for submit <at> debbugs.gnu.org; Sun, 15 Aug 2021 23:21:26 -0400
Received: from mail-wr1-f51.google.com ([209.85.221.51]:38463)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <raaahh@HIDDEN>)
 id 1mFTBj-0007Pc-Us; Sun, 15 Aug 2021 23:21:24 -0400
Received: by mail-wr1-f51.google.com with SMTP id u16so4428362wrn.5;
 Sun, 15 Aug 2021 20:21:23 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=sender:subject:to:cc:references:from:message-id:date:user-agent
 :mime-version:in-reply-to:content-language:content-transfer-encoding;
 bh=/SDdnV/qJpRL1tdLDLIG8DzV4voetJU/pAmP2OsCwmQ=;
 b=INK893s60seq21i3Dj5Kh+rfJDiOwPaCCk5Nt1TrGFQjLttNjPcyQKwFkedSi0/JMg
 ZFiWptBDGU+IByd4h1QasfSz6f+mYeFU5EKKnQ0Vxz7cVhQtV7BwN8G0MnhSecJkpmK+
 8lPEBsaRhE+PXj5zlPjj8IHHCsrliEQ97zZeNBZGCopMFfsHf199L0HVcNMu96mb0zKb
 VV6Cs/FYyM7+XhvSCqY4IQk4u76cYXN1KTPCJqQhD+Y+jXneHJJW2gaLCW8udzkqmtZI
 nhioPrrSSlcWpO2NqKVgHY/OKuvEFJKVuWa+ClvlGjfCMLTfMdcyXmVVbpv4zXWIOj6I
 +Pnw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:sender:subject:to:cc:references:from:message-id
 :date:user-agent:mime-version:in-reply-to:content-language
 :content-transfer-encoding;
 bh=/SDdnV/qJpRL1tdLDLIG8DzV4voetJU/pAmP2OsCwmQ=;
 b=FvfOUWi83UiAk17HICrLJ0e/yOyEEklfNwN7ztrDR5znOUlKO3x01nD5u71kiwdlbl
 i5MpjUF3sBkS5b8IWyQTBbtR55G1spk7YBJi6rHpw3aNfddt37SNsKAQK1sd5O6ctee6
 Eyb2E3eid0Dzj3bvKmZmoo9rJMIUohkGcxsEpuxzt8s4xmE9ybUJ9idnStg6GdufdpSL
 IgTgZ9bbCwLEvFGztA81nKsNwbhU5hhmbcPPx6JAz68/CfPRHhiT4XkvpcgIKlmqbqaR
 1n+PVo+cDGgggqsagvTjvVPwonRbMjtmKoDQXMbJFnc3A3bO1PdOpU9dD6Z8paVPXjEo
 6WVw==
X-Gm-Message-State: AOAM530/V20OZR4i60nwdh+3tIhEHQ57/yHb2+gudBKxnAOSpJ6a2xK6
 eV8Ug9IJNx0PQpJAWONdxffnRsVDli8=
X-Google-Smtp-Source: ABdhPJyYmnt+dnaCSXEoR1/rysVo1gOgb6lV5DwDleVLpb2yhlFNUeLcnSxtUgUrJ9JW8l/NArVtyA==
X-Received: by 2002:adf:e9c3:: with SMTP id l3mr16098689wrn.300.1629084078386; 
 Sun, 15 Aug 2021 20:21:18 -0700 (PDT)
Received: from [192.168.0.6] ([46.251.119.176])
 by smtp.googlemail.com with ESMTPSA id t14sm9177633wmj.2.2021.08.15.20.21.17
 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);
 Sun, 15 Aug 2021 20:21:18 -0700 (PDT)
Subject: Re: bug#48841: bug#47711: [PATCH VERSION 2] Add new
 `completion-filter-completions` API and deferred highlighting
To: Lars Ingebrigtsen <larsi@HIDDEN>, Eli Zaretskii <eliz@HIDDEN>
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <f8c4419a-7e83-b8a9-347c-0221a2052376@HIDDEN>
 <fff6e291-bbcc-93e3-92e7-00af4664b8d4@HIDDEN>
 <3d3f894f-a6fa-53ae-5d50-c3aa9bffc73e@HIDDEN>
 <CALDnm53xh6boc6T9XSC4jZpkCPS3RB8JFNb12BpkoeGYg3Q9RQ@HIDDEN>
 <56ab18b1-4348-9b2c-85bb-af9b76cd429a@HIDDEN>
 <CALDnm52XM+S07OPXO3qtfLN4pup-kT2U=WRO9dUo3tv9uYJWXw@HIDDEN>
 <38a06f3c-4a7a-755c-c99b-708f91afabfa@HIDDEN>
 <CALDnm537=LHAoaGqPpy7=JOPsYn=z+=Q6QW=b9bjrAWOtxqRaQ@HIDDEN>
 <9f59f87c-2489-aaa0-5b3f-0e911b7ffa6c@HIDDEN>
 <CALDnm508EK5qM+AaOOJiNDm-=ORCuOieYpkjO=OSRas+_WTFwQ@HIDDEN>
 <8a36e61a-1c5b-bf3b-a454-077348589c4f@HIDDEN> <87y29471ov.fsf@HIDDEN>
 <837dgob6yo.fsf@HIDDEN> <87wnootec0.fsf@HIDDEN>
From: Dmitry Gutov <dgutov@HIDDEN>
Message-ID: <5d70b0ad-3838-ddb8-d341-9a5531d9da73@HIDDEN>
Date: Mon, 16 Aug 2021 06:21:16 +0300
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.11.0
MIME-Version: 1.0
In-Reply-To: <87wnootec0.fsf@HIDDEN>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
X-Spam-Score: 0.4 (/)
X-Debbugs-Envelope-To: 47711
Cc: mail@HIDDEN, 47711 <at> debbugs.gnu.org, monnier@HIDDEN,
 48841 <at> debbugs.gnu.org, =?UTF-8?B?Sm/Do28gVMOhdm9yYQ==?= <joaotavora@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.6 (/)

On 14.08.2021 15:12, Lars Ingebrigtsen wrote:
> It is a destructive change, but we may just declare that completion
> functions are allowed to destructively change the inputs in certain very
> prescribed ways.  I'd rather avoid that, though, if at all possible,
> because it may lead to subtle bugs all over the place.

That would be a breaking change, but it's a possibility, of course.

If we couldn't find a better way to implement this.

> Stefan just reminded me (in a different bug report) that we've long
> meant to extend the text property machinery with a "namespace" or
> "plane" concept.  The impetus for this is really the font locking
> machinery which wants to manage some text properties that other packages
> also want to manage.

"Planes" for text properties are just prefixed properties, I guess? 
That's different from the situation with font-lock.

> The idea is that the display machinery would combine all the planes
> before displaying, but each package would just manage its own "plane".
> If we had something like this, then using this mechanism in the
> completion context would make sense -- we could then say that completion
> isn't allowed to alter anything except text properties in its private
> plane.

Yes, if the code makes sure to only use prefixed properties, that would 
limit the damage. It could still affect repeated (parallel?) uses of the 
same values in the same piece of code.

And even if the effects are usually not serious, are we really okay with 
evaluating (symbol-name 'car) someday and seeing lots of properties 
attached to it?




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 16 Aug 2021 03:17:43 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Aug 15 23:17:43 2021
Received: from localhost ([127.0.0.1]:48050 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mFT8A-0007IV-Ov
	for submit <at> debbugs.gnu.org; Sun, 15 Aug 2021 23:17:42 -0400
Received: from mail-wm1-f49.google.com ([209.85.128.49]:37399)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <raaahh@HIDDEN>)
 id 1mFT88-0007IE-6k; Sun, 15 Aug 2021 23:17:41 -0400
Received: by mail-wm1-f49.google.com with SMTP id
 c8-20020a7bc008000000b002e6e462e95fso1142688wmb.2; 
 Sun, 15 Aug 2021 20:17:40 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=sender:subject:to:cc:references:from:message-id:date:user-agent
 :mime-version:in-reply-to:content-language:content-transfer-encoding;
 bh=wrMjrV8qQgdb2hCWp4i6jVb0kXGOCWq9X7R/gxFJh3Y=;
 b=VQC7CqZ+QlIdgYzmvEB0JUixt++PP0zheGv4CEg4hvEsfzG9p5Y4uQILjNneEIrOx8
 ExwG3fJCrZRmAyALE57A3jadmlCTJAvYf9Bg4+Hc8jYHRYRx0ZuKuO8MEmlpZ1sx9O0f
 hiotEmN03kV+SeOq3AEEs7o4h/r7cLhHjBBQRDmwPxUchUghIFWRTivbsJdNrkQ+rVas
 WedqUQP8bixNccyZr/s3K9/ED1PurhYinE0NHv1/ZKAIEeNFGip/dqbVem+1DCnIfayH
 kxv2+ilYSV2k2aaGbs3G8pFHfmggYnG256fqqpjEaOI6n5tDiFq2oAI9ZGvhaJ9L0nAd
 ppTA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:sender:subject:to:cc:references:from:message-id
 :date:user-agent:mime-version:in-reply-to:content-language
 :content-transfer-encoding;
 bh=wrMjrV8qQgdb2hCWp4i6jVb0kXGOCWq9X7R/gxFJh3Y=;
 b=gseS7LRraWC/6zNmf/ty8ve+frcEEkECEnOuzrarIgH0IXUa3hPxBozRbyanPziqJ+
 2lBk60tBlbr2WzCyEFO8/tzqYpwNPhmyO+aSPK9axpPYTbfewp+0qX+7EClGk/2TxxZv
 Ve7SZyhHt+qstnWO/SfoVgwVAcCOJg46beyI8/4wde5b4OF5rzaP2r0Ddy8CoBxT5iD1
 fPDKmqymXCyej4En8g3otnseZvVV2tKsa3s/t7FJXcx+/Sb3/wbpAxExdrPm5wIkhxtG
 WkvuWp73jpoWeAn4P1XpUj2aJTNN58A0R8NwFAADAPGS3xPHMv3idLWboyfcC55YlOx1
 rNkw==
X-Gm-Message-State: AOAM531VIB3hR06EuH2+ua0+S8m4Gvrbl63wFDlOUCjtE3LE0dTNjXUA
 jtIh3MQJxtHMRFXf6tuy3aZuOxe4Peg=
X-Google-Smtp-Source: ABdhPJxXaa7rFIciOBkeaJ/YuFNR6EPpQeJNc0JVHlj3aG5kcC4JpzQNtsbWqxWHLS47LPB219CJBQ==
X-Received: by 2002:a05:600c:a08:: with SMTP id
 z8mr13420412wmp.165.1629083854298; 
 Sun, 15 Aug 2021 20:17:34 -0700 (PDT)
Received: from [192.168.0.6] ([46.251.119.176])
 by smtp.googlemail.com with ESMTPSA id b20sm9724439wmj.20.2021.08.15.20.17.33
 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);
 Sun, 15 Aug 2021 20:17:33 -0700 (PDT)
Subject: Re: bug#48841: bug#47711: [PATCH VERSION 2] Add new
 `completion-filter-completions` API and deferred highlighting
To: Eli Zaretskii <eliz@HIDDEN>, =?UTF-8?B?Sm/Do28gVMOhdm9yYQ==?=
 <joaotavora@HIDDEN>
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <f8c4419a-7e83-b8a9-347c-0221a2052376@HIDDEN>
 <fff6e291-bbcc-93e3-92e7-00af4664b8d4@HIDDEN>
 <3d3f894f-a6fa-53ae-5d50-c3aa9bffc73e@HIDDEN>
 <CALDnm53xh6boc6T9XSC4jZpkCPS3RB8JFNb12BpkoeGYg3Q9RQ@HIDDEN>
 <56ab18b1-4348-9b2c-85bb-af9b76cd429a@HIDDEN>
 <CALDnm52XM+S07OPXO3qtfLN4pup-kT2U=WRO9dUo3tv9uYJWXw@HIDDEN>
 <38a06f3c-4a7a-755c-c99b-708f91afabfa@HIDDEN>
 <CALDnm537=LHAoaGqPpy7=JOPsYn=z+=Q6QW=b9bjrAWOtxqRaQ@HIDDEN>
 <9f59f87c-2489-aaa0-5b3f-0e911b7ffa6c@HIDDEN>
 <CALDnm508EK5qM+AaOOJiNDm-=ORCuOieYpkjO=OSRas+_WTFwQ@HIDDEN>
 <8a36e61a-1c5b-bf3b-a454-077348589c4f@HIDDEN> <87y29471ov.fsf@HIDDEN>
 <837dgob6yo.fsf@HIDDEN>
From: Dmitry Gutov <dgutov@HIDDEN>
Message-ID: <31da8f43-02f7-e6dd-ab38-2160af138a4a@HIDDEN>
Date: Mon, 16 Aug 2021 06:17:32 +0300
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.11.0
MIME-Version: 1.0
In-Reply-To: <837dgob6yo.fsf@HIDDEN>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
X-Spam-Score: 0.4 (/)
X-Debbugs-Envelope-To: 47711
Cc: mail@HIDDEN, monnier@HIDDEN, 48841 <at> debbugs.gnu.org,
 47711 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.6 (/)

On 14.08.2021 14:29, Eli Zaretskii wrote:
> Text properties are stored separately from the string, so I don't
> think adding properties can in general be referred to as "change".

Are you thinking of C strings?

Lisp strings carry text properties in addition to the array of 
characters. It doesn't really matter where in the memory the properties 
and the characters reside.

> Whether in some particular situation that could count as a "change"
> depends on that situation and on the particular property, of course.

I was talking in the general sense: modifying a value.

One can talk about whether a certain modification matters in certain 
situations, but that's not the way to discount a general principle.

> I'm not sure in the context of completion there's any reason to count
> as "change" adding properties that don't affect display.

For the context in question, whether the properties affect display is 
not particularly important. Properties affecting display just make it 
easier to notice that something's wrong. Bug involving other properties 
should be more difficult to investigate.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 15 Aug 2021 00:03:29 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Aug 14 20:03:29 2021
Received: from localhost ([127.0.0.1]:45532 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mF3cf-0006KK-5O
	for submit <at> debbugs.gnu.org; Sat, 14 Aug 2021 20:03:29 -0400
Received: from mail-wr1-f41.google.com ([209.85.221.41]:35501)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <joaotavora@HIDDEN>)
 id 1mF3cd-0006K3-D0; Sat, 14 Aug 2021 20:03:27 -0400
Received: by mail-wr1-f41.google.com with SMTP id q10so18386833wro.2;
 Sat, 14 Aug 2021 17:03:27 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=from:to:cc:subject:references:date:in-reply-to:message-id
 :user-agent:mime-version:content-transfer-encoding;
 bh=qklDJ5gPDaNiJGUeObEycQTArsmErnSMftxyioCEZWM=;
 b=PkFu5fYdl2NYm+8n3UE2+ICJducbUpqKse/nlXI8HXlbJGdqcCRbcquZlGnn6X5QcT
 e7XtwTnz1Sq1PASmhDX01c7fHkacEXNzA0HFMdRp8ZxoJetF/K392SngIsF+bJeP6r6m
 1xR8DQbrgYmBKYfgWiao1ra/iJYqjd4ah4SD8+0S4u7nXOuMiDEjrzzFY5/Tbv0AI6Ez
 xXkD8Nr/JPcWKTOq9XvBepPEts5F6Qm1j0wTkfpamcUX5IPZBjoPumdwtw9YoEoRtae9
 oN2MIGqCxj5PYezGZJQvKeFXxcron1zG+pGHh+CcTcQmOQqmW4ABGxeSfg4YKm3tT6PS
 EE0A==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to
 :message-id:user-agent:mime-version:content-transfer-encoding;
 bh=qklDJ5gPDaNiJGUeObEycQTArsmErnSMftxyioCEZWM=;
 b=VXdJWx8IWM31cFiVoetwVhjgcgy5w+4V3qiX0sPb7iHvAgDorxQPewjVrF006QoCMl
 11pLQq0thVvSLgSKVPIEeTFKjV8D9SxsR15fOrFBNlNpffhEimRe0BjshZPTEgh5tnqa
 oEW7elD1CeUGjmlnc3JW8f/CnmqeMqBC956RKI3SDZcsVKrlplbgIjLPgkmF0cjdhBTZ
 pIPEy7IZc79CvRSpc5E1NjryF+SUirnesXy/Jq9rT2xFAhLuKraJFK/lOkQ0tKIQr+Rj
 TAGcq2w6qs4EL6HaW189LRZxc5PDxLGBQdRQF92/6rta6BPFbObeFGGvgA9sZ+lZYeXb
 OqdA==
X-Gm-Message-State: AOAM531YwnxOY6+u54svB3iSQvSON3xlihLMnL+lRdrT2vEKLKKWS5hB
 RcKqe5mqGBrck55u9wXvMt0=
X-Google-Smtp-Source: ABdhPJyUCZaxxxOBi9K8i35OHATog1u0JejE8y4B2qRfotNjDgBltcumqscOgQDXq2Tgy/t0KO4vRQ==
X-Received: by 2002:a5d:5703:: with SMTP id a3mr10510017wrv.333.1628985801441; 
 Sat, 14 Aug 2021 17:03:21 -0700 (PDT)
Received: from krug (a94-133-27-132.cpe.netcabo.pt. [94.133.27.132])
 by smtp.gmail.com with ESMTPSA id s10sm7598232wrv.54.2021.08.14.17.03.19
 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
 Sat, 14 Aug 2021 17:03:20 -0700 (PDT)
From: =?utf-8?B?Sm/Do28gVMOhdm9yYQ==?= <joaotavora@HIDDEN>
To: Eli Zaretskii <eliz@HIDDEN>
Subject: Re: bug#47711: [PATCH VERSION 2] Add new
 `completion-filter-completions` API and deferred highlighting
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <f8c4419a-7e83-b8a9-347c-0221a2052376@HIDDEN>
 <fff6e291-bbcc-93e3-92e7-00af4664b8d4@HIDDEN>
 <3d3f894f-a6fa-53ae-5d50-c3aa9bffc73e@HIDDEN>
 <CALDnm53xh6boc6T9XSC4jZpkCPS3RB8JFNb12BpkoeGYg3Q9RQ@HIDDEN>
 <56ab18b1-4348-9b2c-85bb-af9b76cd429a@HIDDEN>
 <CALDnm52XM+S07OPXO3qtfLN4pup-kT2U=WRO9dUo3tv9uYJWXw@HIDDEN>
 <38a06f3c-4a7a-755c-c99b-708f91afabfa@HIDDEN>
 <CALDnm537=LHAoaGqPpy7=JOPsYn=z+=Q6QW=b9bjrAWOtxqRaQ@HIDDEN>
 <9f59f87c-2489-aaa0-5b3f-0e911b7ffa6c@HIDDEN>
 <83im08bjc3.fsf@HIDDEN> <877dgo8ihf.fsf@HIDDEN>
Date: Sun, 15 Aug 2021 01:03:18 +0100
In-Reply-To: <877dgo8ihf.fsf@HIDDEN> (=?utf-8?Q?=22Jo=C3=A3o_T=C3=A1vor?=
 =?utf-8?Q?a=22's?= message of "Sat, 14 Aug 2021 10:48:28 +0100")
Message-ID: <87tujr7ewp.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 47711
Cc: Daniel Mendler <mail@HIDDEN>, dgutov@HIDDEN,
 monnier@HIDDEN, 48841 <at> debbugs.gnu.org, 47711 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Jo=C3=A3o T=C3=A1vora <joaotavora@HIDDEN> writes:

> in the absence of any controlled benchmarks I did some of
> my own, using the most controlled environment I could devise.  I start
> Emacs like so:
>
>    ~/Source/Emacs/emacs/src/emacs -Q -f fido-mode -f fido-vertical-mode -=
l ~/tmp/benchmark.el ~/tmp/benchmark.el

I have know tweaked the benchmark slightly to make it easier to evaluate
speed qualitatively.  Here's what I've been using.

   (require 'cl-lib)
=20=20=20=20
   (fido-mode 1)
   (fido-vertical-mode 1)
=20=20=20=20
   ;; Introduce 150 000 new functions to really slow things down.
   ;; Probably more than most non-Spacemancs people will have :-)
   (defmacro lotsoffunctions ()
     `(progn
        ,@(cl-loop repeat 150000
                  collect `(defun ,(intern (symbol-name (gensym "heyhey")))=
 () 42))))
=20=20=20=20
   (lotsoffunctions)
=20=20=20=20
   (when nil
     ;; Press C-u C-x C-e C-m quickly to produce a quantitative sample
     (benchmark-run (completing-read "" obarray))
=20=20=20=20
     ;; Or just press C-h f to experience how fast/slow completion is.
     )

The results are the same as the ones I reported in the previous email.

I've also cleaned up my previous patch of the
scratch/icomplete-lazy-highlight-attempt-2 branch slightly.  It is now
fully opt-in for frontends and completion-styles, so the backward
compatibility problems which I speculated seem to have been exaggerated.

I'm still studying it for flaws, but anyone can have a look.  And, of
course, there are many different ways to realize the "opt-in" for
frontends/styles.  I just chose the one that seemed the simplest given
the current completion framework.

The performance is still very good, it reduces the usual waiting time in
long lists of completions to about half of what it currently is.

Jo=C3=A3o




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 14 Aug 2021 13:29:19 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Aug 14 09:29:19 2021
Received: from localhost ([127.0.0.1]:43840 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mEtiw-0007Hv-Mq
	for submit <at> debbugs.gnu.org; Sat, 14 Aug 2021 09:29:18 -0400
Received: from quimby.gnus.org ([95.216.78.240]:33192)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <larsi@HIDDEN>)
 id 1mEtiu-0007He-5k; Sat, 14 Aug 2021 09:29:16 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnus.org;
 s=20200322; h=Content-Type:MIME-Version:Message-ID:In-Reply-To:Date:
 References:Subject:Cc:To:From:Sender:Reply-To:Content-Transfer-Encoding:
 Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender:
 Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:
 List-Subscribe:List-Post:List-Owner:List-Archive;
 bh=xRxGA7F4YEqXq+8FnWlqiVa/Azmuq2pSfPDNleoGrRI=; b=kMAw8LpeatKL2Th0QxC+E7FaCO
 ODH/dK4iF4cu9p9iDHlsF/HGLhfoQ7xmvEcPFkvE/O8AsGxteVztzDc5kf2k3iR0dwIW2DE0wqqjz
 6HGmoD4+4s42x8R5pqGcD5nG8VevB6IWk+Rgag96VKrs6YS5v4iiskLIDe/YFTPZds/s=;
Received: from [84.212.220.105] (helo=elva)
 by quimby.gnus.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.92) (envelope-from <larsi@HIDDEN>)
 id 1mEtii-0001Or-36; Sat, 14 Aug 2021 15:29:08 +0200
From: Lars Ingebrigtsen <larsi@HIDDEN>
To: Eli Zaretskii <eliz@HIDDEN>
Subject: Re: bug#48841: bug#47711: [PATCH VERSION 2] Add new
 `completion-filter-completions` API and deferred highlighting
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <f8c4419a-7e83-b8a9-347c-0221a2052376@HIDDEN>
 <fff6e291-bbcc-93e3-92e7-00af4664b8d4@HIDDEN>
 <3d3f894f-a6fa-53ae-5d50-c3aa9bffc73e@HIDDEN>
 <CALDnm53xh6boc6T9XSC4jZpkCPS3RB8JFNb12BpkoeGYg3Q9RQ@HIDDEN>
 <56ab18b1-4348-9b2c-85bb-af9b76cd429a@HIDDEN>
 <CALDnm52XM+S07OPXO3qtfLN4pup-kT2U=WRO9dUo3tv9uYJWXw@HIDDEN>
 <38a06f3c-4a7a-755c-c99b-708f91afabfa@HIDDEN>
 <CALDnm537=LHAoaGqPpy7=JOPsYn=z+=Q6QW=b9bjrAWOtxqRaQ@HIDDEN>
 <9f59f87c-2489-aaa0-5b3f-0e911b7ffa6c@HIDDEN>
 <CALDnm508EK5qM+AaOOJiNDm-=ORCuOieYpkjO=OSRas+_WTFwQ@HIDDEN>
 <8a36e61a-1c5b-bf3b-a454-077348589c4f@HIDDEN>
 <87y29471ov.fsf@HIDDEN> <837dgob6yo.fsf@HIDDEN>
 <87wnootec0.fsf@HIDDEN> <8335rcb3oo.fsf@HIDDEN>
Date: Sat, 14 Aug 2021 15:29:03 +0200
In-Reply-To: <8335rcb3oo.fsf@HIDDEN> (Eli Zaretskii's message of "Sat, 14 Aug
 2021 15:39:51 +0300")
Message-ID: <87a6lktasg.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Report: Spam detection software, running on the system "quimby.gnus.org",
 has NOT identified this incoming email as spam.  The original
 message has been attached to this so you can view it or label
 similar future email.  If you have any questions, see
 @@CONTACT_ADDRESS@@ for details.
 Content preview: Eli Zaretskii <eliz@HIDDEN> writes: > How can that work if
 at display time all the "planes" are combined? > It would mean that the code
 which produced the original strings will > get them displayed differently
 after completion. 
 Content analysis details:   (-2.9 points, 5.0 required)
 pts rule name              description
 ---- ---------------------- --------------------------------------------------
 -1.0 ALL_TRUSTED            Passed through trusted hosts only via SMTP
 -1.9 BAYES_00               BODY: Bayes spam probability is 0 to 1%
 [score: 0.0000]
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 47711
Cc: mail@HIDDEN, joaotavora@HIDDEN, 48841 <at> debbugs.gnu.org,
 dgutov@HIDDEN, monnier@HIDDEN, 47711 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Eli Zaretskii <eliz@HIDDEN> writes:

> How can that work if at display time all the "planes" are combined?
> It would mean that the code which produced the original strings will
> get them displayed differently after completion.

That's in the font-lock context, where font-lock will do faces on its
"plane" while other packages can do faces on their own "planes", and
they'll be combined on display.

It not relevant in the context of this bug report, but I thought I'd
just mention the general design.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 14 Aug 2021 12:40:18 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Aug 14 08:40:17 2021
Received: from localhost ([127.0.0.1]:43798 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mEsxR-00069g-HA
	for submit <at> debbugs.gnu.org; Sat, 14 Aug 2021 08:40:17 -0400
Received: from eggs.gnu.org ([209.51.188.92]:41018)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>)
 id 1mEsxM-00069G-3k; Sat, 14 Aug 2021 08:40:12 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:49494)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <eliz@HIDDEN>)
 id 1mEsxE-00057C-Rp; Sat, 14 Aug 2021 08:40:00 -0400
Received: from 84.94.185.95.cable.012.net.il ([84.94.185.95]:3166
 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1mEsxE-0002mD-Bm; Sat, 14 Aug 2021 08:40:00 -0400
Date: Sat, 14 Aug 2021 15:39:51 +0300
Message-Id: <8335rcb3oo.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Lars Ingebrigtsen <larsi@HIDDEN>
In-Reply-To: <87wnootec0.fsf@HIDDEN> (message from Lars Ingebrigtsen on Sat, 
 14 Aug 2021 14:12:31 +0200)
Subject: Re: bug#48841: bug#47711: [PATCH VERSION 2] Add new
 `completion-filter-completions` API and deferred highlighting
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <f8c4419a-7e83-b8a9-347c-0221a2052376@HIDDEN>
 <fff6e291-bbcc-93e3-92e7-00af4664b8d4@HIDDEN>
 <3d3f894f-a6fa-53ae-5d50-c3aa9bffc73e@HIDDEN>
 <CALDnm53xh6boc6T9XSC4jZpkCPS3RB8JFNb12BpkoeGYg3Q9RQ@HIDDEN>
 <56ab18b1-4348-9b2c-85bb-af9b76cd429a@HIDDEN>
 <CALDnm52XM+S07OPXO3qtfLN4pup-kT2U=WRO9dUo3tv9uYJWXw@HIDDEN>
 <38a06f3c-4a7a-755c-c99b-708f91afabfa@HIDDEN>
 <CALDnm537=LHAoaGqPpy7=JOPsYn=z+=Q6QW=b9bjrAWOtxqRaQ@HIDDEN>
 <9f59f87c-2489-aaa0-5b3f-0e911b7ffa6c@HIDDEN>
 <CALDnm508EK5qM+AaOOJiNDm-=ORCuOieYpkjO=OSRas+_WTFwQ@HIDDEN>
 <8a36e61a-1c5b-bf3b-a454-077348589c4f@HIDDEN>
 <87y29471ov.fsf@HIDDEN> <837dgob6yo.fsf@HIDDEN>
 <87wnootec0.fsf@HIDDEN>
MIME-version: 1.0
Content-type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: 8bit
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 47711
Cc: mail@HIDDEN, joaotavora@HIDDEN, 48841 <at> debbugs.gnu.org,
 dgutov@HIDDEN, monnier@HIDDEN, 47711 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: Lars Ingebrigtsen <larsi@HIDDEN>
> Cc: Joo Tvora <joaotavora@HIDDEN>,
>   mail@HIDDEN,
>   dgutov@HIDDEN,  monnier@HIDDEN,  48841 <at> debbugs.gnu.org,
>   47711 <at> debbugs.gnu.org
> Date: Sat, 14 Aug 2021 14:12:31 +0200
> 
> Stefan just reminded me (in a different bug report) that we've long
> meant to extend the text property machinery with a "namespace" or
> "plane" concept.  The impetus for this is really the font locking
> machinery which wants to manage some text properties that other packages
> also want to manage.
> 
> The idea is that the display machinery would combine all the planes
> before displaying, but each package would just manage its own "plane".
> If we had something like this, then using this mechanism in the
> completion context would make sense -- we could then say that completion
> isn't allowed to alter anything except text properties in its private
> plane.

How can that work if at display time all the "planes" are combined?
It would mean that the code which produced the original strings will
get them displayed differently after completion.

Anyway, I'm not sure why you are talking about display here: the
properties which are supposed to store the information about
completion aren't supposed to affect display.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 14 Aug 2021 12:12:46 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Aug 14 08:12:46 2021
Received: from localhost ([127.0.0.1]:43757 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mEsWs-000168-1M
	for submit <at> debbugs.gnu.org; Sat, 14 Aug 2021 08:12:46 -0400
Received: from quimby.gnus.org ([95.216.78.240]:60890)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <larsi@HIDDEN>)
 id 1mEsWq-00015s-7p; Sat, 14 Aug 2021 08:12:45 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnus.org;
 s=20200322; h=Content-Type:MIME-Version:Message-ID:In-Reply-To:Date:
 References:Subject:Cc:To:From:Sender:Reply-To:Content-Transfer-Encoding:
 Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender:
 Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:
 List-Subscribe:List-Post:List-Owner:List-Archive;
 bh=0UbbhNCk79EjpLVHOt6hx3/gHrhJ7w9EVHreQXcmOmM=; b=BH2dmC+23gOI7SFNcwIj5Z8c5/
 wkI9ex0yyU6NSnPqlJd/i4K71qa6frp2RdleRovoYKHAODzMfMra3TYC+G4KDjb3mCLqXK4KKq5N6
 RvhQ13CyKxpF1ei1JreFEIIqef0neg38gPSm2QJhfi1oxK9FLpg/sRlvWMjdaLuAR32w=;
Received: from [84.212.220.105] (helo=elva)
 by quimby.gnus.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.92) (envelope-from <larsi@HIDDEN>)
 id 1mEsWe-0000TL-42; Sat, 14 Aug 2021 14:12:36 +0200
From: Lars Ingebrigtsen <larsi@HIDDEN>
To: Eli Zaretskii <eliz@HIDDEN>
Subject: Re: bug#48841: bug#47711: [PATCH VERSION 2] Add new
 `completion-filter-completions` API and deferred highlighting
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <f8c4419a-7e83-b8a9-347c-0221a2052376@HIDDEN>
 <fff6e291-bbcc-93e3-92e7-00af4664b8d4@HIDDEN>
 <3d3f894f-a6fa-53ae-5d50-c3aa9bffc73e@HIDDEN>
 <CALDnm53xh6boc6T9XSC4jZpkCPS3RB8JFNb12BpkoeGYg3Q9RQ@HIDDEN>
 <56ab18b1-4348-9b2c-85bb-af9b76cd429a@HIDDEN>
 <CALDnm52XM+S07OPXO3qtfLN4pup-kT2U=WRO9dUo3tv9uYJWXw@HIDDEN>
 <38a06f3c-4a7a-755c-c99b-708f91afabfa@HIDDEN>
 <CALDnm537=LHAoaGqPpy7=JOPsYn=z+=Q6QW=b9bjrAWOtxqRaQ@HIDDEN>
 <9f59f87c-2489-aaa0-5b3f-0e911b7ffa6c@HIDDEN>
 <CALDnm508EK5qM+AaOOJiNDm-=ORCuOieYpkjO=OSRas+_WTFwQ@HIDDEN>
 <8a36e61a-1c5b-bf3b-a454-077348589c4f@HIDDEN>
 <87y29471ov.fsf@HIDDEN> <837dgob6yo.fsf@HIDDEN>
Date: Sat, 14 Aug 2021 14:12:31 +0200
In-Reply-To: <837dgob6yo.fsf@HIDDEN> (Eli Zaretskii's message of "Sat, 14 Aug
 2021 14:29:03 +0300")
Message-ID: <87wnootec0.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Report: Spam detection software, running on the system "quimby.gnus.org",
 has NOT identified this incoming email as spam.  The original
 message has been attached to this so you can view it or label
 similar future email.  If you have any questions, see
 @@CONTACT_ADDRESS@@ for details.
 Content preview:  Eli Zaretskii <eliz@HIDDEN> writes: > Text properties are
 stored separately from the string, so I don't > think adding properties can
 in general be referred to as "change". > > Whether in some particular
 situation that could count as a [...] 
 Content analysis details:   (-2.9 points, 5.0 required)
 pts rule name              description
 ---- ---------------------- --------------------------------------------------
 -1.0 ALL_TRUSTED            Passed through trusted hosts only via SMTP
 -1.9 BAYES_00               BODY: Bayes spam probability is 0 to 1%
 [score: 0.0000]
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 47711
Cc: mail@HIDDEN,
 =?utf-8?B?Sm/Do28gVMOhdm9yYQ==?= <joaotavora@HIDDEN>, 48841 <at> debbugs.gnu.org,
 dgutov@HIDDEN, monnier@HIDDEN, 47711 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Eli Zaretskii <eliz@HIDDEN> writes:

> Text properties are stored separately from the string, so I don't
> think adding properties can in general be referred to as "change".
>
> Whether in some particular situation that could count as a "change"
> depends on that situation and on the particular property, of course.
> I'm not sure in the context of completion there's any reason to count
> as "change" adding properties that don't affect display.

It is a destructive change, but we may just declare that completion
functions are allowed to destructively change the inputs in certain very
prescribed ways.  I'd rather avoid that, though, if at all possible,
because it may lead to subtle bugs all over the place.

Stefan just reminded me (in a different bug report) that we've long
meant to extend the text property machinery with a "namespace" or
"plane" concept.  The impetus for this is really the font locking
machinery which wants to manage some text properties that other packages
also want to manage.

The idea is that the display machinery would combine all the planes
before displaying, but each package would just manage its own "plane".
If we had something like this, then using this mechanism in the
completion context would make sense -- we could then say that completion
isn't allowed to alter anything except text properties in its private
plane.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 14 Aug 2021 11:29:40 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Aug 14 07:29:40 2021
Received: from localhost ([127.0.0.1]:43683 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mErr5-0006JF-Qu
	for submit <at> debbugs.gnu.org; Sat, 14 Aug 2021 07:29:40 -0400
Received: from eggs.gnu.org ([209.51.188.92]:60850)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>)
 id 1mErr0-0006Iv-6Q; Sat, 14 Aug 2021 07:29:34 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:47960)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <eliz@HIDDEN>)
 id 1mErqu-0006q2-Cr; Sat, 14 Aug 2021 07:29:24 -0400
Received: from 84.94.185.95.cable.012.net.il ([84.94.185.95]:2620
 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1mErqt-0007lR-5U; Sat, 14 Aug 2021 07:29:24 -0400
Date: Sat, 14 Aug 2021 14:29:03 +0300
Message-Id: <837dgob6yo.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: =?utf-8?B?Sm/Do28gVMOhdm9yYQ==?= <joaotavora@HIDDEN>
In-Reply-To: <87y29471ov.fsf@HIDDEN> (message from =?utf-8?B?Sm/Do28g?=
 =?utf-8?B?VMOhdm9yYQ==?= on Sat, 14 Aug 2021 11:36:32 +0100)
Subject: Re: bug#48841: bug#47711: [PATCH VERSION 2] Add new
 `completion-filter-completions` API and deferred highlighting
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <f8c4419a-7e83-b8a9-347c-0221a2052376@HIDDEN>
 <fff6e291-bbcc-93e3-92e7-00af4664b8d4@HIDDEN>
 <3d3f894f-a6fa-53ae-5d50-c3aa9bffc73e@HIDDEN>
 <CALDnm53xh6boc6T9XSC4jZpkCPS3RB8JFNb12BpkoeGYg3Q9RQ@HIDDEN>
 <56ab18b1-4348-9b2c-85bb-af9b76cd429a@HIDDEN>
 <CALDnm52XM+S07OPXO3qtfLN4pup-kT2U=WRO9dUo3tv9uYJWXw@HIDDEN>
 <38a06f3c-4a7a-755c-c99b-708f91afabfa@HIDDEN>
 <CALDnm537=LHAoaGqPpy7=JOPsYn=z+=Q6QW=b9bjrAWOtxqRaQ@HIDDEN>
 <9f59f87c-2489-aaa0-5b3f-0e911b7ffa6c@HIDDEN>
 <CALDnm508EK5qM+AaOOJiNDm-=ORCuOieYpkjO=OSRas+_WTFwQ@HIDDEN>
 <8a36e61a-1c5b-bf3b-a454-077348589c4f@HIDDEN> <87y29471ov.fsf@HIDDEN>
MIME-version: 1.0
Content-type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 47711
Cc: mail@HIDDEN, 47711 <at> debbugs.gnu.org, monnier@HIDDEN,
 48841 <at> debbugs.gnu.org, dgutov@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: João Távora <joaotavora@HIDDEN>
> Date: Sat, 14 Aug 2021 11:36:32 +0100
> Cc: Daniel Mendler <mail@HIDDEN>,
>  Stefan Monnier <monnier@HIDDEN>, 48841 <at> debbugs.gnu.org,
>  47711 <at> debbugs.gnu.org
> 
> > And in the example above, the values are those that the
> > lispref/objects.texi says we should not change (though it gives
> > (symbol-name 'cons) as example). "Not mutable", in its parlance. IIRC
> > the related discussions mentioned that modifying such values could
> > lead to a segfault in some previous Emacs versions. Maybe not anymore,
> > but it's still not a good idea.
> 
> You're extrapolating "change" to also include attaching properties to
> symbols.  I think that document just means that you can't do stuff like
> 
>     (aset "cons" 0 ?z)
> 
> or
> 
>     (aset (symbol-name 'cons) 0 ?z)
> 
> I don't think it means you can't
> 
>     (put-text-property 0 1 'joaot/muahahah 42 (symbol-name 'cons))
> 
> But maybe Eli or someone else more knowledgeable in the deep internals
> of Emacs can correct me.

Text properties are stored separately from the string, so I don't
think adding properties can in general be referred to as "change".

Whether in some particular situation that could count as a "change"
depends on that situation and on the particular property, of course.
I'm not sure in the context of completion there's any reason to count
as "change" adding properties that don't affect display.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 14 Aug 2021 11:22:16 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Aug 14 07:22:16 2021
Received: from localhost ([127.0.0.1]:43675 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mErk0-000684-AV
	for submit <at> debbugs.gnu.org; Sat, 14 Aug 2021 07:22:16 -0400
Received: from mail-wr1-f49.google.com ([209.85.221.49]:45664)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <raaahh@HIDDEN>)
 id 1mErjw-00067h-K4; Sat, 14 Aug 2021 07:22:14 -0400
Received: by mail-wr1-f49.google.com with SMTP id v4so9665078wro.12;
 Sat, 14 Aug 2021 04:22:12 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=sender:subject:to:cc:references:from:message-id:date:user-agent
 :mime-version:in-reply-to:content-language:content-transfer-encoding;
 bh=2lzMRiTN9L+ibgaMoQVjLU1cDSXr8p2TDREAUlYpyHw=;
 b=KtOGOSadTAivI+sMaG6Z6n5tKwV9ww1gyYmNIGTGVZyB97ya1xbDJRnwcp0kzKRZEY
 PoNmwXjiN84dxzXXdFsLHZu1dY+dFGi8R+G7VsmF9uu75doIV470wK+vvrNkIGqq5oF7
 5lXdH2Zh87DLnGcJWdKBHcoSyjd4NDQ54LKV96L/lCRfqUtrWjC2IEgBG8CnZBX/LVse
 Ge/J716dgLZRPAGc9w0Pg4X5tYrIVs7if5uWoY04HN0OQ42Hq3L9HIAtXaoT1kwoF8Vq
 ZfznUT8rQWZZEA50PfQhafBRFekQ+zBB+4N3WW2a5D48rjEUVEaxUnb1zKX8Qtxk+LVd
 YErw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:sender:subject:to:cc:references:from:message-id
 :date:user-agent:mime-version:in-reply-to:content-language
 :content-transfer-encoding;
 bh=2lzMRiTN9L+ibgaMoQVjLU1cDSXr8p2TDREAUlYpyHw=;
 b=dfPwo/eCR6EhPrIzmHuvvv2z/+3RiB3BL5GJU7psxKEWglwZY+MelPg/7InHoF1C2F
 XQTY8yYAyKsBBxRcSSpb0tEgGiwLMg1AVnu28OXDWCDqiYs2wg4IH+7Q1wlDqo3wR1ZM
 PoLwudx744Lbh1EScmPHWOc1C5nOVA7lBhOm4qB6qYyNOJnLntHO1TBqA0trcAvCHvG9
 vG9ECTnNVytY7e2ukBDICfLhCe4fjeIto10DmM442xQrlqfi1RTqtiBN3R8tlL3lcpsm
 AWAha/yueYlnE43yJ8NQgrX0I3BvoLnsU1bN3/vhJUATlXq1WXdcmTJ89gojqf/frn4n
 lL2A==
X-Gm-Message-State: AOAM533B540ekMmbfMcfwwVHVzpDJYcrGbRmNq8CI3zwJAy4vletyysy
 goddsQyHDv4WAT5dFtbWduf10t6/YhI=
X-Google-Smtp-Source: ABdhPJzjf7FPFGHs13PIjXxd+flbf6DnhuFohEbMnGizSM9H9gQDNOAyBHclQjs/QeqAc28qCLDrEA==
X-Received: by 2002:a05:6000:18a4:: with SMTP id
 b4mr8133637wri.162.1628940126633; 
 Sat, 14 Aug 2021 04:22:06 -0700 (PDT)
Received: from [192.168.0.6] ([46.251.119.176])
 by smtp.googlemail.com with ESMTPSA id q75sm4370937wme.40.2021.08.14.04.22.05
 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);
 Sat, 14 Aug 2021 04:22:05 -0700 (PDT)
Subject: Re: bug#48841: bug#47711: [PATCH VERSION 2] Add new
 `completion-filter-completions` API and deferred highlighting
To: Eli Zaretskii <eliz@HIDDEN>
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <f8c4419a-7e83-b8a9-347c-0221a2052376@HIDDEN>
 <fff6e291-bbcc-93e3-92e7-00af4664b8d4@HIDDEN>
 <3d3f894f-a6fa-53ae-5d50-c3aa9bffc73e@HIDDEN>
 <CALDnm53xh6boc6T9XSC4jZpkCPS3RB8JFNb12BpkoeGYg3Q9RQ@HIDDEN>
 <56ab18b1-4348-9b2c-85bb-af9b76cd429a@HIDDEN>
 <CALDnm52XM+S07OPXO3qtfLN4pup-kT2U=WRO9dUo3tv9uYJWXw@HIDDEN>
 <38a06f3c-4a7a-755c-c99b-708f91afabfa@HIDDEN>
 <CALDnm537=LHAoaGqPpy7=JOPsYn=z+=Q6QW=b9bjrAWOtxqRaQ@HIDDEN>
 <9f59f87c-2489-aaa0-5b3f-0e911b7ffa6c@HIDDEN>
 <CALDnm508EK5qM+AaOOJiNDm-=ORCuOieYpkjO=OSRas+_WTFwQ@HIDDEN>
 <8a36e61a-1c5b-bf3b-a454-077348589c4f@HIDDEN> <83h7fsbitl.fsf@HIDDEN>
From: Dmitry Gutov <dgutov@HIDDEN>
Message-ID: <1982cc2e-f955-7f15-0781-65cdbbe96759@HIDDEN>
Date: Sat, 14 Aug 2021 14:22:04 +0300
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.11.0
MIME-Version: 1.0
In-Reply-To: <83h7fsbitl.fsf@HIDDEN>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
X-Spam-Score: 0.4 (/)
X-Debbugs-Envelope-To: 47711
Cc: mail@HIDDEN, 47711 <at> debbugs.gnu.org, joaotavora@HIDDEN,
 48841 <at> debbugs.gnu.org, monnier@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.6 (/)

On 14.08.2021 10:12, Eli Zaretskii wrote:
>> From: Dmitry Gutov <dgutov@HIDDEN>
>> Date: Sat, 14 Aug 2021 05:47:43 +0300
>> Cc: Stefan Monnier <monnier@HIDDEN>, 48841 <at> debbugs.gnu.org,
>>   47711 <at> debbugs.gnu.org
>>
>> I thought I explained the problem with this previously.
>>
>> It's basically this: we cannot mutate what we don't own. Across all of
>> completion functions out there, there will be such that return "shared"
>> strings (meaning, not copied or newly allocated) from their completion
>> tables. And modifying them is bad, with consequences which can present
>> themselves in unexpected, often subtle ways.
>>
>> Since up until now completion-pcm--hilit-commonality copied all strings
>> before modifying, completion tables such as described (with "shared"
>> strings) have all been "legal". Suddenly deciding to stop supporting
>> them would be a major API breakage with consequences that are hard to
>> predict. And while I perhaps agree that it's an inconvenience, I don't
>> think it's a choice we can simply make as this stage in c-a-p-f's
>> development.
> 
> This sounds like an argument against Daniel's approach as well, no?
> Because if a completion API returns strings it "doesn't own", there
> will be restrictions on Lisp programs that use those strings, because
> those Lisp programs previously could do anything they liked with those
> strings, and now they cannot.  Or am I missing something?

Good question. It is not.

The completion tables described above have all been doing "legal" 
things, in our general understanding.

But any callers of completion-all-completions were never really allowed 
to modify the returned strings (those still were strings that code 
"doesn't own").

Of course, some of those callers (I don't know any, though) might have 
taken advantage of being able to modify the strings with impunity 
because of completion-all-completions' implementation detail, but 
they'll have a chance to clean up their act when switching to 
completion-filter-completions.

>>     1. (setq s (symbol-name 'car))
>>
>>     2. (put-text-property 1 3 'face 'error s)
>>
>>     3. Switch to a buffer in fundamental mode
>>
>>     4. (insert (symbol-name 'car)) --> see the error face in the buffer
>>
>> Now imagine that some completion table collects symbol names by passing
>> obarray through #'symbol-name rather than #'all-completions, and voila,
>> if the completion machinery adds properties (any properties, not just
>> face) to those strings, you have just modified a bunch of global values.
>> That's not good.
> 
> How is this different from Daniel's proposal of returning the original
> strings?  AFAIU, he just shifts the responsibility from the completion
> code to the caller of the completion code, but basically leaves the
> problem still very much real and pretty much into our face.

This is a shift of responsibility in the right direction. The callers 
might as well do the string copying when needed, but the fact of the 
matter is, they usually only need to "copy" 20-100 strings (or however 
many is displayed), if they need to modify them at all. That's where we 
win: copying 100 strings is better than copying 10000.

Gotta run now, will reply to other email later.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 14 Aug 2021 10:36:53 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Aug 14 06:36:53 2021
Received: from localhost ([127.0.0.1]:43633 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mEr21-0002iS-B9
	for submit <at> debbugs.gnu.org; Sat, 14 Aug 2021 06:36:53 -0400
Received: from mail-wr1-f54.google.com ([209.85.221.54]:37387)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <joaotavora@HIDDEN>)
 id 1mEr1s-0002i1-Db; Sat, 14 Aug 2021 06:36:44 -0400
Received: by mail-wr1-f54.google.com with SMTP id r6so16717117wrt.4;
 Sat, 14 Aug 2021 03:36:40 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=from:to:cc:subject:references:date:in-reply-to:message-id
 :user-agent:mime-version:content-transfer-encoding;
 bh=Bh27k0LEsx5DFXwN7gWYs7DxXUNYaLIjbu6ommRbxA8=;
 b=kY3CHHhQPgxYfZeYOt4QG8fG3N2dFqnffJS3kMYkIjvl+wEvD+BS/sHE2NuChbqzAa
 D4HElyd6MqwzNTwVJaeT6FQ6LKwoqIKm4g0yKJ3oQpeBsg4se6oPjNTzKkumbkDBYGpr
 pjpnT5CF3bNOU14bbK7Z2PbQs0inEKQOge+OABAkZsSrm69W2Hp4yc5MSP9N6BQzM9Ay
 LK/z2biTj28/xjdOe/Q7EGHgiTIdxCks/Rm5TSYR+fIYJAtdABmAaXaKpNb0Qh/aDkrt
 YhWQHObfly+w0+D5TeJFYHrObATriUCqu6NsJcCTmE+SqBx+ywu14VGjwCJE0W4iVReh
 6oKA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to
 :message-id:user-agent:mime-version:content-transfer-encoding;
 bh=Bh27k0LEsx5DFXwN7gWYs7DxXUNYaLIjbu6ommRbxA8=;
 b=GFi1kooyF9K6kD5WOhruUZCsgahBUr/OtCP0+7FSU4jMRgCUT6On70fIkPBpX0QiVd
 DSQF9O01zrwPhfVtlaTtoNwZIdsEYL8sSU48VjrsGEuylvv0ScV16TI78DjjF6Gl4MPW
 7uc0ZaMetXeX0R9jmqJZS/KK1VfAE6oeFERbBdiSB12UiNLyK6uPZORmgaZI0h3okVnV
 ExRZsX7kjA47zhr9wEv9fN2+wJiXGarW4fHTn6qL74nzAB44Cfi85e89eUF1La2eEqjq
 Y5VoXqiAZfNBlt1o+FIzmxdM7uAGXnUBVrUqOXYHwfeGAsfJrj4SXO9x9g7JgotTECeH
 UlHQ==
X-Gm-Message-State: AOAM531Hp91nzm9zqA0llTP2Yx2Ujt2//uZ1QKC4J1UNmCeNa/NwU7Lk
 bhax5eEIMhKH6PrY6j4mit5GwNMgdfM=
X-Google-Smtp-Source: ABdhPJzhS09Z/f61UZIaCPeue+OeZ21p0Mzyvuu4983LqVr6GkAUSlSIt2ZLLV39lmhe7drkDbsGBg==
X-Received: by 2002:adf:e107:: with SMTP id t7mr7755483wrz.165.1628937394463; 
 Sat, 14 Aug 2021 03:36:34 -0700 (PDT)
Received: from krug (a94-133-27-132.cpe.netcabo.pt. [94.133.27.132])
 by smtp.gmail.com with ESMTPSA id z137sm4350807wmc.14.2021.08.14.03.36.33
 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
 Sat, 14 Aug 2021 03:36:33 -0700 (PDT)
From: =?utf-8?B?Sm/Do28gVMOhdm9yYQ==?= <joaotavora@HIDDEN>
To: Dmitry Gutov <dgutov@HIDDEN>
Subject: Re: bug#47711: [PATCH VERSION 2] Add new
 `completion-filter-completions` API and deferred highlighting
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <f8c4419a-7e83-b8a9-347c-0221a2052376@HIDDEN>
 <fff6e291-bbcc-93e3-92e7-00af4664b8d4@HIDDEN>
 <3d3f894f-a6fa-53ae-5d50-c3aa9bffc73e@HIDDEN>
 <CALDnm53xh6boc6T9XSC4jZpkCPS3RB8JFNb12BpkoeGYg3Q9RQ@HIDDEN>
 <56ab18b1-4348-9b2c-85bb-af9b76cd429a@HIDDEN>
 <CALDnm52XM+S07OPXO3qtfLN4pup-kT2U=WRO9dUo3tv9uYJWXw@HIDDEN>
 <38a06f3c-4a7a-755c-c99b-708f91afabfa@HIDDEN>
 <CALDnm537=LHAoaGqPpy7=JOPsYn=z+=Q6QW=b9bjrAWOtxqRaQ@HIDDEN>
 <9f59f87c-2489-aaa0-5b3f-0e911b7ffa6c@HIDDEN>
 <CALDnm508EK5qM+AaOOJiNDm-=ORCuOieYpkjO=OSRas+_WTFwQ@HIDDEN>
 <8a36e61a-1c5b-bf3b-a454-077348589c4f@HIDDEN>
Date: Sat, 14 Aug 2021 11:36:32 +0100
In-Reply-To: <8a36e61a-1c5b-bf3b-a454-077348589c4f@HIDDEN> (Dmitry Gutov's
 message of "Sat, 14 Aug 2021 05:47:43 +0300")
Message-ID: <87y29471ov.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 47711
Cc: Daniel Mendler <mail@HIDDEN>,
 Stefan Monnier <monnier@HIDDEN>, 48841 <at> debbugs.gnu.org,
 47711 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Dmitry Gutov <dgutov@HIDDEN> writes:

> It's basically this: we cannot mutate what we don't own. Across all of
> completion functions out there, there will be such that return
> "shared" strings (meaning, not copied or newly allocated) from their
> completion tables. And modifying them is bad, with consequences which
> can present themselves in unexpected, often subtle ways.

I agree with this premise.  But would you call putting a uniquely named
text property in them a modification or mutation of said strings?  I
don't.

> Since up until now completion-pcm--hilit-commonality copied all
> strings before modifying, completion tables such as described (with
> "shared" strings) have all been "legal".

Again, if I take one of this shared strings, in whichever environment
running row now, and I secretly attach a privatte "joaot/blargh"
property to it, there is very very low likelyhood that that will hurt
anybody.

You seem to be worrying about re-setting the 'face' property (a public
property by excellence) and that's the very same bug I experienced and
have described early.  It's not even a hard bug to see.  Just remove the
copy-sequence in `completion-pcm--hilit-commonality' and see strange
stuff happening.

But if you set some other property, _that_ bug _doesn't_ occur.  Do some
other bugs occur?  I don't know, I don't think we'll ever know, for
_any_ change.

Furthermore, there are other ways to forego the copying in
`completion-pcm--hilit-commonality and not even touch _ANY_ string
property.

> Suddenly deciding to stop supporting them would be a major API
> breakage with consequences that are hard to predict. And while I
> perhaps agree that it's an inconvenience, I don't think it's a choice
> we can simply make as this stage in c-a-p-f's development.
>
> To give an example of a subtle consequence:
>
>   1. (setq s (symbol-name 'car))
>
>   2. (put-text-property 1 3 'face 'error s)
>
>   3. Switch to a buffer in fundamental mode
>
>   4. (insert (symbol-name 'car)) --> see the error face in the buffer

It's not even subtle :-) Yes this is why have seen from the beginning in
bug#48841.  I think it was even I who reported it to you.

The principle to follow can be summarized as this: "Don't touch values
of properties you don't own in objects you don't own."

So just don't touch the 'face' property in things you don't own!  But
feel free to touch the "dmitry/blargh" property even in objects you
don't own.

So 'c-p--h-l' doesn't "own" face.  So it must either create an object
that it owns or set something that it does own.  'completion-score' is
"owned" by 'c-p--h-l'.  Only it can write it (though others can read
it).

> Now imagine that some completion table collects symbol names by
> passing obarray through #'symbol-name rather than #'all-completions,
> and voila, if the completion machinery adds properties (any
> properties, not just face) to those strings, you have just modified a
> bunch of global values. That's not good.

Why?  Maybe I'm missing something.  Why is adding properties -- that
no-one but the completion machinery knows about -- to those shared
strings "not good"?  What bad thing can happen if I do?

> And in the example above, the values are those that the
> lispref/objects.texi says we should not change (though it gives
> (symbol-name 'cons) as example). "Not mutable", in its parlance. IIRC
> the related discussions mentioned that modifying such values could
> lead to a segfault in some previous Emacs versions. Maybe not anymore,
> but it's still not a good idea.

You're extrapolating "change" to also include attaching properties to
symbols.  I think that document just means that you can't do stuff like

    (aset "cons" 0 ?z)

or

    (aset (symbol-name 'cons) 0 ?z)

I don't think it means you can't

    (put-text-property 0 1 'joaot/muahahah 42 (symbol-name 'cons))

But maybe Eli or someone else more knowledgeable in the deep internals
of Emacs can correct me.

If indeed I'm wrong, there are other ways to forego the copying in
`c-p---hilit-commonality` and still don't incurr in any such "mutation".
We must keep our eyes on the prize: copying -- not property-attaching --
is the real bummer here.

scratch/icomplete-lazy-highlight-attempt-2, although still incomplete,
is one such approach, though it still sets `completion-score` on the
"shared" string, used later for sorting.  But also that could be
prevented (again, only if it turns out to be actually problematic IMO).

Jo=C3=A3o

PS: Maybe I've not stated it clearly enough: I *don't* object to -- or
endorse -- Daniel's patch.  My point was solely that it mixes too many
things for me to be intellectually able to review its functional merits,
and that those things should be separated into multiple problems and
patches to make this evaluation easier.  Maybe someone with superior
intellecutal capacity can review -- on substance -- as it stands.

See my other reply containing benchmarks.  Daniel's patch doesn't
perform well there, but for all I know, it can co-exist with my
non-copying approach, and we can all have our cake.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 14 Aug 2021 09:48:52 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Aug 14 05:48:52 2021
Received: from localhost ([127.0.0.1]:43575 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mEqHX-0007bT-Rc
	for submit <at> debbugs.gnu.org; Sat, 14 Aug 2021 05:48:51 -0400
Received: from mail-wm1-f49.google.com ([209.85.128.49]:53088)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <joaotavora@HIDDEN>)
 id 1mEqHO-0007at-6L; Sat, 14 Aug 2021 05:48:42 -0400
Received: by mail-wm1-f49.google.com with SMTP id f10so5259283wml.2;
 Sat, 14 Aug 2021 02:48:38 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=from:to:cc:subject:references:date:in-reply-to:message-id
 :user-agent:mime-version;
 bh=IS4QwsDKH6F4eWyXx6K0rL7ucoThMsH5rpcQmB+fRc8=;
 b=OJdEtXu9nvilRpB8PfhTS6L2QOIUYP043g1Fv82poBBkAx3FMobRNV1sXK/U6hBEuz
 CKZNa6is7lzW4Tc8z0noc1ahwfzUYbJu8Qo2MiDd5klm3iID/SnCbJ+BYOgX3ZYUPBGK
 TnJrQV1xxyrmdgekIX5EmLf2LoCJHWkWIOPldExlzNAGIpGurhdpVN4wZtU2/Ik4QaiC
 6vdRwmIGFh6WJWO7XhZjO9M+2FLA/OuyI8Mf3VozvOW1idEThkzIyRiJNb7OpeTdFWAk
 8jN8fsvCY/l8DMmY2i50pWkJ9BU5cV726+1CZ4YyRYHAkPTFvOU16GIxTUWcuKgqnQJN
 WF/g==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to
 :message-id:user-agent:mime-version;
 bh=IS4QwsDKH6F4eWyXx6K0rL7ucoThMsH5rpcQmB+fRc8=;
 b=DK8hkUSafjUj7HKuFo4qv4tNEUmLFRwhLN3rddJVozkV8G/tJ8S5f8AHWKyR7agxEu
 A4hCPcrjHp9iexu1GSoqWHGZpFqLzav63fEkQTwLIaoNk/qGHP1xW+MQh4GzG9fvBrMh
 DrHGm/ZG4CQwdmbb5iff6n2UlKGZZTt/L3KAx/QWb2RRrRSquqS6sZmkRaA4Ht8/Jyes
 Fwk0B7jl0q22cTlxji/VcEsHf3Yadm09Joj8GWVPkotFYkEjWkmmqrO1OxkxEqYKeAhQ
 6SRKGr3sgTauMPM/U7Rmo0kliW93TfixyKbcPVi3Dro7StRKP67J6IPCBDX6sF5d22Gz
 bn4Q==
X-Gm-Message-State: AOAM532MiwwLhW9luFJ652V2YTeSeZbH3+gc2Vdf/XmOVqBtXUOODx90
 I/S+h2kmUbxiX81Mmrvt4Ms=
X-Google-Smtp-Source: ABdhPJzMOTtxnVBtJYFETxD9xrGT4o4DfULs9PSItYoSoGFyQeGQPc+agS/dVtKe8a52F5XjCb+cug==
X-Received: by 2002:a1c:9852:: with SMTP id a79mr6632045wme.2.1628934512004;
 Sat, 14 Aug 2021 02:48:32 -0700 (PDT)
Received: from krug (a94-133-27-132.cpe.netcabo.pt. [94.133.27.132])
 by smtp.gmail.com with ESMTPSA id c15sm3983732wrw.93.2021.08.14.02.48.30
 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
 Sat, 14 Aug 2021 02:48:31 -0700 (PDT)
From: =?utf-8?B?Sm/Do28gVMOhdm9yYQ==?= <joaotavora@HIDDEN>
To: Eli Zaretskii <eliz@HIDDEN>
Subject: Re: bug#47711: [PATCH VERSION 2] Add new
 `completion-filter-completions` API and deferred highlighting
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <f8c4419a-7e83-b8a9-347c-0221a2052376@HIDDEN>
 <fff6e291-bbcc-93e3-92e7-00af4664b8d4@HIDDEN>
 <3d3f894f-a6fa-53ae-5d50-c3aa9bffc73e@HIDDEN>
 <CALDnm53xh6boc6T9XSC4jZpkCPS3RB8JFNb12BpkoeGYg3Q9RQ@HIDDEN>
 <56ab18b1-4348-9b2c-85bb-af9b76cd429a@HIDDEN>
 <CALDnm52XM+S07OPXO3qtfLN4pup-kT2U=WRO9dUo3tv9uYJWXw@HIDDEN>
 <38a06f3c-4a7a-755c-c99b-708f91afabfa@HIDDEN>
 <CALDnm537=LHAoaGqPpy7=JOPsYn=z+=Q6QW=b9bjrAWOtxqRaQ@HIDDEN>
 <9f59f87c-2489-aaa0-5b3f-0e911b7ffa6c@HIDDEN>
 <83im08bjc3.fsf@HIDDEN>
Date: Sat, 14 Aug 2021 10:48:28 +0100
In-Reply-To: <83im08bjc3.fsf@HIDDEN> (Eli Zaretskii's message of "Sat, 14 Aug
 2021 10:01:48 +0300")
Message-ID: <877dgo8ihf.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="=-=-="
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 47711
Cc: Daniel Mendler <mail@HIDDEN>, dgutov@HIDDEN,
 monnier@HIDDEN, 48841 <at> debbugs.gnu.org, 47711 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

--=-=-=
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

Eli Zaretskii <eliz@HIDDEN> writes:

>> > On Fri, Aug 13, 2021 at 1:22 PM Daniel Mendler <mail@HIDDEN=
> wrote:
>> >=20
>> >> It is important to keep this property since this will preclude many b=
ugs
>> >> due to string mutation.
>> > I am aware of this, of course.  Can you give examples of these "many b=
ugs"?
>> > Perhaps other than the one I already described and addressed?
>> No, Jo=C3=A3o, this is not how it goes.  I don't have to prove to you th=
at
>> your idea introduces bugs.  You have to show that mutation of the
>> completion table strings (which are not supposed to be mutated) will not
>> lead to bugs, which are hard to find.
> Please calm down, both of you.  No one has to prove anything to anyone
> here, that's not how Emacs development works.  We need to see which
> idea is better, and if none is significantly better, we will probably
> have both of them living side by side.
>
> And while asking for an example of potential bugs is reasonable,
> asking for a proof that a change will NOT lead to bugs isn't.

As far as I remember, I have done the first. I found bugs and addressed
them. I cannot _prove_ that my change will not leads to bugs indeed:
no-one can with any change.  I've just stated repeteadly that I'm not
aware of any such bugs.  I can understand intuition" for bugs to a
certain extent (everyone has intuition), but this intuition must always
resolve into actual reality to be useful in the end.

> So how
> about a couple of examples where having original strings unchanged is
> important, which could then be discussed?

Good idea, so in the absence of any controlled benchmarks I did some of
my own, using the most controlled environment I could devise.  I start
Emacs like so:

   ~/Source/Emacs/emacs/src/emacs -Q -f fido-mode -f fido-vertical-mode -l =
~/tmp/benchmark.el ~/tmp/benchmark.el

I prime the obarry with lots of symblos to make completion purposedly
slow:

   (require 'cl-lib)
   (cl-loop repeat 300000 do (intern (symbol-name (gensym))))

I attach the file. Then I try a run of 10 invocations of

  ;; Press C-u C-x C-e C-m quickly to produce a sample.=20=20
  (benchmark-run (completing-read "" obarray))

This, I think, is a good representation of the responsiveness of the
completion system.  It always prints well after I finish typing, so I
don't think I'm inducing any artificial slowdows while it waits for my
input.  When not measuring quantitatively, I also feel the difference in
responsiveness between different approaches.

Summarized results with an assortment of Emacs builds.

   - the current master (254dc6ab4ca8e6a549a795f9eaf45378ce51b61f).

     20.25 seconds total
=20=20=20
   - Applying Daniel's patch over 254dc6.

     23.41 seconds total
=20=20=20
   - The theoretical best situation where we don't highlight in
     completion-pcm--hilit-commonality (like 254dc6, but just removed
     the copy-sequence)

     10.70 seconds total

   - Experimental patch published in
     scratch/icomplete-lazy-highlight-attempt-2 (not finished, still
     needs a way for frontends to opt into the optimization).

     10.80 seconds total

I invite you all to reproduce these results.

In conclusion, I don't think Daniel's patch is going in the right
direction, *performance-wise*, for the kind of responsiveness scenarios
that I am concerned with, and which were discussed with Dmitry in
bug#48841.  It seems to slow down the process by about 10%.

Note 1: there may be *other* performance scenarios that I am not aware
of, where Daniel's patch excels.  I've requested these benchmarks,
regrettably without any success.

Note 2: doesn't mean that there aren't *other* merits to Daniel's patch,
but I have not understood these yet.  That is due to the stated fact
that the patch is very long, and seems to comprise performance
improvements, refactorings, and API redesign.  It has no documentation
in manual and/or examples on how to use the new API.

>> >> Note that your idea also does not address the other issues which are
>> >> addressed by my patch.
>> >=20
>> > That's for sure.  My patch idea addresses only that single problem.
>> > I think this is a good property of patches: to solve one thing, not ma=
ny.
>>=20
>> No, this is not necessarily true.  This is only good if the problem is
>> solved in a way which is future proof.  The idea of mutating the strings
>> is a hack and not a solution.
>
> Just to make sure we are on the same page: adding a text property to a
> string doesn't mutate a string.  Lisp programs that process these
> strings will not necessarily see any difference, and displaying those
> strings will also not show any difference if the property is not
> related to display.  So the assumption that seems to be made here,
> that adding a property is the same as mutating a string, is IMO
> inaccurate if not incorrect.

Yes, in Lisp it is very common to attach a "private" property to an
object.  If no-one else knows about the existence of that property, then
attaching it is not harmful. Generally, of course: there are situations
where adding a private property brings side-effects to other parts of
the code.  But IMO that other code is in the wrong, not the one that
adds properties.

Also, to be clear, attaching a different property (as in, not 'face') to
the completion string is only _one_ of the ways of the ways to bypass
copying.  According to my measurements, performance doesn't seem to be
decided by property attachments, but by copying or not copying of the
character data of said strings in completion-pcm--hilit-commonality.

Jo=C3=A3o


--=-=-=
Content-Type: application/emacs-lisp
Content-Disposition: attachment; filename=benchmark.el
Content-Transfer-Encoding: quoted-printable
Content-Description: benchmark.el

(require 'cl-lib)

;; Introduce 300 000 new symbols to slow things down.  Probably more
;; than most non-Spacemancs people will have :-)
(cl-loop repeat 300000 do (intern (symbol-name (gensym))))

(when nil
  ;; Press C-u C-x C-e C-m quickly to produce a sample
  (benchmark-run (completing-read "bla" obarray))

  ;; scratch/icomplete-lazy-highlight-attempt-2 (dbfe6f72e3608db4488bbc9bbc=
22d876746b1012)
  (cl-reduce #'+
             '((1.060225172 4 0.40529085799999987)
               (1.084274293 4 0.40401850100000036)
               (1.075857223 4 0.4066735760000002)
               (1.096181884 4 0.41024331699999994)
               (1.090617755 4 0.4027740900000003)
               (1.092172347 4 0.4073497049999997)
               (1.064530759 4 0.40525715400000006)
               (1.088796966 4 0.4075235989999997)
               (1.052578979 4 0.39851351000000035)
               (1.0952534230000002 4 0.4396319659999999))
             :key #'car);; 10.800488801

  ;; Current situation origin/master (254dc6ab4ca8e6a549a795f9eaf45378ce51b=
61f)
  (cl-reduce #'+
             '((2.094681183 15 1.299947048)
               (2.061771249 14 1.248384553000001)
               (1.9915545579999998 14 1.1808336689999983)
               (2.1072676080000003 15 1.2833755230000001)
               (2.100205698 15 1.2943715630000003)
               (1.940760815 14 1.1518027680000005)
               (1.919338815 14 1.127410448)
               (2.0683418259999997 14 1.272936392)
               (1.9932618739999999 15 1.1983156959999999)
               (1.969784269 17 1.140629235))
             :key #'car) ;; 20.246967895000004

  ;; Daniel's patch from Mon, 12 Jul 2021 21:40:32 +0200
  (cl-reduce #'+
             '((2.371949053 12 1.0679449390000002)
               (2.411950542 12 1.0926466229999985)
               (2.3595232590000004 12 1.0562706020000006)
               (2.386636212 12 1.0797333340000002)
               (2.37649102 12 1.0579168220000001)
               (2.315395413 12 0.9729727209999997)
               (2.310257558 12 0.9982636050000004)
               (2.283203076 12 0.9730703639999998)
               (2.302582002 13 0.9867599240000002)
               (2.292846619 15 0.9585957429999998))
             :key #'car)  ;; 23.410834754

  ;; Theoretical best (don't care about highlighting, so don't copy)
  (cl-reduce #'+
             '((1.090906703 4 0.40746227999999984)
               (1.052931397 4 0.4143886020000007)
               (1.114877434 4 0.42559067599999967)
               (1.0578293460000001 4 0.40872522)
               (1.086898854 4 0.4109444489999996)
               (1.0749850980000002 4 0.41140822900000007)
               (1.076554257 4 0.4121859250000002)
               (0.998330274 4 0.3969351310000002)
               (1.064626675 4 0.4286588339999997)
               (1.079113392 4 0.44295618700000006))
             :key #'car) ;; 10.69705343

  )

--=-=-=--




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 14 Aug 2021 08:23:48 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Aug 14 04:23:48 2021
Received: from localhost ([127.0.0.1]:43510 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mEoxF-0005O3-06
	for submit <at> debbugs.gnu.org; Sat, 14 Aug 2021 04:23:48 -0400
Received: from mail-wr1-f49.google.com ([209.85.221.49]:36492)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <joaotavora@HIDDEN>)
 id 1mEox5-0005NY-7j; Sat, 14 Aug 2021 04:23:39 -0400
Received: by mail-wr1-f49.google.com with SMTP id b13so16406279wrs.3;
 Sat, 14 Aug 2021 01:23:35 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=from:to:cc:subject:references:date:in-reply-to:message-id
 :user-agent:mime-version:content-transfer-encoding;
 bh=PbE/O458f4w1AgSVDgGeifvLtJWnYHfLvklmc66ifKw=;
 b=r4GFs6/YIoNbxO9A3AhZshKw8xKPXEi/GCsoOt5lD/zXIZpM4PN5Of0jkm5/MD40LB
 SMUM7p6C0w6y3voUSs52UydRmLrkM0b5KGQPF3FUkDnmIg8dz9Q3EjiK+jzqsGIKdnN2
 ibfngZ3SqmKXAwkybP1fnLcmuKL3bNBfTJBPxRohx+mFHYBTiGCEN4T6wLlujbOkcXqW
 kZN/waUTySLyUJERyOs8+feDDU4zT6TPvi/usj22Y+rMp8J/m/iydEb9TsqoZUFqPrFm
 8kKe7eUrrMtP+uRUyYD8kCwFqsWKQbSX3bufB1YoWRxBonZ+pDRdbFQMBFQEz5izjd3R
 XyVQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to
 :message-id:user-agent:mime-version:content-transfer-encoding;
 bh=PbE/O458f4w1AgSVDgGeifvLtJWnYHfLvklmc66ifKw=;
 b=S7Vj7+9BJ0Xy25H7WuASDth3Q6fVU0VMVNO0H+tWZbBELCcTYRuaGfFLbxTZUHCuU+
 cGE3kzEKOqFz3NCCcrAwx14U6zW43oszTaqmQc2RJdys6vyv2AGhvy3wuYYYNJzfyyG0
 2hXRw/S4+Rrk1OKSS9PHg4WH9ukxZSIPRGajrkd+qzG7Z21IGx7ZxNAkzwfItlYV1XY5
 zkecBvBbjqM599sd06u6QoKHGg1pbXvzknk3Vgj+CbhKdCUOKBbG2nr4/EQFmLbJxtzm
 MYWBtGNuBgamqYXW48LbzKIuk/ulqt262W2ipbHqzM8Cwqy5QDNH+IeYvg/ekYm16Xqz
 5/lg==
X-Gm-Message-State: AOAM532NYOQFPTnIyonOt0s6zM3IRmISYCtLsvuxCVYvz1izjrnP3Cgt
 WRfipPNgR6ru+2YMkgQccD9cTAkgFlc=
X-Google-Smtp-Source: ABdhPJw13EjCIKhW4B0XX0PCwubNK8eU02A7Jj4+dq420uMlLn626Rrwj8Ocwoi2pBUWwsK9BH6wMg==
X-Received: by 2002:adf:d085:: with SMTP id y5mr7301176wrh.209.1628929408907; 
 Sat, 14 Aug 2021 01:23:28 -0700 (PDT)
Received: from krug (a94-133-27-132.cpe.netcabo.pt. [94.133.27.132])
 by smtp.gmail.com with ESMTPSA id x18sm3687486wmc.17.2021.08.14.01.23.27
 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
 Sat, 14 Aug 2021 01:23:27 -0700 (PDT)
From: =?utf-8?B?Sm/Do28gVMOhdm9yYQ==?= <joaotavora@HIDDEN>
To: Dmitry Gutov <dgutov@HIDDEN>
Subject: Re: bug#48841: bug#47711: [PATCH VERSION 2] Add new
 `completion-filter-completions` API and deferred highlighting
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <f8c4419a-7e83-b8a9-347c-0221a2052376@HIDDEN>
 <fff6e291-bbcc-93e3-92e7-00af4664b8d4@HIDDEN>
 <3d3f894f-a6fa-53ae-5d50-c3aa9bffc73e@HIDDEN>
 <CALDnm53xh6boc6T9XSC4jZpkCPS3RB8JFNb12BpkoeGYg3Q9RQ@HIDDEN>
 <56ab18b1-4348-9b2c-85bb-af9b76cd429a@HIDDEN>
 <CALDnm52XM+S07OPXO3qtfLN4pup-kT2U=WRO9dUo3tv9uYJWXw@HIDDEN>
 <328f87eb-6474-1442-e1ca-9ae8deb2a84a@HIDDEN>
Date: Sat, 14 Aug 2021 09:23:25 +0100
In-Reply-To: <328f87eb-6474-1442-e1ca-9ae8deb2a84a@HIDDEN> (Dmitry Gutov's
 message of "Sat, 14 Aug 2021 05:55:17 +0300")
Message-ID: <87sfzca0zm.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 47711
Cc: Daniel Mendler <mail@HIDDEN>,
 Stefan Monnier <monnier@HIDDEN>, 48841 <at> debbugs.gnu.org,
 47711 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Dmitry Gutov <dgutov@HIDDEN> writes:

> Aside from the mutability/ownership issue,
>
> On 13.08.2021 15:05, Jo=C3=A3o T=C3=A1vora wrote:
>> If one removes these lines, the process becomes much faster, but there i=
s a
>> problem with highlighting.  My idea is indeed to defer highlighting by n=
ot
>> setting the 'face property directly on that shared string, but some
>> other property
>> that is read later from the shared string by compliant frontents.
>
> I haven't done any direct benchmarking, but I'm pretty sure that this
> approach cannot, by definition, be as fast as the non-mutating one.
>
> Because you go through the whole list and mutate all of its elements,
> you have to perform a certain bit of work W x N times, where N is the
> size of the whole list.

Let's call W the work that you perform N times in this istuation.  In
the non-mutation, let's call it Z.  So

W <=3D Z, because Z not only propertizes the string with a calculation of
faces but _also copies its character contents_.

Also I think it's better to start about copying rather than mutating.
As Eli points out, putting a text property in a string (my idea) is not
equivalent with "mutating" it.

> Whereas the deferred-mutation approach will only have to do its bit
> (which is likely more work, like, WWW) only 20 times, or 100 times, or
> however many completions are displayed. And this is usually
> negligible.

I think you're going in the same fallacy you went briefly in the other
bug report.  The flex and pcm styles (meaning
completion-pcm--hilit-commonality) has to go through all the completions
when deciding the score to atribute to each completion that we already
know matches the pattern.  That's because this scoring is essential to
sorting.  That's a given in both scenarios, copying and non-copying.

Then, it's true that only a very small set of those eventually have to
be displayed to the user, depending on where wants she wants her
scrolling page to be.  So that's when you have to apply 'face' to, say
20 strings, and that can indeed be pretty fast.  But where does the
information come from?

- Currently, it comes from the string's 'face' itself, which was copied
  entirely.

- In the non-copying approach, it must come from somewhere else.  One
  idea is that it comes from a new "private" property 'lazy-face', also
  in the string itselv, but which was _not_ copied.  Another idea is
  just to remember the pattern and re-match it to those 20 strings.

I think the second alternative is always faster.

> However big the difference is going to be, I can't say in advance, of
> course, or whether it's going to be shadowed by some other performance
> costs. But the non-mutating approach should have the best optimization
> potential when the list is long.

Don't think so.  I'm doing benchmarks, will post soon.

Jo=C3=A3o




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 14 Aug 2021 07:16:26 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Aug 14 03:16:26 2021
Received: from localhost ([127.0.0.1]:43478 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mEnu6-0003di-ES
	for submit <at> debbugs.gnu.org; Sat, 14 Aug 2021 03:16:26 -0400
Received: from eggs.gnu.org ([209.51.188.92]:51596)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>)
 id 1mEnu4-0003dS-7x; Sat, 14 Aug 2021 03:16:25 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:44022)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <eliz@HIDDEN>)
 id 1mEntz-0001rd-33; Sat, 14 Aug 2021 03:16:19 -0400
Received: from 84.94.185.95.cable.012.net.il ([84.94.185.95]:2898
 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1mEnty-0006r5-Iw; Sat, 14 Aug 2021 03:16:18 -0400
Date: Sat, 14 Aug 2021 10:16:08 +0300
Message-Id: <83fsvcbio7.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Dmitry Gutov <dgutov@HIDDEN>
In-Reply-To: <328f87eb-6474-1442-e1ca-9ae8deb2a84a@HIDDEN> (message from
 Dmitry Gutov on Sat, 14 Aug 2021 05:55:17 +0300)
Subject: Re: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new
 `completion-filter-completions` API and deferred highlighting
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <f8c4419a-7e83-b8a9-347c-0221a2052376@HIDDEN>
 <fff6e291-bbcc-93e3-92e7-00af4664b8d4@HIDDEN>
 <3d3f894f-a6fa-53ae-5d50-c3aa9bffc73e@HIDDEN>
 <CALDnm53xh6boc6T9XSC4jZpkCPS3RB8JFNb12BpkoeGYg3Q9RQ@HIDDEN>
 <56ab18b1-4348-9b2c-85bb-af9b76cd429a@HIDDEN>
 <CALDnm52XM+S07OPXO3qtfLN4pup-kT2U=WRO9dUo3tv9uYJWXw@HIDDEN>
 <328f87eb-6474-1442-e1ca-9ae8deb2a84a@HIDDEN>
MIME-version: 1.0
Content-type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 47711
Cc: mail@HIDDEN, 47711 <at> debbugs.gnu.org, joaotavora@HIDDEN,
 48841 <at> debbugs.gnu.org, monnier@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: Dmitry Gutov <dgutov@HIDDEN>
> Date: Sat, 14 Aug 2021 05:55:17 +0300
> Cc: Stefan Monnier <monnier@HIDDEN>, 48841 <at> debbugs.gnu.org,
>  47711 <at> debbugs.gnu.org
> 
> On 13.08.2021 15:05, João Távora wrote:
> > If one removes these lines, the process becomes much faster, but there is a
> > problem with highlighting.  My idea is indeed to defer highlighting by not
> > setting the 'face property directly on that shared string, but some
> > other property
> > that is read later from the shared string by compliant frontents.
> 
> I haven't done any direct benchmarking, but I'm pretty sure that this 
> approach cannot, by definition, be as fast as the non-mutating one.

Daniel seems to think otherwise, AFAIU.

> Because you go through the whole list and mutate all of its elements, 
> you have to perform a certain bit of work W x N times, where N is the 
> size of the whole list.
> 
> Whereas the deferred-mutation approach will only have to do its bit 
> (which is likely more work, like, WWW) only 20 times, or 100 times, or 
> however many completions are displayed. And this is usually negligible.
> 
> However big the difference is going to be, I can't say in advance, of 
> course, or whether it's going to be shadowed by some other performance 
> costs. But the non-mutating approach should have the best optimization 
> potential when the list is long.

So I guess the suggestion to have a benchmark is still useful, because
the estimations of which approach has better performance vary between
you three.  So maybe producing such benchmarks would be a good step?




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 14 Aug 2021 07:13:12 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Aug 14 03:13:12 2021
Received: from localhost ([127.0.0.1]:43470 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mEnqy-0003XW-Im
	for submit <at> debbugs.gnu.org; Sat, 14 Aug 2021 03:13:12 -0400
Received: from eggs.gnu.org ([209.51.188.92]:51076)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>)
 id 1mEnqx-0003X9-3n; Sat, 14 Aug 2021 03:13:11 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:43950)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <eliz@HIDDEN>)
 id 1mEnqr-0007Pp-6e; Sat, 14 Aug 2021 03:13:05 -0400
Received: from 84.94.185.95.cable.012.net.il ([84.94.185.95]:2698
 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1mEnqq-0006TB-UW; Sat, 14 Aug 2021 03:13:05 -0400
Date: Sat, 14 Aug 2021 10:12:54 +0300
Message-Id: <83h7fsbitl.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Dmitry Gutov <dgutov@HIDDEN>
In-Reply-To: <8a36e61a-1c5b-bf3b-a454-077348589c4f@HIDDEN> (message from
 Dmitry Gutov on Sat, 14 Aug 2021 05:47:43 +0300)
Subject: Re: bug#48841: bug#47711: [PATCH VERSION 2] Add new
 `completion-filter-completions` API and deferred highlighting
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <f8c4419a-7e83-b8a9-347c-0221a2052376@HIDDEN>
 <fff6e291-bbcc-93e3-92e7-00af4664b8d4@HIDDEN>
 <3d3f894f-a6fa-53ae-5d50-c3aa9bffc73e@HIDDEN>
 <CALDnm53xh6boc6T9XSC4jZpkCPS3RB8JFNb12BpkoeGYg3Q9RQ@HIDDEN>
 <56ab18b1-4348-9b2c-85bb-af9b76cd429a@HIDDEN>
 <CALDnm52XM+S07OPXO3qtfLN4pup-kT2U=WRO9dUo3tv9uYJWXw@HIDDEN>
 <38a06f3c-4a7a-755c-c99b-708f91afabfa@HIDDEN>
 <CALDnm537=LHAoaGqPpy7=JOPsYn=z+=Q6QW=b9bjrAWOtxqRaQ@HIDDEN>
 <9f59f87c-2489-aaa0-5b3f-0e911b7ffa6c@HIDDEN>
 <CALDnm508EK5qM+AaOOJiNDm-=ORCuOieYpkjO=OSRas+_WTFwQ@HIDDEN>
 <8a36e61a-1c5b-bf3b-a454-077348589c4f@HIDDEN>
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 47711
Cc: mail@HIDDEN, 47711 <at> debbugs.gnu.org, joaotavora@HIDDEN,
 48841 <at> debbugs.gnu.org, monnier@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: Dmitry Gutov <dgutov@HIDDEN>
> Date: Sat, 14 Aug 2021 05:47:43 +0300
> Cc: Stefan Monnier <monnier@HIDDEN>, 48841 <at> debbugs.gnu.org,
>  47711 <at> debbugs.gnu.org
> 
> I thought I explained the problem with this previously.
> 
> It's basically this: we cannot mutate what we don't own. Across all of 
> completion functions out there, there will be such that return "shared" 
> strings (meaning, not copied or newly allocated) from their completion 
> tables. And modifying them is bad, with consequences which can present 
> themselves in unexpected, often subtle ways.
> 
> Since up until now completion-pcm--hilit-commonality copied all strings 
> before modifying, completion tables such as described (with "shared" 
> strings) have all been "legal". Suddenly deciding to stop supporting 
> them would be a major API breakage with consequences that are hard to 
> predict. And while I perhaps agree that it's an inconvenience, I don't 
> think it's a choice we can simply make as this stage in c-a-p-f's 
> development.

This sounds like an argument against Daniel's approach as well, no?
Because if a completion API returns strings it "doesn't own", there
will be restrictions on Lisp programs that use those strings, because
those Lisp programs previously could do anything they liked with those
strings, and now they cannot.  Or am I missing something?

>    1. (setq s (symbol-name 'car))
> 
>    2. (put-text-property 1 3 'face 'error s)
> 
>    3. Switch to a buffer in fundamental mode
> 
>    4. (insert (symbol-name 'car)) --> see the error face in the buffer
> 
> Now imagine that some completion table collects symbol names by passing 
> obarray through #'symbol-name rather than #'all-completions, and voila, 
> if the completion machinery adds properties (any properties, not just 
> face) to those strings, you have just modified a bunch of global values. 
> That's not good.

How is this different from Daniel's proposal of returning the original
strings?  AFAIU, he just shifts the responsibility from the completion
code to the caller of the completion code, but basically leaves the
problem still very much real and pretty much into our face.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 14 Aug 2021 07:02:23 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Aug 14 03:02:23 2021
Received: from localhost ([127.0.0.1]:43465 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mEngR-0003HZ-FC
	for submit <at> debbugs.gnu.org; Sat, 14 Aug 2021 03:02:23 -0400
Received: from eggs.gnu.org ([209.51.188.92]:50200)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>)
 id 1mEngH-0003H8-Ok; Sat, 14 Aug 2021 03:02:14 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:43850)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <eliz@HIDDEN>)
 id 1mEngB-0006iO-5Y; Sat, 14 Aug 2021 03:02:03 -0400
Received: from 84.94.185.95.cable.012.net.il ([84.94.185.95]:2015
 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1mEngA-0003L8-2Y; Sat, 14 Aug 2021 03:02:03 -0400
Date: Sat, 14 Aug 2021 10:01:48 +0300
Message-Id: <83im08bjc3.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Daniel Mendler <mail@HIDDEN>
In-Reply-To: <9f59f87c-2489-aaa0-5b3f-0e911b7ffa6c@HIDDEN> (message
 from Daniel Mendler on Fri, 13 Aug 2021 14:56:38 +0200)
Subject: Re: bug#47711: [PATCH VERSION 2] Add new
 `completion-filter-completions` API and deferred highlighting
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <f8c4419a-7e83-b8a9-347c-0221a2052376@HIDDEN>
 <fff6e291-bbcc-93e3-92e7-00af4664b8d4@HIDDEN>
 <3d3f894f-a6fa-53ae-5d50-c3aa9bffc73e@HIDDEN>
 <CALDnm53xh6boc6T9XSC4jZpkCPS3RB8JFNb12BpkoeGYg3Q9RQ@HIDDEN>
 <56ab18b1-4348-9b2c-85bb-af9b76cd429a@HIDDEN>
 <CALDnm52XM+S07OPXO3qtfLN4pup-kT2U=WRO9dUo3tv9uYJWXw@HIDDEN>
 <38a06f3c-4a7a-755c-c99b-708f91afabfa@HIDDEN>
 <CALDnm537=LHAoaGqPpy7=JOPsYn=z+=Q6QW=b9bjrAWOtxqRaQ@HIDDEN>
 <9f59f87c-2489-aaa0-5b3f-0e911b7ffa6c@HIDDEN>
MIME-version: 1.0
Content-type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 47711
Cc: 48841 <at> debbugs.gnu.org, dgutov@HIDDEN, joaotavora@HIDDEN,
 monnier@HIDDEN, 47711 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: Daniel Mendler <mail@HIDDEN>
> Date: Fri, 13 Aug 2021 14:56:38 +0200
> Cc: 47711 <at> debbugs.gnu.org, Stefan Monnier <monnier@HIDDEN>,
>  48841 <at> debbugs.gnu.org, Dmitry Gutov <dgutov@HIDDEN>
> 
> On 8/13/21 2:37 PM, João Távora wrote:
> > On Fri, Aug 13, 2021 at 1:22 PM Daniel Mendler <mail@HIDDEN> wrote:
> > 
> >> It is important to keep this property since this will preclude many bugs
> >> due to string mutation.
> > 
> > I am aware of this, of course.  Can you give examples of these "many bugs"?
> > Perhaps other than the one I already described and addressed?
> 
> No, João, this is not how it goes.  I don't have to prove to you that
> your idea introduces bugs.  You have to show that mutation of the
> completion table strings (which are not supposed to be mutated) will not
> lead to bugs, which are hard to find.

Please calm down, both of you.  No one has to prove anything to anyone
here, that's not how Emacs development works.  We need to see which
idea is better, and if none is significantly better, we will probably
have both of them living side by side.

And while asking for an example of potential bugs is reasonable,
asking for a proof that a change will NOT lead to bugs isn't.  So how
about a couple of examples where having original strings unchanged is
important, which could then be discussed?

> >> Note that your idea also does not address the other issues which are
> >> addressed by my patch.
> > 
> > That's for sure.  My patch idea addresses only that single problem.
> > I think this is a good property of patches: to solve one thing, not many.
> 
> No, this is not necessarily true.  This is only good if the problem is
> solved in a way which is future proof.  The idea of mutating the strings
> is a hack and not a solution.

Just to make sure we are on the same page: adding a text property to a
string doesn't mutate a string.  Lisp programs that process these
strings will not necessarily see any difference, and displaying those
strings will also not show any difference if the property is not
related to display.  So the assumption that seems to be made here,
that adding a property is the same as mutating a string, is IMO
inaccurate if not incorrect.

And once again: please tone down your responses, both of you.  TIA.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 14 Aug 2021 06:45:31 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Aug 14 02:45:31 2021
Received: from localhost ([127.0.0.1]:43457 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mEnQB-0002rU-M3
	for submit <at> debbugs.gnu.org; Sat, 14 Aug 2021 02:45:31 -0400
Received: from eggs.gnu.org ([209.51.188.92]:48858)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>)
 id 1mEnQ7-0002r6-Sw; Sat, 14 Aug 2021 02:45:28 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:43728)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <eliz@HIDDEN>)
 id 1mEnQ1-0000b7-A2; Sat, 14 Aug 2021 02:45:21 -0400
Received: from 84.94.185.95.cable.012.net.il ([84.94.185.95]:4976
 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1mEnPz-0004qn-W4; Sat, 14 Aug 2021 02:45:20 -0400
Date: Sat, 14 Aug 2021 09:45:09 +0300
Message-Id: <83k0kobk3u.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Daniel Mendler <mail@HIDDEN>
In-Reply-To: <3d3f894f-a6fa-53ae-5d50-c3aa9bffc73e@HIDDEN> (message
 from Daniel Mendler on Fri, 13 Aug 2021 12:38:28 +0200)
Subject: Re: [PATCH VERSION 2] Add new `completion-filter-completions` API and
 deferred highlighting
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <f8c4419a-7e83-b8a9-347c-0221a2052376@HIDDEN>
 <fff6e291-bbcc-93e3-92e7-00af4664b8d4@HIDDEN>
 <3d3f894f-a6fa-53ae-5d50-c3aa9bffc73e@HIDDEN>
MIME-version: 1.0
Content-type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 47711
Cc: dgutov@HIDDEN, monnier@HIDDEN, joaotavora@HIDDEN,
 48841 <at> debbugs.gnu.org, 47711 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: Daniel Mendler <mail@HIDDEN>
> Cc: Eli Zaretskii <eliz@HIDDEN>, João Távora
>  <joaotavora@HIDDEN>, Stefan Monnier <monnier@HIDDEN>,
>  Dmitry Gutov <dgutov@HIDDEN>
> Date: Fri, 13 Aug 2021 12:38:28 +0200
> 
> I attached the overhauled patch, which addresses most of the comments by
> Eli.  In comparison to my last patch, the patch is fully backward
> compatible and preserves all existing tests.  As before, there are tests
> which check the new functionality for each existing completion style.

Thanks.  You were faster than me, so I sent a few more comments to the
old patch today.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 14 Aug 2021 06:27:21 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Aug 14 02:27:21 2021
Received: from localhost ([127.0.0.1]:43443 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mEn8b-0002PJ-8q
	for submit <at> debbugs.gnu.org; Sat, 14 Aug 2021 02:27:21 -0400
Received: from eggs.gnu.org ([209.51.188.92]:47728)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>)
 id 1mEn8Y-0002P1-Ah; Sat, 14 Aug 2021 02:27:20 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:43570)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <eliz@HIDDEN>)
 id 1mEn8Q-0002HA-RY; Sat, 14 Aug 2021 02:27:10 -0400
Received: from 84.94.185.95.cable.012.net.il ([84.94.185.95]:3865
 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1mEn8Q-0005wA-CI; Sat, 14 Aug 2021 02:27:10 -0400
Date: Sat, 14 Aug 2021 09:27:00 +0300
Message-Id: <83mtpkbky3.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Daniel Mendler <mail@HIDDEN>
In-Reply-To: <aff2bd3a-ea3e-1ac6-cf0d-2785c6591c84@HIDDEN> (message
 from Daniel Mendler on Thu, 12 Aug 2021 10:47:17 +0200)
Subject: Re: bug#48841: [PATCH] Add new `completion-filter-completions` API
 and deferred highlighting
References: <f94541d3-5f93-9a23-8256-3bbc453f127a@HIDDEN>
 <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <837dgrdrec.fsf@HIDDEN>
 <aff2bd3a-ea3e-1ac6-cf0d-2785c6591c84@HIDDEN>
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 47711
Cc: 47711 <at> debbugs.gnu.org, monnier@HIDDEN, joaotavora@HIDDEN,
 48841 <at> debbugs.gnu.org, dgutov@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> Cc: 48841 <at> debbugs.gnu.org, dgutov@HIDDEN, joaotavora@HIDDEN,
>  monnier@HIDDEN, 47711 <at> debbugs.gnu.org
> From: Daniel Mendler <mail@HIDDEN>
> Date: Thu, 12 Aug 2021 10:47:17 +0200
> 
> >> The `completions` value is the list of completion strings *without*
> >> applied highlighting.  The completion strings are returned unmodified,
> >> which avoids allocations and results in performance gains for
> > 
> > This is unclear: how can you return a list of strings which you
> > produce without allocating the strings?
> 
> The function 'completion-filter-completions' receives a completion table
> as argument.  The strings produced by this table are returned
> unmodified, but of course the completion table has to produce them.  For
> a static completion table (e.g., in the simplest case a list of strings)
> the completion table itself will not allocate strings.  In this scenario
> 'completion-filter-completions' will not perform any string allocations,
> only the list will be allocated.  This is what leads to major
> performance gains.

My point was that at least some of this should be in the description,
otherwise it will leave the reader wondering.

> >> +(defvar completion--filter-completions nil
> >> +  "Enable the new completions return value format.
> >
> > Btw, why is this an internal variable?  Shouldn't all completion
> > styles ideally support it?  If so, it should be a public variable,
> > documented in the ELisp manual.  And the name should also end with -p,
> > since it's a boolean.  How about completion-filter-completions-format-p?
> 
> (As I understood the style guide '-p' is not a good idea for boolean
> variables, since a value is not a predicate in a strict sense.)
> 
> To address your technical comment - this variable is precisely what one
> of the technical difficulties mentioned in my other mail is about.  The
> question is how we can retain backward compatibility of the completion
> style 'all' functions, e.g., 'completion-basic-all-completions', while
> still allowing the function to return the newly introduced alist format
> with more data, which enables 'completion-filter-completions' to perform
> the efficient deferred highlighting.

I understand, but given that we provide this for other packages, it
shouldn't be an internal variable.

> > Also, the "This function has been superseded..." part should be a new
> > paragraph, so that it stands out.  (And I'm not yet sure we indeed
> > want to say "superseded" here, but that's part of the on-going
> > discussion.  maybe use a more neutral language here, like "See also".)
> 
> The new API 'completion-filter-completions' will substitute the existing
> API 'completion-all-completions'.

That's your hope, and I understand.  But we as a project didn't yet
decide to deprecate the original APIs, so talking about superseding is
premature.

> > Is "filter" really the right word here (in the doc string)?  "Filer"
> > means you take a sequence and produce another sequence with some
> > members removed.  That's not what this API does, is it?  Suggest to
> > use a different name, like completion-completions-alist or
> > completion-all-completions-as-alist.
> 
> "Filter" seems like exactly the right word to me.  The function takes a
> list of strings (or a completion table) and returns a subset of matching
> completion strings without further modifications to the strings. See
> above what I wrote about allocations.

But the name says "filter completions".  Which would mean you take a
list of completions and filter out some of them.  A completion table
is much more general object than a list of strings.  Thus, I think
using "filter" here is sub-optimal.

> >> +Only the elements of table that satisfy predicate PRED are considered.
> >> +POINT is the position of point within STRING.  The METADATA may be
> >> +modified by the completion style.  The return value is a alist with
> >> +the keys:
> >> +
> >> +- base: Base position of the completion (from the start of STRING)
> > 
> > "Base" here means the beginning?  If so, why not call it "beg" or
> > somesuch?
> 
> Base position is a fixed term which is already used in minibuffer.el for
> completions.  See also 'completion-base-position' for example.

Well, we don't have to keep bad habits indefinitely.  It's okay to
lose them and use better terminology.  Or at least to explain that
terminology in parentheses the first time it is used in some context.

> > Are we really losing the completion-score property here?  If so, why?
> 
> Yes, the property is removed in the current patch.  It is not actually
> used for anything in the new implementation.  But it is possible to
> restore the property such that 'completion-all-completions' always
> returns scored candidates as it does now.  See my other mail regarding
> the caveats of the current patch.

I'd prefer not to lose existing features, because that'd potentially
make the changes backward-incompatible.

Thanks.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 14 Aug 2021 03:11:36 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Aug 13 23:11:36 2021
Received: from localhost ([127.0.0.1]:43392 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mEk5A-0005ye-Bl
	for submit <at> debbugs.gnu.org; Fri, 13 Aug 2021 23:11:36 -0400
Received: from mail-wr1-f48.google.com ([209.85.221.48]:40583)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <raaahh@HIDDEN>)
 id 1mEk58-0005yJ-2K; Fri, 13 Aug 2021 23:11:34 -0400
Received: by mail-wr1-f48.google.com with SMTP id k29so15760049wrd.7;
 Fri, 13 Aug 2021 20:11:34 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=sender:subject:to:cc:references:from:message-id:date:user-agent
 :mime-version:in-reply-to:content-language:content-transfer-encoding;
 bh=8qJF3v8JbwJAZ15jhDdtLSeHlHSTougNt1Um+Emevgs=;
 b=uhb3Yu2G/KwF7e23ESrN4j/Fh07Y51CsdrtrHwzY2W3mXnf2hTJ+nyFKDVb1cB09t0
 utWlFJEmCtPOi1lSSISjfuadcx1seTUW8FNXGAaXP5GYHuCrgxhEkyoeckULXB6hYw1d
 hcMvXWlpglvE1U6x6dBxhqqoIqspz8DoK5OFzxfkWH+VOFzUb6sncRrvYWMd8FcL3riD
 LYupotBinwmXxXkps3k3UKa+IArjNeeQ2/uLlBOSckA5X0FT+wanYvp8tuwcWDbT4VW9
 Php/VN8Fu0s0SkbB3WNC4779V6VFZgYoovzxA2+33u//UtCPTPUdPX4iszORMtaq10TC
 tZrQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:sender:subject:to:cc:references:from:message-id
 :date:user-agent:mime-version:in-reply-to:content-language
 :content-transfer-encoding;
 bh=8qJF3v8JbwJAZ15jhDdtLSeHlHSTougNt1Um+Emevgs=;
 b=WQ1H4EbH4EBesGEw0wHtd400mwrs92osOVMjjQS6bKOq1DgBNSZ1KkttUCw5401CLQ
 W2sTz3+tIB4ClHZw3QIM0lOgleBfPRJZ17yei489eI/448D/YnGgO0Y6HP6Paf7IJ6M6
 Lboe/kXXMTljbtRK+JZxXA1PSil9uyzBqkb3JW8v+rIT5zkBLXuNNQBgDZx25U02BJ/u
 AMWT+SPigIwEnyx+XJ1o4F5utwEQMZNt0PIFtiqn6OLArZv/0cGwPwpDqw3O1hqtCYby
 55S8SrGaju4iC2mjMmWQJqEdJE79D7Ty+UfdacCoq9/4WH5H7GJA+8/8e1vlSIgL+GUl
 OVyg==
X-Gm-Message-State: AOAM530YfUQaXZJZVwpeDnHEmo8GDF4SVgH5Gk3XV9KcnbXOKmbabtL2
 sVBNDKeYQ//8MXSUpsICGxxQgiUBchc=
X-Google-Smtp-Source: ABdhPJxsO782oJW6/wOSeAg58zlmFyQfHKUIV+YHUBe0p3NUhjFYEMpSgTKZfhOE8W+l2fTU/YsL6Q==
X-Received: by 2002:a5d:42c2:: with SMTP id t2mr6137986wrr.49.1628910688214;
 Fri, 13 Aug 2021 20:11:28 -0700 (PDT)
Received: from [192.168.0.6] ([46.251.119.176])
 by smtp.googlemail.com with ESMTPSA id z126sm3235705wmc.11.2021.08.13.20.11.27
 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);
 Fri, 13 Aug 2021 20:11:27 -0700 (PDT)
Subject: Re: bug#48841: [PATCH] Add new `completion-filter-completions` API
 and deferred highlighting
To: Daniel Mendler <mail@HIDDEN>,
 "emacs-devel@HIDDEN" <emacs-devel@HIDDEN>
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <f8c4419a-7e83-b8a9-347c-0221a2052376@HIDDEN>
From: Dmitry Gutov <dgutov@HIDDEN>
Message-ID: <5dba40c7-681b-392d-486a-ae71090d27f4@HIDDEN>
Date: Sat, 14 Aug 2021 06:11:26 +0300
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.11.0
MIME-Version: 1.0
In-Reply-To: <f8c4419a-7e83-b8a9-347c-0221a2052376@HIDDEN>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
X-Spam-Score: 0.4 (/)
X-Debbugs-Envelope-To: 47711
Cc: =?UTF-8?B?Sm/Do28gVMOhdm9yYQ==?= <joaotavora@HIDDEN>,
 Stefan Monnier <monnier@HIDDEN>, 48841 <at> debbugs.gnu.org,
 47711 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.6 (/)

Hi Daniel,

I haven't yet read the patch in detail, but it sounds like a move in the 
right direction (even if it doesn't include the long-overdue overhaul of 
the whole API).

A few notes on the new stuff:

 > Finally the
`highlight` value is a function taking a list of completion strings
and returns a new list of new strings with highlighting applied.

First of all, I'd really like it to be a function that applies to 
individual completion strings, not the whole collection. That would make 
it much easier to use in company-capf without having to rewrite a lot of 
code in the presentation layer.

Second, perhaps instead of modifying the strings themselves it could 
return some data (like a list of faces-intervals tuples) that could be 
used to do so?

Again, in company-capf's we end up parsing the face properties in the 
string, so those modifications are just extra work for CPU which we 
could eliminate.

This is less critical, though.

On 11.08.2021 19:11, Daniel Mendler wrote:
> There are currently two issues with the patch with regards to backward
> compatibility. Fortunately they are fixable with a little effort.
> 
> 1. I would like to deprecate `completion-score' or remove it altogether,
>     but unfortunately `completion-score' is used in the wild. In order to
>     preserve `completion-score', bind `completion--filter-completions' in
>     the highlighting functions. Add `completion-score' in
>     `completion-pcm--hilit-commonality' when
>     `completion--filter-completions' is nil.

And third: I think completion-score could ultimately use the same 
treatment as 'highlight'. Meaning, being returned up the stack together 
with completions, so other bits of code could look up those values.

I don't have a clear picture of this yet, but see the recently filed 
bug#49888. If we want to be able to combine matching scores with recency 
scores, simply sorting the completions after matching is not going to 
cut it.

Not sure if this is something we can tackle now, but keeping this 
possible evolution in mind could help us make good choices in the 
current migration.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 14 Aug 2021 02:55:27 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Aug 13 22:55:27 2021
Received: from localhost ([127.0.0.1]:43376 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mEjpW-0005Zq-Vc
	for submit <at> debbugs.gnu.org; Fri, 13 Aug 2021 22:55:27 -0400
Received: from mail-wr1-f45.google.com ([209.85.221.45]:41909)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <raaahh@HIDDEN>)
 id 1mEjpV-0005ZW-9A; Fri, 13 Aug 2021 22:55:25 -0400
Received: by mail-wr1-f45.google.com with SMTP id x10so9434671wrt.8;
 Fri, 13 Aug 2021 19:55:25 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=sender:subject:to:cc:references:from:message-id:date:user-agent
 :mime-version:in-reply-to:content-language:content-transfer-encoding;
 bh=RrMsSouFaapPEAk74ymKj85jn+NHe8o5ot//bYK6Y7U=;
 b=COd7ecBI7d9wbKFTROI+mvyr0iWMzOONcICvbmCVXLgz+ledAdeUe7APuGkKWgY+Jd
 cKsB6Jw6eN/FU+3FG1D22/HfwNJ37eVjpsMDWYBDzOFQ33B+gnUrNiWG7HAPSPBVy+TD
 LIFLJl9g1H6dWbm0aNq9GbrEMbmydt6KJElf76D0Y0ryFayy2XtAhagdDsb82l7IM0Gv
 AXgGm9CDZUH2mKmqSHTwx0ylFJGJ6STudzscB+U0ktueqX+C9ZE/CBlb8qtoKEikHWHf
 5Gig5+TFtb6dIWj1juymBsPFdhWkzWF3UrFHfF+Squ/YfPDI9z/3SpouUUAv8unDrq2J
 2rqg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:sender:subject:to:cc:references:from:message-id
 :date:user-agent:mime-version:in-reply-to:content-language
 :content-transfer-encoding;
 bh=RrMsSouFaapPEAk74ymKj85jn+NHe8o5ot//bYK6Y7U=;
 b=H8t6MI28Sl2fTK2O/up2bWmS2nA5jTdWxCXMHw/hgaQ8CJ6IjlrgyieaUq18wfzWaP
 ormbF0YU/riADgVwyVjUdLGqLVopAtKUIudbrwehAa/era4Ni968D0EdyYJEViHr+kMY
 xI7UEMU4t37EeV7+U2HFf03vpDFyKHoeB5H0zRIbikpvCjBIp4+iS3/kO9q7ySMDW6Er
 N36ABqbcjdD7heNtjBHKRsLSyO5mkUkr8hxCJe/cVkG1brBWeD21cdduhgd2+KDOIQtg
 M/RJTlyur1ts58M5pHNttNRB8Q5M0j0hLy6kTHRfYeJw1VQQYb/EyckDzT9cCXFkvrJh
 4CqQ==
X-Gm-Message-State: AOAM533+KR1BiRY9V2JtmxhPQVfKusq2NE2SrPcn9jFrhGB1VypJuezz
 xcLAqjo8jnh1E/Qz3ZjR+4Hk+9kaqD0=
X-Google-Smtp-Source: ABdhPJxpWonVHMT/qpjjgzVsfYxYiCwu8gMPFaSBLngN8O5F+J1FfE4DFsuty8mw7DKIAbbIYwXJGw==
X-Received: by 2002:adf:9c8c:: with SMTP id d12mr6159409wre.71.1628909719415; 
 Fri, 13 Aug 2021 19:55:19 -0700 (PDT)
Received: from [192.168.0.6] ([46.251.119.176])
 by smtp.googlemail.com with ESMTPSA id i21sm3353014wrb.62.2021.08.13.19.55.18
 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);
 Fri, 13 Aug 2021 19:55:19 -0700 (PDT)
Subject: Re: bug#48841: bug#47711: [PATCH VERSION 2] Add new
 `completion-filter-completions` API and deferred highlighting
To: =?UTF-8?B?Sm/Do28gVMOhdm9yYQ==?= <joaotavora@HIDDEN>,
 Daniel Mendler <mail@HIDDEN>
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <f8c4419a-7e83-b8a9-347c-0221a2052376@HIDDEN>
 <fff6e291-bbcc-93e3-92e7-00af4664b8d4@HIDDEN>
 <3d3f894f-a6fa-53ae-5d50-c3aa9bffc73e@HIDDEN>
 <CALDnm53xh6boc6T9XSC4jZpkCPS3RB8JFNb12BpkoeGYg3Q9RQ@HIDDEN>
 <56ab18b1-4348-9b2c-85bb-af9b76cd429a@HIDDEN>
 <CALDnm52XM+S07OPXO3qtfLN4pup-kT2U=WRO9dUo3tv9uYJWXw@HIDDEN>
From: Dmitry Gutov <dgutov@HIDDEN>
Message-ID: <328f87eb-6474-1442-e1ca-9ae8deb2a84a@HIDDEN>
Date: Sat, 14 Aug 2021 05:55:17 +0300
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.11.0
MIME-Version: 1.0
In-Reply-To: <CALDnm52XM+S07OPXO3qtfLN4pup-kT2U=WRO9dUo3tv9uYJWXw@HIDDEN>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 8bit
X-Spam-Score: 0.4 (/)
X-Debbugs-Envelope-To: 47711
Cc: Stefan Monnier <monnier@HIDDEN>, 48841 <at> debbugs.gnu.org,
 47711 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.6 (/)

Aside from the mutability/ownership issue,

On 13.08.2021 15:05, João Távora wrote:
> If one removes these lines, the process becomes much faster, but there is a
> problem with highlighting.  My idea is indeed to defer highlighting by not
> setting the 'face property directly on that shared string, but some
> other property
> that is read later from the shared string by compliant frontents.

I haven't done any direct benchmarking, but I'm pretty sure that this 
approach cannot, by definition, be as fast as the non-mutating one.

Because you go through the whole list and mutate all of its elements, 
you have to perform a certain bit of work W x N times, where N is the 
size of the whole list.

Whereas the deferred-mutation approach will only have to do its bit 
(which is likely more work, like, WWW) only 20 times, or 100 times, or 
however many completions are displayed. And this is usually negligible.

However big the difference is going to be, I can't say in advance, of 
course, or whether it's going to be shadowed by some other performance 
costs. But the non-mutating approach should have the best optimization 
potential when the list is long.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 14 Aug 2021 02:47:54 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Aug 13 22:47:54 2021
Received: from localhost ([127.0.0.1]:43369 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mEjiD-0005Oo-Rr
	for submit <at> debbugs.gnu.org; Fri, 13 Aug 2021 22:47:54 -0400
Received: from mail-wm1-f45.google.com ([209.85.128.45]:40490)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <raaahh@HIDDEN>)
 id 1mEjiC-0005OW-1a; Fri, 13 Aug 2021 22:47:52 -0400
Received: by mail-wm1-f45.google.com with SMTP id
 f12-20020a05600c4e8c00b002e6bdd6ffe2so5160108wmq.5; 
 Fri, 13 Aug 2021 19:47:51 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=sender:subject:to:cc:references:from:message-id:date:user-agent
 :mime-version:in-reply-to:content-language:content-transfer-encoding;
 bh=YJw6OlV4lASsh/tUTN6SzIwACHP933YFdLTFK0MkuqM=;
 b=Vidv7X/VQNIy18MEfGR9+nO2HYx3p9uH5yds3+TK7an4SST2+36hNaFlToeGos1yzU
 ZXPEaOpJzr5nAm2atp8TRLaxcrJx59uNbGTcfk7ElXQLM6craANPxqMRQ/TJbw12OkVS
 lC0+2s9E6OgAmxOnj8t2Qhui612cejdqTiSIhupU9/FRSTtmV0qUYMJmda0IWsImEzbn
 DKsFPCnqf7m0sx09tPSKXKlkBvncG41XVJ6Cq3K5sANAomQJa8sH4P9GAr6+2fVnmIX9
 w8xO/J7VMNT6j80LLgl56LJNAtV57ifHaDToXoKmypw3CPOhSfbrl6x7N0bZIfhRQ7xR
 2TFg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:sender:subject:to:cc:references:from:message-id
 :date:user-agent:mime-version:in-reply-to:content-language
 :content-transfer-encoding;
 bh=YJw6OlV4lASsh/tUTN6SzIwACHP933YFdLTFK0MkuqM=;
 b=nR1zS8Nn309e6wVeitw1G4GtSPeEcG9GrIwjnVhWO70ig2C6FOM97NZkOHB4MyExCs
 X/DPOTnUIk77LeUS5ETxtYZRizhEKjozvMkKAiWOH2VOxkBTWoJpsbeZaq+BtGp+eKeb
 hWvDeoVOCZ4faOBObfh9j2PgNAW6clgferd+ak0/cx8HUE1Hg706HUK32wirZWauT4zm
 kXzquiRez+FniGAL7lf5pkd26FbkrmgB03mTjrER/KDSwJUHGzKVEOdhV5LJ1EvlTmox
 JC592hdQW9AUDUNbVJncKNQoJ7tR8XDlj9TeiZ/OFnLqvYBmYDc5LvJDilhju+ZeA8n/
 yhLw==
X-Gm-Message-State: AOAM531g56FoYOaYmEL8muZY8+YnZJw7nkYCU37bqilNQgOB1iu8oIO1
 4oK+xKI7C2XpW2L5of8woHtOyR7P9XU=
X-Google-Smtp-Source: ABdhPJzxplFEaS0USQ1x53CqyR9NU8Z7vLpfrynY2I7Tg+tlEzk6uUD8ELyaLrcjfBvTR/qCtD+o/w==
X-Received: by 2002:a1c:a743:: with SMTP id q64mr5205240wme.74.1628909266069; 
 Fri, 13 Aug 2021 19:47:46 -0700 (PDT)
Received: from [192.168.0.6] ([46.251.119.176])
 by smtp.googlemail.com with ESMTPSA id l2sm3072055wme.28.2021.08.13.19.47.44
 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);
 Fri, 13 Aug 2021 19:47:45 -0700 (PDT)
Subject: Re: bug#47711: [PATCH VERSION 2] Add new
 `completion-filter-completions` API and deferred highlighting
To: =?UTF-8?B?Sm/Do28gVMOhdm9yYQ==?= <joaotavora@HIDDEN>,
 Daniel Mendler <mail@HIDDEN>
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <f8c4419a-7e83-b8a9-347c-0221a2052376@HIDDEN>
 <fff6e291-bbcc-93e3-92e7-00af4664b8d4@HIDDEN>
 <3d3f894f-a6fa-53ae-5d50-c3aa9bffc73e@HIDDEN>
 <CALDnm53xh6boc6T9XSC4jZpkCPS3RB8JFNb12BpkoeGYg3Q9RQ@HIDDEN>
 <56ab18b1-4348-9b2c-85bb-af9b76cd429a@HIDDEN>
 <CALDnm52XM+S07OPXO3qtfLN4pup-kT2U=WRO9dUo3tv9uYJWXw@HIDDEN>
 <38a06f3c-4a7a-755c-c99b-708f91afabfa@HIDDEN>
 <CALDnm537=LHAoaGqPpy7=JOPsYn=z+=Q6QW=b9bjrAWOtxqRaQ@HIDDEN>
 <9f59f87c-2489-aaa0-5b3f-0e911b7ffa6c@HIDDEN>
 <CALDnm508EK5qM+AaOOJiNDm-=ORCuOieYpkjO=OSRas+_WTFwQ@HIDDEN>
From: Dmitry Gutov <dgutov@HIDDEN>
Message-ID: <8a36e61a-1c5b-bf3b-a454-077348589c4f@HIDDEN>
Date: Sat, 14 Aug 2021 05:47:43 +0300
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.11.0
MIME-Version: 1.0
In-Reply-To: <CALDnm508EK5qM+AaOOJiNDm-=ORCuOieYpkjO=OSRas+_WTFwQ@HIDDEN>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 8bit
X-Spam-Score: 0.4 (/)
X-Debbugs-Envelope-To: 47711
Cc: Stefan Monnier <monnier@HIDDEN>, 48841 <at> debbugs.gnu.org,
 47711 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.6 (/)

Hi folks,

Sorry I'm late to this party.

On 13.08.2021 16:36, João Távora wrote:
>>>> By separating the filtering and mutation
>>>> (highlighting, scoring) my patch addresses the problem at hand in the
>>>> proper way.
>>>> [ ... ]
>>>> Mutation would be a reasonable choice here if the problem could not be
>>>> solved in a proper way.  But in fact it can be solved in a proper way
>>>> without mutating the strings at all as my patch shows.
>>> "proper" is just an reasonably empty adjective.  There are different ways to
>>> go about this, of course.  What's "proper" and isn't is hard to debate
>>> objectively.
>> You are contradicting yourself here. You agree that string mutation is
>> better be avoid. If we define "proper" as avoids string mutation if this
>> is easily possible, then my patch implements a proper solution to the> problem.
> I didn't say it's better avoided, though of course I will avoid_any_  change if
> I can. I said I have identified one drawback with doing it.  Then I
> have addressed
> that drawback. So that's what I said.
> 
> I am unaware of_other_  drawbacks.  They might exist, but I am unaware of
> them.  Perhaps you are, and indeed you state they exist, but you refuse to
> let me know about them.  Or perhaps others know of them and will let me know.
> In my long-running discussion with Dmitry they were not presented (again,
> except for the one I identified).

I thought I explained the problem with this previously.

It's basically this: we cannot mutate what we don't own. Across all of 
completion functions out there, there will be such that return "shared" 
strings (meaning, not copied or newly allocated) from their completion 
tables. And modifying them is bad, with consequences which can present 
themselves in unexpected, often subtle ways.

Since up until now completion-pcm--hilit-commonality copied all strings 
before modifying, completion tables such as described (with "shared" 
strings) have all been "legal". Suddenly deciding to stop supporting 
them would be a major API breakage with consequences that are hard to 
predict. And while I perhaps agree that it's an inconvenience, I don't 
think it's a choice we can simply make as this stage in c-a-p-f's 
development.

To give an example of a subtle consequence:

   1. (setq s (symbol-name 'car))

   2. (put-text-property 1 3 'face 'error s)

   3. Switch to a buffer in fundamental mode

   4. (insert (symbol-name 'car)) --> see the error face in the buffer

Now imagine that some completion table collects symbol names by passing 
obarray through #'symbol-name rather than #'all-completions, and voila, 
if the completion machinery adds properties (any properties, not just 
face) to those strings, you have just modified a bunch of global values. 
That's not good.

And in the example above, the values are those that the 
lispref/objects.texi says we should not change (though it gives 
(symbol-name 'cons) as example). "Not mutable", in its parlance. IIRC 
the related discussions mentioned that modifying such values could lead 
to a segfault in some previous Emacs versions. Maybe not anymore, but 
it's still not a good idea.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 13 Aug 2021 14:37:52 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Aug 13 10:37:52 2021
Received: from localhost ([127.0.0.1]:42852 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mEYJk-0008N8-0X
	for submit <at> debbugs.gnu.org; Fri, 13 Aug 2021 10:37:52 -0400
Received: from server.qxqx.de ([178.63.65.180]:42485 helo=mail.qxqx.de)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mail@HIDDEN>)
 id 1mEYJh-0008Mn-4w; Fri, 13 Aug 2021 10:37:50 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=qxqx.de;
 s=mail1392553390; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:
 MIME-Version:Date:Message-ID:From:References:Cc:To:Subject:Sender:Reply-To:
 Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender:
 Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:
 List-Subscribe:List-Post:List-Owner:List-Archive;
 bh=xG1wTRTLJhDNwXgVcJFmihoxCZwFJ4j6Wekup9FM4LM=; b=w8l5uwURr2DI8wfigMtIRnxeqk
 XJjqcryeSPLognhKMP8t7idXRiybmfBIhfmDcAk0T0QQZ/gs58g/Ava2RVZMTKDabQ26uPZdz8zXQ
 5v9yaXw+fkdL9H2JxFxGERF90chAe7wGIpHbtp8F/7pcfLg1OBsYao8/z/z/4FYZ5b4Q=;
Subject: Re: bug#48841: bug#47711: [PATCH VERSION 2] Add new
 `completion-filter-completions` API and deferred highlighting
To: =?UTF-8?B?Sm/Do28gVMOhdm9yYQ==?= <joaotavora@HIDDEN>
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <f8c4419a-7e83-b8a9-347c-0221a2052376@HIDDEN>
 <fff6e291-bbcc-93e3-92e7-00af4664b8d4@HIDDEN>
 <3d3f894f-a6fa-53ae-5d50-c3aa9bffc73e@HIDDEN>
 <CALDnm53xh6boc6T9XSC4jZpkCPS3RB8JFNb12BpkoeGYg3Q9RQ@HIDDEN>
 <56ab18b1-4348-9b2c-85bb-af9b76cd429a@HIDDEN>
 <CALDnm52XM+S07OPXO3qtfLN4pup-kT2U=WRO9dUo3tv9uYJWXw@HIDDEN>
 <38a06f3c-4a7a-755c-c99b-708f91afabfa@HIDDEN>
 <CALDnm537=LHAoaGqPpy7=JOPsYn=z+=Q6QW=b9bjrAWOtxqRaQ@HIDDEN>
 <9f59f87c-2489-aaa0-5b3f-0e911b7ffa6c@HIDDEN>
 <CALDnm508EK5qM+AaOOJiNDm-=ORCuOieYpkjO=OSRas+_WTFwQ@HIDDEN>
 <ea37de5b-909a-6467-a274-df193429dabd@HIDDEN>
 <CALDnm52eYiC1Hb2+Emj+_a27ODscCq=gM_s0hWY15332-mN2Dw@HIDDEN>
From: Daniel Mendler <mail@HIDDEN>
Message-ID: <61963a69-1b1c-a635-97dc-a665ecb07a6b@HIDDEN>
Date: Fri, 13 Aug 2021 16:37:38 +0200
MIME-Version: 1.0
In-Reply-To: <CALDnm52eYiC1Hb2+Emj+_a27ODscCq=gM_s0hWY15332-mN2Dw@HIDDEN>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 8bit
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 47711
Cc: Dmitry Gutov <dgutov@HIDDEN>, Stefan Monnier <monnier@HIDDEN>,
 48841 <at> debbugs.gnu.org, 47711 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)



On 8/13/21 4:11 PM, João Távora wrote:
> On Fri, Aug 13, 2021 at 3:03 PM Daniel Mendler <mail@HIDDEN> wrote:
> 
>>> Without facts to back it up, I have to take this as gratuitous disparagement.
>>> Nicht so gut.
>>
>> João, your whole answers are "nicht so gut" or not useful.  What is your
>> point?  Please give constructive technical feedback instead of such
>> empty phrases.
> 
> Look, you disparaged an idea of mine without absolutely any facts. I don't think
> that's good. "Nicht so gut" was a lighthearted way of pointing it out.
> Lighten up.
> Post the benchmarks you say you have and stop the pompous handwaving.

João, the way you argue is not in any way "lighthearted".  It also
depends on what the other party receives as the message.  And here you
just repeat this style by calling my reasoning "pompous handwaving".
This is not a fair way to discuss.  In contrast my arguments were
generally of a technical nature.  I propose we both calm down a bit and
let others chime in here.

Daniel





Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 13 Aug 2021 14:11:47 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Aug 13 10:11:47 2021
Received: from localhost ([127.0.0.1]:42799 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mEXuU-0005Xf-Q9
	for submit <at> debbugs.gnu.org; Fri, 13 Aug 2021 10:11:46 -0400
Received: from mail-pl1-f171.google.com ([209.85.214.171]:36497)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <joaotavora@HIDDEN>)
 id 1mEXuT-0005XO-29; Fri, 13 Aug 2021 10:11:45 -0400
Received: by mail-pl1-f171.google.com with SMTP id f3so12149386plg.3;
 Fri, 13 Aug 2021 07:11:45 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=mime-version:references:in-reply-to:from:date:message-id:subject:to
 :cc:content-transfer-encoding;
 bh=eUEMG04mECiWu+Pg/59yRQTqKhWPV24mxZW+MGMR83g=;
 b=e0f+5n9o+qdDSb6F+N45qk8yKvmMSb2FY0vmM6rOqaQ9Of2rVABsOw2yLQmY1B55RM
 xHfmICGG/kxNCAEieMkU0QAsM2Unu1ed14G1om4axInaVgwLGIez0yHGwSNFkGRT70ga
 1QCFtv5Q+j66S1hQoWh+ZQNoJ0y9xHKXSvisZUZRaavrATVjtkF3TBk14aXrWLqtoBIK
 mwBpxpbYWzOaaONZjQAbAQhHMD8sKQ9BNiRNVLu7Iz/bpPh04kSvZSXzYE96MdokBnl2
 zUY1ALHN61ayUWTMkkqancqmZ/RgHXM9RWdgt+DrbRzKSX17hvauxv+OrR19KcmSUQhx
 2mgw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:mime-version:references:in-reply-to:from:date
 :message-id:subject:to:cc:content-transfer-encoding;
 bh=eUEMG04mECiWu+Pg/59yRQTqKhWPV24mxZW+MGMR83g=;
 b=J/9AhrCpLYmciMdutRi9PDAJajvqL/gilhZdFhaXGr2ADnMBmc9qEAH198VozeP1v4
 6XTttn6CRnlPbnKg9UGrAmz9HmSkWUyxTBYkhaK8IIpBsEoNm0FIgEmyt3SR4ZqfLgHq
 NV32ZrN7xropCfI0xmSz3mitrTJkQ/DMKN5ZzDXkXxvmy6pRIckUTIQ7LWS1EqfHnyRi
 tAQmT/novzlG9cHZQ5o8HTUWNUuQSclIbEmVplEe0C37yi3vYK6uvyG+ToUK0FjTNGKT
 ezQ1Xl2R/vxcQv3/U1tsOwAPESkhwj+IKzb1KIQWJ3MkiV+RcfeeNtxtON8zYOelSdLG
 VAyw==
X-Gm-Message-State: AOAM532ALRMS4yIKbKle8NmtPEbllZ4lSqVlD8eB8GWn5LVcVqSbmMv6
 s4N9pJUY/0XfYgNwZDAdhiccWf19JhMtTsg1jsM=
X-Google-Smtp-Source: ABdhPJwpc22f6+4Nexy2mgyEQcZGioFJ083PVKOOgX27jXzg9BK7q63Zr5o/xaX4+lJyfg9OXQ4JKsxOZ5QXUcxfbR8=
X-Received: by 2002:a17:90a:a091:: with SMTP id
 r17mr2830007pjp.56.1628863899134; 
 Fri, 13 Aug 2021 07:11:39 -0700 (PDT)
MIME-Version: 1.0
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <f8c4419a-7e83-b8a9-347c-0221a2052376@HIDDEN>
 <fff6e291-bbcc-93e3-92e7-00af4664b8d4@HIDDEN>
 <3d3f894f-a6fa-53ae-5d50-c3aa9bffc73e@HIDDEN>
 <CALDnm53xh6boc6T9XSC4jZpkCPS3RB8JFNb12BpkoeGYg3Q9RQ@HIDDEN>
 <56ab18b1-4348-9b2c-85bb-af9b76cd429a@HIDDEN>
 <CALDnm52XM+S07OPXO3qtfLN4pup-kT2U=WRO9dUo3tv9uYJWXw@HIDDEN>
 <38a06f3c-4a7a-755c-c99b-708f91afabfa@HIDDEN>
 <CALDnm537=LHAoaGqPpy7=JOPsYn=z+=Q6QW=b9bjrAWOtxqRaQ@HIDDEN>
 <9f59f87c-2489-aaa0-5b3f-0e911b7ffa6c@HIDDEN>
 <CALDnm508EK5qM+AaOOJiNDm-=ORCuOieYpkjO=OSRas+_WTFwQ@HIDDEN>
 <ea37de5b-909a-6467-a274-df193429dabd@HIDDEN>
In-Reply-To: <ea37de5b-909a-6467-a274-df193429dabd@HIDDEN>
From: =?UTF-8?B?Sm/Do28gVMOhdm9yYQ==?= <joaotavora@HIDDEN>
Date: Fri, 13 Aug 2021 15:11:27 +0100
Message-ID: <CALDnm52eYiC1Hb2+Emj+_a27ODscCq=gM_s0hWY15332-mN2Dw@HIDDEN>
Subject: Re: bug#47711: [PATCH VERSION 2] Add new
 `completion-filter-completions` API and deferred highlighting
To: Daniel Mendler <mail@HIDDEN>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 47711
Cc: Dmitry Gutov <dgutov@HIDDEN>, Stefan Monnier <monnier@HIDDEN>,
 48841 <at> debbugs.gnu.org, 47711 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

On Fri, Aug 13, 2021 at 3:03 PM Daniel Mendler <mail@HIDDEN> wro=
te:

> > Without facts to back it up, I have to take this as gratuitous disparag=
ement.
> > Nicht so gut.
>
> Jo=C3=A3o, your whole answers are "nicht so gut" or not useful.  What is =
your
> point?  Please give constructive technical feedback instead of such
> empty phrases.

Look, you disparaged an idea of mine without absolutely any facts. I don't =
think
that's good. "Nicht so gut" was a lighthearted way of pointing it out.
Lighten up.
Post the benchmarks you say you have and stop the pompous handwaving.

Bye,
Jo=C3=A3o




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 13 Aug 2021 14:03:44 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Aug 13 10:03:44 2021
Received: from localhost ([127.0.0.1]:42776 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mEXme-0005KK-Tz
	for submit <at> debbugs.gnu.org; Fri, 13 Aug 2021 10:03:44 -0400
Received: from server.qxqx.de ([178.63.65.180]:52349 helo=mail.qxqx.de)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mail@HIDDEN>)
 id 1mEXmV-0005Jq-Ns; Fri, 13 Aug 2021 10:03:36 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=qxqx.de;
 s=mail1392553390; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:
 MIME-Version:Date:Message-ID:From:References:Cc:To:Subject:Sender:Reply-To:
 Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender:
 Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:
 List-Subscribe:List-Post:List-Owner:List-Archive;
 bh=U+7jHgcxqSUV0RHeOu3QFSoRtGe0iLwvU6TC36yrcVI=; b=EcPIMUnEp4uE8O01Mqno6vFAsl
 rUQgFBQ6At+ydR0XwESZnD/Q1mhooK6GNCJmS6WcUPXOPRR//qIGlTuFIPoxXHXt5ps4KrGh8Fs0W
 muuQwa8s1CF0tqEG9JJpzW/GomDGoT7mDRug2LZ73hg4xE0AWTjiLc32Ot2ecxPYC9EI=;
Subject: Re: bug#47711: [PATCH VERSION 2] Add new
 `completion-filter-completions` API and deferred highlighting
To: =?UTF-8?B?Sm/Do28gVMOhdm9yYQ==?= <joaotavora@HIDDEN>
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <f8c4419a-7e83-b8a9-347c-0221a2052376@HIDDEN>
 <fff6e291-bbcc-93e3-92e7-00af4664b8d4@HIDDEN>
 <3d3f894f-a6fa-53ae-5d50-c3aa9bffc73e@HIDDEN>
 <CALDnm53xh6boc6T9XSC4jZpkCPS3RB8JFNb12BpkoeGYg3Q9RQ@HIDDEN>
 <56ab18b1-4348-9b2c-85bb-af9b76cd429a@HIDDEN>
 <CALDnm52XM+S07OPXO3qtfLN4pup-kT2U=WRO9dUo3tv9uYJWXw@HIDDEN>
 <38a06f3c-4a7a-755c-c99b-708f91afabfa@HIDDEN>
 <CALDnm537=LHAoaGqPpy7=JOPsYn=z+=Q6QW=b9bjrAWOtxqRaQ@HIDDEN>
 <9f59f87c-2489-aaa0-5b3f-0e911b7ffa6c@HIDDEN>
 <CALDnm508EK5qM+AaOOJiNDm-=ORCuOieYpkjO=OSRas+_WTFwQ@HIDDEN>
From: Daniel Mendler <mail@HIDDEN>
Message-ID: <ea37de5b-909a-6467-a274-df193429dabd@HIDDEN>
Date: Fri, 13 Aug 2021 16:03:21 +0200
MIME-Version: 1.0
In-Reply-To: <CALDnm508EK5qM+AaOOJiNDm-=ORCuOieYpkjO=OSRas+_WTFwQ@HIDDEN>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 8bit
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 47711
Cc: Dmitry Gutov <dgutov@HIDDEN>, Stefan Monnier <monnier@HIDDEN>,
 48841 <at> debbugs.gnu.org, 47711 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

On 8/13/21 3:36 PM, João Távora wrote:
>> You are contradicting yourself here. You agree that string mutation is
>> better be avoid. If we define "proper" as avoids string mutation if this
>> is easily possible, then my patch implements a proper solution to the> problem.
> 
> I didn't say it's better avoided, though of course I will avoid _any_ change if
> I can. I said I have identified one drawback with doing it.  Then I
> have addressed
> that drawback. So that's what I said.
> 
> I am unaware of _other_ drawbacks.  They might exist, but I am unaware of
> them.  Perhaps you are, and indeed you state they exist, but you refuse to
> let me know about them.  Or perhaps others know of them and will let me know.
> In my long-running discussion with Dmitry they were not presented (again,
> except for the one I identified).

In the discussion with Dmitry, I already pointed out that there is an
alternative principled approach implemented by my Vertico UI, which is
in fact the same approach as implemented in this patch.

If there are other useful conclusions from the discussion I will adopt
them here for this patch.

>>> That's for sure.  My patch idea addresses only that single problem.
>>> I think this is a good property of patches: to solve one thing, not many.
>> No, this is not necessarily true.  This is only good if the problem is
>> solved in a way which is future proof.
> 
> OK, but what thing of the future, real or academic, do you envision that
> would bring back the problem, or create other problems?
> 
>> The idea of mutating the strings is a hack and not a solution.
> 
> Without facts to back it up, I have to take this as gratuitous disparagement.
> Nicht so gut.

João, your whole answers are "nicht so gut" or not useful.  What is your
point?  Please give constructive technical feedback instead of such
empty phrases.

>> In contrast, I am presenting a
>> future-proof new API as solution which addresses multiple problems.
> 
> That's the issue.  The completion system is very complex and there are many
> good ideas, different, floated by many people. But if you make a patch to
> address "multiple" fuzzily-described problems, it's hard to judge how good
> your ideas even are! Maybe they are indeed very good,  I never said
> they weren't.  No need to get worked up about it!
> 
> Again, my proposal is to first focus on the performance problems caused by
> string allocation.  _That_ problem is well understood, at least by me (but it
> would help to settle on convenient benchmarks understood by others, too).
> Then we can go from there.

No, it is not the correct approach to fix larger issues by applying
localized patches.  We both have identified the string allocations and
highlighting as problem.  My patch resolves the problem, by exposing
just the right pieces of the already existing completion machinery. More
about this below.

>> you look at the patch, only 196 new lines are added to minibuffer.el.
>> Furthermore the patch adds 213 lines of new tests.
>
> It's a large patch, over 1000 lines.  One does not review a patch
> merely by looking at
> lines added, when one needs to read much more, to understand implications, etc.
> It needs documentation, for one, much more than just docstrings, on
> how to use the
> new API.

I suggest you take a step back here and try to understand the high-level
idea first.  It seems that you are misjudging the complexity of the
patch.  The minibuffer completion machinery is already constructed such
that filtering and highlighting are separate.

If you look at `completion-basic-all-completions` for example, there is
first a filtering step and then the highlighting is applied in a second
step by the function `completion-hilit-commonality`.  This separation
exists for all completion styles.

My patch does nothing else than separating these two processing steps.
The new API `completion-filter-completions` returns the filtered list
and a function to apply highlighting afterwards only to the actually
displayed candidates where highlighting is needed.

In contrast your idea totally misses this.

> Maybe others review patches on other aspects that's fine.   Maybe
> others will. Eli reviewed on minor formatting and documentation aspects.

I am looking forward to more reviews by other people.

Your desire for benchmarks is understandable, but I doubt that it will
lead to progress in the discussion here and I doubt that it will
convince you.

The outcome of the benchmark is the following - my patch only filters
and does not mutate the strings, so it will be slightly faster than your
idea where the strings are mutated first and afterwards the mutation has
to be undone again.  However the mutations are of course not expensive,
so the differences will be small.  The discussion we should be having
here is about technical details and internals and not about the numbers
which won't give any guidance in this case regarding the correct API design.

You seem to always come back to the "scientific method". Note that there
is not only statistics, there is only "scientific reasoning" and
mathematics, which allows to reason about transformations and drawing
conclusions from that.  If you don't do this, you are only doing half of
the science.

Daniel




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 13 Aug 2021 13:36:42 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Aug 13 09:36:42 2021
Received: from localhost ([127.0.0.1]:40915 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mEXMX-0008CF-SH
	for submit <at> debbugs.gnu.org; Fri, 13 Aug 2021 09:36:42 -0400
Received: from mail-pl1-f179.google.com ([209.85.214.179]:43911)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <joaotavora@HIDDEN>)
 id 1mEXMV-0008Bw-1X; Fri, 13 Aug 2021 09:36:40 -0400
Received: by mail-pl1-f179.google.com with SMTP id e19so12007291pla.10;
 Fri, 13 Aug 2021 06:36:38 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=mime-version:references:in-reply-to:from:date:message-id:subject:to
 :cc:content-transfer-encoding;
 bh=tvhjGLLU6CrPIl2BruGEyiA2A8Y1z7e+RBKklFm3qaY=;
 b=WO6Qd/orW+L6sc7R1/ProxKfGtNTRSoLWOpPCBg2HsS4hNQN0IfFCM9vtuwahg7YSc
 y5j9j8J5pFb7pZXkCmI5hJO73LhiVAQA9zwBpVbTCh5w41gt7X/C7slXlMm8cxfhzZVk
 t78oUsxDQXpqMBzVTKenBQTnAQC5LAvIUruT4OBagHR7s35sQXBrrAwUqkjHjz6UcyYa
 uBdIxbgTQ/wP9xejKyFASJfY4Xlvv45tPw55SXPaqn3qXjJ6Q5j4VBnpfDA8GH+Cr+zQ
 PmvqBJHw0y+T5zUd5debsLzFCMKOAkYZQ+zl8/YuyaW0TUIAetq0n859qgTHbpcNY27E
 uEAQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:mime-version:references:in-reply-to:from:date
 :message-id:subject:to:cc:content-transfer-encoding;
 bh=tvhjGLLU6CrPIl2BruGEyiA2A8Y1z7e+RBKklFm3qaY=;
 b=nr+vEnLGKC33Ilcv1/hcPHgqXiNnUKl+mvSF94J3uwqYpp/Ne2P+cJzvzNwfZpw92k
 cdfkGO2BM96+oxkMY+QffGxZDEKJXBfG2p/BodDe+Xvl1MrZJMqXQWma0LRdtZ7Ey0GC
 Va2bOGXHhocqRmn4pGVW8HDObXeOsF2ZXRDZk7ECgiyjMb30j4eGOSJpjTwXF5CHQQ65
 HMzjz2e08rrA80Y7LAnOo9IjPlPzimx4FfJVbWaZi2Rfa1HunW5WdhJKVDqiJ8t5UjBP
 9P22quEBpM+96aq0KYdjgIK8MvYA7ISrKT80Ujv5wOWJwUhgDcjHNu2MS1ucGLO5gBi1
 1nsA==
X-Gm-Message-State: AOAM531aKxIK6KX97S4/wGTRppidvp8nrLKDrhYgnZ+xSMX6Vzu5Ulaz
 ZFeANigcxIyqNHQaoxKIf+cxQsigik5eI8TlwB8=
X-Google-Smtp-Source: ABdhPJyqcaYJLJRUlMaexDe0u7Fuq1zPF1MfteXDQLEs+nt9W0IdOoXkavODVEBpwkrwjUcylKHzniBr4TbNfnLRSis=
X-Received: by 2002:aa7:9050:0:b029:3af:7e99:f48f with SMTP id
 n16-20020aa790500000b02903af7e99f48fmr2534645pfo.2.1628861793055; Fri, 13 Aug
 2021 06:36:33 -0700 (PDT)
MIME-Version: 1.0
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <f8c4419a-7e83-b8a9-347c-0221a2052376@HIDDEN>
 <fff6e291-bbcc-93e3-92e7-00af4664b8d4@HIDDEN>
 <3d3f894f-a6fa-53ae-5d50-c3aa9bffc73e@HIDDEN>
 <CALDnm53xh6boc6T9XSC4jZpkCPS3RB8JFNb12BpkoeGYg3Q9RQ@HIDDEN>
 <56ab18b1-4348-9b2c-85bb-af9b76cd429a@HIDDEN>
 <CALDnm52XM+S07OPXO3qtfLN4pup-kT2U=WRO9dUo3tv9uYJWXw@HIDDEN>
 <38a06f3c-4a7a-755c-c99b-708f91afabfa@HIDDEN>
 <CALDnm537=LHAoaGqPpy7=JOPsYn=z+=Q6QW=b9bjrAWOtxqRaQ@HIDDEN>
 <9f59f87c-2489-aaa0-5b3f-0e911b7ffa6c@HIDDEN>
In-Reply-To: <9f59f87c-2489-aaa0-5b3f-0e911b7ffa6c@HIDDEN>
From: =?UTF-8?B?Sm/Do28gVMOhdm9yYQ==?= <joaotavora@HIDDEN>
Date: Fri, 13 Aug 2021 14:36:21 +0100
Message-ID: <CALDnm508EK5qM+AaOOJiNDm-=ORCuOieYpkjO=OSRas+_WTFwQ@HIDDEN>
Subject: Re: bug#47711: [PATCH VERSION 2] Add new
 `completion-filter-completions` API and deferred highlighting
To: Daniel Mendler <mail@HIDDEN>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 47711
Cc: Dmitry Gutov <dgutov@HIDDEN>, Stefan Monnier <monnier@HIDDEN>,
 48841 <at> debbugs.gnu.org, 47711 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

didnOn Fri, Aug 13, 2021 at 1:56 PM Daniel Mendler
<mail@HIDDEN> wrote:
>
> On 8/13/21 2:37 PM, Jo=C3=A3o T=C3=A1vora wrote:
> > On Fri, Aug 13, 2021 at 1:22 PM Daniel Mendler <mail@HIDDEN>=
 wrote:
> >
> >> It is important to keep this property since this will preclude many bu=
gs
> >> due to string mutation.
> >
> > I am aware of this, of course.  Can you give examples of these "many bu=
gs"?
> > Perhaps other than the one I already described and addressed?
>
> No, Jo=C3=A3o, this is not how it goes.  I don't have to prove to you tha=
t
> your idea introduces bugs.

So you just say it and I have to believe it?  Then I could say the same to
you, right?  I won't of course, that would be silly.

You have to show that mutation of the
> completion table strings (which are not supposed to be mutated) will not
> lead to bugs, which are hard to find.
>
> In contrast with the new API `completion-filter-completions` this entire
> class of bugs is avoided by construction of the API.  Furthermore the
> `completion-filter-completions` API is easy to use in comparison to your
> idea, where "compliant" backends have to apply string manipulations to
> apply the highlighting and revert the strings back to their old pristine
> state.  The only thing the API user has to do is to call the `highlight`
> function returned in the alist by `completion-filter-completions`.
>
> >> By separating the filtering and mutation
> >> (highlighting, scoring) my patch addresses the problem at hand in the
> >> proper way.
> >> [ ... ]
> >> Mutation would be a reasonable choice here if the problem could not be
> >> solved in a proper way.  But in fact it can be solved in a proper way
> >> without mutating the strings at all as my patch shows.
> >
> > "proper" is just an reasonably empty adjective.  There are different wa=
ys to
> > go about this, of course.  What's "proper" and isn't is hard to debate
> > objectively.
>
> You are contradicting yourself here. You agree that string mutation is
> better be avoid. If we define "proper" as avoids string mutation if this
> is easily possible, then my patch implements a proper solution to the> pr=
oblem.

I didn't say it's better avoided, though of course I will avoid _any_ chang=
e if
I can. I said I have identified one drawback with doing it.  Then I
have addressed
that drawback. So that's what I said.

I am unaware of _other_ drawbacks.  They might exist, but I am unaware of
them.  Perhaps you are, and indeed you state they exist, but you refuse to
let me know about them.  Or perhaps others know of them and will let me kno=
w.
In my long-running discussion with Dmitry they were not presented (again,
except for the one I identified).

> > That's for sure.  My patch idea addresses only that single problem.
> > I think this is a good property of patches: to solve one thing, not man=
y.
> No, this is not necessarily true.  This is only good if the problem is
> solved in a way which is future proof.

OK, but what thing of the future, real or academic, do you envision that
would bring back the problem, or create other problems?

> The idea of mutating the strings is a hack and not a solution.

Without facts to back it up, I have to take this as gratuitous disparagemen=
t.
Nicht so gut.

> In contrast, I am presenting a
> future-proof new API as solution which addresses multiple problems.

That's the issue.  The completion system is very complex and there are many
good ideas, different, floated by many people. But if you make a patch to
address "multiple" fuzzily-described problems, it's hard to judge how good
your ideas even are! Maybe they are indeed very good,  I never said
they weren't.  No need to get worked up about it!

Again, my proposal is to first focus on the performance problems caused by
string allocation.  _That_ problem is well understood, at least by me (but =
it
would help to settle on convenient benchmarks understood by others, too).
Then we can go from there.

> you look at the patch, only 196 new lines are added to minibuffer.el.
> Furthermore the patch adds 213 lines of new tests.

It's a large patch, over 1000 lines.  One does not review a patch
merely by looking at
lines added, when one needs to read much more, to understand implications, =
etc.
It needs documentation, for one, much more than just docstrings, on
how to use the
new API.

> Jo=C3=A3o, you don't have to lecture me on these things.  Of course I can
> provide such numbers.

Then please do! Not meaning to lecture you, just that your suggestion that
I try Vertico UI as a substitution for these numbers seemed completely
misguided.  So if you have them (or "can provide them") let's see them.
All I'm asking,  preferably from Emacs -Q recipe.

> You cannot reasonably make the claim that
> `copy-sequence` is the problem and at the same time claim that my patch
> does not solve the performance issues, when in fact my patch avoids this
> exact string copying.

I didn't say it didn't solve them! Now, where did I say that? I would
like to see a
benchmark so that I can witness it _and_ study alternative solutions. With
that, there's a better chance that I will be persuaded there are none
as elegant,
clean, proper, pure, etc as yours!

Maybe others review patches on other aspects that's fine.   Maybe
others will. Eli reviewed on minor formatting and documentation aspects.
I review them on substance, using numbers and conducting my own
experiments and tests. This takes time and help from the scientist on the
other end.

Simple and in summary, let's hope your next reply has some benchmarks
so we can make progress.

Thanks,
Jo=C3=A3o




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 13 Aug 2021 12:57:03 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Aug 13 08:57:03 2021
Received: from localhost ([127.0.0.1]:40820 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mEWk6-0002kW-V9
	for submit <at> debbugs.gnu.org; Fri, 13 Aug 2021 08:57:03 -0400
Received: from server.qxqx.de ([178.63.65.180]:56345 helo=mail.qxqx.de)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mail@HIDDEN>)
 id 1mEWjw-0002k2-Bh; Fri, 13 Aug 2021 08:56:53 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=qxqx.de;
 s=mail1392553390; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:
 MIME-Version:Date:Message-ID:From:References:Cc:To:Subject:Sender:Reply-To:
 Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender:
 Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:
 List-Subscribe:List-Post:List-Owner:List-Archive;
 bh=a89yZI6TfoUJ6rLAH1MakLAhc71tO9b1C8U/CHdpsC8=; b=N4MIoo3oh97UH6pQB1pEy7f66+
 9sONgkN7OIXHdBdyqVEavbTl7vLSVNa0DJfEcPc+jLn1uMcJhu7xLnmEPYWbWDt0Zjt5Xil5g4Qt5
 sgbdZcDkgu7zBVoSKBaf+jKohOVq31IKqv9Yz5dnImPysQjXCqiE1zInTEHWiTn5Mu34=;
Subject: Re: bug#47711: [PATCH VERSION 2] Add new
 `completion-filter-completions` API and deferred highlighting
To: =?UTF-8?B?Sm/Do28gVMOhdm9yYQ==?= <joaotavora@HIDDEN>
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <f8c4419a-7e83-b8a9-347c-0221a2052376@HIDDEN>
 <fff6e291-bbcc-93e3-92e7-00af4664b8d4@HIDDEN>
 <3d3f894f-a6fa-53ae-5d50-c3aa9bffc73e@HIDDEN>
 <CALDnm53xh6boc6T9XSC4jZpkCPS3RB8JFNb12BpkoeGYg3Q9RQ@HIDDEN>
 <56ab18b1-4348-9b2c-85bb-af9b76cd429a@HIDDEN>
 <CALDnm52XM+S07OPXO3qtfLN4pup-kT2U=WRO9dUo3tv9uYJWXw@HIDDEN>
 <38a06f3c-4a7a-755c-c99b-708f91afabfa@HIDDEN>
 <CALDnm537=LHAoaGqPpy7=JOPsYn=z+=Q6QW=b9bjrAWOtxqRaQ@HIDDEN>
From: Daniel Mendler <mail@HIDDEN>
Message-ID: <9f59f87c-2489-aaa0-5b3f-0e911b7ffa6c@HIDDEN>
Date: Fri, 13 Aug 2021 14:56:38 +0200
MIME-Version: 1.0
In-Reply-To: <CALDnm537=LHAoaGqPpy7=JOPsYn=z+=Q6QW=b9bjrAWOtxqRaQ@HIDDEN>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 8bit
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 47711
Cc: Dmitry Gutov <dgutov@HIDDEN>, Stefan Monnier <monnier@HIDDEN>,
 48841 <at> debbugs.gnu.org, 47711 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

On 8/13/21 2:37 PM, João Távora wrote:
> On Fri, Aug 13, 2021 at 1:22 PM Daniel Mendler <mail@HIDDEN> wrote:
> 
>> It is important to keep this property since this will preclude many bugs
>> due to string mutation.
> 
> I am aware of this, of course.  Can you give examples of these "many bugs"?
> Perhaps other than the one I already described and addressed?

No, João, this is not how it goes.  I don't have to prove to you that
your idea introduces bugs.  You have to show that mutation of the
completion table strings (which are not supposed to be mutated) will not
lead to bugs, which are hard to find.

In contrast with the new API `completion-filter-completions` this entire
class of bugs is avoided by construction of the API.  Furthermore the
`completion-filter-completions` API is easy to use in comparison to your
idea, where "compliant" backends have to apply string manipulations to
apply the highlighting and revert the strings back to their old pristine
state.  The only thing the API user has to do is to call the `highlight`
function returned in the alist by `completion-filter-completions`.

>> By separating the filtering and mutation
>> (highlighting, scoring) my patch addresses the problem at hand in the
>> proper way.
>> [ ... ]
>> Mutation would be a reasonable choice here if the problem could not be
>> solved in a proper way.  But in fact it can be solved in a proper way
>> without mutating the strings at all as my patch shows.
> 
> "proper" is just an reasonably empty adjective.  There are different ways to
> go about this, of course.  What's "proper" and isn't is hard to debate
> objectively.

You are contradicting yourself here. You agree that string mutation is
better be avoid. If we define "proper" as avoids string mutation if this
is easily possible, then my patch implements a proper solution to the
problem.

>>> The advantage that I see is that those adaptations apart, it is a small
>>> localized and effective change.
>>
>> Note that your idea also does not address the other issues which are
>> addressed by my patch.
> 
> That's for sure.  My patch idea addresses only that single problem.
> I think this is a good property of patches: to solve one thing, not many.

No, this is not necessarily true.  This is only good if the problem is
solved in a way which is future proof.  The idea of mutating the strings
is a hack and not a solution. In contrast, I am presenting a
future-proof new API as solution which addresses multiple problems.  If
you look at the patch, only 196 new lines are added to minibuffer.el.
Furthermore the patch adds 213 lines of new tests.

> Look, one needs to evaluate things quantitively. Your patch is not
> to Vertico, it's to Emacs. I'm concerned with changes to Emacs and their
> effect on all completion frontends.  So trying Vertico isn't very useful.
> 
> If you're solving a performance problem (and it seems that you are, among
> other things) we really need benchmarks, a description of an experiment whose
> results can be reproduced independently. It's the normal scientific method.

João, you don't have to lecture me on these things.  Of course I can
provide such numbers.  You cannot reasonably make the claim that
`copy-sequence` is the problem and at the same time claim that my patch
does not solve the performance issues, when in fact my patch avoids this
exact string copying.

>>>> The second problem addressed by the new API
>>>> `completion-filter-completions` is that `completion-all-completions` is
>>>> limited in what it can return.  For example it cannot return the end
>>>> position of the completion.
>>>
>>> And why is this a problem? Can you post an example of something you'd
>>> like to do, but can't?  Regardless, it does seem indeed like a "second" problem
>>> (as you state) so perhaps something that can be addressed separately.
>>
>> Please look at the FIXMEs in minibuffer.el which address this.
>> Currently only the beginning position of the completion boundary is
>> returned, which is only half of the information.
> 
> OK. It does seem like a separate problem, so maybe open a new bug for it?

There is already a FIXME in minibuffer.el, so I assume Stefan Monnier is
well aware of these issues.  It is an additional win of the new API that
such open problems can be fixed too.  As I see it, a new API is the way
to go here.

Daniel




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 13 Aug 2021 12:37:58 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Aug 13 08:37:58 2021
Received: from localhost ([127.0.0.1]:40777 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mEWRi-00005M-Ad
	for submit <at> debbugs.gnu.org; Fri, 13 Aug 2021 08:37:58 -0400
Received: from mail-pj1-f54.google.com ([209.85.216.54]:50968)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <joaotavora@HIDDEN>)
 id 1mEWRh-00004t-9k; Fri, 13 Aug 2021 08:37:57 -0400
Received: by mail-pj1-f54.google.com with SMTP id bo18so15156570pjb.0;
 Fri, 13 Aug 2021 05:37:57 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=mime-version:references:in-reply-to:from:date:message-id:subject:to
 :cc:content-transfer-encoding;
 bh=tlFNsJoHhRBkUHPWY4LrARaxXuMGniqUKbtu2G0qdXU=;
 b=W2jhGqYmQiddIO9xrhNq+mgLhNpZDrXgB7E3Nq9ikxl01hbFX3Xd7DKIjge5nk70Vk
 TJ6lEut6rRtmflXg2bZWmdKRLTVld535iIGhkPNPkR2iE5mzIBImO16T3hpJMGEAqQv7
 Yhr8k8CWpX1DATiwscnuWOKn8mfY2+OzkPrdykjc/3wfXZb8o3M/yKk0NdfzqtI6ymFg
 0kPcVTzA66bPapJPUHB8ZIrhZztHJy3tmpAGVeMOUngbv1wGsxMvUTRwlZe1YsLBzK6+
 KKaqcpLVT2U+JgodurScZY/iVu9Yzg7kQaUXfr3xSsiXh8bXW9UwtkjVdPzFp2sSSxI8
 eG/w==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:mime-version:references:in-reply-to:from:date
 :message-id:subject:to:cc:content-transfer-encoding;
 bh=tlFNsJoHhRBkUHPWY4LrARaxXuMGniqUKbtu2G0qdXU=;
 b=MSUMwNM1xDgUgyDOyycbk6t2dYHgo4sV0XHsatXgNpSvKZ9YH34CNMmfKZKZ19xLfW
 M7OfsOt3tHBGN3g95s2YqHPo6swiNlHncTqBJrwIhlJcrTT2Yqx8BiHduUuMGrutw5+h
 VUeHt52P8EcFYZFdgNbtZbLToqq+Mru0MghAtWGYwpI5CyRZYrKB2qesDKpv+vcfIwUs
 HqVtM54gdxBnGzPgKeGuu/W9kkqKyhdTwRmhsDGfwz8NTTfconxgRhFcjaOOvO6agmaz
 Y6MlnngG4cM0S3vjLFWM/gIDIcDQBZbxVY3BzPILmCN2m9RRa5DOVhWCY1DpRJ5s9kZN
 NgPA==
X-Gm-Message-State: AOAM533LyFHgkB8NdTtQk9u31KuoZZ2wotSW8j8BVW/NqfHWrsnUup2m
 aSTQuKFoIbnP+NZEcJXQVbU1f7js7LJI7X9N18w=
X-Google-Smtp-Source: ABdhPJydEtmy84BK6Wfd+Bbslze/kSoY9HX0jvqeGGlmCcf/udvaNVKD9YyLf4HvKp8/XJpyzqs4997sPyThwdhiEiE=
X-Received: by 2002:a62:5c6:0:b029:341:e0b1:a72c with SMTP id
 189-20020a6205c60000b0290341e0b1a72cmr2220586pff.71.1628858271491; Fri, 13
 Aug 2021 05:37:51 -0700 (PDT)
MIME-Version: 1.0
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <f8c4419a-7e83-b8a9-347c-0221a2052376@HIDDEN>
 <fff6e291-bbcc-93e3-92e7-00af4664b8d4@HIDDEN>
 <3d3f894f-a6fa-53ae-5d50-c3aa9bffc73e@HIDDEN>
 <CALDnm53xh6boc6T9XSC4jZpkCPS3RB8JFNb12BpkoeGYg3Q9RQ@HIDDEN>
 <56ab18b1-4348-9b2c-85bb-af9b76cd429a@HIDDEN>
 <CALDnm52XM+S07OPXO3qtfLN4pup-kT2U=WRO9dUo3tv9uYJWXw@HIDDEN>
 <38a06f3c-4a7a-755c-c99b-708f91afabfa@HIDDEN>
In-Reply-To: <38a06f3c-4a7a-755c-c99b-708f91afabfa@HIDDEN>
From: =?UTF-8?B?Sm/Do28gVMOhdm9yYQ==?= <joaotavora@HIDDEN>
Date: Fri, 13 Aug 2021 13:37:38 +0100
Message-ID: <CALDnm537=LHAoaGqPpy7=JOPsYn=z+=Q6QW=b9bjrAWOtxqRaQ@HIDDEN>
Subject: Re: bug#47711: [PATCH VERSION 2] Add new
 `completion-filter-completions` API and deferred highlighting
To: Daniel Mendler <mail@HIDDEN>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 47711
Cc: Dmitry Gutov <dgutov@HIDDEN>, Stefan Monnier <monnier@HIDDEN>,
 48841 <at> debbugs.gnu.org, 47711 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

On Fri, Aug 13, 2021 at 1:22 PM Daniel Mendler <mail@HIDDEN> wro=
te:

> It is important to keep this property since this will preclude many bugs
> due to string mutation.

I am aware of this, of course.  Can you give examples of these "many bugs"?
Perhaps other than the one I already described and addressed?

> By separating the filtering and mutation
> (highlighting, scoring) my patch addresses the problem at hand in the
> proper way.
>[ ... ]
> Mutation would be a reasonable choice here if the problem could not be
> solved in a proper way.  But in fact it can be solved in a proper way
> without mutating the strings at all as my patch shows.

"proper" is just an reasonably empty adjective.  There are different ways t=
o
go about this, of course.  What's "proper" and isn't is hard to debate
objectively.

> This solution is much more ad-hoc and you still mutate the string which
> is not allowed.

It's also difficult to debate "ad-hoc" or not.  If you've studied the
problem, what
makes you say that mutating the string (in this case, adding a
'completion--style-face' property to it) is not allowed? What negative thin=
gs
would derive from it.

> > The advantage that I see is that those adaptations apart, it is a small
> > localized and effective change.
>
> Note that your idea also does not address the other issues which are
> addressed by my patch.

That's for sure.  My patch idea addresses only that single problem.
I think this is a good property of patches: to solve one thing, not many.

We can make more patches to solve other problems, once we
identify them clearly.

> The new API `completion-filter-completions`
> returns data which hasn't been available before, e.g., the end position,
> which cannot be fixed given the existing API.
>
> >> The main problem is that `completion-all-completions` allocates all th=
e
> >> strings every time the completions are filtered.  This is the same
> >> performance issue you encountered in fido-mode/icomplete-mode.
> >
> > OK. I encountered at least two different performance problems there, wi=
th
> > quite different causes. So let's stick to the string-allocation problem=
.  Post
> > a code snippet that demonstrates the problem the way you see it/experie=
nce it?

Look, one needs to evaluate things quantitively. Your patch is not
to Vertico, it's to Emacs. I'm concerned with changes to Emacs and their
effect on all completion frontends.  So trying Vertico isn't very useful.

If you're solving a performance problem (and it seems that you are, among
other things) we really need benchmarks, a description of an experiment who=
se
results can be reproduced independently. It's the normal scientific method.

Something like:

"before my patch, this code takes 123 seconds to run, after my patch it
takes 12."

> >> The second problem addressed by the new API
> >> `completion-filter-completions` is that `completion-all-completions` i=
s
> >> limited in what it can return.  For example it cannot return the end
> >> position of the completion.
> >
> > And why is this a problem? Can you post an example of something you'd
> > like to do, but can't?  Regardless, it does seem indeed like a "second"=
 problem
> > (as you state) so perhaps something that can be addressed separately.
>
> Please look at the FIXMEs in minibuffer.el which address this.
> Currently only the beginning position of the completion boundary is
> returned, which is only half of the information.

OK. It does seem like a separate problem, so maybe open a new bug for it?

Jo=C3=A3o T=C3=A1vora




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 13 Aug 2021 12:23:10 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Aug 13 08:23:10 2021
Received: from localhost ([127.0.0.1]:40705 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mEWDK-000850-Qv
	for submit <at> debbugs.gnu.org; Fri, 13 Aug 2021 08:23:10 -0400
Received: from server.qxqx.de ([178.63.65.180]:52851 helo=mail.qxqx.de)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mail@HIDDEN>)
 id 1mEWDB-000847-Ko; Fri, 13 Aug 2021 08:23:01 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=qxqx.de;
 s=mail1392553390; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:
 MIME-Version:Date:Message-ID:From:References:Cc:To:Subject:Sender:Reply-To:
 Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender:
 Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:
 List-Subscribe:List-Post:List-Owner:List-Archive;
 bh=No8L/5f/jqE7VfR4Zr8RQeJNpAlnovp34dfprC2OpIA=; b=NVDxqieGZDnqxkTN6bD1l5Lybk
 XKH0kqn8ciZRmZw+/MId68EnF9wrXz5XBq9PW3+NEGcLSSbkH7oZ/LvNHUVkYBB3G2MrmqMEWev05
 59CczGWRQCHXazW65/zaniWkgZYzTYafVKgjWRWtqAl9idZuD1K3RAOU5YOdF7dQHWx0=;
Subject: Re: bug#47711: [PATCH VERSION 2] Add new
 `completion-filter-completions` API and deferred highlighting
To: =?UTF-8?B?Sm/Do28gVMOhdm9yYQ==?= <joaotavora@HIDDEN>
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <f8c4419a-7e83-b8a9-347c-0221a2052376@HIDDEN>
 <fff6e291-bbcc-93e3-92e7-00af4664b8d4@HIDDEN>
 <3d3f894f-a6fa-53ae-5d50-c3aa9bffc73e@HIDDEN>
 <CALDnm53xh6boc6T9XSC4jZpkCPS3RB8JFNb12BpkoeGYg3Q9RQ@HIDDEN>
 <56ab18b1-4348-9b2c-85bb-af9b76cd429a@HIDDEN>
 <CALDnm52XM+S07OPXO3qtfLN4pup-kT2U=WRO9dUo3tv9uYJWXw@HIDDEN>
From: Daniel Mendler <mail@HIDDEN>
Message-ID: <38a06f3c-4a7a-755c-c99b-708f91afabfa@HIDDEN>
Date: Fri, 13 Aug 2021 14:22:48 +0200
MIME-Version: 1.0
In-Reply-To: <CALDnm52XM+S07OPXO3qtfLN4pup-kT2U=WRO9dUo3tv9uYJWXw@HIDDEN>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 8bit
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 47711
Cc: Dmitry Gutov <dgutov@HIDDEN>, Stefan Monnier <monnier@HIDDEN>,
 48841 <at> debbugs.gnu.org, 47711 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

On 8/13/21 2:05 PM, João Távora wrote:
>> The existing API `completion-all-completions`
>> necessarily has to allocate all the strings in order to attach
>> highlighting and scoring.  The new API solves this in a clean way by
>> both deferring highlighting and scoring.
> 
> I'm not sure you understand my alternative idea.  As far as I
> understand (and have actually measured) the lines:
> 
>    ;; Don't modify the string itself.
>    (setq str (copy-sequence str))
> 
> in minibuffer.el, in the function completion-pcm--hilit-commonality
> 
> Are the cause of the problem that _I am talking about_ and that
> I have actually measured.  Again you may be referring to a
> _different_ problem that I am unaware of.

You are right that the call to `copy-sequence` is a major bottleneck in
the filtering.  However you are wrong that this line can simply be
removed/disabled and the candidates can be modified.  The API guarantees
and has always guaranteed that the candidate strings are not mutated.
It is important to keep this property since this will preclude many bugs
due to string mutation.  By separating the filtering and mutation
(highlighting, scoring) my patch addresses the problem at hand in the
proper way.

Note that the UI also has no possibility to opt-out of the mutation.
The UI is actually not the one being concerned about the mutation here,
it is the backends (completion tables), which produce the strings.  If
one starts mutating these strings you will see bugs cropping up
throughout Emacs where shared strings suddenly have spurious additional
properties due to the completion filtering.

Mutation would be a reasonable choice here if the problem could not be
solved in a proper way.  But in fact it can be solved in a proper way
without mutating the strings at all as my patch shows.

> If one removes these lines, the process becomes much faster, but there is a
> problem with highlighting.  My idea is indeed to defer highlighting by not
> setting the 'face property directly on that shared string, but some
> other property
> that is read later from the shared string by compliant frontents.

This solution is much more ad-hoc and you still mutate the string which
is not allowed.

> The advantage that I see is that those adaptations apart, it is a small
> localized and effective change.

Note that your idea also does not address the other issues which are
addressed by my patch.  The new API `completion-filter-completions`
returns data which hasn't been available before, e.g., the end position,
which cannot be fixed given the existing API.

>> The main problem is that `completion-all-completions` allocates all the
>> strings every time the completions are filtered.  This is the same
>> performance issue you encountered in fido-mode/icomplete-mode.
> 
> OK. I encountered at least two different performance problems there, with
> quite different causes. So let's stick to the string-allocation problem.  Post
> a code snippet that demonstrates the problem the way you see it/experience it?

You can try my Vertico completion UI, which is available on GNU ELPA.
It implements deferred highlighting and there the performance difference
is perceivable.  Currently Vertico uses an advice-based hack to avoid
the over-eager string-allocations and the highlighting.

>> The second problem addressed by the new API
>> `completion-filter-completions` is that `completion-all-completions` is
>> limited in what it can return.  For example it cannot return the end
>> position of the completion.
> 
> And why is this a problem? Can you post an example of something you'd
> like to do, but can't?  Regardless, it does seem indeed like a "second" problem
> (as you state) so perhaps something that can be addressed separately.

Please look at the FIXMEs in minibuffer.el which address this.
Currently only the beginning position of the completion boundary is
returned, which is only half of the information.

> I understand you've put time and effort into producing this work. We are
> all indebted and I promise to read it. But every API writer in history of
> programming has claimed those things and reality often shows otherwise.
> So it's not that your work can't be those things you claim, maybe it is, but
> generally the larger and broader the work the harder it is to reason about.

I stand by my claim and I also stand by the claim that
removing/disabling `copy-sequence` is not a proper way to address the
issues at hand and will introduce many bugs in the long run.  Please
take your time to look at the patch in earnest.  I would also like to
see others chime in here with their opinion.

Daniel




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 13 Aug 2021 12:06:14 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Aug 13 08:06:14 2021
Received: from localhost ([127.0.0.1]:40678 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mEVww-0007ec-H4
	for submit <at> debbugs.gnu.org; Fri, 13 Aug 2021 08:06:14 -0400
Received: from mail-pl1-f176.google.com ([209.85.214.176]:36863)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <joaotavora@HIDDEN>)
 id 1mEVwr-0007dy-3b; Fri, 13 Aug 2021 08:06:09 -0400
Received: by mail-pl1-f176.google.com with SMTP id f3so11648504plg.3;
 Fri, 13 Aug 2021 05:06:05 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=mime-version:references:in-reply-to:from:date:message-id:subject:to
 :cc:content-transfer-encoding;
 bh=njSiQZeqjhLp38f+q1STdsHcPKVcgTrEHylx+LpFmXQ=;
 b=mRybdEqQZZnXQ+wU8nuO02C7OgdE70ro+46P0pATdrPTNswOqyV8UJ8PUS9VtW0SPf
 VzMVuQpMif7UNSR5qytOEKul+S5kzXJw5cRg0YDCVjkJpi7cBM9PcqAplrtBnUCQw3wo
 NLnASLQXzEy4e6KCvPgVgHnmGImAeETOsALWq5E3R4V1DL/XuzYwwdKoHXJzIm46fikI
 nEAdAoPQXgB0jaarEGkweITDlYPwhyv2TLda0XFpXP6+UutpTkTXhusniX90baGjkHNV
 BugikjEx/lXJhnWusr5PLPs6oddB10h+b4g+Hz+GKa0DpV+k5xP7pCNxlLPprpZiBO7a
 m0kw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:mime-version:references:in-reply-to:from:date
 :message-id:subject:to:cc:content-transfer-encoding;
 bh=njSiQZeqjhLp38f+q1STdsHcPKVcgTrEHylx+LpFmXQ=;
 b=W/imYrU92wciOyw2t/wJhW/DXkxz4KZI31KIKn0N8EWDDO4Qa/2DY0ZRyWfzxdd2XA
 vPd8KbhcfWqhhCgaTU4AqniQocifXMsCpcqILOUCFveLitTTKAJkah0bCEfHfNnzBcGZ
 lrPimWfTG84qiWMykUPaAM3yAINbVI7gVDmC+DZemRiVcb5OjMzH6Pj4rflBrycpYetu
 koVLblfSjjYwte4sCIY0/8Umw8o1vgkmSuZ1l8DWvz5B33opDq3Gz/F7EkgRJSfk3iqo
 D8alBpqI2rNfcx7LqGavxxxDsS/xbjJivMNgRqqjNAWv/0xujzpCy8r6YZ/MMKUYpc23
 jCZQ==
X-Gm-Message-State: AOAM533qhNr1IbKcVGH0kbevY5VqsnO9AGh+bm1INMIzJtDOdesWCtca
 9Rjs+vA2DLBrTkkZhbJriucb9ffWcVR31B9ykwc=
X-Google-Smtp-Source: ABdhPJyQMdCxprMZ56Ulp2KcbQOQamn6j8Gk84j90A/p0TH9H4ZILhFxlt32JRNgYDIOHmjKvSOaUG0hjQuyC993JyM=
X-Received: by 2002:a17:90b:14b:: with SMTP id
 em11mr2340093pjb.125.1628856359053; 
 Fri, 13 Aug 2021 05:05:59 -0700 (PDT)
MIME-Version: 1.0
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <f8c4419a-7e83-b8a9-347c-0221a2052376@HIDDEN>
 <fff6e291-bbcc-93e3-92e7-00af4664b8d4@HIDDEN>
 <3d3f894f-a6fa-53ae-5d50-c3aa9bffc73e@HIDDEN>
 <CALDnm53xh6boc6T9XSC4jZpkCPS3RB8JFNb12BpkoeGYg3Q9RQ@HIDDEN>
 <56ab18b1-4348-9b2c-85bb-af9b76cd429a@HIDDEN>
In-Reply-To: <56ab18b1-4348-9b2c-85bb-af9b76cd429a@HIDDEN>
From: =?UTF-8?B?Sm/Do28gVMOhdm9yYQ==?= <joaotavora@HIDDEN>
Date: Fri, 13 Aug 2021 13:05:47 +0100
Message-ID: <CALDnm52XM+S07OPXO3qtfLN4pup-kT2U=WRO9dUo3tv9uYJWXw@HIDDEN>
Subject: Re: bug#47711: [PATCH VERSION 2] Add new
 `completion-filter-completions` API and deferred highlighting
To: Daniel Mendler <mail@HIDDEN>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 47711
Cc: Dmitry Gutov <dgutov@HIDDEN>, Stefan Monnier <monnier@HIDDEN>,
 48841 <at> debbugs.gnu.org, 47711 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

On Fri, Aug 13, 2021 at 12:21 PM Daniel Mendler <mail@HIDDEN> wr=
ote:

> No, this is not the case. There is no simple fix of the allocation issue
> on the frontend side.

I didn't claim that. At all. I claimed that the frontends that would be
affected by the (small)  backend patch are easy to adapt.  I think
you completely read past my idea.

> The existing API `completion-all-completions`
> necessarily has to allocate all the strings in order to attach
> highlighting and scoring.  The new API solves this in a clean way by
> both deferring highlighting and scoring.

I'm not sure you understand my alternative idea.  As far as I
understand (and have actually measured) the lines:

   ;; Don't modify the string itself.
   (setq str (copy-sequence str))

in minibuffer.el, in the function completion-pcm--hilit-commonality

Are the cause of the problem that _I am talking about_ and that
I have actually measured.  Again you may be referring to a
_different_ problem that I am unaware of.

If one removes these lines, the process becomes much faster, but there is a
problem with highlighting.  My idea is indeed to defer highlighting by not
setting the 'face property directly on that shared string, but some
other property
that is read later from the shared string by compliant frontents.

If you have understood this idea, can you comment on it?
(Preferably in terms of less adjectification regarding "cleanliness", but i=
n
terms of actual drawbacks/advantages?)

The drawback that I can see in it is that frontends directly relying
on 'face are
broken by that patch. But according to Dmitry (and I tend to agree), it's
quite easy to address those frontends.  Most of them live in Emacs core or
GNU Elpa.

The advantage that I see is that those adaptations apart, it is a small
localized and effective change.

> I claim that my patch is easy to reason about and refactors the existing
> code to address the exact problem we are having. Please take some time
> in reviewing it.

I am already taking some time. I need your assistance in explaining the
problems first. I take into account your claims of cleanliness and elegance=
,
but in terms of their power of persuasion, they are much more limited
than hard material evidence.

> The main problem is that `completion-all-completions` allocates all the
> strings every time the completions are filtered.  This is the same
> performance issue you encountered in fido-mode/icomplete-mode.

OK. I encountered at least two different performance problems there, with
quite different causes. So let's stick to the string-allocation problem.  P=
ost
a code snippet that demonstrates the problem the way you see it/experience =
it?

Some benchmark code would be very welcome.  You can probably grab my
benchmarking code from that other bug.

Then it becomes easy to study multiple solutions to that problem and
choose the best one!

> The second problem addressed by the new API
> `completion-filter-completions` is that `completion-all-completions` is
> limited in what it can return.  For example it cannot return the end
> position of the completion.

And why is this a problem? Can you post an example of something you'd
like to do, but can't?  Regardless, it does seem indeed like a "second" pro=
blem
(as you state) so perhaps something that can be addressed separately.

Is your particular solution to this second problem instrumental in solving
the "main problem"

> This is also solved by the new API.  The new API is a clean extensible wa=
y forward.

I understand you've put time and effort into producing this work. We are
all indebted and I promise to read it. But every API writer in history of
programming has claimed those things and reality often shows otherwise.
So it's not that your work can't be those things you claim, maybe it is, bu=
t
generally the larger and broader the work the harder it is to reason about.

Jo=C3=A3o




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 13 Aug 2021 11:21:43 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Aug 13 07:21:43 2021
Received: from localhost ([127.0.0.1]:40605 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mEVFu-0004C5-TH
	for submit <at> debbugs.gnu.org; Fri, 13 Aug 2021 07:21:43 -0400
Received: from server.qxqx.de ([178.63.65.180]:58693 helo=mail.qxqx.de)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mail@HIDDEN>)
 id 1mEVFs-0004Bk-QI; Fri, 13 Aug 2021 07:21:41 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=qxqx.de;
 s=mail1392553390; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:
 MIME-Version:Date:Message-ID:From:References:Cc:To:Subject:Sender:Reply-To:
 Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender:
 Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:
 List-Subscribe:List-Post:List-Owner:List-Archive;
 bh=8hwFNV/aAe3vVUeLSmkDH/7Bo2a4jUM3hyldTTf1mXU=; b=Z0MtMaK/toFejKLtw4dwCcoPv3
 YNf2RJn82xqwC1SZfFohhyetqlPs/IpcFa3HdDwI5YKS+gW480ebGSmW/RKlFSpKr28QVEAgSI7l0
 N4x0J9Nv6N7kXDletNpixpVp6ik9P4RYvzuI/yqa79S6KsPlVCLjehZWNnB0PA5aAHmA=;
Subject: Re: bug#47711: [PATCH VERSION 2] Add new
 `completion-filter-completions` API and deferred highlighting
To: =?UTF-8?B?Sm/Do28gVMOhdm9yYQ==?= <joaotavora@HIDDEN>
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <f8c4419a-7e83-b8a9-347c-0221a2052376@HIDDEN>
 <fff6e291-bbcc-93e3-92e7-00af4664b8d4@HIDDEN>
 <3d3f894f-a6fa-53ae-5d50-c3aa9bffc73e@HIDDEN>
 <CALDnm53xh6boc6T9XSC4jZpkCPS3RB8JFNb12BpkoeGYg3Q9RQ@HIDDEN>
From: Daniel Mendler <mail@HIDDEN>
Message-ID: <56ab18b1-4348-9b2c-85bb-af9b76cd429a@HIDDEN>
Date: Fri, 13 Aug 2021 13:21:32 +0200
MIME-Version: 1.0
In-Reply-To: <CALDnm53xh6boc6T9XSC4jZpkCPS3RB8JFNb12BpkoeGYg3Q9RQ@HIDDEN>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 8bit
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 47711
Cc: Dmitry Gutov <dgutov@HIDDEN>, Stefan Monnier <monnier@HIDDEN>,
 48841 <at> debbugs.gnu.org, 47711 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

On 8/13/21 12:56 PM, João Távora wrote:
> I've read the discussion and am indeed aware of some non-neglibile
> performance problems in the flex and pcm completion styles since
> they need to copy strings around.  Other -- completely different --
> performance problems affect fido-mode specifically (but not
> fido-vertical-mode, curiously).
> 
> In some conversation with Dmitry
> 
>   bug#48841: fido-mode is slower than ido-mode with similar settings
> 
> We discussed this.

I've read the discussion.  You are probably aware of my efforts to in
Vertico to implement deferred highlighting.  The patch I implemented
here implements the deferred highlighting in a clean way.

> There was also talk of removing the string copying with minimal (but not null)
> backward compatibility breakage.  I recall Dmitry saying it was easy
> to fix on the
> completion frontend side.  Many such frontends live in Emacs or GNU Elpa.
> On the other hand, the patch that we (or at least I) envisioned in
> that discussion
> was almost certainly much, much simpler than the one being presented here,
> and thus much easier to reason about and discuss.

No, this is not the case. There is no simple fix of the allocation issue
on the frontend side.  The existing API `completion-all-completions`
necessarily has to allocate all the strings in order to attach
highlighting and scoring.  The new API solves this in a clean way by
both deferring highlighting and scoring.

I claim that my patch is easy to reason about and refactors the existing
code to address the exact problem we are having. Please take some time
in reviewing it.

> But to avoid comparing apples to oranges, I would you to summarize exactly,
> perhaps in the forms of code snippets, and/or benchmarks exactly what problems
> your large patch solves. State the problem(s) first, then the solution
> (to each).

The main problem is that `completion-all-completions` allocates all the
strings every time the completions are filtered.  This is the same
performance issue you encountered in fido-mode/icomplete-mode.

The second problem addressed by the new API
`completion-filter-completions` is that `completion-all-completions` is
limited in what it can return.  For example it cannot return the end
position of the completion.  This is also solved by the new API.  The
new API is a clean extensible way forward.

Daniel




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 13 Aug 2021 10:57:19 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Aug 13 06:57:19 2021
Received: from localhost ([127.0.0.1]:40566 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mEUsJ-0001Nz-36
	for submit <at> debbugs.gnu.org; Fri, 13 Aug 2021 06:57:19 -0400
Received: from mail-pj1-f53.google.com ([209.85.216.53]:50818)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <joaotavora@HIDDEN>)
 id 1mEUsH-0001Nj-0K; Fri, 13 Aug 2021 06:57:17 -0400
Received: by mail-pj1-f53.google.com with SMTP id bo18so14781595pjb.0;
 Fri, 13 Aug 2021 03:57:16 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=mime-version:references:in-reply-to:from:date:message-id:subject:to
 :cc:content-transfer-encoding;
 bh=dSemysIFcr8EOxC+zmRqh0RfmCQlAPXj3PR3O0OqjzY=;
 b=SqubbkDLKuatuVjfKoJQjDb01671J7SAVkAWFgMGPssDNfl+XOjqCEss4W9U97TluU
 n1xJdbjlqGJ1Mj1WhF5MN+ma5y3HnwSBQMUEaAOAjuOyqjQAQavfhX7lVdrYQHDS8EXP
 Gl31ZgY5JLaBP/bQel4FeetOLzBJKAT1ynhYVwIej4H9fOmCtW8yb9dsnRO1ReNForXQ
 GiQFOPO41g/QwAR85Es4hB2/LPoIDSo5mFPJijI6ptl5uJSNY9IrpQRaw9rxcBk2Fyzs
 Y6kk0pfHMFXrKQo+XVgkxRoTawnBf8r3xw5U9plE3Shu1T9ME2GkKjAQbo+mjc0V/1YX
 yR4w==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:mime-version:references:in-reply-to:from:date
 :message-id:subject:to:cc:content-transfer-encoding;
 bh=dSemysIFcr8EOxC+zmRqh0RfmCQlAPXj3PR3O0OqjzY=;
 b=NTQOAVAo1TY2fEroNfNGTntDeXFhg7x7yWweDS2OXVVmtDuuVF28EDOha3kRVMrEqf
 laGHQhPHesf9JMefAa+1FD2bvqX6L21uVLGLTvhZGUG3FY28r3PoAjXEsuo3NITeatP0
 aSReToSpb9CQo+3PaUcKXBLeP1O/z943YGXtjY9dhUdLp8Fp4jAKfkpedlflGw+x75Ex
 E5bHAqLYnd37uC9K6SBRjOp/8+WhJm2rBM5FhfQvgP9Uloqjk943gW5OP20NWL+Jiy6g
 5CkggSl1auOIe20sy4lq72G9S4N6zbIUfclRVbKOlx4DMkCZAsV0V2lIehmd+AjgqWri
 8vdA==
X-Gm-Message-State: AOAM531XFDNPfEYScjVKT2NWHKrwVe/B05k0HPjP5zzDov/lMVwCXExH
 D5k3adbNTjhyrthvgFvnZ0DIsRXzcm2Nfjj3aJc=
X-Google-Smtp-Source: ABdhPJxe/IE4MLihlfvr0Vhji16fJ+zz8X6GRYFjDtC5ScQ9Kf+7vIgbgXzMBZ7NcW8mfhlhyvK1dz3EAnhzzSNx9hY=
X-Received: by 2002:a63:c0a:: with SMTP id b10mr1803467pgl.447.1628852231016; 
 Fri, 13 Aug 2021 03:57:11 -0700 (PDT)
MIME-Version: 1.0
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <f8c4419a-7e83-b8a9-347c-0221a2052376@HIDDEN>
 <fff6e291-bbcc-93e3-92e7-00af4664b8d4@HIDDEN>
 <3d3f894f-a6fa-53ae-5d50-c3aa9bffc73e@HIDDEN>
In-Reply-To: <3d3f894f-a6fa-53ae-5d50-c3aa9bffc73e@HIDDEN>
From: =?UTF-8?B?Sm/Do28gVMOhdm9yYQ==?= <joaotavora@HIDDEN>
Date: Fri, 13 Aug 2021 11:56:59 +0100
Message-ID: <CALDnm53xh6boc6T9XSC4jZpkCPS3RB8JFNb12BpkoeGYg3Q9RQ@HIDDEN>
Subject: Re: [PATCH VERSION 2] Add new `completion-filter-completions` API and
 deferred highlighting
To: Daniel Mendler <mail@HIDDEN>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 47711
Cc: Eli Zaretskii <eliz@HIDDEN>, Dmitry Gutov <dgutov@HIDDEN>,
 Stefan Monnier <monnier@HIDDEN>, 48841 <at> debbugs.gnu.org,
 47711 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

> In comparison to my last patch, the patch is fully backward
> compatible and preserves all existing tests.

This a very good thing (the fact that the patch is fully backward compatibl=
e,
I mean).

It is quite a large patch that touches many completion internals.  I'd like
some time to look it over.

I've read the discussion and am indeed aware of some non-neglibile
performance problems in the flex and pcm completion styles since
they need to copy strings around.  Other -- completely different --
performance problems affect fido-mode specifically (but not
fido-vertical-mode, curiously).

In some conversation with Dmitry

  bug#48841: fido-mode is slower than ido-mode with similar settings

We discussed this.

There was also talk of removing the string copying with minimal (but not nu=
ll)
backward compatibility breakage.  I recall Dmitry saying it was easy
to fix on the
completion frontend side.  Many such frontends live in Emacs or GNU Elpa.
On the other hand, the patch that we (or at least I) envisioned in
that discussion
was almost certainly much, much simpler than the one being presented here,
and thus much easier to reason about and discuss.

But to avoid comparing apples to oranges, I would you to summarize exactly,
perhaps in the forms of code snippets, and/or benchmarks exactly what probl=
ems
your large patch solves. State the problem(s) first, then the solution
(to each).
If there are multiple problems, then there's a good chance that multiple pa=
tches
that address each of these are preferred.

Thank you very much.
Jo=C3=A3o




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 13 Aug 2021 10:38:49 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Aug 13 06:38:49 2021
Received: from localhost ([127.0.0.1]:40555 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mEUaN-0007E5-G7
	for submit <at> debbugs.gnu.org; Fri, 13 Aug 2021 06:38:49 -0400
Received: from server.qxqx.de ([178.63.65.180]:41947 helo=mail.qxqx.de)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mail@HIDDEN>)
 id 1mEUaD-0007DS-NT; Fri, 13 Aug 2021 06:38:42 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=qxqx.de;
 s=mail1392553390; h=Content-Type:In-Reply-To:MIME-Version:Date:Message-ID:
 References:Cc:To:From:Subject:Sender:Reply-To:Content-Transfer-Encoding:
 Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender:
 Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:
 List-Subscribe:List-Post:List-Owner:List-Archive;
 bh=SOhFQ1wSTJnkfn/KBMxtDxfjGodK9cPP6BpCAVQ3Lfs=; b=IEjg+aZkC1HrwtTiwHLxhnp3JL
 JJ35+RmaX6JZqCW3bXdtO3LJrxwz+CsVgnEjjo5GKq6XPoPo4xYp437AFb0l7RscfmEDM4jtCZM+m
 ehIxls6ebgDm8mqhfIY5wqjVe4ZmcRKzTR06jd9pMG4FQqNPBK400UfmBk6qrYndGUhM=;
Subject: Re: [PATCH VERSION 2] Add new `completion-filter-completions` API and
 deferred highlighting
From: Daniel Mendler <mail@HIDDEN>
To: 47711 <at> debbugs.gnu.org, 48841 <at> debbugs.gnu.org
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <f8c4419a-7e83-b8a9-347c-0221a2052376@HIDDEN>
 <fff6e291-bbcc-93e3-92e7-00af4664b8d4@HIDDEN>
Message-ID: <3d3f894f-a6fa-53ae-5d50-c3aa9bffc73e@HIDDEN>
Date: Fri, 13 Aug 2021 12:38:28 +0200
MIME-Version: 1.0
In-Reply-To: <fff6e291-bbcc-93e3-92e7-00af4664b8d4@HIDDEN>
Content-Type: multipart/mixed; boundary="------------E6B7393FF90421271E72F100"
Content-Language: en-US
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 47711
Cc: Eli Zaretskii <eliz@HIDDEN>, Dmitry Gutov <dgutov@HIDDEN>,
 =?UTF-8?B?Sm/Do28gVMOhdm9yYQ==?= <joaotavora@HIDDEN>,
 Stefan Monnier <monnier@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

This is a multi-part message in MIME format.
--------------E6B7393FF90421271E72F100
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

I attached the overhauled patch, which addresses most of the comments by
Eli.  In comparison to my last patch, the patch is fully backward
compatible and preserves all existing tests.  As before, there are tests
which check the new functionality for each existing completion style.

Daniel

--------------E6B7393FF90421271E72F100
Content-Type: text/x-diff; charset=UTF-8;
 name="0001-Add-new-completion-filter-completions-API-and-deferr.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename*0="0001-Add-new-completion-filter-completions-API-and-deferr.pa";
 filename*1="tch"

From 30ca42b49d5b5316abb3ebad38b0e9629eb52920 Mon Sep 17 00:00:00 2001
From: Daniel Mendler <mail@HIDDEN>
Date: Mon, 12 Jul 2021 21:40:32 +0200
Subject: [PATCH] Add new 'completion-filter-completions' API and deferred
 highlighting

Fix bug#47711.

Add a new 'completion-filter-completions' API, which supersedes
'completion-all-completions'.  The new API returns the matching
completion candidates and additional data.  The return value is an
alist, with the keys 'completions', 'base', 'end' and 'highlight'.
The API can be extended in a backward compatible way later on thanks
to the use of an alist as return value.

The 'completions' value is the list of completion strings *without*
applied highlighting.  The completion strings are returned unmodified,
which avoids allocations and results in performance gains for
continuously updating completion UIs, like Icomplete or Vertico (GNU
ELPA).  The value 'base' is the base position of the completion
relative to the beginning of the input string.  Correspondingly the
value 'end' specifies the end position of the completion relative to
the beginning of the input string.  In comparison, the old function
'completion-all-completions' only returned the base position in the
last cdr of the returned completions list, which complicated usage.
The 'end' position was not provided by 'completion-all-completions'.
Given the new API the 'completion-base-position' can be set
accurately.  Finally the 'highlight' value is a function taking a list
of completion strings and returns a new list of new strings with
highlighting applied.  A continously updating UI can use the
highlighting function to apply highlighting only to the visible
completions.

* lisp/minibuffer.el:
(completion--adjust-metadata): Rename to 'completion--style-metadata'
due to change of calling convention.
(completion--nth-completion): Call renamed metadata adjustment
function.  Ignore the old property 'completion--adjust-metadata'.
(completion--flex-adjust-metadata): Rename function.
(completion--twq-all): Attach 'completion--unquoted' text property to
quoted completion strings.
(completion--flex-score-1): Extract new function from
'completion-pcm--hilit-commonality'.
(completion-pcm--hilit-commonality): Use it.  Add SCORE argument.
(completion--flex-score): Use 'completion--flex-score-1'.  Use
'completion--unquoted' text property.
(completion--flex-style-metadata): Use it.
(completion--pattern-compiler): New function.
(completion-substring--all-completions)
(completion--flex-score): Use it.
(completion--hilit-commonality): New function.
(completion-hilit-commonality): Use it.
(completion--deferred-hilit): New function.
(completion-basic-all-completions)
(completion-emacs21-all-completions)
(completion-emacs22-all-completions): Use it.
(completion--pcm-deferred-hilit): New function.
(completion-pcm-all-completions)
(completion-flex-all-completions)
(completion-initials-all-completions)
(completion-substring-all-completions): Use it.
(completion--return-alist-flag): New variable to conditionally enable
the new alist completions result format.  This variable is for
internal use to preserve the existing calling convention of the
completion style 'all' functions.
(completion-filter-completions): New API which returns the completion
strings and additional data as an an alist.  Transparently convert
old list completion style results to the new alist format.
(completion-all-completions): Transparently convert the new
alist completion style result to the old list format.
(minibuffer-completion-help): Use the new API, set
'completion-base-position' correctly.
(completion-try-completion)
(completion-all-completions): Update doc string.
(completion--replace): Fix property removal.

* test/lisp/minibuffer-tests.el:
(completion--test-style)
(completion--test-boundaries): New test helper function.
(completion-emacs22orig-all-completions): New function.
(completion-flex-score-test-*): Add new scoring test functions.
(completion-*-style-test): Add new API tests for each built-in
completion style.
(completion-*-boundaries-test): New boundary tests for each built-in
completion style.
(completion-filter-completions-highlight-test): New API test.
(completion-upgrade-return-type-test): New test of transparent
completion style return value upgrade.
---
 lisp/minibuffer.el            | 580 +++++++++++++++++++++++-----------
 test/lisp/minibuffer-tests.el | 217 ++++++++++++-
 2 files changed, 603 insertions(+), 194 deletions(-)

diff --git a/lisp/minibuffer.el b/lisp/minibuffer.el
index 9f327df28f..66ac6b3763 100644
--- a/lisp/minibuffer.el
+++ b/lisp/minibuffer.el
@@ -692,6 +692,10 @@ completion--twq-all
                                              'completions-common-part)
                                qprefix))))
                         (qcompletion (concat qprefix qnew)))
+                   ;; Attach unquoted completion string, which is needed
+                   ;; to score the completion in `completion--flex-score'.
+                   (put-text-property 0 1 'completion--unquoted
+                                      completion qcompletion)
 		   ;; FIXME: Similarly here, Cygwin's mapping trips this
 		   ;; assertion.
                    ;;(cl-assert
@@ -1035,6 +1039,17 @@ completion--styles
         (delete-dups (append (cdr over) (copy-sequence completion-styles)))
        completion-styles)))
 
+(defvar completion--return-alist-flag nil
+  "Non-nil means to return completions in alist format.
+If this variable is non-nil the `all-completions' function of a
+completion style should return the results in the alist format of
+`completion-filter-completions'.  This variable is purely needed to
+for backward compatibility of the existing builtin completion style
+functions as of Emacs 28.  Newer completion style functions should
+always return their results in the alist format, since
+`completion-all-completions' transparently converts back to a list of
+completions with base size in the last cdr.")
+
 (defun completion--nth-completion (n string table pred point metadata)
   "Call the Nth method of completion styles."
   ;; We provide special support for quoting/unquoting here because it cannot
@@ -1061,6 +1076,15 @@ completion--nth-completion
                  ;; the original table, in that case!
                  (functionp table))
             (let ((new (funcall table string point 'completion--unquote)))
+              ;; FIXME For now do not attempt deferred highlighting if
+              ;; quoting is used.  Not doing deferred highlighting is
+              ;; not too severe in this case, since
+              ;; `completion--twq-all' is already an expensive
+              ;; function, which allocates all completion strings.  In
+              ;; contrast to plain completion tables, the savings of
+              ;; deferred highlighting would be minimal in the case of
+              ;; quoted completion tables.
+              (setq completion--return-alist-flag nil)
               (setq string (pop new))
               (setq table (pop new))
               (setq point (pop new))
@@ -1069,17 +1093,35 @@ completion--nth-completion
          (result-and-style
           (completion--some
            (lambda (style)
-             (let ((probe (funcall (nth n (assq style
-                                                completion-styles-alist))
-                                   string table pred point)))
+             (let* ((fun (nth n (assq style completion-styles-alist)))
+                    ;; Transparently upgrade the return value for
+                    ;; existing built-in styles as of Emacs 28.  No
+                    ;; new styles should be added here. New completion
+                    ;; styles should directly return the new
+                    ;; completion format.el
+                    (completion--return-alist-flag
+                     (and completion--return-alist-flag
+                          (memq style '(emacs21 emacs22 basic substring
+                                        partial-completion initials flex))))
+                    (probe (funcall fun string table pred point)))
                (and probe (cons probe style))))
            (completion--styles md)))
-         (adjust-fn (get (cdr result-and-style) 'completion--adjust-metadata)))
-    (when (and adjust-fn metadata)
-      (setcdr metadata (cdr (funcall adjust-fn metadata))))
+         (style-md (get (cdr result-and-style) 'completion--style-metadata))
+         (result (car result-and-style)))
+    (when (and style-md metadata)
+      (setcdr metadata (cdr (funcall style-md
+                                     string table pred point metadata))))
+    (when (and (not completion--return-alist-flag) (= n 2) (consp (car result)))
+      ;; Give the completion styles some freedom!  If they are
+      ;; targeting Emacs 28 upwards only, they may return a result
+      ;; with deferred highlighting.  We convert back to the old
+      ;; format here by applying the highlighting eagerly.
+      (setq result (nconc (funcall (cdr (assq 'highlight result))
+                                   (cdr (assq 'completions result)))
+                          (cdr (assq 'base result)))))
     (if requote
-        (funcall requote (car result-and-style) n)
-      (car result-and-style))))
+        (funcall requote result n)
+      result)))
 
 (defun completion-try-completion (string table pred point &optional metadata)
   "Try to complete STRING using completion table TABLE.
@@ -1088,7 +1130,8 @@ completion-try-completion
 The return value can be either nil to indicate that there is no completion,
 t to indicate that STRING is the only possible completion,
 or a pair (NEWSTRING . NEWPOINT) of the completed result string together with
-a new position for point."
+a new position for point.
+The METADATA may be modified by the completion style."
   (completion--nth-completion 1 string table pred point metadata))
 
 (defun completion-all-completions (string table pred point &optional metadata)
@@ -1096,10 +1139,47 @@ completion-all-completions
 Only the elements of table that satisfy predicate PRED are considered.
 POINT is the position of point within STRING.
 The return value is a list of completions and may contain the base-size
-in the last `cdr'."
-  ;; FIXME: We need to additionally return the info needed for the
-  ;; second part of completion-base-position.
-  (completion--nth-completion 2 string table pred point metadata))
+in the last `cdr'.
+The METADATA may be modified by the completion style.
+
+This function has been superseded by `completion-filter-completions',
+which returns richer information and supports deferred candidate
+highlighting."
+  (let ((completion--return-alist-flag nil))
+    (completion--nth-completion 2 string table pred point metadata)))
+
+(defun completion-filter-completions (string table pred point metadata)
+  "Filter the possible completions of STRING in completion table TABLE.
+Only the elements of table that satisfy predicate PRED are considered.
+POINT is the position of point within STRING.
+The METADATA may be modified by the completion style.
+The return value is a alist with the keys:
+
+- base: Base position of the completion (from the start of STRING)
+- end: End position of the completion (from the start of STRING)
+- highlight: Highlighting function taking a list of completions and
+  returning a new list of new strings with applied highlighting.
+- completions: The list of completions.
+
+This function supersedes the function `completion-all-completions',
+which does not provide the `end' position of the completion and does
+not support deferred highlighting."
+  (let* ((completion--return-alist-flag t)
+         (result (completion--nth-completion 2 string table
+                                             pred point metadata)))
+    (if (and result (not (consp (car result))))
+        ;; Deferred highlighting has been requested, but the
+        ;; completion style returned a non-deferred result.  Convert
+        ;; the result to the alist format of
+        ;; `completion-filter-completions'.
+        (let* ((last (last result))
+               (base (or (cdr last) 0)))
+          (setcdr last nil)
+          `((base . ,base)
+            (end . ,(length string))
+            (highlight . identity)
+            (completions . ,result)))
+      result)))
 
 (defun minibuffer--bitset (modified completions exact)
   (logior (if modified    4 0)
@@ -1115,7 +1195,8 @@ completion--replace
   (if minibuffer-allow-text-properties
       ;; If we're preserving properties, then just remove the faces
       ;; and other properties added by the completion machinery.
-      (remove-text-properties 0 (length newtext) '(face completion-score)
+      (remove-text-properties 0 (length newtext)
+                              '(face nil completion-score nil)
                               newtext)
     ;; Remove all text properties.
     (set-text-properties 0 (length newtext) nil newtext))
@@ -2021,34 +2102,49 @@ completion-hilit-commonality
 It returns a list with font-lock properties applied to each element,
 and with BASE-SIZE appended as the last element."
   (when completions
-    (let ((com-str-len (- prefix-len (or base-size 0))))
-      (nconc
-       (mapcar
-        (lambda (elem)
-          (let ((str
-                 ;; Don't modify the string itself, but a copy, since the
-                 ;; the string may be read-only or used for other purposes.
-                 ;; Furthermore, since `completions' may come from
-                 ;; display-completion-list, `elem' may be a list.
-                 (if (consp elem)
-                     (car (setq elem (cons (copy-sequence (car elem))
-                                           (cdr elem))))
-                   (setq elem (copy-sequence elem)))))
-            (font-lock-prepend-text-property
-             0
-             ;; If completion-boundaries returns incorrect
-             ;; values, all-completions may return strings
-             ;; that don't contain the prefix.
-             (min com-str-len (length str))
-             'face 'completions-common-part str)
-            (if (> (length str) com-str-len)
-                (font-lock-prepend-text-property com-str-len (1+ com-str-len)
-                                                 'face
-                                                 'completions-first-difference
-                                                 str)))
-          elem)
-        completions)
-       base-size))))
+    (nconc
+     (completion--hilit-commonality (- prefix-len (or base-size 0)) completions)
+     base-size)))
+
+(defun completion--hilit-commonality (com-size completions)
+  (mapcar
+   (lambda (elem)
+     (let ((str
+            ;; Don't modify the string itself, but a copy, since the
+            ;; the string may be read-only or used for other purposes.
+            ;; Furthermore, since `completions' may come from
+            ;; display-completion-list, `elem' may be a list.
+            (if (consp elem)
+                (car (setq elem (cons (copy-sequence (car elem))
+                                      (cdr elem))))
+              (setq elem (copy-sequence elem)))))
+       (font-lock-prepend-text-property
+        0
+        ;; If completion-boundaries returns incorrect
+        ;; values, all-completions may return strings
+        ;; that don't contain the prefix.
+        (min com-size (length str))
+        'face 'completions-common-part str)
+       (if (> (length str) com-size)
+           (font-lock-prepend-text-property com-size (1+ com-size)
+                                            'face
+                                            'completions-first-difference
+                                            str)))
+     elem)
+   completions))
+
+(defun completion--deferred-hilit (completions prefix-len base end)
+  "Return completions as a list or as an alist.
+If `completion--return-alist-flag' is non-nil use the alist format of
+`completion-filter-completions'."
+  (if completion--return-alist-flag
+      (when completions
+        `((base . ,base)
+          (end . ,end)
+          (highlight . ,(apply-partially #'completion--hilit-commonality
+                                         (- prefix-len base)))
+          (completions . ,completions)))
+    (completion-hilit-commonality completions prefix-len base)))
 
 (defun display-completion-list (completions &optional common-substring group-fun)
   "Display the list of completions, COMPLETIONS, using `standard-output'.
@@ -2163,15 +2259,16 @@ minibuffer-completion-help
          (end (or end (point-max)))
          (string (buffer-substring start end))
          (md (completion--field-metadata start))
-         (completions (completion-all-completions
-                       string
-                       minibuffer-completion-table
-                       minibuffer-completion-predicate
-                       (- (point) start)
-                       md)))
+         (filtered-completions (completion-filter-completions
+                                string
+                                minibuffer-completion-table
+                                minibuffer-completion-predicate
+                                (- (point) start)
+                                md))
+         (completions (alist-get 'completions filtered-completions)))
     (message nil)
     (if (or (null completions)
-            (and (not (consp (cdr completions)))
+            (and (not (cdr completions))
                  (equal (car completions) string)))
         (progn
           ;; If there are no completions, or if the current input is already
@@ -2181,8 +2278,7 @@ minibuffer-completion-help
           (completion--message
            (if completions "Sole completion" "No completions")))
 
-      (let* ((last (last completions))
-             (base-size (or (cdr last) 0))
+      (let* ((base-size (alist-get 'base filtered-completions))
              (prefix (unless (zerop base-size) (substring string 0 base-size)))
              (all-md (completion--metadata (buffer-substring-no-properties
                                             start (point))
@@ -2226,9 +2322,12 @@ minibuffer-completion-help
             (body-function
              . ,#'(lambda (_window)
                     (with-current-buffer mainbuf
-                      ;; Remove the base-size tail because `sort' requires a properly
-                      ;; nil-terminated list.
-                      (when last (setcdr last nil))
+                      ;; Apply highlighting using the deferred
+                      ;; highlighting function provided by
+                      ;; `completion-format-completions'.
+                      (setq completions
+                            (funcall (alist-get 'highlight filtered-completions)
+                                     completions))
 
                       ;; Sort first using the `display-sort-function'.
                       ;; FIXME: This function is for the output of
@@ -2267,13 +2366,10 @@ minibuffer-completion-help
                                       completions))))
 
                       (with-current-buffer standard-output
-                        (setq-local completion-base-position
-                             (list (+ start base-size)
-                                   ;; FIXME: We should pay attention to completion
-                                   ;; boundaries here, but currently
-                                   ;; completion-all-completions does not give us the
-                                   ;; necessary information.
-                                   end))
+                        (setq-local
+                         completion-base-position
+                         (list (+ start base-size)
+                               (+ start (alist-get 'end filtered-completions))))
                         (setq-local completion-list-insert-choice-function
                              (let ((ctable minibuffer-completion-table)
                                    (cpred minibuffer-completion-predicate)
@@ -3223,10 +3319,11 @@ completion-emacs21-try-completion
       completion)))
 
 (defun completion-emacs21-all-completions (string table pred _point)
-  (completion-hilit-commonality
+  (completion--deferred-hilit
    (all-completions string table pred)
    (length string)
-   (car (completion-boundaries string table pred ""))))
+   (car (completion-boundaries string table pred ""))
+   (length string)))
 
 (defun completion-emacs22-try-completion (string table pred point)
   (let ((suffix (substring string point))
@@ -3249,11 +3346,12 @@ completion-emacs22-try-completion
       (cons (concat completion suffix) (length completion)))))
 
 (defun completion-emacs22-all-completions (string table pred point)
-  (let ((beforepoint (substring string 0 point)))
-    (completion-hilit-commonality
+  (let* ((beforepoint (substring string 0 point))
+         (afterpoint (substring string point))
+         (bounds (completion-boundaries beforepoint table pred afterpoint)))
+    (completion--deferred-hilit
      (all-completions beforepoint table pred)
-     point
-     (car (completion-boundaries beforepoint table pred "")))))
+     point (car bounds) (+ point (cdr bounds)))))
 
 ;;; Basic completion.
 
@@ -3312,7 +3410,7 @@ completion-basic-all-completions
                             'point
                             (substring afterpoint 0 (cdr bounds)))))
          (all (completion-pcm--all-completions prefix pattern table pred)))
-    (completion-hilit-commonality all point (car bounds))))
+    (completion--deferred-hilit all point (car bounds) (+ point (cdr bounds)))))
 
 ;;; Partial-completion-mode style completion.
 
@@ -3504,13 +3602,26 @@ flex-score-match-tightness
 than the latter (which has two \"holes\" and three
 one-letter-long matches).")
 
-(defun completion-pcm--hilit-commonality (pattern completions)
+(defun completion-pcm--deferred-hilit (pattern completions base end)
+  "Return completions as a list or as an alist.
+If `completion--return-alist-flag' is non-nil use the alist format of
+`completion-filter-completions'."
+  (when completions
+    (if completion--return-alist-flag
+        `((base . ,base)
+          (end . ,end)
+          (highlight . ,(apply-partially
+                         #'completion-pcm--hilit-commonality
+                         pattern))
+          (completions . ,completions))
+      (nconc (completion-pcm--hilit-commonality pattern completions 'score) base))))
+
+(defun completion-pcm--hilit-commonality (pattern completions &optional score)
   "Show where and how well PATTERN matches COMPLETIONS.
 PATTERN, a list of symbols and strings as seen
 `completion-pcm--merge-completions', is assumed to match every
 string in COMPLETIONS.  Return a deep copy of COMPLETIONS where
-each string is propertized with `completion-score', a number
-between 0 and 1, and with faces `completions-common-part',
+each string is propertized with faces `completions-common-part',
 `completions-first-difference' in the relevant segments."
   (when completions
     (let* ((re (completion-pcm--pattern->regex pattern 'group))
@@ -3527,84 +3638,143 @@ completion-pcm--hilit-commonality
                 (match-end (match-end 0))
                 (md (cddr (setq last-md (match-data t last-md))))
                 (from 0)
-                (end (length str))
-                ;; To understand how this works, consider these simple
-                ;; ascii diagrams showing how the pattern "foo"
-                ;; flex-matches "fabrobazo", "fbarbazoo" and
-                ;; "barfoobaz":
-
-                ;;      f abr o baz o
-                ;;      + --- + --- +
-
-                ;;      f barbaz oo
-                ;;      + ------ ++
-
-                ;;      bar foo baz
-                ;;          +++
-
-                ;; "+" indicates parts where the pattern matched.  A
-                ;; "hole" in the middle of the string is indicated by
-                ;; "-".  Note that there are no "holes" near the edges
-                ;; of the string.  The completion score is a number
-                ;; bound by ]0..1]: the higher the better and only a
-                ;; perfect match (pattern equals string) will have
-                ;; score 1.  The formula takes the form of a quotient.
-                ;; For the numerator, we use the number of +, i.e. the
-                ;; length of the pattern.  For the denominator, it
-                ;; first computes
-                ;;
-                ;;     hole_i_contrib = 1 + (Li-1)^(1/tightness)
-                ;;
-                ;; , for each hole "i" of length "Li", where tightness
-                ;; is given by `flex-score-match-tightness'.  The
-                ;; final value for the denominator is then given by:
-                ;;
-                ;;    (SUM_across_i(hole_i_contrib) + 1) * len
-                ;;
-                ;; , where "len" is the string's length.
-                (score-numerator 0)
-                (score-denominator 0)
-                (last-b 0)
-                (update-score-and-face
-                 (lambda (a b)
-                   "Update score and face given match range (A B)."
-                   (add-face-text-property a b
-                                           'completions-common-part
-                                           nil str)
-                   (setq
-                    score-numerator   (+ score-numerator (- b a)))
-                   (unless (or (= a last-b)
-                               (zerop last-b)
-                               (= a (length str)))
-                     (setq
-                      score-denominator (+ score-denominator
-                                           1
-                                           (expt (- a last-b 1)
-                                                 (/ 1.0
-                                                    flex-score-match-tightness)))))
-                   (setq
-                    last-b              b))))
+                (len (length str)))
+           (when (and score (/= 0 len))
+             (put-text-property
+              0 1 'completion-score (- (completion--flex-score-1 md match-end len)) str))
            (while md
-             (funcall update-score-and-face from (pop md))
+             (add-face-text-property from (pop md)
+                                     'completions-common-part
+                                     nil str)
              (setq from (pop md)))
            ;; If `pattern' doesn't have an explicit trailing any, the
            ;; regex `re' won't produce match data representing the
            ;; region after the match.  We need to account to account
            ;; for that extra bit of match (bug#42149).
            (unless (= from match-end)
-             (funcall update-score-and-face from match-end))
-           (if (> (length str) pos)
+             (add-face-text-property from match-end
+                                     'completions-common-part
+                                     nil str))
+           (if (> len pos)
                (add-face-text-property
                 pos (1+ pos)
                 'completions-first-difference
-                nil str))
-           (unless (zerop (length str))
-             (put-text-property
-              0 1 'completion-score
-              (/ score-numerator (* end (1+ score-denominator)) 1.0) str)))
+                nil str)))
          str)
        completions))))
 
+(defun completion--flex-score-1 (md match-end len)
+  "Compute matching score of completion.
+The score lies in the range between-1 and 0, where -1 corresponds to
+the full match.
+MD is the match data.
+MATCH-END is the end of the match.
+LEN is the length of the completion string."
+  (let* ((from 0)
+         ;; To understand how this works, consider these simple
+         ;; ascii diagrams showing how the pattern "foo"
+         ;; flex-matches "fabrobazo", "fbarbazoo" and
+         ;; "barfoobaz":
+
+         ;;      f abr o baz o
+         ;;      + --- + --- +
+
+         ;;      f barbaz oo
+         ;;      + ------ ++
+
+         ;;      bar foo baz
+         ;;          +++
+
+         ;; "+" indicates parts where the pattern matched.  A
+         ;; "hole" in the middle of the string is indicated by
+         ;; "-".  Note that there are no "holes" near the edges
+         ;; of the string.  The completion score is a number
+         ;; bound by ]0..1]: the higher the better and only a
+         ;; perfect match (pattern equals string) will have
+         ;; score 1.  The formula takes the form of a quotient.
+         ;; For the numerator, we use the number of +, i.e. the
+         ;; length of the pattern.  For the denominator, it
+         ;; first computes
+         ;;
+         ;;     hole_i_contrib = 1 + (Li-1)^(1/tightness)
+         ;;
+         ;; , for each hole "i" of length "Li", where tightness
+         ;; is given by `flex-score-match-tightness'.  The
+         ;; final value for the denominator is then given by:
+         ;;
+         ;;    (SUM_across_i(hole_i_contrib) + 1) * len
+         ;;
+         ;; , where "len" is the string's length.
+         (score-numerator 0)
+         (score-denominator 0)
+         (last-b 0))
+    (while md
+      (let ((a from)
+            (b (pop md)))
+        (setq
+         score-numerator   (+ score-numerator (- b a)))
+        (unless (or (= a last-b)
+                    (zerop last-b)
+                    (= a len))
+          (setq
+           score-denominator (+ score-denominator
+                                1
+                                (expt (- a last-b 1)
+                                      (/ 1.0
+                                         flex-score-match-tightness)))))
+        (setq
+         last-b              b))
+      (setq from (pop md)))
+    ;; If `pattern' doesn't have an explicit trailing any, the
+    ;; regex `re' won't produce match data representing the
+    ;; region after the match.  We need to account to account
+    ;; for that extra bit of match (bug#42149).
+    (unless (= from match-end)
+      (let ((a from)
+            (b match-end))
+        (setq
+         score-numerator   (+ score-numerator (- b a)))
+        (unless (or (= a last-b)
+                    (zerop last-b)
+                    (= a len))
+          (setq
+           score-denominator (+ score-denominator
+                                1
+                                (expt (- a last-b 1)
+                                      (/ 1.0
+                                         flex-score-match-tightness)))))
+        (setq
+         last-b              b)))
+    (- (/ score-numerator (* len (1+ score-denominator)) 1.0))))
+
+(defun completion--flex-score (pattern completions)
+  "Compute how well PATTERN matches COMPLETIONS.
+PATTERN, a pcm pattern is assumed to match every string in the
+COMPLETIONS list.  Return a copy of COMPLETIONS where each element is
+a pair of a score and the string.  The score lies in the range between
+-1 and 0, where -1 corresponds to the full match."
+  (when completions
+    (let* ((re (completion-pcm--pattern->regex pattern 'group))
+           (case-fold-search completion-ignore-case)
+           last-md)
+      (mapcar
+       (lambda (str)
+         ;; The flex completion style requires the completion to match
+         ;; the pattern to compute the scoring.  For quoted completion
+         ;; tables the completions are matched against the *unquoted
+         ;; input string*.  However `completion-all-completions' and
+         ;; `completion-filter-completions' return a list of *quoted
+         ;; completions*, which is subsequently sorted.  Therefore we
+         ;; obtain the unquoted completion string which is stored in
+         ;; the text property `completion--unquoted'.
+         (let ((unquoted (or (get-text-property 0 'completion--unquoted str) str)))
+           (unless (string-match re unquoted)
+             (error "Internal error: %s does not match %s" re unquoted))
+           (cons (completion--flex-score-1 (cddr (setq last-md (match-data t last-md)))
+                                           (match-end 0) (length unquoted))
+                 str)))
+       completions))))
+
 (defun completion-pcm--find-all-completions (string table pred point
                                                     &optional filter)
   "Find all completions for STRING at POINT in TABLE, satisfying PRED.
@@ -3700,11 +3870,11 @@ completion-pcm--find-all-completions
         (list pattern all prefix suffix)))))
 
 (defun completion-pcm-all-completions (string table pred point)
-  (pcase-let ((`(,pattern ,all ,prefix ,_suffix)
+  (pcase-let ((`(,pattern ,all ,prefix ,suffix)
                (completion-pcm--find-all-completions string table pred point)))
-    (when all
-      (nconc (completion-pcm--hilit-commonality pattern all)
-             (length prefix)))))
+    (completion-pcm--deferred-hilit pattern all
+                                    (length prefix)
+                                    (- (length string) (length suffix)))))
 
 (defun completion--common-suffix (strs)
   "Return the common suffix of the strings STRS."
@@ -3885,8 +4055,8 @@ completion-pcm-try-completion
 ;;; Substring completion
 ;; Mostly derived from the code of `basic' completion.
 
-(defun completion-substring--all-completions
-    (string table pred point &optional transform-pattern-fn)
+(defun completion--pattern-compiler
+    (string table pred point transform-pattern-fn)
   "Match the presumed substring STRING to the entries in TABLE.
 Respect PRED and POINT.  The pattern used is a PCM-style
 substring pattern, but it be massaged by TRANSFORM-PATTERN-FN, if
@@ -3904,12 +4074,23 @@ completion-substring--all-completions
          (pattern (completion-pcm--optimize-pattern
                    (if transform-pattern-fn
                        (funcall transform-pattern-fn pattern)
-                     pattern)))
-         (all (completion-pcm--all-completions prefix pattern table pred)))
-    (list all pattern prefix suffix (car bounds))))
+                     pattern))))
+    (list pattern prefix suffix)))
+
+(defun completion-substring--all-completions
+    (string table pred point &optional transform-pattern-fn)
+  "Match the presumed substring STRING to the entries in TABLE.
+Respect PRED and POINT.  The pattern used is a PCM-style
+substring pattern, but it be massaged by TRANSFORM-PATTERN-FN, if
+that is non-nil."
+  (pcase-let (((and result `(,pattern ,prefix ,_suffix))
+               (completion--pattern-compiler string table pred point
+                                             transform-pattern-fn)))
+    (cons (completion-pcm--all-completions prefix pattern table pred)
+          result)))
 
 (defun completion-substring-try-completion (string table pred point)
-  (pcase-let ((`(,all ,pattern ,prefix ,suffix ,_carbounds)
+  (pcase-let ((`(,all ,pattern ,prefix ,suffix)
                (completion-substring--all-completions
                 string table pred point)))
     (if minibuffer-completing-file-name
@@ -3917,12 +4098,12 @@ completion-substring-try-completion
     (completion-pcm--merge-try pattern all prefix suffix)))
 
 (defun completion-substring-all-completions (string table pred point)
-  (pcase-let ((`(,all ,pattern ,prefix ,_suffix ,_carbounds)
+  (pcase-let ((`(,all ,pattern ,prefix ,suffix)
                (completion-substring--all-completions
                 string table pred point)))
-    (when all
-      (nconc (completion-pcm--hilit-commonality pattern all)
-             (length prefix)))))
+    (completion-pcm--deferred-hilit pattern all
+                                    (length prefix)
+                                    (- (length string) (length suffix)))))
 
 ;;; "flex" completion, also known as flx/fuzzy/scatter completion
 ;; Completes "foo" to "frodo" and "farfromsober"
@@ -3932,42 +4113,53 @@ completion-flex-nospace
   :version "27.1"
   :type 'boolean)
 
-(put 'flex 'completion--adjust-metadata 'completion--flex-adjust-metadata)
-
-(defun completion--flex-adjust-metadata (metadata)
-  (cl-flet
-      ((compose-flex-sort-fn
-        (existing-sort-fn) ; wish `cl-flet' had proper indentation...
-        (lambda (completions)
-          (let ((pre-sorted
-                 (if existing-sort-fn
-                     (funcall existing-sort-fn completions)
-                   completions)))
-            (cond
-             ((or (not (window-minibuffer-p))
-                  ;; JT@2019-12-23: FIXME: this is still wrong.  What
-                  ;; we need to test here is "some input that actually
-                  ;; leads to flex filtering", not "something after
-                  ;; the minibuffer prompt".  Among other
-                  ;; inconsistencies, the latter is always true for
-                  ;; file searches, meaning the next clauses will be
-                  ;; ignored.
-                  (> (point-max) (minibuffer-prompt-end)))
-              (sort
-               pre-sorted
-               (lambda (c1 c2)
-                 (let ((s1 (get-text-property 0 'completion-score c1))
-                       (s2 (get-text-property 0 'completion-score c2)))
-                   (> (or s1 0) (or s2 0))))))
-             (t pre-sorted))))))
-    `(metadata
-      (display-sort-function
-       . ,(compose-flex-sort-fn
-           (completion-metadata-get metadata 'display-sort-function)))
-      (cycle-sort-function
-       . ,(compose-flex-sort-fn
-           (completion-metadata-get metadata 'cycle-sort-function)))
-      ,@(cdr metadata))))
+(put 'flex 'completion--style-metadata 'completion--flex-style-metadata)
+
+(defun completion--flex-style-metadata (string table pred point metadata)
+  ;; Use the modified flex sorting function only for non-empty input.
+  ;; In an older version of `completion--flex-adjust-metadata', the
+  ;; check (> (point-max) (minibuffer-prompt-end))) was used instead.
+  (unless (eq string "")
+    (let ((pattern (car (completion--pattern-compiler
+                         string table pred point
+                         #'completion-flex--make-flex-pattern))))
+      (cl-flet
+          ((compose-flex-sort-fn
+            (existing-sort-fn) ; wish `cl-flet' had proper indentation...
+            (lambda (completions)
+              (let ((pre-sorted (if existing-sort-fn
+                                    (funcall existing-sort-fn completions)
+                                  completions)))
+                ;; If `completion-scores' are already present use
+                ;; those instead of recomputing the scores with
+                ;; `completion--flex-score'.  The scores are already
+                ;; present, when the candidates have been computed by
+                ;; `completion-all-completions'.  In contrast, the
+                ;; score is not yet present, when the candidates have
+                ;; been computed by `completion-filter-completions'.
+                (if (and (car pre-sorted)
+                         (get-text-property 0 'completion-score (car pre-sorted)))
+                    (sort
+                     pre-sorted
+                     (lambda (c1 c2)
+                       (> (or (get-text-property 0 'completion-score c1) 0)
+                          (or (get-text-property 0 'completion-score c2) 0))))
+                  (let* ((sorted (sort (completion--flex-score pattern pre-sorted)
+                                       #'car-less-than-car))
+                         (cell sorted))
+                    ;; Remove score decorations, reuse the list to avoid allocations.
+                    (while cell
+                      (setcar cell (cdar cell))
+                      (pop cell))
+                    sorted))))))
+        `(metadata
+          (display-sort-function
+           . ,(compose-flex-sort-fn
+               (completion-metadata-get metadata 'display-sort-function)))
+          (cycle-sort-function
+           . ,(compose-flex-sort-fn
+               (completion-metadata-get metadata 'cycle-sort-function)))
+          ,@(cdr metadata))))))
 
 (defun completion-flex--make-flex-pattern (pattern)
   "Convert PCM-style PATTERN into PCM-style flex pattern.
@@ -3989,7 +4181,7 @@ completion-flex--make-flex-pattern
 (defun completion-flex-try-completion (string table pred point)
   "Try to flex-complete STRING in TABLE given PRED and POINT."
   (unless (and completion-flex-nospace (string-search " " string))
-    (pcase-let ((`(,all ,pattern ,prefix ,suffix ,_carbounds)
+    (pcase-let ((`(,all ,pattern ,prefix ,suffix)
                  (completion-substring--all-completions
                   string table pred point
                   #'completion-flex--make-flex-pattern)))
@@ -4006,13 +4198,13 @@ completion-flex-try-completion
 (defun completion-flex-all-completions (string table pred point)
   "Get flex-completions of STRING in TABLE, given PRED and POINT."
   (unless (and completion-flex-nospace (string-search " " string))
-    (pcase-let ((`(,all ,pattern ,prefix ,_suffix ,_carbounds)
+    (pcase-let ((`(,all ,pattern ,prefix ,suffix)
                  (completion-substring--all-completions
                   string table pred point
                   #'completion-flex--make-flex-pattern)))
-      (when all
-        (nconc (completion-pcm--hilit-commonality pattern all)
-               (length prefix))))))
+      (completion-pcm--deferred-hilit pattern all
+                                    (length prefix)
+                                    (- (length string) (length suffix))))))
 
 ;; Initials completion
 ;; Complete /ums to /usr/monnier/src or lch to list-command-history.
@@ -4049,7 +4241,11 @@ completion-initials-expand
 (defun completion-initials-all-completions (string table pred _point)
   (let ((newstr (completion-initials-expand string table pred)))
     (when newstr
-      (completion-pcm-all-completions newstr table pred (length newstr)))))
+      (pcase-let ((`(,pattern ,all ,prefix ,_suffix)
+                   (completion-pcm--find-all-completions newstr table
+                                                         pred (length newstr))))
+        (completion-pcm--deferred-hilit pattern all
+                                        (length prefix) (length string))))))
 
 (defun completion-initials-try-completion (string table pred _point)
   (let ((newstr (completion-initials-expand string table pred)))
diff --git a/test/lisp/minibuffer-tests.el b/test/lisp/minibuffer-tests.el
index c3ba8f9a92..22de3c9ff5 100644
--- a/test/lisp/minibuffer-tests.el
+++ b/test/lisp/minibuffer-tests.el
@@ -28,8 +28,7 @@
 
 (require 'ert)
 (require 'ert-x)
-
-(eval-when-compile (require 'cl-lib))
+(require 'cl-lib)
 
 (ert-deftest completion-test1 ()
   (with-temp-buffer
@@ -331,5 +330,219 @@ completion-flex-test-3
                   "custgroup" '("customize-group-other-window") nil 9)))
            15)))
 
+(ert-deftest completion-flex-score-test-1 ()
+  ;; Full match!
+  (should (equal
+           (completion--flex-score '(prefix "R") '("R"))
+           (list (cons -1.0 "R")))))
+
+(ert-deftest completion-flex-score-test-2 ()
+  ;; One third and half of a match!
+  (should (equal
+           (completion--flex-score '(prefix "foo")
+                                   '("barfoobar" "fooboo"))
+           (list (cons (/ -1.0 3.0) "barfoobar")
+                 (cons (/ -1.0 2.0) "fooboo")))))
+
+(ert-deftest completion-flex-score-test-3 ()
+  ;; One fourth of a match
+  (should (eql
+           (caar (completion--flex-score '(prefix "R" point "O")
+                                         '("RaOb")))
+           (/ -1.0 4.0))))
+
+(ert-deftest completion-flex-score-test-4 ()
+  ;; For quoted completion tables, score the unquoted completion string.
+  (should (equal
+           (completion--flex-score
+            '(prefix "R")
+            (list (propertize "X" 'completion--unquoted "R")))
+           (list (cons -1.0 "X")))))
+
+(defun completion--test-style (style string point table filtered)
+  (let* ((completion-styles (list style))
+         (pred (lambda (x) (not (string-search "!" x))))
+         (result (completion-filter-completions
+                  string table pred point nil)))
+    (should (equal (alist-get 'base result) 0))
+    (should (equal (alist-get 'end result) (length string)))
+    (should (equal (alist-get 'completions result) filtered))
+    ;; The highlighting function should be present.
+    (should (not (memq (alist-get 'highlight result) '(nil identity))))
+    ;; Equal results as `completion-all-completions'.
+    (should (equal (completion-all-completions string table pred point)
+                   (append filtered 0)))
+    ;; The returned strings should be identical to the original strings.
+    ;; The `completion-filter-completions' function avoids allocations!
+    (should (cl-intersection (alist-get 'completions result)
+                             table :test #'eq))))
+
+(ert-deftest completion-basic-style-test-1 ()
+  ;; point at the beginning |foo
+  (completion--test-style 'basic "foo" 0
+                          '("foobar" "foo!" "barfoo" "xfooy" "boobar")
+                          '("foobar" "barfoo" "xfooy")))
+
+(ert-deftest completion-basic-style-test-2 ()
+  ;; point foo
+  (completion--test-style 'basic "foo" 2
+                          '("foobar" "foo!" "fobar" "barfoo" "xfooy" "boobar")
+                          '("foobar")))
+
+(ert-deftest completion-substring-style-test ()
+  (completion--test-style 'substring "foo" 1
+                          '("foobar" "foo!" "barfoo" "xfooy" "boobar")
+                          '("foobar" "barfoo" "xfooy")))
+
+(ert-deftest completion-emacs21-style-test ()
+  (completion--test-style 'emacs21 "foo" 1
+                          '("foobar" "foo!" "fobar" "barfoo" "xfooy" "boobar")
+                          '("foobar")))
+
+(ert-deftest completion-emacs22-style-test ()
+  (completion--test-style 'emacs22 "fo0" 1
+                          '("foobar" "foo!" "fobar" "barfoo" "xfooy" "boobar")
+                          '("foobar" "fobar"))) ;; suffix ignored completely
+
+(ert-deftest completion-flex-style-test ()
+  (completion--test-style 'flex "abc" 1
+                          '("abc" "abc!" "xaybzc" "xaybz")
+                          '("abc" "xaybzc")))
+
+(ert-deftest completion-initials-style-test ()
+  (completion--test-style 'initials "abc" 1
+                          '("a-b-c" "a-b-c!" "ax-by-cz" "xax-by-cz")
+                          '("a-b-c" "ax-by-cz")))
+
+(ert-deftest completion-pcm-style-test ()
+  (completion--test-style 'partial-completion "ax-b-c" 1
+                          '("ax-b-c" "ax-b-c!" "ax-by-cz" "xax-by-cz")
+                          '("ax-b-c" "ax-by-cz")))
+
+(ert-deftest completion-filter-completions-highlight-test ()
+  ;; point at the beginning |foo
+  (let* ((completion-styles '(basic))
+         (result (completion-filter-completions
+                  "foo" '("foobar" "fbarfoo" "fxfooy" "bar")
+                  nil 1 nil)))
+    (should (equal
+             (format "%S" (alist-get 'completions result))
+             (format "%S" '("foobar" "fbarfoo" "fxfooy"))))
+    (should (equal
+             (format "%S" (funcall (alist-get 'highlight result)
+                                   (alist-get 'completions result)))
+             (format "%S"
+                     '(#("foobar" 0 1 (face (completions-common-part))
+                         1 2 (face (completions-first-difference)))
+                       #("fbarfoo" 0 1 (face (completions-common-part))
+                         1 2 (face (completions-first-difference)))
+                       #("fxfooy" 0 1 (face (completions-common-part))
+                         1 2 (face (completions-first-difference)))))))))
+
+(defun completion--test-boundaries (style string table result)
+  (let ((table
+         (lambda (str pred action)
+           (pcase action
+             (`(boundaries . ,suffix) `(boundaries
+                                        ,(1+ (string-match-p "<\\|/" str))
+                                        . ,(or (string-search ">" suffix) (length suffix))))
+             (_ (complete-with-action action table
+                                      (replace-regexp-in-string ".*[</]" "" str)
+                                      pred)))))
+        (point (string-search "|" string))
+        (string (string-replace "|" "" string))
+        (completion-styles (list style)))
+    (should (equal
+             (assq-delete-all
+              (if (assq 'highlight result) '-does-not-exist 'highlight)
+              (completion-filter-completions
+               string table nil point nil))
+             result))
+    (should (equal
+             (completion-all-completions
+              string table nil point)
+             (append (alist-get 'completions result)
+                     (alist-get 'base result))))))
+
+(ert-deftest completion-emacs21-boundaries-test ()
+  (completion--test-boundaries 'emacs21 "before<in|put>after"
+                               '("other") nil)
+  (completion--test-boundaries 'emacs21 "before<in|put>after"
+                               '("ainput>after" "input>after" "inpux>after"
+                                 "inxputy>after" "input>after2")
+                               '((base . 7)
+                                 (end . 18)
+                                 (completions "input>after" "input>after2"))))
+
+(ert-deftest completion-emacs22-boundaries-test ()
+  (completion--test-boundaries 'emacs22 "before<in|put>after"
+                               '("other") nil)
+  (completion--test-boundaries 'emacs22 "before<in|put>after"
+                               '("ainxxx" "inyy" "inzzz")
+                               '((base . 7)
+                                 (end . 12)
+                                 (completions "inyy" "inzzz"))))
+
+(ert-deftest completion-basic-boundaries-test ()
+  (completion--test-boundaries 'basic "before<in|put>after"
+                               '("other") nil)
+  (completion--test-boundaries 'basic "before<in|put>after"
+                               '("ainput" "input" "inpux" "inxputy")
+                               '((base . 7)
+                                 (end . 12)
+                                 (completions "input" "inxputy"))))
+
+(ert-deftest completion-substring-boundaries-test ()
+  (completion--test-boundaries 'substring "before<in|puts>after"
+                               '("other") nil)
+  (completion--test-boundaries 'substring "before<in|puts>after"
+                               '("ainputs" "inputs" "inpux" "inxputsy")
+                               '((base . 7)
+                                 (end . 13)
+                                 (completions "ainputs" "inputs" "inxputsy"))))
+
+(ert-deftest completion-pcm-boundaries-test ()
+  (completion--test-boundaries 'partial-completion "before<in-p|t>after"
+                               '("other") nil)
+  (completion--test-boundaries 'partial-completion "before<in-p|t>after"
+                               '("ain-pu-ts" "in-pts" "in-pu-ts" "in-px" "inx-ptsy")
+                               '((base . 7)
+                                 (end . 12)
+                                 (completions "in-pts" "in-pu-ts" "inx-ptsy"))))
+
+(ert-deftest completion-initials-boundaries-test ()
+  (completion--test-boundaries 'initials "/ip|t"
+                               '("other") nil)
+  (completion--test-boundaries 'initials "/ip|t"
+                               '("ain/pu/ts" "in/pts" "in/pu/ts" "a/in/pu/ts"
+                                 "in/pu/ts/foo" "in/px" "inx/ptsy")
+                               '((base . 1)
+                                 (end . 4)
+                                 (completions "in/pu/ts" "in/pu/ts/foo"))))
+
+(defun completion-emacs22orig-all-completions (string table pred point)
+  (let ((beforepoint (substring string 0 point)))
+    (completion-hilit-commonality
+      (all-completions beforepoint table pred)
+     point
+     (car (completion-boundaries beforepoint table pred "")))))
+
+(ert-deftest completion-upgrade-return-type-test ()
+  ;; Test transparent upgrade of list completion style return value
+  ;; to the alist return value format of `completion-format-completions'.
+  (let ((completion-styles-alist
+         '((emacs22orig completion-emacs22-try-completion
+                        completion-emacs22orig-all-completions nil))))
+  (completion--test-boundaries 'emacs22orig "before<in|put>after"
+                               '("ainxxx" "inyy" "inzzz")
+                               '((base . 7)
+                                 ;; 18 is incorrect, should be 12!
+                                 ;; But the information is not available
+                                 ;; due to the completion-style upgrade.
+                                 (end . 18)
+                                 ;; Identity highlighting function.
+                                 (highlight . identity)
+                                 (completions "inyy" "inzzz")))))
+
 (provide 'minibuffer-tests)
 ;;; minibuffer-tests.el ends here
-- 
2.20.1


--------------E6B7393FF90421271E72F100--




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 12 Aug 2021 09:24:34 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Aug 12 05:24:34 2021
Received: from localhost ([127.0.0.1]:37605 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mE6wz-0001M5-RB
	for submit <at> debbugs.gnu.org; Thu, 12 Aug 2021 05:24:34 -0400
Received: from server.qxqx.de ([178.63.65.180]:51191 helo=mail.qxqx.de)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mail@HIDDEN>)
 id 1mE6ww-0001Lj-MB; Thu, 12 Aug 2021 05:24:31 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=qxqx.de;
 s=mail1392553390; h=Content-Type:In-Reply-To:MIME-Version:Date:Message-ID:
 References:Cc:To:Subject:From:Sender:Reply-To:Content-Transfer-Encoding:
 Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender:
 Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:
 List-Subscribe:List-Post:List-Owner:List-Archive;
 bh=1/NAMhdhCAQoAwSar/mnKa6p1gF9Kj9u70aQDXXvs/I=; b=Vz2rFSIR3R1syGGK8FyEjXXWQk
 /aE6HFJOEWGlR1lQj9n4TNnRltNSdvbHCDMhnPaAq8IYEUTYFvj1uDQnlDzsqR3DpSF4PTU0XrafO
 MGImMuwtL/NgyHisj+6aeYwg6SVQd9IpELjau6Kfk7qpXv1X4HMtV13SIwadGtDQjPaQ=;
From: Daniel Mendler <mail@HIDDEN>
Subject: Re: [PATCH] Add new `completion-filter-completions` API and deferred
 highlighting
To: 47711 <at> debbugs.gnu.org, 48841 <at> debbugs.gnu.org
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <f8c4419a-7e83-b8a9-347c-0221a2052376@HIDDEN>
Message-ID: <fff6e291-bbcc-93e3-92e7-00af4664b8d4@HIDDEN>
Date: Thu, 12 Aug 2021 11:24:22 +0200
MIME-Version: 1.0
In-Reply-To: <f8c4419a-7e83-b8a9-347c-0221a2052376@HIDDEN>
Content-Type: multipart/mixed; boundary="------------4B3E2A198290BEF506AC64C4"
Content-Language: en-US
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 47711
Cc: Eli Zaretskii <eliz@HIDDEN>, Dmitry Gutov <dgutov@HIDDEN>,
 =?UTF-8?B?Sm/Do28gVMOhdm9yYQ==?= <joaotavora@HIDDEN>,
 Stefan Monnier <monnier@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

This is a multi-part message in MIME format.
--------------4B3E2A198290BEF506AC64C4
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

On 8/11/21 6:11 PM, Daniel Mendler wrote:
> 2. In `completion--nth-completion' set `completion--filter-completions'
>    to nil, unless `(memq style '(emacs21 emacs22 basic
>    partial-completion initials flex))' such that custom completion
>    styles which wrap the completion functions don't see the new return
>    value format, except if the custom style opts in explicitly by
>    binding `completion--filter-completions'. An alternative criterion is
>    `(memq fun '(completion-emacs22-all-completions) ...)'. Unfortunately
>    this approach will still not work if the user has advised a
>    `completion-x-all-completions' function. The only 100% safe approach
>    seems to transparently redirect calls to
>    `completion-x-all-completions' to `completion--x-filter-completions',
>    which returns the results in the new format.

I attached two patch variants which can be placed on top of my previous
patch to improve the backward compatibility of the internal API.

Variant 1: Set 'completion--return-alist-flag' only for the existing
completion styles, such that they transparently upgrade to the alist
return format.  If the variable is not set, the completion styles return
the result as plain list retaining backward compatibility.  The variable
is purely for internal use, new completion styles should return their
results as an alist on Emacs 28 and newer.

Variant 2: Add an optional argument FILTER to each of the completion
styles 'all' functions, e.g., 'completion-basic-all-completions'.  In
'completion--nth-completion' try to call the function with the
additional FILTER argument to upgrade to the alist return format.  If
this fails with a 'wrong-number-of-arguments' error, retry again without
the argument.

Daniel

--------------4B3E2A198290BEF506AC64C4
Content-Type: text/plain; charset=UTF-8;
 name="variant1-restrict.el"
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
 filename="variant1-restrict.el"

RnJvbSA1NTM5ZWVhN2RmMTU3MGI2YjUwNDViZTJiMzgzZDJiZmJkNDQ3ODllIE1vbiBTZXAg
MTcgMDA6MDA6MDAgMjAwMQpGcm9tOiBEYW5pZWwgTWVuZGxlciA8bWFpbEBkYW5pZWwtbWVu
ZGxlci5kZT4KRGF0ZTogVGh1LCAxMiBBdWcgMjAyMSAxMDoyMDoxMCArMDIwMApTdWJqZWN0
OiBbUEFUQ0hdIFNldCAnY29tcGxldGlvbi0tcmV0dXJuLWFsaXN0LWZsYWcnIG9ubHkgZm9y
IHRoZSBleGlzdGluZwogc3R5bGVzCgpUaGlzIGF2b2lkcyBwcm9ibGVtcyBpZiBhIHBhY2th
Z2Ugd3JhcHMgYW4gZXhpc3RpbmcgY29tcGxldGlvbgpzdHlsZSBmdW5jdGlvbiBpbiBhIGN1
c3RvbSBjb21wbGV0aW9uIHN0eWxlLgotLS0KIGxpc3AvbWluaWJ1ZmZlci5lbCB8IDY2ICsr
KysrKysrKysrKysrKysrKysrKysrKystLS0tLS0tLS0tLS0tLS0tLS0tLS0KIDEgZmlsZSBj
aGFuZ2VkLCAzNiBpbnNlcnRpb25zKCspLCAzMCBkZWxldGlvbnMoLSkKCmRpZmYgLS1naXQg
YS9saXNwL21pbmlidWZmZXIuZWwgYi9saXNwL21pbmlidWZmZXIuZWwKaW5kZXggYmE4ODU1
YzRlYS4uMzI1NzM0OWExYSAxMDA2NDQKLS0tIGEvbGlzcC9taW5pYnVmZmVyLmVsCisrKyBi
L2xpc3AvbWluaWJ1ZmZlci5lbApAQCAtMTAzOSwxNiArMTAzOSwxNiBAQCBjb21wbGV0aW9u
LS1zdHlsZXMKICAgICAgICAgKGRlbGV0ZS1kdXBzIChhcHBlbmQgKGNkciBvdmVyKSAoY29w
eS1zZXF1ZW5jZSBjb21wbGV0aW9uLXN0eWxlcykpKQogICAgICAgIGNvbXBsZXRpb24tc3R5
bGVzKSkpCiAKLShkZWZ2YXIgY29tcGxldGlvbi0tZmlsdGVyLWNvbXBsZXRpb25zIG5pbAor
KGRlZnZhciBjb21wbGV0aW9uLS1yZXR1cm4tYWxpc3QtZmxhZyBuaWwKICAgIkVuYWJsZSB0
aGUgbmV3IGNvbXBsZXRpb25zIHJldHVybiB2YWx1ZSBmb3JtYXQuCiBJZiB0aGlzIHZhcmlh
YmxlIGlzIG5vbi1uaWwgdGhlIGBhbGwtY29tcGxldGlvbnMnIGZ1bmN0aW9uIG9mIGEKIGNv
bXBsZXRpb24gc3R5bGUgc2hvdWxkIHJldHVybiB0aGUgcmVzdWx0cyBpbiB0aGUgbmV3IGFs
aXN0IGZvcm1hdCBvZgogYGNvbXBsZXRpb24tZmlsdGVyLWNvbXBsZXRpb25zJy4gIFRoaXMg
dmFyaWFibGUgaXMgcHVyZWx5IG5lZWRlZCB0bwogZm9yIGJhY2t3YXJkIGNvbXBhdGliaWxp
dHkgb2YgdGhlIGV4aXN0aW5nIGJ1aWx0aW4gY29tcGxldGlvbiBzdHlsZQotZnVuY3Rpb25z
LiAgTmV3IGNvbXBsZXRpb24gc3R5bGUgZnVuY3Rpb25zIG1heSBhbHdheXMgcmV0dXJuIHRo
ZWlyCi1yZXN1bHRzIGluIHRoZSBuZXcgYWxpc3QgZm9ybWF0LCBzaW5jZSBgY29tcGxldGlv
bi1hbGwtY29tcGxldGlvbnMnCi10cmFuc3BhcmVudGx5IGNvbnZlcnRzIGJhY2sgdG8gdGhl
IG9sZCBpbXByb3BlciBsaXN0IG9mIGNvbXBsZXRpb25zCi13aXRoIGJhc2Ugc2l6ZSBpbiB0
aGUgbGFzdCBjZHIuIikKK2Z1bmN0aW9ucyBhcyBvZiBFbWFjcyAyOC4gIE5ldyBjb21wbGV0
aW9uIHN0eWxlIGZ1bmN0aW9ucyBzaG91bGQKK2Fsd2F5cyByZXR1cm4gdGhlaXIgcmVzdWx0
cyBpbiB0aGUgbmV3IGFsaXN0IGZvcm1hdCwgc2luY2UKK2Bjb21wbGV0aW9uLWFsbC1jb21w
bGV0aW9ucycgdHJhbnNwYXJlbnRseSBjb252ZXJ0cyBiYWNrIHRvIHRoZSBvbGQKK2ltcHJv
cGVyIGxpc3Qgb2YgY29tcGxldGlvbnMgd2l0aCBiYXNlIHNpemUgaW4gdGhlIGxhc3QgY2Ry
LiIpCiAKIChkZWZ1biBjb21wbGV0aW9uLS1udGgtY29tcGxldGlvbiAobiBzdHJpbmcgdGFi
bGUgcHJlZCBwb2ludCBtZXRhZGF0YSkKICAgIkNhbGwgdGhlIE50aCBtZXRob2Qgb2YgY29t
cGxldGlvbiBzdHlsZXMuIgpAQCAtMTA4NCw3ICsxMDg0LDcgQEAgY29tcGxldGlvbi0tbnRo
LWNvbXBsZXRpb24KICAgICAgICAgICAgICAgOzsgY29udHJhc3QgdG8gcGxhaW4gY29tcGxl
dGlvbiB0YWJsZXMsIHRoZSBzYXZpbmdzIG9mCiAgICAgICAgICAgICAgIDs7IGRlZmVycmVk
IGhpZ2hsaWdodGluZyB3b3VsZCBiZSBtaW5pbWFsIGluIHRoZSBjYXNlIG9mCiAgICAgICAg
ICAgICAgIDs7IHF1b3RlZCBjb21wbGV0aW9uIHRhYmxlcy4KLSAgICAgICAgICAgICAgKHNl
dHEgY29tcGxldGlvbi0tZmlsdGVyLWNvbXBsZXRpb25zIG5pbCkKKyAgICAgICAgICAgICAg
KHNldHEgY29tcGxldGlvbi0tcmV0dXJuLWFsaXN0LWZsYWcgbmlsKQogICAgICAgICAgICAg
ICAoc2V0cSBzdHJpbmcgKHBvcCBuZXcpKQogICAgICAgICAgICAgICAoc2V0cSB0YWJsZSAo
cG9wIG5ldykpCiAgICAgICAgICAgICAgIChzZXRxIHBvaW50IChwb3AgbmV3KSkKQEAgLTEw
OTMsMTggKzEwOTMsMzUgQEAgY29tcGxldGlvbi0tbnRoLWNvbXBsZXRpb24KICAgICAgICAg
IChyZXN1bHQtYW5kLXN0eWxlCiAgICAgICAgICAgKGNvbXBsZXRpb24tLXNvbWUKICAgICAg
ICAgICAgKGxhbWJkYSAoc3R5bGUpCi0gICAgICAgICAgICAgKGxldCAoKHByb2JlIChmdW5j
YWxsIChudGggbiAoYXNzcSBzdHlsZQotICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgY29tcGxldGlvbi1zdHlsZXMtYWxpc3QpKQotICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICBzdHJpbmcgdGFibGUgcHJlZCBwb2ludCkpKQor
ICAgICAgICAgICAgIChsZXQqICgoZnVuIChudGggbiAoYXNzcSBzdHlsZSBjb21wbGV0aW9u
LXN0eWxlcy1hbGlzdCkpKQorICAgICAgICAgICAgICAgICAgICA7OyBUcmFuc3BhcmVudGx5
IHVwZ3JhZGUgdGhlIHJldHVybiB2YWx1ZSBmb3IKKyAgICAgICAgICAgICAgICAgICAgOzsg
ZXhpc3RpbmcgYnVpbHQtaW4gc3R5bGVzIGFzIG9mIEVtYWNzIDI4LiAgTm8KKyAgICAgICAg
ICAgICAgICAgICAgOzsgbmV3IHN0eWxlcyBzaG91bGQgYmUgYWRkZWQgaGVyZS4gTmV3IGNv
bXBsZXRpb24KKyAgICAgICAgICAgICAgICAgICAgOzsgc3R5bGVzIHNob3VsZCBkaXJlY3Rs
eSByZXR1cm4gdGhlIG5ldworICAgICAgICAgICAgICAgICAgICA7OyBjb21wbGV0aW9uIGZv
cm1hdC5lbAorICAgICAgICAgICAgICAgICAgICAoY29tcGxldGlvbi0tcmV0dXJuLWFsaXN0
LWZsYWcKKyAgICAgICAgICAgICAgICAgICAgIChhbmQgY29tcGxldGlvbi0tcmV0dXJuLWFs
aXN0LWZsYWcKKyAgICAgICAgICAgICAgICAgICAgICAgICAgKG1lbXEgc3R5bGUgJyhlbWFj
czIxIGVtYWNzMjIgYmFzaWMgc3Vic3RyaW5nCisgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgcGFydGlhbC1jb21wbGV0aW9uIGluaXRpYWxzIGZsZXgpKSkpCisg
ICAgICAgICAgICAgICAgICAgIChwcm9iZSAoZnVuY2FsbCBmdW4gc3RyaW5nIHRhYmxlIHBy
ZWQgcG9pbnQpKSkKICAgICAgICAgICAgICAgIChhbmQgcHJvYmUgKGNvbnMgcHJvYmUgc3R5
bGUpKSkpCiAgICAgICAgICAgIChjb21wbGV0aW9uLS1zdHlsZXMgbWQpKSkKLSAgICAgICAg
IChzdHlsZS1tZCAoZ2V0IChjZHIgcmVzdWx0LWFuZC1zdHlsZSkgJ2NvbXBsZXRpb24tLXN0
eWxlLW1ldGFkYXRhKSkpCisgICAgICAgICAoc3R5bGUtbWQgKGdldCAoY2RyIHJlc3VsdC1h
bmQtc3R5bGUpICdjb21wbGV0aW9uLS1zdHlsZS1tZXRhZGF0YSkpCisgICAgICAgICAocmVz
dWx0IChjYXIgcmVzdWx0LWFuZC1zdHlsZSkpKQogICAgICh3aGVuIChhbmQgc3R5bGUtbWQg
bWV0YWRhdGEpCiAgICAgICAoc2V0Y2RyIG1ldGFkYXRhIChjZHIgKGZ1bmNhbGwgc3R5bGUt
bWQKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBzdHJpbmcgdGFibGUg
cHJlZCBwb2ludCBtZXRhZGF0YSkpKSkKKyAgICAod2hlbiAoYW5kIChub3QgY29tcGxldGlv
bi0tcmV0dXJuLWFsaXN0LWZsYWcpICg9IG4gMikgKGNvbnNwIChjYXIgcmVzdWx0KSkpCisg
ICAgICA7OyBHaXZlIHRoZSBjb21wbGV0aW9uIHN0eWxlcyBzb21lIGZyZWVkb20hICBJZiB0
aGV5IGFyZQorICAgICAgOzsgdGFyZ2V0aW5nIEVtYWNzIDI4IHVwd2FyZHMgb25seSwgdGhl
eSBtYXkgcmV0dXJuIGEgcmVzdWx0CisgICAgICA7OyB3aXRoIGRlZmVycmVkIGhpZ2hsaWdo
dGluZy4gIFdlIGNvbnZlcnQgYmFjayB0byB0aGUgb2xkCisgICAgICA7OyBmb3JtYXQgaGVy
ZSBieSBhcHBseWluZyB0aGUgaGlnaGxpZ2h0aW5nIGVhZ2VybHkuCisgICAgICAoc2V0cSBy
ZXN1bHQgKG5jb25jIChmdW5jYWxsIChjZHIgKGFzc3EgJ2hpZ2hsaWdodCByZXN1bHQpKQor
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAoY2RyIChhc3NxICdjb21wbGV0
aW9ucyByZXN1bHQpKSkKKyAgICAgICAgICAgICAgICAgICAgICAgICAgKGNkciAoYXNzcSAn
YmFzZSByZXN1bHQpKSkpKQogICAgIChpZiByZXF1b3RlCi0gICAgICAgIChmdW5jYWxsIHJl
cXVvdGUgKGNhciByZXN1bHQtYW5kLXN0eWxlKSBuKQotICAgICAgKGNhciByZXN1bHQtYW5k
LXN0eWxlKSkpKQorICAgICAgICAoZnVuY2FsbCByZXF1b3RlIHJlc3VsdCBuKQorICAgICAg
cmVzdWx0KSkpCiAKIChkZWZ1biBjb21wbGV0aW9uLXRyeS1jb21wbGV0aW9uIChzdHJpbmcg
dGFibGUgcHJlZCBwb2ludCAmb3B0aW9uYWwgbWV0YWRhdGEpCiAgICJUcnkgdG8gY29tcGxl
dGUgU1RSSU5HIHVzaW5nIGNvbXBsZXRpb24gdGFibGUgVEFCTEUuCkBAIC0xMTI0LDE5ICsx
MTQxLDggQEAgY29tcGxldGlvbi1hbGwtY29tcGxldGlvbnMKIFRoZSBNRVRBREFUQSBtYXkg
YmUgbW9kaWZpZWQgYnkgdGhlIGNvbXBsZXRpb24gc3R5bGUuICBUaGlzIGZ1bmN0aW9uCiBo
YXMgYmVlbiBzdXBlcnNlZGVkIGJ5IGBjb21wbGV0aW9uLWZpbHRlci1jb21wbGV0aW9ucycs
IHdoaWNoIHJldHVybnMKIHJpY2hlciBpbmZvcm1hdGlvbiBhbmQgc3VwcG9ydHMgZGVmZXJy
ZWQgY2FuZGlkYXRlIGhpZ2hsaWdodGluZy4iCi0gIChsZXQgKChjb21wbGV0aW9uLS1maWx0
ZXItY29tcGxldGlvbnMgbmlsKQotICAgICAgICAocmVzdWx0IChjb21wbGV0aW9uLS1udGgt
Y29tcGxldGlvbiAyIHN0cmluZyB0YWJsZQotICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICBwcmVkIHBvaW50IG1ldGFkYXRhKSkpCi0gICAgKGlmIChhbmQg
cmVzdWx0IChjb25zcCAoY2FyIHJlc3VsdCkpKQotICAgICAgICA7OyBHaXZlIHRoZSBjb21w
bGV0aW9uIHN0eWxlcyBzb21lIGZyZWVkb20hCi0gICAgICAgIDs7IElmIHRoZXkgYXJlIHRh
cmdldGluZyBFbWFjcyAyOCB1cHdhcmRzIG9ubHksIHRoZXkKLSAgICAgICAgOzsgbWF5IGFs
d2F5cyByZXR1cm4gYSByZXN1bHQgd2l0aCBkZWZlcnJlZAotICAgICAgICA7OyBoaWdobGln
aHRpbmcuICBXZSBjb252ZXJ0IGJhY2sgdG8gdGhlIG9sZCBmb3JtYXQKLSAgICAgICAgOzsg
aGVyZSBieSBhcHBseWluZyB0aGUgaGlnaGxpZ2h0aW5nIGVhZ2VybHkuCi0gICAgICAgIChu
Y29uYyAoZnVuY2FsbCAoY2RyIChhc3NxICdoaWdobGlnaHQgcmVzdWx0KSkKLSAgICAgICAg
ICAgICAgICAgICAgICAgIChjZHIgKGFzc3EgJ2NvbXBsZXRpb25zIHJlc3VsdCkpKQotICAg
ICAgICAgICAgICAgKGNkciAoYXNzcSAnYmFzZSByZXN1bHQpKSkKLSAgICAgIHJlc3VsdCkp
KQorICAobGV0ICgoY29tcGxldGlvbi0tcmV0dXJuLWFsaXN0LWZsYWcgbmlsKSkKKyAgICAo
Y29tcGxldGlvbi0tbnRoLWNvbXBsZXRpb24gMiBzdHJpbmcgdGFibGUgcHJlZCBwb2ludCBt
ZXRhZGF0YSkpKQogCiAoZGVmdW4gY29tcGxldGlvbi1maWx0ZXItY29tcGxldGlvbnMgKHN0
cmluZyB0YWJsZSBwcmVkIHBvaW50IG1ldGFkYXRhKQogICAiRmlsdGVyIHRoZSBwb3NzaWJs
ZSBjb21wbGV0aW9ucyBvZiBTVFJJTkcgaW4gY29tcGxldGlvbiB0YWJsZSBUQUJMRS4KQEAg
LTExNTIsNyArMTE1OCw3IEBAIGNvbXBsZXRpb24tZmlsdGVyLWNvbXBsZXRpb25zCiAtIGNv
bXBsZXRpb25zOiBUaGUgbGlzdCBvZiBjb21wbGV0aW9ucy4KIAogVGhpcyBmdW5jdGlvbiBz
dXBlcnNlZGVzIHRoZSBmdW5jdGlvbiBgY29tcGxldGlvbi1hbGwtY29tcGxldGlvbnMnLiIK
LSAgKGxldCogKChjb21wbGV0aW9uLS1maWx0ZXItY29tcGxldGlvbnMgdCkKKyAgKGxldCog
KChjb21wbGV0aW9uLS1yZXR1cm4tYWxpc3QtZmxhZyB0KQogICAgICAgICAgKHJlc3VsdCAo
Y29tcGxldGlvbi0tbnRoLWNvbXBsZXRpb24gMiBzdHJpbmcgdGFibGUKICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHByZWQgcG9pbnQgbWV0YWRhdGEp
KSkKICAgICAoaWYgKGFuZCByZXN1bHQgKG5vdCAoY29uc3AgKGNhciByZXN1bHQpKSkpCkBA
IC0yMTIwLDggKzIxMjYsOCBAQCBjb21wbGV0aW9uLS1oaWxpdC1jb21tb25hbGl0eQogCiAo
ZGVmdW4gY29tcGxldGlvbi0tZGVmZXJyZWQtaGlsaXQgKGNvbXBsZXRpb25zIHByZWZpeC1s
ZW4gYmFzZSBlbmQpCiAgICJSZXR1cm4gY29tcGxldGlvbnMgaW4gb2xkIGZvcm1hdCBvciBu
ZXcgYWxpc3QgZm9ybWF0LgotSWYgYGNvbXBsZXRpb24tLWZpbHRlci1jb21wbGV0aW9ucycg
aXMgbm9uLW5pbCB1c2UgdGhlIG5ldyBmb3JtYXQuIgotICAoaWYgY29tcGxldGlvbi0tZmls
dGVyLWNvbXBsZXRpb25zCitJZiBgY29tcGxldGlvbi0tcmV0dXJuLWFsaXN0LWZsYWcnIGlz
IG5vbi1uaWwgdXNlIHRoZSBuZXcgZm9ybWF0LiIKKyAgKGlmIGNvbXBsZXRpb24tLXJldHVy
bi1hbGlzdC1mbGFnCiAgICAgICAod2hlbiBjb21wbGV0aW9ucwogICAgICAgICBgKChiYXNl
IC4gLGJhc2UpCiAgICAgICAgICAgKGVuZCAuICxlbmQpCkBAIC0zNTg2LDkgKzM1OTIsOSBA
QCBmbGV4LXNjb3JlLW1hdGNoLXRpZ2h0bmVzcwogCiAoZGVmdW4gY29tcGxldGlvbi1wY20t
LWRlZmVycmVkLWhpbGl0IChwYXR0ZXJuIGNvbXBsZXRpb25zIGJhc2UgZW5kKQogICAiUmV0
dXJuIGNvbXBsZXRpb25zIGluIG9sZCBmb3JtYXQgb3IgbmV3IGFsaXN0IGZvcm1hdC4KLUlm
IGBjb21wbGV0aW9uLS1maWx0ZXItY29tcGxldGlvbnMnIGlzIG5vbi1uaWwgdXNlIHRoZSBu
ZXcgZm9ybWF0LiIKK0lmIGBjb21wbGV0aW9uLS1yZXR1cm4tYWxpc3QtZmxhZycgaXMgbm9u
LW5pbCB1c2UgdGhlIG5ldyBmb3JtYXQuIgogICAod2hlbiBjb21wbGV0aW9ucwotICAgIChp
ZiBjb21wbGV0aW9uLS1maWx0ZXItY29tcGxldGlvbnMKKyAgICAoaWYgY29tcGxldGlvbi0t
cmV0dXJuLWFsaXN0LWZsYWcKICAgICAgICAgYCgoYmFzZSAuICxiYXNlKQogICAgICAgICAg
IChlbmQgLiAsZW5kKQogICAgICAgICAgIChoaWdobGlnaHQgLiAsKGFwcGx5LXBhcnRpYWxs
eQotLSAKMi4yMC4xCgo=
--------------4B3E2A198290BEF506AC64C4
Content-Type: text/plain; charset=UTF-8;
 name="variant2-argument.el"
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
 filename="variant2-argument.el"

RnJvbSBmMGRhMWYxZDkyZjIwZGY3ZDkyZDk3NjIyOTVhYzUxMzkwZmNlZTgzIE1vbiBTZXAg
MTcgMDA6MDA6MDAgMjAwMQpGcm9tOiBEYW5pZWwgTWVuZGxlciA8bWFpbEBkYW5pZWwtbWVu
ZGxlci5kZT4KRGF0ZTogVGh1LCAxMiBBdWcgMjAyMSAxMDoxMDowMiArMDIwMApTdWJqZWN0
OiBbUEFUQ0hdIFVzZSBGSUxURVIgYXJndW1lbnQgaW5zdGVhZCBvZiBnbG9iYWwKIGBjb21w
bGV0aW9uLS1maWx0ZXItY29tcGxldGlvbnNgCgotLS0KIGxpc3AvbWluaWJ1ZmZlci5lbCB8
IDExOSArKysrKysrKysrKysrKysrKysrKysrKystLS0tLS0tLS0tLS0tLS0tLS0tLS0KIDEg
ZmlsZSBjaGFuZ2VkLCA2NCBpbnNlcnRpb25zKCspLCA1NSBkZWxldGlvbnMoLSkKCmRpZmYg
LS1naXQgYS9saXNwL21pbmlidWZmZXIuZWwgYi9saXNwL21pbmlidWZmZXIuZWwKaW5kZXgg
YmE4ODU1YzRlYS4uMGY4Yjg1ZGRkNCAxMDA2NDQKLS0tIGEvbGlzcC9taW5pYnVmZmVyLmVs
CisrKyBiL2xpc3AvbWluaWJ1ZmZlci5lbApAQCAtMTAzOSwxOCArMTAzOSw3IEBAIGNvbXBs
ZXRpb24tLXN0eWxlcwogICAgICAgICAoZGVsZXRlLWR1cHMgKGFwcGVuZCAoY2RyIG92ZXIp
IChjb3B5LXNlcXVlbmNlIGNvbXBsZXRpb24tc3R5bGVzKSkpCiAgICAgICAgY29tcGxldGlv
bi1zdHlsZXMpKSkKIAotKGRlZnZhciBjb21wbGV0aW9uLS1maWx0ZXItY29tcGxldGlvbnMg
bmlsCi0gICJFbmFibGUgdGhlIG5ldyBjb21wbGV0aW9ucyByZXR1cm4gdmFsdWUgZm9ybWF0
LgotSWYgdGhpcyB2YXJpYWJsZSBpcyBub24tbmlsIHRoZSBgYWxsLWNvbXBsZXRpb25zJyBm
dW5jdGlvbiBvZiBhCi1jb21wbGV0aW9uIHN0eWxlIHNob3VsZCByZXR1cm4gdGhlIHJlc3Vs
dHMgaW4gdGhlIG5ldyBhbGlzdCBmb3JtYXQgb2YKLWBjb21wbGV0aW9uLWZpbHRlci1jb21w
bGV0aW9ucycuICBUaGlzIHZhcmlhYmxlIGlzIHB1cmVseSBuZWVkZWQgdG8KLWZvciBiYWNr
d2FyZCBjb21wYXRpYmlsaXR5IG9mIHRoZSBleGlzdGluZyBidWlsdGluIGNvbXBsZXRpb24g
c3R5bGUKLWZ1bmN0aW9ucy4gIE5ldyBjb21wbGV0aW9uIHN0eWxlIGZ1bmN0aW9ucyBtYXkg
YWx3YXlzIHJldHVybiB0aGVpcgotcmVzdWx0cyBpbiB0aGUgbmV3IGFsaXN0IGZvcm1hdCwg
c2luY2UgYGNvbXBsZXRpb24tYWxsLWNvbXBsZXRpb25zJwotdHJhbnNwYXJlbnRseSBjb252
ZXJ0cyBiYWNrIHRvIHRoZSBvbGQgaW1wcm9wZXIgbGlzdCBvZiBjb21wbGV0aW9ucwotd2l0
aCBiYXNlIHNpemUgaW4gdGhlIGxhc3QgY2RyLiIpCi0KLShkZWZ1biBjb21wbGV0aW9uLS1u
dGgtY29tcGxldGlvbiAobiBzdHJpbmcgdGFibGUgcHJlZCBwb2ludCBtZXRhZGF0YSkKKyhk
ZWZ1biBjb21wbGV0aW9uLS1udGgtY29tcGxldGlvbiAobiBzdHJpbmcgdGFibGUgcHJlZCBw
b2ludCBtZXRhZGF0YSAmb3B0aW9uYWwgZmlsdGVyKQogICAiQ2FsbCB0aGUgTnRoIG1ldGhv
ZCBvZiBjb21wbGV0aW9uIHN0eWxlcy4iCiAgIDs7IFdlIHByb3ZpZGUgc3BlY2lhbCBzdXBw
b3J0IGZvciBxdW90aW5nL3VucXVvdGluZyBoZXJlIGJlY2F1c2UgaXQgY2Fubm90CiAgIDs7
IHJlbGlhYmx5IGJlIGRvbmUgd2l0aGluIHRoZSBub3JtYWwgY29tcGxldGlvbi10YWJsZSBy
b3V0aW5lczogQ29tcGxldGlvbgpAQCAtMTA4NCw3ICsxMDczLDcgQEAgY29tcGxldGlvbi0t
bnRoLWNvbXBsZXRpb24KICAgICAgICAgICAgICAgOzsgY29udHJhc3QgdG8gcGxhaW4gY29t
cGxldGlvbiB0YWJsZXMsIHRoZSBzYXZpbmdzIG9mCiAgICAgICAgICAgICAgIDs7IGRlZmVy
cmVkIGhpZ2hsaWdodGluZyB3b3VsZCBiZSBtaW5pbWFsIGluIHRoZSBjYXNlIG9mCiAgICAg
ICAgICAgICAgIDs7IHF1b3RlZCBjb21wbGV0aW9uIHRhYmxlcy4KLSAgICAgICAgICAgICAg
KHNldHEgY29tcGxldGlvbi0tZmlsdGVyLWNvbXBsZXRpb25zIG5pbCkKKyAgICAgICAgICAg
ICAgKHNldHEgZmlsdGVyIG5pbCkKICAgICAgICAgICAgICAgKHNldHEgc3RyaW5nIChwb3Ag
bmV3KSkKICAgICAgICAgICAgICAgKHNldHEgdGFibGUgKHBvcCBuZXcpKQogICAgICAgICAg
ICAgICAoc2V0cSBwb2ludCAocG9wIG5ldykpCkBAIC0xMDkzLDE4ICsxMDgyLDM3IEBAIGNv
bXBsZXRpb24tLW50aC1jb21wbGV0aW9uCiAgICAgICAgICAocmVzdWx0LWFuZC1zdHlsZQog
ICAgICAgICAgIChjb21wbGV0aW9uLS1zb21lCiAgICAgICAgICAgIChsYW1iZGEgKHN0eWxl
KQotICAgICAgICAgICAgIChsZXQgKChwcm9iZSAoZnVuY2FsbCAobnRoIG4gKGFzc3Egc3R5
bGUKLSAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGNv
bXBsZXRpb24tc3R5bGVzLWFsaXN0KSkKLSAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgc3RyaW5nIHRhYmxlIHByZWQgcG9pbnQpKSkKKyAgICAgICAgICAgICAobGV0KiAo
KGZ1biAobnRoIG4gKGFzc3Egc3R5bGUgY29tcGxldGlvbi1zdHlsZXMtYWxpc3QpKSkKKyAg
ICAgICAgICAgICAgICAgICAgKHByb2JlCisgICAgICAgICAgICAgICAgICAgICAoaWYgZmls
dGVyCisgICAgICAgICAgICAgICAgICAgICAgICAgOzsgSW4gb3JkZXIgdG8gcmV0YWluIGJh
Y2t3YXJkIGNvbXBhdGliaWxpdHkKKyAgICAgICAgICAgICAgICAgICAgICAgICA7OyBvZiB0
aGUgY2FsbGluZyBjb252ZW50aW9uIG9mIHRoZSBleGlzaXRpbmcKKyAgICAgICAgICAgICAg
ICAgICAgICAgICA7OyBjb21wbGV0aW9uIHN0eWxlcywgYWRkIGEgZm91cnRoIEZJTFRFUgor
ICAgICAgICAgICAgICAgICAgICAgICAgIDs7IGFyZ3VtZW50IHRvIG9wdC1pbiB0byB0aGUg
bmV3IHJldHVybiB2YWx1ZQorICAgICAgICAgICAgICAgICAgICAgICAgIDs7IGZvcm1hdC4K
KyAgICAgICAgICAgICAgICAgICAgICAgICAoY29uZGl0aW9uLWNhc2UgbmlsCisgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgIChmdW5jYWxsIGZ1biBzdHJpbmcgdGFibGUgcHJlZCBw
b2ludCBmaWx0ZXIpCisgICAgICAgICAgICAgICAgICAgICAgICAgICAod3JvbmctbnVtYmVy
LW9mLWFyZ3VtZW50cworICAgICAgICAgICAgICAgICAgICAgICAgICAgIChmdW5jYWxsIGZ1
biBzdHJpbmcgdGFibGUgcHJlZCBwb2ludCkpKQorICAgICAgICAgICAgICAgICAgICAgICAo
ZnVuY2FsbCBmdW4gc3RyaW5nIHRhYmxlIHByZWQgcG9pbnQpKSkpCiAgICAgICAgICAgICAg
ICAoYW5kIHByb2JlIChjb25zIHByb2JlIHN0eWxlKSkpKQogICAgICAgICAgICAoY29tcGxl
dGlvbi0tc3R5bGVzIG1kKSkpCi0gICAgICAgICAoc3R5bGUtbWQgKGdldCAoY2RyIHJlc3Vs
dC1hbmQtc3R5bGUpICdjb21wbGV0aW9uLS1zdHlsZS1tZXRhZGF0YSkpKQorICAgICAgICAg
KHN0eWxlLW1kIChnZXQgKGNkciByZXN1bHQtYW5kLXN0eWxlKSAnY29tcGxldGlvbi0tc3R5
bGUtbWV0YWRhdGEpKQorICAgICAgICAgKHJlc3VsdCAoY2FyIHJlc3VsdC1hbmQtc3R5bGUp
KSkKICAgICAod2hlbiAoYW5kIHN0eWxlLW1kIG1ldGFkYXRhKQogICAgICAgKHNldGNkciBt
ZXRhZGF0YSAoY2RyIChmdW5jYWxsIHN0eWxlLW1kCiAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgc3RyaW5nIHRhYmxlIHByZWQgcG9pbnQgbWV0YWRhdGEpKSkpCisg
ICAgKHdoZW4gKGFuZCAobm90IGZpbHRlcikgKD0gbiAyKSAoY29uc3AgKGNhciByZXN1bHQp
KSkKKyAgICAgIDs7IEdpdmUgdGhlIGNvbXBsZXRpb24gc3R5bGVzIHNvbWUgZnJlZWRvbSEg
IElmIHRoZXkgYXJlCisgICAgICA7OyB0YXJnZXRpbmcgRW1hY3MgMjggdXB3YXJkcyBvbmx5
LCB0aGV5IG1heSByZXR1cm4gYSByZXN1bHQKKyAgICAgIDs7IHdpdGggZGVmZXJyZWQgaGln
aGxpZ2h0aW5nLiAgV2UgY29udmVydCBiYWNrIHRvIHRoZSBvbGQKKyAgICAgIDs7IGZvcm1h
dCBoZXJlIGJ5IGFwcGx5aW5nIHRoZSBoaWdobGlnaHRpbmcgZWFnZXJseS4KKyAgICAgIChz
ZXRxIHJlc3VsdCAobmNvbmMgKGZ1bmNhbGwgKGNkciAoYXNzcSAnaGlnaGxpZ2h0IHJlc3Vs
dCkpCisgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIChjZHIgKGFzc3EgJ2Nv
bXBsZXRpb25zIHJlc3VsdCkpKQorICAgICAgICAgICAgICAgICAgICAgICAgICAoY2RyIChh
c3NxICdiYXNlIHJlc3VsdCkpKSkpCiAgICAgKGlmIHJlcXVvdGUKLSAgICAgICAgKGZ1bmNh
bGwgcmVxdW90ZSAoY2FyIHJlc3VsdC1hbmQtc3R5bGUpIG4pCi0gICAgICAoY2FyIHJlc3Vs
dC1hbmQtc3R5bGUpKSkpCisgICAgICAgIChmdW5jYWxsIHJlcXVvdGUgcmVzdWx0IG4pCisg
ICAgICByZXN1bHQpKSkKIAogKGRlZnVuIGNvbXBsZXRpb24tdHJ5LWNvbXBsZXRpb24gKHN0
cmluZyB0YWJsZSBwcmVkIHBvaW50ICZvcHRpb25hbCBtZXRhZGF0YSkKICAgIlRyeSB0byBj
b21wbGV0ZSBTVFJJTkcgdXNpbmcgY29tcGxldGlvbiB0YWJsZSBUQUJMRS4KQEAgLTExMjQs
MTkgKzExMzIsNyBAQCBjb21wbGV0aW9uLWFsbC1jb21wbGV0aW9ucwogVGhlIE1FVEFEQVRB
IG1heSBiZSBtb2RpZmllZCBieSB0aGUgY29tcGxldGlvbiBzdHlsZS4gIFRoaXMgZnVuY3Rp
b24KIGhhcyBiZWVuIHN1cGVyc2VkZWQgYnkgYGNvbXBsZXRpb24tZmlsdGVyLWNvbXBsZXRp
b25zJywgd2hpY2ggcmV0dXJucwogcmljaGVyIGluZm9ybWF0aW9uIGFuZCBzdXBwb3J0cyBk
ZWZlcnJlZCBjYW5kaWRhdGUgaGlnaGxpZ2h0aW5nLiIKLSAgKGxldCAoKGNvbXBsZXRpb24t
LWZpbHRlci1jb21wbGV0aW9ucyBuaWwpCi0gICAgICAgIChyZXN1bHQgKGNvbXBsZXRpb24t
LW50aC1jb21wbGV0aW9uIDIgc3RyaW5nIHRhYmxlCi0gICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgIHByZWQgcG9pbnQgbWV0YWRhdGEpKSkKLSAgICAoaWYg
KGFuZCByZXN1bHQgKGNvbnNwIChjYXIgcmVzdWx0KSkpCi0gICAgICAgIDs7IEdpdmUgdGhl
IGNvbXBsZXRpb24gc3R5bGVzIHNvbWUgZnJlZWRvbSEKLSAgICAgICAgOzsgSWYgdGhleSBh
cmUgdGFyZ2V0aW5nIEVtYWNzIDI4IHVwd2FyZHMgb25seSwgdGhleQotICAgICAgICA7OyBt
YXkgYWx3YXlzIHJldHVybiBhIHJlc3VsdCB3aXRoIGRlZmVycmVkCi0gICAgICAgIDs7IGhp
Z2hsaWdodGluZy4gIFdlIGNvbnZlcnQgYmFjayB0byB0aGUgb2xkIGZvcm1hdAotICAgICAg
ICA7OyBoZXJlIGJ5IGFwcGx5aW5nIHRoZSBoaWdobGlnaHRpbmcgZWFnZXJseS4KLSAgICAg
ICAgKG5jb25jIChmdW5jYWxsIChjZHIgKGFzc3EgJ2hpZ2hsaWdodCByZXN1bHQpKQotICAg
ICAgICAgICAgICAgICAgICAgICAgKGNkciAoYXNzcSAnY29tcGxldGlvbnMgcmVzdWx0KSkp
Ci0gICAgICAgICAgICAgICAoY2RyIChhc3NxICdiYXNlIHJlc3VsdCkpKQotICAgICAgcmVz
dWx0KSkpCisgIChjb21wbGV0aW9uLS1udGgtY29tcGxldGlvbiAyIHN0cmluZyB0YWJsZSBw
cmVkIHBvaW50IG1ldGFkYXRhKSkKIAogKGRlZnVuIGNvbXBsZXRpb24tZmlsdGVyLWNvbXBs
ZXRpb25zIChzdHJpbmcgdGFibGUgcHJlZCBwb2ludCBtZXRhZGF0YSkKICAgIkZpbHRlciB0
aGUgcG9zc2libGUgY29tcGxldGlvbnMgb2YgU1RSSU5HIGluIGNvbXBsZXRpb24gdGFibGUg
VEFCTEUuCkBAIC0xMTUyLDkgKzExNDgsOCBAQCBjb21wbGV0aW9uLWZpbHRlci1jb21wbGV0
aW9ucwogLSBjb21wbGV0aW9uczogVGhlIGxpc3Qgb2YgY29tcGxldGlvbnMuCiAKIFRoaXMg
ZnVuY3Rpb24gc3VwZXJzZWRlcyB0aGUgZnVuY3Rpb24gYGNvbXBsZXRpb24tYWxsLWNvbXBs
ZXRpb25zJy4iCi0gIChsZXQqICgoY29tcGxldGlvbi0tZmlsdGVyLWNvbXBsZXRpb25zIHQp
Ci0gICAgICAgICAocmVzdWx0IChjb21wbGV0aW9uLS1udGgtY29tcGxldGlvbiAyIHN0cmlu
ZyB0YWJsZQotICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
cHJlZCBwb2ludCBtZXRhZGF0YSkpKQorICAobGV0KiAoKHJlc3VsdCAoY29tcGxldGlvbi0t
bnRoLWNvbXBsZXRpb24gMiBzdHJpbmcgdGFibGUKKyAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgIHByZWQgcG9pbnQgbWV0YWRhdGEgJ2ZpbHRlcikpKQog
ICAgIChpZiAoYW5kIHJlc3VsdCAobm90IChjb25zcCAoY2FyIHJlc3VsdCkpKSkKICAgICAg
ICAgOzsgRGVmZXJyZWQgaGlnaGxpZ2h0aW5nIGhhcyBiZWVuIHJlcXVlc3RlZCwgYnV0IHRo
ZSBjb21wbGV0aW9uCiAgICAgICAgIDs7IHN0eWxlIHJldHVybmVkIGEgbm9uLWRlZmVycmVk
IHJlc3VsdC4gQ29udmVydCB0aGUgcmVzdWx0IHRvIHRoZQpAQCAtMjExOCwxMCArMjExMywx
MCBAQCBjb21wbGV0aW9uLS1oaWxpdC1jb21tb25hbGl0eQogICAgICBlbGVtKQogICAgY29t
cGxldGlvbnMpKQogCi0oZGVmdW4gY29tcGxldGlvbi0tZGVmZXJyZWQtaGlsaXQgKGNvbXBs
ZXRpb25zIHByZWZpeC1sZW4gYmFzZSBlbmQpCisoZGVmdW4gY29tcGxldGlvbi0tZGVmZXJy
ZWQtaGlsaXQgKGNvbXBsZXRpb25zIHByZWZpeC1sZW4gYmFzZSBlbmQgZmlsdGVyKQogICAi
UmV0dXJuIGNvbXBsZXRpb25zIGluIG9sZCBmb3JtYXQgb3IgbmV3IGFsaXN0IGZvcm1hdC4K
LUlmIGBjb21wbGV0aW9uLS1maWx0ZXItY29tcGxldGlvbnMnIGlzIG5vbi1uaWwgdXNlIHRo
ZSBuZXcgZm9ybWF0LiIKLSAgKGlmIGNvbXBsZXRpb24tLWZpbHRlci1jb21wbGV0aW9ucwor
SWYgRklMVEVSIGlzIG5vbi1uaWwgdXNlIHRoZSBuZXcgZm9ybWF0LiIKKyAgKGlmIGZpbHRl
cgogICAgICAgKHdoZW4gY29tcGxldGlvbnMKICAgICAgICAgYCgoYmFzZSAuICxiYXNlKQog
ICAgICAgICAgIChlbmQgLiAsZW5kKQpAQCAtMzMwMCwxMiArMzI5NSwxNCBAQCBjb21wbGV0
aW9uLWVtYWNzMjEtdHJ5LWNvbXBsZXRpb24KICAgICAgICAgKGNvbnMgY29tcGxldGlvbiAo
bGVuZ3RoIGNvbXBsZXRpb24pKQogICAgICAgY29tcGxldGlvbikpKQogCi0oZGVmdW4gY29t
cGxldGlvbi1lbWFjczIxLWFsbC1jb21wbGV0aW9ucyAoc3RyaW5nIHRhYmxlIHByZWQgX3Bv
aW50KQorKGRlZnVuIGNvbXBsZXRpb24tZW1hY3MyMS1hbGwtY29tcGxldGlvbnMgKHN0cmlu
ZyB0YWJsZSBwcmVkIF9wb2ludAorICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAmb3B0aW9uYWwgZmlsdGVyKQogICAoY29tcGxldGlvbi0tZGVm
ZXJyZWQtaGlsaXQKICAgIChhbGwtY29tcGxldGlvbnMgc3RyaW5nIHRhYmxlIHByZWQpCiAg
ICAobGVuZ3RoIHN0cmluZykKICAgIChjYXIgKGNvbXBsZXRpb24tYm91bmRhcmllcyBzdHJp
bmcgdGFibGUgcHJlZCAiIikpCi0gICAobGVuZ3RoIHN0cmluZykpKQorICAgKGxlbmd0aCBz
dHJpbmcpCisgICBmaWx0ZXIpKQogCiAoZGVmdW4gY29tcGxldGlvbi1lbWFjczIyLXRyeS1j
b21wbGV0aW9uIChzdHJpbmcgdGFibGUgcHJlZCBwb2ludCkKICAgKGxldCAoKHN1ZmZpeCAo
c3Vic3RyaW5nIHN0cmluZyBwb2ludCkpCkBAIC0zMzI3LDEzICszMzI0LDE0IEBAIGNvbXBs
ZXRpb24tZW1hY3MyMi10cnktY29tcGxldGlvbgogICAgICAgICAgIChzZXRxIHN1ZmZpeCAo
c3Vic3RyaW5nIHN1ZmZpeCAxKSkpCiAgICAgICAoY29ucyAoY29uY2F0IGNvbXBsZXRpb24g
c3VmZml4KSAobGVuZ3RoIGNvbXBsZXRpb24pKSkpKQogCi0oZGVmdW4gY29tcGxldGlvbi1l
bWFjczIyLWFsbC1jb21wbGV0aW9ucyAoc3RyaW5nIHRhYmxlIHByZWQgcG9pbnQpCisoZGVm
dW4gY29tcGxldGlvbi1lbWFjczIyLWFsbC1jb21wbGV0aW9ucyAoc3RyaW5nIHRhYmxlIHBy
ZWQgcG9pbnQKKyAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgJm9wdGlvbmFsIGZpbHRlcikKICAgKGxldCogKChiZWZvcmVwb2ludCAoc3Vic3Ry
aW5nIHN0cmluZyAwIHBvaW50KSkKICAgICAgICAgIChhZnRlcnBvaW50IChzdWJzdHJpbmcg
c3RyaW5nIHBvaW50KSkKICAgICAgICAgIChib3VuZHMgKGNvbXBsZXRpb24tYm91bmRhcmll
cyBiZWZvcmVwb2ludCB0YWJsZSBwcmVkIGFmdGVycG9pbnQpKSkKICAgICAoY29tcGxldGlv
bi0tZGVmZXJyZWQtaGlsaXQKICAgICAgKGFsbC1jb21wbGV0aW9ucyBiZWZvcmVwb2ludCB0
YWJsZSBwcmVkKQotICAgICBwb2ludCAoY2FyIGJvdW5kcykgKCsgcG9pbnQgKGNkciBib3Vu
ZHMpKSkpKQorICAgICBwb2ludCAoY2FyIGJvdW5kcykgKCsgcG9pbnQgKGNkciBib3VuZHMp
KSBmaWx0ZXIpKSkKIAogOzs7IEJhc2ljIGNvbXBsZXRpb24uCiAKQEAgLTMzODEsNyArMzM3
OSw4IEBAIGNvbXBsZXRpb24tYmFzaWMtdHJ5LWNvbXBsZXRpb24KICAgICAgICAgICAgIChz
ZXRxIGFsbCAoY29tcGxldGlvbi1wY20tLWZpbGVuYW1lLXRyeS1maWx0ZXIgYWxsKSkpCiAg
ICAgICAgIChjb21wbGV0aW9uLXBjbS0tbWVyZ2UtdHJ5IHBhdHRlcm4gYWxsIHByZWZpeCBz
dWZmaXgpKSkpKQogCi0oZGVmdW4gY29tcGxldGlvbi1iYXNpYy1hbGwtY29tcGxldGlvbnMg
KHN0cmluZyB0YWJsZSBwcmVkIHBvaW50KQorKGRlZnVuIGNvbXBsZXRpb24tYmFzaWMtYWxs
LWNvbXBsZXRpb25zIChzdHJpbmcgdGFibGUgcHJlZCBwb2ludAorICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgJm9wdGlvbmFsIGZpbHRlcikKICAg
KGxldCogKChiZWZvcmVwb2ludCAoc3Vic3RyaW5nIHN0cmluZyAwIHBvaW50KSkKICAgICAg
ICAgIChhZnRlcnBvaW50IChzdWJzdHJpbmcgc3RyaW5nIHBvaW50KSkKICAgICAgICAgIChi
b3VuZHMgKGNvbXBsZXRpb24tYm91bmRhcmllcyBiZWZvcmVwb2ludCB0YWJsZSBwcmVkIGFm
dGVycG9pbnQpKQpAQCAtMzM5Miw3ICszMzkxLDkgQEAgY29tcGxldGlvbi1iYXNpYy1hbGwt
Y29tcGxldGlvbnMKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAncG9pbnQKICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAoc3Vic3RyaW5nIGFmdGVycG9pbnQgMCAoY2RyIGJv
dW5kcykpKSkpCiAgICAgICAgICAoYWxsIChjb21wbGV0aW9uLXBjbS0tYWxsLWNvbXBsZXRp
b25zIHByZWZpeCBwYXR0ZXJuIHRhYmxlIHByZWQpKSkKLSAgICAoY29tcGxldGlvbi0tZGVm
ZXJyZWQtaGlsaXQgYWxsIHBvaW50IChjYXIgYm91bmRzKSAoKyBwb2ludCAoY2RyIGJvdW5k
cykpKSkpCisgICAgKGNvbXBsZXRpb24tLWRlZmVycmVkLWhpbGl0IGFsbCBwb2ludAorICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAoY2FyIGJvdW5kcykgKCsgcG9pbnQgKGNk
ciBib3VuZHMpKQorICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBmaWx0ZXIpKSkK
IAogOzs7IFBhcnRpYWwtY29tcGxldGlvbi1tb2RlIHN0eWxlIGNvbXBsZXRpb24uCiAKQEAg
LTM1ODQsMTEgKzM1ODUsMTEgQEAgZmxleC1zY29yZS1tYXRjaC10aWdodG5lc3MKIHRoYW4g
dGhlIGxhdHRlciAod2hpY2ggaGFzIHR3byBcImhvbGVzXCIgYW5kIHRocmVlCiBvbmUtbGV0
dGVyLWxvbmcgbWF0Y2hlcykuIikKIAotKGRlZnVuIGNvbXBsZXRpb24tcGNtLS1kZWZlcnJl
ZC1oaWxpdCAocGF0dGVybiBjb21wbGV0aW9ucyBiYXNlIGVuZCkKKyhkZWZ1biBjb21wbGV0
aW9uLXBjbS0tZGVmZXJyZWQtaGlsaXQgKHBhdHRlcm4gY29tcGxldGlvbnMgYmFzZSBlbmQg
ZmlsdGVyKQogICAiUmV0dXJuIGNvbXBsZXRpb25zIGluIG9sZCBmb3JtYXQgb3IgbmV3IGFs
aXN0IGZvcm1hdC4KLUlmIGBjb21wbGV0aW9uLS1maWx0ZXItY29tcGxldGlvbnMnIGlzIG5v
bi1uaWwgdXNlIHRoZSBuZXcgZm9ybWF0LiIKK0lmIEZJTFRFUiBpcyBub24tbmlsIHVzZSB0
aGUgbmV3IGZvcm1hdC4iCiAgICh3aGVuIGNvbXBsZXRpb25zCi0gICAgKGlmIGNvbXBsZXRp
b24tLWZpbHRlci1jb21wbGV0aW9ucworICAgIChpZiBmaWx0ZXIKICAgICAgICAgYCgoYmFz
ZSAuICxiYXNlKQogICAgICAgICAgIChlbmQgLiAsZW5kKQogICAgICAgICAgIChoaWdobGln
aHQgLiAsKGFwcGx5LXBhcnRpYWxseQpAQCAtMzgzOSwxMiArMzg0MCwxNCBAQCBjb21wbGV0
aW9uLXBjbS0tZmluZC1hbGwtY29tcGxldGlvbnMKICAgICAgICAgICAoc2lnbmFsIChjYXIg
Zmlyc3RlcnJvcikgKGNkciBmaXJzdGVycm9yKSkKICAgICAgICAgKGxpc3QgcGF0dGVybiBh
bGwgcHJlZml4IHN1ZmZpeCkpKSkpCiAKLShkZWZ1biBjb21wbGV0aW9uLXBjbS1hbGwtY29t
cGxldGlvbnMgKHN0cmluZyB0YWJsZSBwcmVkIHBvaW50KQorKGRlZnVuIGNvbXBsZXRpb24t
cGNtLWFsbC1jb21wbGV0aW9ucyAoc3RyaW5nIHRhYmxlIHByZWQgcG9pbnQKKyAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAmb3B0aW9uYWwgZmlsdGVy
KQogICAocGNhc2UtbGV0ICgoYCgscGF0dGVybiAsYWxsICxwcmVmaXggLHN1ZmZpeCkKICAg
ICAgICAgICAgICAgIChjb21wbGV0aW9uLXBjbS0tZmluZC1hbGwtY29tcGxldGlvbnMgc3Ry
aW5nIHRhYmxlIHByZWQgcG9pbnQpKSkKICAgICAoY29tcGxldGlvbi1wY20tLWRlZmVycmVk
LWhpbGl0IHBhdHRlcm4gYWxsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAobGVuZ3RoIHByZWZpeCkKLSAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICgtIChsZW5ndGggc3RyaW5nKSAobGVuZ3RoIHN1ZmZpeCkpKSkpCisgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAoLSAobGVuZ3RoIHN0cmluZykgKGxlbmd0aCBzdWZm
aXgpKQorICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZmlsdGVyKSkpCiAK
IChkZWZ1biBjb21wbGV0aW9uLS1jb21tb24tc3VmZml4IChzdHJzKQogICAiUmV0dXJuIHRo
ZSBjb21tb24gc3VmZml4IG9mIHRoZSBzdHJpbmdzIFNUUlMuIgpAQCAtNDA2NywxMyArNDA3
MCwxNSBAQCBjb21wbGV0aW9uLXN1YnN0cmluZy10cnktY29tcGxldGlvbgogICAgICAgICAo
c2V0cSBhbGwgKGNvbXBsZXRpb24tcGNtLS1maWxlbmFtZS10cnktZmlsdGVyIGFsbCkpKQog
ICAgIChjb21wbGV0aW9uLXBjbS0tbWVyZ2UtdHJ5IHBhdHRlcm4gYWxsIHByZWZpeCBzdWZm
aXgpKSkKIAotKGRlZnVuIGNvbXBsZXRpb24tc3Vic3RyaW5nLWFsbC1jb21wbGV0aW9ucyAo
c3RyaW5nIHRhYmxlIHByZWQgcG9pbnQpCisoZGVmdW4gY29tcGxldGlvbi1zdWJzdHJpbmct
YWxsLWNvbXBsZXRpb25zIChzdHJpbmcgdGFibGUgcHJlZCBwb2ludAorICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICZvcHRpb25hbCBmaWx0
ZXIpCiAgIChwY2FzZS1sZXQgKChgKCxhbGwgLHBhdHRlcm4gLHByZWZpeCAsc3VmZml4KQog
ICAgICAgICAgICAgICAgKGNvbXBsZXRpb24tc3Vic3RyaW5nLS1hbGwtY29tcGxldGlvbnMK
ICAgICAgICAgICAgICAgICBzdHJpbmcgdGFibGUgcHJlZCBwb2ludCkpKQogICAgIChjb21w
bGV0aW9uLXBjbS0tZGVmZXJyZWQtaGlsaXQgcGF0dGVybiBhbGwKICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgIChsZW5ndGggcHJlZml4KQotICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgKC0gKGxlbmd0aCBzdHJpbmcpIChsZW5ndGggc3VmZml4
KSkpKSkKKyAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICgtIChsZW5ndGgg
c3RyaW5nKSAobGVuZ3RoIHN1ZmZpeCkpCisgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICBmaWx0ZXIpKSkKIAogOzs7ICJmbGV4IiBjb21wbGV0aW9uLCBhbHNvIGtub3du
IGFzIGZseC9mdXp6eS9zY2F0dGVyIGNvbXBsZXRpb24KIDs7IENvbXBsZXRlcyAiZm9vIiB0
byAiZnJvZG8iIGFuZCAiZmFyZnJvbXNvYmVyIgpAQCAtNDE1Miw3ICs0MTU3LDggQEAgY29t
cGxldGlvbi1mbGV4LXRyeS1jb21wbGV0aW9uCiAgICAgICA7OyAiZmFyZnJvbXNvYmVyIi4K
ICAgICAgIChjb21wbGV0aW9uLXBjbS0tbWVyZ2UtdHJ5IHBhdHRlcm4gYWxsIHByZWZpeCBz
dWZmaXgpKSkpCiAKLShkZWZ1biBjb21wbGV0aW9uLWZsZXgtYWxsLWNvbXBsZXRpb25zIChz
dHJpbmcgdGFibGUgcHJlZCBwb2ludCkKKyhkZWZ1biBjb21wbGV0aW9uLWZsZXgtYWxsLWNv
bXBsZXRpb25zIChzdHJpbmcgdGFibGUgcHJlZCBwb2ludAorICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAmb3B0aW9uYWwgZmlsdGVyKQogICAiR2V0
IGZsZXgtY29tcGxldGlvbnMgb2YgU1RSSU5HIGluIFRBQkxFLCBnaXZlbiBQUkVEIGFuZCBQ
T0lOVC4iCiAgICh1bmxlc3MgKGFuZCBjb21wbGV0aW9uLWZsZXgtbm9zcGFjZSAoc3RyaW5n
LXNlYXJjaCAiICIgc3RyaW5nKSkKICAgICAocGNhc2UtbGV0ICgoYCgsYWxsICxwYXR0ZXJu
ICxwcmVmaXggLHN1ZmZpeCkKQEAgLTQxNjEsNyArNDE2Nyw4IEBAIGNvbXBsZXRpb24tZmxl
eC1hbGwtY29tcGxldGlvbnMKICAgICAgICAgICAgICAgICAgICMnY29tcGxldGlvbi1mbGV4
LS1tYWtlLWZsZXgtcGF0dGVybikpKQogICAgICAgKGNvbXBsZXRpb24tcGNtLS1kZWZlcnJl
ZC1oaWxpdCBwYXR0ZXJuIGFsbAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgKGxlbmd0aCBwcmVmaXgpCi0gICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAoLSAobGVuZ3RoIHN0cmluZykgKGxlbmd0aCBzdWZmaXgpKSkpKSkKKyAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICgtIChsZW5ndGggc3RyaW5nKSAobGVuZ3RoIHN1
ZmZpeCkpCisgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBmaWx0ZXIpKSkp
CiAKIDs7IEluaXRpYWxzIGNvbXBsZXRpb24KIDs7IENvbXBsZXRlIC91bXMgdG8gL3Vzci9t
b25uaWVyL3NyYyBvciBsY2ggdG8gbGlzdC1jb21tYW5kLWhpc3RvcnkuCkBAIC00MTk1LDE0
ICs0MjAyLDE2IEBAIGNvbXBsZXRpb24taW5pdGlhbHMtZXhwYW5kCiAgICAgICAgICAgICAo
Y29uY2F0IChzdWJzdHJpbmcgc3RyIDAgKGNhciBib3VuZHMpKQogICAgICAgICAgICAgICAg
ICAgICAobWFwY29uY2F0ICdzdHJpbmcgKHN1YnN0cmluZyBzdHIgKGNhciBib3VuZHMpKSBz
ZXApKSkpKSkpKQogCi0oZGVmdW4gY29tcGxldGlvbi1pbml0aWFscy1hbGwtY29tcGxldGlv
bnMgKHN0cmluZyB0YWJsZSBwcmVkIF9wb2ludCkKKyhkZWZ1biBjb21wbGV0aW9uLWluaXRp
YWxzLWFsbC1jb21wbGV0aW9ucyAoc3RyaW5nIHRhYmxlIHByZWQgX3BvaW50CisgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAmb3B0aW9uYWwg
ZmlsdGVyKQogICAobGV0ICgobmV3c3RyIChjb21wbGV0aW9uLWluaXRpYWxzLWV4cGFuZCBz
dHJpbmcgdGFibGUgcHJlZCkpKQogICAgICh3aGVuIG5ld3N0cgogICAgICAgKHBjYXNlLWxl
dCAoKGAoLHBhdHRlcm4gLGFsbCAscHJlZml4ICxfc3VmZml4KQogICAgICAgICAgICAgICAg
ICAgIChjb21wbGV0aW9uLXBjbS0tZmluZC1hbGwtY29tcGxldGlvbnMgbmV3c3RyIHRhYmxl
CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICBwcmVkIChsZW5ndGggbmV3c3RyKSkpKQogICAgICAgICAoY29tcGxldGlvbi1wY20t
LWRlZmVycmVkLWhpbGl0IHBhdHRlcm4gYWxsCi0gICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgKGxlbmd0aCBwcmVmaXgpIChsZW5ndGggc3RyaW5nKSkpKSkpCisg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgKGxlbmd0aCBwcmVmaXgp
IChsZW5ndGggc3RyaW5nKQorICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgIGZpbHRlcikpKSkpCiAKIChkZWZ1biBjb21wbGV0aW9uLWluaXRpYWxzLXRyeS1jb21w
bGV0aW9uIChzdHJpbmcgdGFibGUgcHJlZCBfcG9pbnQpCiAgIChsZXQgKChuZXdzdHIgKGNv
bXBsZXRpb24taW5pdGlhbHMtZXhwYW5kIHN0cmluZyB0YWJsZSBwcmVkKSkpCi0tIAoyLjIw
LjEKCg==
--------------4B3E2A198290BEF506AC64C4--




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 12 Aug 2021 08:47:28 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Aug 12 04:47:28 2021
Received: from localhost ([127.0.0.1]:37522 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mE6N6-0000Oh-CB
	for submit <at> debbugs.gnu.org; Thu, 12 Aug 2021 04:47:28 -0400
Received: from server.qxqx.de ([178.63.65.180]:51005 helo=mail.qxqx.de)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mail@HIDDEN>)
 id 1mE6N5-0000OR-Aa; Thu, 12 Aug 2021 04:47:27 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=qxqx.de;
 s=mail1392553390; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:
 MIME-Version:Date:Message-ID:From:References:Cc:To:Subject:Sender:Reply-To:
 Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender:
 Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:
 List-Subscribe:List-Post:List-Owner:List-Archive;
 bh=i2BeI1W5p4ITvwSsYZbpZMP46bquNh7WtZnDbLxnDTs=; b=CkXnzKP04T4dpOSSH2ZJ+E38aD
 YyMJjNahZq7mYM0sODxvUmJLg0HxHEwRzrUnlWF+2rHz9e1q7O4WlG8pPrsbBCyJ/2rOY5IXaMZy5
 AtgefXe2+civGL9/WP4+huRNqrnjQtnWrWEvRL+u+OnHMqHcrVrd8SGqA5EUYts6Zv/8=;
Subject: Re: bug#48841: [PATCH] Add new `completion-filter-completions` API
 and deferred highlighting
To: Eli Zaretskii <eliz@HIDDEN>
References: <f94541d3-5f93-9a23-8256-3bbc453f127a@HIDDEN>
 <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <837dgrdrec.fsf@HIDDEN>
From: Daniel Mendler <mail@HIDDEN>
Message-ID: <aff2bd3a-ea3e-1ac6-cf0d-2785c6591c84@HIDDEN>
Date: Thu, 12 Aug 2021 10:47:17 +0200
MIME-Version: 1.0
In-Reply-To: <837dgrdrec.fsf@HIDDEN>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 7bit
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 47711
Cc: 47711 <at> debbugs.gnu.org, monnier@HIDDEN, joaotavora@HIDDEN,
 48841 <at> debbugs.gnu.org, dgutov@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Eli, thank you for your feedback and for pointing me to the mode which
helps with the formatting.  I will address the documentation and
formatting issues as soon as the discussion has concluded.

In the following I answer to a few of your questions about technical
details.

>> The `completions` value is the list of completion strings *without*
>> applied highlighting.  The completion strings are returned unmodified,
>> which avoids allocations and results in performance gains for
> 
> This is unclear: how can you return a list of strings which you
> produce without allocating the strings?

The function 'completion-filter-completions' receives a completion table
as argument.  The strings produced by this table are returned
unmodified, but of course the completion table has to produce them.  For
a static completion table (e.g., in the simplest case a list of strings)
the completion table itself will not allocate strings.  In this scenario
'completion-filter-completions' will not perform any string allocations,
only the list will be allocated.  This is what leads to major
performance gains.

>> +(defvar completion--filter-completions nil
>> +  "Enable the new completions return value format.
>
> Btw, why is this an internal variable?  Shouldn't all completion
> styles ideally support it?  If so, it should be a public variable,
> documented in the ELisp manual.  And the name should also end with -p,
> since it's a boolean.  How about completion-filter-completions-format-p?

(As I understood the style guide '-p' is not a good idea for boolean
variables, since a value is not a predicate in a strict sense.)

To address your technical comment - this variable is precisely what one
of the technical difficulties mentioned in my other mail is about.  The
question is how we can retain backward compatibility of the completion
style 'all' functions, e.g., 'completion-basic-all-completions', while
still allowing the function to return the newly introduced alist format
with more data, which enables 'completion-filter-completions' to perform
the efficient deferred highlighting.

> Also, the "This function has been superseded..." part should be a new
> paragraph, so that it stands out.  (And I'm not yet sure we indeed
> want to say "superseded" here, but that's part of the on-going
> discussion.  maybe use a more neutral language here, like "See also".)

The new API 'completion-filter-completions' will substitute the existing
API 'completion-all-completions'.  I only didn't go as far as
deprecating the 'completion-all-completions' API right away, but we
could also do this.

> Is "filter" really the right word here (in the doc string)?  "Filer"
> means you take a sequence and produce another sequence with some
> members removed.  That's not what this API does, is it?  Suggest to
> use a different name, like completion-completions-alist or
> completion-all-completions-as-alist.

"Filter" seems like exactly the right word to me.  The function takes a
list of strings (or a completion table) and returns a subset of matching
completion strings without further modifications to the strings. See
above what I wrote about allocations.

>> +Only the elements of table that satisfy predicate PRED are considered.
>> +POINT is the position of point within STRING.  The METADATA may be
>> +modified by the completion style.  The return value is a alist with
>> +the keys:
>> +
>> +- base: Base position of the completion (from the start of STRING)
> 
> "Base" here means the beginning?  If so, why not call it "beg" or
> somesuch?

Base position is a fixed term which is already used in minibuffer.el for
completions.  See also 'completion-base-position' for example.

> Are we really losing the completion-score property here?  If so, why?

Yes, the property is removed in the current patch.  It is not actually
used for anything in the new implementation.  But it is possible to
restore the property such that 'completion-all-completions' always
returns scored candidates as it does now.  See my other mail regarding
the caveats of the current patch.

Daniel




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 12 Aug 2021 08:00:41 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Aug 12 04:00:40 2021
Received: from localhost ([127.0.0.1]:37404 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mE5do-0007UR-3t
	for submit <at> debbugs.gnu.org; Thu, 12 Aug 2021 04:00:40 -0400
Received: from eggs.gnu.org ([209.51.188.92]:46316)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>)
 id 1mE5dl-0007U9-G8; Thu, 12 Aug 2021 04:00:38 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:58862)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <eliz@HIDDEN>)
 id 1mE5dd-0002GC-RB; Thu, 12 Aug 2021 04:00:29 -0400
Received: from 84.94.185.95.cable.012.net.il ([84.94.185.95]:3162
 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1mE5dd-0008LK-Eb; Thu, 12 Aug 2021 04:00:29 -0400
Date: Thu, 12 Aug 2021 11:00:11 +0300
Message-Id: <837dgrdrec.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Daniel Mendler <mail@HIDDEN>
In-Reply-To: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN> (message
 from Daniel Mendler on Wed, 11 Aug 2021 16:16:57 +0200)
Subject: Re: bug#48841: [PATCH] Add new `completion-filter-completions` API and
 deferred highlighting
References: <f94541d3-5f93-9a23-8256-3bbc453f127a@HIDDEN>
 <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
MIME-version: 1.0
Content-type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 47711
Cc: 47711 <at> debbugs.gnu.org, monnier@HIDDEN, joaotavora@HIDDEN,
 48841 <at> debbugs.gnu.org, dgutov@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

[I removed emacs-devel from the CC list, please don't cross-post to
both emacs-devel and the bug tracker.]

> From: Daniel Mendler <mail@HIDDEN>
> Date: Wed, 11 Aug 2021 16:16:57 +0200
> Cc: 48841 <at> debbugs.gnu.org, Dmitry Gutov <dgutov@HIDDEN>,
>  João Távora <joaotavora@HIDDEN>,
>  Stefan Monnier <monnier@HIDDEN>, 47711 <at> debbugs.gnu.org
> 
> I prepared a patch which provides the API
> `completion-filter-completions`. This function supports deferred
> highlighting and returns additional data with the list of matching
> completion candidates. The API supersedes the existing function
> `completion-all-completions`.

Thanks.  The discussion of this is still going on, and I don't
consider myself an expert in this area of Emacs, so below please find
only comments for minor formatting and documentation aspects.

> Add a new `completion-filter-completions` API, which supersedes
> `completion-all-completions`.  The new API returns the matching
> completion candidates and additional data.  The return value is an
> alist, with the keys `completions`, `base`, `end` and `highlight`.
> The API can be extended in a backward compatible way later on thanks
> to the use of an alist as return value.

Please don't use Markdown-style quoting `like this` in our comments
and log messages.  We quote 'like this' in these places.

> The `completions` value is the list of completion strings *without*
> applied highlighting.  The completion strings are returned unmodified,
> which avoids allocations and results in performance gains for

This is unclear: how can you return a list of strings which you
produce without allocating the strings?

>         The value `base` is the base position of the completion.

"Base position" where, or relative to what object?

> Correspondingly the value `end` specifies the end position of the
> completion counted from the beginning of the input strng.

So the base position is also relative to the beginning of the input
string?  If so, please say so explicitly.

>                                                    Finally the
> `highlight` value is a function taking a list of completion strings
> and returns a new list of new strings with highlighting applied.

If you say "taking a list...", then for consistent style please also
say "...and returning a new list...".

> A continously updating UI can use the highlighting function to apply
> highlighting only to the visible completions.

Not, "visible", but "displayed", right?  So I'd rephrase

  ...to apply highlighting only to the completions that are actually
  displayed.

> (completion-basic-all-completions,
> completion-emacs21-all-completions,
> completion-emacs22-all-completions): Use it.

That's incorrect format, I guess you did it by hand rather than
letting change-log-mode do it for you?  The correct format looks like
this:

  (completion-basic-all-completions)
  (completion-emacs21-all-completions)
  (completion-emacs22-all-completions): use it.

IOW, each line ends with a closing parentheses, not a comma.

> +(defvar completion--filter-completions nil
> +  "Enable the new completions return value format.

Using genitive construction should be limited to just 2 words.  Here
you have 4, which produces an awkward, hard to interpret phrase.
Suggest to reword:

  If non-nil, return completions in `completion-filter-completions' format.

Note that I also dropped the "new" part (which generally doesn't stand
the test of time), and instead gave a hint as to what that format is.

Btw, why is this an internal variable?  Shouldn't all completion
styles ideally support it?  If so, it should be a public variable,
documented in the ELisp manual.  And the name should also end with -p,
since it's a boolean.  How about completion-filter-completions-format-p?

> +            New completion style functions may always return their
> +results in the new alist format, since `completion-all-completions'
> +transparently converts back to the old improper list of completions
> +with base size in the last cdr.")

"may" and "always" are a kind of contradiction.

Also, I'd drop the "improper" part, as it sounds like a derogatory
adjective.

>  (defun completion-try-completion (string table pred point &optional metadata)
>    "Try to complete STRING using completion table TABLE.
>  Only the elements of table that satisfy predicate PRED are considered.
> -POINT is the position of point within STRING.
> -The return value can be either nil to indicate that there is no completion,
> -t to indicate that STRING is the only possible completion,
> -or a pair (NEWSTRING . NEWPOINT) of the completed result string together with
> -a new position for point."
> +POINT is the position of point within STRING.  The return value can be
> +either nil to indicate that there is no completion, t to indicate that
> +STRING is the only possible completion, or a pair (NEWSTRING . NEWPOINT)
> +of the completed result string together with a new position for point.
> +The METADATA may be modified by the completion style."

Here you changed whitespace by filling, and that ruined the
intentionally formatted doc string, which made it easy to find the
various forms of the return value and the important parts of the doc
string.  Please keep the original formatting.

>  (defun completion-all-completions (string table pred point &optional metadata)
>    "List the possible completions of STRING in completion table TABLE.
>  Only the elements of table that satisfy predicate PRED are considered.
> -POINT is the position of point within STRING.
> -The return value is a list of completions and may contain the base-size
> -in the last `cdr'."
> -  ;; FIXME: We need to additionally return the info needed for the
> -  ;; second part of completion-base-position.
> -  (completion--nth-completion 2 string table pred point metadata))
> +POINT is the position of point within STRING.  The return value is a
> +list of completions and may contain the base-size in the last `cdr'.
> +The METADATA may be modified by the completion style.  This function
> +has been superseded by `completion-filter-completions', which returns
> +richer information and supports deferred candidate highlighting."

Likewise here.

Also, the "This function has been superseded..." part should be a new
paragraph, so that it stands out.  (And I'm not yet sure we indeed
want to say "superseded" here, but that's part of the on-going
discussion.  maybe use a more neutral language here, like "See also".)

> +    (if (and result (consp (car result)))
> +        ;; Give the completion styles some freedom!
> +        ;; If they are targeting Emacs 28 upwards only, they
> +        ;; may always return a result with deferred
> +        ;; highlighting.  We convert back to the old format
> +        ;; here by applying the highlighting eagerly.

"May always" again.  How about "can always" instead?

> +        (nconc (funcall (cdr (assq 'highlight result))
> +                        (cdr (assq 'completions result)))
> +               (cdr (assq 'base result)))
> +      result)))
> +
> +(defun completion-filter-completions (string table pred point metadata)
> +  "Filter the possible completions of STRING in completion table TABLE.

Is "filter" really the right word here (in the doc string)?  "Filer"
means you take a sequence and produce another sequence with some
members removed.  That's not what this API does, is it?  Suggest to
use a different name, like completion-completions-alist or
completion-all-completions-as-alist.

> +Only the elements of table that satisfy predicate PRED are considered.
> +POINT is the position of point within STRING.  The METADATA may be
> +modified by the completion style.  The return value is a alist with
> +the keys:
> +
> +- base: Base position of the completion (from the start of STRING)

"Base" here means the beginning?  If so, why not call it "beg" or
somesuch?

> +This function supersedes the function `completion-all-completions'."

Again, "supersedes" is too strong, IMO.  I would say "is a variant of"
instead, and explain why this variant could be better suited to some
use cases.  IOW, explain the upsides (and downsides, if any), and let
the programmers decide whether they want this, instead of more-or-less
forcing them to use it.

> +        ;; Deferred highlighting has been requested, but the completion
> +        ;; style returned a non-deferred result. Convert the result to the
                                                  ^^
two spaces between sentences, please.

> +        ;; new alist format.

"New" is not a good word here.

> +      ;; added by the completion machinery.

Please start comments with a capital letter.

> +(defun completion--deferred-hilit (completions prefix-len base end)
> +  "Return completions in old format or new alist format.
> +If `completion--filter-completions' is non-nil use the new format."

Again, please don't use "old" and "new" here, but instead describe
explicitly the differences between them, or provide a hyperlink to
where that is described.

> +                      ;; Apply highlighting

Please end each sentence in a comment with a period.

> +(defun completion-pcm--deferred-hilit (pattern completions base end)
> +  "Return completions in old format or new alist format.
> +If `completion--filter-completions' is non-nil use the new format."

"Old" and "new" again.

>  (defun completion-pcm--hilit-commonality (pattern completions)
>    "Show where and how well PATTERN matches COMPLETIONS.
>  PATTERN, a list of symbols and strings as seen
>  `completion-pcm--merge-completions', is assumed to match every
>  string in COMPLETIONS.  Return a deep copy of COMPLETIONS where
> -each string is propertized with `completion-score', a number
> -between 0 and 1, and with faces `completions-common-part',
> +each string is propertized with faces `completions-common-part',
>  `completions-first-difference' in the relevant segments."

Are we really losing the completion-score property here?  If so, why?

> +           ;; If `pattern' doesn't have an explicit trailing any,

This is confusing: what do you mean by "explicit trailing any" in the
context of patterns?

> +(defun completion--flex-score (pattern completions)
> +  "Compute how well PATTERN matches COMPLETIONS.
> +PATTERN, a list of strings is assumed to match every string in
> +COMPLETIONS.

Is PATTERN really a list?  It would be strange for a list to be called
PATTERN, and how can a list "match every string in COMPLETIONS"?

>               Return a copy of COMPLETIONS where each element is
> +a pair of a score and the completion string.

What is "the completion string" in this case? is it the same string
from COMPLETIONS, or is it something else?  The doc string leaves that
unclear.

>                                                The score lies in
> +the range between -1 and 0, where -1 corresponds to the full
> +match."

What score could a partial match have, and what is the meaning of the
numerical value for a partial match?




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 11 Aug 2021 16:17:28 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Wed Aug 11 12:17:28 2021
Received: from localhost ([127.0.0.1]:36392 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mDqv1-0005ei-S7
	for submit <at> debbugs.gnu.org; Wed, 11 Aug 2021 12:17:28 -0400
Received: from mail-pj1-f52.google.com ([209.85.216.52]:42550)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <joaotavora@HIDDEN>)
 id 1mDquz-0005ZI-D2; Wed, 11 Aug 2021 12:17:26 -0400
Received: by mail-pj1-f52.google.com with SMTP id
 mq2-20020a17090b3802b0290178911d298bso5839129pjb.1; 
 Wed, 11 Aug 2021 09:17:25 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=mime-version:references:in-reply-to:from:date:message-id:subject:to
 :cc:content-transfer-encoding;
 bh=jA8E40VtlAuX0iMEkWlyJ13M+vYAfR9CULGNzYrMWuw=;
 b=nqcIq4vOrpqIwPVP+7LOt4OpWbiWWrogS1eHgcHDa+vtPH3Ha0L1qzKb8X4HN7V5s9
 YZ0gwitcOOhmL8STj5M8Z+dbTpuziDlwrWH6eWNnp75JcAKnCAD+/Vh2y+yBNgzr3a10
 YhROHr/KYUsOIcV/5Auj+FMbg2InwANm0FFPpUXlOktGbLo1376Vm2WL8kBg6PLjoUgX
 3HEkWQg3QGPpF0tkSrmYTnchgUFZKmkaZOEXuJE/+qDE0cpoVjOaE2bzNLGJiX+iMQac
 lZNhoyzC7vgfvHed3pwZIAxGNM+hp5J29mlwnpKLsgpdGXVgOm2HtDlQonDhXSLUMuKs
 FduQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:mime-version:references:in-reply-to:from:date
 :message-id:subject:to:cc:content-transfer-encoding;
 bh=jA8E40VtlAuX0iMEkWlyJ13M+vYAfR9CULGNzYrMWuw=;
 b=SOA8e6x4uRSrmAteCOOcUkm3Wc//TKG1aUAO0VpyNZo23J9oLGpXKmrO4CwiWzawqq
 oMzOmbDbZwvpxK2fht4BSlDfbVlMVT5A4ResIlgi9MokNykxQJ83ObqmYUSrUayD9B/1
 QdNn1r9/JhDMqAJAxod8u10v4bpt4OUDSLg/pm6c8eM+2Ogj4psMkTdfGlAyzHABjrFx
 dhg1Dt0hGftCHeyZ5qtbYX1RWkZwGaQnBeboDpBmkG5kwEHn7jeUFpkH0VDD7wqtypWk
 tk1mkGfjpL16Yu398Z8Oi6jf0W/p87EmReZnuO1tJaflpo/nnWtaAHAXDM3p3gIhxXyj
 3bIg==
X-Gm-Message-State: AOAM533lMhyMxqZtgJ6+NSfSSwwNOHnH5JlHjTOqTWHT4jh7pBDSCOAJ
 urUK93eFKxO3eU+2okNK5X1/EUPTFArte+8c/rw=
X-Google-Smtp-Source: ABdhPJyP0fKKy0TCf0MVCKV4sAlwqv6ZxwwQMS5d//O3nf4QoGUYY3eWz0YesAADm/rqyracNgAd497dBs1yi5ai1Rg=
X-Received: by 2002:aa7:9050:0:b029:3af:7e99:f48f with SMTP id
 n16-20020aa790500000b02903af7e99f48fmr29318366pfo.2.1628698639473; Wed, 11
 Aug 2021 09:17:19 -0700 (PDT)
MIME-Version: 1.0
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
 <f8c4419a-7e83-b8a9-347c-0221a2052376@HIDDEN>
In-Reply-To: <f8c4419a-7e83-b8a9-347c-0221a2052376@HIDDEN>
From: =?UTF-8?B?Sm/Do28gVMOhdm9yYQ==?= <joaotavora@HIDDEN>
Date: Wed, 11 Aug 2021 17:17:07 +0100
Message-ID: <CALDnm52AFkHT7UwncetuK6mzzz0wD2QxAM-zNBG0qLfYQ284Ng@HIDDEN>
Subject: Re: [PATCH] Add new `completion-filter-completions` API and deferred
 highlighting
To: Daniel Mendler <mail@HIDDEN>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 47711
Cc: 47711 <at> debbugs.gnu.org, Dmitry Gutov <dgutov@HIDDEN>,
 Stefan Monnier <monnier@HIDDEN>, 48841 <at> debbugs.gnu.org,
 "emacs-devel@HIDDEN" <emacs-devel@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Perhaps you should first provide a patch with these 2 "little effort" chang=
es,
(that are presumably also backward compatible and don't affect the API) by
themselves.  Reading about these complex ideas isn't as clear as seeing
them in actual code.

Then it'll be easier to evaluate the merits of the patch you proposed in
your first email.

Jo=C3=A3o

On Wed, Aug 11, 2021 at 5:11 PM Daniel Mendler <mail@HIDDEN> wro=
te:
>
> On 8/11/21 4:16 PM, Daniel Mendler wrote:
> > I prepared a patch which provides the API
> > `completion-filter-completions`. This function supports deferred
> > highlighting and returns additional data with the list of matching
> > completion candidates. The API supersedes the existing function
> > `completion-all-completions`.
> >
> > The main goal of the new API is to avoid expensive string allocations
> > and highlighting during completion. This is particularly relevant for
> > continuously updating completion UIs like Icomplete or Vertico.
> > Furthermore the end position of the completion boundaries is returned
> > with the completion results. This information is not provided by the
> > existing `completion-all-completions` API.
> >
> > See also the relevant bugs bug#47711 and bug#48841. I am looking forwar=
d
> > to your feedback. Thank you!
>
> There are currently two issues with the patch with regards to backward
> compatibility. Fortunately they are fixable with a little effort.
>
> 1. I would like to deprecate `completion-score' or remove it altogether,
>    but unfortunately `completion-score' is used in the wild. In order to
>    preserve `completion-score', bind `completion--filter-completions' in
>    the highlighting functions. Add `completion-score' in
>    `completion-pcm--hilit-commonality' when
>    `completion--filter-completions' is nil.
>
> 2. In `completion--nth-completion' set `completion--filter-completions'
>    to nil, unless `(memq style '(emacs21 emacs22 basic
>    partial-completion initials flex))' such that custom completion
>    styles which wrap the completion functions don't see the new return
>    value format, except if the custom style opts in explicitly by
>    binding `completion--filter-completions'. An alternative criterion is
>    `(memq fun '(completion-emacs22-all-completions) ...)'. Unfortunately
>    this approach will still not work if the user has advised a
>    `completion-x-all-completions' function. The only 100% safe approach
>    seems to transparently redirect calls to
>    `completion-x-all-completions' to `completion--x-filter-completions',
>    which returns the results in the new format.
>
> With these changes the patch should be 100% backward compatible.
>
> Daniel



--=20
Jo=C3=A3o T=C3=A1vora




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 11 Aug 2021 16:11:31 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Wed Aug 11 12:11:31 2021
Received: from localhost ([127.0.0.1]:36386 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mDqpH-0003Uv-46
	for submit <at> debbugs.gnu.org; Wed, 11 Aug 2021 12:11:31 -0400
Received: from server.qxqx.de ([178.63.65.180]:60351 helo=mail.qxqx.de)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mail@HIDDEN>)
 id 1mDqpF-0003UZ-3i; Wed, 11 Aug 2021 12:11:29 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=qxqx.de;
 s=mail1392553390; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:
 MIME-Version:Date:Message-ID:References:Cc:To:From:Subject:Sender:Reply-To:
 Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender:
 Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:
 List-Subscribe:List-Post:List-Owner:List-Archive;
 bh=L9YD2Rbs2yUeev3cPOFCPRt64IqdN05dDBBAfML4R0Y=; b=PYVBp5sBBCAoJ4YvgPqbbwiiXL
 IuVLkgk+39f09D7qa9siQeSn5VpG+WNb4+9wvgzxTMubp8aXRfRL9GfhgFqd7ORoZhAwiv0T5G5KC
 i2ckBeXWHmESGnVZ3uZ7cQ/fP1vXU8fQja7S3gO1j634WEiBrgFDUNflZQz1n2BV8CGc=;
Subject: Re: [PATCH] Add new `completion-filter-completions` API and deferred
 highlighting
From: Daniel Mendler <mail@HIDDEN>
To: "emacs-devel@HIDDEN" <emacs-devel@HIDDEN>
References: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
Message-ID: <f8c4419a-7e83-b8a9-347c-0221a2052376@HIDDEN>
Date: Wed, 11 Aug 2021 18:11:21 +0200
MIME-Version: 1.0
In-Reply-To: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 7bit
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 47711
Cc: 48841 <at> debbugs.gnu.org, Dmitry Gutov <dgutov@HIDDEN>,
 =?UTF-8?B?Sm/Do28gVMOhdm9yYQ==?= <joaotavora@HIDDEN>,
 Stefan Monnier <monnier@HIDDEN>, 47711 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

On 8/11/21 4:16 PM, Daniel Mendler wrote:
> I prepared a patch which provides the API
> `completion-filter-completions`. This function supports deferred
> highlighting and returns additional data with the list of matching
> completion candidates. The API supersedes the existing function
> `completion-all-completions`.
> 
> The main goal of the new API is to avoid expensive string allocations
> and highlighting during completion. This is particularly relevant for
> continuously updating completion UIs like Icomplete or Vertico.
> Furthermore the end position of the completion boundaries is returned
> with the completion results. This information is not provided by the
> existing `completion-all-completions` API.
> 
> See also the relevant bugs bug#47711 and bug#48841. I am looking forward
> to your feedback. Thank you!

There are currently two issues with the patch with regards to backward
compatibility. Fortunately they are fixable with a little effort.

1. I would like to deprecate `completion-score' or remove it altogether,
   but unfortunately `completion-score' is used in the wild. In order to
   preserve `completion-score', bind `completion--filter-completions' in
   the highlighting functions. Add `completion-score' in
   `completion-pcm--hilit-commonality' when
   `completion--filter-completions' is nil.

2. In `completion--nth-completion' set `completion--filter-completions'
   to nil, unless `(memq style '(emacs21 emacs22 basic
   partial-completion initials flex))' such that custom completion
   styles which wrap the completion functions don't see the new return
   value format, except if the custom style opts in explicitly by
   binding `completion--filter-completions'. An alternative criterion is
   `(memq fun '(completion-emacs22-all-completions) ...)'. Unfortunately
   this approach will still not work if the user has advised a
   `completion-x-all-completions' function. The only 100% safe approach
   seems to transparently redirect calls to
   `completion-x-all-completions' to `completion--x-filter-completions',
   which returns the results in the new format.

With these changes the patch should be 100% backward compatible.

Daniel




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 11 Aug 2021 14:17:13 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Wed Aug 11 10:17:13 2021
Received: from localhost ([127.0.0.1]:36222 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mDp2d-0006Yh-4Z
	for submit <at> debbugs.gnu.org; Wed, 11 Aug 2021 10:17:13 -0400
Received: from server.qxqx.de ([178.63.65.180]:59847 helo=mail.qxqx.de)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mail@HIDDEN>)
 id 1mDp2X-0006Xz-HJ; Wed, 11 Aug 2021 10:17:08 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=qxqx.de;
 s=mail1392553390; h=Content-Type:MIME-Version:Date:Message-ID:Subject:From:Cc
 :To:Sender:Reply-To:Content-Transfer-Encoding:Content-ID:Content-Description:
 Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:
 In-Reply-To:References:List-Id:List-Help:List-Unsubscribe:List-Subscribe:
 List-Post:List-Owner:List-Archive;
 bh=jI1hVGefRnIqqV63epya37LbCwXprNTn1ArIfYVUC30=; b=yxBwqrKs8WqUGCKrekohWXbjMR
 m08SIThso/uGJtkcBilQwb/05lPWJ9HN8PxrQOP6eiiT7fsug3Re7hkbNRdqo6Unt9xIxRfl70zp3
 NsowJoPS7ZMi4mClGdJmTb33/gve/EMaHQ/WkjdJpco+1za1E/Y8Z0vV91g0rWb0eXh8=;
To: "emacs-devel@HIDDEN" <emacs-devel@HIDDEN>
From: Daniel Mendler <mail@HIDDEN>
Subject: [PATCH] Add new `completion-filter-completions` API and deferred
 highlighting
Message-ID: <aa52bb9b-2adb-e579-6da1-d784e57e3c68@HIDDEN>
Date: Wed, 11 Aug 2021 16:16:57 +0200
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="------------3147CDE26CD3C1A11006A179"
Content-Language: en-US
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 47711
Cc: =?UTF-8?B?Sm/Do28gVMOhdm9yYQ==?= <joaotavora@HIDDEN>,
 47711 <at> debbugs.gnu.org, Stefan Monnier <monnier@HIDDEN>,
 48841 <at> debbugs.gnu.org, Dmitry Gutov <dgutov@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

This is a multi-part message in MIME format.
--------------3147CDE26CD3C1A11006A179
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

I prepared a patch which provides the API
`completion-filter-completions`. This function supports deferred
highlighting and returns additional data with the list of matching
completion candidates. The API supersedes the existing function
`completion-all-completions`.

The main goal of the new API is to avoid expensive string allocations
and highlighting during completion. This is particularly relevant for
continuously updating completion UIs like Icomplete or Vertico.
Furthermore the end position of the completion boundaries is returned
with the completion results. This information is not provided by the
existing `completion-all-completions` API.

See also the relevant bugs bug#47711 and bug#48841. I am looking forward
to your feedback. Thank you!

Daniel Mendler

--------------3147CDE26CD3C1A11006A179
Content-Type: text/x-diff; charset=UTF-8;
 name="0001-Add-new-completion-filter-completions-API-and-deferr.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename*0="0001-Add-new-completion-filter-completions-API-and-deferr.pa";
 filename*1="tch"

From e7f26abc520ac36cc154a92bfb4744837d9f7e5e Mon Sep 17 00:00:00 2001
From: Daniel Mendler <mail@HIDDEN>
Date: Mon, 12 Jul 2021 21:40:32 +0200
Subject: [PATCH] Add new `completion-filter-completions` API and deferred
 highlighting

Fix bug#47711.

Add a new `completion-filter-completions` API, which supersedes
`completion-all-completions`.  The new API returns the matching
completion candidates and additional data.  The return value is an
alist, with the keys `completions`, `base`, `end` and `highlight`.
The API can be extended in a backward compatible way later on thanks
to the use of an alist as return value.

The `completions` value is the list of completion strings *without*
applied highlighting.  The completion strings are returned unmodified,
which avoids allocations and results in performance gains for
continuously updating completion UIs, like Icomplete or Vertico (GNU
ELPA).  The value `base` is the base position of the completion.
Correspondingly the value `end` specifies the end position of the
completion counted from the beginning of the input strng.  In
comparison, the old function `completion-all-completions` only
returned the base position in the last cdr of the returned completions
list, which complicated usage.  The `end` position was not provided by
`completion-all-completions`.  Given the new API the
`completion-base-position` can be set accurately.  Finally the
`highlight` value is a function taking a list of completion strings
and returns a new list of new strings with highlighting applied.  A
continously updating UI can use the highlighting function to apply
highlighting only to the visible completions.

* lisp/minibuffer.el:
(completion-pcm--hilit-commonality): Remove scoring computation.
(completion--adjust-metadata): Rename to `completion--style-metadata`
due to change of calling convention.
(completion--nth-completion): Call renamed metadata adjustment
function.  Ignore the old property `completion--adjust-metadata`.
(completion--flex-adjust-metadata): Rename function.
(completion--twq-all): Attach `completion--unquoted` text property to
quoted completion strings.
(completion--flex-score): New optimized scoring function.  Use
`completion--unquoted` text property.
(completion--flex-style-metadata): Use it.
(completion--pattern-compiler): New function.
(completion-substring--all-completions,
completion--flex-score): Use it.
(completion--hilit-commonality): New function.
(completion-hilit-commonality): Use it.
(completion--deferred-hilit): New function.
(completion-basic-all-completions,
completion-emacs21-all-completions,
completion-emacs22-all-completions): Use it.
(completion--pcm-deferred-hilit): New function.
(completion-pcm-all-completions,
completion-flex-all-completions,
completion-initials-all-completions,
completion-substring-all-completions): Use it.
(completion--filter-completions): New variable to conditionally enable
the new alist completions result format.  This variable is for
internal use to preserve the existing calling convention of the
completion style `all` functions.
(completion-filter-completions): New API which returns the completion
strings and additional data as an an alist.  Transparently convert
old-fashioned completion style results to the new format.
(completion-all-completions): Transparently downgrade the
new-fashioned completion style result to the old list format.
(minibuffer-completion-help): Use the new API, set
`completion-base-position` correctly.
(completion-try-completion,
completion-all-completions): Update doc string.
(completion--replace): Remove unnecessary property removal.

* test/lisp/minibuffer-tests.el:
(completion--pcm-score): Remove obsolete function.
(completion-*-test-*): Remove obsolete functions, rename.
(completion-flex-score-test-*): Add new scoring test functions.
(completion--test-style): New test helper function.
(completion-*-style-test): Add new API tests for each built-in
completion style.
(completion--test-boundaries): New test helper function.
(completion-*-boundaries-test): New boundary tests for each built-in
completion style.
(completion-filter-completions-highlight-test): New API test.
(completion-emacs22orig-all-completions): New function.
(completion-upgrade-return-type-test): New test of transparent
completion style return value upgrade.
---
 lisp/minibuffer.el            | 453 +++++++++++++++++++++++-----------
 test/lisp/minibuffer-tests.el | 257 +++++++++++++++----
 2 files changed, 516 insertions(+), 194 deletions(-)

diff --git a/lisp/minibuffer.el b/lisp/minibuffer.el
index 9f327df28f..ba8855c4ea 100644
--- a/lisp/minibuffer.el
+++ b/lisp/minibuffer.el
@@ -692,6 +692,10 @@ completion--twq-all
                                              'completions-common-part)
                                qprefix))))
                         (qcompletion (concat qprefix qnew)))
+                   ;; Attach unquoted completion string, which is needed
+                   ;; to score the completion in `completion--flex-score'.
+                   (put-text-property 0 1 'completion--unquoted
+                                      completion qcompletion)
 		   ;; FIXME: Similarly here, Cygwin's mapping trips this
 		   ;; assertion.
                    ;;(cl-assert
@@ -1035,6 +1039,17 @@ completion--styles
         (delete-dups (append (cdr over) (copy-sequence completion-styles)))
        completion-styles)))
 
+(defvar completion--filter-completions nil
+  "Enable the new completions return value format.
+If this variable is non-nil the `all-completions' function of a
+completion style should return the results in the new alist format of
+`completion-filter-completions'.  This variable is purely needed to
+for backward compatibility of the existing builtin completion style
+functions.  New completion style functions may always return their
+results in the new alist format, since `completion-all-completions'
+transparently converts back to the old improper list of completions
+with base size in the last cdr.")
+
 (defun completion--nth-completion (n string table pred point metadata)
   "Call the Nth method of completion styles."
   ;; We provide special support for quoting/unquoting here because it cannot
@@ -1061,6 +1076,15 @@ completion--nth-completion
                  ;; the original table, in that case!
                  (functionp table))
             (let ((new (funcall table string point 'completion--unquote)))
+              ;; FIXME For now do not attempt deferred highlighting if
+              ;; quoting is used.  Not doing deferred highlighting is
+              ;; not too severe in this case, since
+              ;; `completion--twq-all' is already an expensive
+              ;; function, which allocates all completion strings.  In
+              ;; contrast to plain completion tables, the savings of
+              ;; deferred highlighting would be minimal in the case of
+              ;; quoted completion tables.
+              (setq completion--filter-completions nil)
               (setq string (pop new))
               (setq table (pop new))
               (setq point (pop new))
@@ -1074,9 +1098,10 @@ completion--nth-completion
                                    string table pred point)))
                (and probe (cons probe style))))
            (completion--styles md)))
-         (adjust-fn (get (cdr result-and-style) 'completion--adjust-metadata)))
-    (when (and adjust-fn metadata)
-      (setcdr metadata (cdr (funcall adjust-fn metadata))))
+         (style-md (get (cdr result-and-style) 'completion--style-metadata)))
+    (when (and style-md metadata)
+      (setcdr metadata (cdr (funcall style-md
+                                     string table pred point metadata))))
     (if requote
         (funcall requote (car result-and-style) n)
       (car result-and-style))))
@@ -1084,22 +1109,64 @@ completion--nth-completion
 (defun completion-try-completion (string table pred point &optional metadata)
   "Try to complete STRING using completion table TABLE.
 Only the elements of table that satisfy predicate PRED are considered.
-POINT is the position of point within STRING.
-The return value can be either nil to indicate that there is no completion,
-t to indicate that STRING is the only possible completion,
-or a pair (NEWSTRING . NEWPOINT) of the completed result string together with
-a new position for point."
+POINT is the position of point within STRING.  The return value can be
+either nil to indicate that there is no completion, t to indicate that
+STRING is the only possible completion, or a pair (NEWSTRING . NEWPOINT)
+of the completed result string together with a new position for point.
+The METADATA may be modified by the completion style."
   (completion--nth-completion 1 string table pred point metadata))
 
 (defun completion-all-completions (string table pred point &optional metadata)
   "List the possible completions of STRING in completion table TABLE.
 Only the elements of table that satisfy predicate PRED are considered.
-POINT is the position of point within STRING.
-The return value is a list of completions and may contain the base-size
-in the last `cdr'."
-  ;; FIXME: We need to additionally return the info needed for the
-  ;; second part of completion-base-position.
-  (completion--nth-completion 2 string table pred point metadata))
+POINT is the position of point within STRING.  The return value is a
+list of completions and may contain the base-size in the last `cdr'.
+The METADATA may be modified by the completion style.  This function
+has been superseded by `completion-filter-completions', which returns
+richer information and supports deferred candidate highlighting."
+  (let ((completion--filter-completions nil)
+        (result (completion--nth-completion 2 string table
+                                            pred point metadata)))
+    (if (and result (consp (car result)))
+        ;; Give the completion styles some freedom!
+        ;; If they are targeting Emacs 28 upwards only, they
+        ;; may always return a result with deferred
+        ;; highlighting.  We convert back to the old format
+        ;; here by applying the highlighting eagerly.
+        (nconc (funcall (cdr (assq 'highlight result))
+                        (cdr (assq 'completions result)))
+               (cdr (assq 'base result)))
+      result)))
+
+(defun completion-filter-completions (string table pred point metadata)
+  "Filter the possible completions of STRING in completion table TABLE.
+Only the elements of table that satisfy predicate PRED are considered.
+POINT is the position of point within STRING.  The METADATA may be
+modified by the completion style.  The return value is a alist with
+the keys:
+
+- base: Base position of the completion (from the start of STRING)
+- end: End position of the completion (from the start of STRING)
+- highlight: Highlighting function taking a list of completions and
+  returning a new list of new strings with applied highlighting.
+- completions: The list of completions.
+
+This function supersedes the function `completion-all-completions'."
+  (let* ((completion--filter-completions t)
+         (result (completion--nth-completion 2 string table
+                                             pred point metadata)))
+    (if (and result (not (consp (car result))))
+        ;; Deferred highlighting has been requested, but the completion
+        ;; style returned a non-deferred result. Convert the result to the
+        ;; new alist format.
+        (let* ((last (last result))
+               (base (or (cdr last) 0)))
+          (setcdr last nil)
+          `((base . ,base)
+            (end . ,(length string))
+            (highlight . identity)
+            (completions . ,result)))
+      result)))
 
 (defun minibuffer--bitset (modified completions exact)
   (logior (if modified    4 0)
@@ -1114,9 +1181,8 @@ completion--replace
   ;; include upon insertion.
   (if minibuffer-allow-text-properties
       ;; If we're preserving properties, then just remove the faces
-      ;; and other properties added by the completion machinery.
-      (remove-text-properties 0 (length newtext) '(face completion-score)
-                              newtext)
+      ;; added by the completion machinery.
+      (remove-text-properties 0 (length newtext) '(face nil) newtext)
     ;; Remove all text properties.
     (set-text-properties 0 (length newtext) nil newtext))
   ;; Maybe this should be in subr.el.
@@ -2021,34 +2087,48 @@ completion-hilit-commonality
 It returns a list with font-lock properties applied to each element,
 and with BASE-SIZE appended as the last element."
   (when completions
-    (let ((com-str-len (- prefix-len (or base-size 0))))
-      (nconc
-       (mapcar
-        (lambda (elem)
-          (let ((str
-                 ;; Don't modify the string itself, but a copy, since the
-                 ;; the string may be read-only or used for other purposes.
-                 ;; Furthermore, since `completions' may come from
-                 ;; display-completion-list, `elem' may be a list.
-                 (if (consp elem)
-                     (car (setq elem (cons (copy-sequence (car elem))
-                                           (cdr elem))))
-                   (setq elem (copy-sequence elem)))))
-            (font-lock-prepend-text-property
-             0
-             ;; If completion-boundaries returns incorrect
-             ;; values, all-completions may return strings
-             ;; that don't contain the prefix.
-             (min com-str-len (length str))
-             'face 'completions-common-part str)
-            (if (> (length str) com-str-len)
-                (font-lock-prepend-text-property com-str-len (1+ com-str-len)
-                                                 'face
-                                                 'completions-first-difference
-                                                 str)))
-          elem)
-        completions)
-       base-size))))
+    (nconc
+     (completion--hilit-commonality (- prefix-len (or base-size 0)) completions)
+     base-size)))
+
+(defun completion--hilit-commonality (com-size completions)
+  (mapcar
+   (lambda (elem)
+     (let ((str
+            ;; Don't modify the string itself, but a copy, since the
+            ;; the string may be read-only or used for other purposes.
+            ;; Furthermore, since `completions' may come from
+            ;; display-completion-list, `elem' may be a list.
+            (if (consp elem)
+                (car (setq elem (cons (copy-sequence (car elem))
+                                      (cdr elem))))
+              (setq elem (copy-sequence elem)))))
+       (font-lock-prepend-text-property
+        0
+        ;; If completion-boundaries returns incorrect
+        ;; values, all-completions may return strings
+        ;; that don't contain the prefix.
+        (min com-size (length str))
+        'face 'completions-common-part str)
+       (if (> (length str) com-size)
+           (font-lock-prepend-text-property com-size (1+ com-size)
+                                            'face
+                                            'completions-first-difference
+                                            str)))
+     elem)
+   completions))
+
+(defun completion--deferred-hilit (completions prefix-len base end)
+  "Return completions in old format or new alist format.
+If `completion--filter-completions' is non-nil use the new format."
+  (if completion--filter-completions
+      (when completions
+        `((base . ,base)
+          (end . ,end)
+          (highlight . ,(apply-partially #'completion--hilit-commonality
+                                         (- prefix-len base)))
+          (completions . ,completions)))
+    (completion-hilit-commonality completions prefix-len base)))
 
 (defun display-completion-list (completions &optional common-substring group-fun)
   "Display the list of completions, COMPLETIONS, using `standard-output'.
@@ -2163,15 +2243,16 @@ minibuffer-completion-help
          (end (or end (point-max)))
          (string (buffer-substring start end))
          (md (completion--field-metadata start))
-         (completions (completion-all-completions
-                       string
-                       minibuffer-completion-table
-                       minibuffer-completion-predicate
-                       (- (point) start)
-                       md)))
+         (filtered-completions (completion-filter-completions
+                                string
+                                minibuffer-completion-table
+                                minibuffer-completion-predicate
+                                (- (point) start)
+                                md))
+         (completions (alist-get 'completions filtered-completions)))
     (message nil)
     (if (or (null completions)
-            (and (not (consp (cdr completions)))
+            (and (not (cdr completions))
                  (equal (car completions) string)))
         (progn
           ;; If there are no completions, or if the current input is already
@@ -2181,8 +2262,7 @@ minibuffer-completion-help
           (completion--message
            (if completions "Sole completion" "No completions")))
 
-      (let* ((last (last completions))
-             (base-size (or (cdr last) 0))
+      (let* ((base-size (alist-get 'base filtered-completions))
              (prefix (unless (zerop base-size) (substring string 0 base-size)))
              (all-md (completion--metadata (buffer-substring-no-properties
                                             start (point))
@@ -2226,9 +2306,10 @@ minibuffer-completion-help
             (body-function
              . ,#'(lambda (_window)
                     (with-current-buffer mainbuf
-                      ;; Remove the base-size tail because `sort' requires a properly
-                      ;; nil-terminated list.
-                      (when last (setcdr last nil))
+                      ;; Apply highlighting
+                      (setq completions
+                            (funcall (alist-get 'highlight filtered-completions)
+                                     completions))
 
                       ;; Sort first using the `display-sort-function'.
                       ;; FIXME: This function is for the output of
@@ -2267,13 +2348,10 @@ minibuffer-completion-help
                                       completions))))
 
                       (with-current-buffer standard-output
-                        (setq-local completion-base-position
-                             (list (+ start base-size)
-                                   ;; FIXME: We should pay attention to completion
-                                   ;; boundaries here, but currently
-                                   ;; completion-all-completions does not give us the
-                                   ;; necessary information.
-                                   end))
+                        (setq-local
+                         completion-base-position
+                         (list (+ start base-size)
+                               (+ start (alist-get 'end filtered-completions))))
                         (setq-local completion-list-insert-choice-function
                              (let ((ctable minibuffer-completion-table)
                                    (cpred minibuffer-completion-predicate)
@@ -3223,10 +3301,11 @@ completion-emacs21-try-completion
       completion)))
 
 (defun completion-emacs21-all-completions (string table pred _point)
-  (completion-hilit-commonality
+  (completion--deferred-hilit
    (all-completions string table pred)
    (length string)
-   (car (completion-boundaries string table pred ""))))
+   (car (completion-boundaries string table pred ""))
+   (length string)))
 
 (defun completion-emacs22-try-completion (string table pred point)
   (let ((suffix (substring string point))
@@ -3249,11 +3328,12 @@ completion-emacs22-try-completion
       (cons (concat completion suffix) (length completion)))))
 
 (defun completion-emacs22-all-completions (string table pred point)
-  (let ((beforepoint (substring string 0 point)))
-    (completion-hilit-commonality
+  (let* ((beforepoint (substring string 0 point))
+         (afterpoint (substring string point))
+         (bounds (completion-boundaries beforepoint table pred afterpoint)))
+    (completion--deferred-hilit
      (all-completions beforepoint table pred)
-     point
-     (car (completion-boundaries beforepoint table pred "")))))
+     point (car bounds) (+ point (cdr bounds)))))
 
 ;;; Basic completion.
 
@@ -3312,7 +3392,7 @@ completion-basic-all-completions
                             'point
                             (substring afterpoint 0 (cdr bounds)))))
          (all (completion-pcm--all-completions prefix pattern table pred)))
-    (completion-hilit-commonality all point (car bounds))))
+    (completion--deferred-hilit all point (car bounds) (+ point (cdr bounds)))))
 
 ;;; Partial-completion-mode style completion.
 
@@ -3504,13 +3584,25 @@ flex-score-match-tightness
 than the latter (which has two \"holes\" and three
 one-letter-long matches).")
 
+(defun completion-pcm--deferred-hilit (pattern completions base end)
+  "Return completions in old format or new alist format.
+If `completion--filter-completions' is non-nil use the new format."
+  (when completions
+    (if completion--filter-completions
+        `((base . ,base)
+          (end . ,end)
+          (highlight . ,(apply-partially
+                         #'completion-pcm--hilit-commonality
+                         pattern))
+          (completions . ,completions))
+      (nconc (completion-pcm--hilit-commonality pattern completions) base))))
+
 (defun completion-pcm--hilit-commonality (pattern completions)
   "Show where and how well PATTERN matches COMPLETIONS.
 PATTERN, a list of symbols and strings as seen
 `completion-pcm--merge-completions', is assumed to match every
 string in COMPLETIONS.  Return a deep copy of COMPLETIONS where
-each string is propertized with `completion-score', a number
-between 0 and 1, and with faces `completions-common-part',
+each string is propertized with faces `completions-common-part',
 `completions-first-difference' in the relevant segments."
   (when completions
     (let* ((re (completion-pcm--pattern->regex pattern 'group))
@@ -3525,6 +3617,54 @@ completion-pcm--hilit-commonality
            (error "Internal error: %s does not match %s" re str))
          (let* ((pos (if point-idx (match-beginning point-idx) (match-end 0)))
                 (match-end (match-end 0))
+                (md (cddr (setq last-md (match-data t last-md))))
+                (from 0))
+           (while md
+             (add-face-text-property from (pop md)
+                                     'completions-common-part
+                                     nil str)
+             (setq from (pop md)))
+           ;; If `pattern' doesn't have an explicit trailing any, the
+           ;; regex `re' won't produce match data representing the
+           ;; region after the match.  We need to account to account
+           ;; for that extra bit of match (bug#42149).
+           (unless (= from match-end)
+             (add-face-text-property from match-end
+                                     'completions-common-part
+                                     nil str))
+           (if (> (length str) pos)
+               (add-face-text-property
+                pos (1+ pos)
+                'completions-first-difference
+                nil str)))
+         str)
+       completions))))
+
+(defun completion--flex-score (pattern completions)
+  "Compute how well PATTERN matches COMPLETIONS.
+PATTERN, a list of strings is assumed to match every string in
+COMPLETIONS.  Return a copy of COMPLETIONS where each element is
+a pair of a score and the completion string.  The score lies in
+the range between -1 and 0, where -1 corresponds to the full
+match."
+  (when completions
+    (let* ((re (completion-pcm--pattern->regex pattern 'group))
+           (case-fold-search completion-ignore-case)
+           last-md)
+      (mapcar
+       (lambda (str)
+         ;; The flex completion style requires the completion to match
+         ;; the pattern to compute the scoring.  For quoted completion
+         ;; tables the completions are matched against the *unquoted
+         ;; input string*.  However `completion-all-completions' and
+         ;; `completion-filter-completions' return a list of *quoted
+         ;; completions*, which is subsequently sorted.  Therefore we
+         ;; obtain the unquoted completion string which is stored in
+         ;; the text property `completion--unquoted'.
+         (setq str (or (get-text-property 0 'completion--unquoted str) str))
+         (unless (string-match re str)
+           (error "Internal error: %s does not match %s" re str))
+         (let* ((match-end (match-end 0))
                 (md (cddr (setq last-md (match-data t last-md))))
                 (from 0)
                 (end (length str))
@@ -3564,13 +3704,10 @@ completion-pcm--hilit-commonality
                 ;; , where "len" is the string's length.
                 (score-numerator 0)
                 (score-denominator 0)
-                (last-b 0)
-                (update-score-and-face
-                 (lambda (a b)
-                   "Update score and face given match range (A B)."
-                   (add-face-text-property a b
-                                           'completions-common-part
-                                           nil str)
+                (last-b 0))
+           (while md
+             (let ((a from)
+                   (b (pop md)))
                    (setq
                     score-numerator   (+ score-numerator (- b a)))
                    (unless (or (= a last-b)
@@ -3583,26 +3720,29 @@ completion-pcm--hilit-commonality
                                                  (/ 1.0
                                                     flex-score-match-tightness)))))
                    (setq
-                    last-b              b))))
-           (while md
-             (funcall update-score-and-face from (pop md))
+                    last-b              b))
              (setq from (pop md)))
            ;; If `pattern' doesn't have an explicit trailing any, the
            ;; regex `re' won't produce match data representing the
            ;; region after the match.  We need to account to account
            ;; for that extra bit of match (bug#42149).
            (unless (= from match-end)
-             (funcall update-score-and-face from match-end))
-           (if (> (length str) pos)
-               (add-face-text-property
-                pos (1+ pos)
-                'completions-first-difference
-                nil str))
-           (unless (zerop (length str))
-             (put-text-property
-              0 1 'completion-score
-              (/ score-numerator (* end (1+ score-denominator)) 1.0) str)))
-         str)
+             (let ((a from)
+                   (b match-end))
+                   (setq
+                    score-numerator   (+ score-numerator (- b a)))
+                   (unless (or (= a last-b)
+                               (zerop last-b)
+                               (= a (length str)))
+                     (setq
+                      score-denominator (+ score-denominator
+                                           1
+                                           (expt (- a last-b 1)
+                                                 (/ 1.0
+                                                    flex-score-match-tightness)))))
+                   (setq
+                    last-b              b)))
+           (cons (- (/ score-numerator (* end (1+ score-denominator)) 1.0)) str)))
        completions))))
 
 (defun completion-pcm--find-all-completions (string table pred point
@@ -3700,11 +3840,11 @@ completion-pcm--find-all-completions
         (list pattern all prefix suffix)))))
 
 (defun completion-pcm-all-completions (string table pred point)
-  (pcase-let ((`(,pattern ,all ,prefix ,_suffix)
+  (pcase-let ((`(,pattern ,all ,prefix ,suffix)
                (completion-pcm--find-all-completions string table pred point)))
-    (when all
-      (nconc (completion-pcm--hilit-commonality pattern all)
-             (length prefix)))))
+    (completion-pcm--deferred-hilit pattern all
+                                    (length prefix)
+                                    (- (length string) (length suffix)))))
 
 (defun completion--common-suffix (strs)
   "Return the common suffix of the strings STRS."
@@ -3885,8 +4025,8 @@ completion-pcm-try-completion
 ;;; Substring completion
 ;; Mostly derived from the code of `basic' completion.
 
-(defun completion-substring--all-completions
-    (string table pred point &optional transform-pattern-fn)
+(defun completion--pattern-compiler
+    (string table pred point transform-pattern-fn)
   "Match the presumed substring STRING to the entries in TABLE.
 Respect PRED and POINT.  The pattern used is a PCM-style
 substring pattern, but it be massaged by TRANSFORM-PATTERN-FN, if
@@ -3904,12 +4044,23 @@ completion-substring--all-completions
          (pattern (completion-pcm--optimize-pattern
                    (if transform-pattern-fn
                        (funcall transform-pattern-fn pattern)
-                     pattern)))
-         (all (completion-pcm--all-completions prefix pattern table pred)))
-    (list all pattern prefix suffix (car bounds))))
+                     pattern))))
+    (list pattern prefix suffix)))
+
+(defun completion-substring--all-completions
+    (string table pred point &optional transform-pattern-fn)
+  "Match the presumed substring STRING to the entries in TABLE.
+Respect PRED and POINT.  The pattern used is a PCM-style
+substring pattern, but it be massaged by TRANSFORM-PATTERN-FN, if
+that is non-nil."
+  (pcase-let (((and result `(,pattern ,prefix ,_suffix))
+               (completion--pattern-compiler string table pred point
+                                             transform-pattern-fn)))
+    (cons (completion-pcm--all-completions prefix pattern table pred)
+          result)))
 
 (defun completion-substring-try-completion (string table pred point)
-  (pcase-let ((`(,all ,pattern ,prefix ,suffix ,_carbounds)
+  (pcase-let ((`(,all ,pattern ,prefix ,suffix)
                (completion-substring--all-completions
                 string table pred point)))
     (if minibuffer-completing-file-name
@@ -3917,12 +4068,12 @@ completion-substring-try-completion
     (completion-pcm--merge-try pattern all prefix suffix)))
 
 (defun completion-substring-all-completions (string table pred point)
-  (pcase-let ((`(,all ,pattern ,prefix ,_suffix ,_carbounds)
+  (pcase-let ((`(,all ,pattern ,prefix ,suffix)
                (completion-substring--all-completions
                 string table pred point)))
-    (when all
-      (nconc (completion-pcm--hilit-commonality pattern all)
-             (length prefix)))))
+    (completion-pcm--deferred-hilit pattern all
+                                    (length prefix)
+                                    (- (length string) (length suffix)))))
 
 ;;; "flex" completion, also known as flx/fuzzy/scatter completion
 ;; Completes "foo" to "frodo" and "farfromsober"
@@ -3932,42 +4083,40 @@ completion-flex-nospace
   :version "27.1"
   :type 'boolean)
 
-(put 'flex 'completion--adjust-metadata 'completion--flex-adjust-metadata)
-
-(defun completion--flex-adjust-metadata (metadata)
-  (cl-flet
-      ((compose-flex-sort-fn
-        (existing-sort-fn) ; wish `cl-flet' had proper indentation...
-        (lambda (completions)
-          (let ((pre-sorted
-                 (if existing-sort-fn
-                     (funcall existing-sort-fn completions)
-                   completions)))
-            (cond
-             ((or (not (window-minibuffer-p))
-                  ;; JT@2019-12-23: FIXME: this is still wrong.  What
-                  ;; we need to test here is "some input that actually
-                  ;; leads to flex filtering", not "something after
-                  ;; the minibuffer prompt".  Among other
-                  ;; inconsistencies, the latter is always true for
-                  ;; file searches, meaning the next clauses will be
-                  ;; ignored.
-                  (> (point-max) (minibuffer-prompt-end)))
-              (sort
-               pre-sorted
-               (lambda (c1 c2)
-                 (let ((s1 (get-text-property 0 'completion-score c1))
-                       (s2 (get-text-property 0 'completion-score c2)))
-                   (> (or s1 0) (or s2 0))))))
-             (t pre-sorted))))))
-    `(metadata
-      (display-sort-function
-       . ,(compose-flex-sort-fn
-           (completion-metadata-get metadata 'display-sort-function)))
-      (cycle-sort-function
-       . ,(compose-flex-sort-fn
-           (completion-metadata-get metadata 'cycle-sort-function)))
-      ,@(cdr metadata))))
+(put 'flex 'completion--style-metadata 'completion--flex-style-metadata)
+
+(defun completion--flex-style-metadata (string table pred point metadata)
+  ;; Use the modified flex sorting function only for non-empty input.
+  ;; In an older version of `completion--flex-adjust-metadata', the
+  ;; check (> (point-max) (minibuffer-prompt-end))) was used instead.
+  (unless (eq string "")
+    (let ((pattern (car (completion--pattern-compiler
+                         string table pred point
+                         #'completion-flex--make-flex-pattern))))
+      (cl-flet
+          ((compose-flex-sort-fn
+            (existing-sort-fn) ; wish `cl-flet' had proper indentation...
+            (lambda (completions)
+              (let* ((sorted (sort (completion--flex-score
+                                    pattern
+                                    (if existing-sort-fn
+                                        (funcall existing-sort-fn completions)
+                                      completions))
+                                   #'car-less-than-car))
+                     (cell sorted))
+                ;; Remove score decorations, reuse the list to avoid allocations.
+                (while cell
+                  (setcar cell (cdar cell))
+                  (pop cell))
+                sorted))))
+        `(metadata
+          (display-sort-function
+           . ,(compose-flex-sort-fn
+               (completion-metadata-get metadata 'display-sort-function)))
+          (cycle-sort-function
+           . ,(compose-flex-sort-fn
+               (completion-metadata-get metadata 'cycle-sort-function)))
+          ,@(cdr metadata))))))
 
 (defun completion-flex--make-flex-pattern (pattern)
   "Convert PCM-style PATTERN into PCM-style flex pattern.
@@ -3989,7 +4138,7 @@ completion-flex--make-flex-pattern
 (defun completion-flex-try-completion (string table pred point)
   "Try to flex-complete STRING in TABLE given PRED and POINT."
   (unless (and completion-flex-nospace (string-search " " string))
-    (pcase-let ((`(,all ,pattern ,prefix ,suffix ,_carbounds)
+    (pcase-let ((`(,all ,pattern ,prefix ,suffix)
                  (completion-substring--all-completions
                   string table pred point
                   #'completion-flex--make-flex-pattern)))
@@ -4006,13 +4155,13 @@ completion-flex-try-completion
 (defun completion-flex-all-completions (string table pred point)
   "Get flex-completions of STRING in TABLE, given PRED and POINT."
   (unless (and completion-flex-nospace (string-search " " string))
-    (pcase-let ((`(,all ,pattern ,prefix ,_suffix ,_carbounds)
+    (pcase-let ((`(,all ,pattern ,prefix ,suffix)
                  (completion-substring--all-completions
                   string table pred point
                   #'completion-flex--make-flex-pattern)))
-      (when all
-        (nconc (completion-pcm--hilit-commonality pattern all)
-               (length prefix))))))
+      (completion-pcm--deferred-hilit pattern all
+                                    (length prefix)
+                                    (- (length string) (length suffix))))))
 
 ;; Initials completion
 ;; Complete /ums to /usr/monnier/src or lch to list-command-history.
@@ -4049,7 +4198,11 @@ completion-initials-expand
 (defun completion-initials-all-completions (string table pred _point)
   (let ((newstr (completion-initials-expand string table pred)))
     (when newstr
-      (completion-pcm-all-completions newstr table pred (length newstr)))))
+      (pcase-let ((`(,pattern ,all ,prefix ,_suffix)
+                   (completion-pcm--find-all-completions newstr table
+                                                         pred (length newstr))))
+        (completion-pcm--deferred-hilit pattern all
+                                        (length prefix) (length string))))))
 
 (defun completion-initials-try-completion (string table pred _point)
   (let ((newstr (completion-initials-expand string table pred)))
diff --git a/test/lisp/minibuffer-tests.el b/test/lisp/minibuffer-tests.el
index c3ba8f9a92..4ebf27fd1d 100644
--- a/test/lisp/minibuffer-tests.el
+++ b/test/lisp/minibuffer-tests.el
@@ -188,10 +188,6 @@ completion-all-sorted-completions
                   '("some/alpha" "base/epsilon" "base/delta"))
                  `("epsilon" "delta" "beta" "alpha" "gamma"  . 5))))
 
-(defun completion--pcm-score (comp)
-  "Get `completion-score' from COMP."
-  (get-text-property 0 'completion-score comp))
-
 (defun completion--pcm-first-difference-pos (comp)
   "Get `completions-first-difference' from COMP."
   (cl-loop for pos = (next-single-property-change 0 'face comp)
@@ -215,25 +211,12 @@ completion-pcm-test-2
            "barfoobar")))
 
 (ert-deftest completion-pcm-test-3 ()
-  ;; Full match!
-  (should (eql
-           (completion--pcm-score
-            (car (completion-pcm-all-completions
-                  "R" '("R" "hello") nil 1)))
-           1.0)))
-
-(ert-deftest completion-pcm-test-4 ()
-  ;; One fourth of a match and no match due to point being at the end
-  (should (eql
-           (completion--pcm-score
-            (car (completion-pcm-all-completions
-                  "RO" '("RaOb") nil 1)))
-           (/ 1.0 4.0)))
+  ;; No match due to point being at the end
   (should (null
            (completion-pcm-all-completions
             "RO" '("RaOb") nil 2))))
 
-(ert-deftest completion-pcm-test-5 ()
+(ert-deftest completion-pcm-test-4 ()
   ;; Since point is at the beginning, there is nothing that can really
   ;; be typed anymore
   (should (null
@@ -241,7 +224,7 @@ completion-pcm-test-5
             (car (completion-pcm-all-completions
                   "f" '("few" "many") nil 0))))))
 
-(ert-deftest completion-pcm-test-6 ()
+(ert-deftest completion-pcm-test-5 ()
   ;; Wildcards and delimiters work
   (should (equal
            (car (completion-pcm-all-completions
@@ -252,26 +235,12 @@ completion-pcm-test-6
                  "li-pac*" '("do-not-list-packages") nil 7)))))
 
 (ert-deftest completion-substring-test-1 ()
-  ;; One third of a match!
   (should (equal
            (car (completion-substring-all-completions
                  "foo" '("hello" "world" "barfoobar") nil 3))
-           "barfoobar"))
-  (should (eql
-           (completion--pcm-score
-            (car (completion-substring-all-completions
-                  "foo" '("hello" "world" "barfoobar") nil 3)))
-           (/ 1.0 3.0))))
+           "barfoobar")))
 
 (ert-deftest completion-substring-test-2 ()
-  ;; Full match!
-  (should (eql
-           (completion--pcm-score
-            (car (completion-substring-all-completions
-                  "R" '("R" "hello") nil 1)))
-           1.0)))
-
-(ert-deftest completion-substring-test-3 ()
   ;; Substring match
   (should (equal
            (car (completion-substring-all-completions
@@ -281,7 +250,7 @@ completion-substring-test-3
            (car (completion-substring-all-completions
                  "custgroup" '("customize-group") nil 5)))))
 
-(ert-deftest completion-substring-test-4 ()
+(ert-deftest completion-substring-test-3 ()
   ;; `completions-first-difference' should be at the right place
   (should (eql
            (completion--pcm-first-difference-pos
@@ -306,14 +275,6 @@ completion-flex-test-1
            "fabrobazo")))
 
 (ert-deftest completion-flex-test-2 ()
-  ;; Full match!
-  (should (eql
-           (completion--pcm-score
-            (car (completion-flex-all-completions
-                  "R" '("R" "hello") nil 1)))
-           1.0)))
-
-(ert-deftest completion-flex-test-3 ()
   ;; Another fuzzy match, but more of a "substring" one
   (should (equal
            (car (completion-flex-all-completions
@@ -331,5 +292,213 @@ completion-flex-test-3
                   "custgroup" '("customize-group-other-window") nil 9)))
            15)))
 
+(ert-deftest completion-flex-score-test-1 ()
+  ;; Full match!
+  (should (equal
+           (completion--flex-score '(prefix "R") '("R"))
+           (list (cons -1.0 "R")))))
+
+(ert-deftest completion-flex-score-test-2 ()
+  ;; One third and half of a match!
+  (should (equal
+           (completion--flex-score '(prefix "foo")
+                                   '("barfoobar" "fooboo"))
+           (list (cons (/ -1.0 3.0) "barfoobar")
+                 (cons (/ -1.0 2.0) "fooboo")))))
+
+(ert-deftest completion-flex-score-test-3 ()
+  ;; One fourth of a match
+  (should (eql
+           (caar (completion--flex-score '(prefix "R" point "O")
+                                         '("RaOb")))
+           (/ -1.0 4.0))))
+
+(ert-deftest completion-flex-score-test-4 ()
+  ;; For quoted completion tables, score the unquoted completion string.
+  (should (equal
+           (completion--flex-score
+            '(prefix "R")
+            (list (propertize "X" 'completion--unquoted "R")))
+           (list (cons -1.0 "R")))))
+
+(defun completion--test-style (style string point table filtered)
+  (let* ((completion-styles (list style))
+         (pred (lambda (x) (not (string-search "!" x))))
+         (result (completion-filter-completions
+                  string table pred point nil)))
+    (should (equal (alist-get 'base result) 0))
+    (should (equal (alist-get 'end result) (length string)))
+    (should (equal (alist-get 'completions result) filtered))
+    (should (not (memq (alist-get 'highlight result) '(nil identity))))
+    (should (equal (completion-all-completions string table pred point)
+                   (append filtered 0)))))
+
+(ert-deftest completion-basic-style-test-1 ()
+  ;; point at the beginning |foo
+  (completion--test-style 'basic "foo" 0
+                          '("foobar" "foo!" "barfoo" "xfooy" "boobar")
+                          '("foobar" "barfoo" "xfooy")))
+
+(ert-deftest completion-basic-style-test-2 ()
+  ;; point foo
+  (completion--test-style 'basic "foo" 2
+                          '("foobar" "foo!" "fobar" "barfoo" "xfooy" "boobar")
+                          '("foobar")))
+
+(ert-deftest completion-substring-style-test ()
+  (completion--test-style 'substring "foo" 1
+                          '("foobar" "foo!" "barfoo" "xfooy" "boobar")
+                          '("foobar" "barfoo" "xfooy")))
+
+(ert-deftest completion-emacs21-style-test ()
+  (completion--test-style 'emacs21 "foo" 1
+                          '("foobar" "foo!" "fobar" "barfoo" "xfooy" "boobar")
+                          '("foobar")))
+
+(ert-deftest completion-emacs22-style-test ()
+  (completion--test-style 'emacs22 "fo0" 1
+                          '("foobar" "foo!" "fobar" "barfoo" "xfooy" "boobar")
+                          '("foobar" "fobar"))) ;; suffix ignored completely
+
+(ert-deftest completion-flex-style-test ()
+  (completion--test-style 'flex "abc" 1
+                          '("abc" "abc!" "xaybzc" "xaybz")
+                          '("abc" "xaybzc")))
+
+(ert-deftest completion-initials-style-test ()
+  (completion--test-style 'initials "abc" 1
+                          '("a-b-c" "a-b-c!" "ax-by-cz" "xax-by-cz")
+                          '("a-b-c" "ax-by-cz")))
+
+(ert-deftest completion-pcm-style-test ()
+  (completion--test-style 'partial-completion "ax-b-c" 1
+                          '("ax-b-c" "ax-b-c!" "ax-by-cz" "xax-by-cz")
+                          '("ax-b-c" "ax-by-cz")))
+
+(ert-deftest completion-filter-completions-highlight-test ()
+  ;; point at the beginning |foo
+  (let* ((completion-styles '(basic))
+         (result (completion-filter-completions
+                  "foo" '("foobar" "fbarfoo" "fxfooy" "bar")
+                  nil 1 nil)))
+    (should (equal
+             (format "%S" (alist-get 'completions result))
+             (format "%S" '("foobar" "fbarfoo" "fxfooy"))))
+    (should (equal
+             (format "%S" (funcall (alist-get 'highlight result)
+                                   (alist-get 'completions result)))
+             (format "%S"
+                     '(#("foobar" 0 1 (face (completions-common-part))
+                         1 2 (face (completions-first-difference)))
+                       #("fbarfoo" 0 1 (face (completions-common-part))
+                         1 2 (face (completions-first-difference)))
+                       #("fxfooy" 0 1 (face (completions-common-part))
+                         1 2 (face (completions-first-difference)))))))))
+
+(defun completion--test-boundaries (style string table result)
+  (let ((table
+         (lambda (str pred action)
+           (pcase action
+             (`(boundaries . ,suffix) `(boundaries
+                                        ,(1+ (string-match-p "<\\|/" str))
+                                        . ,(or (string-search ">" suffix) (length suffix))))
+             (_ (complete-with-action action table
+                                      (replace-regexp-in-string ".*[</]" "" str)
+                                      pred)))))
+        (point (string-search "|" string))
+        (string (string-replace "|" "" string))
+        (completion-styles (list style)))
+    (should (equal
+             (assq-delete-all
+              (if (assq 'highlight result) '-does-not-exist 'highlight)
+              (completion-filter-completions
+               string table nil point nil))
+             result))
+    (should (equal
+             (completion-all-completions
+              string table nil point)
+             (append (alist-get 'completions result)
+                     (alist-get 'base result))))))
+
+(ert-deftest completion-emacs21-boundaries-test ()
+  (completion--test-boundaries 'emacs21 "before<in|put>after"
+                               '("other") nil)
+  (completion--test-boundaries 'emacs21 "before<in|put>after"
+                               '("ainput>after" "input>after" "inpux>after"
+                                 "inxputy>after" "input>after2")
+                               '((base . 7)
+                                 (end . 18)
+                                 (completions "input>after" "input>after2"))))
+
+(ert-deftest completion-emacs22-boundaries-test ()
+  (completion--test-boundaries 'emacs22 "before<in|put>after"
+                               '("other") nil)
+  (completion--test-boundaries 'emacs22 "before<in|put>after"
+                               '("ainxxx" "inyy" "inzzz")
+                               '((base . 7)
+                                 (end . 12)
+                                 (completions "inyy" "inzzz"))))
+
+(ert-deftest completion-basic-boundaries-test ()
+  (completion--test-boundaries 'basic "before<in|put>after"
+                               '("other") nil)
+  (completion--test-boundaries 'basic "before<in|put>after"
+                               '("ainput" "input" "inpux" "inxputy")
+                               '((base . 7)
+                                 (end . 12)
+                                 (completions "input" "inxputy"))))
+
+(ert-deftest completion-substring-boundaries-test ()
+  (completion--test-boundaries 'substring "before<in|puts>after"
+                               '("other") nil)
+  (completion--test-boundaries 'substring "before<in|puts>after"
+                               '("ainputs" "inputs" "inpux" "inxputsy")
+                               '((base . 7)
+                                 (end . 13)
+                                 (completions "ainputs" "inputs" "inxputsy"))))
+
+(ert-deftest completion-pcm-boundaries-test ()
+  (completion--test-boundaries 'partial-completion "before<in-p|t>after"
+                               '("other") nil)
+  (completion--test-boundaries 'partial-completion "before<in-p|t>after"
+                               '("ain-pu-ts" "in-pts" "in-pu-ts" "in-px" "inx-ptsy")
+                               '((base . 7)
+                                 (end . 12)
+                                 (completions "in-pts" "in-pu-ts" "inx-ptsy"))))
+
+(ert-deftest completion-initials-boundaries-test ()
+  (completion--test-boundaries 'initials "/ip|t"
+                               '("other") nil)
+  (completion--test-boundaries 'initials "/ip|t"
+                               '("ain/pu/ts" "in/pts" "in/pu/ts" "a/in/pu/ts"
+                                 "in/pu/ts/foo" "in/px" "inx/ptsy")
+                               '((base . 1)
+                                 (end . 4)
+                                 (completions "in/pu/ts" "in/pu/ts/foo"))))
+
+(defun completion-emacs22orig-all-completions (string table pred point)
+  (let ((beforepoint (substring string 0 point)))
+    (completion-hilit-commonality
+      (all-completions beforepoint table pred)
+     point
+     (car (completion-boundaries beforepoint table pred "")))))
+
+(ert-deftest completion-upgrade-return-type-test ()
+  ;; Test transparent upgrade of old completion style return value
+  ;; to new return value format.
+  (let ((completion-styles-alist
+         '((emacs22orig completion-emacs22-try-completion
+                        completion-emacs22orig-all-completions nil))))
+  (completion--test-boundaries 'emacs22orig "before<in|put>after"
+                               '("ainxxx" "inyy" "inzzz")
+                               '((base . 7)
+                                 ;; 18 is incorrect, should be 12!
+                                 ;; But the information is not available
+                                 ;; due to the completion-style upgrade.
+                                 (end . 18)
+                                 ;; Identity highlighting function.
+                                 (highlight . identity)
+                                 (completions "inyy" "inzzz")))))
+
 (provide 'minibuffer-tests)
 ;;; minibuffer-tests.el ends here
-- 
2.20.1


--------------3147CDE26CD3C1A11006A179--




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 7 Jul 2021 08:56:17 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Wed Jul 07 04:56:17 2021
Received: from localhost ([127.0.0.1]:50828 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1m13Ls-0003eE-Ph
	for submit <at> debbugs.gnu.org; Wed, 07 Jul 2021 04:56:16 -0400
Received: from server.qxqx.de ([178.63.65.180]:47469 helo=mail.qxqx.de)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mail@HIDDEN>)
 id 1m13Lq-0003dw-Aq; Wed, 07 Jul 2021 04:56:15 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=qxqx.de;
 s=mail1392553390; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:
 MIME-Version:Date:Message-ID:From:References:Cc:To:Subject:Sender:Reply-To:
 Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender:
 Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:
 List-Subscribe:List-Post:List-Owner:List-Archive;
 bh=o3u4a1/ZU7W8pFLoHziFDm4XIG8WTn7yTgWzxLbAWns=; b=gK3nxdsmphgKTeETBcWmUA4Qo4
 RXioGPDTTt2+6BBibZiNXl6TkIjZzWODFkUe4nCHqnjBNe/RTTOf4gv21nCSPj+CVPUOx3IDvkJd/
 MRQUXLGIzxJLD9wS5GLJQI1sU87P9dSvztcdan9oTMWD+M8HvgMeDm+MEwY3McP+gD5s=;
Subject: Re: bug#48841: fido-mode is slower than ido-mode with similar settings
To: Dmitry Gutov <dgutov@HIDDEN>, =?UTF-8?B?Sm/Do28gVMOhdm9yYQ==?=
 <joaotavora@HIDDEN>
References: <f94541d3-5f93-9a23-8256-3bbc453f127a@HIDDEN>
 <87eedgy7pt.fsf@HIDDEN> <1f659c88-4d9d-8fc9-733a-5e6068f9ed4a@HIDDEN>
 <87a6o3x5j7.fsf@HIDDEN> <dd69952b-bb7b-7e86-466c-c2117f839cc4@HIDDEN>
 <87y2bnv5xc.fsf@HIDDEN> <35be6652-9c8d-ee21-e9eb-9598ad6777eb@HIDDEN>
 <CALDnm510An-+2b31oT_TE0akNjU_KWA7iv9RjN1Hr-KXSZ68sw@HIDDEN>
 <d57185f3-c28a-8c48-a50c-0b4b8bdb95c8@HIDDEN>
 <CALDnm51En25oyL8i+g-_oxBtCgmts0skjqwNE8hBn66k_DyV_g@HIDDEN>
 <858682b2-b8fd-898b-bef3-97dbe5e4debc@HIDDEN> <87mtrwuy4v.fsf@HIDDEN>
 <2234991b-c2e0-81e3-c1ef-b1d94d35a728@HIDDEN> <87v96hu845.fsf@HIDDEN>
 <310ab8d8-2bba-33bb-1aa4-1dc88dcb57d8@HIDDEN> <877disb30s.fsf@HIDDEN>
 <526eeb14-31c2-f414-ec44-192180d59164@HIDDEN>
From: Daniel Mendler <mail@HIDDEN>
Message-ID: <7fc8ce6d-20ba-79ed-56d8-f10be72cddc8@HIDDEN>
Date: Wed, 7 Jul 2021 10:56:05 +0200
MIME-Version: 1.0
In-Reply-To: <526eeb14-31c2-f414-ec44-192180d59164@HIDDEN>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 7bit
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 47711
Cc: Stefan Monnier <monnier@HIDDEN>, 48841 <at> debbugs.gnu.org,
 47711 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

On 7/4/21 3:53 AM, Dmitry Gutov wrote:
>> - icomplete.el? for fido-mode & friends
>> - minibuffer.el, for the *Completions* buffer
>> - company.el
>> - Any notable others?
> 
> corfu, consult, etc? Probably Ivy too. All of these are in GNU ELPA.
> 
> BTW, I think Daniel had some ideas about applying the face property 
> lazily as well. I can't find the particular discussion now, but perhaps 
> he can add to this discussion as well.

Yes, Vertico and Corfu apply highlighting lazily. This leads to
significant performance wins. See `vertico--all-completions` in
https://github.com/minad/vertico/blob/main/vertico.el#L243-L279 and
bug#47711.

The technique I am using in Vertico and Corfu retains backward
compatibility, such that the strings are returned unmodified by the
completion style. Highlighting is applied lazily by copying the
candidate strings and mutating the copies. For now I am relying on advices.

One could add an optional argument (or dynamically bound variable) to
completion styles which tell the completion style to opt out of copying
the candidates and the highlighting.

Daniel




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at 47711 <at> debbugs.gnu.org:


Received: (at 47711) by debbugs.gnu.org; 18 Apr 2021 21:26:57 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Apr 18 17:26:57 2021
Received: from localhost ([127.0.0.1]:47979 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1lYEwS-0004GL-Vu
	for submit <at> debbugs.gnu.org; Sun, 18 Apr 2021 17:26:57 -0400
Received: from server.qxqx.de ([178.63.65.180]:43317 helo=mail.qxqx.de)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mail@HIDDEN>) id 1lYEwQ-0004G5-5P
 for 47711 <at> debbugs.gnu.org; Sun, 18 Apr 2021 17:26:55 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=qxqx.de;
 s=mail1392553390; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:
 MIME-Version:Date:Message-ID:From:References:To:Subject:Sender:Reply-To:Cc:
 Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender:
 Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:
 List-Subscribe:List-Post:List-Owner:List-Archive;
 bh=1i52Ca6rZx32peDDFalbTqTolCv2NoCa8/Axw8v2Ch8=; b=G7SLFZWkGEcnOzpaALmVUjqf7t
 F5gysiVZxDsq+foQInGF+P7h4da0m3ArgVg9ZrwPbMGtL0AqCCzcUry92E8Mw51lHou52OxRGi8Wa
 1y0pYvsyFbhh61n5wJ2/pdiBOol0FrDlv+r9HSMoJdaam08/RSpgBK7avk25zyGd21M4=;
Subject: Re: bug#47711: Acknowledgement (27.1; Deferred highlighting support
 in `completion-all-completions', `vertico--all-completions`)
To: 47711 <at> debbugs.gnu.org
References: <3cc8aae8-58fc-abce-728c-090595281da2@HIDDEN>
 <handler.47711.B.16181742862702.ack <at> debbugs.gnu.org>
From: Daniel Mendler <mail@HIDDEN>
Message-ID: <4e53fde0-4c08-73d6-f500-5085ea4ecdfc@HIDDEN>
Date: Sun, 18 Apr 2021 23:26:44 +0200
MIME-Version: 1.0
In-Reply-To: <handler.47711.B.16181742862702.ack <at> debbugs.gnu.org>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 47711
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Deferred highlighting is also useful for completion-at-point, see the 
ELPA Corfu package, `corfu--all-completions`, which uses the same method 
as Vertico.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.

Message received at submit <at> debbugs.gnu.org:


Received: (at submit) by debbugs.gnu.org; 11 Apr 2021 20:51:26 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Apr 11 16:51:26 2021
Received: from localhost ([127.0.0.1]:55936 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1lVh3F-0000hW-O7
	for submit <at> debbugs.gnu.org; Sun, 11 Apr 2021 16:51:26 -0400
Received: from lists.gnu.org ([209.51.188.17]:52524)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mail@HIDDEN>) id 1lVh3D-0000hO-VM
 for submit <at> debbugs.gnu.org; Sun, 11 Apr 2021 16:51:24 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:38496)
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <mail@HIDDEN>)
 id 1lVh3D-0000MJ-NK
 for bug-gnu-emacs@HIDDEN; Sun, 11 Apr 2021 16:51:23 -0400
Received: from server.qxqx.de ([2a01:4f8:121:346::180]:36941 helo=mail.qxqx.de)
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <mail@HIDDEN>)
 id 1lVh3B-0000s7-0p
 for bug-gnu-emacs@HIDDEN; Sun, 11 Apr 2021 16:51:23 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=qxqx.de;
 s=mail1392553390; h=Content-Transfer-Encoding:Content-Type:MIME-Version:Date:
 Message-ID:Subject:From:Cc:To:Sender:Reply-To:Content-ID:Content-Description:
 Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:
 In-Reply-To:References:List-Id:List-Help:List-Unsubscribe:List-Subscribe:
 List-Post:List-Owner:List-Archive;
 bh=phweBwQ12UjBVQH3aBBq3QC32jFVbUVJO67bDBnsOG4=; b=zW5FvyGZ5fSOyOBNNxuk8MRqDS
 tWjdWfWuC1C5+wVxXulc6UsLs0wFuD3gpOCVGMJAlLkL9sYpG5zBpNe7EBMl4WZO77YjKBJ8WKXe0
 1VI1j2goeMO6fl0i6FYYo9LR85K2sUaWc1YfefNiDNrIBzYUKYy9EhrI5tcFG4ZzYEKo=;
To: bug-gnu-emacs@HIDDEN
From: Daniel Mendler <mail@HIDDEN>
Subject: 27.1; Deferred highlighting support in `completion-all-completions',
 `vertico--all-completions`
Message-ID: <3cc8aae8-58fc-abce-728c-090595281da2@HIDDEN>
Date: Sun, 11 Apr 2021 22:51:14 +0200
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Received-SPF: pass client-ip=2a01:4f8:121:346::180;
 envelope-from=mail@HIDDEN; helo=mail.qxqx.de
X-Spam_score_int: -41
X-Spam_score: -4.2
X-Spam_bar: ----
X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,
 DKIM_VALID=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_PASS=-0.001,
 SPF_PASS=-0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-Spam-Score: -1.4 (-)
X-Debbugs-Envelope-To: submit
Cc: Stefan Monnier <monnier@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -2.4 (--)

Emacs is lacking a possibility to defer the completion highlighting when
computing completions via `completion-all-completions'. This feature is
important for the performance of completion UIs when the set of all
completions is much larger than the set of completions which are
displayed.

The Vertico package defers highlighting by modifying the
`completion*-hilit-*' function with advices.

(declare-function orderless-highlight-matches "ext:orderless")
(defun vertico--all-completions (&rest args)
   "Compute all completions for ARGS with deferred highlighting."
   (cl-letf* ((orig-pcm (symbol-function 
#'completion-pcm--hilit-commonality))
              (orig-flex (symbol-function 
#'completion-flex-all-completions))
              ((symbol-function #'completion-flex-all-completions)
               (lambda (&rest args)
                 ;; Unfortunately for flex we have to undo the deferred 
highlighting, since flex uses
                 ;; the completion-score for sorting, which is applied 
during highlighting.
                 (cl-letf (((symbol-function 
#'completion-pcm--hilit-commonality) orig-pcm))
                   (apply orig-flex args))))
              ;; Defer the following highlighting functions
              (hl #'identity)
              ((symbol-function #'completion-hilit-commonality)
               (lambda (cands prefix &optional base)
                 (setq hl (lambda (x) (nconc 
(completion-hilit-commonality x prefix base) nil)))
                 (and cands (nconc cands base))))
              ((symbol-function #'completion-pcm--hilit-commonality)
               (lambda (pattern cands)
                 (setq hl (lambda (x) (completion-pcm--hilit-commonality 
pattern x)))
                 cands))
              ((symbol-function #'orderless-highlight-matches)
               (lambda (pattern cands)
                 (setq hl (lambda (x) (orderless-highlight-matches 
pattern x)))
                 cands)))
     (cons (apply #'completion-all-completions args) hl)))

This function `vertico--all-completions` returns the list of completions 
and a highlighting
function which can then be used to highlight the completions on the fly.
It is a prototype of how some improved functionality in Emacs could look 
like.

(completion-all-completions STRING TABLE PRED POINT &optional METADATA 
DEFER-HL)

or

(completion-all-completions-defer-hl STRING TABLE PRED POINT &optional 
METADATA)

If DEFER-HL=t, then the function returns the completions and a
highlighting function. One may consider returning a triple of base,
completions and highlighting functions. Internally the completion styles
should be adapted such that they support the deferred highlighting.

It could be that this feature becomes less needed with the introduction
of gccemacs in the future.





Acknowledgement sent to Daniel Mendler <mail@HIDDEN>:
New bug report received and forwarded. Copy sent to bug-gnu-emacs@HIDDEN. Full text available.
Report forwarded to bug-gnu-emacs@HIDDEN:
bug#47711; Package emacs. Full text available.
Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.
Last modified: Mon, 16 Aug 2021 13:45:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.