GNU logs - #60690, boring messages

Message sent to bug-grep@HIDDEN:

Subject: bug#60690: [PATCH v2] grep: correctly identify utf-8 characters with \{b,w} in -P
Resent-From: =?UTF-8?Q?=C3=86var_?= =?UTF-8?Q?Arnfj=C3=B6r=C3=B0?= Bjarmason <avarab@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-grep@HIDDEN
Resent-Date: Mon, 09 Jan 2023 12:19:01 +0000
Resent-Message-ID: <handler.60690.B.167326669420778 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: Carlo Marcelo Arenas =?UTF-8?Q?Bel=C3=B3n?= <carenas@HIDDEN>
Cc: demerphq@HIDDEN, pcre-dev@HIDDEN, 60690 <at> debbugs.gnu.org, gitster@HIDDEN, git@HIDDEN
From: =?UTF-8?Q?=C3=86var_?= =?UTF-8?Q?Arnfj=C3=B6r=C3=B0?= Bjarmason <avarab@HIDDEN>
Date: Mon, 09 Jan 2023 12:35:05 +0100
References: <20230108062335.72114-1-carenas@HIDDEN>
 <20230108155217.2817-1-carenas@HIDDEN>
User-agent: Debian GNU/Linux bookworm/sid; Emacs 28.2; mu4e 1.9.0
In-reply-to: <20230108155217.2817-1-carenas@HIDDEN>
Message-ID: <230109.86v8lf297g.gmgdl@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Received-SPF: pass client-ip=2a00:1450:4864:20::42d;
 envelope-from=avarab@HIDDEN; helo=mail-wr1-x42d.google.com
X-Spam_score_int: -20
X-Spam_score: -2.1
X-Spam_bar: --
X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,
 DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001,
 RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001,
 SPF_PASS=-0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-Spam-Score: -1.3 (-)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -2.3 (--)

On Sun, Jan 08 2023, Carlo Marcelo Arenas Bel=C3=B3n wrote:

> When UTF is enabled for a PCRE match, the corresponding flags are
> added to the pcre2_compile() call, but PCRE2_UCP wasn't included.
>
> This prevents extending the meaning of the character classes to
> include those new valid characters and therefore result in failed
> matches for expressions that rely on that extention, for ex:
>
>   $ git grep -P '\b=C3=86var'
>
> Add PCRE2_UCP so that \w will include =C3=86 and therefore \b could
> correctly match the beginning of that word.
>
> This has an impact on performance that has been estimated to be
> between 20% to 40% and that is shown through the added performance
> test.
>
> Signed-off-by: Carlo Marcelo Arenas Bel=C3=B3n <carenas@HIDDEN>
> ---
>  grep.c                              |  2 +-
>  t/perf/p7822-grep-perl-character.sh | 42 +++++++++++++++++++++++++++++
>  2 files changed, 43 insertions(+), 1 deletion(-)
>  create mode 100755 t/perf/p7822-grep-perl-character.sh
>
> diff --git a/grep.c b/grep.c
> index 06eed69493..1687f65b64 100644
> --- a/grep.c
> +++ b/grep.c
> @@ -293,7 +293,7 @@ static void compile_pcre2_pattern(struct grep_pat *p,=
 const struct grep_opt *opt
>  		options |=3D PCRE2_CASELESS;
>  	}
>  	if (!opt->ignore_locale && is_utf8_locale() && !literal)
> -		options |=3D (PCRE2_UTF | PCRE2_MATCH_INVALID_UTF);
> +		options |=3D (PCRE2_UTF | PCRE2_UCP | PCRE2_MATCH_INVALID_UTF);

I have a definite bias towards liking this change, it would help my find
myself :)

But I don't think it's safe to change the default behavior "git-grep",
it's not a mere bug fix, but a major behavior change for existing users
of grep.patternType=3Dperl. E.g. on git.git:
=09
	$ diff <(git -P grep -P '\d+') <(git -P grep -P '(*UCP)\d')
	53360a53361,53362
	> git-gui/po/ja.po:"- =E7=AC=AC=EF=BC=91=E8=A1=8C: =E4=BD=95=E3=82=92=E3=
=81=97=E3=81=9F=E3=81=8B=E3=80=81=E3=82=92=EF=BC=91=E8=A1=8C=E3=81=A7=E8=A6=
=81=E7=B4=84=E3=80=82\n"
	> git-gui/po/ja.po:"- =E7=AC=AC=EF=BC=92=E8=A1=8C: =E7=A9=BA=E7=99=BD\n"

So, it will help "do the right thing" on e.g. "\b=C3=86", but it will also
find e.g. CJK numeric characters for \d etc.

I see per the discussion on
https://github.com/PCRE2Project/pcre2/issues/185 and
https://lists.gnu.org/archive/html/bug-grep/2023-01/threads.html that
you submitted similar fixes to GNU grep & PCRE itself.

I see that GNU grep integrated it a couple of days ago as
https://git.savannah.gnu.org/cgit/grep.git/commit/?id=3D5e3b760f65f13856e57=
17e5b9d935f5b4a615be3

As most discussions about PCRE will eventually devolve into "what does
Perl do?": "Perl" itself will promiscuously use this behavior by
default.

E.g. here the same "=EF=BC=91" character (not the ASCII digit "1") will be
matched from the command-line:

	$ perl -Mre=3Ddebug -CA -wE 'shift =3D~ /\d/' "=EF=BC=91"
	Compiling REx "\d"
	Final program:
	   1: POSIXU[\d] (2)
	   2: END (0)
	stclass POSIXU[\d] minlen 1
	Matching REx "\d" against "%x{ff11}"
	UTF-8 string...
	Matching stclass POSIXU[\d] against "%x{ff11}" (3 bytes)
	   0 <> <%x{ff11}>           |   0| 1:POSIXU[\d](2)
	   3 <%x{ff11}> <>           |   0| 2:END(0)
	Match successful!
	Freeing REx: "\d"

But I don't think it makes sense for "git grep" (or GNU "grep") to
follow Perl in this particular case.

For those not familiar with its Unicode model it doesn't assume by
default that strings are Unicode, they have to be explicitly marked as
such. in the above example I'm declaring that all of "argv" is UTF-8
(via the "-CA" flag).

If I didn't supply that flag the string wouldn't have the UTF-8 flag,
and wouldn't match, as the Perl regex engine won't use Unicode semantics
except on Unicode target strings.

Even for Perl, this behavior has been troublesome. Opinions differ, but
I think many would agree (and I've CC'd the main authority on Perl's
regex engine) that doing this by default was *probably* a mistake.

You almost never want "everything Unicode considers a digit", and if you
do using e.g. \p{Nd} instead of \d would be better in terms of
expressing your intent. I see you're running into this on the PCRE
tracker, where you're suggesting that the equivalent of /a (or /aa)
would be needed.

	https://github.com/PCRE2Project/pcre2/issues/185#issuecomment-1374796393

Which brings me home to the seeming digression about "Perl"
above.

Unlike a programming language where you'll typically "mark" your data as
it comes in, natural text as UTF-8, binary data as such etc., a "grep"
utility has to operate on more of an "all or nothing" basis (except in
the case of "-a"). I.e. we're usually searching through unknown data.

Enabling this by default means that we'll pick up characters most people
probably wouldn't expect, particularly from near-binary data formats
(those that won't require "-a", but contain non-Unicode non-ASCII
sequences).

I don't have some completely holistic view of what we should do in every
case, e.g. we turned on PCRE2_UTF so that things like "-i" would Just
Work, but even case-insensitivity has its own unexpected edge cases in
Unicode.

But I don't think those edge cases are nearly as common as those we'd
run into by enabling PCRE2_UCP. Rather than trying to opt-out with "/a"
or "/aa" I think this should be opt-in.

As the example at the start shows you can already do this with "(*UCP)"
in the pattern, so perhaps we should just link to the pcre2pattern(3)
manual from git-grep(1)?

Message sent:

Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Mailer: MIME-tools 5.505 (Entity 5.505)
Content-Type: text/plain; charset=utf-8
X-Loop: help-debbugs@HIDDEN
From: help-debbugs@HIDDEN (GNU bug Tracking System)
To: =?UTF-8?Q?=C3=86var_?= =?UTF-8?Q?Arnfj=C3=B6r=C3=B0?= Bjarmason
 <avarab@HIDDEN>
Subject: bug#60690: Acknowledgement ([PATCH v2] grep: correctly identify
 utf-8 characters with \{b,w} in -P)
Message-ID: <handler.60690.B.167326669420778.ack <at> debbugs.gnu.org>
References: <230109.86v8lf297g.gmgdl@HIDDEN>
X-Gnu-PR-Message: ack 60690
X-Gnu-PR-Package: grep
X-Gnu-PR-Keywords: patch
Reply-To: 60690 <at> debbugs.gnu.org
Date: Mon, 09 Jan 2023 12:19:01 +0000

Thank you for filing a new bug report with debbugs.gnu.org.

This is an automatically generated reply to let you know your message
has been received.

Your message is being forwarded to the package maintainers and other
interested parties for their attention; they will reply in due course.

Your message has been sent to the package maintainer(s):
 bug-grep@HIDDEN

If you wish to submit further information on this problem, please
send it to 60690 <at> debbugs.gnu.org.

Please do not send mail to help-debbugs@HIDDEN unless you wish
to report a problem with the Bug-tracking system.

--=20
60690: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D60690
GNU Bug Tracking System
Contact help-debbugs@HIDDEN with problems

Message sent to bug-grep@HIDDEN:

Subject: bug#60690: [PATCH v2] grep: correctly identify utf-8 characters with \{b, w} in -P
Resent-From: Paul Eggert <eggert@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-grep@HIDDEN
Resent-Date: Mon, 09 Jan 2023 18:41:01 +0000
Resent-Message-ID: <handler.60690.B60690.167328963122679 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: =?UTF-8?Q?=C3=86var_?= =?UTF-8?Q?Arnfj=C3=B6r=C3=B0?= Bjarmason <avarab@HIDDEN>, Carlo Marcelo Arenas =?UTF-8?Q?Bel=C3=B3n?= <carenas@HIDDEN>
Cc: demerphq@HIDDEN, pcre-dev@HIDDEN, 60690 <at> debbugs.gnu.org, git@HIDDEN, gitster@HIDDEN
Message-ID: <d6814350-10a3-55c0-68da-7e691976cd45@HIDDEN>
Date: Mon, 9 Jan 2023 10:40:16 -0800
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.6.0
Content-Language: en-US
References: <20230108062335.72114-1-carenas@HIDDEN>
 <20230108155217.2817-1-carenas@HIDDEN>
 <230109.86v8lf297g.gmgdl@HIDDEN>
From: Paul Eggert <eggert@HIDDEN>
Organization: UCLA Computer Science Department
In-Reply-To: <230109.86v8lf297g.gmgdl@HIDDEN>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: quoted-printable
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

On 1/9/23 03:35, =C3=86var Arnfj=C3=B6r=C3=B0 Bjarmason wrote:

> You almost never want "everything Unicode considers a digit", and if yo=
u
> do using e.g. \p{Nd} instead of \d would be better in terms of
> expressing your intent.

For GNU grep, PCRE2_UCP is needed because of examples like what Gro-Tsen=20
and Karl Petterssen supplied. If there's some diagreement about how \d=20
should behave with UTF-8 data the GNU grep hackers should let the Perl=20
community decide that; that is, GNU grep can simply follow PCRE2's lead.=20
But GNU grep does need PCRE2_UCP for \b etc.

> 	$ diff <(git -P grep -P '\d+') <(git -P grep -P '(*UCP)\d')
> 	53360a53361,53362
> 	> git-gui/po/ja.po:"- =E7=AC=AC=EF=BC=91=E8=A1=8C: =E4=BD=95=E3=82=92=E3=
=81=97=E3=81=9F=E3=81=8B=E3=80=81=E3=82=92=EF=BC=91=E8=A1=8C=E3=81=A7=E8=A6=
=81=E7=B4=84=E3=80=82\n"
> 	> git-gui/po/ja.po:"- =E7=AC=AC=EF=BC=92=E8=A1=8C: =E7=A9=BA=E7=99=BD\=
n"

Although I don't speak Japanese I have dealt with quite a bit of=20
Japanese text in a previous job, and personally I would prefer \d to=20
match those two lines as they do contain digits. So to me this=20
particular case is not a good argument that git grep should not match=20
those lines.

Of course other people might prefer differently, and there are cases=20
where I want to match only ASCII digits. I've learned in the past to use=20
[0-9] for that. I hope PCRE2 never changes [0-9] to match anything but=20
ASCII digits when searching UTF-8 text.

Message sent to bug-grep@HIDDEN:

Subject: bug#60690: [PATCH v2] grep: correctly identify utf-8 characters with \{b, w} in -P
Resent-From: =?UTF-8?Q?=C3=86var_?= =?UTF-8?Q?Arnfj=C3=B6r=C3=B0?= Bjarmason <avarab@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-grep@HIDDEN
Resent-Date: Mon, 09 Jan 2023 20:31:02 +0000
Resent-Message-ID: <handler.60690.B60690.16732962472757 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: Paul Eggert <eggert@HIDDEN>
Cc: demerphq@HIDDEN, 60690 <at> debbugs.gnu.org, Carlo Marcelo Arenas =?UTF-8?Q?Bel=C3=B3n?= <carenas@HIDDEN>, pcre-dev@HIDDEN, gitster@HIDDEN, git@HIDDEN
From: =?UTF-8?Q?=C3=86var_?= =?UTF-8?Q?Arnfj=C3=B6r=C3=B0?= Bjarmason <avarab@HIDDEN>
Date: Mon, 09 Jan 2023 20:51:00 +0100
References: <20230108062335.72114-1-carenas@HIDDEN>
 <20230108155217.2817-1-carenas@HIDDEN>
 <230109.86v8lf297g.gmgdl@HIDDEN>
 <d6814350-10a3-55c0-68da-7e691976cd45@HIDDEN>
User-agent: Debian GNU/Linux bookworm/sid; Emacs 28.2; mu4e 1.9.0
In-reply-to: <d6814350-10a3-55c0-68da-7e691976cd45@HIDDEN>
Message-ID: <230109.865ydf1mdu.gmgdl@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>


On Mon, Jan 09 2023, Paul Eggert wrote:

> On 1/9/23 03:35, =C3=86var Arnfj=C3=B6r=C3=B0 Bjarmason wrote:
>
>> You almost never want "everything Unicode considers a digit", and if you
>> do using e.g. \p{Nd} instead of \d would be better in terms of
>> expressing your intent.
>
> For GNU grep, PCRE2_UCP is needed because of examples like what
> Gro-Tsen and Karl Petterssen supplied.

[For reference, referring to this Twitter thread:
https://twitter.com/gro_tsen/status/1610972356972875777]

Those examples compared -E and -P. I think it's correct that UCP brings
the behavior closer to -E, but it's also different in various ways.

E.g. on emacs.git (which I've been finding to be quite a nice test case)
a comparison of the two, with "git grep" because I found it easier to
test, but GNU grep will presumably find the same for those files:
=09
	for c in b s w
	do
		for pfx in '' '(*UCP)'
		do
			echo "$pfx/$c:" &&
			diff -u <(git -P grep -E "\\$c") <(git -P grep -P "$pfx\\$c") | wc -l
		done
	done

Yields:

	/b:
	155781
	(*UCP)/b:
	46035
	/s:
	0
	(*UCP)/s:
	0
	/w:
	142468
	(*UCP)/w:
	9706

So the output still differs, and some of those differences may or may
not be wanted.

> If there's some diagreement
> about how \d should behave with UTF-8 data the GNU grep hackers should
> let the Perl community decide that; that is, GNU grep can simply
> follow PCRE2's lead.

PCRE2 tends to follow Perl, I'm mainly trying to point out here that it
isn't a-priory clear how "let Perl decide" is supposed to map to the of
a "grep"-like utility, since the Perl behavior is inherently tied up
with knowing the encoding of the target data.

For GNU grep and "git grep" that's more of an all-or-nothing with
locales, although in this case being as close as possible to -E is
probably more correct than not.

>> 	$ diff <(git -P grep -P '\d+') <(git -P grep -P '(*UCP)\d')
>> 	53360a53361,53362
>> 	> git-gui/po/ja.po:"- =E7=AC=AC=EF=BC=91=E8=A1=8C: =E4=BD=95=E3=82=92=
=E3=81=97=E3=81=9F=E3=81=8B=E3=80=81=E3=82=92=EF=BC=91=E8=A1=8C=E3=81=A7=E8=
=A6=81=E7=B4=84=E3=80=82\n"
>> 	> git-gui/po/ja.po:"- =E7=AC=AC=EF=BC=92=E8=A1=8C: =E7=A9=BA=E7=99=BD\n"
>
> Although I don't speak Japanese I have dealt with quite a bit of
> Japanese text in a previous job, and personally I would prefer \d to
> match those two lines as they do contain digits. So to me this
> particular case is not a good argument that git grep should not match
> those lines.

I'm mainly raising the backwards compatibility concern, which GNU grep
and git grep may or may not want to handle differently, but let's at
least be aware of the various edge cases.

For \b I think it mostly does the right thing.

For \w and \d in particular I'm mainly noting that yes, sometimes you
want to match [0-9], and sometimes you'd want to match Japanese numbers,
but you rarely (or at least I haven't) want to match everything Unicode
considers X, unless you're doing some self-reflection on Unicode itself.

E.g. for \d it's at least (up from just 10):

	$ perl -CO -wE 'for (1..2**20) { say chr if chr =3D~ /\d/ }'|wc -l
	650

For \w you similarly go from ~60 to ~130k:

	$ perl -CO -wE 'for (1..2**24) { say chr if chr =3D~ /\w/ }'|wc -l
	134564

If all you're doing is matching either ASCII or Japanese text and you
want "locale-aware numbers" it might do the wrong thing.

But I've found it to be too promiscuous when casting a wider net, which
is the usual use-case with 'grep".

> Of course other people might prefer differently, and there are cases
> where I want to match only ASCII digits. I've learned in the past to
> use [0-9] for that. I hope PCRE2 never changes [0-9] to match anything
> but ASCII digits when searching UTF-8 text.

I think that'll never change.

Message sent to bug-grep@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#60690: [PATCH v2] grep: correctly identify utf-8 characters with \{b, w} in -P
Resent-From: Paul Eggert <eggert@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-grep@HIDDEN
Resent-Date: Mon, 09 Jan 2023 23:13:02 +0000
Resent-Message-ID: <handler.60690.B60690.167330595319076 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 60690
X-GNU-PR-Package: grep
X-GNU-PR-Keywords: patch
To: =?UTF-8?Q?=C3=86var_?= =?UTF-8?Q?Arnfj=C3=B6r=C3=B0?= Bjarmason <avarab@HIDDEN>
Cc: demerphq@HIDDEN, 60690 <at> debbugs.gnu.org, Carlo Marcelo Arenas =?UTF-8?Q?Bel=C3=B3n?= <carenas@HIDDEN>, pcre-dev@HIDDEN, gitster@HIDDEN, git@HIDDEN
Received: via spool by 60690-submit <at> debbugs.gnu.org id=B60690.167330595319076
          (code B ref 60690); Mon, 09 Jan 2023 23:13:02 +0000
Received: (at 60690) by debbugs.gnu.org; 9 Jan 2023 23:12:33 +0000
Received: from localhost ([127.0.0.1]:38384 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1pF1Jh-0004xc-Cu
	for submit <at> debbugs.gnu.org; Mon, 09 Jan 2023 18:12:33 -0500
Received: from zimbra.cs.ucla.edu ([131.179.128.68]:54480)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eggert@HIDDEN>) id 1pF1Jf-0004xH-0V
 for 60690 <at> debbugs.gnu.org; Mon, 09 Jan 2023 18:12:32 -0500
Received: from localhost (localhost [127.0.0.1])
 by zimbra.cs.ucla.edu (Postfix) with ESMTP id E3DC2160054;
 Mon,  9 Jan 2023 15:12:24 -0800 (PST)
Received: from zimbra.cs.ucla.edu ([127.0.0.1])
 by localhost (zimbra.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10032)
 with ESMTP id QUKYHr1oDdhJ; Mon,  9 Jan 2023 15:12:24 -0800 (PST)
Received: from localhost (localhost [127.0.0.1])
 by zimbra.cs.ucla.edu (Postfix) with ESMTP id 0E66D160056;
 Mon,  9 Jan 2023 15:12:24 -0800 (PST)
DKIM-Filter: OpenDKIM Filter v2.9.2 zimbra.cs.ucla.edu 0E66D160056
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cs.ucla.edu;
 s=78364E5A-2AF3-11ED-87FA-8298ECA2D365; t=1673305944;
 bh=BjAoBqk7Uzf8782MPAQGxCGw88gce3GnII+peA/HPj8=;
 h=Message-ID:Date:MIME-Version:Subject:To:From:Content-Type:
 Content-Transfer-Encoding;
 b=ZCl0Y/BcJcKj5hCPFkegIR4vlqONfqZRG4dB2EtSGSzY05belsjIHgkeIBzmvZ0U7
 BbM4HgPJBYzfghxfciZG3wKzgKmrG0INprlLZRmUTx+oNGKRjFkaiOopMNL4cLiOfo
 VM6e7aRUAh+FfoeiTY+DyikrricO6FqHC87BLtiE=
X-Virus-Scanned: amavisd-new at zimbra.cs.ucla.edu
Received: from zimbra.cs.ucla.edu ([127.0.0.1])
 by localhost (zimbra.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10026)
 with ESMTP id O3xC3cGB2yBT; Mon,  9 Jan 2023 15:12:23 -0800 (PST)
Received: from [131.179.64.200] (Penguin.CS.UCLA.EDU [131.179.64.200])
 by zimbra.cs.ucla.edu (Postfix) with ESMTPSA id DAE02160054;
 Mon,  9 Jan 2023 15:12:23 -0800 (PST)
Message-ID: <80b42740-c85b-cef2-622c-c5b2450e264c@HIDDEN>
Date: Mon, 9 Jan 2023 15:12:23 -0800
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.6.0
Content-Language: en-US
References: <20230108062335.72114-1-carenas@HIDDEN>
 <20230108155217.2817-1-carenas@HIDDEN>
 <230109.86v8lf297g.gmgdl@HIDDEN>
 <d6814350-10a3-55c0-68da-7e691976cd45@HIDDEN>
 <230109.865ydf1mdu.gmgdl@HIDDEN>
From: Paul Eggert <eggert@HIDDEN>
Organization: UCLA Computer Science Department
In-Reply-To: <230109.865ydf1mdu.gmgdl@HIDDEN>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -3.4 (---)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -4.4 (----)

On 1/9/23 11:51, =C3=86var Arnfj=C3=B6r=C3=B0 Bjarmason wrote:

> 	/b:
> 	155781
> 	(*UCP)/b:
> 	46035
> 	/s:
> 	0
> 	(*UCP)/s:
> 	0
> 	/w:
> 	142468
> 	(*UCP)/w:
> 	9706
>=20
> So the output still differs, and some of those differences may or may
> not be wanted.

I took a look at the output, and by and large I'd want the differences;=20
that is, I'd want the UCP version, which generates less output. This is=20
because several Emacs source files are not UTF-8, and \b has nonsense=20
matches when searching text files encoded via Shift-JIS or Big 5 or=20
whatever. For this sort of thing, the fewer matches the better.


> If all you're doing is matching either ASCII or Japanese text and you
> want "locale-aware numbers" it might do the wrong thing.

I'm not seeing much of a problem here. When searching Japanese text, I=20
would expect \d and [0-9=EF=BC=90-=EF=BC=99] (using both ASCII and full-w=
idth digits) to=20
be equivalent so (assuming UCP) it's not a big deal as to which regex=20
you use, since Japanese text won't contain Bengali (or whatever) digits.=20
And when searching binary data, I'd expect a bunch of garbage no matter=20
how \d is interpreted.

Here I'm assuming [=EF=BC=90-=EF=BC=99] (using full-width digits) has the=
 expected=20
meaning in PCRE2, i.e., that PCRE2 didn't make the same mistake that=20
POSIX made.

Message received at control <at> debbugs.gnu.org:

Received: (at control) by debbugs.gnu.org; 3 Apr 2023 20:44:42 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Apr 03 16:44:42 2023
Received: from localhost ([127.0.0.1]:45394 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1pjR2f-0003vr-Qy
	for submit <at> debbugs.gnu.org; Mon, 03 Apr 2023 16:44:42 -0400
Received: from mail.cs.ucla.edu ([131.179.128.66]:40168)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eggert@HIDDEN>) id 1pjR2d-0003vW-8B
 for control <at> debbugs.gnu.org; Mon, 03 Apr 2023 16:44:40 -0400
Received: from localhost (localhost [127.0.0.1])
 by mail.cs.ucla.edu (Postfix) with ESMTP id D84F33C097AFC
 for <control <at> debbugs.gnu.org>; Mon,  3 Apr 2023 13:44:32 -0700 (PDT)
Received: from mail.cs.ucla.edu ([127.0.0.1])
 by localhost (mail.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10032)
 with ESMTP id NjXhv9YZ1vkB for <control <at> debbugs.gnu.org>;
 Mon,  3 Apr 2023 13:44:32 -0700 (PDT)
Received: from localhost (localhost [127.0.0.1])
 by mail.cs.ucla.edu (Postfix) with ESMTP id 866363C09FA04
 for <control <at> debbugs.gnu.org>; Mon,  3 Apr 2023 13:44:32 -0700 (PDT)
DKIM-Filter: OpenDKIM Filter v2.10.3 mail.cs.ucla.edu 866363C09FA04
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cs.ucla.edu;
 s=9D0B346E-2AEB-11ED-9476-E14B719DCE6C; t=1680554672;
 bh=EfkwPFAdMvuQM4+OUYD+FPgIUhIcD1pUZ8w2eaT0zl8=;
 h=Message-ID:Date:MIME-Version:To:From;
 b=qxK/2OM0ojYaXpJXDjScM9GJfDi0N/b3NEM2xbbxX/sR2OqgiAzooVuxKUFwxE1BH
 e1Q6m5O5AV2XtU0wUdc9jyXyyBXz+ObThg73HL+e4FStSo5EpIfRR19envT/zXYSu7
 Iq0zssPJmkLTxRIVUkzf0OEmwvSAZhqSJOB98t+lNCw/7wFFQ3TXgE8PXikrDm6FrC
 SHj/81Wer6vEnLUZBXKaD4lG55xFXX0fUCTxCdFMqIc7lzecCHE/7oZN7+GCXa1zS+
 ujPOePLzSv6YVXLiolXL2QsRzvyJU0B9XGYs01byzr9whsyHctkno2GOhwR/hv1DXb
 j0ESczqasGDNQ==
X-Virus-Scanned: amavisd-new at mail.cs.ucla.edu
Received: from mail.cs.ucla.edu ([127.0.0.1])
 by localhost (mail.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10026)
 with ESMTP id hM5R3rzqa6Fq for <control <at> debbugs.gnu.org>;
 Mon,  3 Apr 2023 13:44:32 -0700 (PDT)
Received: from [131.179.64.200] (Penguin.CS.UCLA.EDU [131.179.64.200])
 by mail.cs.ucla.edu (Postfix) with ESMTPSA id 73C193C097AFC
 for <control <at> debbugs.gnu.org>; Mon,  3 Apr 2023 13:44:32 -0700 (PDT)
Message-ID: <d14924af-5ff1-e782-844f-7a6bd2a010e4@HIDDEN>
Date: Mon, 3 Apr 2023 13:44:32 -0700
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.9.1
Content-Language: en-US
To: GNU bug control <control <at> debbugs.gnu.org>
From: Paul Eggert <eggert@HIDDEN>
Subject: merge a related \d bug report
Organization: UCLA Computer Science Department
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: control
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

tags 60690 + patch
merge 60690 62605

Message sent to bug-grep@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#60690: -P '\d' in GNU and git grep
References: <230109.86v8lf297g.gmgdl@HIDDEN>
In-Reply-To: <230109.86v8lf297g.gmgdl@HIDDEN>
Resent-From: Paul Eggert <eggert@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-grep@HIDDEN
Resent-Date: Mon, 03 Apr 2023 21:39:02 +0000
Resent-Message-ID: <handler.60690.B60690.168055793630592 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 60690
X-GNU-PR-Package: grep
X-GNU-PR-Keywords: patch
To: 60690 <at> debbugs.gnu.org
Cc: demerphq@HIDDEN, mega lith01 <megalith01@HIDDEN>, Carlo Marcelo Arenas =?UTF-8?Q?Bel=C3=B3n?= <carenas@HIDDEN>, =?UTF-8?Q?=C3=86var_?= =?UTF-8?Q?Arnfj=C3=B6r=C3=B0?= Bjarmason <avarab@HIDDEN>, git@HIDDEN, gitster@HIDDEN, =?UTF-8?Q?Tukusej=E2=80=99s?= Sirs <tukusejssirs@HIDDEN>, pcre-dev@HIDDEN
Received: via spool by 60690-submit <at> debbugs.gnu.org id=B60690.168055793630592
          (code B ref 60690); Mon, 03 Apr 2023 21:39:02 +0000
Received: (at 60690) by debbugs.gnu.org; 3 Apr 2023 21:38:56 +0000
Received: from localhost ([127.0.0.1]:45428 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1pjRt9-0007xL-M4
	for submit <at> debbugs.gnu.org; Mon, 03 Apr 2023 17:38:56 -0400
Received: from mail.cs.ucla.edu ([131.179.128.66]:60202)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eggert@HIDDEN>) id 1pjRt4-0007x1-Au
 for 60690 <at> debbugs.gnu.org; Mon, 03 Apr 2023 17:38:54 -0400
Received: from localhost (localhost [127.0.0.1])
 by mail.cs.ucla.edu (Postfix) with ESMTP id 6EF8C3C097AFC;
 Mon,  3 Apr 2023 14:38:43 -0700 (PDT)
Received: from mail.cs.ucla.edu ([127.0.0.1])
 by localhost (mail.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10032)
 with ESMTP id 01c1HguO6g6L; Mon,  3 Apr 2023 14:38:43 -0700 (PDT)
Received: from localhost (localhost [127.0.0.1])
 by mail.cs.ucla.edu (Postfix) with ESMTP id F395C3C097AFD;
 Mon,  3 Apr 2023 14:38:42 -0700 (PDT)
DKIM-Filter: OpenDKIM Filter v2.10.3 mail.cs.ucla.edu F395C3C097AFD
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cs.ucla.edu;
 s=9D0B346E-2AEB-11ED-9476-E14B719DCE6C; t=1680557923;
 bh=mPk8IoktGpCOFvvbm6FfDFpRZjgCNQY0M93nZRbaYvA=;
 h=Message-ID:Date:MIME-Version:To:From;
 b=Mb6Ob0rNFkTbzDgf/zQA3lF1a3BgMs8pOKjlhZSCYQThLrVMFUCs/n0whkksKwcaV
 qSAmHwcLn7ult6QYNTe1YG9vbwz3iTL857NVLQ95BX7Z0d6WHxmdz1n725C5qXqCma
 zupJnWHHeOsnhbsLvcAk+jw4t3bzBibhqNqSVnh/UzHFuAnQL/azagvwM9/ioUKgV0
 VXSBvWHBBG/w8S2Jmh44IeCL+Z+zfF0GpRtk/kZKJhKPF2M59RcPg57bw3Y3323p+k
 mttcUF6Ldx3Wq85uGKFURxqrziDdWVpw21RgBTbuy8HeUMRAAkJvGb2hVUWudxbHRN
 Ti9q5U/oqGzQw==
X-Virus-Scanned: amavisd-new at mail.cs.ucla.edu
Received: from mail.cs.ucla.edu ([127.0.0.1])
 by localhost (mail.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10026)
 with ESMTP id GXWQ8CbnROrO; Mon,  3 Apr 2023 14:38:42 -0700 (PDT)
Received: from [131.179.64.200] (Penguin.CS.UCLA.EDU [131.179.64.200])
 by mail.cs.ucla.edu (Postfix) with ESMTPSA id D2C2F3C097AFC;
 Mon,  3 Apr 2023 14:38:42 -0700 (PDT)
Message-ID: <2554712d-e386-3bab-bc6c-1f0e85d999db@HIDDEN>
Date: Mon, 3 Apr 2023 14:38:42 -0700
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.9.1
Content-Language: en-US
From: Paul Eggert <eggert@HIDDEN>
Organization: UCLA Computer Science Department
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Spam-Score: 0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

I've recently done some bug-report maintenance about a set of GNU grep 
bug reports related to whether whether "grep -P '\d'" should match 
non-ASCII digits, and have some thoughts about coordinating GNU grep 
with git grep in this department.

GNU Bug#62605[1] "`[\d]` does not work with PCRE" has been fixed on 
Savannah's copy of GNU grep, and some sort of fix should appear in the 
next grep release. However, I'm leaving the GNU grep bug report open for 
now because it's related to Bug#60690[2] "[PATCH v2] grep: correctly 
identify utf-8 characters with \{b,w} in -P" and to Bug#62552[3] "Bug 
found in latest stable release v3.10 of grep". I merged these related 
bug reports, and the oldest one, Bug#60690, is now the representative 
displayed in the GNU grep bug list[4].

For this set of grep bug reports there's still a pending issue discussed 
in my recent email[5], which proposes a patch so I've tagged Bug#60690 
with "patch". The proposal is that GNU grep -P '\d' should revert to the 
grep 3.9 behavior, i.e., that in a UTF-8 locale, \d should also match 
non-ASCII decimal digits.

In researching this a bit further, I found that on March 23 Git disabled 
the use of PCRE2_UCP in PCRE2 10.34 or earlier[6], due to a PCRE2 bug 
that can cause a crash when PCRE2_UCP is used[7]. A bug fix[8] should 
appear in the next PCRE2 release.

When PCRE2 10.35 comes out, it appears that 'git grep -P' will behave 
like 'grep -P' only if GNU grep adopts something like the solution 
proposed in [5].

[1]: https://bugs.gnu.org/62605
[2]: https://bugs.gnu.org/60690
[3]: https://bugs.gnu.org/62552
[4]: https://debbugs.gnu.org/cgi/pkgreport.cgi?package=grep
[5]: https://lists.gnu.org/archive/html/grep-devel/2023-04/msg00004.html
[6]: 
https://github.com/git/git/commit/14b9a044798ebb3858a1f1a1377309a3d6054ac8
[7]: 
https://lore.kernel.org/git/7E83DAA1-F9A9-4151-8D07-D80EA6D59EEA@HIDDEN/
[8]: 
https://github.com/git/git/commit/14b9a044798ebb3858a1f1a1377309a3d6054ac8

Message sent to bug-grep@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#60690: -P '\d' in GNU and git grep
Resent-From: Jim Meyering <jim@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-grep@HIDDEN
Resent-Date: Tue, 04 Apr 2023 03:31:01 +0000
Resent-Message-ID: <handler.60690.B60690.168057902317359 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 60690
X-GNU-PR-Package: grep
X-GNU-PR-Keywords: patch
To: Paul Eggert <eggert@HIDDEN>
Cc: demerphq@HIDDEN, 60690 <at> debbugs.gnu.org, mega lith01 <megalith01@HIDDEN>, Carlo Marcelo Arenas =?UTF-8?Q?Bel=C3=B3n?= <carenas@HIDDEN>, =?UTF-8?Q?=C3=86var_?= =?UTF-8?Q?Arnfj=C3=B6r=C3=B0?= Bjarmason <avarab@HIDDEN>, gitster@HIDDEN, pcre-dev@HIDDEN, =?UTF-8?Q?Tukusej=E2=80=99s?= Sirs <tukusejssirs@HIDDEN>, git@HIDDEN
Received: via spool by 60690-submit <at> debbugs.gnu.org id=B60690.168057902317359
          (code B ref 60690); Tue, 04 Apr 2023 03:31:01 +0000
Received: (at 60690) by debbugs.gnu.org; 4 Apr 2023 03:30:23 +0000
Received: from localhost ([127.0.0.1]:46176 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1pjXNH-0004Vu-DC
	for submit <at> debbugs.gnu.org; Mon, 03 Apr 2023 23:30:23 -0400
Received: from mail-lf1-f52.google.com ([209.85.167.52]:45921)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <meyering@HIDDEN>) id 1pjXNF-0004VV-Mt
 for 60690 <at> debbugs.gnu.org; Mon, 03 Apr 2023 23:30:22 -0400
Received: by mail-lf1-f52.google.com with SMTP id bi9so40607786lfb.12
 for <60690 <at> debbugs.gnu.org>; Mon, 03 Apr 2023 20:30:21 -0700 (PDT)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20210112; t=1680579015;
 h=content-transfer-encoding:cc:to:subject:message-id:date:from
 :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc
 :subject:date:message-id:reply-to;
 bh=RxLvDywim81BDoQlVW2zTMwgbTw8PpwLQvG0B7AxiqM=;
 b=44XnBW/jFZsS6NNoSxgDlIODu1TXcsi0haFm2msgi+cwgiVGrII+ja8Ns/dsNav0f8
 dHucp5wMJlHO/Sv9shu/MbQtRFVmiquZ49yL1aQ32Q3lpbVin8j5bUv0seyEbzFHK6lv
 wnCXqdRoPWK4G6Es4zzHe+VwA1J3ePQpUszQt4yZQUsSpC0ljLrAIiYwXjjaxXxnYvGG
 ZOqbzXZyEEhtcFpbvFCtEjbSm3Sig2h/M1sYLnFDfWR5zshKIT1da3Rl1DUsKusiXti+
 ayAV96JtP9eo3cS+qIv/xRfBg9lTmw0a7BElkqLtpTjnU3NIq6DEmMV+HEzoAcXq+Jx5
 O1uQ==
X-Gm-Message-State: AAQBX9ceov84q9xb1HN17PHoyQOybdKF+tKAqjcNYZ7e0xSXhuGRwnfs
 oWTKEO1Y9Z9FaZu3AGV/1xb6YD3lzKctK2A04HE=
X-Google-Smtp-Source: AKy350buA7U1HUotqrdLUjYZ0ef4wICYwOpKaxxNkWrBVNxWp/yfxQR+dKDIjFqxCeAk36IpJF1epjCYQUTvgBWJolc=
X-Received: by 2002:a19:f607:0:b0:4d5:ca32:6ed5 with SMTP id
 x7-20020a19f607000000b004d5ca326ed5mr388039lfe.3.1680579015313; Mon, 03 Apr
 2023 20:30:15 -0700 (PDT)
MIME-Version: 1.0
References: <230109.86v8lf297g.gmgdl@HIDDEN>
 <2554712d-e386-3bab-bc6c-1f0e85d999db@HIDDEN>
In-Reply-To: <2554712d-e386-3bab-bc6c-1f0e85d999db@HIDDEN>
From: Jim Meyering <jim@HIDDEN>
Date: Mon, 3 Apr 2023 20:30:02 -0700
Message-ID: <CA+8g5KHuE-kQqmi9cVjeJbpyt54v9m9omh9A9we1zmR0+aTDHg@HIDDEN>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: 0.2 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.8 (/)

On Mon, Apr 3, 2023 at 2:39=E2=80=AFPM Paul Eggert <eggert@HIDDEN> wro=
te:
> I've recently done some bug-report maintenance about a set of GNU grep
> bug reports related to whether whether "grep -P '\d'" should match
> non-ASCII digits, and have some thoughts about coordinating GNU grep
> with git grep in this department.
>
> GNU Bug#62605[1] "`[\d]` does not work with PCRE" has been fixed on
> Savannah's copy of GNU grep, and some sort of fix should appear in the
> next grep release. However, I'm leaving the GNU grep bug report open for
> now because it's related to Bug#60690[2] "[PATCH v2] grep: correctly
> identify utf-8 characters with \{b,w} in -P" and to Bug#62552[3] "Bug
> found in latest stable release v3.10 of grep". I merged these related
> bug reports, and the oldest one, Bug#60690, is now the representative
> displayed in the GNU grep bug list[4].
>
> For this set of grep bug reports there's still a pending issue discussed
> in my recent email[5], which proposes a patch so I've tagged Bug#60690
> with "patch". The proposal is that GNU grep -P '\d' should revert to the
> grep 3.9 behavior, i.e., that in a UTF-8 locale, \d should also match
> non-ASCII decimal digits.
>
> In researching this a bit further, I found that on March 23 Git disabled
> the use of PCRE2_UCP in PCRE2 10.34 or earlier[6], due to a PCRE2 bug
> that can cause a crash when PCRE2_UCP is used[7]. A bug fix[8] should
> appear in the next PCRE2 release.
>
> When PCRE2 10.35 comes out,

Thanks for finding that.
It's clearly a good idea to disable PCRE2_UCP for those using those
older, known-buggy versions of pcre2.

The latest is 10.42, per https://github.com/PCRE2Project/pcre2/releases

> it appears that 'git grep -P' will behave
> like 'grep -P' only if GNU grep adopts something like the solution
> proposed in [5].
>
> [1]: https://bugs.gnu.org/62605
> [2]: https://bugs.gnu.org/60690
> [3]: https://bugs.gnu.org/62552
> [4]: https://debbugs.gnu.org/cgi/pkgreport.cgi?package=3Dgrep
> [5]: https://lists.gnu.org/archive/html/grep-devel/2023-04/msg00004.html
> [6]:
> https://github.com/git/git/commit/14b9a044798ebb3858a1f1a1377309a3d6054ac=
8
> [7]:
> https://lore.kernel.org/git/7E83DAA1-F9A9-4151-8D07-D80EA6D59EEA@HIDDEN=
om/
> [8]:
> https://github.com/git/git/commit/14b9a044798ebb3858a1f1a1377309a3d6054ac=
8

Thanks for all of the links. However, have you seen justification
(other than for compatibility with some other tool or language) for
allowing \d to match non-ASCII by default, in spite of the risks?
IMHO, we have an obligation to retain compatibility with how grep -P
'\d' has worked since -P was added. I'd be happy to see an option to
enable the match-multibyte-digits behavior, but making it the default
seems too likely to introduce unwarranted risk.

Message sent to bug-grep@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#60690: -P '\d' in GNU and git grep
Resent-From: Paul Eggert <eggert@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-grep@HIDDEN
Resent-Date: Tue, 04 Apr 2023 06:48:02 +0000
Resent-Message-ID: <handler.60690.B60690.168059082620193 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 60690
X-GNU-PR-Package: grep
X-GNU-PR-Keywords: patch
To: Jim Meyering <jim@HIDDEN>
Cc: demerphq@HIDDEN, 60690 <at> debbugs.gnu.org, mega lith01 <megalith01@HIDDEN>, Carlo Marcelo Arenas =?UTF-8?Q?Bel=C3=B3n?= <carenas@HIDDEN>, =?UTF-8?Q?=C3=86var_?= =?UTF-8?Q?Arnfj=C3=B6r=C3=B0?= Bjarmason <avarab@HIDDEN>, gitster@HIDDEN, pcre-dev@HIDDEN, =?UTF-8?Q?Tukusej=E2=80=99s?= Sirs <tukusejssirs@HIDDEN>, git@HIDDEN
Received: via spool by 60690-submit <at> debbugs.gnu.org id=B60690.168059082620193
          (code B ref 60690); Tue, 04 Apr 2023 06:48:02 +0000
Received: (at 60690) by debbugs.gnu.org; 4 Apr 2023 06:47:06 +0000
Received: from localhost ([127.0.0.1]:46385 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1pjaRe-0005Fc-BZ
	for submit <at> debbugs.gnu.org; Tue, 04 Apr 2023 02:47:06 -0400
Received: from mail.cs.ucla.edu ([131.179.128.66]:50556)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eggert@HIDDEN>) id 1pjaRc-0005F8-7M
 for 60690 <at> debbugs.gnu.org; Tue, 04 Apr 2023 02:47:04 -0400
Received: from localhost (localhost [127.0.0.1])
 by mail.cs.ucla.edu (Postfix) with ESMTP id B809F3C09FA06;
 Mon,  3 Apr 2023 23:46:58 -0700 (PDT)
Received: from mail.cs.ucla.edu ([127.0.0.1])
 by localhost (mail.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10032)
 with ESMTP id dRGahyowkSL6; Mon,  3 Apr 2023 23:46:58 -0700 (PDT)
Received: from localhost (localhost [127.0.0.1])
 by mail.cs.ucla.edu (Postfix) with ESMTP id 50CD53C09FA08;
 Mon,  3 Apr 2023 23:46:58 -0700 (PDT)
DKIM-Filter: OpenDKIM Filter v2.10.3 mail.cs.ucla.edu 50CD53C09FA08
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cs.ucla.edu;
 s=9D0B346E-2AEB-11ED-9476-E14B719DCE6C; t=1680590818;
 bh=OPKGbZHEgoMi8NRsw8a0G2vgxL3IygYevxOn970mno8=;
 h=Message-ID:Date:MIME-Version:To:From;
 b=SzxiOLdccdooebFoT64zRlOK1yMXrRpaZZTb6G913crnXjlaCR9GtOg7enjqjrHRK
 dK4wVg8cHY1IiOyaP/vEPnM4n7aMZUXu40/SZ6H7gTZbJKQH7qECPXC1QoDbvfyAmo
 WKPy28P+4mI1eAGN0UlagJWtNNmlB7GcLnMm2nGGfV3DqqKcCzgBxnLp94pzrMBDj/
 c+U9KvQtBHXCFeOhE/X8i+HyltGsZr23a8t2L0f9bA8EQezx+KHCSB4dVoB4MO+up7
 iOvwIOSimA/sZl71UWJUTYE3VlF3UmpKgP1Afd7UILJ8urXbEbEA0JjTBbi011K/C1
 ycpJq+AqoDfJg==
X-Virus-Scanned: amavisd-new at mail.cs.ucla.edu
Received: from mail.cs.ucla.edu ([127.0.0.1])
 by localhost (mail.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10026)
 with ESMTP id V-Qx7gZwcg3n; Mon,  3 Apr 2023 23:46:58 -0700 (PDT)
Received: from [192.168.1.9] (cpe-172-91-119-151.socal.res.rr.com
 [172.91.119.151])
 by mail.cs.ucla.edu (Postfix) with ESMTPSA id 0AB563C09FA06;
 Mon,  3 Apr 2023 23:46:58 -0700 (PDT)
Message-ID: <920dcc8d-9e45-a03e-af06-6b420c6e0f81@HIDDEN>
Date: Mon, 3 Apr 2023 23:46:57 -0700
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.9.0
Content-Language: en-US
References: <230109.86v8lf297g.gmgdl@HIDDEN>
 <2554712d-e386-3bab-bc6c-1f0e85d999db@HIDDEN>
 <CA+8g5KHuE-kQqmi9cVjeJbpyt54v9m9omh9A9we1zmR0+aTDHg@HIDDEN>
From: Paul Eggert <eggert@HIDDEN>
Organization: UCLA Computer Science Department
In-Reply-To: <CA+8g5KHuE-kQqmi9cVjeJbpyt54v9m9omh9A9we1zmR0+aTDHg@HIDDEN>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -1.1 (-)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -2.1 (--)

On 2023-04-03 20:30, Jim Meyering wrote:
> have you seen justification
> (other than for compatibility with some other tool or language) for
> allowing \d to match non-ASCII by default, in spite of the risks?

In the example =C3=86var supplied in <https://bugs.gnu.org/60690>, my=20
impression was that it was better when \d matched non-ASCII digits. That=20
is, in a UTF-8 locale it's better when \d finds matches in these lines:

>> 	> git-gui/po/ja.po:"- =E7=AC=AC=EF=BC=91=E8=A1=8C: =E4=BD=95=E3=82=92=
=E3=81=97=E3=81=9F=E3=81=8B=E3=80=81=E3=82=92=EF=BC=91=E8=A1=8C=E3=81=A7=E8=
=A6=81=E7=B4=84=E3=80=82\n"
>> 	> git-gui/po/ja.po:"- =E7=AC=AC=EF=BC=92=E8=A1=8C: =E7=A9=BA=E7=99=BD=
\n"

because they contain the Japanese digits "=EF=BC=91" and "=EF=BC=92". Thi=
s was the only=20
example I recall being given.

Also, I find it odd that grep -P '^[\w\d]*$' matches lines containing=20
any sort of Arabic word characters, but it rejects lines containing=20
Arabic digits like "=D9=A3" that are perfectly reasonable in Arabic-langu=
age=20
text. I also find it odd that [\d] and [[:digit:]] mean different things.

There are arguments on the other side, otherwise we wouldn't be having=20
this discussion. And it's true that grep -P '\d' formerly rejected=20
Arabic digits (though it's also true that grep -P '\w' formerly rejected=20
Arabic letters...). Still, the cure's oddness and incompatibility with=20
Git, Perl, etc. appears to me to be worse than the disease of dealing=20
with grep -P invocations that need to use [0-9] or LC_ALL=3D"C" anyway if=
=20
they want to be portable to any program other than GNU grep.

Message sent to bug-grep@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#60690: -P '\d' in GNU and git grep
Resent-From: Carlo Arenas <carenas@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-grep@HIDDEN
Resent-Date: Tue, 04 Apr 2023 06:58:02 +0000
Resent-Message-ID: <handler.60690.B60690.168059143322032 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 60690
X-GNU-PR-Package: grep
X-GNU-PR-Keywords: patch
To: Paul Eggert <eggert@HIDDEN>
Cc: demerphq@HIDDEN, 60690 <at> debbugs.gnu.org, mega lith01 <megalith01@HIDDEN>, =?UTF-8?Q?=C3=86var_?= =?UTF-8?Q?Arnfj=C3=B6r=C3=B0?= Bjarmason <avarab@HIDDEN>, git@HIDDEN, gitster@HIDDEN, =?UTF-8?Q?Tukusej=E2=80=99s?= Sirs <tukusejssirs@HIDDEN>, pcre-dev@HIDDEN
Received: via spool by 60690-submit <at> debbugs.gnu.org id=B60690.168059143322032
          (code B ref 60690); Tue, 04 Apr 2023 06:58:02 +0000
Received: (at 60690) by debbugs.gnu.org; 4 Apr 2023 06:57:13 +0000
Received: from localhost ([127.0.0.1]:46389 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1pjabR-0005jG-H0
	for submit <at> debbugs.gnu.org; Tue, 04 Apr 2023 02:57:13 -0400
Received: from mail-wm1-f53.google.com ([209.85.128.53]:33654)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <carenas@HIDDEN>) id 1pjabP-0005j2-DK
 for 60690 <at> debbugs.gnu.org; Tue, 04 Apr 2023 02:57:11 -0400
Received: by mail-wm1-f53.google.com with SMTP id
 v20-20020a05600c471400b003ed8826253aso442870wmo.0
 for <60690 <at> debbugs.gnu.org>; Mon, 03 Apr 2023 23:57:11 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=gmail.com; s=20210112; t=1680591425;
 h=content-transfer-encoding:cc:to:subject:message-id:date:from
 :in-reply-to:references:mime-version:from:to:cc:subject:date
 :message-id:reply-to;
 bh=L+z/XBfmEv3ffS4KRALIKDiiP+nzelhlbDnf7JG/2/I=;
 b=Ze/kUqG1BDVhp1NkyRxsa/Sz5NYonV2JetzPVy1toCgdoSwnGFOi6SQTTuEyOTMwfz
 Za3jS1ahSOYwF0IL7PrRmRf4y5hk3AJkNzK64/m1SxIOEEAXHQKz/4s8D0dSQrwr/w1p
 1vwwN2h5nQzQUOezT8K1n9TGi89eSeD4gF/YXYRMGWQVbvqUbWvgZOilDOkgat8Kdmd1
 qinheoZJ7UV1x9GfL1TwUpIbBHC9+gfjuHNA1w7DDAMPJcuQOqDi1C5coOKVBziS49Wa
 tjPUAeZO/g4N/Kk93xXtpRwR8eH0xNWMHRZVLjaksTWkyAEtg/vLty+oX/o2eLpod49V
 pH8g==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20210112; t=1680591425;
 h=content-transfer-encoding:cc:to:subject:message-id:date:from
 :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc
 :subject:date:message-id:reply-to;
 bh=L+z/XBfmEv3ffS4KRALIKDiiP+nzelhlbDnf7JG/2/I=;
 b=V8kcY8tpgqjAVlmbtOvytw32LJl6gm3p0pxUKDv0o3YOgzeHxj0LquEvTDUpV/00OP
 sYQjwrAxmNUrymI2wmPKNSwvy/01Z8yoewSgbXgz6jAYxCHTZEt3v74JpjtS/gV8Qdc4
 44QQ97l+WTaQA357GC+JnYUQIPTJBWRhXQWe4VNx0Cg7Rnb/MZ1zOzvwEav8vETOQFji
 /csfGMevk9HM3yduvSb6hCtB/kynfxMKleCZyWhKdY41XoQvmqEaVcemC3GC76fwCslN
 z/0KrhurCFEh4pbPzFzM6purmBI75AaV1C6YD+MdcFWu2rBqVKXXm+tl8TdF7oWVrieF
 5fOQ==
X-Gm-Message-State: AAQBX9fgrCX4g+ullqgLcnHmdJbjNBj505IqXGblOH2/Hqbh9uPqA+ir
 dbd+Uz6XsE6X35CE4C6ItZMXBVyci0vUa32A9M0=
X-Google-Smtp-Source: AKy350bAChGT+QnAeagE/UnDfrmPXehjEhK9ou4j99vXGkjQ8Wy0ipglwG7z086+Es08phDNXXTYt68GbK8Kvg8EzJc=
X-Received: by 2002:a05:600c:2202:b0:3ed:6979:3ab with SMTP id
 z2-20020a05600c220200b003ed697903abmr464916wml.4.1680591425352; Mon, 03 Apr
 2023 23:57:05 -0700 (PDT)
MIME-Version: 1.0
References: <2554712d-e386-3bab-bc6c-1f0e85d999db@HIDDEN>
In-Reply-To: <2554712d-e386-3bab-bc6c-1f0e85d999db@HIDDEN>
From: Carlo Arenas <carenas@HIDDEN>
Date: Mon, 3 Apr 2023 23:56:54 -0700
Message-ID: <CAPUEspj1m6F0_XgOFUVaq3Aq_Ah3PzCUs7YUyFH9_Zz-MOYTTA@HIDDEN>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: 0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

On Mon, Apr 3, 2023 at 2:38=E2=80=AFPM Paul Eggert <eggert@HIDDEN> wro=
te:
>
> In researching this a bit further, I found that on March 23 Git disabled
> the use of PCRE2_UCP in PCRE2 10.34 or earlier[6], due to a PCRE2 bug
> that can cause a crash when PCRE2_UCP is used[7]. A bug fix[8] should
> appear in the next PCRE2 release.

Presume PCRE2 is a typo and should have been "git" here?

FWIW the PCRE2 fix[1] has been released already with 10.35 and
backporting to the Ubuntu 20.04 package that crashed in the original
report would also solve the crash with 10.34.

Carlo

[1] https://github.com/PCRE2Project/pcre2/commit/c21bd977547d

Message sent to bug-grep@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#60690: -P '\d' in GNU and git grep
Resent-From: Jim Meyering <jim@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-grep@HIDDEN
Resent-Date: Tue, 04 Apr 2023 15:33:02 +0000
Resent-Message-ID: <handler.60690.B60690.168062232615573 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 60690
X-GNU-PR-Package: grep
X-GNU-PR-Keywords: patch
To: Paul Eggert <eggert@HIDDEN>
Cc: demerphq@HIDDEN, 60690 <at> debbugs.gnu.org, mega lith01 <megalith01@HIDDEN>, Carlo Marcelo Arenas =?UTF-8?Q?Bel=C3=B3n?= <carenas@HIDDEN>, =?UTF-8?Q?=C3=86var_?= =?UTF-8?Q?Arnfj=C3=B6r=C3=B0?= Bjarmason <avarab@HIDDEN>, gitster@HIDDEN, pcre-dev@HIDDEN, =?UTF-8?Q?Tukusej=E2=80=99s?= Sirs <tukusejssirs@HIDDEN>, git@HIDDEN
Received: via spool by 60690-submit <at> debbugs.gnu.org id=B60690.168062232615573
          (code B ref 60690); Tue, 04 Apr 2023 15:33:02 +0000
Received: (at 60690) by debbugs.gnu.org; 4 Apr 2023 15:32:06 +0000
Received: from localhost ([127.0.0.1]:50022 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1pjidh-000437-Qs
	for submit <at> debbugs.gnu.org; Tue, 04 Apr 2023 11:32:06 -0400
Received: from mail-lf1-f41.google.com ([209.85.167.41]:42513)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <meyering@HIDDEN>) id 1pjidf-00042Z-GF
 for 60690 <at> debbugs.gnu.org; Tue, 04 Apr 2023 11:32:04 -0400
Received: by mail-lf1-f41.google.com with SMTP id g19so29735047lfr.9
 for <60690 <at> debbugs.gnu.org>; Tue, 04 Apr 2023 08:32:03 -0700 (PDT)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20210112; t=1680622317;
 h=content-transfer-encoding:cc:to:subject:message-id:date:from
 :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc
 :subject:date:message-id:reply-to;
 bh=wnAS7PnZ+qI47ad2cTwmspqJ/bYge73O/i5SR9X3vUw=;
 b=CieOdYctcadTYiE5v1YzIhjjQNykINWkd3lfEs6pYleJblKoEaaN4hrFx8Fjjqsbla
 Dky7EXhHv3YEb4pqSoXhxXhbS5h5Uk52t2sxNzCH4FR0fM+Wy9KWZeGmkSodgj5vb1Mk
 GhgeWDKOrYnNRO3IKvPNK8UETnq72PCohlw9ZirI+kLPVKQQLHo1EWwPW6HPFp5Z/IHA
 Y4guuD3n6JJ6ghcf879cOBfYRTPCg3SoOFdUBt70r+xfjfvGpIxah2aLymfFCZbuVFi8
 2GcVyFd3LISYy5hyV/XSjgdBwsZnsasmZrJQ72pcisEaKoNU+4qoFGstsohNq8Kyqy7A
 keoA==
X-Gm-Message-State: AAQBX9dYacTbIb7iheG1YvctDhs3TtIIqS8HcVrNfBWppmMZYkiWMGzD
 ssKS8M6wyr2lbP+nkzRGXl5JvambAopp5+X/k+o=
X-Google-Smtp-Source: AKy350aQK+9RCaSofpCjRENW8HmYTkll3IqdCd7d/eG1RmpuOsmcEPOT7NDhkJJdYa25UePBK2pfpReUFNKhcodRJdE=
X-Received: by 2002:a19:ad09:0:b0:4d5:ca32:6aea with SMTP id
 t9-20020a19ad09000000b004d5ca326aeamr879865lfc.10.1680622317044; Tue, 04 Apr
 2023 08:31:57 -0700 (PDT)
MIME-Version: 1.0
References: <230109.86v8lf297g.gmgdl@HIDDEN>
 <2554712d-e386-3bab-bc6c-1f0e85d999db@HIDDEN>
 <CA+8g5KHuE-kQqmi9cVjeJbpyt54v9m9omh9A9we1zmR0+aTDHg@HIDDEN>
 <920dcc8d-9e45-a03e-af06-6b420c6e0f81@HIDDEN>
In-Reply-To: <920dcc8d-9e45-a03e-af06-6b420c6e0f81@HIDDEN>
From: Jim Meyering <jim@HIDDEN>
Date: Tue, 4 Apr 2023 08:31:44 -0700
Message-ID: <CA+8g5KGGnsf0xMCXO28R1m8-z76=kG_AiYRh6=OgRL+x5C1yqQ@HIDDEN>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: 0.2 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.8 (/)

On Mon, Apr 3, 2023 at 11:47=E2=80=AFPM Paul Eggert <eggert@HIDDEN> wr=
ote:
> On 2023-04-03 20:30, Jim Meyering wrote:
> > have you seen justification
> > (other than for compatibility with some other tool or language) for
> > allowing \d to match non-ASCII by default, in spite of the risks?
>
> In the example =C3=86var supplied in <https://bugs.gnu.org/60690>, my
> impression was that it was better when \d matched non-ASCII digits. That
> is, in a UTF-8 locale it's better when \d finds matches in these lines:
>
> >>      > git-gui/po/ja.po:"- =E7=AC=AC=EF=BC=91=E8=A1=8C: =E4=BD=95=E3=
=82=92=E3=81=97=E3=81=9F=E3=81=8B=E3=80=81=E3=82=92=EF=BC=91=E8=A1=8C=E3=81=
=A7=E8=A6=81=E7=B4=84=E3=80=82\n"
> >>      > git-gui/po/ja.po:"- =E7=AC=AC=EF=BC=92=E8=A1=8C: =E7=A9=BA=E7=
=99=BD\n"
>
> because they contain the Japanese digits "=EF=BC=91" and "=EF=BC=92". Thi=
s was the only
> example I recall being given.

Before it was unintentionally enabled in grep-3.9, lines like that have
never been matched by grep -P's '\d'. By relaxing \d, we'd weaken
any application that uses say grep -P '^\d+$' to perform input
validation intending to ensure that some input is all ASCII digits.
It's not a big stretch to imagine that some downstream processor
of that "verified" data is not prepared to deal with multi-byte digits.

> Also, I find it odd that grep -P '^[\w\d]*$' matches lines containing
> any sort of Arabic word characters, but it rejects lines containing
> Arabic digits like "=D9=A3" that are perfectly reasonable in Arabic-langu=
age
> text. I also find it odd that [\d] and [[:digit:]] mean different things.
>
> There are arguments on the other side, otherwise we wouldn't be having
> this discussion. And it's true that grep -P '\d' formerly rejected
> Arabic digits (though it's also true that grep -P '\w' formerly rejected
> Arabic letters...). Still, the cure's oddness and incompatibility with
> Git, Perl, etc. appears to me to be worse than the disease of dealing
> with grep -P invocations that need to use [0-9] or LC_ALL=3D"C" anyway if
> they want to be portable to any program other than GNU grep.

I'm primarily concerned about not introducing a persistent regression in
how GNU grep's -P '\d' works in multibyte locales. The corner cases you
mention do matter, of course, but are far less likely to matter in practice=
.

Message sent to bug-grep@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#60690: -P '\d' in GNU and git grep
Resent-From: Paul Eggert <eggert@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-grep@HIDDEN
Resent-Date: Tue, 04 Apr 2023 18:27:01 +0000
Resent-Message-ID: <handler.60690.B60690.16806327724620 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 60690
X-GNU-PR-Package: grep
X-GNU-PR-Keywords: patch
To: Carlo Arenas <carenas@HIDDEN>
Cc: demerphq@HIDDEN, 60690 <at> debbugs.gnu.org, mega lith01 <megalith01@HIDDEN>, =?UTF-8?Q?=C3=86var_?= =?UTF-8?Q?Arnfj=C3=B6r=C3=B0?= Bjarmason <avarab@HIDDEN>, gitster@HIDDEN, pcre-dev@HIDDEN, =?UTF-8?Q?Tukusej=E2=80=99s?= Sirs <tukusejssirs@HIDDEN>, git@HIDDEN
Received: via spool by 60690-submit <at> debbugs.gnu.org id=B60690.16806327724620
          (code B ref 60690); Tue, 04 Apr 2023 18:27:01 +0000
Received: (at 60690) by debbugs.gnu.org; 4 Apr 2023 18:26:12 +0000
Received: from localhost ([127.0.0.1]:50179 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1pjlMB-0001CS-Q4
	for submit <at> debbugs.gnu.org; Tue, 04 Apr 2023 14:26:12 -0400
Received: from mail.cs.ucla.edu ([131.179.128.66]:39920)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eggert@HIDDEN>) id 1pjlM7-0001Bs-BR
 for 60690 <at> debbugs.gnu.org; Tue, 04 Apr 2023 14:26:10 -0400
Received: from localhost (localhost [127.0.0.1])
 by mail.cs.ucla.edu (Postfix) with ESMTP id 999673C09FA04;
 Tue,  4 Apr 2023 11:26:00 -0700 (PDT)
Received: from mail.cs.ucla.edu ([127.0.0.1])
 by localhost (mail.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10032)
 with ESMTP id 95-7Sq2pH7TB; Tue,  4 Apr 2023 11:26:00 -0700 (PDT)
Received: from localhost (localhost [127.0.0.1])
 by mail.cs.ucla.edu (Postfix) with ESMTP id 57B053C09FA08;
 Tue,  4 Apr 2023 11:26:00 -0700 (PDT)
DKIM-Filter: OpenDKIM Filter v2.10.3 mail.cs.ucla.edu 57B053C09FA08
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cs.ucla.edu;
 s=9D0B346E-2AEB-11ED-9476-E14B719DCE6C; t=1680632760;
 bh=FYW4ECUW8MUCylXzUhT21AzRYAuNwByZMArOjfDZ+6A=;
 h=Message-ID:Date:MIME-Version:To:From;
 b=HYhxH1aNvKIJjxXiUlUK7/K6lLKOMOLMCVz/ZSwngWyMpluC1Reh+RB6VOYjmggNR
 uVCF237LceyP1veG3HZBJr3Rfe5S7tojXIi3WOP3BqEHg8j2frOmu1x87sacvnNBWl
 InfUvQzVvzvyr4FmPv4dkUtzsIs0IKRQ45/0CK/1c3D4NcVPBLv5oRYKqEIDOaitkc
 UCoUe9JG5vKjI8QxiD+fQIBIPNXxMyqVbFIjr42xitME9nYQ+g1uRYh7xOkCsCiXd0
 Xt3yrvgUGMGnYqIyCusnaGmuWuVqhld4j4DhdFQNhkpP91bjYE7LOnCWos3JONtGEd
 2/9x9rz3yJnEg==
X-Virus-Scanned: amavisd-new at mail.cs.ucla.edu
Received: from mail.cs.ucla.edu ([127.0.0.1])
 by localhost (mail.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10026)
 with ESMTP id H9BFLE10cWjv; Tue,  4 Apr 2023 11:26:00 -0700 (PDT)
Received: from [131.179.64.200] (Penguin.CS.UCLA.EDU [131.179.64.200])
 by mail.cs.ucla.edu (Postfix) with ESMTPSA id 2DB283C09FA04;
 Tue,  4 Apr 2023 11:26:00 -0700 (PDT)
Message-ID: <96358c4e-7200-e5a5-869e-5da9d0de3503@HIDDEN>
Date: Tue, 4 Apr 2023 11:25:59 -0700
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.9.1
Content-Language: en-US
References: <2554712d-e386-3bab-bc6c-1f0e85d999db@HIDDEN>
 <CAPUEspj1m6F0_XgOFUVaq3Aq_Ah3PzCUs7YUyFH9_Zz-MOYTTA@HIDDEN>
From: Paul Eggert <eggert@HIDDEN>
Organization: UCLA Computer Science Department
In-Reply-To: <CAPUEspj1m6F0_XgOFUVaq3Aq_Ah3PzCUs7YUyFH9_Zz-MOYTTA@HIDDEN>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -1.1 (-)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -2.1 (--)

On 4/3/23 23:56, Carlo Arenas wrote:
> On Mon, Apr 3, 2023 at 2:38=E2=80=AFPM Paul Eggert <eggert@HIDDEN>=
 wrote:
>>
>> on March 23 Git disabled
>> the use of PCRE2_UCP in PCRE2 10.34 or earlier[6], due to a PCRE2 bug
>> that can cause a crash when PCRE2_UCP is used[7]. A bug fix[8] should
>> appear in the next PCRE2 release.
>=20
> Presume PCRE2 is a typo and should have been "git" here?

No, I was talking about what options Git uses when it calls PCRE2=20
functions. In other words, this is about whether GNU 'grep -P' should be=20
compatible with 'git grep -P' (as well as with Perl and with pcregrep),=20
when interpreting \d and similar constructs.

This is an evolving area. Git master is fiddling with flags and options,=20
and so is GNU grep master, and so is PCRE2, and there are bugs. If=20
you're running bleeding-edge versions of this code you'll get different=20
behavior than if you're running grep 3.8, pcregrep 8.45, Perl 5.36, and=20
git 2.39.2 (which is what Fedora 37 has).

What I'm fearing is that we may evolve into mutually incompatible=20
interpretations of how Perl regular expressions deal with UTF-8 text.=20
That'd be a recipe for confusion down the road.

Message sent to bug-grep@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#60690: -P '\d' in GNU and git grep
Resent-From: Junio C Hamano <gitster@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-grep@HIDDEN
Resent-Date: Tue, 04 Apr 2023 19:32:02 +0000
Resent-Message-ID: <handler.60690.B60690.168063672121354 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 60690
X-GNU-PR-Package: grep
X-GNU-PR-Keywords: patch
To: Paul Eggert <eggert@HIDDEN>
Cc: demerphq@HIDDEN, 60690 <at> debbugs.gnu.org, mega lith01 <megalith01@HIDDEN>, Carlo Arenas <carenas@HIDDEN>, =?UTF-8?Q?=C3=86var_?= =?UTF-8?Q?Arnfj=C3=B6r=C3=B0?= Bjarmason <avarab@HIDDEN>, pcre-dev@HIDDEN, =?UTF-8?Q?Tukusej=E2=80=99s?= Sirs <tukusejssirs@HIDDEN>, git@HIDDEN
Received: via spool by 60690-submit <at> debbugs.gnu.org id=B60690.168063672121354
          (code B ref 60690); Tue, 04 Apr 2023 19:32:02 +0000
Received: (at 60690) by debbugs.gnu.org; 4 Apr 2023 19:32:01 +0000
Received: from localhost ([127.0.0.1]:50254 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1pjmNt-0005YJ-9l
	for submit <at> debbugs.gnu.org; Tue, 04 Apr 2023 15:32:01 -0400
Received: from mail-pl1-f176.google.com ([209.85.214.176]:40570)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <jch2355@HIDDEN>) id 1pjmNq-0005Xz-Dc
 for 60690 <at> debbugs.gnu.org; Tue, 04 Apr 2023 15:31:59 -0400
Received: by mail-pl1-f176.google.com with SMTP id u10so32318069plz.7
 for <60690 <at> debbugs.gnu.org>; Tue, 04 Apr 2023 12:31:58 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=gmail.com; s=20210112; t=1680636712;
 h=mime-version:user-agent:message-id:in-reply-to:date:references
 :subject:cc:to:from:sender:from:to:cc:subject:date:message-id
 :reply-to; bh=6u+YUnimTHNqsVP7F3dghDPdmr+RHhSOP18RibhhSu8=;
 b=P0SA8CYKyvL8Q0TRQ9El2yEbBx7T3paBdY7iEvLp+1f9C5AP9YUhWNh5RPCvOjVzpZ
 2Jn9G/vaXknKDJ37lrtH8T7HXNWs/Syi8Y3vt1pTm80K2s34qgwSRdSkEWAqs/0YXy75
 kZQATSCd+nKm/Gp4pqmpE80Onyds2cSsGFLcU/zFGDGyBPiW6/Oe93oRIGhWDnTrSRq9
 2lO8qmyoaaH+FVXIuS+nhXbjx+772OswBd3q6Pgvxrlr7zxtFEihe0kPeVMvlEolH7C1
 bdjlriax5kfz6mm8kaUy10CEB0LWseVihV7NINZtCPduKJAJP2FhoRAveu2UaslSyhcQ
 WA8Q==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20210112; t=1680636712;
 h=mime-version:user-agent:message-id:in-reply-to:date:references
 :subject:cc:to:from:sender:x-gm-message-state:from:to:cc:subject
 :date:message-id:reply-to;
 bh=6u+YUnimTHNqsVP7F3dghDPdmr+RHhSOP18RibhhSu8=;
 b=38mxy/+8zX2X0UNBQYB2VEs0r1bK1v8Spu/ldRgA23gUHKSoiQcRbZaciMUwSnS78E
 PL+3bFUJC/LdX7yGlViqsgBpBSjkbSpRA60UNDmoWUOK+LQuh2sBYTN5VfKbLtQITLA4
 8YVJQx+pi10IDdNIAUlACK5ru76vxN71jg1AfYwX6vpfs2pgURyGhbIrX1pHlu10x7o5
 tGPO+1fe4agKHYLnUXaalvl3wtJFnspKmK++gkoRofbS6J3OVuMIm0TNT1Hx3r5xRoV4
 /nWm5zMtvmN3D9wPlOjYYVvQWPG7nfXCUof9CgBzUeDFiZQ+8DR2/5qAxkWDyAR+g5bn
 +f8g==
X-Gm-Message-State: AAQBX9fApm75vCLqVpWjtEHQVfoCUSSFcrJGHHVsLNEfaNoDA1auI2hk
 ZLQ6HmszTKjSvVATci9NHbg=
X-Google-Smtp-Source: AKy350btd5q4jt7/5/l0+bfz8VRalTaP/tRnSwu4zFULoE+u6flcSrAWNJc8KxHe7hKmdCnkPhiEaQ==
X-Received: by 2002:a05:6a20:619a:b0:e3:9d4e:b340 with SMTP id
 x26-20020a056a20619a00b000e39d4eb340mr3460394pzd.12.1680636711819; 
 Tue, 04 Apr 2023 12:31:51 -0700 (PDT)
Received: from localhost (254.80.82.34.bc.googleusercontent.com.
 [34.82.80.254]) by smtp.gmail.com with ESMTPSA id
 e5-20020a62ee05000000b005e099d7c30bsm8971154pfi.205.2023.04.04.12.31.51
 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
 Tue, 04 Apr 2023 12:31:51 -0700 (PDT)
From: Junio C Hamano <gitster@HIDDEN>
References: <2554712d-e386-3bab-bc6c-1f0e85d999db@HIDDEN>
 <CAPUEspj1m6F0_XgOFUVaq3Aq_Ah3PzCUs7YUyFH9_Zz-MOYTTA@HIDDEN>
 <96358c4e-7200-e5a5-869e-5da9d0de3503@HIDDEN>
Date: Tue, 04 Apr 2023 12:31:51 -0700
In-Reply-To: <96358c4e-7200-e5a5-869e-5da9d0de3503@HIDDEN> (Paul Eggert's
 message of "Tue, 4 Apr 2023 11:25:59 -0700")
Message-ID: <xmqqttxvzbo8.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: 0.5 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.5 (/)

Paul Eggert <eggert@HIDDEN> writes:

> This is an evolving area. Git master is fiddling with flags and
> options, and so is GNU grep master, and so is PCRE2, and there are
> bugs. If you're running bleeding-edge versions of this code you'll get
> different behavior than if you're running grep 3.8, pcregrep 8.45,
> Perl 5.36, and git 2.39.2 (which is what Fedora 37 has).
>
> What I'm fearing is that we may evolve into mutually incompatible
> interpretations of how Perl regular expressions deal with UTF-8
> text. That'd be a recipe for confusion down the road.

Nicely said.  My personal inclination is to let Perl folks decide
and follow them (even though I am skeptical about the wisdom of
letting '\d' match anything other than [0-9]), but even in Git
circle there would be different opinions, so I am glad that the
discussion is visible on the list to those who are intrested.

Message sent to bug-grep@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#60690: -P '\d' in GNU and git grep
Resent-From: Paul Eggert <eggert@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-grep@HIDDEN
Resent-Date: Wed, 05 Apr 2023 18:33:02 +0000
Resent-Message-ID: <handler.60690.B60690.168071956818069 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 60690
X-GNU-PR-Package: grep
X-GNU-PR-Keywords: patch
To: Junio C Hamano <gitster@HIDDEN>
Cc: demerphq@HIDDEN, Philip.Hazel@HIDDEN, 60690 <at> debbugs.gnu.org, mega lith01 <megalith01@HIDDEN>, Carlo Arenas <carenas@HIDDEN>, =?UTF-8?Q?=C3=86var_?= =?UTF-8?Q?Arnfj=C3=B6r=C3=B0?= Bjarmason <avarab@HIDDEN>, pcre-dev@HIDDEN, =?UTF-8?Q?Tukusej=E2=80=99s?= Sirs <tukusejssirs@HIDDEN>, git@HIDDEN
Received: via spool by 60690-submit <at> debbugs.gnu.org id=B60690.168071956818069
          (code B ref 60690); Wed, 05 Apr 2023 18:33:02 +0000
Received: (at 60690) by debbugs.gnu.org; 5 Apr 2023 18:32:48 +0000
Received: from localhost ([127.0.0.1]:52628 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1pk7w7-0004hN-Hd
	for submit <at> debbugs.gnu.org; Wed, 05 Apr 2023 14:32:47 -0400
Received: from mail.cs.ucla.edu ([131.179.128.66]:48256)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eggert@HIDDEN>) id 1pk7w5-0004h8-Ku
 for 60690 <at> debbugs.gnu.org; Wed, 05 Apr 2023 14:32:46 -0400
Received: from localhost (localhost [127.0.0.1])
 by mail.cs.ucla.edu (Postfix) with ESMTP id 76A843C09FA00;
 Wed,  5 Apr 2023 11:32:39 -0700 (PDT)
Received: from mail.cs.ucla.edu ([127.0.0.1])
 by localhost (mail.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10032)
 with ESMTP id lgOUApS0xxtS; Wed,  5 Apr 2023 11:32:39 -0700 (PDT)
Received: from localhost (localhost [127.0.0.1])
 by mail.cs.ucla.edu (Postfix) with ESMTP id 2221F3C09FA03;
 Wed,  5 Apr 2023 11:32:39 -0700 (PDT)
DKIM-Filter: OpenDKIM Filter v2.10.3 mail.cs.ucla.edu 2221F3C09FA03
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cs.ucla.edu;
 s=9D0B346E-2AEB-11ED-9476-E14B719DCE6C; t=1680719559;
 bh=bRYhVoaglb+oHA8EWDW1Fqok6GB2iW6NSs3OaO/A+wY=;
 h=Message-ID:Date:MIME-Version:To:From;
 b=ddPN/A9we4BUHk9jtuP3e/d7yYjuzKL+mI1OKkx4N7NuG6ksZPWNUvHGSbLckV00L
 fLV9O7BD0cvUUSIjRGHGHSSMCwYTWdutl0Ed8r9ye/L7asLcyjEcy77TrD62thRZXQ
 Op/3jqZAHJcJsSbP8ejjthm4wXf9u7OkNgr2F5yDYT0JWT7DOiW0auT6SAYXu234CV
 G/0KL1akOCluJBnF2XtLl+t6ftlrAIzfX48LqlSEI129fMNS/D/DPqJeM4nEHD595X
 2HI54efP8ArWRj//LA2JVhJaYfDJAk8qP/4ga/l0n5XJCPagkY6mjgfDMJ9Ooxj4A4
 ArTlLflXvcBtw==
X-Virus-Scanned: amavisd-new at mail.cs.ucla.edu
Received: from mail.cs.ucla.edu ([127.0.0.1])
 by localhost (mail.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10026)
 with ESMTP id L7eQ5AvG5bMC; Wed,  5 Apr 2023 11:32:39 -0700 (PDT)
Received: from [192.168.1.9] (cpe-172-91-119-151.socal.res.rr.com
 [172.91.119.151])
 by mail.cs.ucla.edu (Postfix) with ESMTPSA id D20763C09FA00;
 Wed,  5 Apr 2023 11:32:38 -0700 (PDT)
Message-ID: <6d86214a-1b80-eb88-1efb-36e61fd3203e@HIDDEN>
Date: Wed, 5 Apr 2023 11:32:38 -0700
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.9.0
Content-Language: en-US
References: <2554712d-e386-3bab-bc6c-1f0e85d999db@HIDDEN>
 <CAPUEspj1m6F0_XgOFUVaq3Aq_Ah3PzCUs7YUyFH9_Zz-MOYTTA@HIDDEN>
 <96358c4e-7200-e5a5-869e-5da9d0de3503@HIDDEN>
 <xmqqttxvzbo8.fsf@HIDDEN>
From: Paul Eggert <eggert@HIDDEN>
Organization: UCLA Computer Science Department
In-Reply-To: <xmqqttxvzbo8.fsf@HIDDEN>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -1.1 (-)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -2.1 (--)

On 2023-04-04 12:31, Junio C Hamano wrote:

> My personal inclination is to let Perl folks decide
> and follow them (even though I am skeptical about the wisdom of
> letting '\d' match anything other than [0-9])

I looked into what pcre2grep does. It has always done only 8-bit=20
processing unless you use the -u or --utf option, so plain "pcre2grep=20
'\d'" matches only ASCII digits.

Although this causes pcre2grep to mishandle Unicode characters:

   $ echo '=C3=86var' | pcre2grep '[Ss=C3=9F]'
   =C3=86var

it mimics Perl 5.36:

   $ echo '=C3=86var' | perl -ne 'print $_ if /[Ss=C3=9F]/'
   =C3=86var

so this seems to be what Perl users expect, despite its infelicities.

For better Unicode handling one can use pcre2grep's -u or --utf option,=20
which causes pcre2grep to behave more like GNU grep -P and git grep -P:=20
"echo '=C3=86var' | pcre2grep -u '[Ss=C3=9F]'" outputs nothing, which I t=
hink is=20
what most people would expect (unless they're Perl users :-).

Neither git grep -P nor the current release of pcre2grep -u have \d=20
matching non-ASCII digits, because they do not use PCRE2_UCP. However,=20
in a February 8 commit[1], Philip Hazel changed pcre2grep to use=20
PCRE2_UCP, so this will mean 10.43 pcre2grep -u will behave like 3.9 GNU=20
grep -P did (though 3.10 has changed this).

That February commit also added a --no-ucp option, to disable PCRE2_UCP.=20
So as I understand it, if you're in a UTF-8 locale:

* 10.43 pcre2grep -u will behave like 3.9 GNU grep -P.

* 10.43 pcre2grep -u --no-ucp will behave like git grep -P.

* Current GNU grep -P is different from everybody else.

This incompatibility is not good.

Here are two ways forward to fix this incompatibility (there are other=20
possibilities of course):

(A) GNU grep adds a --no-ucp option that acts like 10.43 pcre2grep=20
--no-ucp, and git grep -P follows suit. That is, both GNU and git grep=20
act like 10.43 pcre2grep -u, in that they enable PCRE2_UTF, and also=20
enable PCRE2_UCP unless --no-ucp is given. This would cause \d to match=20
non-ASCII digits unless --no-ucp is given.

(B) GNU grep -P and git grep -P mimic pcre2grep in both -u and --no-ucp.=20
That is, they would both do 8-bit-only by default, and use PCRE2_UTF=20
only when -u or --utf is given, and use PCRE2_UCP only when --no-ucp is=20
absent. This would cause \d to match non-ASCII digits only when -u is=20
given but --no-ucp is not.

Under either (A) or (B), future pcre2grep -u, GNU grep -P, and git grep=20
-P would be consistent.

I mildly prefer (B) but (A) would also work. (One advantage of (B) is=20
that it should be faster....)

[1]:=20
https://github.com/PCRE2Project/pcre2/commit/8385df8c97b6f8069a48e600c7e4=
e94cc3e3ebd9ht

Message sent to bug-grep@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#60690: -P '\d' in GNU and git grep
Resent-From: Paul Eggert <eggert@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-grep@HIDDEN
Resent-Date: Wed, 05 Apr 2023 19:05:02 +0000
Resent-Message-ID: <handler.60690.B60690.168072147822173 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 60690
X-GNU-PR-Package: grep
X-GNU-PR-Keywords: patch
To: Junio C Hamano <gitster@HIDDEN>
Cc: demerphq@HIDDEN, Philip.Hazel@HIDDEN, 60690 <at> debbugs.gnu.org, mega lith01 <megalith01@HIDDEN>, Carlo Arenas <carenas@HIDDEN>, =?UTF-8?Q?=C3=86var_?= =?UTF-8?Q?Arnfj=C3=B6r=C3=B0?= Bjarmason <avarab@HIDDEN>, git@HIDDEN, =?UTF-8?Q?Tukusej=E2=80=99s?= Sirs <tukusejssirs@HIDDEN>, pcre-dev@HIDDEN
Received: via spool by 60690-submit <at> debbugs.gnu.org id=B60690.168072147822173
          (code B ref 60690); Wed, 05 Apr 2023 19:05:02 +0000
Received: (at 60690) by debbugs.gnu.org; 5 Apr 2023 19:04:38 +0000
Received: from localhost ([127.0.0.1]:52670 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1pk8Qw-0005lZ-Bf
	for submit <at> debbugs.gnu.org; Wed, 05 Apr 2023 15:04:38 -0400
Received: from mail.cs.ucla.edu ([131.179.128.66]:44430)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eggert@HIDDEN>) id 1pk8Qt-0005lJ-Q2
 for 60690 <at> debbugs.gnu.org; Wed, 05 Apr 2023 15:04:36 -0400
Received: from localhost (localhost [127.0.0.1])
 by mail.cs.ucla.edu (Postfix) with ESMTP id 7CC873C09FA00;
 Wed,  5 Apr 2023 12:04:29 -0700 (PDT)
Received: from mail.cs.ucla.edu ([127.0.0.1])
 by localhost (mail.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10032)
 with ESMTP id RQFYaT8pCWkd; Wed,  5 Apr 2023 12:04:29 -0700 (PDT)
Received: from localhost (localhost [127.0.0.1])
 by mail.cs.ucla.edu (Postfix) with ESMTP id 3F3603C09FA02;
 Wed,  5 Apr 2023 12:04:29 -0700 (PDT)
DKIM-Filter: OpenDKIM Filter v2.10.3 mail.cs.ucla.edu 3F3603C09FA02
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cs.ucla.edu;
 s=9D0B346E-2AEB-11ED-9476-E14B719DCE6C; t=1680721469;
 bh=A5QqC6pTJiuMWmHDn0PeTFbe/cing73GoNhYwwNMCXk=;
 h=Message-ID:Date:MIME-Version:From:To;
 b=Kshsl4GHwHgUO4jd/s3/C8xBzgZPwTnyInFsabrj1l3fg9P69ZaIKk7zsHF+Osx4T
 dWG6MNbxOt5BTc6303ooOhQIXsrR19858BZzNjyoaeZRYyU5ilMjeXg2T9NV0C5uxP
 029lCtfR6OAEJy6TmMSpAQWknN76bRdRniFGNnGLhw6TIkkcOj96qsqdGLTMyKard0
 TVvdCmmbCgetrxK/5njbSxnEEVn/beLt86vpVKmbDGhWBNuIuPZOIenjrjsTKjPf6q
 vVl192QJ+rjcuGPxW9/M5JqLBt20Tk+YvP6aDRN1KnijKorU69GPfgEqSaHZBRH5kH
 HYuFDvX6+NF8Q==
X-Virus-Scanned: amavisd-new at mail.cs.ucla.edu
Received: from mail.cs.ucla.edu ([127.0.0.1])
 by localhost (mail.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10026)
 with ESMTP id IigyzgVcjc-m; Wed,  5 Apr 2023 12:04:29 -0700 (PDT)
Received: from [192.168.1.9] (cpe-172-91-119-151.socal.res.rr.com
 [172.91.119.151])
 by mail.cs.ucla.edu (Postfix) with ESMTPSA id EA1B33C09FA00;
 Wed,  5 Apr 2023 12:04:28 -0700 (PDT)
Message-ID: <33b3eb15-73e2-8004-9f06-19e5ec5c5877@HIDDEN>
Date: Wed, 5 Apr 2023 12:04:28 -0700
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.9.0
Content-Language: en-US
From: Paul Eggert <eggert@HIDDEN>
References: <2554712d-e386-3bab-bc6c-1f0e85d999db@HIDDEN>
 <CAPUEspj1m6F0_XgOFUVaq3Aq_Ah3PzCUs7YUyFH9_Zz-MOYTTA@HIDDEN>
 <96358c4e-7200-e5a5-869e-5da9d0de3503@HIDDEN>
 <xmqqttxvzbo8.fsf@HIDDEN>
 <6d86214a-1b80-eb88-1efb-36e61fd3203e@HIDDEN>
Organization: UCLA Computer Science Department
In-Reply-To: <6d86214a-1b80-eb88-1efb-36e61fd3203e@HIDDEN>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Spam-Score: -1.1 (-)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -2.1 (--)

On 2023-04-05 11:32, Paul Eggert wrote:

> in a February 8 commit[1], Philip Hazel changed pcre2grep to use 
> PCRE2_UCP, so this will mean 10.43 pcre2grep -u will behave like 3.9 GNU 
> grep -P did (though 3.10 has changed this).

Sorry, due to fumblefingers I gave the wrong URL for [1]. Here's a 
corrected URL:

https://github.com/PCRE2Project/pcre2/commit/8385df8c97b6f8069a48e600c7e4e94cc3e3ebd9

It also mentions a new --case-restrict option, intended for 10.43 
pcre2grep. Given Perl's and PCRE2's plethora of options I suppose one 
could imagine several other options of that ilk.

Message sent to bug-grep@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#60690: -P '\d' in GNU and git grep
Resent-From: Junio C Hamano <gitster@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-grep@HIDDEN
Resent-Date: Wed, 05 Apr 2023 19:38:01 +0000
Resent-Message-ID: <handler.60690.B60690.168072347825594 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 60690
X-GNU-PR-Package: grep
X-GNU-PR-Keywords: patch
To: Paul Eggert <eggert@HIDDEN>
Cc: demerphq@HIDDEN, Philip.Hazel@HIDDEN, 60690 <at> debbugs.gnu.org, mega lith01 <megalith01@HIDDEN>, Carlo Arenas <carenas@HIDDEN>, =?UTF-8?Q?=C3=86var_?= =?UTF-8?Q?Arnfj=C3=B6r=C3=B0?= Bjarmason <avarab@HIDDEN>, pcre-dev@HIDDEN, =?UTF-8?Q?Tukusej=E2=80=99s?= Sirs <tukusejssirs@HIDDEN>, git@HIDDEN
Received: via spool by 60690-submit <at> debbugs.gnu.org id=B60690.168072347825594
          (code B ref 60690); Wed, 05 Apr 2023 19:38:01 +0000
Received: (at 60690) by debbugs.gnu.org; 5 Apr 2023 19:37:58 +0000
Received: from localhost ([127.0.0.1]:52683 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1pk8xB-0006ek-Kk
	for submit <at> debbugs.gnu.org; Wed, 05 Apr 2023 15:37:57 -0400
Received: from mail-pj1-f42.google.com ([209.85.216.42]:43918)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <jch2355@HIDDEN>) id 1pk8xA-0006eX-Eq
 for 60690 <at> debbugs.gnu.org; Wed, 05 Apr 2023 15:37:57 -0400
Received: by mail-pj1-f42.google.com with SMTP id
 lr16-20020a17090b4b9000b0023f187954acso38353189pjb.2
 for <60690 <at> debbugs.gnu.org>; Wed, 05 Apr 2023 12:37:56 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=gmail.com; s=20210112; t=1680723470;
 h=content-transfer-encoding:mime-version:user-agent:message-id
 :in-reply-to:date:references:subject:cc:to:from:sender:from:to:cc
 :subject:date:message-id:reply-to;
 bh=sTfI/fZISXlpPkW/OBzm8tUQj2Dur1rRA4qGlvG70uM=;
 b=f1+JvFrG4DD6jt/4TOFG1ZgMBT/x/hxOk2oMMZjC09iK/Hc+lFbYdyMimmRxjKxtFl
 wq8+Mu9yV/Sg7igX9TIQ1BCyitEJWo4XyX0caxB/JilRdUhH16GSXEycfie/ouc1DUxO
 Cp3ooHyOM/Zgg69byygJmL7kBrYxawNALNTyEiuJi/fOsX1WPoXIO+p1Vx7T9FzUVm2s
 3vYKG0hzRKSfZdGMvHSW0K7m3YS8M+tAf1cHSBBKlCmVFklW3GII1tKLAqQgxyn5NLtj
 h3VgkIYDneuYMuA8G4miZl2CWEkhT6qGPQxbZxc7DjFZvtrK/XBJ+caX7EgCRV/E9/wk
 s0EA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20210112; t=1680723470;
 h=content-transfer-encoding:mime-version:user-agent:message-id
 :in-reply-to:date:references:subject:cc:to:from:sender
 :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to;
 bh=sTfI/fZISXlpPkW/OBzm8tUQj2Dur1rRA4qGlvG70uM=;
 b=fZO3CI2IhW+IhHkcvO+EgmkKzkpyI66t00X89c5VxQeGpCpqp8NSQ0l+qWMZSkViGV
 WPatBjnjWS2TpR47DEwBu8zaf4y2UT/bHP3qiQoj0vCSAaSbDVyR9jxz2QY4OCYoM9+d
 F2U+A7L/nOKAXetYy7+AZpWNh2gkDO2Y1Mdo5TPfOKYmoAlgphLTIVdZIH+Ueg4ygPBJ
 exINC3rFBqiwdjrp7JncJwN2bkAsxcvLtDAvM5rSlQ4hWBLnHJP85GLdOcuBiQKHFYmL
 gB0wxBgAp1te/KfWYApPfvoMiaqiiN/6wPOFaNdJbFCX3MM4zTx3RLBP8PaXTJe28IgS
 G+jQ==
X-Gm-Message-State: AAQBX9dPJQkVl0aYn2q4xQYoKoGTccRoin4wbd04O9EEHbSdOOwbtOJk
 qOwRLTOL/Q9/5eLVairqiUA=
X-Google-Smtp-Source: AKy350YXcN80SrkZsGvKU62pKhIdePWRYuIHZk0iDdgSIYaikoeedNI1AvNAT+UloODxJnzxnS4JGQ==
X-Received: by 2002:a17:902:e884:b0:19a:a520:b203 with SMTP id
 w4-20020a170902e88400b0019aa520b203mr9034298plg.25.1680723470325; 
 Wed, 05 Apr 2023 12:37:50 -0700 (PDT)
Received: from localhost (254.80.82.34.bc.googleusercontent.com.
 [34.82.80.254]) by smtp.gmail.com with ESMTPSA id
 p10-20020a170902a40a00b001a01bb92273sm10432863plq.279.2023.04.05.12.37.49
 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
 Wed, 05 Apr 2023 12:37:49 -0700 (PDT)
From: Junio C Hamano <gitster@HIDDEN>
References: <2554712d-e386-3bab-bc6c-1f0e85d999db@HIDDEN>
 <CAPUEspj1m6F0_XgOFUVaq3Aq_Ah3PzCUs7YUyFH9_Zz-MOYTTA@HIDDEN>
 <96358c4e-7200-e5a5-869e-5da9d0de3503@HIDDEN>
 <xmqqttxvzbo8.fsf@HIDDEN>
 <6d86214a-1b80-eb88-1efb-36e61fd3203e@HIDDEN>
Date: Wed, 05 Apr 2023 12:37:49 -0700
In-Reply-To: <6d86214a-1b80-eb88-1efb-36e61fd3203e@HIDDEN> (Paul Eggert's
 message of "Wed, 5 Apr 2023 11:32:38 -0700")
Message-ID: <xmqqlej6unle.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
X-Spam-Score: 0.5 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.5 (/)

Paul Eggert <eggert@HIDDEN> writes:

> Here are two ways forward to fix this incompatibility (there are other
> possibilities of course):
>
> (A) GNU grep adds a --no-ucp option that acts like 10.43 pcre2grep
> --no-ucp, and git grep -P follows suit. That is, both GNU and git grep
> act like 10.43 pcre2grep -u, in that they enable PCRE2_UTF, and also
> enable PCRE2_UCP unless --no-ucp is given. This would cause \d to
> match non-ASCII digits unless --no-ucp is given.
>
> (B) GNU grep -P and git grep -P mimic pcre2grep in both -u and
> --no-ucp. That is, they would both do 8-bit-only by default, and use
> PCRE2_UTF only when -u or --utf is given, and use PCRE2_UCP only when
> --no-ucp is absent. This would cause \d to match non-ASCII digits only
> when -u is given but --no-ucp is not.
>
> Under either (A) or (B), future pcre2grep -u, GNU grep -P, and git
> grep -P would be consistent.
>
> I mildly prefer (B) but (A) would also work. (One advantage of (B) is
> that it should be faster....)

For "git grep -P", I would like to hear from Carlo and Ævar; I agree
both (A) and (B) would be workable solutions, and have a slight
preference on a solution that does not add more options that take
only in effect when -P is given, simply because these options are
cumbersome to document and explain, but that is a very minor point.

Thanks.

Message sent to bug-grep@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#60690: -P '\d' in GNU and git grep
Resent-From: Jim Meyering <jim@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-grep@HIDDEN
Resent-Date: Wed, 05 Apr 2023 19:41:02 +0000
Resent-Message-ID: <handler.60690.B60690.168072364325899 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 60690
X-GNU-PR-Package: grep
X-GNU-PR-Keywords: patch
To: Paul Eggert <eggert@HIDDEN>
Cc: demerphq@HIDDEN, Philip.Hazel@HIDDEN, 60690 <at> debbugs.gnu.org, mega lith01 <megalith01@HIDDEN>, Carlo Arenas <carenas@HIDDEN>, =?UTF-8?Q?=C3=86var_?= =?UTF-8?Q?Arnfj=C3=B6r=C3=B0?= Bjarmason <avarab@HIDDEN>, git@HIDDEN, Junio C Hamano <gitster@HIDDEN>, =?UTF-8?Q?Tukusej=E2=80=99s?= Sirs <tukusejssirs@HIDDEN>, pcre-dev@HIDDEN
Received: via spool by 60690-submit <at> debbugs.gnu.org id=B60690.168072364325899
          (code B ref 60690); Wed, 05 Apr 2023 19:41:02 +0000
Received: (at 60690) by debbugs.gnu.org; 5 Apr 2023 19:40:43 +0000
Received: from localhost ([127.0.0.1]:52689 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1pk8zr-0006je-7L
	for submit <at> debbugs.gnu.org; Wed, 05 Apr 2023 15:40:43 -0400
Received: from mail-lj1-f174.google.com ([209.85.208.174]:40749)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <meyering@HIDDEN>) id 1pk8zm-0006jM-2T
 for 60690 <at> debbugs.gnu.org; Wed, 05 Apr 2023 15:40:42 -0400
Received: by mail-lj1-f174.google.com with SMTP id s20so18116840ljp.7
 for <60690 <at> debbugs.gnu.org>; Wed, 05 Apr 2023 12:40:38 -0700 (PDT)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20210112; t=1680723632;
 h=content-transfer-encoding:cc:to:subject:message-id:date:from
 :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc
 :subject:date:message-id:reply-to;
 bh=wq98D7+2AkYfDLkf1OqLxyRym/GbLikvkppAQQaLhwc=;
 b=MDPpE0NqlWRz5EDQXnGR/f+n/9LtVDPcAGRkfAB52JB+Q891RPQA2kezYdzUJJ6QDZ
 +0ROypcNDLbrf+HgSN16na6XCErFFOJOE++VE6tvmILsg2HIQ41WQ267ttpHAqj7J06g
 ybfI7AdfdcN2zMZ8PrIwSgVhDd0CNys2FshXcs4xH4nAVUh3gzqDK0oa+IPRBO7RRWmu
 0kp4bDeK8WVfwVh6Jn6xR7HhdJw0qmP7NKyYPQAEVPX9F3N2RKrwV3PPthn7zEH//KOZ
 GsIJOG5Z3JktNDD+vwZPUFDuva1nV156CJscnKYFgRiJhkPXWIS/9YcK4Z9J4mp89gmK
 esLQ==
X-Gm-Message-State: AAQBX9c25CN3rK19fvNSrDv5l7pKF8JB9zqXZDmBQpwokUGSUCl9QL5i
 RMiUOi25jtaK+jvcYUN8+/EXGraDEngJAxw1GCY=
X-Google-Smtp-Source: AKy350Y7hXDqiTw4u77IrwBslHwTeFnWZ3TuHB0VTmrcrUtuVprIqOAtICyO5QtbQuEFutlYez2HWzx2n69d1vhbQNw=
X-Received: by 2002:a2e:a401:0:b0:295:acea:5875 with SMTP id
 p1-20020a2ea401000000b00295acea5875mr1410079ljn.2.1680723631858; Wed, 05 Apr
 2023 12:40:31 -0700 (PDT)
MIME-Version: 1.0
References: <2554712d-e386-3bab-bc6c-1f0e85d999db@HIDDEN>
 <CAPUEspj1m6F0_XgOFUVaq3Aq_Ah3PzCUs7YUyFH9_Zz-MOYTTA@HIDDEN>
 <96358c4e-7200-e5a5-869e-5da9d0de3503@HIDDEN>
 <xmqqttxvzbo8.fsf@HIDDEN>
 <6d86214a-1b80-eb88-1efb-36e61fd3203e@HIDDEN>
In-Reply-To: <6d86214a-1b80-eb88-1efb-36e61fd3203e@HIDDEN>
From: Jim Meyering <jim@HIDDEN>
Date: Wed, 5 Apr 2023 12:40:18 -0700
Message-ID: <CA+8g5KHYqgAZPpTOXWekDpWv-mvj-rBkGu+4MXy4OB1VDeS4Lw@HIDDEN>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: 3.3 (+++)
X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org",
 has NOT identified this incoming email as spam.  The original
 message has been attached to this so you can view it or label
 similar future email.  If you have any questions, see
 the administrator of that system for details.
 
 Content preview:  On Wed, Apr 5, 2023 at =?UTF-8?Q?11:33=E2=80=AFAM?= Paul Eggert <eggert@HIDDEN>
    wrote: > On 2023-04-04 12:31, Junio C Hamano wrote: > > My personal inclination
    is to let Perl folks decide > > and follow them (even [...] 
 
 Content analysis details:   (3.3 points, 10.0 required)
 
  pts rule name              description
 ---- ---------------------- --------------------------------------------------
  3.0 MANY_TO_CC             Sent to 10+ recipients
  0.0 FREEMAIL_FROM          Sender email is commonly abused enduser mail
                             provider (meyering[at]gmail.com)
  0.0 SPF_HELO_NONE          SPF: HELO does not publish an SPF Record
 -0.0 RCVD_IN_MSPIKE_H2      RBL: Average reputation (+2)
                             [209.85.208.174 listed in wl.mailspike.net]
 -0.0 RCVD_IN_DNSWL_NONE     RBL: Sender listed at https://www.dnswl.org/,
                              no trust
                             [209.85.208.174 listed in list.dnswl.org]
  0.2 HEADER_FROM_DIFFERENT_DOMAINS From and EnvelopeFrom 2nd level
                             mail domains are different
  0.0 T_SPF_TEMPERROR        SPF: test of record failed (temperror)
  0.0 FREEMAIL_FORGED_FROMDOMAIN 2nd level domains in From and
                             EnvelopeFrom freemail headers are
                             different
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: 2.2 (++)
X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org",
 has NOT identified this incoming email as spam.  The original
 message has been attached to this so you can view it or label
 similar future email.  If you have any questions, see
 the administrator of that system for details.
 
 Content preview:  On Wed, Apr 5, 2023 at =?UTF-8?Q?11:33=E2=80=AFAM?= Paul Eggert <eggert@HIDDEN>
    wrote: > On 2023-04-04 12:31, Junio C Hamano wrote: > > My personal inclination
    is to let Perl folks decide > > and follow them (even [...] 
 
 Content analysis details:   (2.2 points, 10.0 required)
 
  pts rule name              description
 ---- ---------------------- --------------------------------------------------
 -0.0 RCVD_IN_MSPIKE_H2      RBL: Average reputation (+2)
                             [209.85.208.174 listed in wl.mailspike.net]
 -0.0 RCVD_IN_DNSWL_NONE     RBL: Sender listed at https://www.dnswl.org/,
                              no trust
                             [209.85.208.174 listed in list.dnswl.org]
  3.0 MANY_TO_CC             Sent to 10+ recipients
  0.0 FREEMAIL_FROM          Sender email is commonly abused enduser mail
                             provider (meyering[at]gmail.com)
  0.0 SPF_HELO_NONE          SPF: HELO does not publish an SPF Record
 -0.0 SPF_PASS               SPF: sender matches SPF record
  0.2 HEADER_FROM_DIFFERENT_DOMAINS From and EnvelopeFrom 2nd level
                             mail domains are different
  0.0 FREEMAIL_FORGED_FROMDOMAIN 2nd level domains in From and
                             EnvelopeFrom freemail headers are
                             different
 -1.0 MAILING_LIST_MULTI     Multiple indicators imply a widely-seen list
                             manager

On Wed, Apr 5, 2023 at 11:33=E2=80=AFAM Paul Eggert <eggert@HIDDEN> wr=
ote:
> On 2023-04-04 12:31, Junio C Hamano wrote:
> > My personal inclination is to let Perl folks decide
> > and follow them (even though I am skeptical about the wisdom of
> > letting '\d' match anything other than [0-9])
>
> I looked into what pcre2grep does. It has always done only 8-bit
> processing unless you use the -u or --utf option, so plain "pcre2grep
> '\d'" matches only ASCII digits.
>
> Although this causes pcre2grep to mishandle Unicode characters:
>
>    $ echo '=C3=86var' | pcre2grep '[Ss=C3=9F]'
>    =C3=86var
>
> it mimics Perl 5.36:
>
>    $ echo '=C3=86var' | perl -ne 'print $_ if /[Ss=C3=9F]/'
>    =C3=86var
>
> so this seems to be what Perl users expect, despite its infelicities.
>
> For better Unicode handling one can use pcre2grep's -u or --utf option,
> which causes pcre2grep to behave more like GNU grep -P and git grep -P:
> "echo '=C3=86var' | pcre2grep -u '[Ss=C3=9F]'" outputs nothing, which I t=
hink is
> what most people would expect (unless they're Perl users :-).

Good argument for making PCRE2_UCP the default.

> Neither git grep -P nor the current release of pcre2grep -u have \d
> matching non-ASCII digits, because they do not use PCRE2_UCP. However,
> in a February 8 commit[1], Philip Hazel changed pcre2grep to use
> PCRE2_UCP, so this will mean 10.43 pcre2grep -u will behave like 3.9 GNU
> grep -P did (though 3.10 has changed this).
>
> That February commit also added a --no-ucp option, to disable PCRE2_UCP.
> So as I understand it, if you're in a UTF-8 locale:
>
> * 10.43 pcre2grep -u will behave like 3.9 GNU grep -P.
>
> * 10.43 pcre2grep -u --no-ucp will behave like git grep -P.
>
> * Current GNU grep -P is different from everybody else.
>
> This incompatibility is not good.
>
> Here are two ways forward to fix this incompatibility (there are other
> possibilities of course):
>
> (A) GNU grep adds a --no-ucp option that acts like 10.43 pcre2grep
> --no-ucp, and git grep -P follows suit. That is, both GNU and git grep
> act like 10.43 pcre2grep -u, in that they enable PCRE2_UTF, and also
> enable PCRE2_UCP unless --no-ucp is given. This would cause \d to match
> non-ASCII digits unless --no-ucp is given.
>
> (B) GNU grep -P and git grep -P mimic pcre2grep in both -u and --no-ucp.
> That is, they would both do 8-bit-only by default, and use PCRE2_UTF
> only when -u or --utf is given, and use PCRE2_UCP only when --no-ucp is
> absent. This would cause \d to match non-ASCII digits only when -u is
> given but --no-ucp is not.

Changing grep -P's \d to match multibyte digits by default would break
an important contract. Avoiding that feels like it must outweigh any
cross-tool portability concern.

(C)  preserve grep -P's tradition of \d matching only 0..9, and once
grep uses 10.43 or newer, \b and \w will also work as desired.

> Under either (A) or (B), future pcre2grep -u, GNU grep -P, and git grep
> -P would be consistent.

I hope git grep -P's \d will also stick to ASCII-only by default.
Those rare few who desire multibyte matches can always specify \p{Nd}
instead of \d, or (with new enough PCRE2), use (?-aD) and (?aD) to
toggle the digit-matching mode.

Message sent to bug-grep@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#60690: -P '\d' in GNU and git grep
Resent-From: Paul Eggert <eggert@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-grep@HIDDEN
Resent-Date: Wed, 05 Apr 2023 20:04:01 +0000
Resent-Message-ID: <handler.60690.B60690.168072504029197 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 60690
X-GNU-PR-Package: grep
X-GNU-PR-Keywords: patch
To: Jim Meyering <jim@HIDDEN>
Cc: demerphq@HIDDEN, Philip.Hazel@HIDDEN, 60690 <at> debbugs.gnu.org, mega lith01 <megalith01@HIDDEN>, Carlo Arenas <carenas@HIDDEN>, =?UTF-8?Q?=C3=86var_?= =?UTF-8?Q?Arnfj=C3=B6r=C3=B0?= Bjarmason <avarab@HIDDEN>, git@HIDDEN, Junio C Hamano <gitster@HIDDEN>, =?UTF-8?Q?Tukusej=E2=80=99s?= Sirs <tukusejssirs@HIDDEN>, pcre-dev@HIDDEN
Received: via spool by 60690-submit <at> debbugs.gnu.org id=B60690.168072504029197
          (code B ref 60690); Wed, 05 Apr 2023 20:04:01 +0000
Received: (at 60690) by debbugs.gnu.org; 5 Apr 2023 20:04:00 +0000
Received: from localhost ([127.0.0.1]:52707 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1pk9MN-0007ar-K1
	for submit <at> debbugs.gnu.org; Wed, 05 Apr 2023 16:03:59 -0400
Received: from mail.cs.ucla.edu ([131.179.128.66]:46502)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eggert@HIDDEN>) id 1pk9ML-0007ab-Ut
 for 60690 <at> debbugs.gnu.org; Wed, 05 Apr 2023 16:03:58 -0400
Received: from localhost (localhost [127.0.0.1])
 by mail.cs.ucla.edu (Postfix) with ESMTP id DAD523C09FA02;
 Wed,  5 Apr 2023 13:03:51 -0700 (PDT)
Received: from mail.cs.ucla.edu ([127.0.0.1])
 by localhost (mail.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10032)
 with ESMTP id 5aZEMnBhpMz3; Wed,  5 Apr 2023 13:03:51 -0700 (PDT)
Received: from localhost (localhost [127.0.0.1])
 by mail.cs.ucla.edu (Postfix) with ESMTP id 905833C09FA04;
 Wed,  5 Apr 2023 13:03:51 -0700 (PDT)
DKIM-Filter: OpenDKIM Filter v2.10.3 mail.cs.ucla.edu 905833C09FA04
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cs.ucla.edu;
 s=9D0B346E-2AEB-11ED-9476-E14B719DCE6C; t=1680725031;
 bh=8PkslEbHjoWrpxk5RHwWEEVpxilllW7PcwBgH/aBFtk=;
 h=Message-ID:Date:MIME-Version:To:From;
 b=CftoLGsHsh5+2RqTPkPiiFimYzMr4dQgc415GFbs6dKBjA2ISMPv1p2YYlZn/6k4W
 yOuq+Fv6giOfadQqgV56YsHcz4rZfo+tCY7rlNTo8MLLSTahRqwc3buVoBzY6zKLVT
 JC8Ea7lsV+gVxbOwyBIVlj6+pJLNXW1GP/wJ3+mB6e39YJg1MqYj/kv3oF6YCuAaBs
 LUIWivy/Lnwa+ozTMLlxqmC50PBwy9cZaQ8116gwM5KLLBeWBOwuK4XawQOGxkQa49
 DSOvtaQKY/TYKIiVDmqEwrlsZ8ERVJ36HcSlVPQuo11NPCVlb3F7goLAwUQhNes8vk
 DX5sQ4hLPkE7Q==
X-Virus-Scanned: amavisd-new at mail.cs.ucla.edu
Received: from mail.cs.ucla.edu ([127.0.0.1])
 by localhost (mail.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10026)
 with ESMTP id EB-czLSrBsYx; Wed,  5 Apr 2023 13:03:51 -0700 (PDT)
Received: from [192.168.1.9] (cpe-172-91-119-151.socal.res.rr.com
 [172.91.119.151])
 by mail.cs.ucla.edu (Postfix) with ESMTPSA id 4A6033C09FA02;
 Wed,  5 Apr 2023 13:03:51 -0700 (PDT)
Message-ID: <ed237a07-2f77-74eb-2f52-49b9b8f08873@HIDDEN>
Date: Wed, 5 Apr 2023 13:03:51 -0700
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.9.0
Content-Language: en-US
References: <2554712d-e386-3bab-bc6c-1f0e85d999db@HIDDEN>
 <CAPUEspj1m6F0_XgOFUVaq3Aq_Ah3PzCUs7YUyFH9_Zz-MOYTTA@HIDDEN>
 <96358c4e-7200-e5a5-869e-5da9d0de3503@HIDDEN>
 <xmqqttxvzbo8.fsf@HIDDEN>
 <6d86214a-1b80-eb88-1efb-36e61fd3203e@HIDDEN>
 <CA+8g5KHYqgAZPpTOXWekDpWv-mvj-rBkGu+4MXy4OB1VDeS4Lw@HIDDEN>
From: Paul Eggert <eggert@HIDDEN>
Organization: UCLA Computer Science Department
In-Reply-To: <CA+8g5KHYqgAZPpTOXWekDpWv-mvj-rBkGu+4MXy4OB1VDeS4Lw@HIDDEN>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Spam-Score: 1.9 (+)
X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org",
 has NOT identified this incoming email as spam.  The original
 message has been attached to this so you can view it or label
 similar future email.  If you have any questions, see
 the administrator of that system for details.
 Content preview:  On 2023-04-05 12:40, Jim Meyering wrote: > (C) preserve grep
 -P's tradition of \d matching only 0..9, and once > grep uses 10.43 or newer, 
 \b and \w will also work as desired. If I understand you correctly, (C) would
 mean that GNU grep -P, git grep -P, and pcre2grep -u would all use PCRE2_UTF
 | PCRE2_UCP, and would also use the extra option PCRE2_EXTRA_ASCII_BSD that
 is pla [...] 
 Content analysis details:   (1.9 points, 10.0 required)
 pts rule name              description
 ---- ---------------------- --------------------------------------------------
 3.0 MANY_TO_CC             Sent to 10+ recipients
 0.0 SPF_HELO_NONE          SPF: HELO does not publish an SPF Record
 -0.0 SPF_PASS               SPF: sender matches SPF record
 -1.1 NICE_REPLY_A           Looks like a legit reply (A)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: 0.9 (/)

On 2023-04-05 12:40, Jim Meyering wrote:
> (C)  preserve grep -P's tradition of \d matching only 0..9, and once
> grep uses 10.43 or newer, \b and \w will also work as desired.

If I understand you correctly, (C) would mean that GNU grep -P, git grep 
-P, and pcre2grep -u would all use PCRE2_UTF | PCRE2_UCP, and would also 
use the extra option PCRE2_EXTRA_ASCII_BSD that is planned for 10.43 PCRE2.

This would require changes to bleeding-edge pcre2grep -u (since it would 
need to add PCRE2_EXTRA_ASCII_BSD unless --no-ucp is also given), and to 
git grep -P (which would need to add PCRE2_UCP and 
PCRE2_EXTRA_ASCII_BSD, when libpcre2 is new enough to #define 
PCRE2_EXTRA_ASCII_BSD).

This option works for me as well. In fact it's the least work for me 
since I already implemented it in bleeding-edge GNU grep (so it works 
this way already :-).

Message sent to bug-grep@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#60690: -P '\d' in GNU and git grep
Resent-From: Carlo Arenas <carenas@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-grep@HIDDEN
Resent-Date: Wed, 05 Apr 2023 21:22:01 +0000
Resent-Message-ID: <handler.60690.B60690.16807296735440 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 60690
X-GNU-PR-Package: grep
X-GNU-PR-Keywords: patch
To: Jim Meyering <jim@HIDDEN>
Cc: demerphq@HIDDEN, Paul Eggert <eggert@HIDDEN>, pcre2-dev@HIDDEN, 60690 <at> debbugs.gnu.org, mega lith01 <megalith01@HIDDEN>, Philip.Hazel@HIDDEN, =?UTF-8?Q?=C3=86var_?= =?UTF-8?Q?Arnfj=C3=B6r=C3=B0?= Bjarmason <avarab@HIDDEN>, Junio C Hamano <gitster@HIDDEN>, =?UTF-8?Q?Tukusej=E2=80=99s?= Sirs <tukusejssirs@HIDDEN>, git@HIDDEN
Received: via spool by 60690-submit <at> debbugs.gnu.org id=B60690.16807296735440
          (code B ref 60690); Wed, 05 Apr 2023 21:22:01 +0000
Received: (at 60690) by debbugs.gnu.org; 5 Apr 2023 21:21:13 +0000
Received: from localhost ([127.0.0.1]:52785 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1pkAZ6-0001Pg-TR
	for submit <at> debbugs.gnu.org; Wed, 05 Apr 2023 17:21:13 -0400
Received: from mail-wr1-f54.google.com ([209.85.221.54]:36713)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <carenas@HIDDEN>) id 1pkAZ3-0001PQ-Ao
 for 60690 <at> debbugs.gnu.org; Wed, 05 Apr 2023 17:21:11 -0400
Received: by mail-wr1-f54.google.com with SMTP id i9so37538167wrp.3
 for <60690 <at> debbugs.gnu.org>; Wed, 05 Apr 2023 14:21:09 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=gmail.com; s=20210112; t=1680729663;
 h=content-transfer-encoding:cc:to:subject:message-id:date:from
 :in-reply-to:references:mime-version:from:to:cc:subject:date
 :message-id:reply-to;
 bh=aCatREPKZ1k9qVjjaXtsQbJYEyUR5HAkP3fROIQp2zc=;
 b=Ta47UMFMnsY5uoJHl8ZWLWyVcShlCVEcV9LOHBtRHZVisgQyH4JJ4plkso7aUUrFoT
 O+uLv92ZbmL83qlXvwLWwOiZfATHV4qpsHbAR55Sk+kCyy6KFLOFTD4UFzMvQCZcHhA7
 2X1wKhD14DjgqFxevFM3H0Et5TOA9PtmTUvQrgdKbITJ9XiDVCv2Ik4HPutUtJrNjHRH
 7QIv181+HI0FBkOECWKiG7d3yWk8ATxn8RpSSHdtlq1Jg1SyPTiWrgEbG41a5fbchxHD
 zvmoMS8fqbroTCbJlcKk72xXIe4l7yrdvznPRsgiQPsHjPvZqRRN2IZHLNxf3b0CIIg4
 /NjQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20210112; t=1680729663;
 h=content-transfer-encoding:cc:to:subject:message-id:date:from
 :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc
 :subject:date:message-id:reply-to;
 bh=aCatREPKZ1k9qVjjaXtsQbJYEyUR5HAkP3fROIQp2zc=;
 b=voj/9ydXaVTHQrmh2bvemO2Yu3HFYiu64y+x5WtAKVsPikkqLB2VRRGAyp0lpcAkTl
 k3GmRKR5gdPRJdijb+q05YS3xgw8qxZCzurLchLdYjLCVad7rwOpF03HDz+DB2VBsK8W
 sqdhS6YDnhOq0eep9kmvSxArVaeFtEmbXxiYIT3cCG648t5GIJ+79ipUFwGtDjAFSunF
 rtGYFkVVEtV7H7DekYXCwuvvmNWBt/JXLRFRHcPN0HaHfXy6GVYWNrPmxiWd8CfGfyFR
 a1C1JIXdJryfmIAtNQXKFtMI3wBp7V3dj23exugy5Lb1xEan1sRVjCdDkBkUMCucGHK5
 zoIQ==
X-Gm-Message-State: AAQBX9eKFmBm6CB0ocK4dxExRa3/0Le5Sq//NFONy6FTNZ1kF+Sm+XD3
 /ljm3hs6m1M87fKSm2yp8aLuVQ8OREaeOue7HAY=
X-Google-Smtp-Source: AKy350Yug+z1y1f69227N9A9y1YgjMZIWHm96uHR51/doAU4PfBXpU9KcCLPwSBbscke8hJLLb7BFyul+DEL7tmhHjs=
X-Received: by 2002:adf:e30e:0:b0:2e4:cbfe:da50 with SMTP id
 b14-20020adfe30e000000b002e4cbfeda50mr1424397wrj.1.1680729663146; Wed, 05 Apr
 2023 14:21:03 -0700 (PDT)
MIME-Version: 1.0
References: <2554712d-e386-3bab-bc6c-1f0e85d999db@HIDDEN>
 <CAPUEspj1m6F0_XgOFUVaq3Aq_Ah3PzCUs7YUyFH9_Zz-MOYTTA@HIDDEN>
 <96358c4e-7200-e5a5-869e-5da9d0de3503@HIDDEN>
 <xmqqttxvzbo8.fsf@HIDDEN>
 <6d86214a-1b80-eb88-1efb-36e61fd3203e@HIDDEN>
 <CA+8g5KHYqgAZPpTOXWekDpWv-mvj-rBkGu+4MXy4OB1VDeS4Lw@HIDDEN>
In-Reply-To: <CA+8g5KHYqgAZPpTOXWekDpWv-mvj-rBkGu+4MXy4OB1VDeS4Lw@HIDDEN>
From: Carlo Arenas <carenas@HIDDEN>
Date: Wed, 5 Apr 2023 14:20:51 -0700
Message-ID: <CAPUEspjM6PtsY9LiK9Lqb2+H2UrWEfPziWVrOPwZGVpVbx7aJQ@HIDDEN>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: 3.0 (+++)
X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org",
 has NOT identified this incoming email as spam.  The original
 message has been attached to this so you can view it or label
 similar future email.  If you have any questions, see
 the administrator of that system for details.
 
 Content preview:  On Wed, Apr 5, 2023 at =?UTF-8?Q?12:40=E2=80=AFPM?= Jim Meyering wrote: > >
   Changing grep -P's \d to match multibyte digits by default would break > an
    important contract. While I tend to agree[1] (and indeed that is why PCRE2_EXTRA_ASCII_BSD
    was invented), it would be also important to note that it goes against the
    Unicode recommendation[2] and it is actually not true [...] 
 
 Content analysis details:   (3.0 points, 10.0 required)
 
  pts rule name              description
 ---- ---------------------- --------------------------------------------------
  3.0 MANY_TO_CC             Sent to 10+ recipients
  0.0 FREEMAIL_FROM          Sender email is commonly abused enduser mail
                             provider (carenas[at]gmail.com)
  0.0 SPF_HELO_NONE          SPF: HELO does not publish an SPF Record
 -0.0 SPF_PASS               SPF: sender matches SPF record
 -0.0 RCVD_IN_DNSWL_NONE     RBL: Sender listed at https://www.dnswl.org/,
                              no trust
                             [209.85.221.54 listed in list.dnswl.org]
 -0.0 RCVD_IN_MSPIKE_H2      RBL: Average reputation (+2)
                             [209.85.221.54 listed in wl.mailspike.net]
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: 2.0 (++)
X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org",
 has NOT identified this incoming email as spam.  The original
 message has been attached to this so you can view it or label
 similar future email.  If you have any questions, see
 the administrator of that system for details.
 
 Content preview:  On Wed, Apr 5, 2023 at =?UTF-8?Q?12:40=E2=80=AFPM?= Jim Meyering wrote: > >
   Changing grep -P's \d to match multibyte digits by default would break > an
    important contract. While I tend to agree[1] (and indeed that is why PCRE2_EXTRA_ASCII_BSD
    was invented), it would be also important to note that it goes against the
    Unicode recommendation[2] and it is actually not true [...] 
 
 Content analysis details:   (2.0 points, 10.0 required)
 
  pts rule name              description
 ---- ---------------------- --------------------------------------------------
 -0.0 RCVD_IN_MSPIKE_H2      RBL: Average reputation (+2)
                             [209.85.221.54 listed in wl.mailspike.net]
 -0.0 RCVD_IN_DNSWL_NONE     RBL: Sender listed at https://www.dnswl.org/,
                              no trust
                             [209.85.221.54 listed in list.dnswl.org]
  3.0 MANY_TO_CC             Sent to 10+ recipients
  0.0 FREEMAIL_FROM          Sender email is commonly abused enduser mail
                             provider (carenas[at]gmail.com)
  0.0 SPF_HELO_NONE          SPF: HELO does not publish an SPF Record
 -0.0 SPF_PASS               SPF: sender matches SPF record
 -1.0 MAILING_LIST_MULTI     Multiple indicators imply a widely-seen list
                             manager

On Wed, Apr 5, 2023 at 12:40=E2=80=AFPM Jim Meyering <jim@HIDDEN> wro=
te:
>
> Changing grep -P's \d to match multibyte digits by default would break
> an important contract.

While I tend to agree[1] (and indeed that is why PCRE2_EXTRA_ASCII_BSD
was invented), it would be also important to note that it goes against
the Unicode recommendation[2] and it is actually not true already[3]
for Python, .NET or Rust (which means ripgrep behaves like GNU grep -P
3.9).

FWIW I also agree that (at least `git grep -P`) should use
PCRE2_EXTRA_ASCII_BSD by default as that is what makes more sense in
the context of matching source code and using instead `\P{Nd}` if you
really want all Unicode digits is not much of a burden, but I am also
not sure if that makes sense in other contexts, specially considering
that I am obviously biased since the languages I mostly interact with
ONLY use arabic numerals and therefore `\d` meaning `[0-9]` seems
"normal".

Carlo

CC: changed to the real email address for PCRE2 development, for full
context on this thread use [4]

[1] https://github.com/PCRE2Project/pcre2/pull/186
[2] https://unicode.org/reports/tr18/
[3] https://regex101.com/r/S5RW4c/1
[4] https://lore.kernel.org/git/230109.86v8lf297g.gmgdl@HIDDEN=
/T/

Message sent to bug-grep@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#60690: -P '\d' in GNU and git grep
Resent-From: demerphq <demerphq@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-grep@HIDDEN
Resent-Date: Thu, 06 Apr 2023 13:40:02 +0000
Resent-Message-ID: <handler.60690.B60690.16807883917622 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 60690
X-GNU-PR-Package: grep
X-GNU-PR-Keywords: patch
To: Junio C Hamano <gitster@HIDDEN>
Cc: Paul Eggert <eggert@HIDDEN>, 60690 <at> debbugs.gnu.org, mega lith01 <megalith01@HIDDEN>, Carlo Arenas <carenas@HIDDEN>, =?UTF-8?Q?=C3=86var_?= =?UTF-8?Q?Arnfj=C3=B6r=C3=B0?= Bjarmason <avarab@HIDDEN>, pcre-dev@HIDDEN, =?UTF-8?Q?Tukusej=E2=80=99s?= Sirs <tukusejssirs@HIDDEN>, git@HIDDEN
Received: via spool by 60690-submit <at> debbugs.gnu.org id=B60690.16807883917622
          (code B ref 60690); Thu, 06 Apr 2023 13:40:02 +0000
Received: (at 60690) by debbugs.gnu.org; 6 Apr 2023 13:39:51 +0000
Received: from localhost ([127.0.0.1]:53679 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1pkPqB-0001yr-49
	for submit <at> debbugs.gnu.org; Thu, 06 Apr 2023 09:39:51 -0400
Received: from mail-qv1-f53.google.com ([209.85.219.53]:36810)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <demerphq@HIDDEN>) id 1pkPq8-0001yZ-BS
 for 60690 <at> debbugs.gnu.org; Thu, 06 Apr 2023 09:39:49 -0400
Received: by mail-qv1-f53.google.com with SMTP id cu4so28359624qvb.3
 for <60690 <at> debbugs.gnu.org>; Thu, 06 Apr 2023 06:39:48 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=gmail.com; s=20210112; t=1680788382;
 h=cc:to:subject:message-id:date:from:in-reply-to:references
 :mime-version:from:to:cc:subject:date:message-id:reply-to;
 bh=qHG4jeUv5wcUw/5Ov0dUmw116z+k+P3f0ikY/j9srHA=;
 b=TzJ51q3xpt2NBeiYICRYi18x5HIblyYRB56/90ewB7G9pvQnl/l1Es41UEI8S/mbYe
 P6GHj1TDWzDdnf324jBOcx0ruyJoHjZBfDMVwH6WyLHPL0Ur7CHxJzclmE/EGbtEiUlN
 uVTDgyTf+RY4zPvNfR0g5dbfY+adei57zRCD6neJ3a9KrXdUzaXY7nNHAL4aZBPrIRmS
 8OYSbJ3X5VW/q30VxR1wTDAWrhMPHuiGyVdgwtD/tNOqRGIEgIs32FImASbr2BdvINI8
 HGmjHRwXeV16+EpIoUUmkp3T4MPa3p5C/H8WGPRMZSXJo9fU2fB51RRVkNAAYf5TFyOW
 g2eA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20210112; t=1680788382;
 h=cc:to:subject:message-id:date:from:in-reply-to:references
 :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id
 :reply-to;
 bh=qHG4jeUv5wcUw/5Ov0dUmw116z+k+P3f0ikY/j9srHA=;
 b=SElmFFKhXL8dt0nnTl1zNZ67WOIGJOHf/KPyol7IgRWjp+GjWHNRNTQnZRDga9Misy
 pe62NTUUoijpbLBqrS1nNF0kx+fn/qgkbcMV1kD+LveawbFlUhLBwSfSXvT/+zjd5lsZ
 Z5YLtIjaBmlAPZlZrliNxG9+B49boa+6ygww3PNpzJ681zvqKkxknQPsoPTHI14pi3Kq
 zQkagvy2iM/KhlqKKNfQMBLfSqxunS1ibGNzKnAO2gaOXgQOrD0d/tuzi9nZTBxtFGfI
 hfmTXva+ALCZ66qAe1atQqFDiK7EYXYPaWpGEHiINfvj02A22xzwG6M6rGGN6rrXwohB
 V51w==
X-Gm-Message-State: AAQBX9f9WmRmnq2ZD8k3GRly3AMIL68Dp7BVR4L+v/kNyH6jBfVNVDOs
 /eOzUgVGqo8bZNYHAbIWmd2MyLOuG8DsqbtmL20=
X-Google-Smtp-Source: AKy350Z5c7XEECkeGZ6yZ/3bRWz1vgXrT0l6ZxLjSewl8PQDX5zhWKkhT1VUsRPntK8S9FLhI9nSIgEjyBevA+AplXA=
X-Received: by 2002:ad4:58b2:0:b0:56f:378:951 with SMTP id
 ea18-20020ad458b2000000b0056f03780951mr542261qvb.1.1680788382666; 
 Thu, 06 Apr 2023 06:39:42 -0700 (PDT)
MIME-Version: 1.0
References: <2554712d-e386-3bab-bc6c-1f0e85d999db@HIDDEN>
 <CAPUEspj1m6F0_XgOFUVaq3Aq_Ah3PzCUs7YUyFH9_Zz-MOYTTA@HIDDEN>
 <96358c4e-7200-e5a5-869e-5da9d0de3503@HIDDEN>
 <xmqqttxvzbo8.fsf@HIDDEN>
In-Reply-To: <xmqqttxvzbo8.fsf@HIDDEN>
From: demerphq <demerphq@HIDDEN>
Date: Thu, 6 Apr 2023 15:39:31 +0200
Message-ID: <CANgJU+U+xXsh9psd0z5Xjr+Se5QgdKkjQ7LUQ-PdUULSN3n4+g@HIDDEN>
Content-Type: text/plain; charset="UTF-8"
X-Spam-Score: 0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

On Tue, 4 Apr 2023 at 21:31, Junio C Hamano <gitster@HIDDEN> wrote:
>
> Paul Eggert <eggert@HIDDEN> writes:
>
> > This is an evolving area. Git master is fiddling with flags and
> > options, and so is GNU grep master, and so is PCRE2, and there are
> > bugs. If you're running bleeding-edge versions of this code you'll get
> > different behavior than if you're running grep 3.8, pcregrep 8.45,
> > Perl 5.36, and git 2.39.2 (which is what Fedora 37 has).
> >
> > What I'm fearing is that we may evolve into mutually incompatible
> > interpretations of how Perl regular expressions deal with UTF-8
> > text. That'd be a recipe for confusion down the road.
>
> Nicely said.  My personal inclination is to let Perl folks decide
> and follow them (even though I am skeptical about the wisdom of
> letting '\d' match anything other than [0-9]), but even in Git
> circle there would be different opinions, so I am glad that the
> discussion is visible on the list to those who are intrested.


Perl matches Unicode text according to the rules specified by the
Unicode consortium. It is the reference implementation for Unicode
regular expression matching. Unicode specifies that \d match any digit
in any script that it supports. Thus \d matches far more codepoints
than \p{PosixDigit} or [0-9] would.  Be aware that Unicode contains
and separates numbers and digits, eg, \x{1EC9E} represents a Lakh,
which is used in many Indian languages for 100,000, but which is not
considered a *digit* for obvious reasons.

FWIW, someone mentioned [[:digit:]] which matches the same as \d does
on Unicode strings and under the /u matching flag for regexes in Perl.
Arguably this was a mistake, [[:digit:]] is a POSIX character class,
and POSIX doesn't support Unicode so it should have matched [0-9] or
\p{PosixDigit}. But historically \d and [[:digit:]] in Perl were the
same and when \d was extended to meet the Unicode specification
[[:digit:]] came along for the ride likely inadvertently, thus
\p{PosixDigit} is equivalent to [0-9], but \p{XPosixDigit} is
equivalent to \d and [[:digit:]].

I notice that other posts in this thread have moved the conversation
on, and covered most of the points I wanted to make here. However I
wanted to say that there seem to be two different issues here. The
first is "what semantics do i expect from my regular expressions",
Unicode or legacy-ASCII, mostly this relates to case-insensitive
matching, but things like \d also surface discrepancies. The second is
"what encodings does the regular expression engine understand".
Unfortunately on *nix there is no tradition of using BOM's to
distinguish the 6 different possible encodings of Unicode (UTF-8,
UTF-EBCDIC, UTF-16LE, UTF-16BE, UTF-32LE, UTF-32BE), and there seems
to be some level of desire of matching with unicode semantics against
files that are not uniformly encoded in one of these formats.

So the question comes up, A) how do you tell the regular expression
engine what semantics you want and B) how does the regular expression
library identify the encoding in the file, and how does it handle
malformed content in that file. For instance if I have a file which
contains snippets of UTF8 encoded data, *and* snippets of data that is
illegal in UTF8, what should the regular expression engine do if it is
asked to do a case insensitive match against that file.

cheers,
yves

Message sent to bug-grep@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#60690: -P '\d' in GNU and git grep
Resent-From: demerphq <demerphq@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-grep@HIDDEN
Resent-Date: Thu, 06 Apr 2023 15:46:01 +0000
Resent-Message-ID: <handler.60690.B60690.168079593129422 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 60690
X-GNU-PR-Package: grep
X-GNU-PR-Keywords: patch
To: Paul Eggert <eggert@HIDDEN>
Cc: Philip.Hazel@HIDDEN, 60690 <at> debbugs.gnu.org, mega lith01 <megalith01@HIDDEN>, Carlo Arenas <carenas@HIDDEN>, =?UTF-8?Q?=C3=86var_?= =?UTF-8?Q?Arnfj=C3=B6r=C3=B0?= Bjarmason <avarab@HIDDEN>, Junio C Hamano <gitster@HIDDEN>, pcre-dev@HIDDEN, =?UTF-8?Q?Tukusej=E2=80=99s?= Sirs <tukusejssirs@HIDDEN>, git@HIDDEN
Received: via spool by 60690-submit <at> debbugs.gnu.org id=B60690.168079593129422
          (code B ref 60690); Thu, 06 Apr 2023 15:46:01 +0000
Received: (at 60690) by debbugs.gnu.org; 6 Apr 2023 15:45:31 +0000
Received: from localhost ([127.0.0.1]:54768 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1pkRnn-0007cd-5p
	for submit <at> debbugs.gnu.org; Thu, 06 Apr 2023 11:45:31 -0400
Received: from mail-qk1-f172.google.com ([209.85.222.172]:43725)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <demerphq@HIDDEN>) id 1pkRni-0007AX-5P
 for 60690 <at> debbugs.gnu.org; Thu, 06 Apr 2023 11:45:29 -0400
Received: by mail-qk1-f172.google.com with SMTP id dw4so12443410qkb.10
 for <60690 <at> debbugs.gnu.org>; Thu, 06 Apr 2023 08:45:26 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=gmail.com; s=20210112; t=1680795920;
 h=content-transfer-encoding:cc:to:subject:message-id:date:from
 :in-reply-to:references:mime-version:from:to:cc:subject:date
 :message-id:reply-to;
 bh=RFsD2+GzDkJ4XHTOiptOBEv1CaiCwfmfe5UABd76rH8=;
 b=P19rU7JXHzUzmPhWhz/BlYF4wzDQ2YORVksL6gFF6XQacSJKbG+x+xYd2UPbZCG2kR
 Q3MO22MeT5kYQXRKbNRFBqqfuChlSRkwBMzbpCKErFbRDngXTfPX6qUxHnkSu7wReGxj
 6MUlOosx50IBPExa6bXsYOd6hLlVhGSbec70gqQV23Tqa3zrl9zB5dapx6I98WKPxkEJ
 IrgwCz7lumL6LXjPkHLOnMMQJBoC4qSXmAJBgKulK6szgJ5KmGjNl6JT7b8u9vGVsNf1
 UKnvlbrgv5obqZL/zFzB2HOyvtpQJ7Sx4D+wYAJATbqunx5rFvwvfsy8AxeNUe2xOInc
 FXtA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20210112; t=1680795920;
 h=content-transfer-encoding:cc:to:subject:message-id:date:from
 :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc
 :subject:date:message-id:reply-to;
 bh=RFsD2+GzDkJ4XHTOiptOBEv1CaiCwfmfe5UABd76rH8=;
 b=xp8r4PliDGyv1z6zYTmRqtCfn3sUexpZy6y6rZMvUeIKzcdd3Ir1KeZ2akKkfsUm5O
 R2vA4bBvTjGhK1jqvCCqveNYlcxiNZbqJHV4P1NC7/HBHCC29AqyjUhmtThIvIcHfjMr
 wjIhisAjAv8+qqiokfmenkN20/YDWPfKj/qlWDz09PjHWNlJou7u8//GBooGrdwghY5A
 JDh7Zu5AXiLUvtRBqCoIzeXp4smj1MburO0xUE1oxij5eUvtrYWMcvt+JBV1jnUe9PSK
 +Onx3FkrwVppKJ7WDIe/Epnt30hhm5jj7kVht6txYux3oPdzKNPpcPrMsDsMS4YiuJ0J
 rVog==
X-Gm-Message-State: AAQBX9e8qBPZzWlGeSLCBG0LE29k1gz0Wi1DQ9qP3pzdN5S45zPSbIMm
 34sp5ZZ6aZ4B/wQ4IScaUvh5QCiDH6sUN5bwswo=
X-Google-Smtp-Source: AKy350Z84z2cPpW/yIWeSZZfvcjmNdEMr5kQdi9PtQ0/c+S2q2SYM1PZRitNdtPVA2x7AWMjKqze6Lb7brxBxuFNegk=
X-Received: by 2002:a05:620a:410a:b0:742:8868:bfd1 with SMTP id
 j10-20020a05620a410a00b007428868bfd1mr2737838qko.7.1680795920160; Thu, 06 Apr
 2023 08:45:20 -0700 (PDT)
MIME-Version: 1.0
References: <2554712d-e386-3bab-bc6c-1f0e85d999db@HIDDEN>
 <CAPUEspj1m6F0_XgOFUVaq3Aq_Ah3PzCUs7YUyFH9_Zz-MOYTTA@HIDDEN>
 <96358c4e-7200-e5a5-869e-5da9d0de3503@HIDDEN>
 <xmqqttxvzbo8.fsf@HIDDEN>
 <6d86214a-1b80-eb88-1efb-36e61fd3203e@HIDDEN>
In-Reply-To: <6d86214a-1b80-eb88-1efb-36e61fd3203e@HIDDEN>
From: demerphq <demerphq@HIDDEN>
Date: Thu, 6 Apr 2023 17:45:09 +0200
Message-ID: <CANgJU+XoyptS8NU+f6uMLrKjQakv=iN2c4DQydVaBVH3dK3s-w@HIDDEN>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: 0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

On Wed, 5 Apr 2023 at 20:32, Paul Eggert <eggert@HIDDEN> wrote:
>
> On 2023-04-04 12:31, Junio C Hamano wrote:
>
> > My personal inclination is to let Perl folks decide
> > and follow them (even though I am skeptical about the wisdom of
> > letting '\d' match anything other than [0-9])
>
> I looked into what pcre2grep does. It has always done only 8-bit
> processing unless you use the -u or --utf option, so plain "pcre2grep
> '\d'" matches only ASCII digits.
>
> Although this causes pcre2grep to mishandle Unicode characters:
>
>    $ echo '=C3=86var' | pcre2grep '[Ss=C3=9F]'
>    =C3=86var
>
> it mimics Perl 5.36:
>
>    $ echo '=C3=86var' | perl -ne 'print $_ if /[Ss=C3=9F]/'
>    =C3=86var
>
> so this seems to be what Perl users expect, despite its infelicities.

Actually no, I think you have misunderstood what is happening at the
different layers involved here.

Your terminal is rendering =C3=9F as a glyph. But it is almost certainly
actually the octets C3 9F (which is the UTF8 canonical representation
of the codepoint U+DF). So the code you provided to perl is close to
the equivalent of

echo '=C3=86var' | perl -ne 'print $_ if /[Ss\x{C3}\x{9F}]/'

And if you check, you will see that U+C6 "=C3=86" in utf8 is represented as
the octets C3 86.

So what you have done is the equivalent of:

perl -le'print "\x{C3}\x{86}"' | perl -ne'print $_ if /[Ss\x{C3}\x{9F}]/'

which of course matches. \x{C3} matches \x{C3} always and everywhere.

What you should have done is something like this:

$ echo '=C3=86var' | perl -ne 'utf8::decode($_); print $_ if /[Ss\x{DF}]/u'
$ echo 'ba=C3=9F' | perl -MEncode -ne 'utf8::decode($_); print
encode_utf8($_) if /[Ss\x{DF}]/u'
ba=C3=9F
$ echo '=C3=86var' | perl -MEncode -ne 'utf8::decode($_); print
encode_utf8($_) if /[Ss\x{C6}]/u'
=C3=86var
$ echo '=C3=86var' | perl -MEncode -ne 'utf8::decode($_); print
encode_utf8($_) if /[Ss\x{e6}]/ui'
=C3=86var

The "utf8::decode($_)" tells perl to decode the input string as though
it contained utf8 (which in this case it does). THe /u suffix tells
the regex engine that you want Unicode semantics.

I believe that the same thing is true of your pcre2grep example. You
simply aren't checking what you think you are checking. You terminal
renders UTF8 as glyphs, but the programs you are feeding those glyphs
to aren't seeing glyphs, they are seeing UTF8 sequences as distinct
octets, and are not decoding their input back as codepoints.

You could have checked your assumptions by using the -Mre=3Ddebug option to=
 perl:

$ echo '=C3=86var' | perl -Mre=3Ddebug -ne 'print $_ if /[Ss=C3=9F]/'
Compiling REx "[Ss%x{c3}%x{9f}]"
Final program:
   1: ANYOF[Ss\x9F\xC3] (11)
  11: END (0)
stclass ANYOF[Ss\x9F\xC3] minlen 1
Matching REx "[Ss%x{c3}%x{9f}]" against "%x{c3}%x{86}var%n"
Matching stclass ANYOF[Ss\x9F\xC3] against "%x{c3}%x{86}var%n" (6 bytes)
   0 <> <%x{c3}>             |   0| 1:ANYOF[Ss\x9F\xC3](11)
   1 <%x{c3}> <%x{86}var>    |   0| 11:END(0)
Match successful!
=C3=86var
Freeing REx: "[Ss%x{c3}%x{9f}]"

The line: Matching REx "[Ss%x{c3}%x{9f}]" against "%x{c3}%x{86}var%n"

basically says it all. Perl has not decoded the UTF8 into U+C6, and it
has not decoded the UTF8 for U+DF either. Instead you have asked it if
the UTF8 sequence that represents U+C6 contains any of the same octets
as the UTF8 representation of U+53, U+73 and U+DF would. Which gives
the common octet of \x{c3}.

> For better Unicode handling one can use pcre2grep's -u or --utf option,
> which causes pcre2grep to behave more like GNU grep -P and git grep -P:
> "echo '=C3=86var' | pcre2grep -u '[Ss=C3=9F]'" outputs nothing, which I t=
hink is
> what most people would expect (unless they're Perl users :-).

It is what Perl users would expect also, assuming you actually wrote
the character class [Ss\x{DF}] and asked for unicode semantics. \x{DF}
is the Latin1 codepoint range, so perl will assume that you meant
ASCII semantics unless you tell it otherwise.

Basically these tests you have quoted here are just examples of
garbage in garbage out.

Perl has been working together with the Unicode consortium for over 20
years. Afaik we were and are the reference implementation for the spec
on regular expression matching in Unicode and we have a long history
of working together with the Unicode consortium to refine and
implement the spec. You should assume that if Perl seems to have made
a gross error in how it does Unicode matching that you are simply
using it wrong, we take a great deal of pride in having the best
Unicode support there is.

https://unicode.org/reports/tr18/

FWIW, i think this email nicely illustrates the issues with git and
regular expressions. To do regular expressions properly you need to
know a) what semantics do you expect, b) how to decode the text you
are matching against. If you want unicode semantics you need to have a
way to ask for it. If you want to match against Unicode data then you
need a way to determine which of the 6 possible encodings[1] of
Unicode data you are using. If you get either wrong you will not get
the results you expect.  You may even want to deal with cases where
you want Unicode semantics, but to match against non-unicode data. For
instance Latin-1. In Latin-1 the codepoint U+DF is the *octet* 0xDF.
Maybe you want that octet to match "ss" case-insensitively, as a
German speaker would expect and as Unicode specifies is correct.  Or
vice versa, maybe you are like some of the posters to this thread who
seem to expect that \d should not match U+16B51 (as a Hmong speaker
might expect). Perl resolves these problems at the pattern level by
supporting the suffixes /a and /u (for ascii and unicode), and at the
string level it supports two type of string, unicode strings, and
binary/ASCII strings. By default input is the latter but there are a
variety of ways of saying that a file handle should decode to Unicode
instead.

cheers,
Yves
[1] UTF-EBCDIC, UTF-8, UTF-16LE, UTF-16BE, UTF-32LE, UTF-32BE.

--
perl -Mre=3Ddebug -e "/just|another|perl|hacker/"

Message sent to bug-grep@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#60690: -P '\d' in GNU and git grep
Resent-From: Paul Eggert <eggert@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-grep@HIDDEN
Resent-Date: Fri, 07 Apr 2023 16:49:02 +0000
Resent-Message-ID: <handler.60690.B60690.16808861333010 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 60690
X-GNU-PR-Package: grep
X-GNU-PR-Keywords: patch
To: demerphq <demerphq@HIDDEN>
Cc: Carlo Arenas <carenas@HIDDEN>, 60690 <at> debbugs.gnu.org, mega lith01 <megalith01@HIDDEN>, Philip.Hazel@HIDDEN, =?UTF-8?Q?=C3=86var_?= =?UTF-8?Q?Arnfj=C3=B6r=C3=B0?= Bjarmason <avarab@HIDDEN>, git@HIDDEN, Junio C Hamano <gitster@HIDDEN>, =?UTF-8?Q?Tukusej=E2=80=99s?= Sirs <tukusejssirs@HIDDEN>, pcre-dev@HIDDEN
Received: via spool by 60690-submit <at> debbugs.gnu.org id=B60690.16808861333010
          (code B ref 60690); Fri, 07 Apr 2023 16:49:02 +0000
Received: (at 60690) by debbugs.gnu.org; 7 Apr 2023 16:48:53 +0000
Received: from localhost ([127.0.0.1]:57153 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1pkpGf-0000mU-8T
	for submit <at> debbugs.gnu.org; Fri, 07 Apr 2023 12:48:53 -0400
Received: from mail.cs.ucla.edu ([131.179.128.66]:45272)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eggert@HIDDEN>) id 1pkpGc-0000mF-SA
 for 60690 <at> debbugs.gnu.org; Fri, 07 Apr 2023 12:48:51 -0400
Received: from localhost (localhost [127.0.0.1])
 by mail.cs.ucla.edu (Postfix) with ESMTP id 6AE183C020F7C;
 Fri,  7 Apr 2023 09:48:44 -0700 (PDT)
Received: from mail.cs.ucla.edu ([127.0.0.1])
 by localhost (mail.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10032)
 with ESMTP id kPQurlYB1bY0; Fri,  7 Apr 2023 09:48:42 -0700 (PDT)
Received: from localhost (localhost [127.0.0.1])
 by mail.cs.ucla.edu (Postfix) with ESMTP id 7DDA13C097AFC;
 Fri,  7 Apr 2023 09:48:42 -0700 (PDT)
DKIM-Filter: OpenDKIM Filter v2.10.3 mail.cs.ucla.edu 7DDA13C097AFC
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cs.ucla.edu;
 s=9D0B346E-2AEB-11ED-9476-E14B719DCE6C; t=1680886122;
 bh=F+PW77ErQhFQHkkfv0GgRGwScVA/JOfmevt2y4yx7FU=;
 h=Message-ID:Date:MIME-Version:To:From;
 b=WlG/Z6XGA9EERDV2TMyOvewGtofk1t15OtmMu2AjQK5Ud/wxn6Av2FWxVJSODV9xt
 LnOD34IwqAt0s1QtFmVR75c4nJgyc8OepvFXwwlIlnvYjnicUgnD+ueWuRz962V9pw
 NOQ0pxL35251f5jxoMCRl6hRdxm8N9hCz61yeMIav5YYGhcJ6Vw6um2439yOFuZ5jV
 hPujxPTSPbKG0lWlYxVVC2Uqeg4hn6cgGODsALqjl5WP55qeoiY/leXchm/fVlu5WZ
 mETBAPuvzrmFIz6tZ6mdD2hXrn1ft6S0CAGXL0bSYpyQetP4l38QnOxaGQmAGl0J8K
 VdpEfWuq2Vt7w==
X-Virus-Scanned: amavisd-new at mail.cs.ucla.edu
Received: from mail.cs.ucla.edu ([127.0.0.1])
 by localhost (mail.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10026)
 with ESMTP id qML49UNS655D; Fri,  7 Apr 2023 09:48:42 -0700 (PDT)
Received: from [192.168.1.9] (cpe-172-91-119-151.socal.res.rr.com
 [172.91.119.151])
 by mail.cs.ucla.edu (Postfix) with ESMTPSA id 385E83C020F7C;
 Fri,  7 Apr 2023 09:48:42 -0700 (PDT)
Message-ID: <767d3617-e35a-a693-6ec8-f65421c68e5f@HIDDEN>
Date: Fri, 7 Apr 2023 09:48:40 -0700
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.9.0
References: <2554712d-e386-3bab-bc6c-1f0e85d999db@HIDDEN>
 <CAPUEspj1m6F0_XgOFUVaq3Aq_Ah3PzCUs7YUyFH9_Zz-MOYTTA@HIDDEN>
 <96358c4e-7200-e5a5-869e-5da9d0de3503@HIDDEN>
 <xmqqttxvzbo8.fsf@HIDDEN>
 <6d86214a-1b80-eb88-1efb-36e61fd3203e@HIDDEN>
 <CANgJU+XoyptS8NU+f6uMLrKjQakv=iN2c4DQydVaBVH3dK3s-w@HIDDEN>
Content-Language: en-US
From: Paul Eggert <eggert@HIDDEN>
Organization: UCLA Computer Science Department
In-Reply-To: <CANgJU+XoyptS8NU+f6uMLrKjQakv=iN2c4DQydVaBVH3dK3s-w@HIDDEN>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -1.1 (-)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -2.1 (--)

On 2023-04-06 08:45, demerphq wrote:
>> Although this causes pcre2grep to mishandle Unicode characters:
>>
>>     $ echo '=C3=86var' | pcre2grep '[Ss=C3=9F]'
>>     =C3=86var
>>
>> it mimics Perl 5.36:
>>
>>     $ echo '=C3=86var' | perl -ne 'print $_ if /[Ss=C3=9F]/'
>>     =C3=86var
>>
>> so this seems to be what Perl users expect, despite its infelicities.
> Actually no, I think you have misunderstood what is happening at the
> different layers involved here.

No, I understood what was going on. My point was that Perl users seem to=20
have accepted this behavior, even though it does not match what people=20
would ordinarily expect.


> What you should have done is something like this:

No, for two reasons. First, I'm no Perl expert and so I don't know (and=20
don't particularly want to learn) its complicated Unicode options and=20
calls. Second, /[Ss\x{DF}]/u is hard to read. If I want the S letters of=20
traditional German, I'll write them in the obvious way, as [Ss=C3=9F]. No=
=20
doubt Perl will let me do this somehow - but it is telling that none of=20
your examples do it in such a straightforward way.

> $ echo '=C3=86var' | perl -ne 'utf8::decode($_); print $_ if /[Ss\x{DF}=
]/u'
> $ echo 'ba=C3=9F' | perl -MEncode -ne 'utf8::decode($_); print
> encode_utf8($_) if /[Ss\x{DF}]/u'
> ba=C3=9F
> $ echo '=C3=86var' | perl -MEncode -ne 'utf8::decode($_); print
> encode_utf8($_) if /[Ss\x{C6}]/u'
> =C3=86var
> $ echo '=C3=86var' | perl -MEncode -ne 'utf8::decode($_); print
> encode_utf8($_) if /[Ss\x{e6}]/ui'
> =C3=86var

Message sent to bug-grep@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#60690: -P '\d' in GNU and git grep
Resent-From: Paul Eggert <eggert@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-grep@HIDDEN
Resent-Date: Fri, 07 Apr 2023 19:01:02 +0000
Resent-Message-ID: <handler.60690.B60690.168089402617541 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 60690
X-GNU-PR-Package: grep
X-GNU-PR-Keywords: patch
To: demerphq <demerphq@HIDDEN>
Cc: pcre2-dev@HIDDEN, 60690 <at> debbugs.gnu.org, mega lith01 <megalith01@HIDDEN>, Carlo Arenas <carenas@HIDDEN>, =?UTF-8?Q?=C3=86var_?= =?UTF-8?Q?Arnfj=C3=B6r=C3=B0?= Bjarmason <avarab@HIDDEN>, Junio C Hamano <gitster@HIDDEN>, =?UTF-8?Q?Tukusej=E2=80=99s?= Sirs <tukusejssirs@HIDDEN>, git@HIDDEN
Received: via spool by 60690-submit <at> debbugs.gnu.org id=B60690.168089402617541
          (code B ref 60690); Fri, 07 Apr 2023 19:01:02 +0000
Received: (at 60690) by debbugs.gnu.org; 7 Apr 2023 19:00:26 +0000
Received: from localhost ([127.0.0.1]:57252 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1pkrJy-0004Yr-As
	for submit <at> debbugs.gnu.org; Fri, 07 Apr 2023 15:00:26 -0400
Received: from mail.cs.ucla.edu ([131.179.128.66]:54636)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eggert@HIDDEN>) id 1pkrJv-0004Ya-IF
 for 60690 <at> debbugs.gnu.org; Fri, 07 Apr 2023 15:00:24 -0400
Received: from localhost (localhost [127.0.0.1])
 by mail.cs.ucla.edu (Postfix) with ESMTP id 64E6E3C09FA01;
 Fri,  7 Apr 2023 12:00:17 -0700 (PDT)
Received: from mail.cs.ucla.edu ([127.0.0.1])
 by localhost (mail.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10032)
 with ESMTP id zp66LFy4pHE7; Fri,  7 Apr 2023 12:00:17 -0700 (PDT)
Received: from localhost (localhost [127.0.0.1])
 by mail.cs.ucla.edu (Postfix) with ESMTP id 0E32C3C097AFC;
 Fri,  7 Apr 2023 12:00:17 -0700 (PDT)
DKIM-Filter: OpenDKIM Filter v2.10.3 mail.cs.ucla.edu 0E32C3C097AFC
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cs.ucla.edu;
 s=9D0B346E-2AEB-11ED-9476-E14B719DCE6C; t=1680894017;
 bh=RLGYiM+qbMJT97xdVCzt/fP+O1nLWgInQnT7uysumQU=;
 h=Message-ID:Date:MIME-Version:To:From;
 b=ihF/bUcmkB0vPm8EweR+bhESphyJsVE2A7pZD1bxge1K/Cra6+NeQzVjYAe5GeL/j
 vaDyolqDxZnVUVXzM2Hd1PtptcnFudh/Qh35TBFhU6mxh2ydjnJn6Ld36fuKNdswI/
 ni/1Q4NScfFlhwCsJ21c/CAIySv8QO0sz6my2gtds3PT88nO3+egndruMz8p8ihg7m
 EOKYEeW5pMXGxkO7EvhyGnTSQg1Yg5sx/0iruvSSIMTUyvWe40UweTpMAIe4NX2GQz
 M+Ph7M68u3adlMdxx6c4iHzdSaNW22mP8owoQSVfgfA4CCiR4QTAoMwOo44EKwSvHW
 nRXRIjBoM3wxg==
X-Virus-Scanned: amavisd-new at mail.cs.ucla.edu
Received: from mail.cs.ucla.edu ([127.0.0.1])
 by localhost (mail.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10026)
 with ESMTP id hpZkBCsrcJpZ; Fri,  7 Apr 2023 12:00:16 -0700 (PDT)
Received: from [192.168.1.9] (cpe-172-91-119-151.socal.res.rr.com
 [172.91.119.151])
 by mail.cs.ucla.edu (Postfix) with ESMTPSA id C1CF63C09FA02;
 Fri,  7 Apr 2023 12:00:16 -0700 (PDT)
Message-ID: <065bcdcb-5770-5384-5afe-4a4d29272274@HIDDEN>
Date: Fri, 7 Apr 2023 12:00:16 -0700
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.9.0
Content-Language: en-US
References: <2554712d-e386-3bab-bc6c-1f0e85d999db@HIDDEN>
 <CAPUEspj1m6F0_XgOFUVaq3Aq_Ah3PzCUs7YUyFH9_Zz-MOYTTA@HIDDEN>
 <96358c4e-7200-e5a5-869e-5da9d0de3503@HIDDEN>
 <xmqqttxvzbo8.fsf@HIDDEN>
 <CANgJU+U+xXsh9psd0z5Xjr+Se5QgdKkjQ7LUQ-PdUULSN3n4+g@HIDDEN>
From: Paul Eggert <eggert@HIDDEN>
Organization: UCLA Computer Science Department
In-Reply-To: <CANgJU+U+xXsh9psd0z5Xjr+Se5QgdKkjQ7LUQ-PdUULSN3n4+g@HIDDEN>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Spam-Score: -1.1 (-)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -2.1 (--)

On 2023-04-06 06:39, demerphq wrote:

> Unicode specifies that \d match any digit
> in any script that it supports.

"Specifies" is too strong. The Unicode Regular Expressions technical 
standard (UTS#18) mentions \d only in Annex C[1], next to the word 
"digit" in a column labeled "Property" (even though \d is really syntax 
not a property). This is at best an informal recommendation, not a 
requirement, as UTS#18 0.2[2] says that UTS#18's syntax is only for 
illustration and that although it's similar to Perl's, the two syntax 
forms may not be exactly the same. So we can't look to UTS#18 for a 
definitive way out of the \d mess, as the Unicode folks specifically 
delegated matters to us.

Even ignoring the \d issue the digit situation is messy. UTS#18 Annex C 
says "\p{gc=Decimal_Number}" is the standard recommended syntax 
assignment for digits. However, PCRE2 does not support this syntax; it 
supports another variant \p{Nd} that UTS#18 also recommends. So it 
appears that PCRE2 already does not implement every recommended aspect 
of UTS#18 syntax. PCRE2 also doesn't match Perl, which does support 
"\p{gc=Decimal_Number}".

Anyway, since grep -P '\p{Nd}' implements Unicode's decimal digit class, 
that's clearly enough for grep -P to conform to UTS#18 with respect to 
digits.


> A) how do you tell the regular expression
> engine what semantics you want and B) how does the regular expression
> library identify the encoding in the file, and how does it handle
> malformed content in that file.

Here's how GNU grep does it:

* RE semantics are specified via command-line options like -P.

* Text encoding is specified by locale, e.g., LC_ALL='en_US.utf8'.

* REs do not match encoding errors.


> on *nix there is no tradition of using BOM's to
> distinguish the 6 different possible encodings of Unicode (UTF-8,
> UTF-EBCDIC, UTF-16LE, UTF-16BE, UTF-32LE, UTF-32BE)

Yes, GNU/Linux never really experienced the joys of UTF-EBCDIC, Oracle 
UTFE, UTF-16LE vs UTF-16BE etc. If you're running legacy IBM mainframe 
or MS-Windows code these legacy encodings are obviously a big deal. 
However, there seems little reason to force their nontrivial hassles 
onto every GNU/Linux program that processes text. A few specialized apps 
like 'iconv' deal with offbeat encodings, and that is probably a better 
approach all around.


> there seems
> to be some level of desire of matching with unicode semantics against
> files that are not uniformly encoded in one of these formats.

That is a use case, yes. It's what 'strings' and 'grep' do.


[1]: https://unicode.org/reports/tr18/#Compatibility_Properties
[2]: https://unicode.org/reports/tr18/#Conformance

Message sent to bug-grep@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#60690: -P '\d' in GNU and git grep
Resent-From: Carlo Arenas <carenas@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-grep@HIDDEN
Resent-Date: Sat, 08 Apr 2023 05:02:01 +0000
Resent-Message-ID: <handler.60690.B60690.168093009615566 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 60690
X-GNU-PR-Package: grep
X-GNU-PR-Keywords: patch
To: Paul Eggert <eggert@HIDDEN>
Cc: demerphq <demerphq@HIDDEN>, pcre2-dev@HIDDEN, 60690 <at> debbugs.gnu.org, mega lith01 <megalith01@HIDDEN>, =?UTF-8?Q?=C3=86var_?= =?UTF-8?Q?Arnfj=C3=B6r=C3=B0?= Bjarmason <avarab@HIDDEN>, Junio C Hamano <gitster@HIDDEN>, =?UTF-8?Q?Tukusej=E2=80=99s?= Sirs <tukusejssirs@HIDDEN>, git@HIDDEN
Received: via spool by 60690-submit <at> debbugs.gnu.org id=B60690.168093009615566
          (code B ref 60690); Sat, 08 Apr 2023 05:02:01 +0000
Received: (at 60690) by debbugs.gnu.org; 8 Apr 2023 05:01:36 +0000
Received: from localhost ([127.0.0.1]:57538 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1pl0hj-000430-Li
	for submit <at> debbugs.gnu.org; Sat, 08 Apr 2023 01:01:36 -0400
Received: from mail-wm1-f49.google.com ([209.85.128.49]:33695)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <carenas@HIDDEN>) id 1pl0hg-00042m-Ut
 for 60690 <at> debbugs.gnu.org; Sat, 08 Apr 2023 01:01:33 -0400
Received: by mail-wm1-f49.google.com with SMTP id
 v20-20020a05600c471400b003ed8826253aso5156107wmo.0
 for <60690 <at> debbugs.gnu.org>; Fri, 07 Apr 2023 22:01:32 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=gmail.com; s=20210112; t=1680930087; x=1683522087;
 h=content-transfer-encoding:cc:to:subject:message-id:date:from
 :in-reply-to:references:mime-version:from:to:cc:subject:date
 :message-id:reply-to;
 bh=NF72RZy6Lg3Abkq5r49TDt8ODdQqgH+JoDE9ap8+4F4=;
 b=RkDOIn1VvUUUJJTAd8XS+cL4rPVeUVl6zGK2hKOgtuZnqqV5QUJX5S5t+mNnBuMt8H
 Nnn8zTuPckM2vaCqMSs5H8PbF4nwSpe9xEaFNYkQXIVjVW6aZpPgzS0Xcp4XNMm9R5hE
 MMBXIMv0Q0XIXfibQQhUZL7zeKWgjB7JW/rY2D+uxAYfKbzTYENLOwCSBL8N6h9fnebv
 WVtAogGGlTp8QYIHQzY/BowSqZ/EC9I2JNScbKTviV2ph8DU7Bxbd745pJltWYv9+Z+L
 E4LmisqGfK/SNiaTHn//+aks0Oig10/Y8FRKOdObAIMDEfCZY1/6cRnhoQge/nm1Z5IB
 zzzA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20210112; t=1680930087; x=1683522087;
 h=content-transfer-encoding:cc:to:subject:message-id:date:from
 :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc
 :subject:date:message-id:reply-to;
 bh=NF72RZy6Lg3Abkq5r49TDt8ODdQqgH+JoDE9ap8+4F4=;
 b=zJ7IISlj8KF471B27GePPl1oWlU16Wr7lTjEa1MxMK5pKamgghULeOObUuEA2Zzokp
 PuHqon0Yh7MGxLCCb1coHqQldGKfigLKC7Pp5TVAs/i8hbaUSNubFKeO+CXlLAYhiDBy
 Xh65EZiAeE/0OiFThzK91upIWhPjXDqJUlhNqYNYxd6Ihm2InhyzmkroHj9Oazm1LUHP
 74P7XyyAovLzTyECVhv1Dn7EvjVcdS6+9fUn4qfsxjS+Mi8al25q/iIHJ/ePQBBdcJl7
 Nnoonn3CwTlLvtBQ9vPxPDWzb10vdYvGH7FY77sltuC5FVEIfxtxt5gPz3BZm8+oJl2u
 ibdw==
X-Gm-Message-State: AAQBX9djMu84y91wGw/0+mry0mhcBN5G8XAaiJV3Hst+R4OLGNdiDALZ
 4Xd0J+zxFMWPbxoOZ5mm1C55x8GCQ3okxjT3lCHmL02VHCA=
X-Google-Smtp-Source: AKy350bH7nP8YmEVysXIlQlls42DSYQTM4X3Sfk0P0GokhzgQMdR1aXzdqLAIRuksRug0GVyeuyA1sO3LinceVZDyBY=
X-Received: by 2002:a05:600c:3786:b0:3ee:143f:786d with SMTP id
 o6-20020a05600c378600b003ee143f786dmr158379wmr.4.1680930086816; Fri, 07 Apr
 2023 22:01:26 -0700 (PDT)
MIME-Version: 1.0
References: <2554712d-e386-3bab-bc6c-1f0e85d999db@HIDDEN>
 <CAPUEspj1m6F0_XgOFUVaq3Aq_Ah3PzCUs7YUyFH9_Zz-MOYTTA@HIDDEN>
 <96358c4e-7200-e5a5-869e-5da9d0de3503@HIDDEN>
 <xmqqttxvzbo8.fsf@HIDDEN>
 <CANgJU+U+xXsh9psd0z5Xjr+Se5QgdKkjQ7LUQ-PdUULSN3n4+g@HIDDEN>
 <065bcdcb-5770-5384-5afe-4a4d29272274@HIDDEN>
In-Reply-To: <065bcdcb-5770-5384-5afe-4a4d29272274@HIDDEN>
From: Carlo Arenas <carenas@HIDDEN>
Date: Fri, 7 Apr 2023 22:01:14 -0700
Message-ID: <CAPUEspjtN-cwm=Nn=hMCcbOcOgPaVHsBfLW9TXn1HZrxtRR3BQ@HIDDEN>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: 0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

On Fri, Apr 7, 2023 at 12:00=E2=80=AFPM Paul Eggert <eggert@HIDDEN> wr=
ote:
>
> On 2023-04-06 06:39, demerphq wrote:
>
> > Unicode specifies that \d match any digit
> > in any script that it supports.
>
> "Specifies" is too strong. The Unicode Regular Expressions technical
> standard (UTS#18) mentions \d only in Annex C[1], next to the word
> "digit" in a column labeled "Property" (even though \d is really syntax
> not a property). This is at best an informal recommendation, not a
> requirement, as UTS#18 0.2[2] says that UTS#18's syntax is only for
> illustration and that although it's similar to Perl's, the two syntax
> forms may not be exactly the same. So we can't look to UTS#18 for a
> definitive way out of the \d mess, as the Unicode folks specifically
> delegated matters to us.
>
> Even ignoring the \d issue the digit situation is messy. UTS#18 Annex C
> says "\p{gc=3DDecimal_Number}" is the standard recommended syntax
> assignment for digits. However, PCRE2 does not support this syntax; it
> supports another variant \p{Nd} that UTS#18 also recommends. So it
> appears that PCRE2 already does not implement every recommended aspect
> of UTS#18 syntax. PCRE2 also doesn't match Perl, which does support
> "\p{gc=3DDecimal_Number}".

Not sure I follow the whole logic here, but PCRE2[3] (search for
"general category" which is what the "gc" above stands for) only
supports the abbreviated form of the unicode classes and `Nd` is
indeed the one that corresponds to `Decimal_Number`.

Carlo

[1]: https://unicode.org/reports/tr18/#Compatibility_Properties
[2]: https://unicode.org/reports/tr18/#Conformance
[3]: https://pcre2project.github.io/pcre2/doc/html/pcre2pattern.html

Message sent to bug-grep@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#60690: -P '\d' in GNU and git grep
Resent-From: Paul Eggert <eggert@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-grep@HIDDEN
Resent-Date: Sat, 08 Apr 2023 22:46:02 +0000
Resent-Message-ID: <handler.60690.B60690.16809939307090 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 60690
X-GNU-PR-Package: grep
X-GNU-PR-Keywords: patch
To: Carlo Arenas <carenas@HIDDEN>
Cc: demerphq <demerphq@HIDDEN>, pcre2-dev@HIDDEN, 60690 <at> debbugs.gnu.org, mega lith01 <megalith01@HIDDEN>, =?UTF-8?Q?=C3=86var_?= =?UTF-8?Q?Arnfj=C3=B6r=C3=B0?= Bjarmason <avarab@HIDDEN>, Junio C Hamano <gitster@HIDDEN>, =?UTF-8?Q?Tukusej=E2=80=99s?= Sirs <tukusejssirs@HIDDEN>, git@HIDDEN
Received: via spool by 60690-submit <at> debbugs.gnu.org id=B60690.16809939307090
          (code B ref 60690); Sat, 08 Apr 2023 22:46:02 +0000
Received: (at 60690) by debbugs.gnu.org; 8 Apr 2023 22:45:30 +0000
Received: from localhost ([127.0.0.1]:59791 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1plHJJ-0001qI-Nz
	for submit <at> debbugs.gnu.org; Sat, 08 Apr 2023 18:45:29 -0400
Received: from mail.cs.ucla.edu ([131.179.128.66]:37202)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eggert@HIDDEN>) id 1plHJI-0001q4-96
 for 60690 <at> debbugs.gnu.org; Sat, 08 Apr 2023 18:45:29 -0400
Received: from localhost (localhost [127.0.0.1])
 by mail.cs.ucla.edu (Postfix) with ESMTP id 41B333C09FA05;
 Sat,  8 Apr 2023 15:45:22 -0700 (PDT)
Received: from mail.cs.ucla.edu ([127.0.0.1])
 by localhost (mail.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10032)
 with ESMTP id bqtIu2mkYvMC; Sat,  8 Apr 2023 15:45:21 -0700 (PDT)
Received: from localhost (localhost [127.0.0.1])
 by mail.cs.ucla.edu (Postfix) with ESMTP id 3060E3C09FA06;
 Sat,  8 Apr 2023 15:45:21 -0700 (PDT)
DKIM-Filter: OpenDKIM Filter v2.10.3 mail.cs.ucla.edu 3060E3C09FA06
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cs.ucla.edu;
 s=9D0B346E-2AEB-11ED-9476-E14B719DCE6C; t=1680993921;
 bh=wfSiy/8QBl9l5t1AHfLCq1MtZCAo+ITeL91CiR2IfSI=;
 h=Message-ID:Date:MIME-Version:To:From;
 b=KEAt4wad8SgYYOCJHM5fxsHoUZJRX8HMQdovS7SeowO071mZ5rKSb3j5fisu7KfY0
 HKx539tdnVAGUS8PpHxS+2xvBneQjlFM/rpwCuaR+55NnBKyCw8oUhBxQCsjKbfDHL
 G+9b6XuJv5tqiMmX/cF+YUzD4/awMgdO2O86Tc9NvK7mflbXqj7ss0wwxcoV95KsDf
 kZS+LpBXDUe0Lp8qXMBfc895cHiAIu29FGcl1eAwt8CD5iS1gP3l5wC+LiLSEx8/IP
 BIDcWIPeIVj/TDsT6k3rHL5QlgWN+C6ZtajvPYgwZasLyKkwxLo5E4z32GBrVEEcE+
 ubVRY8Xvb/0yw==
X-Virus-Scanned: amavisd-new at mail.cs.ucla.edu
Received: from mail.cs.ucla.edu ([127.0.0.1])
 by localhost (mail.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10026)
 with ESMTP id tok1hIEic2OY; Sat,  8 Apr 2023 15:45:21 -0700 (PDT)
Received: from [192.168.1.9] (cpe-172-91-119-151.socal.res.rr.com
 [172.91.119.151])
 by mail.cs.ucla.edu (Postfix) with ESMTPSA id E68933C09FA05;
 Sat,  8 Apr 2023 15:45:20 -0700 (PDT)
Message-ID: <43d04851-2463-2922-44e3-075080129ec3@HIDDEN>
Date: Sat, 8 Apr 2023 15:45:20 -0700
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.9.0
Content-Language: en-US
References: <2554712d-e386-3bab-bc6c-1f0e85d999db@HIDDEN>
 <CAPUEspj1m6F0_XgOFUVaq3Aq_Ah3PzCUs7YUyFH9_Zz-MOYTTA@HIDDEN>
 <96358c4e-7200-e5a5-869e-5da9d0de3503@HIDDEN>
 <xmqqttxvzbo8.fsf@HIDDEN>
 <CANgJU+U+xXsh9psd0z5Xjr+Se5QgdKkjQ7LUQ-PdUULSN3n4+g@HIDDEN>
 <065bcdcb-5770-5384-5afe-4a4d29272274@HIDDEN>
 <CAPUEspjtN-cwm=Nn=hMCcbOcOgPaVHsBfLW9TXn1HZrxtRR3BQ@HIDDEN>
From: Paul Eggert <eggert@HIDDEN>
Organization: UCLA Computer Science Department
In-Reply-To: <CAPUEspjtN-cwm=Nn=hMCcbOcOgPaVHsBfLW9TXn1HZrxtRR3BQ@HIDDEN>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Spam-Score: -1.1 (-)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -2.1 (--)

On 2023-04-07 22:01, Carlo Arenas wrote:

> Not sure I follow the whole logic here, but PCRE2[3] (search for
> "general category" which is what the "gc" above stands for) only
> supports the abbreviated form of the unicode classes and `Nd` is
> indeed the one that corresponds to `Decimal_Number`.

That's fine: all that UTS#18[1] requires is that PCRE2 provide syntax 
for a regular expression that matches the Decimal Number class. Which 
PCRE2 does, via \p{Nd}.

The logic is that UTF#18 does not require that \d must behave like 
\p{Nd}, or even that \p{gc=Decimal_Number} must behave like \p{Nd}. It 
merely requires that there be some syntax for matching Decimal Number, 
and it says the choice of syntax is up to the implementer. This is why 
UTF#18 doesn't require that \d must also match non-ASCII digits (which 
is what I think Yves was saying).

[1]: https://unicode.org/reports/tr18/

Last modified: Sat, 8 Apr 2023 23:00:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.