GNU logs - #64277, boring messages

Message sent to bug-grep@HIDDEN:

Subject: bug#64277: [feature request] handle files encoded in utf-16le
Resent-From: Jeremy Hetzler <jeremyhetzler@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-grep@HIDDEN
Resent-Date: Sat, 24 Jun 2023 21:24:02 +0000
Resent-Message-ID: <handler.64277.B.168764182926557 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: 64277 <at> debbugs.gnu.org
MIME-Version: 1.0
From: Jeremy Hetzler <jeremyhetzler@HIDDEN>
Date: Sat, 24 Jun 2023 17:23:05 -0400
Message-ID: <CAOh4nmnCN1GHyGwbR85UEHNYpBKSQ1pjEioRvw88i5M-0Rh8bg@HIDDEN>
Content-Type: multipart/alternative; boundary="000000000000c3ee2e05fee6ba42"
Received-SPF: pass client-ip=2a00:1450:4864:20::130;
 envelope-from=jeremyhetzler@HIDDEN; helo=mail-lf1-x130.google.com
X-Spam_score_int: -20
X-Spam_score: -2.1
X-Spam_bar: --
X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,
 DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001,
 HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001,
 SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-Spam-Score: -1.3 (-)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -2.3 (--)

--000000000000c3ee2e05fee6ba42
Content-Type: text/plain; charset="UTF-8"

Maintainers,

I recently was confused as to why GNU grep did not find any matches in
certain files, when vim clearly showed that the search string was present.

Turns out the files (log files from a Windows application) are encoded in
UTF16-LE.

$ file '06-21-2023 03-22-46'
> 06-21-2023 03-22-46: Unicode text, UTF-16, little-endian text, with CRLF
> line terminators


> $ /bin/od -Ad -w16 -t cz '06-21-2023 03-22-46' | head -10
> 0000000 377 376   [  \0   H  \0   E  \0   A  \0   D  \0   E  \0   R  \0
>  >..[.H.E.A.D.E.R.<
> 0000016   :  \0   ]  \0  \r  \0  \n  \0   [  \0   I  \0   D  \0   r  \0
>  >:.].....[.I.D.r.<
> 0000032   i  \0   v  \0   e  \0      \0   v  \0   e  \0   r  \0   s  \0
>  >i.v.e. .v.e.r.s.<
> 0000048   i  \0   o  \0   n  \0   :  \0      \0   6  \0   .  \0   7  \0
>  >i.o.n.:. .6...7.<
> 0000064   .  \0   4  \0   .  \0   4  \0   6  \0      \0   R  \0   e  \0
>  >..4...4.6. .R.e.<
> 0000080   l  \0   e  \0   a  \0   s  \0   e  \0      \0   D  \0   a  \0
>  >l.e.a.s.e. .D.a.<
> 0000096   t  \0   e  \0   :  \0      \0   0  \0   6  \0   /  \0   1  \0
>  >t.e.:. .0.6./.1.<
> 0000112   6  \0   /  \0   2  \0   0  \0   2  \0   3  \0   ]  \0  \r  \0
>  >6./.2.0.2.3.]...<
> 0000128  \n  \0   [  \0   I  \0   n  \0   t  \0   e  \0   r  \0   a  \0
>  >..[.I.n.t.e.r.a.<
> 0000144   c  \0   t  \0   i  \0   v  \0   e  \0      \0   B  \0   a  \0
>  >c.t.i.v.e. .B.a.<


> $ grep --version
> grep (GNU grep) 3.11
> Packaged by Cygwin (3.11-1)
> Copyright (C) 2023 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <
> https://gnu.org/licenses/gpl.html>.
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.


>
There is no easy way to use grep to search these files, even if one knows
the encoding in advance.

I would like to request a feature to be added to grep which would enable it
to transparently decode UTF16-LE files so they can be conveniently searched.

Thanks,
Jeremy Hetzler
(he/him)

--000000000000c3ee2e05fee6ba42
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Maintainers,<br><br>I recently was confused as to why GNU =
grep did not find any matches in certain files, when vim clearly showed tha=
t the search string was present.<br><br>Turns out the files (log files from=
 a Windows application) are encoded in UTF16-LE.<br><br><blockquote class=
=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rg=
b(204,204,204);padding-left:1ex"><font face=3D"monospace">$ file &#39;06-21=
-2023 03-22-46&#39;<br></font><font face=3D"monospace">06-21-2023 03-22-46:=
 Unicode text, UTF-16, little-endian text, with CRLF line terminators</font=
></blockquote><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px=
 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><font face=
=3D"monospace"><br></font><font face=3D"monospace">$ /bin/od -Ad -w16 -t cz=
 &#39;06-21-2023 03-22-46&#39; | head -10<br></font><font face=3D"monospace=
">0000000 377 376 =C2=A0 [ =C2=A0\0 =C2=A0 H =C2=A0\0 =C2=A0 E =C2=A0\0 =C2=
=A0 A =C2=A0\0 =C2=A0 D =C2=A0\0 =C2=A0 E =C2=A0\0 =C2=A0 R =C2=A0\0 =C2=A0=
&gt;..[.H.E.A.D.E.R.&lt;<br></font><font face=3D"monospace">0000016 =C2=A0 =
: =C2=A0\0 =C2=A0 ] =C2=A0\0 =C2=A0\r =C2=A0\0 =C2=A0\n =C2=A0\0 =C2=A0 [ =
=C2=A0\0 =C2=A0 I =C2=A0\0 =C2=A0 D =C2=A0\0 =C2=A0 r =C2=A0\0 =C2=A0&gt;:.=
].....[.I.D.r.&lt;<br></font><font face=3D"monospace">0000032 =C2=A0 i =C2=
=A0\0 =C2=A0 v =C2=A0\0 =C2=A0 e =C2=A0\0 =C2=A0 =C2=A0 =C2=A0\0 =C2=A0 v =
=C2=A0\0 =C2=A0 e =C2=A0\0 =C2=A0 r =C2=A0\0 =C2=A0 s =C2=A0\0 =C2=A0&gt;i.=
v.e. .v.e.r.s.&lt;<br></font><font face=3D"monospace">0000048 =C2=A0 i =C2=
=A0\0 =C2=A0 o =C2=A0\0 =C2=A0 n =C2=A0\0 =C2=A0 : =C2=A0\0 =C2=A0 =C2=A0 =
=C2=A0\0 =C2=A0 6 =C2=A0\0 =C2=A0 . =C2=A0\0 =C2=A0 7 =C2=A0\0 =C2=A0&gt;i.=
o.n.:. .6...7.&lt;<br></font><font face=3D"monospace">0000064 =C2=A0 . =C2=
=A0\0 =C2=A0 4 =C2=A0\0 =C2=A0 . =C2=A0\0 =C2=A0 4 =C2=A0\0 =C2=A0 6 =C2=A0=
\0 =C2=A0 =C2=A0 =C2=A0\0 =C2=A0 R =C2=A0\0 =C2=A0 e =C2=A0\0 =C2=A0&gt;..4=
...4.6. .R.e.&lt;<br></font><font face=3D"monospace">0000080 =C2=A0 l =C2=
=A0\0 =C2=A0 e =C2=A0\0 =C2=A0 a =C2=A0\0 =C2=A0 s =C2=A0\0 =C2=A0 e =C2=A0=
\0 =C2=A0 =C2=A0 =C2=A0\0 =C2=A0 D =C2=A0\0 =C2=A0 a =C2=A0\0 =C2=A0&gt;l.e=
.a.s.e. .D.a.&lt;<br></font><font face=3D"monospace">0000096 =C2=A0 t =C2=
=A0\0 =C2=A0 e =C2=A0\0 =C2=A0 : =C2=A0\0 =C2=A0 =C2=A0 =C2=A0\0 =C2=A0 0 =
=C2=A0\0 =C2=A0 6 =C2=A0\0 =C2=A0 / =C2=A0\0 =C2=A0 1 =C2=A0\0 =C2=A0&gt;t.=
e.:. .0.6./.1.&lt;<br></font><font face=3D"monospace">0000112 =C2=A0 6 =C2=
=A0\0 =C2=A0 / =C2=A0\0 =C2=A0 2 =C2=A0\0 =C2=A0 0 =C2=A0\0 =C2=A0 2 =C2=A0=
\0 =C2=A0 3 =C2=A0\0 =C2=A0 ] =C2=A0\0 =C2=A0\r =C2=A0\0 =C2=A0&gt;6./.2.0.=
2.3.]...&lt;<br></font><font face=3D"monospace">0000128 =C2=A0\n =C2=A0\0 =
=C2=A0 [ =C2=A0\0 =C2=A0 I =C2=A0\0 =C2=A0 n =C2=A0\0 =C2=A0 t =C2=A0\0 =C2=
=A0 e =C2=A0\0 =C2=A0 r =C2=A0\0 =C2=A0 a =C2=A0\0 =C2=A0&gt;..[.I.n.t.e.r.=
a.&lt;<br></font><font face=3D"monospace">0000144 =C2=A0 c =C2=A0\0 =C2=A0 =
t =C2=A0\0 =C2=A0 i =C2=A0\0 =C2=A0 v =C2=A0\0 =C2=A0 e =C2=A0\0 =C2=A0 =C2=
=A0 =C2=A0\0 =C2=A0 B =C2=A0\0 =C2=A0 a =C2=A0\0 =C2=A0&gt;c.t.i.v.e. .B.a.=
&lt;</font></blockquote><blockquote class=3D"gmail_quote" style=3D"margin:0=
px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><=
font face=3D"monospace"><br></font><font face=3D"monospace">$ grep --versio=
n<br></font><font face=3D"monospace">grep (GNU grep) 3.11<br></font><font f=
ace=3D"monospace">Packaged by Cygwin (3.11-1)<br></font><font face=3D"monos=
pace">Copyright (C) 2023 Free Software Foundation, Inc.<br></font><font fac=
e=3D"monospace">License GPLv3+: GNU GPL version 3 or later &lt;<a href=3D"h=
ttps://gnu.org/licenses/gpl.html">https://gnu.org/licenses/gpl.html</a>&gt;=
.<br></font><font face=3D"monospace">This is free software: you are free to=
 change and redistribute it.<br></font><font face=3D"monospace">There is NO=
 WARRANTY, to the extent permitted by law.</font></blockquote><div><blockqu=
ote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px=
 solid rgb(204,204,204);padding-left:1ex"><br></blockquote></div><br>There =
is no easy way to use grep to search these files, even if one knows the enc=
oding in advance.<div><br></div><div><div>I would like to request a feature=
 to be added to grep which would enable it to transparently decode UTF16-LE=
 files so they can be conveniently searched.</div><div><br></div><div><div>=
<div><div><div><div><div><div>Thanks,<br>Jeremy Hetzler<br>(he/him)</div></=
div></div></div></div></div></div></div></div><div><br></div><div><br></div=
></div>

--000000000000c3ee2e05fee6ba42--

Message sent:

Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Mailer: MIME-tools 5.505 (Entity 5.505)
Content-Type: text/plain; charset=utf-8
X-Loop: help-debbugs@HIDDEN
From: help-debbugs@HIDDEN (GNU bug Tracking System)
To: Jeremy Hetzler <jeremyhetzler@HIDDEN>
Subject: bug#64277: Acknowledgement ([feature request] handle files
 encoded in utf-16le)
Message-ID: <handler.64277.B.168764182926557.ack <at> debbugs.gnu.org>
References: <CAOh4nmnCN1GHyGwbR85UEHNYpBKSQ1pjEioRvw88i5M-0Rh8bg@HIDDEN>
X-Gnu-PR-Message: ack 64277
X-Gnu-PR-Package: grep
Reply-To: 64277 <at> debbugs.gnu.org
Date: Sat, 24 Jun 2023 21:24:02 +0000

Thank you for filing a new bug report with debbugs.gnu.org.

This is an automatically generated reply to let you know your message
has been received.

Your message is being forwarded to the package maintainers and other
interested parties for their attention; they will reply in due course.

Your message has been sent to the package maintainer(s):
 bug-grep@HIDDEN

If you wish to submit further information on this problem, please
send it to 64277 <at> debbugs.gnu.org.

Please do not send mail to help-debbugs@HIDDEN unless you wish
to report a problem with the Bug-tracking system.

--=20
64277: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D64277
GNU Bug Tracking System
Contact help-debbugs@HIDDEN with problems

Message sent to bug-grep@HIDDEN:

Subject: bug#64277: [feature request] handle files encoded in utf-16le
Resent-From: Paul Eggert <eggert@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-grep@HIDDEN
Resent-Date: Sat, 24 Jun 2023 22:18:01 +0000
Resent-Message-ID: <handler.64277.B64277.16876450641520 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: Jeremy Hetzler <jeremyhetzler@HIDDEN>
Cc: 64277 <at> debbugs.gnu.org
Message-ID: <d57fa2a2-642e-87c4-d4dd-44dc744a2ec0@HIDDEN>
Date: Sat, 24 Jun 2023 15:17:34 -0700
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.11.0
Content-Language: en-US
References: <CAOh4nmnCN1GHyGwbR85UEHNYpBKSQ1pjEioRvw88i5M-0Rh8bg@HIDDEN>
From: Paul Eggert <eggert@HIDDEN>
Organization: UCLA Computer Science Department
In-Reply-To: <CAOh4nmnCN1GHyGwbR85UEHNYpBKSQ1pjEioRvw88i5M-0Rh8bg@HIDDEN>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

On 2023-06-24 14:23, Jeremy Hetzler wrote:
> I would like to request a feature to be added to grep which would enable it
> to transparently decode UTF16-LE files so they can be conveniently searched.

Not sure it's worth the effort as this format is not that common for GNU 
grep, it'd be a pain to add proper support for it, and anyway 16-bit 
encodings have been problematic ever since Unicode crossed the 16-bit 
Rubicon. I'm not saying we'd reject a patch if someone wrote it, but I'd 
say it should be low priority.

Last modified: Sat, 24 Jun 2023 22:30:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.