GNU logs - #31185, boring messages

Message sent to bug-diffutils@HIDDEN:

Subject: bug#31185: Why is there no full support for Unicode?
Resent-From: Keepun <keepun@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-diffutils@HIDDEN
Resent-Date: Mon, 16 Apr 2018 22:02:01 +0000
Resent-Message-ID: <handler.31185.B.152391608331935 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: 31185 <at> debbugs.gnu.org
From: Keepun <keepun@HIDDEN>
Message-ID: <7f6138f2-dec1-034d-7414-7c9749315291@HIDDEN>
Date: Tue, 17 Apr 2018 00:38:10 +0300
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101
 Thunderbird/45.8.0
MIME-Version: 1.0
Content-Type: multipart/alternative;
 boundary="------------1271067F6861B3321FDA72A8"
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

This is a multi-part message in MIME format.
--------------1271067F6861B3321FDA72A8
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit

Why is there no full support for Unicode?

Set the encoding using BOM.

The status of the binary file should be given only after checking 0x00 
characters.

BOM is part of the Unicode standard. 
http://www.unicode.org/faq/utf_bom.html#bom4

Files with encoding greater than 8 bits without BOM at the beginning can 
be immediately identified as binary.

My function in C#:

/// <summary>
/// </summary>
/// <param name="stream"></param>
/// <returns>null - binary</returns>
public static Encoding GetEncodingStream(Stream stream)
{
     BinaryReader bin = new BinaryReader(stream);
     byte[] bom = new byte[4];
     bin.BaseStream.Seek(0, SeekOrigin.Begin);
     bin.BaseStream.Read(bom, 0, bom.Length);
     bin.BaseStream.Seek(0, SeekOrigin.Begin);
     if (bom[0] == 0x00 && bom[1] == 0x00 && bom[2] == 0xFE && bom[3] == 
0xFF) {
         return new UTF32Encoding(true, true); // UTF-32, big-endian
     } else if (bom[0] == 0xFE && bom[1] == 0xFF) {
         return new UnicodeEncoding(true, true); // UTF-16, big-endian
     } else if (bom[0] == 0xFF && bom[1] == 0xFE) {
         if (bom[2] == 0x00 && bom[2] == 0x00) {
             return new UTF32Encoding(false, true); // UTF-32, little-endian
         } else {
             return new UnicodeEncoding(false, true); // UTF-16, 
little-endian
         }
     } else if (bom[0] == 0xEF && bom[1] == 0xBB && bom[2] == 0xBF) {
         return new UTF8Encoding(true);
     } else {
         bool binary = false;
         long fsize = bin.BaseStream.Length;
         if (fsize > 100000) {
             fsize = 100000;
         }
         byte[] bts = new byte[fsize];
         bin.BaseStream.Seek(0, SeekOrigin.Begin);
         bin.BaseStream.Read(bts, 0, (int)fsize);
         bin.BaseStream.Seek(0, SeekOrigin.Begin);
         for (int x = 0; x < fsize; x++) {
             if (bts[x] == 0) {
                 binary = true;
                 break;
             }
         }
         if (binary) {
             return null;
         }

         return Encoding.Default;
     }
}


--------------1271067F6861B3321FDA72A8
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit

<html>
  <head>

    <meta http-equiv="content-type" content="text/html; charset=utf-8">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    <span id="result_box" class="" lang="en"><span class="">Why is there
        no full support for Unicode?</span><br>
      <br>
      <span class="">Set the encoding using BOM.</span><br>
      <br>
      <span class="">The status of the binary file should be given only
        after checking 0x00 characters.</span><br>
      <br>
      <span class="">BOM is part of the Unicode standard.</span> <span><a class="moz-txt-link-freetext" href="http://www.unicode.org/faq/utf_bom.html#bom4">http://www.unicode.org/faq/utf_bom.html#bom4</a></span><br>
      <br>
      <span class="">Files with encoding greater than 8 bits without BOM
        at the beginning can be immediately identified as binary.</span><br>
      <br>
      <span class="">My function in C#:<br>
        <br>
      </span></span>/// &lt;summary&gt;<br>
    /// &lt;/summary&gt;<br>
    /// &lt;param name="stream"&gt;&lt;/param&gt;<br>
    /// &lt;returns&gt;null - binary&lt;/returns&gt;<br>
    public static Encoding GetEncodingStream(Stream stream)<br>
    {<br>
        BinaryReader bin = new BinaryReader(stream);<br>
        byte[] bom = new byte[4];<br>
        bin.BaseStream.Seek(0, SeekOrigin.Begin);<br>
        bin.BaseStream.Read(bom, 0, bom.Length);<br>
        bin.BaseStream.Seek(0, SeekOrigin.Begin);<br>
        if (bom[0] == 0x00 &amp;&amp; bom[1] == 0x00 &amp;&amp; bom[2]
    == 0xFE &amp;&amp; bom[3] == 0xFF) {<br>
            return new UTF32Encoding(true, true); // UTF-32, big-endian<br>
        } else if (bom[0] == 0xFE &amp;&amp; bom[1] == 0xFF) {<br>
            return new UnicodeEncoding(true, true); // UTF-16,
    big-endian<br>
        } else if (bom[0] == 0xFF &amp;&amp; bom[1] == 0xFE) {<br>
            if (bom[2] == 0x00 &amp;&amp; bom[2] == 0x00) {<br>
                return new UTF32Encoding(false, true); // UTF-32,
    little-endian<br>
            } else {<br>
                return new UnicodeEncoding(false, true); // UTF-16,
    little-endian<br>
            }<br>
        } else if (bom[0] == 0xEF &amp;&amp; bom[1] == 0xBB &amp;&amp;
    bom[2] == 0xBF) {<br>
            return new UTF8Encoding(true);<br>
        } else {<br>
            bool binary = false;<br>
            long fsize = bin.BaseStream.Length;<br>
            if (fsize &gt; 100000) {<br>
                fsize = 100000;<br>
            }<br>
            byte[] bts = new byte[fsize];<br>
            bin.BaseStream.Seek(0, SeekOrigin.Begin);<br>
            bin.BaseStream.Read(bts, 0, (int)fsize);<br>
            bin.BaseStream.Seek(0, SeekOrigin.Begin);<br>
            for (int x = 0; x &lt; fsize; x++) {<br>
                if (bts[x] == 0) {<br>
                    binary = true;<br>
                    break;<br>
                }<br>
            }<br>
            if (binary) {<br>
                return null;<br>
            }<br>
    <br>
            return Encoding.Default;<br>
        }<br>
    }<br>
    <br>
  </body>
</html>

--------------1271067F6861B3321FDA72A8--

Message sent:

Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Mailer: MIME-tools 5.505 (Entity 5.505)
Content-Type: text/plain; charset=utf-8
X-Loop: help-debbugs@HIDDEN
From: help-debbugs@HIDDEN (GNU bug Tracking System)
To: Keepun <keepun@HIDDEN>
Subject: bug#31185: Acknowledgement (Why is there no full support for
 Unicode?)
Message-ID: <handler.31185.B.152391608331935.ack <at> debbugs.gnu.org>
References: <7f6138f2-dec1-034d-7414-7c9749315291@HIDDEN>
X-Gnu-PR-Message: ack 31185
X-Gnu-PR-Package: diffutils
Reply-To: 31185 <at> debbugs.gnu.org
Date: Mon, 16 Apr 2018 22:02:02 +0000

Thank you for filing a new bug report with debbugs.gnu.org.

This is an automatically generated reply to let you know your message
has been received.

Your message is being forwarded to the package maintainers and other
interested parties for their attention; they will reply in due course.

Your message has been sent to the package maintainer(s):
 bug-diffutils@HIDDEN

If you wish to submit further information on this problem, please
send it to 31185 <at> debbugs.gnu.org.

Please do not send mail to help-debbugs@HIDDEN unless you wish
to report a problem with the Bug-tracking system.

--=20
31185: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D31185
GNU Bug Tracking System
Contact help-debbugs@HIDDEN with problems

Message sent to bug-diffutils@HIDDEN:

Subject: bug#31185: [bug-diffutils] bug#31185: Why is there no full support for Unicode?
Resent-From: Paul Eggert <eggert@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-diffutils@HIDDEN
Resent-Date: Tue, 17 Apr 2018 07:38:01 +0000
Resent-Message-ID: <handler.31185.B31185.15239506461398 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: Keepun <keepun@HIDDEN>, 31185 <at> debbugs.gnu.org
References: <7f6138f2-dec1-034d-7414-7c9749315291@HIDDEN>
From: Paul Eggert <eggert@HIDDEN>
Organization: UCLA Computer Science Department
Message-ID: <1e6ea3ef-86af-a130-30b2-df5e74207668@HIDDEN>
Date: Tue, 17 Apr 2018 00:37:18 -0700
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101
 Thunderbird/52.7.0
MIME-Version: 1.0
In-Reply-To: <7f6138f2-dec1-034d-7414-7c9749315291@HIDDEN>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

Keepun wrote:
> Files with encoding greater than 8 bits without BOM at the beginning can be 
> immediately identified as binary.

No, the BOM is not required or recommended in UTF-8, so it would be a mistake to 
identify GNU/Linux text files as binary merely because they lack a BOM. 
Typically these files do not have a BOM, and when they do one of the first 
things many users do is remove the BOM because it can cause trouble in practice.

Diffutils does not support UTF-16, where a BOM would make more sense, and there 
are no plans to add support for UTF-16 (or for UTF-32, for that matter).

Message sent to bug-diffutils@HIDDEN:

Subject: bug#31185: [bug-diffutils] bug#31185: Why is there no full support for Unicode?
Resent-From: Keepun <keepun@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-diffutils@HIDDEN
Resent-Date: Tue, 17 Apr 2018 20:28:02 +0000
Resent-Message-ID: <handler.31185.B31185.15239968702733 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: Paul Eggert <eggert@HIDDEN>, 31185 <at> debbugs.gnu.org
References: <7f6138f2-dec1-034d-7414-7c9749315291@HIDDEN>
 <1e6ea3ef-86af-a130-30b2-df5e74207668@HIDDEN>
From: Keepun <keepun@HIDDEN>
Message-ID: <b9cfe89a-fbf9-72d1-be65-06f5b1eba227@HIDDEN>
Date: Tue, 17 Apr 2018 23:27:36 +0300
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101
 Thunderbird/45.8.0
MIME-Version: 1.0
In-Reply-To: <1e6ea3ef-86af-a130-30b2-df5e74207668@HIDDEN>
Content-Type: multipart/alternative;
 boundary="------------2C700FED09D584661B5FE60C"
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

This is a multi-part message in MIME format.
--------------2C700FED09D584661B5FE60C
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit

UTF-8 does not require BOM, but for UTF-16 and UTF-32 BOM is always 
present. Files with UTF-16 and UTF-32 without the BOM should be 
identified as binary.

But why there are no plans to support UTF-16 and UTF-32? Diff is part of 
the Git and is used all over the world. Now 2018 and Unicode solved 
problems with encodings.


17.04.2018 10:37, Paul Eggert:
> Keepun wrote:
>> Files with encoding greater than 8 bits without BOM at the beginning 
>> can be immediately identified as binary.
>
> No, the BOM is not required or recommended in UTF-8, so it would be a 
> mistake to identify GNU/Linux text files as binary merely because they 
> lack a BOM. Typically these files do not have a BOM, and when they do 
> one of the first things many users do is remove the BOM because it can 
> cause trouble in practice.
>
> Diffutils does not support UTF-16, where a BOM would make more sense, 
> and there are no plans to add support for UTF-16 (or for UTF-32, for 
> that matter).


--------------2C700FED09D584661B5FE60C
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 7bit

<html>
  <head>
    <meta content="text/html; charset=utf-8" http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    <p><span id="result_box" class="" lang="en"><span>UTF-8 does not
          require BOM, but for UTF-16 and UTF-32 BOM is always present.</span>
        <span>Files with UTF-16 and UTF-32 without the BOM should be
          identified as binary.</span><br>
        <br>
        <span>But why there are no plans to support UTF-16 and UTF-32?</span>
        Diff <span class="">is part of the Git and is used all over the
          world.</span> <span class="">Now 2018 and Unicode solved
          problems with encodings.</span></span></p>
    <br>
    <div class="moz-cite-prefix">17.04.2018 10:37, Paul Eggert:<br>
    </div>
    <blockquote
      cite="mid:1e6ea3ef-86af-a130-30b2-df5e74207668@HIDDEN"
      type="cite">
      <div class="moz-text-flowed" style="font-family: -moz-fixed;
        font-size: 14px;" lang="x-unicode">Keepun wrote:
        <br>
        <blockquote type="cite" style="color: #000000;">Files with
          encoding greater than 8 bits without BOM at the beginning can
          be immediately identified as binary.
          <br>
        </blockquote>
        <br>
        No, the BOM is not required or recommended in UTF-8, so it would
        be a mistake to identify GNU/Linux text files as binary merely
        because they lack a BOM. Typically these files do not have a
        BOM, and when they do one of the first things many users do is
        remove the BOM because it can cause trouble in practice.
        <br>
        <br>
        Diffutils does not support UTF-16, where a BOM would make more
        sense, and there are no plans to add support for UTF-16 (or for
        UTF-32, for that matter).
        <br>
      </div>
    </blockquote>
    <br>
  </body>
</html>

--------------2C700FED09D584661B5FE60C--

Message sent to bug-diffutils@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#31185: [bug-diffutils] bug#31185: Why is there no full support for Unicode?
Resent-From: Paul Eggert <eggert@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-diffutils@HIDDEN
Resent-Date: Tue, 17 Apr 2018 20:46:01 +0000
Resent-Message-ID: <handler.31185.B31185.152399795415538 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 31185
X-GNU-PR-Package: diffutils
X-GNU-PR-Keywords: 
To: Keepun <keepun@HIDDEN>, 31185 <at> debbugs.gnu.org
Received: via spool by 31185-submit <at> debbugs.gnu.org id=B31185.152399795415538
          (code B ref 31185); Tue, 17 Apr 2018 20:46:01 +0000
Received: (at 31185) by debbugs.gnu.org; 17 Apr 2018 20:45:54 +0000
Received: from localhost ([127.0.0.1]:58004 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1f8XU9-000429-Hp
	for submit <at> debbugs.gnu.org; Tue, 17 Apr 2018 16:45:53 -0400
Received: from zimbra.cs.ucla.edu ([131.179.128.68]:33518)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eggert@HIDDEN>) id 1f8XU7-0003uP-KT
 for 31185 <at> debbugs.gnu.org; Tue, 17 Apr 2018 16:45:52 -0400
Received: from localhost (localhost [127.0.0.1])
 by zimbra.cs.ucla.edu (Postfix) with ESMTP id B327F1615FB;
 Tue, 17 Apr 2018 13:45:45 -0700 (PDT)
Received: from zimbra.cs.ucla.edu ([127.0.0.1])
 by localhost (zimbra.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10032)
 with ESMTP id n3FFr21POCVP; Tue, 17 Apr 2018 13:45:40 -0700 (PDT)
Received: from localhost (localhost [127.0.0.1])
 by zimbra.cs.ucla.edu (Postfix) with ESMTP id E3958161611;
 Tue, 17 Apr 2018 13:45:40 -0700 (PDT)
X-Virus-Scanned: amavisd-new at zimbra.cs.ucla.edu
Received: from zimbra.cs.ucla.edu ([127.0.0.1])
 by localhost (zimbra.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10026)
 with ESMTP id R-OmRoN61aGZ; Tue, 17 Apr 2018 13:45:40 -0700 (PDT)
Received: from Penguin.CS.UCLA.EDU (Penguin.CS.UCLA.EDU [131.179.64.200])
 by zimbra.cs.ucla.edu (Postfix) with ESMTPSA id C8DD51615FB;
 Tue, 17 Apr 2018 13:45:40 -0700 (PDT)
References: <7f6138f2-dec1-034d-7414-7c9749315291@HIDDEN>
 <1e6ea3ef-86af-a130-30b2-df5e74207668@HIDDEN>
 <b9cfe89a-fbf9-72d1-be65-06f5b1eba227@HIDDEN>
From: Paul Eggert <eggert@HIDDEN>
Organization: UCLA Computer Science Department
Message-ID: <3cdfd123-75fa-2783-df8c-c0250236334b@HIDDEN>
Date: Tue, 17 Apr 2018 13:45:40 -0700
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101
 Thunderbird/52.7.0
MIME-Version: 1.0
In-Reply-To: <b9cfe89a-fbf9-72d1-be65-06f5b1eba227@HIDDEN>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
Content-Language: en-US
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

On 04/17/2018 01:27 PM, Keepun wrote:
> why there are no plans to support UTF-16 and UTF-32?

Nobody has volunteered to do it, and there hasn't been a pressing need. 
UTF-16 and UTF-32 are primarily used for internal representation, not 
for text files. For more on the subject, please see:

http://utf8everywhere.org/

Last modified: Mon, 25 Nov 2019 12:00:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.