GNU bug report logs - #30076
[PATCH] web: Recognize JSON content type as text.

Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.

Package: guile; Reported by: Arun Isaac <arunisaac@HIDDEN>; Keywords: patch; dated Thu, 11 Jan 2018 05:33:01 UTC; Maintainer for guile is bug-guile@HIDDEN.

Message received at 30076 <at> debbugs.gnu.org:


Received: (at 30076) by debbugs.gnu.org; 31 Jan 2018 03:31:49 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Jan 30 22:31:49 2018
Received: from localhost ([127.0.0.1]:50085 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1egj7l-0003TT-F0
	for submit <at> debbugs.gnu.org; Tue, 30 Jan 2018 22:31:49 -0500
Received: from world.peace.net ([50.252.239.5]:51134)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mhw@HIDDEN>) id 1egj7j-0003T9-1U
 for 30076 <at> debbugs.gnu.org; Tue, 30 Jan 2018 22:31:47 -0500
Received: from pool-72-93-27-251.bstnma.east.verizon.net ([72.93.27.251]
 helo=jojen)
 by world.peace.net with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.89) (envelope-from <mhw@HIDDEN>)
 id 1egj7c-0001US-I2; Tue, 30 Jan 2018 22:31:40 -0500
From: Mark H Weaver <mhw@HIDDEN>
To: Arun Isaac <arunisaac@HIDDEN>
Subject: Re: bug#30076: [PATCH] web: Recognize JSON content type as text.
References: <20180111053117.4597-1-arunisaac@HIDDEN>
Date: Tue, 30 Jan 2018 22:31:04 -0500
In-Reply-To: <20180111053117.4597-1-arunisaac@HIDDEN> (Arun Isaac's
 message of "Thu, 11 Jan 2018 11:01:17 +0530")
Message-ID: <87y3kevh53.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.3 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 30076
Cc: 30076 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: 0.0 (/)

Hi Arun,

Arun Isaac <arunisaac@HIDDEN> writes:
> * module/web/response.scm (text-content-type?): Recognize JSON content
>   type as text.

While this would seem reasonable at first glance, it seems to me that
this will result in JSON texts with non-ASCII characters being
mishandled in many cases.

Within Guile, 'text-content-type?' is currently used in two places:

* 'decode-response-body' in (web client), and
* 'response-body-port' in (web response).

In both places, if 'text-content-type?' returns true, the encoding of
the response is assumed to be "ISO-8859-1" if not otherwise specified by
an explicit 'charset' parameter.  This is what RFC 2616 specifies for
text/plain, although RFC 6657 would change the default to US-ASCII, as
it was in RFC 2046, and maybe we should look into that.

However, things are quite different for the application/json MIME type,
as specified in RFCs 4627 and 7159.  Those RFCs specify that JSON text
"SHALL" (i.e. MUST) be encoded in Unicode (UTF-8, UTF-16 or UTF-32),
that the default encoding is UTF-8, and furthermore that no charset
parameter is defined for application/json.

So, we can expect at least some conforming implementations to omit the
'charset' parameter, and yet in that case we must assume that the
encoding is Unicode, and most definitely not ISO-8859-1.

RFC 4627 makes the additional interesting observation (in section 3,
"encoding") that since the first two characters of JSON text will always
be ASCII, and since UTF-8/UTF-16/UTF-32 are the only valid encodings for
JSON text, we can reliably determine the encoding by looking at the
pattern of nul bytes in the first four octets:

           00 00 00 xx  UTF-32BE
           00 xx 00 xx  UTF-16BE
           xx 00 00 00  UTF-32LE
           xx 00 xx 00  UTF-16LE
           xx xx xx xx  UTF-8

Given that any of these encodings above are possible, and that there is
no 'charset' parameter defined for "application/json", it seems to me
that we have no choice but to be prepared to auto-detect the encoding,
as described in RFC 4627 section 3 if the 'charset' parameter is
missing.

What do you think?

      Mark




Information forwarded to bug-guile@HIDDEN:
bug#30076; Package guile. Full text available.

Message received at submit <at> debbugs.gnu.org:


Received: (at submit) by debbugs.gnu.org; 11 Jan 2018 05:32:04 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Jan 11 00:32:04 2018
Received: from localhost ([127.0.0.1]:51636 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1eZVTA-0007bQ-D4
	for submit <at> debbugs.gnu.org; Thu, 11 Jan 2018 00:32:04 -0500
Received: from eggs.gnu.org ([208.118.235.92]:53455)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <arunisaac@HIDDEN>) id 1eZVT8-0007aw-L5
 for submit <at> debbugs.gnu.org; Thu, 11 Jan 2018 00:32:03 -0500
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <arunisaac@HIDDEN>) id 1eZVT2-0001zH-Nb
 for submit <at> debbugs.gnu.org; Thu, 11 Jan 2018 00:31:57 -0500
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00 autolearn=disabled
 version=3.3.2
Received: from lists.gnu.org ([2001:4830:134:3::11]:40661)
 by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32)
 (Exim 4.71) (envelope-from <arunisaac@HIDDEN>)
 id 1eZVT2-0001z4-KN
 for submit <at> debbugs.gnu.org; Thu, 11 Jan 2018 00:31:56 -0500
Received: from eggs.gnu.org ([2001:4830:134:3::10]:44418)
 by lists.gnu.org with esmtp (Exim 4.71)
 (envelope-from <arunisaac@HIDDEN>) id 1eZVT1-0001x0-Mr
 for bug-guile@HIDDEN; Thu, 11 Jan 2018 00:31:56 -0500
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <arunisaac@HIDDEN>) id 1eZVSy-0001v1-Fy
 for bug-guile@HIDDEN; Thu, 11 Jan 2018 00:31:55 -0500
Received: from vultr.systemreboot.net ([45.77.148.100]:53872)
 by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)
 (Exim 4.71) (envelope-from <arunisaac@HIDDEN>)
 id 1eZVSy-0001qK-0u
 for bug-guile@HIDDEN; Thu, 11 Jan 2018 00:31:52 -0500
Received: from [117.192.104.45] (helo=localhost.localdomain)
 by systemreboot.net with esmtpsa (TLSv1.2:ECDHE-RSA-AES128-GCM-SHA256:128)
 (Exim 4.90) (envelope-from <arunisaac@HIDDEN>)
 id 1eZVSg-00024Q-Oq; Thu, 11 Jan 2018 11:01:34 +0530
From: Arun Isaac <arunisaac@HIDDEN>
To: bug-guile@HIDDEN
Subject: [PATCH] web: Recognize JSON content type as text.
Date: Thu, 11 Jan 2018 11:01:17 +0530
Message-Id: <20180111053117.4597-1-arunisaac@HIDDEN>
X-Mailer: git-send-email 2.15.1
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
 [fuzzy]
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x
X-Received-From: 2001:4830:134:3::11
X-Spam-Score: -4.1 (----)
X-Debbugs-Envelope-To: submit
Cc: Arun Isaac <arunisaac@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -4.1 (----)

* module/web/response.scm (text-content-type?): Recognize JSON content
  type as text.
---
 module/web/response.scm | 1 +
 1 file changed, 1 insertion(+)

diff --git a/module/web/response.scm b/module/web/response.scm
index 06e1c6dc1..679304c4d 100644
--- a/module/web/response.scm
+++ b/module/web/response.scm
@@ -184,6 +184,7 @@ reason phrase for the response's code."
 represents a textual type such as `text/plain'."
   (let ((type (symbol->string type)))
     (or (string-prefix? "text/" type)
+        (string-suffix? "/json" type)
         (string-suffix? "/xml" type)
         (string-suffix? "+xml" type))))
 
-- 
2.15.1





Acknowledgement sent to Arun Isaac <arunisaac@HIDDEN>:
New bug report received and forwarded. Copy sent to bug-guile@HIDDEN. Full text available.
Report forwarded to bug-guile@HIDDEN:
bug#30076; Package guile. Full text available.
Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.
Last modified: Wed, 31 Jan 2018 03:45:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.