GNU bug report logs - #33133
26.1.50; zlib-decompress-region too rigid

Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.

Package: emacs; Reported by: Katsumi Yamaoka <yamaoka@HIDDEN>; Keywords: patch; dated Tue, 23 Oct 2018 23:10:02 UTC; Maintainer for emacs is bug-gnu-emacs@HIDDEN.
Added tag(s) patch. Request was from Noam Postavsky <npostavs@HIDDEN> to control <at> debbugs.gnu.org. Full text available.

Message received at 33133 <at> debbugs.gnu.org:


Received: (at 33133) by debbugs.gnu.org; 27 Oct 2018 21:48:36 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Oct 27 17:48:36 2018
Received: from localhost ([127.0.0.1]:45869 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gGWRf-0004Fe-Tm
	for submit <at> debbugs.gnu.org; Sat, 27 Oct 2018 17:48:36 -0400
Received: from mail-it1-f180.google.com ([209.85.166.180]:55239)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <npostavs@HIDDEN>)
 id 1gGWRe-0004FM-08; Sat, 27 Oct 2018 17:48:34 -0400
Received: by mail-it1-f180.google.com with SMTP id l191-v6so5460625ita.4;
 Sat, 27 Oct 2018 14:48:33 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=from:to:cc:subject:references:date:in-reply-to:message-id
 :user-agent:mime-version;
 bh=QCHdACLKeS8jzzr8o1vAVC4Bn4t/JajO+czw+Mm5V/8=;
 b=M24ashCPLlIc7mhvtcSck0NouLi1e358wZLglW/wWnLrKuoE3TGOl62k7mLvtwTmqJ
 /pu7fUFLgmGyWgoCsgVQLzoRE12IQGLlxW5KVv8RtV4udAV6HlBkvR4SONv78bCjwufB
 Ic8N2pjPjgi9m0RgLuU/ySufmMtCJ4kscKSXwpRynXZk/3aEATfIGBqaNDNpXFAv5GDb
 GcZuk3+cZHelO8iZe1K/CykmVRSDHXb3/MLM/NRWmknRfYe2ZelZogRXFi2fFVffXXrz
 rS3cGdtxj9k9nyL+enI6fAfvaLOkwm/nfTn96DEVQXNfIhOQBLArpY7XJzJwOJKVPkWu
 29BA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to
 :message-id:user-agent:mime-version;
 bh=QCHdACLKeS8jzzr8o1vAVC4Bn4t/JajO+czw+Mm5V/8=;
 b=ons+D0oalNnVQ8T3r7go9gCa+JZE49aAPBgNuXEgMtRrxVxXfuha/H8SW2AUV8mz5i
 lQkEyLRbDchws++u+VOZBhC5CFGjZYldOTnUeWZ9l1+LIrTkcndUbcEpH4tAKpDHsDjA
 hgK1eYRkDq3WGWRFTSpur5eE7jxSxJglvoH6Q/2TaT314g3azedc0jMHSYGlUnhmiizb
 Abx0W5PZOGhKhusCVx+3JrnRntfMMU9fTTQwmh+pEoqjo0CFC1LNie8SXbsk/pA5KEl5
 dLDeGrCbAq+MR9WB8K97TbwHEJEcjdU/eySuRSjJpGez3zKde2Qo/MWxTWqfvRcyUnVe
 mc7Q==
X-Gm-Message-State: AGRZ1gLhUGInG1KWaKdFGkJfLCQi12ueHyEnfPp2KaosQ+ZpyFr2leNO
 KAVF1vmhlfcxYbCnoKMMi5lFutK7
X-Google-Smtp-Source: AJdET5exkDXcHLoyvuEhC7FACp0w6QmcIjdVj2m3EM8WTgKIqmEM7Ll59NueDo2On+vO/YBitmV4mA==
X-Received: by 2002:a02:c18:: with SMTP id
 g24-v6mr6457762jad.131.1540676908126; 
 Sat, 27 Oct 2018 14:48:28 -0700 (PDT)
Received: from zebian (cbl-45-2-119-34.yyz.frontiernetworks.ca. [45.2.119.34])
 by smtp.googlemail.com with ESMTPSA id
 u68-v6sm3208552itd.1.2018.10.27.14.48.27
 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256);
 Sat, 27 Oct 2018 14:48:27 -0700 (PDT)
From: Noam Postavsky <npostavs@HIDDEN>
To: Katsumi Yamaoka <yamaoka@HIDDEN>
Subject: Re: bug#33133: 26.1.50; zlib-decompress-region too rigid
References: <b4my3aow9bt.fsf@HIDDEN> <87a7n4mbos.fsf@HIDDEN>
 <b4mpnw0qh3z.fsf@HIDDEN>
Date: Sat, 27 Oct 2018 17:48:26 -0400
In-Reply-To: <b4mpnw0qh3z.fsf@HIDDEN> (Katsumi Yamaoka's message of "Wed, 24
 Oct 2018 10:16:16 +0900")
Message-ID: <87efcbjc2d.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux)
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="=-=-="
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 33133
Cc: Kevin Ryde <user42_kevin@HIDDEN>, 33133 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

--=-=-=
Content-Type: text/plain

tags 33133 + patch
quit

Katsumi Yamaoka <yamaoka@HIDDEN> writes:

> On Tue, 23 Oct 2018 20:26:59 -0400, Noam Postavsky wrote:
>> --- i/src/decompress.c
>> +++ w/src/decompress.c
>> @@ -206,7 +206,7 @@ DEFUN ("zlib-decompress-region", Fzlib_decompress_region,
>
>>    while (inflate_status == Z_OK);
>
>> -  if (inflate_status != Z_STREAM_END)
>> +  if (inflate_status != Z_STREAM_END && inflate_status != Z_BUF_ERROR)
>>      return unbind_to (count, Qnil);
>
>>    unwind_data.start = 0;
>
> I confirmed that it makes it work for the corrupted web site in
> question.  Thank you!

Here's a proper patch.


--=-=-=
Content-Type: text/plain
Content-Disposition: attachment;
 filename=v1-0001-Allow-partial-decompression-Bug-33133.patch
Content-Description: patch

From 430ebd936b0bc41bd3e33e171938161846597196 Mon Sep 17 00:00:00 2001
From: Noam Postavsky <npostavs@HIDDEN>
Date: Sat, 27 Oct 2018 17:45:52 -0400
Subject: [PATCH v1] Allow partial decompression (Bug#33133)

* src/decompress.c (Fzlib_decompress_region): Add optional
ALLOW-PARTIAL parameter.
* lisp/url/url-http.el (url-handle-content-transfer-encoding): Use it.
* doc/lispref/text.texi (Decompression): Document it.
* etc/NEWS: Announce it.
---
 doc/lispref/text.texi | 10 ++++++----
 etc/NEWS              |  6 ++++++
 lisp/url/url-http.el  |  5 +++--
 src/decompress.c      | 22 +++++++++++++++++-----
 4 files changed, 32 insertions(+), 11 deletions(-)

diff --git a/doc/lispref/text.texi b/doc/lispref/text.texi
index 6c38d8eed0..e39ba6a192 100644
--- a/doc/lispref/text.texi
+++ b/doc/lispref/text.texi
@@ -4462,14 +4462,16 @@ Decompression
 available.
 @end defun
 
-@defun zlib-decompress-region start end
+@defun zlib-decompress-region start end &optional allow-partial
 This function decompresses the region between @var{start} and
 @var{end}, using built-in zlib decompression.  The region should
 contain data that were compressed with gzip or zlib.  On success, the
 function replaces the contents of the region with the decompressed
-data.  On failure, the function leaves the region unchanged and
-returns @code{nil}.  This function can be called only in unibyte
-buffers.
+data.  If @var{allow-partial} is @code{nil}, on failure, the function
+leaves the region unchanged and returns @code{nil}.  Otherwise, it
+returns the number of bytes that were not decompressed and replaces
+the region text by whatever data was successfully decompressed.  This
+function can be called only in unibyte buffers.
 @end defun
 
 
diff --git a/etc/NEWS b/etc/NEWS
index 3f86195695..395169253d 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -1159,6 +1159,12 @@ to mean that it is not known whether DST is in effect.
 'json-insert', 'json-parse-string', and 'json-parse-buffer'.  These
 are implemented in C using the Jansson library.
 
++++
+** 'zlib-decompress-region' can partially decompress corrupted data.
+If the new optional ALLOW-PARTIAL argument is passed, then the data
+that was decompressed successfully before failing will be inserted
+into the buffer.
+
 ** Mailcap
 
 ---
diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
index 6b5749e1bc..94ac660fcf 100644
--- a/lisp/url/url-http.el
+++ b/lisp/url/url-http.el
@@ -939,7 +939,8 @@ url-http-parse-headers
     (goto-char (point-min))
     success))
 
-(declare-function zlib-decompress-region "decompress.c" (start end))
+(declare-function zlib-decompress-region "decompress.c"
+                  (start end &optional allow-partial))
 
 (defun url-handle-content-transfer-encoding ()
   (let ((encoding (mail-fetch-field "content-encoding")))
@@ -951,7 +952,7 @@ url-handle-content-transfer-encoding
 	(widen)
 	(goto-char (point-min))
 	(when (search-forward "\n\n")
-	  (zlib-decompress-region (point) (point-max)))))))
+	  (zlib-decompress-region (point) (point-max) t))))))
 
 ;; Miscellaneous
 (defun url-http-activate-callback ()
diff --git a/src/decompress.c b/src/decompress.c
index 2836338216..3872014739 100644
--- a/src/decompress.c
+++ b/src/decompress.c
@@ -120,12 +120,18 @@ DEFUN ("zlib-available-p", Fzlib_available_p, Szlib_available_p, 0, 0, 0,
 
 DEFUN ("zlib-decompress-region", Fzlib_decompress_region,
        Szlib_decompress_region,
-       2, 2, 0,
+       2, 3, 0,
        doc: /* Decompress a gzip- or zlib-compressed region.
 Replace the text in the region by the decompressed data.
-On failure, return nil and leave the data in place.
+
+If optional parameter ALLOW-PARTIAL is nil or omitted, on failure,
+return nil and leave the data in place.  Otherwise, return the number
+of bytes that were not decompressed and replace the region text by
+whatever data was successfully decompressed.  If decompression is
+completely successful return t.
+
 This function can be called only in unibyte buffers.  */)
-  (Lisp_Object start, Lisp_Object end)
+  (Lisp_Object start, Lisp_Object end, Lisp_Object allow_partial)
 {
   ptrdiff_t istart, iend, pos_byte;
   z_stream stream;
@@ -206,8 +212,14 @@ DEFUN ("zlib-decompress-region", Fzlib_decompress_region,
     }
   while (inflate_status == Z_OK);
 
+  Lisp_Object ret = Qt;
   if (inflate_status != Z_STREAM_END)
-    return unbind_to (count, Qnil);
+    {
+      if (!NILP (allow_partial))
+        ret = make_int (iend - pos_byte);
+      else
+        return unbind_to (count, Qnil);
+    }
 
   unwind_data.start = 0;
 
@@ -218,7 +230,7 @@ DEFUN ("zlib-decompress-region", Fzlib_decompress_region,
   signal_after_change (istart, iend - istart, unwind_data.nbytes);
   update_compositions (istart, istart, CHECK_HEAD);
 
-  return unbind_to (count, Qt);
+  return unbind_to (count, ret);
 }
 
 
-- 
2.11.0


--=-=-=--




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33133; Package emacs. Full text available.

Message received at 33133 <at> debbugs.gnu.org:


Received: (at 33133) by debbugs.gnu.org; 24 Oct 2018 01:16:28 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Oct 23 21:16:28 2018
Received: from localhost ([127.0.0.1]:38979 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gF7me-0006Yf-3U
	for submit <at> debbugs.gnu.org; Tue, 23 Oct 2018 21:16:28 -0400
Received: from bonobo.maple.relay.mailchannels.net ([23.83.214.22]:4624)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <yamaoka@HIDDEN>) id 1gF7mb-0006YU-Dv
 for 33133 <at> debbugs.gnu.org; Tue, 23 Oct 2018 21:16:26 -0400
X-Sender-Id: tih5qno0ow|x-authuser|yamaoka@HIDDEN
Received: from relay.mailchannels.net (localhost [127.0.0.1])
 by relay.mailchannels.net (Postfix) with ESMTP id D7D94502121;
 Wed, 24 Oct 2018 01:16:20 +0000 (UTC)
Received: from hampton.hostforweb.net (unknown [100.96.26.166])
 (Authenticated sender: tih5qno0ow)
 by relay.mailchannels.net (Postfix) with ESMTPA id 53C70501839;
 Wed, 24 Oct 2018 01:16:20 +0000 (UTC)
X-Sender-Id: tih5qno0ow|x-authuser|yamaoka@HIDDEN
Received: from hampton.hostforweb.net (hampton.hostforweb.net
 [172.245.115.217])
 (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384)
 by 0.0.0.0:2500 (trex/5.16.2); Wed, 24 Oct 2018 01:16:20 +0000
X-MC-Relay: Neutral
X-MailChannels-SenderId: tih5qno0ow|x-authuser|yamaoka@HIDDEN
X-MailChannels-Auth-Id: tih5qno0ow
X-Ruddy-Ruddy: 45b2524842cb0e12_1540343780732_1419938232
X-MC-Loop-Signature: 1540343780732:102784458
X-MC-Ingress-Time: 1540343780731
Received: from s70.gtokyofl21.vectant.ne.jp ([202.215.75.70]:60000
 helo=localhost) by hampton.hostforweb.net with esmtpsa
 (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.89_1)
 (envelope-from <yamaoka@HIDDEN>)
 id 1gF7mT-009umR-50; Tue, 23 Oct 2018 20:16:18 -0500
Date: Wed, 24 Oct 2018 10:16:16 +0900
Message-ID: <b4mpnw0qh3z.fsf@HIDDEN>
From: Katsumi Yamaoka <yamaoka@HIDDEN>
To: Noam Postavsky <npostavs@HIDDEN>
Subject: Re: bug#33133: 26.1.50; zlib-decompress-region too rigid
References: <b4my3aow9bt.fsf@HIDDEN> <87a7n4mbos.fsf@HIDDEN>
Organization: Emacsen advocacy group
X-Face: #kKnN,xUnmKia.'[pp`;
 Omh}odZK)?7wQSl"4o04=EixTF+V[""w~iNbM9ZL+.b*_CxUmFk
 B#Fu[*?MZZH@IkN:!"\w%I_zt>[$nm7nQosZ<3eu; B:$Q_:p!',P.c0-_Cy[dz4oIpw0ESA^D*1Lw=
 L&i*6&(
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (x86_64-unknown-cygwin)
Cancel-Lock: sha1:Qo2Rzd+w62/tHdHM1fjtafiG3KE=
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-OutGoing-Spam-Status: No, score=-0.2
X-AuthUser: yamaoka@HIDDEN
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 33133
Cc: Kevin Ryde <user42_kevin@HIDDEN>, 33133 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

On Tue, 23 Oct 2018 20:26:59 -0400, Noam Postavsky wrote:
> We could have zlib-decompress-region ignore unexpected eof as well,
> e.g., the below (though it should obviously be enhanced to depend on a
> new parameter):

> --- i/src/decompress.c
> +++ w/src/decompress.c
> @@ -206,7 +206,7 @@ DEFUN ("zlib-decompress-region", Fzlib_decompress_region,

>    while (inflate_status == Z_OK);

> -  if (inflate_status != Z_STREAM_END)
> +  if (inflate_status != Z_STREAM_END && inflate_status != Z_BUF_ERROR)
>      return unbind_to (count, Qnil);

>    unwind_data.start = 0;

I confirmed that it makes it work for the corrupted web site in
question.  Thank you!




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33133; Package emacs. Full text available.

Message received at 33133 <at> debbugs.gnu.org:


Received: (at 33133) by debbugs.gnu.org; 24 Oct 2018 00:27:08 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Oct 23 20:27:08 2018
Received: from localhost ([127.0.0.1]:38945 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gF70u-0005PH-H7
	for submit <at> debbugs.gnu.org; Tue, 23 Oct 2018 20:27:08 -0400
Received: from mail-it1-f176.google.com ([209.85.166.176]:34023)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <npostavs@HIDDEN>) id 1gF70t-0005Os-2T
 for 33133 <at> debbugs.gnu.org; Tue, 23 Oct 2018 20:27:07 -0400
Received: by mail-it1-f176.google.com with SMTP id e81-v6so2882668itc.1
 for <33133 <at> debbugs.gnu.org>; Tue, 23 Oct 2018 17:27:07 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=from:to:cc:subject:references:date:in-reply-to:message-id
 :user-agent:mime-version;
 bh=EEzXwbSIFbD+aSL5HevF0ZWUzEItGnHYg8rIAkpruyE=;
 b=POhnoupInDzkTCkhpHkSFmoFV8CGU6v8VcgZB4nXYZypUkJZ2qzBInhysqX7e+flbb
 BRCO4qAKXReT+fhmwQsW3UnBNItMXucranB4N6adr4jm9DDiv+vhEDDI7VdQ7UNu91T6
 /kxxZVpjV7J88V/go5ORgDttcmSgtiRk60BAgI1ZzhSbI26CRWXHAu14RBrM0cW3eoXo
 5vaSrrOZE2PRrw5lA4VhpI7g1r9doT9FCOv/80jPZQkizBT1OJuKYahNhMcn7j0je9DX
 2i4YjmE7RHr8OYEp4SKQXhZfxroXLVxHZK4KSz3Ah25TFrQ8h7QUsbB3REdJV3viX9HL
 6Vjg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to
 :message-id:user-agent:mime-version;
 bh=EEzXwbSIFbD+aSL5HevF0ZWUzEItGnHYg8rIAkpruyE=;
 b=pcBMKU55S5kWDKeUWj/Fq47ZAkX5RPZTNH3T6Udgdh6Bw0P+zFA+tpXPUrCag35MKw
 O+neVNYGhg+2Iv8Ac74s8zO6XTcQf1agQsMqcRdV/dBV3XjiwxGbxg/5fcLm1HuRe3/e
 SpFBW1536qSuF9mfqI9lxlkZ1d/vsyhLn0dsXb9SaUk95+AxQy3Gn55OSrhuyTE6L7Us
 zXQrfCIoC0+V+lewuDHLrC7zoJvtq1Fa36C3r+WXmJbYty9aeUsiSyDxFjPGLXHBWWtA
 9WoqEkdtUynLJHofOoVMDpDgojNOYEtnymDRgOgQWROSm1htx1qstoNIGCsry3gqMFZm
 YTrA==
X-Gm-Message-State: AGRZ1gLRnMxADytR1FFICb4SDtXqWiGUZ8JLQXbyxQdvqbkrGZhJVQLm
 7snTHl8EdajIDXNo3baTKZI=
X-Google-Smtp-Source: AJdET5dQ45avk9OaawuuS0GgQaDfkOpfptgsYcPezBfE9/v+b54DrM84MkGUCVZawr35HiYLv9pK3w==
X-Received: by 2002:a24:a10b:: with SMTP id
 y11-v6mr260903ite.148.1540340821571; 
 Tue, 23 Oct 2018 17:27:01 -0700 (PDT)
Received: from zebian (cbl-45-2-119-34.yyz.frontiernetworks.ca. [45.2.119.34])
 by smtp.googlemail.com with ESMTPSA id
 197-v6sm1102757iou.87.2018.10.23.17.27.00
 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256);
 Tue, 23 Oct 2018 17:27:00 -0700 (PDT)
From: Noam Postavsky <npostavs@HIDDEN>
To: Katsumi Yamaoka <yamaoka@HIDDEN>
Subject: Re: bug#33133: 26.1.50; zlib-decompress-region too rigid
References: <b4my3aow9bt.fsf@HIDDEN>
Date: Tue, 23 Oct 2018 20:26:59 -0400
In-Reply-To: <b4my3aow9bt.fsf@HIDDEN> (Katsumi Yamaoka's message of "Wed, 24
 Oct 2018 08:07:50 +0900")
Message-ID: <87a7n4mbos.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 33133
Cc: Kevin Ryde <user42_kevin@HIDDEN>, 33133 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Katsumi Yamaoka <yamaoka@HIDDEN> writes:

> Whereas `gzip -d' does, zlib-decompress-region doesn't decompress
> corrupted data of a certain kind.  For instance, visiting
>
> https://www.gutenberg.org/no-such-page-exists
>
> using eww shows raw gzipped data.  The data extracted is attached.
> As for `gzip -d', it says "unexpected end of file" in stderr.

> If there is no prospect to improve zlib-decompress-region, how
> about this workaround?

We could have zlib-decompress-region ignore unexpected eof as well,
e.g., the below (though it should obviously be enhanced to depend on a
new parameter):

--- i/src/decompress.c
+++ w/src/decompress.c
@@ -206,7 +206,7 @@ DEFUN ("zlib-decompress-region", Fzlib_decompress_region,
     }
   while (inflate_status == Z_OK);
 
-  if (inflate_status != Z_STREAM_END)
+  if (inflate_status != Z_STREAM_END && inflate_status != Z_BUF_ERROR)
     return unbind_to (count, Qnil);
 
   unwind_data.start = 0;






Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33133; Package emacs. Full text available.

Message received at submit <at> debbugs.gnu.org:


Received: (at submit) by debbugs.gnu.org; 23 Oct 2018 23:09:10 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Oct 23 19:09:09 2018
Received: from localhost ([127.0.0.1]:38900 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gF5nR-0007yN-MY
	for submit <at> debbugs.gnu.org; Tue, 23 Oct 2018 19:09:09 -0400
Received: from eggs.gnu.org ([208.118.235.92]:52424)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <yamaoka@HIDDEN>) id 1gF5nO-0007xv-U6
 for submit <at> debbugs.gnu.org; Tue, 23 Oct 2018 19:09:08 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <yamaoka@HIDDEN>) id 1gF5nE-0006ja-HT
 for submit <at> debbugs.gnu.org; Tue, 23 Oct 2018 19:09:01 -0400
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50, T_TVD_MIME_NO_HEADERS
 autolearn=disabled version=3.3.2
Received: from lists.gnu.org ([2001:4830:134:3::11]:58596)
 by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32)
 (Exim 4.71) (envelope-from <yamaoka@HIDDEN>) id 1gF5nE-0006is-1w
 for submit <at> debbugs.gnu.org; Tue, 23 Oct 2018 19:08:56 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:33664)
 by lists.gnu.org with esmtp (Exim 4.71)
 (envelope-from <yamaoka@HIDDEN>) id 1gF5nC-00023f-IN
 for bug-gnu-emacs@HIDDEN; Tue, 23 Oct 2018 19:08:55 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <yamaoka@HIDDEN>) id 1gF5n3-0006Vz-UR
 for bug-gnu-emacs@HIDDEN; Tue, 23 Oct 2018 19:08:52 -0400
Received: from coral.maple.relay.mailchannels.net ([23.83.214.39]:42673)
 by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)
 (Exim 4.71) (envelope-from <yamaoka@HIDDEN>) id 1gF5mx-0005e2-FS
 for bug-gnu-emacs@HIDDEN; Tue, 23 Oct 2018 19:08:41 -0400
X-Sender-Id: tih5qno0ow|x-authuser|yamaoka@HIDDEN
Received: from relay.mailchannels.net (localhost [127.0.0.1])
 by relay.mailchannels.net (Postfix) with ESMTP id 24D99282822;
 Tue, 23 Oct 2018 23:07:56 +0000 (UTC)
Received: from hampton.hostforweb.net (unknown [100.96.19.78])
 (Authenticated sender: tih5qno0ow)
 by relay.mailchannels.net (Postfix) with ESMTPA id 90CB9282C63;
 Tue, 23 Oct 2018 23:07:55 +0000 (UTC)
X-Sender-Id: tih5qno0ow|x-authuser|yamaoka@HIDDEN
Received: from hampton.hostforweb.net (hampton.hostforweb.net
 [172.245.115.217])
 (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384)
 by 0.0.0.0:2500 (trex/5.16.2); Tue, 23 Oct 2018 23:07:56 +0000
X-MC-Relay: Neutral
X-MailChannels-SenderId: tih5qno0ow|x-authuser|yamaoka@HIDDEN
X-MailChannels-Auth-Id: tih5qno0ow
X-Vacuous-Abaft: 111069091fa92dfb_1540336075988_2192594722
X-MC-Loop-Signature: 1540336075987:3337611917
X-MC-Ingress-Time: 1540336075987
Received: from s70.gtokyofl21.vectant.ne.jp ([202.215.75.70]:60000
 helo=localhost) by hampton.hostforweb.net with esmtpsa
 (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.89_1)
 (envelope-from <yamaoka@HIDDEN>)
 id 1gF5mC-009Pi0-41; Tue, 23 Oct 2018 18:07:53 -0500
Date: Wed, 24 Oct 2018 08:07:50 +0900
Message-ID: <b4my3aow9bt.fsf@HIDDEN>
From: Katsumi Yamaoka <yamaoka@HIDDEN>
To: bug-gnu-emacs@HIDDEN
Subject: 26.1.50; zlib-decompress-region too rigid
Organization: Emacsen advocacy group
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1.50 (x86_64-unknown-cygwin)
Cancel-Lock: sha1:cQKpMnMykhq6rg+1wFwf2GKc/GU=
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="=-=-="
X-OutGoing-Spam-Status: No, score=-0.2
X-AuthUser: yamaoka@HIDDEN
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x
X-Received-From: 2001:4830:134:3::11
X-Spam-Score: -4.1 (----)
X-Debbugs-Envelope-To: submit
Cc: Kevin Ryde <user42_kevin@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -5.1 (-----)

--=-=-=

Hi,

Whereas `gzip -d' does, zlib-decompress-region doesn't decompress
corrupted data of a certain kind.  For instance, visiting

https://www.gutenberg.org/no-such-page-exists

using eww shows raw gzipped data.  The data extracted is attached.
As for `gzip -d', it says "unexpected end of file" in stderr.
Here is a recipe to reproduce zlib-decompress-region not working:

(let ((buffer (get-buffer-create "*testing*"))
      (coding-system-for-read 'binary)
      (cw (selected-window))
      jka-compr-compression-info-list format-alist)
  (switch-to-buffer-other-window buffer)
  (erase-buffer)
  (set-buffer-multibyte nil)
  (insert-file-contents "/TEMP/corrupted-data.gz")
  (sit-for 1)
  (prog1
      (zlib-decompress-region (point-min) (point-max))
    (select-window cw)))

If there is no prospect to improve zlib-decompress-region, how
about this workaround?

--- url-http.el~	2018-09-12 07:48:16.110765500 +0000
+++ url-http.el	2018-10-23 23:04:48.060829900 +0000
@@ -951,7 +951,12 @@
 	(widen)
 	(goto-char (point-min))
 	(when (search-forward "\n\n")
-	  (zlib-decompress-region (point) (point-max)))))))
+	  (or (zlib-decompress-region (point) (point-max))
+	      (let ((coding-system-for-write 'binary)
+		    (coding-system-for-read 'binary)
+		    (default-process-coding-system (cons 'binary 'binary)))
+		(zerop (call-process-region (point) (point-max) "gzip"
+					    t '(t nil) nil "-d")))))))))
 
 ;; Miscellaneous
 (defun url-http-activate-callback ()


In GNU Emacs 26.1.50 (build 1, x86_64-unknown-cygwin, GTK+ Version 3.22.28)
 of 2018-10-22 built on localhost
Windowing system distributor 'The Cygwin/X Project', version 11.0.12001000

--=-=-=
Content-Type: application/octet-stream
Content-Disposition: attachment; filename=corrupted-data.gz
Content-Transfer-Encoding: base64

H4sIAAAAAAAAA8xYW2/bNhR+rn/FCYelLVpbtpMNaWNraJO0C9CmQZqh6FNAS5TEmhI1krLrDfvv
OyRlWbKdZEBfFqANRZ7Lx3NnJgfnn85uv15fwO+3Hz/A9R9vP1yeAekHwZejsyA4vz33B8eD4SgI
Lq4IkMyY8nUQLJfLwfJoIFUa3N4EmcnFcaCN4pEZxCYmYa83sZsgaJFOCStwB2CSMRrbBS5zZihY
YX32Z8UXU3ImC8MK079dlYxA5L+mxLDvxsk/hSijSjMzvfz8qX9y8sur/oi0hRU0Z1MyZ6ulVLEm
sP3TiGQzKef6JbjfL6H+8h+JYuwlyELwAn/TKuZyj5KY6Ujx0nBZkHuUHB0Nh8MXTp5XpGupe8RF
gmrNEx7RbYmNuLKaCR6Rx4z32awE22vCSGvrFcfuwYNBsimhZSlqzcE3uqD+sNYEwBN4ZmQJB1PQ
TCTP4e/eGhxuD4T0rAPFSkEjBs+etiIkrRDBjKnUBsrT56cNKxVMGaS9VvIbiwy8XxMC10Dh3c3F
BapTC44Sl9xkcPUJcpYjhc54CcpeW7F4AJcJrGQFJeUxaJmjpeMVMKEZooOUGciYQkfmdI47Gcsh
5QvmWPCfglwWbAUzGs0PNvD+8VYKvCVCqK2GzpujZjEl2lpZZ4wZUhuxMTFkiiVTYtdBmfaHw/HA
WX5bQiaViSoDPPIOr9kSurA7A/xv7WvDjWDhNU0ZXEkD72RVxNCHHctNAk9p8yzwiWaX1iC1pJgv
wIXalOSsqGZUNV52ZzyeEiFTSdZU7iNsnDahYOiMFzH7PiUj0uzbOG0xtPfray35nAcfKS/u7D06
FA70lLyX1mGWBBxJOOF52qZ7QAn+aBWhmpJHzuqWwpm+LNJtyiWPTYbwx6/QV4ynGSbIyXCbigrc
RhABbSwUoInqBNq1ZdtKW0f9VMmqhM2yL8u6HrYMWzNkI9Ky8fGQAI0ipjVWtSkZk661MJ53w2Bj
QwySNw3zmNzji7Ch95ed6JIWbfyalSScHqbmFDMCz0LoAG9FxKiDNtmD9krCmdQGpIJ3WBhjmf+G
qbsFNWmgJp6GhHv4HkLb7z8GddyBKv6TYT9gMSqwsOwAFg1g4UlIWNP+GMijDsj5HpDvKx4z21Q0
JGgbW194ke4inDcIM7k0sm8JLUhPfoPFIv9BrMcdrOpxrJW2qikoOcOqtoNYNYgdQUDCG0fYxroG
4jPz4fz73+bbvkt0P9alOZYFNWy7OD/p+OHVCYHeE6/Z9mFdN+KSrkoqBpHMg+8RRuk8mFkHINip
F/vz8bDTrA9pXp5yw/I7N6KcWyJs8y+MfFGb4EVjAl+s4d5+8GRTdn8db6ru0cgftWu3h5nyZF+P
cCAYFHJpvbHrCZOhp9MMjbu6psILd4W8ZrztnoddB5ThrsQCa432k0JcW+AA2v2w7eOG7XUt6K7Z
ubuygu6+oqC7tSkR3zq6cEzJpWLAi0QCnUmcC9bq9N6rIm/40XJcIoe9xiQoHw6eDMsni+/p7Jt8
2O6tm0iqA4nPOA6i0oZIm66+yd5xzquG2Qpq5p3mfo/yVmDUnH0vC3v7sevt7ZhbN/bRsBVi4w7J
vV29tdwZleoxWndnpXWBkfaxM3GcGEEuD+qjREiK82H7EugsD3uN7xjTtcZtlx7f2pjC0sCsEjM/
JYa9cxxy7eg6dymQUYPl2ihZpCHuToJ6DRFW8IyJEmsslErGVbR+gvTQCbgjk77COmoL8LfKttWi
7moUYrr6refK3etNfLf8X6ZxOSiYLcjn3D74ZpV17rUVamXigN7E4yM1bV2CHVd7gMpGW+MuzrOj
R+XYQde9O8vws1RqhS86zCNbs93VsOVYEzj7IWzEjInli70N0qrApw8XdCbYoOfQW0Ff0CSYZami
cUWFWEFVxpiWtnEJAeytf9YJxG/dUcBPo+EAn30voUdxSMc679QrabXgI0QuLKt7hljlWF6WkHBh
N/UKAzsfNJp7WCucIzdFxrsQDa8ZVVHmuih1l7IWH6A5ehZvKTGP+F/MnVv9vMDoWbCCsyJiSNa7
Rahz34UXXHN3n53MxccvXiHD1+EK6mA52K0ye/yQSGm6/uwkU7lSNq67/bhsfwF8YTNAXAwacji0
y1MYD4dH/fFwNNw7n6FeqlbwBq1jX3kudlwVhcM8pjo7hTfothsrUOMkYR+Y+IZsAwnKx1tys5wE
/nE1cX+fCHv/AgAA//8=
--=-=-=--




Acknowledgement sent to Katsumi Yamaoka <yamaoka@HIDDEN>:
New bug report received and forwarded. Copy sent to bug-gnu-emacs@HIDDEN. Full text available.
Report forwarded to bug-gnu-emacs@HIDDEN:
bug#33133; Package emacs. Full text available.
Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.
Last modified: Sat, 27 Oct 2018 22:00:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.