GNU bug report logs - #35841
[PATCH] IBM Z DFLTCC: fix three data corruption issues

Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.

Package: gzip; Reported by: Ilya Leoshkevich <iii@HIDDEN>; Keywords: patch; dated Tue, 21 May 2019 11:26:02 UTC; Maintainer for gzip is bug-gzip@HIDDEN.

Message received at submit <at> debbugs.gnu.org:


Received: (at submit) by debbugs.gnu.org; 21 May 2019 11:25:53 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue May 21 07:25:53 2019
Received: from localhost ([127.0.0.1]:38760 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1hT2u1-000264-2m
	for submit <at> debbugs.gnu.org; Tue, 21 May 2019 07:25:53 -0400
Received: from eggs.gnu.org ([209.51.188.92]:48593)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <iii@HIDDEN>) id 1hT2tz-00025s-3C
 for submit <at> debbugs.gnu.org; Tue, 21 May 2019 07:25:51 -0400
Received: from lists.gnu.org ([209.51.188.17]:35292)
 by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32)
 (Exim 4.71) (envelope-from <iii@HIDDEN>) id 1hT2tr-0003tz-46
 for submit <at> debbugs.gnu.org; Tue, 21 May 2019 07:25:45 -0400
Received: from eggs.gnu.org ([209.51.188.92]:34741)
 by lists.gnu.org with esmtp (Exim 4.71)
 (envelope-from <iii@HIDDEN>) id 1hT2tp-0001Y8-Jg
 for bug-gzip@HIDDEN; Tue, 21 May 2019 07:25:42 -0400
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=0.1 required=5.0 tests=BAYES_50,RCVD_IN_DNSWL_LOW,
 URIBL_BLOCKED autolearn=disabled version=3.3.2
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <iii@HIDDEN>) id 1hT2tl-0003px-Sq
 for bug-gzip@HIDDEN; Tue, 21 May 2019 07:25:41 -0400
Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:43298
 helo=mx0a-001b2d01.pphosted.com)
 by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32)
 (Exim 4.71) (envelope-from <iii@HIDDEN>) id 1hT2tl-0003n6-Nj
 for bug-gzip@HIDDEN; Tue, 21 May 2019 07:25:37 -0400
Received: from pps.filterd (m0098416.ppops.net [127.0.0.1])
 by mx0b-001b2d01.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id
 x4LBIEqZ029883
 for <bug-gzip@HIDDEN>; Tue, 21 May 2019 07:25:34 -0400
Received: from e06smtp07.uk.ibm.com (e06smtp07.uk.ibm.com [195.75.94.103])
 by mx0b-001b2d01.pphosted.com with ESMTP id 2smg25s569-1
 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT)
 for <bug-gzip@HIDDEN>; Tue, 21 May 2019 07:25:33 -0400
Received: from localhost
 by e06smtp07.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only!
 Violators will be prosecuted
 for <bug-gzip@HIDDEN> from <iii@HIDDEN>;
 Tue, 21 May 2019 12:25:32 +0100
Received: from b06cxnps4074.portsmouth.uk.ibm.com (9.149.109.196)
 by e06smtp07.uk.ibm.com (192.168.101.137) with IBM ESMTP SMTP Gateway:
 Authorized Use Only! Violators will be prosecuted; 
 (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256)
 Tue, 21 May 2019 12:25:30 +0100
Received: from d06av22.portsmouth.uk.ibm.com (d06av22.portsmouth.uk.ibm.com
 [9.149.105.58])
 by b06cxnps4074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id
 x4LBPSlE45154444
 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK);
 Tue, 21 May 2019 11:25:28 GMT
Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1])
 by IMSVA (Postfix) with ESMTP id 886C94C044;
 Tue, 21 May 2019 11:25:28 +0000 (GMT)
Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1])
 by IMSVA (Postfix) with ESMTP id 639314C04A;
 Tue, 21 May 2019 11:25:28 +0000 (GMT)
Received: from white.boeblingen.de.ibm.com (unknown [9.152.96.43])
 by d06av22.portsmouth.uk.ibm.com (Postfix) with ESMTP;
 Tue, 21 May 2019 11:25:28 +0000 (GMT)
From: Ilya Leoshkevich <iii@HIDDEN>
To: bug-gzip@HIDDEN
Subject: [PATCH] IBM Z DFLTCC: fix three data corruption issues
Date: Tue, 21 May 2019 13:25:23 +0200
X-Mailer: git-send-email 2.21.0
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
X-TM-AS-GCONF: 00
x-cbid: 19052111-0028-0000-0000-0000036FF130
X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused
x-cbparentid: 19052111-0029-0000-0000-0000242F9AEC
Message-Id: <20190521112523.68548-1-iii@HIDDEN>
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, ,
 definitions=2019-05-21_02:, , signatures=0
X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0
 priorityscore=1501
 malwarescore=0 suspectscore=1 phishscore=0 bulkscore=0 spamscore=0
 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0
 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx
 scancount=1 engine=8.0.1-1810050000 definitions=main-1905210071
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic]
X-Received-From: 148.163.158.5
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x
X-Spam-Score: -1.3 (-)
X-Debbugs-Envelope-To: submit
Cc: Ilya Leoshkevich <iii@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -2.3 (--)

Hello,

This patch fixes three bugs found during internal testing. It also adds
feature detection for sys/sdt.h, adds diagnostic output when hardware
decompression fails, and fixes formatting in a few places.

Best regards,
Ilya

---

* configure.ac (AC_CHECK_HEADERS_ONCE): Add feature detection for
sys/sdt.h probes.
* dfltcc.c (dfltcc_cc): Minor formatting improvements.
(HB_BITS): Remove.
(HB_SIZE): Likewise.
(dfltcc): Use sys/sdt.h feature detection.
(bi_load): New function.
(bi_close_block): Use bi_load.
(close_stream): Fix overwriting the End-of-block Symbol.
(dfltcc_deflate): Fix losing partial byte on flush.
Fix setting Block-Continuation Flag when DFLTCC-CMPR outputs 0 bits and
requests a retry.
Minor formatting improvements.
(dfltcc_inflate): Retry immediately if requested.
Print the hardware error code and flush the output buffer on error.
Minor formatting improvements.
* tests/hufts: Ignore the hardware error code.
---
 configure.ac |  2 +-
 dfltcc.c     | 63 +++++++++++++++++++++++++++++++++++-----------------
 tests/hufts  |  2 ++
 3 files changed, 46 insertions(+), 21 deletions(-)

diff --git a/configure.ac b/configure.ac
index 76ac26f..b4aea34 100644
--- a/configure.ac
+++ b/configure.ac
@@ -263,7 +263,7 @@ AC_SUBST([ASFLAGS_config])
 AC_ISC_POSIX
 AC_C_CONST
 AC_HEADER_STDC
-AC_CHECK_HEADERS_ONCE(fcntl.h limits.h memory.h time.h)
+AC_CHECK_HEADERS_ONCE(fcntl.h limits.h memory.h time.h sys/sdt.h)
 AC_CHECK_FUNCS_ONCE([chown fchmod fchown lstat siginterrupt])
 AC_HEADER_DIRENT
 AC_TYPE_SIGNAL
diff --git a/dfltcc.c b/dfltcc.c
index ba62968..6ad41a8 100644
--- a/dfltcc.c
+++ b/dfltcc.c
@@ -22,7 +22,7 @@
 #include <stdbool.h>
 #include <stdlib.h>
 
-#ifdef DFLTCC_USDT
+#ifdef HAVE_SYS_SDT_H
 # include <sys/sdt.h>
 #endif
 
@@ -39,11 +39,11 @@
 
 typedef enum
 {
- DFLTCC_CC_OK = 0,
- DFLTCC_CC_OP1_TOO_SHORT = 1,
- DFLTCC_CC_OP2_TOO_SHORT = 2,
- DFLTCC_CC_OP2_CORRUPT = 2,
- DFLTCC_CC_AGAIN = 3,
+  DFLTCC_CC_OK = 0,
+  DFLTCC_CC_OP1_TOO_SHORT = 1,
+  DFLTCC_CC_OP2_TOO_SHORT = 2,
+  DFLTCC_CC_OP2_CORRUPT = 2,
+  DFLTCC_CC_AGAIN = 3,
 } dfltcc_cc;
 
 #define DFLTCC_QAF 0
@@ -51,8 +51,6 @@ typedef enum
 #define DFLTCC_CMPR 2
 #define DFLTCC_XPND 4
 #define HBT_CIRCULAR (1 << 7)
-/* #define HB_BITS 15 */
-/* #define HB_SIZE (1 << HB_BITS) */
 #define DFLTCC_FACILITY 151
 #define DFLTCC_FMT0 0
 #define CVT_CRC32 0
@@ -180,12 +178,12 @@ dfltcc (int fn, void *param,
   int cc;
 
   __asm__ volatile (
-#ifdef DFLTCC_USDT
+#ifdef HAVE_SYS_SDT_H
                     STAP_PROBE_ASM (zlib, dfltcc_entry,
                                     STAP_PROBE_ASM_TEMPLATE (5))
 #endif
                     ".insn rrf,0xb9390000,%[r2],%[r4],%[hist],0\n"
-#ifdef DFLTCC_USDT
+#ifdef HAVE_SYS_SDT_H
                     STAP_PROBE_ASM (zlib, dfltcc_exit,
                                     STAP_PROBE_ASM_TEMPLATE (5))
 #endif
@@ -198,7 +196,7 @@ dfltcc (int fn, void *param,
                     : [r0] "r" (r0)
                       , [r1] "r" (r1)
                       , [hist] "r" (hist)
-#ifdef DFLTCC_USDT
+#ifdef HAVE_SYS_SDT_H
                       , STAP_PROBE_ASM_OPERANDS (5, r2, r3, r4, r5, hist)
 #endif
                     : "cc", "memory");
@@ -264,10 +262,16 @@ init_param (union aligned_dfltcc_param_v0 *ctx)
 }
 
 static void
-bi_close_block (struct dfltcc_param_v0 *param)
+bi_load (struct dfltcc_param_v0 *param)
 {
   bi_valid = param->sbb;
   bi_buf = bi_valid == 0 ? 0 : outbuf[outcnt] & ((1 << bi_valid) - 1);
+}
+
+static void
+bi_close_block (struct dfltcc_param_v0 *param)
+{
+  bi_load (param);
   send_bits (bi_reverse (param->eobs >> (15 - param->eobl), param->eobl),
              param->eobl);
   param->bcf = 0;
@@ -278,6 +282,7 @@ close_block (struct dfltcc_param_v0 *param)
 {
   bi_close_block (param);
   bi_windup ();
+  /* bi_windup has written out a possibly partial byte, fix up the position */
   param->sbb = (param->sbb + param->eobl) % 8;
   if (param->sbb != 0)
     {
@@ -291,6 +296,8 @@ close_stream (struct dfltcc_param_v0 *param)
 {
   if (param->bcf)
     bi_close_block (param);
+  else
+    bi_load (param);
   send_bits (1, 3); /* BFINAL=1, BTYPE=00 */
   bi_windup ();
   put_short (0x0000);
@@ -334,7 +341,16 @@ dfltcc_deflate (int pack_level)
     {
       /* Flush the output data.  */
       if (outcnt > OUTBUFSIZ - 8)
-        flush_outbuf ();
+        {
+          if (param->sbb == 0)
+            flush_outbuf ();
+          else
+            {
+              uch partial = outbuf[outcnt];
+              flush_outbuf ();
+              outbuf[outcnt] = partial;
+            }
+        }
 
       /* Close the block.  */
       if (param->bcf && total_in == block_threshold && !param->cf)
@@ -360,14 +376,16 @@ dfltcc_deflate (int pack_level)
         {
           if (total_in == 0 && block_threshold > 0)
             param->htt = HTT_FIXED;
-          else {
-            param->htt = HTT_DYNAMIC;
-            dfltcc_gdht (param);
-          }
+          else
+            {
+              param->htt = HTT_DYNAMIC;
+              dfltcc_gdht (param);
+            }
         }
 
       /* Compress inbuf into outbuf.  */
-      dfltcc_cmpr_xpnd (param, DFLTCC_CMPR);
+      while (dfltcc_cmpr_xpnd (param, DFLTCC_CMPR) == DFLTCC_CC_AGAIN)
+        ;
 
       /* Unmask the input data.  */
       insize += extra;
@@ -413,7 +431,9 @@ dfltcc_inflate (void)
         }
 
         /* Decompress inbuf into outbuf.  */
-        dfltcc_cc cc = dfltcc_cmpr_xpnd (param, DFLTCC_XPND);
+        dfltcc_cc cc;
+        while ((cc = dfltcc_cmpr_xpnd (param, DFLTCC_XPND)) == DFLTCC_CC_AGAIN)
+          ;
         if (cc == DFLTCC_CC_OK)
           {
             /* The entire deflate stream has been successfully decompressed.  */
@@ -422,6 +442,9 @@ dfltcc_inflate (void)
         if (cc == DFLTCC_CC_OP2_CORRUPT && param->oesc != 0)
           {
             /* The deflate stream is corrupted.  */
+            fprintf (stderr, "Operation-Ending-Supplemental Code 0x%x\n",
+                     param->oesc);
+            flush_outbuf ();
             return 2;
           }
         /* There must be more data to decompress.  */
@@ -430,7 +453,7 @@ dfltcc_inflate (void)
   if (param->sbb != 0)
     {
       /* The deflate stream has ended in the middle of a byte.  Go to
-        the next byte boundary, so that unzip can read CRC and length.  */
+         the next byte boundary, so that unzip can read CRC and length.  */
       inptr++;
     }
 
diff --git a/tests/hufts b/tests/hufts
index cd8368a..7ca22af 100755
--- a/tests/hufts
+++ b/tests/hufts
@@ -28,6 +28,7 @@ returns_ 1 gzip -dc "$abs_srcdir/hufts-segv.gz" > out 2> err || fail=1
 compare /dev/null out || fail=1
 
 sed 's/.*hufts-segv.gz: /...: /' err > k; mv k err || fail=1
+grep -v 'Operation-Ending-Supplemental Code' err > k; mv k err || fail=1
 compare exp err || fail=1
 
 printf '\037\213\010\000\060\060\060\060\060\060\144\000\000\000' > bug33501 \
@@ -35,6 +36,7 @@ printf '\037\213\010\000\060\060\060\060\060\060\144\000\000\000' > bug33501 \
 printf '\ngzip: stdin: invalid compressed data--format violated\n' >exp33501 \
   || framework_failure_
 returns_ 1 gzip -d <bug33501 >out33501 2> err33501 || fail=1
+grep -v 'Operation-Ending-Supplemental Code' err33501 > k; mv k err33501 || fail=1
 compare exp33501 err33501 || fail=1
 
 Exit $fail
-- 
2.21.0





Acknowledgement sent to Ilya Leoshkevich <iii@HIDDEN>:
New bug report received and forwarded. Copy sent to bug-gzip@HIDDEN. Full text available.
Report forwarded to bug-gzip@HIDDEN:
bug#35841; Package gzip. Full text available.
Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.
Last modified: Tue, 21 May 2019 11:30:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.