Received: (at submit) by debbugs.gnu.org; 15 Sep 2013 13:36:49 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Sun Sep 15 09:36:49 2013 Received: from localhost ([127.0.0.1]:38374 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1VLCVM-0001GP-FN for submit <at> debbugs.gnu.org; Sun, 15 Sep 2013 09:36:48 -0400 Received: from eggs.gnu.org ([208.118.235.92]:45086) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from <martin.langhoff@HIDDEN>) id 1VLCVJ-0001G5-5m for submit <at> debbugs.gnu.org; Sun, 15 Sep 2013 09:36:45 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from <martin.langhoff@HIDDEN>) id 1VLCVC-0001A2-LS for submit <at> debbugs.gnu.org; Sun, 15 Sep 2013 09:36:39 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50,FREEMAIL_FROM, T_DKIM_INVALID autolearn=disabled version=3.3.2 Received: from lists.gnu.org ([2001:4830:134:3::11]:50846) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <martin.langhoff@HIDDEN>) id 1VLCVC-00019y-IL for submit <at> debbugs.gnu.org; Sun, 15 Sep 2013 09:36:38 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:46808) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from <martin.langhoff@HIDDEN>) id 1VLCVB-0004jt-IC for bug-gzip@HIDDEN; Sun, 15 Sep 2013 09:36:38 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from <martin.langhoff@HIDDEN>) id 1VLCVA-00019R-G0 for bug-gzip@HIDDEN; Sun, 15 Sep 2013 09:36:37 -0400 Received: from mail-qa0-x22a.google.com ([2607:f8b0:400d:c00::22a]:45537) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <martin.langhoff@HIDDEN>) id 1VLCVA-00019N-B2 for bug-gzip@HIDDEN; Sun, 15 Sep 2013 09:36:36 -0400 Received: by mail-qa0-f42.google.com with SMTP id cm18so517826qab.8 for <bug-gzip@HIDDEN>; Sun, 15 Sep 2013 06:36:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=zg0OVw5/I7Prb7xie2Vae/iXbPPaXYZ9pGp3Yfpz9qo=; b=HUJ92vFTT4IIAHJzCbxr/bD7EiK18s8ZD/JBV/qhojv81qQNHBFT2/hGNJeoGogPV0 Ez+iYNpCajPcCWLZa8b8sQ35PjC8dK0gYZSIJw7nEG2O40Y1Eg6Zjar1vK6gBt+UB4Bl 9h2qcNGYcbJNRKWu9Xv4qthc+U7tzDb0C+hLsCx0QDv1Jrev2fuy5eEDzR/cf5/cAucO gqYrtSR88exOaRFnTChK4GVoMcGhdzciVSXEGWCncaeu6rtyeXUs6OBzKvUVrUsspOiW XKr2Yk2P/irHixsbaE0mFB5qLqC9mbCIbY6ZuPExZfAXUQnHUOSYbkSOWUabVS7HzLnu E13Q== X-Received: by 10.49.30.66 with SMTP id q2mr40121330qeh.38.1379252195913; Sun, 15 Sep 2013 06:36:35 -0700 (PDT) MIME-Version: 1.0 Received: by 10.49.61.35 with HTTP; Sun, 15 Sep 2013 06:36:15 -0700 (PDT) In-Reply-To: <523533E2.2090603@HIDDEN> References: <CACPiFCJF1+_+8mZ4YM3Wrp2OuuffCwJb1gj6Sox_6Qc=FhNvLQ@HIDDEN> <523533E2.2090603@HIDDEN> From: Martin Langhoff <martin.langhoff@HIDDEN> Date: Sun, 15 Sep 2013 09:36:15 -0400 Message-ID: <CACPiFC+Cq3sD+Qe=31WSW-sXA+ZmnEAGE=RQZQ8fbsKZ9kdYcA@HIDDEN> Subject: Re: Bug in gzip.c buffering, and two feature requests To: Paul Eggert <eggert@HIDDEN> Content-Type: text/plain; charset=ISO-8859-1 X-detected-operating-system: by eggs.gnu.org: Error: Malformed IPv6 address (bad octet value). X-detected-operating-system: by eggs.gnu.org: Error: Malformed IPv6 address (bad octet value). X-Received-From: 2001:4830:134:3::11 X-Spam-Score: -4.0 (----) X-Debbugs-Envelope-To: submit Cc: Jim Meyering <jim@HIDDEN>, bug-gzip@HIDDEN X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <http://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <http://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <http://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -4.0 (----) On Sun, Sep 15, 2013 at 12:13 AM, Paul Eggert <eggert@HIDDEN> wrote: > Your gzip feature requests sound reasonable, except for this one: Thanks for taking the time to reply! >> Bug: gzip.c discards its buffers when it gets SIGTERM, with no >> effort made to flush them to disk. > > That's the normal behavior of utilites when they get SIGTERM; they > exit right away with minimal fuss, and they typically do not attempt > to flush output buffers. I am no POSIX lawyer. Googling, I find two diverging interpretations of SIGTERM -- one is close to SIGABRT, the other one is for a more elegant exit. For example, Wikipedia has: "SIGTERM - The SIGTERM signal is sent to a process to request its termination. Unlike the SIGKILL signal, it can be caught and interpreted or ignored by the process. This allows the process to perform nice termination releasing resources and saving state if appropriate. It should be noted that SIGINT is nearly identical to SIGTERM." Did a bit of reading around POSIX 2008 at http://pubs.opengroup.org/onlinepubs/9699919799/ -- if I read http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/signal.h.html I do find that SIGTERM should behave as per the "T" key, "Abnormal termination of process". But it also says SIGALRM, SIGUSR1 and SIGUSR2 are "T", so this is not the information we are looking for :-) At the end of the day, we didn't get SIGABRT. We got SIGTERM. IMHO, flushing of buffers is reasonable in this context. >> Logging via gzip adds buffering, when apache is stopped (or >> restarted, usually daily to rotate logs), apache sends SIGTERM >> before it closes the fh; > > That seems wrong. Apache should close the file handle, and let gzip > finish up cleanly. Maybe Apache should kill gzip if gzip doesn't exit I am double-checking the apache sources to confirm my initial reading, but TBH I am not yet convinced that SIGTERM is wrong in this context. (closing the fh would be nice, though). cheers, m -- martin.langhoff@HIDDEN - ask interesting questions - don't get distracted with shiny stuff - working code first ~ http://docs.moodle.org/en/User:Martin_Langhoff
bug-gzip@HIDDEN:bug#15380; Package gzip.
Full text available.Received: (at submit) by debbugs.gnu.org; 15 Sep 2013 04:14:13 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Sun Sep 15 00:14:13 2013 Received: from localhost ([127.0.0.1]:37677 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1VL3iu-00032F-9V for submit <at> debbugs.gnu.org; Sun, 15 Sep 2013 00:14:12 -0400 Received: from eggs.gnu.org ([208.118.235.92]:37502) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from <eggert@HIDDEN>) id 1VL3is-000322-6W for submit <at> debbugs.gnu.org; Sun, 15 Sep 2013 00:14:10 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from <eggert@HIDDEN>) id 1VL3id-0007GJ-9t for submit <at> debbugs.gnu.org; Sun, 15 Sep 2013 00:14:04 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50 autolearn=disabled version=3.3.2 Received: from lists.gnu.org ([2001:4830:134:3::11]:51023) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eggert@HIDDEN>) id 1VL3id-0007GF-7E for submit <at> debbugs.gnu.org; Sun, 15 Sep 2013 00:13:55 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:39224) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from <eggert@HIDDEN>) id 1VL3iV-0000mz-U4 for bug-gzip@HIDDEN; Sun, 15 Sep 2013 00:13:55 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from <eggert@HIDDEN>) id 1VL3iO-0007Fq-K9 for bug-gzip@HIDDEN; Sun, 15 Sep 2013 00:13:47 -0400 Received: from smtp.cs.ucla.edu ([131.179.128.62]:49251) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eggert@HIDDEN>) id 1VL3iO-0007FP-EQ for bug-gzip@HIDDEN; Sun, 15 Sep 2013 00:13:40 -0400 Received: from localhost (localhost.localdomain [127.0.0.1]) by smtp.cs.ucla.edu (Postfix) with ESMTP id E624039E8106; Sat, 14 Sep 2013 21:13:32 -0700 (PDT) X-Virus-Scanned: amavisd-new at smtp.cs.ucla.edu Received: from smtp.cs.ucla.edu ([127.0.0.1]) by localhost (smtp.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id M43b0LGfDYBV; Sat, 14 Sep 2013 21:13:32 -0700 (PDT) Received: from [192.168.10.104] (ip70-178-190-200.ks.ks.cox.net [70.178.190.200]) by smtp.cs.ucla.edu (Postfix) with ESMTPSA id 1770F39E80F8; Sat, 14 Sep 2013 21:13:31 -0700 (PDT) Message-ID: <523533E2.2090603@HIDDEN> Date: Sat, 14 Sep 2013 23:13:22 -0500 From: Paul Eggert <eggert@HIDDEN> User-Agent: Mozilla/5.0 (X11; Linux i686; rv:17.0) Gecko/20130803 Thunderbird/17.0.8 MIME-Version: 1.0 To: Martin Langhoff <martin.langhoff@HIDDEN> Subject: Re: Bug in gzip.c buffering, and two feature requests References: <CACPiFCJF1+_+8mZ4YM3Wrp2OuuffCwJb1gj6Sox_6Qc=FhNvLQ@HIDDEN> In-Reply-To: <CACPiFCJF1+_+8mZ4YM3Wrp2OuuffCwJb1gj6Sox_6Qc=FhNvLQ@HIDDEN> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x X-detected-operating-system: by eggs.gnu.org: Error: Malformed IPv6 address (bad octet value). X-Received-From: 2001:4830:134:3::11 X-Spam-Score: -4.0 (----) X-Debbugs-Envelope-To: submit Cc: Jim Meyering <jim@HIDDEN>, bug-gzip@HIDDEN X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <http://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <http://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <http://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -4.0 (----) Your gzip feature requests sound reasonable, except for this one: On 09/14/2013 07:57 PM, Martin Langhoff wrote: > Bug: gzip.c discards its buffers when it gets SIGTERM, with no > effort made to flush them to disk. That's the normal behavior of utilites when they get SIGTERM; they exit right away with minimal fuss, and they typically do not attempt to flush output buffers. > Logging via gzip adds buffering, when apache is stopped (or > restarted, usually daily to rotate logs), apache sends SIGTERM > before it closes the fh; That seems wrong. Apache should close the file handle, and let gzip finish up cleanly. Maybe Apache should kill gzip if gzip doesn't exit reasonably soon after the file handle is closed, but it'd be misguided for Apache to kill off gzip first, before closing the file handle. If this is really Apache's behavior, perhaps you should file an Apache bug report.
bug-gzip@HIDDEN:bug#15380; Package gzip.
Full text available.Received: (at submit) by debbugs.gnu.org; 15 Sep 2013 00:59:02 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Sat Sep 14 20:59:02 2013 Received: from localhost ([127.0.0.1]:37453 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1VL0g1-0006hy-K7 for submit <at> debbugs.gnu.org; Sat, 14 Sep 2013 20:59:02 -0400 Received: from eggs.gnu.org ([208.118.235.92]:51359) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from <martin.langhoff@HIDDEN>) id 1VL0ep-0006fT-Hk for submit <at> debbugs.gnu.org; Sat, 14 Sep 2013 20:57:48 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from <martin.langhoff@HIDDEN>) id 1VL0ei-0005B1-Ld for submit <at> debbugs.gnu.org; Sat, 14 Sep 2013 20:57:41 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50,FREEMAIL_FROM, T_DKIM_INVALID autolearn=disabled version=3.3.2 Received: from lists.gnu.org ([2001:4830:134:3::11]:44103) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <martin.langhoff@HIDDEN>) id 1VL0ei-0005Aw-Hq for submit <at> debbugs.gnu.org; Sat, 14 Sep 2013 20:57:40 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:53072) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from <martin.langhoff@HIDDEN>) id 1VL0eh-0002Wi-2A for bug-gzip@HIDDEN; Sat, 14 Sep 2013 20:57:40 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from <martin.langhoff@HIDDEN>) id 1VL0ee-0005AV-K2 for bug-gzip@HIDDEN; Sat, 14 Sep 2013 20:57:38 -0400 Received: from mail-qa0-x235.google.com ([2607:f8b0:400d:c00::235]:58939) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <martin.langhoff@HIDDEN>) id 1VL0ee-0005AQ-Fp for bug-gzip@HIDDEN; Sat, 14 Sep 2013 20:57:36 -0400 Received: by mail-qa0-f53.google.com with SMTP id k4so339762qaq.12 for <bug-gzip@HIDDEN>; Sat, 14 Sep 2013 17:57:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:from:date:message-id:subject:to:content-type; bh=/28/hioPV0/agXefxBgWFCpPtUn/gnhCSGiPu4FmsD8=; b=vm+8Auq3iDM5tInMX5yZTQLCKeU/W27+az3mAncEgZWADW9jsaTwO88IhchPerj2FZ rkHOypvEU0gF0MoYTZRV+0VA5zw1VhP+GakQ849KHzA4yptegnq5VUpBGG+0zs39Vh2n TROzMfUcXOZnlYdm4Kt9oTX3dlAS43p8vFyFKVwSd1qSr1/fXFN4EtkQznzYg2j6I2ty bhgqtdRZj1Qcxnja31iLy2F3gExLO+SHydxFSvZQ+mLaPHz4f26Exj+d4WJow8lZOwmt 9j6K/TKPBGNHvrlc8PfhpHiHu/LBR2G4DTEN9Pp4Yth6tGsGbPYw72O+bAEFSs3SrdpM fBhg== X-Received: by 10.49.127.179 with SMTP id nh19mr36985286qeb.1.1379206655583; Sat, 14 Sep 2013 17:57:35 -0700 (PDT) MIME-Version: 1.0 Received: by 10.49.61.35 with HTTP; Sat, 14 Sep 2013 17:57:14 -0700 (PDT) From: Martin Langhoff <martin.langhoff@HIDDEN> Date: Sat, 14 Sep 2013 20:57:14 -0400 Message-ID: <CACPiFCJF1+_+8mZ4YM3Wrp2OuuffCwJb1gj6Sox_6Qc=FhNvLQ@HIDDEN> Subject: Bug in gzip.c buffering, and two feature requests To: bug-gzip@HIDDEN, Jim Meyering <jim@HIDDEN>, Paul Eggert <eggert@HIDDEN> Content-Type: text/plain; charset=ISO-8859-1 X-detected-operating-system: by eggs.gnu.org: Error: Malformed IPv6 address (bad octet value). X-detected-operating-system: by eggs.gnu.org: Error: Malformed IPv6 address (bad octet value). X-Received-From: 2001:4830:134:3::11 X-Spam-Score: -4.0 (----) X-Debbugs-Envelope-To: submit X-Mailman-Approved-At: Sat, 14 Sep 2013 20:58:58 -0400 X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <http://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <http://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <http://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -4.0 (----) Hi gzip maintainers! In the course of trying to use gzip as a stream-compressor with Apache's "piped logs" feature I hit what I think is a bug in gzip. Additionally, using gzip as a stream log compressor triggered two related "feature requests". For the time being, I have rolled my own mini-reimplementation of gzip.c (in python, I was in a rush...) and the tool I will describe below. However, I think these improvements belong in gzip proper. Bug: gzip.c discards its buffers when it gets SIGTERM, with no effort made to flush them to disk. Logging via gzip adds buffering, when apache is stopped (or restarted, usually daily to rotate logs), apache sends SIGTERM before it closes the fh; gzip discards everything it has buffered. [ Apache logs are highly compressible, so the loss is quite significant! ] Feature requests: - for debugging / diagnostics of live services, gzip (when in "stream" mode) should flush its buffers to disk when it receives a SIGUSR1 - zcat needs to support a "followtail" option, where it reads to the end of an open gzipped file and then "follows" it (as in tail -f). This tool would be one user of the "flush on SIGUSR1" feature. Working implementations of "compresslog" (a gzip-stream-compressor) and "ztail", implemented in Python, are available here: http://repo.or.cz/w/compresslog.git Is there any interest in this? I am not familiar with gzip.c sources; and hesitant to throw a lot of effort without hearing from the maintainers. cheers, m -- martin.langhoff@HIDDEN - ask interesting questions - don't get distracted with shiny stuff - working code first ~ http://docs.moodle.org/en/User:Martin_Langhoff
Martin Langhoff <martin.langhoff@HIDDEN>:bug-gzip@HIDDEN.
Full text available.bug-gzip@HIDDEN:bug#15380; Package gzip.
Full text available.
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997 nCipher Corporation Ltd,
1994-97 Ian Jackson.