GNU logs - #15380, boring messages

Message sent to bug-gzip@HIDDEN:

Subject: bug#15380: Bug in gzip.c buffering, and two feature requests
Resent-From: Martin Langhoff <martin.langhoff@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gzip@HIDDEN
Resent-Date: Sun, 15 Sep 2013 01:00:04 +0000
Resent-Message-ID: <handler.15380.B.137920674225805 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: 15380 <at> debbugs.gnu.org, jim@HIDDEN, eggert@HIDDEN
MIME-Version: 1.0
From: Martin Langhoff <martin.langhoff@HIDDEN>
Date: Sat, 14 Sep 2013 20:57:14 -0400
Message-ID: <CACPiFCJF1+_+8mZ4YM3Wrp2OuuffCwJb1gj6Sox_6Qc=FhNvLQ@HIDDEN>
Content-Type: text/plain; charset=ISO-8859-1
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

Hi gzip maintainers!

In the course of trying to use gzip as a stream-compressor with
Apache's "piped logs" feature I hit what I think is a bug in gzip.
Additionally, using gzip as a stream log compressor triggered two
related "feature requests".

For the time being, I have rolled my own mini-reimplementation of
gzip.c (in python, I was in a rush...) and the tool I will describe
below. However, I think these improvements belong in gzip proper.

Bug: gzip.c discards its buffers when it gets SIGTERM, with no effort
made to flush them to disk. Logging via gzip adds buffering, when
apache is stopped (or restarted, usually daily to rotate logs), apache
sends SIGTERM before it closes the fh; gzip discards everything it has
buffered.

[ Apache logs are highly compressible, so the loss is quite significant! ]

Feature requests:

 - for debugging / diagnostics of live services, gzip (when in
"stream" mode) should flush its buffers to disk when it receives a
SIGUSR1

 - zcat needs to support a "followtail" option, where it reads to the
end of an open gzipped file and then "follows" it (as in tail -f).
This tool would be one user of the "flush on SIGUSR1" feature.

Working implementations of "compresslog" (a gzip-stream-compressor)
and "ztail", implemented in Python, are available here:

   http://repo.or.cz/w/compresslog.git

Is there any interest in this? I am not familiar with gzip.c sources;
and hesitant to throw a lot of effort without hearing from the
maintainers.

cheers,

m
-- 
 martin.langhoff@HIDDEN
 -  ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff

Message sent:

Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Mailer: MIME-tools 5.503 (Entity 5.503)
Content-Type: text/plain; charset=utf-8
X-Loop: help-debbugs@HIDDEN
From: help-debbugs@HIDDEN (GNU bug Tracking System)
To: Martin Langhoff <martin.langhoff@HIDDEN>
Subject: bug#15380: Acknowledgement (Bug in gzip.c buffering, and two
 feature requests)
Message-ID: <handler.15380.B.137920674225805.ack <at> debbugs.gnu.org>
References: <CACPiFCJF1+_+8mZ4YM3Wrp2OuuffCwJb1gj6Sox_6Qc=FhNvLQ@HIDDEN>
X-Gnu-PR-Message: ack 15380
X-Gnu-PR-Package: gzip
Reply-To: 15380 <at> debbugs.gnu.org
Date: Sun, 15 Sep 2013 01:00:06 +0000

Thank you for filing a new bug report with debbugs.gnu.org.

This is an automatically generated reply to let you know your message
has been received.

Your message is being forwarded to the package maintainers and other
interested parties for their attention; they will reply in due course.

Your message has been sent to the package maintainer(s):
 bug-gzip@HIDDEN

If you wish to submit further information on this problem, please
send it to 15380 <at> debbugs.gnu.org.

Please do not send mail to help-debbugs@HIDDEN unless you wish
to report a problem with the Bug-tracking system.

--=20
15380: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D15380
GNU Bug Tracking System
Contact help-debbugs@HIDDEN with problems

Message sent to bug-gzip@HIDDEN:

Subject: bug#15380: Bug in gzip.c buffering, and two feature requests
Resent-From: Paul Eggert <eggert@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gzip@HIDDEN
Resent-Date: Sun, 15 Sep 2013 04:15:02 +0000
Resent-Message-ID: <handler.15380.B.137921845311676 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: Martin Langhoff <martin.langhoff@HIDDEN>
Cc: jim@HIDDEN, 15380 <at> debbugs.gnu.org
Message-ID: <523533E2.2090603@HIDDEN>
Date: Sat, 14 Sep 2013 23:13:22 -0500
From: Paul Eggert <eggert@HIDDEN>
User-Agent: Mozilla/5.0 (X11; Linux i686;
 rv:17.0) Gecko/20130803 Thunderbird/17.0.8
MIME-Version: 1.0
References: <CACPiFCJF1+_+8mZ4YM3Wrp2OuuffCwJb1gj6Sox_6Qc=FhNvLQ@HIDDEN>
In-Reply-To: <CACPiFCJF1+_+8mZ4YM3Wrp2OuuffCwJb1gj6Sox_6Qc=FhNvLQ@HIDDEN>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

Your gzip feature requests sound reasonable, except for this one:

On 09/14/2013 07:57 PM, Martin Langhoff wrote:

> Bug: gzip.c discards its buffers when it gets SIGTERM, with no
> effort made to flush them to disk.

That's the normal behavior of utilites when they get SIGTERM; they
exit right away with minimal fuss, and they typically do not attempt
to flush output buffers.

> Logging via gzip adds buffering, when apache is stopped (or
> restarted, usually daily to rotate logs), apache sends SIGTERM
> before it closes the fh;

That seems wrong.  Apache should close the file handle, and let gzip
finish up cleanly.  Maybe Apache should kill gzip if gzip doesn't exit
reasonably soon after the file handle is closed, but it'd be misguided
for Apache to kill off gzip first, before closing the file handle.  If
this is really Apache's behavior, perhaps you should file an Apache
bug report.

Message sent to bug-gzip@HIDDEN:

Subject: bug#15380: Bug in gzip.c buffering, and two feature requests
Resent-From: Martin Langhoff <martin.langhoff@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gzip@HIDDEN
Resent-Date: Sun, 15 Sep 2013 13:37:02 +0000
Resent-Message-ID: <handler.15380.B.13792522094866 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: Paul Eggert <eggert@HIDDEN>
Cc: jim@HIDDEN, 15380 <at> debbugs.gnu.org
MIME-Version: 1.0
In-Reply-To: <523533E2.2090603@HIDDEN>
References: <CACPiFCJF1+_+8mZ4YM3Wrp2OuuffCwJb1gj6Sox_6Qc=FhNvLQ@HIDDEN>
 <523533E2.2090603@HIDDEN>
From: Martin Langhoff <martin.langhoff@HIDDEN>
Date: Sun, 15 Sep 2013 09:36:15 -0400
Message-ID: <CACPiFC+Cq3sD+Qe=31WSW-sXA+ZmnEAGE=RQZQ8fbsKZ9kdYcA@HIDDEN>
Content-Type: text/plain; charset=ISO-8859-1
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

On Sun, Sep 15, 2013 at 12:13 AM, Paul Eggert <eggert@HIDDEN> wrote:
> Your gzip feature requests sound reasonable, except for this one:

Thanks for taking the time to reply!

>> Bug: gzip.c discards its buffers when it gets SIGTERM, with no
>> effort made to flush them to disk.
>
> That's the normal behavior of utilites when they get SIGTERM; they
> exit right away with minimal fuss, and they typically do not attempt
> to flush output buffers.

I am no POSIX lawyer. Googling, I find two diverging interpretations
of SIGTERM -- one is close to SIGABRT, the other one is for a more
elegant exit.

For example, Wikipedia has:

"SIGTERM - The SIGTERM signal is sent to a process to request its
termination. Unlike the SIGKILL signal, it can be caught and
interpreted or ignored by the process. This allows the process to
perform nice termination releasing resources and saving state if
appropriate. It should be noted that SIGINT is nearly identical to
SIGTERM."

Did a bit of reading around POSIX 2008 at
http://pubs.opengroup.org/onlinepubs/9699919799/ -- if I read
http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/signal.h.html
I do find that SIGTERM should behave as per the "T" key, "Abnormal
termination of process". But it also says SIGALRM, SIGUSR1 and SIGUSR2
are "T", so this is not the information we are looking for :-)

At the end of the day, we didn't get SIGABRT. We got SIGTERM. IMHO,
flushing of buffers is reasonable in this context.

>> Logging via gzip adds buffering, when apache is stopped (or
>> restarted, usually daily to rotate logs), apache sends SIGTERM
>> before it closes the fh;
>
> That seems wrong.  Apache should close the file handle, and let gzip
> finish up cleanly.  Maybe Apache should kill gzip if gzip doesn't exit

I am double-checking the apache sources to confirm my initial reading,
but TBH I am not yet convinced that SIGTERM is wrong in this context.
(closing the fh would be nice, though).

cheers,

m
-- 
 martin.langhoff@HIDDEN
 -  ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff

Last modified: Mon, 25 Nov 2019 12:00:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.