GNU bug report logs - #19760
[bug] "tail -f" with inotify fails to follow a file after a rename()

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 19760 in the body.
You can then email your comments to 19760 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Stephane Chazelas <stephane.chazelas <at> gmail.com>
To: bug-coreutils <at> gnu.org
Subject: [bug] "tail -f" with inotify fails to follow a file after a rename()
Date: Tue, 3 Feb 2015 22:04:11 +0000

[Message part 1 (text/plain, inline)]

Hello,

On Linux, when inotify is used,

   tail -f file

follows a file only until it's renamed. After it is renamed, the
inotify watch is removed, which means tail sits there doing
nothing and any further modifications to the file are ignored.

To reproduce:

    echo 1 > file
    tail -f file &
    exec 3>> file
    echo 2 >&3
    sleep 1
    mv file file2
    sleep 1
    echo 3 >&3
    sleep 1
    : > file2

"3" is not displayed. No message about the file being truncated
either.

Work arounds:

   tail ---disable-inotify -f file
   tail -f < file # effectively disables inotify

   or rename the file with a link() followed by an unlink()
   ln file newfile && rm -f file

Note that the IN_DELETED_SELF event is not reached in
follow-descriptor mode because tail has the file open preventing
it from being deleted even after it's unlinked from the last
directory.

Path attached (on the current git head).

I don't think IN_DELETED_SELF is useful in follow-name mode
either, but I've not removed it.

-- 
Stephane

[0001-tail-fix-rename-in-follow-descriptor-mode.patch (text/x-diff, attachment)]

Message #8 received at 19760 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Stephane Chazelas <stephane.chazelas <at> gmail.com>, 
 19760 <at> debbugs.gnu.org
Subject: Re: bug#19760: [bug] "tail -f" with inotify fails to follow a file
 after a rename()
Date: Tue, 03 Feb 2015 23:30:34 +0000

On 03/02/15 22:04, Stephane Chazelas wrote:
> Hello,
> 
> On Linux, when inotify is used,
> 
>    tail -f file
> 
> follows a file only until it's renamed. After it is renamed, the
> inotify watch is removed, which means tail sits there doing
> nothing and any further modifications to the file are ignored.
> 
> To reproduce:
> 
>     echo 1 > file
>     tail -f file &
>     exec 3>> file
>     echo 2 >&3
>     sleep 1
>     mv file file2
>     sleep 1
>     echo 3 >&3
>     sleep 1
>     : > file2
> 
> "3" is not displayed. No message about the file being truncated
> either.
> 
> Work arounds:
> 
>    tail ---disable-inotify -f file
>    tail -f < file # effectively disables inotify
> 
>    or rename the file with a link() followed by an unlink()
>    ln file newfile && rm -f file
> 
> Note that the IN_DELETED_SELF event is not reached in
> follow-descriptor mode because tail has the file open preventing
> it from being deleted even after it's unlinked from the last
> directory.
> 
> Patch attached (on the current git head).

Ouch. The patch makes sense on first glance,
and all existing tests pass with it.
I'll check some more and add a test.

thanks!
Pádraig.

Message #11 received at 19760 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Stephane Chazelas <stephane.chazelas <at> gmail.com>, 
 19760 <at> debbugs.gnu.org
Subject: Re: bug#19760: [bug] "tail -f" with inotify fails to follow a file
 after a rename()
Date: Tue, 31 Mar 2015 04:15:27 +0100

[Message part 1 (text/plain, inline)]

On 03/02/15 23:30, Pádraig Brady wrote:
> On 03/02/15 22:04, Stephane Chazelas wrote:
>> Hello,
>>
>> On Linux, when inotify is used,
>>
>>    tail -f file
>>
>> follows a file only until it's renamed. After it is renamed, the
>> inotify watch is removed, which means tail sits there doing
>> nothing and any further modifications to the file are ignored.
>>
>> To reproduce:
>>
>>     echo 1 > file
>>     tail -f file &
>>     exec 3>> file
>>     echo 2 >&3
>>     sleep 1
>>     mv file file2
>>     sleep 1
>>     echo 3 >&3
>>     sleep 1
>>     : > file2
>>
>> "3" is not displayed. No message about the file being truncated
>> either.
>>
>> Work arounds:
>>
>>    tail ---disable-inotify -f file
>>    tail -f < file # effectively disables inotify
>>
>>    or rename the file with a link() followed by an unlink()
>>    ln file newfile && rm -f file
>>
>> Note that the IN_DELETED_SELF event is not reached in
>> follow-descriptor mode because tail has the file open preventing
>> it from being deleted even after it's unlinked from the last
>> directory.
>>
>> Patch attached (on the current git head).
> 
> Ouch. The patch makes sense on first glance,
> and all existing tests pass with it.
> I'll check some more and add a test.

Sorry for the delay.
I'll apply the attached in your name soon.

thanks,
Pádraig.

[tail-rename-desc.patch (text/x-patch, attachment)]

Message #14 received at 19760 <at> debbugs.gnu.org (full text, mbox):

From: Bernhard Voelker <mail <at> bernhard-voelker.de>
To: Pádraig Brady <P <at> draigBrady.com>, 
 Stephane Chazelas <stephane.chazelas <at> gmail.com>, 19760 <at> debbugs.gnu.org
Subject: Re: bug#19760: [bug] "tail -f" with inotify fails to follow a file
 after a rename()
Date: Tue, 31 Mar 2015 08:30:17 +0200

On 03/31/2015 05:15 AM, Pádraig Brady wrote:
> +  tail -f continues to follow changes to a file even after it's renamed.
> +  [bug introduced in coreutils-7.5]
> +

It is not 100% clear to me by this sentence what was the actual change;
maybe a little "again" or "now" would help?

> --- /dev/null
> +++ b/tests/tail-2/f-vs-rename.sh
> @@ -0,0 +1,51 @@
> +#!/bin/sh
> +# demonstrate that tail -f works when renaming the tailed files

s/^d/D/; s/$/./

> +# Before coreutils-8.24, tail -f a would stop tracking additions to b
> +# after "mv a b".
> +
> +# Copyright (C) 2015 Free Software Foundation, Inc.
> +
> +# This program is free software: you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation, either version 3 of the License, or
> +# (at your option) any later version.
> +
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +
> +# You should have received a copy of the GNU General Public License
> +# along with this program.  If not, see <http://www.gnu.org/licenses/>.
> +
> +. "${srcdir=.}/tests/init.sh"; path_prepend_ ./src
> +print_ver_ tail
> +
> +touch a || framework_failure_
> +
> +debug='---disable-inotify'
> +debug=
> +tail $debug -f -s.1 a > out 2>&1 & pid=$!

Shouldn't $debug be removed?  Otherwise maybe a loop over both the
inotify and the non-inotify mode would make sense?

> +
> +check_tail_output()
> +{
> +  local delay="$1"
> +  grep "$tail_re" out > /dev/null ||
> +    { sleep $delay; return 1; }
> +}

Please don't discard grep's output: reading the test's log file is
easier with this included.

> +
> +# Wait up to 12.7s for tail to start

s/$/./

> +echo x > a
> +tail_re='^x$' retry_delay_ check_tail_output .1 7 || fail=1
> +
> +mv a b || fail=1
> +
> +echo y >> b
> +# Wait up to 12.7s for "y" to appear in the output:
> +tail_re='^y$' retry_delay_ check_tail_output .1 7 || fail=1
> +
> +kill $pid
> +
> +wait
> +
> +Exit $fail

Otherwise +1 (including the changes in tail.c).

Thanks & have a nice day,
Berny

Message #19 received at 19760-done <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Bernhard Voelker <mail <at> bernhard-voelker.de>, 
 Stephane Chazelas <stephane.chazelas <at> gmail.com>, 19760-done <at> debbugs.gnu.org
Subject: Re: bug#19760: [bug] "tail -f" with inotify fails to follow a file
 after a rename()
Date: Tue, 31 Mar 2015 11:56:40 +0100

[Message part 1 (text/plain, inline)]

On 31/03/15 07:30, Bernhard Voelker wrote:
> On 03/31/2015 05:15 AM, Pádraig Brady wrote:
>> +  tail -f continues to follow changes to a file even after it's renamed.
>> +  [bug introduced in coreutils-7.5]
>> +
> 
> It is not 100% clear to me by this sentence what was the actual change;
> maybe a little "again" or "now" would help?
> 
>> --- /dev/null
>> +++ b/tests/tail-2/f-vs-rename.sh
>> @@ -0,0 +1,51 @@
>> +#!/bin/sh
>> +# demonstrate that tail -f works when renaming the tailed files
> 
> s/^d/D/; s/$/./
> 
>> +# Before coreutils-8.24, tail -f a would stop tracking additions to b
>> +# after "mv a b".
>> +
>> +# Copyright (C) 2015 Free Software Foundation, Inc.
>> +
>> +# This program is free software: you can redistribute it and/or modify
>> +# it under the terms of the GNU General Public License as published by
>> +# the Free Software Foundation, either version 3 of the License, or
>> +# (at your option) any later version.
>> +
>> +# This program is distributed in the hope that it will be useful,
>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> +# GNU General Public License for more details.
>> +
>> +# You should have received a copy of the GNU General Public License
>> +# along with this program.  If not, see <http://www.gnu.org/licenses/>.
>> +
>> +. "${srcdir=.}/tests/init.sh"; path_prepend_ ./src
>> +print_ver_ tail
>> +
>> +touch a || framework_failure_
>> +
>> +debug='---disable-inotify'
>> +debug=
>> +tail $debug -f -s.1 a > out 2>&1 & pid=$!
> 
> Shouldn't $debug be removed?  Otherwise maybe a loop over both the
> inotify and the non-inotify mode would make sense?
> 
>> +
>> +check_tail_output()
>> +{
>> +  local delay="$1"
>> +  grep "$tail_re" out > /dev/null ||
>> +    { sleep $delay; return 1; }
>> +}
> 
> Please don't discard grep's output: reading the test's log file is
> easier with this included.
> 
>> +
>> +# Wait up to 12.7s for tail to start
> 
> s/$/./
> 
>> +echo x > a
>> +tail_re='^x$' retry_delay_ check_tail_output .1 7 || fail=1
>> +
>> +mv a b || fail=1
>> +
>> +echo y >> b
>> +# Wait up to 12.7s for "y" to appear in the output:
>> +tail_re='^y$' retry_delay_ check_tail_output .1 7 || fail=1
>> +
>> +kill $pid
>> +
>> +wait
>> +
>> +Exit $fail
> 
> Otherwise +1 (including the changes in tail.c).

All good suggestions.
Latest attached.

thanks for the review!
Pádraig.

[tail-rename-desc.patch (text/x-patch, attachment)]

Message #22 received at 19760 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Bernhard Voelker <mail <at> bernhard-voelker.de>, 
 Stephane Chazelas <stephane.chazelas <at> gmail.com>, 19760 <at> debbugs.gnu.org
Subject: Re: bug#19760: [bug] "tail -f" with inotify fails to follow a file
 after a rename()
Date: Tue, 31 Mar 2015 12:35:49 +0100

BTW given that -f was broken for so long (6 years)
it lends more weight to making -f behave like -F by default.

Note POSIX allows (and even implies this),
and openBSD -f behaves like -F for example.

Not something appropriate for coreutils 8.x,
and I'd be 60:40 against changing in a later major release,
but it's worth mentioning.

cheers,
Pádraig.

Message #25 received at 19760 <at> debbugs.gnu.org (full text, mbox):

From: Bob Proulx <bob <at> proulx.com>
To: 19760 <at> debbugs.gnu.org
Cc: Stephane Chazelas <stephane.chazelas <at> gmail.com>
Subject: Re: bug#19760: [bug] "tail -f" with inotify fails to follow a file
 after a rename()
Date: Wed, 1 Apr 2015 14:23:08 -0600

Pádraig Brady wrote:
> BTW given that -f was broken for so long (6 years)

In retrospect the inotify support, while appearing nifty, has been the
source of many bugs and problems for such a simple utility.  It
certainly destabilized it.

> it lends more weight to making -f behave like -F by default.
>
> Note POSIX allows (and even implies this),
> and openBSD -f behaves like -F for example.

I think the reason this problem isn't often noticed is because once a
file has been renamed it usually is no longer written anymore.  This
makes the behavior be mostly invisible in the typical case.  Except
for the cases where it isn't and the people count on it working as
expected and it isn't.  Then it is a shame that such a simple utility
is misbehaving.

> Not something appropriate for coreutils 8.x,
> and I'd be 60:40 against changing in a later major release,
> but it's worth mentioning.

I use the -F behavior often but I think changing to -F behavior by
default actually makes it more complicated.  For example the -f
behavior is that tail follows a file.  And if I rename the file?  Then
it follows the original file.  If I unlink the file?  Then it follows
the original file.  No complexity.  Simple.  Does what it says.  (Or
at least should.)  The -F is more complicated and needs explanation
for how each of those cases is handled.

Bob

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.