GNU bug report logs - #25930
optimize mv for multiple bind mounts

Previous Next

Package: coreutils;

Reported by: Ruediger Meier <sweet_f_a <at> gmx.de>

Date: Thu, 2 Mar 2017 09:52:01 UTC

Severity: normal

Done: Assaf Gordon <assafgordon <at> gmail.com>

Bug is archived. No further changes may be made.

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 25930 in the body.
You can then email your comments to 25930 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to bug-coreutils <at> gnu.org:
bug#25930; Package coreutils. (Thu, 02 Mar 2017 09:52:02 GMT) Full text and rfc822 format available.

Acknowledgement sent to Ruediger Meier <sweet_f_a <at> gmx.de>:
New bug report received and forwarded. Copy sent to bug-coreutils <at> gnu.org. (Thu, 02 Mar 2017 09:52:03 GMT) Full text and rfc822 format available.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Ruediger Meier <sweet_f_a <at> gmx.de>
To: bug-coreutils <at> gnu.org
Subject: optimize mv for multiple bind mounts
Date: Thu, 2 Mar 2017 10:51:40 +0100
Hi,

I have two bind mounts of the same filesystem

$ grep "/tmp"  /etc/fstab
/dev/vg0/tmpdirs     /mnt/tmpdirs         ext4       acl,user_xattr,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0  1 2
/mnt/tmpdirs/tmp     /tmp                 none       bind                  0 0
/mnt/tmpdirs/var/tmp /var/tmp             none       bind                  0 0


Using mv to move files between the bind mounts makes a slow copy:

$ time mv /var/tmp/BIGFILE  /tmp/

real    0m0.622s
user    0m0.001s
sys     0m0.621s


In theory mv could know somehow that's the same filesystem
and do it like this:

$ time mv /mnt/tmpdirs/tmp/BIGFILE /mnt/tmpdirs/var/tmp/

real    0m0.004s
user    0m0.001s
sys     0m0.003s


Would it be possible to optimize mv regarding bind mounts? Or perhaps
are there other filesystems which do this better than ext4? Specially
the case that the original whole filesystem (/mnt/tmpdirs) is umounted
should work too somewhow.

cu,
Rudi




Information forwarded to bug-coreutils <at> gnu.org:
bug#25930; Package coreutils. (Thu, 02 Mar 2017 10:07:01 GMT) Full text and rfc822 format available.

Message #8 received at 25930 <at> debbugs.gnu.org (full text, mbox):

From: Sven Joachim <svenjoac <at> gmx.de>
To: Ruediger Meier <sweet_f_a <at> gmx.de>
Cc: 25930 <at> debbugs.gnu.org
Subject: Re: bug#25930: optimize mv for multiple bind mounts
Date: Thu, 02 Mar 2017 11:05:57 +0100
Am 02.03.2017 um 10:51 schrieb Ruediger Meier:

> I have two bind mounts of the same filesystem
>
> $ grep "/tmp"  /etc/fstab
> /dev/vg0/tmpdirs /mnt/tmpdirs ext4
> acl,user_xattr,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0
> 1 2
> /mnt/tmpdirs/tmp     /tmp                 none       bind                  0 0
> /mnt/tmpdirs/var/tmp /var/tmp             none       bind                  0 0
>
>
> Using mv to move files between the bind mounts makes a slow copy:
>
> $ time mv /var/tmp/BIGFILE  /tmp/
>
> real    0m0.622s
> user    0m0.001s
> sys     0m0.621s
>
>
> In theory mv could know somehow that's the same filesystem
> and do it like this:
>
> $ time mv /mnt/tmpdirs/tmp/BIGFILE /mnt/tmpdirs/var/tmp/
>
> real    0m0.004s
> user    0m0.001s
> sys     0m0.003s
>
>
> Would it be possible to optimize mv regarding bind mounts?

Apparently not, according to the rename(2) manpage:

,----
|   EXDEV  oldpath and newpath are not on the same mounted filesystem.
|          (Linux  permits  a  filesystem  to  be  mounted at multiple
|          points, but rename() does not work across  different  mount
|          points, even if the same filesystem is mounted on both.)
`----

Cheers,
       Sven





Information forwarded to bug-coreutils <at> gnu.org:
bug#25930; Package coreutils. (Thu, 02 Mar 2017 21:18:01 GMT) Full text and rfc822 format available.

Message #11 received at submit <at> debbugs.gnu.org (full text, mbox):

From: L A Walsh <coreutils <at> tlinx.org>
To: Sven Joachim <svenjoac <at> gmx.de>
Cc: Ruediger Meier <sweet_f_a <at> gmx.de>, bug-coreutils <at> gnu.org
Subject: Re: bug#25930: optimize mv for multiple bind mounts
Date: Thu, 02 Mar 2017 13:16:49 -0800
Sven Joachim wrote:
> ,----
> |   EXDEV  oldpath and newpath are not on the same mounted filesystem.
> |          (Linux  permits  a  filesystem  to  be  mounted at multiple
> |          points, but rename() does not work across  different  mount
> |          points, even if the same filesystem is mounted on both.)
>   
----
   That's unfortunate, as Windows recognizes moves between
the same device and does a rename vs. a copy (i.e. it doesn't
matter if the mounted object from the mount is different, as
long as the rename happens between the same devices).

   Seems like the linux kernel should have a check for that case.
(I.e. coreutils isn't likely the right place to try to fix this, since
you'd want the 'rename' optimization to happen no matter what tool
you used.

   Anyone know why Linux doesn't do detection by device vs.
by mount point?  Both pieces of info have their use, but for rename
seems that 'by device' would be optimal.








Information forwarded to bug-coreutils <at> gnu.org:
bug#25930; Package coreutils. (Thu, 02 Mar 2017 22:47:02 GMT) Full text and rfc822 format available.

Message #14 received at 25930 <at> debbugs.gnu.org (full text, mbox):

From: Bernhard Voelker <mail <at> bernhard-voelker.de>
To: L A Walsh <coreutils <at> tlinx.org>, Sven Joachim <svenjoac <at> gmx.de>
Cc: sweet_f_a <at> gmx.de, 25930 <at> debbugs.gnu.org
Subject: Re: bug#25930: optimize mv for multiple bind mounts
Date: Thu, 2 Mar 2017 23:45:49 +0100
On 03/02/2017 10:16 PM, L A Walsh wrote:
>     Anyone know why Linux doesn't do detection by device vs.
> by mount point?  Both pieces of info have their use, but for rename
> seems that 'by device' would be optimal.

quick guess: because not only the device matters.  What if e.g. the
other bind mount is read-only?

Have a nice day,
Berny




Information forwarded to bug-coreutils <at> gnu.org:
bug#25930; Package coreutils. (Fri, 03 Mar 2017 02:25:01 GMT) Full text and rfc822 format available.

Message #17 received at 25930 <at> debbugs.gnu.org (full text, mbox):

From: L A Walsh <coreutils <at> tlinx.org>
To: Bernhard Voelker <mail <at> bernhard-voelker.de>
Cc: sweet_f_a <at> gmx.de, Sven Joachim <svenjoac <at> gmx.de>, 25930 <at> debbugs.gnu.org
Subject: Re: bug#25930: optimize mv for multiple bind mounts
Date: Thu, 02 Mar 2017 18:24:36 -0800

Bernhard Voelker wrote:
> On 03/02/2017 10:16 PM, L A Walsh wrote:
>>     Anyone know why Linux doesn't do detection by device vs.
>> by mount point?  Both pieces of info have their use, but for rename
>> seems that 'by device' would be optimal.
> 
> quick guess: because not only the device matters.  What if e.g. the
> other bind mount is read-only?
---
Not "cleanly" supported.  mount man page:

  Note  that  the filesystem mount options will remain the same as
  those on the original mount point,  and  cannot  be  changed  by
  passing  the  -o  option  along  with --bind/--rbind.  The mount
  options can be changed by a separate remount command, for  exam-
  ple:

         mount --bind olddir newdir
         mount -o remount,ro newdir

  Note  that  the behavior of the remount operation depends on the
  /etc/mtab file.  The first command stores the 'bind' flag in the
  /etc/mtab  file  and  the second command reads the flag from the
  file.  If you have a system without the /etc/mtab file or if you
  explicitly  define  source  and  target  for the remount command
  (then mount(8) does not read /etc/mtab), then you  have  to  use
  the  bind  flag  (or  option)  for the remount command too.  For
  example:

         mount --bind olddir newdir
         mount -o remount,ro,bind olddir newdir

  Note that remount,ro,bind will  create  a  read-only  mountpoint
  (VFS  entry),  but the original filesystem superblock will still
  be writable, meaning that the olddir will be writable,  but  the
  newdir will be read-only.

----
	It's a bit bothersome that the behavior of the remount
is dependent on existence of /etc/mtab, contents and mount order.


	Interestingly, though, on Windows, I haven't found 
any way of mounting a file system "Read-Only", so maybe that's
not something Windows normally has to worry about in moving files.





Information forwarded to bug-coreutils <at> gnu.org:
bug#25930; Package coreutils. (Sat, 04 Mar 2017 08:02:02 GMT) Full text and rfc822 format available.

Message #20 received at 25930 <at> debbugs.gnu.org (full text, mbox):

From: Sven Joachim <svenjoac <at> gmx.de>
To: L A Walsh <coreutils <at> tlinx.org>
Cc: Ruediger Meier <sweet_f_a <at> gmx.de>, 25930 <at> debbugs.gnu.org
Subject: Re: bug#25930: optimize mv for multiple bind mounts
Date: Sat, 04 Mar 2017 09:01:14 +0100
On 2017-03-02 13:16 -0800, L A Walsh wrote:

> Sven Joachim wrote:
>> ,----
>> |   EXDEV  oldpath and newpath are not on the same mounted filesystem.
>> |          (Linux  permits  a  filesystem  to  be  mounted at multiple
>> |          points, but rename() does not work across  different  mount
>> |          points, even if the same filesystem is mounted on both.)
>>   
> ----
>    That's unfortunate, as Windows recognizes moves between
> the same device and does a rename vs. a copy (i.e. it doesn't
> matter if the mounted object from the mount is different, as
> long as the rename happens between the same devices).

Linux used to do the same in kernel 2.2, but changed the behavior when
bind mounts were introduced in 2.4.0.  In other words, it's deliberate.

>    Anyone know why Linux doesn't do detection by device vs.
> by mount point?  Both pieces of info have their use, but for rename
> seems that 'by device' would be optimal.

It is meant as a security measure, bind mounts give you the option to
confine file renames and hard links to a directory subtree even if that
directory does not live on its own filesystem.  See this mail by Al Viro
on that matter: http://yarchive.net/comp/linux/bind_mounts.html.

Cheers,
       Sven




Information forwarded to bug-coreutils <at> gnu.org:
bug#25930; Package coreutils. (Mon, 29 Oct 2018 02:51:01 GMT) Full text and rfc822 format available.

Message #23 received at 25930 <at> debbugs.gnu.org (full text, mbox):

From: Assaf Gordon <assafgordon <at> gmail.com>
To: 25930 <at> debbugs.gnu.org
Subject: Re: bug#25930: optimize mv for multiple bind mounts
Date: Sun, 28 Oct 2018 20:50:17 -0600
close 25930
stop

(triaging old bugs)

On 2017-03-04 1:01 a.m., Sven Joachim wrote:
> On 2017-03-02 13:16 -0800, L A Walsh wrote:
> 
>> Sven Joachim wrote:
>>> ,----
>>> |   EXDEV  oldpath and newpath are not on the same mounted filesystem.
>>> |          (Linux  permits  a  filesystem  to  be  mounted at multiple
>>> |          points, but rename() does not work across  different  mount
>>> |          points, even if the same filesystem is mounted on both.)
>>>    
>> ----
>>     That's unfortunate, as Windows recognizes moves between
>> the same device and does a rename vs. a copy (i.e. it doesn't
>> matter if the mounted object from the mount is different, as
>> long as the rename happens between the same devices).
> 
> Linux used to do the same in kernel 2.2, but changed the behavior when
> bind mounts were introduced in 2.4.0.  In other words, it's deliberate.
> 

Given the above (kernel limitation), I'm closing this bug.
Discussion can continue by replying to this thread.

-assaf





bug closed, send any further explanations to 25930 <at> debbugs.gnu.org and Ruediger Meier <sweet_f_a <at> gmx.de> Request was from Assaf Gordon <assafgordon <at> gmail.com> to control <at> debbugs.gnu.org. (Mon, 29 Oct 2018 02:51:02 GMT) Full text and rfc822 format available.

bug archived. Request was from Debbugs Internal Request <help-debbugs <at> gnu.org> to internal_control <at> debbugs.gnu.org. (Mon, 26 Nov 2018 12:24:06 GMT) Full text and rfc822 format available.

This bug report was last modified 5 years and 123 days ago.

Previous Next


GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.