GNU bug report logs - #69532
mv's new -x option should be made orthogonal to -t/-T/default

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 69532 in the body.
You can then email your comments to 69532 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Coreutils bugs <bug-coreutils <at> gnu.org>
Cc: Petr Malat <oss <at> malat.biz>
Subject: mv's new -x option should be made orthogonal to -t/-T/default
Date: Sun, 3 Mar 2024 16:44:52 -0800

Although I like the idea of exposing file swaps to the user, the first 
cut of 'mv -x' has significant problems.

I expect 'mv -x A B' to act like 'mv A B' except the destination must 
exist and is renamed back to A. However, this is not true for 'mv -x A 
B' when B is a directory; it renames B to A rather than renaming B/A to 
A as I expect. That is, 'mv -x' acts as if -T (--no-target-directory) is 
also specified. There is no way to get mv's traditional behavior, or to 
get mv -t behavior.

To fix this, 'mv -x' should respect the usual mv behavior with respect 
to directories. For example, when D is a directory 'mv -x A B C D' 
should act like 'mv A B C D' except that D's old entries should be 
renamed back to A B and C. And the -t and -T options should work with -x 
the same way they work when -x is not specified.

This needs to happen before the next coreutils release, to avoid 
confusion about 'mv -x'.

Message #8 received at 69532 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Paul Eggert <eggert <at> cs.ucla.edu>, 69532 <at> debbugs.gnu.org
Cc: Petr Malat <oss <at> malat.biz>
Subject: Re: bug#69532: mv's new -x option should be made orthogonal to
 -t/-T/default
Date: Mon, 4 Mar 2024 15:47:23 +0000

On 04/03/2024 00:44, Paul Eggert wrote:
> Although I like the idea of exposing file swaps to the user, the first
> cut of 'mv -x' has significant problems.
> 
> I expect 'mv -x A B' to act like 'mv A B' except the destination must
> exist and is renamed back to A. However, this is not true for 'mv -x A
> B' when B is a directory; it renames B to A rather than renaming B/A to
> A as I expect. That is, 'mv -x' acts as if -T (--no-target-directory) is
> also specified. There is no way to get mv's traditional behavior, or to
> get mv -t behavior.
> 
> To fix this, 'mv -x' should respect the usual mv behavior with respect
> to directories. For example, when D is a directory 'mv -x A B C D'
> should act like 'mv A B C D' except that D's old entries should be
> renamed back to A B and C. And the -t and -T options should work with -x
> the same way they work when -x is not specified.
> 
> This needs to happen before the next coreutils release, to avoid
> confusion about 'mv -x'.

So you're saying the current mv -x behavior should only be with -T explicitly specified.
With the current implementation, we should at least document that -x implies -T.

By changing as you suggest, we'd not be adding any significant functionality,
but potentially reducing confusion by following mv traditional arg handling.
I agree with you I think, but not strongly.

It's worth stating though that if users want to swap 2 directories
they'd have to `mv -Tx d1 d2`, which may also be a little confusing.

The use case we don't currently support directly is:
cd source_dir && mv -x * "$dest_dir"
Instead that currently requires:
for f in *; do mv -x "$f" "$dest_dir"; done

For completeness, stating disadvantages of pulling this use case within mv,
is that by requiring the for loop for this would allow users
to programatically determine which swap failed, and I see --swap
as primarily a programatic interface used by scripts.
Also if we made this change, We'd have to document that `mv -x 1 2 ... d`
was not atomic over the whole set.

I will look at making the change before the upcoming release.

cheers,
Pádraig

Message #11 received at 69532 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Paul Eggert <eggert <at> cs.ucla.edu>, 69532 <at> debbugs.gnu.org
Cc: Petr Malat <oss <at> malat.biz>
Subject: Re: bug#69532: mv's new -x option should be made orthogonal to
 -t/-T/default
Date: Mon, 4 Mar 2024 20:35:03 +0000

On 04/03/2024 15:47, Pádraig Brady wrote:
> On 04/03/2024 00:44, Paul Eggert wrote:
>> Although I like the idea of exposing file swaps to the user, the first
>> cut of 'mv -x' has significant problems.
>>
>> I expect 'mv -x A B' to act like 'mv A B' except the destination must
>> exist and is renamed back to A. However, this is not true for 'mv -x A
>> B' when B is a directory; it renames B to A rather than renaming B/A to
>> A as I expect. That is, 'mv -x' acts as if -T (--no-target-directory) is
>> also specified. There is no way to get mv's traditional behavior, or to
>> get mv -t behavior.
>>
>> To fix this, 'mv -x' should respect the usual mv behavior with respect
>> to directories. For example, when D is a directory 'mv -x A B C D'
>> should act like 'mv A B C D' except that D's old entries should be
>> renamed back to A B and C. And the -t and -T options should work with -x
>> the same way they work when -x is not specified.
>>
>> This needs to happen before the next coreutils release, to avoid
>> confusion about 'mv -x'.
> 
> So you're saying the current mv -x behavior should only be with -T explicitly specified.
> With the current implementation, we should at least document that -x implies -T.
> 
> By changing as you suggest, we'd not be adding any significant functionality,
> but potentially reducing confusion by following mv traditional arg handling.
> I agree with you I think, but not strongly.
> 
> It's worth stating though that if users want to swap 2 directories
> they'd have to `mv -Tx d1 d2`, which may also be a little confusing.
> 
> The use case we don't currently support directly is:
> cd source_dir && mv -x * "$dest_dir"
> Instead that currently requires:
> for f in *; do mv -x "$f" "$dest_dir"; done
> 
> For completeness, stating disadvantages of pulling this use case within mv,
> is that by requiring the for loop for this would allow users
> to programatically determine which swap failed, and I see --swap
> as primarily a programatic interface used by scripts.
> Also if we made this change, We'd have to document that `mv -x 1 2 ... d`
> was not atomic over the whole set.
> 
> I will look at making the change before the upcoming release.

Another point worth mentioning before changing this,
is that changing would make the --swap operation non symmetric.
I.e. `mv -x a d` would be different to `mv -x d a` where d in a directory.

cheers,
Pádraig

p.s. Re dir swapping I noticed that specifying '/' or '.' dirs always gives EBUSY:
  $ mv -x . .
  mv: swap of '.' and '.' failed: Device or resource busy

Message #14 received at 69532 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Petr Malat <oss <at> malat.biz>, P??draig Brady <P <at> draigbrady.com>
Cc: 69532 <at> debbugs.gnu.org
Subject: Re: bug#69532: mv's new -x option should be made orthogonal to
 -t/-T/default
Date: Mon, 4 Mar 2024 16:08:30 -0800

On 3/4/24 15:16, Petr Malat wrote:
>   I prefer KISS principle and allowing
> swapping just 2 paths.

In that case, the option should be added to the 'rename' command, not to 
'mv'.

It is not KISS to add an option to 'mv' that makes it act completely 
differently, such that most of mv's other options don't work or don't 
make sense. That would be complexity, not simplicity.

If you want simplicity, use the 'rename' command.

Message #17 received at 69532 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Petr Malat <oss <at> malat.biz>
Cc: 69532 <at> debbugs.gnu.org
Subject: Re: mv's new -x option should be made orthogonal to -t/-T/default
Date: Mon, 4 Mar 2024 16:24:27 -0800

On 3/4/24 15:37, Petr Malat wrote:
> Why do you expect this?

I expect it because mv has always treated destination directories that 
way. This has been true since the 1970s. We should not change this basic 
mode of operation.


>> To fix this, 'mv -x' should respect the usual mv behavior with respect to
>> directories. For example, when D is a directory 'mv -x A B C D' should act
>> like 'mv A B C D' except that D's old entries should be renamed back to A B
>> and C. And the -t and -T options should work with -x the same way they work
>> when -x is not specified.
> 
> I do not think this is a good idea, because that operation is not
> atomic.

There's nothing wrong with 'mv -x a b c d/' being nonatomic. "mv a b c 
d/" is not atomic either, and that's fine too.


> Probably, the user wants to prepare a temporary directory
> instead and then atomically swap it with the directory in use.

This usage was not at all obvious to me. If it's the intended use, I 
suggest inventing a new command, or adding -x to the 'rename' command 
instead. Pushing this flag onto 'mv', without making the flag reasonably 
orthogonal to mv's other options, would be a mistake because it would 
cause too much confusion.


> Supporting swap for more than 2 files also brings a problem, when
> the operation fails as then one doesn't know what has been swapped
> and what not.

I don't see why not. The user can look at mv's diagnostics. It's the 
same as regular "mv a b c d/".

Message #20 received at 69532 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Dominique Martinet <asmadeus <at> codewreck.org>,
 Pádraig Brady <P <at> draigBrady.com>
Cc: 69532 <at> debbugs.gnu.org, Petr Malat <oss <at> malat.biz>,
 Rob Landley <rob <at> landley.net>
Subject: Re: bug#69532: mv's new -x option should be made orthogonal to
 -t/-T/default
Date: Mon, 4 Mar 2024 20:10:35 -0800

On 3/4/24 16:43, Dominique Martinet wrote:
> Adding Rob to the loop because this impacts compatibility with
> toybox/maybe busybox implementations

Busybox does not use RENAME_EXCHANGE, so this isn't a Busybox issue.

Toybox mv added -x to its development version yesterday:

https://github.com/landley/toybox/commit/a2419ad52d489bf1a84a9f3aa73afb351642c765

so there's little prior art there, and there's still plenty of time to 
fix its problems before exposing it to the world.


> I also see --swap mostly used by scripts and this actually feels a bit
> dangerous to me -- I'd *always* use this with -T.

Yes, it's a problem.

By "see --swap mostly used by scripts" I assume you mean scripts that 
haven't been written yet, assuming that nobody had -x until yesterday....


> (by the way, what's this "rename" command you speak of?

https://mirrors.edge.kernel.org/pub/linux/utils/util-linux/

Now that I've looked into it further, util-linux already has an "exch" 
command that does exactly what you want. This is the command that toybox 
should implement rather than try to simulate it with "mv -x" (which 
causes all sorts of problems).

That is, toybox should revert yesterday's change to "mv", and should 
implement "exch" instead.

Message #23 received at 69532 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Pádraig Brady <P <at> draigBrady.com>, 69532 <at> debbugs.gnu.org
Cc: Petr Malat <oss <at> malat.biz>
Subject: Re: bug#69532: mv's new -x option should be made orthogonal to
 -t/-T/default
Date: Mon, 4 Mar 2024 20:35:00 -0800

On 3/4/24 12:35, Pádraig Brady wrote:
> Another point worth mentioning before changing this,
> is that changing would make the --swap operation non symmetric.
> I.e. `mv -x a d` would be different to `mv -x d a` where d in a directory.

Yes, of course. It's just like mv without the -x. After you've done an 
'mv a b', 'mv b a' will undo the rename, unless b is a directory.

I realize this business of the destination being a directory is a 
hassle. That's why I put in mv's -t and -T options years ago. But at 
this point it's such a well-known hassle that we really gotta stay 
consistent with it if we add options.

The "exch" shell command (in util-linux) is a different matter, as it 
has never had the target-directory hassle (i.e., it has always assumed -T).

In looking into this a bit more, I see a bunch of other gotchas. For 
example, 'mv -b' does not work with 'mv -x'. A bunch of other mv options 
also don't work with 'mv -x'. It's a real mess as-is and we can't ship 
it this way.

Message #26 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Petr Malat <oss <at> malat.biz>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: Coreutils bugs <bug-coreutils <at> gnu.org>
Subject: Re: mv's new -x option should be made orthogonal to -t/-T/default
Date: Tue, 5 Mar 2024 00:37:44 +0100

Hi Paul,
On Sun, Mar 03, 2024 at 04:44:52PM -0800, Paul Eggert wrote:
> Although I like the idea of exposing file swaps to the user, the first cut
> of 'mv -x' has significant problems.
> 
> I expect 'mv -x A B' to act like 'mv A B' except the destination must exist
> and is renamed back to A. However, this is not true for 'mv -x A B' when B
> is a directory; it renames B to A rather than renaming B/A to A as I expect.
> That is, 'mv -x' acts as if -T (--no-target-directory) is also specified.
> There is no way to get mv's traditional behavior, or to get mv -t behavior.

Why do you expect this? If the description in the help output was not
clear enough we can improve it by mentioning it implies -T.

> To fix this, 'mv -x' should respect the usual mv behavior with respect to
> directories. For example, when D is a directory 'mv -x A B C D' should act
> like 'mv A B C D' except that D's old entries should be renamed back to A B
> and C. And the -t and -T options should work with -x the same way they work
> when -x is not specified.

I do not think this is a good idea, because that operation is not
atomic. Probably, the user wants to prepare a temporary directory
instead and then atomically swap it with the directory in use.

The main motivation for this option is not to preserve old files, 
but to be able to atomically replace one file with another. If one
needs to preserve replaced files, he can use the backup option.

Supporting swap for more than 2 files also brings a problem, when
the operation fails as then one doesn't know what has been swapped
and what not. With plain mv one knows the file was moved if the
original location is unlinked.
BR,
  Petr

Message #29 received at 69532 <at> debbugs.gnu.org (full text, mbox):

From: Petr Malat <oss <at> malat.biz>
To: P??draig Brady <P <at> draigbrady.com>
Cc: 69532 <at> debbugs.gnu.org, Paul Eggert <eggert <at> cs.ucla.edu>
Subject: Re: bug#69532: mv's new -x option should be made orthogonal to
 -t/-T/default
Date: Tue, 5 Mar 2024 00:16:36 +0100

On Mon, Mar 04, 2024 at 08:35:03PM +0000, P??draig Brady wrote:
> On 04/03/2024 15:47, P??draig Brady wrote:
> > On 04/03/2024 00:44, Paul Eggert wrote:
> > > Although I like the idea of exposing file swaps to the user, the first
> > > cut of 'mv -x' has significant problems.
> > > 
> > > I expect 'mv -x A B' to act like 'mv A B' except the destination must
> > > exist and is renamed back to A. However, this is not true for 'mv -x A
> > > B' when B is a directory; it renames B to A rather than renaming B/A to
> > > A as I expect. That is, 'mv -x' acts as if -T (--no-target-directory) is
> > > also specified. There is no way to get mv's traditional behavior, or to
> > > get mv -t behavior.
> > > 
> > > To fix this, 'mv -x' should respect the usual mv behavior with respect
> > > to directories. For example, when D is a directory 'mv -x A B C D'
> > > should act like 'mv A B C D' except that D's old entries should be
> > > renamed back to A B and C. And the -t and -T options should work with -x
> > > the same way they work when -x is not specified.
> > > 
> > > This needs to happen before the next coreutils release, to avoid
> > > confusion about 'mv -x'.
> > 
> > So you're saying the current mv -x behavior should only be with -T explicitly specified.
> > With the current implementation, we should at least document that -x implies -T.
> > 
> > By changing as you suggest, we'd not be adding any significant functionality,
> > but potentially reducing confusion by following mv traditional arg handling.
> > I agree with you I think, but not strongly.
> > 
> > It's worth stating though that if users want to swap 2 directories
> > they'd have to `mv -Tx d1 d2`, which may also be a little confusing.
> > 
> > The use case we don't currently support directly is:
> > cd source_dir && mv -x * "$dest_dir"
> > Instead that currently requires:
> > for f in *; do mv -x "$f" "$dest_dir"; done
> > 
> > For completeness, stating disadvantages of pulling this use case within mv,
> > is that by requiring the for loop for this would allow users
> > to programatically determine which swap failed, and I see --swap
> > as primarily a programatic interface used by scripts.
> > Also if we made this change, We'd have to document that `mv -x 1 2 ... d`
> > was not atomic over the whole set.
I do not think this would be ever needed in a real life. It's more likely
one actually wants something like this:
  mkdir d.tmp
  ln -s -t d.tmp d/*
  # Modify d.tmp as needed (without propagating changes to d)
  mv --swap d d.tmp

> > I will look at making the change before the upcoming release.
> 
> Another point worth mentioning before changing this,
> is that changing would make the --swap operation non symmetric.
> I.e. `mv -x a d` would be different to `mv -x d a` where d in a directory.
I think this would lead to bugs. I prefer KISS principle and allowing
swapping just 2 paths. Explicitly mentioning -x automatically implies
-t may be added to the documentation. In general, I do not think there
is any need to make mv -x behave similar to plain mv. It's normal an
option changes behavior, at the end -t does the same.

> p.s. Re dir swapping I noticed that specifying '/' or '.' dirs always gives EBUSY:
>   $ mv -x . .
>   mv: swap of '.' and '.' failed: Device or resource busy
That's because it's CWD of mv and your shell.
  Petr

Message #32 received at 69532 <at> debbugs.gnu.org (full text, mbox):

From: Dominique Martinet <asmadeus <at> codewreck.org>
To: Pádraig Brady <P <at> draigBrady.com>
Cc: 69532 <at> debbugs.gnu.org, Petr Malat <oss <at> malat.biz>,
 Paul Eggert <eggert <at> cs.ucla.edu>, Rob Landley <rob <at> landley.net>
Subject: Re: bug#69532: mv's new -x option should be made orthogonal to
 -t/-T/default
Date: Tue, 5 Mar 2024 09:43:56 +0900

Adding Rob to the loop because this impacts compatibility with
toybox/maybe busybox implementations
(Quoting in full for convenience, there's a few more mails in
https://lists.nongnu.org/archive/html/bug-coreutils/2024-03/msg00002.html
but we seem to be missing Petr's reply)

Pádraig Brady wrote on Mon, Mar 04, 2024 at 03:47:23PM +0000:
> On 04/03/2024 00:44, Paul Eggert wrote:
> > Although I like the idea of exposing file swaps to the user, the first
> > cut of 'mv -x' has significant problems.
> > 
> > I expect 'mv -x A B' to act like 'mv A B' except the destination must
> > exist and is renamed back to A. However, this is not true for 'mv -x A
> > B' when B is a directory; it renames B to A rather than renaming B/A to
> > A as I expect. That is, 'mv -x' acts as if -T (--no-target-directory) is
> > also specified. There is no way to get mv's traditional behavior, or to
> > get mv -t behavior.
> > 
> > To fix this, 'mv -x' should respect the usual mv behavior with respect
> > to directories. For example, when D is a directory 'mv -x A B C D'
> > should act like 'mv A B C D' except that D's old entries should be
> > renamed back to A B and C. And the -t and -T options should work with -x
> > the same way they work when -x is not specified.
> > 
> > This needs to happen before the next coreutils release, to avoid
> > confusion about 'mv -x'.
> 
> So you're saying the current mv -x behavior should only be with -T explicitly specified.
> With the current implementation, we should at least document that -x implies -T.
> 
> By changing as you suggest, we'd not be adding any significant functionality,
> but potentially reducing confusion by following mv traditional arg handling.
> I agree with you I think, but not strongly.
> 
> It's worth stating though that if users want to swap 2 directories
> they'd have to `mv -Tx d1 d2`, which may also be a little confusing.
> 
> The use case we don't currently support directly is:
> cd source_dir && mv -x * "$dest_dir"
> Instead that currently requires:
> for f in *; do mv -x "$f" "$dest_dir"; done
> 
> For completeness, stating disadvantages of pulling this use case within mv,
> is that by requiring the for loop for this would allow users
> to programatically determine which swap failed, and I see --swap
> as primarily a programatic interface used by scripts.

I also see --swap mostly used by scripts and this actually feels a bit
dangerous to me -- I'd *always* use this with -T.
The point of --swap is that it's safe to use concurrently or when
something else might happen in parallel, but if you behave differently
when something else places a directory that'll likely lead to weird
script failures at some point.

I also find it a bit weird that `mv --swap a b` is different from `mv
--swap b a` -- this is normal for mv without --swap, but swap kinds of
changes my expectation of mv...
I agree with Petr's reply that operating on arguments directly is
simpler to reason about, especially in the context of scripts.

(by the way, what's this "rename" command you speak of? I know of a few
different implementations that often get placed as "rename" in $PATH but
I'm not aware of any such command that'd be installed on most systems)

> Also if we made this change, We'd have to document that `mv -x 1 2 ... d`
> was not atomic over the whole set.
> 
> I will look at making the change before the upcoming release.

Thanks,
-- 
Dominique Martinet | Asmadeus

Message #35 received at 69532 <at> debbugs.gnu.org (full text, mbox):

From: Rob Landley <rob <at> landley.net>
To: Dominique Martinet <asmadeus <at> codewreck.org>,
 Pádraig Brady <P <at> draigBrady.com>
Cc: 69532 <at> debbugs.gnu.org, Petr Malat <oss <at> malat.biz>,
 Paul Eggert <eggert <at> cs.ucla.edu>
Subject: Re: bug#69532: mv's new -x option should be made orthogonal to
 -t/-T/default
Date: Mon, 4 Mar 2024 20:24:01 -0600

On 3/4/24 18:43, Dominique Martinet wrote:
> Adding Rob to the loop because this impacts compatibility with
> toybox/maybe busybox implementations
> (Quoting in full for convenience, there's a few more mails in
> https://lists.nongnu.org/archive/html/bug-coreutils/2024-03/msg00002.html
> but we seem to be missing Petr's reply)
> 
> Pádraig Brady wrote on Mon, Mar 04, 2024 at 03:47:23PM +0000:
>> On 04/03/2024 00:44, Paul Eggert wrote:
>> > Although I like the idea of exposing file swaps to the user, the first
>> > cut of 'mv -x' has significant problems.
>> > 
>> > I expect 'mv -x A B' to act like 'mv A B' except the destination must
>> > exist and is renamed back to A. However, this is not true for 'mv -x A
>> > B' when B is a directory; it renames B to A rather than renaming B/A to
>> > A as I expect.

I don't expect that. I expect it to swap the two. I don't care if one is a
directory and the other is a symlink or file or device node or what.

I asked it to swap A and B. It did that. If I wanted to ask it to swap with B/A
I could have said so.

>> > That is, 'mv -x' acts as if -T (--no-target-directory) is
>> > also specified.

Then document that?

>> Also if we made this change, We'd have to document that `mv -x 1 2 ... d`
>> was not atomic over the whole set.

Making the atomic exchange not be atomic kind of defeats the purpose...

Rob

Message #38 received at 69532 <at> debbugs.gnu.org (full text, mbox):

From: Dominique Martinet <asmadeus <at> codewreck.org>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: 69532 <at> debbugs.gnu.org, Petr Malat <oss <at> malat.biz>,
 Pádraig Brady <P <at> draigBrady.com>,
 Rob Landley <rob <at> landley.net>
Subject: Re: bug#69532: mv's new -x option should be made orthogonal to
 -t/-T/default
Date: Tue, 5 Mar 2024 14:05:36 +0900

Paul Eggert wrote on Mon, Mar 04, 2024 at 08:10:35PM -0800:
> so there's little prior art there, and there's still plenty of time to fix
> its problems before exposing it to the world.

Yes, I just meant that everyone should agree, or there's little point in
implementing these for toybox/busybox, which is why I added Rob to Ccs
as I do not know if he's subscribed to the debbugs list.
(for busybox, Rob sent a mail to the busybox list about the feature,
someone should reply to that mail once this discussion has settled)

> > I also see --swap mostly used by scripts and this actually feels a bit
> > dangerous to me -- I'd *always* use this with -T.
> 
> Yes, it's a problem.
> 
> By "see --swap mostly used by scripts" I assume you mean scripts that
> haven't been written yet, assuming that nobody had -x until yesterday....

I have scripts that use an alternate implementation (since I've sent my
tentative patch two years ago);
I will likely be updating them to use mv -x once that is available on
the platform they're intended to run on.

> > (by the way, what's this "rename" command you speak of?
> 
> https://mirrors.edge.kernel.org/pub/linux/utils/util-linux/

Right, that's installed as rename.ul on debian because they've
historically always had another rename command.
It's not a command that can be relied on easily.... And I guess
util-linux folks decided rename_exchange didn't belong there anyway
given your next point:

> Now that I've looked into it further, util-linux already has an "exch"
> command that does exactly what you want. This is the command that toybox
> should implement rather than try to simulate it with "mv -x" (which causes
> all sorts of problems).

eh, exch has been implemented last December and isn't part of any
release yet either (although it is in v2.40-rc1 so it will be part of
v2.40 shortly)

I guess it's definitely simpler than whatever will happen to mv and
would go as far as suggesting coreutils could also drop this
implementation as well now there's an alternative, but I'll leave that
up to you.


(
  And I really wish people would stop adding new commands all the time...
  This is bad both from a discoverability point of view (I look at the
  help/man pages of commands I know more often than I go through the list
  of new commands installed on my computer) and size (util-linux doesn't
  have a single binary mode and there's a lot of overhead - especially
  since binutils have been taking up 68KB per trivial binaries for a few
  releases now)
  But anyway these two points have nothing to do with coreutils, so I'll
  stop my rant here - thanks for noticing exch; I'll switch to lurking
  from now and decide on what to do with my scripts in another while.
)

-- 
Dominique Martinet | Asmadeus

Message #41 received at 69532 <at> debbugs.gnu.org (full text, mbox):

From: Petr Malat <oss <at> malat.biz>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: 69532 <at> debbugs.gnu.org
Subject: Re: mv's new -x option should be made orthogonal to -t/-T/default
Date: Tue, 5 Mar 2024 08:17:34 +0100

On Mon, Mar 04, 2024 at 04:24:27PM -0800, Paul Eggert wrote:
> On 3/4/24 15:37, Petr Malat wrote:
> > Why do you expect this?
> 
> I expect it because mv has always treated destination directories that way.
> This has been true since the 1970s. We should not change this basic mode of
> operation.

But it doesn't behave like that when one uses -T, which is fine, because it's
documented.
 
> > > To fix this, 'mv -x' should respect the usual mv behavior with respect to
> > > directories. For example, when D is a directory 'mv -x A B C D' should act
> > > like 'mv A B C D' except that D's old entries should be renamed back to A B
> > > and C. And the -t and -T options should work with -x the same way they work
> > > when -x is not specified.
> > 
> > I do not think this is a good idea, because that operation is not
> > atomic.
> 
> There's nothing wrong with 'mv -x a b c d/' being nonatomic. "mv a b c d/"
> is not atomic either, and that's fine too.

The swap option description explicitly says it's atomic and the atomicity
was the only motivation for adding the new option.
 
> > Probably, the user wants to prepare a temporary directory
> > instead and then atomically swap it with the directory in use.
> 
> This usage was not at all obvious to me. If it's the intended use, I suggest
> inventing a new command, or adding -x to the 'rename' command instead.
> Pushing this flag onto 'mv', without making the flag reasonably orthogonal
> to mv's other options, would be a mistake because it would cause too much
> confusion.

The problem with a new command is that it's hard to find and rename
seems worst fitting than mv, because its main purpose is to replace
a pattern in a filename (e.g. change .JPG to .jpeg), so to "implement"
  mv A B
with rename one has to do
  rename A B A

Implementing atomic swap operation there doesn't seem logical to me,
I would never look for such a feature there. On the other hand when
I call mv A B, I'm generally interested in knowing if there is a time
frame, when no version of B exists.

> > Supporting swap for more than 2 files also brings a problem, when
> > the operation fails as then one doesn't know what has been swapped
> > and what not.
> 
> I don't see why not. The user can look at mv's diagnostics. It's the same as
> regular "mv a b c d/".

I don't find parsing diagnostics reliable and in general, it's something
I try to avoid in scripts. In a case I would do something like mv *.X d/,
and mv would fail, I would rather iterate over *.X than parse the output
to handle the error.
  Petr

Message #44 received at 69532 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Paul Eggert <eggert <at> cs.ucla.edu>,
 Dominique Martinet <asmadeus <at> codewreck.org>
Cc: 69532 <at> debbugs.gnu.org, Petr Malat <oss <at> malat.biz>,
 util-linux <util-linux <at> vger.kernel.org>, Rob Landley <rob <at> landley.net>
Subject: Re: bug#69532: mv's new -x option should be made orthogonal to
 -t/-T/default
Date: Tue, 5 Mar 2024 14:16:05 +0000

On 05/03/2024 04:10, Paul Eggert wrote:
> On 3/4/24 16:43, Dominique Martinet wrote:
>> Adding Rob to the loop because this impacts compatibility with
>> toybox/maybe busybox implementations
> 
> Busybox does not use RENAME_EXCHANGE, so this isn't a Busybox issue.
> 
> Toybox mv added -x to its development version yesterday:
> 
> https://github.com/landley/toybox/commit/a2419ad52d489bf1a84a9f3aa73afb351642c765
> 
> so there's little prior art there, and there's still plenty of time to
> fix its problems before exposing it to the world.
> 
> 
>> I also see --swap mostly used by scripts and this actually feels a bit
>> dangerous to me -- I'd *always* use this with -T.
> 
> Yes, it's a problem.
> 
> By "see --swap mostly used by scripts" I assume you mean scripts that
> haven't been written yet, assuming that nobody had -x until yesterday....
> 
> 
>> (by the way, what's this "rename" command you speak of?
> 
> https://mirrors.edge.kernel.org/pub/linux/utils/util-linux/
> 
> Now that I've looked into it further, util-linux already has an "exch"
> command that does exactly what you want. This is the command that toybox
> should implement rather than try to simulate it with "mv -x" (which
> causes all sorts of problems).
> 
> That is, toybox should revert yesterday's change to "mv", and should
> implement "exch" instead.

I think having the functionality in mv(1) is better than in rename(1),
but since exch(1) is already released that's probably
the best place for this functionality now.

A separate exch command may be overkill for just this,
but perhaps related functionality might be added to that command in future.
For e.g. some of the discussed functionality for a "replace" command
might reside there.

So I think I'll remove the as yet unreleased mv --swap from coreutils, given that
util-linux is as widely available as coreutils on GNU/Linux platforms.

cheers,
Pádraig

Message #47 received at 69532 <at> debbugs.gnu.org (full text, mbox):

From: Karel Zak <kzak <at> redhat.com>
To: Pádraig Brady <P <at> draigBrady.com>
Cc: Petr Malat <oss <at> malat.biz>, util-linux <util-linux <at> vger.kernel.org>,
 Paul Eggert <eggert <at> cs.ucla.edu>, Dominique Martinet <asmadeus <at> codewreck.org>,
 69532 <at> debbugs.gnu.org, Rob Landley <rob <at> landley.net>
Subject: Re: bug#69532: mv's new -x option should be made orthogonal to
 -t/-T/default
Date: Tue, 5 Mar 2024 21:41:02 +0100

On Tue, Mar 05, 2024 at 02:16:05PM +0000, Pádraig Brady wrote:

> I think having the functionality in mv(1) is better than in rename(1),
> but since exch(1) is already released that's probably
> the best place for this functionality now.
> 
> A separate exch command may be overkill for just this,

rename(1) was also my initial idea, but it's too complex and rarely used
by users for simple tasks like those we can now achieve with the new simple
command exch(1).

> but perhaps related functionality might be added to that command in future.
> For e.g. some of the discussed functionality for a "replace" command
> might reside there.
> 
> So I think I'll remove the as yet unreleased mv --swap from coreutils, given that
> util-linux is as widely available as coreutils on GNU/Linux platforms.

Yes, it seems better to have this Linux-specific feature in
util-linux. We should discuss such changes early next time ;-) 

Thanks for CC:

Karel

-- 
 Karel Zak  <kzak <at> redhat.com>
 http://karelzak.blogspot.com

Message #50 received at 69532 <at> debbugs.gnu.org (full text, mbox):

From: Masatake YAMATO <yamato <at> redhat.com>
To: P <at> draigBrady.com
Cc: oss <at> malat.biz, util-linux <at> vger.kernel.org, eggert <at> cs.ucla.edu,
 asmadeus <at> codewreck.org, 69532 <at> debbugs.gnu.org, rob <at> landley.net
Subject: Re: bug#69532: mv's new -x option should be made orthogonal to
 -t/-T/default
Date: Wed, 06 Mar 2024 07:13:21 +0900 (JST)

When I knew RENAME_EXCHANGE, I thought we should extend mv command as
you did: adding --swap.  However, after researching the past
challenges, I decided not to propose the feature to coreutils.

    https://www.gnu.org/software/coreutils/rejected_requests.html
    https://lists.gnu.org/archive/html/coreutils/2018-12/msg00004.html
    https://www.mail-archive.com/coreutils <at> gnu.org/msg10276.html

Masatake YAMATO

> On 05/03/2024 04:10, Paul Eggert wrote:
>> On 3/4/24 16:43, Dominique Martinet wrote:
>>> Adding Rob to the loop because this impacts compatibility with
>>> toybox/maybe busybox implementations
>> Busybox does not use RENAME_EXCHANGE, so this isn't a Busybox issue.
>> Toybox mv added -x to its development version yesterday:
>> https://github.com/landley/toybox/commit/a2419ad52d489bf1a84a9f3aa73afb351642c765
>> so there's little prior art there, and there's still plenty of time to
>> fix its problems before exposing it to the world.
>> 
>>> I also see --swap mostly used by scripts and this actually feels a bit
>>> dangerous to me -- I'd *always* use this with -T.
>> Yes, it's a problem.
>> By "see --swap mostly used by scripts" I assume you mean scripts that
>> haven't been written yet, assuming that nobody had -x until
>> yesterday....
>> 
>>> (by the way, what's this "rename" command you speak of?
>> https://mirrors.edge.kernel.org/pub/linux/utils/util-linux/
>> Now that I've looked into it further, util-linux already has an "exch"
>> command that does exactly what you want. This is the command that
>> toybox
>> should implement rather than try to simulate it with "mv -x" (which
>> causes all sorts of problems).
>> That is, toybox should revert yesterday's change to "mv", and should
>> implement "exch" instead.
> 
> I think having the functionality in mv(1) is better than in rename(1),
> but since exch(1) is already released that's probably
> the best place for this functionality now.
> 
> A separate exch command may be overkill for just this,
> but perhaps related functionality might be added to that command in
> future.
> For e.g. some of the discussed functionality for a "replace" command
> might reside there.
> 
> So I think I'll remove the as yet unreleased mv --swap from coreutils,
> given that
> util-linux is as widely available as coreutils on GNU/Linux platforms.
> 
> cheers,
> Pádraig
> 
> 

Message #53 received at 69532 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Pádraig Brady <P <at> draigBrady.com>
Cc: Petr Malat <oss <at> malat.biz>, util-linux <util-linux <at> vger.kernel.org>,
 Dominique Martinet <asmadeus <at> codewreck.org>, 69532 <at> debbugs.gnu.org,
 Karel Zak <kzak <at> redhat.com>, Rob Landley <rob <at> landley.net>
Subject: Re: bug#69532: mv's new -x option should be made orthogonal to
 -t/-T/default
Date: Sat, 16 Mar 2024 23:10:50 -0700

[Message part 1 (text/plain, inline)]

On 2024-03-05 06:16, Pádraig Brady wrote:
> I think I'll remove the as yet unreleased mv --swap from coreutils, 
> given that
> util-linux is as widely available as coreutils on GNU/Linux platforms.

Although removing that "mv --swap" implementation was a win, I don't 
think we can simply delegate this to util-linux's exch command. 
Exchanging files via a renameat-like call is not limited to the Linux 
kernel; it's also possible on macOS via renameatx_np with RENAME_SWAP, 
and there have been noises about adding similar things to other 
operating systems.

I just now added support for macOS renameatx_np to Gnulib, so coreutils 
does not need to worry about the macOS details; it can simply use 
renameatu with the Linux flags. See:

https://git.savannah.gnu.org/cgit/gnulib.git/commit/?id=af32ee824ee18255839f9812b8ed61aa5257a82b

Even with Linux it's dicey. People may have older util-linux installed 
and so lack the 'exch' utility; this is true for both Fedora 39 and 
Ubuntu 23.10, the current releases. Ubuntu is also odd in that it 
doesn't install all the util-linux utilities as part of the util-linux 
package, so it's not clear what they will do with 'exch'.

So I propose that we implement the idea in coreutils in a better way, 
that interacts more nicely with -t, -T, etc. Also, I suggest using the 
Linuxish name "--exchange" instead of the macOSish name "--swap", and 
(for now at least) not giving the option a single-letter equivalent as I 
expect it to be useful from scripts, not interactively.

After looking at various ways to do it I came up with the attached 
proposed patch. This should work on both GNU/Linux and macOS, if your OS 
is recent enough and the file system supports atomic exchange.

[0001-mv-new-option-exchange.patch (text/x-patch, attachment)]

Message #56 received at 69532 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: Petr Malat <oss <at> malat.biz>, util-linux <util-linux <at> vger.kernel.org>,
 Dominique Martinet <asmadeus <at> codewreck.org>, 69532 <at> debbugs.gnu.org,
 Karel Zak <kzak <at> redhat.com>, Rob Landley <rob <at> landley.net>
Subject: Re: bug#69532: mv's new -x option should be made orthogonal to
 -t/-T/default
Date: Sun, 17 Mar 2024 11:32:15 +0000

On 17/03/2024 06:10, Paul Eggert wrote:
> On 2024-03-05 06:16, Pádraig Brady wrote:
>> I think I'll remove the as yet unreleased mv --swap from coreutils,
>> given that
>> util-linux is as widely available as coreutils on GNU/Linux platforms.
> 
> Although removing that "mv --swap" implementation was a win, I don't
> think we can simply delegate this to util-linux's exch command.
> Exchanging files via a renameat-like call is not limited to the Linux
> kernel; it's also possible on macOS via renameatx_np with RENAME_SWAP,
> and there have been noises about adding similar things to other
> operating systems.
> 
> I just now added support for macOS renameatx_np to Gnulib, so coreutils
> does not need to worry about the macOS details; it can simply use
> renameatu with the Linux flags. See:
> 
> https://git.savannah.gnu.org/cgit/gnulib.git/commit/?id=af32ee824ee18255839f9812b8ed61aa5257a82b
> 
> Even with Linux it's dicey. People may have older util-linux installed
> and so lack the 'exch' utility; this is true for both Fedora 39 and
> Ubuntu 23.10, the current releases. Ubuntu is also odd in that it
> doesn't install all the util-linux utilities as part of the util-linux
> package, so it's not clear what they will do with 'exch'.
> 
> So I propose that we implement the idea in coreutils in a better way,
> that interacts more nicely with -t, -T, etc. Also, I suggest using the
> Linuxish name "--exchange" instead of the macOSish name "--swap", and
> (for now at least) not giving the option a single-letter equivalent as I
> expect it to be useful from scripts, not interactively.
> 
> After looking at various ways to do it I came up with the attached
> proposed patch. This should work on both GNU/Linux and macOS, if your OS
> is recent enough and the file system supports atomic exchange.

The implementation looks good.

Re exch(1) on macos, I see util-linux is on homebrew,
so it would be relatively easy to ifdef renameatx_np in util-linux also.

I think the --no-copy situation is brittle, as scripts not using it now
would be atomic, but then if we ever supported cross fs swaps
it may become non atomic. I'd at least doc with a line in the --exchange
description in usage() to say something like:
  "Use --no-copy to enforce atomic operation"

While the most flexible, it's also quite awkward to need
`mv --exchange --no-copy --no-target-directory` for most uses.
I.e. it's tempting to imply the --no-... options with --exchange,
but I suppose since scripting is the primary use case for this
flexibility trumps conciseness, so I'm ok with the verbosity I think.

thanks,
Pádraig

Message #59 received at 69532 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: Petr Malat <oss <at> malat.biz>, util-linux <util-linux <at> vger.kernel.org>,
 Dominique Martinet <asmadeus <at> codewreck.org>, 69532 <at> debbugs.gnu.org,
 Karel Zak <kzak <at> redhat.com>, Rob Landley <rob <at> landley.net>
Subject: Re: bug#69532: mv's new -x option should be made orthogonal to
 -t/-T/default
Date: Sun, 17 Mar 2024 11:40:00 +0000

On 17/03/2024 11:32, Pádraig Brady wrote:
> On 17/03/2024 06:10, Paul Eggert wrote:
>> On 2024-03-05 06:16, Pádraig Brady wrote:
>>> I think I'll remove the as yet unreleased mv --swap from coreutils,
>>> given that
>>> util-linux is as widely available as coreutils on GNU/Linux platforms.
>>
>> Although removing that "mv --swap" implementation was a win, I don't
>> think we can simply delegate this to util-linux's exch command.
>> Exchanging files via a renameat-like call is not limited to the Linux
>> kernel; it's also possible on macOS via renameatx_np with RENAME_SWAP,
>> and there have been noises about adding similar things to other
>> operating systems.
>>
>> I just now added support for macOS renameatx_np to Gnulib, so coreutils
>> does not need to worry about the macOS details; it can simply use
>> renameatu with the Linux flags. See:
>>
>> https://git.savannah.gnu.org/cgit/gnulib.git/commit/?id=af32ee824ee18255839f9812b8ed61aa5257a82b
>>
>> Even with Linux it's dicey. People may have older util-linux installed
>> and so lack the 'exch' utility; this is true for both Fedora 39 and
>> Ubuntu 23.10, the current releases. Ubuntu is also odd in that it
>> doesn't install all the util-linux utilities as part of the util-linux
>> package, so it's not clear what they will do with 'exch'.
>>
>> So I propose that we implement the idea in coreutils in a better way,
>> that interacts more nicely with -t, -T, etc. Also, I suggest using the
>> Linuxish name "--exchange" instead of the macOSish name "--swap", and
>> (for now at least) not giving the option a single-letter equivalent as I
>> expect it to be useful from scripts, not interactively.
>>
>> After looking at various ways to do it I came up with the attached
>> proposed patch. This should work on both GNU/Linux and macOS, if your OS
>> is recent enough and the file system supports atomic exchange.
> 
> The implementation looks good.
> 
> Re exch(1) on macos, I see util-linux is on homebrew,
> so it would be relatively easy to ifdef renameatx_np in util-linux also.
> 
> I think the --no-copy situation is brittle, as scripts not using it now
> would be atomic, but then if we ever supported cross fs swaps
> it may become non atomic. I'd at least doc with a line in the --exchange
> description in usage() to say something like:
>     "Use --no-copy to enforce atomic operation"
> 
> While the most flexible, it's also quite awkward to need
> `mv --exchange --no-copy --no-target-directory` for most uses.
> I.e. it's tempting to imply the --no-... options with --exchange,
> but I suppose since scripting is the primary use case for this
> flexibility trumps conciseness, so I'm ok with the verbosity I think.

Oh also in the texinfo I think it's important to mention that the swap
will "exchange all data and metadata". That's not obvious otherwise.
For example users may be wondering if only data was being exchanged
with the macos exchangedata(2) or equivalent.

cheers,
Pádraig

Message #62 received at 69532 <at> debbugs.gnu.org (full text, mbox):

From: Bernhard Voelker <mail <at> bernhard-voelker.de>
To: Paul Eggert <eggert <at> cs.ucla.edu>, Pádraig Brady
 <P <at> draigBrady.com>
Cc: 69532 <at> debbugs.gnu.org, util-linux <util-linux <at> vger.kernel.org>,
 Dominique Martinet <asmadeus <at> codewreck.org>, Petr Malat <oss <at> malat.biz>,
 Karel Zak <kzak <at> redhat.com>, Rob Landley <rob <at> landley.net>
Subject: Re: bug#69532: mv's new -x option should be made orthogonal to
 -t/-T/default
Date: Wed, 20 Mar 2024 20:43:40 +0100

On 3/17/24 07:10, Paul Eggert wrote:
> Although removing that "mv --swap" implementation was a win, I don't
> think we can simply delegate this to util-linux's exch command.

I still have some headache adding this.

This stems from the fact that although mv(1) is a userland frontend
for renameat(2), the user interface is different:
while renameat(2) deals exactly with 2 operands, mv(1) has always
been able to work on more arguments.

Now, extending "exchange" to more arguments is confusing and the
use is not intuitive:
  mv -v --exchange  a b c d

An "exchange" can literally only be applied to 2 files,
and 'exch' is IMO fine.

I have the gut feeling that we didn't think through all cases,
and that some might be surprising, e.g.:

  $ mkdir d; echo 1 > a; echo 2 > d/a
  $ src/mv --exchange a a a a d/a

versus

  $ src/mv --exchange a a a a d/a

Have a nice day,
Berny

Message #65 received at 69532 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Bernhard Voelker <mail <at> bernhard-voelker.de>,
 Pádraig Brady <P <at> draigBrady.com>
Cc: 69532 <at> debbugs.gnu.org, util-linux <util-linux <at> vger.kernel.org>,
 Dominique Martinet <asmadeus <at> codewreck.org>, Petr Malat <oss <at> malat.biz>,
 Karel Zak <kzak <at> redhat.com>, Rob Landley <rob <at> landley.net>
Subject: Re: bug#69532: mv's new -x option should be made orthogonal to
 -t/-T/default
Date: Wed, 20 Mar 2024 13:56:44 -0700

On 3/20/24 12:43, Bernhard Voelker wrote:

> This stems from the fact that although mv(1) is a userland frontend
> for renameat(2), the user interface is different:
> while renameat(2) deals exactly with 2 operands, mv(1) has always
> been able to work on more arguments.

Yes, that's mv's original sin, which we cannot realistically change now.


> Now, extending "exchange" to more arguments is confusing and the
> use is not intuitive:
>    mv -v --exchange  a b c d
> 
> An "exchange" can literally only be applied to 2 files,

Sure, but that's true for "rename" too: a "rename" can be applied only 
to 2 files.

When d is a directory, "mv a b c d" does three renames so it is like "mv 
a d/a; mv b d/b; mv c d/c". This remains true if you uniformly replace 
"mv" with "mv --exchange", which does three exchanges.


> I have the gut feeling that we didn't think through all cases,
> and that some might be surprising, e.g.:
> 
>    $ mkdir d; echo 1 > a; echo 2 > d/a
>    $ src/mv --exchange a a a a d/a
> 
> versus
> 
>    $ src/mv --exchange a a a a d/a

I don't understand the word "versus" here, as the two examples look the 
same to me.

If d/a is not a directory, the example is an error, just as it would be 
without --exchange.

If d/a is a directory and you have permissions etc., "mv a a a a d/a" is 
like attempting "mv -T a d/a/a; mv -T a d/a/a; mv -T a d/a/a; mv -T a 
d/a/a". If you use plain "mv" only the first "mv -T a d/a/a" succeeds 
because "a" goes away, so you get three diagnostics for the remaining 
three "a"s. If you use "mv --exchange" all four "mv --exchange -T a 
d/a/a" attempts succeed, and since there are an even number of exchanges 
the end result is a no-op except for updated directory timestamps. So I 
don't see any ambiguity about what mv should do with this contrived example.

Message #68 received at 69532 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Pádraig Brady <P <at> draigBrady.com>
Cc: Petr Malat <oss <at> malat.biz>, util-linux <util-linux <at> vger.kernel.org>,
 Dominique Martinet <asmadeus <at> codewreck.org>, 69532 <at> debbugs.gnu.org,
 Karel Zak <kzak <at> redhat.com>, Rob Landley <rob <at> landley.net>
Subject: Re: bug#69532: mv's new -x option should be made orthogonal to
 -t/-T/default
Date: Wed, 20 Mar 2024 15:10:26 -0700

On 3/17/24 04:32, Pádraig Brady wrote:
> I think the --no-copy situation is brittle, as scripts not using it now
> would be atomic, but then if we ever supported cross fs swaps
> it may become non atomic. I'd at least doc with a line in the --exchange
> description in usage() to say something like:
>    "Use --no-copy to enforce atomic operation"

But --no-copy doesn't mean atomic operation; it simply means don't copy.

On systems that don't have an atomic exchange, we can emulate "mv 
--exchange --no-copy a b" with three calls link("b", "b.tmp"); 
rename("a","b"); rename("b.tmp","a"). This wouldn't copy, but it 
wouldn't be atomic.

Although atomicity is important, currently the coreutils documentation 
doesn't cover it and the code doesn't always handle it well. For 
example, if A and B are regular files "mv -b A B" briefly has a moment 
when B stops existing. To my mind this is a bug: an existing destination 
shouldn't stop existing merely because you're replacing it. At some 
point this stuff should be documented better and this (and probably some 
other) atomicity bugs fixed.

One thought I had while looking into this was that we could add an 
--atomic option to mv and ln, such that the command fails if the 
destination cannot be updated atomically. That would be a stronger 
option than --no-copy. (In some cases we could support --atomic even for 
'cp', no?)

Anyway, for now I installed the patch with some minor changes to the 
documentation's --exchange section to try to document this tricky area 
more clearly. Here's the revised doc section. It also incorporates your 
later suggestion to mention both data and metadata.

----

@item --exchange
@opindex --exchange
Exchange source and destination instead of renaming source to destination.
Both files must exist; they need not be the same type.
This exchanges all data and metadata.

This option can be used to replace one directory with another.
When used this way, it should be combined with
@code{--no-target-directory} (@option{-T})
to avoid confusion about the destination location.
For example, you might use @samp{mv -T --exchange @var{d1} @var{d2}}
to exchange two directories @var{d1} and @var{d2}.

Exchanges are atomic if the source and destination are both in a
single file system that supports atomic exchange.
Non-atomic exchanges are not yet supported.

If the source and destination might not be on the same file system,
using @code{--no-copy} will prevent future versions of @command{mv}
from implementing the exchange by copying.

Message #73 received at 69532 <at> debbugs.gnu.org (full text, mbox):

From: Bernhard Voelker <mail <at> bernhard-voelker.de>
To: Paul Eggert <eggert <at> cs.ucla.edu>, Pádraig Brady
 <P <at> draigBrady.com>
Cc: Petr Malat <oss <at> malat.biz>, util-linux <util-linux <at> vger.kernel.org>,
 Dominique Martinet <asmadeus <at> codewreck.org>, 69532 <at> debbugs.gnu.org,
 Karel Zak <kzak <at> redhat.com>, Rob Landley <rob <at> landley.net>
Subject: Re: bug#69532: mv's new -x option should be made orthogonal to
 -t/-T/default
Date: Wed, 20 Mar 2024 23:53:05 +0100

On 3/20/24 21:56, Paul Eggert wrote:
> On 3/20/24 12:43, Bernhard Voelker wrote:
>
>> This stems from the fact that although mv(1) is a userland frontend
>> for renameat(2), the user interface is different:
>> while renameat(2) deals exactly with 2 operands, mv(1) has always
>> been able to work on more arguments.
>
> Yes, that's mv's original sin, which we cannot realistically change now.

I wouldn't go that far that it was a sin.  It's useful and people got
used to it without having to think about it.

>> I have the gut feeling that we didn't think through all cases,
>> and that some might be surprising, e.g.:
>>
>>     $ mkdir d; echo 1 > a; echo 2 > d/a
>>     $ src/mv --exchange a a a a d/a
>>
>> versus
>>
>>     $ src/mv --exchange a a a a d/a
>
> I don't understand the word "versus" here, as the two examples look the
> same to me.

sorry, I messed the example up.

  $ echo 1 > a
  $ mkdir d
  $ echo 2 > d/a
  $ src/mv -v --exchange a a a d
  renamed 'a' -> 'd/a'
  renamed 'a' -> 'd/a'
  renamed 'a' -> 'd/a'
  $ cat a
  2
  $ src/mv -v --exchange a a a d
  renamed 'a' -> 'd/a'
  renamed 'a' -> 'd/a'
  renamed 'a' -> 'd/a'
  $ cat a
  1
  $ src/mv -v --exchange a a a a d
  renamed 'a' -> 'd/a'
  renamed 'a' -> 'd/a'
  renamed 'a' -> 'd/a'
  renamed 'a' -> 'd/a'
  $ cat a
  1

I remember some implementation where mv(1) really was just a rename(2),
which failed when crossing file systems.  Was it some HP-UX or Solaris mv(1)?
mv(1) has learned to copy+delete over time, which is what people would
expect from a "move".

My point is that "exchange" is a different functionality.
It's well somehow belonging and related to what renameat(2) is doing in the kernel,
and therefore it comes in handy that we can simply call it with an additional flag.
Yet it's IMO a different operation.  I bet there had been discussions whether
to create a new syscall, but apparently it was easier to put it with a flag
into an existing one.  Fine for the kernel.

On userland OTOH, we have broader choice.
Karel did his choice in util-linux for exch(1), and coreutils could expose
the same functionality.

For other feature requests, we were much more reluctant in coreutils ... for
good reasons: feature bloat, maintainability, etc.

So I'm asking myself what is different this time?
- The feature already exists -> util-linux.
- Okay, we're using the same syscall, renameat(2) -> it's tempting.
- How large is the useful overlap with the existing code of mv(1)?
  Not much: no traditional rename nor copy.
- How large is the useful overlap with the existing options/modes of mv(1)?
  - exchange contradicts --backup,
  - exchange is not useful together with options working with a regular
    rename of copy, at least: --update, -Z, -n.
  - not sure if exchange works well together with -f.

I'm currently only 20:80 for adding it to mv(1).
The functionality is cool, but do we need to press it into mv(1) with so many
incompatibilities just because it's requiring renameat(2) we already use?
Maybe to consider: One tool for one thing ... means another tool for another thing.

Again, I have the gut feeling that we've missed some cases to think about.
And once the feature would be in ...

Furthermore, why does exchange not work to exchange a regular with a directory file?
We've all learned that everything's a file, so it cannot be explained to users that
exchanging a regular file with a directory doesn't work.

Finally, the test cases are very sparse: no cases with different owners, different
directory permissions, different file types (if we know already f<->d doesn't work),
triggering races, etc.

I don't really want to object to add it, but I find it quite odd as of today.

Have a nice day,
Berny

Message #76 received at 69532 <at> debbugs.gnu.org (full text, mbox):

From: Rob Landley <rob <at> landley.net>
To: Bernhard Voelker <mail <at> bernhard-voelker.de>,
 Paul Eggert <eggert <at> cs.ucla.edu>, Pádraig Brady
 <P <at> draigBrady.com>
Cc: Petr Malat <oss <at> malat.biz>, 69532 <at> debbugs.gnu.org,
 util-linux <util-linux <at> vger.kernel.org>, Karel Zak <kzak <at> redhat.com>,
 Dominique Martinet <asmadeus <at> codewreck.org>
Subject: Re: bug#69532: mv's new -x option should be made orthogonal to
 -t/-T/default
Date: Wed, 20 Mar 2024 19:03:09 -0500

On 3/20/24 14:43, Bernhard Voelker wrote:
> On 3/17/24 07:10, Paul Eggert wrote:
> Now, extending "exchange" to more arguments is confusing and the
> use is not intuitive:
>    mv -v --exchange  a b c d

It's also pointless. An atomic exchange on more than 2 files ISN'T ATOMIC.
That's why I didn't do it.

You already had "mv -T" requiring exactly two arguments, so thinking mv -x has
cooties because it works the same way is just weird.

> I have the gut feeling that we didn't think through all cases,

Sounds like. Having mv modify its source directory during recursive descent is
creepy.

Toybox implemented:

-x	Atomically exchange source/dest (--swap)

Which behaves like:

  $ ./mv -x one two
  $ ./mv -x one two three
  mv: -x needs 2 args

My change from this discussion was adding the "--swap" synonym you wanted.

Rob

Message #79 received at 69532 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Bernhard Voelker <mail <at> bernhard-voelker.de>,
 Pádraig Brady <P <at> draigBrady.com>
Cc: 69532 <at> debbugs.gnu.org, util-linux <util-linux <at> vger.kernel.org>,
 Dominique Martinet <asmadeus <at> codewreck.org>, Petr Malat <oss <at> malat.biz>,
 Karel Zak <kzak <at> redhat.com>, Rob Landley <rob <at> landley.net>
Subject: Re: bug#69532: mv's new -x option should be made orthogonal to
 -t/-T/default
Date: Wed, 20 Mar 2024 16:56:32 -0700

On 3/20/24 15:53, Bernhard Voelker wrote:

>    $ echo 1 > a
>    $ mkdir d
>    $ echo 2 > d/a
>    $ src/mv -v --exchange a a a d
>    renamed 'a' -> 'd/a'
>    renamed 'a' -> 'd/a'
>    renamed 'a' -> 'd/a'
>    $ cat a
>    2
>    $ src/mv -v --exchange a a a d
>    renamed 'a' -> 'd/a'
>    renamed 'a' -> 'd/a'
>    renamed 'a' -> 'd/a'
>    $ cat a
>    1
>    $ src/mv -v --exchange a a a a d
>    renamed 'a' -> 'd/a'
>    renamed 'a' -> 'd/a'
>    renamed 'a' -> 'd/a'
>    renamed 'a' -> 'd/a'
>    $ cat a
>    1

Yes, that's the expected behavior for this contrived case. Just as one 
would get odd behavior if one did the same thing without --exchange.


> I remember some implementation where mv(1) really was just a rename(2),
> which failed when crossing file systems.  Was it some HP-UX or Solaris mv(1)?

I doubt it. Even 7th Edition 'mv' (1979) fell back on 'cp' when the link 
syscall failed (this was before 'rename' existed).


> My point is that "exchange" is a different functionality.

Yes, but it's closely related. Arguably --backup is also a different 
functionality too (and arguably --exchange is simply an alternative 
backup scheme!) but 'mv' has --backup.


> - How large is the useful overlap with the existing code of mv(1)?
>    Not much: no traditional rename nor copy.

I don't follow this point. The code change was fairly small, which 
indicates there was a lot of overlap with existing functionality.


> - How large is the useful overlap with the existing options/modes of mv(1)?
>    - exchange contradicts --backup,

That could be fixed for regular files, if there's a need, by backing up 
the destination via 'link' before exchanging. For directories it's 
admittedly a problem, but that's also the case for plain 'mv' (or for 
'cp' or 'ln', for that matter) so there's not much new here.


>    - exchange is not useful together with options working with a regular
>      rename of copy, at least: --update, -Z, -n.

It should work with --update and -Z. -n of course is logically 
incompatible, but this not the only set of logically incompatible 
options (e.g., -t vs -T).


>    - not sure if exchange works well together with -f.

What problems do you see there?



> why does exchange not work to exchange a regular with a 
> directory file?

It works. I don't see a problem there.

  $ touch a
  $ mkdir d
  $ ./mv -T --exchange a d
  $ ls -ld a d
  drwxr-xr-x. 2 eggert eggert 4096 Mar 20 16:52 a
  -rw-r--r--. 1 eggert eggert    0 Mar 20 16:52 d


> Finally, the test cases are very sparse:

Feel free to add some. :-)

Message #82 received at 69532 <at> debbugs.gnu.org (full text, mbox):

From: Bernhard Voelker <mail <at> bernhard-voelker.de>
To: Paul Eggert <eggert <at> cs.ucla.edu>, Pádraig Brady
 <P <at> draigBrady.com>
Cc: Petr Malat <oss <at> malat.biz>, util-linux <util-linux <at> vger.kernel.org>,
 Dominique Martinet <asmadeus <at> codewreck.org>, 69532 <at> debbugs.gnu.org,
 Karel Zak <kzak <at> redhat.com>, Rob Landley <rob <at> landley.net>
Subject: Re: bug#69532: mv's new -x option should be made orthogonal to
 -t/-T/default
Date: Thu, 21 Mar 2024 22:45:24 +0100

On 3/21/24 00:56, Paul Eggert wrote:
> On 3/20/24 15:53, Bernhard Voelker wrote:
> Yes, that's the expected behavior for this contrived case. Just as one
> would get odd behavior if one did the same thing without --exchange.

There's another which is not consistent with/without --exchange:

  $ src/mv -v a a
  src/mv: 'a' and 'a' are the same file

  $ src/mv -v --exchange a a
  renamed 'a' -> 'a'

RENAME_EXCHANGE is allowed (but useless?) for 1 file.

BTW: shouldn't the -v diagnostic better say "exchanged 'a' <-> 'a'"
because that's what happened?

>>     - not sure if exchange works well together with -f.
>
> What problems do you see there?

it's up to the tests to proof that.

>> why does exchange not work to exchange a regular with a
>> directory file?
>
> It works. I don't see a problem there.
>
>     $ touch a
>     $ mkdir d
>     $ ./mv -T --exchange a d
>     $ ls -ld a d
>     drwxr-xr-x. 2 eggert eggert 4096 Mar 20 16:52 a
>     -rw-r--r--. 1 eggert eggert    0 Mar 20 16:52 d

indeed, it works.  It seems my test was wrong, sorry.

>> Finally, the test cases are very sparse:
>
> Feel free to add some. :-)

Unfortunately, I cannot currently spend as much time as I'd love to.

It seems that -i is skipped:

  $ src/mv -iv --exchange a b
  renamed 'a' -> 'b'

As far as I know Padraig would like to do the next release very soon,
so I would recommend to not hurrying this one into it, and instead
ironing all out after the release.

Have a nice day,
Berny

Message #85 received at 69532 <at> debbugs.gnu.org (full text, mbox):

From: Karel Zak <kzak <at> redhat.com>
To: Bernhard Voelker <mail <at> bernhard-voelker.de>
Cc: Petr Malat <oss <at> malat.biz>, util-linux <util-linux <at> vger.kernel.org>,
 Paul Eggert <eggert <at> cs.ucla.edu>, Dominique Martinet <asmadeus <at> codewreck.org>,
 69532 <at> debbugs.gnu.org, Rob Landley <rob <at> landley.net>,
 Pádraig Brady <P <at> draigBrady.com>
Subject: Re: bug#69532: mv's new -x option should be made orthogonal to
 -t/-T/default
Date: Fri, 22 Mar 2024 11:22:57 +0100

On Wed, Mar 20, 2024 at 11:53:05PM +0100, Bernhard Voelker wrote:
> On userland OTOH, we have broader choice.
> Karel did his choice in util-linux for exch(1), and coreutils could expose
> the same functionality.
> 
> For other feature requests, we were much more reluctant in coreutils ... for
> good reasons: feature bloat, maintainability, etc.
> 
> So I'm asking myself what is different this time?
> - The feature already exists -> util-linux.

Note that we can move exch(1) from util-linux to coreutils if, at the
end of the brainstorming session, the conclusion will be that mv(1) is
a bad choice :-)

> I'm currently only 20:80 for adding it to mv(1).

I think the functionality will be lost in the mv(1) for many users.

    Karel

-- 
 Karel Zak  <kzak <at> redhat.com>
 http://karelzak.blogspot.com

Message #88 received at 69532 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Bernhard Voelker <mail <at> bernhard-voelker.de>,
 Pádraig Brady <P <at> draigBrady.com>
Cc: Petr Malat <oss <at> malat.biz>, util-linux <util-linux <at> vger.kernel.org>,
 Dominique Martinet <asmadeus <at> codewreck.org>, 69532 <at> debbugs.gnu.org,
 Karel Zak <kzak <at> redhat.com>, Rob Landley <rob <at> landley.net>
Subject: Re: bug#69532: mv's new -x option should be made orthogonal to
 -t/-T/default
Date: Fri, 22 Mar 2024 18:44:39 -0700

[Message part 1 (text/plain, inline)]

On 3/21/24 14:45, Bernhard Voelker wrote:
> On 3/21/24 00:56, Paul Eggert wrote:
>> On 3/20/24 15:53, Bernhard Voelker wrote:
>> Yes, that's the expected behavior for this contrived case. Just as one
>> would get odd behavior if one did the same thing without --exchange.
> 
> There's another which is not consistent with/without --exchange:
> 
>    $ src/mv -v a a
>    src/mv: 'a' and 'a' are the same file
> 
>    $ src/mv -v --exchange a a
>    renamed 'a' -> 'a'
> 
> RENAME_EXCHANGE is allowed (but useless?) for 1 file.

Yes, thanks, --exchange should act more like non --exchange there.


> BTW: shouldn't the -v diagnostic better say "exchanged 'a' <-> 'a'"
> because that's what happened?

Good suggestion.


> It seems that -i is skipped:
> 
>    $ src/mv -iv --exchange a b
>    renamed 'a' -> 'b'

Yes, I suppose -i should be treated similarly too.

I installed the attached patches to do the above. (Basically, the 
problem was that my earlier patches were too ambitious; these patches 
scale things back to avoid some optimizations so that mv --exchange is 
more like ordinary mv.)

The first patch simplifies the code (and fixes a diagnostic to be more 
useful) without otherwise changing behavior; it's more of a refactoring. 
The second patch does the real work.

Thanks again.

[0001-cp-ln-mv-improve-dir-vs-nondir-diagnostics.patch (text/x-patch, attachment)]

[0002-mv-treat-exchange-more-like-non-exchange.patch (text/x-patch, attachment)]

Message #91 received at 69532 <at> debbugs.gnu.org (full text, mbox):

From: Bernhard Voelker <mail <at> bernhard-voelker.de>
To: Paul Eggert <eggert <at> cs.ucla.edu>, Pádraig Brady
 <P <at> draigBrady.com>
Cc: Petr Malat <oss <at> malat.biz>, util-linux <util-linux <at> vger.kernel.org>,
 Dominique Martinet <asmadeus <at> codewreck.org>, 69532 <at> debbugs.gnu.org,
 Karel Zak <kzak <at> redhat.com>, Rob Landley <rob <at> landley.net>
Subject: Re: bug#69532: mv's new -x option should be made orthogonal to
 -t/-T/default
Date: Sat, 23 Mar 2024 11:24:15 +0100

On 3/23/24 02:44, Paul Eggert wrote:
> I installed the attached patches to do the above. (Basically, the
> problem was that my earlier patches were too ambitious; these patches
> scale things back to avoid some optimizations so that mv --exchange is
> more like ordinary mv.)
>
> The first patch simplifies the code (and fixes a diagnostic to be more
> useful) without otherwise changing behavior; it's more of a refactoring.
> The second patch does the real work.

Thanks.

We should put adding more tests on our TODO list still.

Have a nice day,
Berny

Message #94 received at 69532 <at> debbugs.gnu.org (full text, mbox):

From: Bernhard Voelker <mail <at> bernhard-voelker.de>
To: Karel Zak <kzak <at> redhat.com>
Cc: Petr Malat <oss <at> malat.biz>, util-linux <util-linux <at> vger.kernel.org>,
 Paul Eggert <eggert <at> cs.ucla.edu>, Dominique Martinet <asmadeus <at> codewreck.org>,
 69532 <at> debbugs.gnu.org, Rob Landley <rob <at> landley.net>,
 Pádraig Brady <P <at> draigBrady.com>
Subject: Re: bug#69532: mv's new -x option should be made orthogonal to
 -t/-T/default
Date: Sat, 23 Mar 2024 11:24:18 +0100

On 3/22/24 11:22, Karel Zak wrote:
> On Wed, Mar 20, 2024 at 11:53:05PM +0100, Bernhard Voelker wrote:>> On userland OTOH, we have broader choice.
>> Karel did his choice in util-linux for exch(1), and coreutils could expose
>> the same functionality.
>>
>> For other feature requests, we were much more reluctant in coreutils ... for
>> good reasons: feature bloat, maintainability, etc.
>>
>> So I'm asking myself what is different this time?
>> - The feature already exists -> util-linux.
>
> Note that we can move exch(1) from util-linux to coreutils if, at the
> end of the brainstorming session, the conclusion will be that mv(1) is
> a bad choice :-)

I'd be for that as well, because ...

>> I'm currently only 20:80 for adding it to mv(1).
>
> I think the functionality will be lost in the mv(1) for many users.

... that's a fair point.

The code for the functionality is in copy.c, so - as with mv.c/cp.c/install.c -
we could have a exch.c using just that part, and thus expose a clearer interface
to the users.

Have a nice day,
Berny

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.