GNU logs - #15806, boring messages

Message sent to bug-coreutils@HIDDEN:

Subject: bug#15806: [cp] -R modifies dereferencing settings
Resent-From: Gian Piero Carrubba <gpiero@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-coreutils@HIDDEN
Resent-Date: Mon, 04 Nov 2013 23:38:02 +0000
Resent-Message-ID: <handler.15806.B.138360825717738 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: 15806 <at> debbugs.gnu.org
Date: Tue, 5 Nov 2013 00:37:06 +0100
From: Gian Piero Carrubba <gpiero@HIDDEN>
Message-ID: <20131104233706.GA5665@HIDDEN>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="45Z9DzgjV8m4Oswq"
Content-Disposition: inline
User-Agent: Mutt/1.5.21 (2010-09-15)
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

--45Z9DzgjV8m4Oswq
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Disposition: inline

Severity: wishlist

Please Cc: me as I'm not subscribed.

POSIX.1-2008 (ed. 2013) [0] states:

     If source_file is a file of type symbolic link:

     If the -R option was not specified, cp shall take actions based on
     the type and contents of the file referenced by the symbolic link,
     and not by the symbolic link itself, unless the -P option was
     specified.

So far, so good. As far as I know, coreutils' cp fully respect this.

     If the -R option was specified:

     If none of the options -H, -L, nor -P were specified, it is
     unspecified which of -H, -L, or -P will be used as a default.

Somewhere in the past, coreutils team has opted for using '-P' 
(DEREF_NEVER) in this case.  Anyway this leads to an inconsistence in 
the behaviour of cp:

     $ touch file ; ln -s file symlink ; for o in "" -r ; do cp $o
     symlink cp$o ; done ; ls -liog
     total 0
     8224776 -rw-r--r-- 1 0 Nov  4 23:15 cp
     8224777 lrwxrwxrwx 1 4 Nov  4 23:15 cp-r -> file
     8224774 -rw-r--r-- 1 0 Nov  4 23:15 file
     8224775 lrwxrwxrwx 1 4 Nov  4 23:15 symlink -> file

What wrong with it ? The problem is that we got a modified dereference 
setting just by using '--recursive', an unrelated option. Moreover, 
recursion and dereferencing are orthogonal features and there's no 
reason one should imply the other.

As for my opinion, there are three possibilities for, at least, reduce 
this perceived inconsistency. 

1. leave `cp -R` acting as it currently does...

...but properly document it. The texinfo doc already cites this 
behaviour, but both the man page and the help (imo, the two most used  
sources while searching for an option) don't. I would suggest adding 
"Implies '--no-dereference'" to the description of the -r/-R option.

2. let `cp -R` act as if `-L` is used.

This is my preferred option, but it's also the most invasive. All the 
scripts using `cp -r` for copying a directory tree without dereferencing 
the symbolic links would fail. The texinfo doc warns against that use 
[1]:

     -- doc/coreutils.texi:8334
     [...] It is not portable to use
     @option{-r} to copy symbolic links or special files.  On some
     non-GNU systems, @option{-r} implies the equivalent of
     @option{-L} and @option{--copy-contents} for historical reasons.
     Also, it is not portable to use @option{-R} to copy symbolic links
     unless you also specify @option{-P}, as POSIX allows
     implementations that dereference symbolic links by default.

so maybe this option could be considered. But honestly, despite being my 
preferred option (i.e.: the one I would implement if I had to write a 
'cp' utility from scratch), I'm not sure I can endorse it as it would be 
highly backward incompatible.

3. let `cp -R` act as if `-H` is used.

This is a sort of compromise between the other two options. It gains 
some consistency ( `cp` and `cp -R` would act the same with regard to 
dereferencing the symlinks passed on the command line ) and would 
preserve a certain level of backward compatibility. I expect the 
percentage of real-case scripts that would be broken by this change to 
be really minimal.
I don't particularly like this option, but if option 2. is judged 
impractical (as it probably is), I personally largely prefer this 
solution over option 1.

The attached preliminary patch implements option 3.

After applying the patch the following tests fail:

= tests/cp/r-vs-symlink.sh
This is expected, as it is the point of the patch. Anyway the comments 
in the test file are suspicious:

     # cp -r should not create symlinks.  Fixed in fileutils-4.1.5.
     # Restored old behavior (whereby cp -r preserves symlinks) in 4.1.6,
     # though now such usage evokes a warning:
     # cp: 'slink': WARNING: using -r to copy symbolic links is not
     # portable

So maybe a similar change (probably option 2. above) was implemented, 
and reverted, in the past ? I wasn't able to find a reference to it in 
the Changelog, so cannot tell why it has been reverted. Anyway the 
warning cited in the comment doesn't seem to exist anymore.

= tests/cp/special-f.sh [2]
This is unexpected. At a first look, the failure should be caused by the 
following test not being fully inclusive:

     -- src/copy.c:1930
     else if (! S_ISDIR (dst_sb.st_mode)
         /* Never unlink dst_name when in move mode.  */
         && ! x->move_mode
         && (x->unlink_dest_before_opening
             || (x->preserve_links && 1 < dst_sb.st_nlink)
             || (x->dereference == DEREF_NEVER
                 && !  S_ISREG (src_sb.st_mode))
             ))

I hadn't the time for investigate further, will do in the next days. At 
the moment, however, I suspect the previous dereference settings was 
simply masking and underlying issue.

Ciao,
Gian Piero.

[0] http://pubs.opengroup.org/onlinepubs/9699919799/utilities/cp.html

[1] Anyway I'm not sure why it differentiates between '-r' and '-R', 
given that in coreutils' cp they are aliases:

     -- src/copy.c:1045
     case 'r':
     case 'R':
       x.recursive = true;
       break;

[2] A side note about this test. The comment seems to contradict the 
code:

     -- tests/cp/special-f.sh:28
     # Without -f, expect it to fail
     cp -R fifo e || fail=1

As for my impression, it also contradicts POSIX specifications (w/o -f 
it should fail). Anyway this belongs to another bug report.

--45Z9DzgjV8m4Oswq
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment; filename="draft.patch"

Tue Nov  5 00:35:28 CET 2013  Gian Piero Carrubba <gpiero@HIDDEN>
  * DRAFT
diff -rN -u old-dereference-R/src/copy.h new-dereference-R/src/copy.h
--- old-dereference-R/src/copy.h	2013-11-05 00:35:44.062894654 +0100
+++ new-dereference-R/src/copy.h	2013-11-05 00:35:44.066894686 +0100
@@ -68,10 +68,8 @@
 /* How to handle symbolic links.  */
 enum Dereference_symlink
 {
-  DEREF_UNDEFINED = 1,
-
   /* Copy the symbolic link itself.  -P  */
-  DEREF_NEVER,
+  DEREF_NEVER = 1,

   /* If the symbolic is a command line argument, then copy
      its referent.  Otherwise, copy the symbolic link itself.  -H  */
diff -rN -u old-dereference-R/src/cp.c new-dereference-R/src/cp.c
--- old-dereference-R/src/cp.c	2013-11-05 00:35:44.062894654 +0100
+++ new-dereference-R/src/cp.c	2013-11-05 00:35:44.066894686 +0100
@@ -768,7 +768,7 @@
 {
   cp_options_default (x);
   x->copy_as_regular = true;
-  x->dereference = DEREF_UNDEFINED;
+  x->dereference = DEREF_COMMAND_LINE_ARGUMENTS;
   x->unlink_dest_before_opening = false;
   x->unlink_dest_after_failed_open = false;
   x->hard_link = false;
@@ -1133,15 +1133,6 @@
                                    version_control_string)
                    : no_backups);

-  if (x.dereference == DEREF_UNDEFINED)
-    {
-      if (x.recursive)
-        /* This is compatible with FreeBSD.  */
-        x.dereference = DEREF_NEVER;
-      else
-        x.dereference = DEREF_ALWAYS;
-    }
-
   if (x.recursive)
     x.copy_as_regular = copy_contents;

--45Z9DzgjV8m4Oswq--

Message sent:

Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Mailer: MIME-tools 5.503 (Entity 5.503)
Content-Type: text/plain; charset=utf-8
X-Loop: help-debbugs@HIDDEN
From: help-debbugs@HIDDEN (GNU bug Tracking System)
To: Gian Piero Carrubba <gpiero@HIDDEN>
Subject: bug#15806: Acknowledgement ([cp] -R modifies dereferencing settings)
Message-ID: <handler.15806.B.138360825717738.ack <at> debbugs.gnu.org>
References: <20131104233706.GA5665@HIDDEN>
X-Gnu-PR-Message: ack 15806
X-Gnu-PR-Package: coreutils
Reply-To: 15806 <at> debbugs.gnu.org
Date: Mon, 04 Nov 2013 23:38:03 +0000

Thank you for filing a new bug report with debbugs.gnu.org.

This is an automatically generated reply to let you know your message
has been received.

Your message is being forwarded to the package maintainers and other
interested parties for their attention; they will reply in due course.

Your message has been sent to the package maintainer(s):
 bug-coreutils@HIDDEN

If you wish to submit further information on this problem, please
send it to 15806 <at> debbugs.gnu.org.

Please do not send mail to help-debbugs@HIDDEN unless you wish
to report a problem with the Bug-tracking system.

--=20
15806: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D15806
GNU Bug Tracking System
Contact help-debbugs@HIDDEN with problems

Message sent to bug-coreutils@HIDDEN:

Subject: bug#15806: [cp] -R modifies dereferencing settings
Resent-From: Jim Meyering <jim@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-coreutils@HIDDEN
Resent-Date: Thu, 07 Nov 2013 17:29:02 +0000
Resent-Message-ID: <handler.15806.B15806.138384529110419 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: Gian Piero Carrubba <gpiero@HIDDEN>
Cc: 15806 <at> debbugs.gnu.org
MIME-Version: 1.0
In-Reply-To: <20131104233706.GA5665@HIDDEN>
References: <20131104233706.GA5665@HIDDEN>
From: Jim Meyering <jim@HIDDEN>
Date: Thu, 7 Nov 2013 09:27:41 -0800
Message-ID: <CA+8g5KHZjkuzEC0nqfBxmeWoTV=vwpqyZa59WRgaU4R7Wuhnxg@HIDDEN>
Content-Type: text/plain; charset=ISO-8859-1
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

On Mon, Nov 4, 2013 at 3:37 PM, Gian Piero Carrubba <gpiero@HIDDEN> wrote:
> Severity: wishlist
>
> Please Cc: me as I'm not subscribed.
>
> POSIX.1-2008 (ed. 2013) [0] states:
>
>     If source_file is a file of type symbolic link:
>
>     If the -R option was not specified, cp shall take actions based on
>     the type and contents of the file referenced by the symbolic link,
>     and not by the symbolic link itself, unless the -P option was
>     specified.
>
> So far, so good. As far as I know, coreutils' cp fully respect this.
>
>     If the -R option was specified:
>
>     If none of the options -H, -L, nor -P were specified, it is
>     unspecified which of -H, -L, or -P will be used as a default.
>
> Somewhere in the past, coreutils team has opted for using '-P' (DEREF_NEVER)
> in this case.  Anyway this leads to an inconsistence in the behaviour of cp:
>
>     $ touch file ; ln -s file symlink ; for o in "" -r ; do cp $o
>     symlink cp$o ; done ; ls -liog
>     total 0
>     8224776 -rw-r--r-- 1 0 Nov  4 23:15 cp
>     8224777 lrwxrwxrwx 1 4 Nov  4 23:15 cp-r -> file
>     8224774 -rw-r--r-- 1 0 Nov  4 23:15 file
>     8224775 lrwxrwxrwx 1 4 Nov  4 23:15 symlink -> file
>
> What wrong with it ? The problem is that we got a modified dereference
> setting just by using '--recursive', an unrelated option. Moreover,
> recursion and dereferencing are orthogonal features and there's no reason
> one should imply the other.
>
> As for my opinion, there are three possibilities for, at least, reduce this
> perceived inconsistency.
> 1. leave `cp -R` acting as it currently does...
>
> ...but properly document it. The texinfo doc already cites this behaviour,
> but both the man page and the help (imo, the two most used  sources while
> searching for an option) don't. I would suggest adding "Implies
> '--no-dereference'" to the description of the -r/-R option.

Thanks for your report and the patch.

My first reaction is that it would take a strong argument to justify
changing the existing semantics. Have you compared the semantics
of any other cp implementations?

Message sent to bug-coreutils@HIDDEN:

Subject: bug#15806: [cp] -R modifies dereferencing settings
References: <20131104233706.GA5665@HIDDEN>
Resent-From: Gian Piero Carrubba <gpiero@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-coreutils@HIDDEN
Resent-Date: Fri, 08 Nov 2013 16:46:01 +0000
Resent-Message-ID: <handler.15806.B15806.138392915225974 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: Jim Meyering <jim@HIDDEN>
Cc: 15806 <at> debbugs.gnu.org
Date: Fri, 8 Nov 2013 17:45:40 +0100
From: Gian Piero Carrubba <gpiero@HIDDEN>
Message-ID: <20131108164540.GC2303@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Disposition: inline
In-Reply-To: <CA+8g5KHZjkuzEC0nqfBxmeWoTV=vwpqyZa59WRgaU4R7Wuhnxg@HIDDEN>
 <20131104233706.GA5665@HIDDEN>
User-Agent: Mutt/1.5.21 (2010-09-15)
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

* [Tue, Nov 05, 2013 at 12:37:06AM +0100] Gian Piero Carrubba:
>1. leave `cp -R` acting as it currently does...
>
>...but properly document it. The texinfo doc already cites this 
>behaviour, but both the man page and the help (imo, the two most used  
>sources while searching for an option) don't. I would suggest adding 
>"Implies '--no-dereference'" to the description of the -r/-R option.

...except that it's not exactly true. If '-r' implied '-P', `cp -rL` 
would dereference and `cp -Lr` would not. Actually they both 
dereference, so a better wording is needed. Something along the line: 
"By default, does not dereference symbolic links in the source".

>= tests/cp/special-f.sh [2]
>This is unexpected. At a first look, the failure should be caused by 
>the following test not being fully inclusive:
>
>    -- src/copy.c:1930
>    else if (! S_ISDIR (dst_sb.st_mode)
>        /* Never unlink dst_name when in move mode.  */
>        && ! x->move_mode
>        && (x->unlink_dest_before_opening
>            || (x->preserve_links && 1 < dst_sb.st_nlink)
>            || (x->dereference == DEREF_NEVER
>                && !  S_ISREG (src_sb.st_mode))
>            ))
>
>I hadn't the time for investigate further, will do in the next days. 
>At the moment, however, I suspect the previous dereference settings was 
>simply masking and underlying issue.

Will diverge to another report, as imho it is a bug by itself.

>[2] A side note about this test. The comment seems to contradict the 
>code:
>
>    -- tests/cp/special-f.sh:28
>    # Without -f, expect it to fail
>    cp -R fifo e || fail=1
>
>As for my impression, it also contradicts POSIX specifications (w/o 
>-f it should fail). Anyway this belongs to another bug report.

mmh, no... a better reading of the specs was enough to understand that 
it should NOT fail, so the test is right (but the comment is 
misleading).

* [Thu, Nov 07, 2013 at 09:27:41AM -0800] Jim Meyering:
>My first reaction is that it would take a strong argument to justify
>changing the existing semantics. Have you compared the semantics
>of any other cp implementations?

I have to admit I have no strong argument, and I suspect I cannot obtain 
one just by comparing different implementations, given that this 
behaviour is "implementation dependant" (so different implementations 
differ in this regard).

Just for reference, both cp implementations on Solaris 10 dereference:

$ echo TEST > file ; ln -s file link
$ for o in "" -R -r -PR -Pr -RP -rP ; do /usr/bin/cp $o link cp$o ; 
/usr/xpg4/bin/cp $o link cp-xpg4$o ; done
$ ls -liog
      678491 -rw-r--r--   1       5 Nov  8 16:55 cp
      678499 lrwxrwxrwx   1       4 Nov  8 16:55 cp-Pr -> file
      678497 lrwxrwxrwx   1       4 Nov  8 16:55 cp-PR -> file
      678495 -rw-r--r--   1       5 Nov  8 16:55 cp-r
      678493 -rw-r--r--   1       5 Nov  8 16:55 cp-R
      678503 lrwxrwxrwx   1       4 Nov  8 16:55 cp-rP -> file
      678501 lrwxrwxrwx   1       4 Nov  8 16:55 cp-RP -> file
      678492 -rw-r--r--   1       5 Nov  8 16:55 cp-xpg4
      678500 lrwxrwxrwx   1       4 Nov  8 16:55 cp-xpg4-Pr -> file
      678498 lrwxrwxrwx   1       4 Nov  8 16:55 cp-xpg4-PR -> file
      678496 -rw-r--r--   1       5 Nov  8 16:55 cp-xpg4-r
      678494 -rw-r--r--   1       5 Nov  8 16:55 cp-xpg4-R
      678504 lrwxrwxrwx   1       4 Nov  8 16:55 cp-xpg4-rP -> file
      678502 lrwxrwxrwx   1       4 Nov  8 16:55 cp-xpg4-RP -> file
      678489 -rw-r--r--   1       5 Nov  8 16:54 file
      678490 lrwxrwxrwx   1       4 Nov  8 16:54 link -> file

The same is true for AIX 6.1:

$ for o in "" -R -r -P -PR -Pr -RP -rP ; do cp $o link cp$o ; done
$ ls -liog
      32831 -rw-r--r--    1          5 Nov 08 16:53 cp
      32834 lrwxrwxrwx    1          4 Nov 08 16:53 cp-P
      32835 lrwxrwxrwx    1          4 Nov 08 16:53 cp-PR
      32836 lrwxrwxrwx    1          4 Nov 08 16:53 cp-Pr
      32832 -rw-r--r--    1          5 Nov 08 16:53 cp-R
      32837 lrwxrwxrwx    1          4 Nov 08 16:53 cp-RP
      32833 -rw-r--r--    1          5 Nov 08 16:53 cp-r
      32838 lrwxrwxrwx    1          4 Nov 08 16:53 cp-rP
      32829 -rw-r--r--    1          5 Nov 08 16:50 file
      32830 lrwxrwxrwx    1          4 Nov 08 16:50 link

On the other hand, if I recall correctly, the FreeBSD implementation 
acts like the coreutils one (doesn't dereference). Don't know about 
other implementations.

Ciao,
Gian Piero.

Last modified: Mon, 25 Nov 2019 12:00:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.