GNU bug report logs - #70586
cp walks dir differently than rm and is hitting "File name too long" where this could be avoided

Previous Next

To reply to this bug, email your comments to 70586 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Arkadiusz Miśkiewicz <arekm <at> maven.pl>
To: bug-coreutils <at> gnu.org
Subject: cp walks dir differently than rm and is hitting "File name too long"
 where this could be avoided
Date: Fri, 26 Apr 2024 10:35:40 +0200

Hello.

rm -r while deleting a directory that's longer than PATH_MAX walks it in 
a way to avoid hitting max limit

$ (for i in `seq 1 2000`; do mkdir 
1234567890123456789012345678901234567890; cd 
1234567890123456789012345678901234567890; done)
$ rm -r 1234567890123456789012345678901234567890


but cp doesn't do that:

$ (for i in `seq 1 2000`; do mkdir 
1234567890123456789012345678901234567890; cd 
1234567890123456789012345678901234567890; done)
$ cp -a 1234567890123456789012345678901234567890 2
> cp: cannot stat '1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890/1234567890123456789012345678901234567890': File name too long

I wonder (+ report as a enhancement request) why cp isn't made to do the 
same smart thing and avoid hitting ENAMETOOLONG?

$ cp --version
cp (GNU coreutils) 9.5

-- 
Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org )

Message #8 received at 70586 <at> debbugs.gnu.org (full text, mbox):

From: Collin Funk <collin.funk1 <at> gmail.com>
To: 70586 <at> debbugs.gnu.org
Subject: Re: bug#70586: cp walks dir differently than rm and is hitting
 "File name too long" where this could be avoided
Date: Tue, 14 Oct 2025 22:13:35 -0700

Hi Arkadiusz,

You wrote:

> rm -r while deleting a directory that's longer than PATH_MAX walks it in 
> a way to avoid hitting max limit
> 
> $ (for i in `seq 1 2000`; do mkdir 
> 1234567890123456789012345678901234567890; cd 
> 1234567890123456789012345678901234567890; done)
> $ rm -r 1234567890123456789012345678901234567890
>
> 
> but cp doesn't do that:
> 
> $ (for i in `seq 1 2000`; do mkdir 
> 1234567890123456789012345678901234567890; cd 
> 1234567890123456789012345678901234567890; done)
> $ cp -a 1234567890123456789012345678901234567890 2
> > cp: cannot stat '[long filename snipped]': File name too long

An easier way to test this is (assuming your system supports it):

    $ mkdir -p `python3 -c 'print("./" + "a/" * 32768)'`
    $ cp -r a b
    cp: cannot stat '[long file name snipped]': File name too long
    $ rm -rf a b

> I wonder (+ report as a enhancement request) why cp isn't made to do the 
> same smart thing and avoid hitting ENAMETOOLONG?

It is known about, and has been mentioned in the TODO file for a very
long time:

    cp --recursive: use fts and *at functions to perform directory
    traversals in source and destination hierarchy rather than forming
    full file names. The latter (current) approach fails unnecessarily
    when the names become very long, and requires space and time that is
    quadratic in the depth of the hierarchy.

I suppose very few have run into the limit with real usage.

I agree it should be fixed though. I'll have a look at it.

Thanks,
Collin

Message #11 received at 70586 <at> debbugs.gnu.org (full text, mbox):

From: Collin Funk <collin.funk1 <at> gmail.com>
To: 70586 <at> debbugs.gnu.org
Subject: Re: bug#70586: cp walks dir differently than rm and is hitting
 "File name too long" where this could be avoided
Date: Wed, 15 Oct 2025 09:36:58 -0700

Collin Funk <collin.funk1 <at> gmail.com> writes:

> It is known about, and has been mentioned in the TODO file for a very
> long time:
>
>     cp --recursive: use fts and *at functions to perform directory
>     traversals in source and destination hierarchy rather than forming
>     full file names. The latter (current) approach fails unnecessarily
>     when the names become very long, and requires space and time that is
>     quadratic in the depth of the hierarchy.
>
> I suppose very few have run into the limit with real usage.
>
> I agree it should be fixed though. I'll have a look at it.

Also, 'ls -R' has the same problem.

     $ mkdir -p `python3 -c 'print("./" + "a/" * 32768)'`
     $ ls -R a > /dev/null
     ls: cannot open directory '[long file name]': File name too long

I'll have a look at that one too. Finding a reasonable way to define the
sort functions in terms of 'FTSENT **' will not be very fun though.

Collin

Message #14 received at 70586 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Collin Funk <collin.funk1 <at> gmail.com>
Cc: 70586 <at> debbugs.gnu.org
Subject: Re: bug#70586: cp walks dir differently than rm and is hitting "File
 name too long" where this could be avoided
Date: Wed, 15 Oct 2025 10:41:04 -0700

On 10/15/25 09:36, Collin Funk wrote:
> I'll have a look at that one too. Finding a reasonable way to define the
> sort functions in terms of 'FTSENT **' will not be very fun though.

Should be easier for ls than for cp, as fts isn't really designed for 
destinations so you'll have to traverse the destination by hand.

As you say, this is all reasonably low priority, as only buggy uses and 
attackers create file names longer than PATH_MAX (and rm is the cure for 
that...).

Message #17 received at 70586 <at> debbugs.gnu.org (full text, mbox):

From: Collin Funk <collin.funk1 <at> gmail.com>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: 70586 <at> debbugs.gnu.org
Subject: Re: bug#70586: cp walks dir differently than rm and is hitting
 "File name too long" where this could be avoided
Date: Wed, 15 Oct 2025 19:05:56 -0700

Paul Eggert <eggert <at> cs.ucla.edu> writes:

> On 10/15/25 09:36, Collin Funk wrote:
>> I'll have a look at that one too. Finding a reasonable way to define the
>> sort functions in terms of 'FTSENT **' will not be very fun though.
>
> Should be easier for ls than for cp, as fts isn't really designed for
> destinations so you'll have to traverse the destination by hand.

Good to know, thanks.

> As you say, this is all reasonably low priority, as only buggy uses
> and attackers create file names longer than PATH_MAX (and rm is the
> cure for that...).

True, I don't think many, if any, have run into path length limitations
without explicitly trying to (*).

It still would be nice to fix though. We should be able to create
files/directories if the operating system supports it.

Collin

(*) In recent years at least. I am ignoring things like FAT32 filenames
which were limited to 11 characters or 255 with an extension.

Message #20 received at 70586 <at> debbugs.gnu.org (full text, mbox):

From: Collin Funk <collin.funk1 <at> gmail.com>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: 70586 <at> debbugs.gnu.org
Subject: Re: bug#70586: cp walks dir differently than rm and is hitting
 "File name too long" where this could be avoided
Date: Sun, 19 Oct 2025 19:18:48 -0700

Paul Eggert <eggert <at> cs.ucla.edu> writes:

> On 10/15/25 09:36, Collin Funk wrote:
>> I'll have a look at that one too. Finding a reasonable way to define the
>> sort functions in terms of 'FTSENT **' will not be very fun though.
>
> Should be easier for ls than for cp, as fts isn't really designed for
> destinations so you'll have to traverse the destination by hand.

I thought I had something close to working for 'cp', but realized
another problem with this.

Here is the limit on file descriptors on my Fedora 42 system without
messing with configurations:

     $ ulimit -n
     1024

It is easy to hit this limit even if you only open file descriptors to
the parents of the file that you want to copy.

I'm considering a gnulib module with an implemented version of the
following:

   int
   open_deep (char const *filename, int flags, ...)
   {
     mode_t mode = 0;

     /* Insert code to use va_list to get optional value here.  */

     /* Optimize for the common case of a sane file name length.  */
     int result = open (filename, flags, mode);
     if (0 <= result || errno != ENAMETOOLONG)
       return result;

     /* Loop through the patch components using openat on each
        until we open FILENAME.  */
     return result;
   }

While thinking about that, I'm curious is there a reason why savewd and
mkancesdirs don't use openat + mkdirat instead of using chdir in a child
process? Do these functions just predate the *at interfaces, or is there
another reason that I am missing?

Collin

Message #23 received at 70586 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Collin Funk <collin.funk1 <at> gmail.com>
Cc: 70586 <at> debbugs.gnu.org
Subject: Re: bug#70586: cp walks dir differently than rm and is hitting "File
 name too long" where this could be avoided
Date: Sun, 19 Oct 2025 22:30:21 -0700

On 2025-10-19 19:18, Collin Funk wrote:

>       $ ulimit -n
>       1024
> 
> It is easy to hit this limit even if you only open file descriptors to
> the parents of the file that you want to copy.

(I assume by "parents" you mean "ancestors".)

Yes, and that's why Gnulib fts doesn't do that.


>     int
>     open_deep (char const *filename, int flags, ...)
>     {
...>       /* Loop through the patch components using openat on each
>          until we open FILENAME.  */
>       return result;

That's O(N * D) if you open N files nested D deep, which is not a good 
thing. We should be able to do O(N + D).


> is there a reason why savewd and
> mkancesdirs don't use openat + mkdirat instead of using chdir in a child
> process? Do these functions just predate the *at interfaces

Yes, precisely. Some day it will be time to give up on those old 
platforms, I suppose. We haven't done that yet because there hasn't been 
motivation to get rid of them, or rewrite them.

Message #26 received at 70586 <at> debbugs.gnu.org (full text, mbox):

From: Collin Funk <collin.funk1 <at> gmail.com>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: 70586 <at> debbugs.gnu.org
Subject: Re: bug#70586: cp walks dir differently than rm and is hitting
 "File name too long" where this could be avoided
Date: Mon, 20 Oct 2025 19:27:53 -0700

Paul Eggert <eggert <at> cs.ucla.edu> writes:

>> is there a reason why savewd and
>> mkancesdirs don't use openat + mkdirat instead of using chdir in a child
>> process? Do these functions just predate the *at interfaces
>
> Yes, precisely. Some day it will be time to give up on those old
> platforms, I suppose. We haven't done that yet because there hasn't
> been motivation to get rid of them, or rewrite them.

I see. With this trivial test it seems that chdir is faster than open on
GNU/Linux:

    $ cat main.c 
    #define _GNU_SOURCE 1
    #include <fcntl.h>
    #include <unistd.h>
    #include <stdlib.h>
    int
    main (void)
    {
      for (int i = 0; i < 100000; ++i)
        {
    #ifdef DO_OPEN
          int fd = open ("/", O_DIRECTORY | O_PATH);
          if (fd < 0 || close (fd) < 0)
            abort ();
    #else
          if (chdir ("/") < 0)
            abort ();
    #endif
        }
      return 0;
    }
    $ gcc main.c
    $ perf stat --repeat 10 ./a.out 2>&1 | grep -F 'seconds time'
    $ perf stat --repeat 10 ./a.out 2>&1 | grep -F 'seconds time'
           0.08036 +- 0.00194 seconds time elapsed  ( +-  2.41% )
    $ gcc -DDO_OPEN main.c
    $ perf stat --repeat 10 ./a.out 2>&1 | grep -F 'seconds time'
           0.13929 +- 0.00102 seconds time elapsed  ( +-  0.73% )

Collin

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.