GNU bug report logs - #80106
Don't sort dirents (in du) for Lustre

Previous Next

To reply to this bug, email your comments to 80106 AT debbugs.gnu.org.
There is no need to reopen the bug first.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: "Day, Timothy" <timday <at> amazon.com>
To: "bug-coreutils <at> gnu.org" <bug-coreutils <at> gnu.org>
Subject: Don't sort dirents (in du) for Lustre
Date: Wed, 31 Dec 2025 19:12:53 +0000

The gnulib function dirent_inode_sort_may_be_useful() should return
false for Lustre (i.e. #define S_MAGIC_LUSTRE 0x0BD00BD0 as seen
in lustre/include/uapi/linux/lustre/lustre_user.h in the Lustre source
tree as LL_SUPER_MAGIC [1]). Sorting dirents negatively impacts du
performance on Lustre because it interferes with Lustre's ability to
prefetch file metadata (via statahead).

For context, Lustre is an open-source (GPLv2) out-of-tree Linux filesystem
commonly used for HPC applications.

Let me know if you need anymore context. Thanks!

Tim Day

[1] https://git.whamcloud.com/?p=fs/lustre-release.git;a=blob;f=lustre/include/uapi/linux/lustre/lustre_user.h

Message #8 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Collin Funk <collin.funk1 <at> gmail.com>
To: "Day, Timothy" via GNU coreutils Bug Reports <bug-coreutils <at> gnu.org>
Cc: 80106 <at> debbugs.gnu.org, "Day, Timothy" <timday <at> amazon.com>
Subject: Re: bug#80106: Don't sort dirents (in du) for Lustre
Date: Wed, 31 Dec 2025 21:16:21 -0800

"Day, Timothy" via GNU coreutils Bug Reports <bug-coreutils <at> gnu.org>
writes:

> The gnulib function dirent_inode_sort_may_be_useful() should return
> false for Lustre (i.e. #define S_MAGIC_LUSTRE 0x0BD00BD0 as seen
> in lustre/include/uapi/linux/lustre/lustre_user.h in the Lustre source
> tree as LL_SUPER_MAGIC [1]). Sorting dirents negatively impacts du
> performance on Lustre because it interferes with Lustre's ability to
> prefetch file metadata (via statahead).

Thanks for the investigation.

I think the inode sorting is done any time there is no sort function
given to fts_open and there are 10000 or more entries. So it probably
affects these other programs as well:

     $ grep 'fts_open (.*, nullptr);' src/*.c
     src/chcon.c:  FTS *fts = xfts_open (files, bit_flags, nullptr);
     src/chmod.c:  FTS *fts = xfts_open (files, bit_flags, nullptr);
     src/chown-core.c:  FTS *fts = xfts_open (files, bit_flags | stat_flags, nullptr);
     src/du.c:      FTS *fts = xfts_open (files, bit_flags, nullptr);
     src/remove.c:      FTS *fts = xfts_open (file, bit_flags, nullptr);
     src/selinux.c:  FTS *fts = xfts_open ((char *const *) ftspath, FTS_PHYSICAL, nullptr);

> For context, Lustre is an open-source (GPLv2) out-of-tree Linux filesystem
> commonly used for HPC applications.
>
> Let me know if you need anymore context. Thanks!

Could you check coreutils built with this patch and see if it fixes the
performance? I can push it to Gnulib afterwards:

diff --git a/lib/fts.c b/lib/fts.c
index e7e0be5a98..a65e7ab771 100644
--- a/lib/fts.c
+++ b/lib/fts.c
@@ -652,6 +652,7 @@ enum leaf_optimization
 /* Linux-specific constants from coreutils' src/fs.h */
 # define S_MAGIC_AFS 0x5346414F
 # define S_MAGIC_CIFS 0xFF534D42
+# define S_MAGIC_LUSTRE 0x0BD00BD0
 # define S_MAGIC_NFS 0x6969
 # define S_MAGIC_PROC 0x9FA0
 # define S_MAGIC_TMPFS 0x1021994
@@ -760,6 +761,7 @@ dirent_inode_sort_may_be_useful (FTSENT const *p, int dir_fd)
     {
     case S_MAGIC_CIFS:
     case S_MAGIC_NFS:
+    case S_MAGIC_LUSTRE:
     case S_MAGIC_TMPFS:
       /* On a file system of any of these types, sorting
          is unnecessary, and hence wasteful.  */

Message #14 received at 80106 <at> debbugs.gnu.org (full text, mbox):

From: "Day, Timothy" <timday <at> amazon.com>
To: Collin Funk <collin.funk1 <at> gmail.com>, "Day, Timothy via GNU coreutils Bug
 Reports" <bug-coreutils <at> gnu.org>
Cc: "80106 <at> debbugs.gnu.org" <80106 <at> debbugs.gnu.org>
Subject: Re: bug#80106: Don't sort dirents (in du) for Lustre
Date: Thu, 1 Jan 2026 23:15:45 +0000

>> For context, Lustre is an open-source (GPLv2) out-of-tree Linux filesystem
>> commonly used for HPC applications.
>>
>> Let me know if you need anymore context. Thanks!
>
>Could you check coreutils built with this patch and see if it fixes the
>performance? I can push it to Gnulib afterwards:

Works perfectly:

$ echo 3 | sudo tee /proc/sys/vm/drop_caches 
3
$ time coreutils-9.9-unpatched/src/du /fsx/test_100K > /dev/null

real	1m54.268s
user	0m0.229s
sys	0m14.136s
$ echo 3 | sudo tee /proc/sys/vm/drop_caches 
3
$ time coreutils-9.9-patched/src/du /fsx/test_100K > /dev/null

real	0m13.150s
user	0m0.075s
sys	0m1.956s

That's around a 9x speedup. I haven't tested any other commands.
Lustre statahead (which is giving the performance boost) is designed
for "stat everything in a directory" type workloads. The other commands
wouldn't do that, I think. Perhaps chown would? Either way, thanks for
the help!

Tim Day

Message #22 received at 80106-done <at> debbugs.gnu.org (full text, mbox):

From: Collin Funk <collin.funk1 <at> gmail.com>
To: "Day, Timothy" <timday <at> amazon.com>
Cc: 80106-done <at> debbugs.gnu.org,
 Pádraig Brady <P <at> draigBrady.com>
Subject: Re: bug#80106: Don't sort dirents (in du) for Lustre
Date: Thu, 01 Jan 2026 21:07:06 -0800

[Message part 1 (text/plain, inline)]

"Day, Timothy" <timday <at> amazon.com> writes:

>>> For context, Lustre is an open-source (GPLv2) out-of-tree Linux filesystem
>>> commonly used for HPC applications.
>>>
>>> Let me know if you need anymore context. Thanks!
>>
>>Could you check coreutils built with this patch and see if it fixes the
>>performance? I can push it to Gnulib afterwards:
>
> Works perfectly:
>
> $ echo 3 | sudo tee /proc/sys/vm/drop_caches 
> 3
> $ time coreutils-9.9-unpatched/src/du /fsx/test_100K > /dev/null
>
> real	1m54.268s
> user	0m0.229s
> sys	0m14.136s
> $ echo 3 | sudo tee /proc/sys/vm/drop_caches 
> 3
> $ time coreutils-9.9-patched/src/du /fsx/test_100K > /dev/null
>
> real	0m13.150s
> user	0m0.075s
> sys	0m1.956s
>
> That's around a 9x speedup. I haven't tested any other commands.
> Lustre statahead (which is giving the performance boost) is designed
> for "stat everything in a directory" type workloads. The other commands
> wouldn't do that, I think. Perhaps chown would? Either way, thanks for
> the help!

Great, thanks for your help investigating!

I think 'chown -R' and 'rm -r' would probably be affected as well:

    $ mkdir -p dir && (cd dir && mkdir $(seq 10000))
    $ strace chown -R collin:collin dir 2>&1 > /dev/null | grep -Fc stat
    20022
    $ strace rm -r dir 2>&1 > /dev/null | grep -Fc stat
    40011

Anyways, I pushed the Gnulib patch and updated the submodule in
coreutils with the attached patch.

Pádraig, I assume this warrants a NEWS entry under "Bug Fixes"?

Collin

[0001-build-update-gnulib-submodule-to-latest.patch (text/x-patch, attachment)]

Message #25 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: bug-coreutils <at> gnu.org
Subject: Re: bug#80106: Don't sort dirents (in du) for Lustre
Date: Thu, 1 Jan 2026 21:29:27 -0800

On 2026-01-01 21:07, Collin Funk wrote:
> Pádraig, I assume this warrants a NEWS entry under "Bug Fixes"?

It's just a performance improvement, no? And not such a big improvement 
that it warrants being called aa "bug fix".

Message #28 received at 80106 <at> debbugs.gnu.org (full text, mbox):

From: Collin Funk <collin.funk1 <at> gmail.com>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: 80106 <at> debbugs.gnu.org, "Day, Timothy" <timday <at> amazon.com>
Subject: Re: bug#80106: Don't sort dirents (in du) for Lustre
Date: Fri, 02 Jan 2026 19:47:30 -0800

[Message part 1 (text/plain, inline)]

Paul Eggert <eggert <at> cs.ucla.edu> writes:

> On 2026-01-01 21:07, Collin Funk wrote:
>> Pádraig, I assume this warrants a NEWS entry under "Bug Fixes"?
>
> It's just a performance improvement, no? And not such a big
> improvement that it warrants being called aa "bug fix".

I was conflicted on that since it might be considered a bug in fts. I
guess it worked fine though, just much slower.

I pushed the attached patch to mention the improvement in NEWS.

By the way Tim, the fts_* functions are a part of glibc as well.
However, the inode sorting code is not contained there. I assume it was
a gnulib invention among some other extensions we have, e.g., using
openat() instead of chdir(). So no need for any changes there. I forgot
to mention that in my previous mail.

Collin

[0001-doc-NEWS-mention-du-performance-improvement-on-the-L.patch (text/x-patch, attachment)]

Message #31 received at 80106 <at> debbugs.gnu.org (full text, mbox):

From: "Day, Timothy" <timday <at> amazon.com>
To: Collin Funk <collin.funk1 <at> gmail.com>, Paul Eggert <eggert <at> cs.ucla.edu>
Cc: "80106 <at> debbugs.gnu.org" <80106 <at> debbugs.gnu.org>
Subject: Re: bug#80106: Don't sort dirents (in du) for Lustre
Date: Wed, 7 Jan 2026 22:32:22 +0000

>>> Pádraig, I assume this warrants a NEWS entry under "Bug Fixes"?
>>
>> It's just a performance improvement, no? And not such a big
>> improvement that it warrants being called aa "bug fix".
>
>I was conflicted on that since it might be considered a bug in fts. I
>guess it worked fine though, just much slower.
>
>I pushed the attached patch to mention the improvement in NEWS.

It might be considered a performance regression. Although I haven't tested
old versions of coreutils to be sure or looked to see where this
sorting was originally introduced.

>By the way Tim, the fts_* functions are a part of glibc as well.
>However, the inode sorting code is not contained there. I assume it was
>a gnulib invention among some other extensions we have, e.g., using
>openat() instead of chdir(). So no need for any changes there. I forgot
>to mention that in my previous mail.

Thanks for the heads up and thanks for looking into this.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.