GNU bug report logs -
#7355
coreutils df on OSX reports wrong sizes for large filesystems
Previous Next
Reported by: Tim Spriggs <tims <at> uahirise.org>
Date: Mon, 8 Nov 2010 21:37:01 UTC
Severity: normal
Done: Pádraig Brady <P <at> draigBrady.com>
Bug is archived. No further changes may be made.
To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 7355 in the body.
You can then email your comments to 7355 AT debbugs.gnu.org in the normal way.
Toggle the display of automated, internal messages from the tracker.
Report forwarded
to
owner <at> debbugs.gnu.org, bug-coreutils <at> gnu.org:
bug#7355; Package
coreutils.
(Mon, 08 Nov 2010 21:37:01 GMT)
Full text and
rfc822 format available.
Acknowledgement sent
to
Tim Spriggs <tims <at> uahirise.org>:
New bug report received and forwarded. Copy sent to
bug-coreutils <at> gnu.org.
(Mon, 08 Nov 2010 21:37:02 GMT)
Full text and
rfc822 format available.
Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):
Hello,
I am running gentoo-prefix on OSX which builds coreutils. When I
compare the output of the system df to the standard GNU df I get:
$ /bin/df -h /data/hirise06
Filesystem Size Used Avail Capacity Mounted on
hiserve2:/vol/hirise06 7.9Ti 430Gi 7.5Ti 6% /data/hirise06
$ df -h /data/hirise06
Filesystem Size Used Avail Use% Mounted on
hiserve2:/vol/hirise06
2.0T 431G -523G - /data/hirise06
The 2.0T size looks suspicious to me. Maybe an integer is overflowing somewhere?
I'll also note that this was reported some time ago to the macports
bug tracking software (https://trac.macports.org/ticket/16480) but
their resolution was requesting an upstream fix.
Thanks,
-Tim
Information forwarded
to
owner <at> debbugs.gnu.org, bug-coreutils <at> gnu.org:
bug#7355; Package
coreutils.
(Mon, 08 Nov 2010 23:10:02 GMT)
Full text and
rfc822 format available.
Message #8 received at 7355 <at> debbugs.gnu.org (full text, mbox):
On 08/11/10 20:37, Tim Spriggs wrote:
> Hello,
>
> I am running gentoo-prefix on OSX which builds coreutils. When I
> compare the output of the system df to the standard GNU df I get:
>
>
> $ /bin/df -h /data/hirise06
> Filesystem Size Used Avail Capacity Mounted on
> hiserve2:/vol/hirise06 7.9Ti 430Gi 7.5Ti 6% /data/hirise06
>
> $ df -h /data/hirise06
> Filesystem Size Used Avail Use% Mounted on
> hiserve2:/vol/hirise06
> 2.0T 431G -523G - /data/hirise06
>
> The 2.0T size looks suspicious to me. Maybe an integer is overflowing somewhere?
Well 2T is between 32 bit and 64 bit.
If I had to guess I'd say df was using a 32 bit statfs
that was returning 1024 blksize?
Could you compare the system calls made by both versions of df.
On GNU/Linux I can trace the statfs calls like:
strace -v -e statfs,statfs64 df /data/hirise06
If the trace for the above gives no output
for either command, then can you include the
full trace output.
thanks,
Pádraig.
Information forwarded
to
owner <at> debbugs.gnu.org, bug-coreutils <at> gnu.org:
bug#7355; Package
coreutils.
(Mon, 08 Nov 2010 23:26:02 GMT)
Full text and
rfc822 format available.
Message #11 received at 7355 <at> debbugs.gnu.org (full text, mbox):
On 08/11/10 23:13, Pádraig Brady wrote:
> On 08/11/10 20:37, Tim Spriggs wrote:
>> Hello,
>>
>> I am running gentoo-prefix on OSX which builds coreutils. When I
>> compare the output of the system df to the standard GNU df I get:
>>
>>
>> $ /bin/df -h /data/hirise06
>> Filesystem Size Used Avail Capacity Mounted on
>> hiserve2:/vol/hirise06 7.9Ti 430Gi 7.5Ti 6% /data/hirise06
>>
>> $ df -h /data/hirise06
>> Filesystem Size Used Avail Use% Mounted on
>> hiserve2:/vol/hirise06
>> 2.0T 431G -523G - /data/hirise06
>>
>> The 2.0T size looks suspicious to me. Maybe an integer is overflowing somewhere?
>
> Well 2T is between 32 bit and 64 bit.
> If I had to guess I'd say df was using a 32 bit statfs
> that was returning 1024 blksize?
>
> Could you compare the system calls made by both versions of df.
> On GNU/Linux I can trace the statfs calls like:
>
> strace -v -e statfs,statfs64 df /data/hirise06
>
> If the trace for the above gives no output
> for either command, then can you include the
> full trace output.
Also could you give your exact version of OSX.
I had a quick google for statfs64 and OSX and
it seems that _DARWIN_USE_64_BIT_INODE needs to
be defined (and your OSX version to support symbol variants).
Could you try adding that as the first line in df.c? I.E.
#define _DARWIN_USE_64_BIT_INODE 1
If that's the case, then it seems like the AC_SYS_LARGEFILE
would need to be updated in autoconf to handle this.
cheers,
Pádraig.
Information forwarded
to
owner <at> debbugs.gnu.org, bug-coreutils <at> gnu.org:
bug#7355; Package
coreutils.
(Mon, 08 Nov 2010 23:35:01 GMT)
Full text and
rfc822 format available.
Message #14 received at 7355 <at> debbugs.gnu.org (full text, mbox):
2010/11/8 Pádraig Brady <P <at> draigbrady.com>:
> On 08/11/10 20:37, Tim Spriggs wrote:
>> Hello,
>>
>> I am running gentoo-prefix on OSX which builds coreutils. When I
>> compare the output of the system df to the standard GNU df I get:
>>
>>
>> $ /bin/df -h /data/hirise06
>> Filesystem Size Used Avail Capacity Mounted on
>> hiserve2:/vol/hirise06 7.9Ti 430Gi 7.5Ti 6% /data/hirise06
>>
>> $ df -h /data/hirise06
>> Filesystem Size Used Avail Use% Mounted on
>> hiserve2:/vol/hirise06
>> 2.0T 431G -523G - /data/hirise06
>>
>> The 2.0T size looks suspicious to me. Maybe an integer is overflowing somewhere?
>
> Well 2T is between 32 bit and 64 bit.
> If I had to guess I'd say df was using a 32 bit statfs
> that was returning 1024 blksize?
>
> Could you compare the system calls made by both versions of df.
> On GNU/Linux I can trace the statfs calls like:
>
> strace -v -e statfs,statfs64 df /data/hirise06
>
> If the trace for the above gives no output
> for either command, then can you include the
> full trace output.
>
> thanks,
> Pádraig.
>
I ran:
# dtruss -c -f /opt/prefix/bin/df /data/hirise06
CALL COUNT
exit 1
fcntl_nocancel 1
fstatfs64 1
getgid 1
gethostuuid 1
getrlimit 1
kqueue 1
shared_region_check_np 1
shm_open 1
statfs64 1
workq_open 1
bsdthread_register 2
fcntl 2
getdirentries64 2
getfsstat64 2
getpid 2
ioctl 2
lseek 2
madvise 2
thread_selfid 3
write_nocancel 3
issetugid 4
pread 4
getegid 7
read 7
workq_kernreturn 9
close 10
open 10
kevent 11
munmap 11
open_nocancel 12
close_nocancel 13
fstat64 13
read_nocancel 15
__sysctl 19
getuid 19
geteuid 27
mmap 30
stat64 38
and similarly:
# dtruss -c -f /bin/df /data/hirise06
CALL COUNT
bsdthread_register 1
exit 1
fcntl 1
fstat64 1
geteuid 1
getrlimit 1
issetugid 1
shared_region_check_np 1
statfs64 1
thread_selfid 1
getfsstat64 2
getpid 2
ioctl 2
madvise 2
write_nocancel 2
close 3
close_nocancel 3
open 3
open_nocancel 3
read_nocancel 3
pread 4
munmap 5
stat64 6
__sysctl 15
mmap 16
Information forwarded
to
owner <at> debbugs.gnu.org, bug-coreutils <at> gnu.org:
bug#7355; Package
coreutils.
(Tue, 09 Nov 2010 00:28:02 GMT)
Full text and
rfc822 format available.
Message #17 received at 7355 <at> debbugs.gnu.org (full text, mbox):
On 08/11/10 23:38, Tim Spriggs wrote:
> 2010/11/8 Pádraig Brady <P <at> draigbrady.com>:
>> On 08/11/10 20:37, Tim Spriggs wrote:
>>> Hello,
>>>
>>> I am running gentoo-prefix on OSX which builds coreutils. When I
>>> compare the output of the system df to the standard GNU df I get:
>>>
>>>
>>> $ /bin/df -h /data/hirise06
>>> Filesystem Size Used Avail Capacity Mounted on
>>> hiserve2:/vol/hirise06 7.9Ti 430Gi 7.5Ti 6% /data/hirise06
>>>
>>> $ df -h /data/hirise06
>>> Filesystem Size Used Avail Use% Mounted on
>>> hiserve2:/vol/hirise06
>>> 2.0T 431G -523G - /data/hirise06
>>>
>>> The 2.0T size looks suspicious to me. Maybe an integer is overflowing somewhere?
>>
>> Well 2T is between 32 bit and 64 bit.
>> If I had to guess I'd say df was using a 32 bit statfs
>> that was returning 1024 blksize?
>>
>> Could you compare the system calls made by both versions of df.
>> On GNU/Linux I can trace the statfs calls like:
>>
>> strace -v -e statfs,statfs64 df /data/hirise06
>>
>> If the trace for the above gives no output
>> for either command, then can you include the
>> full trace output.
>>
>> thanks,
>> Pádraig.
>>
>
It does look like the right functions are being called.
If you could concentrate on the calls below
(with the dtruss -t option according to google),
you might notice some differences.
> # dtruss -c -f /opt/prefix/bin/df /data/hirise06
>
>
> CALL COUNT
> fstatfs64 1
> statfs64 1
> getfsstat64 2
>
> and similarly:
>
> # dtruss -c -f /bin/df /data/hirise06
>
> CALL COUNT
> statfs64 1
> getfsstat64 2
Without an OSX box to work on, it's awkward.
cheers,
Pádraig.
Information forwarded
to
owner <at> debbugs.gnu.org, bug-coreutils <at> gnu.org:
bug#7355; Package
coreutils.
(Tue, 09 Nov 2010 03:37:01 GMT)
Full text and
rfc822 format available.
Message #20 received at 7355 <at> debbugs.gnu.org (full text, mbox):
2010/11/8 Pádraig Brady <P <at> draigbrady.com>:
> Also could you give your exact version of OSX.
> I had a quick google for statfs64 and OSX and
> it seems that _DARWIN_USE_64_BIT_INODE needs to
> be defined (and your OSX version to support symbol variants).
> Could you try adding that as the first line in df.c? I.E.
>
> #define _DARWIN_USE_64_BIT_INODE 1
>
> If that's the case, then it seems like the AC_SYS_LARGEFILE
> would need to be updated in autoconf to handle this.
>
> cheers,
> Pádraig.
I added the define and then ran a vanilla ./configure && make with the
same results.
From the system_profiler command I can get the version of OSX:
System Software Overview:
System Version: Mac OS X 10.6.4 (10F2521)
Kernel Version: Darwin 10.4.2
Boot Volume: guru
Boot Mode: Normal
Computer Name: guru
User Name: Prefix User (prefix)
Secure Virtual Memory: Not Enabled
64-bit Kernel and Extensions: Yes
Time since boot: 4 days 19:42
The coreutils package in prefix is labelled as coreutils-8.5-r01.1
I can reproduce this on a 10.5 (Leopard) host as well with identical results.
Information forwarded
to
owner <at> debbugs.gnu.org, bug-coreutils <at> gnu.org:
bug#7355; Package
coreutils.
(Tue, 09 Nov 2010 03:56:02 GMT)
Full text and
rfc822 format available.
Message #23 received at 7355 <at> debbugs.gnu.org (full text, mbox):
2010/11/8 Pádraig Brady <P <at> draigbrady.com>:
> On 08/11/10 23:38, Tim Spriggs wrote:
>> 2010/11/8 Pádraig Brady <P <at> draigbrady.com>:
>>> On 08/11/10 20:37, Tim Spriggs wrote:
>>>> Hello,
>>>>
>>>> I am running gentoo-prefix on OSX which builds coreutils. When I
>>>> compare the output of the system df to the standard GNU df I get:
>>>>
>>>>
>>>> $ /bin/df -h /data/hirise06
>>>> Filesystem Size Used Avail Capacity Mounted on
>>>> hiserve2:/vol/hirise06 7.9Ti 430Gi 7.5Ti 6% /data/hirise06
>>>>
>>>> $ df -h /data/hirise06
>>>> Filesystem Size Used Avail Use% Mounted on
>>>> hiserve2:/vol/hirise06
>>>> 2.0T 431G -523G - /data/hirise06
>>>>
>>>> The 2.0T size looks suspicious to me. Maybe an integer is overflowing somewhere?
>>>
>>> Well 2T is between 32 bit and 64 bit.
>>> If I had to guess I'd say df was using a 32 bit statfs
>>> that was returning 1024 blksize?
>>>
>>> Could you compare the system calls made by both versions of df.
>>> On GNU/Linux I can trace the statfs calls like:
>>>
>>> strace -v -e statfs,statfs64 df /data/hirise06
>>>
>>> If the trace for the above gives no output
>>> for either command, then can you include the
>>> full trace output.
>>>
>>> thanks,
>>> Pádraig.
>>>
>>
>
> It does look like the right functions are being called.
> If you could concentrate on the calls below
> (with the dtruss -t option according to google),
> you might notice some differences.
>
>> # dtruss -c -f /opt/prefix/bin/df /data/hirise06
>>
>>
>> CALL COUNT
>> fstatfs64 1
>> statfs64 1
>> getfsstat64 2
>>
>> and similarly:
>>
>> # dtruss -c -f /bin/df /data/hirise06
>>
>> CALL COUNT
>> statfs64 1
>> getfsstat64 2
>
> Without an OSX box to work on, it's awkward.
>
> cheers,
> Pádraig.
>
Thanks for your patience. If this gets too much further I may just
make an OSX box available to you if you want. I had to modify the
dtruss command to dtruss.stats in order to grab all the syscalls into
one output. Otherwise you can only specify one call with -t afaict.
:-/ Anyway, with stack information included:
guru ~ # ./dtruss.stats -s -t fstatfs64,statfs64,getfsstat64 -f
/bin/df -h /data/hirise07
Filesystem Size Used Avail Capacity Mounted on
hiserve1:/vol/hirise07 4.5Ti 3.4Ti 1.1Ti 77% /data/hirise07
PID/THRD SYSCALL(args) = return
754/0x296efe: getfsstat64(0x0, 0x0, 0x2) = 11 0
libSystem.B.dylib`getfsstat64+0xa
df`0x100002505
df`0x1000015c4
df`0x3
754/0x296efe: getfsstat64(0x100800000, 0x65A0, 0x2) = 11 0
libSystem.B.dylib`getfsstat64+0xa
df`0x100002505
df`0x1000015c4
df`0x3
754/0x296efe: statfs64(0x7FFF5FBFEA83, 0x7FFF5FBFE070, 0x0) = 0 0
libSystem.B.dylib`statfs$INODE64+0xa
df`0x1000015c4
df`0x3
guru ~ # ./dtruss.stats -s -t fstatfs64,statfs64,getfsstat64 -f
/opt/prefix/bin/df -h /data/hirise07
Filesystem Size Used Avail Use% Mounted on
hiserve1:/vol/hirise07
512G -591G -946G 39% /data/hirise07
PID/THRD SYSCALL(args) = return
762/0x296f30: fstatfs64(0x3, 0x7FFF5FBFA850, 0x0) = 0 0
libSystem.B.dylib`fstatfs64+0xa
CoreFoundation`_CFBundleCopyDirectoryContentsAtPath+0x63f
CoreFoundation`_CFBundleURLLooksLikeBundleVersion+0x41
CoreFoundation`_CFBundleCreate+0x14d
CoreFoundation`_CFBundleGetMainBundleAlreadyLocked+0xad
CoreFoundation`CFBundleGetMainBundle+0x69
CoreFoundation`cacheBundleInfo+0xc
CoreFoundation`_CFAppVersionCheckLessThan+0x90
CoreFoundation`__CFInitialize+0x6a6
dyld`ImageLoaderMachO::doImageInit(ImageLoader::LinkContext
const&)+0x8a
dyld`ImageLoaderMachO::doInitialization(ImageLoader::LinkContext
const&)+0x1b
dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext
const&, unsigned int)+0xec
dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext
const&, unsigned int)+0x9d
dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext
const&, unsigned int)+0x9d
dyld`ImageLoader::runInitializers(ImageLoader::LinkContext
const&)+0x3a
dyld`dyld::initializeMainExecutable()+0x85
dyld`dyld::_main(macho_header const*, unsigned long,
int, char const**, char const**, char const**)+0xbb1
dyld`dyldbootstrap::start(macho_header const*, int, char
const**, long)+0x31f
dyld`_dyld_start+0x2a
df`0x100000000
762/0x296f30: getfsstat64(0x0, 0x0, 0x2) = 11 0
libSystem.B.dylib`getfsstat64+0xa
df`read_file_system_list+0x1f
df`main+0x578
df`start+0x34
df`0x3
762/0x296f30: getfsstat64(0x101010C00, 0x65A0, 0x2) = 11 0
libSystem.B.dylib`getfsstat64+0xa
df`read_file_system_list+0x1f
df`main+0x578
df`start+0x34
df`0x3
762/0x296f30: statfs64(0x7FFF5FBFEA5E, 0x7FFF5FBFD980,
0x7FFF5FBFE610) = 0 0
libSystem.B.dylib`statfs$INODE64+0xa
df`get_fs_usage+0x15
df`show_dev+0xad2
df`main+0x9b0
df`start+0x34
df`0x3
Thanks,
-Tim
Information forwarded
to
owner <at> debbugs.gnu.org, bug-coreutils <at> gnu.org:
bug#7355; Package
coreutils.
(Mon, 13 Dec 2010 20:41:01 GMT)
Full text and
rfc822 format available.
Message #26 received at 7355 <at> debbugs.gnu.org (full text, mbox):
Hi,
Is there anything else I can provide to help resolve this bug?
Thanks,
-Tim
Information forwarded
to
owner <at> debbugs.gnu.org, bug-coreutils <at> gnu.org:
bug#7355; Package
coreutils.
(Mon, 13 Dec 2010 21:56:01 GMT)
Full text and
rfc822 format available.
Message #29 received at 7355 <at> debbugs.gnu.org (full text, mbox):
On 13/12/10 20:46, Tim Spriggs wrote:
> Hi,
>
> Is there anything else I can provide to help resolve this bug?
I'll beg for an OSX a/c somewhere tomorrow
Information forwarded
to
owner <at> debbugs.gnu.org, bug-coreutils <at> gnu.org:
bug#7355; Package
coreutils.
(Mon, 03 Jan 2011 20:49:01 GMT)
Full text and
rfc822 format available.
Message #32 received at 7355 <at> debbugs.gnu.org (full text, mbox):
2010/12/13 Pádraig Brady <P <at> draigbrady.com>:
> On 13/12/10 20:46, Tim Spriggs wrote:
>> Hi,
>>
>> Is there anything else I can provide to help resolve this bug?
>
> I'll beg for an OSX a/c somewhere tomorrow
>
If this is still hard to come by I can setup a machine that you can
ssh into for this. Contact me offline for details.
Reply sent
to
Pádraig Brady <P <at> draigBrady.com>:
You have taken responsibility.
(Mon, 28 Nov 2011 23:05:01 GMT)
Full text and
rfc822 format available.
Notification sent
to
Tim Spriggs <tims <at> uahirise.org>:
bug acknowledged by developer.
(Mon, 28 Nov 2011 23:05:02 GMT)
Full text and
rfc822 format available.
Message #37 received at 7355-done <at> debbugs.gnu.org (full text, mbox):
On 11/28/2011 05:53 PM, Tim Spriggs wrote:
> 2011/1/3 Pádraig Brady <P <at> draigbrady.com>:
>> On 03/01/11 20:55, Tim Spriggs wrote:
>>> 2010/12/13 Pádraig Brady <P <at> draigbrady.com>:
>>>> On 13/12/10 20:46, Tim Spriggs wrote:
>>>>> Hi,
>>>>>
>>>>> Is there anything else I can provide to help resolve this bug?
>>>>
>>>> I'll beg for an OSX a/c somewhere tomorrow
>>>>
>>>
>>> If this is still hard to come by I can setup a machine that you can
>>> ssh into for this. Contact me offline for details.
>>
>> It would help a lot to have
>> the latest release compiled
>> so I could run it on your machine
>>
>> cheers,
>> Pádraig.
>>
>
> Hello again,
>
> Sorry for taking so long (almost a year!) but we have an OSX host on
> our external networks now and a filesystem mounted that shows the bug.
> If you send me an ssh pub key then I'll put it into place and send you
> the rest of the details.
Cool, as I initially suspected, defining _DARWIN_USE_64_BIT_INODE
at the appropriate place fixes things. This has been in place
since coreutils 8.13. I verified 8.14 works on your system.
See also http://debbugs.gnu.org/9140
$ src/df -a # coreutils-8.14
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/disk0s2 312235312 26613868 285365444 9% /
devfs 110 110 0 100% /dev
map -hosts 0 0 0 - /net
map auto_home 0 0 0 - /home
hiserve1.lpl.arizona.edu:/vol/lroc 11220602064 9762365096 1458236968 88% /data/lroc
/dev/disk1s2 31121368 4491368 26630000 15% /opt/prefix
$ df
Filesystem 512-blocks Used Available Capacity Mounted on
/dev/disk0s2 624470624 53103352 570855272 9% /
devfs 220 220 0 100% /dev
map -hosts 0 0 0 100% /net
map auto_home 0 0 0 100% /home
hiserve1.lpl.arizona.edu:/vol/lroc 22441204128 19524730200 2916473928 88% /data/lroc
/dev/disk1s2 62242736 8982736 53260000 15% /opt/prefix
$ /opt/prefix/bin/df -a # coreutils-8.9
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/disk0s2 312235312 26613868 285365444 9% /
devfs 110 110 0 100% /dev
map -hosts 0 0 0 - /net
map auto_home 0 0 0 - /home
hiserve1.lpl.arizona.edu:/vol/lroc
483183824 -975053160 -689246664 59% /data/lroc
/dev/disk1s2 31121368 4491368 26630000 15% /opt/prefix
Notes:
Newer coreutils also fixed up the alignment
coreutils diverges from POSIX and defaults to 1024 byte blocks
The system df displays /net and /home even without -a. bug?
The system df displays the capacity as 100%. bug?
cheers,
Pádraig.
bug archived.
Request was from
Debbugs Internal Request <help-debbugs <at> gnu.org>
to
internal_control <at> debbugs.gnu.org.
(Tue, 27 Dec 2011 12:24:03 GMT)
Full text and
rfc822 format available.
This bug report was last modified 12 years and 123 days ago.
Previous Next
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997,2003 nCipher Corporation Ltd,
1994-97 Ian Jackson.