GNU bug report logs -
#25630
multibyte: df: unicode is not supported on mounted point
Previous Next
To reply to this bug, email your comments to 25630 AT debbugs.gnu.org.
Toggle the display of automated, internal messages from the tracker.
Report forwarded
to
bug-coreutils <at> gnu.org
:
bug#25630
; Package
coreutils
.
(Mon, 06 Feb 2017 06:24:02 GMT)
Full text and
rfc822 format available.
Acknowledgement sent
to
Jaeseok Park <jaeseok.park <at> daliworks.net>
:
New bug report received and forwarded. Copy sent to
bug-coreutils <at> gnu.org
.
(Mon, 06 Feb 2017 06:24:02 GMT)
Full text and
rfc822 format available.
Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
Hi.
I'm using coreutils on OSX.
I have a USB storage which has the name "무제".
It is Korean characters and it means "NO NAME".
However, when I execute "df", it shows as below:
/dev/disk3s1 7563232 2510624 5052608 34% /Volumes/�??�?��??�?�
It seems not to support Unicode charaters.
Could you please fix it?
Thank you.
Best Regards,
Jaeseok
--
--------------------------
*"Good People, Valuable Challenge, Better World"*
Jaeseok Park | Daliworks Inc.
Mobile : +82-10-5759-5853
Office : +82-02-2274-3254
----------------------------
[Message part 2 (text/html, inline)]
Information forwarded
to
bug-coreutils <at> gnu.org
:
bug#25630
; Package
coreutils
.
(Tue, 07 Feb 2017 03:45:01 GMT)
Full text and
rfc822 format available.
Message #8 received at 25630 <at> debbugs.gnu.org (full text, mbox):
On 05/02/17 21:49, Jaeseok Park wrote:
> Hi.
>
> I'm using coreutils on OSX.
> I have a USB storage which has the name "무제".
> It is Korean characters and it means "NO NAME".
>
> However, when I execute "df", it shows as below:
>
> /dev/disk3s1 7563232 2510624 5052608 34% /Volumes/�??�?��??�?�
>
> It seems not to support Unicode charaters.
>
> Could you please fix it?
Yes our replacement function in df is simplistic,
and doesn't deal with all encodings. See:
http://git.sv.gnu.org/cgit/coreutils.git/commit/?id=v8.17-51-g3ed70fd
Now the fact that you have both types of replacement chars
� (from mbsalign) and ? (from df),
suggests there is some mismatch in encodings.
What is the value of $LC_ALL and $LANG on your system?
What encoding is the file on disk? I presume some variant of ISO-2022-KR,
though I couldn't correlate that with the above output.
The encoding of the file on disk should be indicated by:
LC_ALL=C gls --quoting=shell-escape -1 /Volumes
If it was possible to use UTF-8 representation of /Volumes/무제
then it would be displayed without issue.
thanks,
Pádraig
Information forwarded
to
bug-coreutils <at> gnu.org
:
bug#25630
; Package
coreutils
.
(Tue, 07 Feb 2017 16:05:01 GMT)
Full text and
rfc822 format available.
Message #11 received at 25630 <at> debbugs.gnu.org (full text, mbox):
On 07/02/17 05:11, Jaeseok Park wrote:
> Dear Pádraig
>
> Thank you for your reply.
>
> 1. Maybe its volume name seems to be encoded by UTF-8. I formatted my USB storage on OSX.
>
> 2. My $LC_ALL is empty and $LANG is "ko_KR.UTF-8".
>
> 3. The result of gls is as below:
>
> BEAGLEBONE
> boot.tar
> ext
> ssd
> ''$'\341\204\206\341\205\256\341\204\214\341\205\246'
Ah right that's the decomposed form.
HFS must use that for normalization of file names.
Using that I can reproduce your issue with an incorrect locale:
# LC_ALL=ko_KR df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 100M 120K 100M 1% /root/�??�?��??�?�
# LC_ALL=ko_KR.UTF-8 df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 100M 120K 100M 1% /root/무제
I.E. It looks like you have the wrong locale settings for the df command.
Please try setting LC_ALL as above, or otherwise setting UTF-8.
thanks,
Pádraig
Information forwarded
to
bug-coreutils <at> gnu.org
:
bug#25630
; Package
coreutils
.
(Tue, 07 Feb 2017 16:32:01 GMT)
Full text and
rfc822 format available.
Message #14 received at 25630 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
Dear Pádraig
Thank you for your reply.
1. Maybe its volume name seems to be encoded by UTF-8. I formatted my USB
storage on OSX.
2. My $LC_ALL is empty and $LANG is "ko_KR.UTF-8".
3. The result of gls is as below:
BEAGLEBONE
boot.tar
ext
ssd
''$'\341\204\206\341\205\256\341\204\214\341\205\246'
If any test is needed, please tell me.
Thank you.
Best regards,
Jaeseok
2017-02-07 12:44 GMT+09:00 Pádraig Brady <P <at> draigbrady.com>:
> On 05/02/17 21:49, Jaeseok Park wrote:
> > Hi.
> >
> > I'm using coreutils on OSX.
> > I have a USB storage which has the name "무제".
> > It is Korean characters and it means "NO NAME".
> >
> > However, when I execute "df", it shows as below:
> >
> > /dev/disk3s1 7563232 2510624 5052608 34% /Volumes/�??�?��??�?�
> >
> > It seems not to support Unicode charaters.
> >
> > Could you please fix it?
>
> Yes our replacement function in df is simplistic,
> and doesn't deal with all encodings. See:
> http://git.sv.gnu.org/cgit/coreutils.git/commit/?id=v8.17-51-g3ed70fd
>
> Now the fact that you have both types of replacement chars
> � (from mbsalign) and ? (from df),
> suggests there is some mismatch in encodings.
>
> What is the value of $LC_ALL and $LANG on your system?
> What encoding is the file on disk? I presume some variant of ISO-2022-KR,
> though I couldn't correlate that with the above output.
> The encoding of the file on disk should be indicated by:
> LC_ALL=C gls --quoting=shell-escape -1 /Volumes
>
> If it was possible to use UTF-8 representation of /Volumes/무제
> then it would be displayed without issue.
>
> thanks,
> Pádraig
>
--
--------------------------
*"Good People, Valuable Challenge, Better World"*
Jaeseok Park | Daliworks Inc.
Mobile : +82-10-5759-5853
Office : +82-02-2274-3254
----------------------------
[Message part 2 (text/html, inline)]
Information forwarded
to
bug-coreutils <at> gnu.org
:
bug#25630
; Package
coreutils
.
(Wed, 08 Feb 2017 01:31:01 GMT)
Full text and
rfc822 format available.
Message #17 received at 25630 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
I tried to execute the commands as you guided me, however the result is the
same.
Mini:~$ LC_ALL=ko_KR df -h
Filesystem Size Used Avail Use% Mounted on
/dev/disk0s2 233G 176G 56G 76% /
/dev/disk1s2 931G 686G 246G 74% /Volumes/ext
/dev/disk2s1 39M 33M 6.4M 84% /Volumes/BEAGLEBONE
/dev/disk3s1 15G 2.5M 15G 1% /Volumes/�??�?��??�?�
Mini:~$ LC_ALL=ko_KR.UTF-8 df -h
Filesystem Size Used Avail Use% Mounted on
/dev/disk0s2 233G 176G 56G 76% /
/dev/disk1s2 931G 686G 246G 74% /Volumes/ext
/dev/disk2s1 39M 33M 6.4M 84% /Volumes/BEAGLEBONE
/dev/disk3s1 15G 2.5M 15G 1% /Volumes/�??�?��??�?�
Mini:~$ gls /Volumes/
BEAGLEBONE boot.tar ext ssd 무제
Did I do wrong something?
Regards,
Jaeseok
2017-02-08 1:04 GMT+09:00 Pádraig Brady <P <at> draigbrady.com>:
> On 07/02/17 05:11, Jaeseok Park wrote:
> > Dear Pádraig
> >
> > Thank you for your reply.
> >
> > 1. Maybe its volume name seems to be encoded by UTF-8. I formatted my
> USB storage on OSX.
> >
> > 2. My $LC_ALL is empty and $LANG is "ko_KR.UTF-8".
> >
> > 3. The result of gls is as below:
> >
> > BEAGLEBONE
> > boot.tar
> > ext
> > ssd
> > ''$'\341\204\206\341\205\256\341\204\214\341\205\246'
>
> Ah right that's the decomposed form.
> HFS must use that for normalization of file names.
> Using that I can reproduce your issue with an incorrect locale:
>
> # LC_ALL=ko_KR df -h
> Filesystem Size Used Avail Use% Mounted on
> /dev/sda1 100M 120K 100M 1% /root/�??�?��??�?�
> # LC_ALL=ko_KR.UTF-8 df -h
> Filesystem Size Used Avail Use% Mounted on
> /dev/sda1 100M 120K 100M 1% /root/무제
>
>
> I.E. It looks like you have the wrong locale settings for the df command.
> Please try setting LC_ALL as above, or otherwise setting UTF-8.
>
> thanks,
> Pádraig
>
--
--------------------------
*"Good People, Valuable Challenge, Better World"*
Jaeseok Park | Daliworks Inc.
Mobile : +82-10-5759-5853
Office : +82-02-2274-3254
----------------------------
[Message part 2 (text/html, inline)]
Information forwarded
to
bug-coreutils <at> gnu.org
:
bug#25630
; Package
coreutils
.
(Wed, 08 Feb 2017 03:53:01 GMT)
Full text and
rfc822 format available.
Message #20 received at 25630 <at> debbugs.gnu.org (full text, mbox):
On 07/02/17 17:30, Jaeseok Park wrote:
> 2017-02-08 1:04 GMT+09:00 Pádraig Brady <P <at> draigbrady.com <mailto:P <at> draigbrady.com>>:
>
> On 07/02/17 05:11, Jaeseok Park wrote:
> > Dear Pádraig
> >
> > Thank you for your reply.
> >
> > 1. Maybe its volume name seems to be encoded by UTF-8. I formatted my USB storage on OSX.
> >
> > 2. My $LC_ALL is empty and $LANG is "ko_KR.UTF-8".
> >
> > 3. The result of gls is as below:
> >
> > BEAGLEBONE
> > boot.tar
> > ext
> > ssd
> > ''$'\341\204\206\341\205\256\341\204\214\341\205\246'
>
> Ah right that's the decomposed form.
> HFS must use that for normalization of file names.
> Using that I can reproduce your issue with an incorrect locale:
>
> # LC_ALL=ko_KR df -h
> Filesystem Size Used Avail Use% Mounted on
> /dev/sda1 100M 120K 100M 1% /root/�??�?��??�?�
> # LC_ALL=ko_KR.UTF-8 df -h
> Filesystem Size Used Avail Use% Mounted on
> /dev/sda1 100M 120K 100M 1% /root/무제
>
>
> I.E. It looks like you have the wrong locale settings for the df command.
> Please try setting LC_ALL as above, or otherwise setting UTF-8.
> I tried to execute the commands as you guided me, however the result is the same.
>
> Mini:~$ LC_ALL=ko_KR df -h
> Filesystem Size Used Avail Use% Mounted on
> /dev/disk0s2 233G 176G 56G 76% /
> /dev/disk1s2 931G 686G 246G 74% /Volumes/ext
> /dev/disk2s1 39M 33M 6.4M 84% /Volumes/BEAGLEBONE
> /dev/disk3s1 15G 2.5M 15G 1% /Volumes/�??�?��??�?�
>
> Mini:~$ LC_ALL=ko_KR.UTF-8 df -h
> Filesystem Size Used Avail Use% Mounted on
> /dev/disk0s2 233G 176G 56G 76% /
> /dev/disk1s2 931G 686G 246G 74% /Volumes/ext
> /dev/disk2s1 39M 33M 6.4M 84% /Volumes/BEAGLEBONE
> /dev/disk3s1 15G 2.5M 15G 1% /Volumes/�??�?��??�?�
>
> Mini:~$ gls /Volumes/
> BEAGLEBONE boot.tar ext ssd무제
>
> Did I do wrong something?
>
> Regards,
> Jaeseok
>
>
I don't have access to OSX to try out things,
but I suspect ko_KR.UTF-8 may not be supported on your system?
Can you try instead with LC_ALL=en_US.UTF-8 or LC_ALL=C,
both of which display correctly on Fedora Linux here.
Also what's the output from:
LC_ALL=ko_KR.UTF-8 locale
If that suggests ko_KR.UTF-8 is supported,
then I'll need to get access to an OSX system to debug.
Perhaps there is some shennanigans with the returned charset on OSX.
thanks,
Pádraig
Information forwarded
to
bug-coreutils <at> gnu.org
:
bug#25630
; Package
coreutils
.
(Wed, 08 Feb 2017 06:02:02 GMT)
Full text and
rfc822 format available.
Message #23 received at 25630 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
I think that df may support Korean characters because gls supports Korean
characters without any modification of settings.
The results of the commands you requested are as below:
Mini:~$ LC_ALL=en_US.UTF-8 df -h
Filesystem Size Used Avail Use% Mounted on
/dev/disk0s2 233G 177G 55G 77% /
/dev/disk1s2 931G 686G 245G 74% /Volumes/ext
/dev/disk2s1 39M 33M 6.4M 84% /Volumes/BEAGLEBONE
/dev/disk3s1 15G 2.5M 15G 1% /Volumes/�??�?��??�?�
Mini:~$ LC_ALL=C df -h
Filesystem Size Used Avail Use% Mounted on
/dev/disk0s2 233G 177G 55G 77% /
/dev/disk1s2 931G 686G 245G 74% /Volumes/ext
/dev/disk2s1 39M 33M 6.4M 84% /Volumes/BEAGLEBONE
/dev/disk3s1 15G 2.5M 15G 1% /Volumes/무제
Mini:~$ LC_ALL=ko_KR.UTF-8 locale
LANG="ko_KR.UTF-8"
LC_COLLATE="ko_KR.UTF-8"
LC_CTYPE="ko_KR.UTF-8"
LC_MESSAGES="ko_KR.UTF-8"
LC_MONETARY="ko_KR.UTF-8"
LC_NUMERIC="ko_KR.UTF-8"
LC_TIME="ko_KR.UTF-8"
LC_ALL="ko_KR.UTF-8"
LC_ALL=C makes it work well.
By the way, why does gls work well without LC_ALL setting?
Regards,
Jaeseok
2017-02-08 12:52 GMT+09:00 Pádraig Brady <P <at> draigbrady.com>:
> On 07/02/17 17:30, Jaeseok Park wrote:
> > 2017-02-08 1:04 GMT+09:00 Pádraig Brady <P <at> draigbrady.com <mailto:
> P <at> draigbrady.com>>:
> >
> > On 07/02/17 05:11, Jaeseok Park wrote:
> > > Dear Pádraig
> > >
> > > Thank you for your reply.
> > >
> > > 1. Maybe its volume name seems to be encoded by UTF-8. I formatted
> my USB storage on OSX.
> > >
> > > 2. My $LC_ALL is empty and $LANG is "ko_KR.UTF-8".
> > >
> > > 3. The result of gls is as below:
> > >
> > > BEAGLEBONE
> > > boot.tar
> > > ext
> > > ssd
> > > ''$'\341\204\206\341\205\256\341\204\214\341\205\246'
> >
> > Ah right that's the decomposed form.
> > HFS must use that for normalization of file names.
> > Using that I can reproduce your issue with an incorrect locale:
> >
> > # LC_ALL=ko_KR df -h
> > Filesystem Size Used Avail Use% Mounted on
> > /dev/sda1 100M 120K 100M 1% /root/�??�?��??�?�
> > # LC_ALL=ko_KR.UTF-8 df -h
> > Filesystem Size Used Avail Use% Mounted on
> > /dev/sda1 100M 120K 100M 1% /root/무제
> >
> >
> > I.E. It looks like you have the wrong locale settings for the df
> command.
> > Please try setting LC_ALL as above, or otherwise setting UTF-8.
>
> > I tried to execute the commands as you guided me, however the result is
> the same.
> >
> > Mini:~$ LC_ALL=ko_KR df -h
> > Filesystem Size Used Avail Use% Mounted on
> > /dev/disk0s2 233G 176G 56G 76% /
> > /dev/disk1s2 931G 686G 246G 74% /Volumes/ext
> > /dev/disk2s1 39M 33M 6.4M 84% /Volumes/BEAGLEBONE
> > /dev/disk3s1 15G 2.5M 15G 1% /Volumes/�??�?��??�?�
> >
> > Mini:~$ LC_ALL=ko_KR.UTF-8 df -h
> > Filesystem Size Used Avail Use% Mounted on
> > /dev/disk0s2 233G 176G 56G 76% /
> > /dev/disk1s2 931G 686G 246G 74% /Volumes/ext
> > /dev/disk2s1 39M 33M 6.4M 84% /Volumes/BEAGLEBONE
> > /dev/disk3s1 15G 2.5M 15G 1% /Volumes/�??�?��??�?�
> >
> > Mini:~$ gls /Volumes/
> > BEAGLEBONE boot.tar ext ssd무제
> >
> > Did I do wrong something?
> >
> > Regards,
> > Jaeseok
> >
> >
>
> I don't have access to OSX to try out things,
> but I suspect ko_KR.UTF-8 may not be supported on your system?
> Can you try instead with LC_ALL=en_US.UTF-8 or LC_ALL=C,
> both of which display correctly on Fedora Linux here.
>
> Also what's the output from:
>
> LC_ALL=ko_KR.UTF-8 locale
>
> If that suggests ko_KR.UTF-8 is supported,
> then I'll need to get access to an OSX system to debug.
> Perhaps there is some shennanigans with the returned charset on OSX.
>
> thanks,
> Pádraig
>
--
--------------------------
*"Good People, Valuable Challenge, Better World"*
Jaeseok Park | Daliworks Inc.
Mobile : +82-10-5759-5853
Office : +82-02-2274-3254
----------------------------
[Message part 2 (text/html, inline)]
Information forwarded
to
bug-coreutils <at> gnu.org
:
bug#25630
; Package
coreutils
.
(Wed, 08 Feb 2017 18:08:02 GMT)
Full text and
rfc822 format available.
Message #26 received at 25630 <at> debbugs.gnu.org (full text, mbox):
On 07/02/17 22:01, Jaeseok Park wrote:
> I think that df may support Korean characters because gls supports Korean
> characters without any modification of settings.
>
> The results of the commands you requested are as below:
>
> Mini:~$ LC_ALL=en_US.UTF-8 df -h
> Filesystem Size Used Avail Use% Mounted on
> /dev/disk0s2 233G 177G 55G 77% /
> /dev/disk1s2 931G 686G 245G 74% /Volumes/ext
> /dev/disk2s1 39M 33M 6.4M 84% /Volumes/BEAGLEBONE
> /dev/disk3s1 15G 2.5M 15G 1% /Volumes/�??�?��??�?�
>
> Mini:~$ LC_ALL=C df -h
> Filesystem Size Used Avail Use% Mounted on
> /dev/disk0s2 233G 177G 55G 77% /
> /dev/disk1s2 931G 686G 245G 74% /Volumes/ext
> /dev/disk2s1 39M 33M 6.4M 84% /Volumes/BEAGLEBONE
> /dev/disk3s1 15G 2.5M 15G 1% /Volumes/무제
>
> Mini:~$ LC_ALL=ko_KR.UTF-8 locale
> LANG="ko_KR.UTF-8"
> LC_COLLATE="ko_KR.UTF-8"
> LC_CTYPE="ko_KR.UTF-8"
> LC_MESSAGES="ko_KR.UTF-8"
> LC_MONETARY="ko_KR.UTF-8"
> LC_NUMERIC="ko_KR.UTF-8"
> LC_TIME="ko_KR.UTF-8"
> LC_ALL="ko_KR.UTF-8"
>
> LC_ALL=C makes it work well.
> By the way, why does gls work well without LC_ALL setting?
Right. LC_ALL=C will avoid the multibyte processing.
Now ls and df have similar multibyte processing,
with ls using mbrtowc() and df using mbstowcs().
Perhaps there is some issue with our use of the later on OSX,
with decomposed form?
I'll try to get access. What version of OSX is this?
thanks,
Pádraig
Information forwarded
to
bug-coreutils <at> gnu.org
:
bug#25630
; Package
coreutils
.
(Thu, 09 Feb 2017 01:27:01 GMT)
Full text and
rfc822 format available.
Message #29 received at 25630 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
It is OS X El Capitan, version 10.11.6.
Thank you.
Best regards,
Jaeseok
2017-02-09 3:07 GMT+09:00 Pádraig Brady <P <at> draigbrady.com>:
> On 07/02/17 22:01, Jaeseok Park wrote:
> > I think that df may support Korean characters because gls supports Korean
> > characters without any modification of settings.
> >
> > The results of the commands you requested are as below:
> >
> > Mini:~$ LC_ALL=en_US.UTF-8 df -h
> > Filesystem Size Used Avail Use% Mounted on
> > /dev/disk0s2 233G 177G 55G 77% /
> > /dev/disk1s2 931G 686G 245G 74% /Volumes/ext
> > /dev/disk2s1 39M 33M 6.4M 84% /Volumes/BEAGLEBONE
> > /dev/disk3s1 15G 2.5M 15G 1% /Volumes/�??�?��??�?�
> >
> > Mini:~$ LC_ALL=C df -h
> > Filesystem Size Used Avail Use% Mounted on
> > /dev/disk0s2 233G 177G 55G 77% /
> > /dev/disk1s2 931G 686G 245G 74% /Volumes/ext
> > /dev/disk2s1 39M 33M 6.4M 84% /Volumes/BEAGLEBONE
> > /dev/disk3s1 15G 2.5M 15G 1% /Volumes/무제
> >
> > Mini:~$ LC_ALL=ko_KR.UTF-8 locale
> > LANG="ko_KR.UTF-8"
> > LC_COLLATE="ko_KR.UTF-8"
> > LC_CTYPE="ko_KR.UTF-8"
> > LC_MESSAGES="ko_KR.UTF-8"
> > LC_MONETARY="ko_KR.UTF-8"
> > LC_NUMERIC="ko_KR.UTF-8"
> > LC_TIME="ko_KR.UTF-8"
> > LC_ALL="ko_KR.UTF-8"
> >
> > LC_ALL=C makes it work well.
> > By the way, why does gls work well without LC_ALL setting?
>
> Right. LC_ALL=C will avoid the multibyte processing.
> Now ls and df have similar multibyte processing,
> with ls using mbrtowc() and df using mbstowcs().
> Perhaps there is some issue with our use of the later on OSX,
> with decomposed form?
> I'll try to get access. What version of OSX is this?
>
> thanks,
> Pádraig
>
--
--------------------------
*"Good People, Valuable Challenge, Better World"*
Jaeseok Park | Daliworks Inc.
Mobile : +82-10-5759-5853
Office : +82-02-2274-3254
----------------------------
[Message part 2 (text/html, inline)]
Changed bug title to 'multibyte: df: unicode is not supported on mounted point' from '[bug/df] Unicode is not supported on mounted point when I execute "df" command'
Request was from
Assaf Gordon <assafgordon <at> gmail.com>
to
control <at> debbugs.gnu.org
.
(Mon, 29 Oct 2018 02:38:02 GMT)
Full text and
rfc822 format available.
Severity set to 'wishlist' from 'normal'
Request was from
Assaf Gordon <assafgordon <at> gmail.com>
to
control <at> debbugs.gnu.org
.
(Tue, 30 Oct 2018 09:02:01 GMT)
Full text and
rfc822 format available.
This bug report was last modified 6 years and 92 days ago.
Previous Next
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997,2003 nCipher Corporation Ltd,
1994-97 Ian Jackson.