GNU logs - #29832, boring messages

Message sent to bug-coreutils@HIDDEN:

Subject: bug#29832: Bug in 'ls -FQ': incorrectly quoted characters
Resent-From: hackerb9@HIDDEN
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-coreutils@HIDDEN
Resent-Date: Sun, 24 Dec 2017 07:15:01 +0000
Resent-Message-ID: <handler.29832.B.151409968311459 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: 29832 <at> debbugs.gnu.org
MIME-Version: 1.0
From: hackerb9@HIDDEN
Date: Sat, 23 Dec 2017 23:14:28 -0800
Message-ID: <CAO+cPoMAHZfLthJa55aq2OQjhyPFgZ+bG5ERaA_Q=GsK06wiMw@HIDDEN>
Content-Type: multipart/mixed; boundary="001a113ac54cf31658056110cc39"
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

--001a113ac54cf31658056110cc39
Content-Type: multipart/alternative; boundary="001a113ac54cf31655056110cc37"

--001a113ac54cf31655056110cc37
Content-Type: text/plain; charset="UTF-8"

Hello!

Thank you for your continued improvements and maintenance of 'ls' in
coreutils-8.28.

I have found a somewhat subtle bug. I've included a patch and a test script
to make sure there are no regressions.

In short, the problem is that a backlash is incorrectly put in front of the
characters @, =, >, and |. This only happens when using filetype indicators
and C style quoting.

  $ touch @
  $ ls
  @
  $ ls -F
  @
  $ ls -Q
  "@"
  $ ls -FQ
  "\@"        # <-- incorrect

There should be no backslash before the @.

This was a little bit tricky to track down as the code responsible is not
commented and appears to have been prematurely optimized by the programmer:

  ls.c:2141
  if (file_type <= indicator_style)
    {
      char const *p;
      for (p = &"*=>@|"[indicator_style - file_type]; *p; p++)
        set_char_quoting (filename_quoting_options, *p, 1);
    }

Basically, it says, if --file-type was used, then we should be sure to
quote '=', '>', and '@'. If --classify is used, then we should do the same
but also add in '*'. Or, to put it in less compressed C:

  if (indicator_style == file_type)
    {
      char const *p;
      for (p = "=>@|"; *p; p++)
        set_char_quoting (filename_quoting_options, *p, 1);
    }
  if (indicator_style == classify)
    {
      char const *p;
      for (p = "*=>@|"; *p; p++)
        set_char_quoting (filename_quoting_options, *p, 1);
    }

However, there is no need to quote those characters when the quoting-style
is set to C. In fact, it is an error to do so as these are not valid C
escape sequences.

I am guessing the idea of the code was that those characters would not be
backslash escaped, but rather the entire filename would be wrapped in
quotes so the indicator character cannot be confused for part of the
filename. And, indeed, this does work for some of the quoting styles:
"shell[-escape][-always]" work that way. Strangely, "c-maybe" also works
that way.

  $ ls -F --quoting-style=literal
  @
  $ ls -F --quoting-style=shell
  '@'
  $ ls -F --quoting-style=shell-always
  '@'
  $ ls -F --quoting-style=shell-escape
  '@'
  $ ls -F --quoting-style=shell-escape-always
  '@'
  $ ls -F --quoting-style=c-maybe
  "@"        # <-- This is correct!

I spent some time debugging it, but I didn't find the perfect fix as I
don't understand why it was written the way it was. Rather, I found
something that I think is an improvement and a step towards doing things
right. I've made a patch that simply checks if the quoting style is C and
skips the problematic lines. Ideally, somebody who understands why this
code was written this way in the first place can make a better patch, but
this should work for now and not break anything for anybody else.

Please find attached below, my patch for src/ls.c and a regression test
script to be placed in tests/ls/quoting-style-c.sh.

--b9

P.S. This bug may also apply to the -b (--quoting-style=escape). However,
the output from it appears to be more like a mish-mash of C and shell
escapes. (For example, tabs are shown as "\t", as in C, but spaces are
shown as "\ ", as in bourne shell. ) The -Q option, on the other hand,
generates perfectly acceptable C strings (tested with gcc and python),
except for those few buggy characters fixed in this patch.

--001a113ac54cf31655056110cc37
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div><div><div><div><div><div>Hello!<br><br></div>Thank yo=
u for your continued improvements and maintenance of &#39;ls&#39; in coreut=
ils-8.28.<br><br></div>I have found a somewhat subtle bug. I&#39;ve include=
d a patch and a test script to make sure there are no regressions.<br><br><=
/div>In short, the problem is that a backlash is incorrectly put in front o=
f the characters @, =3D, &gt;, and |. This only happens when using filetype=
 indicators and C style quoting.<br><br>=C2=A0 $ touch @<br>=C2=A0 $ ls <br=
>=C2=A0 @<br>=C2=A0 $ ls -F<br>=C2=A0 @<br>=C2=A0 $ ls -Q<br>=C2=A0 &quot;@=
&quot;<br>=C2=A0 $ ls -FQ<br></div><div>=C2=A0 &quot;\@&quot;=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 # &lt;-- incorrect<br><br></div>There should=
 be no backslash before the @. <br><br>This was a little bit tricky to trac=
k down as the code responsible is not commented and appears to have been pr=
ematurely optimized by the programmer:<br><br></div>=C2=A0 ls.c:2141<br>=C2=
=A0 if (file_type &lt;=3D indicator_style)<br>=C2=A0=C2=A0=C2=A0 {<br>=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0 char const *p;<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0 for (p =3D &amp;&quot;*=3D&gt;@|&quot;[indicator_style - file_type]; *p=
; p++)<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 set_char_quoting (file=
name_quoting_options, *p, 1);<br>=C2=A0=C2=A0=C2=A0 }<br><br></div>Basicall=
y, it says, if --file-type was used, then we should be sure to quote &#39;=
=3D&#39;, &#39;&gt;&#39;, and &#39;@&#39;. If --classify is used, then we s=
hould do the same but also add in &#39;*&#39;. Or, to put it in less compre=
ssed C:<br><div><div><br>=C2=A0 if (indicator_style =3D=3D file_type)<br>=
=C2=A0=C2=A0=C2=A0 {<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 char const *p;<br>=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 for (p =3D &quot;=3D&gt;@|&quot;; *p; p++)<b=
r>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 set_char_quoting (filename_quo=
ting_options, *p, 1);<br>=C2=A0=C2=A0=C2=A0 }<br>=C2=A0 if (indicator_style=
 =3D=3D classify)<br>=C2=A0=C2=A0=C2=A0 {<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
 char const *p;<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 for (p =3D &quot;*=3D&gt;=
@|&quot;; *p; p++)<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 set_char_q=
uoting (filename_quoting_options, *p, 1);<br>=C2=A0=C2=A0=C2=A0 }<br><br></=
div><div>However, there is no need to quote those characters when the quoti=
ng-style is set to C. In fact, it is an error to do so as these are not val=
id C escape sequences.<br><br></div><div>I am guessing the idea of the code=
 was that those characters would not be backslash escaped, but rather the e=
ntire filename would be wrapped in quotes so the indicator character cannot=
 be confused for part of the filename. And, indeed, this does work for some=
 of the quoting styles: &quot;shell[-escape][-always]&quot; work that way. =
Strangely, &quot;c-maybe&quot; also works that way.<br><br>=C2=A0 $ ls -F -=
-quoting-style=3Dliteral<br>=C2=A0 @<br>=C2=A0 $ ls -F --quoting-style=3Dsh=
ell<br>=C2=A0 &#39;@&#39;<br>=C2=A0 $ ls -F --quoting-style=3Dshell-always<=
br>=C2=A0 &#39;@&#39;<br>=C2=A0 $ ls -F --quoting-style=3Dshell-escape<br>=
=C2=A0 &#39;@&#39;<br>=C2=A0 $ ls -F --quoting-style=3Dshell-escape-<wbr>al=
ways<br>=C2=A0 &#39;@&#39;<br>=C2=A0 $ ls -F --quoting-style=3Dc-maybe<br><=
/div><div>=C2=A0 &quot;@&quot;=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 # =
&lt;-- This is correct!<br></div><div><br></div><div>I spent some time debu=
gging it, but I didn&#39;t find the perfect fix as I don&#39;t understand w=
hy it was written the way it was. Rather, I found something that I think is=
 an improvement and a step towards doing things right. I&#39;ve made a patc=
h that simply checks if the quoting style is C and skips the problematic li=
nes. Ideally, somebody who understands why this code was written this way i=
n the first place can make a better patch, but this should work for now and=
 not break anything for anybody else.<br><br></div><div>Please find attache=
d below, my patch for src/ls.c and a regression test script to be placed in=
 tests/ls/quoting-style-c.sh. <br><br></div><div>--b9<br><br></div><div>P.S=
. This bug may also apply to the -b (--quoting-style=3Descape). However, th=
e output from it appears to be more like a mish-mash of C and shell escapes=
. (For example, tabs are shown as &quot;\t&quot;, as in C, but spaces are s=
hown as &quot;\ &quot;, as in bourne shell. ) The -Q option, on the other h=
and, generates perfectly acceptable C strings (tested with gcc and python),=
 except for those few buggy characters fixed in this patch. <br><br><br></d=
iv></div></div>

--001a113ac54cf31655056110cc37--

--001a113ac54cf31658056110cc39
Content-Type: text/x-patch; charset="US-ASCII"; name="ls-FQ.patch"
Content-Disposition: attachment; filename="ls-FQ.patch"
Content-Transfer-Encoding: base64
X-Attachment-Id: f_jbkf5quy0

LS0tIHNyYy9scy5jLm9yaWcJMjAxNy0wOS0wMSAwMDoxMTowMy4wMDAwMDAwMDAgLTA3MDAKKysr
IHNyYy9scy5jCTIwMTctMTItMjMgMjI6NTM6NTcuNDQ4NTk0NTA3IC0wODAwCkBAIC0yMTM4LDEy
ICsyMTM4LDE5IEBACiAgIGZpbGVuYW1lX3F1b3Rpbmdfb3B0aW9ucyA9IGNsb25lX3F1b3Rpbmdf
b3B0aW9ucyAoTlVMTCk7CiAgIGlmIChxcyA9PSBlc2NhcGVfcXVvdGluZ19zdHlsZSkKICAgICBz
ZXRfY2hhcl9xdW90aW5nIChmaWxlbmFtZV9xdW90aW5nX29wdGlvbnMsICcgJywgMSk7Ci0gIGlm
IChmaWxlX3R5cGUgPD0gaW5kaWNhdG9yX3N0eWxlKQotICAgIHsKLSAgICAgIGNoYXIgY29uc3Qg
KnA7Ci0gICAgICBmb3IgKHAgPSAmIio9PkB8IltpbmRpY2F0b3Jfc3R5bGUgLSBmaWxlX3R5cGVd
OyAqcDsgcCsrKQotICAgICAgICBzZXRfY2hhcl9xdW90aW5nIChmaWxlbmFtZV9xdW90aW5nX29w
dGlvbnMsICpwLCAxKTsKLSAgICB9CisKKworICAvKiBUaGlzIGNvZGUgd2FzIGNhdXNpbmcgYnVn
cyBpbiBscyAtRlEuIFRoZXJlIHdlcmUgbm8gY29tbWVudHMsIHNvCisgICAgIGl0J3MgdW5jbGVh
ciBleGFjdGx5IHdoYXQgaXQgd2FzIHRyeWluZyB0byBhY2NvbXBsaXNoLiAKKyAgICAgRm9yIG5v
dywgd2UnbGwgc2tpcCBpdCBmb3IgQyBxdW90aW5nLXN0eWxlLiAoLVEpLiAgKi8KKyAgaWYgKHFz
ICE9IGNfcXVvdGluZ19zdHlsZSkgeworICAgIGlmIChmaWxlX3R5cGUgPD0gaW5kaWNhdG9yX3N0
eWxlKQorICAgICAgeworCWNoYXIgY29uc3QgKnA7CisJZm9yIChwID0gJiIqPT5AfCJbaW5kaWNh
dG9yX3N0eWxlIC0gZmlsZV90eXBlXTsgKnA7IHArKykKKwkgIHNldF9jaGFyX3F1b3RpbmcgKGZp
bGVuYW1lX3F1b3Rpbmdfb3B0aW9ucywgKnAsIDEpOworICAgICAgfQorICB9CiAKICAgZGlybmFt
ZV9xdW90aW5nX29wdGlvbnMgPSBjbG9uZV9xdW90aW5nX29wdGlvbnMgKE5VTEwpOwogICBzZXRf
Y2hhcl9xdW90aW5nIChkaXJuYW1lX3F1b3Rpbmdfb3B0aW9ucywgJzonLCAxKTsK
--001a113ac54cf31658056110cc39
Content-Type: application/x-sh; name="quoting-style-c.sh"
Content-Disposition: attachment; filename="quoting-style-c.sh"
Content-Transfer-Encoding: base64
X-Attachment-Id: f_jbkfbbu41

IyEvYmluL3NoCiMgVGhpcyBpcyB0ZXN0cy9scy9xdW90aW5nLXN0eWxlLWMuc2gKIyB0ZXN0IC0t
cXVvdGluZy1zdHlsZT1jICgtUSkKCiMgVXNhZ2U6CiMgICBjZCB0ZXN0cwojICAgbWFrZSBjaGVj
ayBURVNUUz1scy9xdW90aW5nLXN0eWxlLWMuc2gKCiMgdjE6IEZvciBub3csIHdlJ3JlIGp1c3Qg
d2F0Y2hpbmcgZm9yIHJlZ3Jlc3Npb25zIHdpdGggLS1jbGFzc2lmeSAoLUYpCiMKCiMgQ29weXJp
Z2h0IChDKSAyMDE3IEZyZWUgU29mdHdhcmUgRm91bmRhdGlvbiwgSW5jLgoKIyBUaGlzIHByb2dy
YW0gaXMgZnJlZSBzb2Z0d2FyZTogeW91IGNhbiByZWRpc3RyaWJ1dGUgaXQgYW5kL29yIG1vZGlm
eQojIGl0IHVuZGVyIHRoZSB0ZXJtcyBvZiB0aGUgR05VIEdlbmVyYWwgUHVibGljIExpY2Vuc2Ug
YXMgcHVibGlzaGVkIGJ5CiMgdGhlIEZyZWUgU29mdHdhcmUgRm91bmRhdGlvbiwgZWl0aGVyIHZl
cnNpb24gMyBvZiB0aGUgTGljZW5zZSwgb3IKIyAoYXQgeW91ciBvcHRpb24pIGFueSBsYXRlciB2
ZXJzaW9uLgoKIyBUaGlzIHByb2dyYW0gaXMgZGlzdHJpYnV0ZWQgaW4gdGhlIGhvcGUgdGhhdCBp
dCB3aWxsIGJlIHVzZWZ1bCwKIyBidXQgV0lUSE9VVCBBTlkgV0FSUkFOVFk7IHdpdGhvdXQgZXZl
biB0aGUgaW1wbGllZCB3YXJyYW50eSBvZgojIE1FUkNIQU5UQUJJTElUWSBvciBGSVRORVNTIEZP
UiBBIFBBUlRJQ1VMQVIgUFVSUE9TRS4gIFNlZSB0aGUKIyBHTlUgR2VuZXJhbCBQdWJsaWMgTGlj
ZW5zZSBmb3IgbW9yZSBkZXRhaWxzLgoKIyBZb3Ugc2hvdWxkIGhhdmUgcmVjZWl2ZWQgYSBjb3B5
IG9mIHRoZSBHTlUgR2VuZXJhbCBQdWJsaWMgTGljZW5zZQojIGFsb25nIHdpdGggdGhpcyBwcm9n
cmFtLiAgSWYgbm90LCBzZWUgPGh0dHA6Ly93d3cuZ251Lm9yZy9saWNlbnNlcy8+LgoKLiAiJHtz
cmNkaXI9Ln0vdGVzdHMvaW5pdC5zaCI7IHBhdGhfcHJlcGVuZF8gLi9zcmMKcHJpbnRfdmVyXyBs
cwoKIyBDcmVhdGUgYWxsIHBvc3NpYmxlIDgtYml0IGNoYXJhY3RlcnMgaW4gZmlsZW5hbWVzLgoj
IE5vdGUgdGhhdCwgd2hpbGUgYmFzaCB3b3JrcyBmaW5lLCBkYXNoJ3MgYnVpbHRpbiBwcmludGYg
aXMgdW5hYmxlIHRvCiMgZG8gdGhpcywgc28gd2UgdXNlIC91c3IvYmluL3ByaW50ZiBqdXN0IGlu
IGNhc2UuCmZvciB4IGluICQoc2VxIDAgMjU1KTsgZG8KICAgIHk9JChwcmludGYgIiUwMngiICR4
KQogICAgdG91Y2ggIiQoL3Vzci9iaW4vcHJpbnRmICJiYXJceCR7eX1mb28iKSIgfHwgZWNobyAi
U2tpcHBpbmc6IDB4JHkiCmRvbmUgfHwgZnJhbWV3b3JrX2ZhaWx1cmVfCgojIEluY2x1ZGUgc29t
ZSBzcGVjaWZpYyBmaWxlcyB3ZSBrbm93IHNob3VsZCAob3Igc2hvdWxkIG5vdCkgYmUgcXVvdGVk
Lgpta25vZCAnQ2VjaSBlc3QgdW5lIHBpcGUnIHAgfHwgZnJhbWV3b3JrX2ZhaWx1cmVfCm1rZGly
IDQxMSB8fCBmcmFtZXdvcmtfZmFpbHVyZV8KbG4gLXMgb3Vyb2Jvcm9zIHx8IGZyYW1ld29ya19m
YWlsdXJlXwoKdG91Y2ggJ3llc2JhY2tzbGFzaGJlZm9yZWRvdWJsZXF1b3RlIicgJ3llc2JhY2tz
bGFzaGJlZm9yZWJhY2tzbGFzaFwnCnRvdWNoICdub2JhY2tzbGFzaD0nICdub2JhY2tzbGFzaD4n
ICdub2JhY2tzbGFzaEAnICdub2JhY2tzbGFzaHwnIFwKICAgICAgJ25vYmFja3NsYXNoOicgJ25v
YmFja3NsYXNoICcgJ25vYmFja3NsYXNoKicgJ25vYmFja3NsYXNoPCcKCmNhdCA8PCdFT0YnID5l
eHAgfHwgZnJhbWV3b3JrX2ZhaWx1cmVfCiI0MTEiLwoiQ2VjaSBlc3QgdW5lIHBpcGUifAoiYmFy
XDAwMWZvbyIKImJhclwwMDJmb28iCiJiYXJcMDAzZm9vIgoiYmFyXDAwNGZvbyIKImJhclwwMDVm
b28iCiJiYXJcMDA2Zm9vIgoiYmFyXGFmb28iCiJiYXJcYmZvbyIKImJhclx0Zm9vIgoiYmFyXG5m
b28iCiJiYXJcdmZvbyIKImJhclxmZm9vIgoiYmFyXHJmb28iCiJiYXJcMDE2Zm9vIgoiYmFyXDAx
N2ZvbyIKImJhclwwMjBmb28iCiJiYXJcMDIxZm9vIgoiYmFyXDAyMmZvbyIKImJhclwwMjNmb28i
CiJiYXJcMDI0Zm9vIgoiYmFyXDAyNWZvbyIKImJhclwwMjZmb28iCiJiYXJcMDI3Zm9vIgoiYmFy
XDAzMGZvbyIKImJhclwwMzFmb28iCiJiYXJcMDMyZm9vIgoiYmFyXDAzM2ZvbyIKImJhclwwMzRm
b28iCiJiYXJcMDM1Zm9vIgoiYmFyXDAzNmZvbyIKImJhclwwMzdmb28iCiJiYXIgZm9vIgoiYmFy
IWZvbyIKImJhclwiZm9vIgoiYmFyI2ZvbyIKImJhciRmb28iCiJiYXIlZm9vIgoiYmFyJmZvbyIK
ImJhcidmb28iCiJiYXIoZm9vIgoiYmFyKWZvbyIKImJhcipmb28iCiJiYXIrZm9vIgoiYmFyLGZv
byIKImJhci1mb28iCiJiYXIuZm9vIgoiYmFyMGZvbyIKImJhcjFmb28iCiJiYXIyZm9vIgoiYmFy
M2ZvbyIKImJhcjRmb28iCiJiYXI1Zm9vIgoiYmFyNmZvbyIKImJhcjdmb28iCiJiYXI4Zm9vIgoi
YmFyOWZvbyIKImJhcjpmb28iCiJiYXI7Zm9vIgoiYmFyPGZvbyIKImJhcj1mb28iCiJiYXI+Zm9v
IgoiYmFyP2ZvbyIKImJhckBmb28iCiJiYXJBZm9vIgoiYmFyQmZvbyIKImJhckNmb28iCiJiYXJE
Zm9vIgoiYmFyRWZvbyIKImJhckZmb28iCiJiYXJHZm9vIgoiYmFySGZvbyIKImJhcklmb28iCiJi
YXJKZm9vIgoiYmFyS2ZvbyIKImJhckxmb28iCiJiYXJNZm9vIgoiYmFyTmZvbyIKImJhck9mb28i
CiJiYXJQZm9vIgoiYmFyUWZvbyIKImJhclJmb28iCiJiYXJTZm9vIgoiYmFyVGZvbyIKImJhclVm
b28iCiJiYXJWZm9vIgoiYmFyV2ZvbyIKImJhclhmb28iCiJiYXJZZm9vIgoiYmFyWmZvbyIKImJh
cltmb28iCiJiYXJcXGZvbyIKImJhcl1mb28iCiJiYXJeZm9vIgoiYmFyX2ZvbyIKImJhcmBmb28i
CiJiYXJhZm9vIgoiYmFyYmZvbyIKImJhcmNmb28iCiJiYXJkZm9vIgoiYmFyZWZvbyIKImJhcmZm
b28iCiJiYXJmb28iCiJiYXJnZm9vIgoiYmFyaGZvbyIKImJhcmlmb28iCiJiYXJqZm9vIgoiYmFy
a2ZvbyIKImJhcmxmb28iCiJiYXJtZm9vIgoiYmFybmZvbyIKImJhcm9mb28iCiJiYXJwZm9vIgoi
YmFycWZvbyIKImJhcnJmb28iCiJiYXJzZm9vIgoiYmFydGZvbyIKImJhcnVmb28iCiJiYXJ2Zm9v
IgoiYmFyd2ZvbyIKImJhcnhmb28iCiJiYXJ5Zm9vIgoiYmFyemZvbyIKImJhcntmb28iCiJiYXJ8
Zm9vIgoiYmFyfWZvbyIKImJhcn5mb28iCiJiYXJcMTc3Zm9vIgoiYmFyXDIwMGZvbyIKImJhclwy
MDFmb28iCiJiYXJcMjAyZm9vIgoiYmFyXDIwM2ZvbyIKImJhclwyMDRmb28iCiJiYXJcMjA1Zm9v
IgoiYmFyXDIwNmZvbyIKImJhclwyMDdmb28iCiJiYXJcMjEwZm9vIgoiYmFyXDIxMWZvbyIKImJh
clwyMTJmb28iCiJiYXJcMjEzZm9vIgoiYmFyXDIxNGZvbyIKImJhclwyMTVmb28iCiJiYXJcMjE2
Zm9vIgoiYmFyXDIxN2ZvbyIKImJhclwyMjBmb28iCiJiYXJcMjIxZm9vIgoiYmFyXDIyMmZvbyIK
ImJhclwyMjNmb28iCiJiYXJcMjI0Zm9vIgoiYmFyXDIyNWZvbyIKImJhclwyMjZmb28iCiJiYXJc
MjI3Zm9vIgoiYmFyXDIzMGZvbyIKImJhclwyMzFmb28iCiJiYXJcMjMyZm9vIgoiYmFyXDIzM2Zv
byIKImJhclwyMzRmb28iCiJiYXJcMjM1Zm9vIgoiYmFyXDIzNmZvbyIKImJhclwyMzdmb28iCiJi
YXJcMjQwZm9vIgoiYmFyXDI0MWZvbyIKImJhclwyNDJmb28iCiJiYXJcMjQzZm9vIgoiYmFyXDI0
NGZvbyIKImJhclwyNDVmb28iCiJiYXJcMjQ2Zm9vIgoiYmFyXDI0N2ZvbyIKImJhclwyNTBmb28i
CiJiYXJcMjUxZm9vIgoiYmFyXDI1MmZvbyIKImJhclwyNTNmb28iCiJiYXJcMjU0Zm9vIgoiYmFy
XDI1NWZvbyIKImJhclwyNTZmb28iCiJiYXJcMjU3Zm9vIgoiYmFyXDI2MGZvbyIKImJhclwyNjFm
b28iCiJiYXJcMjYyZm9vIgoiYmFyXDI2M2ZvbyIKImJhclwyNjRmb28iCiJiYXJcMjY1Zm9vIgoi
YmFyXDI2NmZvbyIKImJhclwyNjdmb28iCiJiYXJcMjcwZm9vIgoiYmFyXDI3MWZvbyIKImJhclwy
NzJmb28iCiJiYXJcMjczZm9vIgoiYmFyXDI3NGZvbyIKImJhclwyNzVmb28iCiJiYXJcMjc2Zm9v
IgoiYmFyXDI3N2ZvbyIKImJhclwzMDBmb28iCiJiYXJcMzAxZm9vIgoiYmFyXDMwMmZvbyIKImJh
clwzMDNmb28iCiJiYXJcMzA0Zm9vIgoiYmFyXDMwNWZvbyIKImJhclwzMDZmb28iCiJiYXJcMzA3
Zm9vIgoiYmFyXDMxMGZvbyIKImJhclwzMTFmb28iCiJiYXJcMzEyZm9vIgoiYmFyXDMxM2ZvbyIK
ImJhclwzMTRmb28iCiJiYXJcMzE1Zm9vIgoiYmFyXDMxNmZvbyIKImJhclwzMTdmb28iCiJiYXJc
MzIwZm9vIgoiYmFyXDMyMWZvbyIKImJhclwzMjJmb28iCiJiYXJcMzIzZm9vIgoiYmFyXDMyNGZv
byIKImJhclwzMjVmb28iCiJiYXJcMzI2Zm9vIgoiYmFyXDMyN2ZvbyIKImJhclwzMzBmb28iCiJi
YXJcMzMxZm9vIgoiYmFyXDMzMmZvbyIKImJhclwzMzNmb28iCiJiYXJcMzM0Zm9vIgoiYmFyXDMz
NWZvbyIKImJhclwzMzZmb28iCiJiYXJcMzM3Zm9vIgoiYmFyXDM0MGZvbyIKImJhclwzNDFmb28i
CiJiYXJcMzQyZm9vIgoiYmFyXDM0M2ZvbyIKImJhclwzNDRmb28iCiJiYXJcMzQ1Zm9vIgoiYmFy
XDM0NmZvbyIKImJhclwzNDdmb28iCiJiYXJcMzUwZm9vIgoiYmFyXDM1MWZvbyIKImJhclwzNTJm
b28iCiJiYXJcMzUzZm9vIgoiYmFyXDM1NGZvbyIKImJhclwzNTVmb28iCiJiYXJcMzU2Zm9vIgoi
YmFyXDM1N2ZvbyIKImJhclwzNjBmb28iCiJiYXJcMzYxZm9vIgoiYmFyXDM2MmZvbyIKImJhclwz
NjNmb28iCiJiYXJcMzY0Zm9vIgoiYmFyXDM2NWZvbyIKImJhclwzNjZmb28iCiJiYXJcMzY3Zm9v
IgoiYmFyXDM3MGZvbyIKImJhclwzNzFmb28iCiJiYXJcMzcyZm9vIgoiYmFyXDM3M2ZvbyIKImJh
clwzNzRmb28iCiJiYXJcMzc1Zm9vIgoiYmFyXDM3NmZvbyIKImJhclwzNzdmb28iCiJleHAiCiJu
b2JhY2tzbGFzaCAiCiJub2JhY2tzbGFzaCoiCiJub2JhY2tzbGFzaDoiCiJub2JhY2tzbGFzaDwi
CiJub2JhY2tzbGFzaD0iCiJub2JhY2tzbGFzaD4iCiJub2JhY2tzbGFzaEAiCiJub2JhY2tzbGFz
aHwiCiJvdXJvYm9yb3MiQAoib3V0IgoieWVzYmFja3NsYXNoYmVmb3JlYmFja3NsYXNoXFwiCiJ5
ZXNiYWNrc2xhc2hiZWZvcmVkb3VibGVxdW90ZVwiIgpFT0YKCmxzIC0tY2xhc3NpZnkgLS1xdW90
aW5nLXN0eWxlPWMgPiBvdXQgfHwgZmFpbD0xCgpjb21wYXJlIGV4cCBvdXQgfHwgZmFpbD0xCgpF
eGl0ICRmYWlsCg==
--001a113ac54cf31658056110cc39--

Message sent:

Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Mailer: MIME-tools 5.505 (Entity 5.505)
Content-Type: text/plain; charset=utf-8
X-Loop: help-debbugs@HIDDEN
From: help-debbugs@HIDDEN (GNU bug Tracking System)
To: hackerb9@HIDDEN
Subject: bug#29832: Acknowledgement (Bug in 'ls -FQ': incorrectly quoted
 characters)
Message-ID: <handler.29832.B.151409968311459.ack <at> debbugs.gnu.org>
References: <CAO+cPoMAHZfLthJa55aq2OQjhyPFgZ+bG5ERaA_Q=GsK06wiMw@HIDDEN>
X-Gnu-PR-Message: ack 29832
X-Gnu-PR-Package: coreutils
Reply-To: 29832 <at> debbugs.gnu.org
Date: Sun, 24 Dec 2017 07:15:02 +0000

Thank you for filing a new bug report with debbugs.gnu.org.

This is an automatically generated reply to let you know your message
has been received.

Your message is being forwarded to the package maintainers and other
interested parties for their attention; they will reply in due course.

Your message has been sent to the package maintainer(s):
 bug-coreutils@HIDDEN

If you wish to submit further information on this problem, please
send it to 29832 <at> debbugs.gnu.org.

Please do not send mail to help-debbugs@HIDDEN unless you wish
to report a problem with the Bug-tracking system.

--=20
29832: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D29832
GNU Bug Tracking System
Contact help-debbugs@HIDDEN with problems

Message sent to bug-coreutils@HIDDEN:

Subject: bug#29832: Bug in 'ls -FQ': incorrectly quoted characters
Resent-From: =?UTF-8?Q?P=C3=A1draig?= Brady <P@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-coreutils@HIDDEN
Resent-Date: Sun, 24 Dec 2017 12:27:02 +0000
Resent-Message-ID: <handler.29832.B29832.15141183948592 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: hackerb9@HIDDEN, 29832 <at> debbugs.gnu.org
References: <CAO+cPoMAHZfLthJa55aq2OQjhyPFgZ+bG5ERaA_Q=GsK06wiMw@HIDDEN>
From: =?UTF-8?Q?P=C3=A1draig?= Brady <P@HIDDEN>
Message-ID: <642746a6-f1e1-5546-159f-4c84342677b5@HIDDEN>
Date: Sun, 24 Dec 2017 12:26:30 +0000
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101
 Thunderbird/45.8.0
MIME-Version: 1.0
In-Reply-To: <CAO+cPoMAHZfLthJa55aq2OQjhyPFgZ+bG5ERaA_Q=GsK06wiMw@HIDDEN>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 8bit
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

On 24/12/17 07:14, hackerb9@HIDDEN wrote:
> Hello!
> 
> Thank you for your continued improvements and maintenance of 'ls' in
> coreutils-8.28.
> 
> I have found a somewhat subtle bug. I've included a patch and a test script
> to make sure there are no regressions.
> 
> In short, the problem is that a backlash is incorrectly put in front of the
> characters @, =, >, and |. This only happens when using filetype indicators
> and C style quoting.
> 
>   $ touch @
>   $ ls
>   @
>   $ ls -F
>   @
>   $ ls -Q
>   "@"
>   $ ls -FQ
>   "\@"        # <-- incorrect
> 
> There should be no backslash before the @.
> 
> This was a little bit tricky to track down as the code responsible is not
> commented and appears to have been prematurely optimized by the programmer:
> 
>   ls.c:2141
>   if (file_type <= indicator_style)
>     {
>       char const *p;
>       for (p = &"*=>@|"[indicator_style - file_type]; *p; p++)
>         set_char_quoting (filename_quoting_options, *p, 1);
>     }
> 
> Basically, it says, if --file-type was used, then we should be sure to
> quote '=', '>', and '@'. If --classify is used, then we should do the same
> but also add in '*'. Or, to put it in less compressed C:
> 
>   if (indicator_style == file_type)
>     {
>       char const *p;
>       for (p = "=>@|"; *p; p++)
>         set_char_quoting (filename_quoting_options, *p, 1);
>     }
>   if (indicator_style == classify)
>     {
>       char const *p;
>       for (p = "*=>@|"; *p; p++)
>         set_char_quoting (filename_quoting_options, *p, 1);
>     }
> 
> However, there is no need to quote those characters when the quoting-style
> is set to C. In fact, it is an error to do so as these are not valid C
> escape sequences.
> 
> I am guessing the idea of the code was that those characters would not be
> backslash escaped, but rather the entire filename would be wrapped in
> quotes so the indicator character cannot be confused for part of the
> filename. And, indeed, this does work for some of the quoting styles:
> "shell[-escape][-always]" work that way. Strangely, "c-maybe" also works
> that way.
> 
>   $ ls -F --quoting-style=literal
>   @
>   $ ls -F --quoting-style=shell
>   '@'
>   $ ls -F --quoting-style=shell-always
>   '@'
>   $ ls -F --quoting-style=shell-escape
>   '@'
>   $ ls -F --quoting-style=shell-escape-always
>   '@'
>   $ ls -F --quoting-style=c-maybe
>   "@"        # <-- This is correct!
> 
> I spent some time debugging it, but I didn't find the perfect fix as I
> don't understand why it was written the way it was. Rather, I found
> something that I think is an improvement and a step towards doing things
> right. I've made a patch that simply checks if the quoting style is C and
> skips the problematic lines. Ideally, somebody who understands why this
> code was written this way in the first place can make a better patch, but
> this should work for now and not break anything for anybody else.
> 
> Please find attached below, my patch for src/ls.c and a regression test
> script to be placed in tests/ls/quoting-style-c.sh.
> 
> --b9
> 
> P.S. This bug may also apply to the -b (--quoting-style=escape). However,
> the output from it appears to be more like a mish-mash of C and shell
> escapes. (For example, tabs are shown as "\t", as in C, but spaces are
> shown as "\ ", as in bourne shell. ) The -Q option, on the other hand,
> generates perfectly acceptable C strings (tested with gcc and python),
> except for those few buggy characters fixed in this patch.

Thanks for the detailed analysis.

c-maybe is correct here as quotearg explicitly ignores these extra chars
to be quoted when outer quotes are added.  Therefore we should
probably ignore these extra chars to be quoted when we're adding
outer quotes unconditionally.

This gnulib patch would handle this specific case.
I'll look into generalizing for all outer quoted cases:

diff --git a/lib/quotearg.c b/lib/quotearg.c
index 8e432e1..47ffc5c 100644
--- a/lib/quotearg.c
+++ b/lib/quotearg.c
@@ -709,7 +709,9 @@ quotearg_buffer_restyled (char *buffer, size_t buffersize,
           }
         }

-      if (! (((backslash_escapes && quoting_style != shell_always_quoting_style)
+      if (! (((backslash_escapes
+               && quoting_style != shell_always_quoting_style
+               && quoting_style != c_quoting_style)
               || elide_outer_quotes)
              && quote_these_too
              && quote_these_too[c / INT_BITS] >> (c % INT_BITS) & 1)

cheers,
P�draig

Message received at control <at> debbugs.gnu.org:

Received: (at control) by debbugs.gnu.org; 30 Oct 2018 02:22:58 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Oct 29 22:22:58 2018
Received: from localhost ([127.0.0.1]:52621 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gHJgI-0003gd-6S
	for submit <at> debbugs.gnu.org; Mon, 29 Oct 2018 22:22:58 -0400
Received: from mail-pg1-f169.google.com ([209.85.215.169]:39841)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <assafgordon@HIDDEN>) id 1gHJgG-0003gR-BM
 for control <at> debbugs.gnu.org; Mon, 29 Oct 2018 22:22:56 -0400
Received: by mail-pg1-f169.google.com with SMTP id r9-v6so4839799pgv.6
 for <control <at> debbugs.gnu.org>; Mon, 29 Oct 2018 19:22:56 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=to:from:message-id:date:user-agent:mime-version:content-language
 :content-transfer-encoding;
 bh=E/8gu7d7RP3Ig0ZQOU+aUvoPe6sCGg9GXhpgnbvSA74=;
 b=ka4bg4mQssEVF3sSoiCl0WMMivUgXMxjHFe990BnG3JOPGQwH7b+bjDO3ArZFsDNtV
 rMWrYnxaf2uKyDel4ELpnpC/iyVJNSw72dKAQza+qoKkPJuL+7CKf6QnA/N85oy2nAM+
 XcpIIBudNcsy/WvjAj3JJBFwNf2U/MgT1QdrwEBDleRGTqPOs4meOngrfH/wGLyWaepe
 vXdqCqZXXWiJ2vqja3jlzFihadfcbQFJBxgie/w8isD1ySLmEyOD3Rht0wUWqeVRw/+2
 Rkz34yBrcb2XO6SfmrbaVjWsF5d2NrjZlPc+i0VArBqinRQZBaX9NvSng6yI3ZEGSrnI
 MoXg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:to:from:message-id:date:user-agent:mime-version
 :content-language:content-transfer-encoding;
 bh=E/8gu7d7RP3Ig0ZQOU+aUvoPe6sCGg9GXhpgnbvSA74=;
 b=E46dyzUA67fvv3h0rnHNSWw0vJDlhgzb2B4a4385gv7LZWedEjCHiblI4WHOj0HVcV
 SP/ruDLNyTVZKLuGpRuvB6jtRG2PPfh4w6C+1bjsFiZ8eL307vbFcG7Bm5Q+SRR4OqSf
 JK8tUOv5fxLQxxj+XmlVustMVAegSbN3FveXjVg4hOO1gDQPDV857TBVA/GyKaArcSgT
 6kg8Hvpg56qDxlf2gc+6bzyQ9JI+Hqbdt7PMj9c7PRUDeOEQSnUXaSbOak108QTJCiLU
 hfbBiv4nqnQRDWicsGIo49PPKO6GwyvbV6ZJgQH3UdUQqyONiivDRFjIEVtX+WU4JJZ2
 7PHQ==
X-Gm-Message-State: AGRZ1gJa08g8bFc8Deacb0899cDtiUppnbWyrXqH3RM17Ov/6J9WNrN3
 PlO0RHNUqGVyYr9UYpA3u+U9xUo8mdI=
X-Google-Smtp-Source: AJdET5cLvR0RTwCy9YVFVjIy+fYbKQdJ9t9IkMP3uQaIllDCeOzA21iewcg4QCdZmd5DpBg22YuAsQ==
X-Received: by 2002:a63:f501:: with SMTP id
 w1-v6mr15409464pgh.336.1540866170169; 
 Mon, 29 Oct 2018 19:22:50 -0700 (PDT)
Received: from tomato.housegordon.com (moose.housegordon.com. [184.68.105.38])
 by smtp.googlemail.com with ESMTPSA id
 o12-v6sm32969462pfh.20.2018.10.29.19.22.48
 for <control <at> debbugs.gnu.org>
 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);
 Mon, 29 Oct 2018 19:22:49 -0700 (PDT)
To: control <at> debbugs.gnu.org
From: Assaf Gordon <assafgordon@HIDDEN>
Message-ID: <b7095b67-b2fc-1791-e2ad-ce7e8c10c3cb@HIDDEN>
Date: Mon, 29 Oct 2018 20:22:47 -0600
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101
 Thunderbird/60.2.1
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
X-Spam-Score: 2.0 (++)
X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org",
 has NOT identified this incoming email as spam.  The original
 message has been attached to this so you can view it or label
 similar future email.  If you have any questions, see
 the administrator of that system for details.
 Content preview: retitle 29832 ls: bug in 'ls -FQ': incorrectly quoted
 characters [...] 
 Content analysis details:   (2.0 points, 10.0 required)
 pts rule name              description
 ---- ---------------------- --------------------------------------------------
 -0.0 RCVD_IN_DNSWL_NONE     RBL: Sender listed at http://www.dnswl.org/, no
 trust [209.85.215.169 listed in list.dnswl.org]
 -0.0 SPF_PASS               SPF: sender matches SPF record
 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider
 (assafgordon[at]gmail.com)
 0.0 RCVD_IN_MSPIKE_H3      RBL: Good reputation (+3)
 [209.85.215.169 listed in wl.mailspike.net]
 0.0 RCVD_IN_MSPIKE_WL      Mailspike good senders
 1.8 MISSING_SUBJECT        Missing Subject: header
 0.2 NO_SUBJECT             Extra score for no subject
X-Debbugs-Envelope-To: control
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: 1.0 (+)

retitle 29832 ls: bug in 'ls -FQ': incorrectly quoted characters

Last modified: Mon, 25 Nov 2019 12:00:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.