GNU logs - #28180, boring messages

Message sent to bug-gnu-emacs@HIDDEN:

Subject: bug#28180: [w32] Unicode characters in subprocess (git) arguments changed to space
Resent-From: npostavs@HIDDEN
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Tue, 22 Aug 2017 02:35:02 +0000
Resent-Message-ID: <handler.28180.B.150336924213797 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: 28180 <at> debbugs.gnu.org
From: npostavs@HIDDEN
Date: Mon, 21 Aug 2017 22:35:24 -0400
Message-ID: <87d17oba8j.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="=-=-="
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

--=-=-=
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

In w32.c there is a comment saying

   . Running subprocesses in non-ASCII directories and with non-ASCII
     file arguments is limited to the current codepage [...]
     This should be fixed, but will also require changes in cmdproxy.
     The current limitation is not terribly bad anyway, since very
     few, if any, Windows console programs that are likely to be
     invoked by Emacs support UTF-16 encoded command lines.

I believe we're running into this limitation with git: staging a file
named =E5=A5=BD.txt fails from magit[1] (I tried also with vc, same problem=
).
A quick way to see the problem is evaluating the call-process form
below, the output shows that the Chinese character has been transformed
into a space.  This happens even if I execute 'chcp 65001' before
starting Emacs (a possible workaround I saw suggested in a few places).
The short file name trick doesn't help either.

(call-process "git" nil '(t t) nil
              "-c" "alias.x=3D!x() { printf '%s' \"$1\" | od -tx1; }; x" "x=
" "(=E5=A5=BD)")
0000000 28 20 29
0000003

As far as I can tell, git does support UTF-16 encoded command lines, as
demonstrated by the attached git-args.c, which produces the utf8
encoding of the character (this is also what the call-process form
produces when I run it on GNU/Linux):

C:\Users\npostavs\src\win32-args>.\git-args.exe
0000000 28 e5 a5 bd 29
0000005


--=-=-=
Content-Type: text/plain
Content-Disposition: attachment; filename=git-args.c
Content-Description: test program, C source

#include <windows.h>
#include <stdio.h>

int main(int argc, char **argv)
{
    STARTUPINFOW wsi;
    PROCESS_INFORMATION pi;

    WCHAR wcmdline[] = L"git -c \"alias.x=!x() { printf '%s' \\\"$1\\\" | od -tx1; }; x\" x (\x597d)";

    ZeroMemory( &wsi, sizeof(wsi) );
    wsi.cb = sizeof(wsi);
    ZeroMemory( &pi, sizeof(pi) );

    if (!CreateProcessW(NULL, wcmdline,
            NULL, NULL, // proc, thread attrs
            FALSE, // handle inheritance
            0, // creation flags
            NULL, // env
            NULL, // cur dir
            &wsi,
            &pi)) {
        printf( "CreateProcess failed (%d).\n", GetLastError() );
        return 1;
    }

    // Wait until child process exits.
    WaitForSingleObject( pi.hProcess, INFINITE );

    // Close process and thread handles. 
    CloseHandle( pi.hProcess );
    CloseHandle( pi.hThread );
    return 0;
}

--=-=-=
Content-Type: text/plain


Am I correct that this problem is related the w32.c comment?  It's not
clear to me what changes are needed in cmdproxy (and other places?) to
address it.

[1]: https://github.com/magit/magit/issues/3111

--=-=-=--

Message sent:

Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Mailer: MIME-tools 5.505 (Entity 5.505)
Content-Type: text/plain; charset=utf-8
X-Loop: help-debbugs@HIDDEN
From: help-debbugs@HIDDEN (GNU bug Tracking System)
To: npostavs@HIDDEN
Subject: bug#28180: Acknowledgement ([w32] Unicode characters in
 subprocess (git) arguments changed to space)
Message-ID: <handler.28180.B.150336924213797.ack <at> debbugs.gnu.org>
References: <87d17oba8j.fsf@HIDDEN>
X-Gnu-PR-Message: ack 28180
X-Gnu-PR-Package: emacs
Reply-To: 28180 <at> debbugs.gnu.org
Date: Tue, 22 Aug 2017 02:35:02 +0000

Thank you for filing a new bug report with debbugs.gnu.org.

This is an automatically generated reply to let you know your message
has been received.

Your message is being forwarded to the package maintainers and other
interested parties for their attention; they will reply in due course.

Your message has been sent to the package maintainer(s):
 bug-gnu-emacs@HIDDEN

If you wish to submit further information on this problem, please
send it to 28180 <at> debbugs.gnu.org.

Please do not send mail to help-debbugs@HIDDEN unless you wish
to report a problem with the Bug-tracking system.

--=20
28180: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D28180
GNU Bug Tracking System
Contact help-debbugs@HIDDEN with problems

Message sent to bug-gnu-emacs@HIDDEN:

Subject: bug#28180: [w32] Unicode characters in subprocess (git) arguments changed to space
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Tue, 22 Aug 2017 14:56:02 +0000
Resent-Message-ID: <handler.28180.B28180.150341371330084 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: npostavs@HIDDEN
Cc: 28180 <at> debbugs.gnu.org
Reply-To: Eli Zaretskii <eliz@HIDDEN>
Date: Tue, 22 Aug 2017 17:54:59 +0300
Message-Id: <83y3qbabzw.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-reply-to: <87d17oba8j.fsf@HIDDEN>
 (npostavs@HIDDEN)
References: <87d17oba8j.fsf@HIDDEN>
MIME-version: 1.0
Content-type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

> From: npostavs@HIDDEN
> Date: Mon, 21 Aug 2017 22:35:24 -0400
> 
> In w32.c there is a comment saying
> 
>    . Running subprocesses in non-ASCII directories and with non-ASCII
>      file arguments is limited to the current codepage [...]
>      This should be fixed, but will also require changes in cmdproxy.
>      The current limitation is not terribly bad anyway, since very
>      few, if any, Windows console programs that are likely to be
>      invoked by Emacs support UTF-16 encoded command lines.
> 
> I believe we're running into this limitation with git: staging a file
> named 好.txt fails from magit[1] (I tried also with vc, same problem).
> A quick way to see the problem is evaluating the call-process form
> below, the output shows that the Chinese character has been transformed
> into a space.

I'd expect that in a non-Chinese locale (which I believe was what you
did), but the OP of the Magit issue has Windows set up for a Chinese
locale, so there has to be some other explanation, because passing
Chinese characters on the command line ought to work in that case.

> Am I correct that this problem is related the w32.c comment?

The comment is accurate, but it can only explain why command-line
arguments with characters outside of the current Windows locale cannot
be safely passed to sub-processes.  Which AFAIU is not the case with
the OP of that Magit issue.

> It's not clear to me what changes are needed in cmdproxy (and other
> places?) to address it.

cmdproxy is not involved in call-process, but it is involved in
shell-command and its ilk.  As it makes no sense to support Unicode in
the former, but not in the latter, if we want to lift this limitation,
we must teach cmdproxy to use "wide" APIs both for receiving
command-line arguments from Emacs and for passing them to programs it
invokes.

As to the "other places", the only problem I'm aware of is that the
encoding of the command-line arguments, when they arrive at w32proc.c,
is not known in advance, so this must be somehow fixed/changed,
otherwise we will be unable to re-encode them in UTF-16.  I believe
the comment in w32.c does mention that.

Message sent to bug-gnu-emacs@HIDDEN:

Subject: bug#28180: [w32] Unicode characters in subprocess (git) arguments changed to space
Resent-From: Noam Postavsky <npostavs@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Mon, 28 Aug 2017 14:43:01 +0000
Resent-Message-ID: <handler.28180.B28180.150393134224474 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: Eli Zaretskii <eliz@HIDDEN>
Cc: 28180 <at> debbugs.gnu.org
MIME-Version: 1.0
In-Reply-To: <83y3qbabzw.fsf@HIDDEN>
References: <87d17oba8j.fsf@HIDDEN> <83y3qbabzw.fsf@HIDDEN>
From: Noam Postavsky <npostavs@HIDDEN>
Date: Mon, 28 Aug 2017 10:42:14 -0400
Message-ID: <CAM-tV--3p_JKL5d+65BvMR2pMS97wfH9-ZVNW3DR2tPf4AObJQ@HIDDEN>
Content-Type: multipart/mixed; boundary="001a11c183aa03278b0557d14df9"
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

--001a11c183aa03278b0557d14df9
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Tue, Aug 22, 2017 at 10:54 AM, Eli Zaretskii <eliz@HIDDEN> wrote:

> As to the "other places", the only problem I'm aware of is that the
> encoding of the command-line arguments, when they arrive at w32proc.c,
> is not known in advance, so this must be somehow fixed/changed,
> otherwise we will be unable to re-encode them in UTF-16.  I believe
> the comment in w32.c does mention that.

Just to understand the issue better, I applied the attached diff to
use CreateProcessW. It seemed to work, but only when I start emacs
from mingw's msys shell. When running from cmd.exe it still translates
to space.

Furthermore, when I run an unpatched Emacs from the msys shell, the
output of the test I posted above is different:

(call-process "git" nil '(t t) nil
              "-c" "alias.x=3D!x() { printf '%s' \"$1\" | od -tx1; }; x"
"x" "(=E5=A5=BD)")
0000000 28 c3 a5 c2 a5 c2 bd 29
0000010

Do you have any idea what setting could cause this?

--001a11c183aa03278b0557d14df9
Content-Type: text/plain; charset="US-ASCII"; name="CreateProcessW.diff"
Content-Disposition: attachment; filename="CreateProcessW.diff"
Content-Transfer-Encoding: base64
X-Attachment-Id: f_j6w9ncre0

ZGlmZiAtLWdpdCBjL3NyYy9wcm9jZXNzLmMgaS9zcmMvcHJvY2Vzcy5jCmluZGV4IGU3ZWU5OWEu
LjJjMTdiNWEgMTAwNjQ0Ci0tLSBjL3NyYy9wcm9jZXNzLmMKKysrIGkvc3JjL3Byb2Nlc3MuYwpA
QCAtMTg4MSw4ICsxODgxLDcgQEAgdXNhZ2U6IChtYWtlLXByb2Nlc3MgJnJlc3QgQVJHUykgICov
KQogCSAgaWYgKFNUUklOR19NVUxUSUJZVEUgKGFyZykpCiAJICAgIHsKIAkgICAgICBpZiAoTklM
UCAoYXJnX2VuY29kaW5nKSkKLQkJYXJnX2VuY29kaW5nID0gKGNvbXBsZW1lbnRfcHJvY2Vzc19l
bmNvZGluZ19zeXN0ZW0KLQkJCQkoWFBST0NFU1MgKHByb2MpLT5lbmNvZGVfY29kaW5nX3N5c3Rl
bSkpOworCQlhcmdfZW5jb2RpbmcgPSBRdXRmXzg7CiAJICAgICAgYXJnID0gY29kZV9jb252ZXJ0
X3N0cmluZ19ub3JlY29yZCAoYXJnLCBhcmdfZW5jb2RpbmcsIDEpOwogCSAgICB9CiAJICB0ZW0g
PSBGY29ucyAoYXJnLCB0ZW0pOwpkaWZmIC0tZ2l0IGMvc3JjL3czMnByb2MuYyBpL3NyYy93MzJw
cm9jLmMKaW5kZXggNzZhZjU1Zi4uODZhYWVlMiAxMDA2NDQKLS0tIGMvc3JjL3czMnByb2MuYwor
KysgaS9zcmMvdzMycHJvYy5jCkBAIC0xMjA0LDE0ICsxMjA0LDEzIEBAIHN0YXRpYyBCT09MCiBj
cmVhdGVfY2hpbGQgKGNoYXIgKmV4ZSwgY2hhciAqY21kbGluZSwgY2hhciAqZW52LCBpbnQgaXNf
Z3VpX2FwcCwKIAkgICAgICBwaWRfdCAqIHBQaWQsIGNoaWxkX3Byb2Nlc3MgKmNwKQogewotICBT
VEFSVFVQSU5GTyBzdGFydDsKKyAgU1RBUlRVUElORk9XIHN0YXJ0OwogICBTRUNVUklUWV9BVFRS
SUJVVEVTIHNlY19hdHRyczsKICNpZiAwCiAgIFNFQ1VSSVRZX0RFU0NSSVBUT1Igc2VjX2Rlc2M7
CiAjZW5kaWYKICAgRFdPUkQgZmxhZ3M7CiAgIGNoYXIgZGlyWyBNQVhfUEFUSCBdOwotICBjaGFy
ICpwOwogICBjb25zdCBjaGFyICpleHQ7CiAKICAgaWYgKGNwID09IE5VTEwpIGVtYWNzX2Fib3J0
ICgpOwpAQCAtMTI0MiwxNCArMTI0MSw4IEBAIGNyZWF0ZV9jaGlsZCAoY2hhciAqZXhlLCBjaGFy
ICpjbWRsaW5lLCBjaGFyICplbnYsIGludCBpc19ndWlfYXBwLAogICBzZWNfYXR0cnMubHBTZWN1
cml0eURlc2NyaXB0b3IgPSBOVUxMIC8qICZzZWNfZGVzYyAqLzsKICAgc2VjX2F0dHJzLmJJbmhl
cml0SGFuZGxlID0gRkFMU0U7CiAKLSAgZmlsZW5hbWVfdG9fYW5zaSAocHJvY2Vzc19kaXIsIGRp
cik7Ci0gIC8qIENhbid0IHVzZSB1bml4dG9kb3NfZmlsZW5hbWUgaGVyZSwgc2luY2UgdGhhdCBu
ZWVkcyBpdHMgZmlsZSBuYW1lCi0gICAgIGFyZ3VtZW50IGVuY29kZWQgaW4gVVRGLTguICBPVE9I
LCBwcm9jZXNzX2Rpciwgd2hpY2ggX2lzXyBpbgotICAgICBVVEYtOCwgcG9pbnRzLCB0byB0aGUg
ZGlyZWN0b3J5IGNvbXB1dGVkIGJ5IG91ciBjYWxsZXIsIGFuZCB3ZQotICAgICBkb24ndCB3YW50
IHRvIG1vZGlmeSB0aGF0LCBlaXRoZXIuICAqLwotICBmb3IgKHAgPSBkaXI7ICpwOyBwID0gQ2hh
ck5leHRBIChwKSkKLSAgICBpZiAoKnAgPT0gJy8nKQotICAgICAgKnAgPSAnXFwnOworICBzdHJj
cHkgKGRpciwgcHJvY2Vzc19kaXIpOworICB1bml4dG9kb3NfZmlsZW5hbWUgKGRpcik7CiAKICAg
LyogQ3JlYXRlUHJvY2VzcyBoYW5kbGVzIGJhdGNoIGZpbGVzIGFzIGV4ZSBzcGVjaWFsbHkuICBU
aGlzIHNwZWNpYWwKICAgICAgaGFuZGxpbmcgZmFpbHMgd2hlbiBib3RoIHRoZSBiYXRjaCBmaWxl
IGFuZCBhcmd1bWVudHMgYXJlIHF1b3RlZC4KQEAgLTEyNjUsOCArMTI1OCwxMyBAQCBjcmVhdGVf
Y2hpbGQgKGNoYXIgKmV4ZSwgY2hhciAqY21kbGluZSwgY2hhciAqZW52LCBpbnQgaXNfZ3VpX2Fw
cCwKIAkgICA6IENSRUFURV9ORVdfQ09OU09MRSk7CiAgIGlmIChOSUxQIChWdzMyX3N0YXJ0X3By
b2Nlc3NfaW5oZXJpdF9lcnJvcl9tb2RlKSkKICAgICBmbGFncyB8PSBDUkVBVEVfREVGQVVMVF9F
UlJPUl9NT0RFOwotICBpZiAoIUNyZWF0ZVByb2Nlc3NBIChleGUsIGNtZGxpbmUsICZzZWNfYXR0
cnMsIE5VTEwsIFRSVUUsCi0JCSAgICAgICBmbGFncywgZW52LCBkaXIsICZzdGFydCwgJmNwLT5w
cm9jaW5mbykpCisKKyAgd2NoYXJfdCBleGVXW01BWF9QQVRIXSwgY21kbGluZVdbTUFYX1BBVEhd
LCBkaXJXW01BWF9QQVRIXTsKKyAgZmlsZW5hbWVfdG9fdXRmMTYgKGV4ZSwgZXhlVyk7CisgIGZp
bGVuYW1lX3RvX3V0ZjE2IChjbWRsaW5lLCBjbWRsaW5lVyk7CisgIGZpbGVuYW1lX3RvX3V0ZjE2
IChkaXIsIGRpclcpOworICBpZiAoIUNyZWF0ZVByb2Nlc3NXIChleGVXLCBjbWRsaW5lVywgJnNl
Y19hdHRycywgTlVMTCwgVFJVRSwKKyAgICAgICAgICAgICAgICAgICAgICAgZmxhZ3MsIGVudiwg
ZGlyVywgJnN0YXJ0LCAmY3AtPnByb2NpbmZvKSkKICAgICBnb3RvIEVIX0ZhaWw7CiAKICAgY3At
PnBpZCA9IChpbnQpIGNwLT5wcm9jaW5mby5kd1Byb2Nlc3NJZDsK
--001a11c183aa03278b0557d14df9--

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#28180: [w32] Unicode characters in subprocess (git) arguments changed to space
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Mon, 28 Aug 2017 17:17:02 +0000
Resent-Message-ID: <handler.28180.B28180.150394057310378 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 28180
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Noam Postavsky <npostavs@HIDDEN>
Cc: 28180 <at> debbugs.gnu.org
Reply-To: Eli Zaretskii <eliz@HIDDEN>
Received: via spool by 28180-submit <at> debbugs.gnu.org id=B28180.150394057310378
          (code B ref 28180); Mon, 28 Aug 2017 17:17:02 +0000
Received: (at 28180) by debbugs.gnu.org; 28 Aug 2017 17:16:13 +0000
Received: from localhost ([127.0.0.1]:59600 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1dmNe0-0002h2-Go
	for submit <at> debbugs.gnu.org; Mon, 28 Aug 2017 13:16:12 -0400
Received: from eggs.gnu.org ([208.118.235.92]:35897)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1dmNdz-0002at-3R
 for 28180 <at> debbugs.gnu.org; Mon, 28 Aug 2017 13:16:11 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <eliz@HIDDEN>) id 1dmNdp-0000CF-V0
 for 28180 <at> debbugs.gnu.org; Mon, 28 Aug 2017 13:16:05 -0400
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=-0.5 required=5.0 tests=BAYES_05,RP_MATCHES_RCVD
 autolearn=disabled version=3.3.2
Received: from fencepost.gnu.org ([2001:4830:134:3::e]:49486)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>)
 id 1dmNdp-0000Bt-RY; Mon, 28 Aug 2017 13:16:01 -0400
Received: from 84.94.185.246.cable.012.net.il ([84.94.185.246]:4267
 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1dmNdo-0002US-1V; Mon, 28 Aug 2017 13:16:01 -0400
Date: Mon, 28 Aug 2017 20:15:46 +0300
Message-Id: <83bmmz4nr1.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-reply-to: <CAM-tV--3p_JKL5d+65BvMR2pMS97wfH9-ZVNW3DR2tPf4AObJQ@HIDDEN>
 (message from Noam Postavsky on Mon, 28 Aug 2017 10:42:14 -0400)
References: <87d17oba8j.fsf@HIDDEN> <83y3qbabzw.fsf@HIDDEN>
 <CAM-tV--3p_JKL5d+65BvMR2pMS97wfH9-ZVNW3DR2tPf4AObJQ@HIDDEN>
MIME-version: 1.0
Content-type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2001:4830:134:3::e
X-Spam-Score: -5.0 (-----)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -5.0 (-----)

> From: Noam Postavsky <npostavs@HIDDEN>
> Date: Mon, 28 Aug 2017 10:42:14 -0400
> Cc: 28180 <at> debbugs.gnu.org
> 
> Just to understand the issue better, I applied the attached diff to
> use CreateProcessW.

I hope you realize that this is just a quick hack, which cannot work
in general, yes?  For starters, the command line is not a file name,
in general, so using filename_to_utf16 is inappropriate.  Also, I
think the environment variables need to be converted to UTF-16.

> It seemed to work, but only when I start emacs from mingw's msys
> shell. When running from cmd.exe it still translates to space.

What exactly did you run from cmd.exe?  What command?

> Furthermore, when I run an unpatched Emacs from the msys shell, the
> output of the test I posted above is different:
> 
> (call-process "git" nil '(t t) nil
>               "-c" "alias.x=!x() { printf '%s' \"$1\" | od -tx1; }; x"
> "x" "(好)")
> 0000000 28 c3 a5 c2 a5 c2 bd 29
> 0000010
> 
> Do you have any idea what setting could cause this?

Windows tries to interpret UTF-8 as something else?

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#28180: [w32] Unicode characters in subprocess (git) arguments changed to space
Resent-From: Noam Postavsky <npostavs@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Tue, 29 Aug 2017 22:07:02 +0000
Resent-Message-ID: <handler.28180.B28180.150404437223985 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 28180
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: 28180 <at> debbugs.gnu.org
Received: via spool by 28180-submit <at> debbugs.gnu.org id=B28180.150404437223985
          (code B ref 28180); Tue, 29 Aug 2017 22:07:02 +0000
Received: (at 28180) by debbugs.gnu.org; 29 Aug 2017 22:06:12 +0000
Received: from localhost ([127.0.0.1]:32986 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1dmoeB-0006Em-C5
	for submit <at> debbugs.gnu.org; Tue, 29 Aug 2017 18:06:11 -0400
Received: from mail-oi0-f41.google.com ([209.85.218.41]:33849)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <npostavs@HIDDEN>) id 1dmoe8-0006EY-5h
 for 28180 <at> debbugs.gnu.org; Tue, 29 Aug 2017 18:06:08 -0400
Received: by mail-oi0-f41.google.com with SMTP id w10so38415319oie.1
 for <28180 <at> debbugs.gnu.org>; Tue, 29 Aug 2017 15:06:08 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=mime-version:sender:in-reply-to:references:from:date:message-id
 :subject:to:cc;
 bh=bRpN0mkKHMuYEgR+TarcpOXLPoF2AxiXOcSdOQtuHME=;
 b=gPd1gqJtr0vmtDqhsya77fHrWgqHtMCnUF2JrYNwWGBfPg1OnIdk/I+i6UEWzyjNl0
 XGgIJOL+joT3ZBtA+z1wTeVR/sTMC+syjsE08U7m1qB35ptZkFy8VjL6JcMt14vss3nh
 6HC9hCQ5Qr25GQHNjpn8v5dbwG04MPlXCX13rtebXA6FX2FSOOIoElVXX30scn05Tabw
 azLy6EKP+thjr+SDO1lkMZh96VlsVwm/MP1+G0Toy7Y5kZ7OsRDtbLB1e6PEnhq9Mr0+
 RL0PJFshqX5TnzALGy5fFXhNRExUOGOP9CuUs0fwBukdPiegf9VeIlzgpr9S1+S76rFj
 Qrbw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:mime-version:sender:in-reply-to:references:from
 :date:message-id:subject:to:cc;
 bh=bRpN0mkKHMuYEgR+TarcpOXLPoF2AxiXOcSdOQtuHME=;
 b=LZbC+7dmaKIO1fAz/+1kKcEUcRghcByt7g/dNfxNJ9exDgQ3gWt5t2gKbvcD98oXht
 w7Rq9QodSubp66E+TzFy5PVT/1KcJVTPzgomNBAUUy8UnHCTUeesenIOdSS8JN75RS7J
 V1+hYKbNexg4fwHQyiU2B4fw7t9VRiCyOi2jnKJcGtgfL6UdaFZuon5SQGuLP2UCnN2H
 mnE5/COPOSKyEi/VSG8TjJmeG///taW+iw9rrPRh+cHZlTwhYpZrjb4f/uzzjPSXIRKd
 pQkxhBHDVFCSLY4wF/w/gn2jG37lWunC3unVW87wH2mfQ/430Ak63fVTmvgGQ9imRLuc
 yhGw==
X-Gm-Message-State: AHYfb5gHkI51a/nhIy2RFwGMGCjWuKFhBgPMaR36HbKW4PzPRAFqpZNI
 V25VmdJFR5i/jB/98MHswRKO6PbOPA==
X-Received: by 10.202.117.78 with SMTP id q75mr1530210oic.247.1504044362534;
 Tue, 29 Aug 2017 15:06:02 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.74.17.199 with HTTP; Tue, 29 Aug 2017 15:06:01 -0700 (PDT)
In-Reply-To: <CAM-tV--3p_JKL5d+65BvMR2pMS97wfH9-ZVNW3DR2tPf4AObJQ@HIDDEN>
References: <87d17oba8j.fsf@HIDDEN> <83y3qbabzw.fsf@HIDDEN>
 <CAM-tV--3p_JKL5d+65BvMR2pMS97wfH9-ZVNW3DR2tPf4AObJQ@HIDDEN>
From: Noam Postavsky <npostavs@HIDDEN>
Date: Tue, 29 Aug 2017 18:06:01 -0400
X-Google-Sender-Auth: e2zKye2abO5_nD7p02jsrFa7mfs
Message-ID: <CAM-tV-8bovY45Mo3Ma8_gXCKQFt12kaPiCo81dU7u4g8xfSw-w@HIDDEN>
Content-Type: text/plain; charset="UTF-8"
X-Spam-Score: 0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: 0.7 (/)

On Mon, Aug 28, 2017 at 10:42 AM, Noam Postavsky
<npostavs@HIDDEN> wrote:

> Just to understand the issue better, I applied the attached diff to
> use CreateProcessW. It seemed to work, but only when I start emacs
> from mingw's msys shell. When running from cmd.exe it still translates
> to space.

Ugh, it's just a stupid mistake on my part, I changed the argument
encoding to utf8 in make-process instead of call-process. It
*appeared* to work because msys sets LANG to en_US.UTF-8.

Last modified: Mon, 25 Nov 2019 12:00:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.