GNU bug report logs - #28180
[w32] Unicode characters in subprocess (git) arguments changed to space

Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.
Package: emacs; Reported by: npostavs@HIDDEN; dated Tue, 22 Aug 2017 02:35:02 UTC; Maintainer for emacs is bug-gnu-emacs@HIDDEN.

Message received at 28180 <at> debbugs.gnu.org:


Received: (at 28180) by debbugs.gnu.org; 29 Aug 2017 22:06:12 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Aug 29 18:06:12 2017
Received: from localhost ([127.0.0.1]:32986 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1dmoeB-0006Em-C5
	for submit <at> debbugs.gnu.org; Tue, 29 Aug 2017 18:06:11 -0400
Received: from mail-oi0-f41.google.com ([209.85.218.41]:33849)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <npostavs@HIDDEN>) id 1dmoe8-0006EY-5h
 for 28180 <at> debbugs.gnu.org; Tue, 29 Aug 2017 18:06:08 -0400
Received: by mail-oi0-f41.google.com with SMTP id w10so38415319oie.1
 for <28180 <at> debbugs.gnu.org>; Tue, 29 Aug 2017 15:06:08 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=mime-version:sender:in-reply-to:references:from:date:message-id
 :subject:to:cc;
 bh=bRpN0mkKHMuYEgR+TarcpOXLPoF2AxiXOcSdOQtuHME=;
 b=gPd1gqJtr0vmtDqhsya77fHrWgqHtMCnUF2JrYNwWGBfPg1OnIdk/I+i6UEWzyjNl0
 XGgIJOL+joT3ZBtA+z1wTeVR/sTMC+syjsE08U7m1qB35ptZkFy8VjL6JcMt14vss3nh
 6HC9hCQ5Qr25GQHNjpn8v5dbwG04MPlXCX13rtebXA6FX2FSOOIoElVXX30scn05Tabw
 azLy6EKP+thjr+SDO1lkMZh96VlsVwm/MP1+G0Toy7Y5kZ7OsRDtbLB1e6PEnhq9Mr0+
 RL0PJFshqX5TnzALGy5fFXhNRExUOGOP9CuUs0fwBukdPiegf9VeIlzgpr9S1+S76rFj
 Qrbw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:mime-version:sender:in-reply-to:references:from
 :date:message-id:subject:to:cc;
 bh=bRpN0mkKHMuYEgR+TarcpOXLPoF2AxiXOcSdOQtuHME=;
 b=LZbC+7dmaKIO1fAz/+1kKcEUcRghcByt7g/dNfxNJ9exDgQ3gWt5t2gKbvcD98oXht
 w7Rq9QodSubp66E+TzFy5PVT/1KcJVTPzgomNBAUUy8UnHCTUeesenIOdSS8JN75RS7J
 V1+hYKbNexg4fwHQyiU2B4fw7t9VRiCyOi2jnKJcGtgfL6UdaFZuon5SQGuLP2UCnN2H
 mnE5/COPOSKyEi/VSG8TjJmeG///taW+iw9rrPRh+cHZlTwhYpZrjb4f/uzzjPSXIRKd
 pQkxhBHDVFCSLY4wF/w/gn2jG37lWunC3unVW87wH2mfQ/430Ak63fVTmvgGQ9imRLuc
 yhGw==
X-Gm-Message-State: AHYfb5gHkI51a/nhIy2RFwGMGCjWuKFhBgPMaR36HbKW4PzPRAFqpZNI
 V25VmdJFR5i/jB/98MHswRKO6PbOPA==
X-Received: by 10.202.117.78 with SMTP id q75mr1530210oic.247.1504044362534;
 Tue, 29 Aug 2017 15:06:02 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.74.17.199 with HTTP; Tue, 29 Aug 2017 15:06:01 -0700 (PDT)
In-Reply-To: <CAM-tV--3p_JKL5d+65BvMR2pMS97wfH9-ZVNW3DR2tPf4AObJQ@HIDDEN>
References: <87d17oba8j.fsf@HIDDEN> <83y3qbabzw.fsf@HIDDEN>
 <CAM-tV--3p_JKL5d+65BvMR2pMS97wfH9-ZVNW3DR2tPf4AObJQ@HIDDEN>
From: Noam Postavsky <npostavs@HIDDEN>
Date: Tue, 29 Aug 2017 18:06:01 -0400
X-Google-Sender-Auth: e2zKye2abO5_nD7p02jsrFa7mfs
Message-ID: <CAM-tV-8bovY45Mo3Ma8_gXCKQFt12kaPiCo81dU7u4g8xfSw-w@HIDDEN>
Subject: Re: bug#28180: [w32] Unicode characters in subprocess (git) arguments
 changed to space
To: Eli Zaretskii <eliz@HIDDEN>
Content-Type: text/plain; charset="UTF-8"
X-Spam-Score: 0.7 (/)
X-Debbugs-Envelope-To: 28180
Cc: 28180 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: 0.7 (/)

On Mon, Aug 28, 2017 at 10:42 AM, Noam Postavsky
<npostavs@HIDDEN> wrote:

> Just to understand the issue better, I applied the attached diff to
> use CreateProcessW. It seemed to work, but only when I start emacs
> from mingw's msys shell. When running from cmd.exe it still translates
> to space.

Ugh, it's just a stupid mistake on my part, I changed the argument
encoding to utf8 in make-process instead of call-process. It
*appeared* to work because msys sets LANG to en_US.UTF-8.
Information forwarded to bug-gnu-emacs@HIDDEN:
bug#28180; Package emacs. Full text available.

Message received at 28180 <at> debbugs.gnu.org:


Received: (at 28180) by debbugs.gnu.org; 28 Aug 2017 17:16:13 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Aug 28 13:16:12 2017
Received: from localhost ([127.0.0.1]:59600 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1dmNe0-0002h2-Go
	for submit <at> debbugs.gnu.org; Mon, 28 Aug 2017 13:16:12 -0400
Received: from eggs.gnu.org ([208.118.235.92]:35897)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1dmNdz-0002at-3R
 for 28180 <at> debbugs.gnu.org; Mon, 28 Aug 2017 13:16:11 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <eliz@HIDDEN>) id 1dmNdp-0000CF-V0
 for 28180 <at> debbugs.gnu.org; Mon, 28 Aug 2017 13:16:05 -0400
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=-0.5 required=5.0 tests=BAYES_05,RP_MATCHES_RCVD
 autolearn=disabled version=3.3.2
Received: from fencepost.gnu.org ([2001:4830:134:3::e]:49486)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>)
 id 1dmNdp-0000Bt-RY; Mon, 28 Aug 2017 13:16:01 -0400
Received: from 84.94.185.246.cable.012.net.il ([84.94.185.246]:4267
 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1dmNdo-0002US-1V; Mon, 28 Aug 2017 13:16:01 -0400
Date: Mon, 28 Aug 2017 20:15:46 +0300
Message-Id: <83bmmz4nr1.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Noam Postavsky <npostavs@HIDDEN>
In-reply-to: <CAM-tV--3p_JKL5d+65BvMR2pMS97wfH9-ZVNW3DR2tPf4AObJQ@HIDDEN>
 (message from Noam Postavsky on Mon, 28 Aug 2017 10:42:14 -0400)
Subject: Re: bug#28180: [w32] Unicode characters in subprocess (git) arguments
 changed to space
References: <87d17oba8j.fsf@HIDDEN> <83y3qbabzw.fsf@HIDDEN>
 <CAM-tV--3p_JKL5d+65BvMR2pMS97wfH9-ZVNW3DR2tPf4AObJQ@HIDDEN>
MIME-version: 1.0
Content-type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2001:4830:134:3::e
X-Spam-Score: -5.0 (-----)
X-Debbugs-Envelope-To: 28180
Cc: 28180 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Reply-To: Eli Zaretskii <eliz@HIDDEN>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -5.0 (-----)

> From: Noam Postavsky <npostavs@HIDDEN>
> Date: Mon, 28 Aug 2017 10:42:14 -0400
> Cc: 28180 <at> debbugs.gnu.org
> 
> Just to understand the issue better, I applied the attached diff to
> use CreateProcessW.

I hope you realize that this is just a quick hack, which cannot work
in general, yes?  For starters, the command line is not a file name,
in general, so using filename_to_utf16 is inappropriate.  Also, I
think the environment variables need to be converted to UTF-16.

> It seemed to work, but only when I start emacs from mingw's msys
> shell. When running from cmd.exe it still translates to space.

What exactly did you run from cmd.exe?  What command?

> Furthermore, when I run an unpatched Emacs from the msys shell, the
> output of the test I posted above is different:
> 
> (call-process "git" nil '(t t) nil
>               "-c" "alias.x=!x() { printf '%s' \"$1\" | od -tx1; }; x"
> "x" "(好)")
> 0000000 28 c3 a5 c2 a5 c2 bd 29
> 0000010
> 
> Do you have any idea what setting could cause this?

Windows tries to interpret UTF-8 as something else?
Information forwarded to bug-gnu-emacs@HIDDEN:
bug#28180; Package emacs. Full text available.

Message received at 28180 <at> debbugs.gnu.org:


Received: (at 28180) by debbugs.gnu.org; 28 Aug 2017 14:42:22 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Aug 28 10:42:22 2017
Received: from localhost ([127.0.0.1]:59543 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1dmLF8-0006Mg-3r
	for submit <at> debbugs.gnu.org; Mon, 28 Aug 2017 10:42:22 -0400
Received: from mail-oi0-f47.google.com ([209.85.218.47]:35350)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <npostavs@HIDDEN>) id 1dmLF7-0006MU-Am
 for 28180 <at> debbugs.gnu.org; Mon, 28 Aug 2017 10:42:21 -0400
Received: by mail-oi0-f47.google.com with SMTP id k77so5446303oib.2
 for <28180 <at> debbugs.gnu.org>; Mon, 28 Aug 2017 07:42:21 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=mime-version:sender:in-reply-to:references:from:date:message-id
 :subject:to:cc;
 bh=lecipUaW1VIs8rlAlFIqgVsJZGhd4R8AzPcX/MwKNOg=;
 b=d7UFUYV+Kath5heyXDw8Rd4y9iD+l6c0PbcSfYpLcPuxaHo5zKmsUOQCYB4q2wVzc7
 ix8C47TAYK5wOHzXCNSMHWAbIBWNRP6MFj0idlchPk5GfKMIH3vxfqmbtaaAOzM0ih5t
 yyucdYZpimin0tGI9d3EmE0Wc+on4Vjt27rnWlcIfP+v7fGCGyDC2rfzvyit2EtDAb74
 pXdvfVp2tHJKXvjTRv1rcKdNPQJnCH2tYX+Z5caHEmGH8aRbHHl7b4Zs1rsFwlxEHRfY
 ruU6SIPoaxTSDVxk+YMNIu0gXp3X5Pe8sqnQbe5g5LODBypC4jCXVWjQFOKpzH3fIQ3/
 gVSw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:mime-version:sender:in-reply-to:references:from
 :date:message-id:subject:to:cc;
 bh=lecipUaW1VIs8rlAlFIqgVsJZGhd4R8AzPcX/MwKNOg=;
 b=ay11a7lf/rBAgek75PgVR2eqbXO4KlsDZgbKc3DGQfrSipLVRh5QbFeIJaP6k7l7kZ
 KNZbr5smA7kFhQ6HQ069enmt4x4PcERwAD8LC+A+iSbF3rZBbDgis14cMYZ/Ql5KN9Pr
 PzLnCTBHH5oy1YAH24rXB7TGuucLUo406Sh5vd/1azxrrsq1QEhGjUaSV8+cK8MlgXbZ
 4jAJOX28bM/nBIvSjotNf+JRdFfvZCbbMyYuFmhYzBEWq0kO58hOdNfrQSEa+NYEzZal
 brOUCa+JlxvYbtrAJ5IsCvHCf5TyzFXEMfHT6EaGdtxEtUzwX1mFzYW8UVEe2CpbjIP3
 Y/Eg==
X-Gm-Message-State: AHYfb5hTwgCqrnd3LU62DVaMpQKZD9mPtqhyL5A9ErWyvmVhsgzHdliN
 vKjJ5z4rtg56ietcGoF9nk9Omno4Ng==
X-Received: by 10.202.78.85 with SMTP id c82mr669662oib.96.1503931335364; Mon,
 28 Aug 2017 07:42:15 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.74.17.199 with HTTP; Mon, 28 Aug 2017 07:42:14 -0700 (PDT)
In-Reply-To: <83y3qbabzw.fsf@HIDDEN>
References: <87d17oba8j.fsf@HIDDEN> <83y3qbabzw.fsf@HIDDEN>
From: Noam Postavsky <npostavs@HIDDEN>
Date: Mon, 28 Aug 2017 10:42:14 -0400
X-Google-Sender-Auth: UV20xlhB8iPjccvU4OLCrNrpU84
Message-ID: <CAM-tV--3p_JKL5d+65BvMR2pMS97wfH9-ZVNW3DR2tPf4AObJQ@HIDDEN>
Subject: Re: bug#28180: [w32] Unicode characters in subprocess (git) arguments
 changed to space
To: Eli Zaretskii <eliz@HIDDEN>
Content-Type: multipart/mixed; boundary="001a11c183aa03278b0557d14df9"
X-Spam-Score: -2.1 (--)
X-Debbugs-Envelope-To: 28180
Cc: 28180 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -2.1 (--)

--001a11c183aa03278b0557d14df9
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Tue, Aug 22, 2017 at 10:54 AM, Eli Zaretskii <eliz@HIDDEN> wrote:

> As to the "other places", the only problem I'm aware of is that the
> encoding of the command-line arguments, when they arrive at w32proc.c,
> is not known in advance, so this must be somehow fixed/changed,
> otherwise we will be unable to re-encode them in UTF-16.  I believe
> the comment in w32.c does mention that.

Just to understand the issue better, I applied the attached diff to
use CreateProcessW. It seemed to work, but only when I start emacs
from mingw's msys shell. When running from cmd.exe it still translates
to space.

Furthermore, when I run an unpatched Emacs from the msys shell, the
output of the test I posted above is different:

(call-process "git" nil '(t t) nil
              "-c" "alias.x=3D!x() { printf '%s' \"$1\" | od -tx1; }; x"
"x" "(=E5=A5=BD)")
0000000 28 c3 a5 c2 a5 c2 bd 29
0000010

Do you have any idea what setting could cause this?

--001a11c183aa03278b0557d14df9
Content-Type: text/plain; charset="US-ASCII"; name="CreateProcessW.diff"
Content-Disposition: attachment; filename="CreateProcessW.diff"
Content-Transfer-Encoding: base64
X-Attachment-Id: f_j6w9ncre0

ZGlmZiAtLWdpdCBjL3NyYy9wcm9jZXNzLmMgaS9zcmMvcHJvY2Vzcy5jCmluZGV4IGU3ZWU5OWEu
LjJjMTdiNWEgMTAwNjQ0Ci0tLSBjL3NyYy9wcm9jZXNzLmMKKysrIGkvc3JjL3Byb2Nlc3MuYwpA
QCAtMTg4MSw4ICsxODgxLDcgQEAgdXNhZ2U6IChtYWtlLXByb2Nlc3MgJnJlc3QgQVJHUykgICov
KQogCSAgaWYgKFNUUklOR19NVUxUSUJZVEUgKGFyZykpCiAJICAgIHsKIAkgICAgICBpZiAoTklM
UCAoYXJnX2VuY29kaW5nKSkKLQkJYXJnX2VuY29kaW5nID0gKGNvbXBsZW1lbnRfcHJvY2Vzc19l
bmNvZGluZ19zeXN0ZW0KLQkJCQkoWFBST0NFU1MgKHByb2MpLT5lbmNvZGVfY29kaW5nX3N5c3Rl
bSkpOworCQlhcmdfZW5jb2RpbmcgPSBRdXRmXzg7CiAJICAgICAgYXJnID0gY29kZV9jb252ZXJ0
X3N0cmluZ19ub3JlY29yZCAoYXJnLCBhcmdfZW5jb2RpbmcsIDEpOwogCSAgICB9CiAJICB0ZW0g
PSBGY29ucyAoYXJnLCB0ZW0pOwpkaWZmIC0tZ2l0IGMvc3JjL3czMnByb2MuYyBpL3NyYy93MzJw
cm9jLmMKaW5kZXggNzZhZjU1Zi4uODZhYWVlMiAxMDA2NDQKLS0tIGMvc3JjL3czMnByb2MuYwor
KysgaS9zcmMvdzMycHJvYy5jCkBAIC0xMjA0LDE0ICsxMjA0LDEzIEBAIHN0YXRpYyBCT09MCiBj
cmVhdGVfY2hpbGQgKGNoYXIgKmV4ZSwgY2hhciAqY21kbGluZSwgY2hhciAqZW52LCBpbnQgaXNf
Z3VpX2FwcCwKIAkgICAgICBwaWRfdCAqIHBQaWQsIGNoaWxkX3Byb2Nlc3MgKmNwKQogewotICBT
VEFSVFVQSU5GTyBzdGFydDsKKyAgU1RBUlRVUElORk9XIHN0YXJ0OwogICBTRUNVUklUWV9BVFRS
SUJVVEVTIHNlY19hdHRyczsKICNpZiAwCiAgIFNFQ1VSSVRZX0RFU0NSSVBUT1Igc2VjX2Rlc2M7
CiAjZW5kaWYKICAgRFdPUkQgZmxhZ3M7CiAgIGNoYXIgZGlyWyBNQVhfUEFUSCBdOwotICBjaGFy
ICpwOwogICBjb25zdCBjaGFyICpleHQ7CiAKICAgaWYgKGNwID09IE5VTEwpIGVtYWNzX2Fib3J0
ICgpOwpAQCAtMTI0MiwxNCArMTI0MSw4IEBAIGNyZWF0ZV9jaGlsZCAoY2hhciAqZXhlLCBjaGFy
ICpjbWRsaW5lLCBjaGFyICplbnYsIGludCBpc19ndWlfYXBwLAogICBzZWNfYXR0cnMubHBTZWN1
cml0eURlc2NyaXB0b3IgPSBOVUxMIC8qICZzZWNfZGVzYyAqLzsKICAgc2VjX2F0dHJzLmJJbmhl
cml0SGFuZGxlID0gRkFMU0U7CiAKLSAgZmlsZW5hbWVfdG9fYW5zaSAocHJvY2Vzc19kaXIsIGRp
cik7Ci0gIC8qIENhbid0IHVzZSB1bml4dG9kb3NfZmlsZW5hbWUgaGVyZSwgc2luY2UgdGhhdCBu
ZWVkcyBpdHMgZmlsZSBuYW1lCi0gICAgIGFyZ3VtZW50IGVuY29kZWQgaW4gVVRGLTguICBPVE9I
LCBwcm9jZXNzX2Rpciwgd2hpY2ggX2lzXyBpbgotICAgICBVVEYtOCwgcG9pbnRzLCB0byB0aGUg
ZGlyZWN0b3J5IGNvbXB1dGVkIGJ5IG91ciBjYWxsZXIsIGFuZCB3ZQotICAgICBkb24ndCB3YW50
IHRvIG1vZGlmeSB0aGF0LCBlaXRoZXIuICAqLwotICBmb3IgKHAgPSBkaXI7ICpwOyBwID0gQ2hh
ck5leHRBIChwKSkKLSAgICBpZiAoKnAgPT0gJy8nKQotICAgICAgKnAgPSAnXFwnOworICBzdHJj
cHkgKGRpciwgcHJvY2Vzc19kaXIpOworICB1bml4dG9kb3NfZmlsZW5hbWUgKGRpcik7CiAKICAg
LyogQ3JlYXRlUHJvY2VzcyBoYW5kbGVzIGJhdGNoIGZpbGVzIGFzIGV4ZSBzcGVjaWFsbHkuICBU
aGlzIHNwZWNpYWwKICAgICAgaGFuZGxpbmcgZmFpbHMgd2hlbiBib3RoIHRoZSBiYXRjaCBmaWxl
IGFuZCBhcmd1bWVudHMgYXJlIHF1b3RlZC4KQEAgLTEyNjUsOCArMTI1OCwxMyBAQCBjcmVhdGVf
Y2hpbGQgKGNoYXIgKmV4ZSwgY2hhciAqY21kbGluZSwgY2hhciAqZW52LCBpbnQgaXNfZ3VpX2Fw
cCwKIAkgICA6IENSRUFURV9ORVdfQ09OU09MRSk7CiAgIGlmIChOSUxQIChWdzMyX3N0YXJ0X3By
b2Nlc3NfaW5oZXJpdF9lcnJvcl9tb2RlKSkKICAgICBmbGFncyB8PSBDUkVBVEVfREVGQVVMVF9F
UlJPUl9NT0RFOwotICBpZiAoIUNyZWF0ZVByb2Nlc3NBIChleGUsIGNtZGxpbmUsICZzZWNfYXR0
cnMsIE5VTEwsIFRSVUUsCi0JCSAgICAgICBmbGFncywgZW52LCBkaXIsICZzdGFydCwgJmNwLT5w
cm9jaW5mbykpCisKKyAgd2NoYXJfdCBleGVXW01BWF9QQVRIXSwgY21kbGluZVdbTUFYX1BBVEhd
LCBkaXJXW01BWF9QQVRIXTsKKyAgZmlsZW5hbWVfdG9fdXRmMTYgKGV4ZSwgZXhlVyk7CisgIGZp
bGVuYW1lX3RvX3V0ZjE2IChjbWRsaW5lLCBjbWRsaW5lVyk7CisgIGZpbGVuYW1lX3RvX3V0ZjE2
IChkaXIsIGRpclcpOworICBpZiAoIUNyZWF0ZVByb2Nlc3NXIChleGVXLCBjbWRsaW5lVywgJnNl
Y19hdHRycywgTlVMTCwgVFJVRSwKKyAgICAgICAgICAgICAgICAgICAgICAgZmxhZ3MsIGVudiwg
ZGlyVywgJnN0YXJ0LCAmY3AtPnByb2NpbmZvKSkKICAgICBnb3RvIEVIX0ZhaWw7CiAKICAgY3At
PnBpZCA9IChpbnQpIGNwLT5wcm9jaW5mby5kd1Byb2Nlc3NJZDsK
--001a11c183aa03278b0557d14df9--
Information forwarded to bug-gnu-emacs@HIDDEN:
bug#28180; Package emacs. Full text available.

Message received at 28180 <at> debbugs.gnu.org:


Received: (at 28180) by debbugs.gnu.org; 22 Aug 2017 14:55:13 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Aug 22 10:55:13 2017
Received: from localhost ([127.0.0.1]:49703 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1dkAaH-0007pA-Lu
	for submit <at> debbugs.gnu.org; Tue, 22 Aug 2017 10:55:13 -0400
Received: from eggs.gnu.org ([208.118.235.92]:57464)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1dkAaG-0007oy-Ct
 for 28180 <at> debbugs.gnu.org; Tue, 22 Aug 2017 10:55:12 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <eliz@HIDDEN>) id 1dkAa8-0000e4-03
 for 28180 <at> debbugs.gnu.org; Tue, 22 Aug 2017 10:55:07 -0400
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=-0.0 required=5.0 tests=BAYES_20,RP_MATCHES_RCVD
 autolearn=disabled version=3.3.2
Received: from fencepost.gnu.org ([2001:4830:134:3::e]:59673)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>)
 id 1dkAa7-0000dx-So; Tue, 22 Aug 2017 10:55:03 -0400
Received: from 84.94.185.246.cable.012.net.il ([84.94.185.246]:2450
 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1dkAa7-0002LU-7V; Tue, 22 Aug 2017 10:55:03 -0400
Date: Tue, 22 Aug 2017 17:54:59 +0300
Message-Id: <83y3qbabzw.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: npostavs@HIDDEN
In-reply-to: <87d17oba8j.fsf@HIDDEN>
 (npostavs@HIDDEN)
Subject: Re: bug#28180: [w32] Unicode characters in subprocess (git) arguments
 changed to space
References: <87d17oba8j.fsf@HIDDEN>
MIME-version: 1.0
Content-type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2001:4830:134:3::e
X-Spam-Score: -5.0 (-----)
X-Debbugs-Envelope-To: 28180
Cc: 28180 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Reply-To: Eli Zaretskii <eliz@HIDDEN>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -5.0 (-----)

> From: npostavs@HIDDEN
> Date: Mon, 21 Aug 2017 22:35:24 -0400
> 
> In w32.c there is a comment saying
> 
>    . Running subprocesses in non-ASCII directories and with non-ASCII
>      file arguments is limited to the current codepage [...]
>      This should be fixed, but will also require changes in cmdproxy.
>      The current limitation is not terribly bad anyway, since very
>      few, if any, Windows console programs that are likely to be
>      invoked by Emacs support UTF-16 encoded command lines.
> 
> I believe we're running into this limitation with git: staging a file
> named 好.txt fails from magit[1] (I tried also with vc, same problem).
> A quick way to see the problem is evaluating the call-process form
> below, the output shows that the Chinese character has been transformed
> into a space.

I'd expect that in a non-Chinese locale (which I believe was what you
did), but the OP of the Magit issue has Windows set up for a Chinese
locale, so there has to be some other explanation, because passing
Chinese characters on the command line ought to work in that case.

> Am I correct that this problem is related the w32.c comment?

The comment is accurate, but it can only explain why command-line
arguments with characters outside of the current Windows locale cannot
be safely passed to sub-processes.  Which AFAIU is not the case with
the OP of that Magit issue.

> It's not clear to me what changes are needed in cmdproxy (and other
> places?) to address it.

cmdproxy is not involved in call-process, but it is involved in
shell-command and its ilk.  As it makes no sense to support Unicode in
the former, but not in the latter, if we want to lift this limitation,
we must teach cmdproxy to use "wide" APIs both for receiving
command-line arguments from Emacs and for passing them to programs it
invokes.

As to the "other places", the only problem I'm aware of is that the
encoding of the command-line arguments, when they arrive at w32proc.c,
is not known in advance, so this must be somehow fixed/changed,
otherwise we will be unable to re-encode them in UTF-16.  I believe
the comment in w32.c does mention that.
Information forwarded to bug-gnu-emacs@HIDDEN:
bug#28180; Package emacs. Full text available.

Message received at submit <at> debbugs.gnu.org:


Received: (at submit) by debbugs.gnu.org; 22 Aug 2017 02:34:02 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Aug 21 22:34:02 2017
Received: from localhost ([127.0.0.1]:48419 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1djz10-0003aT-9l
	for submit <at> debbugs.gnu.org; Mon, 21 Aug 2017 22:34:02 -0400
Received: from eggs.gnu.org ([208.118.235.92]:50819)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <npostavs@HIDDEN>) id 1djz0y-0003a0-I3
 for submit <at> debbugs.gnu.org; Mon, 21 Aug 2017 22:34:01 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <npostavs@HIDDEN>) id 1djz0s-0002rM-24
 for submit <at> debbugs.gnu.org; Mon, 21 Aug 2017 22:33:55 -0400
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=-0.5 required=5.0 tests=BAYES_05,FREEMAIL_FROM,
 T_DKIM_INVALID autolearn=disabled version=3.3.2
Received: from lists.gnu.org ([2001:4830:134:3::11]:40748)
 by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32)
 (Exim 4.71) (envelope-from <npostavs@HIDDEN>) id 1djz0r-0002rG-UN
 for submit <at> debbugs.gnu.org; Mon, 21 Aug 2017 22:33:53 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:41785)
 by lists.gnu.org with esmtp (Exim 4.71)
 (envelope-from <npostavs@HIDDEN>) id 1djz0q-0007S5-Fq
 for bug-gnu-emacs@HIDDEN; Mon, 21 Aug 2017 22:33:53 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <npostavs@HIDDEN>) id 1djz0l-0002pB-OX
 for bug-gnu-emacs@HIDDEN; Mon, 21 Aug 2017 22:33:52 -0400
Received: from mail-it0-x22d.google.com ([2607:f8b0:4001:c0b::22d]:37703)
 by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16)
 (Exim 4.71) (envelope-from <npostavs@HIDDEN>) id 1djz0l-0002ow-Ho
 for bug-gnu-emacs@HIDDEN; Mon, 21 Aug 2017 22:33:47 -0400
Received: by mail-it0-x22d.google.com with SMTP id 76so40288085ith.0
 for <bug-gnu-emacs@HIDDEN>; Mon, 21 Aug 2017 19:33:47 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=sender:from:to:subject:date:message-id:mime-version;
 bh=PeXHU65WctXgemL2+Un3L/NOUHEZKv4Ue6Q2teo/Noc=;
 b=YHxRF7+G3hQSfvVl67GqN6iuy4Ozl3NkVgBBCmMtqMIbeOtoXkWrwKvtE/YIh6c/p1
 /FQy1SGGAFapmP87cKNxVyiCf0DE/7wORkCjCQMYsbIX+aOlKdkBd5kjb6yWXHlxsnHh
 aLmQXQ9byQa7AJrghNKPtzAZf0S1GVQOZ5QFQ3EMGgLQHdvqxaKTz4vAjnGQw1zDkS5n
 pmex6uHdhwINzCxygSJOXH3ju2alKBWok6pvynPSxxJYO6Hc6SH652vt34VOVpAwoo64
 sVQfHnE6XCfIPKGOi7JY67TuYMOO+4RT01Gg11OwqTtnJOC7EKEGtAySP1Us0y5vd1sr
 aQzA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:sender:from:to:subject:date:message-id
 :mime-version;
 bh=PeXHU65WctXgemL2+Un3L/NOUHEZKv4Ue6Q2teo/Noc=;
 b=ruMl5AXgXV4zykqqyMLTyF+TmFkCbiKiK9SzXe+gFwlbdki0F/x3ytfv6ahebL/zrv
 FkczT4kzB1On5Jb1CvKKL7OUzCCnPkZiEfrxQcTQlZuE76Rz2HKrHDoBGKF27ujgNOIE
 H1VAxeDD8XNLmqN+QsIxYr5YoNu89LpspK4W+39jfzmatf43oKijoSX8MMPyjpmOuAaQ
 tOlEcNM9iFuPuXoWFQ6Rt2gEokIAT03NdIdm7jC/CuZ94Nv5aFdPbRpVf3ENHQ/N8opB
 rZlD9DLiGHgWYG+KGxw4xur7PzMze0ZLXEDxYgkTPLoVgeoiZ1hOF3yDNkksTglEPIjj
 qmXQ==
X-Gm-Message-State: AHYfb5hCcC/Nccy+JM31ZmHlIEkBwZH8fDidQHxPyJMdaVaGnd3Exuyt
 iQjNTP90jFDp8krl
X-Received: by 10.36.40.147 with SMTP id h141mr8042989ith.85.1503369226713;
 Mon, 21 Aug 2017 19:33:46 -0700 (PDT)
Received: from zony ([45.2.119.49])
 by smtp.googlemail.com with ESMTPSA id f11sm631037itf.1.2017.08.21.19.33.45
 for <bug-gnu-emacs@HIDDEN>
 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256);
 Mon, 21 Aug 2017 19:33:45 -0700 (PDT)
From: npostavs@HIDDEN
To: bug-gnu-emacs@HIDDEN
Subject: [w32] Unicode characters in subprocess (git) arguments changed to
 space
Date: Mon, 21 Aug 2017 22:35:24 -0400
Message-ID: <87d17oba8j.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="=-=-="
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
 recognized.
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x
X-Received-From: 2001:4830:134:3::11
X-Spam-Score: -3.8 (---)
X-Debbugs-Envelope-To: submit
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.8 (---)

--=-=-=
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

In w32.c there is a comment saying

   . Running subprocesses in non-ASCII directories and with non-ASCII
     file arguments is limited to the current codepage [...]
     This should be fixed, but will also require changes in cmdproxy.
     The current limitation is not terribly bad anyway, since very
     few, if any, Windows console programs that are likely to be
     invoked by Emacs support UTF-16 encoded command lines.

I believe we're running into this limitation with git: staging a file
named =E5=A5=BD.txt fails from magit[1] (I tried also with vc, same problem=
).
A quick way to see the problem is evaluating the call-process form
below, the output shows that the Chinese character has been transformed
into a space.  This happens even if I execute 'chcp 65001' before
starting Emacs (a possible workaround I saw suggested in a few places).
The short file name trick doesn't help either.

(call-process "git" nil '(t t) nil
              "-c" "alias.x=3D!x() { printf '%s' \"$1\" | od -tx1; }; x" "x=
" "(=E5=A5=BD)")
0000000 28 20 29
0000003

As far as I can tell, git does support UTF-16 encoded command lines, as
demonstrated by the attached git-args.c, which produces the utf8
encoding of the character (this is also what the call-process form
produces when I run it on GNU/Linux):

C:\Users\npostavs\src\win32-args>.\git-args.exe
0000000 28 e5 a5 bd 29
0000005


--=-=-=
Content-Type: text/plain
Content-Disposition: attachment; filename=git-args.c
Content-Description: test program, C source

#include <windows.h>
#include <stdio.h>

int main(int argc, char **argv)
{
    STARTUPINFOW wsi;
    PROCESS_INFORMATION pi;

    WCHAR wcmdline[] = L"git -c \"alias.x=!x() { printf '%s' \\\"$1\\\" | od -tx1; }; x\" x (\x597d)";

    ZeroMemory( &wsi, sizeof(wsi) );
    wsi.cb = sizeof(wsi);
    ZeroMemory( &pi, sizeof(pi) );

    if (!CreateProcessW(NULL, wcmdline,
            NULL, NULL, // proc, thread attrs
            FALSE, // handle inheritance
            0, // creation flags
            NULL, // env
            NULL, // cur dir
            &wsi,
            &pi)) {
        printf( "CreateProcess failed (%d).\n", GetLastError() );
        return 1;
    }

    // Wait until child process exits.
    WaitForSingleObject( pi.hProcess, INFINITE );

    // Close process and thread handles. 
    CloseHandle( pi.hProcess );
    CloseHandle( pi.hThread );
    return 0;
}

--=-=-=
Content-Type: text/plain


Am I correct that this problem is related the w32.c comment?  It's not
clear to me what changes are needed in cmdproxy (and other places?) to
address it.

[1]: https://github.com/magit/magit/issues/3111

--=-=-=--
Acknowledgement sent to npostavs@HIDDEN:
New bug report received and forwarded. Copy sent to bug-gnu-emacs@HIDDEN. Full text available.
Report forwarded to bug-gnu-emacs@HIDDEN:
bug#28180; Package emacs. Full text available.
Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.
Last modified: Mon, 25 Nov 2019 12:00:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.