GNU logs - #20823, boring messages

Message sent to bug-guile@HIDDEN:

Subject: bug#20823: argv mangled by locale
Resent-From: Zefram <zefram@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guile@HIDDEN
Resent-Date: Tue, 16 Jun 2015 04:34:02 +0000
Resent-Message-ID: <handler.20823.B.14344291988266 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: 20823 <at> debbugs.gnu.org
Date: Tue, 16 Jun 2015 05:33:00 +0100
From: Zefram <zefram@HIDDEN>
Message-ID: <20150616043300.GB2718@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

When guile-2.0 stores argv for later access via program-arguments,
it sometimes decodes the underlying octet string according to the
nominal character encoding of the locale suggested by the environment.
This is a problem, because the arguments are not necessarily encoded
that way, and may not even be encodings of character strings at all.
The decoding is lossy, where the octet string isn't consistent with the
character encoding, so the original octet string cannot be recovered
from the mangled form.  I don't see any Scheme interface that reliably
retrieves the command line arguments without locale decoding.

The decoding doesn't follow the usual rules for locale control.  It is
not at all sensitive to setlocale, which is understandable due to the
arguments being acquired before any of the actual program's code runs.
Empirically, if the environment nominates no locale, "POSIX", or a
non-existent locale, then argv is decoded according to ISO-8859-1, thus
preserving the octets.  If the environment nominates an extant locale
other than "POSIX", then argv is decoded according to that locale's
nominal character encoding.

Demos:

$ env - guile-2.0 -c '(write (map char->integer (string->list (cadr (program-arguments))))) (newline)' $'L\xc3\xa9on'  
(76 195 169 111 110)
$ env - LANG=C guile-2.0 -c '(write (map char->integer (string->list (cadr (program-arguments))))) (newline)' $'L\xc3\xa9on' 
(76 63 63 111 110)
$ env - LANG=de_DE.utf8 guile-2.0 -c '(write (map char->integer (string->list (cadr (program-arguments))))) (newline)' $'L\xc3\xa9on' 
(76 233 111 110)
$ env - LANG=de_DE.iso88591 guile-2.0 -c '(write (map char->integer (string->list (cadr (program-arguments))))) (newline)' $'L\xc3\xa9on' 
(76 195 169 111 110)

The actual data passed between processes is an octet string, and
there really needs to be some reliable way to access that octet string.
My comments about resolution in bug#20822 "environment mangled by locale"
mostly apply here too, with a slight change: it seems necessary to store
the original octet strings and decode at the time program-arguments is
called.  With that change, the decoding can be responsive to setlocale
(and in particular can reliably use ISO-8859-1 in the absence of
setlocale).

-zefram

Message sent:

Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Mailer: MIME-tools 5.503 (Entity 5.503)
Content-Type: text/plain; charset=utf-8
X-Loop: help-debbugs@HIDDEN
From: help-debbugs@HIDDEN (GNU bug Tracking System)
To: Zefram <zefram@HIDDEN>
Subject: bug#20823: Acknowledgement (argv mangled by locale)
Message-ID: <handler.20823.B.14344291988266.ack <at> debbugs.gnu.org>
References: <20150616043300.GB2718@HIDDEN>
X-Gnu-PR-Message: ack 20823
X-Gnu-PR-Package: guile
Reply-To: 20823 <at> debbugs.gnu.org
Date: Tue, 16 Jun 2015 04:34:02 +0000

Thank you for filing a new bug report with debbugs.gnu.org.

This is an automatically generated reply to let you know your message
has been received.

Your message is being forwarded to the package maintainers and other
interested parties for their attention; they will reply in due course.

Your message has been sent to the package maintainer(s):
 bug-guile@HIDDEN

If you wish to submit further information on this problem, please
send it to 20823 <at> debbugs.gnu.org.

Please do not send mail to help-debbugs@HIDDEN unless you wish
to report a problem with the Bug-tracking system.

--=20
20823: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D20823
GNU Bug Tracking System
Contact help-debbugs@HIDDEN with problems

Message sent to bug-guile@HIDDEN:

Subject: bug#20823: argv mangled by locale
Resent-From: Zefram <zefram@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guile@HIDDEN
Resent-Date: Fri, 04 Mar 2016 23:25:01 +0000
Resent-Message-ID: <handler.20823.B20823.145713388621076 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: 20823 <at> debbugs.gnu.org
Date: Fri, 4 Mar 2016 23:24:41 +0000
From: Zefram <zefram@HIDDEN>
Message-ID: <20160304232441.GB13009@HIDDEN>
References: <20150616043300.GB2718@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20150616043300.GB2718@HIDDEN>
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

I wrote:
>My comments about resolution in bug#20822 "environment mangled by locale"
>mostly apply here too,

The revised comments that I have just made on that ticket also apply
here.  Short version: "absence of setlocale" isn't a useful criterion,
so explicit control of encoding will be necessary.

-zefram

Message sent to bug-guile@HIDDEN:

Subject: bug#20823: argv mangled by locale
Resent-From: Andy Wingo <wingo@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guile@HIDDEN
Resent-Date: Fri, 24 Jun 2016 06:12:01 +0000
Resent-Message-ID: <handler.20823.B20823.146674870025823 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: mhw@HIDDEN, ludo@HIDDEN
Cc: 20823 <at> debbugs.gnu.org, Zefram <zefram@HIDDEN>
DomainKey-Signature: a=rsa-sha1; c=nofws; d=pobox.com; h=from:to:cc
 :subject:references:date:in-reply-to:message-id:mime-version
 :content-type; q=dns; s=sasl; b=vOpER5KUHWi2QvuDUIq+gdtt6lDR4NYJ
 2L1MIZWfuNwGowgLVNZARLvqmfLGTjgPZQeToy2xuNO6ZbgDnVG3pPqnueFTPo5o
 17ANNjwve+zAb70aCD3ESsGackLwP55h8Qg8R5M89jaKZsFNna6AredTdac0ns7Z
 HQDR5YTXL64=
From: Andy Wingo <wingo@HIDDEN>
References: <20150616043300.GB2718@HIDDEN>
Date: Fri, 24 Jun 2016 08:11:29 +0200
In-Reply-To: <20150616043300.GB2718@HIDDEN> (zefram@HIDDEN's message of
 "Tue, 16 Jun 2015 05:33:00 +0100")
Message-ID: <87a8ibjeum.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

On Tue 16 Jun 2015 06:33, Zefram <zefram@HIDDEN> writes:

> I don't see any Scheme interface that reliably retrieves the command
> line arguments without locale decoding.
[...]
> The actual data passed between processes is an octet string, and
> there really needs to be some reliable way to access that octet string.
> My comments about resolution in bug#20822 "environment mangled by locale"
> mostly apply here too, with a slight change: it seems necessary to store
> the original octet strings and decode at the time program-arguments is
> called.  With that change, the decoding can be responsive to setlocale
> (and in particular can reliably use ISO-8859-1 in the absence of
> setlocale).

Proposal: scm_i_set_boot_program_arguments just copies the bytes, and
scm_program_arguments decodes them.  I don't know whether to save the
locale that was current at program start and use that locale to decode
the arguments, or default the current locale, or what.  I also don't
know whether to supply an optional "encoding" argument, and use that
encoding to decode the command line arguments.  Thoughts, Mark and
Ludovic?

Andy

Message sent to bug-guile@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#20823: argv mangled by locale
Resent-From: Zefram <zefram@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guile@HIDDEN
Resent-Date: Fri, 24 Jun 2016 08:43:02 +0000
Resent-Message-ID: <handler.20823.B20823.14667577798853 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 20823
X-GNU-PR-Package: guile
X-GNU-PR-Keywords: 
To: Andy Wingo <wingo@HIDDEN>
Cc: 20823 <at> debbugs.gnu.org, mhw@HIDDEN, ludo@HIDDEN
Received: via spool by 20823-submit <at> debbugs.gnu.org id=B20823.14667577798853
          (code B ref 20823); Fri, 24 Jun 2016 08:43:02 +0000
Received: (at 20823) by debbugs.gnu.org; 24 Jun 2016 08:42:59 +0000
Received: from localhost ([127.0.0.1]:53375 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1bGMhX-0002Ij-8c
	for submit <at> debbugs.gnu.org; Fri, 24 Jun 2016 04:42:59 -0400
Received: from river.fysh.org ([87.98.248.19]:57633 ident=Debian-exim)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <zefram@HIDDEN>) id 1bGMhW-0002Ia-9b
 for 20823 <at> debbugs.gnu.org; Fri, 24 Jun 2016 04:42:58 -0400
Received: from zefram by river.fysh.org with local (Exim 4.84_2 #1 (Debian))
 id 1bGMhQ-0008FR-Ri; Fri, 24 Jun 2016 09:42:52 +0100
Date: Fri, 24 Jun 2016 09:42:52 +0100
From: Zefram <zefram@HIDDEN>
Message-ID: <20160624084252.GE1170@HIDDEN>
References: <20150616043300.GB2718@HIDDEN>
 <87a8ibjeum.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <87a8ibjeum.fsf@HIDDEN>
X-Spam-Score: -1.4 (-)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.4 (-)

Andy Wingo wrote:
>                                                        I also don't
>know whether to supply an optional "encoding" argument, and use that
>encoding to decode the command line arguments.

That, or something that just retrieves octets, is necessary.  Decoding via
the selected locale does not suffice, because there's no guarantee that
there'll be a locale with a cooperative encoding.

-zefram

Message sent to bug-guile@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#20823: argv mangled by locale
Resent-From: Zefram <zefram@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guile@HIDDEN
Resent-Date: Sun, 14 Aug 2016 21:37:01 +0000
Resent-Message-ID: <handler.20823.B20823.147121058112967 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 20823
X-GNU-PR-Package: guile
X-GNU-PR-Keywords: 
To: 20823 <at> debbugs.gnu.org
Received: via spool by 20823-submit <at> debbugs.gnu.org id=B20823.147121058112967
          (code B ref 20823); Sun, 14 Aug 2016 21:37:01 +0000
Received: (at 20823) by debbugs.gnu.org; 14 Aug 2016 21:36:21 +0000
Received: from localhost ([127.0.0.1]:57636 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1bZ34v-0003N5-JI
	for submit <at> debbugs.gnu.org; Sun, 14 Aug 2016 17:36:21 -0400
Received: from river.fysh.org ([87.98.248.19]:41886 ident=Debian-exim)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <zefram@HIDDEN>) id 1bZ34t-0003Mv-UV
 for 20823 <at> debbugs.gnu.org; Sun, 14 Aug 2016 17:36:20 -0400
Received: from zefram by river.fysh.org with local (Exim 4.84_2 #1 (Debian))
 id 1bZ34q-0006Mk-Jb; Sun, 14 Aug 2016 22:36:16 +0100
Date: Sun, 14 Aug 2016 22:36:16 +0100
From: Zefram <zefram@HIDDEN>
Message-ID: <20160814213616.GA22491@HIDDEN>
References: <20150616043300.GB2718@HIDDEN>
 <87a8ibjeum.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <87a8ibjeum.fsf@HIDDEN>
X-Spam-Score: -0.5 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.5 (/)

Andy Wingo wrote:
>                                                        I also don't
>know whether to supply an optional "encoding" argument, and use that
>encoding to decode the command line arguments.

If you don't fancy the profusion of extra "encoding" parameters on
argv access (this ticket), environment access (bug#20822), and all
sorts of syscalls (bug#22913), you could bundle them all together in
a fluid.  This would be a bit like the %default-port-encoding fluid,
but setlocale should absolutely not modify it.  It should follow the
scheme that I laid out in bug#24186: its value can be either a string
naming an encoding, or #:locale-at-io meaning that whenever encoding
is required the currently selected locale is consulted.  There should
also be a fluid determining the conversion strategy, like the existing
%default-port-conversion-strategy.  These two fluids together would
control the encoding and decoding for all operations that currently
apply the locale encoding to arbitrary data.  (Decoding locale-supplied
messages is a different matter.)

-zefram

Last modified: Mon, 25 Nov 2019 12:00:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.