GNU bug report logs - #33848
Store references in SBCL-compiled code are "invisible"

Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.

Package: guix; Severity: important; Reported by: Ludovic Courtès <ludo@HIDDEN>; dated Sun, 23 Dec 2018 14:21:02 UTC; Maintainer for guix is bug-guix@HIDDEN.

Message received at 33848 <at> debbugs.gnu.org:


Received: (at 33848) by debbugs.gnu.org; 28 Dec 2018 07:47:41 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Dec 28 02:47:41 2018
Received: from localhost ([127.0.0.1]:39690 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gcmrt-0005rz-Js
	for submit <at> debbugs.gnu.org; Fri, 28 Dec 2018 02:47:41 -0500
Received: from relay2-d.mail.gandi.net ([217.70.183.194]:51625)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mail@HIDDEN>) id 1gcmrr-0005rr-OW
 for 33848 <at> debbugs.gnu.org; Fri, 28 Dec 2018 02:47:40 -0500
X-Originating-IP: 78.199.129.170
Received: from mimimi (moi44-1-78-199-129-170.fbx.proxad.net [78.199.129.170])
 (Authenticated sender: mail@HIDDEN)
 by relay2-d.mail.gandi.net (Postfix) with ESMTPSA id A7DFB40002;
 Fri, 28 Dec 2018 07:47:37 +0000 (UTC)
References: <87r2e8jpfx.fsf@HIDDEN> <877eg0i43j.fsf@HIDDEN>
 <87d0psi1xo.fsf@HIDDEN> <874lb3kin6.fsf@HIDDEN>
 <87sgynezha.fsf@HIDDEN> <87tvj2yesd.fsf@HIDDEN> <877efwe04u.fsf@HIDDEN>
 <8736qji7c1.fsf@HIDDEN> <87tvizvzgk.fsf@HIDDEN>
 <87o9979gfn.fsf@HIDDEN> <87tvizgghs.fsf@HIDDEN>
 <87k1juaomo.fsf@HIDDEN> <87muoqhk62.fsf@HIDDEN>
 <87zhsq8wkj.fsf@HIDDEN> <87d0pmhbgn.fsf@HIDDEN>
 <87r2e28tkv.fsf@HIDDEN>
User-agent: mu4e 1.0; emacs 26.1
From: Pierre Neidhardt <mail@HIDDEN>
To: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Subject: Re: bug#33848: Store references in SBCL-compiled code are "invisible"
In-reply-to: <87r2e28tkv.fsf@HIDDEN>
Date: Fri, 28 Dec 2018 08:47:33 +0100
Message-ID: <874laygkiy.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="=-=-=";
 micalg=pgp-sha256; protocol="application/pgp-signature"
X-Spam-Score: -0.7 (/)
X-Debbugs-Envelope-To: 33848
Cc: Mark H Weaver <mhw@HIDDEN>, 33848 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

--=-=-=
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


> I=E2=80=99m not sure what you call =E2=80=9Cthe binary=E2=80=9D.  Do you =
mean the nar?

No, in this case I referred to "/bin/next" in sbcl-next.  So any file in th=
e nar
passed to the reference scanner.

=2D-=20
Pierre Neidhardt
https://ambrevar.xyz/

--=-=-=
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEUPM+LlsMPZAEJKvom9z0l6S7zH8FAlwl1RUACgkQm9z0l6S7
zH/vtAgAq1KOgRvdEXvdvjPy//H0f/nhzu9Z1qYhwxV9tGOoeBcHiG/vUo6mc7/h
Rkht5zFI35d6L1o09XB6oJKfsVSi0P5VE0APDZzlc3y1pifNcjFekBHUuiva5jR4
zCYeOsXc8AKZSSC7Lf+OPYb2CTFV2nvoANo9dhl5OUPZHZ0B7GcCyHZDtnk4jtiZ
D3p/FjlLcmZ9hom0CbvHjffkDlq/0nmO22kqY5tuKqY/p6UDyJDXROeMdYi2u2lb
hM3kgrw5j7MQrQthmt00PUmqDRPoudjnjc4btoGUjUIjZlVgfZidv1qygC4Bxi/C
9LZ3luopQVjPL/m2is4gwUp2DctYjg==
=3PX/
-----END PGP SIGNATURE-----
--=-=-=--




Information forwarded to bug-guix@HIDDEN:
bug#33848; Package guix. Full text available.

Message received at 33848 <at> debbugs.gnu.org:


Received: (at 33848) by debbugs.gnu.org; 27 Dec 2018 22:59:16 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Dec 27 17:59:16 2018
Received: from localhost ([127.0.0.1]:39612 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gcecW-0007wH-94
	for submit <at> debbugs.gnu.org; Thu, 27 Dec 2018 17:59:16 -0500
Received: from hera.aquilenet.fr ([185.233.100.1]:54844)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1gcecV-0007wA-4M
 for 33848 <at> debbugs.gnu.org; Thu, 27 Dec 2018 17:59:15 -0500
Received: from localhost (localhost [127.0.0.1])
 by hera.aquilenet.fr (Postfix) with ESMTP id 127AEE16;
 Thu, 27 Dec 2018 23:59:14 +0100 (CET)
X-Virus-Scanned: Debian amavisd-new at aquilenet.fr
Received: from hera.aquilenet.fr ([127.0.0.1])
 by localhost (hera.aquilenet.fr [127.0.0.1]) (amavisd-new, port 10024)
 with ESMTP id mkus5lLJDbeT; Thu, 27 Dec 2018 23:59:13 +0100 (CET)
Received: from ribbon (unknown [IPv6:2a01:e0a:1d:7270:af76:b9b:ca24:c465])
 by hera.aquilenet.fr (Postfix) with ESMTPSA id DBAD91CF;
 Thu, 27 Dec 2018 23:59:12 +0100 (CET)
From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>
To: Pierre Neidhardt <mail@HIDDEN>
Subject: Re: bug#33848: Store references in SBCL-compiled code are "invisible"
References: <87r2e8jpfx.fsf@HIDDEN> <877eg0i43j.fsf@HIDDEN>
 <87d0psi1xo.fsf@HIDDEN> <874lb3kin6.fsf@HIDDEN>
 <87sgynezha.fsf@HIDDEN> <87tvj2yesd.fsf@HIDDEN>
 <877efwe04u.fsf@HIDDEN> <8736qji7c1.fsf@HIDDEN>
 <87tvizvzgk.fsf@HIDDEN> <87o9979gfn.fsf@HIDDEN>
 <87tvizgghs.fsf@HIDDEN> <87k1juaomo.fsf@HIDDEN>
 <87muoqhk62.fsf@HIDDEN> <87zhsq8wkj.fsf@HIDDEN>
 <87d0pmhbgn.fsf@HIDDEN>
X-URL: http://www.fdn.fr/~lcourtes/
X-Revolutionary-Date: 7 =?utf-8?Q?Niv=C3=B4se?= an 227 de la =?utf-8?Q?R?=
 =?utf-8?Q?=C3=A9volution?=
X-PGP-Key-ID: 0x090B11993D9AEBB5
X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc
X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4  0CFB 090B 1199 3D9A EBB5
X-OS: x86_64-pc-linux-gnu
Date: Thu, 27 Dec 2018 23:59:12 +0100
In-Reply-To: <87d0pmhbgn.fsf@HIDDEN> (Pierre Neidhardt's message of
 "Thu, 27 Dec 2018 23:05:44 +0100")
Message-ID: <87r2e28tkv.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: 1.0 (+)
X-Debbugs-Envelope-To: 33848
Cc: Mark H Weaver <mhw@HIDDEN>, 33848 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.0 (/)

Pierre Neidhardt <mail@HIDDEN> skribis:

>> The reference scanner, currently written in C++, traverses whole
>> directory trees.  Being C++ it treats file names as byte arrays so it
>> doesn=E2=80=99t matter what the file name encoding is.
>
> But what matters then is that the filename encodings on the filesystem an=
d in the
> binary match, right?

I=E2=80=99m not sure what you call =E2=80=9Cthe binary=E2=80=9D.  Do you me=
an the nar?

Ludo=E2=80=99.




Information forwarded to bug-guix@HIDDEN:
bug#33848; Package guix. Full text available.

Message received at 33848 <at> debbugs.gnu.org:


Received: (at 33848) by debbugs.gnu.org; 27 Dec 2018 22:05:53 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Dec 27 17:05:53 2018
Received: from localhost ([127.0.0.1]:39601 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gcdmq-0006hq-OK
	for submit <at> debbugs.gnu.org; Thu, 27 Dec 2018 17:05:52 -0500
Received: from relay10.mail.gandi.net ([217.70.178.230]:53759)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mail@HIDDEN>) id 1gcdmp-0006hh-6j
 for 33848 <at> debbugs.gnu.org; Thu, 27 Dec 2018 17:05:51 -0500
Received: from mimimi (moi44-1-78-199-129-170.fbx.proxad.net [78.199.129.170])
 (Authenticated sender: mail@HIDDEN)
 by relay10.mail.gandi.net (Postfix) with ESMTPSA id 301A8240007;
 Thu, 27 Dec 2018 22:05:47 +0000 (UTC)
References: <87r2e8jpfx.fsf@HIDDEN> <877eg0i43j.fsf@HIDDEN>
 <87d0psi1xo.fsf@HIDDEN> <874lb3kin6.fsf@HIDDEN>
 <87sgynezha.fsf@HIDDEN> <87tvj2yesd.fsf@HIDDEN> <877efwe04u.fsf@HIDDEN>
 <8736qji7c1.fsf@HIDDEN> <87tvizvzgk.fsf@HIDDEN>
 <87o9979gfn.fsf@HIDDEN> <87tvizgghs.fsf@HIDDEN>
 <87k1juaomo.fsf@HIDDEN> <87muoqhk62.fsf@HIDDEN>
 <87zhsq8wkj.fsf@HIDDEN>
User-agent: mu4e 1.0; emacs 26.1
From: Pierre Neidhardt <mail@HIDDEN>
To: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Subject: Re: bug#33848: Store references in SBCL-compiled code are "invisible"
In-reply-to: <87zhsq8wkj.fsf@HIDDEN>
Date: Thu, 27 Dec 2018 23:05:44 +0100
Message-ID: <87d0pmhbgn.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="=-=-=";
 micalg=pgp-sha256; protocol="application/pgp-signature"
X-Spam-Score: -0.7 (/)
X-Debbugs-Envelope-To: 33848
Cc: Mark H Weaver <mhw@HIDDEN>, 33848 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

--=-=-=
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


> The reference scanner, currently written in C++, traverses whole
> directory trees.  Being C++ it treats file names as byte arrays so it
> doesn=E2=80=99t matter what the file name encoding is.

But what matters then is that the filename encodings on the filesystem and =
in the
binary match, right?

> Note also that the reference scanner only looks for =E2=80=9Cxyz=E2=80=A6=
-foo=E2=80=9D; what
> comes before and after doesn=E2=80=99t matter.  So for example if you have
> =E2=80=9C/gnu/store/xyz=E2=80=A6-foo/=C3=A0=E2=80=9D, what=E2=80=99s impo=
rtant is the =E2=80=9Cxyz=E2=80=A6-foo=E2=80=9D bit.

OK, makes sense, then my main worry is just moot :)

=2D-=20
Pierre Neidhardt
https://ambrevar.xyz/

--=-=-=
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEUPM+LlsMPZAEJKvom9z0l6S7zH8FAlwlTLgACgkQm9z0l6S7
zH/S3Qf/W9Oy7e1p3LkCiKM2l9t7jW3TezaUlHflLGmVd1zdiaTq/aeLdTfY2r+i
+/aEweAHmQGD1oHWmSbnDMyBOQalzBNAQi8dg+oOSVNiMASWk+aHCj5OohE1mxrd
dSLwTxk0a4BIM03GbEc/qFtI2nOZEJGjphkHJGjSKHB/5gzilsLVXRjajYWXVh/P
qVaIH3xJzjA4zIyg711PQTKMqB8qIAKpr0OKA23vpZ1FaRKoNY5NRx/g1wQpFTfi
4ZdwgJwPwh3bDkU61C5mPiHcVARi8X3M65h96Aj+9RX9TCZ/+3oQTWexKTv2o0xH
ZfSYbHpvtAb8xkg9sKLaN47TFoM2Zg==
=djyT
-----END PGP SIGNATURE-----
--=-=-=--




Information forwarded to bug-guix@HIDDEN:
bug#33848; Package guix. Full text available.

Message received at 33848 <at> debbugs.gnu.org:


Received: (at 33848) by debbugs.gnu.org; 27 Dec 2018 21:54:42 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Dec 27 16:54:42 2018
Received: from localhost ([127.0.0.1]:39598 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gcdc2-0006QE-K9
	for submit <at> debbugs.gnu.org; Thu, 27 Dec 2018 16:54:42 -0500
Received: from hera.aquilenet.fr ([185.233.100.1]:54502)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1gcdbz-0006Q4-ER
 for 33848 <at> debbugs.gnu.org; Thu, 27 Dec 2018 16:54:39 -0500
Received: from localhost (localhost [127.0.0.1])
 by hera.aquilenet.fr (Postfix) with ESMTP id 4C8EC1665;
 Thu, 27 Dec 2018 22:54:38 +0100 (CET)
X-Virus-Scanned: Debian amavisd-new at aquilenet.fr
Received: from hera.aquilenet.fr ([127.0.0.1])
 by localhost (hera.aquilenet.fr [127.0.0.1]) (amavisd-new, port 10024)
 with ESMTP id 1N__u0rUgLmU; Thu, 27 Dec 2018 22:54:37 +0100 (CET)
Received: from ribbon (unknown [IPv6:2a01:e0a:1d:7270:af76:b9b:ca24:c465])
 by hera.aquilenet.fr (Postfix) with ESMTPSA id 522671639;
 Thu, 27 Dec 2018 22:54:37 +0100 (CET)
From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>
To: Pierre Neidhardt <mail@HIDDEN>
Subject: Re: bug#33848: Store references in SBCL-compiled code are "invisible"
References: <87r2e8jpfx.fsf@HIDDEN> <877eg0i43j.fsf@HIDDEN>
 <87d0psi1xo.fsf@HIDDEN> <874lb3kin6.fsf@HIDDEN>
 <87sgynezha.fsf@HIDDEN> <87tvj2yesd.fsf@HIDDEN>
 <877efwe04u.fsf@HIDDEN> <8736qji7c1.fsf@HIDDEN>
 <87tvizvzgk.fsf@HIDDEN> <87o9979gfn.fsf@HIDDEN>
 <87tvizgghs.fsf@HIDDEN> <87k1juaomo.fsf@HIDDEN>
 <87muoqhk62.fsf@HIDDEN>
X-URL: http://www.fdn.fr/~lcourtes/
X-Revolutionary-Date: 7 =?utf-8?Q?Niv=C3=B4se?= an 227 de la =?utf-8?Q?R?=
 =?utf-8?Q?=C3=A9volution?=
X-PGP-Key-ID: 0x090B11993D9AEBB5
X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc
X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4  0CFB 090B 1199 3D9A EBB5
X-OS: x86_64-pc-linux-gnu
Date: Thu, 27 Dec 2018 22:54:36 +0100
In-Reply-To: <87muoqhk62.fsf@HIDDEN> (Pierre Neidhardt's message of
 "Thu, 27 Dec 2018 19:57:41 +0100")
Message-ID: <87zhsq8wkj.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: 1.0 (+)
X-Debbugs-Envelope-To: 33848
Cc: Mark H Weaver <mhw@HIDDEN>, 33848 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.0 (/)

Pierre Neidhardt <mail@HIDDEN> skribis:

>> Every file in the store is properly scanned for references.  It=E2=80=99=
s just
>> that users cannot create top-level items with a non-ASCII file name.
>
> So if '/gnu/store/...-foo/=C3=A1' is stored as UTF-8 in a binary, then it=
 will be
> found?  Is it because the filesystem encoding is also UTF-8 and Guix scan=
s over
> byte arrays?

The reference scanner, currently written in C++, traverses whole
directory trees.  Being C++ it treats file names as byte arrays so it
doesn=E2=80=99t matter what the file name encoding is.

Note also that the reference scanner only looks for =E2=80=9Cxyz=E2=80=A6-f=
oo=E2=80=9D; what
comes before and after doesn=E2=80=99t matter.  So for example if you have
=E2=80=9C/gnu/store/xyz=E2=80=A6-foo/=C3=A0=E2=80=9D, what=E2=80=99s import=
ant is the =E2=80=9Cxyz=E2=80=A6-foo=E2=80=9D bit.

This is all happening in libstore/references.cc (which is surprisingly
small) and in (guix build graft) for the grafting part, which Mark wrote
a while back.

HTH,
Ludo=E2=80=99.




Information forwarded to bug-guix@HIDDEN:
bug#33848; Package guix. Full text available.

Message received at 33848 <at> debbugs.gnu.org:


Received: (at 33848) by debbugs.gnu.org; 27 Dec 2018 18:57:53 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Dec 27 13:57:53 2018
Received: from localhost ([127.0.0.1]:39541 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gcaqr-0008Nq-C9
	for submit <at> debbugs.gnu.org; Thu, 27 Dec 2018 13:57:53 -0500
Received: from relay12.mail.gandi.net ([217.70.178.232]:52773)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mail@HIDDEN>) id 1gcaqm-0008Nd-Iv
 for 33848 <at> debbugs.gnu.org; Thu, 27 Dec 2018 13:57:48 -0500
Received: from mimimi (moi44-1-78-199-129-170.fbx.proxad.net [78.199.129.170])
 (Authenticated sender: mail@HIDDEN)
 by relay12.mail.gandi.net (Postfix) with ESMTPSA id 508F2200003;
 Thu, 27 Dec 2018 18:57:41 +0000 (UTC)
References: <87r2e8jpfx.fsf@HIDDEN> <877eg0i43j.fsf@HIDDEN>
 <87d0psi1xo.fsf@HIDDEN> <874lb3kin6.fsf@HIDDEN>
 <87sgynezha.fsf@HIDDEN> <87tvj2yesd.fsf@HIDDEN> <877efwe04u.fsf@HIDDEN>
 <8736qji7c1.fsf@HIDDEN> <87tvizvzgk.fsf@HIDDEN>
 <87o9979gfn.fsf@HIDDEN> <87tvizgghs.fsf@HIDDEN>
 <87k1juaomo.fsf@HIDDEN>
User-agent: mu4e 1.0; emacs 26.1
From: Pierre Neidhardt <mail@HIDDEN>
To: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Subject: Re: bug#33848: Store references in SBCL-compiled code are "invisible"
In-reply-to: <87k1juaomo.fsf@HIDDEN>
Date: Thu, 27 Dec 2018 19:57:41 +0100
Message-ID: <87muoqhk62.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="=-=-=";
 micalg=pgp-sha256; protocol="application/pgp-signature"
X-Spam-Score: -0.7 (/)
X-Debbugs-Envelope-To: 33848
Cc: Mark H Weaver <mhw@HIDDEN>, 33848 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

--=-=-=
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


> Every file in the store is properly scanned for references.  It=E2=80=99s=
 just
> that users cannot create top-level items with a non-ASCII file name.

So if '/gnu/store/...-foo/=C3=A1' is stored as UTF-8 in a binary, then it w=
ill be
found?  Is it because the filesystem encoding is also UTF-8 and Guix scans =
over
byte arrays?

Sorry for dragging on this, I guess I should look at the code at this point=
 but
I have very little time these days.

=2D-=20
Pierre Neidhardt
https://ambrevar.xyz/

--=-=-=
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEUPM+LlsMPZAEJKvom9z0l6S7zH8FAlwlIKUACgkQm9z0l6S7
zH+Efgf/TSVtek2CEmPI+/oF8lD2xe6oSUhog4zvhirSXvsrDvLpY/R7i8lqjhw0
ZSqU/yqkPiz/b/sFZsWOdyjWUsniVjcOjujuy7tzwZynifj0RF9sCCnjJcM3j8Dm
3ioKwO5ppyAZBQRwt+UbjBoOC9NNyT3oDrjs9DWsWEL9cXBvkBbzoKzh/9kH5aaP
vY1HoCx7mVuHeRsmKDR+YnaclArbux5jAseNCnWszqUJjkFvuDgGlmup5supsO3H
t+tFPlVeeEKu414jynZUktyKrJZdRplgmpfqBpuB+6TlQTCG9AW4gqvjoqrgfGbR
VrM1kAU01xtEILl9Zx7C2wRE2+RAtg==
=oyjv
-----END PGP SIGNATURE-----
--=-=-=--




Information forwarded to bug-guix@HIDDEN:
bug#33848; Package guix. Full text available.

Message received at 33848 <at> debbugs.gnu.org:


Received: (at 33848) by debbugs.gnu.org; 27 Dec 2018 17:03:23 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Dec 27 12:03:23 2018
Received: from localhost ([127.0.0.1]:39496 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gcZ44-0005Us-VS
	for submit <at> debbugs.gnu.org; Thu, 27 Dec 2018 12:03:23 -0500
Received: from hera.aquilenet.fr ([185.233.100.1]:52386)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1gcZ3y-0005Uc-6m
 for 33848 <at> debbugs.gnu.org; Thu, 27 Dec 2018 12:03:18 -0500
Received: from localhost (localhost [127.0.0.1])
 by hera.aquilenet.fr (Postfix) with ESMTP id 28B921FA;
 Thu, 27 Dec 2018 18:03:13 +0100 (CET)
X-Virus-Scanned: Debian amavisd-new at aquilenet.fr
Received: from hera.aquilenet.fr ([127.0.0.1])
 by localhost (hera.aquilenet.fr [127.0.0.1]) (amavisd-new, port 10024)
 with ESMTP id jrtGguFx35Uq; Thu, 27 Dec 2018 18:03:12 +0100 (CET)
Received: from ribbon (unknown [IPv6:2a01:e0a:1d:7270:af76:b9b:ca24:c465])
 by hera.aquilenet.fr (Postfix) with ESMTPSA id 50E12170;
 Thu, 27 Dec 2018 18:03:12 +0100 (CET)
From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>
To: Pierre Neidhardt <mail@HIDDEN>
Subject: Re: bug#33848: Store references in SBCL-compiled code are "invisible"
References: <87r2e8jpfx.fsf@HIDDEN> <877eg0i43j.fsf@HIDDEN>
 <87d0psi1xo.fsf@HIDDEN> <874lb3kin6.fsf@HIDDEN>
 <87sgynezha.fsf@HIDDEN> <87tvj2yesd.fsf@HIDDEN>
 <877efwe04u.fsf@HIDDEN> <8736qji7c1.fsf@HIDDEN>
 <87tvizvzgk.fsf@HIDDEN> <87o9979gfn.fsf@HIDDEN>
 <87tvizgghs.fsf@HIDDEN>
X-URL: http://www.fdn.fr/~lcourtes/
X-Revolutionary-Date: 7 =?utf-8?Q?Niv=C3=B4se?= an 227 de la =?utf-8?Q?R?=
 =?utf-8?Q?=C3=A9volution?=
X-PGP-Key-ID: 0x090B11993D9AEBB5
X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc
X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4  0CFB 090B 1199 3D9A EBB5
X-OS: x86_64-pc-linux-gnu
Date: Thu, 27 Dec 2018 18:03:11 +0100
In-Reply-To: <87tvizgghs.fsf@HIDDEN> (Pierre Neidhardt's message of
 "Thu, 27 Dec 2018 16:02:23 +0100")
Message-ID: <87k1juaomo.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: 1.0 (+)
X-Debbugs-Envelope-To: 33848
Cc: Mark H Weaver <mhw@HIDDEN>, 33848 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.0 (/)

Pierre Neidhardt <mail@HIDDEN> skribis:

> Just to be sure I understand: non-toplevel, non-ASCII file names will
> not be scanned properly, right?

Every file in the store is properly scanned for references.  It=E2=80=99s j=
ust
that users cannot create top-level items with a non-ASCII file name.

I hope this clarifies things!

Ludo=E2=80=99.




Information forwarded to bug-guix@HIDDEN:
bug#33848; Package guix. Full text available.

Message received at 33848 <at> debbugs.gnu.org:


Received: (at 33848) by debbugs.gnu.org; 27 Dec 2018 16:15:59 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Dec 27 11:15:59 2018
Received: from localhost ([127.0.0.1]:39472 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gcYKF-0004Lk-BQ
	for submit <at> debbugs.gnu.org; Thu, 27 Dec 2018 11:15:59 -0500
Received: from relay10.mail.gandi.net ([217.70.178.230]:43167)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mail@HIDDEN>) id 1gcYKE-0004Lc-3q
 for 33848 <at> debbugs.gnu.org; Thu, 27 Dec 2018 11:15:58 -0500
Received: from mimimi (moi44-1-78-199-129-170.fbx.proxad.net [78.199.129.170])
 (Authenticated sender: mail@HIDDEN)
 by relay10.mail.gandi.net (Postfix) with ESMTPSA id 97CC2240004;
 Thu, 27 Dec 2018 16:15:55 +0000 (UTC)
References: <87r2e8jpfx.fsf@HIDDEN> <877eg0i43j.fsf@HIDDEN>
 <87d0psi1xo.fsf@HIDDEN> <874lb3kin6.fsf@HIDDEN>
 <87sgynezha.fsf@HIDDEN> <87tvj2yesd.fsf@HIDDEN> <877efwe04u.fsf@HIDDEN>
 <8736qji7c1.fsf@HIDDEN> <87tvizvzgk.fsf@HIDDEN>
 <87o9979gfn.fsf@HIDDEN> <87tvizgghs.fsf@HIDDEN>
User-agent: mu4e 1.0; emacs 26.1
From: Pierre Neidhardt <mail@HIDDEN>
To: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Subject: Re: bug#33848: Store references in SBCL-compiled code are "invisible"
In-reply-to: <87tvizgghs.fsf@HIDDEN>
Date: Thu, 27 Dec 2018 17:15:51 +0100
Message-ID: <87r2e3gd3c.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="=-=-=";
 micalg=pgp-sha256; protocol="application/pgp-signature"
X-Spam-Score: -0.7 (/)
X-Debbugs-Envelope-To: 33848
Cc: Mark H Weaver <mhw@HIDDEN>, 33848 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

--=-=-=
Content-Type: text/plain
Content-Transfer-Encoding: quoted-printable


Danny Milosavljevic <dannym@HIDDEN> writes:
> In 1992, UTF-8 was invented.  Subsequently, most of the Internet,
> all new GNU Linux distributions etc, all UNIX GUI frameworks, Subversion
> etc standardized on UTF-8, with the eventual goal of standardizing all
> network transfer and storage to UTF-8.  I think that by now the outliers
> are the ones who need to change, otherwise these senseless encoding
> conversions will never cease.  It's not like different encodings allow for
> better expression of writings or anything useful to the end user.
>=20
> As a distribution we can't force upstream to change, but just filing
> bug reports upstream would make us see where they stand on this.

I agree with this.  Reporting upstream should be a first step.

=2D-=20
Pierre Neidhardt
https://ambrevar.xyz/

--=-=-=
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEUPM+LlsMPZAEJKvom9z0l6S7zH8FAlwk+rcACgkQm9z0l6S7
zH9gUgf+Nns9ka7lqwHLI14NoJfKk77nUli0A0ANr+I8yQqyll3u0KdiqulNStOc
KnJtLQYny/co27/mMIMllL8im2pwoVhZ6hUxvwyp1AetR4CW5ArPqma2aFKpDFKx
d+T5W1ZUA/fwyB3S1hc3qVIVOzxAHSKQp/Ik/tb++ZDmoHCEg5qlAFxJovlcsCPU
nOlee9bqMXfweqZhckl+97xXmK9mJ3tZ3ijZKQ/ceBmvJvcf7t+XEOSOQQ3FQxsq
YlkUh0jB39NrSTH/HbLxRzPUaihuwZRCEXJu0c29E6S8u+MmXHF04wdH9TXeHoWB
BEAnq4txR6tjKKMEjpDAAeCJfHqnOw==
=QuoM
-----END PGP SIGNATURE-----
--=-=-=--




Information forwarded to bug-guix@HIDDEN:
bug#33848; Package guix. Full text available.

Message received at 33848 <at> debbugs.gnu.org:


Received: (at 33848) by debbugs.gnu.org; 27 Dec 2018 15:02:30 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Dec 27 10:02:30 2018
Received: from localhost ([127.0.0.1]:39442 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gcXB7-0002YM-RC
	for submit <at> debbugs.gnu.org; Thu, 27 Dec 2018 10:02:30 -0500
Received: from relay8-d.mail.gandi.net ([217.70.183.201]:46739)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mail@HIDDEN>) id 1gcXB6-0002YE-Qr
 for 33848 <at> debbugs.gnu.org; Thu, 27 Dec 2018 10:02:29 -0500
X-Originating-IP: 78.199.129.170
Received: from mimimi (moi44-1-78-199-129-170.fbx.proxad.net [78.199.129.170])
 (Authenticated sender: mail@HIDDEN)
 by relay8-d.mail.gandi.net (Postfix) with ESMTPSA id CD9721BF206;
 Thu, 27 Dec 2018 15:02:26 +0000 (UTC)
References: <87r2e8jpfx.fsf@HIDDEN> <877eg0i43j.fsf@HIDDEN>
 <87d0psi1xo.fsf@HIDDEN> <874lb3kin6.fsf@HIDDEN>
 <87sgynezha.fsf@HIDDEN> <87tvj2yesd.fsf@HIDDEN> <877efwe04u.fsf@HIDDEN>
 <8736qji7c1.fsf@HIDDEN> <87tvizvzgk.fsf@HIDDEN>
 <87o9979gfn.fsf@HIDDEN>
User-agent: mu4e 1.0; emacs 26.1
From: Pierre Neidhardt <mail@HIDDEN>
To: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Subject: Re: bug#33848: Store references in SBCL-compiled code are "invisible"
In-reply-to: <87o9979gfn.fsf@HIDDEN>
Date: Thu, 27 Dec 2018 16:02:23 +0100
Message-ID: <87tvizgghs.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="=-=-=";
 micalg=pgp-sha256; protocol="application/pgp-signature"
X-Spam-Score: -0.7 (/)
X-Debbugs-Envelope-To: 33848
Cc: Mark H Weaver <mhw@HIDDEN>, 33848 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

--=-=-=
Content-Type: text/plain
Content-Transfer-Encoding: quoted-printable

Just to be sure I understand: non-toplevel, non-ASCII file names will
not be scanned properly, right?

=2D-=20
Pierre Neidhardt
https://ambrevar.xyz/

--=-=-=
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEUPM+LlsMPZAEJKvom9z0l6S7zH8FAlwk6X8ACgkQm9z0l6S7
zH/cXQgAnsjU66YtPvV2m1Mu2mRYRZfOFYicrKrjPiUkHKeQDY+CnBfPZQFNbMSL
EICwx7JBZ86X48q0jJgJQ2PggQ7L17/4IhLKgL1brVTkuMIWyYXAj6hDF7qJ/ZK1
mkUve432gbMkavmBWSqddENa38T/XdUxij3SxeYyDp0YjhRjbeLddZH2eIji+B8m
qavVmR/wsyTb0u8+xdScTGaB5QoOiYKiE58g02lrZTF2PUkNXG58LhbmPwl1hpfs
0abt5p4k/wvY5d9V8baXLlW9s4NQVE3/vpiE+ycQqPj90iUIummb83trbAshfUHH
whUGRZvYwnWAXVebivC8jWmR2UQdog==
=3pew
-----END PGP SIGNATURE-----
--=-=-=--




Information forwarded to bug-guix@HIDDEN:
bug#33848; Package guix. Full text available.

Message received at 33848 <at> debbugs.gnu.org:


Received: (at 33848) by debbugs.gnu.org; 27 Dec 2018 14:45:37 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Dec 27 09:45:37 2018
Received: from localhost ([127.0.0.1]:38283 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gcWun-0001qy-A7
	for submit <at> debbugs.gnu.org; Thu, 27 Dec 2018 09:45:37 -0500
Received: from hera.aquilenet.fr ([185.233.100.1]:51388)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1gcWul-0001qn-4y
 for 33848 <at> debbugs.gnu.org; Thu, 27 Dec 2018 09:45:35 -0500
Received: from localhost (localhost [127.0.0.1])
 by hera.aquilenet.fr (Postfix) with ESMTP id 2AE99140C;
 Thu, 27 Dec 2018 15:45:34 +0100 (CET)
X-Virus-Scanned: Debian amavisd-new at aquilenet.fr
Received: from hera.aquilenet.fr ([127.0.0.1])
 by localhost (hera.aquilenet.fr [127.0.0.1]) (amavisd-new, port 10024)
 with ESMTP id 5f68Bm7pGLpS; Thu, 27 Dec 2018 15:45:33 +0100 (CET)
Received: from ribbon (unknown [IPv6:2a01:e0a:1d:7270:af76:b9b:ca24:c465])
 by hera.aquilenet.fr (Postfix) with ESMTPSA id 49669B27;
 Thu, 27 Dec 2018 15:45:33 +0100 (CET)
From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>
To: Mark H Weaver <mhw@HIDDEN>
Subject: Re: bug#33848: Store references in SBCL-compiled code are "invisible"
References: <87r2e8jpfx.fsf@HIDDEN> <877eg0i43j.fsf@HIDDEN>
 <87d0psi1xo.fsf@HIDDEN> <874lb3kin6.fsf@HIDDEN>
 <87sgynezha.fsf@HIDDEN> <87tvj2yesd.fsf@HIDDEN>
 <877efwe04u.fsf@HIDDEN> <8736qji7c1.fsf@HIDDEN>
 <87tvizvzgk.fsf@HIDDEN>
X-URL: http://www.fdn.fr/~lcourtes/
X-Revolutionary-Date: 7 =?utf-8?Q?Niv=C3=B4se?= an 227 de la =?utf-8?Q?R?=
 =?utf-8?Q?=C3=A9volution?=
X-PGP-Key-ID: 0x090B11993D9AEBB5
X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc
X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4  0CFB 090B 1199 3D9A EBB5
X-OS: x86_64-pc-linux-gnu
Date: Thu, 27 Dec 2018 15:45:32 +0100
In-Reply-To: <87tvizvzgk.fsf@HIDDEN> (Mark H. Weaver's message of "Thu, 27
 Dec 2018 09:03:12 -0500")
Message-ID: <87o9979gfn.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: 1.0 (+)
X-Debbugs-Envelope-To: 33848
Cc: Pierre Neidhardt <mail@HIDDEN>, 33848 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.0 (/)

Hello,

Mark H Weaver <mhw@HIDDEN> skribis:

> Pierre Neidhardt <mail@HIDDEN> writes:
>
>>> : > Store file names are always ASCII so problems arise when they are s=
tored
>>> : > as UTF-16 or UTF-32/UCS-4.
>>> :=20
>>> : I understand that most programs stick to ASCII filenames, but what ab=
out the odd
>>> : one using non-English, special characters?
>>>=20
>>> That=E2=80=99s a separate debate.  :-)  Essentially this restriction on=
 store
>>> file names has always been there in Guix (and Nix before that).  If we
>>> were to change it, that would raise compatibility issues.
>>
>> But what happens if we attempt to store "=C3=A1" in the store?
>
> Indeed.  Although we might restrict the immediate entries within
> /gnu/store to ASCII characters, file names deeper within those
> directories may have non-ASCII characters.  More generally, store
> references may occur within larger strings which might include non-ASCII
> characters.

Right.  For example =E2=80=98nss-certs=E2=80=99 contains non-ASCII, UTF-8-e=
ncoded file
names.

For =E2=80=9Ctop-level=E2=80=9D store file names, the restriction is enforc=
ed by
=E2=80=98checkStoreName=E2=80=99 in libstore/store-api.cc.

Ludo=E2=80=99.




Information forwarded to bug-guix@HIDDEN:
bug#33848; Package guix. Full text available.

Message received at 33848 <at> debbugs.gnu.org:


Received: (at 33848) by debbugs.gnu.org; 27 Dec 2018 14:30:57 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Dec 27 09:30:57 2018
Received: from localhost ([127.0.0.1]:38280 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gcWga-0001Ua-Oa
	for submit <at> debbugs.gnu.org; Thu, 27 Dec 2018 09:30:57 -0500
Received: from world.peace.net ([64.112.178.59]:50632)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mhw@HIDDEN>) id 1gcWgY-0001UM-MY
 for 33848 <at> debbugs.gnu.org; Thu, 27 Dec 2018 09:30:55 -0500
Received: from mhw by world.peace.net with esmtpsa
 (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89)
 (envelope-from <mhw@HIDDEN>)
 id 1gcWgS-0003Wu-St; Thu, 27 Dec 2018 09:30:48 -0500
From: Mark H Weaver <mhw@HIDDEN>
To: Danny Milosavljevic <dannym@HIDDEN>
Subject: Re: bug#33848: Store references in SBCL-compiled code are "invisible"
References: <87r2e8jpfx.fsf@HIDDEN> <877eg0i43j.fsf@HIDDEN>
 <87d0psi1xo.fsf@HIDDEN> <874lb3kin6.fsf@HIDDEN>
 <87sgynezha.fsf@HIDDEN> <87tvj2yesd.fsf@HIDDEN>
 <20181227145258.0c420eac@HIDDEN>
Date: Thu, 27 Dec 2018 09:29:42 -0500
In-Reply-To: <20181227145258.0c420eac@HIDDEN> (Danny Milosavljevic's
 message of "Thu, 27 Dec 2018 14:52:58 +0100")
Message-ID: <87pntnvy8e.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 33848
Cc: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>,
 Pierre Neidhardt <mail@HIDDEN>, 33848 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Hi Danny,

Danny Milosavljevic <dannym@HIDDEN> writes:

> On Mon, 24 Dec 2018 13:12:23 -0500
> Mark H Weaver <mhw@HIDDEN> wrote:
>
>> Of course, the usual reason to choose UTF-32 is to support non-ASCII
>> characters while retaining fixed-width code points, so that string
>> lookups are straightforward and efficient.
>
> This kind of lookup is almost never what is necessary.  There are many
> people who assume character is the same as codepoint and to those people
> UTF-32 brings something to the table, but it's really not useful if people
> do text processing correctly, see below.
>
> (Of course whether packages actually do this remains to be seen)
>
>>  That extra
>> complexity is what I guess we would need to add to each program that
>> currently uses UTF-32.
>
> Yes, but they usually have to do stream processing even with UTF-32 (because
> a character can be composed of possibly infinite number of codepoints),

I agree with you.  However, as silly as it might be, the fact remains
that almost every modern programming language and string library uses
code points as the base units by which to index strings.

> so the infrastructure should be already there and the effort should be
> minimal.

The infrastructure might or might not be there, depending on the
sophistication of the program's unicode support, but even if it _is_
there, it will most likely be a layer that expects to iterate over
strings indexed by code point to find graphemes, etc.

Anyway, if you truly believe the effort should be minimal, feel free to
investigate and propose patches to fix our 5 common lisp compilers and
Fish to avoid storing UTF-32 in the object code.

> Also, if both UTF-32 and UTF-8 are used on disk, care needs to not misdetect
> an UTF-8 sequence as an UTF-32 sequence of different text - or the other way
> around -, but that's unlikely for ASCII strings.

This is not an issue because the substrings that the reference scanner
and grafter are looking for are ASCII-only, even if they are part of a
larger non-ASCII string.  Specifically, they only need to look for the
nix hashes.

>> I really think it would be a mistake to try to force every program and
>> language implementation to use our preferred string representation.  I
>> suspect it would be vastly easier to compromise and support a few other
>> popular string representations in Guix, namely UTF-16 and UTF-32.
>
> In 1992, UTF-8 was invented.  Subsequently, most of the Internet,
> all new GNU Linux distributions etc, all UNIX GUI frameworks, Subversion
> etc standardized on UTF-8, with the eventual goal of standardizing all
> network transfer and storage to UTF-8.  I think that by now the outliers
> are the ones who need to change,

I agree that we need to standardize on Unicode.  However, given the
perhaps unfortunate fact that almost everyone has standardized on code
points as the units by which to index strings, choosing UTF-32 as an
internal representation is a very reasonable choice, IMO.

Anyway, feel free to engage with the developers of the Common Lisp
implementations that use UTF-32 and try to convince them to change.

The remaining question is: what to do if upstream refuses to change?  Do
we exclude that software in Guix, or do we maintain our own patches to
override upstream's decision?

>> If you don't want to change the daemon, it could be worked around in our
>> build-side code as follows: we could add a new phase to certain build
>> systems (or possibly gnu-build-system) that scans each output for
>> UTF-16/32 encoded store references that are never referenced in UTF-8.
>> If such references exist, a file with an unobtrusive name would be added
>> to that output containing those references encoded in UTF-8.  This would
>> enable our daemon's existing reference scanner to find all of the
>> references.
>
> I agree that that would be nice.  As a first step, even just detecting
> problems like that and erroring out would be okay - in order to find them
> in the first place.  Right now, it's difficult to detect and so also difficult
> to say how wide-spread the problem is.  If the problem is wide-spread enough
> my tune could change very quickly.

Sure, it would be useful to have more data on what packages are
currently affected by this issue.

      Regards,
        Mark




Information forwarded to bug-guix@HIDDEN:
bug#33848; Package guix. Full text available.

Message received at 33848 <at> debbugs.gnu.org:


Received: (at 33848) by debbugs.gnu.org; 27 Dec 2018 14:04:35 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Dec 27 09:04:35 2018
Received: from localhost ([127.0.0.1]:38275 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gcWH1-0000nz-Ql
	for submit <at> debbugs.gnu.org; Thu, 27 Dec 2018 09:04:35 -0500
Received: from world.peace.net ([64.112.178.59]:50574)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mhw@HIDDEN>) id 1gcWGw-0000na-J5
 for 33848 <at> debbugs.gnu.org; Thu, 27 Dec 2018 09:04:30 -0500
Received: from mhw by world.peace.net with esmtpsa
 (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89)
 (envelope-from <mhw@HIDDEN>)
 id 1gcWGp-0003PY-UJ; Thu, 27 Dec 2018 09:04:19 -0500
From: Mark H Weaver <mhw@HIDDEN>
To: Pierre Neidhardt <mail@HIDDEN>
Subject: Re: bug#33848: Store references in SBCL-compiled code are "invisible"
References: <87r2e8jpfx.fsf@HIDDEN> <877eg0i43j.fsf@HIDDEN>
 <87d0psi1xo.fsf@HIDDEN> <874lb3kin6.fsf@HIDDEN>
 <87sgynezha.fsf@HIDDEN> <87tvj2yesd.fsf@HIDDEN>
 <877efwe04u.fsf@HIDDEN> <8736qji7c1.fsf@HIDDEN>
Date: Thu, 27 Dec 2018 09:03:12 -0500
In-Reply-To: <8736qji7c1.fsf@HIDDEN> (Pierre Neidhardt's message of
 "Thu, 27 Dec 2018 11:37:18 +0100")
Message-ID: <87tvizvzgk.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 33848
Cc: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>, 33848 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Pierre Neidhardt <mail@HIDDEN> writes:

>> : > Store file names are always ASCII so problems arise when they are st=
ored
>> : > as UTF-16 or UTF-32/UCS-4.
>> :=20
>> : I understand that most programs stick to ASCII filenames, but what abo=
ut the odd
>> : one using non-English, special characters?
>>=20
>> That=E2=80=99s a separate debate.  :-)  Essentially this restriction on =
store
>> file names has always been there in Guix (and Nix before that).  If we
>> were to change it, that would raise compatibility issues.
>
> But what happens if we attempt to store "=C3=A1" in the store?

Indeed.  Although we might restrict the immediate entries within
/gnu/store to ASCII characters, file names deeper within those
directories may have non-ASCII characters.  More generally, store
references may occur within larger strings which might include non-ASCII
characters.

       Mark




Information forwarded to bug-guix@HIDDEN:
bug#33848; Package guix. Full text available.

Message received at 33848 <at> debbugs.gnu.org:


Received: (at 33848) by debbugs.gnu.org; 27 Dec 2018 13:53:13 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Dec 27 08:53:13 2018
Received: from localhost ([127.0.0.1]:38272 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gcW64-0000SO-JX
	for submit <at> debbugs.gnu.org; Thu, 27 Dec 2018 08:53:12 -0500
Received: from dd26836.kasserver.com ([85.13.145.193]:50892)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dannym@HIDDEN>) id 1gcW61-0000SE-8u
 for 33848 <at> debbugs.gnu.org; Thu, 27 Dec 2018 08:53:11 -0500
Received: from localhost (77.116.200.150.wireless.dyn.drei.com
 [77.116.200.150])
 by dd26836.kasserver.com (Postfix) with ESMTPSA id E30FC3360147;
 Thu, 27 Dec 2018 14:53:06 +0100 (CET)
Date: Thu, 27 Dec 2018 14:52:58 +0100
From: Danny Milosavljevic <dannym@HIDDEN>
To: Mark H Weaver <mhw@HIDDEN>
Subject: Re: bug#33848: Store references in SBCL-compiled code are "invisible"
Message-ID: <20181227145258.0c420eac@HIDDEN>
In-Reply-To: <87tvj2yesd.fsf@HIDDEN>
References: <87r2e8jpfx.fsf@HIDDEN> <877eg0i43j.fsf@HIDDEN>
 <87d0psi1xo.fsf@HIDDEN> <874lb3kin6.fsf@HIDDEN>
 <87sgynezha.fsf@HIDDEN> <87tvj2yesd.fsf@HIDDEN>
X-Mailer: Claws Mail 3.17.1 (GTK+ 2.24.32; x86_64-unknown-linux-gnu)
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha256;
 boundary="Sig_/psKMAB3Tik_KCO1Se691Qt1"; protocol="application/pgp-signature"
X-Spam-Score: -0.7 (/)
X-Debbugs-Envelope-To: 33848
Cc: Ludovic =?ISO-8859-1?Q?Court=E8s?= <ludo@HIDDEN>,
 Pierre Neidhardt <mail@HIDDEN>, 33848 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

--Sig_/psKMAB3Tik_KCO1Se691Qt1
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: quoted-printable

Hi Mark,

On Mon, 24 Dec 2018 13:12:23 -0500
Mark H Weaver <mhw@HIDDEN> wrote:

> Of course, the usual reason to choose UTF-32 is to support non-ASCII
> characters while retaining fixed-width code points, so that string
> lookups are straightforward and efficient.

This kind of lookup is almost never what is necessary.  There are many
people who assume character is the same as codepoint and to those people
UTF-32 brings something to the table, but it's really not useful if people
do text processing correctly, see below.

(Of course whether packages actually do this remains to be seen)

>  Using UTF-8 improves space efficiency, but at the cost of extra code
>complexity.

I agree.

>  That extra
> complexity is what I guess we would need to add to each program that
> currently uses UTF-32.

Yes, but they usually have to do stream processing even with UTF-32 (because
a character can be composed of possibly infinite number of codepoints),
so the infrastructure should be already there and the effort should be
minimal.

>  Alternatively, we could extend the on-disk
> format to support UTF-8 and then add some kind of "load hook" that
> converts the string to UTF-32 at load time.  Either way, it's likely to
> be a can of worms.

If it ever came to that, a pluggable reference scanner would be=20
preferrable.  But really, it would irk me to have so much complexity
in something so basic (the reference scanner) for no end-user gain
(as a distribution we could just mandate UTF-8 for references and the
problem would be gone for the user with no loss of functionality).

It's always easy to add special cases - but more code means more bugs
and I think if possible it's best to have only the simple case implemented
in the core - because it's less complicated which means more likely
to be correct (for the case it does handle).  In the end it depends on
what would be more code, and more widely used.

Also, if we wanted to debug reference errors, we couldn't use grep anymore
because it can't handle utf-32 either (neither can any of the other UNIX to=
ols).

Also, I really don't want to return to the time where I had to call iconv
once every three commands to be able to do anything useful on UNIX.

Also, the build daemon is written in C++ and C++ strings are widely
known to have very very bad codepoint awareness (to say nothing about
the horrible conversion facilities).

Also, if both UTF-32 and UTF-8 are used on disk, care needs to not misdetect
an UTF-8 sequence as an UTF-32 sequence of different text - or the other way
around -, but that's unlikely for ASCII strings.

> I really think it would be a mistake to try to force every program and
> language implementation to use our preferred string representation.  I
> suspect it would be vastly easier to compromise and support a few other
> popular string representations in Guix, namely UTF-16 and UTF-32.

In 1992, UTF-8 was invented.  Subsequently, most of the Internet,
all new GNU Linux distributions etc, all UNIX GUI frameworks, Subversion
etc standardized on UTF-8, with the eventual goal of standardizing all
network transfer and storage to UTF-8.  I think that by now the outliers
are the ones who need to change, otherwise these senseless encoding
conversions will never cease.  It's not like different encodings allow for
better expression of writings or anything useful to the end user.

As a distribution we can't force upstream to change, but just filing
bug reports upstream would make us see where they stand on this.

> If you don't want to change the daemon, it could be worked around in our
> build-side code as follows: we could add a new phase to certain build
> systems (or possibly gnu-build-system) that scans each output for
> UTF-16/32 encoded store references that are never referenced in UTF-8.
> If such references exist, a file with an unobtrusive name would be added
> to that output containing those references encoded in UTF-8.  This would
> enable our daemon's existing reference scanner to find all of the
> references.

I agree that that would be nice.  As a first step, even just detecting
problems like that and erroring out would be okay - in order to find them
in the first place.  Right now, it's difficult to detect and so also diffic=
ult
to say how wide-spread the problem is.  If the problem is wide-spread enough
my tune could change very quickly.

What you propose is similar to what I did in Java in Guix, only it gives
us even more advantages in the Java case (faster class loading and
eventual non-propagated inputs).

--Sig_/psKMAB3Tik_KCO1Se691Qt1
Content-Type: application/pgp-signature
Content-Description: OpenPGP digital signature

-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEds7GsXJ0tGXALbPZ5xo1VCwwuqUFAlwk2ToACgkQ5xo1VCww
uqUUzAgApbxUHv/XlbjYMXvV4cOY0maxbx92ndZlJiukCN+bIiMqhuCd7PdEoL7Z
1d9ABxe+2oXO4Nkjpez71nhK8ym8KwRYNDuTkCSZbzUJwNEee2pF/OlU2Y+Jugz5
ICSlYGCFfwx6Buf9bZReYq1e5qjO//QSytgYC061gYURw/abtGSEyvllHWv4qrl6
DFfQuQilycHAOqrT/ACBtMgFFnsV7miHs6CrKTSPPBWKKuA3BM4STNUfHlMeb8un
gNUH3ijbDviqBgRiqDy50dZ0kbFv8zSm1LytoySX0qZ7j5oidDJGHATGbEXp4sDU
1dPmBSYeasLAVfJn4RQSQFAKba6TuA==
=gkV6
-----END PGP SIGNATURE-----

--Sig_/psKMAB3Tik_KCO1Se691Qt1--




Information forwarded to bug-guix@HIDDEN:
bug#33848; Package guix. Full text available.

Message received at 33848 <at> debbugs.gnu.org:


Received: (at 33848) by debbugs.gnu.org; 27 Dec 2018 10:37:25 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Dec 27 05:37:25 2018
Received: from localhost ([127.0.0.1]:38203 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gcT2b-0008Jf-8V
	for submit <at> debbugs.gnu.org; Thu, 27 Dec 2018 05:37:25 -0500
Received: from relay10.mail.gandi.net ([217.70.178.230]:51675)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mail@HIDDEN>) id 1gcT2Y-0008JW-Hp
 for 33848 <at> debbugs.gnu.org; Thu, 27 Dec 2018 05:37:23 -0500
Received: from mimimi (moi44-1-78-199-129-170.fbx.proxad.net [78.199.129.170])
 (Authenticated sender: mail@HIDDEN)
 by relay10.mail.gandi.net (Postfix) with ESMTPSA id 0AC2B240002;
 Thu, 27 Dec 2018 10:37:19 +0000 (UTC)
References: <87r2e8jpfx.fsf@HIDDEN> <877eg0i43j.fsf@HIDDEN>
 <87d0psi1xo.fsf@HIDDEN> <874lb3kin6.fsf@HIDDEN>
 <87sgynezha.fsf@HIDDEN> <87tvj2yesd.fsf@HIDDEN> <877efwe04u.fsf@HIDDEN>
User-agent: mu4e 1.0; emacs 26.1
From: Pierre Neidhardt <mail@HIDDEN>
To: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Subject: Re: bug#33848: Store references in SBCL-compiled code are "invisible"
In-reply-to: <877efwe04u.fsf@HIDDEN>
Date: Thu, 27 Dec 2018 11:37:18 +0100
Message-ID: <8736qji7c1.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="=-=-=";
 micalg=pgp-sha256; protocol="application/pgp-signature"
X-Spam-Score: -0.7 (/)
X-Debbugs-Envelope-To: 33848
Cc: Mark H Weaver <mhw@HIDDEN>, 33848 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

--=-=-=
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


> : > Store file names are always ASCII so problems arise when they are sto=
red
> : > as UTF-16 or UTF-32/UCS-4.
> :=20
> : I understand that most programs stick to ASCII filenames, but what abou=
t the odd
> : one using non-English, special characters?
>=20
> That=E2=80=99s a separate debate.  :-)  Essentially this restriction on s=
tore
> file names has always been there in Guix (and Nix before that).  If we
> were to change it, that would raise compatibility issues.

But what happens if we attempt to store "=C3=A1" in the store?

> For example I guess we could always store the file name as a literal
> byte vector/list and add a call to turn that into a string.

In the case of Next, that would be a simple patch, but other programs could=
 get
much more complicated.  In the end, this approach requires a linear amount =
of
work.  Conversely, adding UCS-* support to the scanner would fix this issue=
 once
and for all.

> : > We did have a problem with Fish but I can no longer find it.  Do you
> : > remember what it was?  Something with C++, no?
> :=20
> : I think bug #30265.
>=20
> Oh I see, UCS-4 as well.  (I can=E2=80=99t believe this bug is still open=
 given
> the relatively simple solutions outlined at
> <https://issues.guix.info/issue/30265#8>.  :-))

Well, if currently only two packages out of 8500+ suffer from this, then I =
think
it's easier to go with Ludo's suggestion of patching the code to use ASCII
strings.

Does anyone know about more packages with this issue?  It could also be that
more packages suffer from this, unbeknownst to us.

=2D-=20
Pierre Neidhardt
https://ambrevar.xyz/

--=-=-=
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEUPM+LlsMPZAEJKvom9z0l6S7zH8FAlwkq14ACgkQm9z0l6S7
zH9zbAf8C7alkC/FiNu4pb3HkuSWZXKkZ/pccOIXH0ErCiND6SwQC9pBXTgxoYew
p9Y3J0SrKyUMVKHidWERkA1EnVR6wBUT3sru6idmiNF2JIBw5JC+UiNdiS5RqvXd
Ka3eHjqxVXfL2kEINOOSoiB1t6P6chQsxHJjxOs9TTk+8UgFgDMF9VhtYubiaLYf
oBOP7FVAIojHHGxth14ekyohT65TD4mgRqK3mTsLxPjrQ43/nAayo6aJWilx5BB1
YoRe8bjUNzHS1G0JSsM6E8ZRwwUfwBBhwdqFml2O76LpJoWi/xi358JNldRqD7j/
eV0ZNuJAZjONvVJZ9qtfJDifLPJkJQ==
=IU/3
-----END PGP SIGNATURE-----
--=-=-=--




Information forwarded to bug-guix@HIDDEN:
bug#33848; Package guix. Full text available.

Message received at 33848 <at> debbugs.gnu.org:


Received: (at 33848) by debbugs.gnu.org; 26 Dec 2018 16:14:16 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Wed Dec 26 11:14:15 2018
Received: from localhost ([127.0.0.1]:37888 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gcBp0-0001Gz-E6
	for submit <at> debbugs.gnu.org; Wed, 26 Dec 2018 11:14:15 -0500
Received: from hera.aquilenet.fr ([185.233.100.1]:44360)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1gcBoy-0001Gr-0t
 for 33848 <at> debbugs.gnu.org; Wed, 26 Dec 2018 11:14:12 -0500
Received: from localhost (localhost [127.0.0.1])
 by hera.aquilenet.fr (Postfix) with ESMTP id 7B83917CE;
 Wed, 26 Dec 2018 17:14:11 +0100 (CET)
X-Virus-Scanned: Debian amavisd-new at aquilenet.fr
Received: from hera.aquilenet.fr ([127.0.0.1])
 by localhost (hera.aquilenet.fr [127.0.0.1]) (amavisd-new, port 10024)
 with ESMTP id DDHfqvVGlezg; Wed, 26 Dec 2018 17:14:10 +0100 (CET)
Received: from ribbon (unknown [IPv6:2a01:e0a:1d:7270:af76:b9b:ca24:c465])
 by hera.aquilenet.fr (Postfix) with ESMTPSA id BF6031753;
 Wed, 26 Dec 2018 17:14:09 +0100 (CET)
From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>
To: Mark H Weaver <mhw@HIDDEN>
Subject: Re: bug#33848: Store references in SBCL-compiled code are "invisible"
References: <87r2e8jpfx.fsf@HIDDEN> <877eg0i43j.fsf@HIDDEN>
 <87d0psi1xo.fsf@HIDDEN> <874lb3kin6.fsf@HIDDEN>
 <87sgynezha.fsf@HIDDEN> <87tvj2yesd.fsf@HIDDEN>
X-URL: http://www.fdn.fr/~lcourtes/
X-Revolutionary-Date: 6 =?utf-8?Q?Niv=C3=B4se?= an 227 de la =?utf-8?Q?R?=
 =?utf-8?Q?=C3=A9volution?=
X-PGP-Key-ID: 0x090B11993D9AEBB5
X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc
X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4  0CFB 090B 1199 3D9A EBB5
X-OS: x86_64-pc-linux-gnu
Date: Wed, 26 Dec 2018 17:14:09 +0100
In-Reply-To: <87tvj2yesd.fsf@HIDDEN> (Mark H. Weaver's message of "Mon, 24
 Dec 2018 13:12:23 -0500")
Message-ID: <877efwe04u.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: 1.0 (+)
X-Debbugs-Envelope-To: 33848
Cc: Pierre Neidhardt <mail@HIDDEN>, 33848 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.0 (/)

Hello!

Mark H Weaver <mhw@HIDDEN> skribis:

> Ludovic Court=C3=A8s <ludo@HIDDEN> writes:
>
>> Pierre Neidhardt <mail@HIDDEN> skribis:
>>
>>>> For now I lean towards looking for a way to address the issue
>>>> specifically for SBCL.
>>>
>>> Don't forget that we currently have 5 Lisp compilers.
>>> Besides, it's not clear that this can be fixed on the compiler's side, =
it could
>>> very well be that patches will be required  on a per-project basis.
>>
>> I know little about CL but maybe we can find a solution that works for
>> all five compilers.  At least that would be the first approach I would
>> suggest following.
>
> I can't imagine a solution that would work for all five compilers, but
> perhaps that's a failure of imagination on my part.  Of course, you're
> welcome to search for such a solution.  Can you give me a rough outline
> of what you have in mind?

I have nothing specific in mind, I=E2=80=99m just brainstorming with everyo=
ne
here.  :-)

For a similar situation in C++, there=E2=80=99s a fairly simple and local
workaround:

  https://issues.guix.info/issue/30265#8

I=E2=80=99m not familiar with CL but I thought that it we could achieve
something similar, that would be great=E2=80=94I=E2=80=99m not suggesting t=
o change the
CL compilers in any non-trivial way.

For example I guess we could always store the file name as a literal
byte vector/list and add a call to turn that into a string.

Does that make sense?

Thanks,
Ludo=E2=80=99.




Information forwarded to bug-guix@HIDDEN:
bug#33848; Package guix. Full text available.

Message received at 33848 <at> debbugs.gnu.org:


Received: (at 33848) by debbugs.gnu.org; 26 Dec 2018 16:07:40 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Wed Dec 26 11:07:40 2018
Received: from localhost ([127.0.0.1]:37878 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gcBie-00016Y-4U
	for submit <at> debbugs.gnu.org; Wed, 26 Dec 2018 11:07:40 -0500
Received: from hera.aquilenet.fr ([185.233.100.1]:44310)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1gcBia-00016N-Ka
 for 33848 <at> debbugs.gnu.org; Wed, 26 Dec 2018 11:07:38 -0500
Received: from localhost (localhost [127.0.0.1])
 by hera.aquilenet.fr (Postfix) with ESMTP id AE20C1753;
 Wed, 26 Dec 2018 17:07:35 +0100 (CET)
X-Virus-Scanned: Debian amavisd-new at aquilenet.fr
Received: from hera.aquilenet.fr ([127.0.0.1])
 by localhost (hera.aquilenet.fr [127.0.0.1]) (amavisd-new, port 10024)
 with ESMTP id N7ShGo81cct7; Wed, 26 Dec 2018 17:07:34 +0100 (CET)
Received: from ribbon (unknown [IPv6:2a01:e0a:1d:7270:af76:b9b:ca24:c465])
 by hera.aquilenet.fr (Postfix) with ESMTPSA id 65F6CAAB;
 Wed, 26 Dec 2018 17:07:34 +0100 (CET)
From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>
To: Pierre Neidhardt <mail@HIDDEN>
Subject: Re: bug#33848: Store references in SBCL-compiled code are "invisible"
References: <87r2e8jpfx.fsf@HIDDEN> <877eg0i43j.fsf@HIDDEN>
 <87d0psi1xo.fsf@HIDDEN> <874lb3kin6.fsf@HIDDEN>
 <87sgynezha.fsf@HIDDEN> <87r2e6j1hw.fsf@HIDDEN>
X-URL: http://www.fdn.fr/~lcourtes/
X-Revolutionary-Date: 6 =?utf-8?Q?Niv=C3=B4se?= an 227 de la =?utf-8?Q?R?=
 =?utf-8?Q?=C3=A9volution?=
X-PGP-Key-ID: 0x090B11993D9AEBB5
X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc
X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4  0CFB 090B 1199 3D9A EBB5
X-OS: x86_64-pc-linux-gnu
Date: Wed, 26 Dec 2018 17:07:33 +0100
In-Reply-To: <87r2e6j1hw.fsf@HIDDEN> (Pierre Neidhardt's message of
 "Mon, 24 Dec 2018 18:08:59 +0100")
Message-ID: <87r2e4e0fu.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: 1.0 (+)
X-Debbugs-Envelope-To: 33848
Cc: Mark H Weaver <mhw@HIDDEN>, 33848 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.0 (/)

Pierre Neidhardt <mail@HIDDEN> skribis:

>> Store file names are always ASCII so problems arise when they are stored
>> as UTF-16 or UTF-32/UCS-4.
>
> I understand that most programs stick to ASCII filenames, but what about =
the odd
> one using non-English, special characters?

That=E2=80=99s a separate debate.  :-)  Essentially this restriction on sto=
re
file names has always been there in Guix (and Nix before that).  If we
were to change it, that would raise compatibility issues.

>> We did have a problem with Fish but I can no longer find it.  Do you
>> remember what it was?  Something with C++, no?
>
> I think bug #30265.

Oh I see, UCS-4 as well.  (I can=E2=80=99t believe this bug is still open g=
iven
the relatively simple solutions outlined at
<https://issues.guix.info/issue/30265#8>.  :-))

Thanks,
Ludo=E2=80=99.




Information forwarded to bug-guix@HIDDEN:
bug#33848; Package guix. Full text available.

Message received at 33848 <at> debbugs.gnu.org:


Received: (at 33848) by debbugs.gnu.org; 24 Dec 2018 23:58:44 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Dec 24 18:58:44 2018
Received: from localhost ([127.0.0.1]:35668 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gba7M-0002K7-Av
	for submit <at> debbugs.gnu.org; Mon, 24 Dec 2018 18:58:44 -0500
Received: from relay12.mail.gandi.net ([217.70.178.232]:57635)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mail@HIDDEN>) id 1gba7H-0002Jt-5P
 for 33848 <at> debbugs.gnu.org; Mon, 24 Dec 2018 18:58:39 -0500
Received: from mimimi (moi44-1-78-199-129-170.fbx.proxad.net [78.199.129.170])
 (Authenticated sender: mail@HIDDEN)
 by relay12.mail.gandi.net (Postfix) with ESMTPSA id 0C571200004;
 Mon, 24 Dec 2018 23:58:32 +0000 (UTC)
References: <87r2e8jpfx.fsf@HIDDEN> <877eg0i43j.fsf@HIDDEN>
 <87d0psi1xo.fsf@HIDDEN> <874lb3kin6.fsf@HIDDEN>
 <87sgynezha.fsf@HIDDEN> <87tvj2yesd.fsf@HIDDEN>
User-agent: mu4e 1.0; emacs 26.1
From: Pierre Neidhardt <mail@HIDDEN>
To: Mark H Weaver <mhw@HIDDEN>
Subject: Re: bug#33848: Store references in SBCL-compiled code are "invisible"
In-reply-to: <87tvj2yesd.fsf@HIDDEN>
Date: Tue, 25 Dec 2018 00:58:31 +0100
Message-ID: <87pntqiijc.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="=-=-=";
 micalg=pgp-sha256; protocol="application/pgp-signature"
X-Spam-Score: -0.7 (/)
X-Debbugs-Envelope-To: 33848
Cc: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>, 33848 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

--=-=-=
Content-Type: text/plain
Content-Transfer-Encoding: quoted-printable

I find Mark's points reasonable, although to be honest I have very little
knowledge of the daemon.

=2D-=20
Pierre Neidhardt
https://ambrevar.xyz/

--=-=-=
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEUPM+LlsMPZAEJKvom9z0l6S7zH8FAlwhcqcACgkQm9z0l6S7
zH/OWAf9EAHvRknh8vP7fQlEH0koh30BL0A62gzcn25Aw/Un5zPR7ZZ5vBhm7FqQ
fOS3fGF+ZYwTd5QWHz/49sNMJes4caaNqxN9x1xm/IBU374/MEkxtvnJqNctYL0z
MAIXlruvch+cBYFfAyuCDjkNqFBHuqlFPP1lZCbal6xHvirMLLzNfRhQcFtYXD0T
y3YN0D5T9KfgQcrDEf78ShJSBto7lyBMKe9PqJBeKJexrzkD1XsY+sZB0PXiSrTN
CT/tC2MqT8QMRrfGNpEepQIHMqowjVheJ3vcC5NDEKT7IFdY5art5d96+QpFlXNl
RrCgC+3b8NGpxoWSR4oGqpR3FrjzEw==
=Apxv
-----END PGP SIGNATURE-----
--=-=-=--




Information forwarded to bug-guix@HIDDEN:
bug#33848; Package guix. Full text available.

Message received at 33848 <at> debbugs.gnu.org:


Received: (at 33848) by debbugs.gnu.org; 24 Dec 2018 18:13:38 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Dec 24 13:13:38 2018
Received: from localhost ([127.0.0.1]:35618 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gbUjR-0002Px-Ot
	for submit <at> debbugs.gnu.org; Mon, 24 Dec 2018 13:13:38 -0500
Received: from world.peace.net ([64.112.178.59]:37592)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mhw@HIDDEN>) id 1gbUjQ-0002Pl-Im
 for 33848 <at> debbugs.gnu.org; Mon, 24 Dec 2018 13:13:36 -0500
Received: from mhw by world.peace.net with esmtpsa
 (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89)
 (envelope-from <mhw@HIDDEN>)
 id 1gbUjJ-0004Wz-Ok; Mon, 24 Dec 2018 13:13:29 -0500
From: Mark H Weaver <mhw@HIDDEN>
To: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Subject: Re: bug#33848: Store references in SBCL-compiled code are "invisible"
References: <87r2e8jpfx.fsf@HIDDEN> <877eg0i43j.fsf@HIDDEN>
 <87d0psi1xo.fsf@HIDDEN> <874lb3kin6.fsf@HIDDEN>
 <87sgynezha.fsf@HIDDEN>
Date: Mon, 24 Dec 2018 13:12:23 -0500
In-Reply-To: <87sgynezha.fsf@HIDDEN> ("Ludovic
 \=\?utf-8\?Q\?Court\=C3\=A8s\=22'\?\=
 \=\?utf-8\?Q\?s\?\= message of "Mon, 24 Dec 2018 16:06:09 +0100")
Message-ID: <87tvj2yesd.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 33848
Cc: Pierre Neidhardt <mail@HIDDEN>, 33848 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Hi Ludovic,

Ludovic Court=C3=A8s <ludo@HIDDEN> writes:

> Pierre Neidhardt <mail@HIDDEN> skribis:
>
>>> For now I lean towards looking for a way to address the issue
>>> specifically for SBCL.
>>
>> Don't forget that we currently have 5 Lisp compilers.
>> Besides, it's not clear that this can be fixed on the compiler's side, i=
t could
>> very well be that patches will be required  on a per-project basis.
>
> I know little about CL but maybe we can find a solution that works for
> all five compilers.  At least that would be the first approach I would
> suggest following.

I can't imagine a solution that would work for all five compilers, but
perhaps that's a failure of imagination on my part.  Of course, you're
welcome to search for such a solution.  Can you give me a rough outline
of what you have in mind?

Of course, the usual reason to choose UTF-32 is to support non-ASCII
characters while retaining fixed-width code points, so that string
lookups are straightforward and efficient.  Using UTF-8 improves space
efficiency, but at the cost of extra code complexity.  That extra
complexity is what I guess we would need to add to each program that
currently uses UTF-32.  Alternatively, we could extend the on-disk
format to support UTF-8 and then add some kind of "load hook" that
converts the string to UTF-32 at load time.  Either way, it's likely to
be a can of worms.

Consider the case of Guile.  Years ago we agreed to switch to UTF-8 as
its sole internal string encoding, but it hasn't yet been done because
it's a big job, even for those of us already intimately familiar with
the code.

Now imagine how hard it would be for someone who barely uses Guile, but
nevertheless felt compelled to change our internal string representation
to use UTF-8.  Moreover, imagine that they hoped to find a single
solution that would work for several different Scheme implementations.

What would you say to them if they proposed to find a general solution
to convert several Scheme implementations to use UTF-8 as their string
representation, to save themselves the trouble of having to understand
each implementation individually?

I really think it would be a mistake to try to force every program and
language implementation to use our preferred string representation.  I
suspect it would be vastly easier to compromise and support a few other
popular string representations in Guix, namely UTF-16 and UTF-32.

If you don't want to change the daemon, it could be worked around in our
build-side code as follows: we could add a new phase to certain build
systems (or possibly gnu-build-system) that scans each output for
UTF-16/32 encoded store references that are never referenced in UTF-8.
If such references exist, a file with an unobtrusive name would be added
to that output containing those references encoded in UTF-8.  This would
enable our daemon's existing reference scanner to find all of the
references.

Our grafting code would then need to be extended to recognize and
transform store references encoded in UTF-16/32 as well as UTF-8.

What do you think?

      Regards,
        Mark




Information forwarded to bug-guix@HIDDEN:
bug#33848; Package guix. Full text available.

Message received at 33848 <at> debbugs.gnu.org:


Received: (at 33848) by debbugs.gnu.org; 24 Dec 2018 17:09:04 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Dec 24 12:09:04 2018
Received: from localhost ([127.0.0.1]:35589 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gbTiy-0000iy-71
	for submit <at> debbugs.gnu.org; Mon, 24 Dec 2018 12:09:04 -0500
Received: from relay9-d.mail.gandi.net ([217.70.183.199]:58059)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mail@HIDDEN>) id 1gbTiw-0000id-4i
 for 33848 <at> debbugs.gnu.org; Mon, 24 Dec 2018 12:09:02 -0500
X-Originating-IP: 78.199.129.170
Received: from mimimi (moi44-1-78-199-129-170.fbx.proxad.net [78.199.129.170])
 (Authenticated sender: mail@HIDDEN)
 by relay9-d.mail.gandi.net (Postfix) with ESMTPSA id 09C0EFF808;
 Mon, 24 Dec 2018 17:08:59 +0000 (UTC)
References: <87r2e8jpfx.fsf@HIDDEN> <877eg0i43j.fsf@HIDDEN>
 <87d0psi1xo.fsf@HIDDEN> <874lb3kin6.fsf@HIDDEN>
 <87sgynezha.fsf@HIDDEN>
User-agent: mu4e 1.0; emacs 26.1
From: Pierre Neidhardt <mail@HIDDEN>
To: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Subject: Re: bug#33848: Store references in SBCL-compiled code are "invisible"
In-reply-to: <87sgynezha.fsf@HIDDEN>
Date: Mon, 24 Dec 2018 18:08:59 +0100
Message-ID: <87r2e6j1hw.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="=-=-=";
 micalg=pgp-sha256; protocol="application/pgp-signature"
X-Spam-Score: -0.7 (/)
X-Debbugs-Envelope-To: 33848
Cc: Mark H Weaver <mhw@HIDDEN>, 33848 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

--=-=-=
Content-Type: text/plain
Content-Transfer-Encoding: quoted-printable


> Store file names are always ASCII so problems arise when they are stored
> as UTF-16 or UTF-32/UCS-4.

I understand that most programs stick to ASCII filenames, but what about th=
e odd
one using non-English, special characters?

> We did have a problem with Fish but I can no longer find it.  Do you
> remember what it was?  Something with C++, no?

I think bug #30265.

=2D-=20
Pierre Neidhardt
https://ambrevar.xyz/

--=-=-=
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEUPM+LlsMPZAEJKvom9z0l6S7zH8FAlwhEqsACgkQm9z0l6S7
zH8nhAf9Hv2U1ajhnsl50XrKSr629VR3LFtu6whoiU3WJOygmulOIdlaWJ2IRFSR
mwCvD8I/pE+BokAgT28BQNpvyG78+vgJeevb4adTD8eUxQsS2aRLPvJ9Js3B4epY
tTDxtm6xp/kFKmxk/9WFYX/lxuyXfSYv/A7m8q3qWfngzvizZjCZVY0iQHrDLlfS
xP1TVlUoiudIUo9BCjLmQQuyAkgxgDln9idzgXZKWXZMrW6HcK3Q4Ji2ymowCUf0
vHRGj2mjBHo+QSYhOz/NduJPG717THk9C+9xG6eOyFa712VIwEJZc5dPKA50J/s0
OaeNE8fk/mCyJ2y3yyE2V61wqhgRvA==
=E6l9
-----END PGP SIGNATURE-----
--=-=-=--




Information forwarded to bug-guix@HIDDEN:
bug#33848; Package guix. Full text available.

Message received at 33848 <at> debbugs.gnu.org:


Received: (at 33848) by debbugs.gnu.org; 24 Dec 2018 15:06:13 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Dec 24 10:06:13 2018
Received: from localhost ([127.0.0.1]:35540 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gbRo5-00063E-01
	for submit <at> debbugs.gnu.org; Mon, 24 Dec 2018 10:06:13 -0500
Received: from hera.aquilenet.fr ([185.233.100.1]:58974)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1gbRo3-000636-Iy
 for 33848 <at> debbugs.gnu.org; Mon, 24 Dec 2018 10:06:11 -0500
Received: from localhost (localhost [127.0.0.1])
 by hera.aquilenet.fr (Postfix) with ESMTP id 0079C1494;
 Mon, 24 Dec 2018 16:06:11 +0100 (CET)
X-Virus-Scanned: Debian amavisd-new at aquilenet.fr
Received: from hera.aquilenet.fr ([127.0.0.1])
 by localhost (hera.aquilenet.fr [127.0.0.1]) (amavisd-new, port 10024)
 with ESMTP id 57lQzFcVz4IN; Mon, 24 Dec 2018 16:06:10 +0100 (CET)
Received: from ribbon (unknown [IPv6:2a01:e0a:1d:7270:af76:b9b:ca24:c465])
 by hera.aquilenet.fr (Postfix) with ESMTPSA id C72FA138F;
 Mon, 24 Dec 2018 16:06:09 +0100 (CET)
From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>
To: Pierre Neidhardt <mail@HIDDEN>
Subject: Re: bug#33848: Store references in SBCL-compiled code are "invisible"
References: <87r2e8jpfx.fsf@HIDDEN> <877eg0i43j.fsf@HIDDEN>
 <87d0psi1xo.fsf@HIDDEN> <874lb3kin6.fsf@HIDDEN>
Date: Mon, 24 Dec 2018 16:06:09 +0100
In-Reply-To: <874lb3kin6.fsf@HIDDEN> (Pierre Neidhardt's message of
 "Sun, 23 Dec 2018 23:01:01 +0100")
Message-ID: <87sgynezha.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: 1.0 (+)
X-Debbugs-Envelope-To: 33848
Cc: Mark H Weaver <mhw@HIDDEN>, 33848 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.0 (/)

Hi Pierre,

Pierre Neidhardt <mail@HIDDEN> skribis:

>> I don=E2=80=99t think we=E2=80=99ve encountered the problem before.
>
> Actually it does ring a bell for me.  Didn't we have a similar issue with=
 Fish,
> or some dependency?

We did have a problem with Fish but I can no longer find it.  Do you
remember what it was?  Something with C++, no?

>> For now I lean towards looking for a way to address the issue
>> specifically for SBCL.
>
> Don't forget that we currently have 5 Lisp compilers.
> Besides, it's not clear that this can be fixed on the compiler's side, it=
 could
> very well be that patches will be required  on a per-project basis.

I know little about CL but maybe we can find a solution that works for
all five compilers.  At least that would be the first approach I would
suggest following.

Thanks,
Ludo=E2=80=99.




Information forwarded to bug-guix@HIDDEN:
bug#33848; Package guix. Full text available.

Message received at 33848 <at> debbugs.gnu.org:


Received: (at 33848) by debbugs.gnu.org; 24 Dec 2018 14:57:55 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Dec 24 09:57:54 2018
Received: from localhost ([127.0.0.1]:35532 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gbRg2-0003pB-Ny
	for submit <at> debbugs.gnu.org; Mon, 24 Dec 2018 09:57:54 -0500
Received: from hera.aquilenet.fr ([185.233.100.1]:58924)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1gbRg1-0003p2-55
 for 33848 <at> debbugs.gnu.org; Mon, 24 Dec 2018 09:57:53 -0500
Received: from localhost (localhost [127.0.0.1])
 by hera.aquilenet.fr (Postfix) with ESMTP id 9035E1468;
 Mon, 24 Dec 2018 15:57:52 +0100 (CET)
X-Virus-Scanned: Debian amavisd-new at aquilenet.fr
Received: from hera.aquilenet.fr ([127.0.0.1])
 by localhost (hera.aquilenet.fr [127.0.0.1]) (amavisd-new, port 10024)
 with ESMTP id OWrLy6bCjPYf; Mon, 24 Dec 2018 15:57:52 +0100 (CET)
Received: from ribbon (unknown [IPv6:2a01:e0a:1d:7270:af76:b9b:ca24:c465])
 by hera.aquilenet.fr (Postfix) with ESMTPSA id B71BA1102;
 Mon, 24 Dec 2018 15:57:51 +0100 (CET)
From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>
To: Pierre Neidhardt <mail@HIDDEN>
Subject: Re: bug#33848: Store references in SBCL-compiled code are "invisible"
References: <87r2e8jpfx.fsf@HIDDEN> <87a7kwjnai.fsf@HIDDEN>
Date: Mon, 24 Dec 2018 15:57:50 +0100
In-Reply-To: <87a7kwjnai.fsf@HIDDEN> (Pierre Neidhardt's message of
 "Sun, 23 Dec 2018 16:05:57 +0100")
Message-ID: <875zvjgefl.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: 1.0 (+)
X-Debbugs-Envelope-To: 33848
Cc: ajpatter@HIDDEN, 33848 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.0 (/)

Hi!

Pierre Neidhardt <mail@HIDDEN> skribis:

> Thanks for looking into this, Ludo.
>
> At first glance, I'd say that this is not a compilation option but the way
> strings are encoded by default.  It seems that multibyte encoding is used=
 all
> over the place by a few compilers including SBCL (and CCL I think).
>
> One way I know around this (I'm by no mean a Common Lisp expert) is the
> flexi-streams package for re-encoding.

OK, we need to investigate.

> More generally, shouldn't we make the reference scanner a bit smarter?  In
> particular, how does it handle non-ASCII references?  Maybe it would not =
be
> unreasonable to handle UTF-8 and UCS-4 for instance?

Store file names are always ASCII so problems arise when they are stored
as UTF-16 or UTF-32/UCS-4.

Ludo=E2=80=99.




Information forwarded to bug-guix@HIDDEN:
bug#33848; Package guix. Full text available.
Severity set to 'important' from 'normal' Request was from Ludovic Courtès <ludo@HIDDEN> to control <at> debbugs.gnu.org. Full text available.

Message received at 33848 <at> debbugs.gnu.org:


Received: (at 33848) by debbugs.gnu.org; 23 Dec 2018 22:01:09 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Dec 23 17:01:09 2018
Received: from localhost ([127.0.0.1]:34100 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gbBo5-0005Du-6q
	for submit <at> debbugs.gnu.org; Sun, 23 Dec 2018 17:01:09 -0500
Received: from relay8-d.mail.gandi.net ([217.70.183.201]:46911)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mail@HIDDEN>) id 1gbBo4-0005Dn-7D
 for 33848 <at> debbugs.gnu.org; Sun, 23 Dec 2018 17:01:08 -0500
X-Originating-IP: 78.199.129.170
Received: from mimimi (moi44-1-78-199-129-170.fbx.proxad.net [78.199.129.170])
 (Authenticated sender: mail@HIDDEN)
 by relay8-d.mail.gandi.net (Postfix) with ESMTPSA id 48A421BF208;
 Sun, 23 Dec 2018 22:01:03 +0000 (UTC)
References: <87r2e8jpfx.fsf@HIDDEN> <877eg0i43j.fsf@HIDDEN>
 <87d0psi1xo.fsf@HIDDEN>
User-agent: mu4e 1.0; emacs 26.1
From: Pierre Neidhardt <mail@HIDDEN>
To: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Subject: Re: bug#33848: Store references in SBCL-compiled code are "invisible"
In-reply-to: <87d0psi1xo.fsf@HIDDEN>
Date: Sun, 23 Dec 2018 23:01:01 +0100
Message-ID: <874lb3kin6.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="=-=-=";
 micalg=pgp-sha256; protocol="application/pgp-signature"
X-Spam-Score: -0.7 (/)
X-Debbugs-Envelope-To: 33848
Cc: Mark H Weaver <mhw@HIDDEN>, 33848 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

--=-=-=
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


> I don=E2=80=99t think we=E2=80=99ve encountered the problem before.

Actually it does ring a bell for me.  Didn't we have a similar issue with F=
ish,
or some dependency?

> For now I lean towards looking for a way to address the issue
> specifically for SBCL.

Don't forget that we currently have 5 Lisp compilers.
Besides, it's not clear that this can be fixed on the compiler's side, it c=
ould
very well be that patches will be required  on a per-project basis.

=2D-=20
Pierre Neidhardt
https://ambrevar.xyz/

--=-=-=
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEUPM+LlsMPZAEJKvom9z0l6S7zH8FAlwgBZ0ACgkQm9z0l6S7
zH8XGAgAqUkpkfyLkBTmGkB0E4yhMQ2Mo5elI/PvQQN4NGPHl/VDJysQMZJPnYDS
N4FAooCf3v5oenfr+VgZKr7NDDkDgVIdZbUCjzEw0La7FFl8DpB4+riJ0WqtghiB
jCr4KRNfuSn1tgIenvMFsswH3otTaAllMIlfqMhxYJDtGYTzcjP059xgDQ0rPlKF
PoAOv839rILx0AfdXAp7knIV+q4iN623ZEiGFIJQ3K2JuaPoBkBBVUHkk/lJSOvW
naGrIE56gqOnjMKJuTx9FuhhYPtN8ieNYj/VLV3y5V9v5JyvO3zwV49ahemBEK84
n9PxAlU4A0D66Gy6ZmCBN38Ewgfhkw==
=s6io
-----END PGP SIGNATURE-----
--=-=-=--




Information forwarded to bug-guix@HIDDEN:
bug#33848; Package guix. Full text available.

Message received at 33848 <at> debbugs.gnu.org:


Received: (at 33848) by debbugs.gnu.org; 23 Dec 2018 17:32:40 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Dec 23 12:32:40 2018
Received: from localhost ([127.0.0.1]:33981 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gb7cF-0006VM-MU
	for submit <at> debbugs.gnu.org; Sun, 23 Dec 2018 12:32:40 -0500
Received: from hera.aquilenet.fr ([185.233.100.1]:52410)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1gb7cE-0006VE-44
 for 33848 <at> debbugs.gnu.org; Sun, 23 Dec 2018 12:32:38 -0500
Received: from localhost (localhost [127.0.0.1])
 by hera.aquilenet.fr (Postfix) with ESMTP id 7CD6312BB;
 Sun, 23 Dec 2018 18:32:37 +0100 (CET)
X-Virus-Scanned: Debian amavisd-new at aquilenet.fr
Received: from hera.aquilenet.fr ([127.0.0.1])
 by localhost (hera.aquilenet.fr [127.0.0.1]) (amavisd-new, port 10024)
 with ESMTP id qtk9Zqe9FfNj; Sun, 23 Dec 2018 18:32:36 +0100 (CET)
Received: from ribbon (unknown [IPv6:2a01:e0a:1d:7270:af76:b9b:ca24:c465])
 by hera.aquilenet.fr (Postfix) with ESMTPSA id 6997F1282;
 Sun, 23 Dec 2018 18:32:36 +0100 (CET)
From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>
To: Mark H Weaver <mhw@HIDDEN>
Subject: Re: bug#33848: Store references in SBCL-compiled code are "invisible"
References: <87r2e8jpfx.fsf@HIDDEN> <877eg0i43j.fsf@HIDDEN>
X-URL: http://www.fdn.fr/~lcourtes/
X-Revolutionary-Date: 3 =?utf-8?Q?Niv=C3=B4se?= an 227 de la =?utf-8?Q?R?=
 =?utf-8?Q?=C3=A9volution?=
X-PGP-Key-ID: 0x090B11993D9AEBB5
X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc
X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4  0CFB 090B 1199 3D9A EBB5
X-OS: x86_64-pc-linux-gnu
Date: Sun, 23 Dec 2018 18:32:35 +0100
In-Reply-To: <877eg0i43j.fsf@HIDDEN> (Mark H. Weaver's message of "Sun, 23
 Dec 2018 11:45:25 -0500")
Message-ID: <87d0psi1xo.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: 1.0 (+)
X-Debbugs-Envelope-To: 33848
Cc: Pierre Neidhardt <mail@HIDDEN>, 33848 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.0 (/)

Hi Mark,

Mark H Weaver <mhw@HIDDEN> skribis:

> Ludovic Court=C3=A8s <ludo@HIDDEN> writes:

[...]

>> Apparently this string literal is stored as UTF-32 (UCS-4) or similar,
>> which prevents the reference scanner and the grafting code from finding
>> it, and problems ensue.  :-)
>
> IMO, we should consider modifying Guix to search for store references
> encoded in UTF-32 and/or UTF-16.  I wouldn't be surprised if some other
> programs use those encodings.  I'd be willing to work on it.

I don=E2=80=99t think we=E2=80=99ve encountered the problem before.  This w=
ould require
fixing both the scanner and the grafting code (though eventually that
might be a single code base when the Scheme-implemented daemon is
merged) in non-trivial ways.

One issue is that users of an old daemon would get a different behavior
than users of a new daemon.  It would be the first time we introduce
such a significant change in the daemon since Guix was started.

For now I lean towards looking for a way to address the issue
specifically for SBCL.  I=E2=80=99d be tempted to generalize if and only if=
 we
find other occurrences of the problem that would make the benefits
outweigh the development and maintenance costs.

WDYT?

I remember discussing in the past some sort of =E2=80=9Cpluggable=E2=80=9D =
reference
scanning mechanism that could also work for compressed archives, etc.
That also looks like the right thing, but it has a development and
maintenance cost that=E2=80=99s pretty high whereas we might be able to add=
ress
the same problems in much simpler ways.

Thanks,
Ludo=E2=80=99.




Information forwarded to bug-guix@HIDDEN:
bug#33848; Package guix. Full text available.

Message received at 33848 <at> debbugs.gnu.org:


Received: (at 33848) by debbugs.gnu.org; 23 Dec 2018 16:46:41 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Dec 23 11:46:41 2018
Received: from localhost ([127.0.0.1]:33947 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gb6tk-0005MJ-Lx
	for submit <at> debbugs.gnu.org; Sun, 23 Dec 2018 11:46:40 -0500
Received: from world.peace.net ([64.112.178.59]:32792)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mhw@HIDDEN>) id 1gb6tj-0005M6-1x
 for 33848 <at> debbugs.gnu.org; Sun, 23 Dec 2018 11:46:39 -0500
Received: from mhw by world.peace.net with esmtpsa
 (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89)
 (envelope-from <mhw@HIDDEN>)
 id 1gb6tb-0006Wd-M3; Sun, 23 Dec 2018 11:46:31 -0500
From: Mark H Weaver <mhw@HIDDEN>
To: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Subject: Re: bug#33848: Store references in SBCL-compiled code are "invisible"
References: <87r2e8jpfx.fsf@HIDDEN>
Date: Sun, 23 Dec 2018 11:45:25 -0500
In-Reply-To: <87r2e8jpfx.fsf@HIDDEN> ("Ludovic
 \=\?utf-8\?Q\?Court\=C3\=A8s\=22'\?\=
 \=\?utf-8\?Q\?s\?\= message of "Sun, 23 Dec 2018 15:19:30 +0100")
Message-ID: <877eg0i43j.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 33848
Cc: Pierre Neidhardt <mail@HIDDEN>, 33848 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Hi Ludovic,

Ludovic Court=C3=A8s <ludo@HIDDEN> writes:

> As discussed with Pierre at the R-B Summit, =E2=80=98sbcl-next=E2=80=99 l=
acks a
> reference to =E2=80=98next-gtk-webkit=E2=80=99 even though is invokes it:
>
> $ guix gc --references $(type -P next) | grep next-
> /gnu/store/9d66xb8wvggsp0x9pxj61mzqy007978f-sbcl-next-1.1.0
> /gnu/store/pqy064fw3vkfld6lw95vi0zavj19zvrc-sbcl-next-1.1.0-lib
> $ ./pre-inst-env guix run next
>
> WARNING: Setting locale failed.
>   Check the following variables for correct values:
>   LANG=3Den_US.utf8
> Unhandled SIMPLE-ERROR in thread #<SB-THREAD:THREAD "main thread" RUNNING
>                                     {10005885B3}>:
>   Couldn't execute "/gnu/store/7p6pbcmdgr53dff6033gcfl2jq0d762h-next-gtk-=
webkit-1.1.0/bin/next-gtk-webkit": No such file or directory
>
>
> (Here =E2=80=98guix run=E2=80=99 runs =E2=80=98next=E2=80=99 in a contain=
er with exactly the closure of
> =E2=80=98next=E2=80=99, nothing more, and the =E2=80=98next=E2=80=99 bina=
ry is grafted.)
>
> So the problem looks a lot like that this GCC issue we fixed a while
> back: <https://bugs.gnu.org/24703>.
>
> Looking at the =E2=80=98sbcl-next=E2=80=99 package, the reference to =E2=
=80=98next-gtk-webkit=E2=80=99
> is inserted in gtk-webkit.lisp:
>
> (defvar *gtk-webkit-command* "next-gtk-webkit"
>   "Path to the GTK-Webkit platform port executable.")
>
>
> Through hexl-mode on the =E2=80=98next=E2=80=99 binary, we can find that =
reference:
>
> 01d0bac0: 2f00 0000 6700 0000 6e00 0000 7500 0000  /...g...n...u...
> 01d0bad0: 2f00 0000 7300 0000 7400 0000 6f00 0000  /...s...t...o...
> 01d0bae0: 7200 0000 6500 0000 2f00 0000 3700 0000  r...e.../...7...
> 01d0baf0: 7000 0000 3600 0000 7000 0000 6200 0000  p...6...p...b...
> 01d0bb00: 6300 0000 6d00 0000 6400 0000 6700 0000  c...m...d...g...
> 01d0bb10: 7200 0000 3500 0000 3300 0000 6400 0000  r...5...3...d...
> 01d0bb20: 6600 0000 6600 0000 3600 0000 3000 0000  f...f...6...0...
> 01d0bb30: 3300 0000 3300 0000 6700 0000 6300 0000  3...3...g...c...
> 01d0bb40: 6600 0000 6c00 0000 3200 0000 6a00 0000  f...l...2...j...
> 01d0bb50: 7100 0000 3000 0000 6400 0000 3700 0000  q...0...d...7...
> 01d0bb60: 3600 0000 3200 0000 6800 0000 2d00 0000  6...2...h...-...
> 01d0bb70: 6e00 0000 6500 0000 7800 0000 7400 0000  n...e...x...t...
> 01d0bb80: 2d00 0000 6700 0000 7400 0000 6b00 0000  -...g...t...k...
> 01d0bb90: 2d00 0000 7700 0000 6500 0000 6200 0000  -...w...e...b...
> 01d0bba0: 6b00 0000 6900 0000 7400 0000 2d00 0000  k...i...t...-...
> 01d0bbb0: 3100 0000 2e00 0000 3100 0000 2e00 0000  1.......1.......
> 01d0bbc0: 3000 0000 2f00 0000 6200 0000 6900 0000  0.../...b...i...
> 01d0bbd0: 6e00 0000 2f00 0000 6e00 0000 6500 0000  n.../...n...e...
> 01d0bbe0: 7800 0000 7400 0000 2d00 0000 6700 0000  x...t...-...g...
> 01d0bbf0: 7400 0000 6b00 0000 2d00 0000 7700 0000  t...k...-...w...
> 01d0bc00: 6500 0000 6200 0000 6b00 0000 6900 0000  e...b...k...i...
> 01d0bc10: 7400 0000 0000 0000 0000 0000 0000 0000  t...............
> 01d0bc20: e100 0100 0000 0000 2800 0000 0000 0000  ........(.......
> 01d0bc30: 2a47 544b 2d57 4542 4b49 542d 434f 4d4d  *GTK-WEBKIT-COMM
> 01d0bc40: 414e 442a 0000 0000 0000 0000 0000 0000  AND*............
>
> Apparently this string literal is stored as UTF-32 (UCS-4) or similar,
> which prevents the reference scanner and the grafting code from finding
> it, and problems ensue.  :-)

IMO, we should consider modifying Guix to search for store references
encoded in UTF-32 and/or UTF-16.  I wouldn't be surprised if some other
programs use those encodings.  I'd be willing to work on it.

What do you think?

      Mark




Information forwarded to bug-guix@HIDDEN:
bug#33848; Package guix. Full text available.

Message received at submit <at> debbugs.gnu.org:


Received: (at submit) by debbugs.gnu.org; 23 Dec 2018 15:06:14 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Dec 23 10:06:14 2018
Received: from localhost ([127.0.0.1]:33890 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gb5KY-0002xv-C4
	for submit <at> debbugs.gnu.org; Sun, 23 Dec 2018 10:06:14 -0500
Received: from eggs.gnu.org ([208.118.235.92]:40163)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mail@HIDDEN>) id 1gb5KW-0002xj-RA
 for submit <at> debbugs.gnu.org; Sun, 23 Dec 2018 10:06:13 -0500
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <mail@HIDDEN>) id 1gb5KQ-0001eY-QF
 for submit <at> debbugs.gnu.org; Sun, 23 Dec 2018 10:06:07 -0500
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00 autolearn=disabled
 version=3.3.2
Received: from lists.gnu.org ([2001:4830:134:3::11]:52557)
 by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32)
 (Exim 4.71) (envelope-from <mail@HIDDEN>) id 1gb5KQ-0001eK-NL
 for submit <at> debbugs.gnu.org; Sun, 23 Dec 2018 10:06:06 -0500
Received: from eggs.gnu.org ([2001:4830:134:3::10]:48343)
 by lists.gnu.org with esmtp (Exim 4.71)
 (envelope-from <mail@HIDDEN>) id 1gb5KP-0003W7-Qs
 for bug-guix@HIDDEN; Sun, 23 Dec 2018 10:06:06 -0500
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <mail@HIDDEN>) id 1gb5KM-0001YX-I2
 for bug-guix@HIDDEN; Sun, 23 Dec 2018 10:06:05 -0500
Received: from relay4-d.mail.gandi.net ([217.70.183.196]:50737)
 by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)
 (Exim 4.71) (envelope-from <mail@HIDDEN>)
 id 1gb5KM-0001Wl-Bp; Sun, 23 Dec 2018 10:06:02 -0500
X-Originating-IP: 78.199.129.170
Received: from mimimi (moi44-1-78-199-129-170.fbx.proxad.net [78.199.129.170])
 (Authenticated sender: mail@HIDDEN)
 by relay4-d.mail.gandi.net (Postfix) with ESMTPSA id 7BDFBE0008;
 Sun, 23 Dec 2018 15:05:58 +0000 (UTC)
References: <87r2e8jpfx.fsf@HIDDEN>
User-agent: mu4e 1.0; emacs 26.1
From: Pierre Neidhardt <mail@HIDDEN>
To: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Subject: Re: Store references in SBCL-compiled code are "invisible"
In-reply-to: <87r2e8jpfx.fsf@HIDDEN>
Date: Sun, 23 Dec 2018 16:05:57 +0100
Message-ID: <87a7kwjnai.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="=-=-=";
 micalg=pgp-sha256; protocol="application/pgp-signature"
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
 [fuzzy]
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x
X-Received-From: 2001:4830:134:3::11
X-Spam-Score: -5.0 (-----)
X-Debbugs-Envelope-To: submit
Cc: Andy Patterson <ajpatter@HIDDEN>, Bug Guix <bug-guix@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -6.0 (------)

--=-=-=
Content-Type: text/plain
Content-Transfer-Encoding: quoted-printable

Thanks for looking into this, Ludo.

At first glance, I'd say that this is not a compilation option but the way
strings are encoded by default.  It seems that multibyte encoding is used a=
ll
over the place by a few compilers including SBCL (and CCL I think).

One way I know around this (I'm by no mean a Common Lisp expert) is the
flexi-streams package for re-encoding.

More generally, shouldn't we make the reference scanner a bit smarter?  In
particular, how does it handle non-ASCII references?  Maybe it would not be
unreasonable to handle UTF-8 and UCS-4 for instance?

=2D-=20
Pierre Neidhardt
https://ambrevar.xyz/

--=-=-=
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEUPM+LlsMPZAEJKvom9z0l6S7zH8FAlwfpFUACgkQm9z0l6S7
zH/rdggAnQg4EGjzFVDFFNwofXuTBydyu+uLV6FHjagMA1ijxaEPtL27NhmdWUdM
oFgRKIKabQpixyDSWJhAPGPIv2JHdrqiBwNRUfaDWSKhoh8/qA654QF8NiSFCHaf
JpXMcdYVGvQ92Fo9OUFls8CWeWSpaEgQcTeIeeTNLwDCid8ob5gFW8doaqxraGw6
dkLeqIOuenB7jI/7cBs4yD4e+r8V/IAY/mVvuTZ+gRFGu+StbMo01KRX2X1xOVaL
L4WiaGhJHyDYqw3otCPfZduOvsOyfhSrov3HpPT6vocQ11Wb8tW7t+JipO+w1DLi
FMG7JYdAN3ibsDXXncsKcSxgeszj1g==
=oKMR
-----END PGP SIGNATURE-----
--=-=-=--




Information forwarded to bug-guix@HIDDEN:
bug#33848; Package guix. Full text available.

Message received at submit <at> debbugs.gnu.org:


Received: (at submit) by debbugs.gnu.org; 23 Dec 2018 14:20:55 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Dec 23 09:20:55 2018
Received: from localhost ([127.0.0.1]:60618 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gb4cg-0001Wj-OS
	for submit <at> debbugs.gnu.org; Sun, 23 Dec 2018 09:20:55 -0500
Received: from eggs.gnu.org ([208.118.235.92]:35153)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1gb4cd-0001WV-QJ
 for submit <at> debbugs.gnu.org; Sun, 23 Dec 2018 09:20:52 -0500
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <ludo@HIDDEN>) id 1gb4c6-0006x6-G2
 for submit <at> debbugs.gnu.org; Sun, 23 Dec 2018 09:20:46 -0500
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50 autolearn=disabled
 version=3.3.2
Received: from lists.gnu.org ([2001:4830:134:3::11]:42664)
 by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32)
 (Exim 4.71) (envelope-from <ludo@HIDDEN>) id 1gb4c5-0006wI-Lq
 for submit <at> debbugs.gnu.org; Sun, 23 Dec 2018 09:20:17 -0500
Received: from eggs.gnu.org ([2001:4830:134:3::10]:53071)
 by lists.gnu.org with esmtp (Exim 4.71)
 (envelope-from <ludo@HIDDEN>) id 1gb4bi-00018j-WD
 for bug-guix@HIDDEN; Sun, 23 Dec 2018 09:20:17 -0500
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <ludo@HIDDEN>) id 1gb4bO-0006IA-KZ
 for bug-guix@HIDDEN; Sun, 23 Dec 2018 09:19:54 -0500
Received: from hera.aquilenet.fr ([185.233.100.1]:41666)
 by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)
 (Exim 4.71) (envelope-from <ludo@HIDDEN>) id 1gb4bO-0006Ge-8W
 for bug-guix@HIDDEN; Sun, 23 Dec 2018 09:19:34 -0500
Received: from localhost (localhost [127.0.0.1])
 by hera.aquilenet.fr (Postfix) with ESMTP id B1233126E;
 Sun, 23 Dec 2018 15:19:32 +0100 (CET)
X-Virus-Scanned: Debian amavisd-new at aquilenet.fr
Received: from hera.aquilenet.fr ([127.0.0.1])
 by localhost (hera.aquilenet.fr [127.0.0.1]) (amavisd-new, port 10024)
 with ESMTP id vknIypg0FXuW; Sun, 23 Dec 2018 15:19:31 +0100 (CET)
Received: from ribbon (unknown [IPv6:2a01:e0a:1d:7270:af76:b9b:ca24:c465])
 by hera.aquilenet.fr (Postfix) with ESMTPSA id 779FB1D8;
 Sun, 23 Dec 2018 15:19:31 +0100 (CET)
From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>
To: Bug Guix <bug-guix@HIDDEN>
Subject: Store references in SBCL-compiled code are "invisible"
X-URL: http://www.fdn.fr/~lcourtes/
X-Revolutionary-Date: 3 =?utf-8?Q?Niv=C3=B4se?= an 227 de la =?utf-8?Q?R?=
 =?utf-8?Q?=C3=A9volution?=
X-PGP-Key-ID: 0x090B11993D9AEBB5
X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc
X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4  0CFB 090B 1199 3D9A EBB5
X-OS: x86_64-pc-linux-gnu
Date: Sun, 23 Dec 2018 15:19:30 +0100
Message-ID: <87r2e8jpfx.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
 [fuzzy]
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x
X-Received-From: 2001:4830:134:3::11
X-Spam-Score: -5.0 (-----)
X-Debbugs-Envelope-To: submit
Cc: Pierre Neidhardt <mail@HIDDEN>,
 Andy Patterson <ajpatter@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -6.0 (------)

Hello,

As discussed with Pierre at the R-B Summit, =E2=80=98sbcl-next=E2=80=99 lac=
ks a
reference to =E2=80=98next-gtk-webkit=E2=80=99 even though is invokes it:

--8<---------------cut here---------------start------------->8---
$ guix gc --references $(type -P next) | grep next-
/gnu/store/9d66xb8wvggsp0x9pxj61mzqy007978f-sbcl-next-1.1.0
/gnu/store/pqy064fw3vkfld6lw95vi0zavj19zvrc-sbcl-next-1.1.0-lib
$ ./pre-inst-env guix run next

WARNING: Setting locale failed.
  Check the following variables for correct values:
  LANG=3Den_US.utf8
Unhandled SIMPLE-ERROR in thread #<SB-THREAD:THREAD "main thread" RUNNING
                                    {10005885B3}>:
  Couldn't execute "/gnu/store/7p6pbcmdgr53dff6033gcfl2jq0d762h-next-gtk-we=
bkit-1.1.0/bin/next-gtk-webkit": No such file or directory
--8<---------------cut here---------------end--------------->8---

(Here =E2=80=98guix run=E2=80=99 runs =E2=80=98next=E2=80=99 in a container=
 with exactly the closure of
=E2=80=98next=E2=80=99, nothing more, and the =E2=80=98next=E2=80=99 binary=
 is grafted.)

So the problem looks a lot like that this GCC issue we fixed a while
back: <https://bugs.gnu.org/24703>.

Looking at the =E2=80=98sbcl-next=E2=80=99 package, the reference to =E2=80=
=98next-gtk-webkit=E2=80=99
is inserted in gtk-webkit.lisp:

--8<---------------cut here---------------start------------->8---
(defvar *gtk-webkit-command* "next-gtk-webkit"
  "Path to the GTK-Webkit platform port executable.")
--8<---------------cut here---------------end--------------->8---

Through hexl-mode on the =E2=80=98next=E2=80=99 binary, we can find that re=
ference:

--8<---------------cut here---------------start------------->8---
01d0bac0: 2f00 0000 6700 0000 6e00 0000 7500 0000  /...g...n...u...
01d0bad0: 2f00 0000 7300 0000 7400 0000 6f00 0000  /...s...t...o...
01d0bae0: 7200 0000 6500 0000 2f00 0000 3700 0000  r...e.../...7...
01d0baf0: 7000 0000 3600 0000 7000 0000 6200 0000  p...6...p...b...
01d0bb00: 6300 0000 6d00 0000 6400 0000 6700 0000  c...m...d...g...
01d0bb10: 7200 0000 3500 0000 3300 0000 6400 0000  r...5...3...d...
01d0bb20: 6600 0000 6600 0000 3600 0000 3000 0000  f...f...6...0...
01d0bb30: 3300 0000 3300 0000 6700 0000 6300 0000  3...3...g...c...
01d0bb40: 6600 0000 6c00 0000 3200 0000 6a00 0000  f...l...2...j...
01d0bb50: 7100 0000 3000 0000 6400 0000 3700 0000  q...0...d...7...
01d0bb60: 3600 0000 3200 0000 6800 0000 2d00 0000  6...2...h...-...
01d0bb70: 6e00 0000 6500 0000 7800 0000 7400 0000  n...e...x...t...
01d0bb80: 2d00 0000 6700 0000 7400 0000 6b00 0000  -...g...t...k...
01d0bb90: 2d00 0000 7700 0000 6500 0000 6200 0000  -...w...e...b...
01d0bba0: 6b00 0000 6900 0000 7400 0000 2d00 0000  k...i...t...-...
01d0bbb0: 3100 0000 2e00 0000 3100 0000 2e00 0000  1.......1.......
01d0bbc0: 3000 0000 2f00 0000 6200 0000 6900 0000  0.../...b...i...
01d0bbd0: 6e00 0000 2f00 0000 6e00 0000 6500 0000  n.../...n...e...
01d0bbe0: 7800 0000 7400 0000 2d00 0000 6700 0000  x...t...-...g...
01d0bbf0: 7400 0000 6b00 0000 2d00 0000 7700 0000  t...k...-...w...
01d0bc00: 6500 0000 6200 0000 6b00 0000 6900 0000  e...b...k...i...
01d0bc10: 7400 0000 0000 0000 0000 0000 0000 0000  t...............
01d0bc20: e100 0100 0000 0000 2800 0000 0000 0000  ........(.......
01d0bc30: 2a47 544b 2d57 4542 4b49 542d 434f 4d4d  *GTK-WEBKIT-COMM
01d0bc40: 414e 442a 0000 0000 0000 0000 0000 0000  AND*............
--8<---------------cut here---------------end--------------->8---

Apparently this string literal is stored as UTF-32 (UCS-4) or similar,
which prevents the reference scanner and the grafting code from finding
it, and problems ensue.  :-)

Pierre, Andy: is there any way to tell SBCL to store this literal as
ASCII/UTF-8?  That would be an easy fix, though we should discuss the
pros and cons and whether to enable that globally.

Thanks in advance!

Ludo=E2=80=99.




Acknowledgement sent to Ludovic Courtès <ludo@HIDDEN>:
New bug report received and forwarded. Copy sent to bug-guix@HIDDEN. Full text available.
Report forwarded to bug-guix@HIDDEN:
bug#33848; Package guix. Full text available.
Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.
Last modified: Mon, 25 Nov 2019 12:00:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.