GNU bug report logs - #32942
'guix offload' fails to restore non-ASCII file names when running in C locale

Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.

Package: guix; Severity: important; Reported by: Julien Lepiller <julien@HIDDEN>; dated Fri, 5 Oct 2018 08:19:02 UTC; Maintainer for guix is bug-guix@HIDDEN.
Changed bug title to ''guix offload' fails to restore non-ASCII file names when running in C locale' from 'nss-certs not deterministic' Request was from ludo@HIDDEN (Ludovic Courtès) to control <at> debbugs.gnu.org. Full text available.

Message received at 32942 <at> debbugs.gnu.org:


Received: (at 32942) by debbugs.gnu.org; 5 Dec 2018 14:01:43 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Wed Dec 05 09:01:43 2018
Received: from localhost ([127.0.0.1]:33251 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gUXkC-0001RS-VQ
	for submit <at> debbugs.gnu.org; Wed, 05 Dec 2018 09:01:43 -0500
Received: from eggs.gnu.org ([208.118.235.92]:48725)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1gUXkA-0001R8-5r
 for 32942 <at> debbugs.gnu.org; Wed, 05 Dec 2018 09:01:38 -0500
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <ludo@HIDDEN>) id 1gUXk0-0006y0-0E
 for 32942 <at> debbugs.gnu.org; Wed, 05 Dec 2018 09:01:33 -0500
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00 autolearn=disabled
 version=3.3.2
Received: from fencepost.gnu.org ([2001:4830:134:3::e]:54403)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <ludo@HIDDEN>)
 id 1gUXjz-0006xg-RZ; Wed, 05 Dec 2018 09:01:27 -0500
Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=40576 helo=ribbon)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <ludo@HIDDEN>)
 id 1gUXjz-0002Dz-Iv; Wed, 05 Dec 2018 09:01:27 -0500
From: ludo@HIDDEN (Ludovic =?utf-8?Q?Court=C3=A8s?=)
To: Julien Lepiller <julien@HIDDEN>
Subject: Re: bug#32942: nss-certs not deterministic
References: <3974e5005881951012bb5e55a5bfabe2@HIDDEN>
Date: Wed, 05 Dec 2018 15:01:25 +0100
In-Reply-To: <3974e5005881951012bb5e55a5bfabe2@HIDDEN> (Julien Lepiller's
 message of "Fri, 05 Oct 2018 10:17:45 +0200")
Message-ID: <87woooxebu.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux)
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="=-=-="
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2001:4830:134:3::e
X-Spam-Score: -5.0 (-----)
X-Debbugs-Envelope-To: 32942
Cc: 32942 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -6.0 (------)

--=-=-=
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

Hello,

Julien Lepiller <julien@HIDDEN> skribis:

> While updating a profile, I found that nss-certs was not
> deterministic. From ludo:
>
> $ wget -O - -q=20
> https://mirror.hydra.gnu.org/mbs5mavs3gi4y7xkywcwwjj9g3p1yjmv.narinfo| gr=
ep Hash
> NarHash: sha256:101v69xp1qzw9v6pgmbhw7gfdaic8vvs4v5l567lx7f2mjp25rla
> $ wget -O - -q=20
> https://berlin.guixsd.org/mbs5mavs3gi4y7xkywcwwjj9g3p1yjmv.narinfo |
> grep Hash
> NarHash: sha256:08ziz714diyfq2klxy1nc0nhr5wa2vd356n9vizlq913a7an9a9s

As shown above, berlin and hydra disagree on nss-certs.

The difference is an encoding bug:

--8<---------------cut here---------------start------------->8---
$ wget -O - https://berlin.guixsd.org/nar/gzip/xbj4fhad0lnz0ziflwi90gyqbls8=
ains-nss-certs-3.39 |gunzip -c |guix archive -x /tmp/nss-certs.berlin
$ wget -O - https://mirror.hydra.gnu.org/nar/gzip/xbj4fhad0lnz0ziflwi90gyqb=
ls8ains-nss-certs-3.39 |gunzip -c |guix archive -x /tmp/nss-certs.hydra
$ diff -ru /tmp/nss-certs.{hydra,berlin}
Only in /tmp/nss-certs.hydra/etc/ssl/certs: AC_Ra=C3=ADz_Certic=C3=A1mara_S=
.A.:2.15.7.126.82.147.123.224.21.227.87.240.105.140.203.236.12.pem
Only in /tmp/nss-certs.berlin/etc/ssl/certs: AC_Ra?z_Certic?mara_S.A.:2.15.=
7.126.82.147.123.224.21.227.87.240.105.140.203.236.12.pem
Only in /tmp/nss-certs.hydra/etc/ssl/certs: NetLock_Arany_=3DClass_Gold=3D_=
F=C5=91tan=C3=BAs=C3=ADtv=C3=A1ny:2.6.73.65.44.228.0.16.pem
Only in /tmp/nss-certs.berlin/etc/ssl/certs: NetLock_Arany_=3DClass_Gold=3D=
_F?tan?s?tv?ny:2.6.73.65.44.228.0.16.pem
--8<---------------cut here---------------end--------------->8---

The problem was already reported as <https://bugs.gnu.org/26948> and
since commit 412701b0e5e073e6767eed162c14698db99df69c (July 2017) =E2=80=98=
guix
publish=E2=80=99 on GuixSD runs in a UTF-8 locale to avoid that problem.

The faulty narinfo/nar on berlin were generated on Oct. 17, 2018, so
clearly the above commit was in effect.  Indeed, after removing them and
regenerating them, I=E2=80=99m still getting
08ziz714diyfq2klxy1nc0nhr5wa2vd356n9vizlq913a7an9a9s (aka. the wrong
hash).

On closer inspection the problem is elsewhere: the
/gnu/store/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39 directory on
berlin has question marks in file names, so =E2=80=98guix publish=E2=80=99 =
is not to
blame; instead the problem likely comes from =E2=80=98guix offload=E2=80=99.

Indeed =E2=80=98guix-daemon=E2=80=99 and its child processes such as =E2=80=
=98guix offload=E2=80=99 run
with an empty environment, and thus in the C locale.  Specifically,
=E2=80=98restore-file-set=E2=80=99 on the build farm front-end must be the =
one
substituting question marks to the non-ASCII characters.

If this analysis is correct, the patch below should fix it.  I=E2=80=99ll t=
ry it
later.

Thanks,
Ludo=E2=80=99.


--=-=-=
Content-Type: text/x-patch
Content-Disposition: inline

diff --git a/gnu/services/base.scm b/gnu/services/base.scm
index cee9898d79..9fe64e8087 100644
--- a/gnu/services/base.scm
+++ b/gnu/services/base.scm
@@ -1603,7 +1603,15 @@ failed to register public key '~a': ~a~%" key status))))))))
                             '())
                      #$@(if tmpdir
                             (list (string-append "TMPDIR=" tmpdir))
-                            '()))
+                            '())
+
+                     ;; Make sure we run in a UTF-8 locale so that 'guix
+                     ;; offload' correctly restores nars that contain UTF-8
+                     ;; file names such as 'nss-certs'.  See
+                     ;; <https://bugs.gnu.org/32942>.
+                     (string-append "GUIX_LOCPATH="
+                                    #$glibc-utf8-locales "/lib/locale")
+                     "LC_ALL=en_US.utf8")
 
                #:log-file #$log-file))
            (stop #~(make-kill-destructor))))))

--=-=-=--




Information forwarded to bug-guix@HIDDEN:
bug#32942; Package guix. Full text available.
Severity set to 'important' from 'normal' Request was from ludo@HIDDEN (Ludovic Courtès) to control <at> debbugs.gnu.org. Full text available.

Message received at submit <at> debbugs.gnu.org:


Received: (at submit) by debbugs.gnu.org; 5 Oct 2018 08:18:37 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Oct 05 04:18:37 2018
Received: from localhost ([127.0.0.1]:37374 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1g8LJj-0002yV-Sx
	for submit <at> debbugs.gnu.org; Fri, 05 Oct 2018 04:18:37 -0400
Received: from eggs.gnu.org ([208.118.235.92]:51801)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <julien@HIDDEN>) id 1g8LJi-0002yG-37
 for submit <at> debbugs.gnu.org; Fri, 05 Oct 2018 04:18:34 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <julien@HIDDEN>) id 1g8LJV-0002At-TJ
 for submit <at> debbugs.gnu.org; Fri, 05 Oct 2018 04:18:27 -0400
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00 autolearn=disabled
 version=3.3.2
Received: from lists.gnu.org ([2001:4830:134:3::11]:37872)
 by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32)
 (Exim 4.71) (envelope-from <julien@HIDDEN>) id 1g8LJQ-0001m0-QS
 for submit <at> debbugs.gnu.org; Fri, 05 Oct 2018 04:18:18 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:33106)
 by lists.gnu.org with esmtp (Exim 4.71)
 (envelope-from <julien@HIDDEN>) id 1g8LJP-0007hB-S2
 for bug-guix@HIDDEN; Fri, 05 Oct 2018 04:18:16 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <julien@HIDDEN>) id 1g8LJH-00017z-W7
 for bug-guix@HIDDEN; Fri, 05 Oct 2018 04:18:14 -0400
Received: from lepiller.eu ([2a00:5884:8208::1]:45394)
 by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)
 (Exim 4.71) (envelope-from <julien@HIDDEN>) id 1g8LJH-0008Of-Hy
 for bug-guix@HIDDEN; Fri, 05 Oct 2018 04:18:07 -0400
Received: from webmail.lepiller.eu (static-176-182-42-79.ncc.abo.bbox.fr
 [176.182.42.79]) by lepiller.eu (OpenSMTPD) with ESMTPSA id 37a73608
 (TLSv1.2:ECDHE-RSA-CHACHA20-POLY1305:256:NO) for <bug-guix@HIDDEN>;
 Fri, 5 Oct 2018 08:19:43 +0000 (UTC)
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII;
 format=flowed
Content-Transfer-Encoding: 7bit
Date: Fri, 05 Oct 2018 10:17:45 +0200
From: Julien Lepiller <julien@HIDDEN>
To: bug-guix@HIDDEN
Subject: nss-certs not deterministic
Message-ID: <3974e5005881951012bb5e55a5bfabe2@HIDDEN>
X-Sender: julien@HIDDEN
User-Agent: Roundcube Webmail/1.3.6
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
 recognized.
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x
X-Received-From: 2001:4830:134:3::11
X-Spam-Score: -5.0 (-----)
X-Debbugs-Envelope-To: submit
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -6.0 (------)

While updating a profile, I found that nss-certs was not deterministic. 
 From ludo:

$ wget -O - -q 
https://mirror.hydra.gnu.org/mbs5mavs3gi4y7xkywcwwjj9g3p1yjmv.narinfo | 
grep Hash
NarHash: sha256:101v69xp1qzw9v6pgmbhw7gfdaic8vvs4v5l567lx7f2mjp25rla
$ wget -O - -q 
https://berlin.guixsd.org/mbs5mavs3gi4y7xkywcwwjj9g3p1yjmv.narinfo | 
grep Hash
NarHash: sha256:08ziz714diyfq2klxy1nc0nhr5wa2vd356n9vizlq913a7an9a9s




Acknowledgement sent to Julien Lepiller <julien@HIDDEN>:
New bug report received and forwarded. Copy sent to bug-guix@HIDDEN. Full text available.
Report forwarded to bug-guix@HIDDEN:
bug#32942; Package guix. Full text available.
Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.
Last modified: Wed, 5 Dec 2018 14:15:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.