GNU bug report logs - #22533
Non-determinism in python-3 ".pyc" bytecode

Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.

Package: guix; Severity: important; Reported by: Leo Famulari <leo@HIDDEN>; Done: Ricardo Wurmus <rekado@HIDDEN>; Maintainer for guix is bug-guix@HIDDEN.

Message received at 22533 <at> debbugs.gnu.org:


Received: (at 22533) by debbugs.gnu.org; 4 Feb 2019 22:39:25 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Feb 04 17:39:25 2019
Received: from localhost ([127.0.0.1]:59835 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gqmth-0002H0-Ez
	for submit <at> debbugs.gnu.org; Mon, 04 Feb 2019 17:39:25 -0500
Received: from hera.aquilenet.fr ([185.233.100.1]:41818)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1gqmtf-0002Gs-Nh
 for 22533 <at> debbugs.gnu.org; Mon, 04 Feb 2019 17:39:24 -0500
Received: from localhost (localhost [127.0.0.1])
 by hera.aquilenet.fr (Postfix) with ESMTP id 2A87DBAD2;
 Mon,  4 Feb 2019 23:39:23 +0100 (CET)
X-Virus-Scanned: Debian amavisd-new at aquilenet.fr
Received: from hera.aquilenet.fr ([127.0.0.1])
 by localhost (hera.aquilenet.fr [127.0.0.1]) (amavisd-new, port 10024)
 with ESMTP id V6xIuOkh4eYD; Mon,  4 Feb 2019 23:39:22 +0100 (CET)
Received: from ribbon (unknown [IPv6:2a01:e0a:1d:7270:af76:b9b:ca24:c465])
 by hera.aquilenet.fr (Postfix) with ESMTPSA id 32F66B80B;
 Mon,  4 Feb 2019 23:39:22 +0100 (CET)
From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>
To: 22533 <at> debbugs.gnu.org
Subject: Re: bug#22533: Python bytecode reproducibility
References: <20160202051544.GA11744@jasmine> <87bmqfu44s.fsf@HIDDEN>
 <87606c23bq.fsf@HIDDEN>
 <CAE4v=phoeRvKHG96Y1-_A2TRAK_wNPUyBgMFgq6+Cd9enFScUg@HIDDEN>
 <874llw101c.fsf@HIDDEN> <871sgz1wg0.fsf@HIDDEN>
 <CAE4v=pjfpPfuihBbFVrB_Tx5py3hp+3aaFDJ6E++tHR2YPwXYA@HIDDEN>
 <87sgxvtl19.fsf@HIDDEN> <87va20wos0.fsf@HIDDEN>
X-URL: http://www.fdn.fr/~lcourtes/
X-Revolutionary-Date: 16 =?utf-8?Q?Pluvi=C3=B4se?= an 227 de la
 =?utf-8?Q?R=C3=A9volution?=
X-PGP-Key-ID: 0x090B11993D9AEBB5
X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc
X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4  0CFB 090B 1199 3D9A EBB5
X-OS: x86_64-pc-linux-gnu
Date: Mon, 04 Feb 2019 23:39:21 +0100
In-Reply-To: <87va20wos0.fsf@HIDDEN> (Ricardo Wurmus's message of "Sun,
 03 Feb 2019 22:22:23 +0100")
Message-ID: <87h8djmb52.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: 1.0 (+)
X-Debbugs-Envelope-To: 22533
Cc: rekado@HIDDEN, leo@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.0 (/)

Ricardo Wurmus <rekado@HIDDEN> skribis:

> Ricardo Wurmus <rekado@HIDDEN> writes:
>
>> Now that we=E2=80=99re using Python 3.7 and this version supports hash-b=
ased pyc
>> files, is this still an issue?  Do we need to do anything to enable
>> hash-based pyc compilation?
>>
>> See:
>>   https://docs.python.org/3/whatsnew/3.7.html#pep-552-hash-based-pyc-fil=
es
>>   https://www.python.org/dev/peps/pep-0552/
>
> It looks like this is no longer a problem.  I built borg just now and
> the pyc files are reproducible.

Yay! \o/

Ludo'.




Information forwarded to bug-guix@HIDDEN:
bug#22533; Package guix. Full text available.

Message received at 22533-done <at> debbugs.gnu.org:


Received: (at 22533-done) by debbugs.gnu.org; 3 Feb 2019 21:22:54 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Feb 03 16:22:54 2019
Received: from localhost ([127.0.0.1]:57821 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gqPE5-00056c-Nr
	for submit <at> debbugs.gnu.org; Sun, 03 Feb 2019 16:22:53 -0500
Received: from sender-of-o51.zoho.com ([135.84.80.216]:21111)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <rekado@HIDDEN>) id 1gqPE3-00056S-4H
 for 22533-done <at> debbugs.gnu.org; Sun, 03 Feb 2019 16:22:52 -0500
ARC-Seal: i=1; a=rsa-sha256; t=1549228948; cv=none; d=zoho.com; s=zohoarc; 
 b=ICO3bVVlfJKS45PNXiI5e4N2bDj8to7VvzfEF9Isb2r2G+sEZZlRZVmEgKS+hSGk7L4wfHE5sPU0XvOAQ5o0B1NSVrFnT2Z6v6iQOFDpmwYstsMCLnc/K/e7vkm+/xjcs6Vvfkb1wnBizwQ8/bdfPh9mGnFot/tfH5sb0kqV0LU=
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com;
 s=zohoarc; t=1549228948;
 h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:To:ARC-Authentication-Results;
 bh=Qv5/AGQCxkAGV7x/YAbL0JwXg9Hggy5+bH4tvyKM7Ts=; 
 b=l76/o0yUOZQjNhD6zZByF24yy0fwF88wo+ISLKjZKr4TNbsLRijUNWg90JTEtbgq4Kde42TNlp1mRJD9lGLjkGFNEKFuXpWZ9LGjczs6c6dDx17OT7IefK98eg8FzOQj1KZDEUHuMgGUTdvG5uuVV6XdyuThhb522FaK8VXXE+0=
ARC-Authentication-Results: i=1; mx.zoho.com; dkim=pass  header.i=elephly.net;
 spf=pass  smtp.mailfrom=rekado@HIDDEN;
 dmarc=pass header.from=<rekado@HIDDEN> header.from=<rekado@HIDDEN>
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1549228948; 
 s=zoho; d=elephly.net; i=rekado@HIDDEN;
 h=References:From:To:Cc:Subject:In-reply-to:Date:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding;
 l=581; bh=Qv5/AGQCxkAGV7x/YAbL0JwXg9Hggy5+bH4tvyKM7Ts=;
 b=hAzYtYosdon/CXH1P5CIGZpL3Va9h3GFAx3ZhG5z3L4bx/Nl4Un7GwaRU1Dsdop/
 Cd18fsrsGLgPbyX8Ag+odsw+oIBzVJRs9UgK4ifYrHx59ySaCHCdu9FueothJHh7d4S
 CYQavK0xZwQ6wg6tkRQ3ICIWVupmmRGaUPUFJ8xc=
Received: from localhost (252.139-78-194.adsl-static.isp.belgacom.be
 [194.78.139.252]) by mx.zohomail.com
 with SMTPS id 1549228946135147.2639457165526;
 Sun, 3 Feb 2019 13:22:26 -0800 (PST)
References: <20160202051544.GA11744@jasmine> <87bmqfu44s.fsf@HIDDEN>
 <87606c23bq.fsf@HIDDEN>
 <CAE4v=phoeRvKHG96Y1-_A2TRAK_wNPUyBgMFgq6+Cd9enFScUg@HIDDEN>
 <874llw101c.fsf@HIDDEN> <871sgz1wg0.fsf@HIDDEN>
 <CAE4v=pjfpPfuihBbFVrB_Tx5py3hp+3aaFDJ6E++tHR2YPwXYA@HIDDEN>
 <87sgxvtl19.fsf@HIDDEN>
User-agent: mu4e 1.0; emacs 26.1
From: Ricardo Wurmus <rekado@HIDDEN>
To: =?utf-8?Q?G=C3=A1bor?= Boskovits <boskovits@HIDDEN>
Subject: Re: bug#22533: Python bytecode reproducibility
In-reply-to: <87sgxvtl19.fsf@HIDDEN>
X-URL: https://elephly.net
X-PGP-Key: https://elephly.net/rekado.pubkey
X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
Date: Sun, 03 Feb 2019 22:22:23 +0100
Message-ID: <87va20wos0.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-ZohoMailClient: External
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 22533-done
Cc: Marius Bakke <mbakke@HIDDEN>, 22533-done <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)


Ricardo Wurmus <rekado@HIDDEN> writes:

> Now that we=E2=80=99re using Python 3.7 and this version supports hash-ba=
sed pyc
> files, is this still an issue?  Do we need to do anything to enable
> hash-based pyc compilation?
>
> See:
>   https://docs.python.org/3/whatsnew/3.7.html#pep-552-hash-based-pyc-files
>   https://www.python.org/dev/peps/pep-0552/

It looks like this is no longer a problem.  I built borg just now and
the pyc files are reproducible.

(The man pages include a date stamp, though, which I=E2=80=99m trying to pa=
tch
now.)

--
Ricardo





Notification sent to Leo Famulari <leo@HIDDEN>:
bug acknowledged by developer. Full text available.
Reply sent to Ricardo Wurmus <rekado@HIDDEN>:
You have taken responsibility. Full text available.

Message received at 22533 <at> debbugs.gnu.org:


Received: (at 22533) by debbugs.gnu.org; 14 Jan 2019 13:40:42 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Jan 14 08:40:42 2019
Received: from localhost ([127.0.0.1]:58379 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gj2Tq-0004Ef-D9
	for submit <at> debbugs.gnu.org; Mon, 14 Jan 2019 08:40:42 -0500
Received: from sender-of-o53.zoho.com ([135.84.80.218]:21812)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <rekado@HIDDEN>) id 1gj2To-0004EW-3L
 for 22533 <at> debbugs.gnu.org; Mon, 14 Jan 2019 08:40:41 -0500
ARC-Seal: i=1; a=rsa-sha256; t=1547473223; cv=none; d=zoho.com; s=zohoarc; 
 b=eJUUpakaAFQWDaUGP/Hv9G39tPxReyPqKvbxdqwc8ZyyXSTbt530aJ+1N4bPqd6F8MUCE84ACApsnc+kbIO/v3adjMTqc9lXAlYFVCID9jTrjo4r870YZF3ZG/witZ46UN9KVdbHTlD03ddvj7MfKzz0D6FVFK9JnowUiXZLJDA=
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com;
 s=zohoarc; t=1547473223;
 h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:To:ARC-Authentication-Results;
 bh=/18kGDRKl2kk9R5xc4f1l7+/gcZOeNDiKiIEyIqfOhY=; 
 b=TDcZDUc6gFABMNRAxLnF/WpPE6g/MqLDvo1lUTxTyGDqzrXP/AE52WEGSTEFkgE3pI3jBjLu8pQVUxl/GXaaoBUNuJIStPMuMXrwbXnnYiKArPA6FV5OZvq8O4pry95juJxWBtT+GmqRtfvi/kiTP4m9IXSMX3trJQ8XcsN3Cio=
ARC-Authentication-Results: i=1; mx.zoho.com; dkim=pass  header.i=elephly.net;
 spf=pass  smtp.mailfrom=rekado@HIDDEN;
 dmarc=pass header.from=<rekado@HIDDEN> header.from=<rekado@HIDDEN>
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1547473223; 
 s=zoho; d=elephly.net; i=rekado@HIDDEN;
 h=References:From:To:Cc:Subject:In-reply-to:Date:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding;
 l=329; bh=/18kGDRKl2kk9R5xc4f1l7+/gcZOeNDiKiIEyIqfOhY=;
 b=FYIddyFw8N5H8E1gfCf9DxEHJVpqMk4wUbd4GumfGE60UkzmQTBAUtSVbHuuea62
 CPPx+AVMm155fMwogz0tWGqBBr6khQP6yg6bnIIuUd6KNH5LcssoD6YIp9NXeGaC6hf
 Y9XWd9uhbCZDozxGLGuhcD+AOZ7mmvECFL3q+71M=
Received: from localhost (141.80.247.192 [141.80.247.192]) by mx.zohomail.com
 with SMTPS id 1547473222247455.17296416035083;
 Mon, 14 Jan 2019 05:40:22 -0800 (PST)
References: <20160202051544.GA11744@jasmine> <87bmqfu44s.fsf@HIDDEN>
 <87606c23bq.fsf@HIDDEN>
 <CAE4v=phoeRvKHG96Y1-_A2TRAK_wNPUyBgMFgq6+Cd9enFScUg@HIDDEN>
 <874llw101c.fsf@HIDDEN> <871sgz1wg0.fsf@HIDDEN>
 <CAE4v=pjfpPfuihBbFVrB_Tx5py3hp+3aaFDJ6E++tHR2YPwXYA@HIDDEN>
User-agent: mu4e 1.0; emacs 26.1
From: Ricardo Wurmus <rekado@HIDDEN>
To: =?utf-8?Q?G=C3=A1bor?= Boskovits <boskovits@HIDDEN>
Subject: Re: bug#22533: Python bytecode reproducibility
In-reply-to: <CAE4v=pjfpPfuihBbFVrB_Tx5py3hp+3aaFDJ6E++tHR2YPwXYA@HIDDEN>
X-URL: https://elephly.net
X-PGP-Key: https://elephly.net/rekado.pubkey
X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
Date: Mon, 14 Jan 2019 14:40:18 +0100
Message-ID: <87sgxvtl19.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-ZohoMailClient: External
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 22533
Cc: Marius Bakke <mbakke@HIDDEN>, 22533 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)


Now that we=E2=80=99re using Python 3.7 and this version supports hash-base=
d pyc
files, is this still an issue?  Do we need to do anything to enable
hash-based pyc compilation?

See:
  https://docs.python.org/3/whatsnew/3.7.html#pep-552-hash-based-pyc-files
  https://www.python.org/dev/peps/pep-0552/

--=20
Ricardo





Information forwarded to bug-guix@HIDDEN:
bug#22533; Package guix. Full text available.

Message received at 22533 <at> debbugs.gnu.org:


Received: (at 22533) by debbugs.gnu.org; 8 Mar 2018 10:40:01 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Mar 08 05:40:01 2018
Received: from localhost ([127.0.0.1]:50224 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1etsxt-0005NM-0A
	for submit <at> debbugs.gnu.org; Thu, 08 Mar 2018 05:40:01 -0500
Received: from mail-io0-f179.google.com ([209.85.223.179]:34298)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <boskovits@HIDDEN>) id 1etsxq-0005N7-R6
 for 22533 <at> debbugs.gnu.org; Thu, 08 Mar 2018 05:39:59 -0500
Received: by mail-io0-f179.google.com with SMTP id e7so6571426ioj.1
 for <22533 <at> debbugs.gnu.org>; Thu, 08 Mar 2018 02:39:58 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=mime-version:in-reply-to:references:from:date:message-id:subject:to
 :cc; bh=gH+oV29piaFeIxyPi0KtOVbkhHxRPe4yrBv4bE0U4Z4=;
 b=bHIjQMCUKo8q1LUQDHFRVm8v1/YzQqu/mhsSm9HWGPdD2A5ddk3DR2BtPSL0P5VaWS
 ErGcfDYart4gxpqPf+p7maHyld4Rrd1QxerW/mYjvW2e2GF+JyU5tyx86oWNnNQFTLFU
 8h/Hp0EH9tGXy8YgH1WYaEdwNkoCWPbdC/HkdkqL3VrjUeSzrQ3Kz5O0WdDkHcFyKc7u
 kKZyD0sMTGg+oVsbgvsJrgApdZ9IbpVJrUEKQC2nBPd8qkUgcBKxLosTaxb6sLW732d4
 277E4M1bFSIIg5fIvX28rHi4H7qqzKTR7cYCn0PxGQx68q47/S0Jyr/pExpldhSlgOMU
 oUTA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:mime-version:in-reply-to:references:from:date
 :message-id:subject:to:cc;
 bh=gH+oV29piaFeIxyPi0KtOVbkhHxRPe4yrBv4bE0U4Z4=;
 b=lAlbj9mllbVFmFPUCJCPFv14bNMpSoI+Q0LpvGG3nMKeJ61HxYluaAgv+HdxLuNv8y
 kYUtWDzqIkWehbRbU7PVQj7QJsTMbw75NmVmisVHj0PiZa3kCIuIJazNcaLaRptpP/O/
 QgpXkqMcBjLleLFS+aAoC2a5cMvNDvgRv9131pLhbcK6kDdSskYKJLRJnIVEV/tMuP32
 Gh55hSNAWxCRNOELvdTpCm0FvQ6n/9KkSHPi8JtALEOIQRwi30T3Lq1ZiHL5/CHW2Vmo
 bugjP/+h7yCKyJJvjfHpJ+IM9BYxJu9MqG25DV3jFPpquyB1+a7Pc3R4st+NrO7xK+wx
 x6EA==
X-Gm-Message-State: AElRT7Hsb77rDPeL3e34GsvYEPWMKqf1yiU87bOxaJR7680Ohb45XKy2
 mRegbKa66ePj59eEjG9OuVEMdEB4nxCfOdGd7Q==
X-Google-Smtp-Source: AG47ELukVLtnj49nhofsDNYPLRr3u7KPT5QLkpCdPkKmZc+uTkrYGo90LiDwBYzYpoi9Wh0w45vovjQJ/PlXKxit5Bg=
X-Received: by 10.107.178.206 with SMTP id b197mr32072660iof.9.1520505593104; 
 Thu, 08 Mar 2018 02:39:53 -0800 (PST)
MIME-Version: 1.0
Received: by 10.79.157.15 with HTTP; Thu, 8 Mar 2018 02:39:52 -0800 (PST)
In-Reply-To: <871sgz1wg0.fsf@HIDDEN>
References: <20160202051544.GA11744@jasmine> <87bmqfu44s.fsf@HIDDEN>
 <87606c23bq.fsf@HIDDEN>
 <CAE4v=phoeRvKHG96Y1-_A2TRAK_wNPUyBgMFgq6+Cd9enFScUg@HIDDEN>
 <874llw101c.fsf@HIDDEN> <871sgz1wg0.fsf@HIDDEN>
From: =?UTF-8?Q?G=C3=A1bor_Boskovits?= <boskovits@HIDDEN>
Date: Thu, 8 Mar 2018 11:39:52 +0100
Message-ID: <CAE4v=pjfpPfuihBbFVrB_Tx5py3hp+3aaFDJ6E++tHR2YPwXYA@HIDDEN>
Subject: Re: bug#22533: Python bytecode reproducibility
To: Ricardo Wurmus <rekado@HIDDEN>
Content-Type: multipart/alternative; boundary="001a114c9d88c18df00566e44b12"
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 22533
Cc: Marius Bakke <mbakke@HIDDEN>, 22533 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: 0.0 (/)

--001a114c9d88c18df00566e44b12
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

2018-03-04 20:18 GMT+01:00 Ricardo Wurmus <rekado@HIDDEN>:

> I have applied this patch locally:
>
>
> diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
> index 5f701701a..0d1ecc3c6 100644
> --- a/gnu/packages/python.scm
> +++ b/gnu/packages/python.scm
> @@ -359,8 +359,42 @@ data types.")
>                                "Lib/ctypes/test/test_win32.py" ; fails on
> aarch64
>                                "Lib/test/test_fcntl.py")) ; fails on
> aarch64
>                    #t))))
> -    (arguments (substitute-keyword-arguments (package-arguments python-2=
)
> -                 ((#:tests? _) #t)))
> +    (arguments
> +     (substitute-keyword-arguments (package-arguments python-2)
> +       ((#:tests? _) #t)
> +       ((#:phases phases)
> +        `(modify-phases ,phases
> +           (add-after 'unpack 'patch-timestamp-for-pyc-files
> +             (lambda _
> +               ;; We set DETERMINISTIC_BUILD to only override the mtime
> when
> +               ;; building with Guix, lest we break auto-compilation in
> +               ;; environments.
> +               (setenv "DETERMINISTIC_BUILD" "1")
> +               (substitute* "Lib/py_compile.py"
> +                 (("source_stats\\['mtime'\\]")
> +                  "(1 if 'DETERMINISTIC_BUILD' in os.environ else
> source_stats['mtime'])"))
> +
> +               ;; Use deterministic hashes for strings, bytes, and
> datetime
> +               ;; objects.
> +               (setenv "PYTHONHASHSEED" "0")
> +
> +               ;; Reset mtime when validating bytecode header.
> +               (substitute* "Lib/importlib/_bootstrap_external.py"
> +                 (("source_mtime =3D int\\(source_stats\\['mtime'\\]\\)"=
)
> +                  "source_mtime =3D 1"))
> +               #t))
> +           (add-after 'unpack 'disable-timestamp-tests
> +             (lambda _
> +               (substitute* "Lib/test/test_importlib/
> source/test_file_loader.py"
> +                 (("test_bad_marshal")
> +                  "disable_test_bad_marshal")
> +                 (("test_no_marshal")
> +                  "disable_test_no_marshal")
> +                 (("test_non_code_marshal")
> +                  "disable_test_non_code_marshal"))
> +               #t))
> +           (add-before 'check 'allow-non-deterministic-compilation
> +             (lambda _ (unsetenv "DETERMINISTIC_BUILD") #t))))))
>      (native-search-paths
>       (list (search-path-specification
>              (variable "PYTHONPATH")
>
>
> It allows me to build python-six and python-sip reproducibly.  It does
> not fix problems with Python 2, and I haven=E2=80=99t yet tested if it ca=
uses
> any new problems.
>
> It=E2=80=99s a little worrying that I had to disable three more tests tha=
t I
> think shouldn=E2=80=99t have failed.
>
>
Ok, I've checked the test issue again. If we change the
_bootstrap_external.py
substitution to:
"source_mtime =3D 1 if 'DETERMINISTIC_BUILD' in _os.environ else
int(source_stats['mtime'])"
the test do not fail any more. WDYT?



> What do you think?
>
> --
> Ricardo
>
> GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
> https://elephly.net
>
>

--001a114c9d88c18df00566e44b12
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div class=3D"gmail_extra"><div class=3D"gmail_quote">2018=
-03-04 20:18 GMT+01:00 Ricardo Wurmus <span dir=3D"ltr">&lt;<a href=3D"mail=
to:rekado@HIDDEN" target=3D"_blank">rekado@HIDDEN</a>&gt;</span>:=
<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-lef=
t:1px #ccc solid;padding-left:1ex">I have applied this patch locally:<br>
<br>
<br>diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm<br>
index 5f701701a..0d1ecc3c6 100644<br>
--- a/gnu/packages/python.scm<br>
+++ b/gnu/packages/python.scm<br>
@@ -359,8 +359,42 @@ data types.&quot;)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0&quot;Lib/ctypes/test/test_win32.<wbr=
>py&quot; ; fails on aarch64<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0&quot;Lib/test/test_fcntl.py&quot;)) =
; fails on aarch64<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0#t))))=
<br>
-=C2=A0 =C2=A0 (arguments (substitute-keyword-arguments (package-arguments =
python-2)<br>
-=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0((#:tests? _=
) #t)))<br>
+=C2=A0 =C2=A0 (arguments<br>
+=C2=A0 =C2=A0 =C2=A0(substitute-keyword-arguments (package-arguments pytho=
n-2)<br>
+=C2=A0 =C2=A0 =C2=A0 =C2=A0((#:tests? _) #t)<br>
+=C2=A0 =C2=A0 =C2=A0 =C2=A0((#:phases phases)<br>
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 `(modify-phases ,phases<br>
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(add-after &#39;unpack &#39;patch=
-timestamp-for-pyc-files<br>
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(lambda _<br>
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0;; We set DETERMINI=
STIC_BUILD to only override the mtime when<br>
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0;; building with Gu=
ix, lest we break auto-compilation in<br>
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0;; environments.<br=
>
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(setenv &quot;DETER=
MINISTIC_BUILD&quot; &quot;1&quot;)<br>
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(substitute* &quot;=
Lib/py_compile.py&quot;<br>
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0((&quot;sour=
ce_stats\\[&#39;mtime&#39;\\]&quot;)<br>
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 &quot;(1 if=
 &#39;DETERMINISTIC_BUILD&#39; in os.environ else source_stats[&#39;mtime&#=
39;])&quot;))<br>
+<br>
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0;; Use deterministi=
c hashes for strings, bytes, and datetime<br>
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0;; objects.<br>
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(setenv &quot;PYTHO=
NHASHSEED&quot; &quot;0&quot;)<br>
+<br>
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0;; Reset mtime when=
 validating bytecode header.<br>
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(substitute* &quot;=
Lib/importlib/_bootstrap_<wbr>external.py&quot;<br>
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0((&quot;sour=
ce_mtime =3D int\\(source_stats\\[&#39;mtime&#39;\\<wbr>]\\)&quot;)<br>
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 &quot;sourc=
e_mtime =3D 1&quot;))<br>
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0#t))<br>
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(add-after &#39;unpack &#39;disab=
le-timestamp-tests<br>
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(lambda _<br>
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(substitute* &quot;=
Lib/test/test_importlib/<wbr>source/test_file_loader.py&quot;<br>
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0((&quot;test=
_bad_marshal&quot;)<br>
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 &quot;disab=
le_test_bad_marshal&quot;)<br>
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0((&quot;test=
_no_marshal&quot;)<br>
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 &quot;disab=
le_test_no_marshal&quot;)<br>
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0((&quot;test=
_non_code_marshal&quot;)<br>
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 &quot;disab=
le_test_non_code_<wbr>marshal&quot;))<br>
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0#t))<br>
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(add-before &#39;check &#39;allow=
-non-deterministic-<wbr>compilation<br>
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(lambda _ (unsetenv &quot;=
DETERMINISTIC_BUILD&quot;) #t))))))<br>
=C2=A0 =C2=A0 =C2=A0(native-search-paths<br>
=C2=A0 =C2=A0 =C2=A0 (list (search-path-specification<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(variable &quot;PYTHONPATH&=
quot;)<br>
<br><br>
It allows me to build python-six and python-sip reproducibly.=C2=A0 It does=
<br>
not fix problems with Python 2, and I haven=E2=80=99t yet tested if it caus=
es<br>
any new problems.<br>
<br>
It=E2=80=99s a little worrying that I had to disable three more tests that =
I<br>
think shouldn=E2=80=99t have failed.<br>
<br></blockquote><div><br></div><div>Ok, I&#39;ve checked the test issue ag=
ain. If we change the _bootstrap_external.py</div><div>substitution to:</di=
v><div>&quot;source_mtime =3D 1 if &#39;DETERMINISTIC_BUILD&#39; in _os.env=
iron else int(source_stats[&#39;mtime&#39;])&quot;</div><div>the test do no=
t fail any more. WDYT?</div><div><br></div><div>=C2=A0</div><blockquote cla=
ss=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;pa=
dding-left:1ex">
What do you think?<br>
<br>
--<br>
Ricardo<br>
<br>
GPG: BCA6 89B6 3655 3801 C3C6=C2=A0 2150 197A 5888 235F ACAC<br>
<a href=3D"https://elephly.net" rel=3D"noreferrer" target=3D"_blank">https:=
//elephly.net</a><br>
<br></blockquote></div><br></div></div>

--001a114c9d88c18df00566e44b12--




Information forwarded to bug-guix@HIDDEN:
bug#22533; Package guix. Full text available.

Message received at 22533 <at> debbugs.gnu.org:


Received: (at 22533) by debbugs.gnu.org; 6 Mar 2018 14:57:11 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Mar 06 09:57:11 2018
Received: from localhost ([127.0.0.1]:47852 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1etE1e-0002WL-Mz
	for submit <at> debbugs.gnu.org; Tue, 06 Mar 2018 09:57:10 -0500
Received: from mail-io0-f170.google.com ([209.85.223.170]:38873)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <boskovits@HIDDEN>) id 1etE1d-0002W7-2S
 for 22533 <at> debbugs.gnu.org; Tue, 06 Mar 2018 09:57:09 -0500
Received: by mail-io0-f170.google.com with SMTP id g21so22321177ioj.5
 for <22533 <at> debbugs.gnu.org>; Tue, 06 Mar 2018 06:57:08 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=mime-version:in-reply-to:references:from:date:message-id:subject:to
 :cc; bh=kZGw1z2xO281mLUXadEKwJZhPrIRLsn1F5fAawgUpPo=;
 b=ouYW6is7EaGhk/Mhn6I9GDd0GMGlw+n/O6ceyl1GynO2207Stvr0TK33EDkINBGgpY
 6OUYvLHSQ9tjuJmSdICW2ZQHhs5cZ1jM30Ubq4+TipRXbrauM6t5rXh2XuRbDlXEShkc
 rFhyX9Hfa5//B/zquWjCaDa8aQmQMJ/Cgf8cy8Pyae2yHPiAkMYA3MZ8Vd62Yf02fZIV
 QMlKdXZefp6Pt2CheTSJLqFq/kstxEQP4kjmIS+IWCkD9ACLaO1wHyF9DvAW5k6I2qYH
 0udEXLZOJGanl8OCcry9noNvo6t7NouUqAjg1gPzvRhgoBSGltStl7JZgPRqqnWRp6H9
 9GMQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:mime-version:in-reply-to:references:from:date
 :message-id:subject:to:cc;
 bh=kZGw1z2xO281mLUXadEKwJZhPrIRLsn1F5fAawgUpPo=;
 b=Cr0sr0qqjiWEaIF+Quph7uS7lrvOo+iRQb9EAZ+AW8OwyaWuaHOEwfw3SrJIloWe9A
 gYctQiCqB/YMUv2lwzHML/00q1Fso6q3YR0zOSMbJZYVe85t3zkmGzEoJiUmCoT8SF23
 o1r/nc4I4ckFweGds++mcbGVAz8PfmPLutApbA0n4Ir7HD0/hOJiI2kLBLy5ZW9p4YcK
 SytcwBBwdiuEor3uR8kwIonq+a2EUFhRpxL+ISQH9lWZfUV6alCKbe5OvuslVzmUiOeM
 JDpFi3R9WSYwegQjj4/HaVckBCf3WKPafW9j60OIPr0Veg5PdGBlOgdsRQbO7coR5JwI
 0xtg==
X-Gm-Message-State: APf1xPAHqxUOD7X6NR3eKChLfxzOn7iTrEvKDfaD/eOCI/MYxpb2DXWJ
 h7motXx5AMnZiREdCkvYr+CJTJ2PM2c63s8gKw==
X-Google-Smtp-Source: AG47ELumAfpHsuSfWsadJWUzlQVk23JbOdurWrtifRNgrjZf8KnH80Og4+1Sqbe/+CBTIP1ao7sEmoA+/406A58Kmdc=
X-Received: by 10.107.53.231 with SMTP id k100mr23601943ioo.292.1520348222937; 
 Tue, 06 Mar 2018 06:57:02 -0800 (PST)
MIME-Version: 1.0
Received: by 10.79.157.15 with HTTP; Tue, 6 Mar 2018 06:57:02 -0800 (PST)
In-Reply-To: <87ina9i7sw.fsf@HIDDEN>
References: <20160202051544.GA11744@jasmine> <87bmqfu44s.fsf@HIDDEN>
 <87606c23bq.fsf@HIDDEN>
 <CAE4v=phoeRvKHG96Y1-_A2TRAK_wNPUyBgMFgq6+Cd9enFScUg@HIDDEN>
 <874llw101c.fsf@HIDDEN> <871sgz1wg0.fsf@HIDDEN>
 <87h8pu153i.fsf@HIDDEN>
 <87muzlib8u.fsf@HIDDEN> <87ina9i7sw.fsf@HIDDEN>
From: =?UTF-8?Q?G=C3=A1bor_Boskovits?= <boskovits@HIDDEN>
Date: Tue, 6 Mar 2018 15:57:02 +0100
Message-ID: <CAE4v=pj=Jbnw82A7NMmjH3DGKk_hkKq0kMNZAjM_QmyMTWbt5w@HIDDEN>
Subject: Re: bug#22533: Python bytecode reproducibility
To: Ricardo Wurmus <rekado@HIDDEN>
Content-Type: multipart/alternative; boundary="001a1144a158c35d230566bfa7af"
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 22533
Cc: Marius Bakke <mbakke@HIDDEN>, 22533 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: 0.0 (/)

--001a1144a158c35d230566bfa7af
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

2018-03-06 15:43 GMT+01:00 Ricardo Wurmus <rekado@HIDDEN>:

>
> Ricardo Wurmus <rekado@HIDDEN> writes:
>
> > Marius Bakke <mbakke@HIDDEN> writes:
> >
> >> I suppose we'll have to set PYTHONHASHSEED somewhere in
> >> python-build-system as well.  Did you check if that makes a difference
> >> for numpy?  Perhaps it's enough to set it if we add an auto-compilatio=
n
> >> step?
> >
> > Right, I=E2=80=99m going to test this with numpy now.  Thanks for the h=
int!
>
> It did help with one file, which is now built reproducibly, namely
>
>   lib/python3.6/site-packages/numpy/testing/nose_tools/__
> pycache__/utils.cpython-36.pyc
>
> This leaves five files in numpy that shouldn=E2=80=99t be but unfortunate=
ly are
> different.
>
>
Unfortunately backporting the upstream version is not straightforward at
all.
There are too many changes. I will have a look at those test failures
instead.


> --
> Ricardo
>
> GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
> https://elephly.net
>
>
>

--001a1144a158c35d230566bfa7af
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div class=3D"gmail_extra"><div class=3D"gmail_quote">2018=
-03-06 15:43 GMT+01:00 Ricardo Wurmus <span dir=3D"ltr">&lt;<a href=3D"mail=
to:rekado@HIDDEN" target=3D"_blank">rekado@HIDDEN</a>&gt;</span>:=
<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-lef=
t:1px #ccc solid;padding-left:1ex"><br>
Ricardo Wurmus &lt;<a href=3D"mailto:rekado@HIDDEN">rekado@HIDDEN=
</a>&gt; writes:<br>
<span class=3D""><br>
&gt; Marius Bakke &lt;<a href=3D"mailto:mbakke@HIDDEN">mbakke@fastmai=
l.com</a>&gt; writes:<br>
&gt;<br>
&gt;&gt; I suppose we&#39;ll have to set PYTHONHASHSEED somewhere in<br>
&gt;&gt; python-build-system as well.=C2=A0 Did you check if that makes a d=
ifference<br>
&gt;&gt; for numpy?=C2=A0 Perhaps it&#39;s enough to set it if we add an au=
to-compilation<br>
&gt;&gt; step?<br>
&gt;<br>
&gt; Right, I=E2=80=99m going to test this with numpy now.=C2=A0 Thanks for=
 the hint!<br>
<br>
</span>It did help with one file, which is now built reproducibly, namely<b=
r>
<br>
=C2=A0 lib/python3.6/site-packages/<wbr>numpy/testing/nose_tools/__<wbr>pyc=
ache__/utils.cpython-36.pyc<br>
<br>
This leaves five files in numpy that shouldn=E2=80=99t be but unfortunately=
 are<br>
different.<br>
<div class=3D"HOEnZb"><div class=3D"h5"><br></div></div></blockquote><div><=
br></div><div>Unfortunately backporting the upstream version is not straigh=
tforward at all.</div><div>There are too many changes. I will have a look a=
t those test failures instead.</div><div>=C2=A0</div><blockquote class=3D"g=
mail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-l=
eft:1ex"><div class=3D"HOEnZb"><div class=3D"h5">
--<br>
Ricardo<br>
<br>
GPG: BCA6 89B6 3655 3801 C3C6=C2=A0 2150 197A 5888 235F ACAC<br>
<a href=3D"https://elephly.net" rel=3D"noreferrer" target=3D"_blank">https:=
//elephly.net</a><br>
<br>
<br>
</div></div></blockquote></div><br></div></div>

--001a1144a158c35d230566bfa7af--




Information forwarded to bug-guix@HIDDEN:
bug#22533; Package guix. Full text available.

Message received at 22533 <at> debbugs.gnu.org:


Received: (at 22533) by debbugs.gnu.org; 6 Mar 2018 14:43:37 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Mar 06 09:43:37 2018
Received: from localhost ([127.0.0.1]:46953 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1etDoX-0001yw-7C
	for submit <at> debbugs.gnu.org; Tue, 06 Mar 2018 09:43:37 -0500
Received: from sender-of-o51.zoho.com ([135.84.80.216]:21056)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <rekado@HIDDEN>) id 1etDoV-0001yo-F3
 for 22533 <at> debbugs.gnu.org; Tue, 06 Mar 2018 09:43:36 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1520347394; 
 s=zoho; d=elephly.net; i=rekado@HIDDEN;
 h=References:From:To:Cc:Subject:In-reply-to:Date:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-ID;
 l=756; bh=gvrXlJH7v9grkQZFy6dwUpjOZZlyUM6p8LDM3YbmYbw=;
 b=aZfHRggoP5qCWvOZuEifJ2AEOtCja18gw4WpEY1qBud8S9vbDI+36ArSFJbggsIk
 vXZscaEP55wHrT9WTCFks7e3flMt5I+qBgCqIidWDKDurk6K6kmJcGf0330AzA36ora
 62LMh57EeFeUBjlBqKI4LYHsja5/OXUFNBGWmmpk=
Received: from localhost (141.80.247.222 [141.80.247.222]) by mx.zohomail.com
 with SMTPS id 1520347394133802.4637948601691;
 Tue, 6 Mar 2018 06:43:14 -0800 (PST)
References: <20160202051544.GA11744@jasmine> <87bmqfu44s.fsf@HIDDEN>
 <87606c23bq.fsf@HIDDEN>
 <CAE4v=phoeRvKHG96Y1-_A2TRAK_wNPUyBgMFgq6+Cd9enFScUg@HIDDEN>
 <874llw101c.fsf@HIDDEN> <871sgz1wg0.fsf@HIDDEN>
 <87h8pu153i.fsf@HIDDEN> <87muzlib8u.fsf@HIDDEN>
User-agent: mu4e 1.0; emacs 25.3.1
From: Ricardo Wurmus <rekado@HIDDEN>
To: Marius Bakke <mbakke@HIDDEN>
Subject: Re: bug#22533: Python bytecode reproducibility
In-reply-to: <87muzlib8u.fsf@HIDDEN>
X-URL: https://elephly.net
X-PGP-Key: https://elephly.net/rekado.pubkey
X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
Date: Tue, 06 Mar 2018 15:43:11 +0100
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Message-ID: <87ina9i7sw.fsf@HIDDEN>
X-ZohoMailClient: External
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 22533
Cc: =?utf-8?Q?G=C3=A1bor?= Boskovits <boskovits@HIDDEN>,
 22533 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.0 (/)


Ricardo Wurmus <rekado@HIDDEN> writes:

> Marius Bakke <mbakke@HIDDEN> writes:
>
>> I suppose we'll have to set PYTHONHASHSEED somewhere in
>> python-build-system as well.  Did you check if that makes a difference
>> for numpy?  Perhaps it's enough to set it if we add an auto-compilation
>> step?
>
> Right, I=E2=80=99m going to test this with numpy now.  Thanks for the hin=
t!

It did help with one file, which is now built reproducibly, namely

  lib/python3.6/site-packages/numpy/testing/nose_tools/__pycache__/utils.cp=
ython-36.pyc

This leaves five files in numpy that shouldn=E2=80=99t be but unfortunately=
 are
different.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net






Information forwarded to bug-guix@HIDDEN:
bug#22533; Package guix. Full text available.

Message received at 22533 <at> debbugs.gnu.org:


Received: (at 22533) by debbugs.gnu.org; 6 Mar 2018 13:29:14 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Mar 06 08:29:14 2018
Received: from localhost ([127.0.0.1]:46919 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1etCeY-0006jO-4W
	for submit <at> debbugs.gnu.org; Tue, 06 Mar 2018 08:29:14 -0500
Received: from sender-of-o51.zoho.com ([135.84.80.216]:21111)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <rekado@HIDDEN>) id 1etCeW-0006jF-6M
 for 22533 <at> debbugs.gnu.org; Tue, 06 Mar 2018 08:29:12 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1520342932; 
 s=zoho; d=elephly.net; i=rekado@HIDDEN;
 h=References:From:To:Cc:Subject:In-reply-to:Date:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-ID;
 l=1070; bh=v66ZWnvWbKyG4FZccvwK5Il8gg4bgyP+Gir93W6aVHM=;
 b=byP5w1sn2X8M+Nj4j42xiuT8Jmi9itCRVqY9SG74jV+WdKn0//PS8mWHFLnIKCVJ
 rW1y2BhDniyEG+P9QIrm73rvQPM2RIhFw9Mwp/FkUurrx4vq1Epsx5CjuiuGeXq3pTU
 cZR0OF9vn1CVpy2epZ28s38y/l0J291IXT2wlDq4=
Received: from localhost (141.80.247.222 [141.80.247.222]) by mx.zohomail.com
 with SMTPS id 152034293253063.76400782171072;
 Tue, 6 Mar 2018 05:28:52 -0800 (PST)
References: <20160202051544.GA11744@jasmine> <87bmqfu44s.fsf@HIDDEN>
 <87606c23bq.fsf@HIDDEN>
 <CAE4v=phoeRvKHG96Y1-_A2TRAK_wNPUyBgMFgq6+Cd9enFScUg@HIDDEN>
 <874llw101c.fsf@HIDDEN> <871sgz1wg0.fsf@HIDDEN>
 <87h8pu153i.fsf@HIDDEN>
User-agent: mu4e 1.0; emacs 25.3.1
From: Ricardo Wurmus <rekado@HIDDEN>
To: Marius Bakke <mbakke@HIDDEN>
Subject: Re: bug#22533: Python bytecode reproducibility
In-reply-to: <87h8pu153i.fsf@HIDDEN>
X-URL: https://elephly.net
X-PGP-Key: https://elephly.net/rekado.pubkey
X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
Date: Tue, 06 Mar 2018 14:28:49 +0100
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Message-ID: <87muzlib8u.fsf@HIDDEN>
X-ZohoMailClient: External
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 22533
Cc: =?utf-8?Q?G=C3=A1bor?= Boskovits <boskovits@HIDDEN>,
 22533 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.0 (/)


Marius Bakke <mbakke@HIDDEN> writes:

> The only remark I have is: is introducing a new variable necessary?
> SOURCE_DATE_EPOCH implies that the user wants a deterministic build;
> the upstream patch doesn't actually honor it outside of making the
> hashing method deterministic.  So, I think it might be enough to just
> test for SOURCE_DATE_EPOCH instead of DETERMINISTIC_BUILD.  The former
> is also already set in the build environment.

> However, I just noticed that you unset DETERMINISTIC_BUILD before the
> 'check' phase.  Did it break more things?

Yes, it broke a bunch of tests that are all about recompiling files when
they are considered stale.

> I suppose we'll have to set PYTHONHASHSEED somewhere in
> python-build-system as well.  Did you check if that makes a difference
> for numpy?  Perhaps it's enough to set it if we add an auto-compilation
> step?

Right, I=E2=80=99m going to test this with numpy now.  Thanks for the hint!

--=20
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net






Information forwarded to bug-guix@HIDDEN:
bug#22533; Package guix. Full text available.

Message received at 22533 <at> debbugs.gnu.org:


Received: (at 22533) by debbugs.gnu.org; 5 Mar 2018 23:21:33 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Mar 05 18:21:33 2018
Received: from localhost ([127.0.0.1]:46426 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1eszQ6-0004Ly-Ko
	for submit <at> debbugs.gnu.org; Mon, 05 Mar 2018 18:21:33 -0500
Received: from out5-smtp.messagingengine.com ([66.111.4.29]:44553)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mbakke@HIDDEN>) id 1eszQ4-0004Lp-Jf
 for 22533 <at> debbugs.gnu.org; Mon, 05 Mar 2018 18:21:25 -0500
Received: from compute5.internal (compute5.nyi.internal [10.202.2.45])
 by mailout.nyi.internal (Postfix) with ESMTP id 0C40D21957;
 Mon,  5 Mar 2018 18:21:24 -0500 (EST)
Received: from frontend2 ([10.202.2.161])
 by compute5.internal (MEProxy); Mon, 05 Mar 2018 18:21:24 -0500
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastmail.com; h=
 cc:content-type:date:from:in-reply-to:message-id:mime-version
 :references:subject:to:x-me-sender:x-me-sender:x-sasl-enc; s=
 fm2; bh=cW2DVlZ6SG5dq7uWsaqu+k/yeEHYKMBRRWmGKdj8DaE=; b=AdeOmMoU
 J9/jUhYrok8Sn9cK/w2jW3ta04affTGT83bazNXmE1TavSiDGWwJKBxnLWa4xqwQ
 BP6lgWy3RWaFfmwxD+i3yQt0p9KkICasIB9DtS4739VUdglEZPxygRN/WbHWgtDr
 1hgOiDdZRBIyuZ1gd0uokSxRvYkmrRmVHGdtNGpFljp2Xp4xcbAMVLi/ELRY5SPP
 3iOMV0gsONEu5yL4xjMpxOcD2zxqJogmXFewe4eJlOCwOQbf4cqX3qF1VaalPHQD
 zaMaxNdDu3RJ+OoAX7ymdjfhmWOfWvobFio0n1pqBCyM3RRcPkijL2ftFQG3malX
 dyVg4J8DXRS0VA==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:content-type:date:from:in-reply-to
 :message-id:mime-version:references:subject:to:x-me-sender
 :x-me-sender:x-sasl-enc; s=fm2; bh=cW2DVlZ6SG5dq7uWsaqu+k/yeEHYK
 MBRRWmGKdj8DaE=; b=VFL6mofot1iC2kxyOs38HlknAKFfWkAly0d9XJPNsJUHk
 1hLeQFzfI8jxpb+/QhpEdnfLFvnpFamXi/eFECB491YqEek0y7Eh+aeDmHxEBc0Y
 mjApygMgr1ZDtWbED4/+lWq04PMufENkxgwkcB7W3vPbxoKq9sEQ6ovNoLGjEmbb
 UiATmJ8WFBpBAk9bEpbCD/96A4z5mymEYtSl5khc5rJTz0SdbdWfIjqCAtEwZ7lJ
 TBjJcuaq3QryxYPXQkVwjTc0S69tdk5tM620XpP3mtZC07BG7nHldAX1QYvGHgYG
 vZXAv8krzh37sC1c6rVj6gc01i03lhQfTVvb7ei2w==
X-ME-Sender: <xms:89CdWpWt7oWSy4sxpKLEGkWKLnvS3Ocqd-qsGzMb3TOdqqCM9oicSQ>
Received: from localhost (unknown [185.3.4.227])
 by mail.messagingengine.com (Postfix) with ESMTPA id 7D72024838;
 Mon,  5 Mar 2018 18:21:23 -0500 (EST)
From: Marius Bakke <mbakke@HIDDEN>
To: Ricardo Wurmus <rekado@HIDDEN>, =?utf-8?Q?G=C3=A1bor?= Boskovits
 <boskovits@HIDDEN>
Subject: Re: bug#22533: Python bytecode reproducibility
In-Reply-To: <871sgz1wg0.fsf@HIDDEN>
References: <20160202051544.GA11744@jasmine> <87bmqfu44s.fsf@HIDDEN>
 <87606c23bq.fsf@HIDDEN>
 <CAE4v=phoeRvKHG96Y1-_A2TRAK_wNPUyBgMFgq6+Cd9enFScUg@HIDDEN>
 <874llw101c.fsf@HIDDEN> <871sgz1wg0.fsf@HIDDEN>
User-Agent: Notmuch/0.26 (https://notmuchmail.org) Emacs/25.3.1
 (x86_64-pc-linux-gnu)
Date: Tue, 06 Mar 2018 00:21:21 +0100
Message-ID: <87h8pu153i.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="=-=-=";
 micalg=pgp-sha512; protocol="application/pgp-signature"
X-Spam-Score: -0.7 (/)
X-Debbugs-Envelope-To: 22533
Cc: 22533 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.7 (/)

--=-=-=
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

Ricardo Wurmus <rekado@HIDDEN> writes:

> I have applied this patch locally:
>
> diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
> index 5f701701a..0d1ecc3c6 100644
> --- a/gnu/packages/python.scm
> +++ b/gnu/packages/python.scm
> @@ -359,8 +359,42 @@ data types.")
>                                "Lib/ctypes/test/test_win32.py" ; fails on=
 aarch64
>                                "Lib/test/test_fcntl.py")) ; fails on aarc=
h64
>                    #t))))
> -    (arguments (substitute-keyword-arguments (package-arguments python-2)
> -                 ((#:tests? _) #t)))
> +    (arguments
> +     (substitute-keyword-arguments (package-arguments python-2)
> +       ((#:tests? _) #t)
> +       ((#:phases phases)
> +        `(modify-phases ,phases
> +           (add-after 'unpack 'patch-timestamp-for-pyc-files
> +             (lambda _
> +               ;; We set DETERMINISTIC_BUILD to only override the mtime =
when
> +               ;; building with Guix, lest we break auto-compilation in
> +               ;; environments.
> +               (setenv "DETERMINISTIC_BUILD" "1")
> +               (substitute* "Lib/py_compile.py"
> +                 (("source_stats\\['mtime'\\]")
> +                  "(1 if 'DETERMINISTIC_BUILD' in os.environ else source=
_stats['mtime'])"))
> +
> +               ;; Use deterministic hashes for strings, bytes, and datet=
ime
> +               ;; objects.
> +               (setenv "PYTHONHASHSEED" "0")
> +
> +               ;; Reset mtime when validating bytecode header.
> +               (substitute* "Lib/importlib/_bootstrap_external.py"
> +                 (("source_mtime =3D int\\(source_stats\\['mtime'\\]\\)")
> +                  "source_mtime =3D 1"))
> +               #t))
> +           (add-after 'unpack 'disable-timestamp-tests
> +             (lambda _
> +               (substitute* "Lib/test/test_importlib/source/test_file_lo=
ader.py"
> +                 (("test_bad_marshal")
> +                  "disable_test_bad_marshal")
> +                 (("test_no_marshal")
> +                  "disable_test_no_marshal")
> +                 (("test_non_code_marshal")
> +                  "disable_test_non_code_marshal"))
> +               #t))
> +           (add-before 'check 'allow-non-deterministic-compilation
> +             (lambda _ (unsetenv "DETERMINISTIC_BUILD") #t))))))
>      (native-search-paths
>       (list (search-path-specification
>              (variable "PYTHONPATH")
>
> It allows me to build python-six and python-sip reproducibly.  It does
> not fix problems with Python 2, and I haven=E2=80=99t yet tested if it ca=
uses
> any new problems.
>
> It=E2=80=99s a little worrying that I had to disable three more tests tha=
t I
> think shouldn=E2=80=99t have failed.

Woow, nice work!  I can't tell what's going on with the tests, they do
some bytecode manipulation stuff.  Maybe it does not expect the low
timestamp somehow?

https://github.com/python/cpython/blob/374c6e178a7599aae46c857b17c6c8bc19df=
e4c2/Lib/test/test_importlib/source/test_file_loader.py#L457-L484

I guess we'll do at least one 'core-updates' before 3.7 is released, so
it makes sense to include this.  It should also give us some experience
that might be relevant for 2.7, since it probably won't get the upstream
reproducibility patch that relies on 3.7 features.

The only remark I have is: is introducing a new variable necessary?
SOURCE_DATE_EPOCH implies that the user wants a deterministic build;
the upstream patch doesn't actually honor it outside of making the
hashing method deterministic.  So, I think it might be enough to just
test for SOURCE_DATE_EPOCH instead of DETERMINISTIC_BUILD.  The former
is also already set in the build environment.

However, I just noticed that you unset DETERMINISTIC_BUILD before the
'check' phase.  Did it break more things?

I suppose we'll have to set PYTHONHASHSEED somewhere in
python-build-system as well.  Did you check if that makes a difference
for numpy?  Perhaps it's enough to set it if we add an auto-compilation
step?

--=-=-=
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQEzBAEBCgAdFiEEu7At3yzq9qgNHeZDoqBt8qM6VPoFAlqd0PEACgkQoqBt8qM6
VPrDLwf7BrDvvnHtKJms7+uZjPg3VNiBNqXjdLm3+Ic8hXcD+T/rUnPbTDjQfKWS
WlsmaUsDa8W5Xs6nTOaalEw2ifwrB0pqGpEZGOnHsGuldDRgt4AMZmKvojt9A4Bx
kjuz06y9i1R1QQel04rQRpFoiC2D9MYgktJczpxeZJyp/wzbhw6BDBtEdCq7+Fap
crIe2CdIepnHlDu44imBawPu59YHLfzXKA2GdwLS3zjfns6ZLaigQTqSNqqAYxML
mpiwDsReZV/OmAa+xhStZMur2LxBIMSkh+PCm2xgW0H8ZaPdRV69bWmqopRynyLN
uzzKniezMl+HM2SrVuljKa1RMMbfFw==
=vpXg
-----END PGP SIGNATURE-----
--=-=-=--




Information forwarded to bug-guix@HIDDEN:
bug#22533; Package guix. Full text available.

Message received at 22533 <at> debbugs.gnu.org:


Received: (at 22533) by debbugs.gnu.org; 5 Mar 2018 22:07:04 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Mar 05 17:07:04 2018
Received: from localhost ([127.0.0.1]:46362 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1esyG8-0000eC-73
	for submit <at> debbugs.gnu.org; Mon, 05 Mar 2018 17:07:04 -0500
Received: from sender-of-o51.zoho.com ([135.84.80.216]:21047)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <rekado@HIDDEN>) id 1esyG5-0000dk-0C
 for 22533 <at> debbugs.gnu.org; Mon, 05 Mar 2018 17:07:03 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1520287614; 
 s=zoho; d=elephly.net; i=rekado@HIDDEN;
 h=References:From:To:Cc:Subject:In-reply-to:Date:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-ID;
 l=2528; bh=woesyRo1lU0C4gmgneg2GpwmkgX7D+3pq8U95JycRAk=;
 b=H7ViicHgBqaQBxGWK/uJdA2EHe7MfVUfj2RlhJh7UWl84Rycs9X5AfStisUdpVZ0
 HZTnV0WBFL//Bgq7crITF5lWO8kPc1U7jH3cQmLVgN9vSSRmQyWzddd0sLYmMtkq8cu
 54PM8ETMap8ZnauU6rPElbSTIG9CF+D//O9lgZbw=
Received: from localhost (port-92-200-95-79.dynamic.qsc.de [92.200.95.79]) by
 mx.zohomail.com with SMTPS id 1520287614401592.5012332278376;
 Mon, 5 Mar 2018 14:06:54 -0800 (PST)
References: <20160202051544.GA11744@jasmine> <87bmqfu44s.fsf@HIDDEN>
 <87606c23bq.fsf@HIDDEN>
 <CAE4v=phoeRvKHG96Y1-_A2TRAK_wNPUyBgMFgq6+Cd9enFScUg@HIDDEN>
 <874llw101c.fsf@HIDDEN> <871sgz1wg0.fsf@HIDDEN>
 <87y3j7z8xk.fsf@HIDDEN>
User-agent: mu4e 1.0; emacs 25.3.1
From: Ricardo Wurmus <rekado@HIDDEN>
To: =?utf-8?Q?G=C3=A1bor?= Boskovits <boskovits@HIDDEN>
Subject: Re: bug#22533: Python bytecode reproducibility
In-reply-to: <87y3j7z8xk.fsf@HIDDEN>
X-URL: https://elephly.net
X-PGP-Key: https://elephly.net/rekado.pubkey
X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
Date: Mon, 05 Mar 2018 23:06:51 +0100
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Message-ID: <87woyqqiro.fsf@HIDDEN>
X-ZohoMailClient: External
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 22533
Cc: 22533 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.0 (/)


Ricardo Wurmus <rekado@HIDDEN> writes:

> Ricardo Wurmus <rekado@HIDDEN> writes:
>
>> I have applied this patch locally:
>>
>> diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
>> index 5f701701a..0d1ecc3c6 100644
>> --- a/gnu/packages/python.scm
>> +++ b/gnu/packages/python.scm
>> @@ -359,8 +359,42 @@ data types.")
>>                                "Lib/ctypes/test/test_win32.py" ; fails o=
n aarch64
>>                                "Lib/test/test_fcntl.py")) ; fails on aar=
ch64
>>                    #t))))
>> -    (arguments (substitute-keyword-arguments (package-arguments python-=
2)
>> -                 ((#:tests? _) #t)))
>> +    (arguments
>> +     (substitute-keyword-arguments (package-arguments python-2)
>> +       ((#:tests? _) #t)
>> +       ((#:phases phases)
>> +        `(modify-phases ,phases
>> +           (add-after 'unpack 'patch-timestamp-for-pyc-files
>> +             (lambda _
>> +               ;; We set DETERMINISTIC_BUILD to only override the mtime=
 when
>> +               ;; building with Guix, lest we break auto-compilation in
>> +               ;; environments.
>> +               (setenv "DETERMINISTIC_BUILD" "1")
>> +               (substitute* "Lib/py_compile.py"
>> +                 (("source_stats\\['mtime'\\]")
>> +                  "(1 if 'DETERMINISTIC_BUILD' in os.environ else sourc=
e_stats['mtime'])"))
>> +
>> +               ;; Use deterministic hashes for strings, bytes, and date=
time
>> +               ;; objects.
>> +               (setenv "PYTHONHASHSEED" "0")
>> +
>> +               ;; Reset mtime when validating bytecode header.
>> +               (substitute* "Lib/importlib/_bootstrap_external.py"
>> +                 (("source_mtime =3D int\\(source_stats\\['mtime'\\]\\)=
")
>> +                  "source_mtime =3D 1"))
>> +               #t))
>> +           (add-after 'unpack 'disable-timestamp-tests
>> +             (lambda _
>> +               (substitute* "Lib/test/test_importlib/source/test_file_l=
oader.py"
>> +                 (("test_bad_marshal")
>> +                  "disable_test_bad_marshal")
>> +                 (("test_no_marshal")
>> +                  "disable_test_no_marshal")
>> +                 (("test_non_code_marshal")
>> +                  "disable_test_non_code_marshal"))
>> +               #t))
>> +           (add-before 'check 'allow-non-deterministic-compilation
>> +             (lambda _ (unsetenv "DETERMINISTIC_BUILD") #t))))))
>>      (native-search-paths
>>       (list (search-path-specification
>>              (variable "PYTHONPATH")
>>
>> It allows me to build python-six and python-sip reproducibly.  It does
>> not fix problems with Python 2, and I haven=E2=80=99t yet tested if it c=
auses
>> any new problems.

I should also note that Python 3 itself still contains pyc files with
timestamps.  This could be the reason why in Nix all pyc files are
rebuilt (more than once).

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net






Information forwarded to bug-guix@HIDDEN:
bug#22533; Package guix. Full text available.

Message received at 22533 <at> debbugs.gnu.org:


Received: (at 22533) by debbugs.gnu.org; 5 Mar 2018 22:02:58 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Mar 05 17:02:58 2018
Received: from localhost ([127.0.0.1]:46353 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1esyCA-0000XL-HF
	for submit <at> debbugs.gnu.org; Mon, 05 Mar 2018 17:02:58 -0500
Received: from sender-of-o51.zoho.com ([135.84.80.216]:21061)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <rekado@HIDDEN>) id 1esyC9-0000XC-4M
 for 22533 <at> debbugs.gnu.org; Mon, 05 Mar 2018 17:02:57 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1520287352; 
 s=zoho; d=elephly.net; i=rekado@HIDDEN;
 h=References:From:To:Cc:Subject:In-reply-to:Date:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-ID;
 l=3905; bh=izzqYGxRAQxble/mdBDQcoQPP4NELOzBPwMIa3dvQMA=;
 b=dhx2ynvJibiXvvgqGLqnDeFGkLRmkYw0JG37YONnfhecqCqHZnL9+7PTnZy+RlL/
 a7lEor4V/uAaWlp/ICRGaD4NJq+K79f9IJunJOar2hPeu17F0jVPClS70xo6BdG0zTn
 nxK5wjdrv1EVuMPaW7+U1i305bm5nV3TOTkjjk+Q=
Received: from localhost (port-92-200-95-79.dynamic.qsc.de [92.200.95.79]) by
 mx.zohomail.com with SMTPS id 1520287352075495.14100502512724;
 Mon, 5 Mar 2018 14:02:32 -0800 (PST)
References: <20160202051544.GA11744@jasmine> <87bmqfu44s.fsf@HIDDEN>
 <87606c23bq.fsf@HIDDEN>
 <CAE4v=phoeRvKHG96Y1-_A2TRAK_wNPUyBgMFgq6+Cd9enFScUg@HIDDEN>
 <874llw101c.fsf@HIDDEN> <871sgz1wg0.fsf@HIDDEN>
 <87y3j7z8xk.fsf@HIDDEN> <87woyrz8sv.fsf@HIDDEN>
User-agent: mu4e 1.0; emacs 25.3.1
From: Ricardo Wurmus <rekado@HIDDEN>
To: =?utf-8?Q?G=C3=A1bor?= Boskovits <boskovits@HIDDEN>
Subject: Re: bug#22533: Python bytecode reproducibility
In-reply-to: <87woyrz8sv.fsf@HIDDEN>
X-URL: https://elephly.net
X-PGP-Key: https://elephly.net/rekado.pubkey
X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
Date: Mon, 05 Mar 2018 23:02:29 +0100
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Message-ID: <87y3j6qiyy.fsf@HIDDEN>
X-ZohoMailClient: External
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 22533
Cc: 22533 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.0 (/)


Ricardo Wurmus <rekado@HIDDEN> writes:

> Ricardo Wurmus <rekado@HIDDEN> writes:
>
>> Unfortunately, this doesn=E2=80=99t fix all reproducibility problems wit=
h numpy:
>>
>> --8<---------------cut here---------------start------------->8---
>> Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.=
14.0-check/lib/python3.6/site-packages/numpy/distutils/__pycache__/__config=
__.cpython-36.pyc and /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-nu=
mpy-1.14.0/lib/python3.6/site-packages/numpy/distutils/__pycache__/__config=
__.cpython-36.pyc differ
>> Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.=
14.0-check/lib/python3.6/site-packages/numpy/distutils/__pycache__/exec_com=
mand.cpython-36.pyc and /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-=
numpy-1.14.0/lib/python3.6/site-packages/numpy/distutils/__pycache__/exec_c=
ommand.cpython-36.pyc differ
>> Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.=
14.0-check/lib/python3.6/site-packages/numpy/distutils/__pycache__/system_i=
nfo.cpython-36.pyc and /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-n=
umpy-1.14.0/lib/python3.6/site-packages/numpy/distutils/__pycache__/system_=
info.cpython-36.pyc differ
>> Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.=
14.0-check/lib/python3.6/site-packages/numpy/__pycache__/__config__.cpython=
-36.pyc and /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0=
/lib/python3.6/site-packages/numpy/__pycache__/__config__.cpython-36.pyc di=
ffer
>> Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.=
14.0-check/lib/python3.6/site-packages/numpy/__pycache__/version.cpython-36=
.pyc and /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0/li=
b/python3.6/site-packages/numpy/__pycache__/version.cpython-36.pyc differ
>> Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.=
14.0-check/lib/python3.6/site-packages/numpy/testing/nose_tools/__pycache__=
/utils.cpython-36.pyc and /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-pytho=
n-numpy-1.14.0/lib/python3.6/site-packages/numpy/testing/nose_tools/__pycac=
he__/utils.cpython-36.pyc differ
>> --8<---------------cut here---------------end--------------->8---
>
> Here=E2=80=99s what diffoscope says:
>
> --8<---------------cut here---------------start------------->8---
> diffoscope /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.=
0{-check,}/lib/python3.6/site-packages/numpy/__pycache__/version.cpython-36=
.pyc
> --- /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0-check=
/lib/python3.6/site-packages/numpy/__pycache__/version.cpython-36.pyc
> +++ /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0/lib/p=
ython3.6/site-packages/numpy/__pycache__/version.cpython-36.pyc
> @@ -1,8 +1,8 @@
> -00000000: 330d 0d0a fa87 9c5a 2601 0000 e300 0000  3......Z&.......
> +00000000: 330d 0d0a c485 9c5a 2601 0000 e300 0000  3......Z&.......
>  00000010: 0000 0000 0000 0000 0001 0000 0040 0000  .............@..
>  00000020: 0073 2000 0000 6400 5a00 6400 5a01 6400  .s ...d.Z.d.Z.d.
>  00000030: 5a02 6401 5a03 6402 5a04 6504 731c 6502  Z.d.Z.d.Z.e.s.e.
>  00000040: 5a01 6403 5300 2904 7a06 312e 3134 2e30  Z.d.S.).z.1.14.0
>  00000050: da28 3639 3134 6262 3431 6630 6662 3363  .(6914bb41f0fb3c
>  00000060: 3162 6135 3030 6261 6534 6537 6436 3731  1ba500bae4e7d671
>  00000070: 6461 3935 3336 3738 3666 544e 2905 da0d  da9536786fTN)...
> --8<---------------cut here---------------end--------------->8---
>
> In other words: this is the timestamp field of the pyc file.
>
> Maybe this can be avoided by setting DETERMINISTIC_BUILD in the
> python-build-system?

It cannot.

So, something=E2=80=99s still missing from my patch.  Does anyone see what =
might
be missing?

--=20
Ricardo






Information forwarded to bug-guix@HIDDEN:
bug#22533; Package guix. Full text available.

Message received at 22533 <at> debbugs.gnu.org:


Received: (at 22533) by debbugs.gnu.org; 5 Mar 2018 21:47:03 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Mar 05 16:47:03 2018
Received: from localhost ([127.0.0.1]:46349 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1esxwk-0000AZ-T4
	for submit <at> debbugs.gnu.org; Mon, 05 Mar 2018 16:47:03 -0500
Received: from sender-of-o51.zoho.com ([135.84.80.216]:21029)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <rekado@HIDDEN>) id 1esxwi-0000AA-NZ
 for 22533 <at> debbugs.gnu.org; Mon, 05 Mar 2018 16:47:01 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1520286401; 
 s=zoho; d=elephly.net; i=rekado@HIDDEN;
 h=References:From:To:Cc:Subject:In-reply-to:Date:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-ID;
 l=4779; bh=rB8EHrJXIoTMq5ngQ+i7eNck+g8O/1ezFUf9KIMWCDA=;
 b=a+Ml+GCo5V8AVbbCIvU+dsLwmEBVlLU3GqR+XAMlLx/XHg45Y4Xq6dJyH4qJTRPe
 hdXnHE+RNETAoOvfLkODM2ZzI0fAguZwDvUkdDjGganobSI7dJhl54czV+aCXfWBUXJ
 CAAEHXhRAmSZSmpx12KyAOWOaqhroJ269qR5iKok=
Received: from localhost (port-92-200-95-79.dynamic.qsc.de [92.200.95.79]) by
 mx.zohomail.com with SMTPS id 1520286400945738.2980576769127;
 Mon, 5 Mar 2018 13:46:40 -0800 (PST)
References: <20160202051544.GA11744@jasmine> <87bmqfu44s.fsf@HIDDEN>
 <87606c23bq.fsf@HIDDEN>
 <CAE4v=phoeRvKHG96Y1-_A2TRAK_wNPUyBgMFgq6+Cd9enFScUg@HIDDEN>
 <874llw101c.fsf@HIDDEN> <871sgz1wg0.fsf@HIDDEN>
 <87y3j7z8xk.fsf@HIDDEN> <87woyrz8sv.fsf@HIDDEN>
 <CAE4v=pjbSGnkGvB5emBMYNXkwy+2CbE2bVAfE5Ph_DUqV4ezug@HIDDEN>
 <CAE4v=piEhWjz9Jei-WFgYke20QgQWvPks6rcoizagpjida-RsQ@HIDDEN>
User-agent: mu4e 1.0; emacs 25.3.1
From: Ricardo Wurmus <rekado@HIDDEN>
To: =?utf-8?Q?G=C3=A1bor?= Boskovits <boskovits@HIDDEN>
Subject: Re: bug#22533: Python bytecode reproducibility
In-reply-to: <CAE4v=piEhWjz9Jei-WFgYke20QgQWvPks6rcoizagpjida-RsQ@HIDDEN>
X-URL: https://elephly.net
X-PGP-Key: https://elephly.net/rekado.pubkey
X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
Date: Mon, 05 Mar 2018 22:46:38 +0100
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Message-ID: <87zi3mqjpd.fsf@HIDDEN>
X-ZohoMailClient: External
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 22533
Cc: Marius Bakke <mbakke@HIDDEN>, 22533 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.0 (/)


G=C3=A1bor Boskovits <boskovits@HIDDEN> writes:

> 2018-03-05 16:36 GMT+01:00 G=C3=A1bor Boskovits <boskovits@HIDDEN>:
>
>> 2018-03-05 1:05 GMT+01:00 Ricardo Wurmus <rekado@HIDDEN>:
>>
>>>
>>> Ricardo Wurmus <rekado@HIDDEN> writes:
>>>
>>> > Unfortunately, this doesn=E2=80=99t fix all reproducibility problems =
with numpy:
>>> >
>>> > --8<---------------cut here---------------start------------->8---
>>> > Binary files /gnu/store/kd06ql8fynlydymzhhn
>>> wk2lh0778dwcc-python-numpy-1.14.0-check/lib/python3.6/site-
>>> packages/numpy/distutils/__pycache__/__config__.cpython-36.pyc and
>>> /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.
>>> 14.0/lib/python3.6/site-packages/numpy/distutils/__
>>> pycache__/__config__.cpython-36.pyc differ
>>> > Binary files /gnu/store/kd06ql8fynlydymzhhn
>>> wk2lh0778dwcc-python-numpy-1.14.0-check/lib/python3.6/site-
>>> packages/numpy/distutils/__pycache__/exec_command.cpython-36.pyc and
>>> /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.
>>> 14.0/lib/python3.6/site-packages/numpy/distutils/__
>>> pycache__/exec_command.cpython-36.pyc differ
>>> > Binary files /gnu/store/kd06ql8fynlydymzhhn
>>> wk2lh0778dwcc-python-numpy-1.14.0-check/lib/python3.6/site-
>>> packages/numpy/distutils/__pycache__/system_info.cpython-36.pyc and
>>> /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.
>>> 14.0/lib/python3.6/site-packages/numpy/distutils/__
>>> pycache__/system_info.cpython-36.pyc differ
>>> > Binary files /gnu/store/kd06ql8fynlydymzhhn
>>> wk2lh0778dwcc-python-numpy-1.14.0-check/lib/python3.6/site-
>>> packages/numpy/__pycache__/__config__.cpython-36.pyc and
>>> /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.
>>> 14.0/lib/python3.6/site-packages/numpy/__pycache__/__config__.cpython-3=
6.pyc
>>> differ
>>> > Binary files /gnu/store/kd06ql8fynlydymzhhn
>>> wk2lh0778dwcc-python-numpy-1.14.0-check/lib/python3.6/site-
>>> packages/numpy/__pycache__/version.cpython-36.pyc and
>>> /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.
>>> 14.0/lib/python3.6/site-packages/numpy/__pycache__/version.cpython-36.p=
yc
>>> differ
>>> > Binary files /gnu/store/kd06ql8fynlydymzhhn
>>> wk2lh0778dwcc-python-numpy-1.14.0-check/lib/python3.6/site-
>>> packages/numpy/testing/nose_tools/__pycache__/utils.cpython-36.pyc and
>>> /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.
>>> 14.0/lib/python3.6/site-packages/numpy/testing/nose_
>>> tools/__pycache__/utils.cpython-36.pyc differ
>>> > --8<---------------cut here---------------end--------------->8---
>>>
>>> Here=E2=80=99s what diffoscope says:
>>>
>>> --8<---------------cut here---------------start------------->8---
>>> diffoscope /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.
>>> 14.0{-check,}/lib/python3.6/site-packages/numpy/__pycache_
>>> _/version.cpython-36.pyc
>>> --- /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.
>>> 14.0-check/lib/python3.6/site-packages/numpy/__pycache__/
>>> version.cpython-36.pyc
>>> +++ /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.
>>> 14.0/lib/python3.6/site-packages/numpy/__pycache__/version.cpython-36.p=
yc
>>> @@ -1,8 +1,8 @@
>>> -00000000: 330d 0d0a fa87 9c5a 2601 0000 e300 0000  3......Z&.......
>>> +00000000: 330d 0d0a c485 9c5a 2601 0000 e300 0000  3......Z&.......
>>>  00000010: 0000 0000 0000 0000 0001 0000 0040 0000  .............@..
>>>  00000020: 0073 2000 0000 6400 5a00 6400 5a01 6400  .s ...d.Z.d.Z.d.
>>>  00000030: 5a02 6401 5a03 6402 5a04 6504 731c 6502  Z.d.Z.d.Z.e.s.e.
>>>  00000040: 5a01 6403 5300 2904 7a06 312e 3134 2e30  Z.d.S.).z.1.14.0
>>>  00000050: da28 3639 3134 6262 3431 6630 6662 3363  .(6914bb41f0fb3c
>>>  00000060: 3162 6135 3030 6261 6534 6537 6436 3731  1ba500bae4e7d671
>>>  00000070: 6461 3935 3336 3738 3666 544e 2905 da0d  da9536786fTN)...
>>> --8<---------------cut here---------------end--------------->8---
>>>
>>> In other words: this is the timestamp field of the pyc file.
>>>
>>> Maybe this can be avoided by setting DETERMINISTIC_BUILD in the
>>> python-build-system?
>>>
>>>
>> It seems that the deterministic build patch already landed upstream
>> https://github.com/python/cpython/pull/5200, so we might consider
>> applying the upstream patches. WDYT?
>>
>
> And also this: https://github.com/python/cpython/pull/4575.
> I'm now having a look at this approach. However this second one
> seems quite invasive...

These patches are for what will become Python 3.7.  Python 3.6 does not
have support for =E2=80=9Cinvalidation_mode=E2=80=9D, so at least the first=
 patch would
not work for us.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net






Information forwarded to bug-guix@HIDDEN:
bug#22533; Package guix. Full text available.

Message received at 22533 <at> debbugs.gnu.org:


Received: (at 22533) by debbugs.gnu.org; 5 Mar 2018 20:33:10 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Mar 05 15:33:10 2018
Received: from localhost ([127.0.0.1]:46274 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1eswnG-0004tN-Bs
	for submit <at> debbugs.gnu.org; Mon, 05 Mar 2018 15:33:10 -0500
Received: from mail-it0-f48.google.com ([209.85.214.48]:51868)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <boskovits@HIDDEN>) id 1eswnE-0004tB-Oa
 for 22533 <at> debbugs.gnu.org; Mon, 05 Mar 2018 15:33:09 -0500
Received: by mail-it0-f48.google.com with SMTP id u66so11000435ith.1
 for <22533 <at> debbugs.gnu.org>; Mon, 05 Mar 2018 12:33:08 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=mime-version:in-reply-to:references:from:date:message-id:subject:to
 :cc; bh=kGMBdOxSMOgl1QRj6NBLebjmjr4kq6K0XUYrt7eCyy4=;
 b=Q9QYVe0tCl7sapJdS78aIgj9KQPXBXvtLEZdSPXBQjBQ0heUHfjphQ0TARyQr8Q6UI
 PTI+c5qUa/eob20XpTJ/ds+gI2IzuyUqWdobiJYHD32s9y0wQB/V7dmG2rgqW2W7WDi2
 jt0CuEvsPY4FnbUu9aTubSAfHr0Zqn2f/VooHj8kVvOVD0MC+M2YFl9bgTdfRBe3Y/vj
 ZztAh507ZaQhLGNUCjUGFMOSZrZt2AtBhClYRtebgWq2coO4oiZ4XLE/TrrimfRemCzu
 zEibm7dSIVsn3+jZriMGaj00HkhY9dXvO56I1lk3P2akc4JIkF9zYTtutAnmT5AFIJUE
 K7KQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:mime-version:in-reply-to:references:from:date
 :message-id:subject:to:cc;
 bh=kGMBdOxSMOgl1QRj6NBLebjmjr4kq6K0XUYrt7eCyy4=;
 b=WHIF1ssqMSCSLAAUq1dkyGnFlaw4EhKKq+NXuDHT9QpbkV+5oRH81HJUyfWLc9CJJO
 93CNQxhhxP+XOMuCIMhW28cFpGZSXuy+nSfGusEtNtp6GaocrJs3c/VQpANLfviuElrq
 A8c08HZEhKYcvARdVu2uuZaVtvqRwEz5kQ/0ut1jRIyMnqiFIfWUCcoirFQAt5SRSa2t
 LOYDnr0VhL5QFiId/AHHhyiG9J5//nJ21l4T7VkoZtMCe9lLHPEDlCl7/JH1XtZX/ss4
 6HOQzTHlCRQiZMgLj/KllSItSIjGDwN0ZnI2ozhibZ+wv0hx5sH5nUiqMT8AGVuk5JNq
 gzVA==
X-Gm-Message-State: AElRT7EeWF6tPJHS2F4Ctmzlur0OjgqAIMwyJRTNB0o+FLHAMlz96279
 28lghXZndk3Q/rurtoG8CfKzEnvz5mB7Zk6yzQ==
X-Google-Smtp-Source: AG47ELtv5f9DaB5Ul7DIaECVs4kdR7HGbheHwMTP8jg4w9bU1QuJwUyR0SpLe6exXvi+Zjp9evPXRhsRrc+znPy4cXA=
X-Received: by 10.36.2.200 with SMTP id 191mr7590270itu.108.1520281982894;
 Mon, 05 Mar 2018 12:33:02 -0800 (PST)
MIME-Version: 1.0
Received: by 10.79.157.15 with HTTP; Mon, 5 Mar 2018 12:33:02 -0800 (PST)
In-Reply-To: <CAE4v=pjbSGnkGvB5emBMYNXkwy+2CbE2bVAfE5Ph_DUqV4ezug@HIDDEN>
References: <20160202051544.GA11744@jasmine> <87bmqfu44s.fsf@HIDDEN>
 <87606c23bq.fsf@HIDDEN>
 <CAE4v=phoeRvKHG96Y1-_A2TRAK_wNPUyBgMFgq6+Cd9enFScUg@HIDDEN>
 <874llw101c.fsf@HIDDEN> <871sgz1wg0.fsf@HIDDEN>
 <87y3j7z8xk.fsf@HIDDEN> <87woyrz8sv.fsf@HIDDEN>
 <CAE4v=pjbSGnkGvB5emBMYNXkwy+2CbE2bVAfE5Ph_DUqV4ezug@HIDDEN>
From: =?UTF-8?Q?G=C3=A1bor_Boskovits?= <boskovits@HIDDEN>
Date: Mon, 5 Mar 2018 21:33:02 +0100
Message-ID: <CAE4v=piEhWjz9Jei-WFgYke20QgQWvPks6rcoizagpjida-RsQ@HIDDEN>
Subject: Re: bug#22533: Python bytecode reproducibility
To: Ricardo Wurmus <rekado@HIDDEN>
Content-Type: multipart/alternative; boundary="001a11446d988c8c0f0566b03bdf"
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 22533
Cc: Marius Bakke <mbakke@HIDDEN>, 22533 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.0 (/)

--001a11446d988c8c0f0566b03bdf
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

2018-03-05 16:36 GMT+01:00 G=C3=A1bor Boskovits <boskovits@HIDDEN>:

> 2018-03-05 1:05 GMT+01:00 Ricardo Wurmus <rekado@HIDDEN>:
>
>>
>> Ricardo Wurmus <rekado@HIDDEN> writes:
>>
>> > Unfortunately, this doesn=E2=80=99t fix all reproducibility problems w=
ith numpy:
>> >
>> > --8<---------------cut here---------------start------------->8---
>> > Binary files /gnu/store/kd06ql8fynlydymzhhn
>> wk2lh0778dwcc-python-numpy-1.14.0-check/lib/python3.6/site-
>> packages/numpy/distutils/__pycache__/__config__.cpython-36.pyc and
>> /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.
>> 14.0/lib/python3.6/site-packages/numpy/distutils/__
>> pycache__/__config__.cpython-36.pyc differ
>> > Binary files /gnu/store/kd06ql8fynlydymzhhn
>> wk2lh0778dwcc-python-numpy-1.14.0-check/lib/python3.6/site-
>> packages/numpy/distutils/__pycache__/exec_command.cpython-36.pyc and
>> /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.
>> 14.0/lib/python3.6/site-packages/numpy/distutils/__
>> pycache__/exec_command.cpython-36.pyc differ
>> > Binary files /gnu/store/kd06ql8fynlydymzhhn
>> wk2lh0778dwcc-python-numpy-1.14.0-check/lib/python3.6/site-
>> packages/numpy/distutils/__pycache__/system_info.cpython-36.pyc and
>> /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.
>> 14.0/lib/python3.6/site-packages/numpy/distutils/__
>> pycache__/system_info.cpython-36.pyc differ
>> > Binary files /gnu/store/kd06ql8fynlydymzhhn
>> wk2lh0778dwcc-python-numpy-1.14.0-check/lib/python3.6/site-
>> packages/numpy/__pycache__/__config__.cpython-36.pyc and
>> /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.
>> 14.0/lib/python3.6/site-packages/numpy/__pycache__/__config__.cpython-36=
.pyc
>> differ
>> > Binary files /gnu/store/kd06ql8fynlydymzhhn
>> wk2lh0778dwcc-python-numpy-1.14.0-check/lib/python3.6/site-
>> packages/numpy/__pycache__/version.cpython-36.pyc and
>> /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.
>> 14.0/lib/python3.6/site-packages/numpy/__pycache__/version.cpython-36.py=
c
>> differ
>> > Binary files /gnu/store/kd06ql8fynlydymzhhn
>> wk2lh0778dwcc-python-numpy-1.14.0-check/lib/python3.6/site-
>> packages/numpy/testing/nose_tools/__pycache__/utils.cpython-36.pyc and
>> /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.
>> 14.0/lib/python3.6/site-packages/numpy/testing/nose_
>> tools/__pycache__/utils.cpython-36.pyc differ
>> > --8<---------------cut here---------------end--------------->8---
>>
>> Here=E2=80=99s what diffoscope says:
>>
>> --8<---------------cut here---------------start------------->8---
>> diffoscope /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.
>> 14.0{-check,}/lib/python3.6/site-packages/numpy/__pycache_
>> _/version.cpython-36.pyc
>> --- /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.
>> 14.0-check/lib/python3.6/site-packages/numpy/__pycache__/
>> version.cpython-36.pyc
>> +++ /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.
>> 14.0/lib/python3.6/site-packages/numpy/__pycache__/version.cpython-36.py=
c
>> @@ -1,8 +1,8 @@
>> -00000000: 330d 0d0a fa87 9c5a 2601 0000 e300 0000  3......Z&.......
>> +00000000: 330d 0d0a c485 9c5a 2601 0000 e300 0000  3......Z&.......
>>  00000010: 0000 0000 0000 0000 0001 0000 0040 0000  .............@..
>>  00000020: 0073 2000 0000 6400 5a00 6400 5a01 6400  .s ...d.Z.d.Z.d.
>>  00000030: 5a02 6401 5a03 6402 5a04 6504 731c 6502  Z.d.Z.d.Z.e.s.e.
>>  00000040: 5a01 6403 5300 2904 7a06 312e 3134 2e30  Z.d.S.).z.1.14.0
>>  00000050: da28 3639 3134 6262 3431 6630 6662 3363  .(6914bb41f0fb3c
>>  00000060: 3162 6135 3030 6261 6534 6537 6436 3731  1ba500bae4e7d671
>>  00000070: 6461 3935 3336 3738 3666 544e 2905 da0d  da9536786fTN)...
>> --8<---------------cut here---------------end--------------->8---
>>
>> In other words: this is the timestamp field of the pyc file.
>>
>> Maybe this can be avoided by setting DETERMINISTIC_BUILD in the
>> python-build-system?
>>
>>
> It seems that the deterministic build patch already landed upstream
> https://github.com/python/cpython/pull/5200, so we might consider
> applying the upstream patches. WDYT?
>

And also this: https://github.com/python/cpython/pull/4575.
I'm now having a look at this approach. However this second one
seems quite invasive...


>
>
>> --
>> Ricardo
>>
>> GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
>> https://elephly.net
>>
>>
>>
>

--001a11446d988c8c0f0566b03bdf
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div class=3D"gmail_extra"><div class=3D"gmail_quote">2018=
-03-05 16:36 GMT+01:00 G=C3=A1bor Boskovits <span dir=3D"ltr">&lt;<a href=
=3D"mailto:boskovits@HIDDEN" target=3D"_blank">boskovits@HIDDEN</a>&g=
t;</span>:<br><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px=
 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir=3D=
"ltr"><div class=3D"gmail_extra"><div class=3D"gmail_quote"><div><div class=
=3D"gmail-h5">2018-03-05 1:05 GMT+01:00 Ricardo Wurmus <span dir=3D"ltr">&l=
t;<a href=3D"mailto:rekado@HIDDEN" target=3D"_blank">rekado@HIDDEN=
t</a>&gt;</span>:<br><blockquote class=3D"gmail_quote" style=3D"margin:0px =
0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><spa=
n class=3D"gmail-m_5904028528801827314gmail-"><br>
Ricardo Wurmus &lt;<a href=3D"mailto:rekado@HIDDEN" target=3D"_blank">=
rekado@HIDDEN</a>&gt; writes:<br>
<br>
&gt; Unfortunately, this doesn=E2=80=99t fix all reproducibility problems w=
ith numpy:<br>
&gt;<br>
&gt; --8&lt;---------------cut here---------------start------<wbr>-------&g=
t;8---<br>
&gt; Binary files /gnu/store/kd06ql8fynlydymzhhn<wbr>wk2lh0778dwcc-python-n=
umpy-1.<wbr>14.0-check/lib/python3.6/site-<wbr>packages/numpy/distutils/__<=
wbr>pycache__/__config__.cpython-<wbr>36.pyc and /gnu/store/kd06ql8fynlydym=
zhhn<wbr>wk2lh0778dwcc-python-numpy-1.<wbr>14.0/lib/python3.6/site-<wbr>pac=
kages/numpy/distutils/__<wbr>pycache__/__config__.cpython-<wbr>36.pyc diffe=
r<br>
&gt; Binary files /gnu/store/kd06ql8fynlydymzhhn<wbr>wk2lh0778dwcc-python-n=
umpy-1.<wbr>14.0-check/lib/python3.6/site-<wbr>packages/numpy/distutils/__<=
wbr>pycache__/exec_command.<wbr>cpython-36.pyc and /gnu/store/kd06ql8fynlyd=
ymzhhn<wbr>wk2lh0778dwcc-python-numpy-1.<wbr>14.0/lib/python3.6/site-<wbr>p=
ackages/numpy/distutils/__<wbr>pycache__/exec_command.<wbr>cpython-36.pyc d=
iffer<br>
&gt; Binary files /gnu/store/kd06ql8fynlydymzhhn<wbr>wk2lh0778dwcc-python-n=
umpy-1.<wbr>14.0-check/lib/python3.6/site-<wbr>packages/numpy/distutils/__<=
wbr>pycache__/system_info.cpython-<wbr>36.pyc and /gnu/store/kd06ql8fynlydy=
mzhhn<wbr>wk2lh0778dwcc-python-numpy-1.<wbr>14.0/lib/python3.6/site-<wbr>pa=
ckages/numpy/distutils/__<wbr>pycache__/system_info.cpython-<wbr>36.pyc dif=
fer<br>
&gt; Binary files /gnu/store/kd06ql8fynlydymzhhn<wbr>wk2lh0778dwcc-python-n=
umpy-1.<wbr>14.0-check/lib/python3.6/site-<wbr>packages/numpy/__pycache__/_=
_<wbr>config__.cpython-36.pyc and /gnu/store/kd06ql8fynlydymzhhn<wbr>wk2lh0=
778dwcc-python-numpy-1.<wbr>14.0/lib/python3.6/site-<wbr>packages/numpy/__p=
ycache__/__<wbr>config__.cpython-36.pyc differ<br>
&gt; Binary files /gnu/store/kd06ql8fynlydymzhhn<wbr>wk2lh0778dwcc-python-n=
umpy-1.<wbr>14.0-check/lib/python3.6/site-<wbr>packages/numpy/__pycache__/<=
wbr>version.cpython-36.pyc and /gnu/store/kd06ql8fynlydymzhhn<wbr>wk2lh0778=
dwcc-python-numpy-1.<wbr>14.0/lib/python3.6/site-<wbr>packages/numpy/__pyca=
che__/<wbr>version.cpython-36.pyc differ<br>
&gt; Binary files /gnu/store/kd06ql8fynlydymzhhn<wbr>wk2lh0778dwcc-python-n=
umpy-1.<wbr>14.0-check/lib/python3.6/site-<wbr>packages/numpy/testing/nose_=
<wbr>tools/__pycache__/utils.<wbr>cpython-36.pyc and /gnu/store/kd06ql8fynl=
ydymzhhn<wbr>wk2lh0778dwcc-python-numpy-1.<wbr>14.0/lib/python3.6/site-<wbr=
>packages/numpy/testing/nose_<wbr>tools/__pycache__/utils.<wbr>cpython-36.p=
yc differ<br>
&gt; --8&lt;---------------cut here---------------end--------<wbr>-------&g=
t;8---<br>
<br>
</span>Here=E2=80=99s what diffoscope says:<br>
<br>
--8&lt;---------------cut here---------------start------<wbr>-------&gt;8--=
-<br>
diffoscope /gnu/store/kd06ql8fynlydymzhhn<wbr>wk2lh0778dwcc-python-numpy-1.=
<wbr>14.0{-check,}/lib/python3.6/<wbr>site-packages/numpy/__pycache_<wbr>_/=
version.cpython-36.pyc<br>
--- /gnu/store/kd06ql8fynlydymzhhn<wbr>wk2lh0778dwcc-python-numpy-1.<wbr>14=
.0-check/lib/python3.6/site-<wbr>packages/numpy/__pycache__/<wbr>version.cp=
ython-36.pyc<br>
+++ /gnu/store/kd06ql8fynlydymzhhn<wbr>wk2lh0778dwcc-python-numpy-1.<wbr>14=
.0/lib/python3.6/site-<wbr>packages/numpy/__pycache__/<wbr>version.cpython-=
36.pyc<br>
@@ -1,8 +1,8 @@<br>
-00000000: 330d 0d0a fa87 9c5a 2601 0000 e300 0000=C2=A0 3......Z&amp;.....=
..<br>
+00000000: 330d 0d0a c485 9c5a 2601 0000 e300 0000=C2=A0 3......Z&amp;.....=
..<br>
=C2=A000000010: 0000 0000 0000 0000 0001 0000 0040 0000=C2=A0 .............=
@..<br>
=C2=A000000020: 0073 2000 0000 6400 5a00 6400 5a01 6400=C2=A0 .s ...d.Z.d.Z=
.d.<br>
=C2=A000000030: 5a02 6401 5a03 6402 5a04 6504 731c 6502=C2=A0 Z.d.Z.d.Z.e.s=
.e.<br>
=C2=A000000040: 5a01 6403 5300 2904 7a06 312e 3134 2e30=C2=A0 Z.d.S.).z.1.1=
4.0<br>
=C2=A000000050: da28 3639 3134 6262 3431 6630 6662 3363=C2=A0 .(6914bb41f0f=
b3c<br>
=C2=A000000060: 3162 6135 3030 6261 6534 6537 6436 3731=C2=A0 1ba500bae4e7d=
671<br>
=C2=A000000070: 6461 3935 3336 3738 3666 544e 2905 da0d=C2=A0 da9536786fTN)=
...<br>
--8&lt;---------------cut here---------------end--------<wbr>-------&gt;8--=
-<br>
<br>
In other words: this is the timestamp field of the pyc file.<br>
<br>
Maybe this can be avoided by setting DETERMINISTIC_BUILD in the<br>
python-build-system?<br>
<div class=3D"gmail-m_5904028528801827314gmail-HOEnZb"><div class=3D"gmail-=
m_5904028528801827314gmail-h5"><br></div></div></blockquote><div><br></div>=
</div></div><div>It seems that the deterministic build patch already landed=
 upstream</div><div><a href=3D"https://github.com/python/cpython/pull/5200"=
 target=3D"_blank">https://github.com/python/<wbr>cpython/pull/5200</a>, so=
 we might consider<br></div><div>applying the upstream patches. WDYT?</div>=
<span class=3D"gmail-"><div></div></span></div></div></div></blockquote><di=
v><br></div><div>And also this:=C2=A0<a href=3D"https://github.com/python/c=
python/pull/4575">https://github.com/python/cpython/pull/4575</a>.</div><di=
v>I&#39;m now having a look at this approach. However this second one</div>=
<div>seems quite invasive...</div><div>=C2=A0</div><blockquote class=3D"gma=
il_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,2=
04,204);padding-left:1ex"><div dir=3D"ltr"><div class=3D"gmail_extra"><div =
class=3D"gmail_quote"><span class=3D"gmail-"><div>=C2=A0</div><blockquote c=
lass=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px soli=
d rgb(204,204,204);padding-left:1ex"><div class=3D"gmail-m_5904028528801827=
314gmail-HOEnZb"><div class=3D"gmail-m_5904028528801827314gmail-h5">
--<br>
Ricardo<br>
<br>
GPG: BCA6 89B6 3655 3801 C3C6=C2=A0 2150 197A 5888 235F ACAC<br>
<a href=3D"https://elephly.net" rel=3D"noreferrer" target=3D"_blank">https:=
//elephly.net</a><br>
<br>
<br>
</div></div></blockquote></span></div><br></div></div>
</blockquote></div><br></div></div>

--001a11446d988c8c0f0566b03bdf--




Information forwarded to bug-guix@HIDDEN:
bug#22533; Package guix. Full text available.

Message received at 22533 <at> debbugs.gnu.org:


Received: (at 22533) by debbugs.gnu.org; 5 Mar 2018 15:36:39 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Mar 05 10:36:39 2018
Received: from localhost ([127.0.0.1]:46097 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1essAJ-0002CO-4S
	for submit <at> debbugs.gnu.org; Mon, 05 Mar 2018 10:36:39 -0500
Received: from mail-it0-f48.google.com ([209.85.214.48]:56037)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <boskovits@HIDDEN>) id 1essAH-0002CC-Dx
 for 22533 <at> debbugs.gnu.org; Mon, 05 Mar 2018 10:36:37 -0500
Received: by mail-it0-f48.google.com with SMTP id n7so9855370ita.5
 for <22533 <at> debbugs.gnu.org>; Mon, 05 Mar 2018 07:36:37 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=mime-version:in-reply-to:references:from:date:message-id:subject:to
 :cc; bh=d5VLzDe3uM/OmHCvMJ956GZEhIsgT4wg7Hpf19yyIUE=;
 b=SWbWkyzUX6vQLYt97HzOM2BORD4LZ/TdIb+nISkqCx97pNnKrq3IqhmoHtWdIf0PJD
 C9Il4SlPbYOxa0uUldeSlYMG5QHG96QYUVpqmxCHloMPLSRHlvu4hSmM30IanrM/hMqT
 oJJtp81HcebiYm5NWStm6iJkYUQjLLlQbqxbC6MOvMRnUut4u5Ow7+IlOx+CNUG2P6VZ
 XPw0VJ+bPzU+OWxd2WyKSMgOSYisJPEcniFTNusiic/bUFc7Qx+i+YZvMusORgmPJxVY
 Cpnoe4CdMTUoFe/qqxaNJ5Ul5Vd+eqbCi4tK1ktGP/PfnklXc6hcSMY+qtgYqHiIIxTS
 6g+g==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:mime-version:in-reply-to:references:from:date
 :message-id:subject:to:cc;
 bh=d5VLzDe3uM/OmHCvMJ956GZEhIsgT4wg7Hpf19yyIUE=;
 b=mLmpQRaAWAiVgYEB+4vPQZ/PFphmq4XRdHHvdlMTyI4WMQt3Kn3BIj0sEEYsQNq1ko
 RNtYlaRudoxD38yOAWb2/FHKx3EV09QM52ornt/vv9jMQgoyh9GGlLuCqgxjYoLT+uTm
 rupZdCkhqLF3323De/D4gOvc5HKWTvhNzysymMOmCD+mnpALGlFRTIB65OoCxPyQAV32
 eFzVdnIlv6nco9CXfM7RoWTxYHSQi5GPb/+POgq6nXiX/yUmr4hwLzDCYE978YWPFHib
 Xkh6qxeYE7IQGJXdCzDVXkVABvkY+GdPzROmdKJvQQu7hqSKepT6GVnso0Vf2s+Dtbls
 3kLQ==
X-Gm-Message-State: AElRT7GSv0qS9xdiMmWdPm6ehtqbZCNVwmH1QxkLdf+EJd+AUKrPZcsv
 6+vQKb9mcoibLLZkg9v/pLTBjYHZMrKSrePNpw==
X-Google-Smtp-Source: AG47ELtuw/rfXX37ClbgxqTFx7FXHNdTAb1aujP0JNmK7nfsfiIGcpvrJx5Sy4Fe2c6Rd7sSCM/cfmiXF6GfFcd4M7o=
X-Received: by 10.36.118.211 with SMTP id z202mr14819343itb.104.1520264191744; 
 Mon, 05 Mar 2018 07:36:31 -0800 (PST)
MIME-Version: 1.0
Received: by 10.79.157.15 with HTTP; Mon, 5 Mar 2018 07:36:31 -0800 (PST)
In-Reply-To: <87woyrz8sv.fsf@HIDDEN>
References: <20160202051544.GA11744@jasmine> <87bmqfu44s.fsf@HIDDEN>
 <87606c23bq.fsf@HIDDEN>
 <CAE4v=phoeRvKHG96Y1-_A2TRAK_wNPUyBgMFgq6+Cd9enFScUg@HIDDEN>
 <874llw101c.fsf@HIDDEN> <871sgz1wg0.fsf@HIDDEN>
 <87y3j7z8xk.fsf@HIDDEN> <87woyrz8sv.fsf@HIDDEN>
From: =?UTF-8?Q?G=C3=A1bor_Boskovits?= <boskovits@HIDDEN>
Date: Mon, 5 Mar 2018 16:36:31 +0100
Message-ID: <CAE4v=pjbSGnkGvB5emBMYNXkwy+2CbE2bVAfE5Ph_DUqV4ezug@HIDDEN>
Subject: Re: bug#22533: Python bytecode reproducibility
To: Ricardo Wurmus <rekado@HIDDEN>
Content-Type: multipart/alternative; boundary="001a114412361d206f0566ac1712"
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 22533
Cc: Marius Bakke <mbakke@HIDDEN>, 22533 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.0 (/)

--001a114412361d206f0566ac1712
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

2018-03-05 1:05 GMT+01:00 Ricardo Wurmus <rekado@HIDDEN>:

>
> Ricardo Wurmus <rekado@HIDDEN> writes:
>
> > Unfortunately, this doesn=E2=80=99t fix all reproducibility problems wi=
th numpy:
> >
> > --8<---------------cut here---------------start------------->8---
> > Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dw
> cc-python-numpy-1.14.0-check/lib/python3.6/site-packages/
> numpy/distutils/__pycache__/__config__.cpython-36.pyc and /gnu/store/
> kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0/lib/
> python3.6/site-packages/numpy/distutils/__pycache__/__config__.cpython-36=
.pyc
> differ
> > Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dw
> cc-python-numpy-1.14.0-check/lib/python3.6/site-packages/
> numpy/distutils/__pycache__/exec_command.cpython-36.pyc and /gnu/store/
> kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0/lib/
> python3.6/site-packages/numpy/distutils/__pycache__/exec_command.cpython-=
36.pyc
> differ
> > Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dw
> cc-python-numpy-1.14.0-check/lib/python3.6/site-packages/
> numpy/distutils/__pycache__/system_info.cpython-36.pyc and /gnu/store/
> kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0/lib/
> python3.6/site-packages/numpy/distutils/__pycache__/system_info.cpython-3=
6.pyc
> differ
> > Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dw
> cc-python-numpy-1.14.0-check/lib/python3.6/site-packages/
> numpy/__pycache__/__config__.cpython-36.pyc and /gnu/store/
> kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0/lib/
> python3.6/site-packages/numpy/__pycache__/__config__.cpython-36.pyc diffe=
r
> > Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dw
> cc-python-numpy-1.14.0-check/lib/python3.6/site-packages/
> numpy/__pycache__/version.cpython-36.pyc and /gnu/store/
> kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0/lib/
> python3.6/site-packages/numpy/__pycache__/version.cpython-36.pyc differ
> > Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dw
> cc-python-numpy-1.14.0-check/lib/python3.6/site-packages/
> numpy/testing/nose_tools/__pycache__/utils.cpython-36.pyc and /gnu/store/
> kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0/lib/
> python3.6/site-packages/numpy/testing/nose_tools/__pycache__/utils.cpytho=
n-36.pyc
> differ
> > --8<---------------cut here---------------end--------------->8---
>
> Here=E2=80=99s what diffoscope says:
>
> --8<---------------cut here---------------start------------->8---
> diffoscope /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dw
> cc-python-numpy-1.14.0{-check,}/lib/python3.6/site-packages/
> numpy/__pycache__/version.cpython-36.pyc
> --- /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0-check=
/
> lib/python3.6/site-packages/numpy/__pycache__/version.cpython-36.pyc
> +++ /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0/lib/
> python3.6/site-packages/numpy/__pycache__/version.cpython-36.pyc
> @@ -1,8 +1,8 @@
> -00000000: 330d 0d0a fa87 9c5a 2601 0000 e300 0000  3......Z&.......
> +00000000: 330d 0d0a c485 9c5a 2601 0000 e300 0000  3......Z&.......
>  00000010: 0000 0000 0000 0000 0001 0000 0040 0000  .............@..
>  00000020: 0073 2000 0000 6400 5a00 6400 5a01 6400  .s ...d.Z.d.Z.d.
>  00000030: 5a02 6401 5a03 6402 5a04 6504 731c 6502  Z.d.Z.d.Z.e.s.e.
>  00000040: 5a01 6403 5300 2904 7a06 312e 3134 2e30  Z.d.S.).z.1.14.0
>  00000050: da28 3639 3134 6262 3431 6630 6662 3363  .(6914bb41f0fb3c
>  00000060: 3162 6135 3030 6261 6534 6537 6436 3731  1ba500bae4e7d671
>  00000070: 6461 3935 3336 3738 3666 544e 2905 da0d  da9536786fTN)...
> --8<---------------cut here---------------end--------------->8---
>
> In other words: this is the timestamp field of the pyc file.
>
> Maybe this can be avoided by setting DETERMINISTIC_BUILD in the
> python-build-system?
>
>
It seems that the deterministic build patch already landed upstream
https://github.com/python/cpython/pull/5200, so we might consider
applying the upstream patches. WDYT?


> --
> Ricardo
>
> GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
> https://elephly.net
>
>
>

--001a114412361d206f0566ac1712
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div class=3D"gmail_extra"><div class=3D"gmail_quote">2018=
-03-05 1:05 GMT+01:00 Ricardo Wurmus <span dir=3D"ltr">&lt;<a href=3D"mailt=
o:rekado@HIDDEN" target=3D"_blank">rekado@HIDDEN</a>&gt;</span>:<=
br><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;bord=
er-left:1px solid rgb(204,204,204);padding-left:1ex"><span class=3D"gmail-"=
><br>
Ricardo Wurmus &lt;<a href=3D"mailto:rekado@HIDDEN">rekado@HIDDEN=
</a>&gt; writes:<br>
<br>
&gt; Unfortunately, this doesn=E2=80=99t fix all reproducibility problems w=
ith numpy:<br>
&gt;<br>
&gt; --8&lt;---------------cut here---------------start------<wbr>-------&g=
t;8---<br>
&gt; Binary files /gnu/store/<wbr>kd06ql8fynlydymzhhnwk2lh0778dw<wbr>cc-pyt=
hon-numpy-1.14.0-check/<wbr>lib/python3.6/site-packages/<wbr>numpy/distutil=
s/__pycache__/__<wbr>config__.cpython-36.pyc and /gnu/store/<wbr>kd06ql8fyn=
lydymzhhnwk2lh0778dw<wbr>cc-python-numpy-1.14.0/lib/<wbr>python3.6/site-pac=
kages/numpy/<wbr>distutils/__pycache__/__<wbr>config__.cpython-36.pyc diffe=
r<br>
&gt; Binary files /gnu/store/<wbr>kd06ql8fynlydymzhhnwk2lh0778dw<wbr>cc-pyt=
hon-numpy-1.14.0-check/<wbr>lib/python3.6/site-packages/<wbr>numpy/distutil=
s/__pycache__/<wbr>exec_command.cpython-36.pyc and /gnu/store/<wbr>kd06ql8f=
ynlydymzhhnwk2lh0778dw<wbr>cc-python-numpy-1.14.0/lib/<wbr>python3.6/site-p=
ackages/numpy/<wbr>distutils/__pycache__/exec_<wbr>command.cpython-36.pyc d=
iffer<br>
&gt; Binary files /gnu/store/<wbr>kd06ql8fynlydymzhhnwk2lh0778dw<wbr>cc-pyt=
hon-numpy-1.14.0-check/<wbr>lib/python3.6/site-packages/<wbr>numpy/distutil=
s/__pycache__/<wbr>system_info.cpython-36.pyc and /gnu/store/<wbr>kd06ql8fy=
nlydymzhhnwk2lh0778dw<wbr>cc-python-numpy-1.14.0/lib/<wbr>python3.6/site-pa=
ckages/numpy/<wbr>distutils/__pycache__/system_<wbr>info.cpython-36.pyc dif=
fer<br>
&gt; Binary files /gnu/store/<wbr>kd06ql8fynlydymzhhnwk2lh0778dw<wbr>cc-pyt=
hon-numpy-1.14.0-check/<wbr>lib/python3.6/site-packages/<wbr>numpy/__pycach=
e__/__config__.<wbr>cpython-36.pyc and /gnu/store/<wbr>kd06ql8fynlydymzhhnw=
k2lh0778dw<wbr>cc-python-numpy-1.14.0/lib/<wbr>python3.6/site-packages/nump=
y/<wbr>__pycache__/__config__.<wbr>cpython-36.pyc differ<br>
&gt; Binary files /gnu/store/<wbr>kd06ql8fynlydymzhhnwk2lh0778dw<wbr>cc-pyt=
hon-numpy-1.14.0-check/<wbr>lib/python3.6/site-packages/<wbr>numpy/__pycach=
e__/version.<wbr>cpython-36.pyc and /gnu/store/<wbr>kd06ql8fynlydymzhhnwk2l=
h0778dw<wbr>cc-python-numpy-1.14.0/lib/<wbr>python3.6/site-packages/numpy/<=
wbr>__pycache__/version.cpython-<wbr>36.pyc differ<br>
&gt; Binary files /gnu/store/<wbr>kd06ql8fynlydymzhhnwk2lh0778dw<wbr>cc-pyt=
hon-numpy-1.14.0-check/<wbr>lib/python3.6/site-packages/<wbr>numpy/testing/=
nose_tools/__<wbr>pycache__/utils.cpython-36.pyc and /gnu/store/<wbr>kd06ql=
8fynlydymzhhnwk2lh0778dw<wbr>cc-python-numpy-1.14.0/lib/<wbr>python3.6/site=
-packages/numpy/<wbr>testing/nose_tools/__pycache__<wbr>/utils.cpython-36.p=
yc differ<br>
&gt; --8&lt;---------------cut here---------------end--------<wbr>-------&g=
t;8---<br>
<br>
</span>Here=E2=80=99s what diffoscope says:<br>
<br>
--8&lt;---------------cut here---------------start------<wbr>-------&gt;8--=
-<br>
diffoscope /gnu/store/<wbr>kd06ql8fynlydymzhhnwk2lh0778dw<wbr>cc-python-num=
py-1.14.0{-check,<wbr>}/lib/python3.6/site-packages/<wbr>numpy/__pycache__/=
version.<wbr>cpython-36.pyc<br>
--- /gnu/store/<wbr>kd06ql8fynlydymzhhnwk2lh0778dw<wbr>cc-python-numpy-1.14=
.0-check/<wbr>lib/python3.6/site-packages/<wbr>numpy/__pycache__/version.<w=
br>cpython-36.pyc<br>
+++ /gnu/store/<wbr>kd06ql8fynlydymzhhnwk2lh0778dw<wbr>cc-python-numpy-1.14=
.0/lib/<wbr>python3.6/site-packages/numpy/<wbr>__pycache__/version.cpython-=
<wbr>36.pyc<br>
@@ -1,8 +1,8 @@<br>
-00000000: 330d 0d0a fa87 9c5a 2601 0000 e300 0000=C2=A0 3......Z&amp;.....=
..<br>
+00000000: 330d 0d0a c485 9c5a 2601 0000 e300 0000=C2=A0 3......Z&amp;.....=
..<br>
=C2=A000000010: 0000 0000 0000 0000 0001 0000 0040 0000=C2=A0 .............=
@..<br>
=C2=A000000020: 0073 2000 0000 6400 5a00 6400 5a01 6400=C2=A0 .s ...d.Z.d.Z=
.d.<br>
=C2=A000000030: 5a02 6401 5a03 6402 5a04 6504 731c 6502=C2=A0 Z.d.Z.d.Z.e.s=
.e.<br>
=C2=A000000040: 5a01 6403 5300 2904 7a06 312e 3134 2e30=C2=A0 Z.d.S.).z.1.1=
4.0<br>
=C2=A000000050: da28 3639 3134 6262 3431 6630 6662 3363=C2=A0 .(6914bb41f0f=
b3c<br>
=C2=A000000060: 3162 6135 3030 6261 6534 6537 6436 3731=C2=A0 1ba500bae4e7d=
671<br>
=C2=A000000070: 6461 3935 3336 3738 3666 544e 2905 da0d=C2=A0 da9536786fTN)=
...<br>
--8&lt;---------------cut here---------------end--------<wbr>-------&gt;8--=
-<br>
<br>
In other words: this is the timestamp field of the pyc file.<br>
<br>
Maybe this can be avoided by setting DETERMINISTIC_BUILD in the<br>
python-build-system?<br>
<div class=3D"gmail-HOEnZb"><div class=3D"gmail-h5"><br></div></div></block=
quote><div><br></div><div>It seems that the deterministic build patch alrea=
dy landed upstream</div><div><a href=3D"https://github.com/python/cpython/p=
ull/5200">https://github.com/python/cpython/pull/5200</a>, so we might cons=
ider<br></div><div>applying the upstream patches. WDYT?</div><div>=C2=A0</d=
iv><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;bord=
er-left:1px solid rgb(204,204,204);padding-left:1ex"><div class=3D"gmail-HO=
EnZb"><div class=3D"gmail-h5">
--<br>
Ricardo<br>
<br>
GPG: BCA6 89B6 3655 3801 C3C6=C2=A0 2150 197A 5888 235F ACAC<br>
<a href=3D"https://elephly.net" rel=3D"noreferrer" target=3D"_blank">https:=
//elephly.net</a><br>
<br>
<br>
</div></div></blockquote></div><br></div></div>

--001a114412361d206f0566ac1712--




Information forwarded to bug-guix@HIDDEN:
bug#22533; Package guix. Full text available.

Message received at 22533 <at> debbugs.gnu.org:


Received: (at 22533) by debbugs.gnu.org; 5 Mar 2018 09:25:43 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Mar 05 04:25:43 2018
Received: from localhost ([127.0.0.1]:45095 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1esmNL-0008DG-5G
	for submit <at> debbugs.gnu.org; Mon, 05 Mar 2018 04:25:43 -0500
Received: from hera.aquilenet.fr ([185.233.100.1]:35672)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1esmNJ-0008D7-T3
 for 22533 <at> debbugs.gnu.org; Mon, 05 Mar 2018 04:25:42 -0500
Received: from localhost (localhost [127.0.0.1])
 by hera.aquilenet.fr (Postfix) with ESMTP id 68AE011DF9;
 Mon,  5 Mar 2018 10:25:41 +0100 (CET)
X-Virus-Scanned: Debian amavisd-new at aquilenet.fr
Received: from hera.aquilenet.fr ([127.0.0.1])
 by localhost (hera.aquilenet.fr [127.0.0.1]) (amavisd-new, port 10024)
 with ESMTP id rcWp7iJK2RVr; Mon,  5 Mar 2018 10:25:40 +0100 (CET)
Received: from ribbon (unknown [193.50.110.134])
 by hera.aquilenet.fr (Postfix) with ESMTPSA id 89CDC10CB1;
 Mon,  5 Mar 2018 10:25:40 +0100 (CET)
From: ludo@HIDDEN (Ludovic =?utf-8?Q?Court=C3=A8s?=)
To: Ricardo Wurmus <rekado@HIDDEN>
Subject: Re: bug#22533: Python bytecode reproducibility
References: <20160202051544.GA11744@jasmine> <87bmqfu44s.fsf@HIDDEN>
 <87606c23bq.fsf@HIDDEN>
X-URL: http://www.fdn.fr/~lcourtes/
X-Revolutionary-Date: 15 =?utf-8?Q?Vent=C3=B4se?= an 226 de la =?utf-8?Q?R?=
 =?utf-8?Q?=C3=A9volution?=
X-PGP-Key-ID: 0x090B11993D9AEBB5
X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc
X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4  0CFB 090B 1199 3D9A EBB5
X-OS: x86_64-pc-linux-gnu
Date: Mon, 05 Mar 2018 10:25:40 +0100
In-Reply-To: <87606c23bq.fsf@HIDDEN> (Ricardo Wurmus's message of "Sat,
 03 Mar 2018 23:37:29 +0100")
Message-ID: <87h8pudgbv.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.3 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: 1.0 (+)
X-Debbugs-Envelope-To: 22533
Cc: Marius Bakke <mbakke@HIDDEN>, 22533 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: 1.0 (+)

Hello!

Ricardo Wurmus <rekado@HIDDEN> skribis:

> Is it a bad idea to override the timestamps in the generated binaries?
> I think that we could avoid the recency check then, which was an
> obstacle to resetting the timestamps of the source files.

I think it=E2=80=99s good if we can fix Python itself to honor SOURCE_DATE_=
EPOCH
for its timestamps, but it=E2=80=99s also OK to patch timestamps in generat=
ed
binaries.

We do that already in gzip headers, with =E2=80=98reset-gzip-timestamp=E2=
=80=99.

Thanks for tackling this!

Ludo=E2=80=99.




Information forwarded to bug-guix@HIDDEN:
bug#22533; Package guix. Full text available.

Message received at 22533 <at> debbugs.gnu.org:


Received: (at 22533) by debbugs.gnu.org; 5 Mar 2018 00:05:12 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Mar 04 19:05:12 2018
Received: from localhost ([127.0.0.1]:44865 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1esdct-0002qA-Lv
	for submit <at> debbugs.gnu.org; Sun, 04 Mar 2018 19:05:11 -0500
Received: from sender-of-o51.zoho.com ([135.84.80.216]:21063)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <rekado@HIDDEN>) id 1esdcs-0002q1-Df
 for 22533 <at> debbugs.gnu.org; Sun, 04 Mar 2018 19:05:10 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1520208306; 
 s=zoho; d=elephly.net; i=rekado@HIDDEN;
 h=References:From:To:Cc:Subject:In-reply-to:Date:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-ID;
 l=3773; bh=kDf+kAvi1x68kUrwTEkVh+TvJ7yDjSovBp2TKCmYxos=;
 b=YsfyrUe0VuqLT6iT3ZoUuXITVlpAuv3jnIKjLK9vWzfwW6UltdaozBM4S2eMsXmA
 1dIkSr/0eIUW1Bah+cOPtF3KIEWe7b5JQyfIADmrZo7e1hUK4lFL8SZmHlfyKnKz2nW
 bj5aOHVIllneMalmEbCtkkyTPgtJnoOhg9QtatSc=
Received: from localhost (port-92-200-95-79.dynamic.qsc.de [92.200.95.79]) by
 mx.zohomail.com with SMTPS id 1520208306760731.7313417155125;
 Sun, 4 Mar 2018 16:05:06 -0800 (PST)
References: <20160202051544.GA11744@jasmine> <87bmqfu44s.fsf@HIDDEN>
 <87606c23bq.fsf@HIDDEN>
 <CAE4v=phoeRvKHG96Y1-_A2TRAK_wNPUyBgMFgq6+Cd9enFScUg@HIDDEN>
 <874llw101c.fsf@HIDDEN> <871sgz1wg0.fsf@HIDDEN>
 <87y3j7z8xk.fsf@HIDDEN>
User-agent: mu4e 1.0; emacs 25.3.1
From: Ricardo Wurmus <rekado@HIDDEN>
To: =?utf-8?Q?G=C3=A1bor?= Boskovits <boskovits@HIDDEN>
Subject: Re: bug#22533: Python bytecode reproducibility
In-reply-to: <87y3j7z8xk.fsf@HIDDEN>
X-URL: https://elephly.net
X-PGP-Key: https://elephly.net/rekado.pubkey
X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
Date: Mon, 05 Mar 2018 01:05:04 +0100
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Message-ID: <87woyrz8sv.fsf@HIDDEN>
X-ZohoMailClient: External
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 22533
Cc: Marius Bakke <mbakke@HIDDEN>, 22533 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.0 (/)


Ricardo Wurmus <rekado@HIDDEN> writes:

> Unfortunately, this doesn=E2=80=99t fix all reproducibility problems with=
 numpy:
>
> --8<---------------cut here---------------start------------->8---
> Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.1=
4.0-check/lib/python3.6/site-packages/numpy/distutils/__pycache__/__config_=
_.cpython-36.pyc and /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-num=
py-1.14.0/lib/python3.6/site-packages/numpy/distutils/__pycache__/__config_=
_.cpython-36.pyc differ
> Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.1=
4.0-check/lib/python3.6/site-packages/numpy/distutils/__pycache__/exec_comm=
and.cpython-36.pyc and /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-n=
umpy-1.14.0/lib/python3.6/site-packages/numpy/distutils/__pycache__/exec_co=
mmand.cpython-36.pyc differ
> Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.1=
4.0-check/lib/python3.6/site-packages/numpy/distutils/__pycache__/system_in=
fo.cpython-36.pyc and /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-nu=
mpy-1.14.0/lib/python3.6/site-packages/numpy/distutils/__pycache__/system_i=
nfo.cpython-36.pyc differ
> Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.1=
4.0-check/lib/python3.6/site-packages/numpy/__pycache__/__config__.cpython-=
36.pyc and /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0/=
lib/python3.6/site-packages/numpy/__pycache__/__config__.cpython-36.pyc dif=
fer
> Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.1=
4.0-check/lib/python3.6/site-packages/numpy/__pycache__/version.cpython-36.=
pyc and /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0/lib=
/python3.6/site-packages/numpy/__pycache__/version.cpython-36.pyc differ
> Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.1=
4.0-check/lib/python3.6/site-packages/numpy/testing/nose_tools/__pycache__/=
utils.cpython-36.pyc and /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python=
-numpy-1.14.0/lib/python3.6/site-packages/numpy/testing/nose_tools/__pycach=
e__/utils.cpython-36.pyc differ
> --8<---------------cut here---------------end--------------->8---

Here=E2=80=99s what diffoscope says:

--8<---------------cut here---------------start------------->8---
diffoscope /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0{=
-check,}/lib/python3.6/site-packages/numpy/__pycache__/version.cpython-36.p=
yc
--- /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0-check/l=
ib/python3.6/site-packages/numpy/__pycache__/version.cpython-36.pyc
+++ /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0/lib/pyt=
hon3.6/site-packages/numpy/__pycache__/version.cpython-36.pyc
@@ -1,8 +1,8 @@
-00000000: 330d 0d0a fa87 9c5a 2601 0000 e300 0000  3......Z&.......
+00000000: 330d 0d0a c485 9c5a 2601 0000 e300 0000  3......Z&.......
 00000010: 0000 0000 0000 0000 0001 0000 0040 0000  .............@..
 00000020: 0073 2000 0000 6400 5a00 6400 5a01 6400  .s ...d.Z.d.Z.d.
 00000030: 5a02 6401 5a03 6402 5a04 6504 731c 6502  Z.d.Z.d.Z.e.s.e.
 00000040: 5a01 6403 5300 2904 7a06 312e 3134 2e30  Z.d.S.).z.1.14.0
 00000050: da28 3639 3134 6262 3431 6630 6662 3363  .(6914bb41f0fb3c
 00000060: 3162 6135 3030 6261 6534 6537 6436 3731  1ba500bae4e7d671
 00000070: 6461 3935 3336 3738 3666 544e 2905 da0d  da9536786fTN)...
--8<---------------cut here---------------end--------------->8---

In other words: this is the timestamp field of the pyc file.

Maybe this can be avoided by setting DETERMINISTIC_BUILD in the
python-build-system?

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net






Information forwarded to bug-guix@HIDDEN:
bug#22533; Package guix. Full text available.

Message received at 22533 <at> debbugs.gnu.org:


Received: (at 22533) by debbugs.gnu.org; 5 Mar 2018 00:02:54 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Mar 04 19:02:54 2018
Received: from localhost ([127.0.0.1]:44860 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1esdag-0002mc-4e
	for submit <at> debbugs.gnu.org; Sun, 04 Mar 2018 19:02:54 -0500
Received: from sender-of-o51.zoho.com ([135.84.80.216]:21094)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <rekado@HIDDEN>) id 1esdae-0002mR-02
 for 22533 <at> debbugs.gnu.org; Sun, 04 Mar 2018 19:02:52 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1520208139; 
 s=zoho; d=elephly.net; i=rekado@HIDDEN;
 h=References:From:To:Cc:Subject:In-reply-to:Date:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-ID;
 l=4597; bh=aCWG4IL3DL3qcLo895frCums7f4YqqqNjRsmPhgPbIk=;
 b=eCRUCw7beM2pVDlTfUgv0z3LfspCTt+d5/i+2kfPDXzGLWpJrGs6C9iaSPD8l7xc
 WJUBIbDMweHFcI97TftNPZMOPL0Akc0LH8pap5u2JDs55QZSFCPzB20wa6vjOmMmlYz
 I3uB6WiWc/EK0FUuYMkNNUQZxdIU2l9i4+FP3Q88=
Received: from localhost (port-92-200-95-79.dynamic.qsc.de [92.200.95.79]) by
 mx.zohomail.com with SMTPS id 1520208138979261.13922832052685;
 Sun, 4 Mar 2018 16:02:18 -0800 (PST)
References: <20160202051544.GA11744@jasmine> <87bmqfu44s.fsf@HIDDEN>
 <87606c23bq.fsf@HIDDEN>
 <CAE4v=phoeRvKHG96Y1-_A2TRAK_wNPUyBgMFgq6+Cd9enFScUg@HIDDEN>
 <874llw101c.fsf@HIDDEN> <871sgz1wg0.fsf@HIDDEN>
User-agent: mu4e 1.0; emacs 25.3.1
From: Ricardo Wurmus <rekado@HIDDEN>
To: =?utf-8?Q?G=C3=A1bor?= Boskovits <boskovits@HIDDEN>
Subject: Re: bug#22533: Python bytecode reproducibility
In-reply-to: <871sgz1wg0.fsf@HIDDEN>
X-URL: https://elephly.net
X-PGP-Key: https://elephly.net/rekado.pubkey
X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
Date: Mon, 05 Mar 2018 01:02:15 +0100
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Message-ID: <87y3j7z8xk.fsf@HIDDEN>
X-ZohoMailClient: External
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 22533
Cc: Marius Bakke <mbakke@HIDDEN>, 22533 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.0 (/)


Ricardo Wurmus <rekado@HIDDEN> writes:

> I have applied this patch locally:
>
> diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
> index 5f701701a..0d1ecc3c6 100644
> --- a/gnu/packages/python.scm
> +++ b/gnu/packages/python.scm
> @@ -359,8 +359,42 @@ data types.")
>                                "Lib/ctypes/test/test_win32.py" ; fails on=
 aarch64
>                                "Lib/test/test_fcntl.py")) ; fails on aarc=
h64
>                    #t))))
> -    (arguments (substitute-keyword-arguments (package-arguments python-2=
)
> -                 ((#:tests? _) #t)))
> +    (arguments
> +     (substitute-keyword-arguments (package-arguments python-2)
> +       ((#:tests? _) #t)
> +       ((#:phases phases)
> +        `(modify-phases ,phases
> +           (add-after 'unpack 'patch-timestamp-for-pyc-files
> +             (lambda _
> +               ;; We set DETERMINISTIC_BUILD to only override the mtime =
when
> +               ;; building with Guix, lest we break auto-compilation in
> +               ;; environments.
> +               (setenv "DETERMINISTIC_BUILD" "1")
> +               (substitute* "Lib/py_compile.py"
> +                 (("source_stats\\['mtime'\\]")
> +                  "(1 if 'DETERMINISTIC_BUILD' in os.environ else source=
_stats['mtime'])"))
> +
> +               ;; Use deterministic hashes for strings, bytes, and datet=
ime
> +               ;; objects.
> +               (setenv "PYTHONHASHSEED" "0")
> +
> +               ;; Reset mtime when validating bytecode header.
> +               (substitute* "Lib/importlib/_bootstrap_external.py"
> +                 (("source_mtime =3D int\\(source_stats\\['mtime'\\]\\)"=
)
> +                  "source_mtime =3D 1"))
> +               #t))
> +           (add-after 'unpack 'disable-timestamp-tests
> +             (lambda _
> +               (substitute* "Lib/test/test_importlib/source/test_file_lo=
ader.py"
> +                 (("test_bad_marshal")
> +                  "disable_test_bad_marshal")
> +                 (("test_no_marshal")
> +                  "disable_test_no_marshal")
> +                 (("test_non_code_marshal")
> +                  "disable_test_non_code_marshal"))
> +               #t))
> +           (add-before 'check 'allow-non-deterministic-compilation
> +             (lambda _ (unsetenv "DETERMINISTIC_BUILD") #t))))))
>      (native-search-paths
>       (list (search-path-specification
>              (variable "PYTHONPATH")
>
> It allows me to build python-six and python-sip reproducibly.  It does
> not fix problems with Python 2, and I haven=E2=80=99t yet tested if it ca=
uses
> any new problems.

I tested importing modules in an ad-hoc environment =E2=80=94 no problems.

Unfortunately, this doesn=E2=80=99t fix all reproducibility problems with n=
umpy:

--8<---------------cut here---------------start------------->8---
Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.=
0-check/lib/python3.6/site-packages/numpy/distutils/__pycache__/__config__.=
cpython-36.pyc and /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy=
-1.14.0/lib/python3.6/site-packages/numpy/distutils/__pycache__/__config__.=
cpython-36.pyc differ
Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.=
0-check/lib/python3.6/site-packages/numpy/distutils/__pycache__/exec_comman=
d.cpython-36.pyc and /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-num=
py-1.14.0/lib/python3.6/site-packages/numpy/distutils/__pycache__/exec_comm=
and.cpython-36.pyc differ
Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.=
0-check/lib/python3.6/site-packages/numpy/distutils/__pycache__/system_info=
.cpython-36.pyc and /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-nump=
y-1.14.0/lib/python3.6/site-packages/numpy/distutils/__pycache__/system_inf=
o.cpython-36.pyc differ
Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.=
0-check/lib/python3.6/site-packages/numpy/__pycache__/__config__.cpython-36=
.pyc and /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0/li=
b/python3.6/site-packages/numpy/__pycache__/__config__.cpython-36.pyc diffe=
r
Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.=
0-check/lib/python3.6/site-packages/numpy/__pycache__/version.cpython-36.py=
c and /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0/lib/p=
ython3.6/site-packages/numpy/__pycache__/version.cpython-36.pyc differ
Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.=
0-check/lib/python3.6/site-packages/numpy/testing/nose_tools/__pycache__/ut=
ils.cpython-36.pyc and /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-n=
umpy-1.14.0/lib/python3.6/site-packages/numpy/testing/nose_tools/__pycache_=
_/utils.cpython-36.pyc differ
--8<---------------cut here---------------end--------------->8---

But the successes with simpler Python packages are promising.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net






Information forwarded to bug-guix@HIDDEN:
bug#22533; Package guix. Full text available.

Message received at 22533 <at> debbugs.gnu.org:


Received: (at 22533) by debbugs.gnu.org; 4 Mar 2018 19:18:58 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Mar 04 14:18:58 2018
Received: from localhost ([127.0.0.1]:44710 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1esZ9u-0004Tc-6A
	for submit <at> debbugs.gnu.org; Sun, 04 Mar 2018 14:18:58 -0500
Received: from sender-of-o51.zoho.com ([135.84.80.216]:21078)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <rekado@HIDDEN>) id 1esZ9s-0004TS-Fs
 for 22533 <at> debbugs.gnu.org; Sun, 04 Mar 2018 14:18:57 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1520191107; 
 s=zoho; d=elephly.net; i=rekado@HIDDEN;
 h=References:From:To:Cc:Subject:In-reply-to:Date:MIME-Version:Content-Type:Message-ID;
 l=2456; bh=H11D0GMfJkaKSkwL+A7ge/nCnoQjs64ZBB1kEmd5inI=;
 b=CW2KPW/uN3M+Mrw0RutnIYqb2HqRT/CGti5D2WS31+zjwr/KU5dp0DfmjY9g44q/
 EBLSsFnx9AoXl6uX2HA3eTgEKlx8+6SPVaeppM48YeRTT8mG/wBzoGqXA0Biw5ce8jg
 Vk8CCQSyk8Xi2l+a/NzOVFC/uyyS4Z343Djsxueg=
Received: from localhost (port-92-200-95-79.dynamic.qsc.de [92.200.95.79]) by
 mx.zohomail.com with SMTPS id 152019110705415.552663867062392;
 Sun, 4 Mar 2018 11:18:27 -0800 (PST)
References: <20160202051544.GA11744@jasmine> <87bmqfu44s.fsf@HIDDEN>
 <87606c23bq.fsf@HIDDEN>
 <CAE4v=phoeRvKHG96Y1-_A2TRAK_wNPUyBgMFgq6+Cd9enFScUg@HIDDEN>
 <874llw101c.fsf@HIDDEN>
User-agent: mu4e 1.0; emacs 25.3.1
From: Ricardo Wurmus <rekado@HIDDEN>
To: =?utf-8?Q?G=C3=A1bor?= Boskovits <boskovits@HIDDEN>
Subject: Re: bug#22533: Python bytecode reproducibility
In-reply-to: <874llw101c.fsf@HIDDEN>
X-URL: https://elephly.net
X-PGP-Key: https://elephly.net/rekado.pubkey
X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
Date: Sun, 04 Mar 2018 20:18:23 +0100
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="=-=-="
Message-ID: <871sgz1wg0.fsf@HIDDEN>
X-ZohoMailClient: External
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 22533
Cc: Marius Bakke <mbakke@HIDDEN>, 22533 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.0 (/)

--=-=-=
Content-Type: text/plain

I have applied this patch locally:


--=-=-=
Content-Type: text/x-patch
Content-Disposition: inline; filename=1.diff

diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index 5f701701a..0d1ecc3c6 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -359,8 +359,42 @@ data types.")
                               "Lib/ctypes/test/test_win32.py" ; fails on aarch64
                               "Lib/test/test_fcntl.py")) ; fails on aarch64
                   #t))))
-    (arguments (substitute-keyword-arguments (package-arguments python-2)
-                 ((#:tests? _) #t)))
+    (arguments
+     (substitute-keyword-arguments (package-arguments python-2)
+       ((#:tests? _) #t)
+       ((#:phases phases)
+        `(modify-phases ,phases
+           (add-after 'unpack 'patch-timestamp-for-pyc-files
+             (lambda _
+               ;; We set DETERMINISTIC_BUILD to only override the mtime when
+               ;; building with Guix, lest we break auto-compilation in
+               ;; environments.
+               (setenv "DETERMINISTIC_BUILD" "1")
+               (substitute* "Lib/py_compile.py"
+                 (("source_stats\\['mtime'\\]")
+                  "(1 if 'DETERMINISTIC_BUILD' in os.environ else source_stats['mtime'])"))
+
+               ;; Use deterministic hashes for strings, bytes, and datetime
+               ;; objects.
+               (setenv "PYTHONHASHSEED" "0")
+
+               ;; Reset mtime when validating bytecode header.
+               (substitute* "Lib/importlib/_bootstrap_external.py"
+                 (("source_mtime = int\\(source_stats\\['mtime'\\]\\)")
+                  "source_mtime = 1"))
+               #t))
+           (add-after 'unpack 'disable-timestamp-tests
+             (lambda _
+               (substitute* "Lib/test/test_importlib/source/test_file_loader.py"
+                 (("test_bad_marshal")
+                  "disable_test_bad_marshal")
+                 (("test_no_marshal")
+                  "disable_test_no_marshal")
+                 (("test_non_code_marshal")
+                  "disable_test_non_code_marshal"))
+               #t))
+           (add-before 'check 'allow-non-deterministic-compilation
+             (lambda _ (unsetenv "DETERMINISTIC_BUILD") #t))))))
     (native-search-paths
      (list (search-path-specification
             (variable "PYTHONPATH")

--=-=-=
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


It allows me to build python-six and python-sip reproducibly.  It does
not fix problems with Python 2, and I haven=E2=80=99t yet tested if it caus=
es
any new problems.

It=E2=80=99s a little worrying that I had to disable three more tests that =
I
think shouldn=E2=80=99t have failed.

What do you think?

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

--=-=-=--






Information forwarded to bug-guix@HIDDEN:
bug#22533; Package guix. Full text available.

Message received at 22533 <at> debbugs.gnu.org:


Received: (at 22533) by debbugs.gnu.org; 4 Mar 2018 15:31:08 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Mar 04 10:31:08 2018
Received: from localhost ([127.0.0.1]:44569 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1esVbP-0003gA-Pr
	for submit <at> debbugs.gnu.org; Sun, 04 Mar 2018 10:31:08 -0500
Received: from mail-it0-f52.google.com ([209.85.214.52]:35835)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <boskovits@HIDDEN>) id 1esVbO-0003ff-1Q
 for 22533 <at> debbugs.gnu.org; Sun, 04 Mar 2018 10:31:06 -0500
Received: by mail-it0-f52.google.com with SMTP id v194so7093024itb.0
 for <22533 <at> debbugs.gnu.org>; Sun, 04 Mar 2018 07:31:06 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=mime-version:in-reply-to:references:from:date:message-id:subject:to
 :cc; bh=Aq4zBcYuWi7LuDL2UGQrwZQXb38/y6JTcb2jFeNWMls=;
 b=MwW5cdBdshn2QueZU46mm+RrfCnxfDQuhZ5phWAEEU87CmFIob0KM9t4p5Ed5PBPHv
 w1JWHalFkqu7WW3wlrhUcdu1A3j/XzQIb0JtaVXxu9oi9w3K+2w5vF+3ei0TvtiInayE
 P6coZHeT1C32K8DUMUv/5kk3TUtI1/xTn3JdKWVN0cvHlc3t5b9Ic+iiWK9gmOp2zoxr
 2kzHZ816KoB7Xb1/KsEr3LvGX49cp9XF8IJWxCYjTQMby13IuU+3eBQG8icRk4LTtRR1
 vA+a9WGmLgPedLxCvSAKoHAXQN926UhTdEQCi34r7Z7DDAcq9DUCpuEvVYUUr8dSx5ct
 CCxw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:mime-version:in-reply-to:references:from:date
 :message-id:subject:to:cc;
 bh=Aq4zBcYuWi7LuDL2UGQrwZQXb38/y6JTcb2jFeNWMls=;
 b=WOso0Zqyfto/OP/kIbS/vFPbRqz2oGrg+0ipmkcl9LfgLpHf2bSelw13QsZubUIOW7
 ZTFSR9gwJ1tB93AMMeJw6CIazQ8zIoab7zhnXTxn+JY0plQcYvam6qfemXEigPIsaJjL
 qNT5WR3sNQKnHY5GullSvHiQqexUfJa39xk7N1TxbN5SQ5UEMfgQPyqY/v8APstwYBsK
 77fjnAMEKXnU2j1TnbOimUjDLdB7WAwe7N1GVaiODRydlRL4WCiRB9dZz4b6rQ08oQyr
 XpHgBrhAAXqI0xN/VQ5FlQAgmGaPEst0A8SamDKWTl9PHz7G+qaBhQ1y8GdK65Vyh6XV
 s5Ew==
X-Gm-Message-State: AElRT7GdnhrNa6MKBHAIubmYiZgqEZSxOu2KmgDC+KhaPLRPFDzcmUWA
 iVuDZxAyglpHUCCXfqJ6YqjBZmRA3PRVeQVKwQ==
X-Google-Smtp-Source: AG47ELuf6FOGWnyCgn8UK0ONnKTAbWc8N4p0ydbFOJYpERGUv7rRWxWcLvHeDAbiYWVVGBP1QmZ7+lg+TtzYh2LpfmE=
X-Received: by 10.36.118.211 with SMTP id z202mr10913044itb.104.1520177460333; 
 Sun, 04 Mar 2018 07:31:00 -0800 (PST)
MIME-Version: 1.0
Received: by 10.79.157.15 with HTTP; Sun, 4 Mar 2018 07:30:59 -0800 (PST)
In-Reply-To: <874llw101c.fsf@HIDDEN>
References: <20160202051544.GA11744@jasmine> <87bmqfu44s.fsf@HIDDEN>
 <87606c23bq.fsf@HIDDEN>
 <CAE4v=phoeRvKHG96Y1-_A2TRAK_wNPUyBgMFgq6+Cd9enFScUg@HIDDEN>
 <874llw101c.fsf@HIDDEN>
From: =?UTF-8?Q?G=C3=A1bor_Boskovits?= <boskovits@HIDDEN>
Date: Sun, 4 Mar 2018 16:30:59 +0100
Message-ID: <CAE4v=pjmdLujBt8uSQduwft2x1-kqjWFbZqWpmam+cKbaeby1Q@HIDDEN>
Subject: Re: bug#22533: Python bytecode reproducibility
To: Ricardo Wurmus <rekado@HIDDEN>
Content-Type: multipart/alternative; boundary="001a1144123684cf89056697e53c"
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 22533
Cc: Marius Bakke <mbakke@HIDDEN>, 22533 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.0 (/)

--001a1144123684cf89056697e53c
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

2018-03-04 13:46 GMT+01:00 Ricardo Wurmus <rekado@HIDDEN>:

>
> Hi G=C3=A1bor,
>
> > Nix had this issue, it seems they have a python 3.5 solution, which
> > should be easy to adopt: https://github.com/NixOS/nixpkgs/issues/22570.
> > WDYT?
>
> Here=E2=80=99s the patch for Nix:
>
>   https://patch-diff.githubusercontent.com/raw/
> NixOS/nixpkgs/pull/22585.diff
>
> Here are the relevant changes to the Python packages:
>
> * Python 3.4
>
>   substituteInPlace "Lib/py_compile.py" --replace "source_stats['mtime']"
> "(1 if 'DETERMINISTIC_BUILD' in os.environ else source_stats['mtime'])"
>   substituteInPlace "Lib/importlib/_bootstrap.py" --replace "source_mtime
> =3D int(source_stats['mtime'])" "source_mtime =3D 1"
>
> * Python 3.5
>
>   substituteInPlace "Lib/py_compile.py" --replace "source_stats['mtime']"
> "(1 if 'DETERMINISTIC_BUILD' in os.environ else source_stats['mtime'])"
>   substituteInPlace "Lib/importlib/_bootstrap_external.py" --replace
> "source_mtime =3D int(st['mtime'])" "source_mtime =3D 1"
>
> * Python 3.6
>   substituteInPlace "Lib/py_compile.py" --replace "source_stats['mtime']"
> "(1 if 'DETERMINISTIC_BUILD' in os.environ else source_stats['mtime'])"
>   substituteInPlace "Lib/importlib/_bootstrap_external.py" --replace
> "source_mtime =3D int(st['mtime'])" "source_mtime =3D 1"
>
>
>
Nice, thanks for the summary.
Can we adopt this as is?
Do we need the 3.4 and 3.5 fix or the 3.6 one is enough?


> For all packages they set these environment variables:
>
>   - set PYTHONHASHSEED=3D0 (for hashes of str, bytes and datetime objects=
)
>
>   - set DETERMINISTIC_BUILD; for conditional patching of the timestamp
>     for package builds.  The timestamp is not patched in ad-hoc
>     environments, because that would mess with Python=E2=80=99s ability t=
o
>     determine whether to compile source files.
>
>
Should we set these in python-build-system? What about python booststrap?
I guess we use gnu-build-system there, so bootstrap packages might need to
set these explicitly?


> They also rebuild all bytecode (with the exception of lib2to3 because it
> is Python 2 code) three times, once for each optimization level.
>
> --8<---------------cut here---------------start------------->8---
> +    # Determinism: rebuild all bytecode
> +    # We exclude lib2to3 because that's Python 2 code which fails
> +    # We rebuild three times, once for each optimization level
> +    find $out -name "*.py" | $out/bin/python -m compileall -q -f -x
> "lib2to3" -i -
> +    find $out -name "*.py" | $out/bin/python -O -m compileall -q -f -x
> "lib2to3" -i -
> +    find $out -name "*.py" | $out/bin/python -OO -m compileall -q -f -x
> "lib2to3" -i -
> --8<---------------cut here---------------end--------------->8---
>
>
Do we also have to do this, or should we settle with one optimization
level? Which one?


> --
> Ricardo
>
> GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
> https://elephly.net
>
>
>

--001a1144123684cf89056697e53c
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div class=3D"gmail_extra"><div class=3D"gmail_quote">2018=
-03-04 13:46 GMT+01:00 Ricardo Wurmus <span dir=3D"ltr">&lt;<a href=3D"mail=
to:rekado@HIDDEN" target=3D"_blank">rekado@HIDDEN</a>&gt;</span>:=
<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-lef=
t:1px #ccc solid;padding-left:1ex"><br>
Hi G=C3=A1bor,<br>
<span class=3D""><br>
&gt; Nix had this issue, it seems they have a python 3.5 solution, which<br=
>
&gt; should be easy to adopt: <a href=3D"https://github.com/NixOS/nixpkgs/i=
ssues/22570" rel=3D"noreferrer" target=3D"_blank">https://github.com/NixOS/=
<wbr>nixpkgs/issues/22570</a>.<br>
&gt; WDYT?<br>
<br>
</span>Here=E2=80=99s the patch for Nix:<br>
<br>
=C2=A0 <a href=3D"https://patch-diff.githubusercontent.com/raw/NixOS/nixpkg=
s/pull/22585.diff" rel=3D"noreferrer" target=3D"_blank">https://patch-diff.=
<wbr>githubusercontent.com/raw/<wbr>NixOS/nixpkgs/pull/22585.diff</a><br>
<br>
Here are the relevant changes to the Python packages:<br>
<br>
* Python 3.4<br>
<br>
=C2=A0 substituteInPlace &quot;Lib/py_compile.py&quot; --replace &quot;sour=
ce_stats[&#39;mtime&#39;]&quot; &quot;(1 if &#39;DETERMINISTIC_BUILD&#39; i=
n os.environ else source_stats[&#39;mtime&#39;])&quot;<br>
=C2=A0 substituteInPlace &quot;Lib/importlib/_bootstrap.py&quot; --replace =
&quot;source_mtime =3D int(source_stats[&#39;mtime&#39;])&quot; &quot;sourc=
e_mtime =3D 1&quot;<br>
<br>
* Python 3.5<br>
<br>
=C2=A0 substituteInPlace &quot;Lib/py_compile.py&quot; --replace &quot;sour=
ce_stats[&#39;mtime&#39;]&quot; &quot;(1 if &#39;DETERMINISTIC_BUILD&#39; i=
n os.environ else source_stats[&#39;mtime&#39;])&quot;<br>
=C2=A0 substituteInPlace &quot;Lib/importlib/_bootstrap_<wbr>external.py&qu=
ot; --replace &quot;source_mtime =3D int(st[&#39;mtime&#39;])&quot; &quot;s=
ource_mtime =3D 1&quot;<br>
<br>
* Python 3.6<br>
=C2=A0 substituteInPlace &quot;Lib/py_compile.py&quot; --replace &quot;sour=
ce_stats[&#39;mtime&#39;]&quot; &quot;(1 if &#39;DETERMINISTIC_BUILD&#39; i=
n os.environ else source_stats[&#39;mtime&#39;])&quot;<br>
=C2=A0 substituteInPlace &quot;Lib/importlib/_bootstrap_<wbr>external.py&qu=
ot; --replace &quot;source_mtime =3D int(st[&#39;mtime&#39;])&quot; &quot;s=
ource_mtime =3D 1&quot;<br>
<br>
<br></blockquote><div><br></div><div>Nice, thanks for the summary.</div><di=
v>Can we adopt this as is?</div><div>Do we need the 3.4 and 3.5 fix or the =
3.6 one is enough?=C2=A0</div><div>=C2=A0</div><blockquote class=3D"gmail_q=
uote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1e=
x">
For all packages they set these environment variables:<br>
<br>
=C2=A0 - set PYTHONHASHSEED=3D0 (for hashes of str, bytes and datetime obje=
cts)<br>
<br>
=C2=A0 - set DETERMINISTIC_BUILD; for conditional patching of the timestamp=
<br>
=C2=A0 =C2=A0 for package builds.=C2=A0 The timestamp is not patched in ad-=
hoc<br>
=C2=A0 =C2=A0 environments, because that would mess with Python=E2=80=99s a=
bility to<br>
=C2=A0 =C2=A0 determine whether to compile source files.<br>
<br></blockquote><div><br></div><div>Should we set these in python-build-sy=
stem? What about python booststrap?</div><div>I guess we use gnu-build-syst=
em there, so bootstrap packages might need to</div><div>set these explicitl=
y?</div><div>=C2=A0</div><blockquote class=3D"gmail_quote" style=3D"margin:=
0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
They also rebuild all bytecode (with the exception of lib2to3 because it<br=
>
is Python 2 code) three times, once for each optimization level.<br>
<br>
--8&lt;---------------cut here---------------start------<wbr>-------&gt;8--=
-<br>
+=C2=A0 =C2=A0 # Determinism: rebuild all bytecode<br>
+=C2=A0 =C2=A0 # We exclude lib2to3 because that&#39;s Python 2 code which =
fails<br>
+=C2=A0 =C2=A0 # We rebuild three times, once for each optimization level<b=
r>
+=C2=A0 =C2=A0 find $out -name &quot;*.py&quot; | $out/bin/python -m compil=
eall -q -f -x &quot;lib2to3&quot; -i -<br>
+=C2=A0 =C2=A0 find $out -name &quot;*.py&quot; | $out/bin/python -O -m com=
pileall -q -f -x &quot;lib2to3&quot; -i -<br>
+=C2=A0 =C2=A0 find $out -name &quot;*.py&quot; | $out/bin/python -OO -m co=
mpileall -q -f -x &quot;lib2to3&quot; -i -<br>
--8&lt;---------------cut here---------------end--------<wbr>-------&gt;8--=
-<br>
<div class=3D"HOEnZb"><div class=3D"h5"><br></div></div></blockquote><div><=
br></div><div>Do we also have to do this, or should we settle with one opti=
mization level? Which one?</div><div>=C2=A0</div><blockquote class=3D"gmail=
_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:=
1ex"><div class=3D"HOEnZb"><div class=3D"h5">
--<br>
Ricardo<br>
<br>
GPG: BCA6 89B6 3655 3801 C3C6=C2=A0 2150 197A 5888 235F ACAC<br>
<a href=3D"https://elephly.net" rel=3D"noreferrer" target=3D"_blank">https:=
//elephly.net</a><br>
<br>
<br>
</div></div></blockquote></div><br></div></div>

--001a1144123684cf89056697e53c--




Information forwarded to bug-guix@HIDDEN:
bug#22533; Package guix. Full text available.

Message received at 22533 <at> debbugs.gnu.org:


Received: (at 22533) by debbugs.gnu.org; 4 Mar 2018 12:46:37 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Mar 04 07:46:37 2018
Received: from localhost ([127.0.0.1]:43524 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1esT2D-00065G-0O
	for submit <at> debbugs.gnu.org; Sun, 04 Mar 2018 07:46:37 -0500
Received: from sender-of-o51.zoho.com ([135.84.80.216]:21088)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <rekado@HIDDEN>) id 1esT2A-000658-LT
 for 22533 <at> debbugs.gnu.org; Sun, 04 Mar 2018 07:46:35 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1520167571; 
 s=zoho; d=elephly.net; i=rekado@HIDDEN;
 h=References:From:To:Cc:Subject:In-reply-to:Date:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-ID;
 l=2427; bh=3cE1rro51XNl5hK6+8kx7xMSfqrfmzWIAD0eMhv8tOU=;
 b=a+ze8Rck0dRuF9iIhnJJP15V/l68lWgWtnyN9tVqsusEq+n7L6+w1jGhIAd2WSZf
 uyCioB3c5Z4Bi2fxpsw9epIqL6aXouLreTerE5k8qUn02WorpBoGyucc+t0e90bdfvn
 lxodLH9ET30xx4jhA/elRFvtE9qnGKp+WhkyeYjM=
Received: from localhost (port-92-200-95-79.dynamic.qsc.de [92.200.95.79]) by
 mx.zohomail.com with SMTPS id 1520167571020712.8951878439287;
 Sun, 4 Mar 2018 04:46:11 -0800 (PST)
References: <20160202051544.GA11744@jasmine> <87bmqfu44s.fsf@HIDDEN>
 <87606c23bq.fsf@HIDDEN>
 <CAE4v=phoeRvKHG96Y1-_A2TRAK_wNPUyBgMFgq6+Cd9enFScUg@HIDDEN>
User-agent: mu4e 1.0; emacs 25.3.1
From: Ricardo Wurmus <rekado@HIDDEN>
To: =?utf-8?Q?G=C3=A1bor?= Boskovits <boskovits@HIDDEN>
Subject: Re: bug#22533: Python bytecode reproducibility
In-reply-to: <CAE4v=phoeRvKHG96Y1-_A2TRAK_wNPUyBgMFgq6+Cd9enFScUg@HIDDEN>
Date: Sun, 04 Mar 2018 13:46:07 +0100
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Message-ID: <874llw101c.fsf@HIDDEN>
X-ZohoMailClient: External
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 22533
Cc: Marius Bakke <mbakke@HIDDEN>, 22533 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.0 (/)


Hi G=C3=A1bor,

> Nix had this issue, it seems they have a python 3.5 solution, which
> should be easy to adopt: https://github.com/NixOS/nixpkgs/issues/22570.
> WDYT?

Here=E2=80=99s the patch for Nix:

  https://patch-diff.githubusercontent.com/raw/NixOS/nixpkgs/pull/22585.dif=
f

Here are the relevant changes to the Python packages:

* Python 3.4

  substituteInPlace "Lib/py_compile.py" --replace "source_stats['mtime']" "=
(1 if 'DETERMINISTIC_BUILD' in os.environ else source_stats['mtime'])"
  substituteInPlace "Lib/importlib/_bootstrap.py" --replace "source_mtime =
=3D int(source_stats['mtime'])" "source_mtime =3D 1"

* Python 3.5

  substituteInPlace "Lib/py_compile.py" --replace "source_stats['mtime']" "=
(1 if 'DETERMINISTIC_BUILD' in os.environ else source_stats['mtime'])"
  substituteInPlace "Lib/importlib/_bootstrap_external.py" --replace "sourc=
e_mtime =3D int(st['mtime'])" "source_mtime =3D 1"

* Python 3.6
  substituteInPlace "Lib/py_compile.py" --replace "source_stats['mtime']" "=
(1 if 'DETERMINISTIC_BUILD' in os.environ else source_stats['mtime'])"
  substituteInPlace "Lib/importlib/_bootstrap_external.py" --replace "sourc=
e_mtime =3D int(st['mtime'])" "source_mtime =3D 1"


For all packages they set these environment variables:

  - set PYTHONHASHSEED=3D0 (for hashes of str, bytes and datetime objects)

  - set DETERMINISTIC_BUILD; for conditional patching of the timestamp
    for package builds.  The timestamp is not patched in ad-hoc
    environments, because that would mess with Python=E2=80=99s ability to
    determine whether to compile source files.

They also rebuild all bytecode (with the exception of lib2to3 because it
is Python 2 code) three times, once for each optimization level.

--8<---------------cut here---------------start------------->8---
+    # Determinism: rebuild all bytecode
+    # We exclude lib2to3 because that's Python 2 code which fails
+    # We rebuild three times, once for each optimization level
+    find $out -name "*.py" | $out/bin/python -m compileall -q -f -x "lib2t=
o3" -i -
+    find $out -name "*.py" | $out/bin/python -O -m compileall -q -f -x "li=
b2to3" -i -
+    find $out -name "*.py" | $out/bin/python -OO -m compileall -q -f -x "l=
ib2to3" -i -
--8<---------------cut here---------------end--------------->8---

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net






Information forwarded to bug-guix@HIDDEN:
bug#22533; Package guix. Full text available.

Message received at 22533 <at> debbugs.gnu.org:


Received: (at 22533) by debbugs.gnu.org; 4 Mar 2018 09:21:25 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Mar 04 04:21:25 2018
Received: from localhost ([127.0.0.1]:43489 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1esPpd-0007mu-ID
	for submit <at> debbugs.gnu.org; Sun, 04 Mar 2018 04:21:25 -0500
Received: from mail-io0-f178.google.com ([209.85.223.178]:33087)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <boskovits@HIDDEN>) id 1esPpb-0007mg-32
 for 22533 <at> debbugs.gnu.org; Sun, 04 Mar 2018 04:21:23 -0500
Received: by mail-io0-f178.google.com with SMTP id f1so14899711iob.0
 for <22533 <at> debbugs.gnu.org>; Sun, 04 Mar 2018 01:21:23 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=mime-version:in-reply-to:references:from:date:message-id:subject:to
 :cc; bh=qlpGl8u0zY4e7E7tyVG7IOPyJ+IUQTATj0uodQIyF5o=;
 b=c7I91Azd3jhrjRXTw3UcJ58d/81JKPFTj/TeKFkEjy+SSql9BTNiEyZZqgCvQFOldF
 PsPBFc5/klu2l1KAEU/znqVCiGKkYyRcZRWMtei8z0RO09DopLTd3IxrrpQjBH7mSk8w
 NDFu5FhVc+35G4qFZagdFW+h1Q4HVZOUlGSHvKN8GcSTyCPrGotYSOKQLgEEpD0XuYq0
 mu+cvIMzPO4CI6gP11WmH8QKl0U3VFoRsNEyT3lLqf/3hJF2ilyKRUmGzxwLi0Z27JLS
 Ntvt9hPsxlYQK6ohuQQWld+gySUB5fv9d3kVhgRTLANEGiGkko+v5rCZuUkyzb4PGx9+
 OCTA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:mime-version:in-reply-to:references:from:date
 :message-id:subject:to:cc;
 bh=qlpGl8u0zY4e7E7tyVG7IOPyJ+IUQTATj0uodQIyF5o=;
 b=pqj2mWIILPCUsG4rEvmjg34FVWFXNiGob0Am3ru+dNrAWICjxxjBdzfEQAYunthoS9
 p0ughU/dqKxw8RjMi7X2BnJ68uTF3rc2TejWB/iWIV2kvNLDX5JNmgpm6Pe0KLLxXzIH
 BUGMqxIJcWMm59mjXmz8jU5SdAZ3Flh9M634ZreLRf1Kzz+y8KzP/Kd825dvWlzLqlgd
 ehwRKTcOy3hoy6Urw/+Kr4rowPWklzlqmfzPFS+sRNtwEORC91iHt4qbQ1sRhDzDiEpE
 GQsfJ6dklwLxifYYFJtQUH3krckuH/XqMFBP9XGrfFCnnraznpScQQKXzhVGeBwMMGSo
 GdmQ==
X-Gm-Message-State: APf1xPCGpf+oxvVnaQDnu5AO8auxDMNH77+8Tgi2dfSlcCXCWj7y65Eu
 4lIXp+tLfMZGvk1Yl8n/4rnMfvmvqQiDypGQHg==
X-Google-Smtp-Source: AG47ELsGHlRL86MAim1b83X0nL6jjCQZFEeL7fSFJf/8ZAOnUA5a8u1uGsIGI9jDpYSfHpgKAIPMHRT0hEkNdz1arVY=
X-Received: by 10.107.53.231 with SMTP id k100mr14361685ioo.292.1520155277529; 
 Sun, 04 Mar 2018 01:21:17 -0800 (PST)
MIME-Version: 1.0
Received: by 10.79.157.15 with HTTP; Sun, 4 Mar 2018 01:21:17 -0800 (PST)
In-Reply-To: <87606c23bq.fsf@HIDDEN>
References: <20160202051544.GA11744@jasmine> <87bmqfu44s.fsf@HIDDEN>
 <87606c23bq.fsf@HIDDEN>
From: =?UTF-8?Q?G=C3=A1bor_Boskovits?= <boskovits@HIDDEN>
Date: Sun, 4 Mar 2018 10:21:17 +0100
Message-ID: <CAE4v=phoeRvKHG96Y1-_A2TRAK_wNPUyBgMFgq6+Cd9enFScUg@HIDDEN>
Subject: Re: bug#22533: Python bytecode reproducibility
To: Ricardo Wurmus <rekado@HIDDEN>
Content-Type: multipart/alternative; boundary="001a1144a158521715056692bbd8"
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 22533
Cc: Marius Bakke <mbakke@HIDDEN>, 22533 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: 0.0 (/)

--001a1144a158521715056692bbd8
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

2018-03-03 23:37 GMT+01:00 Ricardo Wurmus <rekado@HIDDEN>:

> Hi Guix,
>
> Marius Bakke <mbakke@HIDDEN> writes:
>
> > It would be great to revive this longstanding bug!
>
> Indeed.
>
> Here=E2=80=99s another attempt.  As far as I understand, the timestamp in=
 the
> pyc files only affects the header.
>
> Up until Python 3.6 (incl) the header looks like this:
>
>   magic | timestamp | size
>
> Since Python 3.7 the header may either contain a timestamp or a hash:
>
>   magic | 00000000000000000000000000000000 | timestamp | size
>   magic | 00000000000000000000000000000001 | hash      | size
>
> This means we likely won=E2=80=99t have this problem any more with Python=
 3.7.
> For Python 3.6 I guess we could add a final build phase that overwrites
> the timestamp in the *binary*.  This needs to happen before any of the
> compiled files are wrapped up in a wheel.
>
> Should we just wait for Python 3.7 which is expected to be released in
> June 2018?  We=E2=80=99d still have to deal with this problem in Python 2=
,
> though.
>
> Is it a bad idea to override the timestamps in the generated binaries?
> I think that we could avoid the recency check then, which was an
> obstacle to resetting the timestamps of the source files.

--
> Ricardo
>
> GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
> https://elephly.net
>
>
Nix had this issue, it seems they have a python 3.5 solution, which
should be easy to adopt: https://github.com/NixOS/nixpkgs/issues/22570.
WDYT?

--001a1144a158521715056692bbd8
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div class=3D"gmail_extra"><div class=3D"gmail_quote">2018=
-03-03 23:37 GMT+01:00 Ricardo Wurmus <span dir=3D"ltr">&lt;<a href=3D"mail=
to:rekado@HIDDEN" target=3D"_blank">rekado@HIDDEN</a>&gt;</span>:=
<br><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;bor=
der-left:1px solid rgb(204,204,204);padding-left:1ex">Hi Guix,<br>
<br>
Marius Bakke &lt;<a href=3D"mailto:mbakke@HIDDEN">mbakke@HIDDEN=
</a>&gt; writes:<br>
<br>
&gt; It would be great to revive this longstanding bug!<br>
<br>
Indeed.<br>
<br>
Here=E2=80=99s another attempt.=C2=A0 As far as I understand, the timestamp=
 in the<br>
pyc files only affects the header.<br>
<br>
Up until Python 3.6 (incl) the header looks like this:<br>
<br>
=C2=A0 magic | timestamp | size<br>
<br>
Since Python 3.7 the header may either contain a timestamp or a hash:<br>
<br>
=C2=A0 magic | 000000000000000000000000000000<wbr>00 | timestamp | size<br>
=C2=A0 magic | 000000000000000000000000000000<wbr>01 | hash=C2=A0 =C2=A0 =
=C2=A0 | size<br>
<br>
This means we likely won=E2=80=99t have this problem any more with Python 3=
.7.<br>
For Python 3.6 I guess we could add a final build phase that overwrites<br>
the timestamp in the *binary*.=C2=A0 This needs to happen before any of the=
<br>
compiled files are wrapped up in a wheel.<br>
<br>
Should we just wait for Python 3.7 which is expected to be released in<br>
June 2018?=C2=A0 We=E2=80=99d still have to deal with this problem in Pytho=
n 2,<br>
though.<br>
<br>
Is it a bad idea to override the timestamps in the generated binaries?<br>
I think that we could avoid the recency check then, which was an<br>
obstacle to resetting the timestamps of the source files.</blockquote><bloc=
kquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:=
1px solid rgb(204,204,204);padding-left:1ex">
--<br>
Ricardo<br>
<br>
GPG: BCA6 89B6 3655 3801 C3C6=C2=A0 2150 197A 5888 235F ACAC<br>
<a href=3D"https://elephly.net" rel=3D"noreferrer" target=3D"_blank">https:=
//elephly.net</a><br>
<br></blockquote><div><br></div><div>Nix had this issue, it seems they have=
 a python 3.5 solution, which</div><div>should be easy to adopt:=C2=A0<a hr=
ef=3D"https://github.com/NixOS/nixpkgs/issues/22570">https://github.com/Nix=
OS/nixpkgs/issues/22570</a>.</div><div>WDYT?</div></div><br></div></div>

--001a1144a158521715056692bbd8--




Information forwarded to bug-guix@HIDDEN:
bug#22533; Package guix. Full text available.

Message received at 22533 <at> debbugs.gnu.org:


Received: (at 22533) by debbugs.gnu.org; 3 Mar 2018 22:37:51 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Mar 03 17:37:51 2018
Received: from localhost ([127.0.0.1]:43287 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1esFmp-0007cL-Hi
	for submit <at> debbugs.gnu.org; Sat, 03 Mar 2018 17:37:51 -0500
Received: from sender-of-o51.zoho.com ([135.84.80.216]:21130)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <rekado@HIDDEN>) id 1esFmn-0007cC-N9
 for 22533 <at> debbugs.gnu.org; Sat, 03 Mar 2018 17:37:50 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1520116652; 
 s=zoho; d=elephly.net; i=rekado@HIDDEN;
 h=References:From:To:Cc:Subject:In-reply-to:Date:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-ID;
 l=1246; bh=pbfIpUnK4KJw4E9/30piOmYcIWs0PH3xUbo7vmxLWj4=;
 b=ifWAJwe5pepJzPXxL2C0rCZrbHg0EeVa4F16dooQn2PFPMlMxjI482ze6wtakoox
 KD8Zgt2TR6+3DFznMBGn982HbdiBagMxanUqOWp1xl81sPgILH5bArIjn01FTymKooy
 fAR/EXnClSgwFqdhE5XcvetavO+9tCEKp0FmN2vs=
Received: from localhost (port-92-200-66-188.dynamic.qsc.de [92.200.66.188])
 by mx.zohomail.com with SMTPS id 1520116652148789.4105675915988;
 Sat, 3 Mar 2018 14:37:32 -0800 (PST)
References: <20160202051544.GA11744@jasmine> <87bmqfu44s.fsf@HIDDEN>
User-agent: mu4e 1.0; emacs 25.3.1
From: Ricardo Wurmus <rekado@HIDDEN>
To: Marius Bakke <mbakke@HIDDEN>
Subject: Re: bug#22533: Python bytecode reproducibility
In-reply-to: <87bmqfu44s.fsf@HIDDEN>
X-URL: https://elephly.net
X-PGP-Key: https://elephly.net/rekado.pubkey
X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
Date: Sat, 03 Mar 2018 23:37:29 +0100
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Message-ID: <87606c23bq.fsf@HIDDEN>
X-ZohoMailClient: External
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 22533
Cc: 22533 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.0 (/)

Hi Guix,

Marius Bakke <mbakke@HIDDEN> writes:

> It would be great to revive this longstanding bug!

Indeed.

Here=E2=80=99s another attempt.  As far as I understand, the timestamp in t=
he
pyc files only affects the header.

Up until Python 3.6 (incl) the header looks like this:

  magic | timestamp | size

Since Python 3.7 the header may either contain a timestamp or a hash:

  magic | 00000000000000000000000000000000 | timestamp | size
  magic | 00000000000000000000000000000001 | hash      | size

This means we likely won=E2=80=99t have this problem any more with Python 3=
.7.
For Python 3.6 I guess we could add a final build phase that overwrites
the timestamp in the *binary*.  This needs to happen before any of the
compiled files are wrapped up in a wheel.

Should we just wait for Python 3.7 which is expected to be released in
June 2018?  We=E2=80=99d still have to deal with this problem in Python 2,
though.

Is it a bad idea to override the timestamps in the generated binaries?
I think that we could avoid the recency check then, which was an
obstacle to resetting the timestamps of the source files.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net






Information forwarded to bug-guix@HIDDEN:
bug#22533; Package guix. Full text available.

Message received at 22533 <at> debbugs.gnu.org:


Received: (at 22533) by debbugs.gnu.org; 26 May 2017 13:41:43 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri May 26 09:41:43 2017
Received: from localhost ([127.0.0.1]:38792 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1dEFUt-0004Yd-Ei
	for submit <at> debbugs.gnu.org; Fri, 26 May 2017 09:41:43 -0400
Received: from out1-smtp.messagingengine.com ([66.111.4.25]:44977)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mbakke@HIDDEN>) id 1dEFUs-0004YV-9W
 for 22533 <at> debbugs.gnu.org; Fri, 26 May 2017 09:41:42 -0400
Received: from compute5.internal (compute5.nyi.internal [10.202.2.45])
 by mailout.nyi.internal (Postfix) with ESMTP id 0625420ACD
 for <22533 <at> debbugs.gnu.org>; Fri, 26 May 2017 09:41:42 -0400 (EDT)
Received: from frontend1 ([10.202.2.160])
 by compute5.internal (MEProxy); Fri, 26 May 2017 09:41:42 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastmail.com; h=
 content-type:date:from:message-id:mime-version:subject:to
 :x-me-sender:x-me-sender:x-sasl-enc:x-sasl-enc; s=fm1; bh=IgEIln
 MVsjdekXQQEGJTQocPmCnOdXnZj5zyWtVp3fc=; b=syW7NjdYJlH9fHp71cOmFZ
 5k71JDJOHbCn4GrylJCkvF14O8IRbZwv7/BaSeQj6T8EaOdfGZq2a1GhFjKr9XCB
 h8Vqu6kMiVn0Hau7/OTCZdtJN1qhP2VAEHx1Bj5mOaeyJObN2CauJ/PU/PUdfZHE
 uWaMszsPYOKum54W5s/M+HWQa/ig8YDtHYIEzo9JjrTSMAdqgyQF6oXoPf7YNMMA
 KA6yeBRsm40Ac8j86pIC18sq+qM+OiTlXuyZiOkLcSQ7j3xkb7WY8nj8MXyOPBGE
 8BU76oWc/1Kd3UXyTPIcpIUV3JSYk5nFyUd/3PwS1O1L5EI2UUnZKMneck6prT4g
 ==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=content-type:date:from:message-id
 :mime-version:subject:to:x-me-sender:x-me-sender:x-sasl-enc
 :x-sasl-enc; s=fm1; bh=IgEIlnMVsjdekXQQEGJTQocPmCnOdXnZj5zyWtVp3
 fc=; b=fiw48lCW31Mjc4Fp9B7tLDPeCAUvwErUM0hYb6zThkp0B1PE6RNTW1OhI
 RHwyL8MFZx1X6WbmS2aSMFyu399e5e/am72FkMKZQ4CNRgLuhL6smA/JllU35GhV
 HZWNnypeYDedQZdn/XII8bdUX9dr+1YArogHCiVAr7HJtHfe10gRqrMzIg3P3bbN
 lryVhs6jhx0A+PaSxbf5VUkN6ovaw/FleHC+PQM9qTHad0+RurEP6E32e/Tzewbh
 ENdFHgdL+2fhCOwpZOhrMoFtoGKt92g758OVHmFFSg7mKz30VlO+ddIQLZ3pcIXs
 t5uEfYL7zp2OxADKE6Az6kJEC5uyg==
X-ME-Sender: <xms:lTAoWazxbChF5350ZIqw8EhNfIY4vjO45aqKUzsumq7p9aUmpBvDZg>
X-Sasl-enc: p4C/GXJvdi49ufXlsjww5mid40l0d3y1VpUdiiWj8u+D 1495806101
Received: from localhost (unknown [188.113.81.93])
 by mail.messagingengine.com (Postfix) with ESMTPA id 876797E84B
 for <22533 <at> debbugs.gnu.org>; Fri, 26 May 2017 09:41:41 -0400 (EDT)
From: Marius Bakke <mbakke@HIDDEN>
To: 22533 <at> debbugs.gnu.org
Subject: Python bytecode reproducibility
User-Agent: Notmuch/0.24.1 (https://notmuchmail.org) Emacs/25.2.1
 (x86_64-unknown-linux-gnu)
Date: Fri, 26 May 2017 15:41:39 +0200
Message-ID: <87bmqfu44s.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="=-=-=";
 micalg=pgp-sha512; protocol="application/pgp-signature"
X-Spam-Score: -0.7 (/)
X-Debbugs-Envelope-To: 22533
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.7 (/)

--=-=-=
Content-Type: text/plain

Hello!

I stumbled across this bug after re-discovering that Python bytecode is
not reproducible (through "glib"). Just sharing some notes..

Nix recently made an effort to fix this. AFAICT the ".pyc" files are
still a problem, but at least they got the interpreters building
reproducibly:

https://github.com/NixOS/nixpkgs/issues/22570
https://github.com/NixOS/nixpkgs/pull/22585

It would be great to revive this longstanding bug!

*walks away slowly before anyone notices*

--=-=-=
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQEzBAEBCgAdFiEEu7At3yzq9qgNHeZDoqBt8qM6VPoFAlkoMJQACgkQoqBt8qM6
VPoLTQgAsNXfMc86cXSiDAJaJCSTbdUEgeEBCEQDFCqcZA8eTxaosuWwjJy3J9lV
9D+nGYols54UeCypEGgfrrrgB49s9XnSuZNlbkMyvPYzASpHmVpx2OWp6NBiLUf3
Rn9qwMTwqchqZNoRJ2F8MGAbklavS7bqruiI9c/38zA/Sg4aET42vDEOZiB3DDIW
bwKFKrZs/BqWSfJXLj9nqzMsbjzn00FzPsOmdxx0rVv07lDdOAUWL0DCm2FbL2OD
NhFNt8pW1+0Y6CmYkln+cwmk4XJraSKvH3+QaEHNEBFmL8FuEKixZ0DTRK+bWdzf
HUB5JYA2299APLld+UFM9qMaMYA5sQ==
=QJ16
-----END PGP SIGNATURE-----
--=-=-=--




Information forwarded to bug-guix@HIDDEN:
bug#22533; Package guix. Full text available.

Message received at 22533 <at> debbugs.gnu.org:


Received: (at 22533) by debbugs.gnu.org; 6 Apr 2016 08:30:14 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Wed Apr 06 04:30:13 2016
Received: from localhost ([127.0.0.1]:50955 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1aniqr-0007jD-ID
	for submit <at> debbugs.gnu.org; Wed, 06 Apr 2016 04:30:13 -0400
Received: from eggs.gnu.org ([208.118.235.92]:49331)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1aniqq-0007j1-P3
 for 22533 <at> debbugs.gnu.org; Wed, 06 Apr 2016 04:30:13 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <ludo@HIDDEN>) id 1aniqf-0005wV-Dz
 for 22533 <at> debbugs.gnu.org; Wed, 06 Apr 2016 04:30:07 -0400
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=-0.2 required=5.0 tests=BAYES_50,RP_MATCHES_RCVD
 autolearn=disabled version=3.3.2
Received: from fencepost.gnu.org ([2001:4830:134:3::e]:36475)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <ludo@HIDDEN>)
 id 1aniqf-0005wR-9f; Wed, 06 Apr 2016 04:30:01 -0400
Received: from reverse-83.fdn.fr ([80.67.176.83]:58896 helo=pluto)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_128_CBC_SHA1:128)
 (Exim 4.82) (envelope-from <ludo@HIDDEN>)
 id 1aniqe-0002c3-Bs; Wed, 06 Apr 2016 04:30:01 -0400
From: ludo@HIDDEN (Ludovic =?utf-8?Q?Court=C3=A8s?=)
To: Cyril Roelandt <tipecaml@HIDDEN>
Subject: Re: bug#22533: Non-determinism in python-3 ".pyc" bytecode
References: <20160202051544.GA11744@jasmine> <87powezvyo.fsf@HIDDEN>
 <20160204231708.GA1297@jasmine> <56FB0C14.9010909@HIDDEN>
X-URL: http://www.fdn.fr/~lcourtes/
X-Revolutionary-Date: 18 Germinal an 224 de la =?utf-8?Q?R=C3=A9volution?=
X-PGP-Key-ID: 0x3D9AEBB5
X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc
X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4  0CFB 090B 1199 3D9A EBB5
X-OS: x86_64-unknown-linux-gnu
Date: Wed, 06 Apr 2016 10:29:57 +0200
In-Reply-To: <56FB0C14.9010909@HIDDEN> (Cyril Roelandt's message of "Wed,
 30 Mar 2016 01:13:24 +0200")
Message-ID: <87oa9nuol6.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux)
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="=-=-="
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2001:4830:134:3::e
X-Spam-Score: -6.0 (------)
X-Debbugs-Envelope-To: 22533
Cc: 22533 <at> debbugs.gnu.org, Leo Famulari <leo@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -6.0 (------)

--=-=-=
Content-Type: text/plain

Cyril Roelandt <tipecaml@HIDDEN> skribis:

> Here is a version of the patch that works with the upstream Python, but
> that I cannot get to work with our Guix recipe.

At first sight the patch LGTM.  How does it not work for you? :-)

I applied this:


--=-=-=
Content-Type: text/x-patch
Content-Disposition: inline

diff --git a/gnu/packages/patches/python-3-deterministic-build-info.patch b/gnu/packages/patches/python-3-deterministic-build-info.patch
index 22c372a..bdf9f20 100644
--- a/gnu/packages/patches/python-3-deterministic-build-info.patch
+++ b/gnu/packages/patches/python-3-deterministic-build-info.patch
@@ -15,3 +15,28 @@ We cannot pass it in CPPFLAGS due to whitespace in the DATE string.
  #ifndef DATE
  #ifdef __DATE__
  #define DATE __DATE__
+
+--- Lib/importlib/_bootstrap.py
++++ Lib/importlib/_bootstrap.py
+@@ -1443,7 +1443,8 @@ class SourceLoader(_LoaderBasics):
+         Implementing this method allows the loader to read bytecode files.
+         Raises IOError when the path cannot be handled.
+         """
+-        return {'mtime': self.path_mtime(path)}
++        return {'mtime': float(_os.environ.get(b'SOURCE_DATE_EPOCH',
++                                               st.st_mtime))}
+ 
+     def _cache_bytecode(self, source_path, cache_path, data):
+         """Optional method which writes data (bytes) to a file path (a str).
+@@ -1580,7 +1581,10 @@ class SourceFileLoader(FileLoader, SourceLoader):
+     def path_stats(self, path):
+         """Return the metadata for the path."""
+         st = _path_stat(path)
+-        return {'mtime': st.st_mtime, 'size': st.st_size}
++        return {
++            'mtime':  float(_os.environ.get(b'SOURCE_DATE_EPOCH', st.st_mtime)),
++            'size': st.st_size
++        }
+ 
+     def _cache_bytecode(self, source_path, bytecode_path, data):
+         # Adapt between the two APIs

--=-=-=
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


=E2=80=A6 and that leads to these test failures:

--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env guix build python@3 --rounds=3D2 -K

[...]

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
FAIL: test_bad_marshal (test.test_importlib.source.test_file_loader.Source_=
SourceLoaderBadBytecodeTestPEP302)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/te=
st_importlib/source/util.py", line 22, in wrapper
    to_return =3D fxn(*args, **kwargs)
  File "/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/te=
st_importlib/source/test_file_loader.py", line 452, in test_bad_marshal
    self._test_bad_marshal()
  File "/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/te=
st_importlib/source/test_file_loader.py", line 342, in _test_bad_marshal
    self.import_(file_path, '_temp')
AssertionError: EOFError not raised

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
FAIL: test_no_marshal (test.test_importlib.source.test_file_loader.Source_S=
ourceLoaderBadBytecodeTestPEP302)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/te=
st_importlib/source/util.py", line 22, in wrapper
    to_return =3D fxn(*args, **kwargs)
  File "/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/te=
st_importlib/source/test_file_loader.py", line 441, in test_no_marshal
    self._test_no_marshal()
  File "/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/te=
st_importlib/source/test_file_loader.py", line 322, in _test_no_marshal
    self.import_(file_path, '_temp')
AssertionError: EOFError not raised

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
FAIL: test_non_code_marshal (test.test_importlib.source.test_file_loader.So=
urce_SourceLoaderBadBytecodeTestPEP302)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/te=
st_importlib/source/util.py", line 22, in wrapper
    to_return =3D fxn(*args, **kwargs)
  File "/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/te=
st_importlib/source/test_file_loader.py", line 445, in test_non_code_marshal
    self._test_non_code_marshal()
  File "/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/te=
st_importlib/source/test_file_loader.py", line 331, in _test_non_code_marsh=
al
    self.import_(file_path, '_temp')
AssertionError: ImportError not raised

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
FAIL: test_old_timestamp (test.test_importlib.source.test_file_loader.Sourc=
e_SourceLoaderBadBytecodeTestPEP302)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/te=
st_importlib/source/util.py", line 22, in wrapper
    to_return =3D fxn(*args, **kwargs)
  File "/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/te=
st_importlib/source/test_file_loader.py", line 471, in test_old_timestamp
    self.assertEqual(bytecode_file.read(4), source_timestamp)
AssertionError: b'\x01\x00\x00\x00' !=3D b'\x7f\xc7\x04W'

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
FAIL: test_bad_marshal (test.test_importlib.source.test_file_loader.Source_=
SourceLoaderBadBytecodeTestPEP451)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/te=
st_importlib/source/util.py", line 22, in wrapper
    to_return =3D fxn(*args, **kwargs)
  File "/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/te=
st_importlib/source/test_file_loader.py", line 452, in test_bad_marshal
    self._test_bad_marshal()
  File "/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/te=
st_importlib/source/test_file_loader.py", line 342, in _test_bad_marshal
    self.import_(file_path, '_temp')
AssertionError: EOFError not raised

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
FAIL: test_no_marshal (test.test_importlib.source.test_file_loader.Source_S=
ourceLoaderBadBytecodeTestPEP451)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/te=
st_importlib/source/util.py", line 22, in wrapper
    to_return =3D fxn(*args, **kwargs)
  File "/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/te=
st_importlib/source/test_file_loader.py", line 441, in test_no_marshal
    self._test_no_marshal()
  File "/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/te=
st_importlib/source/test_file_loader.py", line 322, in _test_no_marshal
    self.import_(file_path, '_temp')
AssertionError: EOFError not raised

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
FAIL: test_non_code_marshal (test.test_importlib.source.test_file_loader.So=
urce_SourceLoaderBadBytecodeTestPEP451)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/te=
st_importlib/source/util.py", line 22, in wrapper
    to_return =3D fxn(*args, **kwargs)
  File "/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/te=
st_importlib/source/test_file_loader.py", line 445, in test_non_code_marshal
    self._test_non_code_marshal()
  File "/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/te=
st_importlib/source/test_file_loader.py", line 331, in _test_non_code_marsh=
al
    self.import_(file_path, '_temp')
AssertionError: ImportError not raised

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
FAIL: test_old_timestamp (test.test_importlib.source.test_file_loader.Sourc=
e_SourceLoaderBadBytecodeTestPEP451)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/te=
st_importlib/source/util.py", line 22, in wrapper
    to_return =3D fxn(*args, **kwargs)
  File "/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/te=
st_importlib/source/test_file_loader.py", line 471, in test_old_timestamp
    self.assertEqual(bytecode_file.read(4), source_timestamp)
AssertionError: b'\x01\x00\x00\x00' !=3D b'\x7f\xc7\x04W'

----------------------------------------------------------------------
Ran 951 tests in 1.102s

FAILED (failures=3D8, skipped=3D19, expected failures=3D1)
Makefile:958: recipe for target 'test' failed
--8<---------------cut here---------------end--------------->8---

=E2=80=98test_old_timestamp=E2=80=99 clearly needs to be adjusted to accoun=
t for the
change.  The others have to do with the bytecode loader, so it=E2=80=99s
probably a similar story.  Could you look into it?

Perhaps you tested with SOURCE_DATE_EPOCH unset?

Thanks for working on this, it=E2=80=99s an important bug to fix!

Ludo=E2=80=99.

--=-=-=--




Information forwarded to bug-guix@HIDDEN:
bug#22533; Package guix. Full text available.

Message received at 22533 <at> debbugs.gnu.org:


Received: (at 22533) by debbugs.gnu.org; 29 Mar 2016 23:13:34 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Mar 29 19:13:34 2016
Received: from localhost ([127.0.0.1]:44231 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1al2pK-0007S2-EW
	for submit <at> debbugs.gnu.org; Tue, 29 Mar 2016 19:13:34 -0400
Received: from mail-wm0-f67.google.com ([74.125.82.67]:35064)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <tipecaml@HIDDEN>) id 1al2pI-0007Rm-9e
 for 22533 <at> debbugs.gnu.org; Tue, 29 Mar 2016 19:13:32 -0400
Received: by mail-wm0-f67.google.com with SMTP id 139so9978491wmn.2
 for <22533 <at> debbugs.gnu.org>; Tue, 29 Mar 2016 16:13:32 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;
 h=subject:to:references:cc:from:message-id:date:user-agent
 :mime-version:in-reply-to;
 bh=M1Ax1OHxMsLgZexYUGLjaG0RM6s8H16JvP5YAtS1aqQ=;
 b=h3p8J2xhER/34cYAc2ISTF625k9rOFtKonYgYugCZ1O9ms1FuApPtiGel36k3dwajd
 aFkBtUSlCuo943M7l7+BIwdDtX9NJ4uCL3h25DoL57BbzGaew3dXk/5r8kCREc091sDs
 c9RjPCI2dcdH4gKbKeQKbYvDnJoTJFHOKtbkhLPYMdxgGnO1pHUqDV2hxKOSG9WCRp9B
 hTaONnOJploR6PmTGcjfQA23jfqIWhKGgj/KTCizzYxrsQWk17Tjad3LY9mdCFAWI1c8
 uvjFgkPKAkdvGMiyzs68RXP3IzdmB6th60JDhi+UMQT2emTh95zHi8Xb9FXYWrblWEXt
 xqiA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20130820;
 h=x-gm-message-state:subject:to:references:cc:from:message-id:date
 :user-agent:mime-version:in-reply-to;
 bh=M1Ax1OHxMsLgZexYUGLjaG0RM6s8H16JvP5YAtS1aqQ=;
 b=f21zgn9PHeAl3z465x5U1+u1hieYLedqIYT8tMz9Sp6TGVCPaEi0jyKLjOlgz3Ydp7
 6st4omZ+7RPJUL6f3StJFkwQuFOEmy/xbQ9eCivL/8gb/OZWmdqNcIRvaDo506i2Uh9B
 ifxKZhU7OZW3MmGr62hfh9WA4IddHgMfto4HhVS3NRZGzLjelPRz9C3FcIlVxZEsB8KW
 skrlg3v+TekJUGLTJfRyhQKQnfmRjpZwhTEwBe++3lRMZ2MY9GoWtjyGszj+w4KfyEFh
 pweW2H7jXAjl9tgCPn0mdn1gQe30+9q1y7wlSe6HWbLe1k02mAHLs4ccOtkEb+7JWsmf
 Fieg==
X-Gm-Message-State: AD7BkJKLkFUd76ou2pEEaQoJSacYkRMG+PtjvYckSnfH2UXl3C6ARvO+M5sD2nVvFw4klw==
X-Received: by 10.194.9.201 with SMTP id c9mr5824078wjb.47.1459293206714;
 Tue, 29 Mar 2016 16:13:26 -0700 (PDT)
Received: from [192.168.0.10] (tal33-3-82-233-82-24.fbx.proxad.net.
 [82.233.82.24])
 by smtp.googlemail.com with ESMTPSA id w184sm1411901wmb.1.2016.03.29.16.13.25
 (version=TLSv1/SSLv3 cipher=OTHER);
 Tue, 29 Mar 2016 16:13:26 -0700 (PDT)
Subject: Re: bug#22533: Non-determinism in python-3 ".pyc" bytecode
To: Leo Famulari <leo@HIDDEN>, =?UTF-8?Q?Ludovic_Court=c3=a8s?=
 <ludo@HIDDEN>
References: <20160202051544.GA11744@jasmine> <87powezvyo.fsf@HIDDEN>
 <20160204231708.GA1297@jasmine>
From: Cyril Roelandt <tipecaml@HIDDEN>
Message-ID: <56FB0C14.9010909@HIDDEN>
Date: Wed, 30 Mar 2016 01:13:24 +0200
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
 Icedove/38.7.0
MIME-Version: 1.0
In-Reply-To: <20160204231708.GA1297@jasmine>
Content-Type: multipart/mixed; boundary="------------030500010504060804020309"
X-Spam-Score: -0.7 (/)
X-Debbugs-Envelope-To: 22533
Cc: 22533 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.7 (/)

This is a multi-part message in MIME format.
--------------030500010504060804020309
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 7bit

Here is a version of the patch that works with the upstream Python, but
that I cannot get to work with our Guix recipe.

Could you test it and tell me what you think? I intend to push this to
CPython.

Cyril.

--------------030500010504060804020309
Content-Type: text/x-diff;
 name="upstream.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename="upstream.patch"

diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py
index c4ee41a..d9885c9 100644
--- Lib/importlib/_bootstrap.py
+++ Lib/importlib/_bootstrap.py
@@ -1443,7 +1443,8 @@ class SourceLoader(_LoaderBasics):
         Implementing this method allows the loader to read bytecode files.
         Raises IOError when the path cannot be handled.
         """
-        return {'mtime': self.path_mtime(path)}
+        return {'mtime': float(_os.environ.get(b'SOURCE_DATE_EPOCH',
+                                               st.st_mtime))}
 
     def _cache_bytecode(self, source_path, cache_path, data):
         """Optional method which writes data (bytes) to a file path (a str).
@@ -1580,7 +1581,10 @@ class SourceFileLoader(FileLoader, SourceLoader):
     def path_stats(self, path):
         """Return the metadata for the path."""
         st = _path_stat(path)
-        return {'mtime': st.st_mtime, 'size': st.st_size}
+        return {
+            'mtime':  float(_os.environ.get(b'SOURCE_DATE_EPOCH', st.st_mtime)),
+            'size': st.st_size
+        }
 
     def _cache_bytecode(self, source_path, bytecode_path, data):
         # Adapt between the two APIs

--------------030500010504060804020309--




Information forwarded to bug-guix@HIDDEN:
bug#22533; Package guix. Full text available.

Message received at submit <at> debbugs.gnu.org:


Received: (at submit) by debbugs.gnu.org; 29 Mar 2016 23:12:03 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Mar 29 19:12:03 2016
Received: from localhost ([127.0.0.1]:44227 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1al2nr-0007Ph-3K
	for submit <at> debbugs.gnu.org; Tue, 29 Mar 2016 19:12:03 -0400
Received: from eggs.gnu.org ([208.118.235.92]:59955)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <tipecaml@HIDDEN>) id 1al2np-0007PC-N2
 for submit <at> debbugs.gnu.org; Tue, 29 Mar 2016 19:12:02 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <tipecaml@HIDDEN>) id 1al2nj-00089t-Mb
 for submit <at> debbugs.gnu.org; Tue, 29 Mar 2016 19:11:56 -0400
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50,FREEMAIL_FROM,
 T_DKIM_INVALID autolearn=disabled version=3.3.2
Received: from lists.gnu.org ([2001:4830:134:3::11]:44252)
 by eggs.gnu.org with esmtp (Exim 4.71)
 (envelope-from <tipecaml@HIDDEN>) id 1al2nj-00089o-Ja
 for submit <at> debbugs.gnu.org; Tue, 29 Mar 2016 19:11:55 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:50918)
 by lists.gnu.org with esmtp (Exim 4.71)
 (envelope-from <tipecaml@HIDDEN>) id 1al2ni-0003na-Ex
 for bug-guix@HIDDEN; Tue, 29 Mar 2016 19:11:55 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <tipecaml@HIDDEN>) id 1al2nf-00087S-9N
 for bug-guix@HIDDEN; Tue, 29 Mar 2016 19:11:54 -0400
Received: from mail-wm0-x22c.google.com ([2a00:1450:400c:c09::22c]:34838)
 by eggs.gnu.org with esmtp (Exim 4.71)
 (envelope-from <tipecaml@HIDDEN>) id 1al2nf-00087K-2H
 for bug-guix@HIDDEN; Tue, 29 Mar 2016 19:11:51 -0400
Received: by mail-wm0-x22c.google.com with SMTP id 191so68952326wmq.0
 for <bug-guix@HIDDEN>; Tue, 29 Mar 2016 16:11:50 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;
 h=subject:to:references:from:message-id:date:user-agent:mime-version
 :in-reply-to; bh=AX9Ihvgu9ENVK87/c0r0d2P9X2fkRIm/y3bIKNnsCZg=;
 b=WKK1bW2i2YHmBnrwqgm0p5Ce8CF+7FJuOS8Khd3e0brGyG080QdEYpNRLpFn1gOdo4
 Hgw8G+Z3QD30N0e6rPAU5ZHSFagK0g3OexY4TFkr3i654Pqh/XPcyeLZwSMBDr8I2aXc
 DYAjytG1BHIqAIhMtE0NdwepDwOS260ycII6K8VxV0bkknxKD7LSsnHDShDgsDdziW0N
 2Ju6VNhyeEbA+2JETOI8VYnHlV0Hx5gp/2x0rR64JWWoCm/a5b1ylHW0VejHubkG/qEn
 7Bl3K/GC52YXBx3g2n20F9F+xFrWINSOucXqBjtxb3ViM8RKNg/bkGZix4cTaUcppGVf
 uzSw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20130820;
 h=x-gm-message-state:subject:to:references:from:message-id:date
 :user-agent:mime-version:in-reply-to;
 bh=AX9Ihvgu9ENVK87/c0r0d2P9X2fkRIm/y3bIKNnsCZg=;
 b=TzFKiuA7TUuiZ8TVNqI2ke+1JTo6O6O5bLyPjoMutz/2bIc//40KxgylJi0U/vr19p
 G/e6KzxltjfDKh+e1MUFfC1WuWut9Mx52kJMrVUiCbzV2/tIFxR9fpH9xVOX+XOqQ4LO
 fsiXWGt2S1hEvX+naF9mcl2oblp7rQshbracODGny8LUakLrZYbnEr07y2gt5EzKmEhm
 y0IgkMokV6I3TcwIxOc+ninYUWj7wJiCeMTqN5ekMq+CiMldqfpwI/aUAWaYj9z4MF1n
 neCXGk8DZbwGI5r38vOVKZtjXk/lr8AhKSi+zGfLCiHdMqo843SoGtIZCiaU9GSf+ZqT
 4w1w==
X-Gm-Message-State: AD7BkJLuxUYHRm1e7IDIsuHrvPSjJBo3XayE9EO8x8nAY14N0HKAUfmo7BNWIkGeOYW7bg==
X-Received: by 10.28.105.5 with SMTP id e5mr6324824wmc.87.1459293110268;
 Tue, 29 Mar 2016 16:11:50 -0700 (PDT)
Received: from [192.168.0.10] (tal33-3-82-233-82-24.fbx.proxad.net.
 [82.233.82.24])
 by smtp.googlemail.com with ESMTPSA id w184sm1407375wmb.1.2016.03.29.16.11.48
 for <bug-guix@HIDDEN> (version=TLSv1/SSLv3 cipher=OTHER);
 Tue, 29 Mar 2016 16:11:49 -0700 (PDT)
Subject: Re: bug#22533: Non-determinism in python-3 ".pyc" bytecode
To: bug-guix@HIDDEN
References: <20160202051544.GA11744@jasmine> <87powezvyo.fsf@HIDDEN>
 <20160204231708.GA1297@jasmine>
From: Cyril Roelandt <tipecaml@HIDDEN>
Message-ID: <56FB0BB3.5040303@HIDDEN>
Date: Wed, 30 Mar 2016 01:11:47 +0200
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
 Icedove/38.7.0
MIME-Version: 1.0
In-Reply-To: <20160204231708.GA1297@jasmine>
Content-Type: multipart/mixed; boundary="------------060100070105040008030705"
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x
X-Received-From: 2001:4830:134:3::11
X-Spam-Score: -4.0 (----)
X-Debbugs-Envelope-To: submit
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -4.0 (----)

This is a multi-part message in MIME format.
--------------060100070105040008030705
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 8bit

Here is a version of the patch that works with the upstream Python, but
that I cannot get to work with our Guix recipe.

Could you test it and tell me what you think? I intend to push this to
CPython.

Cyril.

--------------060100070105040008030705
Content-Type: text/x-diff;
 name="upstream.patch"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: attachment;
 filename="upstream.patch"

diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py
index c4ee41a..d9885c9 100644
--- Lib/importlib/_bootstrap.py
+++ Lib/importlib/_bootstrap.py
@@ -1443,7 +1443,8 @@ class SourceLoader(_LoaderBasics):
         Implementing this method allows the loader to read bytecode file=
s.
         Raises IOError when the path cannot be handled.
         """
-        return {'mtime': self.path_mtime(path)}
+        return {'mtime': float(_os.environ.get(b'SOURCE_DATE_EPOCH',
+                                               st.st_mtime))}
=20
     def _cache_bytecode(self, source_path, cache_path, data):
         """Optional method which writes data (bytes) to a file path (a s=
tr).
@@ -1580,7 +1581,10 @@ class SourceFileLoader(FileLoader, SourceLoader):
     def path_stats(self, path):
         """Return the metadata for the path."""
         st =3D _path_stat(path)
-        return {'mtime': st.st_mtime, 'size': st.st_size}
+        return {
+            'mtime':  float(_os.environ.get(b'SOURCE_DATE_EPOCH', st.st_=
mtime)),
+            'size': st.st_size
+        }
=20
     def _cache_bytecode(self, source_path, bytecode_path, data):
         # Adapt between the two APIs

--------------060100070105040008030705--




Information forwarded to bug-guix@HIDDEN:
bug#22533; Package guix. Full text available.
Severity set to 'important' from 'normal' Request was from ludo@HIDDEN (Ludovic Courtès) to control <at> debbugs.gnu.org. Full text available.

Message received at 22533 <at> debbugs.gnu.org:


Received: (at 22533) by debbugs.gnu.org; 4 Feb 2016 23:17:20 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Feb 04 18:17:19 2016
Received: from localhost ([127.0.0.1]:60729 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1aRT9L-0007bc-J2
	for submit <at> debbugs.gnu.org; Thu, 04 Feb 2016 18:17:19 -0500
Received: from out3-smtp.messagingengine.com ([66.111.4.27]:57913)
 by debbugs.gnu.org with esmtp (Exim 4.84)
 (envelope-from <leo@HIDDEN>) id 1aRT9J-0007bS-7c
 for 22533 <at> debbugs.gnu.org; Thu, 04 Feb 2016 18:17:18 -0500
Received: from compute6.internal (compute6.nyi.internal [10.202.2.46])
 by mailout.nyi.internal (Postfix) with ESMTP id C323420720;
 Thu,  4 Feb 2016 18:17:16 -0500 (EST)
Received: from frontend2 ([10.202.2.161])
 by compute6.internal (MEProxy); Thu, 04 Feb 2016 18:17:16 -0500
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=famulari.name; h=
 cc:content-transfer-encoding:content-type:date:from:in-reply-to
 :message-id:mime-version:references:subject:to:x-sasl-enc
 :x-sasl-enc; s=mesmtp; bh=iYp5UYcGOiilUrakLqJxWPAohj8=; b=oe7+1Z
 O5RjR9NTqA0P7TfJjw/pi10gHcELe2OEFVbvY4RrKYtfvHndlA85RaLlOK7EZsBH
 OPrXXb1Ov81O+PfLc5RhcI+uknWV++KOtKLnOQqqy+2yF4QRxrzEon1IBejazi+I
 HMR0mi3dDJCz5hRQx0XBMlNI9jr44m/evCP2w=
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:content-transfer-encoding:content-type
 :date:from:in-reply-to:message-id:mime-version:references
 :subject:to:x-sasl-enc:x-sasl-enc; s=smtpout; bh=iYp5UYcGOiilUra
 kLqJxWPAohj8=; b=ge/i+pom4MHNYNooRgoCfsDAd1lsXhxw3+najyXn++4hgyn
 5KUa6l94RWZmsPDr1HDC/oe2ND80zc3ttah5XbEGTbuAz36h5yh3CpId+ASMHHDg
 asmt3nGOiZteoWOicuw/fVui0xPZ97c8scqEr3sIBp/1EX3rFv9d438YR5FY=
X-Sasl-enc: KHVm3y2NVYUuKBXBK2QYdNubSfyUeNru07B6t6kXhHk0 1454627834
Received: from localhost (c-73-233-160-251.hsd1.pa.comcast.net
 [73.233.160.251])
 by mail.messagingengine.com (Postfix) with ESMTPA id EF279680109;
 Thu,  4 Feb 2016 18:17:14 -0500 (EST)
Date: Thu, 4 Feb 2016 18:17:08 -0500
From: Leo Famulari <leo@HIDDEN>
To: Ludovic =?iso-8859-1?Q?Court=E8s?= <ludo@HIDDEN>
Subject: Re: bug#22533: Non-determinism in python-3 ".pyc" bytecode
Message-ID: <20160204231708.GA1297@jasmine>
References: <20160202051544.GA11744@jasmine>
 <87powezvyo.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="LQksG6bCIzRHxTLp"
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <87powezvyo.fsf@HIDDEN>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Spam-Score: -0.7 (/)
X-Debbugs-Envelope-To: 22533
Cc: 22533 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.7 (/)


--LQksG6bCIzRHxTLp
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
Content-Transfer-Encoding: 8bit

On Tue, Feb 02, 2016 at 09:41:19PM +0100, Ludovic Courts wrote:
> Could you give it a try and refine as needed?  :-)

I altered your example as shown in the attached patch. It causes some
tests related to timestamps to fail, so I disabled them in a very crude
way. The final patch should address those tests more carefully.

But, the patch doesn't seem to have the desired effect so I'm asking for
help!

Here is how I tested the patch:

I build python-3 with it, and then `export SOURCE_DATE_EPOCH=1` and
enter the resulting Python shell. I manually define the '_w_long'
function used by the patched function. Then: 

print (_w_long(locale.atoi(os.getenv('SOURCE_DATE_EPOCH'))))
b'\x01\x00\x00\x00'

But, when I leave the Python shell and issue `python3 -m compileall
helloworld.py`, the timestamps are present in the compiled bytecode. I
can watch the clock "tick" by doing this repeatedly:

$ touch helloworld.py && rm -r __pycache__ && \
python3 -m compileall helloworld.py &&  \
hexdump __pycache__/helloworld.cpython-34.pyc | head -n1

I'm not much of a Python programmer, so I'm stumped.

--LQksG6bCIzRHxTLp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment; filename="0001-SOURCE_DATE_EPOCH.patch"

From d34a71e4ec4501cb53acd3e15633bc1a05665be9 Mon Sep 17 00:00:00 2001
Message-Id: <d34a71e4ec4501cb53acd3e15633bc1a05665be9.1454625404.git.leo@HIDDEN>
From: Leo Famulari <leo@HIDDEN>
Date: Wed, 3 Feb 2016 20:44:02 -0500
Subject: [PATCH 1/1] SOURCE_DATE_EPOCH

---
 .../patches/python-3.4.3-source-date-epoch.patch    | 21 +++++++++++++++++++++
 gnu/packages/python.scm                             | 14 +++++++++++++-
 2 files changed, 34 insertions(+), 1 deletion(-)
 create mode 100644 gnu/packages/patches/python-3.4.3-source-date-epoch.patch

diff --git a/gnu/packages/patches/python-3.4.3-source-date-epoch.patch b/gnu/packages/patches/python-3.4.3-source-date-epoch.patch
new file mode 100644
index 0000000..403b2df
--- /dev/null
+++ b/gnu/packages/patches/python-3.4.3-source-date-epoch.patch
@@ -0,0 +1,21 @@
+diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py
+index 5b91c05..a87d178 100644
+--- Lib/importlib/_bootstrap.py
++++ Lib/importlib/_bootstrap.py
+@@ -666,8 +666,15 @@ def _compile_bytecode(data, name=None, bytecode_path=None, source_path=None):
+ def _code_to_bytecode(code, mtime=0, source_size=0):
+     """Compile a code object into bytecode for writing out to a byte-compiled
+     file."""
++    """os and locale are required for the SOURCE_DATE_EPOCH
++    deterministic timestamp conditional."""
++    import os
++    import locale
+     data = bytearray(MAGIC_NUMBER)
+-    data.extend(_w_long(mtime))
++    if os.getenv('SOURCE_DATE_EPOCH'):
++        data.extend(_w_long(locale.atoi(os.getenv('SOURCE_DATE_EPOCH'))))
++    else:
++        data.extend(_w_long(mtime))
+     data.extend(_w_long(source_size))
+     data.extend(marshal.dumps(code))
+     return data
diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index 48f65b5..cd366f5 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -173,6 +173,17 @@
              ;; gnu-build-system.scm.
              (setenv "SOURCE_DATE_EPOCH" "1")
              #t))
+          (add-before 'configure 'disable-timestamp-tests
+            (lambda _
+              ;; Filter for existing files, since this only affects
+              ;; Python-3 if the SOURCE_DATE_EPOCH patch is applied.
+              (substitute* (filter file-exists?
+                                   '("Lib/test/test_importlib/test_abc.py"))
+                           (("test_code_bad_timestamp") "disable_test_code_bad_timestamp"))
+              (substitute* (filter file-exists?
+                                   '("Lib/test/test_importlib/source/test_file_loader.py"))
+                           (("test_old_timestamp") "disable_test_old_timestamp"))
+              ))
           (add-before 'configure 'do-not-record-configure-flags
             (lambda* (#:key configure-flags #:allow-other-keys)
               ;; Remove configure flags from the installed '_sysconfigdata.py'
@@ -268,7 +279,8 @@ data types.")
                               ;; XXX Try removing this patch for python > 3.4.3
                               "python-disable-ssl-test.patch"
                               "python-3-deterministic-build-info.patch"
-                              "python-3-search-paths.patch")))
+                              "python-3-search-paths.patch"
+                              "python-3.4.3-source-date-epoch.patch")))
               (patch-flags '("-p0"))
               (sha256
                (base32
-- 
2.6.3


--LQksG6bCIzRHxTLp--




Information forwarded to bug-guix@HIDDEN:
bug#22533; Package guix. Full text available.

Message received at 22533 <at> debbugs.gnu.org:


Received: (at 22533) by debbugs.gnu.org; 2 Feb 2016 20:41:33 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Feb 02 15:41:33 2016
Received: from localhost ([127.0.0.1]:57143 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1aQhlV-0004Ki-IL
	for submit <at> debbugs.gnu.org; Tue, 02 Feb 2016 15:41:33 -0500
Received: from eggs.gnu.org ([208.118.235.92]:56669)
 by debbugs.gnu.org with esmtp (Exim 4.84)
 (envelope-from <ludo@HIDDEN>) id 1aQhlT-0004KW-T6
 for 22533 <at> debbugs.gnu.org; Tue, 02 Feb 2016 15:41:32 -0500
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <ludo@HIDDEN>) id 1aQhlK-0000bN-N1
 for 22533 <at> debbugs.gnu.org; Tue, 02 Feb 2016 15:41:26 -0500
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=-2.3 required=5.0 tests=BAYES_00,RP_MATCHES_RCVD
 autolearn=disabled version=3.3.2
Received: from fencepost.gnu.org ([2001:4830:134:3::e]:59431)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <ludo@HIDDEN>)
 id 1aQhlK-0000bJ-IY; Tue, 02 Feb 2016 15:41:22 -0500
Received: from reverse-83.fdn.fr ([80.67.176.83]:48172 helo=pluto)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_128_CBC_SHA1:128)
 (Exim 4.82) (envelope-from <ludo@HIDDEN>)
 id 1aQhlJ-0006gH-W0; Tue, 02 Feb 2016 15:41:22 -0500
From: ludo@HIDDEN (Ludovic =?utf-8?Q?Court=C3=A8s?=)
To: Leo Famulari <leo@HIDDEN>
Subject: Re: bug#22533: Non-determinism in python-3 ".pyc" bytecode
References: <20160202051544.GA11744@jasmine>
X-URL: http://www.fdn.fr/~lcourtes/
X-Revolutionary-Date: 14 =?utf-8?Q?Pluvi=C3=B4se?= an 224 de la
 =?utf-8?Q?R=C3=A9volution?=
X-PGP-Key-ID: 0x3D9AEBB5
X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc
X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4  0CFB 090B 1199 3D9A EBB5
X-OS: x86_64-unknown-linux-gnu
Date: Tue, 02 Feb 2016 21:41:19 +0100
In-Reply-To: <20160202051544.GA11744@jasmine> (Leo Famulari's message of "Tue, 
 2 Feb 2016 00:15:44 -0500")
Message-ID: <87powezvyo.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux)
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="=-=-="
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2001:4830:134:3::e
X-Spam-Score: -5.5 (-----)
X-Debbugs-Envelope-To: 22533
Cc: 22533 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -5.5 (-----)

--=-=-=
Content-Type: text/plain

Leo Famulari <leo@HIDDEN> skribis:

> We fixed this in python-2 with the patch
> python-2.7-source-date-epoch.patch, but I don't know how to write this
> patch for python-3.

I would imagine something like this (untested):


--=-=-=
Content-Type: text/x-patch
Content-Disposition: inline

--- Python-3.4.3/Lib/importlib/_bootstrap.py	2016-02-02 21:38:48.655809055 +0100
+++ Python-3.4.3/Lib/importlib/_bootstrap.py.new	2016-02-02 21:38:43.659769251 +0100
@@ -667,7 +667,10 @@ def _code_to_bytecode(code, mtime=0, sou
     """Compile a code object into bytecode for writing out to a byte-compiled
     file."""
     data = bytearray(MAGIC_NUMBER)
-    data.extend(_w_long(mtime))
+    if 'SOURCE_DATE_EPOCH' in _os.environ:
+        data.extend(_w_long(string.atoi(_os.environ['SOURCE_DATE_EPOCH'])))
+    else:
+        data.extend(_w_long(mtime))
     data.extend(_w_long(source_size))
     data.extend(marshal.dumps(code))
     return data

--=-=-=
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


Could you give it a try and refine as needed?  :-)

> I asked about this on #debian-reproducible and they said that it wasn't
> an issue for Debian since they don't ship bytecode, but instead generate
> it at install time. Of course, that doesn't really apply to Guix.

I=E2=80=99d recommend trying #reproducible-builds on OFTC, which is more
generic.  Also, in some cases, it=E2=80=99s useful to look at
<git://git.debian.org/git/reproducible/notes.git>, which contains notes
about non-reproducible packages (currently partly Debian-specific, but
we need to lobby to make it more generic.  ;-))

Thanks,
Ludo=E2=80=99.

--=-=-=--




Information forwarded to bug-guix@HIDDEN:
bug#22533; Package guix. Full text available.

Message received at 22533 <at> debbugs.gnu.org:


Received: (at 22533) by debbugs.gnu.org; 2 Feb 2016 08:54:42 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Feb 02 03:54:42 2016
Received: from localhost ([127.0.0.1]:55658 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1aQWjS-0003VB-B9
	for submit <at> debbugs.gnu.org; Tue, 02 Feb 2016 03:54:42 -0500
Received: from out4-smtp.messagingengine.com ([66.111.4.28]:50073)
 by debbugs.gnu.org with esmtp (Exim 4.84)
 (envelope-from <leo@HIDDEN>) id 1aQWjQ-0003V3-Hb
 for 22533 <at> debbugs.gnu.org; Tue, 02 Feb 2016 03:54:41 -0500
Received: from compute1.internal (compute1.nyi.internal [10.202.2.41])
 by mailout.nyi.internal (Postfix) with ESMTP id C6AD9205DE;
 Tue,  2 Feb 2016 03:54:39 -0500 (EST)
Received: from frontend2 ([10.202.2.161])
 by compute1.internal (MEProxy); Tue, 02 Feb 2016 03:54:39 -0500
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=famulari.name; h=
 content-type:date:from:in-reply-to:message-id:mime-version
 :references:subject:to:x-sasl-enc:x-sasl-enc; s=mesmtp; bh=L2HKc
 E4TzcUeANp+yNRUPyNkbcE=; b=UawYEHZNfIztpeUFwt6K065k9JH28l1hsTwTU
 eoUHzMcZ2m1lEmSa3GVffKZcoLcQ4wlgazm/SYZsl6eI+aRjqPgMYd3pLMMeruy1
 jXflT2j2lRC+sgzCXTTs/OayQ1Qh+VAScc9+G7uQ4ekxCcEKZzwH59pH+RlTVKEU
 u7zPSQ=
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=
 messagingengine.com; h=content-type:date:from:in-reply-to
 :message-id:mime-version:references:subject:to:x-sasl-enc
 :x-sasl-enc; s=smtpout; bh=L2HKcE4TzcUeANp+yNRUPyNkbcE=; b=Kq4JM
 eDYYkWRsr6ePZdAEDjMPMPI0YmVhLIJ28NEodi25Tj45ckKi8jEkLhg9HIRjglN9
 sL4pvGAWWQsvd4J9hzgh2EWBtwB+nsWLLCC8kO2mMffm0oxJJw5d5fW4jS9qAr+Z
 +J247JrWBfmIuvHGooQd3t4C4djDlUhRyfHt0A=
X-Sasl-enc: d3pgw1jdNMW+kL3KrW51ivi7grGZ8EDPVmwrIzr+Hz8z 1454403279
Received: from localhost (c-69-249-5-231.hsd1.pa.comcast.net [69.249.5.231])
 by mail.messagingengine.com (Postfix) with ESMTPA id 7D74F68010F
 for <22533 <at> debbugs.gnu.org>; Tue,  2 Feb 2016 03:54:39 -0500 (EST)
Date: Tue, 2 Feb 2016 03:54:39 -0500
From: Leo Famulari <leo@HIDDEN>
To: 22533 <at> debbugs.gnu.org
Subject: Re: bug#22533: Non-determinism in python-3 ".pyc" bytecode
Message-ID: <20160202085439.GA14802@jasmine>
References: <20160202051544.GA11744@jasmine>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160202051544.GA11744@jasmine>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Spam-Score: -0.7 (/)
X-Debbugs-Envelope-To: 22533
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.7 (/)

On Tue, Feb 02, 2016 at 12:15:44AM -0500, Leo Famulari wrote:
> While preparing a package for borg [0], I found that the built output
> was not reproducible. The problem is that the bytecode compiler [1] for
> Python 3.4.3 (our current version) encodes the mtime of the
> corresponding Python source file in the output. This is described in
> PEP-3147 [2], and the responsible Python code is referenced below [3].
> 
> I tested a few of our existing python-3 packages: python-ccm,
> python-pysam, and python-scripttest all exhibit the same problem.
> 
> We fixed this in python-2 with the patch
> python-2.7-source-date-epoch.patch, but I don't know how to write this
> patch for python-3.

mark_weaver suggested setting the timestamps of the source files before
building. I think this is a better option if it doesn't break anything.
It would allow the bytecode "staleness" check to work as expected while
keeping the output consistent.

> 
> Can somebody write this patch?
> 
> I asked about this on #debian-reproducible and they said that it wasn't
> an issue for Debian since they don't ship bytecode, but instead generate
> it at install time. Of course, that doesn't really apply to Guix.
> 
> I used diffoscope-34 to inspect the build outputs to find this, and you
> can see the report here:
> https://famulari.name/misc/7c55c9e97f668234ddea50299d986f14/borg-diffoscope-report.html
> 
> It's first demonstrated in the file
> ...-borg-0.30.0/lib/python3.4/site-packages/__pycache__/site.cpython-34.pyc.
> 
> The first 2 bytes are the "magic numbers" described in PEP-3147, which
> specify the version of the bytecode format. The next 2 bytes are the
> problematic timestamp, as described in the PEP-3147.
> 
> [0]
> http://borgbackup.github.io/
> 
> [1]
> https://docs.python.org/3/library/py_compile.html
> 
> [2]
> https://www.python.org/dev/peps/pep-3147/
> 
> [3] Check out the Guix git commit 4efc8eb27502c, and from there:
> $ tar xf $(./pre-inst-env guix build --source python-3)
> $ sed -n 139,140p Python-3.4.3/Lib/py_compile.py
>     bytecode = importlib._bootstrap._code_to_bytecode(
>             code, source_stats['mtime'], source_stats['size'])
> 
> 
> 




Information forwarded to bug-guix@HIDDEN:
bug#22533; Package guix. Full text available.

Message received at submit <at> debbugs.gnu.org:


Received: (at submit) by debbugs.gnu.org; 2 Feb 2016 05:16:03 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Feb 02 00:16:03 2016
Received: from localhost ([127.0.0.1]:55601 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1aQTJn-0006fy-F4
	for submit <at> debbugs.gnu.org; Tue, 02 Feb 2016 00:16:03 -0500
Received: from eggs.gnu.org ([208.118.235.92]:54384)
 by debbugs.gnu.org with esmtp (Exim 4.84)
 (envelope-from <leo@HIDDEN>) id 1aQTJm-0006fl-3x
 for submit <at> debbugs.gnu.org; Tue, 02 Feb 2016 00:15:58 -0500
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <leo@HIDDEN>) id 1aQTJf-0004vt-Ow
 for submit <at> debbugs.gnu.org; Tue, 02 Feb 2016 00:15:52 -0500
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=-0.5 required=5.0 tests=BAYES_05,T_DKIM_INVALID
 autolearn=disabled version=3.3.2
Received: from lists.gnu.org ([2001:4830:134:3::11]:46320)
 by eggs.gnu.org with esmtp (Exim 4.71)
 (envelope-from <leo@HIDDEN>) id 1aQTJf-0004vp-M9
 for submit <at> debbugs.gnu.org; Tue, 02 Feb 2016 00:15:51 -0500
Received: from eggs.gnu.org ([2001:4830:134:3::10]:45355)
 by lists.gnu.org with esmtp (Exim 4.71)
 (envelope-from <leo@HIDDEN>) id 1aQTJe-0008R3-HA
 for bug-guix@HIDDEN; Tue, 02 Feb 2016 00:15:51 -0500
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <leo@HIDDEN>) id 1aQTJa-0004ve-GU
 for bug-guix@HIDDEN; Tue, 02 Feb 2016 00:15:50 -0500
Received: from out1-smtp.messagingengine.com ([66.111.4.25]:43960)
 by eggs.gnu.org with esmtp (Exim 4.71)
 (envelope-from <leo@HIDDEN>) id 1aQTJZ-0004va-TZ
 for bug-guix@HIDDEN; Tue, 02 Feb 2016 00:15:46 -0500
Received: from compute1.internal (compute1.nyi.internal [10.202.2.41])
 by mailout.nyi.internal (Postfix) with ESMTP id 7051C2008E;
 Tue,  2 Feb 2016 00:15:45 -0500 (EST)
Received: from frontend1 ([10.202.2.160])
 by compute1.internal (MEProxy); Tue, 02 Feb 2016 00:15:45 -0500
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=famulari.name; h=
 content-type:date:from:message-id:mime-version:subject:to
 :x-sasl-enc:x-sasl-enc; s=mesmtp; bh=shM1vEUSai5utsd2gGa5cQtc4Fg
 =; b=XrqYcbRelhKGYhDSkSy4x/b9cl3Q5ka0J3IQ6NTZK5DhylzHGbzSfNiTY3T
 pztXI4byn0520DZFwG9w6CNvO/lU9FIO9SXzo+Q5fDqzlQzmlXq0EpmJKBz23xyV
 9UnEVTGLQ4nHprEqT636c3OaiG7GSi34TXOcS/odY0q+r1Hg=
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=
 messagingengine.com; h=content-type:date:from:message-id
 :mime-version:subject:to:x-sasl-enc:x-sasl-enc; s=smtpout; bh=sh
 M1vEUSai5utsd2gGa5cQtc4Fg=; b=pnfI2paP0e5ceG3bZRr/w3dfJLOP67dTkc
 GesfihhlJ40LbK++Qq97OuLoxkYgCMaxvpVK75Onx2MHN6s+yKSntz++10HfCJNK
 ONeMPeEaRlhNu7mu7bW3n5lWAWmc9Yj2TAt0zVNFljxtTFe75oelPhahwZwEWXR1
 foYdqX/3c=
X-Sasl-enc: BKrZe55VqbINpIbpEwCRGdYTAFb+aV2/Bn6yKfxLsfDS 1454390145
Received: from localhost (c-69-249-5-231.hsd1.pa.comcast.net [69.249.5.231])
 by mail.messagingengine.com (Postfix) with ESMTPA id 1C742C0001A
 for <bug-guix@HIDDEN>; Tue,  2 Feb 2016 00:15:45 -0500 (EST)
Date: Tue, 2 Feb 2016 00:15:44 -0500
From: Leo Famulari <leo@HIDDEN>
To: bug-guix@HIDDEN
Subject: Non-determinism in python-3 ".pyc" bytecode
Message-ID: <20160202051544.GA11744@jasmine>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.5.24 (2015-08-30)
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x
X-Received-From: 2001:4830:134:3::11
X-Spam-Score: -4.1 (----)
X-Debbugs-Envelope-To: submit
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: 0.9 (/)

While preparing a package for borg [0], I found that the built output
was not reproducible. The problem is that the bytecode compiler [1] for
Python 3.4.3 (our current version) encodes the mtime of the
corresponding Python source file in the output. This is described in
PEP-3147 [2], and the responsible Python code is referenced below [3].

I tested a few of our existing python-3 packages: python-ccm,
python-pysam, and python-scripttest all exhibit the same problem.

We fixed this in python-2 with the patch
python-2.7-source-date-epoch.patch, but I don't know how to write this
patch for python-3.

Can somebody write this patch?

I asked about this on #debian-reproducible and they said that it wasn't
an issue for Debian since they don't ship bytecode, but instead generate
it at install time. Of course, that doesn't really apply to Guix.

I used diffoscope-34 to inspect the build outputs to find this, and you
can see the report here:
https://famulari.name/misc/7c55c9e97f668234ddea50299d986f14/borg-diffoscope-report.html

It's first demonstrated in the file
...-borg-0.30.0/lib/python3.4/site-packages/__pycache__/site.cpython-34.pyc.

The first 2 bytes are the "magic numbers" described in PEP-3147, which
specify the version of the bytecode format. The next 2 bytes are the
problematic timestamp, as described in the PEP-3147.

[0]
http://borgbackup.github.io/

[1]
https://docs.python.org/3/library/py_compile.html

[2]
https://www.python.org/dev/peps/pep-3147/

[3] Check out the Guix git commit 4efc8eb27502c, and from there:
$ tar xf $(./pre-inst-env guix build --source python-3)
$ sed -n 139,140p Python-3.4.3/Lib/py_compile.py
    bytecode = importlib._bootstrap._code_to_bytecode(
            code, source_stats['mtime'], source_stats['size'])




Acknowledgement sent to Leo Famulari <leo@HIDDEN>:
New bug report received and forwarded. Copy sent to bug-guix@HIDDEN. Full text available.
Report forwarded to bug-guix@HIDDEN:
bug#22533; Package guix. Full text available.
Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.
Last modified: Mon, 4 Feb 2019 22:45:01 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.