GNU bug report logs - #22533
Non-determinism in python-3 ".pyc" bytecode

Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.

Package: guix; Severity: important; Reported by: Leo Famulari <leo@HIDDEN>; dated Tue, 2 Feb 2016 05:17:02 UTC; Maintainer for guix is bug-guix@HIDDEN.
Severity set to 'important' from 'normal' Request was from ludo@HIDDEN (Ludovic Courtès) to control <at> debbugs.gnu.org. Full text available.

Message received at 22533 <at> debbugs.gnu.org:


Received: (at 22533) by debbugs.gnu.org; 4 Feb 2016 23:17:20 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Feb 04 18:17:19 2016
Received: from localhost ([127.0.0.1]:60729 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1aRT9L-0007bc-J2
	for submit <at> debbugs.gnu.org; Thu, 04 Feb 2016 18:17:19 -0500
Received: from out3-smtp.messagingengine.com ([66.111.4.27]:57913)
 by debbugs.gnu.org with esmtp (Exim 4.84)
 (envelope-from <leo@HIDDEN>) id 1aRT9J-0007bS-7c
 for 22533 <at> debbugs.gnu.org; Thu, 04 Feb 2016 18:17:18 -0500
Received: from compute6.internal (compute6.nyi.internal [10.202.2.46])
 by mailout.nyi.internal (Postfix) with ESMTP id C323420720;
 Thu,  4 Feb 2016 18:17:16 -0500 (EST)
Received: from frontend2 ([10.202.2.161])
 by compute6.internal (MEProxy); Thu, 04 Feb 2016 18:17:16 -0500
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=famulari.name; h=
 cc:content-transfer-encoding:content-type:date:from:in-reply-to
 :message-id:mime-version:references:subject:to:x-sasl-enc
 :x-sasl-enc; s=mesmtp; bh=iYp5UYcGOiilUrakLqJxWPAohj8=; b=oe7+1Z
 O5RjR9NTqA0P7TfJjw/pi10gHcELe2OEFVbvY4RrKYtfvHndlA85RaLlOK7EZsBH
 OPrXXb1Ov81O+PfLc5RhcI+uknWV++KOtKLnOQqqy+2yF4QRxrzEon1IBejazi+I
 HMR0mi3dDJCz5hRQx0XBMlNI9jr44m/evCP2w=
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:content-transfer-encoding:content-type
 :date:from:in-reply-to:message-id:mime-version:references
 :subject:to:x-sasl-enc:x-sasl-enc; s=smtpout; bh=iYp5UYcGOiilUra
 kLqJxWPAohj8=; b=ge/i+pom4MHNYNooRgoCfsDAd1lsXhxw3+najyXn++4hgyn
 5KUa6l94RWZmsPDr1HDC/oe2ND80zc3ttah5XbEGTbuAz36h5yh3CpId+ASMHHDg
 asmt3nGOiZteoWOicuw/fVui0xPZ97c8scqEr3sIBp/1EX3rFv9d438YR5FY=
X-Sasl-enc: KHVm3y2NVYUuKBXBK2QYdNubSfyUeNru07B6t6kXhHk0 1454627834
Received: from localhost (c-73-233-160-251.hsd1.pa.comcast.net
 [73.233.160.251])
 by mail.messagingengine.com (Postfix) with ESMTPA id EF279680109;
 Thu,  4 Feb 2016 18:17:14 -0500 (EST)
Date: Thu, 4 Feb 2016 18:17:08 -0500
From: Leo Famulari <leo@HIDDEN>
To: Ludovic =?iso-8859-1?Q?Court=E8s?= <ludo@HIDDEN>
Subject: Re: bug#22533: Non-determinism in python-3 ".pyc" bytecode
Message-ID: <20160204231708.GA1297@jasmine>
References: <20160202051544.GA11744@jasmine>
 <87powezvyo.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="LQksG6bCIzRHxTLp"
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <87powezvyo.fsf@HIDDEN>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Spam-Score: -0.7 (/)
X-Debbugs-Envelope-To: 22533
Cc: 22533 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.7 (/)


--LQksG6bCIzRHxTLp
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
Content-Transfer-Encoding: 8bit

On Tue, Feb 02, 2016 at 09:41:19PM +0100, Ludovic Courts wrote:
> Could you give it a try and refine as needed?  :-)

I altered your example as shown in the attached patch. It causes some
tests related to timestamps to fail, so I disabled them in a very crude
way. The final patch should address those tests more carefully.

But, the patch doesn't seem to have the desired effect so I'm asking for
help!

Here is how I tested the patch:

I build python-3 with it, and then `export SOURCE_DATE_EPOCH=1` and
enter the resulting Python shell. I manually define the '_w_long'
function used by the patched function. Then: 

print (_w_long(locale.atoi(os.getenv('SOURCE_DATE_EPOCH'))))
b'\x01\x00\x00\x00'

But, when I leave the Python shell and issue `python3 -m compileall
helloworld.py`, the timestamps are present in the compiled bytecode. I
can watch the clock "tick" by doing this repeatedly:

$ touch helloworld.py && rm -r __pycache__ && \
python3 -m compileall helloworld.py &&  \
hexdump __pycache__/helloworld.cpython-34.pyc | head -n1

I'm not much of a Python programmer, so I'm stumped.

--LQksG6bCIzRHxTLp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment; filename="0001-SOURCE_DATE_EPOCH.patch"

From d34a71e4ec4501cb53acd3e15633bc1a05665be9 Mon Sep 17 00:00:00 2001
Message-Id: <d34a71e4ec4501cb53acd3e15633bc1a05665be9.1454625404.git.leo@HIDDEN>
From: Leo Famulari <leo@HIDDEN>
Date: Wed, 3 Feb 2016 20:44:02 -0500
Subject: [PATCH 1/1] SOURCE_DATE_EPOCH

---
 .../patches/python-3.4.3-source-date-epoch.patch    | 21 +++++++++++++++++++++
 gnu/packages/python.scm                             | 14 +++++++++++++-
 2 files changed, 34 insertions(+), 1 deletion(-)
 create mode 100644 gnu/packages/patches/python-3.4.3-source-date-epoch.patch

diff --git a/gnu/packages/patches/python-3.4.3-source-date-epoch.patch b/gnu/packages/patches/python-3.4.3-source-date-epoch.patch
new file mode 100644
index 0000000..403b2df
--- /dev/null
+++ b/gnu/packages/patches/python-3.4.3-source-date-epoch.patch
@@ -0,0 +1,21 @@
+diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py
+index 5b91c05..a87d178 100644
+--- Lib/importlib/_bootstrap.py
++++ Lib/importlib/_bootstrap.py
+@@ -666,8 +666,15 @@ def _compile_bytecode(data, name=None, bytecode_path=None, source_path=None):
+ def _code_to_bytecode(code, mtime=0, source_size=0):
+     """Compile a code object into bytecode for writing out to a byte-compiled
+     file."""
++    """os and locale are required for the SOURCE_DATE_EPOCH
++    deterministic timestamp conditional."""
++    import os
++    import locale
+     data = bytearray(MAGIC_NUMBER)
+-    data.extend(_w_long(mtime))
++    if os.getenv('SOURCE_DATE_EPOCH'):
++        data.extend(_w_long(locale.atoi(os.getenv('SOURCE_DATE_EPOCH'))))
++    else:
++        data.extend(_w_long(mtime))
+     data.extend(_w_long(source_size))
+     data.extend(marshal.dumps(code))
+     return data
diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index 48f65b5..cd366f5 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -173,6 +173,17 @@
              ;; gnu-build-system.scm.
              (setenv "SOURCE_DATE_EPOCH" "1")
              #t))
+          (add-before 'configure 'disable-timestamp-tests
+            (lambda _
+              ;; Filter for existing files, since this only affects
+              ;; Python-3 if the SOURCE_DATE_EPOCH patch is applied.
+              (substitute* (filter file-exists?
+                                   '("Lib/test/test_importlib/test_abc.py"))
+                           (("test_code_bad_timestamp") "disable_test_code_bad_timestamp"))
+              (substitute* (filter file-exists?
+                                   '("Lib/test/test_importlib/source/test_file_loader.py"))
+                           (("test_old_timestamp") "disable_test_old_timestamp"))
+              ))
           (add-before 'configure 'do-not-record-configure-flags
             (lambda* (#:key configure-flags #:allow-other-keys)
               ;; Remove configure flags from the installed '_sysconfigdata.py'
@@ -268,7 +279,8 @@ data types.")
                               ;; XXX Try removing this patch for python > 3.4.3
                               "python-disable-ssl-test.patch"
                               "python-3-deterministic-build-info.patch"
-                              "python-3-search-paths.patch")))
+                              "python-3-search-paths.patch"
+                              "python-3.4.3-source-date-epoch.patch")))
               (patch-flags '("-p0"))
               (sha256
                (base32
-- 
2.6.3


--LQksG6bCIzRHxTLp--




Information forwarded to bug-guix@HIDDEN:
bug#22533; Package guix. Full text available.

Message received at 22533 <at> debbugs.gnu.org:


Received: (at 22533) by debbugs.gnu.org; 2 Feb 2016 20:41:33 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Feb 02 15:41:33 2016
Received: from localhost ([127.0.0.1]:57143 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1aQhlV-0004Ki-IL
	for submit <at> debbugs.gnu.org; Tue, 02 Feb 2016 15:41:33 -0500
Received: from eggs.gnu.org ([208.118.235.92]:56669)
 by debbugs.gnu.org with esmtp (Exim 4.84)
 (envelope-from <ludo@HIDDEN>) id 1aQhlT-0004KW-T6
 for 22533 <at> debbugs.gnu.org; Tue, 02 Feb 2016 15:41:32 -0500
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <ludo@HIDDEN>) id 1aQhlK-0000bN-N1
 for 22533 <at> debbugs.gnu.org; Tue, 02 Feb 2016 15:41:26 -0500
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=-2.3 required=5.0 tests=BAYES_00,RP_MATCHES_RCVD
 autolearn=disabled version=3.3.2
Received: from fencepost.gnu.org ([2001:4830:134:3::e]:59431)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <ludo@HIDDEN>)
 id 1aQhlK-0000bJ-IY; Tue, 02 Feb 2016 15:41:22 -0500
Received: from reverse-83.fdn.fr ([80.67.176.83]:48172 helo=pluto)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_128_CBC_SHA1:128)
 (Exim 4.82) (envelope-from <ludo@HIDDEN>)
 id 1aQhlJ-0006gH-W0; Tue, 02 Feb 2016 15:41:22 -0500
From: ludo@HIDDEN (Ludovic =?utf-8?Q?Court=C3=A8s?=)
To: Leo Famulari <leo@HIDDEN>
Subject: Re: bug#22533: Non-determinism in python-3 ".pyc" bytecode
References: <20160202051544.GA11744@jasmine>
X-URL: http://www.fdn.fr/~lcourtes/
X-Revolutionary-Date: 14 =?utf-8?Q?Pluvi=C3=B4se?= an 224 de la
 =?utf-8?Q?R=C3=A9volution?=
X-PGP-Key-ID: 0x3D9AEBB5
X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc
X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4  0CFB 090B 1199 3D9A EBB5
X-OS: x86_64-unknown-linux-gnu
Date: Tue, 02 Feb 2016 21:41:19 +0100
In-Reply-To: <20160202051544.GA11744@jasmine> (Leo Famulari's message of "Tue, 
 2 Feb 2016 00:15:44 -0500")
Message-ID: <87powezvyo.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux)
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="=-=-="
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2001:4830:134:3::e
X-Spam-Score: -5.5 (-----)
X-Debbugs-Envelope-To: 22533
Cc: 22533 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -5.5 (-----)

--=-=-=
Content-Type: text/plain

Leo Famulari <leo@HIDDEN> skribis:

> We fixed this in python-2 with the patch
> python-2.7-source-date-epoch.patch, but I don't know how to write this
> patch for python-3.

I would imagine something like this (untested):


--=-=-=
Content-Type: text/x-patch
Content-Disposition: inline

--- Python-3.4.3/Lib/importlib/_bootstrap.py	2016-02-02 21:38:48.655809055 +0100
+++ Python-3.4.3/Lib/importlib/_bootstrap.py.new	2016-02-02 21:38:43.659769251 +0100
@@ -667,7 +667,10 @@ def _code_to_bytecode(code, mtime=0, sou
     """Compile a code object into bytecode for writing out to a byte-compiled
     file."""
     data = bytearray(MAGIC_NUMBER)
-    data.extend(_w_long(mtime))
+    if 'SOURCE_DATE_EPOCH' in _os.environ:
+        data.extend(_w_long(string.atoi(_os.environ['SOURCE_DATE_EPOCH'])))
+    else:
+        data.extend(_w_long(mtime))
     data.extend(_w_long(source_size))
     data.extend(marshal.dumps(code))
     return data

--=-=-=
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


Could you give it a try and refine as needed?  :-)

> I asked about this on #debian-reproducible and they said that it wasn't
> an issue for Debian since they don't ship bytecode, but instead generate
> it at install time. Of course, that doesn't really apply to Guix.

I=E2=80=99d recommend trying #reproducible-builds on OFTC, which is more
generic.  Also, in some cases, it=E2=80=99s useful to look at
<git://git.debian.org/git/reproducible/notes.git>, which contains notes
about non-reproducible packages (currently partly Debian-specific, but
we need to lobby to make it more generic.  ;-))

Thanks,
Ludo=E2=80=99.

--=-=-=--




Information forwarded to bug-guix@HIDDEN:
bug#22533; Package guix. Full text available.

Message received at 22533 <at> debbugs.gnu.org:


Received: (at 22533) by debbugs.gnu.org; 2 Feb 2016 08:54:42 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Feb 02 03:54:42 2016
Received: from localhost ([127.0.0.1]:55658 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1aQWjS-0003VB-B9
	for submit <at> debbugs.gnu.org; Tue, 02 Feb 2016 03:54:42 -0500
Received: from out4-smtp.messagingengine.com ([66.111.4.28]:50073)
 by debbugs.gnu.org with esmtp (Exim 4.84)
 (envelope-from <leo@HIDDEN>) id 1aQWjQ-0003V3-Hb
 for 22533 <at> debbugs.gnu.org; Tue, 02 Feb 2016 03:54:41 -0500
Received: from compute1.internal (compute1.nyi.internal [10.202.2.41])
 by mailout.nyi.internal (Postfix) with ESMTP id C6AD9205DE;
 Tue,  2 Feb 2016 03:54:39 -0500 (EST)
Received: from frontend2 ([10.202.2.161])
 by compute1.internal (MEProxy); Tue, 02 Feb 2016 03:54:39 -0500
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=famulari.name; h=
 content-type:date:from:in-reply-to:message-id:mime-version
 :references:subject:to:x-sasl-enc:x-sasl-enc; s=mesmtp; bh=L2HKc
 E4TzcUeANp+yNRUPyNkbcE=; b=UawYEHZNfIztpeUFwt6K065k9JH28l1hsTwTU
 eoUHzMcZ2m1lEmSa3GVffKZcoLcQ4wlgazm/SYZsl6eI+aRjqPgMYd3pLMMeruy1
 jXflT2j2lRC+sgzCXTTs/OayQ1Qh+VAScc9+G7uQ4ekxCcEKZzwH59pH+RlTVKEU
 u7zPSQ=
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=
 messagingengine.com; h=content-type:date:from:in-reply-to
 :message-id:mime-version:references:subject:to:x-sasl-enc
 :x-sasl-enc; s=smtpout; bh=L2HKcE4TzcUeANp+yNRUPyNkbcE=; b=Kq4JM
 eDYYkWRsr6ePZdAEDjMPMPI0YmVhLIJ28NEodi25Tj45ckKi8jEkLhg9HIRjglN9
 sL4pvGAWWQsvd4J9hzgh2EWBtwB+nsWLLCC8kO2mMffm0oxJJw5d5fW4jS9qAr+Z
 +J247JrWBfmIuvHGooQd3t4C4djDlUhRyfHt0A=
X-Sasl-enc: d3pgw1jdNMW+kL3KrW51ivi7grGZ8EDPVmwrIzr+Hz8z 1454403279
Received: from localhost (c-69-249-5-231.hsd1.pa.comcast.net [69.249.5.231])
 by mail.messagingengine.com (Postfix) with ESMTPA id 7D74F68010F
 for <22533 <at> debbugs.gnu.org>; Tue,  2 Feb 2016 03:54:39 -0500 (EST)
Date: Tue, 2 Feb 2016 03:54:39 -0500
From: Leo Famulari <leo@HIDDEN>
To: 22533 <at> debbugs.gnu.org
Subject: Re: bug#22533: Non-determinism in python-3 ".pyc" bytecode
Message-ID: <20160202085439.GA14802@jasmine>
References: <20160202051544.GA11744@jasmine>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160202051544.GA11744@jasmine>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Spam-Score: -0.7 (/)
X-Debbugs-Envelope-To: 22533
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.7 (/)

On Tue, Feb 02, 2016 at 12:15:44AM -0500, Leo Famulari wrote:
> While preparing a package for borg [0], I found that the built output
> was not reproducible. The problem is that the bytecode compiler [1] for
> Python 3.4.3 (our current version) encodes the mtime of the
> corresponding Python source file in the output. This is described in
> PEP-3147 [2], and the responsible Python code is referenced below [3].
> 
> I tested a few of our existing python-3 packages: python-ccm,
> python-pysam, and python-scripttest all exhibit the same problem.
> 
> We fixed this in python-2 with the patch
> python-2.7-source-date-epoch.patch, but I don't know how to write this
> patch for python-3.

mark_weaver suggested setting the timestamps of the source files before
building. I think this is a better option if it doesn't break anything.
It would allow the bytecode "staleness" check to work as expected while
keeping the output consistent.

> 
> Can somebody write this patch?
> 
> I asked about this on #debian-reproducible and they said that it wasn't
> an issue for Debian since they don't ship bytecode, but instead generate
> it at install time. Of course, that doesn't really apply to Guix.
> 
> I used diffoscope-34 to inspect the build outputs to find this, and you
> can see the report here:
> https://famulari.name/misc/7c55c9e97f668234ddea50299d986f14/borg-diffoscope-report.html
> 
> It's first demonstrated in the file
> ...-borg-0.30.0/lib/python3.4/site-packages/__pycache__/site.cpython-34.pyc.
> 
> The first 2 bytes are the "magic numbers" described in PEP-3147, which
> specify the version of the bytecode format. The next 2 bytes are the
> problematic timestamp, as described in the PEP-3147.
> 
> [0]
> http://borgbackup.github.io/
> 
> [1]
> https://docs.python.org/3/library/py_compile.html
> 
> [2]
> https://www.python.org/dev/peps/pep-3147/
> 
> [3] Check out the Guix git commit 4efc8eb27502c, and from there:
> $ tar xf $(./pre-inst-env guix build --source python-3)
> $ sed -n 139,140p Python-3.4.3/Lib/py_compile.py
>     bytecode = importlib._bootstrap._code_to_bytecode(
>             code, source_stats['mtime'], source_stats['size'])
> 
> 
> 




Information forwarded to bug-guix@HIDDEN:
bug#22533; Package guix. Full text available.

Message received at submit <at> debbugs.gnu.org:


Received: (at submit) by debbugs.gnu.org; 2 Feb 2016 05:16:03 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Feb 02 00:16:03 2016
Received: from localhost ([127.0.0.1]:55601 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1aQTJn-0006fy-F4
	for submit <at> debbugs.gnu.org; Tue, 02 Feb 2016 00:16:03 -0500
Received: from eggs.gnu.org ([208.118.235.92]:54384)
 by debbugs.gnu.org with esmtp (Exim 4.84)
 (envelope-from <leo@HIDDEN>) id 1aQTJm-0006fl-3x
 for submit <at> debbugs.gnu.org; Tue, 02 Feb 2016 00:15:58 -0500
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <leo@HIDDEN>) id 1aQTJf-0004vt-Ow
 for submit <at> debbugs.gnu.org; Tue, 02 Feb 2016 00:15:52 -0500
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=-0.5 required=5.0 tests=BAYES_05,T_DKIM_INVALID
 autolearn=disabled version=3.3.2
Received: from lists.gnu.org ([2001:4830:134:3::11]:46320)
 by eggs.gnu.org with esmtp (Exim 4.71)
 (envelope-from <leo@HIDDEN>) id 1aQTJf-0004vp-M9
 for submit <at> debbugs.gnu.org; Tue, 02 Feb 2016 00:15:51 -0500
Received: from eggs.gnu.org ([2001:4830:134:3::10]:45355)
 by lists.gnu.org with esmtp (Exim 4.71)
 (envelope-from <leo@HIDDEN>) id 1aQTJe-0008R3-HA
 for bug-guix@HIDDEN; Tue, 02 Feb 2016 00:15:51 -0500
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <leo@HIDDEN>) id 1aQTJa-0004ve-GU
 for bug-guix@HIDDEN; Tue, 02 Feb 2016 00:15:50 -0500
Received: from out1-smtp.messagingengine.com ([66.111.4.25]:43960)
 by eggs.gnu.org with esmtp (Exim 4.71)
 (envelope-from <leo@HIDDEN>) id 1aQTJZ-0004va-TZ
 for bug-guix@HIDDEN; Tue, 02 Feb 2016 00:15:46 -0500
Received: from compute1.internal (compute1.nyi.internal [10.202.2.41])
 by mailout.nyi.internal (Postfix) with ESMTP id 7051C2008E;
 Tue,  2 Feb 2016 00:15:45 -0500 (EST)
Received: from frontend1 ([10.202.2.160])
 by compute1.internal (MEProxy); Tue, 02 Feb 2016 00:15:45 -0500
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=famulari.name; h=
 content-type:date:from:message-id:mime-version:subject:to
 :x-sasl-enc:x-sasl-enc; s=mesmtp; bh=shM1vEUSai5utsd2gGa5cQtc4Fg
 =; b=XrqYcbRelhKGYhDSkSy4x/b9cl3Q5ka0J3IQ6NTZK5DhylzHGbzSfNiTY3T
 pztXI4byn0520DZFwG9w6CNvO/lU9FIO9SXzo+Q5fDqzlQzmlXq0EpmJKBz23xyV
 9UnEVTGLQ4nHprEqT636c3OaiG7GSi34TXOcS/odY0q+r1Hg=
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=
 messagingengine.com; h=content-type:date:from:message-id
 :mime-version:subject:to:x-sasl-enc:x-sasl-enc; s=smtpout; bh=sh
 M1vEUSai5utsd2gGa5cQtc4Fg=; b=pnfI2paP0e5ceG3bZRr/w3dfJLOP67dTkc
 GesfihhlJ40LbK++Qq97OuLoxkYgCMaxvpVK75Onx2MHN6s+yKSntz++10HfCJNK
 ONeMPeEaRlhNu7mu7bW3n5lWAWmc9Yj2TAt0zVNFljxtTFe75oelPhahwZwEWXR1
 foYdqX/3c=
X-Sasl-enc: BKrZe55VqbINpIbpEwCRGdYTAFb+aV2/Bn6yKfxLsfDS 1454390145
Received: from localhost (c-69-249-5-231.hsd1.pa.comcast.net [69.249.5.231])
 by mail.messagingengine.com (Postfix) with ESMTPA id 1C742C0001A
 for <bug-guix@HIDDEN>; Tue,  2 Feb 2016 00:15:45 -0500 (EST)
Date: Tue, 2 Feb 2016 00:15:44 -0500
From: Leo Famulari <leo@HIDDEN>
To: bug-guix@HIDDEN
Subject: Non-determinism in python-3 ".pyc" bytecode
Message-ID: <20160202051544.GA11744@jasmine>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.5.24 (2015-08-30)
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x
X-Received-From: 2001:4830:134:3::11
X-Spam-Score: -4.1 (----)
X-Debbugs-Envelope-To: submit
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: 0.9 (/)

While preparing a package for borg [0], I found that the built output
was not reproducible. The problem is that the bytecode compiler [1] for
Python 3.4.3 (our current version) encodes the mtime of the
corresponding Python source file in the output. This is described in
PEP-3147 [2], and the responsible Python code is referenced below [3].

I tested a few of our existing python-3 packages: python-ccm,
python-pysam, and python-scripttest all exhibit the same problem.

We fixed this in python-2 with the patch
python-2.7-source-date-epoch.patch, but I don't know how to write this
patch for python-3.

Can somebody write this patch?

I asked about this on #debian-reproducible and they said that it wasn't
an issue for Debian since they don't ship bytecode, but instead generate
it at install time. Of course, that doesn't really apply to Guix.

I used diffoscope-34 to inspect the build outputs to find this, and you
can see the report here:
https://famulari.name/misc/7c55c9e97f668234ddea50299d986f14/borg-diffoscope-report.html

It's first demonstrated in the file
...-borg-0.30.0/lib/python3.4/site-packages/__pycache__/site.cpython-34.pyc.

The first 2 bytes are the "magic numbers" described in PEP-3147, which
specify the version of the bytecode format. The next 2 bytes are the
problematic timestamp, as described in the PEP-3147.

[0]
http://borgbackup.github.io/

[1]
https://docs.python.org/3/library/py_compile.html

[2]
https://www.python.org/dev/peps/pep-3147/

[3] Check out the Guix git commit 4efc8eb27502c, and from there:
$ tar xf $(./pre-inst-env guix build --source python-3)
$ sed -n 139,140p Python-3.4.3/Lib/py_compile.py
    bytecode = importlib._bootstrap._code_to_bytecode(
            code, source_stats['mtime'], source_stats['size'])




Acknowledgement sent to Leo Famulari <leo@HIDDEN>:
New bug report received and forwarded. Copy sent to bug-guix@HIDDEN. Full text available.
Report forwarded to bug-guix@HIDDEN:
bug#22533; Package guix. Full text available.
Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.
Last modified: Fri, 25 Mar 2016 09:00:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.