GNU bug report logs - #44590
[WIP PATCH CORE-UPDATES] build/python: Check for cythonized files.

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 44590 in the body.
You can then email your comments to 44590 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Efraim Flashner <efraim <at> flashner.co.il>
To: guix-patches <at> gnu.org
Cc: Efraim Flashner <efraim <at> flashner.co.il>
Subject: [WIP PATCH CORE-UPDATES] build/python: Check for cythonized files.
Date: Thu, 12 Nov 2020 12:16:30 +0200

* guix/build/python-build-system.scm (ensure-no-cythonized-files): New procedure.
(%standard-phases): Add it.
---

This patch only checks if a file exists and suggests it may be
cythonized. It doesn't check the first line for the string "generated by
Cython" in that file or in any file. It also doesn't error when it
thinks it finds a file. It does provide a good visual queue that the
source needs more scrutiny before being added.

---
 guix/build/python-build-system.scm | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/guix/build/python-build-system.scm b/guix/build/python-build-system.scm
index 4a07496d22..bc34e22df8 100644
--- a/guix/build/python-build-system.scm
+++ b/guix/build/python-build-system.scm
@@ -8,6 +8,7 @@
 ;;; Copyright © 2018 Arun Isaac <arunisaac <at> systemreboot.net>
 ;;; Copyright © 2019, 2020 Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
 ;;; Copyright © 2020 Jakub Kądziołka <kuba <at> kadziolka.net>
+;;; Copyright © 2020 Efraim Flashner <efraim <at> flashner.co.il>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -262,6 +263,18 @@ installed with setuptools."
   (setenv "PYTHONHASHSEED" "0")
   #t)
 
+(define* (ensure-no-cythonized-files #:rest _)
+  "Check the source code for @code{.c} files which may have been pre-generated
+by Cython."
+  (for-each
+    (lambda (file)
+      (let ((generated-file
+              (string-append (string-drop-right file 3) "c")))
+        (when (file-exists? generated-file)
+          (warning (G_ "Possible Cythonized file found: ~a~%") generated-file))))
+    (find-files "." "\\.pyx$"))
+  #t)
+
 (define %standard-phases
   ;; The build phase only builds C extensions and copies the Python sources,
   ;; while the install phase copies then byte-compiles the sources to the
@@ -271,6 +284,8 @@ installed with setuptools."
     (add-after 'unpack 'ensure-no-mtimes-pre-1980 ensure-no-mtimes-pre-1980)
     (add-after 'ensure-no-mtimes-pre-1980 'enable-bytecode-determinism
       enable-bytecode-determinism)
+    (add-after 'enable-bytecode-determinism 'ensure-no-cythonized-files
+      ensure-no-cythonized-files)
     (delete 'bootstrap)
     (delete 'configure)                 ;not needed
     (replace 'build build)
-- 
2.29.2

Message #10 received at 44590-done <at> debbugs.gnu.org (full text, mbox):

From: Efraim Flashner <efraim <at> flashner.co.il>
To: 44590-done <at> debbugs.gnu.org
Subject: build/python: Check for cythonized files
Date: Wed, 9 Dec 2020 09:28:01 +0200

[Message part 1 (text/plain, inline)]

Patch pushed

-- 
Efraim Flashner   <efraim <at> flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[signature.asc (application/pgp-signature, inline)]

Message #13 received at 44590 <at> debbugs.gnu.org (full text, mbox):

From: Simon South <simon <at> simonsouth.net>
To: Efraim Flashner <efraim <at> flashner.co.il>
Cc: 44590 <at> debbugs.gnu.org
Subject: Re: [bug#44590] [WIP PATCH CORE-UPDATES] build/python: Check for
 cythonized files. 
Date: Sun, 13 Dec 2020 11:53:31 -0500

Efraim,

I'm seeing an issue I think is related to this change. Trying to build
glib in core-updates today fails for me with "Unbound variable: warning"
in python-build-system.scm:

    starting phase `ensure-no-cythonized-files'
    Backtrace:
               9 (primitive-load "/gnu/store/ycv4j8dqk44g6d72m2ykvlwb9wc?")
    In ice-9/eval.scm:
       191:35  8 (_ _)
    In guix/build/gnu-build-system.scm:
        886:2  7 (gnu-build #:source _ #:outputs _ #:inputs _ #:phases . #)
    In ice-9/boot-9.scm:
      1736:10  6 (with-exception-handler _ _ #:unwind? _ # _)
    In srfi/srfi-1.scm:
        634:9  5 (for-each #<procedure 7ffff4f662e0 at guix/build/gnu-b?> ?)
    In guix/build/gnu-build-system.scm:
       895:33  4 (_ _)
    In guix/build/python-build-system.scm:
        272:2  3 (ensure-no-cythonized-files . _)
    In srfi/srfi-1.scm:
        634:9  2 (for-each #<procedure 7ffff5ad3978 at guix/build/pytho?> ?)
    In guix/build/python-build-system.scm:
       277:10  1 (_ _)
    In ice-9/boot-9.scm:
      1669:16  0 (raise-exception _ #:continuable? _)

    ice-9/boot-9.scm:1669:16: In procedure raise-exception:
    Unbound variable: warning
    builder for `/gnu/store/cgmgrddwjvryxmj4nn3naiwy42yi3wzn-meson-for-build-0.55.1.drv' failed with exit code 1
    build of /gnu/store/cgmgrddwjvryxmj4nn3naiwy42yi3wzn-meson-for-build-0.55.1.drv failed
    View build log at '/var/log/guix/drvs/cg/mgrddwjvryxmj4nn3naiwy42yi3wzn-meson-for-build-0.55.1.drv.bz2'.
    cannot build derivation `/gnu/store/a00wr5wj4cd0b98ixxynyxgbw2nyv8kg-glib-2.62.6.drv': 1 dependencies couldn't be built
    guix build: error: build of `/gnu/store/a00wr5wj4cd0b98ixxynyxgbw2nyv8kg-glib-2.62.6.drv' failed

And in fact building python-build-system.scm shows

    [100%] GUILEC   guix/build/python-build-system.go
    WARNING: (guix build python-build-system): imported module (guix build utils) overrides core binding `delete'
    guix/build/python-build-system.scm:277:10: warning: possibly unbound variable `warning'
    guix/build/python-build-system.scm:277:19: warning: possibly unbound variable `G_'

Adding use-module directives for (guix diagnostics) and (guix i18n) to
the module doesn't seem to fix the problem, presumably (if I understand
correctly) because these modules are not then also imported on the build
side of things.

Are you aware of this issue?

-- 
Simon South
simon <at> simonsouth.net

Message #16 received at 44590 <at> debbugs.gnu.org (full text, mbox):

From: Efraim Flashner <efraim <at> flashner.co.il>
To: Simon South <simon <at> simonsouth.net>
Cc: 44590 <at> debbugs.gnu.org
Subject: Re: [bug#44590] [WIP PATCH CORE-UPDATES] build/python: Check for
 cythonized files.
Date: Sun, 13 Dec 2020 21:42:18 +0200

[Message part 1 (text/plain, inline)]

On Sun, Dec 13, 2020 at 11:53:31AM -0500, Simon South wrote:
> Efraim,
> 
> I'm seeing an issue I think is related to this change. Trying to build
> glib in core-updates today fails for me with "Unbound variable: warning"
> in python-build-system.scm:
> 
>     starting phase `ensure-no-cythonized-files'
>     Backtrace:
>                9 (primitive-load "/gnu/store/ycv4j8dqk44g6d72m2ykvlwb9wc?")
>     In ice-9/eval.scm:
>        191:35  8 (_ _)
>     In guix/build/gnu-build-system.scm:
>         886:2  7 (gnu-build #:source _ #:outputs _ #:inputs _ #:phases . #)
>     In ice-9/boot-9.scm:
>       1736:10  6 (with-exception-handler _ _ #:unwind? _ # _)
>     In srfi/srfi-1.scm:
>         634:9  5 (for-each #<procedure 7ffff4f662e0 at guix/build/gnu-b?> ?)
>     In guix/build/gnu-build-system.scm:
>        895:33  4 (_ _)
>     In guix/build/python-build-system.scm:
>         272:2  3 (ensure-no-cythonized-files . _)
>     In srfi/srfi-1.scm:
>         634:9  2 (for-each #<procedure 7ffff5ad3978 at guix/build/pytho?> ?)
>     In guix/build/python-build-system.scm:
>        277:10  1 (_ _)
>     In ice-9/boot-9.scm:
>       1669:16  0 (raise-exception _ #:continuable? _)
> 
>     ice-9/boot-9.scm:1669:16: In procedure raise-exception:
>     Unbound variable: warning
>     builder for `/gnu/store/cgmgrddwjvryxmj4nn3naiwy42yi3wzn-meson-for-build-0.55.1.drv' failed with exit code 1
>     build of /gnu/store/cgmgrddwjvryxmj4nn3naiwy42yi3wzn-meson-for-build-0.55.1.drv failed
>     View build log at '/var/log/guix/drvs/cg/mgrddwjvryxmj4nn3naiwy42yi3wzn-meson-for-build-0.55.1.drv.bz2'.
>     cannot build derivation `/gnu/store/a00wr5wj4cd0b98ixxynyxgbw2nyv8kg-glib-2.62.6.drv': 1 dependencies couldn't be built
>     guix build: error: build of `/gnu/store/a00wr5wj4cd0b98ixxynyxgbw2nyv8kg-glib-2.62.6.drv' failed
> 
> And in fact building python-build-system.scm shows
> 
>     [100%] GUILEC   guix/build/python-build-system.go
>     WARNING: (guix build python-build-system): imported module (guix build utils) overrides core binding `delete'
>     guix/build/python-build-system.scm:277:10: warning: possibly unbound variable `warning'
>     guix/build/python-build-system.scm:277:19: warning: possibly unbound variable `G_'
> 
> Adding use-module directives for (guix diagnostics) and (guix i18n) to
> the module doesn't seem to fix the problem, presumably (if I understand
> correctly) because these modules are not then also imported on the build
> side of things.
> 
> Are you aware of this issue?
> 

I tested it mostly against master. I'll see if I can figure out how I
broke it.

-- 
Efraim Flashner   <efraim <at> flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[signature.asc (application/pgp-signature, inline)]

Message #19 received at 44590-done <at> debbugs.gnu.org (full text, mbox):

From: Efraim Flashner <efraim <at> flashner.co.il>
To: Simon South <simon <at> simonsouth.net>
Cc: 44590-done <at> debbugs.gnu.org
Subject: Re: [bug#44590] [WIP PATCH CORE-UPDATES] build/python: Check for
 cythonized files.
Date: Sun, 13 Dec 2020 22:23:23 +0200

[Message part 1 (text/plain, inline)]

On Sun, Dec 13, 2020 at 11:53:31AM -0500, Simon South wrote:
> Efraim,
> 
> I'm seeing an issue I think is related to this change. Trying to build
> glib in core-updates today fails for me with "Unbound variable: warning"
> in python-build-system.scm:
> 
>     starting phase `ensure-no-cythonized-files'
>     Backtrace:
>                9 (primitive-load "/gnu/store/ycv4j8dqk44g6d72m2ykvlwb9wc?")
>     In ice-9/eval.scm:
>        191:35  8 (_ _)
>     In guix/build/gnu-build-system.scm:
>         886:2  7 (gnu-build #:source _ #:outputs _ #:inputs _ #:phases . #)
>     In ice-9/boot-9.scm:
>       1736:10  6 (with-exception-handler _ _ #:unwind? _ # _)
>     In srfi/srfi-1.scm:
>         634:9  5 (for-each #<procedure 7ffff4f662e0 at guix/build/gnu-b?> ?)
>     In guix/build/gnu-build-system.scm:
>        895:33  4 (_ _)
>     In guix/build/python-build-system.scm:
>         272:2  3 (ensure-no-cythonized-files . _)
>     In srfi/srfi-1.scm:
>         634:9  2 (for-each #<procedure 7ffff5ad3978 at guix/build/pytho?> ?)
>     In guix/build/python-build-system.scm:
>        277:10  1 (_ _)
>     In ice-9/boot-9.scm:
>       1669:16  0 (raise-exception _ #:continuable? _)
> 
>     ice-9/boot-9.scm:1669:16: In procedure raise-exception:
>     Unbound variable: warning
>     builder for `/gnu/store/cgmgrddwjvryxmj4nn3naiwy42yi3wzn-meson-for-build-0.55.1.drv' failed with exit code 1
>     build of /gnu/store/cgmgrddwjvryxmj4nn3naiwy42yi3wzn-meson-for-build-0.55.1.drv failed
>     View build log at '/var/log/guix/drvs/cg/mgrddwjvryxmj4nn3naiwy42yi3wzn-meson-for-build-0.55.1.drv.bz2'.
>     cannot build derivation `/gnu/store/a00wr5wj4cd0b98ixxynyxgbw2nyv8kg-glib-2.62.6.drv': 1 dependencies couldn't be built
>     guix build: error: build of `/gnu/store/a00wr5wj4cd0b98ixxynyxgbw2nyv8kg-glib-2.62.6.drv' failed
> 
> And in fact building python-build-system.scm shows
> 
>     [100%] GUILEC   guix/build/python-build-system.go
>     WARNING: (guix build python-build-system): imported module (guix build utils) overrides core binding `delete'
>     guix/build/python-build-system.scm:277:10: warning: possibly unbound variable `warning'
>     guix/build/python-build-system.scm:277:19: warning: possibly unbound variable `G_'
> 
> Adding use-module directives for (guix diagnostics) and (guix i18n) to
> the module doesn't seem to fix the problem, presumably (if I understand
> correctly) because these modules are not then also imported on the build
> side of things.
> 
> Are you aware of this issue?
> 

I fixed it. I changed the code to output using format and (ice-9 format)
and I tested it building meson-for-build and python-setuptools.

I suppose I should point out now that meson-for-build has some possibly
cythonized files :)

-- 
Efraim Flashner   <efraim <at> flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[signature.asc (application/pgp-signature, inline)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.