GNU bug report logs - #61586
[PATCH RFC 0/2] Add BinaryEn

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 61586 in the body.
You can then email your comments to 61586 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Liliana Marie Prikler <liliana.prikler <at> gmail.com>
To: guix-patches <at> gnu.org
Subject: [PATCH RFC 0/2] Add BinaryEn
Date: Fri, 17 Feb 2023 21:45:33 +0100

Hi Guix,

this series gets us a little closer to having a "full" WebAssembly stack.
It packages binaryen, on top of which other compilers such as emscripten
or AssemblyScript (a sort of Typescript?) are built.

However, there is a grain of salt.  It appears binaryen has some rather
esoteric use for -msse2 on i686: Rather than performance, it wants it
for precision.  Needless to say, this would break compatibility with
older CPUs.  I'm wondering if we should simply drop i686 (and similarly
32-bit ARM) from supported-systems or whether there's a more clever
hack to use here.

Cheers

Liliana Marie Prikler (2):
  gnu: Add python-filecheck.
  gnu: Add binaryen.

 gnu/packages/check.scm | 25 +++++++++++++++++++++++
 gnu/packages/web.scm   | 45 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 70 insertions(+)


base-commit: 312f1f41d3f3f3e5d2c36ff46920c6dce1c21a17
-- 
2.39.1

Message #8 received at 61586 <at> debbugs.gnu.org (full text, mbox):

From: Liliana Marie Prikler <liliana.prikler <at> gmail.com>
To: 61586 <at> debbugs.gnu.org
Subject: [PATCH RFC 1/2] gnu: Add python-filecheck.
Date: Fri, 17 Feb 2023 21:27:54 +0100

* gnu/packages/check.scm (python-filecheck): New variable.
---
 gnu/packages/check.scm | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/gnu/packages/check.scm b/gnu/packages/check.scm
index 3d14cb7585..c4e83d41ef 100644
--- a/gnu/packages/check.scm
+++ b/gnu/packages/check.scm
@@ -2364,6 +2364,31 @@ (define-public python-lit
 failures.")
     (license license:ncsa)))
 
+(define-public python-filecheck
+  (package
+    (name "python-filecheck")
+    (version "0.0.23")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/mull-project/FileCheck.py")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "1gipw7x54nr6raxr681igpxn8jli5mwaznnj40xxks0pa5kvirs7"))))
+    (build-system pyproject-build-system)
+    (native-inputs (list python-lit python-pytest python-pylint
+                         python-invoke poetry))
+    (home-page "https://filecheck.readthedocs.io/")
+    (synopsis "Pattern matching file verifier")
+    (description
+     "This package provides a Python port of LLVM's FileCheck utility.  It
+can be used to assert that certain strings (or regular expressions) are present
+or missing in a given file.  A typical application is the post-processing of
+test logs.")
+    (license license:asl2.0)))
+
 ;;; This is marked as a bootstrap package because it propagates bootstrapped
 ;;; versions of jaraco-context and jaraco-functools.
 (define-public python-pytest-enabler-bootstrap
-- 
2.39.1

Message #11 received at 61586 <at> debbugs.gnu.org (full text, mbox):

From: Liliana Marie Prikler <liliana.prikler <at> gmail.com>
To: 61586 <at> debbugs.gnu.org
Subject: [PATCH RFC 2/2] gnu: Add binaryen.
Date: Fri, 17 Feb 2023 21:28:10 +0100

* gnu/packages/web.scm (binaryen): New variable.
---
 gnu/packages/web.scm | 45 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 45 insertions(+)

diff --git a/gnu/packages/web.scm b/gnu/packages/web.scm
index 7e49f798ea..0b509fa34c 100644
--- a/gnu/packages/web.scm
+++ b/gnu/packages/web.scm
@@ -1558,6 +1558,51 @@ (define-public wabt
 other systems that want to manipulate WebAssembly files.")
     (license license:asl2.0)))
 
+(define-public binaryen
+  (package
+   (name "binaryen")
+   (version "112")
+   (source (origin
+            (method git-fetch)
+            (uri (git-reference
+                  (url "https://github.com/WebAssembly/binaryen")
+                  (commit (string-append "version_" version))))
+            (file-name (git-file-name name version))
+            (sha256
+             (base32
+              "0970iz22yjxgi27d67kwmrx4zq7hig3i6b92vmlp4c4bd1bacny5"))
+            (modules '((guix build utils)))
+            (snippet #~(begin
+                         (substitute* "CMakeLists.txt"
+                           (("add_subdirectory\\(third_party\\)")
+                            "find_package(GTest)"))
+                         (substitute* "test/gtest/CMakeLists.txt"
+                           (("include_directory\\(.*third_pary.*\\)") ""))
+                         (delete-file-recursively "third_party")))))
+   (build-system cmake-build-system)
+   (arguments
+    (list #:out-of-source? #f ; for tests
+          #:configure-flags #~(list "-DBUILD_LLVM_DWARF=OFF")
+          #:phases
+          #~(modify-phases %standard-phases
+              (add-before 'check 'delete-failing-tests
+                (lambda _
+                  ;; DWARF support relies on bundling LLVM, so don't
+                  (for-each delete-file
+                            (find-files "test/passes"
+                                        ".*dwarf.*\\.(bin\\.txt\|wasm)"))
+                  (delete-file "test/unit/test_dwarf.py")))
+              (replace 'check
+                (lambda* (#:key tests? #:allow-other-keys)
+                  (invoke "python" "check.py"))))))
+   (native-inputs (list googletest node-lts python-wrapper
+                        python-lit python-filecheck))
+   (home-page "https://github.com/WebAssembly/binaryen")
+   (synopsis "WebAssembly compiler")
+   (description "Binaryen is a compiler and toolchain infrastructure library
+written in C++, with a single-header C API as well as a Javascript API.")
+   (license license:asl2.0)))
+
 (define-public wasm3
   (package
     (name "wasm3")
-- 
2.39.1

Message #14 received at 61586 <at> debbugs.gnu.org (full text, mbox):

From: Andreas Enge <andreas <at> enge.fr>
To: 61586 <at> debbugs.gnu.org
Subject: BinaryEn
Date: Sat, 11 Mar 2023 23:48:59 +0100

Hello Liliana,

how about you start by pushing the python-filecheck package?

Do you have a pointer to BinaryEn using -msse2 for precision?
I am not familiar with SSE2, but a quick look-up on Wikipedia only shows
(packed) double floating point operations and packed integer arithmetic.
All these should be feasible directly in C, although maybe more slowly.

In the CMakeLists.txt file there are the following lines:
  if(NOT EMSCRIPTEN)
    if(CMAKE_SYSTEM_PROCESSOR MATCHES "^i.86$")
      # wasm doesn't allow for x87 floating point math
      add_compile_flag("-msse2")
      add_compile_flag("-mfpmath=sse")
    elseif(CMAKE_SYSTEM_PROCESSOR MATCHES "^armv[2-6]" AND NOT CMAKE_CXX_FLAGS MATCHES "-mfpu=")
      add_compile_flag("-mfpu=vfpv3")
    endif()
  endif()

So the -msse2 flag will not be added on arm; the package is compiled
successfully on aarch64, and I see no reason why in principal it should
not also work on armhf. Maybe there is a more precise test for x86_64
that could be used instead of
    if(CMAKE_SYSTEM_PROCESSOR MATCHES "^i.86$") ?
Building on armhf and i686 currently fails (is "blocked" in QA parlance):
   https://qa.guix.gnu.org/issue/61586
due to python-aioredis failing on i686
   https://data.qa.guix.gnu.org/build-server/2/build?build_server_build_id=00818a34-5efe-4a73-8ca2-3e095885c7ee
and node, python-simplejson and python-numpy on armhf:
   https://data.qa.guix.gnu.org/build-server/2/build?build_server_build_id=34851d8f-41c9-4a8b-9150-3071e1a6d9f7

Andreas

Message #17 received at 61586 <at> debbugs.gnu.org (full text, mbox):

From: Liliana Marie Prikler <liliana.prikler <at> gmail.com>
To: Andreas Enge <andreas <at> enge.fr>, 61586 <at> debbugs.gnu.org
Subject: Re: BinaryEn
Date: Sun, 12 Mar 2023 09:16:45 +0100

Am Samstag, dem 11.03.2023 um 23:48 +0100 schrieb Andreas Enge:
> Hello Liliana,
Hi Andreas, please don't forget to add me in CC.

> how about you start by pushing the python-filecheck package?
I'd only do that if it has another user and for the time being I don't
see that.  Don't worry, I still got more python packages in my backlog
:)

> Do you have a pointer to BinaryEn using -msse2 for precision?
> I am not familiar with SSE2, but a quick look-up on Wikipedia only
> shows (packed) double floating point operations and packed integer
> arithmetic.
> All these should be feasible directly in C, although maybe more
> slowly.
Possible, but more slowly in C doesn't translate that nicely if you
don't want to code up your own float/double types and you really don't
want that.

The problem here is that expressions like:
  double a, b, c;
  c = sqrt(a * a, b * b);
can use 80 bit intermediaries on x87 chips, which they don't when using
SSE2 – hence the precision argument.  You would have to redefine all
basic operations for your floating point (which would still be doable
in C++ due to operator overloading, but be a major pain in the butt to
do correctly and well-tested, hence the deference to SSE2, I believe).

> In the CMakeLists.txt file there are the following lines:
>   if(NOT EMSCRIPTEN)
>     if(CMAKE_SYSTEM_PROCESSOR MATCHES "^i.86$")
>       # wasm doesn't allow for x87 floating point math
>       add_compile_flag("-msse2")
>       add_compile_flag("-mfpmath=sse")
>     elseif(CMAKE_SYSTEM_PROCESSOR MATCHES "^armv[2-6]" AND NOT
> CMAKE_CXX_FLAGS MATCHES "-mfpu=")
>       add_compile_flag("-mfpu=vfpv3")
>     endif()
>   endif()
> 
> So the -msse2 flag will not be added on arm; the package is compiled
> successfully on aarch64, and I see no reason why in principal it
> should not also work on armhf. 
It does require the vfpv3 fpu, which I believe won't exist on all arms.

> Maybe there is a more precise test for x86_64 that could be used
> instead of
>     if(CMAKE_SYSTEM_PROCESSOR MATCHES "^i.86$") ?
Not for the kind of check they want to make, I believe.

Cheers

Message #20 received at 61586 <at> debbugs.gnu.org (full text, mbox):

From: Andreas Enge <andreas <at> enge.fr>
To: Liliana Marie Prikler <liliana.prikler <at> gmail.com>
Cc: 61586 <at> debbugs.gnu.org
Subject: Re: BinaryEn
Date: Sun, 12 Mar 2023 11:49:47 +0100

Hello,

Am Sun, Mar 12, 2023 at 09:16:45AM +0100 schrieb Liliana Marie Prikler:
> Am Samstag, dem 11.03.2023 um 23:48 +0100 schrieb Andreas Enge:
> > Hello Liliana,
> Hi Andreas, please don't forget to add me in CC.

ah, I thought that debbugs would automatically redispatch the mail
to everybody who contributed to the report.

> The problem here is that expressions like:
>   double a, b, c;
>   c = sqrt(a * a, b * b);
> can use 80 bit intermediaries on x87 chips, which they don't when using
> SSE2 – hence the precision argument.  You would have to redefine all
> basic operations for your floating point (which would still be doable
> in C++ due to operator overloading, but be a major pain in the butt to
> do correctly and well-tested, hence the deference to SSE2, I believe).

Hm, I thought that when using gcc and glibc, floating point operations
now followed the IEEE-754 standard. Then it would not matter what the
internal format is (except for special functions).

Still, I wonder if it would not be possible to change the configure test
so that the package compiles on 32 bit platforms; otherwise we would have
to take them out from the supported systems. And that would not be better
(assuming that the package passes its tests on these platforms, of course).

Andreas

Message #23 received at 61586 <at> debbugs.gnu.org (full text, mbox):

From: "Thompson, David" <dthompson2 <at> worcester.edu>
To: Liliana Marie Prikler <liliana.prikler <at> gmail.com>
Cc: 61586 <at> debbugs.gnu.org
Subject: Re: [bug#61586] [PATCH RFC 2/2] gnu: Add binaryen.
Date: Thu, 6 Apr 2023 17:38:23 -0400

Hi Liliana,

On Fri, Feb 17, 2023 at 6:00 PM Liliana Marie Prikler
<liliana.prikler <at> gmail.com> wrote:
>
> * gnu/packages/web.scm (binaryen): New variable.
> ---
>  gnu/packages/web.scm | 45 ++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 45 insertions(+)
>
> diff --git a/gnu/packages/web.scm b/gnu/packages/web.scm
> index 7e49f798ea..0b509fa34c 100644
> --- a/gnu/packages/web.scm
> +++ b/gnu/packages/web.scm
> @@ -1558,6 +1558,51 @@ (define-public wabt
>  other systems that want to manipulate WebAssembly files.")
>      (license license:asl2.0)))
>
> +(define-public binaryen
> +  (package
> +   (name "binaryen")
> +   (version "112")
> +   (source (origin
> +            (method git-fetch)
> +            (uri (git-reference
> +                  (url "https://github.com/WebAssembly/binaryen")
> +                  (commit (string-append "version_" version))))
> +            (file-name (git-file-name name version))
> +            (sha256
> +             (base32
> +              "0970iz22yjxgi27d67kwmrx4zq7hig3i6b92vmlp4c4bd1bacny5"))
> +            (modules '((guix build utils)))
> +            (snippet #~(begin
> +                         (substitute* "CMakeLists.txt"
> +                           (("add_subdirectory\\(third_party\\)")
> +                            "find_package(GTest)"))
> +                         (substitute* "test/gtest/CMakeLists.txt"
> +                           (("include_directory\\(.*third_pary.*\\)") ""))
> +                         (delete-file-recursively "third_party")))))
> +   (build-system cmake-build-system)
> +   (arguments
> +    (list #:out-of-source? #f ; for tests
> +          #:configure-flags #~(list "-DBUILD_LLVM_DWARF=OFF")
> +          #:phases
> +          #~(modify-phases %standard-phases
> +              (add-before 'check 'delete-failing-tests
> +                (lambda _
> +                  ;; DWARF support relies on bundling LLVM, so don't
> +                  (for-each delete-file
> +                            (find-files "test/passes"
> +                                        ".*dwarf.*\\.(bin\\.txt\|wasm)"))
> +                  (delete-file "test/unit/test_dwarf.py")))
> +              (replace 'check
> +                (lambda* (#:key tests? #:allow-other-keys)
> +                  (invoke "python" "check.py"))))))
> +   (native-inputs (list googletest node-lts python-wrapper
> +                        python-lit python-filecheck))
> +   (home-page "https://github.com/WebAssembly/binaryen")
> +   (synopsis "WebAssembly compiler")
> +   (description "Binaryen is a compiler and toolchain infrastructure library
> +written in C++, with a single-header C API as well as a Javascript API.")
> +   (license license:asl2.0)))
> +
>  (define-public wasm3
>    (package
>      (name "wasm3")
> --
> 2.39.1

This looks good to me!  We are currently using a hackier package
recipe with tests disabled in the Guile Hoot (Guile -> WASM compiler)
project: https://gitlab.com/spritely/guile-hoot-updates/-/blob/main/examples/manifest.scm#L18

Nice job getting the tests working!

- Dave

Message #28 received at 61586-done <at> debbugs.gnu.org (full text, mbox):

From: Liliana Marie Prikler <liliana.prikler <at> gmail.com>
To: "Thompson, David" <dthompson2 <at> worcester.edu>
Cc: 61586-done <at> debbugs.gnu.org
Subject: Re: [bug#61586] [PATCH RFC 2/2] gnu: Add binaryen.
Date: Sun, 11 Jun 2023 10:04:44 +0200

Am Donnerstag, dem 06.04.2023 um 17:38 -0400 schrieb Thompson, David:
> Hi Liliana,
> 
> [...]
> This looks good to me!  We are currently using a hackier package
> recipe with tests disabled in the Guile Hoot (Guile -> WASM compiler)
> project:
> https://gitlab.com/spritely/guile-hoot-updates/-/blob/main/examples/manifest.scm#L18
> 
> Nice job getting the tests working!
Someone (I don't remember who and don't care to look it up, might have
been myself) pushed this without marking the bug as done.  The hoot
link is also dead.  Time to close up.

Cheers

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.