Package: guix-patches;
Reported by: Björn Höfling <bjoern.hoefling <at> bjoernhoefling.de>
Date: Tue, 13 Mar 2018 16:59:01 UTC
Severity: normal
Tags: patch
Done: ludo <at> gnu.org (Ludovic Courtès)
Bug is archived. No further changes may be made.
To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 30801 in the body.
You can then email your comments to 30801 AT debbugs.gnu.org in the normal way.
Toggle the display of automated, internal messages from the tracker.
View this report as an mbox folder, status mbox, maintainer mbox
guix-patches <at> gnu.org:bug#30801; Package guix-patches.
(Tue, 13 Mar 2018 16:59:01 GMT) Full text and rfc822 format available.Björn Höfling <bjoern.hoefling <at> bjoernhoefling.de>:guix-patches <at> gnu.org.
(Tue, 13 Mar 2018 16:59:02 GMT) Full text and rfc822 format available.Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):
From: Björn Höfling <bjoern.hoefling <at> bjoernhoefling.de> To: <guix-patches <at> gnu.org> Subject: [PATCH 0/1] Add opencv Date: Tue, 13 Mar 2018 17:58:09 +0100
[Message part 1 (text/plain, inline)]
This patch adds OpenCV, a computer vision library. It combines the basic and contrib algorithms. Some notes: The test suite consists of an extra package, weighting 465MB compressed. It runs very well. I think the size is worth it. It consists of proprietary things (i.e. lena.jpg). As far as I understand, that is OK, if it doesn't get in the final src/bin store output. Right? CPU-optimization: I hope I have done everything right. Reading the article from Guix HPC a second time helped a lot. So now it should be compiled with SSE2/NEON being the minimum required instruction set, and dispatches to other ISAs where available. Size: Currently I load a bunch of dependencies in and have one big package as output. guix size is 1.1 GiB. I slightly have the feeling someone could ask to split it in several outputs. Though having one big output was the easiest thing first and I don't know how one would handle inter-dependencies between the different outputs. Björn Björn Höfling (1): gnu: Add opencv. gnu/packages/image-processing.scm | 186 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 186 insertions(+) -- 2.16.2
[Message part 2 (application/pgp-signature, inline)]
guix-patches <at> gnu.org:bug#30801; Package guix-patches.
(Tue, 13 Mar 2018 17:09:01 GMT) Full text and rfc822 format available.Message #8 received at 30801 <at> debbugs.gnu.org (full text, mbox):
From: Björn Höfling <bjoern.hoefling <at> bjoernhoefling.de> To: 30801 <at> debbugs.gnu.org Subject: [PATCH 1/1] gnu: Add opencv. Date: Tue, 13 Mar 2018 18:07:56 +0100
[Message part 1 (text/plain, inline)]
* gnu/packages/image-processing.scm (opencv): New variable.
---
gnu/packages/image-processing.scm | 186 ++++++++++++++++++++++++++++++++++++++
1 file changed, 186 insertions(+)
diff --git a/gnu/packages/image-processing.scm b/gnu/packages/image-processing.scm
index 6705a726c..c33075abc 100644
--- a/gnu/packages/image-processing.scm
+++ b/gnu/packages/image-processing.scm
@@ -5,6 +5,7 @@
;;; Copyright © 2014 Mark H Weaver <mhw <at> netris.org>
;;; Copyright © 2016 Eric Bavier <bavier <at> member.fsf.org>
;;; Copyright © 2018 Tobias Geerinckx-Rice <me <at> tobias.gr>
+;;; Copyright © 2018 Björn Höfling <bjoern.hoefling <at> bjoernhoefling.de>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -38,12 +39,16 @@
#:use-module (gnu packages gnome)
#:use-module (gnu packages graphics)
#:use-module (gnu packages graphviz)
+ #:use-module (gnu packages gtk)
#:use-module (gnu packages image)
#:use-module (gnu packages maths)
#:use-module (gnu packages perl)
+ #:use-module (gnu packages photo)
#:use-module (gnu packages pkg-config)
+ #:use-module (gnu packages protobuf)
#:use-module (gnu packages python)
#:use-module (gnu packages serialization)
+ #:use-module (gnu packages video)
#:use-module (gnu packages xiph)
#:use-module (gnu packages xml)
#:use-module (gnu packages xorg))
@@ -191,3 +196,184 @@ triangulation. VTK has an extensive information visualization framework, has
a suite of 3D interaction widgets, supports parallel processing, and
integrates with various databases on GUI toolkits such as Qt and Tk.")
(license license:bsd-3)))
+
+(define-public opencv
+ (package
+ (name "opencv")
+ (version "3.4.1")
+ (source (origin
+ (method url-fetch)
+ (uri (string-append "https://github.com/opencv/opencv/archive/"
+ version ".zip"))
+ (file-name (string-append name "-" version ".zip"))
+ (sha256
+ (base32
+ "1g8pvnlkzzp50amd89149hqsbvsc2hq3vk1d6a9fksdcx8ra9g94"))
+ (modules '((guix build utils)))
+ (snippet
+ '(begin
+ ;; Remove external libraries. We have all available in Guix:
+ (delete-file-recursively "3rdparty")
+
+ ;; Milky icon set is non-free:
+ (delete-file-recursively "modules/highgui/src/files_Qt/Milky")
+
+ ;; Some jars found:
+ (for-each delete-file
+ '("modules/java/test/pure_test/lib/junit-4.11.jar"
+ "samples/java/sbt/sbt/sbt-launch.jar"))))))
+ (build-system cmake-build-system)
+ (arguments
+ `(#:configure-flags
+ (list "-DWITH_IPP=OFF"
+ "-DWITH_ITT=OFF"
+
+ ;; CPU-Features:
+ ;; See cmake/OpenCVCompilerOptimizations.cmake
+ ;; (CPU_ALL_OPTIMIZATIONS) for a list of all optimizations
+ ;; BASELINE is the minimum optimization all CPUs must support
+ ;;
+ ;; DISPATCH is the list of optional dispatches
+ "-DCPU_BASELINE=SSE2, NEON"
+
+ "-DCPU_DISPATCH=SSE3,SSSE3,SSE4_1,SSE4_2,AVX,AVX2"
+ "-DCPU_DISPATCH_REQUIRE=SSE3,SSSE3,SSE4_1,SSE4_2,AVX,AVX2"
+
+ "-DBUILD_PERF_TESTS=OFF"
+ "-D BUILD_TESTS=ON"
+
+ (string-append "-DOPENCV_EXTRA_MODULES_PATH="
+ "/tmp/guix-build-opencv-" ,version ".drv-0"
+ "/opencv-contrib/opencv_contrib-" ,version
+ "/modules")
+
+ ;;Define test data:
+ (string-append "-DOPENCV_TEST_DATA_PATH="
+ "/tmp/guix-build-opencv-" ,version ".drv-0"
+ ;;"/opencv-3.4.0"
+ "/opencv-extra/opencv_extra-" ,version
+ "/testdata")
+
+ ;; Is ON by default and would try to rebuild 3rd-party protobuf,
+ ;; which we had removed, which would lead to an error:
+ "-DBUILD_PROTOBUF=OFF"
+
+ ;; Rebuild protobuf files, because we have a slightly different
+ ;; version than the included one. If we would not update, we
+ ;; would get a compile error later:
+ "-DPROTOBUF_UPDATE_FILES=ON"
+
+ ;; xfeatures2d disabled, because it downloads extra binaries from
+ ;; https://github.com/opencv/opencv_3rdparty
+ ;; defined in xfeatures2d/cmake/download_{vgg|bootdesc}.cmake
+ ;; Cmp this bug entry:
+ ;; https://github.com/opencv/opencv_contrib/issues/1131
+ "-DBUILD_opencv_xfeatures2d=OFF")
+ #:phases
+ (modify-phases %standard-phases
+ (add-after 'unpack 'disable-broken-tests
+ (lambda _
+ ;; These tests fails with:
+ ;; vtkXOpenGLRenderWindow (0x723990): Could not find a decent config
+ ;; I think we have no OpenGL support with the Xvfb.
+ (substitute* '("modules/viz/test/test_tutorial3.cpp"
+ "modules/viz/test/test_main.cpp"
+ "modules/viz/test/tests_simple.cpp"
+ "modules/viz/test/test_viz3d.cpp")
+ (("(TEST\\(Viz, )([a-z].*\\).*)" all pre post)
+ (string-append pre "DISABLED_" post)))
+
+ ;; This one fails with "unknown file: Failure"
+ ;; But I couldn't figure out which file was missing:
+ (substitute* (list (string-append
+ "../opencv-contrib/opencv_contrib-"
+ ,version
+ "/modules/face/test/test_face_align.cpp"))
+ (("(TEST\\(CV_Face_FacemarkKazemi, )(can_detect_landmarks\\).*)"
+ all pre post)
+ (string-append pre "DISABLED_" post)))
+
+ ;; Failure reason: Bad accuracy
+ ;; Incorrect count of accurate poses [2nd case]: 90.000000 / 94.000000
+ (substitute* (list (string-append
+ "../opencv-contrib/opencv_contrib-"
+ ,version
+ "/modules/rgbd/test/test_odometry.cpp"))
+ (("(TEST\\(RGBD_Odometry_Rgbd, )(algorithmic\\).*)" all pre post)
+ (string-append pre "DISABLED_" post)))
+ #t))
+
+ ;; Idea copied from ldc.scm (ldc-bootstrap):
+ (add-after 'unpack 'unpack-submodule-sources
+ (lambda* (#:key inputs #:allow-other-keys)
+ (mkdir "../opencv-extra")
+ (mkdir "../opencv-contrib")
+ (let ((unpack (lambda (source target)
+ (with-directory-excursion target
+ (apply invoke "unzip"
+ (list (assoc-ref inputs source)))))))
+ (unpack "opencv-extra" "../opencv-extra")
+ (unpack "opencv-contrib" "../opencv-contrib"))))
+
+ (add-after 'set-paths 'add-ilmbase-include-path
+ (lambda* (#:key inputs #:allow-other-keys)
+ ;; OpenEXR propagates ilmbase, but its include files do not appear
+ ;; in the CPATH, so we need to add "$ilmbase/include/OpenEXR/" to
+ ;; the CPATH to satisfy the dependency on "ImathVec.h".
+ (setenv "CPATH"
+ (string-append (assoc-ref inputs "ilmbase")
+ "/include/OpenEXR"
+ ":" (or (getenv "CPATH") "")))
+ #t))
+ (add-before 'check 'start-xserver
+ (lambda* (#:key inputs #:allow-other-keys)
+ (let ((xorg-server (assoc-ref inputs "xorg-server"))
+ (disp ":1"))
+ (setenv "HOME" (getcwd))
+ (setenv "DISPLAY" disp)
+ ;; There must be a running X server and make check doesn't start one.
+ ;; Therefore we must do it.
+ (zero? (system (format #f "~a/bin/Xvfb ~a &" xorg-server disp)))))))))
+ (native-inputs
+ `(("unzip" ,unzip)
+ ("pkg-config" ,pkg-config)
+ ("xorg-server" ,xorg-server) ; For running the tests
+ ("opencv-extra"
+ ,(origin
+ (method url-fetch)
+ (uri (string-append "https://codeload.github.com/"
+ "opencv/opencv_extra/zip/" version))
+ (file-name (string-append "opencv-extra-" version ".zip"))
+ (sha256
+ (base32 "0wfh3pvfxqydf7hsccp50npcsg37sf6fqi6cd3zkc4qil9zhpbps"))))
+ ("opencv-contrib"
+ ,(origin
+ (method url-fetch)
+ (uri (string-append "https://codeload.github.com/"
+ "opencv/opencv_contrib/zip/" version))
+ (file-name (string-append "opencv-contrib-" version ".zip"))
+ (sha256
+ (base32 "18zm0qmjcdvg90c33gzv0ws0xdaid1xpqzz2xa9l2x12qkr6zj3p"))))))
+ (inputs `(("ffmpeg" ,ffmpeg)
+ ("libjpeg" ,libjpeg)
+ ("libpng" ,libpng)
+ ("jasper" ,jasper)
+ ("libtiff" ,libtiff)
+ ("hdf5" ,hdf5)
+ ("libgphoto2" ,libgphoto2)
+ ("libwebp" ,libwebp)
+ ("zlib" ,zlib)
+ ("gtkglext" ,gtkglext)
+ ("openexr" ,openexr)
+ ("ilmbase" ,ilmbase)
+ ("gtk+" ,gtk+-2)
+ ("python-numpy" ,python-numpy)
+ ("protobuf" ,protobuf)
+ ("vtk" ,vtk)
+ ("python" ,python)))
+ (synopsis "Computer Vision Library")
+ (description "OpenCV (Open Source Computer Vision) is a library aimed at
+real-time computer vision, including several hundred computer
+vision algorithms.")
+ (home-page "https://opencv.org/")
+ (license license:bsd-3)))
--
2.16.2
[Message part 2 (application/pgp-signature, inline)]
guix-patches <at> gnu.org:bug#30801; Package guix-patches.
(Thu, 15 Mar 2018 21:06:01 GMT) Full text and rfc822 format available.Message #11 received at 30801 <at> debbugs.gnu.org (full text, mbox):
From: ludo <at> gnu.org (Ludovic Courtès) To: Björn Höfling <bjoern.hoefling <at> bjoernhoefling.de> Cc: 30801 <at> debbugs.gnu.org Subject: Re: [bug#30801] [PATCH 0/1] Add opencv Date: Thu, 15 Mar 2018 22:04:54 +0100
Hi Björn, Björn Höfling <bjoern.hoefling <at> bjoernhoefling.de> skribis: > The test suite consists of an extra package, weighting 465MB > compressed. It runs very well. I think the size is worth it. It > consists of proprietary things (i.e. lena.jpg). As far as I understand, > that is OK, if it doesn't get in the final src/bin store output. Right? As a rule of thumb, there should not be non-free stuff in the derivation graph. If there’s non-free software, that’s not OK, even if it doesn’t show up in the output. If it’s “just” data (pictures) that are non-free, that’s OK per the FSDG: <https://www.gnu.org/distros/free-system-distribution-guidelines.html#non-functional-data>. If we could replace it with a free variant, I think we should clearly encourage it, but lena.jpg is hardly replaceable in this context (I’d hope it weren’t around for what it tells about CS, but that’s another story…). > CPU-optimization: I hope I have done everything right. Reading the > article from Guix HPC a second time helped a lot. So now it should be > compiled with SSE2/NEON being the minimum required instruction set, and > dispatches to other ISAs where available. Great. > Size: Currently I load a bunch of dependencies in and have one > big package as output. guix size is 1.1 GiB. I slightly have the > feeling someone could ask to split it in several outputs. Though having > one big output was the easiest thing first and I don't know how one > would handle inter-dependencies between the different outputs. 1.1G is a bit too much IMO, indeed. :-) How much is OpenCV itself? If OpenCV itself is big, it would be nice to look at what’s taking up space in there with ‘du’. For instance, if there are .a files, we might want to not build them and keep only shared objects. Splitting in separate outputs may or may not make sense. If there are examples or large pieces of HTML doc, introducing a “doc” and/or an “examples” output may help. Likewise, if there are binaries that depend on more stuff that the library itself, introducing a “lib” output might be a good idea. Could you take a look? Nitpick: > + (synopsis "Computer Vision Library") No need to capitalize. > + (description "OpenCV (Open Source Computer Vision) is a library aimed at I’d remove the parenthetic part. > +real-time computer vision, including several hundred computer > +vision algorithms.") Bonus points if you can expound a little bit, without just listing those algorithms though. ;-) Apart from that the patch looks very polished and that’s a pleasure to review! Thanks, Ludo’.
guix-patches <at> gnu.org:bug#30801; Package guix-patches.
(Sat, 31 Mar 2018 22:27:02 GMT) Full text and rfc822 format available.Message #14 received at 30801 <at> debbugs.gnu.org (full text, mbox):
From: Björn Höfling <bjoern.hoefling <at> bjoernhoefling.de> To: ludo <at> gnu.org (Ludovic Courtès) Cc: 30801 <at> debbugs.gnu.org Subject: Re: [bug#30801] [PATCH 0/1] Add opencv Date: Sun, 1 Apr 2018 00:26:49 +0200
[Message part 1 (text/plain, inline)]
Hi Ludo,
thanks for reviewing!
On Thu, 15 Mar 2018 22:04:54 +0100
ludo <at> gnu.org (Ludovic Courtès) wrote:
> Hi Björn,
>
> Björn Höfling <bjoern.hoefling <at> bjoernhoefling.de> skribis:
>
> > The test suite consists of an extra package, weighting 465MB
> > compressed. It runs very well. I think the size is worth it. It
> > consists of proprietary things (i.e. lena.jpg). As far as I
> > understand, that is OK, if it doesn't get in the final src/bin
> > store output. Right?
>
> As a rule of thumb, there should not be non-free stuff in the
> derivation graph.
>
> If there’s non-free software, that’s not OK, even if it doesn’t show
> up in the output.
>
> If it’s “just” data (pictures) that are non-free, that’s OK per the
> FSDG:
> <https://www.gnu.org/distros/free-system-distribution-guidelines.html#non-functional-data>.
> If we could replace it with a free variant, I think we should clearly
> encourage it, but lena.jpg is hardly replaceable in this context (I’d
> hope it weren’t around for what it tells about CS, but that’s another
> story…).
What is being downloaded is in the store of cause, but it is only
images and videos, no code. And it is not linked against.
The suite is 400 MB in size and replacing Lena with Linus or even
better pictures and videos of trees would need a fundamental idea of
the algorithms behind OpenCV, with I don't have. So for now I just leave
it as is, though I agree with your fundamental ideas :-)
> > CPU-optimization: I hope I have done everything right. Reading the
> > article from Guix HPC a second time helped a lot. So now it should
> > be compiled with SSE2/NEON being the minimum required instruction
> > set, and dispatches to other ISAs where available.
>
> Great.
>
> > Size: Currently I load a bunch of dependencies in and have one
> > big package as output. guix size is 1.1 GiB. I slightly have the
> > feeling someone could ask to split it in several outputs. Though
> > having one big output was the easiest thing first and I don't know
> > how one would handle inter-dependencies between the different
> > outputs.
>
> 1.1G is a bit too much IMO, indeed. :-)
>
> How much is OpenCV itself? If OpenCV itself is big, it would be nice
> to look at what’s taking up space in there with ‘du’. For instance,
> if there are .a files, we might want to not build them and keep only
> shared objects.
>
> Splitting in separate outputs may or may not make sense. If there are
> examples or large pieces of HTML doc, introducing a “doc” and/or an
> “examples” output may help. Likewise, if there are binaries that
> depend on more stuff that the library itself, introducing a “lib”
> output might be a good idea.
>
> Could you take a look?
I don't know which of the store items was my latest one, I'm currently
recompiling but that takes 1-2 hours to finish.
Most are 50MB, one is 265MB, I suppose that was the latest with all
dependencies and all (free) OpenCV modules. So the 1.1GB was
definitively the convex hull including all dependencies.
The whole thing is more or less only "lib".
I forgot to mention that I ignored the "doc" package: It built one when
I added doxygen as a dependency, but it did not install it. Now that I
know how to manually copy/install files, I could give that another try.
But that would add (to a :doc output, of cause), not substract.
The "bin" part is only 7 MB, so not worth mentioning. There are some
examples which I did not include, because they are not very interesting
in the compiled version, but more for understanding the programming
part from the source code.
OK, I looked into some other packages and tried to split it into pieces
(one piece for now). What I do is just add a phase after install and
copy things manually (copied from the git package definition):
+ (add-after 'install 'split
+ (lambda* (#:key outputs #:allow-other-keys)
+ (let* ((outputs-out (assoc-ref outputs "out"))
+ (outputs-feature2d (assoc-ref outputs "feature2d"))
+ (outputs-face (assoc-ref outputs "face"))
+ (outputs-core (assoc-ref outputs "core"))
+ (libface (string-append outputs-out "/lib/libopencv_face.so"))
+ (libface* (string-append outputs-face "/lib/libopencv_face.so"))
+ (libface3.4 (string-append outputs-out "/lib/libopencv_face.so.3.4"))
+ (libface3.4* (string-append outputs-face "/lib/libopencv_face.so.3.4"))
+ (libface3.4.1 (string-append outputs-out "/lib/libopencv_face.so.3.4.1"))
+ (libface3.4.1* (string-append outputs-face "/lib/libopencv_face.so.3.4.1")))
+
+
+ (mkdir-p (string-append outputs-face "/lib"))
+
+ (for-each (lambda (old new)
+ (copy-file old new)
+ (delete-file old)
+ (chmod new #o555))
+ (list libface libface3.4 libface3.4.1)
+ (list libface* libface3.4* libface3.4.1*))
+ )
+ #t ; TODO: Implement it.
+ ))
+ )))
The problem here is that this doesn't correct the RPATHS and I will get
problems with dependent library parts (The python modules in that case):
phase `strip' succeeded after 0.5 seconds
starting phase `validate-runpath'
validating RUNPATH of 46 binaries in "/gnu/store/f7aqk1my2bdprhgp3gxmnl9227gxf43m-opencv-3.4.1/lib"...
/gnu/store/f7aqk1my2bdprhgp3gxmnl9227gxf43m-opencv-3.4.1/lib/python3.6/site-packages/cv2.cpython-36m-x86_64-linux-gnu.so: error: depends on 'libopencv_face.so.3.4', which cannot be found in RUNPATH ("/gnu/store/f7aqk1my2bdprhgp3gxmnl9227gxf43m-opencv-3.4.1/lib" "/gnu/store/124ymrzp0dwx6qfh4r4r4763sa5k48sv-libsm-1.2.2/lib" "/gnu/store/dbdjmralkrzqn6b093hp69bjljvfr7zm-libice-1.0.9/lib" "/gnu/store/g7sak8qzk7lk06ggn38xpfv5mb8da6kk-libxt-1.1.5/lib" "/gnu/store/n6acaivs0jwiwpidjr551dhdni5kgpcr-glibc-2.26.105-g0890d5379c/lib" "/gnu/store/xfjba1kww8ngdc6nxldd8ly93nh13ayy-gcc-5.5.0-lib/lib" "/gnu/store/xfjba1kww8ngdc6nxldd8ly93nh13ayy-gcc-5.5.0-lib/lib/gcc/x86_64-unknown-linux-gnu/5.5.0/../../..")
validating RUNPATH of 7 binaries in "/gnu/store/f7aqk1my2bdprhgp3gxmnl9227gxf43m-opencv-3.4.1/bin"...
validating RUNPATH of 3 binaries in "/gnu/store/1356brvn9cjilcp2zizm9gx3vigmz74i-opencv-3.4.1-face/lib"...
phase `validate-runpath' failed after 0.3 seconds
So, somehow I have to solve dependencies between the different outputs.
I have no idea how to do that. For a :doc output, that doesn't matter,
but how can I tell Guix the dependency hierachy of different outputs?
> Nitpick:
>
> > + (synopsis "Computer Vision Library")
>
> No need to capitalize.
OK.
> > + (description "OpenCV (Open Source Computer Vision) is a
> > library aimed at
>
> I’d remove the parenthetic part.
OK.
>
> > +real-time computer vision, including several hundred computer
> > +vision algorithms.")
>
> Bonus points if you can expound a little bit, without just listing
> those algorithms though. ;-)
OK, you got me. I will add more lines when the technical problems are
solved.
> Apart from that the patch looks very polished and that’s a pleasure to
> review!
Thanks. It took quite some rounds until that point. But good to know
you enjoyed the result :-)
Björn
PS: Happy Easter and this is not the xxxx-04-01 fool.
[Message part 2 (application/pgp-signature, inline)]
guix-patches <at> gnu.org:bug#30801; Package guix-patches.
(Sun, 01 Apr 2018 12:22:02 GMT) Full text and rfc822 format available.Message #17 received at 30801 <at> debbugs.gnu.org (full text, mbox):
From: ludo <at> gnu.org (Ludovic Courtès) To: Björn Höfling <bjoern.hoefling <at> bjoernhoefling.de> Cc: 30801 <at> debbugs.gnu.org Subject: Re: [bug#30801] [PATCH 0/1] Add opencv Date: Sun, 01 Apr 2018 14:21:38 +0200
Hello Björn,
Björn Höfling <bjoern.hoefling <at> bjoernhoefling.de> skribis:
> On Thu, 15 Mar 2018 22:04:54 +0100
> ludo <at> gnu.org (Ludovic Courtès) wrote:
>
>> Hi Björn,
>>
>> Björn Höfling <bjoern.hoefling <at> bjoernhoefling.de> skribis:
>>
>> > The test suite consists of an extra package, weighting 465MB
>> > compressed. It runs very well. I think the size is worth it. It
>> > consists of proprietary things (i.e. lena.jpg). As far as I
>> > understand, that is OK, if it doesn't get in the final src/bin
>> > store output. Right?
>>
>> As a rule of thumb, there should not be non-free stuff in the
>> derivation graph.
>>
>> If there’s non-free software, that’s not OK, even if it doesn’t show
>> up in the output.
>>
>> If it’s “just” data (pictures) that are non-free, that’s OK per the
>> FSDG:
>> <https://www.gnu.org/distros/free-system-distribution-guidelines.html#non-functional-data>.
>> If we could replace it with a free variant, I think we should clearly
>> encourage it, but lena.jpg is hardly replaceable in this context (I’d
>> hope it weren’t around for what it tells about CS, but that’s another
>> story…).
>
> What is being downloaded is in the store of cause, but it is only
> images and videos, no code. And it is not linked against.
>
> The suite is 400 MB in size and replacing Lena with Linus or even
> better pictures and videos of trees would need a fundamental idea of
> the algorithms behind OpenCV, with I don't have. So for now I just leave
> it as is, though I agree with your fundamental ideas :-)
OK, sounds good.
> Most are 50MB, one is 265MB, I suppose that was the latest with all
> dependencies and all (free) OpenCV modules. So the 1.1GB was
> definitively the convex hull including all dependencies.
>
> The whole thing is more or less only "lib".
>
> I forgot to mention that I ignored the "doc" package: It built one when
> I added doxygen as a dependency, but it did not install it. Now that I
> know how to manually copy/install files, I could give that another try.
> But that would add (to a :doc output, of cause), not substract.
>
> The "bin" part is only 7 MB, so not worth mentioning. There are some
> examples which I did not include, because they are not very interesting
> in the compiled version, but more for understanding the programming
> part from the source code.
OK.
> OK, I looked into some other packages and tried to split it into pieces
> (one piece for now). What I do is just add a phase after install and
> copy things manually (copied from the git package definition):
>
> + (add-after 'install 'split
> + (lambda* (#:key outputs #:allow-other-keys)
> + (let* ((outputs-out (assoc-ref outputs "out"))
> + (outputs-feature2d (assoc-ref outputs "feature2d"))
> + (outputs-face (assoc-ref outputs "face"))
> + (outputs-core (assoc-ref outputs "core"))
> + (libface (string-append outputs-out "/lib/libopencv_face.so"))
> + (libface* (string-append outputs-face "/lib/libopencv_face.so"))
> + (libface3.4 (string-append outputs-out "/lib/libopencv_face.so.3.4"))
> + (libface3.4* (string-append outputs-face "/lib/libopencv_face.so.3.4"))
> + (libface3.4.1 (string-append outputs-out "/lib/libopencv_face.so.3.4.1"))
> + (libface3.4.1* (string-append outputs-face "/lib/libopencv_face.so.3.4.1")))
> +
> +
> + (mkdir-p (string-append outputs-face "/lib"))
> +
> + (for-each (lambda (old new)
> + (copy-file old new)
> + (delete-file old)
> + (chmod new #o555))
> + (list libface libface3.4 libface3.4.1)
> + (list libface* libface3.4* libface3.4.1*))
> + )
> + #t ; TODO: Implement it.
> + ))
> + )))
>
> The problem here is that this doesn't correct the RPATHS and I will get
> problems with dependent library parts (The python modules in that case):
>
> phase `strip' succeeded after 0.5 seconds
> starting phase `validate-runpath'
> validating RUNPATH of 46 binaries in "/gnu/store/f7aqk1my2bdprhgp3gxmnl9227gxf43m-opencv-3.4.1/lib"...
> /gnu/store/f7aqk1my2bdprhgp3gxmnl9227gxf43m-opencv-3.4.1/lib/python3.6/site-packages/cv2.cpython-36m-x86_64-linux-gnu.so: error: depends on 'libopencv_face.so.3.4', which cannot be found in RUNPATH ("/gnu/store/f7aqk1my2bdprhgp3gxmnl9227gxf43m-opencv-3.4.1/lib" "/gnu/store/124ymrzp0dwx6qfh4r4r4763sa5k48sv-libsm-1.2.2/lib" "/gnu/store/dbdjmralkrzqn6b093hp69bjljvfr7zm-libice-1.0.9/lib" "/gnu/store/g7sak8qzk7lk06ggn38xpfv5mb8da6kk-libxt-1.1.5/lib" "/gnu/store/n6acaivs0jwiwpidjr551dhdni5kgpcr-glibc-2.26.105-g0890d5379c/lib" "/gnu/store/xfjba1kww8ngdc6nxldd8ly93nh13ayy-gcc-5.5.0-lib/lib" "/gnu/store/xfjba1kww8ngdc6nxldd8ly93nh13ayy-gcc-5.5.0-lib/lib/gcc/x86_64-unknown-linux-gnu/5.5.0/../../..")
Right, you cannot just move .so files after the fact because
cv2.cpython-36m-x86_64-linux-gnu.so, for instance, has …/lib in its
RUNPATH, not …-lib/lib. If you wanted to do that, the right way to do
it would be by modifying the makefiles/cmakelists so that they
explicitly install libopencv_face to …-lib/lib.
That said, it may be safer/easier to keep all libraries in the same
output.
BTW, are .a files installed? It would probably help save space to not
install them.
Thanks for the update & have a nice week-end!
Ludo’.
guix-patches <at> gnu.org:bug#30801; Package guix-patches.
(Mon, 07 May 2018 18:36:01 GMT) Full text and rfc822 format available.Message #20 received at 30801 <at> debbugs.gnu.org (full text, mbox):
From: Björn Höfling <bjoern.hoefling <at> bjoernhoefling.de> To: ludo <at> gnu.org (Ludovic Courtès) Cc: 30801 <at> debbugs.gnu.org Subject: Re: [bug#30801] Add opencv Date: Mon, 7 May 2018 20:35:47 +0200
[Message part 1 (text/plain, inline)]
Here is another update:
* Kept it in one output.
* No, there are no .a-files.
* I tried to build the documentation, but somehow it did not work.
Postponed for the next update.
* Added more to the description.
* The new ffmpeg 4.0 caused a core dump during test phase, so I set it
explicitly to the old 3.4.
Björn
From 0e234430fec497d3c7702e288474e39d37cb8ae8 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Bj=C3=B6rn=20H=C3=B6fling?=
<bjoern.hoefling <at> bjoernhoefling.de>
Date: Fri, 16 Feb 2018 20:31:37 +0100
Subject: [PATCH] gnu: Add opencv.
* gnu/packages/image-processing.scm (opencv): New variable.
---
gnu/packages/image-processing.scm | 200 ++++++++++++++++++++++++++++++
1 file changed, 200 insertions(+)
diff --git a/gnu/packages/image-processing.scm b/gnu/packages/image-processing.scm
index 6705a726c..ef303e134 100644
--- a/gnu/packages/image-processing.scm
+++ b/gnu/packages/image-processing.scm
@@ -5,6 +5,7 @@
;;; Copyright © 2014 Mark H Weaver <mhw <at> netris.org>
;;; Copyright © 2016 Eric Bavier <bavier <at> member.fsf.org>
;;; Copyright © 2018 Tobias Geerinckx-Rice <me <at> tobias.gr>
+;;; Copyright © 2018 Björn Höfling <bjoern.hoefling <at> bjoernhoefling.de>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -38,12 +39,16 @@
#:use-module (gnu packages gnome)
#:use-module (gnu packages graphics)
#:use-module (gnu packages graphviz)
+ #:use-module (gnu packages gtk)
#:use-module (gnu packages image)
#:use-module (gnu packages maths)
#:use-module (gnu packages perl)
+ #:use-module (gnu packages photo)
#:use-module (gnu packages pkg-config)
+ #:use-module (gnu packages protobuf)
#:use-module (gnu packages python)
#:use-module (gnu packages serialization)
+ #:use-module (gnu packages video)
#:use-module (gnu packages xiph)
#:use-module (gnu packages xml)
#:use-module (gnu packages xorg))
@@ -191,3 +196,198 @@ triangulation. VTK has an extensive information visualization framework, has
a suite of 3D interaction widgets, supports parallel processing, and
integrates with various databases on GUI toolkits such as Qt and Tk.")
(license license:bsd-3)))
+
+(define-public opencv
+ (package
+ (name "opencv")
+ (version "3.4.1")
+ (source (origin
+ (method url-fetch)
+ (uri (string-append "https://github.com/opencv/opencv/archive/"
+ version ".zip"))
+ (file-name (string-append name "-" version ".zip"))
+ (sha256
+ (base32
+ "1g8pvnlkzzp50amd89149hqsbvsc2hq3vk1d6a9fksdcx8ra9g94"))
+ (modules '((guix build utils)))
+ (snippet
+ '(begin
+ ;; Remove external libraries. We have all available in Guix:
+ (delete-file-recursively "3rdparty")
+
+ ;; Milky icon set is non-free:
+ (delete-file-recursively "modules/highgui/src/files_Qt/Milky")
+
+ ;; Some jars found:
+ (for-each delete-file
+ '("modules/java/test/pure_test/lib/junit-4.11.jar"
+ "samples/java/sbt/sbt/sbt-launch.jar"))))))
+ (build-system cmake-build-system)
+ (arguments
+ `(#:configure-flags
+ (list "-DWITH_IPP=OFF"
+ "-DWITH_ITT=OFF"
+
+ ;; CPU-Features:
+ ;; See cmake/OpenCVCompilerOptimizations.cmake
+ ;; (CPU_ALL_OPTIMIZATIONS) for a list of all optimizations
+ ;; BASELINE is the minimum optimization all CPUs must support
+ ;;
+ ;; DISPATCH is the list of optional dispatches
+ "-DCPU_BASELINE=SSE2, NEON"
+
+ "-DCPU_DISPATCH=SSE3,SSSE3,SSE4_1,SSE4_2,AVX,AVX2"
+ "-DCPU_DISPATCH_REQUIRE=SSE3,SSSE3,SSE4_1,SSE4_2,AVX,AVX2"
+
+ "-DBUILD_PERF_TESTS=OFF"
+ "-D BUILD_TESTS=ON"
+
+ (string-append "-DOPENCV_EXTRA_MODULES_PATH="
+ "/tmp/guix-build-opencv-" ,version ".drv-0"
+ "/opencv-contrib/opencv_contrib-" ,version
+ "/modules")
+
+ ;;Define test data:
+ (string-append "-DOPENCV_TEST_DATA_PATH="
+ "/tmp/guix-build-opencv-" ,version ".drv-0"
+ ;;"/opencv-3.4.0"
+ "/opencv-extra/opencv_extra-" ,version
+ "/testdata")
+
+ ;; Is ON by default and would try to rebuild 3rd-party protobuf,
+ ;; which we had removed, which would lead to an error:
+ "-DBUILD_PROTOBUF=OFF"
+
+ ;; Rebuild protobuf files, because we have a slightly different
+ ;; version than the included one. If we would not update, we
+ ;; would get a compile error later:
+ "-DPROTOBUF_UPDATE_FILES=ON"
+
+ ;; xfeatures2d disabled, because it downloads extra binaries from
+ ;; https://github.com/opencv/opencv_3rdparty
+ ;; defined in xfeatures2d/cmake/download_{vgg|bootdesc}.cmake
+ ;; Cmp this bug entry:
+ ;; https://github.com/opencv/opencv_contrib/issues/1131
+ "-DBUILD_opencv_xfeatures2d=OFF")
+ #:phases
+ (modify-phases %standard-phases
+ (add-after 'unpack 'disable-broken-tests
+ (lambda _
+ ;; These tests fails with:
+ ;; vtkXOpenGLRenderWindow (0x723990): Could not find a decent config
+ ;; I think we have no OpenGL support with the Xvfb.
+ (substitute* '("modules/viz/test/test_tutorial3.cpp"
+ "modules/viz/test/test_main.cpp"
+ "modules/viz/test/tests_simple.cpp"
+ "modules/viz/test/test_viz3d.cpp")
+ (("(TEST\\(Viz, )([a-z].*\\).*)" all pre post)
+ (string-append pre "DISABLED_" post)))
+
+ ;; This one fails with "unknown file: Failure"
+ ;; But I couldn't figure out which file was missing:
+ (substitute* (list (string-append
+ "../opencv-contrib/opencv_contrib-"
+ ,version
+ "/modules/face/test/test_face_align.cpp"))
+ (("(TEST\\(CV_Face_FacemarkKazemi, )(can_detect_landmarks\\).*)"
+ all pre post)
+ (string-append pre "DISABLED_" post)))
+
+ ;; Failure reason: Bad accuracy
+ ;; Incorrect count of accurate poses [2nd case]: 90.000000 / 94.000000
+ (substitute* (list (string-append
+ "../opencv-contrib/opencv_contrib-"
+ ,version
+ "/modules/rgbd/test/test_odometry.cpp"))
+ (("(TEST\\(RGBD_Odometry_Rgbd, )(algorithmic\\).*)" all pre post)
+ (string-append pre "DISABLED_" post)))
+ #t))
+
+ ;; Idea copied from ldc.scm (ldc-bootstrap):
+ (add-after 'unpack 'unpack-submodule-sources
+ (lambda* (#:key inputs #:allow-other-keys)
+ (mkdir "../opencv-extra")
+ (mkdir "../opencv-contrib")
+ (let ((unpack (lambda (source target)
+ (with-directory-excursion target
+ (apply invoke "unzip"
+ (list (assoc-ref inputs source)))))))
+ (unpack "opencv-extra" "../opencv-extra")
+ (unpack "opencv-contrib" "../opencv-contrib"))))
+
+ (add-after 'set-paths 'add-ilmbase-include-path
+ (lambda* (#:key inputs #:allow-other-keys)
+ ;; OpenEXR propagates ilmbase, but its include files do not appear
+ ;; in the CPATH, so we need to add "$ilmbase/include/OpenEXR/" to
+ ;; the CPATH to satisfy the dependency on "ImathVec.h".
+ (setenv "CPATH"
+ (string-append (assoc-ref inputs "ilmbase")
+ "/include/OpenEXR"
+ ":" (or (getenv "CPATH") "")))
+ #t))
+ (add-before 'check 'start-xserver
+ (lambda* (#:key inputs #:allow-other-keys)
+ (let ((xorg-server (assoc-ref inputs "xorg-server"))
+ (disp ":1"))
+ (setenv "HOME" (getcwd))
+ (setenv "DISPLAY" disp)
+ ;; There must be a running X server and make check doesn't start one.
+ ;; Therefore we must do it.
+ (zero? (system (format #f "~a/bin/Xvfb ~a &" xorg-server disp)))))))))
+ (native-inputs
+ `(("unzip" ,unzip)
+ ("pkg-config" ,pkg-config)
+ ("xorg-server" ,xorg-server) ; For running the tests
+ ("opencv-extra"
+ ,(origin
+ (method url-fetch)
+ (uri (string-append "https://codeload.github.com/"
+ "opencv/opencv_extra/zip/" version))
+ (file-name (string-append "opencv-extra-" version ".zip"))
+ (sha256
+ (base32 "0wfh3pvfxqydf7hsccp50npcsg37sf6fqi6cd3zkc4qil9zhpbps"))))
+ ("opencv-contrib"
+ ,(origin
+ (method url-fetch)
+ (uri (string-append "https://codeload.github.com/"
+ "opencv/opencv_contrib/zip/" version))
+ (file-name (string-append "opencv-contrib-" version ".zip"))
+ (sha256
+ (base32 "18zm0qmjcdvg90c33gzv0ws0xdaid1xpqzz2xa9l2x12qkr6zj3p"))))))
+ (inputs `(("libjpeg" ,libjpeg)
+ ("libpng" ,libpng)
+ ("jasper" ,jasper)
+ ;; ffmpeg 4.0 causes core dumps in tests.
+ ("ffmpeg" ,ffmpeg-3.4)
+ ("libtiff" ,libtiff)
+ ("hdf5" ,hdf5)
+ ("libgphoto2" ,libgphoto2)
+ ("libwebp" ,libwebp)
+ ("zlib" ,zlib)
+ ("gtkglext" ,gtkglext)
+ ("openexr" ,openexr)
+ ("ilmbase" ,ilmbase)
+ ("gtk+" ,gtk+-2)
+ ("python-numpy" ,python-numpy)
+ ("protobuf" ,protobuf)
+ ("vtk" ,vtk)
+ ("python" ,python)))
+ (synopsis "Computer vision library")
+ (description "OpenCV is a library aimed at
+real-time computer vision, including several hundred computer
+vision algorithms. It can be used to do things like:
+
+@itemize
+@item image and video input and output
+@item image and video processing
+@item displaying
+@item feature recognition
+@item segmentation
+@item facial recognition
+@item stereo vision
+@item structure from motion
+@item augmented reality
+@item machine learning
+@end itemize\n")
+ (home-page "https://opencv.org/")
+ (license license:bsd-3)))
--
2.17.0
[Message part 2 (application/pgp-signature, inline)]
ludo <at> gnu.org (Ludovic Courtès):Björn Höfling <bjoern.hoefling <at> bjoernhoefling.de>:Message #25 received at 30801-done <at> debbugs.gnu.org (full text, mbox):
From: ludo <at> gnu.org (Ludovic Courtès) To: Björn Höfling <bjoern.hoefling <at> bjoernhoefling.de> Cc: 30801-done <at> debbugs.gnu.org Subject: Re: [bug#30801] Add opencv Date: Thu, 10 May 2018 00:01:13 +0200
Hi Björn, Björn Höfling <bjoern.hoefling <at> bjoernhoefling.de> skribis: > Here is another update: > > * Kept it in one output. > * No, there are no .a-files. > * I tried to build the documentation, but somehow it did not work. > Postponed for the next update. > * Added more to the description. > * The new ffmpeg 4.0 caused a core dump during test phase, so I set it > explicitly to the old 3.4. Awesome, thanks for looking into this. > From 0e234430fec497d3c7702e288474e39d37cb8ae8 Mon Sep 17 00:00:00 2001 > From: =?UTF-8?q?Bj=C3=B6rn=20H=C3=B6fling?= > <bjoern.hoefling <at> bjoernhoefling.de> > Date: Fri, 16 Feb 2018 20:31:37 +0100 > Subject: [PATCH] gnu: Add opencv. > > * gnu/packages/image-processing.scm (opencv): New variable. Applied! ‘guix lint’ reports this: gnu/packages/image-processing.scm:201:2: opencv <at> 3.4.1: probably vulnerable to CVE-2018-7712, CVE-2018-7713, CVE-2018-7714 Could you take a look? It could be that 3.4.2 is around the corner and we’ll just update at that point; if not, we may have to apply upstream patches for these issues. Thank you! Ludo’.
guix-patches <at> gnu.org:bug#30801; Package guix-patches.
(Fri, 11 May 2018 09:52:02 GMT) Full text and rfc822 format available.Message #28 received at 30801-done <at> debbugs.gnu.org (full text, mbox):
From: Björn Höfling <bjoern.hoefling <at> bjoernhoefling.de> To: ludo <at> gnu.org (Ludovic Courtès) Cc: 30801-done <at> debbugs.gnu.org Subject: Re: [bug#30801] Add opencv Date: Fri, 11 May 2018 11:51:24 +0200
[Message part 1 (text/plain, inline)]
On Thu, 10 May 2018 00:01:13 +0200 ludo <at> gnu.org (Ludovic Courtès) wrote: > > * gnu/packages/image-processing.scm (opencv): New variable. > > Applied! Thanks. > ‘guix lint’ reports this: > > gnu/packages/image-processing.scm:201:2: opencv <at> 3.4.1: probably > vulnerable to CVE-2018-7712, CVE-2018-7713, CVE-2018-7714 > > Could you take a look? It could be that 3.4.2 is around the corner > and we’ll just update at that point; if not, we may have to apply > upstream patches for these issues. While finally linting, I noticed these too. OpenCV claims this is not an issue: https://github.com/opencv/opencv/issues/10998 Should we mention it somewhere in the code? Is there a formal process to hide or comment specific CVEs? Björn
[Message part 2 (application/pgp-signature, inline)]
guix-patches <at> gnu.org:bug#30801; Package guix-patches.
(Fri, 11 May 2018 12:01:02 GMT) Full text and rfc822 format available.Message #31 received at 30801-done <at> debbugs.gnu.org (full text, mbox):
From: ludo <at> gnu.org (Ludovic Courtès) To: Björn Höfling <bjoern.hoefling <at> bjoernhoefling.de> Cc: 30801-done <at> debbugs.gnu.org Subject: Re: [bug#30801] Add opencv Date: Fri, 11 May 2018 14:00:05 +0200
Hello! Björn Höfling <bjoern.hoefling <at> bjoernhoefling.de> skribis: > On Thu, 10 May 2018 00:01:13 +0200 > ludo <at> gnu.org (Ludovic Courtès) wrote: [...] >> ‘guix lint’ reports this: >> >> gnu/packages/image-processing.scm:201:2: opencv <at> 3.4.1: probably >> vulnerable to CVE-2018-7712, CVE-2018-7713, CVE-2018-7714 >> >> Could you take a look? It could be that 3.4.2 is around the corner >> and we’ll just update at that point; if not, we may have to apply >> upstream patches for these issues. > > While finally linting, I noticed these too. OpenCV claims this is not > an issue: > > https://github.com/opencv/opencv/issues/10998 > > Should we mention it somewhere in the code? Is there a formal process > to hide or comment specific CVEs? The developer’s reasoning makes sense to me (IOW, the CVEs should be against the applications that don’t handle exceptions properly rather than against OpenCV itself.) You can use the ‘lint-hidden-cve’ property to explicitly hide them. Please add a comment with the URL above as well. Thanks, Ludo’.
guix-patches <at> gnu.org:bug#30801; Package guix-patches.
(Sat, 12 May 2018 23:43:02 GMT) Full text and rfc822 format available.Message #34 received at 30801-done <at> debbugs.gnu.org (full text, mbox):
From: Björn Höfling <bjoern.hoefling <at> bjoernhoefling.de> To: ludo <at> gnu.org (Ludovic Courtès) Cc: 30801-done <at> debbugs.gnu.org Subject: Re: [bug#30801] Add opencv Date: Sun, 13 May 2018 01:42:38 +0200
[Message part 1 (text/plain, inline)]
On Fri, 11 May 2018 14:00:05 +0200 ludo <at> gnu.org (Ludovic Courtès) wrote: > >> ‘guix lint’ reports this: > >> > >> gnu/packages/image-processing.scm:201:2: opencv <at> 3.4.1: probably > >> vulnerable to CVE-2018-7712, CVE-2018-7713, CVE-2018-7714 > >> > >> Could you take a look? It could be that 3.4.2 is around the corner > >> and we’ll just update at that point; if not, we may have to apply > >> upstream patches for these issues. > > > > While finally linting, I noticed these too. OpenCV claims this is > > not an issue: > > > > https://github.com/opencv/opencv/issues/10998 > > > > Should we mention it somewhere in the code? Is there a formal > > process to hide or comment specific CVEs? > > The developer’s reasoning makes sense to me (IOW, the CVEs should be > against the applications that don’t handle exceptions properly rather > than against OpenCV itself.) > > You can use the ‘lint-hidden-cve’ property to explicitly hide them. > Please add a comment with the URL above as well. I added a new patch including documentation about lint-hidden-cve: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=31437 Björn
[Message part 2 (application/pgp-signature, inline)]
Debbugs Internal Request <help-debbugs <at> gnu.org>
to internal_control <at> debbugs.gnu.org.
(Sun, 10 Jun 2018 11:24:03 GMT) Full text and rfc822 format available.
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997,2003 nCipher Corporation Ltd,
1994-97 Ian Jackson.