GNU bug report logs - #46830
[PATCH Added hdf5-1.12-parallel-openmpi] * gnu/packages/maths.scm (hdf5-1.12-parallel-openmpi): New package based on HDF5 1.12.0

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 46830 in the body.
You can then email your comments to 46830 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Gerd Heber <gerd.heber <at> gmail.com>
To: guix-patches <at> gnu.org
Cc: Gerd Heber <gheber <at> hdfgroup.org>
Subject: [PATCH Added hdf5-1.12-parallel-openmpi] * gnu/packages/maths.scm
 (hdf5-1.12-parallel-openmpi): New package based on HDF5 1.12.0
Date: Sun, 28 Feb 2021 07:33:14 -0600

---
 gnu/packages/maths.scm | 34 +++++++++++++++++++++++++++++++++-
 1 file changed, 33 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/maths.scm b/gnu/packages/maths.scm
index 58433d4b0c..df101635ab 100644
--- a/gnu/packages/maths.scm
+++ b/gnu/packages/maths.scm
@@ -1451,7 +1451,7 @@ Swath).")
     (license (license:non-copyleft home-page))))
 
 (define-public hdf5-parallel-openmpi
-  (package/inherit hdf5-1.10                      ;use the latest
+  (package/inherit hdf5-1.10
     (name "hdf5-parallel-openmpi")
     (inputs
      `(("mpi" ,openmpi)
@@ -1482,6 +1482,38 @@ Swath).")
                #t))))))
     (synopsis "Management suite for data with parallel IO support")))
 
+(define-public hdf5-1.12-parallel-openmpi
+  (package/inherit hdf5-1.12
+    (name "hdf5-1.12-parallel-openmpi")
+    (inputs
+     `(("mpi" ,openmpi)
+       ,@(package-inputs hdf5)))
+    (arguments
+     (substitute-keyword-arguments (package-arguments hdf5)
+       ((#:configure-flags flags)
+        ``("--enable-parallel"
+           ,@(delete "--enable-cxx"
+                     (delete "--enable-threadsafe" ,flags))))
+       ((#:phases phases)
+        `(modify-phases ,phases
+           (add-after 'build 'mpi-setup
+             ,%openmpi-setup)
+           (add-before 'check 'patch-tests
+             (lambda _
+               ;; OpenMPI's mpirun will exit with non-zero status if it
+               ;; detects an "abnormal termination", i.e. any process not
+               ;; calling MPI_Finalize().  Since the test is explicitly
+               ;; avoiding MPI_Finalize so as not to have at_exit and thus
+               ;; H5C_flush_cache from being called, mpirun will always
+               ;; complain, so turn this test off.
+               (substitute* "testpar/Makefile"
+                 (("(^TEST_PROG_PARA.*)t_pflush1(.*)" front back)
+                  (string-append front back "\n")))
+               (substitute* "tools/test/h5diff/testph5diff.sh"
+                 (("/bin/sh") (which "sh")))
+               #t))))))
+    (synopsis "Management suite for data with parallel IO support")))
+
 (define-public hdf5-blosc
   (package
     (name "hdf5-blosc")
-- 
2.30.1

Message #8 received at 46830 <at> debbugs.gnu.org (full text, mbox):

From: Gerd Heber via web <issues.guix.gnu.org <at> elephly.net>
To: 46830 <at> debbugs.gnu.org
Subject: [PATCH Added hdf5-1.12-parallel-openmpi] * gnu/packages/maths.scm
 (hdf5-1.12-parallel-openmpi): New package based on HDF5 1.12.0
Date: Sun, 28 Feb 2021 23:42:43 +0100

This needs to be a separate package since (as with all HDF5 major releases) it's not 100% API compatible with HDF5 1.10. It would break other packages, e.g., `petsc-lite`

Message #11 received at 46830 <at> debbugs.gnu.org (full text, mbox):

From: zimoun <zimon.toutoune <at> gmail.com>
To: Gerd Heber <gerd.heber <at> gmail.com>, 46830 <at> debbugs.gnu.org
Cc: Gerd Heber <gheber <at> hdfgroup.org>
Subject: Re: [bug#46830] [PATCH Added hdf5-1.12-parallel-openmpi] *
 gnu/packages/maths.scm (hdf5-1.12-parallel-openmpi): New package based on
 HDF5 1.12.0
Date: Mon, 01 Mar 2021 11:41:01 +0100

Hi,

On Sun, 28 Feb 2021 at 07:33, Gerd Heber <gerd.heber <at> gmail.com> wrote:

> +(define-public hdf5-1.12-parallel-openmpi
> +  (package/inherit hdf5-1.12
> +    (name "hdf5-1.12-parallel-openmpi")
> +    (inputs
> +     `(("mpi" ,openmpi)
> +       ,@(package-inputs hdf5)))
> +    (arguments
> +     (substitute-keyword-arguments (package-arguments hdf5)
> +       ((#:configure-flags flags)
> +        ``("--enable-parallel"
> +           ,@(delete "--enable-cxx"
> +                     (delete "--enable-threadsafe" ,flags))))
> +       ((#:phases phases)
> +        `(modify-phases ,phases
> +           (add-after 'build 'mpi-setup
> +             ,%openmpi-setup)
> +           (add-before 'check 'patch-tests
> +             (lambda _
> +               ;; OpenMPI's mpirun will exit with non-zero status if it
> +               ;; detects an "abnormal termination", i.e. any process not
> +               ;; calling MPI_Finalize().  Since the test is explicitly
> +               ;; avoiding MPI_Finalize so as not to have at_exit and thus
> +               ;; H5C_flush_cache from being called, mpirun will always
> +               ;; complain, so turn this test off.
> +               (substitute* "testpar/Makefile"
> +                 (("(^TEST_PROG_PARA.*)t_pflush1(.*)" front back)
> +                  (string-append front back "\n")))
> +               (substitute* "tools/test/h5diff/testph5diff.sh"
> +                 (("/bin/sh") (which "sh")))
> +               #t))))))
> +    (synopsis "Management suite for data with parallel IO support")))
> +

Why duplicates?  Something like:

--8<---------------cut here---------------start------------->8---
(define-public hdf5-1.12-parallel-openmpi
  (package/inherit hdf5-parallel-openmpi
    (version (package-version hdf5-1.12))
    (source (package-source hdf5-1.12)))
--8<---------------cut here---------------end--------------->8---

seems simpler.


All the best,
simon

Message #14 received at 46830 <at> debbugs.gnu.org (full text, mbox):

From: zimoun <zimon.toutoune <at> gmail.com>
To: Gerd Heber <gerd.heber <at> gmail.com>, 46830 <at> debbugs.gnu.org
Subject: Re: [bug#46830] [PATCH Added hdf5-1.12-parallel-openmpi] *
 gnu/packages/maths.scm (hdf5-1.12-parallel-openmpi): New package based on
 HDF5 1.12.0
Date: Tue, 2 Mar 2021 10:41:28 +0100

Hi,

(Please CC the bug number.)

On Tue, 2 Mar 2021 at 01:32, Gerd Heber <gerd.heber <at> gmail.com> wrote:
> On Mon, Mar 1, 2021 at 4:45 AM zimoun <zimon.toutoune <at> gmail.com> wrote:

> > --8<---------------cut here---------------start------------->8---
> > (define-public hdf5-1.12-parallel-openmpi
> >   (package/inherit hdf5-parallel-openmpi
> >     (version (package-version hdf5-1.12))
> >     (source (package-source hdf5-1.12)))
> > --8<---------------cut here---------------end--------------->8---

> Hey, thanks, very clever! Should I submit a new patch or is there a
> way to patch the patch?

The usual way is to send a v2 which is the patch to apply to the Git
tree (and *not* a patch to apply to a patch to apply to the tree).

However, personally I am not in favor to add to Guix proper too much
versions of variant packages; but that's just my humble opinion.
Instead, it could be submitted to specific channels, for instance
<https://gitlab.inria.fr/guix-hpc/guix-hpc> or any other listed in
Guix-HPC.


Cheers,
simon

Message #17 received at 46830 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: zimoun <zimon.toutoune <at> gmail.com>
Cc: 46830 <at> debbugs.gnu.org, Gerd Heber <gerd.heber <at> gmail.com>
Subject: Re: bug#46830: [PATCH Added hdf5-1.12-parallel-openmpi] *
 gnu/packages/maths.scm (hdf5-1.12-parallel-openmpi): New package based on
 HDF5 1.12.0
Date: Tue, 02 Mar 2021 20:39:38 +0100

zimoun <zimon.toutoune <at> gmail.com> skribis:

> On Tue, 2 Mar 2021 at 01:32, Gerd Heber <gerd.heber <at> gmail.com> wrote:
>> On Mon, Mar 1, 2021 at 4:45 AM zimoun <zimon.toutoune <at> gmail.com> wrote:
>
>> > --8<---------------cut here---------------start------------->8---
>> > (define-public hdf5-1.12-parallel-openmpi
>> >   (package/inherit hdf5-parallel-openmpi
>> >     (version (package-version hdf5-1.12))
>> >     (source (package-source hdf5-1.12)))
>> > --8<---------------cut here---------------end--------------->8---
>
>> Hey, thanks, very clever! Should I submit a new patch or is there a
>> way to patch the patch?
>
> The usual way is to send a v2 which is the patch to apply to the Git
> tree (and *not* a patch to apply to a patch to apply to the tree).
>
> However, personally I am not in favor to add to Guix proper too much
> versions of variant packages; but that's just my humble opinion.
> Instead, it could be submitted to specific channels, for instance
> <https://gitlab.inria.fr/guix-hpc/guix-hpc> or any other listed in
> Guix-HPC.

In this particular case, I think the proposed variant is fine; it’s
consistent with what we’re already doing with hdf5.

Ludo’.

Message #20 received at 46830 <at> debbugs.gnu.org (full text, mbox):

From: Gerd Heber <gerd.heber <at> gmail.com>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 46830 <at> debbugs.gnu.org, zimoun <zimon.toutoune <at> gmail.com>
Subject: Re: bug#46830: [PATCH Added hdf5-1.12-parallel-openmpi] *
 gnu/packages/maths.scm (hdf5-1.12-parallel-openmpi): New package based on
 HDF5 1.12.0
Date: Wed, 3 Mar 2021 08:10:05 -0600

Ludovic, how are you? Thanks for taking the time to reply. This
conversation makes
me wonder what the Guix model for packages such as HDF5 might be.
On the one hand, there should be defaults for typical, i.e., non-HPC users, and
they probably belong into gnu/packages/maths.scm. Once you add MPI to the
mix, things get a little more complicated, and channels, such as
Guix-HPC might be
the better place. (?) Should we run our own channel?

In HDF5's case, we also use versions in a peculiar way, where the expectation is
that 1.8.x, 1.10.x, and 1.12.x, etc. version are not compatible for
end-user applications,
mostly because of API changes.

What's your recommendation? Maybe (hdf5-1.6?), hdf5-1.8, hdf-1.10, and hdf5-1.12
belong into maths.scm, plus the thread-safe builds, but not
hdf5-parallel-openmpi?
I tried to build hdf5-1.8.22-parallel-openmpi, but some of the (MPI)
atomicity tests fail
with the OpenMPI version used in the current hdf5-parallel-openmpi.
And then there is MPICH...

I would also like to see HDFView as a Guix package. We have a Spack package, but
the Java story on Guix (I don't blame you ;-) is a little confusing.

I'm sold on the merits of Guix and you are doing fantastic work.
What's your recommendation, and what can we (The HDF Group) do to help?

Best, G.

On Tue, Mar 2, 2021 at 1:39 PM Ludovic Courtès <ludo <at> gnu.org> wrote:
>
> zimoun <zimon.toutoune <at> gmail.com> skribis:
>
> > On Tue, 2 Mar 2021 at 01:32, Gerd Heber <gerd.heber <at> gmail.com> wrote:
> >> On Mon, Mar 1, 2021 at 4:45 AM zimoun <zimon.toutoune <at> gmail.com> wrote:
> >
> >> > --8<---------------cut here---------------start------------->8---
> >> > (define-public hdf5-1.12-parallel-openmpi
> >> >   (package/inherit hdf5-parallel-openmpi
> >> >     (version (package-version hdf5-1.12))
> >> >     (source (package-source hdf5-1.12)))
> >> > --8<---------------cut here---------------end--------------->8---
> >
> >> Hey, thanks, very clever! Should I submit a new patch or is there a
> >> way to patch the patch?
> >
> > The usual way is to send a v2 which is the patch to apply to the Git
> > tree (and *not* a patch to apply to a patch to apply to the tree).
> >
> > However, personally I am not in favor to add to Guix proper too much
> > versions of variant packages; but that's just my humble opinion.
> > Instead, it could be submitted to specific channels, for instance
> > <https://gitlab.inria.fr/guix-hpc/guix-hpc> or any other listed in
> > Guix-HPC.
>
> In this particular case, I think the proposed variant is fine; it’s
> consistent with what we’re already doing with hdf5.
>
> Ludo’.

Message #23 received at 46830 <at> debbugs.gnu.org (full text, mbox):

From: zimoun <zimon.toutoune <at> gmail.com>
To: Gerd Heber <gerd.heber <at> gmail.com>, Ludovic Courtès
 <ludo <at> gnu.org>
Cc: 46830 <at> debbugs.gnu.org
Subject: Re: bug#46830: [PATCH Added hdf5-1.12-parallel-openmpi] *
 gnu/packages/maths.scm (hdf5-1.12-parallel-openmpi): New package based on
 HDF5 1.12.0
Date: Fri, 05 Mar 2021 02:08:57 +0100

Hi Gerd,

On Wed, 03 Mar 2021 at 08:10, Gerd Heber <gerd.heber <at> gmail.com> wrote:

> What's your recommendation? Maybe (hdf5-1.6?), hdf5-1.8, hdf-1.10, and hdf5-1.12
> belong into maths.scm, plus the thread-safe builds, but not
> hdf5-parallel-openmpi?

From my understanding, hdf5 at versions 1.8 and 1.10 are used by other
packages.  When those very packages will not use at one of these
versions, I guess the very version will be removed.

hdf5-parallel-openmpi is used by petsc-openmpi for instance.  And this
hdf5 variant is built with version 1.10.  Since there is no package that
requires hdf5-parallel-openmpi at another version than 1.10, I do not
see the point to include it in Guix proper.  Especially when the custom
variant is straightforward to locally create and buildable on a
reasonable amount of time.

However, these words are not totally acceptable. :-) If I take the shoes
of a regular scientist, then they only wants the package at hand and not
necessary RTFM how to do package transformations.

> I tried to build hdf5-1.8.22-parallel-openmpi, but some of the (MPI)
> atomicity tests fail
> with the OpenMPI version used in the current hdf5-parallel-openmpi.

Yes, and we could imagine different versions of openmpi.  And then
compiled with different compiler versions, etc…

> And then there is MPICH...

…and the matrix of combinations is exploding. ;-)


One issue with the channel is to provide substitutes for that channel.
For example, the channel guix-science uses TravisCI to build the package
from GitHub.

<https://github.com/guix-science/guix-science/blob/master/.github/workflows/build.yml>


That’s said, the cons about channels–and so the pros about include the
hdf5 variant in Guix proper–is to keep consistency and detect breakage:
Guix proper updates a package that becomes incompatible with one the
variant living a channel.  All in Guix proper, then Guix CI will detect
it.  Some dependencies in Guix proper and hdf variant in a channel, then
the channel CI probably not since nothing changed (from the channel
side) and so nothing triggered a build.  I do not know.

Well, let stay pragmatic. :-)


> I would also like to see HDFView as a Guix package. We have a Spack
> package, but

It would be really cool!


> I'm sold on the merits of Guix and you are doing fantastic work.
> What's your recommendation, and what can we (The HDF Group) do to help?

Thanks the HDF Group for their interest on Guix.


Where the package definition ends (channel vs Guix proper) is one thing,
maybe a start should to have these hdf5 variant definitions.  Then from
a pragmatic point of view, depending on these definitions (number,
length, etc.), they could ends in (gnu packages maths) or maybe its own
module (gnu packages hdf) or maybe in a channel.   WDYT?


All the best,
simon

Message #26 received at 46830 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludovic.courtes <at> inria.fr>
To: Gerd Heber <gerd.heber <at> gmail.com>
Cc: 46830 <at> debbugs.gnu.org, zimoun <zimon.toutoune <at> gmail.com>
Subject: Re: bug#46830: [PATCH Added hdf5-1.12-parallel-openmpi] *
 gnu/packages/maths.scm (hdf5-1.12-parallel-openmpi): New package based on
 HDF5 1.12.0
Date: Mon, 08 Mar 2021 15:01:42 +0100

Hi Gerd,

Gerd Heber <gerd.heber <at> gmail.com> skribis:

> Ludovic, how are you? Thanks for taking the time to reply. This
> conversation makes
> me wonder what the Guix model for packages such as HDF5 might be.
> On the one hand, there should be defaults for typical, i.e., non-HPC users, and
> they probably belong into gnu/packages/maths.scm. Once you add MPI to the
> mix, things get a little more complicated, and channels, such as
> Guix-HPC might be
> the better place. (?) Should we run our own channel?

As you write, the goal for packages in Guix proper should be to have
“good defaults”, and possibly useful variants, as is currently the case
with HDF5.

We won’t keep all versions and variants in Guix proper because that’d be
too much of a maintenance burden.  For “unusual” variants, I recommend
maintaining your own channel.  (This is something we did at Inria for
example with an Open MPI variant that includes the MPI1 compatibility
layer.)

Additionally, you can use things like ‘--with-input=openmpi=mpich’.  We
may eventually get support for “parameterized packages”, which will
allow users to choose between HDF5 variants from the command line:

  https://lists.gnu.org/archive/html/guix-devel/2020-11/msg00312.html

> What's your recommendation? Maybe (hdf5-1.6?), hdf5-1.8, hdf-1.10, and hdf5-1.12
> belong into maths.scm, plus the thread-safe builds, but not
> hdf5-parallel-openmpi?

For Guix proper, I think we should keep the number of HDF5 variants
stable.  We should remove old versions once they’re no longer needed by
any in-tree package.

> I tried to build hdf5-1.8.22-parallel-openmpi, but some of the (MPI)
> atomicity tests fail
> with the OpenMPI version used in the current hdf5-parallel-openmpi.
> And then there is MPICH...
>
> I would also like to see HDFView as a Guix package. We have a Spack package, but
> the Java story on Guix (I don't blame you ;-) is a little confusing.

That’s a different story :-) but please don’t hesitate to share your
experience, frustration, and questions regarding Java on the mailing
lists, I’m sure you’ll get some guidance.

> I'm sold on the merits of Guix and you are doing fantastic work.
> What's your recommendation, and what can we (The HDF Group) do to help?

It’s great to get feedback from upstream; you can certainly help us make
wise(r) decisions regarding HDF5 packaging, in particular in deciding
which variants and versions make sense, what defaults are reasonable,
and how well they work in an HPC and non-HPC setting.

WDYT?  What changes would you like to see?

Thanks for your interest!

Ludo’.

Message #31 received at 46830-done <at> debbugs.gnu.org (full text, mbox):

From: Andreas Enge <andreas <at> enge.fr>
To: 46830-done <at> debbugs.gnu.org
Subject: Closing
Date: Wed, 13 Sep 2023 11:29:22 +0200

Hello,

thanks for your patch, that I only see just now (teams were not yet a thing
in 2021...). This version of hdf5 has been retired, so I am closing this issue.
It might be useful to add a parallel package to hdf5 <at> 1.14 instead.

Andreas

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.