GNU bug report logs - #62071
openjdk@9/10 sources not reproducible

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 62071 in the body.
You can then email your comments to 62071 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Lars-Dominik Braun <lars <at> 6xq.net>
To: bug-guix <at> gnu.org
Subject: openjdk <at> 9/10 sources not reproducible
Date: Thu, 9 Mar 2023 10:48:53 +0100

Hi,

it looks like the (auto-generated) tarballs for openjdk <at> 9 and openjdk <at> 10
changed their hash, causing a hash mismatch via

    guix build -S openjdk <at> 9 --no-substitutes --no-grafts

I’m not sure why it uses these tarballs in the first place, since we
have a hg-download.

Lars

Message #8 received at 62071 <at> debbugs.gnu.org (full text, mbox):

From: Björn Höfling <bjoern.hoefling <at> bjoernhoefling.de>
To: Lars-Dominik Braun <lars <at> 6xq.net>
Cc: 62071 <at> debbugs.gnu.org
Subject: Re: bug#62071: openjdk <at> 9/10 sources not reproducible
Date: Sun, 12 Mar 2023 00:06:55 +0100

[Message part 1 (text/plain, inline)]

On Thu, 9 Mar 2023 10:48:53 +0100
Lars-Dominik Braun <lars <at> 6xq.net> wrote:

> Hi,
> 
> it looks like the (auto-generated) tarballs for openjdk <at> 9 and
> openjdk <at> 10 changed their hash, causing a hash mismatch via
> 
>     guix build -S openjdk <at> 9 --no-substitutes --no-grafts
 

I can confirm this.

I found the old versions of openjdk 9 and 10 on a server of mine. I
will compare old/new tomorrow (oh, better say: later today).

> I’m not sure why it uses these tarballs in the first place, since we
> have a hg-download.

I don't know. Maybe there was no hg-download yet when we added OpenJDK9?

Björn

[Message part 2 (application/pgp-signature, inline)]

Message #11 received at 62071 <at> debbugs.gnu.org (full text, mbox):

From: Björn Höfling <bjoern.hoefling <at> bjoernhoefling.de>
To: Lars-Dominik Braun <lars <at> 6xq.net>
Cc: 62071 <at> debbugs.gnu.org
Subject: Re: bug#62071: openjdk <at> 9/10 sources not reproducible
Date: Sun, 12 Mar 2023 22:00:21 +0100

[Message part 1 (text/plain, inline)]

On Thu, 9 Mar 2023 10:48:53 +0100
Lars-Dominik Braun <lars <at> 6xq.net> wrote:

> Hi,
> 
> it looks like the (auto-generated) tarballs for openjdk <at> 9 and
> openjdk <at> 10 changed their hash, causing a hash mismatch via
> 
>     guix build -S openjdk <at> 9 --no-substitutes --no-grafts
> 
> I’m not sure why it uses these tarballs in the first place, since we
> have a hg-download.

I compared for JDK9 the two tarballs (old and new hash) and there is no
difference in the content (according to diffoscope). Also, if I
hg-clone the repository/tag (and add the .hg_archival.txt file), all
three directory trees have the same hash value according to guix hash
-rx

Thus, it seams like their artifacts are not stable, as we saw it
for autogenerated artifacts on github.

I will check the same for JDK10 and will prepare a patch within the
next two days.

Björn

[Message part 2 (application/pgp-signature, inline)]

Message #14 received at 62071 <at> debbugs.gnu.org (full text, mbox):

From: Simon Tournier <zimon.toutoune <at> gmail.com>
To: Björn Höfling <bjoern.hoefling <at> bjoernhoefling.de>,
 Lars-Dominik Braun <lars <at> 6xq.net>
Cc: 62071 <at> debbugs.gnu.org
Subject: Re: bug#62071: openjdk <at> 9/10 sources not reproducible
Date: Mon, 13 Mar 2023 14:50:51 +0100

Hi,

On dim., 12 mars 2023 at 22:00, Björn Höfling <bjoern.hoefling <at> bjoernhoefling.de> wrote:

> I compared for JDK9 the two tarballs (old and new hash) and there is no
> difference in the content (according to diffoscope). Also, if I
> hg-clone the repository/tag (and add the .hg_archival.txt file), all
> three directory trees have the same hash value according to guix hash
> -rx

So maybe it comes from some parameters of the compressor; maybe they
changed their level.  Well, I do not know how to check that.

Maybe the best is to switch from url-fetch to hg-fetch.  WDYT?

Cheers,
simon

Message #19 received at 62071-done <at> debbugs.gnu.org (full text, mbox):

From: Björn Höfling <bjoern.hoefling <at> bjoernhoefling.de>
To: Lars-Dominik Braun <lars <at> 6xq.net>
Cc: 62071-done <at> debbugs.gnu.org
Subject: Re: bug#62071: openjdk <at> 9/10 sources not reproducible
Date: Thu, 16 Mar 2023 10:12:12 +0100

[Message part 1 (text/plain, inline)]

On Sun, 12 Mar 2023 22:00:21 +0100
Björn Höfling <bjoern.hoefling <at> bjoernhoefling.de> wrote:

> On Thu, 9 Mar 2023 10:48:53 +0100
> Lars-Dominik Braun <lars <at> 6xq.net> wrote:
> 
> > Hi,
> > 
> > it looks like the (auto-generated) tarballs for openjdk <at> 9 and
> > openjdk <at> 10 changed their hash, causing a hash mismatch via
> > 
> >     guix build -S openjdk <at> 9 --no-substitutes --no-grafts
> > 
> > I’m not sure why it uses these tarballs in the first place, since we
> > have a hg-download.  

Changed to hg-download in commit(s):

7636c49b45adb9870cf416c64bde032ec858a820

Thanks for pointing this out.

Björn

[Message part 2 (application/pgp-signature, inline)]

Message #22 received at 62071 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludovic.courtes <at> inria.fr>
To: Björn Höfling <bjoern.hoefling <at> bjoernhoefling.de>
Cc: 62071 <at> debbugs.gnu.org, Lars-Dominik Braun <lars <at> 6xq.net>
Subject: Re: bug#62071: openjdk <at> 9/10 sources not reproducible
Date: Thu, 16 Mar 2023 12:48:19 +0100

Hi Björn,

Björn Höfling <bjoern.hoefling <at> bjoernhoefling.de> skribis:

> I will check the same for JDK10 and will prepare a patch within the
> next two days.

Thanks for 7636c49b45adb9870cf416c64bde032ec858a820 and its parent
commit!

For the record, there are two remaining issues:

  1. Reproducibility of past revisions.  If we lose copies of the
     auto-generated tarballs, then OpenJDK in past revisions of Guix is
     irreparably lost.  We should check whether/how to get them in
     Disarchive + SWH.

  2. Mercurial/SWH bridge.  While SWH has a one-to-one mapping with Git
     (you can ask it for a specific Git commit ID), that’s not true for
     hg.  This is a more general problem, but as things are today,
     there’s no automatic SWH fallback if the upstream hg server
     vanishes.

Thanks,
Ludo’.

Message #25 received at 62071 <at> debbugs.gnu.org (full text, mbox):

From: Jonathan Brielmaier <jonathan.brielmaier <at> web.de>
To: 62071 <at> debbugs.gnu.org
Subject: openjdk <at> 9/10 sources not reproducible
Date: Thu, 16 Mar 2023 13:37:29 +0100

I’m not sure why it uses these tarballs in the first place, since we
have a hg-download.

-> I guess a reason could be that downloading via hg is quite slow.
Thats at least my impression when fetching the "comm" repository for
Thunderbird with mecurial. Tarballs and git checkout tend to be way
faster...

Message #28 received at 62071 <at> debbugs.gnu.org (full text, mbox):

From: Björn Höfling <bjoern.hoefling <at> bjoernhoefling.de>
To: Ludovic Courtès <ludovic.courtes <at> inria.fr>
Cc: 62071 <at> debbugs.gnu.org, Lars-Dominik Braun <lars <at> 6xq.net>
Subject: Re: bug#62071: openjdk <at> 9/10 sources not reproducible
Date: Fri, 17 Mar 2023 23:10:31 +0100

[Message part 1 (text/plain, inline)]

On Thu, 16 Mar 2023 12:48:19 +0100
Ludovic Courtès <ludovic.courtes <at> inria.fr> wrote:

> Hi Björn,
> 
> Björn Höfling <bjoern.hoefling <at> bjoernhoefling.de> skribis:
> 
> > I will check the same for JDK10 and will prepare a patch within the
> > next two days.  
> 
> Thanks for 7636c49b45adb9870cf416c64bde032ec858a820 and its parent
> commit!
> 
> For the record, there are two remaining issues:
> 
>   1. Reproducibility of past revisions.  If we lose copies of the
>      auto-generated tarballs, then OpenJDK in past revisions of Guix
> is irreparably lost.  We should check whether/how to get them in
>      Disarchive + SWH.

How do we do that? Adding git repos to SWH is something I can achieve,
but adding tarballs to SWH was always opaque to me.

I find no reference in the manual about Disarchive. Ideally, is there a
linter for just adding a package to the disarchive database?

 
>   2. Mercurial/SWH bridge.  While SWH has a one-to-one mapping with
> Git (you can ask it for a specific Git commit ID), that’s not true for
>      hg.  This is a more general problem, but as things are today,
>      there’s no automatic SWH fallback if the upstream hg server
>      vanishes.

For git, I know and appreciate that the linter can directly add a
missing repo to SWH. During linting, I recogniced this is missing for
hg.

I was thinking a second time about it and found that not only the newer
development of OpenJDK is on GitHub, but also the older versions are
available. So I could add another patch like this: 

+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/openjdk/jdk9")

WDYT?

Björn

[Message part 2 (application/pgp-signature, inline)]

Message #31 received at 62071 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludovic.courtes <at> inria.fr>
To: Björn Höfling <bjoern.hoefling <at> bjoernhoefling.de>
Cc: 62071 <at> debbugs.gnu.org, Lars-Dominik Braun <lars <at> 6xq.net>
Subject: Re: bug#62071: openjdk <at> 9/10 sources not reproducible
Date: Mon, 20 Mar 2023 10:08:34 +0100

Hello,

Björn Höfling <bjoern.hoefling <at> bjoernhoefling.de> skribis:

> On Thu, 16 Mar 2023 12:48:19 +0100
> Ludovic Courtès <ludovic.courtes <at> inria.fr> wrote:
>
>> Hi Björn,
>> 
>> Björn Höfling <bjoern.hoefling <at> bjoernhoefling.de> skribis:
>> 
>> > I will check the same for JDK10 and will prepare a patch within the
>> > next two days.  
>> 
>> Thanks for 7636c49b45adb9870cf416c64bde032ec858a820 and its parent
>> commit!
>> 
>> For the record, there are two remaining issues:
>> 
>>   1. Reproducibility of past revisions.  If we lose copies of the
>>      auto-generated tarballs, then OpenJDK in past revisions of Guix
>> is irreparably lost.  We should check whether/how to get them in
>>      Disarchive + SWH.
>
> How do we do that? Adding git repos to SWH is something I can achieve,
> but adding tarballs to SWH was always opaque to me.
>
> I find no reference in the manual about Disarchive. Ideally, is there a
> linter for just adding a package to the disarchive database?

SWH periodically ingests the contents of tarballs (not tarballs
themselves) that appear in <https://guix.gnu.org/sources.json>.  We’d
need to check whether it has the contents of those tarballs.

Then <https://disarchive.guix.gnu.org> is populated by the CI job at
<https://ci.guix.gnu.org/jobset/disarchive>.

Are the openjdk 9 and 10 tarballs archived?

Let’s look at their origins as of commit
1e6ddceb8318d413745ca1c9d91fde01b1e0364b.  We can construct their
Disarchive URL by first converting their SHA256 to hex:

--8<---------------cut here---------------start------------->8---
scheme@(guile-user)> ,use(guix base32)
scheme@(guile-user)> ,use(guix base16)
scheme@(guile-user)> (bytevector->base16-string (nix-base32-string->bytevector "01ihmyf7k5z17wbr7xig7y40l9f01d5zjgkcmawn1102hw5kchpq"))
$5 = "f842360b87028460b9aa6c3ef94b0bc0250a883f2ff693173fe197799caf3006"
--8<---------------cut here---------------end--------------->8---

That gives us:

  https://disarchive.guix.gnu.org/sha256/f842360b87028460b9aa6c3ef94b0bc0250a883f2ff693173fe197799caf3006
  https://disarchive.guix.gnu.org/sha256/249fd462bdd32571c6d0a45f3cb698a305c9e4e66b275d25e990ac0184c0dc7f

Both are 404.  But like I wrote, this is expected: they are bzip2
tarballs and Disarchive doesn’t support bzip2 (yet).

>>   2. Mercurial/SWH bridge.  While SWH has a one-to-one mapping with
>> Git (you can ask it for a specific Git commit ID), that’s not true for
>>      hg.  This is a more general problem, but as things are today,
>>      there’s no automatic SWH fallback if the upstream hg server
>>      vanishes.
>
> For git, I know and appreciate that the linter can directly add a
> missing repo to SWH. During linting, I recogniced this is missing for
> hg.
>
> I was thinking a second time about it and found that not only the newer
> development of OpenJDK is on GitHub, but also the older versions are
> available. So I could add another patch like this: 
>
> +              (method git-fetch)
> +              (uri (git-reference
> +                    (url "https://github.com/openjdk/jdk9")
>
> WDYT?

That’s a good idea.  It shouldn’t change the SHA256 (if it does,
something’s wrong) so it looks like an no-brainer.

Thanks!

Ludo’.

Message #34 received at 62071 <at> debbugs.gnu.org (full text, mbox):

From: Simon Tournier <zimon.toutoune <at> gmail.com>
To: Ludovic Courtès <ludovic.courtes <at> inria.fr>,
 Björn Höfling <bjoern.hoefling <at> bjoernhoefling.de>
Cc: 62071 <at> debbugs.gnu.org, Lars-Dominik Braun <lars <at> 6xq.net>
Subject: Re: bug#62071: openjdk <at> 9/10 sources not reproducible
Date: Mon, 03 Apr 2023 23:52:48 +0200

Hi,

On Mon, 20 Mar 2023 at 10:08, Ludovic Courtès <ludovic.courtes <at> inria.fr> wrote:

>> I was thinking a second time about it and found that not only the newer
>> development of OpenJDK is on GitHub, but also the older versions are
>> available. So I could add another patch like this: 
>>
>> +              (method git-fetch)
>> +              (uri (git-reference
>> +                    (url "https://github.com/openjdk/jdk9")
>
> That’s a good idea.  It shouldn’t change the SHA256 (if it does,
> something’s wrong) so it looks like an no-brainer.

Well, by experience, it is rare that the released tarball contain the
exact same content as the tagged commit.  If it is the case (same SHA256
for both tarball and GitHub tagged release), indeed no-brainer. :-)

Else, some work still remain for the complete preservation of Guix. ;-)

Björn, what is the status of this SHA256?

Cheers,
simon

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.