GNU bug report logs - #46780
java-snappy: Test failure on ci.guix.gnu.org

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 46780 in the body.
You can then email your comments to 46780 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Björn Höfling <bjoern.hoefling <at> bjoernhoefling.de>
To: <bug-guix <at> gnu.org>
Subject: java-snappy: Test failure on ci.guix.gnu.org
Date: Thu, 25 Feb 2021 21:52:45 +0100

[Message part 1 (text/plain, inline)]

java-snappy fails as a dependency of several Java evaluations, for
example:

http://ci.guix.gnu.org/eval/120281?status=failed
http://ci.guix.gnu.org/build/359649/log/raw

These failures exist only on ci.guix.gnu.org, but locally it build
perfectly, even with --rounds=100!

Why? Because our buildmachine has too much memory :-) [And I can
remember, we once had that for another package, I just can't remember
again which one].

What should we do about it? Just disable the test, or something more
complicated?

Analysis:

To analyze, I let it once fail with the -K switch to have the directory

/tmp/guix-build-java-snappy-1.1.7.drv-0/

available. Enter the environment with

$ guix environment java-snappy
$ cd /tmp/guix-build-java-snappy-1.1.7.drv-0/
$ source ./environment-variables
$ cd source
$ ant check
--> everything is fine.  

Then increase the memory used by junit: Edit the build.xml file, add a
huge "maxmemory" property like this :

<junit maxmemory="30G" printsummary="true" ...

Either have enough RAM or add some extra swap space, then run again:

$ ant check

It fails. The junit logs can be found in

src/test/test-reports/TEST-org.xerial.snappy.pool.CachingBufferPoolTest.txt

Testsuite: org.xerial.snappy.pool.CachingBufferPoolTest
Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 7.363 sec

Testcase: testDirectByteBuffers took 0.002 sec
Testcase: testSoftReferences took 7.353 sec
        FAILED
count: 2048
junit.framework.AssertionFailedError: count: 2048
        at org.xerial.snappy.pool.CachingBufferPoolTest.testSoftReferences(Unknown
Source)

Testcase: testArrays took 0 sec
Testcase: testAdjustSize took 0 sec

If you look at the test case source code, it tries to allocate about
20GB and hopes that this fails. It just does not fail on huge machines...

Björn

[Message part 2 (application/pgp-signature, inline)]

Message #8 received at 46780 <at> debbugs.gnu.org (full text, mbox):

From: zimoun <zimon.toutoune <at> gmail.com>
To: Björn Höfling <bjoern.hoefling <at> bjoernhoefling.de>,
 46780 <at> debbugs.gnu.org
Subject: Re: bug#46780: java-snappy: Test failure on ci.guix.gnu.org
Date: Fri, 26 Feb 2021 23:08:42 +0100

Hi,

On Thu, 25 Feb 2021 at 21:52, Björn Höfling <bjoern.hoefling <at> bjoernhoefling.de> wrote:

> Then increase the memory used by junit: Edit the build.xml file, add a
> huge "maxmemory" property like this :
>
> <junit maxmemory="30G" printsummary="true" ...

[...]

> If you look at the test case source code, it tries to allocate about
> 20GB and hopes that this fails. It just does not fail on huge machines...

What about adding a phase just before the test?  For example in this
build.xml file fix the max memory to 20G if it is greater.


All the best,
simon

Message #11 received at 46780 <at> debbugs.gnu.org (full text, mbox):

From: Björn Höfling <bjoern.hoefling <at> bjoernhoefling.de>
To: zimoun <zimon.toutoune <at> gmail.com>
Cc: 46780 <at> debbugs.gnu.org
Subject: Re: bug#46780: java-snappy: Test failure on ci.guix.gnu.org
Date: Sat, 27 Feb 2021 14:58:47 +0100

[Message part 1 (text/plain, inline)]

On Fri, 26 Feb 2021 23:08:42 +0100
zimoun <zimon.toutoune <at> gmail.com> wrote:

> Hi,
> 
> On Thu, 25 Feb 2021 at 21:52, Björn Höfling
> <bjoern.hoefling <at> bjoernhoefling.de> wrote:
> 
> > Then increase the memory used by junit: Edit the build.xml file,
> > add a huge "maxmemory" property like this :
> >
> > <junit maxmemory="30G" printsummary="true" ...  
> 
> [...]
> 
> > If you look at the test case source code, it tries to allocate about
> > 20GB and hopes that this fails. It just does not fail on huge
> > machines...  
> 
> What about adding a phase just before the test?  For example in this
> build.xml file fix the max memory to 20G if it is greater.

The build.xml is auto-generated by our ant-build-system. So we could
add some parameter into it. But I found this too much of an overkill,
and working on the ant-build-system would mean this goes into staging
(we have too many Java package).

Thus, at first I thought the idea with the additional phase wouldn't
fit here. But at the second glimpse: The build.xml file definitely
already exists before running the tests (even before build). So we
could add a phase that substitutestars and adds the maxmemory parameter
in the build.xml.

Björn

[Message part 2 (application/pgp-signature, inline)]

Message #16 received at 46780-done <at> debbugs.gnu.org (full text, mbox):

From: Björn Höfling <bjoern.hoefling <at> bjoernhoefling.de>
To: zimoun <zimon.toutoune <at> gmail.com>
Cc: 46780-done <at> debbugs.gnu.org
Subject: Re: bug#46780: java-snappy: Test failure on ci.guix.gnu.org
Date: Sun, 28 Feb 2021 00:29:27 +0100

[Message part 1 (text/plain, inline)]

On Sat, 27 Feb 2021 14:58:47 +0100
Björn Höfling <bjoern.hoefling <at> bjoernhoefling.de> wrote:

> On Fri, 26 Feb 2021 23:08:42 +0100
> zimoun <zimon.toutoune <at> gmail.com> wrote:
> 
> > What about adding a phase just before the test?  For example in this
> > build.xml file fix the max memory to 20G if it is greater.  

Thanks for the idea.

I first tested it with maxmemory="1M" and this fails with a OOM-error
on my pc, thus I knew this setting is really honored.

Now I set it to 2G and sent a patch under commit:

23dcf4339d1dc102b2c509a151734f4caff793bd

And it works now on Berlin:

https://ci.guix.gnu.org/build/361691/details

Closing.

Björn

PS: While looking at the Cuirass logs, I remembered the other
memory-problem we head, it was with java-kafka-clients:

https://ci.guix.gnu.org/build/361612/details
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=40554

At first sight, we could try the same trick here. I will investigate
this in the next week, if not someone else is quicker.

[Message part 2 (application/pgp-signature, inline)]

Message #19 received at 46780-done <at> debbugs.gnu.org (full text, mbox):

From: zimoun <zimon.toutoune <at> gmail.com>
To: Björn Höfling <bjoern.hoefling <at> bjoernhoefling.de>
Cc: 46780-done <at> debbugs.gnu.org
Subject: Re: bug#46780: java-snappy: Test failure on ci.guix.gnu.org
Date: Mon, 01 Mar 2021 11:50:34 +0100

Hi,

On Sun, 28 Feb 2021 at 00:29, Björn Höfling <bjoern.hoefling <at> bjoernhoefling.de> wrote:

> I first tested it with maxmemory="1M" and this fails with a OOM-error
> on my pc, thus I knew this setting is really honored.
>
> Now I set it to 2G and sent a patch under commit:
>
> 23dcf4339d1dc102b2c509a151734f4caff793bd
>
> And it works now on Berlin:
>
> https://ci.guix.gnu.org/build/361691/details
>
> Closing.

Cool if it worked! :-)


Cheers,
simon

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.