GNU bug report logs - #32161
(guix store deduplication) fails with large files

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 32161 in the body.
You can then email your comments to 32161 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Ricardo Wurmus <rekado <at> elephly.net>
To: bug-guix <at> gnu.org
Cc: ludo <at> gnu.org
Subject: seek out of range
Date: Sun, 15 Jul 2018 07:05:59 +0200

I’m getting this bug on berlin.guixsd.org.  The store is several hundred
GB in size.  I cannot reproduce this on a machine with a smaller store.

--8<---------------cut here---------------start------------->8---
In guix/store/deduplication.scm:
    62:18  1 (nar-sha256 _)
In unknown file:
           0 (seek #<output: string 11b20e0> 0 1)

ERROR: In procedure seek:
Value out of range -2147483648 to 2147483647: 4770726968
@ hook-failed /gnu/store/qjxwff3fajh350chpswbb6x9q2m4c3sd-texlive-texmf-2017.drv - 256 builder for `/gnu/store/qjxwff3fajh350chpswbb6x9q2m4c3sd-texlive-texmf-2017.drv' failed with exit code 1
--8<---------------cut here---------------end--------------->8---

-- 
Ricardo

Message #8 received at 32161 <at> debbugs.gnu.org (full text, mbox):

From: Ricardo Wurmus <rekado <at> elephly.net>
To: 32161 <at> debbugs.gnu.org
Subject: Re: seek out of range
Date: Sun, 15 Jul 2018 09:30:07 +0200

Ricardo Wurmus <rekado <at> elephly.net> writes:

> I’m getting this bug on berlin.guixsd.org.  The store is several hundred
> GB in size.  I cannot reproduce this on a machine with a smaller store.

This is not correct.  I cannot reproduce this on a machine where
deduplication has been disabled.

> --8<---------------cut here---------------start------------->8---
> In guix/store/deduplication.scm:
>     62:18  1 (nar-sha256 _)
> In unknown file:
>            0 (seek #<output: string 11b20e0> 0 1)
>
> ERROR: In procedure seek:
> Value out of range -2147483648 to 2147483647: 4770726968
> @ hook-failed /gnu/store/qjxwff3fajh350chpswbb6x9q2m4c3sd-texlive-texmf-2017.drv - 256 builder for `/gnu/store/qjxwff3fajh350chpswbb6x9q2m4c3sd-texlive-texmf-2017.drv' failed with exit code 1
> --8<---------------cut here---------------end--------------->8---

Line 62 is (port-position wrapper).  “seek” takes an integer as the
offset and the range it reports is that of the minimum and maximum
values of a 32 bit integer.

So… does this mean that we need some other way of counting written bytes
to get rid of “counting-wrapper-port”?

On berlin.guixsd.org I may have to temporarily disable deduplication.

--
Ricardo

Message #11 received at 32161 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Ricardo Wurmus <rekado <at> elephly.net>
Cc: 32161 <at> debbugs.gnu.org
Subject: Re: bug#32161: seek out of range
Date: Thu, 19 Jul 2018 18:03:24 +0200

Hi Ricardo,

Ricardo Wurmus <rekado <at> elephly.net> skribis:

> Ricardo Wurmus <rekado <at> elephly.net> writes:
>
>> I’m getting this bug on berlin.guixsd.org.  The store is several hundred
>> GB in size.  I cannot reproduce this on a machine with a smaller store.
>
> This is not correct.  I cannot reproduce this on a machine where
> deduplication has been disabled.
>
>> --8<---------------cut here---------------start------------->8---
>> In guix/store/deduplication.scm:
>>     62:18  1 (nar-sha256 _)
>> In unknown file:
>>            0 (seek #<output: string 11b20e0> 0 1)
>>
>> ERROR: In procedure seek:
>> Value out of range -2147483648 to 2147483647: 4770726968
>> @ hook-failed /gnu/store/qjxwff3fajh350chpswbb6x9q2m4c3sd-texlive-texmf-2017.drv - 256 builder for `/gnu/store/qjxwff3fajh350chpswbb6x9q2m4c3sd-texlive-texmf-2017.drv' failed with exit code 1
>> --8<---------------cut here---------------end--------------->8---

For the record, this code can be executed through ‘guix offload’, via
‘restore-file-set’.

> Line 62 is (port-position wrapper).  “seek” takes an integer as the
> offset and the range it reports is that of the minimum and maximum
> values of a 32 bit integer.

I have some good news!  I fixed this in Guile commit
d677aca5c5e5b3a9f71af57243169904ba4a712c.

Bad news, we can’t really work around it on the Guix side.

There’s also another bug: the ‘write’ procedure of a custom binary
output port is never called when passing a bytevector bigger than 2^32.

> So… does this mean that we need some other way of counting written bytes
> to get rid of “counting-wrapper-port”?

Commit 83099892e0cf0d9c59f5e1a0774331026e48baa8 gets rid of it, but
that’s mostly for aesthetic and performance reasons.

> On berlin.guixsd.org I may have to temporarily disable deduplication.

I’m not sure that’d help.

Ludo’.

Message #20 received at 32161-done <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Ricardo Wurmus <rekado <at> elephly.net>
Cc: 32161-done <at> debbugs.gnu.org
Subject: Re: bug#32161: seek out of range
Date: Tue, 09 Oct 2018 09:31:17 +0200

Hi,

ludo <at> gnu.org (Ludovic Courtès) skribis:

> Ricardo Wurmus <rekado <at> elephly.net> skribis:
>
>> Ricardo Wurmus <rekado <at> elephly.net> writes:
>>
>>> I’m getting this bug on berlin.guixsd.org.  The store is several hundred
>>> GB in size.  I cannot reproduce this on a machine with a smaller store.
>>
>> This is not correct.  I cannot reproduce this on a machine where
>> deduplication has been disabled.
>>
>>> --8<---------------cut here---------------start------------->8---
>>> In guix/store/deduplication.scm:
>>>     62:18  1 (nar-sha256 _)
>>> In unknown file:
>>>            0 (seek #<output: string 11b20e0> 0 1)
>>>
>>> ERROR: In procedure seek:
>>> Value out of range -2147483648 to 2147483647: 4770726968
>>> @ hook-failed /gnu/store/qjxwff3fajh350chpswbb6x9q2m4c3sd-texlive-texmf-2017.drv - 256 builder for `/gnu/store/qjxwff3fajh350chpswbb6x9q2m4c3sd-texlive-texmf-2017.drv' failed with exit code 1
>>> --8<---------------cut here---------------end--------------->8---
>
> For the record, this code can be executed through ‘guix offload’, via
> ‘restore-file-set’.
>
>> Line 62 is (port-position wrapper).  “seek” takes an integer as the
>> offset and the range it reports is that of the minimum and maximum
>> values of a 32 bit integer.
>
> I have some good news!  I fixed this in Guile commit
> d677aca5c5e5b3a9f71af57243169904ba4a712c.
>
> Bad news, we can’t really work around it on the Guix side.

Actually Guix commit 4f89a8eec69491b925f084381ea4de37527c9310 provides a
workaround.

Ludo’.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.