GNU bug report logs - #53194
System test partition.img differs in size across hosts(?)

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 53194 in the body.
You can then email your comments to 53194 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Tobias Geerinckx-Rice <me <at> tobias.gr>
To: Bug reports for GNU Guix <bug-guix <at> gnu.org>
Subject: System test partition.img differs in size across hosts(?)
Date: Tue, 11 Jan 2022 20:31:27 +0100

[Message part 1 (text/plain, inline)]

Guix,

This is weird.  On berlin:

--8<---------------cut here---------------start------------->8---
$ guix build 
/gnu/store/91wjmydy556ibl38xydpb8yisp3gvx8w-partition.img.drv
[…]
Creating filesystem with 351 1k blocks and 40 inodes
[…]
/gnu/store/q18ca3ilma0h5hpn4s39xhzn0kc7jm5x-partition.img
--8<---------------cut here---------------end--------------->8---

On my laptop:

--8<---------------cut here---------------start------------->8---
$ guix build 
/gnu/store/91wjmydy556ibl38xydpb8yisp3gvx8w-partition.img.drv
[…]
Creating filesystem with 242 1k blocks and 32 inodes
[…]
Copying files into the device: ext2fs_symlink: Could not allocate 
inode in ext2 filesystem while creating symlink "system"
__populate_fs: Could not allocate inode in ext2 filesystem while 
writing symlink"system"
mke2fs: Could not allocate inode in ext2 filesystem while 
populating file system
--8<---------------cut here---------------end--------------->8---

This happens with both a tmpfs and a bcachefs /tmp.

The same make check-system TESTS="openvswitch" fails for Marius as 
well, although I don't know the exact output.  They tested btrfs 
and tmpfs, and suggested a kernel regression.

I don't understand how that would cause this, but I'm forced to 
agree: something spooky is going on in the chroot and the kernel 
is a big variable.

The attached patch was written before I was aware of above 
weirdness and only works around the issue.

Kind regards,

T G-R

[0001-build-image-Account-for-fixed-size-file-system-struc.patch (text/x-patch, inline)]

From 18f288d4b69faa73ffb75488dbc924640441d7ee Mon Sep 17 00:00:00 2001
From: Tobias Geerinckx-Rice <me <at> tobias.gr>
Date: Tue, 11 Jan 2022 19:56:53 +0100
Subject: [PATCH] build: image: Account for fixed-size file system structures.

* gnu/build/image.scm (estimate-partition-size): Enforce a 1-MiB minimum.
---
 gnu/build/image.scm | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/gnu/build/image.scm b/gnu/build/image.scm
index bdd5ec25a9..81caa424f8 100644
--- a/gnu/build/image.scm
+++ b/gnu/build/image.scm
@@ -3,7 +3,7 @@
 ;;; Copyright © 2016 Christine Lemmer-Webber <cwebber <at> dustycloud.org>
 ;;; Copyright © 2016, 2017 Leo Famulari <leo <at> famulari.name>
 ;;; Copyright © 2017 Marius Bakke <mbakke <at> fastmail.com>
-;;; Copyright © 2020 Tobias Geerinckx-Rice <me <at> tobias.gr>
+;;; Copyright © 2020, 2022 Tobias Geerinckx-Rice <me <at> tobias.gr>
 ;;; Copyright © 2020 Mathieu Othacehe <m.othacehe <at> gmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
@@ -62,8 +62,10 @@ (define (size-in-kib size)
 
 (define (estimate-partition-size root)
   "Given the ROOT directory, evaluate and return its size.  As this doesn't
-take the partition metadata size into account, take a 25% margin."
-  (* 1.25 (file-size root)))
+take the partition metadata size into account, take a 25% margin.  As this in
+turn doesn't take any constant overhead into account, force a 1-MiB minimum."
+  (max (ash 1 20)
+       (* 1.25 (file-size root))))
 
 (define* (make-ext-image partition target root
                          #:key
-- 
2.34.0

[signature.asc (application/pgp-signature, inline)]

Message #8 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Tobias Geerinckx-Rice <me <at> tobias.gr>
Cc: 53194 <at> debbugs.gnu.org, bug-guix <at> gnu.org
Subject: Re: bug#53194: System test partition.img differs in size across
 hosts(?)
Date: Tue, 11 Jan 2022 20:44:11 +0100

[Message part 1 (text/plain, inline)]

The most likely culprit is a change or difference in how the 
kernel answers FILE-SIZE's ‘how much disc space does FILE 
consume?’ — rounding it to N blocks or bytes, including or 
excluding directory sizes, differing reported directory sizes, 
etc.

I'll do more testing.

Kind regards,

T G-R

[signature.asc (application/pgp-signature, inline)]

Message #14 received at 53194 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: Tobias Geerinckx-Rice <me <at> tobias.gr>
Cc: 53194 <at> debbugs.gnu.org
Subject: Re: bug#53194: System test partition.img differs in size across
 hosts(?)
Date: Tue, 25 Jan 2022 12:54:50 -0500

Hi Tobias,

[...]

> diff --git a/gnu/build/image.scm b/gnu/build/image.scm
> index bdd5ec25a9..81caa424f8 100644
> --- a/gnu/build/image.scm
> +++ b/gnu/build/image.scm
> @@ -3,7 +3,7 @@
>  ;;; Copyright © 2016 Christine Lemmer-Webber <cwebber <at> dustycloud.org>
>  ;;; Copyright © 2016, 2017 Leo Famulari <leo <at> famulari.name>
>  ;;; Copyright © 2017 Marius Bakke <mbakke <at> fastmail.com>
> -;;; Copyright © 2020 Tobias Geerinckx-Rice <me <at> tobias.gr>
> +;;; Copyright © 2020, 2022 Tobias Geerinckx-Rice <me <at> tobias.gr>
>  ;;; Copyright © 2020 Mathieu Othacehe <m.othacehe <at> gmail.com>
>  ;;;
>  ;;; This file is part of GNU Guix.
> @@ -62,8 +62,10 @@ (define (size-in-kib size)
>  
>  (define (estimate-partition-size root)
>    "Given the ROOT directory, evaluate and return its size.  As this doesn't
> -take the partition metadata size into account, take a 25% margin."
> -  (* 1.25 (file-size root)))
> +take the partition metadata size into account, take a 25% margin.  As this in
> +turn doesn't take any constant overhead into account, force a 1-MiB minimum."
> +  (max (ash 1 20)
> +       (* 1.25 (file-size root))))
>  
>  (define* (make-ext-image partition target root
>                           #:key

Looks reasonable to me (although it is interesting that the behavior is
not the same across machines...).

While at it, you may want to fix this docstring:

--8<---------------cut here---------------start------------->8---
 (define (file-size file)
-  "Return the size of bytes of FILE, entering it if FILE is a directory."
+  "Return the size in bytes of FILE, entering it if FILE is a directory."
   (file-system-fold (const #t)
                     (lambda (file stat result)    ;leaf
                       (+ (stat:size stat) result))
--8<---------------cut here---------------end--------------->8---

in guix/build/store-copy.scm.

Thanks!

Maxim

Message #17 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Leo Famulari <leo <at> famulari.name>
To: Tobias Geerinckx-Rice via Bug reports for GNU Guix <bug-guix <at> gnu.org>
Cc: 53194 <at> debbugs.gnu.org
Subject: Re: bug#53194: System test partition.img differs in size across
 hosts(?)
Date: Thu, 3 Feb 2022 23:43:42 -0500

On Tue, Jan 11, 2022 at 08:31:27PM +0100, Tobias Geerinckx-Rice via Bug reports for GNU Guix wrote:
> On my laptop:
> 
> --8<---------------cut here---------------start------------->8---
> $ guix build /gnu/store/91wjmydy556ibl38xydpb8yisp3gvx8w-partition.img.drv
> […]
> Creating filesystem with 242 1k blocks and 32 inodes
> […]
> Copying files into the device: ext2fs_symlink: Could not allocate inode in
> ext2 filesystem while creating symlink "system"
> __populate_fs: Could not allocate inode in ext2 filesystem while writing
> symlink"system"
> mke2fs: Could not allocate inode in ext2 filesystem while populating file
> system
> --8<---------------cut here---------------end--------------->8---

Same here.

> This happens with both a tmpfs and a bcachefs /tmp.

And also on btrfs.

Message #23 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Leo Famulari <leo <at> famulari.name>
To: Tobias Geerinckx-Rice via Bug reports for GNU Guix <bug-guix <at> gnu.org>
Cc: 53194 <at> debbugs.gnu.org
Subject: Re: bug#53194: System test partition.img differs in size across
 hosts(?)
Date: Fri, 4 Feb 2022 00:17:08 -0500

On Tue, Jan 11, 2022 at 08:31:27PM +0100, Tobias Geerinckx-Rice via Bug reports for GNU Guix wrote:
> This is weird.  On berlin:

Berlin is using ext4, right?

> On my laptop:
> --8<---------------cut here---------------start------------->8---
[...]
> mke2fs: Could not allocate inode in ext2 filesystem while populating file
> system
> --8<---------------cut here---------------end--------------->8---
> 
> This happens with both a tmpfs and a bcachefs /tmp.

And it fails for me on btrfs, but not on ext4.

I tested with Guix kernels 5.16.5, 5.15.17, and 5.15.15, as well as
Debian's 5.10.0-11-amd64.

Message #29 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Leo Famulari <leo <at> famulari.name>
To: Tobias Geerinckx-Rice via Bug reports for GNU Guix <bug-guix <at> gnu.org>
Cc: 53194 <at> debbugs.gnu.org
Subject: Re: bug#53194: System test partition.img differs in size across
 hosts(?)
Date: Fri, 4 Feb 2022 00:23:30 -0500

On Tue, Jan 11, 2022 at 08:44:11PM +0100, Tobias Geerinckx-Rice via Bug reports for GNU Guix wrote:
> The most likely culprit is a change or difference in how the kernel answers
> FILE-SIZE's ‘how much disc space does FILE consume?’ — rounding it to N
> blocks or bytes, including or excluding directory sizes, differing reported
> directory sizes, etc.

I'm going to build the version of the kernel used on berlin and test
with that.

I do find myself wondering if something in Guix is measuring the wrong
thing: maybe we are measuring the size of files compressed in transit,
rather than their uncompressed size on disk. Or something like that.

Message #35 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Leo Famulari <leo <at> famulari.name>
To: Tobias Geerinckx-Rice via Bug reports for GNU Guix <bug-guix <at> gnu.org>
Cc: 53194 <at> debbugs.gnu.org
Subject: Re: bug#53194: System test partition.img differs in size across
 hosts(?)
Date: Fri, 4 Feb 2022 00:32:04 -0500

On Fri, Feb 04, 2022 at 12:23:30AM -0500, Leo Famulari wrote:
> I'm going to build the version of the kernel used on berlin and test
> with that.

Actually, I already had it built. This bug still manifests on that version
of the kernel. So...

> I do find myself wondering if something in Guix is measuring the wrong
> thing: maybe we are measuring the size of files compressed in transit,
> rather than their uncompressed size on disk. Or something like that.

I'm still leaning towards something besides a change in the kernel.

Message #41 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Leo Famulari <leo <at> famulari.name>
To: Tobias Geerinckx-Rice via Bug reports for GNU Guix <bug-guix <at> gnu.org>
Cc: 53194 <at> debbugs.gnu.org, othacehe <at> gnu.org
Subject: Re: bug#53194: System test partition.img differs in size across
 hosts(?)
Date: Fri, 4 Feb 2022 11:55:30 -0500

On Fri, Feb 04, 2022 at 12:32:04AM -0500, Leo Famulari wrote:
> I'm still leaning towards something besides a change in the kernel.

Using bisection of the Guix Git repo, it seems the problem was
introduced in commit 2d12ec724ea2, "scripts: system: Rationalize
persistency."

Message #49 received at 53194 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: Tobias Geerinckx-Rice <me <at> tobias.gr>
Cc: 53194 <at> debbugs.gnu.org
Subject: Re: bug#53194: System test partition.img differs in size across
 hosts(?)
Date: Sat, 05 Feb 2022 23:42:38 -0500

Hello,

Maxim Cournoyer <maxim.cournoyer <at> gmail.com> writes:

> Hi Tobias,
>
> [...]
>
>> diff --git a/gnu/build/image.scm b/gnu/build/image.scm
>> index bdd5ec25a9..81caa424f8 100644
>> --- a/gnu/build/image.scm
>> +++ b/gnu/build/image.scm
>> @@ -3,7 +3,7 @@
>>  ;;; Copyright © 2016 Christine Lemmer-Webber <cwebber <at> dustycloud.org>
>>  ;;; Copyright © 2016, 2017 Leo Famulari <leo <at> famulari.name>
>>  ;;; Copyright © 2017 Marius Bakke <mbakke <at> fastmail.com>
>> -;;; Copyright © 2020 Tobias Geerinckx-Rice <me <at> tobias.gr>
>> +;;; Copyright © 2020, 2022 Tobias Geerinckx-Rice <me <at> tobias.gr>
>>  ;;; Copyright © 2020 Mathieu Othacehe <m.othacehe <at> gmail.com>
>>  ;;;
>>  ;;; This file is part of GNU Guix.
>> @@ -62,8 +62,10 @@ (define (size-in-kib size)
>>  
>>  (define (estimate-partition-size root)
>>    "Given the ROOT directory, evaluate and return its size.  As this doesn't
>> -take the partition metadata size into account, take a 25% margin."
>> -  (* 1.25 (file-size root)))
>> +take the partition metadata size into account, take a 25% margin.  As this in
>> +turn doesn't take any constant overhead into account, force a 1-MiB minimum."
>> +  (max (ash 1 20)
>> +       (* 1.25 (file-size root))))
>>  
>>  (define* (make-ext-image partition target root
>>                           #:key
>
> Looks reasonable to me (although it is interesting that the behavior is
> not the same across machines...).
>
> While at it, you may want to fix this docstring:
>
>  (define (file-size file)
> -  "Return the size of bytes of FILE, entering it if FILE is a directory."
> +  "Return the size in bytes of FILE, entering it if FILE is a directory."
>    (file-system-fold (const #t)
>                      (lambda (file stat result)    ;leaf
>                        (+ (stat:size stat) result))
>
> in guix/build/store-copy.scm.

FYI, I pushed this workaround in
3c3c9d259f87fbc8c1d9551af32e79f9f168f596.

Thanks,

Maxim

Message #52 received at 53194 <at> debbugs.gnu.org (full text, mbox):

From: Leo Famulari <leo <at> famulari.name>
To: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Cc: 53194 <at> debbugs.gnu.org, Tobias Geerinckx-Rice <me <at> tobias.gr>
Subject: Re: bug#53194: System test partition.img differs in size across
 hosts(?)
Date: Sun, 6 Feb 2022 12:41:44 -0500

On Sat, Feb 05, 2022 at 11:42:38PM -0500, Maxim Cournoyer wrote:
> FYI, I pushed this workaround in
> 3c3c9d259f87fbc8c1d9551af32e79f9f168f596.

I don't see this commit in the repo.

Message #55 received at 53194 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: Leo Famulari <leo <at> famulari.name>
Cc: 53194 <at> debbugs.gnu.org, Tobias Geerinckx-Rice <me <at> tobias.gr>
Subject: Re: bug#53194: System test partition.img differs in size across
 hosts(?)
Date: Mon, 07 Feb 2022 16:29:46 -0500

Hi Leo!

Leo Famulari <leo <at> famulari.name> writes:

> On Sat, Feb 05, 2022 at 11:42:38PM -0500, Maxim Cournoyer wrote:
>> FYI, I pushed this workaround in
>> 3c3c9d259f87fbc8c1d9551af32e79f9f168f596.
>
> I don't see this commit in the repo.

Thank you for letting me know.  I hate when this happens; usually the
'make authenticate' fails in my Emacs env because it doesn't run in a
'guix shell -D guix' environment and 'make authenticate' fails due to a
missing dependency, failing the git push.

Anyway, now pushed the linux-libre series for real (which included
this), as e5c06dce93.

Thanks!

Maxim

Message #58 received at 53194 <at> debbugs.gnu.org (full text, mbox):

From: david larsson <david.larsson <at> selfhosted.xyz>
To: Tobias Geerinckx-Rice <me <at> tobias.gr>
Cc: 53194 <at> debbugs.gnu.org,
 bug-Guix <bug-guix-bounces+someone=selfhosted.xyz <at> gnu.org>
Subject: Re: bug#53194: System test partition.img differs in size across
 hosts(?)
Date: Thu, 17 Feb 2022 17:37:49 +0100

On 2022-01-11 20:31, Tobias Geerinckx-Rice via Bug reports for GNU Guix 
wrote:
> Guix,
> 
> This is weird.  On berlin:
> 
> --8<---------------cut here---------------start------------->8---
> $ guix build 
> /gnu/store/91wjmydy556ibl38xydpb8yisp3gvx8w-partition.img.drv
> […]
> Creating filesystem with 351 1k blocks and 40 inodes
> […]
> /gnu/store/q18ca3ilma0h5hpn4s39xhzn0kc7jm5x-partition.img
> --8<---------------cut here---------------end--------------->8---
> 
> On my laptop:
> 
> --8<---------------cut here---------------start------------->8---
> $ guix build 
> /gnu/store/91wjmydy556ibl38xydpb8yisp3gvx8w-partition.img.drv
> […]
> Creating filesystem with 242 1k blocks and 32 inodes
> […]
> Copying files into the device: ext2fs_symlink: Could not allocate
> inode in ext2 filesystem while creating symlink "system"
> __populate_fs: Could not allocate inode in ext2 filesystem while
> writing symlink"system"
> mke2fs: Could not allocate inode in ext2 filesystem while populating 
> file system
> --8<---------------cut here---------------end--------------->8---
> 
> This happens with both a tmpfs and a bcachefs /tmp.
> 
> The same make check-system TESTS="openvswitch" fails for Marius as
> well, although I don't know the exact output.  They tested btrfs and
> tmpfs, and suggested a kernel regression.
> 
> I don't understand how that would cause this, but I'm forced to agree:
> something spooky is going on in the chroot and the kernel is a big
> variable.
> 
> The attached patch was written before I was aware of above weirdness
> and only works around the issue.
> 
> Kind regards,
> 
> T G-R

I hope Im not totally off here, so Im just hoping this is worth 
mentioning:
Are the hosts using the same version of 
https://github.com/guix-mirror/guix/blob/master/gnu/system/image.scm#96 
? It might produce different sizes if the hosts are on different guix 
commits - or is this not a possibility at all if the derivations have 
the same hashes?

...because I just happened to notice that recently the guix system image 
command produces images that are exactly the additional size of the root 
offset and the esp-partition compared to what's specified with the 
--image-size option. I think this has changed from 1-2 years back (since 
Marius B. blog post reg. Ganeti). I think so because when I set up 
Ganeti according to that blog post I could (IIRC) create guix instances 
with the ganeti-instance-guix create script without problem - and it 
produces images with guix system image --image-size=X command - but when 
I did so again 1-2 weeks ago they failed with the error that Ganeti 
disks were too small. The size issue could be resolved by removing from 
the instance create-script the exact number of bytes to the 
--image-size=X option that corresponded to the root offset and the 
esp-partition sizes as defined in (gnu system image).

Maybe some commit has changed the size output of guix system image?


Best regards,
David

Message #63 received at 53194-done <at> debbugs.gnu.org (full text, mbox):

From: Mathieu Othacehe <othacehe <at> gnu.org>
To: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Cc: Tobias Geerinckx-Rice <me <at> tobias.gr>, 53194-done <at> debbugs.gnu.org
Subject: Re: bug#53194: System test partition.img differs in size across
 hosts(?)
Date: Mon, 31 Oct 2022 09:56:02 +0100

Hello,

> FYI, I pushed this workaround in
> 3c3c9d259f87fbc8c1d9551af32e79f9f168f596.

I'm not able to reproduce this issue with or without the workaround, by
running the openvswitch test on Berlin and on my laptop. I think we can
close it for now and re-open it if someone finds a more reliable
reproducer.

Thanks,

Mathieu

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.