GNU bug report logs - #75080
guix pull fails on arm64 due to cmake DEB self-tests

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 75080 in the body.
You can then email your comments to 75080 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Simon Josefsson <simon <at> josefsson.org>
To: bug-guix <at> gnu.org
Subject: guix pull fails on arm64 due to cmake DEB self-tests
Date: Wed, 25 Dec 2024 09:13:50 +0100

[Message part 1 (text/plain, inline)]

Hi!

I'm running the following commands on a arm64 GitLab shared runner node:

export LC_ALL=C.UTF-8
apt-get update
apt-get install -y guix podman ca-certificates
(guix-daemon --disable-chroot --build-users-group=_guixbuild &
guix pull --url=https://gitlab.com/debdistutils/guix/mirror.git \
 || (ls -la /var/log/guix/drvs/*/*.drv.gz; zcat /var/log/guix/drvs/qz/9fsb0935lpyssgg9428w9rvsypbfgv-cmake-bootstrap-3.24.2.drv.gz)

I get a build failure regarding "cmake-bootstrap" shown here:

https://gitlab.com/debdistutils/guix/container/-/jobs/8717616613

I cannot reproduce this on my own arm64 machine running Trisquel.

It seems only cmake's DEB related self-tests fail:

The following tests FAILED:
	561 - RunCMake.CPack_DEB.CUSTOM_NAMES (Failed)
	562 - RunCMake.CPack_DEB.DEBUGINFO (Failed)
	563 - RunCMake.CPack_DEB.DEFAULT_PERMISSIONS (Failed)
	564 - RunCMake.CPack_DEB.DEPENDENCIES (Failed)
	565 - RunCMake.CPack_DEB.EMPTY_DIR (Failed)
	566 - RunCMake.CPack_DEB.VERSION (Failed)
	567 - RunCMake.CPack_DEB.EXTRA (Failed)
	568 - RunCMake.CPack_DEB.GENERATE_SHLIBS (Failed)
	569 - RunCMake.CPack_DEB.GENERATE_SHLIBS_LDCONFIG (Failed)
	571 - RunCMake.CPack_DEB.MINIMAL (Failed)
	572 - RunCMake.CPack_DEB.PER_COMPONENT_FIELDS (Failed)
	573 - RunCMake.CPack_DEB.TIMESTAMPS (Failed)
	574 - RunCMake.CPack_DEB.MD5SUMS (Failed)
	575 - RunCMake.CPack_DEB.DEB_PACKAGE_VERSION_BACK_COMPATIBILITY (Failed)
	576 - RunCMake.CPack_DEB.DEB_DESCRIPTION (Failed)
	577 - RunCMake.CPack_DEB.PROJECT_META (Failed)
Errors while running CTest

That make me believe that the minimal nature of the container has some
impact.  Maybe some missing environment variable or tool or something?
That is usually present during normal builds but not this build.

The first failure starts here:

https://gitlab.com/debdistutils/guix/container/-/jobs/8717616613#L1746

with output cited below.  I don't spot any obvious problem.  I suspect
something odd is happening when running 'dpkg'.

Any ideas how to debug this further?

/Simon

549/622 Test #561: RunCMake.CPack_DEB.CUSTOM_NAMES .............................***Failed    0.88 sec
-- CUSTOM_NAMES-COMPONENT-type - PASSED
-- CUSTOM_NAMES-COMPONENT-type-Build - PASSED
CMake Error at RunCMake.cmake:216 (message):
  DEB/CUSTOM_NAMES-COMPONENT-type - FAILED:
  Result is [1], not [0].
  stderr does not match that expected.
  Command was:
   command> "/tmp/guix-build-cmake-bootstrap-3.24.2.drv-0/cmake-3.24.2/bin/cmake" "-DRunCMake_TEST=CUSTOM_NAMES-COMPONENT-type" "-DRunCMake_TEST_FILE_PREFIX=CUSTOM_NAMES" "-DRunCMake_SUBTEST_SUFFIX=" "-DGENERATOR_TYPE=DEB" "-DPACKAGING_TYPE=COMPONENT" "-Dsrc_dir=/tmp/guix-build-cmake-bootstrap-3.24.2.drv-0/cmake-3.24.2/Tests/RunCMake/CPack" "-Dbin_dir=/tmp/guix-build-cmake-bootstrap-3.24.2.drv-0/cmake-3.24.2/Tests/RunCMake/DEB.CUSTOM_NAMES/CPack/CUSTOM_NAMES-build" "-Dconfig_file=/tmp/guix-build-cmake-bootstrap-3.24.2.drv-0/cmake-3.24.2/Tests/RunCMake/CPack/conf/DEB.CUSTOM_NAMES_config.cmake" "-P" "/tmp/guix-build-cmake-bootstrap-3.24.2.drv-0/cmake-3.24.2/Tests/RunCMake/CPack/VerifyResult.cmake"
  Actual stdout:
   actual-out> 
  Expected stderr to match:
   expect-err> ^$
  Actual stderr:
   actual-err> CMake Error at /tmp/guix-build-cmake-bootstrap-3.24.2.drv-0/cmake-3.24.2/Tests/RunCMake/CPack/VerifyResult.cmake:84 (message):
   actual-err>   Unexpected file content for file 1!
   actual-err> 
   actual-err>   The content was:
   actual-err> 
   actual-err>    actual> 
   actual-err> 
   actual-err>   which does not match:
   actual-err> 
   actual-err>    expect> /usr;/usr/foo;/usr/foo/CMakeLists.txt
   actual-err> 
   actual-err>   CPack output:
   actual-err> 
   actual-err>    cpack-out> CPack: Create package using DEB
   actual-err>    cpack-out> CPack: Install projects
   actual-err>    cpack-out> CPack: - Run preinstall target for: CUSTOM_NAMES-COMPONENT-type
   actual-err>    cpack-out> CPack: - Install project: CUSTOM_NAMES-COMPONENT-type [Debug]
   actual-err>    cpack-out> CPack: -   Install component: pkg_1
   actual-err>    cpack-out> CPack: -   Install component: pkg_2
   actual-err>    cpack-out> CPack: -   Install component: pkg_3
   actual-err>    cpack-out> CPack: Create package
   actual-err>    cpack-out> -- CPACK_DEBIAN_PACKAGE_DEPENDS not set, the package will have no dependencies.
   actual-err>    cpack-out> -- CPACK_DEBIAN_PACKAGE_DEPENDS not set, the package will have no dependencies.
   actual-err>    cpack-out> -- CPACK_DEBIAN_PACKAGE_DEPENDS not set, the package will have no dependencies.
   actual-err>    cpack-out> CPack: - package: /tmp/guix-build-cmake-bootstrap-3.24.2.drv-0/cmake-3.24.2/Tests/RunCMake/DEB.CUSTOM_NAMES/CPack/CUSTOM_NAMES-build/custom_names-pkg_1_0.1.1_arm64.deb generated.
   actual-err>    cpack-out> CPack: - package: /tmp/guix-build-cmake-bootstrap-3.24.2.drv-0/cmake-3.24.2/Tests/RunCMake/DEB.CUSTOM_NAMES/CPack/CUSTOM_NAMES-build/second_0.1.1_arm64.deb generated.
   actual-err>    cpack-out> CPack: - package: /tmp/guix-build-cmake-bootstrap-3.24.2.drv-0/cmake-3.24.2/Tests/RunCMake/DEB.CUSTOM_NAMES/CPack/CUSTOM_NAMES-build/pkg_3_abc.deb generated.
   actual-err>    cpack-out> 
   actual-err> 
   actual-err>   CPack error:
   actual-err> 
   actual-err>    cpack-err> 
   actual-err> 
   actual-err>   CPack result:
   actual-err> 
   actual-err>    cpack-res> 
   actual-err> 
   actual-err>   CPack config file:
   actual-err> 
   actual-err>    cpack-cfg> set(DPKG_EXECUTABLE "/usr/bin/dpkg")
   actual-err>    cpack-cfg> set(READELF_EXECUTABLE "/gnu/store/pqai4n95zn5wdw430gslb00sb967jdg8-binutils-2.41/bin/readelf")
Call Stack (most recent call first):
  RunCMake.cmake:230 (run_cmake)
  CPack/CPackTestHelpers.cmake:119 (run_cmake_command)
  CPack/CPackTestHelpers.cmake:137 (run_cpack_test_common_)
  CPack/RunCMakeTest.cmake:11 (run_cpack_test)
        Start 562: RunCMake.CPack_DEB.DEBUGINFO

[signature.asc (application/pgp-signature, inline)]

Message #8 received at 75080 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Simon Josefsson <simon <at> josefsson.org>
Cc: 75080 <at> debbugs.gnu.org
Subject: Re: bug#75080: guix pull fails on arm64 due to cmake DEB self-tests
Date: Sat, 11 Jan 2025 19:25:50 +0100

Hi,

Simon Josefsson <simon <at> josefsson.org> skribis:

> I get a build failure regarding "cmake-bootstrap" shown here:
>
> https://gitlab.com/debdistutils/guix/container/-/jobs/8717616613
>
> I cannot reproduce this on my own arm64 machine running Trisquel.
>
> It seems only cmake's DEB related self-tests fail:

Looks like the problem is gone with a current-ish commit since
substitutes are available from both build farms:

--8<---------------cut here---------------start------------->8---
$ guix weather cmake-minimal -s aarch64-linux
computing 1 package derivations for aarch64-linux...
looking for 1 store items on https://ci.guix.gnu.org...
https://ci.guix.gnu.org ☀
  100.0% substitutes available (1 out of 1)
  at least 10.6 MiB of nars (compressed)
  27.7 MiB on disk (uncompressed)
  2.815 seconds per request (2.8 seconds in total)
  0.4 requests per second

  at least 1,000 queued builds
      aarch64-linux: 993 (99.3%)
      x86_64-linux: 4 (.4%)
      powerpc64le-linux: 2 (.2%)
      i686-linux: 1 (.1%)
  build rate: 545.79 builds per hour
      x86_64-linux: 545.79 builds per hour
looking for 1 store items on https://bordeaux.guix.gnu.org...
https://bordeaux.guix.gnu.org ☀
  100.0% substitutes available (1 out of 1)
  4.5 MiB of nars (compressed)
  27.7 MiB on disk (uncompressed)
  0.150 seconds per request (0.2 seconds in total)
  6.7 requests per second
  (continuous integration information unavailable)
looking for 1 store items on https://guix.bordeaux.inria.fr...
https://guix.bordeaux.inria.fr ⛈
  0.0% substitutes available (0 out of 1)
  unknown substitute sizes
  0.0 MiB on disk (uncompressed)
  0.169 seconds per request (0.2 seconds in total)
  5.9 requests per second

  0.0% (0 out of 1) of the missing items are queued
  0 queued builds
  build rate: 17.09 builds per hour
      x86_64-linux: 17.09 builds per hour
$ guix describe
Generation 331  Jan 05 2025 22:28:17    (current)
  shepherd 6d52686
    repository URL: https://git.savannah.gnu.org/git/shepherd.git
    branch: main
    commit: 6d526862375a426c13a52c7343c0ee9215367a00
  guile f6359a4
    repository URL: https://git.savannah.gnu.org/git/guile.git
    branch: main
    commit: f6359a4715d023761454f1bf945633ce4cca98fc
  guix 613c8b8
    repository URL: https://git.savannah.gnu.org/git/guix.git
    commit: 613c8b81702f08ee36f20d15ee8f8c42a37acfef
--8<---------------cut here---------------end--------------->8---

Could you confirm?

Ludo’.

Message #11 received at 75080 <at> debbugs.gnu.org (full text, mbox):

From: Simon Josefsson <simon <at> josefsson.org>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 75080 <at> debbugs.gnu.org
Subject: Re: bug#75080: guix pull fails on arm64 due to cmake DEB self-tests
Date: Wed, 05 Feb 2025 00:18:41 +0100

[Message part 1 (text/plain, inline)]

Ludovic Courtès <ludo <at> gnu.org> writes:

> Hi,
>
> Simon Josefsson <simon <at> josefsson.org> skribis:
>
>> I get a build failure regarding "cmake-bootstrap" shown here:
>>
>> https://gitlab.com/debdistutils/guix/container/-/jobs/8717616613
>>
>> I cannot reproduce this on my own arm64 machine running Trisquel.
>>
>> It seems only cmake's DEB related self-tests fail:
>
> Looks like the problem is gone with a current-ish commit since
> substitutes are available from both build farms:
>
> $ guix weather cmake-minimal -s aarch64-linux
...
> Could you confirm?

Maybe the substitutes are now gone, because I just tried again and the
error looks the same:

https://gitlab.com/debdistutils/guix/container/-/jobs/9042454855

If it got the binary substitutes, I suppose things would be fine.

Looking more carefully at the errors, I notice this part:

   actual-err>    cpack-cfg> set(DPKG_EXECUTABLE "/usr/bin/dpkg")
   actual-err>    cpack-cfg> set(READELF_EXECUTABLE "/gnu/store/pqai4n95zn5wdw430gslb00sb967jdg8-binutils-2.41/bin/readelf")

So for some reason building cmake-bootstrap in guix in Debian picks up
Debian's /usr/bin/dpkg, doesn't it?!

I suppose that Guix's build servers aren't running Debian, so it won't
find any /usr/bin/dpkg.

Maybe this has something to do with why it is only all the DEB-related
cmake self-tests that fails here.

(However it doesn't explain why I couldn't reproduce the same problem on
my own arm64 machine that is running Trisquel.  Maybe substitutes just
happened to be available when I tried it?)

The GitLab run above is in a minimal debian 12 container, and the
commands are essentially these:

  - apt-get update
  - apt-get install -y buildah ca-certificates eatmydata
  - time eatmydata buildah build -t ...

https://gitlab.com/debdistutils/guix/container/-/blob/101642fd8dd576ae9346d90bc1eb4d5ba4fcd80e/.gitlab-ci.yml#L36

With the Containerfile running essentially this:

set -x && \
cat /etc/os-release && \
locale -a && \
apt-get update && \
apt-get install -y guix netbase && \
(env LC_ALL=C.UTF-8 guix-daemon --disable-chroot --build-users-group=_guixbuild --discover=no &) && \
set && \
(guix pull --url=https://gitlab.com/debdistutils/guix/mirror.git \
 || { cp -v /var/log/guix/drvs/*/*.drv.gz .; gzip -d *.gz; tail -v -n1000 *.drv; false; })

https://gitlab.com/debdistutils/guix/container/-/blob/main/debian-with-guix/Containerfile

/Simon

[signature.asc (application/pgp-signature, inline)]

Message #14 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Leo Famulari <leo <at> famulari.name>
To: Simon Josefsson via Bug reports for GNU Guix <bug-guix <at> gnu.org>
Cc: Ludovic Courtès <ludo <at> gnu.org>, 75080 <at> debbugs.gnu.org
Subject: Re: bug#75080: guix pull fails on arm64 due to cmake DEB self-tests
Date: Tue, 4 Feb 2025 20:52:30 -0500

On Wed, Feb 05, 2025 at 12:18:41AM +0100, Simon Josefsson via Bug reports for GNU Guix wrote:
> Looking more carefully at the errors, I notice this part:
> 
>    actual-err>    cpack-cfg> set(DPKG_EXECUTABLE "/usr/bin/dpkg")
>    actual-err>    cpack-cfg> set(READELF_EXECUTABLE "/gnu/store/pqai4n95zn5wdw430gslb00sb967jdg8-binutils-2.41/bin/readelf")
> 
> So for some reason building cmake-bootstrap in guix in Debian picks up
> Debian's /usr/bin/dpkg, doesn't it?!
> 
> I suppose that Guix's build servers aren't running Debian, so it won't
> find any /usr/bin/dpkg.

It would be really surprising if '/usr/bin/dpkg' could be found or
accessed from within the package build environment, unless the build
isolation has been disabled.

Are you able to confirm if that is what's happening?

Message #20 received at 75080 <at> debbugs.gnu.org (full text, mbox):

From: Simon Josefsson <simon <at> josefsson.org>
To: Leo Famulari <leo <at> famulari.name>
Cc: ludo <at> gnu.org, 75080 <at> debbugs.gnu.org
Subject: Re: bug#75080: guix pull fails on arm64 due to cmake DEB self-tests
Date: Wed, 26 Feb 2025 09:41:01 +0100

[Message part 1 (text/plain, inline)]

Leo Famulari <leo <at> famulari.name> writes:

> On Wed, Feb 05, 2025 at 12:18:41AM +0100, Simon Josefsson via Bug
> reports for GNU Guix wrote:
>> Looking more carefully at the errors, I notice this part:
>> 
>>    actual-err>    cpack-cfg> set(DPKG_EXECUTABLE "/usr/bin/dpkg")
>>    actual-err> cpack-cfg> set(READELF_EXECUTABLE
>> "/gnu/store/pqai4n95zn5wdw430gslb00sb967jdg8-binutils-2.41/bin/readelf")
>> 
>> So for some reason building cmake-bootstrap in guix in Debian picks up
>> Debian's /usr/bin/dpkg, doesn't it?!
>> 
>> I suppose that Guix's build servers aren't running Debian, so it won't
>> find any /usr/bin/dpkg.
>
> It would be really surprising if '/usr/bin/dpkg' could be found or
> accessed from within the package build environment, unless the build
> isolation has been disabled.
>
> Are you able to confirm if that is what's happening?

Hi.  What do you mean by disabled isolation?  In the initial e-mail I
quoted the set of commands used which involved:

(guix-daemon --disable-chroot --build-users-group=_guixbuild &

Is that what you mean?

Is building things without build isolation not supposed to work?

Maybe there is now sufficient details to reproduce this: rebuild
cmake-bootstrap in a Debian container with Guix installed without build
isolation.  I'm not sure how to trigger a cmake-bootstrap rebuild
though, guix just seems to use the cached result for me.

Btw, substitutes were available at some point, so there is now arm64
Guix container images suitable for GitLab pipeline!

https://gitlab.com/debdistutils/guix/container/-/pipelines/1682821432
https://gitlab.com/debdistutils/guix/container/-/jobs/9211075760

/Simon

[signature.asc (application/pgp-signature, inline)]

Message #23 received at 75080 <at> debbugs.gnu.org (full text, mbox):

From: "Leo Famulari" <leo <at> famulari.name>
To: "Simon Josefsson" <simon <at> josefsson.org>
Cc: Ludovic Courtès <ludo <at> gnu.org>, 75080 <at> debbugs.gnu.org
Subject: Re: bug#75080: guix pull fails on arm64 due to cmake DEB self-tests
Date: Wed, 26 Feb 2025 13:14:02 -0500

On Wed, Feb 26, 2025, at 03:41, Simon Josefsson wrote:
> Hi.  What do you mean by disabled isolation?  In the initial e-mail I
> quoted the set of commands used which involved:
>
> (guix-daemon --disable-chroot --build-users-group=_guixbuild &
>
> Is that what you mean?

Yes, I'm sorry I missed that and made you spell it out!

> Is building things without build isolation not supposed to work?

Not really. We do accept help patching upstream build scripts to fix issues like this one, but I don't think anyone is regularly testing Guix without the build chroot, and the warranty is definitely void when Guix is used this way. I think that countless build will break without the chroot, as build scripts will look for dependencies in the "wrong" (i.e. non-Guixy) places.

Message #28 received at 75080-done <at> debbugs.gnu.org (full text, mbox):

From: Simon Josefsson <simon <at> josefsson.org>
To: "Leo Famulari" <leo <at> famulari.name>
Cc: 75080-done <at> debbugs.gnu.org,
 Ludovic Courtès <ludo <at> gnu.org>
Subject: Re: bug#75080: guix pull fails on arm64 due to cmake DEB self-tests
Date: Tue, 04 Mar 2025 13:11:51 +0100

[Message part 1 (text/plain, inline)]

"Leo Famulari" <leo <at> famulari.name> writes:

> On Wed, Feb 26, 2025, at 03:41, Simon Josefsson wrote:
>> Hi.  What do you mean by disabled isolation?  In the initial e-mail I
>> quoted the set of commands used which involved:
>>
>> (guix-daemon --disable-chroot --build-users-group=_guixbuild &
>>
>> Is that what you mean?
>
> Yes, I'm sorry I missed that and made you spell it out!
>
>> Is building things without build isolation not supposed to work?
>
> Not really. We do accept help patching upstream build scripts to fix
> issues like this one, but I don't think anyone is regularly testing
> Guix without the build chroot, and the warranty is definitely void
> when Guix is used this way. I think that countless build will break
> without the chroot, as build scripts will look for dependencies in the
> "wrong" (i.e. non-Guixy) places.

Thank you for explaining!  These internal things are still rather
sktechy to me.  It makes perfect sense now that I see it this way.

Fortunately it seems that building images without --disable-chroot now
appears to work for me (at least some of the time).  Here is a
successful arm64 Guix GitLab pipeline job where 'guix install' works:

https://gitlab.com/debdistutils/guix/container-selftests/-/jobs/9305446937

That container image was built from a pure Guix container image, which
in turn was built from a Debian+Guix container image.

So I think my original request was simply a misunderstanding, closing...

/Simon

[signature.asc (application/pgp-signature, inline)]

Message #31 received at 75080-done <at> debbugs.gnu.org (full text, mbox):

From: Leo Famulari <leo <at> famulari.name>
To: Simon Josefsson <simon <at> josefsson.org>
Cc: 75080-done <at> debbugs.gnu.org,
 Ludovic Courtès <ludo <at> gnu.org>
Subject: Re: bug#75080: guix pull fails on arm64 due to cmake DEB self-tests
Date: Sun, 9 Mar 2025 17:44:54 -0400

On Tue, Mar 04, 2025 at 01:11:51PM +0100, Simon Josefsson wrote:
> Fortunately it seems that building images without --disable-chroot now
> appears to work for me (at least some of the time).  Here is a
> successful arm64 Guix GitLab pipeline job where 'guix install' works:

Great! It's definitely expected that everything works with
'--disable-chroot'. And if it doesn't, then it's a bug.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.