GNU bug report logs - #49921
go-1.16 build failing on aarch64: "fatal error: runtime.newosproc"

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 49921 in the body.
You can then email your comments to 49921 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Sarah Morgensen <iskarian <at> mgsn.dev>
To: bug-guix <at> gnu.org
Subject: go-1.16 build failing on aarch64: "fatal error: runtime.newosproc"
Date: Fri, 06 Aug 2021 22:04:19 -0700

[Message part 1 (text/plain, inline)]

Hello Guix,

I just noticed go-1.16 is failing on aarch64 [0]. I am not having any
success tracking down the cause. It looks like the error is the same as
was happening for go-1.14 circa 11 Mar [1], which was fixed by 9 Apr
[2], but I cannot tell what resolved the issue. I've attached the
relevant part of the build log; the full log is available at [0].

Any ideas?

[0] https://ci.guix.gnu.org/build/949823/details
[1] https://ci.guix.gnu.org/build/71004/details
[2] https://ci.guix.gnu.org/build/19478/details

[go-1.16.7.log (text/plain, inline)]

starting phase `build'
runtime: failed to create new OS thread (have 2 already; errno=22)
fatal error: runtime.newosproc

runtime stack:
runtime.throw(0x5342c6)
	/gnu/store/ax3zhpkysy3nl5ipw3qb9yh2g04a0f1s-go-1.4-bootstrap-20171003/src/runtime/panic.go:491 +0xa4
runtime.newosproc(0x10722000, 0x10732000)
	/gnu/store/ax3zhpkysy3nl5ipw3qb9yh2g04a0f1s-go-1.4-bootstrap-20171003/src/runtime/os_linux.c:170 +0xec
newm(0xa4080, 0x0)
	/gnu/store/ax3zhpkysy3nl5ipw3qb9yh2g04a0f1s-go-1.4-bootstrap-20171003/src/runtime/proc.c:1157 +0xbc
runtime.newsysmon()
	/gnu/store/ax3zhpkysy3nl5ipw3qb9yh2g04a0f1s-go-1.4-bootstrap-20171003/src/runtime/proc.c:169 +0x2c
runtime.onM(0x54c228)
	/gnu/store/ax3zhpkysy3nl5ipw3qb9yh2g04a0f1s-go-1.4-bootstrap-20171003/src/runtime/asm_arm.s:256 +0x74
runtime.mstart()
	/gnu/store/ax3zhpkysy3nl5ipw3qb9yh2g04a0f1s-go-1.4-bootstrap-20171003/src/runtime/proc.c:818

goroutine 1 [running]:
runtime.switchtoM()
	/gnu/store/ax3zhpkysy3nl5ipw3qb9yh2g04a0f1s-go-1.4-bootstrap-20171003/src/runtime/asm_arm.s:193 +0x4 fp=0x1071c7c0 sp=0x1071c7bc
runtime.main()
	/gnu/store/ax3zhpkysy3nl5ipw3qb9yh2g04a0f1s-go-1.4-bootstrap-20171003/src/runtime/proc.go:32 +0x4c fp=0x1071c7e4 sp=0x1071c7c0
runtime.goexit()
	/gnu/store/ax3zhpkysy3nl5ipw3qb9yh2g04a0f1s-go-1.4-bootstrap-20171003/src/runtime/asm_arm.s:1322 +0x4 fp=0x1071c7e4 sp=0x1071c7e4
Building Go cmd/dist using /gnu/store/ax3zhpkysy3nl5ipw3qb9yh2g04a0f1s-go-1.4-bootstrap-20171003. ()
runtime: failed to create new OS thread (have 2 already; errno=22)
fatal error: runtime.newosproc

runtime stack:
runtime.throw(0x5342c6)
	/gnu/store/ax3zhpkysy3nl5ipw3qb9yh2g04a0f1s-go-1.4-bootstrap-20171003/src/runtime/panic.go:491 +0xa4
runtime.newosproc(0x10722000, 0x10732000)
	/gnu/store/ax3zhpkysy3nl5ipw3qb9yh2g04a0f1s-go-1.4-bootstrap-20171003/src/runtime/os_linux.c:170 +0xec
newm(0xa4080, 0x0)
	/gnu/store/ax3zhpkysy3nl5ipw3qb9yh2g04a0f1s-go-1.4-bootstrap-20171003/src/runtime/proc.c:1157 +0xbc
runtime.newsysmon()
	/gnu/store/ax3zhpkysy3nl5ipw3qb9yh2g04a0f1s-go-1.4-bootstrap-20171003/src/runtime/proc.c:169 +0x2c
runtime.onM(0x54c228)
	/gnu/store/ax3zhpkysy3nl5ipw3qb9yh2g04a0f1s-go-1.4-bootstrap-20171003/src/runtime/asm_arm.s:256 +0x74
runtime.mstart()
	/gnu/store/ax3zhpkysy3nl5ipw3qb9yh2g04a0f1s-go-1.4-bootstrap-20171003/src/runtime/proc.c:818

goroutine 1 [running]:
runtime.switchtoM()
	/gnu/store/ax3zhpkysy3nl5ipw3qb9yh2g04a0f1s-go-1.4-bootstrap-20171003/src/runtime/asm_arm.s:193 +0x4 fp=0x1071c7c0 sp=0x1071c7bc
runtime.main()
	/gnu/store/ax3zhpkysy3nl5ipw3qb9yh2g04a0f1s-go-1.4-bootstrap-20171003/src/runtime/proc.go:32 +0x4c fp=0x1071c7e4 sp=0x1071c7c0
runtime.goexit()
	/gnu/store/ax3zhpkysy3nl5ipw3qb9yh2g04a0f1s-go-1.4-bootstrap-20171003/src/runtime/asm_arm.s:1322 +0x4 fp=0x1071c7e4 sp=0x1071c7e4
command "sh" "make.bash" "--no-banner" failed with status 2
builder for `/gnu/store/dfqww60vr4gykvz3fz4mj9sgk0x4jypz-go-1.16.7.drv' failed with exit code 1
@ build-failed /gnu/store/dfqww60vr4gykvz3fz4mj9sgk0x4jypz-go-1.16.7.drv - 1 builder for `/gnu/store/dfqww60vr4gykvz3fz4mj9sgk0x4jypz-go-1.16.7.drv' failed with exit code 1

Message #8 received at 49921 <at> debbugs.gnu.org (full text, mbox):

From: Sarah Morgensen <iskarian <at> mgsn.dev>
To: 49921 <at> debbugs.gnu.org
Subject: Re: bug#49921: go-1.16 build failing on aarch64: "fatal error:
 runtime.newosproc"
Date: Thu, 02 Sep 2021 11:47:08 -0700

Sarah Morgensen <iskarian <at> mgsn.dev> writes:

> Hello Guix,
>
> I just noticed go-1.16 is failing on aarch64 [0]. I am not having any
> success tracking down the cause. It looks like the error is the same as
> was happening for go-1.14 circa 11 Mar [1], which was fixed by 9 Apr
> [2], but I cannot tell what resolved the issue. I've attached the
> relevant part of the build log; the full log is available at [0].
>
> Any ideas?
>
> [0] https://ci.guix.gnu.org/build/949823/details
> [1] https://ci.guix.gnu.org/build/71004/details
> [2] https://ci.guix.gnu.org/build/19478/details
>
> starting phase `build'
> runtime: failed to create new OS thread (have 2 already; errno=22)
> fatal error: runtime.newosproc

I think this might be related to [0], although if it's true that CI uses
native builders for aarch64 now, I have no idea.

I've been able to reproduce this with both go-1.14 and go-1.16 when
building with --system=aarch64-linux from an amd64 system.  I tried to
apply the patch in the thread I mentioned, but go-1.4 won't build at all
with QEMU.

[0] <https://github.com/golang/go/issues/20763> runtime: cannot run
cross compiled ARM binary on QEMU

--
Sarah

Message #17 received at 49921 <at> debbugs.gnu.org (full text, mbox):

From: Sarah Morgensen <iskarian <at> mgsn.dev>
To: 49921 <at> debbugs.gnu.org
Cc: Leo Famulari <leo <at> famulari.name>
Subject: Re: bug#49921: go-1.16 build failing on aarch64: "fatal error:
 runtime.newosproc"
Date: Thu, 09 Sep 2021 18:49:04 -0700

[Message part 1 (text/plain, inline)]

Sarah Morgensen <iskarian <at> mgsn.dev> writes:

> Sarah Morgensen <iskarian <at> mgsn.dev> writes:
>
>> Hello Guix,
>>
>> I just noticed go-1.16 is failing on aarch64 [0]. I am not having any
>> success tracking down the cause. It looks like the error is the same as
>> was happening for go-1.14 circa 11 Mar [1], which was fixed by 9 Apr
>> [2], but I cannot tell what resolved the issue. I've attached the
>> relevant part of the build log; the full log is available at [0].
>>
>> Any ideas?
>>
>> [0] https://ci.guix.gnu.org/build/949823/details
>> [1] https://ci.guix.gnu.org/build/71004/details
>> [2] https://ci.guix.gnu.org/build/19478/details
>>
>> starting phase `build'
>> runtime: failed to create new OS thread (have 2 already; errno=22)
>> fatal error: runtime.newosproc
>
> I think this might be related to [0], although if it's true that CI uses
> native builders for aarch64 now, I have no idea.
>
> I've been able to reproduce this with both go-1.14 and go-1.16 when
> building with --system=aarch64-linux from an amd64 system.  I tried to
> apply the patch in the thread I mentioned, but go-1.4 won't build at all
> with QEMU.
>
> [0] <https://github.com/golang/go/issues/20763> runtime: cannot run
> cross compiled ARM binary on QEMU
>
> --
> Sarah

I've written this up into a patch (attached below); I don't think
there's much of a way to test this other than just letting CI build it.
It's a backport, so it shouldn't hurt even if it doesn't fix it.

--
Sarah

[0001-gnu-go-1.4-Fix-running-with-qemu-aarch64.patch (text/x-patch, inline)]

From a5824c2495f5a547499ab200cd5b270b38f571d6 Mon Sep 17 00:00:00 2001
Message-Id: <a5824c2495f5a547499ab200cd5b270b38f571d6.1631237116.git.iskarian <at> mgsn.dev>
From: Sarah Morgensen <iskarian <at> mgsn.dev>
Date: Thu, 9 Sep 2021 18:23:10 -0700
Subject: [PATCH core-updates] gnu: go-1.4: Fix running with qemu-aarch64.

Backport the fix for running go with qemu-aarch64.

* gnu/packages/patches/go-1.4-fix-running-with-qemu.patch: New file.
* gnu/local.mk (dist_patch_DATA): Register it.
* gnu/packages/golang.scm (go-1.4)[origin]: Apply patch.
---
This might fix #49921, but I can't test it as there are a number of issues
preventing go-1.4 from compiling on qemu-aarch64.

It builds fine on x86_64, so it didn't break anything there.

--
Sarah

 gnu/local.mk                                  |  1 +
 gnu/packages/golang.scm                       |  4 +-
 .../go-1.4-fix-running-with-qemu.patch        | 38 +++++++++++++++++++
 3 files changed, 42 insertions(+), 1 deletion(-)
 create mode 100644 gnu/packages/patches/go-1.4-fix-running-with-qemu.patch

diff --git a/gnu/local.mk b/gnu/local.mk
index 20f0b8f081..39e17bb3bc 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -1144,6 +1144,7 @@ dist_patch_DATA =						\
   %D%/packages/patches/gobject-introspection-absolute-shlib-path.patch \
   %D%/packages/patches/gobject-introspection-cc.patch		\
   %D%/packages/patches/gobject-introspection-girepository.patch	\
+  %D%/packages/patches/go-1.4-fix-running-with-qemu.patch	\
   %D%/packages/patches/go-fix-script-tests.patch			\
   %D%/packages/patches/go-skip-gc-test.patch			\
   %D%/packages/patches/gpm-glibc-2.26.patch			\
diff --git a/gnu/packages/golang.scm b/gnu/packages/golang.scm
index d3ef39a2e6..33f3120a09 100644
--- a/gnu/packages/golang.scm
+++ b/gnu/packages/golang.scm
@@ -1035,7 +1035,9 @@ your Go binary to be later served from an http.FileSystem.")
                                   name version ".tar.gz"))
               (sha256
                (base32
-                "0liybk5z00hizsb5ypkbhqcawnwwa6mkwgvjjg4y3jm3ndg5pzzl"))))
+                "0liybk5z00hizsb5ypkbhqcawnwwa6mkwgvjjg4y3jm3ndg5pzzl"))
+              (patches
+               (search-patches "go-1.4-fix-running-with-qemu.patch"))))
     (build-system gnu-build-system)
     (outputs '("out"
                "doc"
diff --git a/gnu/packages/patches/go-1.4-fix-running-with-qemu.patch b/gnu/packages/patches/go-1.4-fix-running-with-qemu.patch
new file mode 100644
index 0000000000..52914c71a5
--- /dev/null
+++ b/gnu/packages/patches/go-1.4-fix-running-with-qemu.patch
@@ -0,0 +1,38 @@
+Backport from upstream: https://github/golang/go/commit/2673f9ed
+
+Original header:
+
+From 2673f9ed23348c634f6331ee589d489e4d9c7a9b Mon Sep 17 00:00:00 2001
+From: Austin Clements <austin <at> google.com>
+Date: Wed, 12 Jul 2017 10:12:50 -0600
+Subject: [PATCH] runtime: pass CLONE_SYSVSEM to clone
+
+SysV semaphore undo lists should be shared by threads, just like
+several other resources listed in cloneFlags. Currently we don't do
+this, but it probably doesn't affect anything because 1) probably
+nobody uses SysV semaphores from Go and 2) Go-created threads never
+exit until the process does. Beyond being the right thing to do,
+user-level QEMU requires this flag because it depends on glibc to
+create new threads and glibc uses this flag.
+
+Fixes #20763.
+
+---
+ src/runtime/os_linux.c | 1 +
+ 1 file changed, 1 insertion(+)
+
+diff --git a/src/runtime/os_linux.c b/src/runtime/os_linux.c
+index 0d8ffc9..573be7c 100644
+--- a/src/runtime/os_linux.c
++++ b/src/runtime/os_linux.c
+@@ -150,6 +150,7 @@ runtime·newosproc(M *mp, void *stk)
+ 		| CLONE_FS	/* share cwd, etc */
+ 		| CLONE_FILES	/* share fd table */
+ 		| CLONE_SIGHAND	/* share sig handler table */
++		| CLONE_SYSVSEM /* share SysV semaphore undo lists (see issue #20763) */
+ 		| CLONE_THREAD	/* revisit - okay for now */
+ 		;
+ 
+-- 
+2.31.1
+

base-commit: 22f7d4bce1e694b7ac38e62410d76a6d46d96c5d
-- 
2.33.0

Message #20 received at 49921 <at> debbugs.gnu.org (full text, mbox):

From: Thiago Jung Bauermann <bauermann <at> kolabnow.com>
To: Sarah Morgensen <iskarian <at> mgsn.dev>
Cc: 49921 <at> debbugs.gnu.org
Subject: Re: bug#49921: go-1.16 build failing on aarch64: "fatal error:
 runtime.newosproc"
Date: Fri, 10 Sep 2021 19:06:10 -0300

Hello Sarah,

Em quinta-feira, 2 de setembro de 2021, às 15:47:08 -03, Sarah Morgensen 
escreveu:
> I think this might be related to [0], although if it's true that CI uses
> native builders for aarch64 now, I have no idea.

The CI has two native aarch64 builders (which also do armhf, despite bugs 
43513 and 43591): dover and overdrive1.

It also uses half of the x86_64 builders (hydra-guix-*) for emulated builds 
of aarch64 and armhf, as can be seen here:

https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/berlin-nodes.scm#n143

-- 
Thanks,
Thiago

Message #23 received at 49921 <at> debbugs.gnu.org (full text, mbox):

From: iskarian <at> mgsn.dev
To: "Thiago Jung Bauermann" <bauermann <at> kolabnow.com>
Cc: 49921 <at> debbugs.gnu.org
Subject: Re: bug#49921: go-1.16 build failing on aarch64: "fatal error:
 runtime.newosproc"
Date: Fri, 10 Sep 2021 22:14:06 +0000

Hi Thiago,

(Re-sent due to missing Cc.)

September 10, 2021 3:06 PM, "Thiago Jung Bauermann" <bauermann <at> kolabnow.com> wrote:

> Hello Sarah,
> 
> Em quinta-feira, 2 de setembro de 2021, às 15:47:08 -03, Sarah Morgensen 
> escreveu:
> 
>> I think this might be related to [0], although if it's true that CI uses
>> native builders for aarch64 now, I have no idea.
> 
> The CI has two native aarch64 builders (which also do armhf, despite bugs
> 43513 and 43591): dover and overdrive1.
> 
> It also uses half of the x86_64 builders (hydra-guix-*) for emulated builds
> of aarch64 and armhf, as can be seen here:
> 
> https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/berlin-nodes.scm#n143

Thanks for the explanation.  Is there a way to tell CI (or Guix itself) that certain packages
shouldn't be built under emulation?

--
Sarah

Message #26 received at 49921 <at> debbugs.gnu.org (full text, mbox):

From: Thiago Jung Bauermann <bauermann <at> kolabnow.com>
To: iskarian <at> mgsn.dev
Cc: 49921 <at> debbugs.gnu.org
Subject: Re: bug#49921: go-1.16 build failing on aarch64: "fatal error:
 runtime.newosproc"
Date: Fri, 10 Sep 2021 20:06:34 -0300

Hello Sarah,

Em sexta-feira, 10 de setembro de 2021, às 19:14:06 -03, iskarian <at> mgsn.dev 
escreveu:
> Hi Thiago,
> 
> (Re-sent due to missing Cc.)
> 
> September 10, 2021 3:06 PM, "Thiago Jung Bauermann" 
<bauermann <at> kolabnow.com> wrote:
> > Hello Sarah,
> > 
> > Em quinta-feira, 2 de setembro de 2021, às 15:47:08 -03, Sarah
> > Morgensen
> > 
> > escreveu:
> >> I think this might be related to [0], although if it's true that CI
> >> uses
> >> native builders for aarch64 now, I have no idea.
> > 
> > The CI has two native aarch64 builders (which also do armhf, despite
> > bugs 43513 and 43591): dover and overdrive1.
> > 
> > It also uses half of the x86_64 builders (hydra-guix-*) for emulated
> > builds of aarch64 and armhf, as can be seen here:
> > 
> > https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/berli
> > n-nodes.scm#n143
> Thanks for the explanation.  Is there a way to tell CI (or Guix itself)
> that certain packages shouldn't be built under emulation?

I don’t know. That would certainly be useful, though.

To be honest, my experience in the past couple of months with emulated 
builds for powerpc64le-linux wasn’t good, so I asked for it to be turned 
off.

-- 
Thanks,
Thiago

Message #31 received at 49921-done <at> debbugs.gnu.org (full text, mbox):

From: Leo Famulari <leo <at> famulari.name>
To: Sarah Morgensen <iskarian <at> mgsn.dev>
Cc: 49921-done <at> debbugs.gnu.org
Subject: Re: bug#49921: go-1.16 build failing on aarch64: "fatal error:
 runtime.newosproc"
Date: Tue, 14 Dec 2021 14:29:51 -0500

On Fri, Aug 06, 2021 at 10:04:19PM -0700, Sarah Morgensen wrote:
> I just noticed go-1.16 is failing on aarch64 [0]. I am not having any
> success tracking down the cause. It looks like the error is the same as
> was happening for go-1.14 circa 11 Mar [1], which was fixed by 9 Apr
> [2], but I cannot tell what resolved the issue. I've attached the
> relevant part of the build log; the full log is available at [0].

I was able to build Go 1.16 on aarch64-linux, by building this
derivation "by hand" on the Berlin server. It was offloaded to real
aarch64 hardware.

/gnu/store/jgbp8bpi86is2y620wqais904lvjmvj8-go-1.16.11.drv
https://ci.guix.gnu.org/build/1952349/details

If I understand correctly, we have totally disabled the aarch64 emulated
builds. They were problematic and I think that we should not re-enable
them. So, I'm closing this bug. Please let us know if you think it
should be re-opened.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.