GNU bug report logs - #36754
SSH connections to hydra-slave{1,2,3} fail during builds

Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.

Package: guix; Severity: important; Reported by: Mark H Weaver <mhw@HIDDEN>; dated Sun, 21 Jul 2019 23:59:01 UTC; Maintainer for guix is bug-guix@HIDDEN.

Message received at 36754 <at> debbugs.gnu.org:


Received: (at 36754) by debbugs.gnu.org; 1 Aug 2019 16:39:59 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Aug 01 12:39:59 2019
Received: from localhost ([127.0.0.1]:55058 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1htE7T-00027q-0e
	for submit <at> debbugs.gnu.org; Thu, 01 Aug 2019 12:39:59 -0400
Received: from world.peace.net ([64.112.178.59]:38788)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mhw@HIDDEN>) id 1htE7R-00027Z-3j
 for 36754 <at> debbugs.gnu.org; Thu, 01 Aug 2019 12:39:57 -0400
Received: from mhw by world.peace.net with esmtpsa
 (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89)
 (envelope-from <mhw@HIDDEN>)
 id 1htE7J-0002Mm-Ol; Thu, 01 Aug 2019 12:39:49 -0400
From: Mark H Weaver <mhw@HIDDEN>
To: Marius Bakke <mbakke@HIDDEN>
Subject: Re: bug#36754: SSH connections to hydra-slave{1, 2,
 3} fail during builds
In-Reply-To: <87k1bx2d1f.fsf@HIDDEN>
References: <87o91n3ps0.fsf@HIDDEN> <87o91n3ps0.fsf@HIDDEN>
 <87zhl656xk.fsf@HIDDEN> <87pnm04s21.fsf@HIDDEN>
 <87ef2gr6z3.fsf@HIDDEN> <878ssor680.fsf@HIDDEN>
 <87lfwo1lz0.fsf@HIDDEN> <87ef2fpu2a.fsf@HIDDEN>
 <87lfwnxyzg.fsf@HIDDEN> <87k1bx2d1f.fsf@HIDDEN>
Date: Thu, 01 Aug 2019 12:37:31 -0400
Message-ID: <87pnlokfix.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 36754
Cc: Ricardo Wurmus <rekado@HIDDEN>, 36754 <at> debbugs.gnu.org,
 Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Hi Marius,

Marius Bakke <mbakke@HIDDEN> wrote:

> The truncated log files seems to happen for other builds as well, even
> within the Berlin data center.
>
> https://ci.guix.gnu.org/log/n3ra1b8ic6qhfinnhb80mrn7snsqws9d-geocode-glib-3.26.0
> https://ci.guix.gnu.org/log/zqhqlib00i8f7f10g4c2dfzprw16h4xv-scintilla-4.2.0
> https://ci.guix.gnu.org/log/718jmbq94mvdgnmjyqgxgy7zaj8xzxk3-htslib-1.9
>
> All of these builds are for i686-linux.

Thanks, that's very useful information.

> Mark: are the armhf nodes still operational?

I assume so.  They all respond to pings anyway, and I haven't touched
them since before they were disconnected from Berlin.  (I would need to
boot up my other, more secure computer to try SSHing into them).

> I would like to re-enable them again, since we desperately need the
> computing power with four huge branches going concurrently at the
> moment.

I have no objection, but since Ludovic made the decision to disconnect
them, it would be good to hear from him first.

       Thanks,
         Mark




Information forwarded to bug-guix@HIDDEN:
bug#36754; Package guix. Full text available.

Message received at 36754 <at> debbugs.gnu.org:


Received: (at 36754) by debbugs.gnu.org; 1 Aug 2019 15:56:33 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Aug 01 11:56:33 2019
Received: from localhost ([127.0.0.1]:54918 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1htDRR-00078q-2I
	for submit <at> debbugs.gnu.org; Thu, 01 Aug 2019 11:56:33 -0400
Received: from sender4-of-o53.zoho.com ([136.143.188.53]:21317)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <rekado@HIDDEN>) id 1htDRO-00078f-Ge
 for 36754 <at> debbugs.gnu.org; Thu, 01 Aug 2019 11:56:31 -0400
ARC-Seal: i=1; a=rsa-sha256; t=1564674978; cv=none; d=zoho.com; s=zohoarc; 
 b=eJgExjFecR9RldkOih3G0/gcsdUv9PiE7570fcuI6tgriSoYeuACcZpoqlPBobMT6aiJ+GBkPhG/uTJRtrEaIYKtfz/O/z6i3cG6SZyeiiAFZH7GSW5dfIfAzgsjznlkHSiWAVpupNAzwrbL0S5mG9NILm+q9DF5L7PhdFPM3Jc=
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com;
 s=zohoarc; t=1564674978;
 h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:To:ARC-Authentication-Results;
 bh=LSw/N+WGCJZnGBvEpR+lK3cMxiJYDJPmX8OBGzkyAoE=; 
 b=VUMJx2VUd7X9kmLPbJ2FayAiRylIVxLtZ3JR7+iLpWzb3pNf2iZnVzjFFaq6p6jpu0XxNvj31VRUESvYoRiy3rnBNtm5Cr2b+hQkUIFw+L2/1KFaYoraHeXqm8Z5ACk3QRZyVN8iP3EHT9TzuOHiyipKZ41O1a4yUsfe3Wz06jE=
ARC-Authentication-Results: i=1; mx.zoho.com; dkim=pass  header.i=elephly.net;
 spf=pass  smtp.mailfrom=rekado@HIDDEN;
 dmarc=pass header.from=<rekado@HIDDEN> header.from=<rekado@HIDDEN>
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1564674978; 
 s=zoho; d=elephly.net; i=rekado@HIDDEN;
 h=References:From:To:Cc:Subject:In-reply-to:Date:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding;
 l=651; bh=LSw/N+WGCJZnGBvEpR+lK3cMxiJYDJPmX8OBGzkyAoE=;
 b=VEPYNikrRY9Ogyd+r4PyH7ph0lkVbYcis8ki2Jw7Kcn5jbadPI8rPsFXbXQv0crH
 CoYlm8L+3A8Qo04jcy58+dyv2Z3DB+0PbMhTRmrqdHE4gZjX40aCpG8r95iytUQl0dD
 +NExqc3yRp3cknAKDwhdbhdYhqZCtZWfDrPAKCEY=
Received: from localhost (141.80.247.250 [141.80.247.250]) by mx.zohomail.com
 with SMTPS id 1564673962656270.3743885542227;
 Thu, 1 Aug 2019 08:39:22 -0700 (PDT)
References: <87o91n3ps0.fsf@HIDDEN> <87zhl656xk.fsf@HIDDEN>
 <87pnm04s21.fsf@HIDDEN> <87ef2gr6z3.fsf@HIDDEN>
 <878ssor680.fsf@HIDDEN> <87lfwo1lz0.fsf@HIDDEN>
 <87pnm0bem6.fsf@HIDDEN>
User-agent: mu4e 1.2.0; emacs 26.2
From: Ricardo Wurmus <rekado@HIDDEN>
To: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Subject: Re: bug#36754: SSH connections to hydra-slave{1, 2,
 3} fail during builds
In-reply-to: <87pnm0bem6.fsf@HIDDEN>
X-URL: https://elephly.net
X-PGP-Key: https://elephly.net/rekado.pubkey
X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
Date: Thu, 01 Aug 2019 17:39:19 +0200
Message-ID: <87v9vgsxnc.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-ZohoMailClient: External
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 36754
Cc: Mark H Weaver <mhw@HIDDEN>, Marius Bakke <mbakke@HIDDEN>,
 36754 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)


Ricardo Wurmus <rekado@HIDDEN> writes:

> Ludovic Court=C3=A8s <ludo@HIDDEN> writes:
>
>> Currently we have an SSH tunnel set up on berlin to connect to each of
>> these machines via overdrive1.guixsd.org.  This setup proved to be
>> robust in the past (we used it to connect to another build machine), so
>> I suspect something=E2=80=99s wrong on =E2=80=9Cyour=E2=80=9D end of the=
 network.  It=E2=80=99s hard to
>> tell exactly what, though.
>
> FWIW by the end of this week we should have the firewall changes
> implemented so we can do without the SSH tunnel.

The firewall changes have been applied today.

--=20
Ricardo





Information forwarded to bug-guix@HIDDEN:
bug#36754; Package guix. Full text available.

Message received at 36754 <at> debbugs.gnu.org:


Received: (at 36754) by debbugs.gnu.org; 1 Aug 2019 14:09:15 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Aug 01 10:09:15 2019
Received: from localhost ([127.0.0.1]:54816 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1htBla-0004H9-Im
	for submit <at> debbugs.gnu.org; Thu, 01 Aug 2019 10:09:15 -0400
Received: from wout2-smtp.messagingengine.com ([64.147.123.25]:44137)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mbakke@HIDDEN>) id 1htBlW-0004Gs-Di
 for 36754 <at> debbugs.gnu.org; Thu, 01 Aug 2019 10:09:13 -0400
Received: from compute5.internal (compute5.nyi.internal [10.202.2.45])
 by mailout.west.internal (Postfix) with ESMTP id 8BDF02E0;
 Thu,  1 Aug 2019 10:09:04 -0400 (EDT)
Received: from mailfrontend1 ([10.202.2.162])
 by compute5.internal (MEProxy); Thu, 01 Aug 2019 10:09:04 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastmail.com; h=
 from:to:cc:subject:in-reply-to:references:date:message-id
 :mime-version:content-type; s=fm3; bh=FlvabjkkI+fID3UZ2mJy/KZEJ8
 Fus3U3YTjoyxuZsFM=; b=KJmAJPJ4IrM+6LY1Nrny1EdH1nAaApWMbLsFfFfS5v
 F/BBzgwYHo+7lRRYLv4EKLeLJZDXk+g3Bk1BxQvzWXPUX4DEFpOE1p2SyDE8R9Rf
 qrGwkWhNPcgGRC7HZrbSUk/4fwrRMpt69FE/6qvlNrvNmFaahIRICWm13SH102L0
 IENNUt+rhvWxy7qOk33vafaR0lBb0m3Tj/Cbf3yVjuu0Q4CNa0+437Q+OaFeNsGa
 ltMA4gZwqhnGxeVQmn6OOBM6zz3IkvYnmEOBbpY4qLcjP3IiAeJShy5NTjG01L9y
 +eeKUj78k1b3o/n2b7xU46mnKrFYj1xwOKfwMg3X+avw==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:content-type:date:from:in-reply-to
 :message-id:mime-version:references:subject:to:x-me-proxy
 :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=Flvabj
 kkI+fID3UZ2mJy/KZEJ8Fus3U3YTjoyxuZsFM=; b=e/E9vD+6uKt5a0dDbKlCB5
 Tw32Gi1argKYHsG2CEI1MONwSFGbko2yjNZLa//U9UrfDrJRjP9GKOnsBZkaMgCE
 F+zeOUgl7VozYkb+ucd1OruBf09AHGYARAdkZC3U94QNRHlVJby3q5HNPNzIxpy6
 SpxNoc4uSNXWGb3iGDU+wHenZONPH8Z1+e30CrjiNOpu9sOOcn+lODU9hizweS3n
 hQ3rYPTbjfhCgyiW1RsWU1VZHUWlJF4dM3wC4MrGQp/F0vtmRc6FQXWqDLlK699L
 ksRFSs5Pu8QPqRxbUfs/S5kxB4YCk8jhJu7iMcM8pOpTlAXqdaHa5Begfejs/xfg
 ==
X-ME-Sender: <xms:f_JCXRY_1GetPra0IkDRbrpfbVir6WWYwlrD9Gq7yRJIakQm4Z89aA>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduvddrleejgdejudcutefuodetggdotefrodftvf
 curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu
 uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc
 fjughrpefhvffujghffgffkfggtgesghdtreertdertdenucfhrhhomhepofgrrhhiuhhs
 uceurghkkhgvuceomhgsrghkkhgvsehfrghsthhmrghilhdrtghomheqnecuffhomhgrih
 hnpehgnhhurdhorhhgnecukfhppeeivddrudeirddvvdeirddugedtnecurfgrrhgrmhep
 mhgrihhlfhhrohhmpehmsggrkhhkvgesfhgrshhtmhgrihhlrdgtohhmnecuvehluhhsth
 gvrhfuihiivgeptd
X-ME-Proxy: <xmx:f_JCXcqVecMGc2RFpHchCUMGO5ppvFrauCKyHaQg9eePV43Lyw7jVA>
 <xmx:f_JCXe8vO3seFNc06agWhr7tn58Tav_NGfm2XTRqi1Fj2blz0QSdUQ>
 <xmx:f_JCXT8slya2_v2nVK-4D_3DHKzVH-64kHy-xwSmOrh7DKC9v_kMLA>
 <xmx:gPJCXYrQmYuFZikdLg3t83YsNFvYENBDFwa7D-pmwEfLa8n1XYiiug>
Received: from localhost (140.226.16.62.customer.cdi.no [62.16.226.140])
 by mail.messagingengine.com (Postfix) with ESMTPA id 20C8780064;
 Thu,  1 Aug 2019 10:09:03 -0400 (EDT)
From: Marius Bakke <mbakke@HIDDEN>
To: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>, Mark H Weaver
 <mhw@HIDDEN>
Subject: Re: bug#36754: SSH connections to hydra-slave{1, 2,
 3} fail during builds
In-Reply-To: <87lfwnxyzg.fsf@HIDDEN>
References: <87o91n3ps0.fsf@HIDDEN> <87o91n3ps0.fsf@HIDDEN>
 <87zhl656xk.fsf@HIDDEN> <87pnm04s21.fsf@HIDDEN>
 <87ef2gr6z3.fsf@HIDDEN> <878ssor680.fsf@HIDDEN>
 <87lfwo1lz0.fsf@HIDDEN> <87ef2fpu2a.fsf@HIDDEN> <87lfwnxyzg.fsf@HIDDEN>
User-Agent: Notmuch/0.29.1 (https://notmuchmail.org) Emacs/26.2
 (x86_64-pc-linux-gnu)
Date: Thu, 01 Aug 2019 16:09:00 +0200
Message-ID: <87k1bx2d1f.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="=-=-=";
 micalg=pgp-sha512; protocol="application/pgp-signature"
X-Spam-Score: -0.7 (/)
X-Debbugs-Envelope-To: 36754
Cc: Ricardo Wurmus <rekado@HIDDEN>, 36754 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

--=-=-=
Content-Type: text/plain

The truncated log files seems to happen for other builds as well, even
within the Berlin data center.

https://ci.guix.gnu.org/log/n3ra1b8ic6qhfinnhb80mrn7snsqws9d-geocode-glib-3.26.0
https://ci.guix.gnu.org/log/zqhqlib00i8f7f10g4c2dfzprw16h4xv-scintilla-4.2.0
https://ci.guix.gnu.org/log/718jmbq94mvdgnmjyqgxgy7zaj8xzxk3-htslib-1.9

All of these builds are for i686-linux.

Mark: are the armhf nodes still operational?  I would like to re-enable
them again, since we desperately need the computing power with four huge
branches going concurrently at the moment.

--=-=-=
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQEzBAEBCgAdFiEEu7At3yzq9qgNHeZDoqBt8qM6VPoFAl1C8nwACgkQoqBt8qM6
VPoYkwgAmYCwtCBizPMO0PQZEaX+jcl7LeRl77opAd4zdgNVoaKyukReqyprYgA3
0Uwp2j/Q4wwfVUd8GP5nKn4kdhhRWAK+DdNaOqcOyaBNH2jHhZgWSAnj9UkLwtA0
ebDC3g8AVTlFMGlVQc/21w2c1zpYOxPRIn+AdI04bLuKeAKTzJLDfRF+njE+Hf55
5w9EjyW+K6beR+A0jPLm/6Z9aig8QCxzkboSQdAEECtjMWyKTpDA1tu2YAAw27sC
pc0hpJPS/mIdNQv36B5rbJ9TCF4zp9DzQj22BkvUMaVBb3MVax5+qizE6StDFpd1
t9uKVGo5zGnPNO9/MxOj9YzMEHVAlg==
=PORt
-----END PGP SIGNATURE-----
--=-=-=--




Information forwarded to bug-guix@HIDDEN:
bug#36754; Package guix. Full text available.

Message received at 36754 <at> debbugs.gnu.org:


Received: (at 36754) by debbugs.gnu.org; 24 Jul 2019 14:56:46 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Wed Jul 24 10:56:46 2019
Received: from localhost ([127.0.0.1]:37856 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1hqIhC-0007vK-4F
	for submit <at> debbugs.gnu.org; Wed, 24 Jul 2019 10:56:46 -0400
Received: from eggs.gnu.org ([209.51.188.92]:53739)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1hqIh9-0007v7-Rk
 for 36754 <at> debbugs.gnu.org; Wed, 24 Jul 2019 10:56:44 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:52253)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <ludo@HIDDEN>)
 id 1hqIh4-0002R7-8D; Wed, 24 Jul 2019 10:56:38 -0400
Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=49850 helo=ribbon)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <ludo@HIDDEN>)
 id 1hqIh3-0006ug-Nf; Wed, 24 Jul 2019 10:56:38 -0400
From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>
To: Mark H Weaver <mhw@HIDDEN>
Subject: Re: bug#36754: SSH connections to hydra-slave{1, 2,
 3} fail during builds
References: <87o91n3ps0.fsf@HIDDEN> <87o91n3ps0.fsf@HIDDEN>
 <87zhl656xk.fsf@HIDDEN> <87pnm04s21.fsf@HIDDEN>
 <87ef2gr6z3.fsf@HIDDEN> <878ssor680.fsf@HIDDEN>
 <87lfwo1lz0.fsf@HIDDEN> <87ef2fpu2a.fsf@HIDDEN>
X-URL: http://www.fdn.fr/~lcourtes/
X-Revolutionary-Date: 6 Thermidor an 227 de la =?utf-8?Q?R=C3=A9volution?=
X-PGP-Key-ID: 0x090B11993D9AEBB5
X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc
X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4  0CFB 090B 1199 3D9A EBB5
X-OS: x86_64-pc-linux-gnu
Date: Wed, 24 Jul 2019 16:56:35 +0200
In-Reply-To: <87ef2fpu2a.fsf@HIDDEN> (Mark H. Weaver's message of "Wed, 24
 Jul 2019 07:09:38 -0400")
Message-ID: <87lfwnxyzg.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 36754
Cc: Ricardo Wurmus <rekado@HIDDEN>, Marius Bakke <mbakke@HIDDEN>,
 36754 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Hello,

Mark H Weaver <mhw@HIDDEN> skribis:

> Ludovic Court=C3=A8s <ludo@HIDDEN> wrote:
>> I noticed that connections to the machines were unstable (using
>> OpenSSH=E2=80=99s client).  That is, the connection would eventually =E2=
=80=9Chang=E2=80=9D,
>> apparently several times a day.
>>
>> Currently we have an SSH tunnel set up on berlin to connect to each of
>> these machines via overdrive1.guixsd.org.  This setup proved to be
>> robust in the past (we used it to connect to another build machine), so
>> I suspect something=E2=80=99s wrong on =E2=80=9Cyour=E2=80=9D end of the=
 network.  It=E2=80=99s hard to
>> tell exactly what, though.
>>
>> Ideas?
>
> Okay, I'll look into it.  I'm very busy with something else for the next
> couple of days, but I'll try to get to it in the next week.

OK!

>> If it=E2=80=99s causing build failures, I=E2=80=99m afraid we=E2=80=99ll=
 have to comment out
>> those machines from berlin=E2=80=99s machines.scm until we=E2=80=99ve fi=
gured it out.
>
> Agreed.

I=E2=80=99ve commented them out now.

Thanks,
Ludo=E2=80=99.




Information forwarded to bug-guix@HIDDEN:
bug#36754; Package guix. Full text available.

Message received at 36754 <at> debbugs.gnu.org:


Received: (at 36754) by debbugs.gnu.org; 24 Jul 2019 11:12:05 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Wed Jul 24 07:12:05 2019
Received: from localhost ([127.0.0.1]:36018 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1hqFBl-0008CI-5k
	for submit <at> debbugs.gnu.org; Wed, 24 Jul 2019 07:12:05 -0400
Received: from world.peace.net ([64.112.178.59]:51398)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mhw@HIDDEN>) id 1hqFBi-0008Bm-OV
 for 36754 <at> debbugs.gnu.org; Wed, 24 Jul 2019 07:12:04 -0400
Received: from mhw by world.peace.net with esmtpsa
 (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89)
 (envelope-from <mhw@HIDDEN>)
 id 1hqFBc-0006Ii-Dt; Wed, 24 Jul 2019 07:11:56 -0400
From: Mark H Weaver <mhw@HIDDEN>
To: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Subject: Re: bug#36754: SSH connections to hydra-slave{1, 2,
 3} fail during builds
In-Reply-To: <87lfwo1lz0.fsf@HIDDEN>
References: <87o91n3ps0.fsf@HIDDEN> <87o91n3ps0.fsf@HIDDEN>
 <87zhl656xk.fsf@HIDDEN> <87pnm04s21.fsf@HIDDEN>
 <87ef2gr6z3.fsf@HIDDEN> <878ssor680.fsf@HIDDEN>
 <87lfwo1lz0.fsf@HIDDEN>
Date: Wed, 24 Jul 2019 07:09:38 -0400
Message-ID: <87ef2fpu2a.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 36754
Cc: Ricardo Wurmus <rekado@HIDDEN>, Marius Bakke <mbakke@HIDDEN>,
 36754 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Hi Ludovic,

Ludovic Court=C3=A8s <ludo@HIDDEN> wrote:
> I noticed that connections to the machines were unstable (using
> OpenSSH=E2=80=99s client).  That is, the connection would eventually =E2=
=80=9Chang=E2=80=9D,
> apparently several times a day.
>
> Currently we have an SSH tunnel set up on berlin to connect to each of
> these machines via overdrive1.guixsd.org.  This setup proved to be
> robust in the past (we used it to connect to another build machine), so
> I suspect something=E2=80=99s wrong on =E2=80=9Cyour=E2=80=9D end of the =
network.  It=E2=80=99s hard to
> tell exactly what, though.
>
> Ideas?

Okay, I'll look into it.  I'm very busy with something else for the next
couple of days, but I'll try to get to it in the next week.

> If it=E2=80=99s causing build failures, I=E2=80=99m afraid we=E2=80=99ll =
have to comment out
> those machines from berlin=E2=80=99s machines.scm until we=E2=80=99ve fig=
ured it out.

Agreed.

     Thanks,
       Mark




Information forwarded to bug-guix@HIDDEN:
bug#36754; Package guix. Full text available.

Message received at 36754 <at> debbugs.gnu.org:


Received: (at 36754) by debbugs.gnu.org; 23 Jul 2019 21:55:26 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Jul 23 17:55:25 2019
Received: from localhost ([127.0.0.1]:35738 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1hq2kn-0000Rs-Ef
	for submit <at> debbugs.gnu.org; Tue, 23 Jul 2019 17:55:25 -0400
Received: from sender4-of-o53.zoho.com ([136.143.188.53]:21305)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <rekado@HIDDEN>) id 1hq2kl-0000Rh-GK
 for 36754 <at> debbugs.gnu.org; Tue, 23 Jul 2019 17:55:24 -0400
ARC-Seal: i=1; a=rsa-sha256; t=1563918917; cv=none; d=zoho.com; s=zohoarc; 
 b=ZQcKZa7V5lwm+FsJh7VLUlJwCPkd+xP8HVW2NNifBc5we9AF6BES95eFA2ZvhcEruxWJHXmUAq3TqyT3jX2mi4ABdW7g2RX7fPxUeiM1gCDVa07Ch4LHAa9e8qZ8dpWVGpMq5CyEftX8M5lll2P4THUFc+m3fTBKVOO0c6rtS4A=
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com;
 s=zohoarc; t=1563918917;
 h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:To:ARC-Authentication-Results;
 bh=7eGyOrb+srBmeDwnQGRsxttF5NjD+kOyTQHbOH+8vXw=; 
 b=Qk38l7WFDJ2DZ4K/XeL0RamnZen5aoBmtmHKP66KFEqBqppOrx3RgA/q1u+uSS0crxreFXYLxjw79Sto+2oWgXOzx1r4SO36TugbjBo4Zjr9h3nU6WPnO8ZK5CyG9gyKWNxKAf8niAK0l54ddEGEfjMoelPs6LoxjFtVKVriAtU=
ARC-Authentication-Results: i=1; mx.zoho.com; dkim=pass  header.i=elephly.net;
 spf=pass  smtp.mailfrom=rekado@HIDDEN;
 dmarc=pass header.from=<rekado@HIDDEN> header.from=<rekado@HIDDEN>
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1563918917; 
 s=zoho; d=elephly.net; i=rekado@HIDDEN;
 h=References:From:To:Cc:Subject:In-reply-to:Date:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding;
 l=539; bh=7eGyOrb+srBmeDwnQGRsxttF5NjD+kOyTQHbOH+8vXw=;
 b=J7y0/0At+cm83mCodU+viysyRjSZO+FHWssAom/8WcgPBjYZzaYqH0COSvrrb+WN
 BzWfwe5KNAHo++TbtaPYRSO/EbfHQbF6kd3m1Ed2YncHpRsyeltfODQATedQsFmNQA2
 msV+yk+ls+1tMln/ciJto7LrN5m2zD8F/iTdDxbI=
Received: from localhost (p54AD4F38.dip0.t-ipconnect.de [84.173.79.56]) by
 mx.zohomail.com with SMTPS id 1563918916979714.2313585498698;
 Tue, 23 Jul 2019 14:55:16 -0700 (PDT)
References: <87o91n3ps0.fsf@HIDDEN> <87zhl656xk.fsf@HIDDEN>
 <87pnm04s21.fsf@HIDDEN> <87ef2gr6z3.fsf@HIDDEN>
 <878ssor680.fsf@HIDDEN> <87lfwo1lz0.fsf@HIDDEN>
User-agent: mu4e 1.2.0; emacs 26.2
From: Ricardo Wurmus <rekado@HIDDEN>
To: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Subject: Re: bug#36754: SSH connections to hydra-slave{1, 2,
 3} fail during builds
In-reply-to: <87lfwo1lz0.fsf@HIDDEN>
X-URL: https://elephly.net
X-PGP-Key: https://elephly.net/rekado.pubkey
X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
Date: Tue, 23 Jul 2019 23:55:13 +0200
Message-ID: <87pnm0bem6.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-ZohoMailClient: External
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 36754
Cc: Mark H Weaver <mhw@HIDDEN>, Marius Bakke <mbakke@HIDDEN>,
 36754 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)


Ludovic Court=C3=A8s <ludo@HIDDEN> writes:

> Currently we have an SSH tunnel set up on berlin to connect to each of
> these machines via overdrive1.guixsd.org.  This setup proved to be
> robust in the past (we used it to connect to another build machine), so
> I suspect something=E2=80=99s wrong on =E2=80=9Cyour=E2=80=9D end of the =
network.  It=E2=80=99s hard to
> tell exactly what, though.

FWIW by the end of this week we should have the firewall changes
implemented so we can do without the SSH tunnel.

--
Ricardo





Information forwarded to bug-guix@HIDDEN:
bug#36754; Package guix. Full text available.

Message received at 36754 <at> debbugs.gnu.org:


Received: (at 36754) by debbugs.gnu.org; 23 Jul 2019 21:26:41 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Jul 23 17:26:41 2019
Received: from localhost ([127.0.0.1]:35709 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1hq2Iy-00084C-Nb
	for submit <at> debbugs.gnu.org; Tue, 23 Jul 2019 17:26:41 -0400
Received: from eggs.gnu.org ([209.51.188.92]:46632)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1hq2Iv-00083x-Ac
 for 36754 <at> debbugs.gnu.org; Tue, 23 Jul 2019 17:26:38 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:36806)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <ludo@HIDDEN>)
 id 1hq2Ip-0003UL-LX; Tue, 23 Jul 2019 17:26:31 -0400
Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=47812 helo=ribbon)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <ludo@HIDDEN>)
 id 1hq2Io-00073T-Hh; Tue, 23 Jul 2019 17:26:31 -0400
From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>
To: Mark H Weaver <mhw@HIDDEN>
Subject: Re: bug#36754: SSH connections to hydra-slave{1, 2,
 3} fail during builds
References: <87o91n3ps0.fsf@HIDDEN> <87o91n3ps0.fsf@HIDDEN>
 <87zhl656xk.fsf@HIDDEN> <87pnm04s21.fsf@HIDDEN>
 <87ef2gr6z3.fsf@HIDDEN> <878ssor680.fsf@HIDDEN>
X-URL: http://www.fdn.fr/~lcourtes/
X-Revolutionary-Date: 5 Thermidor an 227 de la =?utf-8?Q?R=C3=A9volution?=
X-PGP-Key-ID: 0x090B11993D9AEBB5
X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc
X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4  0CFB 090B 1199 3D9A EBB5
X-OS: x86_64-pc-linux-gnu
Date: Tue, 23 Jul 2019 23:26:27 +0200
In-Reply-To: <878ssor680.fsf@HIDDEN> (Mark H. Weaver's message of "Tue, 23
 Jul 2019 13:49:24 -0400")
Message-ID: <87lfwo1lz0.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 36754
Cc: Ricardo Wurmus <rekado@HIDDEN>, Marius Bakke <mbakke@HIDDEN>,
 36754 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Hi Mark,

Mark H Weaver <mhw@HIDDEN> skribis:

> I wrote earlier:
>> I now believe that these failures are related to the newly added armhf
>> build slaves, and that they have nothing to do with the recent changes
>> to our linux-libre packages.
>
> I should mention that the armhf build slaves are on a private network,
> and I use my public-facing internet server to forward TCP connections to
> them, using the following entries in /etc/inetd.conf:
>
> # TCP-level forwards for SSH connections to build machines for the GNU
> # Guix build farm:
> 7275    stream  tcp     nowait  nobody  /bin/nc /bin/nc -w 10 172.19.189.=
11 7275
> 7276    stream  tcp     nowait  nobody  /bin/nc /bin/nc -w 10 172.19.189.=
12 7276
> 7274    stream  tcp     nowait  nobody  /bin/nc /bin/nc -w 10 172.19.189.=
13 7274
>
> It's possible that this arrangement is somehow part of the problem.
> However, note that nothing has changed here in several years, and it
> worked fine on hydra.gnu.org.  The build slaves were running a *very*
> old version of Guix though.  It seems likely that the new Guile-SSH code
> doesn't cope well with this setup.

I noticed that connections to the machines were unstable (using
OpenSSH=E2=80=99s client).  That is, the connection would eventually =E2=80=
=9Chang=E2=80=9D,
apparently several times a day.

Currently we have an SSH tunnel set up on berlin to connect to each of
these machines via overdrive1.guixsd.org.  This setup proved to be
robust in the past (we used it to connect to another build machine), so
I suspect something=E2=80=99s wrong on =E2=80=9Cyour=E2=80=9D end of the ne=
twork.  It=E2=80=99s hard to
tell exactly what, though.

Ideas?

If it=E2=80=99s causing build failures, I=E2=80=99m afraid we=E2=80=99ll ha=
ve to comment out
those machines from berlin=E2=80=99s machines.scm until we=E2=80=99ve figur=
ed it out.

Thanks,
Ludo=E2=80=99.




Information forwarded to bug-guix@HIDDEN:
bug#36754; Package guix. Full text available.

Message received at 36754 <at> debbugs.gnu.org:


Received: (at 36754) by debbugs.gnu.org; 23 Jul 2019 17:51:49 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Jul 23 13:51:49 2019
Received: from localhost ([127.0.0.1]:35595 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1hpyx3-0003nd-Dw
	for submit <at> debbugs.gnu.org; Tue, 23 Jul 2019 13:51:49 -0400
Received: from world.peace.net ([64.112.178.59]:49008)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mhw@HIDDEN>) id 1hpyx2-0003nR-1N
 for 36754 <at> debbugs.gnu.org; Tue, 23 Jul 2019 13:51:48 -0400
Received: from mhw by world.peace.net with esmtpsa
 (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89)
 (envelope-from <mhw@HIDDEN>)
 id 1hpywv-0008A6-Gs; Tue, 23 Jul 2019 13:51:41 -0400
From: Mark H Weaver <mhw@HIDDEN>
To: Marius Bakke <mbakke@HIDDEN>
Subject: Re: bug#36754: SSH connections to hydra-slave{1,2,3} fail during
 builds (was: New linux-libre failed to build on armhf on Berlin)
In-Reply-To: <87ef2gr6z3.fsf@HIDDEN>
References: <87o91n3ps0.fsf@HIDDEN> <87o91n3ps0.fsf@HIDDEN>
 <87zhl656xk.fsf@HIDDEN> <87pnm04s21.fsf@HIDDEN>
 <87ef2gr6z3.fsf@HIDDEN>
Date: Tue, 23 Jul 2019 13:49:24 -0400
Message-ID: <878ssor680.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 36754
Cc: Ricardo Wurmus <rekado@HIDDEN>, 36754 <at> debbugs.gnu.org,
 Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

I wrote earlier:
> I now believe that these failures are related to the newly added armhf
> build slaves, and that they have nothing to do with the recent changes
> to our linux-libre packages.

I should mention that the armhf build slaves are on a private network,
and I use my public-facing internet server to forward TCP connections to
them, using the following entries in /etc/inetd.conf:

--8<---------------cut here---------------start------------->8---
# TCP-level forwards for SSH connections to build machines for the GNU
# Guix build farm:
7275    stream  tcp     nowait  nobody  /bin/nc /bin/nc -w 10 172.19.189.11 7275
7276    stream  tcp     nowait  nobody  /bin/nc /bin/nc -w 10 172.19.189.12 7276
7274    stream  tcp     nowait  nobody  /bin/nc /bin/nc -w 10 172.19.189.13 7274
--8<---------------cut here---------------end--------------->8---

It's possible that this arrangement is somehow part of the problem.
However, note that nothing has changed here in several years, and it
worked fine on hydra.gnu.org.  The build slaves were running a *very*
old version of Guix though.  It seems likely that the new Guile-SSH code
doesn't cope well with this setup.

       Mark




Information forwarded to bug-guix@HIDDEN:
bug#36754; Package guix. Full text available.
Changed bug title to 'SSH connections to hydra-slave{1,2,3} fail during builds' from 'New linux-libre failed to build on armhf on Berlin' Request was from Mark H Weaver <mhw@HIDDEN> to control <at> debbugs.gnu.org. Full text available.

Message received at 36754 <at> debbugs.gnu.org:


Received: (at 36754) by debbugs.gnu.org; 23 Jul 2019 17:35:36 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Jul 23 13:35:36 2019
Received: from localhost ([127.0.0.1]:35579 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1hpyhL-0001Fq-Sk
	for submit <at> debbugs.gnu.org; Tue, 23 Jul 2019 13:35:36 -0400
Received: from world.peace.net ([64.112.178.59]:48912)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mhw@HIDDEN>)
 id 1hpyhK-0001FV-Hn; Tue, 23 Jul 2019 13:35:34 -0400
Received: from mhw by world.peace.net with esmtpsa
 (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89)
 (envelope-from <mhw@HIDDEN>)
 id 1hpyhE-00080a-PP; Tue, 23 Jul 2019 13:35:28 -0400
From: Mark H Weaver <mhw@HIDDEN>
To: Marius Bakke <mbakke@HIDDEN>
Subject: Re: bug#36754: SSH connections to hydra-slave{1,2,3} fail during
 builds (was: New linux-libre failed to build on armhf on Berlin)
In-Reply-To: <87pnm04s21.fsf@HIDDEN>
References: <87o91n3ps0.fsf@HIDDEN> <87o91n3ps0.fsf@HIDDEN>
 <87zhl656xk.fsf@HIDDEN> <87pnm04s21.fsf@HIDDEN>
Date: Tue, 23 Jul 2019 13:33:09 -0400
Message-ID: <87ef2gr6z3.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 36754
Cc: Ricardo Wurmus <rekado@HIDDEN>, 36754 <at> debbugs.gnu.org,
 Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

retitle 36754 SSH connections to hydra-slave{1,2,3} fail during builds
thanks

Hi,

I've added Ludovic to the CC list, since he recently added
hydra-slave{1,2,3} to Berlin.

Marius wrote:
> I tried building 5.2.2 'interactively' on Berlin, and got an SSH error:
>=20
>   CC [M]  net/openvswitch/vport-geneve.o
>   CC [M]  net/openvswitch/vport-gre.o
>   LD [M]  net/openvswitch/openvswitch.o
> ;;; [2019/07/23 05:14:53.501502, 0] read_from_channel_port: [GSSH ERROR] =
Error reading from the channel: #<input-output: channel (closed) 14c0e60>
> Backtrace:
>           16 (apply-smob/1 #<catch-closure b79640>)
> In ice-9/boot-9.scm:
>     705:2 15 (call-with-prompt _ _ #<procedure default-prompt-handle=E2=
=80=A6>)
> In ice-9/eval.scm:
>     619:8 14 (_ #(#(#<directory (guile-user) bfb140>)))
> In guix/ui.scm:
>   1747:12 13 (run-guix-command _ . _)
> In guix/scripts/offload.scm:
>    781:22 12 (guix-offload . _)
> In ice-9/boot-9.scm:
>     829:9 11 (catch _ _ #<procedure 7f576678d910 at guix/ui.scm:703=E2=80=
=A6> =E2=80=A6)
>     829:9 10 (catch _ _ #<procedure 7f576678d928 at guix/ui.scm:826=E2=80=
=A6> =E2=80=A6)
> In guix/scripts/offload.scm:
>    580:19  9 (process-request _ _ _ _ #:print-build-trace? _ # _ # _)
>     531:6  8 (call-with-timeout _ _ _)
>     361:2  7 (transfer-and-offload #<derivation /gnu/store/yfns7ga4=E2=80=
=A6> =E2=80=A6)
> In ice-9/boot-9.scm:
>     829:9  6 (catch _ _ #<procedure dbdab0 at guix/scripts/offload.=E2=80=
=A6> =E2=80=A6)
> In guix/scripts/offload.scm:
>     385:6  5 (_)
> In guix/store.scm:
>   1203:15  4 (_ #<store-connection 256.99 19a0ba0> _ _)
>    692:11  3 (process-stderr #<store-connection 256.99 19a0ba0> _)
> In guix/serialization.scm:
>     87:11  2 (read-int _)
>     73:12  1 (get-bytevector-n* #<input-output: channel (closed) 14=E2=80=
=A6> =E2=80=A6)
> In unknown file:
>            0 (get-bytevector-n #<input-output: channel (closed) 14c=E2=80=
=A6> =E2=80=A6)
>=20
> ERROR: In procedure get-bytevector-n:
> Throw to key `guile-ssh-error' with args `("read_from_channel_port" "Erro=
r reading from the channel" #<input-output: channel (closed) 14c0e60> #f)'.
> guix build: error: build of `/gnu/store/yfns7ga468vmv9jn72snk79b16p8mhfa-=
linux-libre-5.2.2.drv' failed
>=20
> real    637m24.906s
> user    0m6.661s
> sys     0m0.897s

Thank you, this is helpful.

> Unfortunately I failed to record which machine was used and don't know a
> way to find out after the fact.

I believe it was hydra-slave2, one of the three armhf machines that I
host which were formerly part of hydra.gnu.org's build farm and were
recently added to Berlin by Ludovic.  I checked hydra-slave{1,2,3} for
build log files corresponding to the derivation above, and found that
all three of them have been attempted recently:

hydra-slave2 attempted to build it on July 23 08:07 UTC.
hydra-slave3 attempted to build it on July 22 16:40 UTC.
hydra-slave1 attempted to build it on July 22 04:44 UTC.

To be precise, each of those dates correspond to the end of the build
attempt.  All three build logs are truncated on the build machine as
well, with no error message at the end.

I now believe that these failures are related to the newly added armhf
build slaves, and that they have nothing to do with the recent changes
to our linux-libre packages.

Well, except for the silence timeout that sometimes happens on slower
machines while deblobbing linux-libre.  That's a separate issue.

      Thanks,
        Mark




Information forwarded to bug-guix@HIDDEN:
bug#36754; Package guix. Full text available.

Message received at 36754 <at> debbugs.gnu.org:


Received: (at 36754) by debbugs.gnu.org; 23 Jul 2019 16:47:00 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Jul 23 12:47:00 2019
Received: from localhost ([127.0.0.1]:35547 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1hpxwK-0006HI-0L
	for submit <at> debbugs.gnu.org; Tue, 23 Jul 2019 12:47:00 -0400
Received: from out5-smtp.messagingengine.com ([66.111.4.29]:38271)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mbakke@HIDDEN>) id 1hpxwG-0006H2-0w
 for 36754 <at> debbugs.gnu.org; Tue, 23 Jul 2019 12:46:56 -0400
Received: from compute5.internal (compute5.nyi.internal [10.202.2.45])
 by mailout.nyi.internal (Postfix) with ESMTP id 99FE420D2B;
 Tue, 23 Jul 2019 12:46:49 -0400 (EDT)
Received: from mailfrontend1 ([10.202.2.162])
 by compute5.internal (MEProxy); Tue, 23 Jul 2019 12:46:49 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastmail.com; h=
 from:to:cc:subject:in-reply-to:references:date:message-id
 :mime-version:content-type:content-transfer-encoding; s=fm3; bh=
 94RpzwMc+v0Cde5lZiZ6F4Nzf2pdlrmrVWsKdiyo2Sw=; b=G3P67n9UZQKYNgD4
 7slGqZHnF1l/mu6HvvuR/toEfqgQGvayEZK7YYfUydbTW41QnQRe1KfAM7ZK0JY8
 W8+B97Qovxc6b7Dnm5WKloqB73Wd8gPkTT87gm+ZeSdaiD/67XT0FKI2xHULsYUM
 +ix88ioucbI7cqs7+JAWHk56Z4ENeWcfrMIA4JlEuxt5Qch24+cpRuBST8afZ3VW
 hzE7SnPUr6dVoCBMMXSkTJqEyHqhxriGKyFL6I5tODQ0F8/M5rtV5TkmDgb+BDL/
 JtieTHQ5DOnzxKTgGcMakSJyTRLQrOnXWbuxIM5eOEPQGkCEU9Yy9o20oo4bQPW2
 VdEOSQ==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:content-transfer-encoding:content-type
 :date:from:in-reply-to:message-id:mime-version:references
 :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender
 :x-sasl-enc; s=fm3; bh=94RpzwMc+v0Cde5lZiZ6F4Nzf2pdlrmrVWsKdiyo2
 Sw=; b=ndbjgeQBR4fdqEDDFDiHWics2ptAgX4Wq+sprtfdOSVUs0IXHoHDj5ipG
 g8IhTJPovAqFQhlZU5no37ghZwi8b0MXRDx+U7gtnKe6f2n1p2yFSET8l1GiOXL1
 JeekS7dZ0MfRklYPKPFrWN56/dExxfD+0rclGWIIWJWhcJgc5A7zyVB7yR1/3kZt
 DNVJOnmJvtCwFBUMEl0sSQ++FWZRz7vr6s8b6YUbDZh96RctiPJI2LoH90d6iGxp
 PFERWzaz/3O4niLNN4U1FX0RpA9uwVsfhGoNuchu9g3DiFJV9I01M4zdx8jUZNPi
 39xC//BJ/giCs+4wlTKkWrgSXO3Gg==
X-ME-Sender: <xms:-Tk3XVjSIlDotENfK8qsE77Vkj4jpzS7tPEbMa-zC5S91wDm3bjA1A>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduvddrjeekgddutdegucetufdoteggodetrfdotf
 fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen
 uceurghilhhouhhtmecufedttdenucenucfjughrpefhvffujghffgffkfggtgfgsehtqh
 ertddtreejnecuhfhrohhmpeforghrihhushcuuegrkhhkvgcuoehmsggrkhhkvgesfhgr
 shhtmhgrihhlrdgtohhmqeenucffohhmrghinhepghhnuhdrohhrghenucfkphepiedvrd
 duiedrvddviedrudegtdenucfrrghrrghmpehmrghilhhfrhhomhepmhgsrghkkhgvsehf
 rghsthhmrghilhdrtghomhenucevlhhushhtvghrufhiiigvpedt
X-ME-Proxy: <xmx:-Tk3XeSUE0LWnVyB0watxCDhedvCCxbc9zjOHFhs9d9H02DQqwDQAg>
 <xmx:-Tk3XQEq9nWO27CmRTK8IbZB55GKEUGz4TF_DeuR8mq3tMPwjinc-A>
 <xmx:-Tk3XenRdwmXHgmHdkvRx2AsBjzCv8cFJz0XnnMXZiFlFXNRZemvqQ>
 <xmx:-Tk3XREOWEhhJV4aLbTmkbItX3Xgn1gsBcwZGGs7coeeVMXsY7eKKw>
Received: from localhost (140.226.16.62.customer.cdi.no [62.16.226.140])
 by mail.messagingengine.com (Postfix) with ESMTPA id 8AB8080059;
 Tue, 23 Jul 2019 12:46:48 -0400 (EDT)
From: Marius Bakke <mbakke@HIDDEN>
To: Mark H Weaver <mhw@HIDDEN>, Ricardo Wurmus <rekado@HIDDEN>
Subject: Re: bug#36754: New linux-libre failed to build on armhf on Berlin
In-Reply-To: <87zhl656xk.fsf@HIDDEN>
References: <87o91n3ps0.fsf@HIDDEN> <87o91n3ps0.fsf@HIDDEN>
 <87zhl656xk.fsf@HIDDEN>
User-Agent: Notmuch/0.29.1 (https://notmuchmail.org) Emacs/26.2
 (x86_64-pc-linux-gnu)
Date: Tue, 23 Jul 2019 18:46:46 +0200
Message-ID: <87pnm04s21.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -0.7 (/)
X-Debbugs-Envelope-To: 36754
Cc: 36754 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

Mark H Weaver <mhw@HIDDEN> writes:

> Hi Ricardo,
>
> Interesting.  I distinctly remember that there was no log file when I
> looked last time.  Hmm.
>
> Anyway, it seems that now, all of the failed builds have either build
> logs available or else information about which dependency failed.  I
> don't remember seeing any of this last time, but I'm glad to see it now.
>
> A pattern has now emerged, but I don't know what it means.  All of the
> armhf kernel builds failed except for linux-libre-arm-veyron-5.2.2,
> which succeeded:
>
>   https://ci.guix.gnu.org/build/1488502/details  (arm-veyron-5.2.2)
>
> Apart from this anomalous success, all of the armhf 5.2.2 and 4.19.60
> have a truncated log file:
>
>   https://ci.guix.gnu.org/build/1488517/details  (5.2.2)
>   https://ci.guix.gnu.org/build/1488503/details  (4.19.60)
>   https://ci.guix.gnu.org/build/1488513/details  (arm-generic-5.2.2)
>   https://ci.guix.gnu.org/build/1488519/details  (arm-generic-4.19.60)
>   https://ci.guix.gnu.org/build/1488504/details  (arm-omap2plus-5.2.2)
>   https://ci.guix.gnu.org/build/1488501/details  (arm-omap2plus-4.19.60)
>
> This pattern seems too regular to be a coincidence.  Can we find out
> which build machines were used for these builds?

I tried building 5.2.2 'interactively' on Berlin, and got an SSH error:

  CC [M]  net/openvswitch/vport-geneve.o
  CC [M]  net/openvswitch/vport-gre.o
  LD [M]  net/openvswitch/openvswitch.o
;;; [2019/07/23 05:14:53.501502, 0] read_from_channel_port: [GSSH ERROR] Er=
ror reading from the channel: #<input-output: channel (closed) 14c0e60>
Backtrace:
          16 (apply-smob/1 #<catch-closure b79640>)
In ice-9/boot-9.scm:
    705:2 15 (call-with-prompt _ _ #<procedure default-prompt-handle=E2=80=
=A6>)
In ice-9/eval.scm:
    619:8 14 (_ #(#(#<directory (guile-user) bfb140>)))
In guix/ui.scm:
  1747:12 13 (run-guix-command _ . _)
In guix/scripts/offload.scm:
   781:22 12 (guix-offload . _)
In ice-9/boot-9.scm:
    829:9 11 (catch _ _ #<procedure 7f576678d910 at guix/ui.scm:703=E2=80=
=A6> =E2=80=A6)
    829:9 10 (catch _ _ #<procedure 7f576678d928 at guix/ui.scm:826=E2=80=
=A6> =E2=80=A6)
In guix/scripts/offload.scm:
   580:19  9 (process-request _ _ _ _ #:print-build-trace? _ # _ # _)
    531:6  8 (call-with-timeout _ _ _)
    361:2  7 (transfer-and-offload #<derivation /gnu/store/yfns7ga4=E2=80=
=A6> =E2=80=A6)
In ice-9/boot-9.scm:
    829:9  6 (catch _ _ #<procedure dbdab0 at guix/scripts/offload.=E2=80=
=A6> =E2=80=A6)
In guix/scripts/offload.scm:
    385:6  5 (_)
In guix/store.scm:
  1203:15  4 (_ #<store-connection 256.99 19a0ba0> _ _)
   692:11  3 (process-stderr #<store-connection 256.99 19a0ba0> _)
In guix/serialization.scm:
    87:11  2 (read-int _)
    73:12  1 (get-bytevector-n* #<input-output: channel (closed) 14=E2=80=
=A6> =E2=80=A6)
In unknown file:
           0 (get-bytevector-n #<input-output: channel (closed) 14c=E2=80=
=A6> =E2=80=A6)

ERROR: In procedure get-bytevector-n:
Throw to key `guile-ssh-error' with args `("read_from_channel_port" "Error =
reading from the channel" #<input-output: channel (closed) 14c0e60> #f)'.
guix build: error: build of `/gnu/store/yfns7ga468vmv9jn72snk79b16p8mhfa-li=
nux-libre-5.2.2.drv' failed

real    637m24.906s
user    0m6.661s
sys     0m0.897s

Unfortunately I failed to record which machine was used and don't know a
way to find out after the fact.




Information forwarded to bug-guix@HIDDEN:
bug#36754; Package guix. Full text available.

Message received at 36754 <at> debbugs.gnu.org:


Received: (at 36754) by debbugs.gnu.org; 22 Jul 2019 17:14:07 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Jul 22 13:14:07 2019
Received: from localhost ([127.0.0.1]:33567 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1hpbt0-0004Ab-Ng
	for submit <at> debbugs.gnu.org; Mon, 22 Jul 2019 13:14:07 -0400
Received: from world.peace.net ([64.112.178.59]:40750)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mhw@HIDDEN>) id 1hpbsz-00049t-9S
 for 36754 <at> debbugs.gnu.org; Mon, 22 Jul 2019 13:14:05 -0400
Received: from mhw by world.peace.net with esmtpsa
 (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89)
 (envelope-from <mhw@HIDDEN>)
 id 1hpbss-0007oE-Pw; Mon, 22 Jul 2019 13:13:58 -0400
From: Mark H Weaver <mhw@HIDDEN>
To: Ricardo Wurmus <rekado@HIDDEN>
Subject: Re: bug#36754: New linux-libre failed to build on armhf on Berlin
In-Reply-To: <87o91n3ps0.fsf@HIDDEN>
References: <87o91n3ps0.fsf@HIDDEN> <87o91n3ps0.fsf@HIDDEN>
Date: Mon, 22 Jul 2019 13:13:11 -0400
Message-ID: <87zhl656xk.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 36754
Cc: 36754 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Hi Ricardo,

Interesting.  I distinctly remember that there was no log file when I
looked last time.  Hmm.

Anyway, it seems that now, all of the failed builds have either build
logs available or else information about which dependency failed.  I
don't remember seeing any of this last time, but I'm glad to see it now.

A pattern has now emerged, but I don't know what it means.  All of the
armhf kernel builds failed except for linux-libre-arm-veyron-5.2.2,
which succeeded:

  https://ci.guix.gnu.org/build/1488502/details  (arm-veyron-5.2.2)

Apart from this anomalous success, all of the armhf 5.2.2 and 4.19.60
have a truncated log file:

  https://ci.guix.gnu.org/build/1488517/details  (5.2.2)
  https://ci.guix.gnu.org/build/1488503/details  (4.19.60)
  https://ci.guix.gnu.org/build/1488513/details  (arm-generic-5.2.2)
  https://ci.guix.gnu.org/build/1488519/details  (arm-generic-4.19.60)
  https://ci.guix.gnu.org/build/1488504/details  (arm-omap2plus-5.2.2)
  https://ci.guix.gnu.org/build/1488501/details  (arm-omap2plus-4.19.60)

This pattern seems too regular to be a coincidence.  Can we find out
which build machines were used for these builds?

All of the 4.14.134 builds failed in the deblobbing step, due to timeout
(1 hour of silence) while packing the linux-libre tarball:

  https://ci.guix.gnu.org/build/1488514/details  (4.14.134)
  https://ci.guix.gnu.org/build/1488515/details  (arm-generic-4.14.134)
  https://ci.guix.gnu.org/build/1488512/details  (arm-omap2plus-4.14.134)

I'm not sure how to deal with this.  This is a computed origin, not a
normal package, and so I don't see a way to configure a longer timeout.

Perhaps I should make the tarball packing and unpacking operations
verbose, to work around the issue.  Of course that's our usual practice,
but I find it suboptimal because any warnings will be buried in a
mountain of uninteresting output.

Thoughts?  Anyway, thanks for looking into it.

       Mark




Information forwarded to bug-guix@HIDDEN:
bug#36754; Package guix. Full text available.

Message received at 36754 <at> debbugs.gnu.org:


Received: (at 36754) by debbugs.gnu.org; 22 Jul 2019 16:10:55 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Jul 22 12:10:55 2019
Received: from localhost ([127.0.0.1]:33513 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1hpatq-0002bA-TR
	for submit <at> debbugs.gnu.org; Mon, 22 Jul 2019 12:10:55 -0400
Received: from sender-of-o51.zoho.com ([135.84.80.216]:21298)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <rekado@HIDDEN>) id 1hpatp-0002b1-0A
 for 36754 <at> debbugs.gnu.org; Mon, 22 Jul 2019 12:10:54 -0400
ARC-Seal: i=1; a=rsa-sha256; t=1563811851; cv=none; d=zoho.com; s=zohoarc; 
 b=KfxV7kaQZTj3Go26+NG5DshdgGwmrv47Jn41doISyv0ru4DUi19gyUlSDBdj8oPLSXKnbMpH0r8afF6SQJ4U4i4AFi3ShxF///+qwD8sSlxDmcJgyKEZ7Dad48MfDsetlOhYsTAFgFHDA/E5I8ETaKGEFENYXeTmJoYFdRv9bYk=
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com;
 s=zohoarc; t=1563811851;
 h=Content-Type:Cc:Date:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:To:ARC-Authentication-Results;
 bh=Xy+zDoPLaJr3BMAF8LWAlfBzADU20UzytbCj6Nto7PU=; 
 b=GRqZ3+Y+z/b2877M527AzGrJOCqNT5ml9rmQht8RrR0IoOTTHZzKum7GoVO0m5rq76nK8Wd+YGDughyooz/6TUlOgzbtdKt6E5wAf2xMUXgPE7jJL3Hk00x39fUuBfDU+vQmV84b6GI0h5Ve6hM5CWsXvgdhx9MNmm7a35N3a78=
ARC-Authentication-Results: i=1; mx.zoho.com; dkim=pass  header.i=elephly.net;
 spf=pass  smtp.mailfrom=rekado@HIDDEN;
 dmarc=pass header.from=<rekado@HIDDEN> header.from=<rekado@HIDDEN>
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1563811851; 
 s=zoho; d=elephly.net; i=rekado@HIDDEN;
 h=References:From:To:Cc:Subject:In-reply-to:Date:Message-ID:MIME-Version:Content-Type;
 l=563; bh=Xy+zDoPLaJr3BMAF8LWAlfBzADU20UzytbCj6Nto7PU=;
 b=H53/nyRd/fubUmibObMuS8MEmZbaT54xhnLHCWMl7xFYBYs5uj0PvADAsNRsZhen
 agSPhWp6saNdbPdxSQelltqldceNkCuFvyyEmObHn9XUkejrCVCbElw/XFCscwyLjlY
 1Auv1SaezcQH8fIpNM0DhuVox29IY63XDtGgoeR4=
Received: from localhost (141.80.247.250 [141.80.247.250]) by mx.zohomail.com
 with SMTPS id 1563811849061428.6631695868524;
 Mon, 22 Jul 2019 09:10:49 -0700 (PDT)
References: <87o91n3ps0.fsf@HIDDEN>
User-agent: mu4e 1.2.0; emacs 26.2
From: Ricardo Wurmus <rekado@HIDDEN>
To: mhw@HIDDEN
Subject: Re: bug#36754: New linux-libre failed to build on armhf on Berlin
In-reply-to: <87o91n3ps0.fsf@HIDDEN>
X-URL: https://elephly.net
X-PGP-Key: https://elephly.net/rekado.pubkey
X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
Date: Mon, 22 Jul 2019 18:10:46 +0200
Message-ID: <87pnm2cant.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain
X-ZohoMailClient: External
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 36754
Cc: 36754 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)


Mark H Weaver <mhw@HIDDEN> writes:

> Unfortunately, I'm unable to get *any* information about what went wrong
> from Cuirass.  None of the failed builds have associated log files, and
> the build details page has no useful information either.  For example:
>
>   https://ci.guix.gnu.org/build/1488517/details

On that page I see a link to the build log, but it appears to be
truncated:

    https://ci.guix.gnu.org/log/33hv7mij9bqqgf5hqwrw14106z9zgav9-linux-libre-5.2.2

Maybe the build node died before the build could be completed?

-- 
Ricardo





Information forwarded to bug-guix@HIDDEN:
bug#36754; Package guix. Full text available.
Severity set to 'important' from 'normal' Request was from Mark H Weaver <mhw@HIDDEN> to control <at> debbugs.gnu.org. Full text available.

Message received at submit <at> debbugs.gnu.org:


Received: (at submit) by debbugs.gnu.org; 21 Jul 2019 23:58:41 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Jul 21 19:58:41 2019
Received: from localhost ([127.0.0.1]:59703 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1hpLiz-00014s-AA
	for submit <at> debbugs.gnu.org; Sun, 21 Jul 2019 19:58:41 -0400
Received: from lists.gnu.org ([209.51.188.17]:43162)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mhw@HIDDEN>) id 1hpLix-00014k-Pk
 for submit <at> debbugs.gnu.org; Sun, 21 Jul 2019 19:58:40 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:41774)
 by lists.gnu.org with esmtp (Exim 4.86_2)
 (envelope-from <mhw@HIDDEN>) id 1hpLiw-0007Yo-Jm
 for bug-guix@HIDDEN; Sun, 21 Jul 2019 19:58:39 -0400
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,URIBL_BLOCKED
 autolearn=disabled version=3.3.2
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <mhw@HIDDEN>) id 1hpLiv-0003g0-El
 for bug-guix@HIDDEN; Sun, 21 Jul 2019 19:58:38 -0400
Received: from world.peace.net ([64.112.178.59]:52366)
 by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32)
 (Exim 4.71) (envelope-from <mhw@HIDDEN>) id 1hpLiv-0003dd-BO
 for bug-guix@HIDDEN; Sun, 21 Jul 2019 19:58:37 -0400
Received: from mhw by world.peace.net with esmtpsa
 (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89)
 (envelope-from <mhw@HIDDEN>)
 id 1hpLiu-0001Vl-0m; Sun, 21 Jul 2019 19:58:36 -0400
From: Mark H Weaver <mhw@HIDDEN>
To: bug-guix@HIDDEN
Subject: New linux-libre failed to build on armhf on Berlin
Date: Sun, 21 Jul 2019 19:56:20 -0400
Message-ID: <87o91n3ps0.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 64.112.178.59
X-Spam-Score: -1.3 (-)
X-Debbugs-Envelope-To: submit
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -2.3 (--)

In commit 1ad9c105c208caa9059924cbfbe4759c8101f6c9, I changed our
linux-libre packages to deblob the linux-libre source tarballs
ourselves, i.e. to run the deblobbing scripts provided by the
linux-libre project to produce linux-libre source tarballs from the
upstream linux tarballs:

  https://git.savannah.gnu.org/cgit/guix.git/commit/?id=1ad9c105c208caa9059924cbfbe4759c8101f6c9

The following queries show that the updated packages built successfully
on x86_64, i686, and aarch64, but they all failed on armhf:

  https://ci.guix.gnu.org/search?query=linux-libre-5.2.2
  https://ci.guix.gnu.org/search?query=linux-libre-4.19.60
  https://ci.guix.gnu.org/search?query=linux-libre-4.14.134
  https://ci.guix.gnu.org/search?query=linux-libre-4.9.186
  https://ci.guix.gnu.org/search?query=linux-libre-4.4.186
  https://ci.guix.gnu.org/search?query=linux-libre-arm-veyron-5.2.2
  https://ci.guix.gnu.org/search?query=linux-libre-arm-generic-5.2.2
  https://ci.guix.gnu.org/search?query=linux-libre-arm-generic-4.19.60
  https://ci.guix.gnu.org/search?query=linux-libre-arm-generic-4.14.134
  https://ci.guix.gnu.org/search?query=linux-libre-arm-omap2plus-5.2.2
  https://ci.guix.gnu.org/search?query=linux-libre-arm-omap2plus-4.19.60
  https://ci.guix.gnu.org/search?query=linux-libre-arm-omap2plus-4.14.134

Unfortunately, I'm unable to get *any* information about what went wrong
from Cuirass.  None of the failed builds have associated log files, and
the build details page has no useful information either.  For example:

  https://ci.guix.gnu.org/build/1488517/details

My first guess was that something went wrong in the 'computed' origin
that runs the deblobbing script.  However, that's apparently not the
case, because all of the updated 'linux-libre-headers' packages built
successfully on armhf, and those use the same source tarballs as the
main 'linux-libre' packages.

  https://ci.guix.gnu.org/search?query=linux-libre-headers-5.2.2
  https://ci.guix.gnu.org/search?query=linux-libre-headers-4.19.60
  https://ci.guix.gnu.org/search?query=linux-libre-headers-4.14.134

Can someone help me find out what's going on here?  Until then, I'm
sorry to say that armhf-linux users will be unable to update their
systems.

       Mark




Acknowledgement sent to Mark H Weaver <mhw@HIDDEN>:
New bug report received and forwarded. Copy sent to bug-guix@HIDDEN. Full text available.
Report forwarded to bug-guix@HIDDEN:
bug#36754; Package guix. Full text available.
Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.
Last modified: Thu, 1 Aug 2019 16:45:01 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.