X-Loop: help-debbugs@HIDDEN Subject: bug#23666: guix download fails for large files Resent-From: Andreas Enge <andreas@HIDDEN> Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> Resent-CC: bug-guix@HIDDEN Resent-Date: Tue, 31 May 2016 19:57:02 +0000 Resent-Message-ID: <handler.23666.B.146472461217122 <at> debbugs.gnu.org> Resent-Sender: help-debbugs@HIDDEN X-GNU-PR-Message: report 23666 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: 23666 <at> debbugs.gnu.org X-Debbugs-Original-To: bug-guix@HIDDEN Received: via spool by submit <at> debbugs.gnu.org id=B.146472461217122 (code B ref -1); Tue, 31 May 2016 19:57:02 +0000 Received: (at submit) by debbugs.gnu.org; 31 May 2016 19:56:52 +0000 Received: from localhost ([127.0.0.1]:48675 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1b7pmV-0004S5-Si for submit <at> debbugs.gnu.org; Tue, 31 May 2016 15:56:52 -0400 Received: from eggs.gnu.org ([208.118.235.92]:45727) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <andreas@HIDDEN>) id 1b7pmU-0004Rt-CT for submit <at> debbugs.gnu.org; Tue, 31 May 2016 15:56:50 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from <andreas@HIDDEN>) id 1b7pmO-0003T5-DZ for submit <at> debbugs.gnu.org; Tue, 31 May 2016 15:56:45 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50,T_DKIM_INVALID autolearn=disabled version=3.3.2 Received: from lists.gnu.org ([2001:4830:134:3::11]:50378) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <andreas@HIDDEN>) id 1b7pmO-0003Ss-AM for submit <at> debbugs.gnu.org; Tue, 31 May 2016 15:56:44 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:36685) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from <andreas@HIDDEN>) id 1b7pmM-00037g-4y for bug-guix@HIDDEN; Tue, 31 May 2016 15:56:43 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from <andreas@HIDDEN>) id 1b7pmH-0003Rg-PI for bug-guix@HIDDEN; Tue, 31 May 2016 15:56:41 -0400 Received: from mailrelay2.public.one.com ([91.198.169.125]:44634) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <andreas@HIDDEN>) id 1b7pmH-0003R5-Bz for bug-guix@HIDDEN; Tue, 31 May 2016 15:56:37 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=enge.fr; s=20140924; h=from:subject:date:message-id:to:mime-version:content-type; bh=kBj5xaM7LmW8CzWvEwNV+gW0dYczbOtrQZVi7jjxVeg=; b=qmbIZyYtZI4U6TdaBsKC9i3cPCx6CTlH3O5zHsZK3koOnsz/VHO23oGmKZqiIhlU/G5xehIaszwLt Jyc12Tjfn0os/GyDwFEeVVIDkG5Xa0chJihwtT08S7+G5FXIcOP8ez/MkZH/ZMNF8bQDMLaLZ/3aYN H857ZgjDuGCtqVhI= X-HalOne-Cookie: af39bd020affca61a32a26ee296b8dfa93ba0fea X-HalOne-ID: c69c5c69-2769-11e6-8278-b82a72d03b9b Received: from solar (unknown [92.89.73.211]) by smtpfilter2.public.one.com (Halon Mail Gateway) with ESMTPSA; Tue, 31 May 2016 19:56:33 +0000 (UTC) Date: Tue, 31 May 2016 21:56:31 +0200 From: Andreas Enge <andreas@HIDDEN> Message-ID: <20160531195631.GA22897@solar> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.6.0 (2016-04-01) X-detected-operating-system: by eggs.gnu.org: FreeBSD 9.x X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x X-Received-From: 2001:4830:134:3::11 X-Spam-Score: -5.0 (-----) X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -5.0 (-----) Hello, since I am regularly rebuilding texlive (who is not!), I would like to keep the source tarball on my machine, and whenever I garbage collect it from the store, add it back via guix download file:texlive-20150523-texmf.tar.xz On my Novena with a 32 bit processor and 4 GB of main memory, this fails with guix download: error: sendfile: Broken pipe The file itself has 1,8 GB. Downloading the file via http as part of "guix build" works, and "guix download file:" also works on my x86_64 machine with 8 GB of RAM. Andreas
Content-Disposition: inline Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Mailer: MIME-tools 5.505 (Entity 5.505) Content-Type: text/plain; charset=utf-8 X-Loop: help-debbugs@HIDDEN From: help-debbugs@HIDDEN (GNU bug Tracking System) To: Andreas Enge <andreas@HIDDEN> Subject: bug#23666: Acknowledgement (guix download fails for large files) Message-ID: <handler.23666.B.146472461217122.ack <at> debbugs.gnu.org> References: <20160531195631.GA22897@solar> X-Gnu-PR-Message: ack 23666 X-Gnu-PR-Package: guix Reply-To: 23666 <at> debbugs.gnu.org Date: Tue, 31 May 2016 19:57:02 +0000 Thank you for filing a new bug report with debbugs.gnu.org. This is an automatically generated reply to let you know your message has been received. Your message is being forwarded to the package maintainers and other interested parties for their attention; they will reply in due course. Your message has been sent to the package maintainer(s): bug-guix@HIDDEN If you wish to submit further information on this problem, please send it to 23666 <at> debbugs.gnu.org. Please do not send mail to help-debbugs@HIDDEN unless you wish to report a problem with the Bug-tracking system. --=20 23666: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D23666 GNU Bug Tracking System Contact help-debbugs@HIDDEN with problems
X-Loop: help-debbugs@HIDDEN Subject: bug#23666: guix download fails for large files Resent-From: Leo Famulari <leo@HIDDEN> Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> Resent-CC: bug-guix@HIDDEN Resent-Date: Tue, 31 May 2016 20:27:02 +0000 Resent-Message-ID: <handler.23666.B23666.146472641926193 <at> debbugs.gnu.org> Resent-Sender: help-debbugs@HIDDEN X-GNU-PR-Message: followup 23666 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: Andreas Enge <andreas@HIDDEN> Cc: 23666 <at> debbugs.gnu.org Received: via spool by 23666-submit <at> debbugs.gnu.org id=B23666.146472641926193 (code B ref 23666); Tue, 31 May 2016 20:27:02 +0000 Received: (at 23666) by debbugs.gnu.org; 31 May 2016 20:26:59 +0000 Received: from localhost ([127.0.0.1]:48719 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1b7qFe-0006oP-MV for submit <at> debbugs.gnu.org; Tue, 31 May 2016 16:26:58 -0400 Received: from out2-smtp.messagingengine.com ([66.111.4.26]:46228) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <leo@HIDDEN>) id 1b7qFc-0006oH-DW for 23666 <at> debbugs.gnu.org; Tue, 31 May 2016 16:26:56 -0400 Received: from compute6.internal (compute6.nyi.internal [10.202.2.46]) by mailout.nyi.internal (Postfix) with ESMTP id A266C20B35; Tue, 31 May 2016 16:26:55 -0400 (EDT) Received: from frontend1 ([10.202.2.160]) by compute6.internal (MEProxy); Tue, 31 May 2016 16:26:55 -0400 DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=famulari.name; h= cc:content-type:date:from:in-reply-to:message-id:mime-version :references:subject:to:x-sasl-enc:x-sasl-enc; s=mesmtp; bh=ci7ZH kxoAqJcZgbCXEPfzAzbadw=; b=wBvlh2Dy/5+9fcPBbj2ahrFoQrol/AxImXINo KdgW/ts/nBriQQzCBHdx/a+5sudyCzsO44mMLRoTjMgk+PKdrbsw8jxl4t4PF1iU dwAX9pEvTlfTeV1OLG6B/50iDwFopS9SliEApBGEciR5yDz6gaB1VYtIoIvzo2Ev kLMK+g= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-sasl-enc :x-sasl-enc; s=smtpout; bh=ci7ZHkxoAqJcZgbCXEPfzAzbadw=; b=uyaVq shwc57CjZkEA0q11njZK1iv/7jw9T/vWVQjZZtMjNwQenCVfUBp/FS2lxi3gI1y6 ZAwNTwJ5Si8ublCnx5WAaY7RwfwKP9sAeiGcfKFmtKHpl62vCJyx0FiWeg7riSww V0FM3Ttq/WvOFSc/+weG3eSBFw/Nm3NsAd3mbI= X-Sasl-enc: UYl0CvVMMJrAQfmdwx9sRWz48BcYt0/yDMmVarjj/FJ5 1464726415 Received: from localhost (c-73-188-17-148.hsd1.pa.comcast.net [73.188.17.148]) by mail.messagingengine.com (Postfix) with ESMTPA id 57301F2A4D; Tue, 31 May 2016 16:26:55 -0400 (EDT) Date: Tue, 31 May 2016 16:26:51 -0400 From: Leo Famulari <leo@HIDDEN> Message-ID: <20160531202651.GA19070@jasmine> References: <20160531195631.GA22897@solar> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160531195631.GA22897@solar> User-Agent: Mutt/1.6.0 (2016-04-01) X-Spam-Score: -0.7 (/) X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -0.7 (/) On Tue, May 31, 2016 at 09:56:31PM +0200, Andreas Enge wrote: > since I am regularly rebuilding texlive (who is not!), I would like to keep > the source tarball on my machine, and whenever I garbage collect it from the > store, add it back via > guix download file:texlive-20150523-texmf.tar.xz > > On my Novena with a 32 bit processor and 4 GB of main memory, this fails with > guix download: error: sendfile: Broken pipe > > The file itself has 1,8 GB. > > Downloading the file via http as part of "guix build" works, and > "guix download file:" also works on my x86_64 machine with 8 GB of RAM. As a temporary workaround, you could make the tarball a gcroot: # ln -sf /gnu/store/...-texlive-20150523-texmf.tar.xz \ /var/guix/gcroots/texlive-texmf.tar.xz
X-Loop: help-debbugs@HIDDEN Subject: bug#23666: guix download fails for large files Resent-From: ludo@HIDDEN (Ludovic =?UTF-8?Q?Court=C3=A8s?=) Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> Resent-CC: bug-guix@HIDDEN Resent-Date: Wed, 01 Jun 2016 12:41:01 +0000 Resent-Message-ID: <handler.23666.B23666.14647848094206 <at> debbugs.gnu.org> Resent-Sender: help-debbugs@HIDDEN X-GNU-PR-Message: followup 23666 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: Andreas Enge <andreas@HIDDEN> Cc: 23666 <at> debbugs.gnu.org Received: via spool by 23666-submit <at> debbugs.gnu.org id=B23666.14647848094206 (code B ref 23666); Wed, 01 Jun 2016 12:41:01 +0000 Received: (at 23666) by debbugs.gnu.org; 1 Jun 2016 12:40:09 +0000 Received: from localhost ([127.0.0.1]:49171 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1b85RQ-00015m-T3 for submit <at> debbugs.gnu.org; Wed, 01 Jun 2016 08:40:09 -0400 Received: from eggs.gnu.org ([208.118.235.92]:58685) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <ludo@HIDDEN>) id 1b85RP-00015a-8f for 23666 <at> debbugs.gnu.org; Wed, 01 Jun 2016 08:40:07 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from <ludo@HIDDEN>) id 1b85RG-0003Mw-S9 for 23666 <at> debbugs.gnu.org; Wed, 01 Jun 2016 08:40:02 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-3.3 required=5.0 tests=BAYES_00,RP_MATCHES_RCVD autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:41997) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <ludo@HIDDEN>) id 1b85RG-0003Mg-Pc; Wed, 01 Jun 2016 08:39:58 -0400 Received: from reverse-83.fdn.fr ([80.67.176.83]:39376 helo=pluto) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_128_CBC_SHA1:128) (Exim 4.82) (envelope-from <ludo@HIDDEN>) id 1b85RF-0002o3-0i; Wed, 01 Jun 2016 08:39:57 -0400 From: ludo@HIDDEN (Ludovic =?UTF-8?Q?Court=C3=A8s?=) References: <20160531195631.GA22897@solar> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 14 Prairial an 224 de la =?UTF-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-unknown-linux-gnu Date: Wed, 01 Jun 2016 14:39:54 +0200 In-Reply-To: <20160531195631.GA22897@solar> (Andreas Enge's message of "Tue, 31 May 2016 21:56:31 +0200") Message-ID: <878typkrqd.fsf@HIDDEN> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-Spam-Score: -6.4 (------) X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -6.4 (------) Andreas Enge <andreas@HIDDEN> skribis: > On my Novena with a 32 bit processor and 4 GB of main memory, this fails = with > guix download: error: sendfile: Broken pipe > > The file itself has 1,8 GB. This is implemented using the =E2=80=98add-to-store=E2=80=99 RPC, which, af= ter all these years, is still implemented like this (nix/libstore/local-store.cc): --8<---------------cut here---------------start------------->8--- Path LocalStore::addToStore(const string & name, const Path & _srcPath, bool recursive, HashType hashAlgo, PathFilter & filter, bool repair) { Path srcPath(absPath(_srcPath)); debug(format("adding `%1%' to the store") % srcPath); /* Read the whole path into memory. This is not a very scalable method for very large paths, but `copyPath' is mainly used for small files. */ --8<---------------cut here---------------end--------------->8--- This is undoubtedly what=E2=80=99s causing the failure. Until this is fixed, you could wrap the file in a nar and import the nar (nar import does not have this limitation, see =E2=80=99LocalStore::importP= ath=E2=80=99 and co.). To do that, you could do: guix archive --export $(guix build -S texlive-texmf-whatever) > t.nar and later: guix archive --import < t.nar HTH, Ludo=E2=80=99.
Received: (at request) by debbugs.gnu.org; 1 Jun 2016 12:41:11 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Wed Jun 01 08:41:11 2016 Received: from localhost ([127.0.0.1]:49175 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1b85SR-00017V-7U for submit <at> debbugs.gnu.org; Wed, 01 Jun 2016 08:41:11 -0400 Received: from eggs.gnu.org ([208.118.235.92]:58969) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <ludo@HIDDEN>) id 1b85SP-00017J-JL for request <at> debbugs.gnu.org; Wed, 01 Jun 2016 08:41:09 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from <ludo@HIDDEN>) id 1b85SF-0003aS-Mh for request <at> debbugs.gnu.org; Wed, 01 Jun 2016 08:41:04 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-3.3 required=5.0 tests=BAYES_00,RP_MATCHES_RCVD autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:42008) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <ludo@HIDDEN>) id 1b85SF-0003a0-J5 for request <at> debbugs.gnu.org; Wed, 01 Jun 2016 08:40:59 -0400 Received: from reverse-83.fdn.fr ([80.67.176.83]:39378 helo=pluto) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_128_CBC_SHA1:128) (Exim 4.82) (envelope-from <ludo@HIDDEN>) id 1b85SD-0006Ca-Ny for request <at> debbugs.gnu.org; Wed, 01 Jun 2016 08:40:58 -0400 From: ludo@HIDDEN (Ludovic =?utf-8?Q?Court=C3=A8s?=) To: request <at> debbugs.gnu.org Subject: retitle X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 14 Prairial an 224 de la =?utf-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-unknown-linux-gnu Date: Wed, 01 Jun 2016 14:40:55 +0200 Message-ID: <874m9dkroo.fsf@HIDDEN> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-Spam-Score: -6.4 (------) X-Debbugs-Envelope-To: request X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -6.4 (------) retitle 23666 'add-to-store' RPC loads files in memory (daemon-side) thanks
X-Loop: help-debbugs@HIDDEN Subject: bug#23666: guix download fails for large files Resent-From: Andreas Enge <andreas@HIDDEN> Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> Resent-CC: bug-guix@HIDDEN Resent-Date: Sun, 05 Jun 2016 20:59:02 +0000 Resent-Message-ID: <handler.23666.B23666.146516033030696 <at> debbugs.gnu.org> Resent-Sender: help-debbugs@HIDDEN X-GNU-PR-Message: followup 23666 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN> Cc: 23666 <at> debbugs.gnu.org Received: via spool by 23666-submit <at> debbugs.gnu.org id=B23666.146516033030696 (code B ref 23666); Sun, 05 Jun 2016 20:59:02 +0000 Received: (at 23666) by debbugs.gnu.org; 5 Jun 2016 20:58:50 +0000 Received: from localhost ([127.0.0.1]:55929 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1b9f8E-0007z2-8L for submit <at> debbugs.gnu.org; Sun, 05 Jun 2016 16:58:50 -0400 Received: from mailrelay7.public.one.com ([91.198.169.215]:59680) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <andreas@HIDDEN>) id 1b9f8C-0007yo-WB for 23666 <at> debbugs.gnu.org; Sun, 05 Jun 2016 16:58:49 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=enge.fr; s=20140924; h=from:subject:date:message-id:to:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:references; bh=4wcLuDq4Cuys5+EXVjAy4+p92GPUvvZ2E4MgjEgpMy0=; b=nCvhLO8OIo9fS7+vGLEcguOh7ECQKZL0Z15BjTk+VDFKpOD7/kzwYaUfACJD+2G9GXm+ynIbjk/Wv j2BwJLJ9gPe1EWnHrHoJxqBEz7dNs/omkass+LW5x1Gar/XCLb4Ra0VDnf9yv6rMkjtzboVPssjUAy SCmKK80yPM3PTGFU= X-HalOne-Cookie: 3db6237ac8803e7fa0dd8ad0f131e55fc05d0f3c X-HalOne-ID: 488d750b-2b60-11e6-a0c6-b82a72cffc46 Received: from solar (unknown [92.89.73.211]) by smtpfilter4.public.one.com (Halon Mail Gateway) with ESMTPSA; Sun, 5 Jun 2016 20:58:41 +0000 (UTC) Date: Sun, 5 Jun 2016 22:58:39 +0200 From: Andreas Enge <andreas@HIDDEN> Message-ID: <20160605205839.GB20831@solar> References: <20160531195631.GA22897@solar> <878typkrqd.fsf@HIDDEN> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <878typkrqd.fsf@HIDDEN> User-Agent: Mutt/1.6.0 (2016-04-01) X-Spam-Score: 0.0 (/) X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: 0.0 (/) Thanks for your suggestions, Leo and Ludovic! I still see this as a bug; should I report it upstream to Nix? On Wed, Jun 01, 2016 at 02:39:54PM +0200, Ludovic Courtès wrote: > This is implemented using the ‘add-to-store’ RPC, which, after all these > years, is still implemented like this (nix/libstore/local-store.cc): > > --8<---------------cut here---------------start------------->8--- > Path LocalStore::addToStore(const string & name, const Path & _srcPath, > bool recursive, HashType hashAlgo, PathFilter & filter, bool repair) > { > Path srcPath(absPath(_srcPath)); > debug(format("adding `%1%' to the store") % srcPath); > > /* Read the whole path into memory. This is not a very scalable > method for very large paths, but `copyPath' is mainly used for > small files. */ > --8<---------------cut here---------------end--------------->8--- Something that mainly does not fail could indeed be seen as a bug... But how come that "guix download http://" succeeds, where "guix download file://" fails? Andreas
X-Loop: help-debbugs@HIDDEN Subject: bug#23666: guix download fails for large files Resent-From: ludo@HIDDEN (Ludovic =?UTF-8?Q?Court=C3=A8s?=) Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> Resent-CC: bug-guix@HIDDEN Resent-Date: Mon, 06 Jun 2016 07:07:01 +0000 Resent-Message-ID: <handler.23666.B23666.14651967807826 <at> debbugs.gnu.org> Resent-Sender: help-debbugs@HIDDEN X-GNU-PR-Message: followup 23666 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: Andreas Enge <andreas@HIDDEN> Cc: 23666 <at> debbugs.gnu.org Received: via spool by 23666-submit <at> debbugs.gnu.org id=B23666.14651967807826 (code B ref 23666); Mon, 06 Jun 2016 07:07:01 +0000 Received: (at 23666) by debbugs.gnu.org; 6 Jun 2016 07:06:20 +0000 Received: from localhost ([127.0.0.1]:56155 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1b9oc8-00022A-FM for submit <at> debbugs.gnu.org; Mon, 06 Jun 2016 03:06:20 -0400 Received: from eggs.gnu.org ([208.118.235.92]:41148) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <ludo@HIDDEN>) id 1b9oc7-00021x-4E for 23666 <at> debbugs.gnu.org; Mon, 06 Jun 2016 03:06:19 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from <ludo@HIDDEN>) id 1b9oby-0006Zk-W5 for 23666 <at> debbugs.gnu.org; Mon, 06 Jun 2016 03:06:14 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_05,RP_MATCHES_RCVD autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:45263) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <ludo@HIDDEN>) id 1b9oby-0006ZU-SI; Mon, 06 Jun 2016 03:06:10 -0400 Received: from pluto.bordeaux.inria.fr ([193.50.110.57]:45908 helo=pluto) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_128_CBC_SHA1:128) (Exim 4.82) (envelope-from <ludo@HIDDEN>) id 1b9obx-0007yy-6I; Mon, 06 Jun 2016 03:06:09 -0400 From: ludo@HIDDEN (Ludovic =?UTF-8?Q?Court=C3=A8s?=) References: <20160531195631.GA22897@solar> <878typkrqd.fsf@HIDDEN> <20160605205839.GB20831@solar> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 19 Prairial an 224 de la =?UTF-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-unknown-linux-gnu Date: Mon, 06 Jun 2016 09:06:07 +0200 In-Reply-To: <20160605205839.GB20831@solar> (Andreas Enge's message of "Sun, 5 Jun 2016 22:58:39 +0200") Message-ID: <87k2i27q5c.fsf@HIDDEN> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-Spam-Score: -6.4 (------) X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -6.4 (------) Andreas Enge <andreas@HIDDEN> skribis: > On Wed, Jun 01, 2016 at 02:39:54PM +0200, Ludovic Court=C3=A8s wrote: >> This is implemented using the =E2=80=98add-to-store=E2=80=99 RPC, which,= after all these >> years, is still implemented like this (nix/libstore/local-store.cc): >>=20 >> --8<---------------cut here---------------start------------->8--- >> Path LocalStore::addToStore(const string & name, const Path & _srcPath, >> bool recursive, HashType hashAlgo, PathFilter & filter, bool repair) >> { >> Path srcPath(absPath(_srcPath)); >> debug(format("adding `%1%' to the store") % srcPath); >>=20 >> /* Read the whole path into memory. This is not a very scalable >> method for very large paths, but `copyPath' is mainly used for >> small files. */ >> --8<---------------cut here---------------end--------------->8--- > > Something that mainly does not fail could indeed be seen as a bug... > But how come that "guix download http://" succeeds, where > "guix download file://" fails? Both use the =E2=80=98add-to-store=E2=80=99 RPC, so both have equal chances= of failing. Note that, fortunately, fixed-output derivations (=E2=80=98guix build -S fo= o=E2=80=99) do not use =E2=80=98add-to-store=E2=80=99. In general, except in the case = of =E2=80=98guix download=E2=80=99, =E2=80=98add-to-store=E2=80=99 is only used with small f= iles like when adding .scm files to the store, etc. Ludo=E2=80=99.
Received: (at control) by debbugs.gnu.org; 24 Jul 2016 14:03:28 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Sun Jul 24 10:03:27 2016 Received: from localhost ([127.0.0.1]:35927 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1bRK07-0003ZP-OB for submit <at> debbugs.gnu.org; Sun, 24 Jul 2016 10:03:27 -0400 Received: from eggs.gnu.org ([208.118.235.92]:34410) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <ludo@HIDDEN>) id 1bRK06-0003ZC-6e for control <at> debbugs.gnu.org; Sun, 24 Jul 2016 10:03:26 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from <ludo@HIDDEN>) id 1bRK00-0004mW-D3 for control <at> debbugs.gnu.org; Sun, 24 Jul 2016 10:03:21 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-3.2 required=5.0 tests=BAYES_00,RP_MATCHES_RCVD autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:36841) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <ludo@HIDDEN>) id 1bRK00-0004mG-AA for control <at> debbugs.gnu.org; Sun, 24 Jul 2016 10:03:20 -0400 Received: from reverse-83.fdn.fr ([80.67.176.83]:43776 helo=pluto) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_128_CBC_SHA1:128) (Exim 4.82) (envelope-from <ludo@HIDDEN>) id 1bRJzy-0007pb-Gp for control <at> debbugs.gnu.org; Sun, 24 Jul 2016 10:03:19 -0400 Date: Sun, 24 Jul 2016 16:03:16 +0200 Message-Id: <87oa5ncevv.fsf@HIDDEN> To: control <at> debbugs.gnu.org From: ludo@HIDDEN (Ludovic =?utf-8?Q?Court=C3=A8s?=) Subject: control message for bug #24053 MIME-version: 1.0 Content-type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-Spam-Score: -6.3 (------) X-Debbugs-Envelope-To: control X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -6.3 (------) merge 24053 23666
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997 nCipher Corporation Ltd,
1994-97 Ian Jackson.