GNU logs - #23666, boring messages


Message sent to bug-guix@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: bug#23666: guix download fails for large files
Resent-From: Andreas Enge <andreas@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Tue, 31 May 2016 19:57:02 +0000
Resent-Message-ID: <handler.23666.B.146472461217122 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: report 23666
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: 23666 <at> debbugs.gnu.org
X-Debbugs-Original-To: bug-guix@HIDDEN
Received: via spool by submit <at> debbugs.gnu.org id=B.146472461217122
          (code B ref -1); Tue, 31 May 2016 19:57:02 +0000
Received: (at submit) by debbugs.gnu.org; 31 May 2016 19:56:52 +0000
Received: from localhost ([127.0.0.1]:48675 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1b7pmV-0004S5-Si
	for submit <at> debbugs.gnu.org; Tue, 31 May 2016 15:56:52 -0400
Received: from eggs.gnu.org ([208.118.235.92]:45727)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <andreas@HIDDEN>) id 1b7pmU-0004Rt-CT
 for submit <at> debbugs.gnu.org; Tue, 31 May 2016 15:56:50 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <andreas@HIDDEN>) id 1b7pmO-0003T5-DZ
 for submit <at> debbugs.gnu.org; Tue, 31 May 2016 15:56:45 -0400
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50,T_DKIM_INVALID
 autolearn=disabled version=3.3.2
Received: from lists.gnu.org ([2001:4830:134:3::11]:50378)
 by eggs.gnu.org with esmtp (Exim 4.71)
 (envelope-from <andreas@HIDDEN>) id 1b7pmO-0003Ss-AM
 for submit <at> debbugs.gnu.org; Tue, 31 May 2016 15:56:44 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:36685)
 by lists.gnu.org with esmtp (Exim 4.71)
 (envelope-from <andreas@HIDDEN>) id 1b7pmM-00037g-4y
 for bug-guix@HIDDEN; Tue, 31 May 2016 15:56:43 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <andreas@HIDDEN>) id 1b7pmH-0003Rg-PI
 for bug-guix@HIDDEN; Tue, 31 May 2016 15:56:41 -0400
Received: from mailrelay2.public.one.com ([91.198.169.125]:44634)
 by eggs.gnu.org with esmtp (Exim 4.71)
 (envelope-from <andreas@HIDDEN>) id 1b7pmH-0003R5-Bz
 for bug-guix@HIDDEN; Tue, 31 May 2016 15:56:37 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=enge.fr; s=20140924;
 h=from:subject:date:message-id:to:mime-version:content-type;
 bh=kBj5xaM7LmW8CzWvEwNV+gW0dYczbOtrQZVi7jjxVeg=;
 b=qmbIZyYtZI4U6TdaBsKC9i3cPCx6CTlH3O5zHsZK3koOnsz/VHO23oGmKZqiIhlU/G5xehIaszwLt
 Jyc12Tjfn0os/GyDwFEeVVIDkG5Xa0chJihwtT08S7+G5FXIcOP8ez/MkZH/ZMNF8bQDMLaLZ/3aYN
 H857ZgjDuGCtqVhI=
X-HalOne-Cookie: af39bd020affca61a32a26ee296b8dfa93ba0fea
X-HalOne-ID: c69c5c69-2769-11e6-8278-b82a72d03b9b
Received: from solar (unknown [92.89.73.211])
 by smtpfilter2.public.one.com (Halon Mail Gateway) with ESMTPSA;
 Tue, 31 May 2016 19:56:33 +0000 (UTC)
Date: Tue, 31 May 2016 21:56:31 +0200
From: Andreas Enge <andreas@HIDDEN>
Message-ID: <20160531195631.GA22897@solar>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.6.0 (2016-04-01)
X-detected-operating-system: by eggs.gnu.org: FreeBSD 9.x
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x
X-Received-From: 2001:4830:134:3::11
X-Spam-Score: -5.0 (-----)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -5.0 (-----)

Hello,

since I am regularly rebuilding texlive (who is not!), I would like to keep
the source tarball on my machine, and whenever I garbage collect it from the
store, add it back via
   guix download file:texlive-20150523-texmf.tar.xz

On my Novena with a 32 bit processor and 4 GB of main memory, this fails with
   guix download: error: sendfile: Broken pipe

The file itself has 1,8 GB.

Downloading the file via http as part of "guix build" works, and
"guix download file:" also works on my x86_64 machine with 8 GB of RAM.

Andreas





Message sent:


Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Mailer: MIME-tools 5.505 (Entity 5.505)
Content-Type: text/plain; charset=utf-8
X-Loop: help-debbugs@HIDDEN
From: help-debbugs@HIDDEN (GNU bug Tracking System)
To: Andreas Enge <andreas@HIDDEN>
Subject: bug#23666: Acknowledgement (guix download fails for large files)
Message-ID: <handler.23666.B.146472461217122.ack <at> debbugs.gnu.org>
References: <20160531195631.GA22897@solar>
X-Gnu-PR-Message: ack 23666
X-Gnu-PR-Package: guix
Reply-To: 23666 <at> debbugs.gnu.org
Date: Tue, 31 May 2016 19:57:02 +0000

Thank you for filing a new bug report with debbugs.gnu.org.

This is an automatically generated reply to let you know your message
has been received.

Your message is being forwarded to the package maintainers and other
interested parties for their attention; they will reply in due course.

Your message has been sent to the package maintainer(s):
 bug-guix@HIDDEN

If you wish to submit further information on this problem, please
send it to 23666 <at> debbugs.gnu.org.

Please do not send mail to help-debbugs@HIDDEN unless you wish
to report a problem with the Bug-tracking system.

--=20
23666: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D23666
GNU Bug Tracking System
Contact help-debbugs@HIDDEN with problems


Message sent to bug-guix@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: bug#23666: guix download fails for large files
Resent-From: Leo Famulari <leo@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Tue, 31 May 2016 20:27:02 +0000
Resent-Message-ID: <handler.23666.B23666.146472641926193 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 23666
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: Andreas Enge <andreas@HIDDEN>
Cc: 23666 <at> debbugs.gnu.org
Received: via spool by 23666-submit <at> debbugs.gnu.org id=B23666.146472641926193
          (code B ref 23666); Tue, 31 May 2016 20:27:02 +0000
Received: (at 23666) by debbugs.gnu.org; 31 May 2016 20:26:59 +0000
Received: from localhost ([127.0.0.1]:48719 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1b7qFe-0006oP-MV
	for submit <at> debbugs.gnu.org; Tue, 31 May 2016 16:26:58 -0400
Received: from out2-smtp.messagingengine.com ([66.111.4.26]:46228)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <leo@HIDDEN>) id 1b7qFc-0006oH-DW
 for 23666 <at> debbugs.gnu.org; Tue, 31 May 2016 16:26:56 -0400
Received: from compute6.internal (compute6.nyi.internal [10.202.2.46])
 by mailout.nyi.internal (Postfix) with ESMTP id A266C20B35;
 Tue, 31 May 2016 16:26:55 -0400 (EDT)
Received: from frontend1 ([10.202.2.160])
 by compute6.internal (MEProxy); Tue, 31 May 2016 16:26:55 -0400
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=famulari.name; h=
 cc:content-type:date:from:in-reply-to:message-id:mime-version
 :references:subject:to:x-sasl-enc:x-sasl-enc; s=mesmtp; bh=ci7ZH
 kxoAqJcZgbCXEPfzAzbadw=; b=wBvlh2Dy/5+9fcPBbj2ahrFoQrol/AxImXINo
 KdgW/ts/nBriQQzCBHdx/a+5sudyCzsO44mMLRoTjMgk+PKdrbsw8jxl4t4PF1iU
 dwAX9pEvTlfTeV1OLG6B/50iDwFopS9SliEApBGEciR5yDz6gaB1VYtIoIvzo2Ev
 kLMK+g=
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:content-type:date:from:in-reply-to
 :message-id:mime-version:references:subject:to:x-sasl-enc
 :x-sasl-enc; s=smtpout; bh=ci7ZHkxoAqJcZgbCXEPfzAzbadw=; b=uyaVq
 shwc57CjZkEA0q11njZK1iv/7jw9T/vWVQjZZtMjNwQenCVfUBp/FS2lxi3gI1y6
 ZAwNTwJ5Si8ublCnx5WAaY7RwfwKP9sAeiGcfKFmtKHpl62vCJyx0FiWeg7riSww
 V0FM3Ttq/WvOFSc/+weG3eSBFw/Nm3NsAd3mbI=
X-Sasl-enc: UYl0CvVMMJrAQfmdwx9sRWz48BcYt0/yDMmVarjj/FJ5 1464726415
Received: from localhost (c-73-188-17-148.hsd1.pa.comcast.net [73.188.17.148])
 by mail.messagingengine.com (Postfix) with ESMTPA id 57301F2A4D;
 Tue, 31 May 2016 16:26:55 -0400 (EDT)
Date: Tue, 31 May 2016 16:26:51 -0400
From: Leo Famulari <leo@HIDDEN>
Message-ID: <20160531202651.GA19070@jasmine>
References: <20160531195631.GA22897@solar>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160531195631.GA22897@solar>
User-Agent: Mutt/1.6.0 (2016-04-01)
X-Spam-Score: -0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.7 (/)

On Tue, May 31, 2016 at 09:56:31PM +0200, Andreas Enge wrote:
> since I am regularly rebuilding texlive (who is not!), I would like to keep
> the source tarball on my machine, and whenever I garbage collect it from the
> store, add it back via
>    guix download file:texlive-20150523-texmf.tar.xz
>
> On my Novena with a 32 bit processor and 4 GB of main memory, this fails with
>    guix download: error: sendfile: Broken pipe
> 
> The file itself has 1,8 GB.
> 
> Downloading the file via http as part of "guix build" works, and
> "guix download file:" also works on my x86_64 machine with 8 GB of RAM.

As a temporary workaround, you could make the tarball a gcroot:

# ln -sf /gnu/store/...-texlive-20150523-texmf.tar.xz \
/var/guix/gcroots/texlive-texmf.tar.xz




Message sent to bug-guix@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: bug#23666: guix download fails for large files
Resent-From: ludo@HIDDEN (Ludovic =?UTF-8?Q?Court=C3=A8s?=)
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Wed, 01 Jun 2016 12:41:01 +0000
Resent-Message-ID: <handler.23666.B23666.14647848094206 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 23666
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: Andreas Enge <andreas@HIDDEN>
Cc: 23666 <at> debbugs.gnu.org
Received: via spool by 23666-submit <at> debbugs.gnu.org id=B23666.14647848094206
          (code B ref 23666); Wed, 01 Jun 2016 12:41:01 +0000
Received: (at 23666) by debbugs.gnu.org; 1 Jun 2016 12:40:09 +0000
Received: from localhost ([127.0.0.1]:49171 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1b85RQ-00015m-T3
	for submit <at> debbugs.gnu.org; Wed, 01 Jun 2016 08:40:09 -0400
Received: from eggs.gnu.org ([208.118.235.92]:58685)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1b85RP-00015a-8f
 for 23666 <at> debbugs.gnu.org; Wed, 01 Jun 2016 08:40:07 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <ludo@HIDDEN>) id 1b85RG-0003Mw-S9
 for 23666 <at> debbugs.gnu.org; Wed, 01 Jun 2016 08:40:02 -0400
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=-3.3 required=5.0 tests=BAYES_00,RP_MATCHES_RCVD
 autolearn=disabled version=3.3.2
Received: from fencepost.gnu.org ([2001:4830:134:3::e]:41997)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <ludo@HIDDEN>)
 id 1b85RG-0003Mg-Pc; Wed, 01 Jun 2016 08:39:58 -0400
Received: from reverse-83.fdn.fr ([80.67.176.83]:39376 helo=pluto)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_128_CBC_SHA1:128)
 (Exim 4.82) (envelope-from <ludo@HIDDEN>)
 id 1b85RF-0002o3-0i; Wed, 01 Jun 2016 08:39:57 -0400
From: ludo@HIDDEN (Ludovic =?UTF-8?Q?Court=C3=A8s?=)
References: <20160531195631.GA22897@solar>
X-URL: http://www.fdn.fr/~lcourtes/
X-Revolutionary-Date: 14 Prairial an 224 de la =?UTF-8?Q?R=C3=A9volution?=
X-PGP-Key-ID: 0x090B11993D9AEBB5
X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc
X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4  0CFB 090B 1199 3D9A EBB5
X-OS: x86_64-unknown-linux-gnu
Date: Wed, 01 Jun 2016 14:39:54 +0200
In-Reply-To: <20160531195631.GA22897@solar> (Andreas Enge's message of "Tue,
 31 May 2016 21:56:31 +0200")
Message-ID: <878typkrqd.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2001:4830:134:3::e
X-Spam-Score: -6.4 (------)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -6.4 (------)

Andreas Enge <andreas@HIDDEN> skribis:

> On my Novena with a 32 bit processor and 4 GB of main memory, this fails =
with
>    guix download: error: sendfile: Broken pipe
>
> The file itself has 1,8 GB.

This is implemented using the =E2=80=98add-to-store=E2=80=99 RPC, which, af=
ter all these
years, is still implemented like this (nix/libstore/local-store.cc):

--8<---------------cut here---------------start------------->8---
Path LocalStore::addToStore(const string & name, const Path & _srcPath,
    bool recursive, HashType hashAlgo, PathFilter & filter, bool repair)
{
    Path srcPath(absPath(_srcPath));
    debug(format("adding `%1%' to the store") % srcPath);

    /* Read the whole path into memory. This is not a very scalable
       method for very large paths, but `copyPath' is mainly used for
       small files. */
--8<---------------cut here---------------end--------------->8---

This is undoubtedly what=E2=80=99s causing the failure.

Until this is fixed, you could wrap the file in a nar and import the nar
(nar import does not have this limitation, see =E2=80=99LocalStore::importP=
ath=E2=80=99
and co.).

To do that, you could do:

  guix archive --export $(guix build -S texlive-texmf-whatever) > t.nar

and later:

  guix archive --import < t.nar

HTH,
Ludo=E2=80=99.




Message received at request <at> debbugs.gnu.org:


Received: (at request) by debbugs.gnu.org; 1 Jun 2016 12:41:11 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Wed Jun 01 08:41:11 2016
Received: from localhost ([127.0.0.1]:49175 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1b85SR-00017V-7U
	for submit <at> debbugs.gnu.org; Wed, 01 Jun 2016 08:41:11 -0400
Received: from eggs.gnu.org ([208.118.235.92]:58969)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1b85SP-00017J-JL
 for request <at> debbugs.gnu.org; Wed, 01 Jun 2016 08:41:09 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <ludo@HIDDEN>) id 1b85SF-0003aS-Mh
 for request <at> debbugs.gnu.org; Wed, 01 Jun 2016 08:41:04 -0400
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=-3.3 required=5.0 tests=BAYES_00,RP_MATCHES_RCVD
 autolearn=disabled version=3.3.2
Received: from fencepost.gnu.org ([2001:4830:134:3::e]:42008)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <ludo@HIDDEN>)
 id 1b85SF-0003a0-J5
 for request <at> debbugs.gnu.org; Wed, 01 Jun 2016 08:40:59 -0400
Received: from reverse-83.fdn.fr ([80.67.176.83]:39378 helo=pluto)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_128_CBC_SHA1:128)
 (Exim 4.82) (envelope-from <ludo@HIDDEN>) id 1b85SD-0006Ca-Ny
 for request <at> debbugs.gnu.org; Wed, 01 Jun 2016 08:40:58 -0400
From: ludo@HIDDEN (Ludovic =?utf-8?Q?Court=C3=A8s?=)
To: request <at> debbugs.gnu.org
Subject: retitle
X-URL: http://www.fdn.fr/~lcourtes/
X-Revolutionary-Date: 14 Prairial an 224 de la =?utf-8?Q?R=C3=A9volution?=
X-PGP-Key-ID: 0x090B11993D9AEBB5
X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc
X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4  0CFB 090B 1199 3D9A EBB5
X-OS: x86_64-unknown-linux-gnu
Date: Wed, 01 Jun 2016 14:40:55 +0200
Message-ID: <874m9dkroo.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2001:4830:134:3::e
X-Spam-Score: -6.4 (------)
X-Debbugs-Envelope-To: request
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -6.4 (------)

retitle 23666 'add-to-store' RPC loads files in memory (daemon-side)
thanks




Message sent to bug-guix@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: bug#23666: guix download fails for large files
Resent-From: Andreas Enge <andreas@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Sun, 05 Jun 2016 20:59:02 +0000
Resent-Message-ID: <handler.23666.B23666.146516033030696 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 23666
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Cc: 23666 <at> debbugs.gnu.org
Received: via spool by 23666-submit <at> debbugs.gnu.org id=B23666.146516033030696
          (code B ref 23666); Sun, 05 Jun 2016 20:59:02 +0000
Received: (at 23666) by debbugs.gnu.org; 5 Jun 2016 20:58:50 +0000
Received: from localhost ([127.0.0.1]:55929 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1b9f8E-0007z2-8L
	for submit <at> debbugs.gnu.org; Sun, 05 Jun 2016 16:58:50 -0400
Received: from mailrelay7.public.one.com ([91.198.169.215]:59680)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <andreas@HIDDEN>) id 1b9f8C-0007yo-WB
 for 23666 <at> debbugs.gnu.org; Sun, 05 Jun 2016 16:58:49 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=enge.fr; s=20140924;
 h=from:subject:date:message-id:to:cc:mime-version:content-type:
 content-transfer-encoding:in-reply-to:references;
 bh=4wcLuDq4Cuys5+EXVjAy4+p92GPUvvZ2E4MgjEgpMy0=;
 b=nCvhLO8OIo9fS7+vGLEcguOh7ECQKZL0Z15BjTk+VDFKpOD7/kzwYaUfACJD+2G9GXm+ynIbjk/Wv
 j2BwJLJ9gPe1EWnHrHoJxqBEz7dNs/omkass+LW5x1Gar/XCLb4Ra0VDnf9yv6rMkjtzboVPssjUAy
 SCmKK80yPM3PTGFU=
X-HalOne-Cookie: 3db6237ac8803e7fa0dd8ad0f131e55fc05d0f3c
X-HalOne-ID: 488d750b-2b60-11e6-a0c6-b82a72cffc46
Received: from solar (unknown [92.89.73.211])
 by smtpfilter4.public.one.com (Halon Mail Gateway) with ESMTPSA;
 Sun,  5 Jun 2016 20:58:41 +0000 (UTC)
Date: Sun, 5 Jun 2016 22:58:39 +0200
From: Andreas Enge <andreas@HIDDEN>
Message-ID: <20160605205839.GB20831@solar>
References: <20160531195631.GA22897@solar>
 <878typkrqd.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <878typkrqd.fsf@HIDDEN>
User-Agent: Mutt/1.6.0 (2016-04-01)
X-Spam-Score: 0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: 0.0 (/)

Thanks for your suggestions, Leo and Ludovic! I still see this as a bug;
should I report it upstream to Nix?

On Wed, Jun 01, 2016 at 02:39:54PM +0200, Ludovic Courtès wrote:
> This is implemented using the ‘add-to-store’ RPC, which, after all these
> years, is still implemented like this (nix/libstore/local-store.cc):
> 
> --8<---------------cut here---------------start------------->8---
> Path LocalStore::addToStore(const string & name, const Path & _srcPath,
>     bool recursive, HashType hashAlgo, PathFilter & filter, bool repair)
> {
>     Path srcPath(absPath(_srcPath));
>     debug(format("adding `%1%' to the store") % srcPath);
> 
>     /* Read the whole path into memory. This is not a very scalable
>        method for very large paths, but `copyPath' is mainly used for
>        small files. */
> --8<---------------cut here---------------end--------------->8---

Something that mainly does not fail could indeed be seen as a bug...
But how come that "guix download http://" succeeds, where
"guix download file://" fails?

Andreas





Message sent to bug-guix@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: bug#23666: guix download fails for large files
Resent-From: ludo@HIDDEN (Ludovic =?UTF-8?Q?Court=C3=A8s?=)
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Mon, 06 Jun 2016 07:07:01 +0000
Resent-Message-ID: <handler.23666.B23666.14651967807826 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 23666
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: Andreas Enge <andreas@HIDDEN>
Cc: 23666 <at> debbugs.gnu.org
Received: via spool by 23666-submit <at> debbugs.gnu.org id=B23666.14651967807826
          (code B ref 23666); Mon, 06 Jun 2016 07:07:01 +0000
Received: (at 23666) by debbugs.gnu.org; 6 Jun 2016 07:06:20 +0000
Received: from localhost ([127.0.0.1]:56155 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1b9oc8-00022A-FM
	for submit <at> debbugs.gnu.org; Mon, 06 Jun 2016 03:06:20 -0400
Received: from eggs.gnu.org ([208.118.235.92]:41148)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1b9oc7-00021x-4E
 for 23666 <at> debbugs.gnu.org; Mon, 06 Jun 2016 03:06:19 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <ludo@HIDDEN>) id 1b9oby-0006Zk-W5
 for 23666 <at> debbugs.gnu.org; Mon, 06 Jun 2016 03:06:14 -0400
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_05,RP_MATCHES_RCVD
 autolearn=disabled version=3.3.2
Received: from fencepost.gnu.org ([2001:4830:134:3::e]:45263)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <ludo@HIDDEN>)
 id 1b9oby-0006ZU-SI; Mon, 06 Jun 2016 03:06:10 -0400
Received: from pluto.bordeaux.inria.fr ([193.50.110.57]:45908 helo=pluto)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_128_CBC_SHA1:128)
 (Exim 4.82) (envelope-from <ludo@HIDDEN>)
 id 1b9obx-0007yy-6I; Mon, 06 Jun 2016 03:06:09 -0400
From: ludo@HIDDEN (Ludovic =?UTF-8?Q?Court=C3=A8s?=)
References: <20160531195631.GA22897@solar> <878typkrqd.fsf@HIDDEN>
 <20160605205839.GB20831@solar>
X-URL: http://www.fdn.fr/~lcourtes/
X-Revolutionary-Date: 19 Prairial an 224 de la =?UTF-8?Q?R=C3=A9volution?=
X-PGP-Key-ID: 0x090B11993D9AEBB5
X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc
X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4  0CFB 090B 1199 3D9A EBB5
X-OS: x86_64-unknown-linux-gnu
Date: Mon, 06 Jun 2016 09:06:07 +0200
In-Reply-To: <20160605205839.GB20831@solar> (Andreas Enge's message of "Sun, 5
 Jun 2016 22:58:39 +0200")
Message-ID: <87k2i27q5c.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2001:4830:134:3::e
X-Spam-Score: -6.4 (------)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -6.4 (------)

Andreas Enge <andreas@HIDDEN> skribis:

> On Wed, Jun 01, 2016 at 02:39:54PM +0200, Ludovic Court=C3=A8s wrote:
>> This is implemented using the =E2=80=98add-to-store=E2=80=99 RPC, which,=
 after all these
>> years, is still implemented like this (nix/libstore/local-store.cc):
>>=20
>> --8<---------------cut here---------------start------------->8---
>> Path LocalStore::addToStore(const string & name, const Path & _srcPath,
>>     bool recursive, HashType hashAlgo, PathFilter & filter, bool repair)
>> {
>>     Path srcPath(absPath(_srcPath));
>>     debug(format("adding `%1%' to the store") % srcPath);
>>=20
>>     /* Read the whole path into memory. This is not a very scalable
>>        method for very large paths, but `copyPath' is mainly used for
>>        small files. */
>> --8<---------------cut here---------------end--------------->8---
>
> Something that mainly does not fail could indeed be seen as a bug...
> But how come that "guix download http://" succeeds, where
> "guix download file://" fails?

Both use the =E2=80=98add-to-store=E2=80=99 RPC, so both have equal chances=
 of failing.

Note that, fortunately, fixed-output derivations (=E2=80=98guix build -S fo=
o=E2=80=99)
do not use =E2=80=98add-to-store=E2=80=99.  In general, except in the case =
of =E2=80=98guix
download=E2=80=99, =E2=80=98add-to-store=E2=80=99 is only used with small f=
iles like when adding
.scm files to the store, etc.

Ludo=E2=80=99.




Message received at control <at> debbugs.gnu.org:


Received: (at control) by debbugs.gnu.org; 24 Jul 2016 14:03:28 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Jul 24 10:03:27 2016
Received: from localhost ([127.0.0.1]:35927 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1bRK07-0003ZP-OB
	for submit <at> debbugs.gnu.org; Sun, 24 Jul 2016 10:03:27 -0400
Received: from eggs.gnu.org ([208.118.235.92]:34410)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1bRK06-0003ZC-6e
 for control <at> debbugs.gnu.org; Sun, 24 Jul 2016 10:03:26 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <ludo@HIDDEN>) id 1bRK00-0004mW-D3
 for control <at> debbugs.gnu.org; Sun, 24 Jul 2016 10:03:21 -0400
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=-3.2 required=5.0 tests=BAYES_00,RP_MATCHES_RCVD
 autolearn=disabled version=3.3.2
Received: from fencepost.gnu.org ([2001:4830:134:3::e]:36841)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <ludo@HIDDEN>)
 id 1bRK00-0004mG-AA
 for control <at> debbugs.gnu.org; Sun, 24 Jul 2016 10:03:20 -0400
Received: from reverse-83.fdn.fr ([80.67.176.83]:43776 helo=pluto)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_128_CBC_SHA1:128)
 (Exim 4.82) (envelope-from <ludo@HIDDEN>) id 1bRJzy-0007pb-Gp
 for control <at> debbugs.gnu.org; Sun, 24 Jul 2016 10:03:19 -0400
Date: Sun, 24 Jul 2016 16:03:16 +0200
Message-Id: <87oa5ncevv.fsf@HIDDEN>
To: control <at> debbugs.gnu.org
From: ludo@HIDDEN (Ludovic =?utf-8?Q?Court=C3=A8s?=)
Subject: control message for bug #24053
MIME-version: 1.0
Content-type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2001:4830:134:3::e
X-Spam-Score: -6.3 (------)
X-Debbugs-Envelope-To: control
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -6.3 (------)

merge 24053 23666





Last modified: Mon, 25 Nov 2019 12:00:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.