GNU bug report logs - #21093
Web server: response bodies systematically loaded in memory

Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.
Package: guile; Reported by: ludo@HIDDEN (Ludovic Courtès); dated Sun, 19 Jul 2015 22:11:01 UTC; Maintainer for guile is bug-guile@HIDDEN.

Message received at 21093 <at> debbugs.gnu.org:


Received: (at 21093) by debbugs.gnu.org; 24 Jun 2016 08:58:41 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Jun 24 04:58:41 2016
Received: from localhost ([127.0.0.1]:53404 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1bGMwj-0002iG-2G
	for submit <at> debbugs.gnu.org; Fri, 24 Jun 2016 04:58:41 -0400
Received: from eggs.gnu.org ([208.118.235.92]:56386)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1bGMwh-0002i2-PO
 for 21093 <at> debbugs.gnu.org; Fri, 24 Jun 2016 04:58:40 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <ludo@HIDDEN>) id 1bGMwb-0005M2-PL
 for 21093 <at> debbugs.gnu.org; Fri, 24 Jun 2016 04:58:34 -0400
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=-0.6 required=5.0 tests=BAYES_50,RP_MATCHES_RCVD
 autolearn=disabled version=3.3.2
Received: from fencepost.gnu.org ([2001:4830:134:3::e]:55297)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <ludo@HIDDEN>)
 id 1bGMwS-0005I5-Sa; Fri, 24 Jun 2016 04:58:24 -0400
Received: from pluto.bordeaux.inria.fr ([193.50.110.57]:47830 helo=pluto)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_128_CBC_SHA1:128)
 (Exim 4.82) (envelope-from <ludo@HIDDEN>)
 id 1bGMwS-0005Mq-30; Fri, 24 Jun 2016 04:58:24 -0400
From: ludo@HIDDEN (Ludovic =?utf-8?Q?Court=C3=A8s?=)
To: Andy Wingo <wingo@HIDDEN>
Subject: Re: bug#21093: Web server: response bodies systematically loaded in
 memory
References: <87a8uriysr.fsf@HIDDEN> <874m8jhuj7.fsf@HIDDEN>
X-URL: http://www.fdn.fr/~lcourtes/
X-Revolutionary-Date: 7 Messidor an 224 de la =?utf-8?Q?R=C3=A9volution?=
X-PGP-Key-ID: 0x090B11993D9AEBB5
X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc
X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4  0CFB 090B 1199 3D9A EBB5
X-OS: x86_64-unknown-linux-gnu
Date: Fri, 24 Jun 2016 10:58:21 +0200
In-Reply-To: <874m8jhuj7.fsf@HIDDEN> (Andy Wingo's message of "Fri, 24 Jun
 2016 10:15:40 +0200")
Message-ID: <87r3bn7yky.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2001:4830:134:3::e
X-Spam-Score: -6.4 (------)
X-Debbugs-Envelope-To: 21093
Cc: guile-devel@HIDDEN, 21093 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -6.4 (------)

Andy Wingo <wingo@HIDDEN> skribis:

> On Mon 20 Jul 2015 00:10, ludo@HIDDEN (Ludovic Court=C3=A8s) writes:
>
>> The =E2=80=98sanitize-response=E2=80=99 procedure systematically loads t=
he whole
>> response body in memory, which causes obvious scalability issues (this
>> is in 2.0.11.)
>>
>> In particular, when a request handler returns a port-taking procedure as
>> its second return value, =E2=80=98sanitize-request=E2=80=99 will just ca=
ll that
>> procedure passing it a string output port.
>>
>> This procedure should instead be called from the server implementation=
=E2=80=99s
>> =E2=80=98write=E2=80=99 hook, but that would necessitate an API change.
>>
>> Thoughts?
>
> How would you set the Content-Length?  Just leave it off and set
> Connection: close ?  Set it in the headers perhaps?  Then you have to
> verify later, which I dunno how nice that is.  Maybe it is OK.

I think it could work this way:

  1. By default, provide no =E2=80=98Content-Length=E2=80=99 and force chun=
ked encoding
     (so that the recipient can make sure it received everything.)

     This is useful for data generated on the fly.

  2. Provide an optional mechanism allowing the user to specify the
     content length upfront.

     Useful for statically-generated data that cannot fit in memory.

I haven=E2=80=99t thought yet about the actual API (I=E2=80=99ll be happy t=
o do so as
time permits; to be clear, I don=E2=80=99t consider it a 2.0.12 blocker.)

Ludo=E2=80=99.
Information forwarded to bug-guile@HIDDEN:
bug#21093; Package guile. Full text available.

Message received at 21093 <at> debbugs.gnu.org:


Received: (at 21093) by debbugs.gnu.org; 24 Jun 2016 08:15:51 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Jun 24 04:15:51 2016
Received: from localhost ([127.0.0.1]:53328 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1bGMHH-0001XL-DF
	for submit <at> debbugs.gnu.org; Fri, 24 Jun 2016 04:15:51 -0400
Received: from pb-sasl1.pobox.com ([64.147.108.66]:59059
 helo=sasl.smtp.pobox.com) by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <wingo@HIDDEN>) id 1bGMHG-0001XE-3c
 for 21093 <at> debbugs.gnu.org; Fri, 24 Jun 2016 04:15:50 -0400
Received: from sasl.smtp.pobox.com (unknown [127.0.0.1])
 by pb-sasl1.pobox.com (Postfix) with ESMTP id ED05C18DBB;
 Fri, 24 Jun 2016 04:15:47 -0400 (EDT)
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=pobox.com; h=from:to:cc
 :subject:references:date:in-reply-to:message-id:mime-version
 :content-type:content-transfer-encoding; s=sasl; bh=OhRI0wjEP7Uh
 fqQcVkPNJz/9Ymw=; b=sXVnMQUeSSMyuG5Hrd35RsYppxe2IXBFFlyVBmMALi/3
 2a9zIluPm4o8mwKKkGiX25WiXEQDjtZ0RFMwUQfIXeYXNlCNY+2pM914vygmBPU4
 ZbjxlElVggEQA5YewGp7ewnNA4TsLfrA0iRrC0bwP2U+Ngb97FOx1fCL1L08J4o=
DomainKey-Signature: a=rsa-sha1; c=nofws; d=pobox.com; h=from:to:cc
 :subject:references:date:in-reply-to:message-id:mime-version
 :content-type:content-transfer-encoding; q=dns; s=sasl; b=uKIVGt
 Dq0mwrsw1k+AlGBjL/m195lQQl53DwKybyQsWUKYy1O27l3w4cJPQFONA6WIZiZa
 AvySmFNlchQsk1XsDjQ/Y6ocwolfa0oi0gPpUWcAuz6Ua3CwJXFRdTFXRhZp6OJx
 jQgfzDSDPRyUmbJc2zXtF9OMNRlmgKQS3gwQE=
Received: from pb-sasl1.nyi.icgroup.com (unknown [127.0.0.1])
 by pb-sasl1.pobox.com (Postfix) with ESMTP id D6C8618DB8;
 Fri, 24 Jun 2016 04:15:47 -0400 (EDT)
Received: from clucks (unknown [88.160.190.192])
 (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits))
 (No client certificate requested)
 by pb-sasl1.pobox.com (Postfix) with ESMTPSA id 22F5A18DB7;
 Fri, 24 Jun 2016 04:15:47 -0400 (EDT)
From: Andy Wingo <wingo@HIDDEN>
To: ludo@HIDDEN (Ludovic =?utf-8?Q?Court=C3=A8s?=)
Subject: Re: bug#21093: Web server: response bodies systematically loaded in
 memory
References: <87a8uriysr.fsf@HIDDEN>
Date: Fri, 24 Jun 2016 10:15:40 +0200
In-Reply-To: <87a8uriysr.fsf@HIDDEN> ("Ludovic =?utf-8?Q?Court=C3=A8s=22'?=
 =?utf-8?Q?s?= message of "Mon, 20 Jul 2015 00:10:28 +0200")
Message-ID: <874m8jhuj7.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Pobox-Relay-ID: DACDDC84-39E3-11E6-A882-C1836462E9F6-02397024!pb-sasl1.pobox.com
X-Spam-Score: -1.4 (-)
X-Debbugs-Envelope-To: 21093
Cc: guile-devel@HIDDEN, 21093 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.4 (-)

On Mon 20 Jul 2015 00:10, ludo@HIDDEN (Ludovic Court=C3=A8s) writes:

> The =E2=80=98sanitize-response=E2=80=99 procedure systematically loads th=
e whole
> response body in memory, which causes obvious scalability issues (this
> is in 2.0.11.)
>
> In particular, when a request handler returns a port-taking procedure as
> its second return value, =E2=80=98sanitize-request=E2=80=99 will just cal=
l that
> procedure passing it a string output port.
>
> This procedure should instead be called from the server implementation=E2=
=80=99s
> =E2=80=98write=E2=80=99 hook, but that would necessitate an API change.
>
> Thoughts?

How would you set the Content-Length?  Just leave it off and set
Connection: close ?  Set it in the headers perhaps?  Then you have to
verify later, which I dunno how nice that is.  Maybe it is OK.

I would be happy to review patches :)  Cc guile-devel.

Andy
Information forwarded to bug-guile@HIDDEN:
bug#21093; Package guile. Full text available.

Message received at submit <at> debbugs.gnu.org:


Received: (at submit) by debbugs.gnu.org; 19 Jul 2015 22:10:47 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Jul 19 18:10:47 2015
Received: from localhost ([127.0.0.1]:54090 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.80)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1ZGwnH-0001I9-HQ
	for submit <at> debbugs.gnu.org; Sun, 19 Jul 2015 18:10:47 -0400
Received: from eggs.gnu.org ([208.118.235.92]:50000)
 by debbugs.gnu.org with esmtp (Exim 4.80)
 (envelope-from <ludo@HIDDEN>) id 1ZGwnF-0001Hw-B0
 for submit <at> debbugs.gnu.org; Sun, 19 Jul 2015 18:10:45 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <ludo@HIDDEN>) id 1ZGwn9-00052I-8t
 for submit <at> debbugs.gnu.org; Sun, 19 Jul 2015 18:10:39 -0400
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=-3.1 required=5.0 tests=BAYES_00,RP_MATCHES_RCVD
 autolearn=disabled version=3.3.2
Received: from lists.gnu.org ([2001:4830:134:3::11]:39385)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <ludo@HIDDEN>)
 id 1ZGwn9-00052E-6f
 for submit <at> debbugs.gnu.org; Sun, 19 Jul 2015 18:10:39 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:35268)
 by lists.gnu.org with esmtp (Exim 4.71)
 (envelope-from <ludo@HIDDEN>) id 1ZGwn8-0003yi-11
 for bug-guile@HIDDEN; Sun, 19 Jul 2015 18:10:39 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <ludo@HIDDEN>) id 1ZGwn0-0004xo-VJ
 for bug-guile@HIDDEN; Sun, 19 Jul 2015 18:10:37 -0400
Received: from fencepost.gnu.org ([2001:4830:134:3::e]:55996)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <ludo@HIDDEN>)
 id 1ZGwn0-0004xk-QD
 for bug-guile@HIDDEN; Sun, 19 Jul 2015 18:10:30 -0400
Received: from reverse-83.fdn.fr ([80.67.176.83]:44423 helo=pluto)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_128_CBC_SHA1:128)
 (Exim 4.82) (envelope-from <ludo@HIDDEN>) id 1ZGwn0-0006kj-4r
 for bug-guile@HIDDEN; Sun, 19 Jul 2015 18:10:30 -0400
From: ludo@HIDDEN (Ludovic =?utf-8?Q?Court=C3=A8s?=)
To: bug-guile@HIDDEN
Subject: Web server: response bodies systematically loaded in memory
X-URL: http://www.fdn.fr/~lcourtes/
X-Revolutionary-Date: 2 Thermidor an 223 de la =?utf-8?Q?R=C3=A9volution?=
X-PGP-Key-ID: 0x3D9AEBB5
X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc
X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4  0CFB 090B 1199 3D9A EBB5
X-OS: x86_64-unknown-linux-gnu
Date: Mon, 20 Jul 2015 00:10:28 +0200
Message-ID: <87a8uriysr.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: Error: Malformed IPv6 address
 (bad octet value).
X-detected-operating-system: by eggs.gnu.org: Error: Malformed IPv6 address
 (bad octet value).
X-Received-From: 2001:4830:134:3::11
X-Spam-Score: -6.2 (------)
X-Debbugs-Envelope-To: submit
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -6.2 (------)

The =E2=80=98sanitize-response=E2=80=99 procedure systematically loads the =
whole
response body in memory, which causes obvious scalability issues (this
is in 2.0.11.)

In particular, when a request handler returns a port-taking procedure as
its second return value, =E2=80=98sanitize-request=E2=80=99 will just call =
that
procedure passing it a string output port.

This procedure should instead be called from the server implementation=E2=
=80=99s
=E2=80=98write=E2=80=99 hook, but that would necessitate an API change.

Thoughts?

Ludo=E2=80=99.
Acknowledgement sent to ludo@HIDDEN (Ludovic Courtès):
New bug report received and forwarded. Copy sent to bug-guile@HIDDEN. Full text available.
Report forwarded to bug-guile@HIDDEN:
bug#21093; Package guile. Full text available.
Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.
Last modified: Mon, 25 Nov 2019 12:00:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.