GNU logs - #21093, boring messages

Message sent to bug-guile@HIDDEN:

Subject: bug#21093: Web server: response bodies systematically loaded in memory
Resent-From: ludo@HIDDEN (Ludovic =?UTF-8?Q?Court=C3=A8s?=)
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guile@HIDDEN
Resent-Date: Sun, 19 Jul 2015 22:11:01 +0000
Resent-Message-ID: <handler.21093.B.14373438474973 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: 21093 <at> debbugs.gnu.org
From: ludo@HIDDEN (Ludovic =?UTF-8?Q?Court=C3=A8s?=)
Date: Mon, 20 Jul 2015 00:10:28 +0200
Message-ID: <87a8uriysr.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

The =E2=80=98sanitize-response=E2=80=99 procedure systematically loads the =
whole
response body in memory, which causes obvious scalability issues (this
is in 2.0.11.)

In particular, when a request handler returns a port-taking procedure as
its second return value, =E2=80=98sanitize-request=E2=80=99 will just call =
that
procedure passing it a string output port.

This procedure should instead be called from the server implementation=E2=
=80=99s
=E2=80=98write=E2=80=99 hook, but that would necessitate an API change.

Thoughts?

Ludo=E2=80=99.

Message sent:

Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Mailer: MIME-tools 5.503 (Entity 5.503)
Content-Type: text/plain; charset=utf-8
X-Loop: help-debbugs@HIDDEN
From: help-debbugs@HIDDEN (GNU bug Tracking System)
To: ludo@HIDDEN (Ludovic =?UTF-8?Q?Court=C3=A8s?=)
Subject: bug#21093: Acknowledgement (Web server: response bodies
 systematically loaded in memory)
Message-ID: <handler.21093.B.14373438474973.ack <at> debbugs.gnu.org>
References: <87a8uriysr.fsf@HIDDEN>
X-Gnu-PR-Message: ack 21093
X-Gnu-PR-Package: guile
Reply-To: 21093 <at> debbugs.gnu.org
Date: Sun, 19 Jul 2015 22:11:03 +0000

Thank you for filing a new bug report with debbugs.gnu.org.

This is an automatically generated reply to let you know your message
has been received.

Your message is being forwarded to the package maintainers and other
interested parties for their attention; they will reply in due course.

Your message has been sent to the package maintainer(s):
 bug-guile@HIDDEN

If you wish to submit further information on this problem, please
send it to 21093 <at> debbugs.gnu.org.

Please do not send mail to help-debbugs@HIDDEN unless you wish
to report a problem with the Bug-tracking system.

--=20
21093: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D21093
GNU Bug Tracking System
Contact help-debbugs@HIDDEN with problems

Message sent to bug-guile@HIDDEN:

Subject: bug#21093: Web server: response bodies systematically loaded in memory
Resent-From: Andy Wingo <wingo@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guile@HIDDEN
Resent-Date: Fri, 24 Jun 2016 08:16:02 +0000
Resent-Message-ID: <handler.21093.B21093.14667561515915 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: ludo@HIDDEN (Ludovic =?UTF-8?Q?Court=C3=A8s?=)
Cc: guile-devel@HIDDEN, 21093 <at> debbugs.gnu.org
DomainKey-Signature: a=rsa-sha1; c=nofws; d=pobox.com; h=from:to:cc
 :subject:references:date:in-reply-to:message-id:mime-version
 :content-type:content-transfer-encoding; q=dns; s=sasl; b=uKIVGt
 Dq0mwrsw1k+AlGBjL/m195lQQl53DwKybyQsWUKYy1O27l3w4cJPQFONA6WIZiZa
 AvySmFNlchQsk1XsDjQ/Y6ocwolfa0oi0gPpUWcAuz6Ua3CwJXFRdTFXRhZp6OJx
 jQgfzDSDPRyUmbJc2zXtF9OMNRlmgKQS3gwQE=
From: Andy Wingo <wingo@HIDDEN>
References: <87a8uriysr.fsf@HIDDEN>
Date: Fri, 24 Jun 2016 10:15:40 +0200
In-Reply-To: <87a8uriysr.fsf@HIDDEN> ("Ludovic =?UTF-8?Q?Court=C3=A8s?="'s message of "Mon, 20 Jul 2015 00:10:28 +0200")
Message-ID: <874m8jhuj7.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

On Mon 20 Jul 2015 00:10, ludo@HIDDEN (Ludovic Court=C3=A8s) writes:

> The =E2=80=98sanitize-response=E2=80=99 procedure systematically loads th=
e whole
> response body in memory, which causes obvious scalability issues (this
> is in 2.0.11.)
>
> In particular, when a request handler returns a port-taking procedure as
> its second return value, =E2=80=98sanitize-request=E2=80=99 will just cal=
l that
> procedure passing it a string output port.
>
> This procedure should instead be called from the server implementation=E2=
=80=99s
> =E2=80=98write=E2=80=99 hook, but that would necessitate an API change.
>
> Thoughts?

How would you set the Content-Length?  Just leave it off and set
Connection: close ?  Set it in the headers perhaps?  Then you have to
verify later, which I dunno how nice that is.  Maybe it is OK.

I would be happy to review patches :)  Cc guile-devel.

Andy

Message sent to bug-guile@HIDDEN:

Subject: bug#21093: Web server: response bodies systematically loaded in memory
Resent-From: ludo@HIDDEN (Ludovic =?UTF-8?Q?Court=C3=A8s?=)
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guile@HIDDEN
Resent-Date: Fri, 24 Jun 2016 08:59:02 +0000
Resent-Message-ID: <handler.21093.B21093.146675872110436 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: Andy Wingo <wingo@HIDDEN>
Cc: guile-devel@HIDDEN, 21093 <at> debbugs.gnu.org
From: ludo@HIDDEN (Ludovic =?UTF-8?Q?Court=C3=A8s?=)
References: <87a8uriysr.fsf@HIDDEN> <874m8jhuj7.fsf@HIDDEN>
Date: Fri, 24 Jun 2016 10:58:21 +0200
In-Reply-To: <874m8jhuj7.fsf@HIDDEN> (Andy Wingo's message of "Fri, 24 Jun
 2016 10:15:40 +0200")
Message-ID: <87r3bn7yky.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

Andy Wingo <wingo@HIDDEN> skribis:

> On Mon 20 Jul 2015 00:10, ludo@HIDDEN (Ludovic Court=C3=A8s) writes:
>
>> The =E2=80=98sanitize-response=E2=80=99 procedure systematically loads t=
he whole
>> response body in memory, which causes obvious scalability issues (this
>> is in 2.0.11.)
>>
>> In particular, when a request handler returns a port-taking procedure as
>> its second return value, =E2=80=98sanitize-request=E2=80=99 will just ca=
ll that
>> procedure passing it a string output port.
>>
>> This procedure should instead be called from the server implementation=
=E2=80=99s
>> =E2=80=98write=E2=80=99 hook, but that would necessitate an API change.
>>
>> Thoughts?
>
> How would you set the Content-Length?  Just leave it off and set
> Connection: close ?  Set it in the headers perhaps?  Then you have to
> verify later, which I dunno how nice that is.  Maybe it is OK.

I think it could work this way:

  1. By default, provide no =E2=80=98Content-Length=E2=80=99 and force chun=
ked encoding
     (so that the recipient can make sure it received everything.)

     This is useful for data generated on the fly.

  2. Provide an optional mechanism allowing the user to specify the
     content length upfront.

     Useful for statically-generated data that cannot fit in memory.

I haven=E2=80=99t thought yet about the actual API (I=E2=80=99ll be happy t=
o do so as
time permits; to be clear, I don=E2=80=99t consider it a 2.0.12 blocker.)

Ludo=E2=80=99.

Last modified: Mon, 25 Nov 2019 12:00:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.