GNU bug report logs - #39873
[PATCH 0/3] http-multiple-get enhancements

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 39873 in the body.
You can then email your comments to 39873 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Christopher Baines <mail <at> cbaines.net>
To: guix-patches <at> gnu.org
Subject: [PATCH 0/3] http-multiple-get enhancements
Date: Mon, 02 Mar 2020 21:02:57 +0000

[Message part 1 (text/plain, inline)]

These are a few changes I made to the copy of http-multiple-get in the
Guix Data Service.


Christopher Baines (3):
  substitute: Use the same port for multiple request batches.
  substitute: Make http-multiple-get batch size configurable.
  substitute: Close port at the end of http-multiple-get.

 guix/scripts/substitute.scm | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

[signature.asc (application/pgp-signature, inline)]

Message #8 received at 39873 <at> debbugs.gnu.org (full text, mbox):

From: Christopher Baines <mail <at> cbaines.net>
To: 39873 <at> debbugs.gnu.org
Subject: [PATCH 1/3] substitute: Use the same port for multiple request
 batches.
Date: Mon,  2 Mar 2020 21:07:33 +0000

In http-multiple-get.

* guix/scripts/substitute.scm (http-multiple-get): Switch port to p in one
occurrence.
---
 guix/scripts/substitute.scm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm
index dfb975a24a..5ed43d36c9 100755
--- a/guix/scripts/substitute.scm
+++ b/guix/scripts/substitute.scm
@@ -545,7 +545,7 @@ initial connection on which HTTP requests are sent."
              (()
               (reverse result))
              (remainder
-              (connect port remainder result))))
+              (connect p remainder result))))
           ((head tail ...)
            (let* ((resp   (read-response p))
                   (body   (response-body-port resp))
-- 
2.25.0

Message #11 received at 39873 <at> debbugs.gnu.org (full text, mbox):

From: Christopher Baines <mail <at> cbaines.net>
To: 39873 <at> debbugs.gnu.org
Subject: [PATCH 2/3] substitute: Make http-multiple-get batch size
 configurable.
Date: Mon,  2 Mar 2020 21:07:34 +0000

* guix/scripts/substitute.scm (http-multiple-get): Add batch-size parameter.
---
 guix/scripts/substitute.scm | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm
index 5ed43d36c9..a88cb5bcfe 100755
--- a/guix/scripts/substitute.scm
+++ b/guix/scripts/substitute.scm
@@ -501,7 +501,8 @@ MAX-LENGTH first elements."
            (loop (+ 1 len) tail (cons head result)))))))
 
 (define* (http-multiple-get base-uri proc seed requests
-                            #:key port (verify-certificate? #t))
+                            #:key port (verify-certificate? #t)
+                            (batch-size 1000))
   "Send all of REQUESTS to the server at BASE-URI.  Call PROC for each
 response, passing it the request object, the response, a port from which to
 read the response body, and the previous result, starting with SEED, à la
@@ -511,7 +512,7 @@ initial connection on which HTTP requests are sent."
                 (requests requests)
                 (result   seed))
     (define batch
-      (at-most 1000 requests))
+      (at-most batch-size requests))
 
     ;; (format (current-error-port) "connecting (~a requests left)..."
     ;;         (length requests))
-- 
2.25.0

Message #14 received at 39873 <at> debbugs.gnu.org (full text, mbox):

From: Christopher Baines <mail <at> cbaines.net>
To: 39873 <at> debbugs.gnu.org
Subject: [PATCH 3/3] substitute: Close port at the end of http-multiple-get.
Date: Mon,  2 Mar 2020 21:07:35 +0000

* guix/scripts/substitute.scm (http-multiple-get): Add close-port call.
---
 guix/scripts/substitute.scm | 1 +
 1 file changed, 1 insertion(+)

diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm
index a88cb5bcfe..e3f5837a8e 100755
--- a/guix/scripts/substitute.scm
+++ b/guix/scripts/substitute.scm
@@ -544,6 +544,7 @@ initial connection on which HTTP requests are sent."
           (()
            (match (drop requests processed)
              (()
+              (close-port p)
               (reverse result))
              (remainder
               (connect p remainder result))))
-- 
2.25.0

Message #17 received at 39873 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Christopher Baines <mail <at> cbaines.net>
Cc: 39873 <at> debbugs.gnu.org
Subject: Re: [bug#39873] [PATCH 1/3] substitute: Use the same port for
 multiple request batches.
Date: Sat, 07 Mar 2020 22:12:58 +0100

Hi Christopher!

Christopher Baines <mail <at> cbaines.net> skribis:

> In http-multiple-get.

Leftover?  :-)

> * guix/scripts/substitute.scm (http-multiple-get): Switch port to p in one
> occurrence.
> ---
>  guix/scripts/substitute.scm | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm
> index dfb975a24a..5ed43d36c9 100755
> --- a/guix/scripts/substitute.scm
> +++ b/guix/scripts/substitute.scm
> @@ -545,7 +545,7 @@ initial connection on which HTTP requests are sent."
>               (()
>                (reverse result))
>               (remainder
> -              (connect port remainder result))))
> +              (connect p remainder result))))

LGTM!

Did you notice an occurrence of this bug somewhere?

Ludo’.

Message #20 received at 39873 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Christopher Baines <mail <at> cbaines.net>
Cc: 39873 <at> debbugs.gnu.org
Subject: Re: [bug#39873] [PATCH 2/3] substitute: Make http-multiple-get batch
 size configurable.
Date: Sat, 07 Mar 2020 22:13:08 +0100

Christopher Baines <mail <at> cbaines.net> skribis:

> * guix/scripts/substitute.scm (http-multiple-get): Add batch-size parameter.

LGTM!

Message #23 received at 39873 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Christopher Baines <mail <at> cbaines.net>
Cc: 39873 <at> debbugs.gnu.org
Subject: Re: [bug#39873] [PATCH 3/3] substitute: Close port at the end of
 http-multiple-get.
Date: Sat, 07 Mar 2020 22:14:03 +0100

Christopher Baines <mail <at> cbaines.net> scribes:

> * guix/scripts/substitute.scm (http-multiple-get): Add close-port call.
> ---
>  guix/scripts/substitute.scm | 1 +
>  1 file changed, 1 insertion(+)
>
> diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm
> index a88cb5bcfe..e3f5837a8e 100755
> --- a/guix/scripts/substitute.scm
> +++ b/guix/scripts/substitute.scm
> @@ -544,6 +544,7 @@ initial connection on which HTTP requests are sent."
>            (()
>             (match (drop requests processed)
>               (()
> +              (close-port p)

LGTM!

Did you notice a file descriptor leak somewhere?

Thank you!

Ludo’.

Message #26 received at 39873 <at> debbugs.gnu.org (full text, mbox):

From: Christopher Baines <mail <at> cbaines.net>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 39873 <at> debbugs.gnu.org
Subject: Re: [bug#39873] [PATCH 3/3] substitute: Close port at the end of
 http-multiple-get.
Date: Sun, 08 Mar 2020 23:58:51 +0000

[Message part 1 (text/plain, inline)]

Ludovic Courtès <ludo <at> gnu.org> writes:

> Christopher Baines <mail <at> cbaines.net> scribes:
>
>> * guix/scripts/substitute.scm (http-multiple-get): Add close-port call.
>> ---
>>  guix/scripts/substitute.scm | 1 +
>>  1 file changed, 1 insertion(+)
>>
>> diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm
>> index a88cb5bcfe..e3f5837a8e 100755
>> --- a/guix/scripts/substitute.scm
>> +++ b/guix/scripts/substitute.scm
>> @@ -544,6 +544,7 @@ initial connection on which HTTP requests are sent."
>>            (()
>>             (match (drop requests processed)
>>               (()
>> +              (close-port p)
>
> LGTM!
>
> Did you notice a file descriptor leak somewhere?

No, I was looking in to some wierd TLS related errors I had when using
http-multiple-get to query Curiass from the Guix Data Service, and I
noticed that maybe the port wasn't being closed when it should be.

I still haven't done a lot of testing, but I think some of these changes
have helped (although probably not this one).

[signature.asc (application/pgp-signature, inline)]

Message #29 received at 39873 <at> debbugs.gnu.org (full text, mbox):

From: Christopher Baines <mail <at> cbaines.net>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 39873 <at> debbugs.gnu.org
Subject: Re: [bug#39873] [PATCH 1/3] substitute: Use the same port for
 multiple request batches.
Date: Mon, 09 Mar 2020 00:07:46 +0000

[Message part 1 (text/plain, inline)]

Ludovic Courtès <ludo <at> gnu.org> writes:

> Hi Christopher!
>
> Christopher Baines <mail <at> cbaines.net> skribis:
>
>> In http-multiple-get.
>
> Leftover?  :-)

No, just clarifying what function is being referred to. Although the
line below reveals that also.

>> * guix/scripts/substitute.scm (http-multiple-get): Switch port to p in one
>> occurrence.
>> ---
>>  guix/scripts/substitute.scm | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm
>> index dfb975a24a..5ed43d36c9 100755
>> --- a/guix/scripts/substitute.scm
>> +++ b/guix/scripts/substitute.scm
>> @@ -545,7 +545,7 @@ initial connection on which HTTP requests are sent."
>>               (()
>>                (reverse result))
>>               (remainder
>> -              (connect port remainder result))))
>> +              (connect p remainder result))))
>
> LGTM!
>
> Did you notice an occurrence of this bug somewhere?

I was looking for something that could be going wrong here as I was
seeing some kind of error talking to Cuirass using this in the Guix Data
Service, and happened to notice this in the code.

[signature.asc (application/pgp-signature, inline)]

Message #34 received at 39873-done <at> debbugs.gnu.org (full text, mbox):

From: Christopher Baines <mail <at> cbaines.net>
To: 39873-done <at> debbugs.gnu.org
Subject: Re: [bug#39873] [PATCH 0/3] http-multiple-get enhancements
Date: Sun, 26 Apr 2020 19:40:35 +0100

[Message part 1 (text/plain, inline)]

Christopher Baines <mail <at> cbaines.net> writes:

> These are a few changes I made to the copy of http-multiple-get in the
> Guix Data Service.
>
>
> Christopher Baines (3):
>   substitute: Use the same port for multiple request batches.
>   substitute: Make http-multiple-get batch size configurable.
>   substitute: Close port at the end of http-multiple-get.
>
>  guix/scripts/substitute.scm | 8 +++++---
>  1 file changed, 5 insertions(+), 3 deletions(-)

I've finally got around to pushing these as
928dc1bb1c1e96e6dfbe03dac2185ecf41a7b4f5 now.

[signature.asc (application/pgp-signature, inline)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.