GNU bug report logs - #49570
28.0.50; url-http proxy connection-wait

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 49570 in the body.
You can then email your comments to 49570 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Madhu <enometh <at> meer.net>
To: bug-gnu-emacs <at> gnu.org
Subject: 28.0.50; url-http proxy connection-wait
Date: Thu, 15 Jul 2021 15:25:06 +0530

[Message part 1 (text/plain, inline)]

When accessing https://melpa.org (an nginx webserver) through a
http-proxy (by setting url-proxy-services, say to privoxy), url-http
hangs when parsing headers. apparently the connection is dropped with
emacs master. I'm on this commit on master

GNU Emacs 28.0.50 (build 1, x86_64-pc-linux-gnu, X toolkit, cairo
 version 1.16.0, Xaw3d scroll bars) of 2021-06-15
 794ec934a76d0647a72b7be32e20dc5b95e5ec11

#ELISP> (setq url-proxy-services '(("http" . "localhost:8118") ("https" . "localhost:8118")))
ELISP> (url-retrieve-synchronously (url-generic-parse-url "https://melpa.org/"))
#<buffer  *http melpa.org:443*>

And the buffer looks like

#+BEGIN_EXAMPLE

Process melpa.org connection broken by remote peer
HTTP/1.1 200 OK
Server: nginx
Date: Thu, 15 Jul 2021 09:41:12 GMT
Content-Type: text/html; charset=utf-8
Last-Modified: Wed, 10 Feb 2021 23:56:44 GMT
Connection: close
Vary: Accept-Encoding
ETag: W/"602472bc-d92"
Strict-Transport-Security: max-age=15768000
Content-Encoding: gzip
#+END_EXAMPLE

The connection is broken and the "\nProcess HOST connection broken by
remote peer\n" line in the buffer means the spins without parsing the
header.


Now proxied requests are asynchronous.

If I (setf (url-asynchronous url) t) in lisp/url/url.el:
(url-retrieve-internal), as per the appended patch, then eventually
make-network-process gets called with a :nowait t, and the connection
pulls through.

But I'm not sure if this is the right thing or if the problem is
elsewhere as this problem doesn't happen on older emacs - say from
2020-10-15.

Do others see this problem?

[BTW I haven't seen paul eggert on the lists lately, any word if he is
still too busy for emacs?]

[0001-protect-url-retrieve-synchronously-from-broken-proxi.patch (text/x-diff, inline)]

diff --git a/lisp/url/url.el b/lisp/url/url.el
index 00c5740cf8..e40a4ecc29 100644
--- a/lisp/url/url.el
+++ b/lisp/url/url.el
@@ -211,8 +211,9 @@ url-retrieve-internal
 	(buffer nil)
 	(asynch (url-scheme-get-property (url-type url) 'asynchronous-p)))
     (if url-using-proxy
-	(setq asynch t
-	      loader #'url-proxy))
+	(progn (setq asynch t
+	             loader #'url-proxy)
+               (setf (url-asynchronous url) t)))
     (if asynch
 	(let ((url-current-object url))
 	  (setq buffer (funcall loader url callback cbargs)))
-- 
2.31.0

Message #8 received at 49570 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Madhu <enometh <at> meer.net>
Cc: 49570 <at> debbugs.gnu.org
Subject: Re: bug#49570: 28.0.50; url-http proxy connection-wait
Date: Thu, 15 Jul 2021 16:47:53 +0200

Madhu <enometh <at> meer.net> writes:

> When accessing https://melpa.org (an nginx webserver) through a
> http-proxy (by setting url-proxy-services, say to privoxy), url-http
> hangs when parsing headers. apparently the connection is dropped with
> emacs master. I'm on this commit on master
>
> GNU Emacs 28.0.50 (build 1, x86_64-pc-linux-gnu, X toolkit, cairo
>  version 1.16.0, Xaw3d scroll bars) of 2021-06-15
>  794ec934a76d0647a72b7be32e20dc5b95e5ec11

Could you try updating the branch?  Mattias pushed a connection logic
fix a couple of days ago that may (or may not) change things in this
area.

> If I (setf (url-asynchronous url) t) in lisp/url/url.el:
> (url-retrieve-internal), as per the appended patch, then eventually
> make-network-process gets called with a :nowait t, and the connection
> pulls through.
>
> But I'm not sure if this is the right thing or if the problem is
> elsewhere as this problem doesn't happen on older emacs - say from
> 2020-10-15.

It'd be interesting if you could bisect to the commit that broke this.

> [BTW I haven't seen paul eggert on the lists lately, any word if he is
> still too busy for emacs?]

He was here just the other day.  :-)

> -	(setq asynch t
> -	      loader #'url-proxy))
> +	(progn (setq asynch t
> +	             loader #'url-proxy)
> +               (setf (url-asynchronous url) t)))

Hm...  Looking at the code, I think this patch looks correct.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #13 received at 49570 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Madhu <enometh <at> meer.net>
Cc: 49570 <at> debbugs.gnu.org
Subject: Re: bug#49570: 28.0.50; url-http proxy connection-wait
Date: Mon, 19 Jul 2021 17:42:41 +0200

Lars Ingebrigtsen <larsi <at> gnus.org> writes:

>> -	(setq asynch t
>> -	      loader #'url-proxy))
>> +	(progn (setq asynch t
>> +	             loader #'url-proxy)
>> +               (setf (url-asynchronous url) t)))
>
> Hm...  Looking at the code, I think this patch looks correct.

So I applied the patch to Emacs 28.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #18 received at 49570 <at> debbugs.gnu.org (full text, mbox):

From: Madhu <enometh <at> meer.net>
To: larsi <at> gnus.org
Cc: 49570 <at> debbugs.gnu.org
Subject: Re: bug#49570: 28.0.50; url-http proxy connection-wait
Date: Tue, 20 Jul 2021 12:20:50 +0530 (IST)

*  Lars Ingebrigtsen <larsi <at> gnus.org> <87lf62l37y.fsf <at> gnus.org>
Wrote on Mon, 19 Jul 2021 17:42:41 +0200

> So I applied the patch to Emacs 28.

Thanks.  (I'm afraid I haven't found the time to bisect it, and
running native-comp is a disincentive to change HEAD (with debug it
takes 100s of GB) but maybe i can still try to bisect after I update
-- for curiosity's sake)

Message #21 received at 49570 <at> debbugs.gnu.org (full text, mbox):

From: Andrea Corallo <akrl <at> sdf.org>
To: Madhu <enometh <at> meer.net>
Cc: larsi <at> gnus.org, 49570 <at> debbugs.gnu.org
Subject: Re: bug#49570: 28.0.50; url-http proxy connection-wait
Date: Tue, 20 Jul 2021 14:08:07 +0000

Madhu <enometh <at> meer.net> writes:

> *  Lars Ingebrigtsen <larsi <at> gnus.org> <87lf62l37y.fsf <at> gnus.org>
> Wrote on Mon, 19 Jul 2021 17:42:41 +0200
>
>> So I applied the patch to Emacs 28.
>
> Thanks.  (I'm afraid I haven't found the time to bisect it, and
> running native-comp is a disincentive to change HEAD (with debug it
> takes 100s of GB) but maybe i can still try to bisect after I update
> -- for curiosity's sake)

native-comp is merged into master since a while.

Actually I thought I removed feature/native-comp, is it still in place?

Regards

  Andrea

Message #24 received at 49570 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Andrea Corallo <akrl <at> sdf.org>
Cc: Madhu <enometh <at> meer.net>, 49570 <at> debbugs.gnu.org
Subject: Re: bug#49570: 28.0.50; url-http proxy connection-wait
Date: Tue, 20 Jul 2021 16:20:10 +0200

Andrea Corallo <akrl <at> sdf.org> writes:

> Actually I thought I removed feature/native-comp, is it still in place?

I think it's gone from Savannah, but if I remember correctly, that
doesn't make the branch go away in other repos?  And indeed:

larsi <at> elva:~/src/emacs/trunk$ git branch -a | grep native
  remotes/origin/feature/native-comp
  remotes/origin/feature/native-comp-macos-fixes
  remotes/origin/scratch/native-comp-cl
  remotes/origin/scratch/native-comp-eln-caches
  remotes/origin/scratch/native-comp-gcc-driver-options
  remotes/origin/scratch/native-comp-macos-43532

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #27 received at 49570 <at> debbugs.gnu.org (full text, mbox):

From: Andrea Corallo <akrl <at> sdf.org>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: Madhu <enometh <at> meer.net>, 49570 <at> debbugs.gnu.org
Subject: Re: bug#49570: 28.0.50; url-http proxy connection-wait
Date: Tue, 20 Jul 2021 15:13:01 +0000

Lars Ingebrigtsen <larsi <at> gnus.org> writes:

> Andrea Corallo <akrl <at> sdf.org> writes:
>
>> Actually I thought I removed feature/native-comp, is it still in place?
>
> I think it's gone from Savannah, but if I remember correctly, that
> doesn't make the branch go away in other repos?  And indeed:
>
> larsi <at> elva:~/src/emacs/trunk$ git branch -a | grep native
>   remotes/origin/feature/native-comp
>   remotes/origin/feature/native-comp-macos-fixes
>   remotes/origin/scratch/native-comp-cl
>   remotes/origin/scratch/native-comp-eln-caches
>   remotes/origin/scratch/native-comp-gcc-driver-options
>   remotes/origin/scratch/native-comp-macos-43532

I see, that's annoying but I guess we have no workaround for that :/

Thanks for checking.

  Andrea

Message #30 received at 49570 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Andrea Corallo <akrl <at> sdf.org>
Cc: Madhu <enometh <at> meer.net>, 49570 <at> debbugs.gnu.org
Subject: Re: bug#49570: 28.0.50; url-http proxy connection-wait
Date: Tue, 20 Jul 2021 17:24:05 +0200

Andrea Corallo <akrl <at> sdf.org> writes:

> I see, that's annoying but I guess we have no workaround for that :/

Let's see...  The magical git command seems to be:

git fetch origin --prune

And that did make the native-comp branch disappear locally.  But it's
something each user has to do.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #33 received at 49570 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Andrea Corallo <akrl <at> sdf.org>
Cc: larsi <at> gnus.org, 49570 <at> debbugs.gnu.org, enometh <at> meer.net
Subject: Re: bug#49570: 28.0.50; url-http proxy connection-wait
Date: Tue, 20 Jul 2021 18:42:21 +0300

> Cc: Madhu <enometh <at> meer.net>, 49570 <at> debbugs.gnu.org
> Date: Tue, 20 Jul 2021 15:13:01 +0000
> From:  Andrea Corallo via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs <at> gnu.org>
> 
> Lars Ingebrigtsen <larsi <at> gnus.org> writes:
> 
> > Andrea Corallo <akrl <at> sdf.org> writes:
> >
> >> Actually I thought I removed feature/native-comp, is it still in place?
> >
> > I think it's gone from Savannah, but if I remember correctly, that
> > doesn't make the branch go away in other repos?  And indeed:
> >
> > larsi <at> elva:~/src/emacs/trunk$ git branch -a | grep native
> >   remotes/origin/feature/native-comp
> >   remotes/origin/feature/native-comp-macos-fixes
> >   remotes/origin/scratch/native-comp-cl
> >   remotes/origin/scratch/native-comp-eln-caches
> >   remotes/origin/scratch/native-comp-gcc-driver-options
> >   remotes/origin/scratch/native-comp-macos-43532
> 
> I see, that's annoying but I guess we have no workaround for that :/

The workaround is to tell people to say

   git remote prune origin

from time to time.

Message #36 received at 49570 <at> debbugs.gnu.org (full text, mbox):

From: Gregory Heytings <gregory <at> heytings.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: larsi <at> gnus.org, 49570 <at> debbugs.gnu.org, enometh <at> meer.net,
 Andrea Corallo <akrl <at> sdf.org>
Subject: Re: bug#49570: 28.0.50; url-http proxy connection-wait
Date: Tue, 20 Jul 2021 16:22:17 +0000

>>>> Actually I thought I removed feature/native-comp, is it still in 
>>>> place?
>>>
>>> I think it's gone from Savannah, but if I remember correctly, that 
>>> doesn't make the branch go away in other repos?
>>
>> I see, that's annoying but I guess we have no workaround for that :/
>
> The workaround is to tell people to say
>
>   git remote prune origin
>
> from time to time.
>

No, the workaround is to tell people to set

git config remote.origin.prune true

Message #39 received at 49570 <at> debbugs.gnu.org (full text, mbox):

From: Gregory Heytings <gregory <at> heytings.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: larsi <at> gnus.org, 49570 <at> debbugs.gnu.org, enometh <at> meer.net,
 Andrea Corallo <akrl <at> sdf.org>
Subject: Re: bug#49570: 28.0.50; url-http proxy connection-wait
Date: Wed, 21 Jul 2021 11:01:09 +0000

>>>>> Actually I thought I removed feature/native-comp, is it still in 
>>>>> place?
>>>> 
>>>> I think it's gone from Savannah, but if I remember correctly, that 
>>>> doesn't make the branch go away in other repos?
>>> 
>>> I see, that's annoying but I guess we have no workaround for that :/
>> 
>> The workaround is to tell people to say
>>
>>   git remote prune origin
>> 
>> from time to time.
>
> No, the workaround is to tell people to set
>
> git config remote.origin.prune true
>

(Or, of course, to add a "git_config remote.origin.prune true" in 
autogen.sh.)

Message #42 received at 49570 <at> debbugs.gnu.org (full text, mbox):

From: "Basil L. Contovounesios" <contovob <at> tcd.ie>
To: Gregory Heytings <gregory <at> heytings.org>
Cc: Eli Zaretskii <eliz <at> gnu.org>, Andrea Corallo <akrl <at> sdf.org>,
 49570 <at> debbugs.gnu.org, larsi <at> gnus.org, enometh <at> meer.net
Subject: Re: bug#49570: 28.0.50; url-http proxy connection-wait
Date: Wed, 21 Jul 2021 13:46:39 +0100

Gregory Heytings <gregory <at> heytings.org> writes:

>>>>>> Actually I thought I removed feature/native-comp, is it still in place?
>>>>> I think it's gone from Savannah, but if I remember correctly, that doesn't
>>>>> make the branch go away in other repos?
>>>> I see, that's annoying but I guess we have no workaround for that :/
>>> The workaround is to tell people to say
>>>
>>>   git remote prune origin
>>> from time to time.
>>
>> No, the workaround is to tell people to set
>>
>> git config remote.origin.prune true
>
> (Or, of course, to add a "git_config remote.origin.prune true" in autogen.sh.)

If pruning loses data it shouldn't be enabled by default in all
checkouts, e.g. if a branch is accidentally deleted by someone else then
simply fetching shouldn't propagate that deletion for others.  So IMO
the developer should decide when and how to enable this.  (Unless I've
misunderstood something.)

-- 
Basil

Message #45 received at 49570 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: "Basil L. Contovounesios" <contovob <at> tcd.ie>
Cc: gregory <at> heytings.org, akrl <at> sdf.org, 49570 <at> debbugs.gnu.org, larsi <at> gnus.org,
 enometh <at> meer.net
Subject: Re: bug#49570: 28.0.50; url-http proxy connection-wait
Date: Wed, 21 Jul 2021 16:06:59 +0300

> From: "Basil L. Contovounesios" <contovob <at> tcd.ie>
> Cc: Eli Zaretskii <eliz <at> gnu.org>,  larsi <at> gnus.org,  49570 <at> debbugs.gnu.org,
>   enometh <at> meer.net,  Andrea Corallo <akrl <at> sdf.org>
> Date: Wed, 21 Jul 2021 13:46:39 +0100
> 
> Gregory Heytings <gregory <at> heytings.org> writes:
> 
> >> No, the workaround is to tell people to set
> >>
> >> git config remote.origin.prune true
> >
> > (Or, of course, to add a "git_config remote.origin.prune true" in autogen.sh.)
> 
> If pruning loses data it shouldn't be enabled by default in all
> checkouts, e.g. if a branch is accidentally deleted by someone else then
> simply fetching shouldn't propagate that deletion for others.

Of course.  Don't worry, we won't do that.

Message #48 received at 49570 <at> debbugs.gnu.org (full text, mbox):

From: Gregory Heytings <gregory <at> heytings.org>
To: "Basil L. Contovounesios" <contovob <at> tcd.ie>
Cc: Eli Zaretskii <eliz <at> gnu.org>, Andrea Corallo <akrl <at> sdf.org>,
 49570 <at> debbugs.gnu.org, larsi <at> gnus.org, enometh <at> meer.net
Subject: Re: bug#49570: 28.0.50; url-http proxy connection-wait
Date: Wed, 21 Jul 2021 13:11:18 +0000

>
> If pruning loses data it shouldn't be enabled by default in all 
> checkouts, e.g. if a branch is accidentally deleted by someone else then 
> simply fetching shouldn't propagate that deletion for others.  So IMO 
> the developer should decide when and how to enable this.  (Unless I've 
> misunderstood something.)
>

Pruning a branch does not lose data AFAIK, it only removes the pointer to 
the last commit of that branch.

Message #51 received at 49570 <at> debbugs.gnu.org (full text, mbox):

From: Gregory Heytings <gregory <at> heytings.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: "Basil L. Contovounesios" <contovob <at> tcd.ie>, enometh <at> meer.net,
 49570 <at> debbugs.gnu.org, larsi <at> gnus.org, akrl <at> sdf.org
Subject: Re: bug#49570: 28.0.50; url-http proxy connection-wait
Date: Thu, 22 Jul 2021 07:30:30 +0000

>> If pruning loses data it shouldn't be enabled by default in all 
>> checkouts, e.g. if a branch is accidentally deleted by someone else 
>> then simply fetching shouldn't propagate that deletion for others.
>
> Of course.  Don't worry, we won't do that.
>

Why on earth won't you do that?  Deleting a branch (or removing a tag) 
does *not* lose any data / commit, it only removes a pointer / label to a 
particular commit.  For example anyone can still access the commits of the 
feature/native-comp branch with git checkout fa65c044f2.  Should someone 
accidentally delete a branch, it is easy to create it again.

Setting this config variable is much better than telling people to say 
"git remote prune origin" from time to time as you suggested, it does the 
same, except that it happens automatically.

Message #54 received at 49570 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Gregory Heytings <gregory <at> heytings.org>
Cc: contovob <at> tcd.ie, enometh <at> meer.net, 49570 <at> debbugs.gnu.org, larsi <at> gnus.org,
 akrl <at> sdf.org
Subject: Re: bug#49570: 28.0.50; url-http proxy connection-wait
Date: Thu, 22 Jul 2021 11:52:25 +0300

> Date: Thu, 22 Jul 2021 07:30:30 +0000
> From: Gregory Heytings <gregory <at> heytings.org>
> cc: "Basil L. Contovounesios" <contovob <at> tcd.ie>, enometh <at> meer.net, 
>     49570 <at> debbugs.gnu.org, larsi <at> gnus.org, akrl <at> sdf.org
> 
> 
> >> If pruning loses data it shouldn't be enabled by default in all 
> >> checkouts, e.g. if a branch is accidentally deleted by someone else 
> >> then simply fetching shouldn't propagate that deletion for others.
> >
> > Of course.  Don't worry, we won't do that.
> 
> Why on earth won't you do that?

Because it isn't our place to second-guess when the user wants to
purge his/her local Git repository.

Message #57 received at 49570 <at> debbugs.gnu.org (full text, mbox):

From: Gregory Heytings <gregory <at> heytings.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: contovob <at> tcd.ie, enometh <at> meer.net, 49570 <at> debbugs.gnu.org, larsi <at> gnus.org,
 akrl <at> sdf.org
Subject: Re: bug#49570: 28.0.50; url-http proxy connection-wait
Date: Thu, 22 Jul 2021 09:10:41 +0000

>> Why on earth won't you do that?
>
> Because it isn't our place to second-guess when the user wants to purge 
> his/her local Git repository.
>

The word "prune" or "purge" are perhaps a bit unfortunate, they seem to 
imply that something serious is happening (as Basil thought).  Once again 
no actual data is removed, the only thing that is removed is a pointer to 
a particular commit.  Two people (Lars and Andrea) regretted upthread that 
removing a branch does not remove them in local repositories, and I 
remember at least one recent incident in which someone was using the 
native-comp branch after it had been removed.  Which is why I suggested to 
set that configuration variable... but I preach in the wilderness.

Message #60 received at 49570 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Gregory Heytings <gregory <at> heytings.org>
Cc: contovob <at> tcd.ie, enometh <at> meer.net, 49570 <at> debbugs.gnu.org, larsi <at> gnus.org,
 akrl <at> sdf.org
Subject: Re: bug#49570: 28.0.50; url-http proxy connection-wait
Date: Thu, 22 Jul 2021 12:40:47 +0300

> Date: Thu, 22 Jul 2021 09:10:41 +0000
> From: Gregory Heytings <gregory <at> heytings.org>
> cc: contovob <at> tcd.ie, enometh <at> meer.net, 49570 <at> debbugs.gnu.org, larsi <at> gnus.org, 
>     akrl <at> sdf.org
> 
> >> Why on earth won't you do that?
> >
> > Because it isn't our place to second-guess when the user wants to purge 
> > his/her local Git repository.
> >
> 
> The word "prune" or "purge" are perhaps a bit unfortunate, they seem to 
> imply that something serious is happening (as Basil thought).

We are miscommunicating.  Regardless of whether the loss of data is or
isn't significant, it is impolite, to say the least, for the project
to force users to do anything in their local clone that is not
absolutely necessary.  In particular, whether a branch should or
shouldn't be deleted in the local clone is for the user to decide, not
for us.  So we should not do that.

> Two people (Lars and Andrea) regretted upthread that removing a
> branch does not remove them in local repositories, and I remember at
> least one recent incident in which someone was using the native-comp
> branch after it had been removed.  Which is why I suggested to set
> that configuration variable... but I preach in the wilderness.

People who want this automatic pruning can configure that in their
personal ~/.gitconfig.  So the information about doing that
automatically is not lost; thank you for posting it.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.