GNU bug report logs - #44072
27.1; url-retrieve-synchronously via proxy gets into wait loop without process.

Previous Next

To reply to this bug, email your comments to 44072 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Peter Wagemans <pwml_no001spam <at> xs4all.nl>
To: bug-gnu-emacs <at> gnu.org
Subject: 27.1; url-retrieve-synchronously via proxy gets into wait loop
 without process.
Date: Sun, 18 Oct 2020 23:57:33 +0200

The problem was encountered on Windows 10, after migration from Emacs
26.3 to 27.1, with url retrieval via a Cygwin cntlm proxy and VPN to
company proxy (difficult to reproduce the timing characteristics I
guess). Emacs was no longer able to install or update many melpa
packages from the (package-)list-packages buffer.

Package update successfully downloads the package and then looks for a
.sig file that doesn't exist on melpa.org.  The melpa nginx returns
http status 404 with a small gzipped html content body that contains
the "404 Not Found" message. This html body becomes about 100 bytes
gzipped.  For this, the melpa nginx doesn't use chunked transfer but
also doesn't send Content-Length (hmm, rfc7230: ..., a server SHOULD
generate encoding or length-delimited messages whenever possible.), so
Emacs can't use the length to detect that the transfer is complete
(url-debug: "http -> No content-length, being dumb") and should
conclude that the transfer is complete when the connection is
closed. It seems that the loop in url-retrieve-synchronously doesn't
pick up that event and doesn't notice that the process is gone.

Note: melpa nginx not sending Content-Length seems to depend on the
url-privacy-level of Emacs; the setting paranoid (used above) doesn't
send a User-Agent header but the default setting does tell melpa that
the user agent is Emacs (User-Agent: URL/Emacs Emacs/...) and with
that privacy setting you get an uncompressed response with
Content-Length and the problem doesn't occur.

Without proxy, I haven't seen the problem: url-debug shows that
url-http-end-of-document-sentinel gets called and the package install
proceeds.

Below is an example url-debug trace, with the routine modified to
print the buffer content in the loop. The binary gzip data isn't very
readable, so it was replaced by the string BINARYGZIPDATA. In the
first iteration, the complete response is already returned, but
without length indication, so Emacs doesn't know that until the
connection is closed. The second time, the string "Process melpa.org
connection broken by remote peer" has been added and after that the
looping continues with no further change in the content. When you
interrupt the loop, the buffer has no process and the backtrace shows
that the value of proc in the loop is nil.

Workaround for the problem:

    (setq url-mime-encoding-string "identity")

to not compress the response content, or lower the privacy level.
Then you get Content-Length from nginx and the problem is avoided.  Or
arrange for internet access without proxy.

Perhaps similar to

    https://debbugs.gnu.org/cgi/bugreport.cgi?bug=34103

    26.1; url-retrieve-synchronously freezes in an infinite loop when
    behind a proxy because of `url-asynchronous' being bound to nil

although I haven't encountered it with the Emacs 26.x versions.


An example url-debug extract:

http -> Contacting host: melpa.org:443
http -> Marking connection as busy: melpa.org:443 #<process melpa.org>
http -> getting referer from buffer: buffer:#<buffer  *temp*-414823> target-url:#s(url "https" nil nil "melpa.org" nil "/packages/async-20200809.501.tar.sig"
nil nil t nil t nil) lastloc:nil
retrieval -> Spinning in url-retrieve-synchronously: nil ("")
http -> Calling after change function `url-https-proxy-after-change-function' for `#<process melpa.org>'
http -> url-http-parse-response called in ( *http melpa.org:443*-499627)
http -> Request is:
GET /packages/async-20200809.501.tar.sig HTTP/1.1
MIME-Version: 1.0
Connection: close
Extension: Security/Digest Security/SSL
Host: melpa.org
Accept-encoding: gzip
Accept: */*


retrieval -> Spinning in url-retrieve-synchronously: nil ("")
http -> Calling after change function `url-http-wait-for-headers-change-function' for `#<process melpa.org>'
http -> url-http-wait-for-headers-change-function ( *http melpa.org:443*-499627)
http -> Saw end of headers... ( *http melpa.org:443*-499627)
http -> url-http-parse-response called in ( *http melpa.org:443*-499627)
http -> No content-length, being dumb.
http -> Spinning waiting for headers...
retrieval -> Spinning in url-retrieve-synchronously: nil ("HTTP/1.1 404 Not Found
Server: nginx
Date: Fri, 16 Oct 2020 14:33:46 GMT
Content-Type: text/html; charset=utf-8
Connection: close
Vary: Accept-Encoding
Content-Encoding: gzip

BINARYGZIPDATA")
retrieval -> Spinning in url-retrieve-synchronously: nil ("HTTP/1.1 404 Not Found
Server: nginx
Date: Fri, 16 Oct 2020 14:33:46 GMT
Content-Type: text/html; charset=utf-8
Connection: close
Vary: Accept-Encoding
Content-Encoding: gzip

BINARYGZIPDATA
Process melpa.org connection broken by remote peer
")
retrieval -> Spinning in url-retrieve-synchronously: nil ("HTTP/1.1 404 Not Found
Server: nginx
Date: Fri, 16 Oct 2020 14:33:46 GMT
Content-Type: text/html; charset=utf-8
Connection: close
Vary: Accept-Encoding
Content-Encoding: gzip

BINARYGZIPDATA
Process melpa.org connection broken by remote peer
")
retrieval -> Spinning in url-retrieve-synchronously: nil ("HTTP/1.1 404 Not Found
... looping

Regards,

Peter Wagemans

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.