GNU bug report logs - #37187
26.2; url-retrieve redirect lost Authorization headers

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 37187 in the body.
You can then email your comments to 37187 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Romain Ouabdelkader <romain.ouabdelkader <at> gmail.com>
To: bug-gnu-emacs <at> gnu.org
Subject: 26.2; url-retrieve redirect lost Authorization headers
Date: Mon, 26 Aug 2019 00:08:35 +0200

[Message part 1 (text/plain, inline)]

Hello,

I have an issue with the 'url-retrieve' function:
If the target url returns a redirect, the 'Authorization' header is not
sent on the redirect url.

Example:

List of endpoint:

http://localhost/a: 308 redirect to http://localhost/b
http://localhost/b: display all received headers

I will be using 'url-retrieve' on url 'http://localhost/a' with headers:

Authorization: Basic YWxhZGRpbjpvcGVuc2VzYW1l
X-ExampleHeader: foo

The response shows 'X-ExampleHeader' but not 'Authorization'.
Is there a way to forward 'Authorization' aswell?

Here is a code sample:

(let ((url-request-extra-headers '(("Authorization" . "Basic
YWxhZGRpbjpvcGVuc2VzYW1l")
                                   ("X-ExampleHeader" . "foo"))))
  (url-retrieve "http://localhost:5000/a" (lambda (status)
                                            (display-buffer
(current-buffer) t))))

Server in Python with flask (pip install flask):

from flask import Flask, redirect, request, jsonify

app = Flask(__name__)

@app.route('/a')
def a():
    return redirect("http://localhost:5000/b", code=308)

@app.route('/b')
def b():
    return jsonify(dict(request.headers))

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5000)

In GNU Emacs 26.2 (build 1, x86_64-apple-darwin18.2.0, NS appkit-1671.20
Version 10.14.3 (Build 18D109))
 of 2019-04-13 built on builder10-14.porkrind.org
Windowing system distributor 'Apple', version 10.3.1671
Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.
Mark set
user-error: Beginning of history; no preceding item
Contacting host: localhost:8080
Undo!
Making completion list... [2 times]

Configured using:
 'configure --with-ns '--enable-locallisppath=/Library/Application
 Support/Emacs/${version}/site-lisp:/Library/Application
 Support/Emacs/site-lisp' --with-modules'

Configured features:
NOTIFY ACL GNUTLS LIBXML2 ZLIB TOOLKIT_SCROLL_BARS NS MODULES THREADS

Important settings:
  value of $LC_CTYPE: UTF-8
  value of $LANG: en_FR.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message dired dired-loaddefs format-spec
rfc822 mml easymenu mml-sec epa derived epg epg-config gnus-util rmail
rmail-loaddefs mm-decode mm-bodies mm-encode mailabbrev gmm-utils
mailheader sendmail mail-utils network-stream starttls url-cache
url-http tls gnutls mail-parse rfc2231 rfc2047 rfc2045 mm-util
ietf-drums mail-prsvr url-gw nsm rmc puny seq url-auth url url-proxy
url-privacy url-expand url-methods url-history url-cookie url-domsuf
url-util url-parse auth-source cl-seq eieio byte-opt bytecomp
byte-compile cconv eieio-core cl-macs gv eieio-loaddefs cl-loaddefs
cl-lib password-cache url-vars mailcap elec-pair time-date tooltip eldoc
electric uniquify ediff-hook vc-hooks lisp-float-type mwheel term/ns-win
ns-win ucs-normalize mule-util term/common-win tool-bar dnd fontset
image regexp-opt fringe tabulated-list replace newcomment text-mode
elisp-mode lisp-mode prog-mode register page menu-bar rfn-eshadow
isearch timer select scroll-bar mouse jit-lock font-lock syntax facemenu
font-core term/tty-colors frame cl-generic cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese composite charscript charprop case-table epa-hook
jka-cmpr-hook help simple abbrev obarray minibuffer cl-preloaded nadvice
loaddefs button faces cus-face macroexp files text-properties overlay
sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote threads kqueue cocoa ns multi-tty
make-network-process emacs)

Memory information:
((conses 16 215729 11157)
 (symbols 48 21206 1)
 (miscs 40 48 218)
 (strings 32 32992 1542)
 (string-bytes 1 877912)
 (vectors 16 36759)
 (vector-slots 8 744932 10148)
 (floats 8 51 242)
 (intervals 56 379 360)
 (buffers 992 13))

[Message part 2 (text/html, inline)]

Message #8 received at 37187 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Romain Ouabdelkader <romain.ouabdelkader <at> gmail.com>
Cc: Thomas Fitzsimmons <fitzsim <at> fitzsim.org>, 37187 <at> debbugs.gnu.org
Subject: Re: bug#37187: 26.2; url-retrieve redirect lost Authorization headers
Date: Fri, 20 Sep 2019 22:36:18 +0200

Romain Ouabdelkader <romain.ouabdelkader <at> gmail.com> writes:

> I have an issue with the 'url-retrieve' function:
> If the target url returns a redirect, the 'Authorization' header is not
> sent on the redirect url.

This is apparently on purpose:

	   ;; Do not automatically include an authorization header in the
	   ;; redirect.  If needed it will be regenerated by the relevant
	   ;; auth scheme when the new request happens.
	   (setq url-http-extra-headers
		 (cl-remove "Authorization"
			    url-http-extra-headers :key 'car :test 'equal))

It's from this patch:

commit 325200ac1dcf5bed6918ea827d8a48d89487e083
Author: Thomas Fitzsimmons <fitzsim <at> fitzsim.org>
Date:   Wed Sep 23 01:45:29 2015 -0400

    Do not include authorization header in an HTTP redirect
    
    * lisp/url/url-http.el (url-http-parse-headers): Do not
    automatically include Authorization header in redirect.
    (Bug#21350)

And I think that makes sense -- when there's a redirect, the domain may
be new, and the auth should perhaps not be sent there.

I've had a look at the standards, but I can't see that they say anything
about this, so I think that perhaps this works as it's supposed to.  But
I haven't checked what Firefox does, for instance.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #11 received at 37187 <at> debbugs.gnu.org (full text, mbox):

From: Romain Ouabdelkader <romain.ouabdelkader <at> gmail.com>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: Thomas Fitzsimmons <fitzsim <at> fitzsim.org>, 37187 <at> debbugs.gnu.org
Subject: Re: bug#37187: 26.2; url-retrieve redirect lost Authorization headers
Date: Sat, 21 Sep 2019 02:01:24 +0200

[Message part 1 (text/plain, inline)]

Indeed, curl does the same thing:
https://curl.haxx.se/docs/CVE-2018-1000007.html

But it seems to only strip the Authorization header if the redirect is on
another host:

https://github.com/curl/curl/commit/af32cd3859336ab.patch

On Fri, Sep 20, 2019 at 10:36 PM Lars Ingebrigtsen <larsi <at> gnus.org> wrote:

> Romain Ouabdelkader <romain.ouabdelkader <at> gmail.com> writes:
>
> > I have an issue with the 'url-retrieve' function:
> > If the target url returns a redirect, the 'Authorization' header is not
> > sent on the redirect url.
>
> This is apparently on purpose:
>
>            ;; Do not automatically include an authorization header in the
>            ;; redirect.  If needed it will be regenerated by the relevant
>            ;; auth scheme when the new request happens.
>            (setq url-http-extra-headers
>                  (cl-remove "Authorization"
>                             url-http-extra-headers :key 'car :test 'equal))
>
> It's from this patch:
>
> commit 325200ac1dcf5bed6918ea827d8a48d89487e083
> Author: Thomas Fitzsimmons <fitzsim <at> fitzsim.org>
> Date:   Wed Sep 23 01:45:29 2015 -0400
>
>     Do not include authorization header in an HTTP redirect
>
>     * lisp/url/url-http.el (url-http-parse-headers): Do not
>     automatically include Authorization header in redirect.
>     (Bug#21350)
>
> And I think that makes sense -- when there's a redirect, the domain may
> be new, and the auth should perhaps not be sent there.
>
> I've had a look at the standards, but I can't see that they say anything
> about this, so I think that perhaps this works as it's supposed to.  But
> I haven't checked what Firefox does, for instance.
>
> --
> (domestic pets only, the antidote for overdose, milk.)
>    bloggy blog: http://lars.ingebrigtsen.no
>

[Message part 2 (text/html, inline)]

Message #14 received at 37187 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Romain Ouabdelkader <romain.ouabdelkader <at> gmail.com>
Cc: Thomas Fitzsimmons <fitzsim <at> fitzsim.org>, 37187 <at> debbugs.gnu.org
Subject: Re: bug#37187: 26.2; url-retrieve redirect lost Authorization headers
Date: Sat, 21 Sep 2019 09:41:22 +0200

Romain Ouabdelkader <romain.ouabdelkader <at> gmail.com> writes:

> Indeed, curl does the same thing:
> https://curl.haxx.se/docs/CVE-2018-1000007.html
>
> But it seems to only strip the Authorization header if the redirect is on 
> another host:
>
> https://github.com/curl/curl/commit/af32cd3859336ab.patch

Right.  But Thomas seems to imply in Bug#21350 that url.el will
determine when doing the redirected call whether to include auth again,
so if that new URL requires auth, then it'll be regenerated at that
point.

Is that not the case?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #17 received at 37187 <at> debbugs.gnu.org (full text, mbox):

From: Romain Ouabdelkader <romain.ouabdelkader <at> gmail.com>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: Thomas Fitzsimmons <fitzsim <at> fitzsim.org>, 37187 <at> debbugs.gnu.org
Subject: Re: bug#37187: 26.2; url-retrieve redirect lost Authorization headers
Date: Sat, 21 Sep 2019 10:26:04 +0200

[Message part 1 (text/plain, inline)]

It doesn't forward the auth on the first example I sent with flask.
I'm adding the header in 'url-request-extra-headers',
perhaps there is another way to do it.

On Sat, Sep 21, 2019 at 9:41 AM Lars Ingebrigtsen <larsi <at> gnus.org> wrote:

> Romain Ouabdelkader <romain.ouabdelkader <at> gmail.com> writes:
>
> > Indeed, curl does the same thing:
> > https://curl.haxx.se/docs/CVE-2018-1000007.html
> >
> > But it seems to only strip the Authorization header if the redirect is
> on
> > another host:
> >
> > https://github.com/curl/curl/commit/af32cd3859336ab.patch
>
> Right.  But Thomas seems to imply in Bug#21350 that url.el will
> determine when doing the redirected call whether to include auth again,
> so if that new URL requires auth, then it'll be regenerated at that
> point.
>
> Is that not the case?
>
> --
> (domestic pets only, the antidote for overdose, milk.)
>    bloggy blog: http://lars.ingebrigtsen.no
>

[Message part 2 (text/html, inline)]

Message #20 received at 37187 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Romain Ouabdelkader <romain.ouabdelkader <at> gmail.com>
Cc: Thomas Fitzsimmons <fitzsim <at> fitzsim.org>, 37187 <at> debbugs.gnu.org
Subject: Re: bug#37187: 26.2; url-retrieve redirect lost Authorization headers
Date: Sat, 21 Sep 2019 10:40:51 +0200

Romain Ouabdelkader <romain.ouabdelkader <at> gmail.com> writes:

> It doesn't forward the auth on the first example I sent with flask.
> I'm adding the header in 'url-request-extra-headers',
> perhaps there is another way to do it.

The normal way auth happens is that the server returns 401 Unauthorized
and then the client sends the credentials...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #23 received at 37187 <at> debbugs.gnu.org (full text, mbox):

From: Romain Ouabdelkader <romain.ouabdelkader <at> gmail.com>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: Thomas Fitzsimmons <fitzsim <at> fitzsim.org>, 37187 <at> debbugs.gnu.org
Subject: Re: bug#37187: 26.2; url-retrieve redirect lost Authorization headers
Date: Sat, 21 Sep 2019 11:27:19 +0200

[Message part 1 (text/plain, inline)]

Oh alright I didn't realize emacs was prompting for credentials on
Unauthorized.

Thanks.

On Sat, Sep 21, 2019 at 10:40 AM Lars Ingebrigtsen <larsi <at> gnus.org> wrote:

> Romain Ouabdelkader <romain.ouabdelkader <at> gmail.com> writes:
>
> > It doesn't forward the auth on the first example I sent with flask.
> > I'm adding the header in 'url-request-extra-headers',
> > perhaps there is another way to do it.
>
> The normal way auth happens is that the server returns 401 Unauthorized
> and then the client sends the credentials...
>
> --
> (domestic pets only, the antidote for overdose, milk.)
>    bloggy blog: http://lars.ingebrigtsen.no
>

[Message part 2 (text/html, inline)]

Message #26 received at 37187 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Romain Ouabdelkader <romain.ouabdelkader <at> gmail.com>
Cc: 37187 <at> debbugs.gnu.org, Thomas Fitzsimmons <fitzsim <at> fitzsim.org>
Subject: Re: bug#37187: 26.2; url-retrieve redirect lost Authorization headers
Date: Sun, 19 Jul 2020 22:24:12 +0200

Romain Ouabdelkader <romain.ouabdelkader <at> gmail.com> writes:

> Oh alright I didn't realize emacs was prompting for credentials on
> Unauthorized.

OK, so I don't think there's a bug here, and I'm closing this bug
report.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.