GNU bug report logs - #27012
26.0.50; eww does not generate Referer headers

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 27012 in the body.
You can then email your comments to 27012 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: bug-gnu-emacs <at> gnu.org
Subject: 26.0.50; eww does not generate Referer headers
Date: Sun, 21 May 2017 23:41:53 +0200

This makes it impossible to log in to services like

https://secure.last.fm/login

which results in


---
You are seeing this message because this HTTPS site requires a 'Referer
header' to be sent by your Web browser, but none was sent. This header is
required for security reasons, to ensure that your browser is not being hijacked
by third parties.
---


eww should always generate Referer headers when following links or
posting forms.


In GNU Emacs 26.0.50 (build 3, x86_64-pc-linux-gnu, GTK+ Version 3.14.5)
 of 2017-04-24 built on stories
Repository revision: a1f93c1dfa53dbe007faa09ab0c6e913e86e3ffe
Windowing system distributor 'The X.Org Foundation', version 11.0.11604000
System Description:	Debian GNU/Linux 8.7 (jessie)


-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #8 received at 27012 <at> debbugs.gnu.org (full text, mbox):

From: Richard Stallman <rms <at> gnu.org>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: 27012 <at> debbugs.gnu.org
Subject: Re: bug#27012: 26.0.50; eww does not generate Referer headers
Date: Mon, 22 May 2017 08:18:36 -0400

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > eww should always generate Referer headers when following links or
  > posting forms.

For users' privacy, we should not give real values for the referrer field,
except in special cases.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Message #11 received at 27012 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Richard Stallman <rms <at> gnu.org>
Cc: 27012 <at> debbugs.gnu.org
Subject: Re: bug#27012: 26.0.50; eww does not generate Referer headers
Date: Mon, 22 May 2017 14:28:05 +0200

Richard Stallman <rms <at> gnu.org> writes:

> For users' privacy, we should not give real values for the referrer field,
> except in special cases.

What are those special cases?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #14 received at 27012 <at> debbugs.gnu.org (full text, mbox):

From: Glenn Morris <rgm <at> gnu.org>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: Richard Stallman <rms <at> gnu.org>, 27012 <at> debbugs.gnu.org
Subject: Re: bug#27012: 26.0.50; eww does not generate Referer headers
Date: Mon, 22 May 2017 12:37:00 -0400

You could look at how GNU Icecat handles this, eg
network.http.referer.spoofSource. (I would guess every privacy issue eww
might encounter has already been considered by Icecat.)

Message #17 received at 27012 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Glenn Morris <rgm <at> gnu.org>
Cc: Richard Stallman <rms <at> gnu.org>, 27012 <at> debbugs.gnu.org
Subject: Re: bug#27012: 26.0.50; eww does not generate Referer headers
Date: Mon, 22 May 2017 18:53:42 +0200

Glenn Morris <rgm <at> gnu.org> writes:

> You could look at how GNU Icecat handles this, eg
> network.http.referer.spoofSource. (I would guess every privacy issue eww
> might encounter has already been considered by Icecat.)

network.http.referer.XOriginPolicy

    0 - always send referrer (default).
    1 - only send if base domains match.
    2 - only send if hosts match.

Adding something like this (and defaulting to 1) might make sense for
eww.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #20 received at 27012 <at> debbugs.gnu.org (full text, mbox):

From: Richard Stallman <rms <at> gnu.org>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: 27012 <at> debbugs.gnu.org
Subject: Re: bug#27012: 26.0.50; eww does not generate Referer headers
Date: Mon, 22 May 2017 20:54:12 -0400

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > For users' privacy, we should not give real values for the referrer field,
  > > except in special cases.

  > What are those special cases?

I know of one: when page FOO uses Cloudflare, the Cloudflare CAPTCHA
page insists on getting FOO as a referrer.

I suppose there are others, but I only know of that one.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Message #23 received at 27012 <at> debbugs.gnu.org (full text, mbox):

From: peder <at> klingenberg.no (Peder O. Klingenberg)
To: 27012 <at> debbugs.gnu.org
Subject: Re: bug#27012: 26.0.50; eww does not generate Referer headers
Date: Thu, 13 Jul 2017 01:03:48 +0200

[Message part 1 (text/plain, inline)]

Lars Ingebrigtsen <larsi <at> gnus.org> writes:

>     0 - always send referrer (default).
>     1 - only send if base domains match.
>     2 - only send if hosts match.
>
> Adding something like this (and defaulting to 1) might make sense for
> eww.

I took a stab at implementing this.  It was trickier than I had
anticipated.  The URL library already had a mechanism of sorts for
adding Referer headers, but it was as an optional argument to a helper
function, and there was no way (short of code changes) of making the
entry points of url.el pass that argument to the helper.

Changing the signature of the url entry points and tracking down every
caller didn't seem attractive to me.  Instead, I reasoned that the
referring url is a property of the page currently being displayed, so a
buffer-local variable seemed natural.

That hit a snag because eww uses url queues, and the queue runner didn't
care what the current buffer was.  So some requests got referrers, some
didn't.  I fixed the queue mechanism so it always calls url-retrieve
from the same buffer that queued up the job.

With the basic mechanism for sending the Referer header then working, I
looked at limiting the distribution of it.  url-privacy-level already
existed, and had a basic on-off-knob for referrers, or "lastloc", as it's
called in that variable.  I left that alone, but added an additional
user option - url-lastloc-privacy, with possible values "none",
"domain-match" and "host-match" corresponding to the levels mentioned
above, and with "domain-match" the default.

So with this patch, eww sets up the buffer local url-current-lastloc
when the page renders.  Subsequent requests, be they automatic
requests for images or other resources, or user-invoked link-following,
get the correct Referer header if the privacy settings allow it.

[0001-Make-eww-optionally-send-Referer-headers.patch (text/x-patch, attachment)]

[Message part 3 (text/plain, inline)]

--
...Peder...

Message #26 received at 27012 <at> debbugs.gnu.org (full text, mbox):

From: Andreas Schwab <schwab <at> suse.de>
To: peder <at> klingenberg.no (Peder O. Klingenberg)
Cc: 27012 <at> debbugs.gnu.org
Subject: Re: bug#27012: 26.0.50; eww does not generate Referer headers
Date: Thu, 13 Jul 2017 09:16:42 +0200

On Jul 13 2017, peder <at> klingenberg.no (Peder O. Klingenberg) wrote:

> With the basic mechanism for sending the Referer header then working, I
> looked at limiting the distribution of it.  url-privacy-level already
> existed, and had a basic on-off-knob for referrers, or "lastloc", as it's
> called in that variable.  I left that alone, but added an additional
> user option - url-lastloc-privacy, with possible values "none",

Perhaps url-lastloc-privacy-level, to rhyme with url-privacy-level.

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab <at> suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Message #29 received at 27012 <at> debbugs.gnu.org (full text, mbox):

From: peder <at> klingenberg.no (Peder O. Klingenberg)
To: Andreas Schwab <schwab <at> suse.de>
Cc: 27012 <at> debbugs.gnu.org
Subject: Re: bug#27012: 26.0.50; eww does not generate Referer headers
Date: Thu, 13 Jul 2017 16:02:17 +0200

[Message part 1 (text/plain, inline)]

Andreas Schwab <schwab <at> suse.de> writes:

> Perhaps url-lastloc-privacy-level, to rhyme with url-privacy-level.

Sure, that's better.  Updated patch:

[0001-Make-eww-optionally-send-Referer-headers.patch (text/x-patch, attachment)]

[Message part 3 (text/plain, inline)]

-- 
...Peder...

Message #34 received at 27012 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: peder <at> klingenberg.no (Peder O. Klingenberg)
Cc: Andreas Schwab <schwab <at> suse.de>, 27012 <at> debbugs.gnu.org
Subject: Re: bug#27012: 26.0.50; eww does not generate Referer headers
Date: Fri, 13 Apr 2018 15:09:39 +0200

peder <at> klingenberg.no (Peder O. Klingenberg) writes:

> Sure, that's better.  Updated patch:

Great!  Now I can log in on last.fm!  :-)

I've applied this to Emacs master.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #41 received at 27012 <at> debbugs.gnu.org (full text, mbox):

From: peder <at> klingenberg.no (Peder O. Klingenberg)
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: Andreas Schwab <schwab <at> suse.de>, 27012 <at> debbugs.gnu.org
Subject: Re: bug#27012: 26.0.50; eww does not generate Referer headers
Date: Fri, 13 Apr 2018 15:33:44 +0200

On Fri, Apr 13 2018 at 15:09, Lars Ingebrigtsen wrote:

> peder <at> klingenberg.no (Peder O. Klingenberg) writes:
>
>> Sure, that's better.  Updated patch:
>
> Great!  Now I can log in on last.fm!  :-)
>
> I've applied this to Emacs master.

Thanks.  But the patch has been sitting for a long time, so I think
maybe the :version of the defcustom needs to be bumped to 27?  I don't
think this is likely to be backported to emacs 26.

...Peder...
-- 
I wish a new life awaited _me_ in some off-world colony.

Message #44 received at 27012 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: peder <at> klingenberg.no (Peder O. Klingenberg)
Cc: Andreas Schwab <schwab <at> suse.de>, 27012 <at> debbugs.gnu.org
Subject: Re: bug#27012: 26.0.50; eww does not generate Referer headers
Date: Fri, 13 Apr 2018 15:39:22 +0200

peder <at> klingenberg.no (Peder O. Klingenberg) writes:

> Thanks.  But the patch has been sitting for a long time, so I think
> maybe the :version of the defcustom needs to be bumped to 27?  I don't
> think this is likely to be backported to emacs 26.

Yup; I'll update the defcustom...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.