GNU bug report logs - #35969
proxy + excorporate -> Failed: Failed to retrieve https://outlook.office365.com/EWS/Services.wsdl

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 35969 in the body.
You can then email your comments to 35969 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: "tenspd137 ." <dcday137 <at> gmail.com>
To: bug-gnu-emacs <at> gnu.org
Subject: 26.2, Excorporate
Date: Tue, 28 May 2019 15:25:40 -0600

Hello,

I was trying to use Excorporate through a proxy, and it fails.  I am
pretty sure that my configuration is correct because I am able to
retrieve *a* file when I use wget and https://server/EWS/Exchange.asmx
and I can get a Services.wsdl file with
https://server/EWS/Exchange.asmx.

If it helps, here is info from URL-DEBUG (with some identifying info removed):

http -> Contacting host: outlook.office365.com:443
http -> Marking connection as busy: outlook.office365.com:443
#<process outlook.office365.com<1>>
http -> Calling after change function
`url-https-proxy-after-change-function' for `#<process
outlook.office365.com<1>>'
http -> url-http-parse-response called in ( *http
outlook.office365.com:443*-608159)
http -> Request is:
GET https://outlook.office365.com/EWS/Services.wsdl HTTP/1.1
MIME-Version: 1.0
Connection: close
Extension: Security/Digest Security/SSL
Host: outlook.office365.com
Accept-encoding: gzip
Accept-charset: utf-8;q=1, gb2312;q=0.5, iso-8859-1;q=0.5, big5;q=0.5,
iso-2022-jp;q=0.5, shift_jis;q=0.5, euc-tw;q=0.5, euc-jp;q=0.5,
euc-jis-2004;q=0.5, euc-kr;q=0.5, us-ascii;q=0.5, utf-7;q=0.5,
hz-gb-2312;q=0.5, big5-hkscs;q=0.5, gbk;q=0.5, gb18030;q=0.5,
iso-8859-5;q=0.5, koi8-r;q=0.5, koi8-u;q=0.5, cp866;q=0.5,
koi8-t;q=0.5, windows-1251;q=0.5, cp855;q=0.5, iso-8859-2;q=0.5,
iso-8859-3;q=0.5, iso-8859-4;q=0.5, iso-8859-9;q=0.5,
iso-8859-10;q=0.5, iso-8859-13;q=0.5, iso-8859-14;q=0.5,
iso-8859-15;q=0.5, windows-1250;q=0.5, windows-1252;q=0.5,
windows-1254;q=0.5, windows-1257;q=0.5, cp775;q=0.5, cp850;q=0.5,
cp852;q=0.5, cp857;q=0.5, cp858;q=0.5, cp860;q=0.5, cp861;q=0.5,
cp863;q=0.5, cp865;q=0.5, cp437;q=0.5, macintosh;q=0.5, next;q=0.5,
hp-roman8;q=0.5, adobe-standard-encoding;q=0.5, iso-8859-16;q=0.5,
iso-8859-7;q=0.5, windows-1253;q=0.5, cp737;q=0.5, cp851;q=0.5,
cp869;q=0.5, iso-8859-8;q=0.5, windows-1255;q=0.5, cp862;q=0.5,
iso-2022-jp-2004;q=0.5, cp874;q=0.5, iso-8859-11;q=0.5, viscii;q=0.5,
windows-1258;q=0.5, iso-8859-6;q=0.5, windows-1256;q=0.5,
iso-2022-cn;q=0.5, iso-2022-cn-ext;q=0.5, iso-2022-jp-2;q=0.5,
iso-2022-kr;q=0.5, utf-16le;q=0.5, utf-16be;q=0.5, utf-16;q=0.5,
x-ctext;q=0.5
Accept: */*
User-Agent: URL/Emacs Emacs/26.2 (X11; x86_64-pc-linux-gnu)
Authorization: Basic ZGF2aWQuYy5kYXlAaHAuY29tOmwxaTNuN2VTa3kxMzcwJSU=
Cookie: ClientId=8998C5691CD143E784857A0D01537963; OIDC=1


http -> Calling after change function
`url-http-wait-for-headers-change-function' for `#<process
outlook.office365.com<1>>'
http -> url-http-wait-for-headers-change-function ( *http
outlook.office365.com:443*-608159)
http -> Saw end of headers... ( *http outlook.office365.com:443*-608159)
http -> url-http-parse-response called in ( *http
outlook.office365.com:443*-608159)
http -> Got a content-length, being smart about document end.
http -> Calling initial content-length for extra data at end of headers
http -> Marking connection as free: outlook.office365.com:443
#<process outlook.office365.com<1>>
http -> url-http-parse-headers called in ( *http
outlook.office365.com:443*-608159)
http -> url-http-parse-response called in ( *http
outlook.office365.com:443*-608159)
http -> Parsed HTTP headers: class=4 status=404
http -> Finished parsing HTTP headers: t
http -> Marking connection as free: outlook.office365.com:443
#<process outlook.office365.com<1>>
http -> Activating callback in buffer ( *http
outlook.office365.com:443*-608159): #[257
"p\303\304\305\306\307 !\310\"\311$\216
@\312=\203$\0\313\301\314\315\316\302\"#\210\317\300\320\"\202/\0\313\301\321\322
#\210\317\300\323\")\207" [fsm-exco--fsm-127 (:identifier
("my-work-email" . "https://outlook.office365.com/EWS/Exchange.asmx")
:mail-address "my-work-email" :retrying nil :autodiscovery-urls nil
:service-url "https://outlook.office365.com/EWS/Exchange.asmx"
:service-xml nil :service-wsdl nil :next-state-after-success nil
:failure-message nil :server-version nil)
"https://outlook.office365.com/EWS/Services.wsdl" make-byte-code 0
"\301\300!\205    \0\302\300!\207" vconcat vector [buffer-live-p
kill-buffer] 2 :error plist-put :failure-message format "Failed to
retrieve %s" fsm-send :unrecoverable-error :service-xml
exco--parse-xml-in-current-buffer :success] 8 "

(fn STATUS)"] ((:error (error http 404) :peer (:certificate (:version
3 :serial-number "03:fa:5c:d7:7f:8a:96:cf:9a:bf:bc:16:5d:a9:8a:0c"
:issuer "C=US,O=DigiCert Inc,CN=DigiCert Cloud Services CA-1"
:valid-from "2018-10-03" :valid-to "2020-10-03" :subject
"C=US,ST=Washington,L=Redmond,O=Microsoft Corporation,CN=outlook.com"
:public-key-algorithm "RSA" :certificate-security-level "Medium"
:signature-algorithm "RSA-SHA256" :public-key-id
"sha1:b5:44:50:ef:ea:49:4c:ea:39:ed:03:22:32:cd:82:54:72:b2:b2:7d"
:certificate-id
"sha1:fb:51:e0:1f:ed:b9:83:bf:0f:41:df:af:4e:4f:25:82:48:84:40:eb")
:key-exchange "ECDHE-RSA" :protocol "TLS1.2" :cipher "AES-256-GCM"
:mac "AEAD")))
http -> Spinning waiting for headers...
excorporate -> Failed: Failed to retrieve
https://outlook.office365.com/EWS/Services.wsdl

Thanks for any help / suggestions.

Message #8 received at 35969 <at> debbugs.gnu.org (full text, mbox):

From: Thomas Fitzsimmons <fitzsim <at> fitzsim.org>
To: "tenspd137 ." <dcday137 <at> gmail.com>
Cc: 35969 <at> debbugs.gnu.org
Subject: Re: bug#35969: 26.2, Excorporate
Date: Tue, 28 May 2019 21:04:54 -0400

Hi,

"tenspd137 ." <dcday137 <at> gmail.com> writes:

> I was trying to use Excorporate through a proxy, and it fails.  I am
> pretty sure that my configuration is correct because I am able to
> retrieve *a* file when I use wget and https://server/EWS/Exchange.asmx
> and I can get a Services.wsdl file with
> https://server/EWS/Exchange.asmx.

Thanks for reporting.  Is just Emacs (not Excorporate) able to retrieve
the file through the proxy?  That is, starting from an "emacs -Q"
session, can you try just:

M-: (setq url-debug t)
M-: (url-retrieve-synchronously "https://server/EWS/Exchange.asmx")

and see what the *URL-DEBUG* and " *http server:443*" (note leading
space character) buffers contain afterwards?

Thanks,
Thomas

Message #11 received at 35969 <at> debbugs.gnu.org (full text, mbox):

From: "tenspd137 ." <dcday137 <at> gmail.com>
To: Thomas Fitzsimmons <fitzsim <at> fitzsim.org>
Cc: 35969 <at> debbugs.gnu.org
Subject: Re: bug#35969: 26.2, Excorporate
Date: Wed, 29 May 2019 10:17:05 -0600

Hi - thanks for the reply.  Here are the steps I took (I had to
configure the proxy as well....):

M-: (setq url-proxy-services '(("http" . "web-proxy:8080") ("https" .
"web-proxy:8080")))
M-: (setq url-debug t)
M-: (url-retrieve-synchronously "https://server/EWS/Exchange.asmx")

-Asks for name and password-, I fill them in.....

The buffer contents are:
" *http server:443*"
HTTP/1.1 401 Unauthorized
Server: Microsoft-IIS/10.0
request-id: 12d85c25-a5f7-49d5-a794-ef282ec5e6af
X-WSSecurity-Enabled: True
X-WSSecurity-For: Logon
X-FederationTrustTokenIssuerUri: urn:federation:MicrosoftOnline
X-WSSecurity-SymmetricKey-Enabled: True
X-WSSecurity-X509Cert-Enabled: True
X-OAuth-Enabled: True
X-Powered-By: ASP.NET
X-FEServer: SN6PR16CA0070
WWW-Authenticate: Basic Realm=""
Date: Wed, 29 May 2019 16:00:44 GMT
Connection: close
Content-Length: 0

and "*URL-DEBUG*"
http -> Contacting host: outlook.office365.com:443
http -> Marking connection as busy: outlook.office365.com:443
#<process outlook.office365.com>
retrieval -> Spinning in url-retrieve-synchronously: nil (#<buffer
*http outlook.office365.com:443*>)
http -> Calling after change function
`url-https-proxy-after-change-function' for `#<process
outlook.office365.com>'
http -> url-http-parse-response called in ( *http outlook.office365.com:443*)
http -> Request is:
GET https://outlook.office365.com/EWS/Exchange.asmx HTTP/1.1
MIME-Version: 1.0
Connection: close
Extension: Security/Digest Security/SSL
Host: outlook.office365.com
Accept-encoding: gzip
Accept: */*
User-Agent: URL/Emacs Emacs/26.2 (X11; x86_64-pc-linux-gnu)
Cookie: ClientId=8998C5691CD143E784857A0D01537963; OIDC=1


retrieval -> Spinning in url-retrieve-synchronously: nil (#<buffer
*http outlook.office365.com:443*>)
http -> Calling after change function
`url-http-wait-for-headers-change-function' for `#<process
outlook.office365.com>'
http -> url-http-wait-for-headers-change-function ( *http
outlook.office365.com:443*)
http -> Saw end of headers... ( *http outlook.office365.com:443*)
http -> url-http-parse-response called in ( *http outlook.office365.com:443*)
http -> Got a content-length, being smart about document end.
http -> Got 0-length content-length, activating callback immediately.
http -> Marking connection as free: outlook.office365.com:443
#<process outlook.office365.com>
http -> url-http-parse-headers called in ( *http outlook.office365.com:443*)
http -> url-http-parse-response called in ( *http outlook.office365.com:443*)
http -> Parsed HTTP headers: class=4 status=401
http -> Handling normal authentication
http -> Contacting host: outlook.office365.com:443
http -> Marking connection as busy: outlook.office365.com:443
#<process outlook.office365.com>
http -> Finished parsing HTTP headers: nil
http -> Spinning waiting for headers...
retrieval -> Spinning in url-retrieve-synchronously: nil (#<buffer
*http outlook.office365.com:443*>)
http -> Calling after change function
`url-https-proxy-after-change-function' for `#<process
outlook.office365.com>'
http -> url-http-parse-response called in ( *http
outlook.office365.com:443*-229815)
http -> Request is:
GET https://outlook.office365.com/EWS/Exchange.asmx HTTP/1.1
MIME-Version: 1.0
Connection: close
Extension: Security/Digest Security/SSL
Host: outlook.office365.com
Accept-encoding: gzip
Accept: */*
User-Agent: URL/Emacs Emacs/26.2 (X11; x86_64-pc-linux-gnu)
Cookie: ClientId=8998C5691CD143E784857A0D01537963; OIDC=1
Authorization: Basic ZGF2aWQuYy5kYXlAaHAuY29tOmwxaTNuN2VTa3kxMzcwJSU=


http -> Calling after change function
`url-http-wait-for-headers-change-function' for `#<process
outlook.office365.com>'
http -> url-http-wait-for-headers-change-function ( *http
outlook.office365.com:443*-229815)
http -> Saw end of headers... ( *http outlook.office365.com:443*-229815)
http -> url-http-parse-response called in ( *http
outlook.office365.com:443*-229815)
http -> Got a content-length, being smart about document end.
http -> Calling initial content-length for extra data at end of headers
http -> Marking connection as free: outlook.office365.com:443
#<process outlook.office365.com>
http -> url-http-parse-headers called in ( *http
outlook.office365.com:443*-229815)
http -> url-http-parse-response called in ( *http
outlook.office365.com:443*-229815)
http -> Parsed HTTP headers: class=4 status=400
http -> Finished parsing HTTP headers: t
http -> Marking connection as free: outlook.office365.com:443
#<process outlook.office365.com>
http -> Activating callback in buffer ( *http
outlook.office365.com:443*-229815): #[128
"\302\303\304p#\210\300\305\240\210\301p\240\207" [(t) (#<buffer
*http outlook.office365.com:443*>) url-debug retrieval "Synchronous
fetching done (%S)" t] 5 "

(fn &rest IGNORED)"] ((:error (error http 400) :peer (:certificate
(:version 3 :serial-number
"04:78:00:ec:6e:d6:46:74:40:83:3c:83:58:2c:0c:eb" :issuer
"C=US,O=DigiCert Inc,CN=DigiCert Cloud Services CA-1" :valid-from
"2018-11-19" :valid-to "2020-11-19" :subject
"C=US,ST=Washington,L=Redmond,O=Microsoft Corporation,CN=outlook.com"
:public-key-algorithm "RSA" :certificate-security-level "Medium"
:signature-algorithm "RSA-SHA256" :public-key-id
"sha1:18:91:f8:bd:ec:41:11:cd:a8:c8:d0:25:00:0d:a5:e4:10:b4:67:dd"
:certificate-id
"sha1:f2:1d:79:fd:e3:61:5c:02:2d:48:62:fd:cd:87:b8:6a:d6:60:b6:06")
:key-exchange "ECDHE-RSA" :protocol "TLS1.2" :cipher "AES-256-GCM"
:mac "AEAD")))
retrieval -> Synchronous fetching done (#<buffer  *http
outlook.office365.com:443*-229815>)
http -> Spinning waiting for headers...

But - just for reference - here is what is in the file I receive when
I use wget:

 wget --user me <at> work --password xxxxxxxxxxx
https://outlook.office365.com/EWS/Exchange.asmx

<HTML><HEAD><link rel="alternate" type="text/xml"
href="https://tu4pr8401mb1231.namprd84.prod.outlook.com:444/EWS/Exchange.asmx?disco"/><STYLE
type="text/css">#content{ FONT-SIZE: 0.7em; PADDING-BOTTOM: 2em;
MARGIN-LEFT: 30px}BODY{MARGIN-TOP: 0px; MARGIN-LEFT: 0px; COLOR:
#000000; FONT-FAMILY: Verdana; BACKGROUND-COLOR: white}P{MARGIN-TOP:
0px; MARGIN-BOTTOM: 12px; COLOR: #000000; FONT-FAMILY:
Verdana}PRE{BORDER-RIGHT: #f0f0e0 1px solid; PADDING-RIGHT: 5px;
BORDER-TOP: #f0f0e0 1px solid; MARGIN-TOP: -5px; PADDING-LEFT: 5px;
FONT-SIZE: 1.2em; PADDING-BOTTOM: 5px; BORDER-LEFT: #f0f0e0 1px solid;
PADDING-TOP: 5px; BORDER-BOTTOM: #f0f0e0 1px solid; FONT-FAMILY:
Courier New; BACKGROUND-COLOR: #e5e5cc}.heading1{MARGIN-TOP: 0px;
PADDING-LEFT: 15px; FONT-WEIGHT: normal; FONT-SIZE: 26px;
MARGIN-BOTTOM: 0px; PADDING-BOTTOM: 3px; MARGIN-LEFT: -30px; WIDTH:
100%; COLOR: #ffffff; PADDING-TOP: 10px; FONT-FAMILY: Tahoma;
BACKGROUND-COLOR: #003366}.intro{MARGIN-LEFT:
-15px}</STYLE><TITLE>Service</TITLE></HEAD><BODY><DIV id="content"><P
class="heading1">Service</P><BR/><P class="intro">You have created a
service.<P class='intro'>To test this service, you will need to create
a client and use it to call the service. You can do this using the
svcutil.exe tool from the command line with the following syntax:</P>
<BR/><PRE>svcutil.exe <A
HREF="https://tu4pr8401mb1231.namprd84.prod.outlook.com:444/EWS/Services.wsdl">https://tu4pr8401mb1231.namprd84.prod.outlook.com:444/EWS/Services.wsdl</A></PRE></P><P
class="intro"/>This will generate a configuration file and a code file
that contains the client class. Add the two files to your client
application and use the generated client class to call the Service.
For example:<BR/><P class='intro'><B>C#</B></P><PRE><font
color="blue">class </font><font color="teal">Test
</font>{
<font color="blue">    static void </font>Main()
    {
        <font color="teal">HelloClient</font> client = <font
color="blue">new </font><font color="teal">HelloClient</font>();

<font color="green">        // Use the 'client' variable to call
operations on the service.

</font><font color="green">        // Always close the client.
</font>        client.Close();
    }
}
</PRE><BR/><P class='intro'><B>Visual Basic</B></P><PRE><font
color="blue">Class </font><font color="teal">Test
</font><font color="blue">    Shared Sub </font>Main()
<font color="blue">        Dim </font>client As <font
color="teal">HelloClient</font> = <font color="blue">New </font><font
color="teal">HelloClient</font>()
<font color="green">        ' Use the 'client' variable to call
operations on the service.

</font><font color="green">        ' Always close the client.
</font>        client.Close()
<font color="blue">    End Sub

Thank you for taking a look and helping me learn some debug tricks.

-C

PS - Sorry for sending to you twice, I should have hit reply to all to
put this in the bug tracker....

On Tue, May 28, 2019 at 7:04 PM Thomas Fitzsimmons <fitzsim <at> fitzsim.org> wrote:
>
> Hi,
>
> "tenspd137 ." <dcday137 <at> gmail.com> writes:
>
> > I was trying to use Excorporate through a proxy, and it fails.  I am
> > pretty sure that my configuration is correct because I am able to
> > retrieve *a* file when I use wget and https://server/EWS/Exchange.asmx
> > and I can get a Services.wsdl file with
> > https://server/EWS/Exchange.asmx.
>
> Thanks for reporting.  Is just Emacs (not Excorporate) able to retrieve
> the file through the proxy?  That is, starting from an "emacs -Q"
> session, can you try just:
>
> M-: (setq url-debug t)
> M-: (url-retrieve-synchronously "https://server/EWS/Exchange.asmx")
>
> and see what the *URL-DEBUG* and " *http server:443*" (note leading
> space character) buffers contain afterwards?
>
> Thanks,
> Thomas

Message #14 received at 35969 <at> debbugs.gnu.org (full text, mbox):

From: Thomas Fitzsimmons <fitzsim <at> fitzsim.org>
To: "tenspd137 ." <dcday137 <at> gmail.com>
Cc: 35969 <at> debbugs.gnu.org
Subject: Re: bug#35969: 26.2, Excorporate
Date: Wed, 29 May 2019 13:02:23 -0400

Hi,

"tenspd137 ." <dcday137 <at> gmail.com> writes:

> Hi - thanks for the reply.  Here are the steps I took (I had to
> configure the proxy as well....):
>
> M-: (setq url-proxy-services '(("http" . "web-proxy:8080") ("https" . "web-proxy:8080")))
> M-: (setq url-debug t)
> M-: (url-retrieve-synchronously "https://server/EWS/Exchange.asmx")
>
> -Asks for name and password-, I fill them in.....

OK, I think the buffer contents you pasted are fine/expected; it's the
web site saying you're unauthorized, which then causes "url" to prompt
for credentials.  That part seems to have worked through the proxies.

Then the url library created a second buffer:

" *http outlook.office365.com:443*-229815"

which I'm inferring from the logs contained "400 Bad Request" headers.
I'm not sure why that might be; I haven't tried Emacs's url proxy
support before.  That said, I'd like to keep trying to fix this since
I'd like Excorporate to work via proxies.

You could try an unauthenticated site to see if that works through the
proxies, e.g.:

[same initial steps]
M-: (url-retrieve-synchronously "https://www.gnu.org/")

and then a different authenticated site, like this test page:

M-: (url-retrieve-synchronously "https://httpbin.org/basic-auth/user/passwd")

Username: user
Password: passwd

If that all works, then we'll have to figure out why url proxy support
is not working with the Exchange server specifically.

[...]

> But - just for reference - here is what is in the file I receive when
> I use wget:
>
>  wget --user me <at> work --password xxxxxxxxxxx https://outlook.office365.com/EWS/Exchange.asmx
>
> [successful output]

Are you configuring wget to use the same proxies as in the Emacs
attempt, via the http_proxy and https_proxy environment variables?

Thanks,
Thomas

Message #17 received at 35969 <at> debbugs.gnu.org (full text, mbox):

From: "tenspd137 ." <dcday137 <at> gmail.com>
To: Thomas Fitzsimmons <fitzsim <at> fitzsim.org>
Cc: 35969 <at> debbugs.gnu.org
Subject: Re: bug#35969: 26.2, Excorporate
Date: Wed, 29 May 2019 12:13:46 -0600

Thomas -

For the unauthenticated site, the buffer " *http www.gnu.org:443*" returns

HTTP/1.1 200 OK
Date: Wed, 29 May 2019 17:31:43 GMT
Server: Apache/2.4.7
Content-Location: home.html
.....
[successfuol output]
.....

and

URL-DEBUG

http -> Contacting host: www.gnu.org:443
http -> Marking connection as busy: www.gnu.org:443 #<process www.gnu.org>
retrieval -> Spinning in url-retrieve-synchronously: nil (#<buffer
*http www.gnu.org:443*>)
retrieval -> Spinning in url-retrieve-synchronously: nil (#<buffer
*http www.gnu.org:443*>)
http -> Calling after change function
`url-https-proxy-after-change-function' for `#<process www.gnu.org>'
http -> url-http-parse-response called in ( *http www.gnu.org:443*)
http -> Request is:
GET https://www.gnu.org/ HTTP/1.1
MIME-Version: 1.0
Connection: close
Extension: Security/Digest Security/SSL
Host: www.gnu.org
Accept-encoding: gzip
Accept: */*
User-Agent: URL/Emacs Emacs/26.2 (X11; x86_64-pc-linux-gnu)


retrieval -> Spinning in url-retrieve-synchronously: nil (#<buffer
*http www.gnu.org:443*>)
http -> Calling after change function
`url-http-wait-for-headers-change-function' for `#<process
www.gnu.org>'
http -> url-http-wait-for-headers-change-function ( *http www.gnu.org:443*)
http -> Saw end of headers... ( *http www.gnu.org:443*)
http -> url-http-parse-response called in ( *http www.gnu.org:443*)
http -> Got a content-length, being smart about document end.
http -> Calling initial content-length for extra data at end of headers
http -> Spinning waiting for headers...
http -> Calling after change function
`url-http-content-length-after-change-function' for `#<process
www.gnu.org>'
http -> Calling after change function
`url-http-content-length-after-change-function' for `#<process
www.gnu.org>'
http -> Calling after change function
`url-http-content-length-after-change-function' for `#<process
www.gnu.org>'
http -> Calling after change function
`url-http-content-length-after-change-function' for `#<process
www.gnu.org>'
http -> Calling after change function
`url-http-content-length-after-change-function' for `#<process
www.gnu.org>'
http -> Marking connection as free: www.gnu.org:443 #<process www.gnu.org>
http -> url-http-parse-headers called in ( *http www.gnu.org:443*)
http -> url-http-parse-response called in ( *http www.gnu.org:443*)
http -> Parsed HTTP headers: class=2 status=200
http -> Finished parsing HTTP headers: t
http -> Marking connection as free: www.gnu.org:443 #<process www.gnu.org>
http -> Activating callback in buffer ( *http www.gnu.org:443*): #[128
"\302\303\304p#\210\300\305\240\210\301p\240\207" [(nil) (#<buffer
*http www.gnu.org:443*>) url-debug retrieval "Synchronous fetching
done (%S)" t] 5 "

(fn &rest IGNORED)"] ((:peer (:certificate (:version 3 :serial-number
"03:da:73:db:ff:c6:23:a4:16:f3:45:6f:fe:05:8e:04:b8:c2" :issuer
"C=US,O=Let's Encrypt,CN=Let's Encrypt Authority X3" :valid-from
"2019-04-02" :valid-to "2019-07-01" :subject "CN=emacs.org"
:public-key-algorithm "RSA" :certificate-security-level "Medium"
:signature-algorithm "RSA-SHA256" :public-key-id
"sha1:89:0c:f7:08:4e:65:16:0b:43:08:43:eb:e3:33:77:fa:fb:1c:a5:70"
:certificate-id
"sha1:0d:61:97:26:9a:58:d7:04:de:52:1f:29:ca:45:55:ec:67:a9:b1:60")
:key-exchange "ECDHE-RSA" :protocol "TLS1.2" :cipher "AES-128-GCM"
:mac "AEAD")))
retrieval -> Synchronous fetching done (#<buffer  *http www.gnu.org:443*>)

The authenticated site buffer " *http httpbin.org:443" returns
HTTP/1.1 401 UNAUTHORIZED
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Date: Wed, 29 May 2019 17:38:27 GMT
Referrer-Policy: no-referrer-when-downgrade
Server: nginx
WWW-Authenticate: Basic realm="Fake Realm"
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block
Content-Length: 0
Connection: Close

and the URL-DEBUG buffer gives
http -> Contacting host: httpbin.org:443
http -> Marking connection as busy: httpbin.org:443 #<process httpbin.org>
retrieval -> Spinning in url-retrieve-synchronously: nil (#<buffer
*http httpbin.org:443*>)
http -> Calling after change function
`url-https-proxy-after-change-function' for `#<process httpbin.org>'
http -> url-http-parse-response called in ( *http httpbin.org:443*)
http -> Request is:
GET https://httpbin.org/basic-auth/user/passwd HTTP/1.1
MIME-Version: 1.0
Connection: close
Extension: Security/Digest Security/SSL
Host: httpbin.org
Accept-encoding: gzip
Accept: */*
User-Agent: URL/Emacs Emacs/26.2 (X11; x86_64-pc-linux-gnu)


retrieval -> Spinning in url-retrieve-synchronously: nil (#<buffer
*http httpbin.org:443*>)
http -> Calling after change function
`url-http-wait-for-headers-change-function' for `#<process
httpbin.org>'
http -> url-http-wait-for-headers-change-function ( *http httpbin.org:443*)
http -> Saw end of headers... ( *http httpbin.org:443*)
http -> url-http-parse-response called in ( *http httpbin.org:443*)
http -> Got a content-length, being smart about document end.
http -> Got 0-length content-length, activating callback immediately.
http -> Marking connection as free: httpbin.org:443 #<process httpbin.org>
http -> url-http-parse-headers called in ( *http httpbin.org:443*)
http -> url-http-parse-response called in ( *http httpbin.org:443*)
http -> Parsed HTTP headers: class=4 status=401
http -> Handling normal authentication
http -> Contacting host: httpbin.org:443
http -> Marking connection as busy: httpbin.org:443 #<process httpbin.org>
http -> Finished parsing HTTP headers: nil
http -> Spinning waiting for headers...
retrieval -> Spinning in url-retrieve-synchronously: nil (#<buffer
*http httpbin.org:443*>)
http -> Calling after change function
`url-https-proxy-after-change-function' for `#<process httpbin.org>'
http -> url-http-parse-response called in ( *http httpbin.org:443*-54346)
http -> Request is:
GET https://httpbin.org/basic-auth/user/passwd HTTP/1.1
MIME-Version: 1.0
Connection: close
Extension: Security/Digest Security/SSL
Host: httpbin.org
Accept-encoding: gzip
Accept: */*
User-Agent: URL/Emacs Emacs/26.2 (X11; x86_64-pc-linux-gnu)
Authorization: Basic dXNlcjpwYXNzd2Q=


http -> Calling after change function
`url-http-wait-for-headers-change-function' for `#<process
httpbin.org>'
http -> url-http-wait-for-headers-change-function ( *http
httpbin.org:443*-54346)
http -> Saw end of headers... ( *http httpbin.org:443*-54346)
http -> url-http-parse-response called in ( *http httpbin.org:443*-54346)
http -> Got a content-length, being smart about document end.
http -> Calling initial content-length for extra data at end of headers
http -> Marking connection as free: httpbin.org:443 #<process httpbin.org>
http -> url-http-parse-headers called in ( *http httpbin.org:443*-54346)
http -> url-http-parse-response called in ( *http httpbin.org:443*-54346)
http -> Parsed HTTP headers: class=2 status=200
http -> Finished parsing HTTP headers: t
http -> Marking connection as free: httpbin.org:443 #<process httpbin.org>
http -> Activating callback in buffer ( *http httpbin.org:443*-54346):
#[128 "\302\303\304p#\210\300\305\240\210\301p\240\207" [(t) (#<buffer
 *http httpbin.org:443*>) url-debug retrieval "Synchronous fetching
done (%S)" t] 5 "

(fn &rest IGNORED)"] ((:peer (:certificate (:version 3 :serial-number
"06:0c:5d:3e:f4:d0:2e:f5:c1:ac:09:8e:b3:0e:39:ab" :issuer
"C=US,O=Amazon,OU=Server CA 1B,CN=Amazon" :valid-from "2019-02-17"
:valid-to "2020-03-17" :subject "CN=httpbin.org" :public-key-algorithm
"RSA" :certificate-security-level "Medium" :signature-algorithm
"RSA-SHA256" :public-key-id
"sha1:1b:e3:b6:57:70:0a:15:df:dc:05:20:f2:66:da:8e:2f:74:4b:e8:ed"
:certificate-id
"sha1:cd:9a:4d:21:af:57:65:e6:d3:be:3a:d0:1c:dc:88:26:b8:96:bb:f7")
:key-exchange "ECDHE-RSA" :protocol "TLS1.2" :cipher "AES-128-GCM"
:mac "AEAD")))
retrieval -> Synchronous fetching done (#<buffer  *http httpbin.org:443*-54346>)
http -> Spinning waiting for headers...

Just as a check:
echo $http_proxy
http://web-proxy:8080

echo $https_proxy
http://web-proxy:8080

using wget, I get a file with the following contents:
 wget --user user --password passwd https://httpbin.org/basic-auth/user/passwd
--2019-05-29 11:46:25--  https://httpbin.org/basic-auth/user/passwd
Resolving web-proxy... xx.xx.xxx.xxx
Connecting to web-proxy|xx.xx.xxx.xxx]:8080... connected.
Proxy request sent, awaiting response... 401 UNAUTHORIZED
Authentication selected: Basic realm="Fake Realm"
Reusing existing connection to httpbin.org:443.
Proxy request sent, awaiting response... 200 OK
Length: 47 [application/json]
Saving to: ‘passwd’

passwd
100%[======================================================================================================================>]
     47  --.-KB/s    in 0s

2019-05-29 11:46:26 (3.59 MB/s) - ‘passwd’ saved [47/47]


cat passwd
{
  "authenticated": true,
  "user": "user"
}

Thanks!

-C

}


On Wed, May 29, 2019 at 11:02 AM Thomas Fitzsimmons <fitzsim <at> fitzsim.org> wrote:
>
> Hi,
>
> "tenspd137 ." <dcday137 <at> gmail.com> writes:
>
> > Hi - thanks for the reply.  Here are the steps I took (I had to
> > configure the proxy as well....):
> >
> > M-: (setq url-proxy-services '(("http" . "web-proxy:8080") ("https" . "web-proxy:8080")))
> > M-: (setq url-debug t)
> > M-: (url-retrieve-synchronously "https://server/EWS/Exchange.asmx")
> >
> > -Asks for name and password-, I fill them in.....
>
> OK, I think the buffer contents you pasted are fine/expected; it's the
> web site saying you're unauthorized, which then causes "url" to prompt
> for credentials.  That part seems to have worked through the proxies.
>
> Then the url library created a second buffer:
>
> " *http outlook.office365.com:443*-229815"
>
> which I'm inferring from the logs contained "400 Bad Request" headers.
> I'm not sure why that might be; I haven't tried Emacs's url proxy
> support before.  That said, I'd like to keep trying to fix this since
> I'd like Excorporate to work via proxies.
>
> You could try an unauthenticated site to see if that works through the
> proxies, e.g.:
>
> [same initial steps]
> M-: (url-retrieve-synchronously "https://www.gnu.org/")
>
> and then a different authenticated site, like this test page:
>
> M-: (url-retrieve-synchronously "https://httpbin.org/basic-auth/user/passwd")
>
> Username: user
> Password: passwd
>
> If that all works, then we'll have to figure out why url proxy support
> is not working with the Exchange server specifically.
>
> [...]
>
> > But - just for reference - here is what is in the file I receive when
> > I use wget:
> >
> >  wget --user me <at> work --password xxxxxxxxxxx https://outlook.office365.com/EWS/Exchange.asmx
> >
> > [successful output]
>
> Are you configuring wget to use the same proxies as in the Emacs
> attempt, via the http_proxy and https_proxy environment variables?
>
> Thanks,
> Thomas

Message #20 received at 35969 <at> debbugs.gnu.org (full text, mbox):

From: Thomas Fitzsimmons <fitzsim <at> fitzsim.org>
To: "tenspd137 ." <dcday137 <at> gmail.com>
Cc: 35969 <at> debbugs.gnu.org
Subject: Re: bug#35969: 26.2, Excorporate
Date: Wed, 29 May 2019 16:53:58 -0400

Hi,

"tenspd137 ." <dcday137 <at> gmail.com> writes:

> For the unauthenticated site, the buffer " *http www.gnu.org:443*" returns
>
> HTTP/1.1 200 OK
> Date: Wed, 29 May 2019 17:31:43 GMT
> Server: Apache/2.4.7
> Content-Location: home.html
> .....
> [successful output]
[...]

> The authenticated site buffer " *http httpbin.org:443" returns

[...]

> http -> Parsed HTTP headers: class=2 status=200

OK, success through the proxy (both in Emacs and wget) for the simple no
and basic authentication cases, interesting.

Are you in a position to test authenticating against
"https://server/EWS/Exchange.asmx" from within Emacs, without going
through the proxy?  (I realize this may not be possible in your setup.)

I guess I'll have to set up a localhost proxy to debug this further.

Thomas

Message #23 received at 35969 <at> debbugs.gnu.org (full text, mbox):

From: "tenspd137 ." <dcday137 <at> gmail.com>
To: Thomas Fitzsimmons <fitzsim <at> fitzsim.org>
Cc: 35969 <at> debbugs.gnu.org
Subject: Re: bug#35969: 26.2, Excorporate
Date: Wed, 29 May 2019 15:44:25 -0600

Sorry for questioning - but I don't think it is successful in the
basic authentication case - unless I am misreading some of the output.

As for testing authentication against no proxy - yes, I can.  In fact,
I actually have.  It works.  At work we have a general office lan and
a development lan.  The bulk of my work is on the development lan,
which is where I spend most my time.  The office lan is less
restricted proxy wise, but I don't use it for various reasons.  I took
a laptop (running M$ no less) and tried running emacs / excorporate
with the same settings minus the proxy setup and had no problem, so it
works, just wrong computer.  I would be happy to repeat the same steps
and grab you some log files if you like, but it might have to wait
until tomorrow morning.  Sorry I am no good at debugging this stuff
myself, but I really don't know what is even supposed to happen under
the hood.  I probably should have thought of comparing log files
myself earlier.  Basically, I am trying to condense all of my stuff to
the computer I use most so I don't have to constantly switch back and
forth.

Again, thanks for your time!

-C

On Wed, May 29, 2019 at 2:54 PM Thomas Fitzsimmons <fitzsim <at> fitzsim.org> wrote:
>
> Hi,
>
> "tenspd137 ." <dcday137 <at> gmail.com> writes:
>
> > For the unauthenticated site, the buffer " *http www.gnu.org:443*" returns
> >
> > HTTP/1.1 200 OK
> > Date: Wed, 29 May 2019 17:31:43 GMT
> > Server: Apache/2.4.7
> > Content-Location: home.html
> > .....
> > [successful output]
> [...]
>
> > The authenticated site buffer " *http httpbin.org:443" returns
>
> [...]
>
> > http -> Parsed HTTP headers: class=2 status=200
>
> OK, success through the proxy (both in Emacs and wget) for the simple no
> and basic authentication cases, interesting.
>
> Are you in a position to test authenticating against
> "https://server/EWS/Exchange.asmx" from within Emacs, without going
> through the proxy?  (I realize this may not be possible in your setup.)
>
> I guess I'll have to set up a localhost proxy to debug this further.
>
> Thomas

Message #26 received at 35969 <at> debbugs.gnu.org (full text, mbox):

From: Thomas Fitzsimmons <fitzsim <at> fitzsim.org>
To: "tenspd137 ." <dcday137 <at> gmail.com>
Cc: 35969 <at> debbugs.gnu.org
Subject: Re: bug#35969: 26.2, Excorporate
Date: Wed, 29 May 2019 18:01:31 -0400

Hi,

"tenspd137 ." <dcday137 <at> gmail.com> writes:

> Sorry for questioning - but I don't think it is successful in the
> basic authentication case - unless I am misreading some of the output.

No problem; I quoted the "status=200" line which I think indicates that
it was successful.  But url creates another buffer and puts the
authentication response in there; to be sure, you'd need to check that.
I think for httpbin.org, it will contain the headers and then:

{
  "authenticated": true,
  "user": "user"
}

just like your wget experiment.

For your prior test run, the secondary buffer is mentioned in
*URL-DEBUG* here:

> retrieval -> Synchronous fetching done (#<buffer  *http httpbin.org:443*-54346>)
                                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Can you redo the httpbin.org authentication test within Emacs, through
the proxies, and paste the contents of that secondary/response buffer?

It would be helpful if you could paste the redacted secondary/response
buffer contents for the failed Exchange authentication too.

> As for testing authentication against no proxy - yes, I can.  In fact,
> I actually have.  It works.

[...]

OK, good to know, thanks.  The use case makes sense.

Thanks,
Thomas

Message #29 received at 35969 <at> debbugs.gnu.org (full text, mbox):

From: "tenspd137 ." <dcday137 <at> gmail.com>
To: Thomas Fitzsimmons <fitzsim <at> fitzsim.org>
Cc: 35969 <at> debbugs.gnu.org
Subject: Re: bug#35969: 26.2, Excorporate
Date: Wed, 29 May 2019 16:10:35 -0600

Yes - I will try to do it tomorrow morning.  Thanks again!

-C

On Wed, May 29, 2019 at 4:01 PM Thomas Fitzsimmons <fitzsim <at> fitzsim.org> wrote:
>
> Hi,
>
> "tenspd137 ." <dcday137 <at> gmail.com> writes:
>
> > Sorry for questioning - but I don't think it is successful in the
> > basic authentication case - unless I am misreading some of the output.
>
> No problem; I quoted the "status=200" line which I think indicates that
> it was successful.  But url creates another buffer and puts the
> authentication response in there; to be sure, you'd need to check that.
> I think for httpbin.org, it will contain the headers and then:
>
> {
>   "authenticated": true,
>   "user": "user"
> }
>
> just like your wget experiment.
>
> For your prior test run, the secondary buffer is mentioned in
> *URL-DEBUG* here:
>
> > retrieval -> Synchronous fetching done (#<buffer  *http httpbin.org:443*-54346>)
>                                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>
> Can you redo the httpbin.org authentication test within Emacs, through
> the proxies, and paste the contents of that secondary/response buffer?
>
> It would be helpful if you could paste the redacted secondary/response
> buffer contents for the failed Exchange authentication too.
>
> > As for testing authentication against no proxy - yes, I can.  In fact,
> > I actually have.  It works.
>
> [...]
>
> OK, good to know, thanks.  The use case makes sense.
>
> Thanks,
> Thomas

Message #32 received at 35969 <at> debbugs.gnu.org (full text, mbox):

From: "tenspd137 ." <dcday137 <at> gmail.com>
To: Thomas Fitzsimmons <fitzsim <at> fitzsim.org>
Cc: 35969 <at> debbugs.gnu.org
Subject: Re: bug#35969: 26.2, Excorporate
Date: Thu, 30 May 2019 11:06:38 -0600

Thomas-

Here is the httpbin.org authentication test buffer (through emacs / proxy)

HTTP/1.1 401 UNAUTHORIZED
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Date: Thu, 30 May 2019 15:32:25 GMT
Referrer-Policy: no-referrer-when-downgrade
Server: nginx
WWW-Authenticate: Basic realm="Fake Realm"
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block
Content-Length: 0
Connection: Close

and here is the failed excahnge.asmx buffer

HTTP/1.1 401 Unauthorized
Server: Microsoft-IIS/10.0
request-id: fb0b957e-f55c-49b9-9fb5-fd9a58a5f31a
X-WSSecurity-Enabled: True
X-WSSecurity-For: Logon
X-FederationTrustTokenIssuerUri: urn:federation:MicrosoftOnline
X-WSSecurity-SymmetricKey-Enabled: True
X-WSSecurity-X509Cert-Enabled: True
X-OAuth-Enabled: True
X-Powered-By: ASP.NET
X-FEServer: SN6PR1501CA0010
WWW-Authenticate: Basic Realm=""
Date: Thu, 30 May 2019 15:36:43 GMT
Connection: close
Content-Length: 0

and, as a bonus, the result of url-exchange.asmx from Emacs not using
a proxy on the "other" lan using url-retrieve-synchronusly

HTTP/1.1 401 Unauthorized
Server: Microsoft-IIS/10.0
request-id: 369ea2d9-8dce-408c-a38e-fb8035f98065
X-WSSecurity-Enabled: True
X-WSSecurity-For: Logon
X-FederationTrustTokenIssuerUri: urn:federation:MicrosoftOnline
X-WSSecurity-SymmetricKey-Enabled: True
X-WSSecurity-X509Cert-Enabled: True
X-OAuth-Enabled: True
X-Powered-By: ASP.NET
X-FEServer: SN4PR0201CA0037
WWW-Authenticate: Basic Realm=""
Date: Thu, 30 May 2019 16:40:48 GMT
Content-Length: 0

I tried to get logs from a successful exorporate useage, but the http
result buffers appear to come and go, and if I try URL-DEBUG, a lot
happens and eventually emacs on my laptop locks up, but if you want me
to try anything specific or there is a way to look at things step by
step, just let me know.

Thanks!

-C



On Wed, May 29, 2019 at 4:10 PM tenspd137 . <dcday137 <at> gmail.com> wrote:
>
> Yes - I will try to do it tomorrow morning.  Thanks again!
>
> -C
>
> On Wed, May 29, 2019 at 4:01 PM Thomas Fitzsimmons <fitzsim <at> fitzsim.org> wrote:
> >
> > Hi,
> >
> > "tenspd137 ." <dcday137 <at> gmail.com> writes:
> >
> > > Sorry for questioning - but I don't think it is successful in the
> > > basic authentication case - unless I am misreading some of the output.
> >
> > No problem; I quoted the "status=200" line which I think indicates that
> > it was successful.  But url creates another buffer and puts the
> > authentication response in there; to be sure, you'd need to check that.
> > I think for httpbin.org, it will contain the headers and then:
> >
> > {
> >   "authenticated": true,
> >   "user": "user"
> > }
> >
> > just like your wget experiment.
> >
> > For your prior test run, the secondary buffer is mentioned in
> > *URL-DEBUG* here:
> >
> > > retrieval -> Synchronous fetching done (#<buffer  *http httpbin.org:443*-54346>)
> >                                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> >
> > Can you redo the httpbin.org authentication test within Emacs, through
> > the proxies, and paste the contents of that secondary/response buffer?
> >
> > It would be helpful if you could paste the redacted secondary/response
> > buffer contents for the failed Exchange authentication too.
> >
> > > As for testing authentication against no proxy - yes, I can.  In fact,
> > > I actually have.  It works.
> >
> > [...]
> >
> > OK, good to know, thanks.  The use case makes sense.
> >
> > Thanks,
> > Thomas

Message #35 received at 35969 <at> debbugs.gnu.org (full text, mbox):

From: "tenspd137 ." <dcday137 <at> gmail.com>
To: Thomas Fitzsimmons <fitzsim <at> fitzsim.org>
Cc: 35969 <at> debbugs.gnu.org
Subject: Re: bug#35969: 26.2, Excorporate
Date: Thu, 30 May 2019 11:25:39 -0600

So, I don't know what this is indicative of, but if I try to use emacs
through a proxy and use the httpbin.org example, the result buffer
shows:

HTTP/1.1 401 UNAUTHORIZED
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Date: Thu, 30 May 2019 17:22:48 GMT
Referrer-Policy: no-referrer-when-downgrade
Server: nginx
WWW-Authenticate: Basic realm="Fake Realm"
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block
Content-Length: 0
Connection: Close

but then if I repeat the url-retrieve-synchrounously call

HTTP/1.1 200 OK
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Content-Encoding: gzip
Content-Type: application/json
Date: Thu, 30 May 2019 17:23:59 GMT
Referrer-Policy: no-referrer-when-downgrade
Server: nginx
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block
Content-Length: 59
Connection: Close

{
  "authenticated": true,
  "user": "user"
}

So - it need two calls - one to authenticate and one to retrieve?
Thanks!
-C

On Thu, May 30, 2019 at 11:06 AM tenspd137 . <dcday137 <at> gmail.com> wrote:
>
> Thomas-
>
> Here is the httpbin.org authentication test buffer (through emacs / proxy)
>
> HTTP/1.1 401 UNAUTHORIZED
> Access-Control-Allow-Credentials: true
> Access-Control-Allow-Origin: *
> Date: Thu, 30 May 2019 15:32:25 GMT
> Referrer-Policy: no-referrer-when-downgrade
> Server: nginx
> WWW-Authenticate: Basic realm="Fake Realm"
> X-Content-Type-Options: nosniff
> X-Frame-Options: DENY
> X-XSS-Protection: 1; mode=block
> Content-Length: 0
> Connection: Close
>
> and here is the failed excahnge.asmx buffer
>
> HTTP/1.1 401 Unauthorized
> Server: Microsoft-IIS/10.0
> request-id: fb0b957e-f55c-49b9-9fb5-fd9a58a5f31a
> X-WSSecurity-Enabled: True
> X-WSSecurity-For: Logon
> X-FederationTrustTokenIssuerUri: urn:federation:MicrosoftOnline
> X-WSSecurity-SymmetricKey-Enabled: True
> X-WSSecurity-X509Cert-Enabled: True
> X-OAuth-Enabled: True
> X-Powered-By: ASP.NET
> X-FEServer: SN6PR1501CA0010
> WWW-Authenticate: Basic Realm=""
> Date: Thu, 30 May 2019 15:36:43 GMT
> Connection: close
> Content-Length: 0
>
> and, as a bonus, the result of url-exchange.asmx from Emacs not using
> a proxy on the "other" lan using url-retrieve-synchronusly
>
> HTTP/1.1 401 Unauthorized
> Server: Microsoft-IIS/10.0
> request-id: 369ea2d9-8dce-408c-a38e-fb8035f98065
> X-WSSecurity-Enabled: True
> X-WSSecurity-For: Logon
> X-FederationTrustTokenIssuerUri: urn:federation:MicrosoftOnline
> X-WSSecurity-SymmetricKey-Enabled: True
> X-WSSecurity-X509Cert-Enabled: True
> X-OAuth-Enabled: True
> X-Powered-By: ASP.NET
> X-FEServer: SN4PR0201CA0037
> WWW-Authenticate: Basic Realm=""
> Date: Thu, 30 May 2019 16:40:48 GMT
> Content-Length: 0
>
> I tried to get logs from a successful exorporate useage, but the http
> result buffers appear to come and go, and if I try URL-DEBUG, a lot
> happens and eventually emacs on my laptop locks up, but if you want me
> to try anything specific or there is a way to look at things step by
> step, just let me know.
>
> Thanks!
>
> -C
>
>
>
> On Wed, May 29, 2019 at 4:10 PM tenspd137 . <dcday137 <at> gmail.com> wrote:
> >
> > Yes - I will try to do it tomorrow morning.  Thanks again!
> >
> > -C
> >
> > On Wed, May 29, 2019 at 4:01 PM Thomas Fitzsimmons <fitzsim <at> fitzsim.org> wrote:
> > >
> > > Hi,
> > >
> > > "tenspd137 ." <dcday137 <at> gmail.com> writes:
> > >
> > > > Sorry for questioning - but I don't think it is successful in the
> > > > basic authentication case - unless I am misreading some of the output.
> > >
> > > No problem; I quoted the "status=200" line which I think indicates that
> > > it was successful.  But url creates another buffer and puts the
> > > authentication response in there; to be sure, you'd need to check that.
> > > I think for httpbin.org, it will contain the headers and then:
> > >
> > > {
> > >   "authenticated": true,
> > >   "user": "user"
> > > }
> > >
> > > just like your wget experiment.
> > >
> > > For your prior test run, the secondary buffer is mentioned in
> > > *URL-DEBUG* here:
> > >
> > > > retrieval -> Synchronous fetching done (#<buffer  *http httpbin.org:443*-54346>)
> > >                                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > >
> > > Can you redo the httpbin.org authentication test within Emacs, through
> > > the proxies, and paste the contents of that secondary/response buffer?
> > >
> > > It would be helpful if you could paste the redacted secondary/response
> > > buffer contents for the failed Exchange authentication too.
> > >
> > > > As for testing authentication against no proxy - yes, I can.  In fact,
> > > > I actually have.  It works.
> > >
> > > [...]
> > >
> > > OK, good to know, thanks.  The use case makes sense.
> > >
> > > Thanks,
> > > Thomas

Message #40 received at 35969 <at> debbugs.gnu.org (full text, mbox):

From: "tenspd137 ." <dcday137 <at> gmail.com>
To: Thomas Fitzsimmons <fitzsim <at> fitzsim.org>, 35969 <at> debbugs.gnu.org
Subject: Re: bug#35969: 26.2, Excorporate
Date: Fri, 14 Jun 2019 14:13:46 -0600

Thomas -

I was able to try stepping through an Emacs/proxy/Exchange test in an
emacs -Q session.  After setting the proxy and configuring the
debugger to step through url-http and url-http-async-sentilnel, the
only thing I noticed is that it appears url-http-async-sentinel is not
being called.  I also put the url-https-proxy-connect override you
gave me earlier into a file, loaded it and set the debugger to run
through that as well as set up proxies.  A broken down list of steps:

1.  Load file containing proxy and altered url-https-proxyconnect, set
debugger to run through it when hit.
2.  set up url-http and url-http-async-sentinel to be picked up by debugger
3.  eval (url-retrieve-synchronously
"https://outlook.office365.com/EWS/Exchange.asmx"), step through
url-http
4.  Input username and password when asked
5.  Continue stepping until end

url-http-async-sentinel is never called. " *http* ... -####" has text
indicating failure. url-https-proxy-connect is indeed called. I don't
know how to look at the actual <process>, if I try to C-x C-e  or M-:
"connection", it goes into the debugger.  The url's, etc look good as
far as I can tell.  Not sure what else I can do.  If there are certain
pieces of url-http you want me to look at, just let me know, but not
really knowing what has to happen under the hood, I am not going to be
able to do much else.

Thanks!

-C



On Tue, Jun 4, 2019 at 4:28 PM Thomas Fitzsimmons <fitzsim <at> fitzsim.org> wrote:
>
> "tenspd137 ." <dcday137 <at> gmail.com> writes:
>
> > Oh - I see.  Because you want to look at the *HEADERS* flying around
> > in that case.  Understood.
>
> Right.
>
> > Quick question - using emacs -Q on my proxied machine
> >
> > (setq url-http-proxy "myproxy") doesn't seem to work, so I have been just doing:
> >
> > (setq url-proxy-services '(("http" . "myproxy")
> >                ("https" . "myproxy")))
> >
> > then, I evaluate:
> >
> > (defun url-https-proxy-connect (connection)
> >   (setq url-http-after-change-function 'url-https-proxy-after-change-function)
> >   (message "THIS WAS CALLED: url-https-proxy-connect")
> >   (process-send-string connection (format (concat "CONNECT %s:%d HTTP/1.1\r\n"
> >                                                   "Host: %s\r\n"
> >                                                   "\r\n")
> >                                           (url-host url-current-object)
> >                                           (or (url-port url-current-object)
> >                                               url-https-default-port)
> >                                           (url-host url-current-object))))
> >
> > which, if I understand correctly, adds the messaging to the orgiinal
> > url-https-proxy-connect.
>
> That's correct.
>
> > Then I evaluate
> >
> > (url-retrieve-synchronously "https://outlook.office365.com/EWS/Exchange.asmx")
> >
> > after sending username and password, THIS WAS CALLED does not appear
> > in the messages.  Am I using it correctly, or is this not being
> > called?  I have to ask, because at this point in my emacs usage, I
> > always bet first that I have done something wrong.
>
> I think you're doing everything correctly, so this suggests that Emacs
> isn't doing any proxy handling, or at least that it is not initiating
> the "CONNECT" protocol.
>
> From the wget output in your other email, it shows wget
> connecting to the proxy with CONNECT.  This protocol keeps the TLS
> "tunnel" through the proxy open, and as far as I know, is required for
> Exchange authentication to work through a proxy.
>
> Then the logs also show:
>
> Connection: Keep-Alive
> Proxy-Connection: Keep-Alive
>
> which are important; even when you set Connection keep-alive, Emacs was
> still sending both Connection close and Connection keep-alive which is
> probably not valid.
>
> You may have reached the limit of what Emacs can currently do for
> proxies, but it's up to someone (maybe me) to eventually implement
> CONNECT properly if it isn't already.
>
> A next step would be to repeat your Emacs/proxy/Exchange experiments
> until you get url-https-proxy-connect to be called or figure out why it
> isn't being called.
>
> The only two callers are url-http and url-http-async-sentinel.  To step
> through them with edebug do:
>
> C-h f url-http RET C-x o TAB RET C-u C-M-x
>
> Likewise for url-http-async-sentinel, then redo your experiment and step
> through the functions with SPC and eval stuff with C-x e to see where
> they go off the rails before calling url-https-proxy-connect.
>
> > Still looking into wget verbosity....
>
> I saw the other email, nice; is that output just with the -d option?
>
> BTW, it seems like Emacs is supposed to read the _proxy environment
> variables the way wget does.  Have you tried running emacs -Q in an
> environment that has http_proxy and https_proxy set, and then not
> setting any of the Emacs proxy variables and doing the Exchange
> authentication experiment?
>
> Thanks,
> Thomas

Message #43 received at 35969 <at> debbugs.gnu.org (full text, mbox):

From: "tenspd137 ." <dcday137 <at> gmail.com>
To: Thomas Fitzsimmons <fitzsim <at> fitzsim.org>, 35969 <at> debbugs.gnu.org
Subject: Re: bug#35969: 26.2, Excorporate
Date: Fri, 14 Jun 2019 14:16:05 -0600

*if I try to C-x C-e  or M-: "connection", it goes into the debugger.*

I mean it gives me a backtrace telling me the variable is empty, even
though later in lisp expressions, it is evaluated.  I am assuming it
is roughly equivalent to a C++ pointer, but I don't know how to query
it, etc.

Thanks!

-C

On Fri, Jun 14, 2019 at 2:13 PM tenspd137 . <dcday137 <at> gmail.com> wrote:
>
> Thomas -
>
> I was able to try stepping through an Emacs/proxy/Exchange test in an
> emacs -Q session.  After setting the proxy and configuring the
> debugger to step through url-http and url-http-async-sentilnel, the
> only thing I noticed is that it appears url-http-async-sentinel is not
> being called.  I also put the url-https-proxy-connect override you
> gave me earlier into a file, loaded it and set the debugger to run
> through that as well as set up proxies.  A broken down list of steps:
>
> 1.  Load file containing proxy and altered url-https-proxyconnect, set
> debugger to run through it when hit.
> 2.  set up url-http and url-http-async-sentinel to be picked up by debugger
> 3.  eval (url-retrieve-synchronously
> "https://outlook.office365.com/EWS/Exchange.asmx"), step through
> url-http
> 4.  Input username and password when asked
> 5.  Continue stepping until end
>
> url-http-async-sentinel is never called. " *http* ... -####" has text
> indicating failure. url-https-proxy-connect is indeed called. I don't
> know how to look at the actual <process>, if I try to C-x C-e  or M-:
> "connection", it goes into the debugger.  The url's, etc look good as
> far as I can tell.  Not sure what else I can do.  If there are certain
> pieces of url-http you want me to look at, just let me know, but not
> really knowing what has to happen under the hood, I am not going to be
> able to do much else.
>
> Thanks!
>
> -C
>
>
>
> On Tue, Jun 4, 2019 at 4:28 PM Thomas Fitzsimmons <fitzsim <at> fitzsim.org> wrote:
> >
> > "tenspd137 ." <dcday137 <at> gmail.com> writes:
> >
> > > Oh - I see.  Because you want to look at the *HEADERS* flying around
> > > in that case.  Understood.
> >
> > Right.
> >
> > > Quick question - using emacs -Q on my proxied machine
> > >
> > > (setq url-http-proxy "myproxy") doesn't seem to work, so I have been just doing:
> > >
> > > (setq url-proxy-services '(("http" . "myproxy")
> > >                ("https" . "myproxy")))
> > >
> > > then, I evaluate:
> > >
> > > (defun url-https-proxy-connect (connection)
> > >   (setq url-http-after-change-function 'url-https-proxy-after-change-function)
> > >   (message "THIS WAS CALLED: url-https-proxy-connect")
> > >   (process-send-string connection (format (concat "CONNECT %s:%d HTTP/1.1\r\n"
> > >                                                   "Host: %s\r\n"
> > >                                                   "\r\n")
> > >                                           (url-host url-current-object)
> > >                                           (or (url-port url-current-object)
> > >                                               url-https-default-port)
> > >                                           (url-host url-current-object))))
> > >
> > > which, if I understand correctly, adds the messaging to the orgiinal
> > > url-https-proxy-connect.
> >
> > That's correct.
> >
> > > Then I evaluate
> > >
> > > (url-retrieve-synchronously "https://outlook.office365.com/EWS/Exchange.asmx")
> > >
> > > after sending username and password, THIS WAS CALLED does not appear
> > > in the messages.  Am I using it correctly, or is this not being
> > > called?  I have to ask, because at this point in my emacs usage, I
> > > always bet first that I have done something wrong.
> >
> > I think you're doing everything correctly, so this suggests that Emacs
> > isn't doing any proxy handling, or at least that it is not initiating
> > the "CONNECT" protocol.
> >
> > From the wget output in your other email, it shows wget
> > connecting to the proxy with CONNECT.  This protocol keeps the TLS
> > "tunnel" through the proxy open, and as far as I know, is required for
> > Exchange authentication to work through a proxy.
> >
> > Then the logs also show:
> >
> > Connection: Keep-Alive
> > Proxy-Connection: Keep-Alive
> >
> > which are important; even when you set Connection keep-alive, Emacs was
> > still sending both Connection close and Connection keep-alive which is
> > probably not valid.
> >
> > You may have reached the limit of what Emacs can currently do for
> > proxies, but it's up to someone (maybe me) to eventually implement
> > CONNECT properly if it isn't already.
> >
> > A next step would be to repeat your Emacs/proxy/Exchange experiments
> > until you get url-https-proxy-connect to be called or figure out why it
> > isn't being called.
> >
> > The only two callers are url-http and url-http-async-sentinel.  To step
> > through them with edebug do:
> >
> > C-h f url-http RET C-x o TAB RET C-u C-M-x
> >
> > Likewise for url-http-async-sentinel, then redo your experiment and step
> > through the functions with SPC and eval stuff with C-x e to see where
> > they go off the rails before calling url-https-proxy-connect.
> >
> > > Still looking into wget verbosity....
> >
> > I saw the other email, nice; is that output just with the -d option?
> >
> > BTW, it seems like Emacs is supposed to read the _proxy environment
> > variables the way wget does.  Have you tried running emacs -Q in an
> > environment that has http_proxy and https_proxy set, and then not
> > setting any of the Emacs proxy variables and doing the Exchange
> > authentication experiment?
> >
> > Thanks,
> > Thomas

Message #46 received at 35969 <at> debbugs.gnu.org (full text, mbox):

From: Noam Postavsky <npostavs <at> gmail.com>
To: "tenspd137 ." <dcday137 <at> gmail.com>
Cc: Thomas Fitzsimmons <fitzsim <at> fitzsim.org>, 35969 <at> debbugs.gnu.org
Subject: Re: bug#35969: 26.2, Excorporate
Date: Fri, 14 Jun 2019 16:22:37 -0400

"tenspd137 ." <dcday137 <at> gmail.com> writes:

> *if I try to C-x C-e  or M-: "connection", it goes into the debugger.*
>
> I mean it gives me a backtrace telling me the variable is empty, even
> though later in lisp expressions, it is evaluated.

If you're stepping with debug or edebug, use `e' (bound to
debugger-eval-expression or edebug-eval-expression, respectively) to
evaluate expressions in the context of the code you are stepping
through.

Message #49 received at 35969 <at> debbugs.gnu.org (full text, mbox):

From: "tenspd137 ." <dcday137 <at> gmail.com>
To: Noam Postavsky <npostavs <at> gmail.com>
Cc: Thomas Fitzsimmons <fitzsim <at> fitzsim.org>, 35969 <at> debbugs.gnu.org
Subject: Re: bug#35969: 26.2, Excorporate
Date: Fri, 14 Jun 2019 14:32:54 -0600

Ok - so yes, when I hit e and type "connection", it tells me #<process
outlook.office365.com>, so I must have been doing it out of context.

On Fri, Jun 14, 2019 at 2:22 PM Noam Postavsky <npostavs <at> gmail.com> wrote:
>
> "tenspd137 ." <dcday137 <at> gmail.com> writes:
>
> > *if I try to C-x C-e  or M-: "connection", it goes into the debugger.*
> >
> > I mean it gives me a backtrace telling me the variable is empty, even
> > though later in lisp expressions, it is evaluated.
>
> If you're stepping with debug or edebug, use `e' (bound to
> debugger-eval-expression or edebug-eval-expression, respectively) to
> evaluate expressions in the context of the code you are stepping
> through.

Message #52 received at 35969 <at> debbugs.gnu.org (full text, mbox):

From: "tenspd137 ." <dcday137 <at> gmail.com>
To: Thomas Fitzsimmons <fitzsim <at> fitzsim.org>, 35969 <at> debbugs.gnu.org
Subject: Re: bug#35969: 26.2, Excorporate
Date: Fri, 14 Jun 2019 14:45:16 -0600

One thing I noticed is that after entering my username and password,
the url-current-object did not contain a user or password.  Of course,
it could also be that authentication happens during the prompt and
that the username ans password is not supposed to be saved there for
security reasons.

Either way, let me know if there are certain values you want me to look at.

Thanks!

-C

On Fri, Jun 14, 2019 at 2:32 PM tenspd137 . <dcday137 <at> gmail.com> wrote:
>
> Ok - so yes, when I hit e and type "connection", it tells me #<process
> outlook.office365.com>, so I must have been doing it out of context.
>
> On Fri, Jun 14, 2019 at 2:22 PM Noam Postavsky <npostavs <at> gmail.com> wrote:
> >
> > "tenspd137 ." <dcday137 <at> gmail.com> writes:
> >
> > > *if I try to C-x C-e  or M-: "connection", it goes into the debugger.*
> > >
> > > I mean it gives me a backtrace telling me the variable is empty, even
> > > though later in lisp expressions, it is evaluated.
> >
> > If you're stepping with debug or edebug, use `e' (bound to
> > debugger-eval-expression or edebug-eval-expression, respectively) to
> > evaluate expressions in the context of the code you are stepping
> > through.

Message #55 received at 35969 <at> debbugs.gnu.org (full text, mbox):

From: Thomas Fitzsimmons <fitzsim <at> fitzsim.org>
To: "tenspd137 ." <dcday137 <at> gmail.com>
Cc: 35969 <at> debbugs.gnu.org
Subject: Re: bug#35969: 26.2, Excorporate
Date: Fri, 14 Jun 2019 17:02:02 -0400

Hi,

Thanks for following up with these further test results.

"tenspd137 ." <dcday137 <at> gmail.com> writes:

> I was able to try stepping through an Emacs/proxy/Exchange test in an
> emacs -Q session.  After setting the proxy and configuring the
> debugger to step through url-http and url-http-async-sentilnel, the
> only thing I noticed is that it appears url-http-async-sentinel is not
> being called.

OK, that's probably expected.  I listed it for completeness (all the
call sites of url-https-proxy-connect), but I guess it would only be
called under url-retrieve, not under url-retrieve-synchronously.

> I also put the url-https-proxy-connect override you gave me earlier
> into a file, loaded it and set the debugger to run through that as
> well as set up proxies.  A broken down list of steps:
>
> 1.  Load file containing proxy and altered url-https-proxyconnect, set
> debugger to run through it when hit.
> 2.  set up url-http and url-http-async-sentinel to be picked up by debugger
> 3.  eval (url-retrieve-synchronously
> "https://outlook.office365.com/EWS/Exchange.asmx"), step through
> url-http
> 4.  Input username and password when asked
> 5.  Continue stepping until end
>
> url-http-async-sentinel is never called. " *http* ... -####" has text
> indicating failure. url-https-proxy-connect is indeed called.

OK, it's good to know that url-http ultimately calls
url-https-proxy-connect.  Unfortunately, despite this, proxying does not
work.

> I don't know how to look at the actual <process>, if I try to C-x C-e
> or M-: "connection", it goes into the debugger.  The url's, etc look
> good as far as I can tell.  Not sure what else I can do.  If there are
> certain pieces of url-http you want me to look at, just let me know,
> but not really knowing what has to happen under the hood, I am not
> going to be able to do much else.

I think I'll probably have to set up my own test environment.  Are you
in control of the proxy and its configuration?  If so, can you provide
rough configuration instructions (proxy software, version, relevant
config settings)?  If not, that's OK, I can try setting up generic proxy
software.

Thanks,
Thomas

Message #58 received at 35969 <at> debbugs.gnu.org (full text, mbox):

From: "tenspd137 ." <dcday137 <at> gmail.com>
To: Thomas Fitzsimmons <fitzsim <at> fitzsim.org>
Cc: 35969 <at> debbugs.gnu.org
Subject: Re: bug#35969: 26.2, Excorporate
Date: Fri, 14 Jun 2019 15:48:34 -0600

Sorry - I am not in control of my proxy.  So, before I give up on this
for today, I was going over the wget logs from trying the same thing.
One thing I noticed is:

Connection: Keep-Alive
Proxy-Connection: Keep-Alive

I see stuff in the emacs buffers mentioning Connection, and it says
closed (unless marking the connection as busy is the same as keeping
it open), but I don't see anything mentioning the proxy
connection....?  Maybe another place to look?

Just some ideas...

Thanks!

-C

On Fri, Jun 14, 2019 at 3:02 PM Thomas Fitzsimmons <fitzsim <at> fitzsim.org> wrote:
>
> Hi,
>
> Thanks for following up with these further test results.
>
> "tenspd137 ." <dcday137 <at> gmail.com> writes:
>
> > I was able to try stepping through an Emacs/proxy/Exchange test in an
> > emacs -Q session.  After setting the proxy and configuring the
> > debugger to step through url-http and url-http-async-sentilnel, the
> > only thing I noticed is that it appears url-http-async-sentinel is not
> > being called.
>
> OK, that's probably expected.  I listed it for completeness (all the
> call sites of url-https-proxy-connect), but I guess it would only be
> called under url-retrieve, not under url-retrieve-synchronously.
>
> > I also put the url-https-proxy-connect override you gave me earlier
> > into a file, loaded it and set the debugger to run through that as
> > well as set up proxies.  A broken down list of steps:
> >
> > 1.  Load file containing proxy and altered url-https-proxyconnect, set
> > debugger to run through it when hit.
> > 2.  set up url-http and url-http-async-sentinel to be picked up by debugger
> > 3.  eval (url-retrieve-synchronously
> > "https://outlook.office365.com/EWS/Exchange.asmx"), step through
> > url-http
> > 4.  Input username and password when asked
> > 5.  Continue stepping until end
> >
> > url-http-async-sentinel is never called. " *http* ... -####" has text
> > indicating failure. url-https-proxy-connect is indeed called.
>
> OK, it's good to know that url-http ultimately calls
> url-https-proxy-connect.  Unfortunately, despite this, proxying does not
> work.
>
> > I don't know how to look at the actual <process>, if I try to C-x C-e
> > or M-: "connection", it goes into the debugger.  The url's, etc look
> > good as far as I can tell.  Not sure what else I can do.  If there are
> > certain pieces of url-http you want me to look at, just let me know,
> > but not really knowing what has to happen under the hood, I am not
> > going to be able to do much else.
>
> I think I'll probably have to set up my own test environment.  Are you
> in control of the proxy and its configuration?  If so, can you provide
> rough configuration instructions (proxy software, version, relevant
> config settings)?  If not, that's OK, I can try setting up generic proxy
> software.
>
> Thanks,
> Thomas

Message #61 received at 35969 <at> debbugs.gnu.org (full text, mbox):

From: "tenspd137 ." <dcday137 <at> gmail.com>
To: Thomas Fitzsimmons <fitzsim <at> fitzsim.org>
Cc: 35969 <at> debbugs.gnu.org
Subject: Re: bug#35969: 26.2, Excorporate
Date: Fri, 14 Jun 2019 16:07:50 -0600

Just some more things I noticed from the wget log vs. the emacs buffers:

emacs: Accept-encoding:gzip, wget:Accept-encoding: identity

emacs: GET  https://outlook.office365.com/EWS/Exchange.asmx HTTP/1.1,
wget: GET /EWS/Exchange.asmx HTTP/1.1

Don't know if any of that is helpful, but there it is.

Thanks!

-C

On Fri, Jun 14, 2019 at 3:48 PM tenspd137 . <dcday137 <at> gmail.com> wrote:
>
> Sorry - I am not in control of my proxy.  So, before I give up on this
> for today, I was going over the wget logs from trying the same thing.
> One thing I noticed is:
>
> Connection: Keep-Alive
> Proxy-Connection: Keep-Alive
>
> I see stuff in the emacs buffers mentioning Connection, and it says
> closed (unless marking the connection as busy is the same as keeping
> it open), but I don't see anything mentioning the proxy
> connection....?  Maybe another place to look?
>
> Just some ideas...
>
> Thanks!
>
> -C
>
> On Fri, Jun 14, 2019 at 3:02 PM Thomas Fitzsimmons <fitzsim <at> fitzsim.org> wrote:
> >
> > Hi,
> >
> > Thanks for following up with these further test results.
> >
> > "tenspd137 ." <dcday137 <at> gmail.com> writes:
> >
> > > I was able to try stepping through an Emacs/proxy/Exchange test in an
> > > emacs -Q session.  After setting the proxy and configuring the
> > > debugger to step through url-http and url-http-async-sentilnel, the
> > > only thing I noticed is that it appears url-http-async-sentinel is not
> > > being called.
> >
> > OK, that's probably expected.  I listed it for completeness (all the
> > call sites of url-https-proxy-connect), but I guess it would only be
> > called under url-retrieve, not under url-retrieve-synchronously.
> >
> > > I also put the url-https-proxy-connect override you gave me earlier
> > > into a file, loaded it and set the debugger to run through that as
> > > well as set up proxies.  A broken down list of steps:
> > >
> > > 1.  Load file containing proxy and altered url-https-proxyconnect, set
> > > debugger to run through it when hit.
> > > 2.  set up url-http and url-http-async-sentinel to be picked up by debugger
> > > 3.  eval (url-retrieve-synchronously
> > > "https://outlook.office365.com/EWS/Exchange.asmx"), step through
> > > url-http
> > > 4.  Input username and password when asked
> > > 5.  Continue stepping until end
> > >
> > > url-http-async-sentinel is never called. " *http* ... -####" has text
> > > indicating failure. url-https-proxy-connect is indeed called.
> >
> > OK, it's good to know that url-http ultimately calls
> > url-https-proxy-connect.  Unfortunately, despite this, proxying does not
> > work.
> >
> > > I don't know how to look at the actual <process>, if I try to C-x C-e
> > > or M-: "connection", it goes into the debugger.  The url's, etc look
> > > good as far as I can tell.  Not sure what else I can do.  If there are
> > > certain pieces of url-http you want me to look at, just let me know,
> > > but not really knowing what has to happen under the hood, I am not
> > > going to be able to do much else.
> >
> > I think I'll probably have to set up my own test environment.  Are you
> > in control of the proxy and its configuration?  If so, can you provide
> > rough configuration instructions (proxy software, version, relevant
> > config settings)?  If not, that's OK, I can try setting up generic proxy
> > software.
> >
> > Thanks,
> > Thomas

Message #64 received at 35969 <at> debbugs.gnu.org (full text, mbox):

From: "tenspd137 ." <dcday137 <at> gmail.com>
To: Thomas Fitzsimmons <fitzsim <at> fitzsim.org>
Cc: 35969 <at> debbugs.gnu.org
Subject: Re: bug#35969: 26.2, Excorporate
Date: Fri, 14 Jun 2019 17:07:59 -0600

I think I figured it out.  I created a function that changed the line
in  url-http-create-request so that the request looks like:
GET /EWS/Exchange.asmx HTTP/1.1 <-----  *no server / host*
MIME-Version: 1.0
Connection: close
Extension: Security/Digest Security/SSL
Host: outlook.office365.com
Accept-encoding: gzip
Accept: */*
User-Agent: URL/Emacs Emacs/26.2 (X11; x86_64-pc-linux-gnu)
Cookie: OIDC=1; ClientId=8998C5691CD143E784857A0D01537963
Authorization: Basic ZGF2aWQuYy5kYXlAaHAuY29tOlMxa3kzbGk3bmUwNzMxJSU=

instead of

GET https://outlook.office365.com/EWS/Exchange.asmx HTTP/1.1
MIME-Version: 1.0
Connection: close
Extension: Security/Digest Security/SSL
Host: outlook.office365.com
Accept-encoding: gzip
Accept: */*
User-Agent: URL/Emacs Emacs/26.2 (X11; x86_64-pc-linux-gnu)
Cookie: OIDC=1; ClientId=8998C5691CD143E784857A0D01537963
Authorization: Basic ZGF2aWQuYy5kYXlAaHAuY29tOlMxa3kzbGk3bmUwNzMxJSU=

and the results buffer gives: *http....*-#####
HTTP/1.1 200 OK
Cache-Control: private
Content-Length: 1213
Content-Type: text/html; charset=UTF-8
Content-Encoding: gzip
Vary: Accept-Encoding
Server: Microsoft-IIS/10.0
request-id: bcd31568-29fd-44e4-935f-35b54d697f33
X-CalculatedFETarget: CY4PR18CU003.internal.outlook.com
X-BackEndHttpStatus: 200
Set-Cookie: exchangecookie=6c731ced364846f2a979bfaa84496f1e;
expires=Sun, 14-Jun-2020 23:02:57 GMT; path=/; secure; HttpOnly
X-FEProxyInfo: CY4PR18CA0059.NAMPRD18.PROD.OUTLOOK.COM
X-CalculatedBETarget: CS1PR8401MB1223.NAMPRD84.PROD.OUTLOOK.COM
X-BackEndHttpStatus: 200
X-RUM-Validated: 1
X-AspNet-Version: 4.0.30319
X-BeSku: Gen9
X-DiagInfo: CS1PR8401MB1223
X-BEServer: CS1PR8401MB1223
X-FEServer: CY4PR18CA0059
X-Powered-By: ASP.NET
X-FEServer: SN4PR0501CA0003
Date: Fri, 14 Jun 2019 23:02:56 GMT
Connection: close

<HTML><HEAD><link rel="alternate" type="text/xml"
href="https://cs1pr8401mb1223.namprd84.prod.outlook.com:444/EWS/Exchange.asmx?disco"/><STYLE
type="text/css">#content{ FONT-SIZE: 0.7em; PADDING-BOTTOM: 2em;
MARGIN-LEFT: 30px}BODY{MARGIN-TOP: 0px; MARGIN-LEFT: 0px; COLOR:
#000000; FONT-FAMILY: Verdana; BACKGROUND-COLOR: white}P{MARGIN-TOP:
0px; MARGIN-BOTTOM: 12px; COLOR: #000000; FONT-FAMILY:
Verdana}PRE{BORDER-RIGHT: #f0f0e0 1px solid; PADDING-RIGHT: 5px;
BORDER-TOP: #f0f0e0 1px solid; MARGIN-TOP: -5px; PADDING-LEFT: 5px;
FONT-SIZE: 1.2em; PADDING-BOTTOM: 5px; BORDER-LEFT: #f0f0e0 1px solid;
PADDING-TOP: 5px; BORDER-BOTTOM: #f0f0e0 1px solid; FONT-FAMILY:
Courier New; BACKGROUND-COLOR: #e5e5cc}.heading1{MARGIN-TOP: 0px;
PADDING-LEFT: 15px; FONT-WEIGHT: normal; FONT-SIZE: 26px;
MARGIN-BOTTOM: 0px; PADDING-BOTTOM: 3px; MARGIN-LEFT: -30px; WIDTH:
100%; COLOR: #ffffff; PADDING-TOP: 10px; FONT-FAMILY: Tahoma;
BACKGROUND-COLOR: #003366}.intro{MARGIN-LEFT:
-15px}</STYLE><TITLE>Service</TITLE></HEAD><BODY><DIV id="content"><P
class="heading1">Service</P><BR/><P class="intro">You have created a
service.<P class='intro'>To test this service, you will need to create
a client and use it to call the service. You can do this using the
svcutil.exe tool from the command line with the following syntax:</P>
<BR/><PRE>svcutil.exe <A
HREF="https://cs1pr8401mb1223.namprd84.prod.outlook.com:444/EWS/Services.wsdl">https://cs1pr8401mb1223.namprd84.prod.outlook.com:444/EWS/Services.wsdl</A></PRE></P><P
class="intro"/>This will generate a configuration file and a code file
that contains the client class. Add the two files to your client
application and use the generated client class to call the Service.
For example:<BR/><P class='intro'><B>C#</B></P><PRE><font
color="blue">class </font><font color="teal">Test
</font>{
<font color="blue">    static void </font>Main()
    {
        <font color="teal">HelloClient</font> client = <font
color="blue">new </font><font color="teal">HelloClient</font>();

<font color="green">        // Use the 'client' variable to call
operations on the service.

</font><font color="green">        // Always close the client.
</font>        client.Close();
    }
}
</PRE><BR/><P class='intro'><B>Visual Basic</B></P><PRE><font
color="blue">Class </font><font color="teal">Test
</font><font color="blue">    Shared Sub </font>Main()
<font color="blue">        Dim </font>client As <font
color="teal">HelloClient</font> = <font color="blue">New </font><font
color="teal">HelloClient</font>()
<font color="green">        ' Use the 'client' variable to call
operations on the service.

</font><font color="green">        ' Always close the client.
</font>        client.Close()
<font color="blue">    End Sub
</font><font color="blue">End Class</font></PRE></DIV></BODY></HTML>

which matches the file saved with wget.

That has to be worth something....

Thanks!

-C

On Fri, Jun 14, 2019 at 4:07 PM tenspd137 . <dcday137 <at> gmail.com> wrote:
>
> Just some more things I noticed from the wget log vs. the emacs buffers:
>
> emacs: Accept-encoding:gzip, wget:Accept-encoding: identity
>
> emacs: GET  https://outlook.office365.com/EWS/Exchange.asmx HTTP/1.1,
> wget: GET /EWS/Exchange.asmx HTTP/1.1
>
> Don't know if any of that is helpful, but there it is.
>
> Thanks!
>
> -C
>
> On Fri, Jun 14, 2019 at 3:48 PM tenspd137 . <dcday137 <at> gmail.com> wrote:
> >
> > Sorry - I am not in control of my proxy.  So, before I give up on this
> > for today, I was going over the wget logs from trying the same thing.
> > One thing I noticed is:
> >
> > Connection: Keep-Alive
> > Proxy-Connection: Keep-Alive
> >
> > I see stuff in the emacs buffers mentioning Connection, and it says
> > closed (unless marking the connection as busy is the same as keeping
> > it open), but I don't see anything mentioning the proxy
> > connection....?  Maybe another place to look?
> >
> > Just some ideas...
> >
> > Thanks!
> >
> > -C
> >
> > On Fri, Jun 14, 2019 at 3:02 PM Thomas Fitzsimmons <fitzsim <at> fitzsim.org> wrote:
> > >
> > > Hi,
> > >
> > > Thanks for following up with these further test results.
> > >
> > > "tenspd137 ." <dcday137 <at> gmail.com> writes:
> > >
> > > > I was able to try stepping through an Emacs/proxy/Exchange test in an
> > > > emacs -Q session.  After setting the proxy and configuring the
> > > > debugger to step through url-http and url-http-async-sentilnel, the
> > > > only thing I noticed is that it appears url-http-async-sentinel is not
> > > > being called.
> > >
> > > OK, that's probably expected.  I listed it for completeness (all the
> > > call sites of url-https-proxy-connect), but I guess it would only be
> > > called under url-retrieve, not under url-retrieve-synchronously.
> > >
> > > > I also put the url-https-proxy-connect override you gave me earlier
> > > > into a file, loaded it and set the debugger to run through that as
> > > > well as set up proxies.  A broken down list of steps:
> > > >
> > > > 1.  Load file containing proxy and altered url-https-proxyconnect, set
> > > > debugger to run through it when hit.
> > > > 2.  set up url-http and url-http-async-sentinel to be picked up by debugger
> > > > 3.  eval (url-retrieve-synchronously
> > > > "https://outlook.office365.com/EWS/Exchange.asmx"), step through
> > > > url-http
> > > > 4.  Input username and password when asked
> > > > 5.  Continue stepping until end
> > > >
> > > > url-http-async-sentinel is never called. " *http* ... -####" has text
> > > > indicating failure. url-https-proxy-connect is indeed called.
> > >
> > > OK, it's good to know that url-http ultimately calls
> > > url-https-proxy-connect.  Unfortunately, despite this, proxying does not
> > > work.
> > >
> > > > I don't know how to look at the actual <process>, if I try to C-x C-e
> > > > or M-: "connection", it goes into the debugger.  The url's, etc look
> > > > good as far as I can tell.  Not sure what else I can do.  If there are
> > > > certain pieces of url-http you want me to look at, just let me know,
> > > > but not really knowing what has to happen under the hood, I am not
> > > > going to be able to do much else.
> > >
> > > I think I'll probably have to set up my own test environment.  Are you
> > > in control of the proxy and its configuration?  If so, can you provide
> > > rough configuration instructions (proxy software, version, relevant
> > > config settings)?  If not, that's OK, I can try setting up generic proxy
> > > software.
> > >
> > > Thanks,
> > > Thomas

Message #67 received at 35969 <at> debbugs.gnu.org (full text, mbox):

From: "tenspd137 ." <dcday137 <at> gmail.com>
To: Thomas Fitzsimmons <fitzsim <at> fitzsim.org>
Cc: 35969 <at> debbugs.gnu.org
Subject: Re: bug#35969: 26.2, Excorporate
Date: Fri, 14 Jun 2019 17:14:26 -0600

Sorry - I was working fast:

*I created a function that changed the line
in  url-http-create-request so that the request looks like:*

In a file I did:

(defun thefile() "/EWS/Exchange.asmx")

(copied from url-http library)
(defun url-http-create-request (&optional ref-url)
.....
   ;; This was done with a call to `format'.  Concatenating parts has
    ;; the advantage of keeping the parts of each header together and
    ;; allows us to elide null lines directly, at the cost of making
    ;; the layout less clear.
    (setq request
          (concat
             ;; The request
             (or url-http-method "GET") " "
             (url-http--encode-string
              (if using-proxy (thefile) real-fname))
<------------Changed this to "hardcode" the request as proof of
concept
             " HTTP/" url-http-version "\r\n"
             ;; Version of MIME we speak
             "MIME-Version: 1.0\r\n"
             ;; (maybe) Try to keep the connection open
             "Connection: " (if (or using-proxy .....
.......
.......

Loaded the file and then ran (url-retriev-synchronously "https://......")

Thanks!

-C

On Fri, Jun 14, 2019 at 5:07 PM tenspd137 . <dcday137 <at> gmail.com> wrote:
>
> I think I figured it out.  I created a function that changed the line
> in  url-http-create-request so that the request looks like:
> GET /EWS/Exchange.asmx HTTP/1.1 <-----  *no server / host*
> MIME-Version: 1.0
> Connection: close
> Extension: Security/Digest Security/SSL
> Host: outlook.office365.com
> Accept-encoding: gzip
> Accept: */*
> User-Agent: URL/Emacs Emacs/26.2 (X11; x86_64-pc-linux-gnu)
> Cookie: OIDC=1; ClientId=8998C5691CD143E784857A0D01537963
> Authorization: Basic ZGF2aWQuYy5kYXlAaHAuY29tOlMxa3kzbGk3bmUwNzMxJSU=
>
> instead of
>
> GET https://outlook.office365.com/EWS/Exchange.asmx HTTP/1.1
> MIME-Version: 1.0
> Connection: close
> Extension: Security/Digest Security/SSL
> Host: outlook.office365.com
> Accept-encoding: gzip
> Accept: */*
> User-Agent: URL/Emacs Emacs/26.2 (X11; x86_64-pc-linux-gnu)
> Cookie: OIDC=1; ClientId=8998C5691CD143E784857A0D01537963
> Authorization: Basic ZGF2aWQuYy5kYXlAaHAuY29tOlMxa3kzbGk3bmUwNzMxJSU=
>
> and the results buffer gives: *http....*-#####
> HTTP/1.1 200 OK
> Cache-Control: private
> Content-Length: 1213
> Content-Type: text/html; charset=UTF-8
> Content-Encoding: gzip
> Vary: Accept-Encoding
> Server: Microsoft-IIS/10.0
> request-id: bcd31568-29fd-44e4-935f-35b54d697f33
> X-CalculatedFETarget: CY4PR18CU003.internal.outlook.com
> X-BackEndHttpStatus: 200
> Set-Cookie: exchangecookie=6c731ced364846f2a979bfaa84496f1e;
> expires=Sun, 14-Jun-2020 23:02:57 GMT; path=/; secure; HttpOnly
> X-FEProxyInfo: CY4PR18CA0059.NAMPRD18.PROD.OUTLOOK.COM
> X-CalculatedBETarget: CS1PR8401MB1223.NAMPRD84.PROD.OUTLOOK.COM
> X-BackEndHttpStatus: 200
> X-RUM-Validated: 1
> X-AspNet-Version: 4.0.30319
> X-BeSku: Gen9
> X-DiagInfo: CS1PR8401MB1223
> X-BEServer: CS1PR8401MB1223
> X-FEServer: CY4PR18CA0059
> X-Powered-By: ASP.NET
> X-FEServer: SN4PR0501CA0003
> Date: Fri, 14 Jun 2019 23:02:56 GMT
> Connection: close
>
> <HTML><HEAD><link rel="alternate" type="text/xml"
> href="https://cs1pr8401mb1223.namprd84.prod.outlook.com:444/EWS/Exchange.asmx?disco"/><STYLE
> type="text/css">#content{ FONT-SIZE: 0.7em; PADDING-BOTTOM: 2em;
> MARGIN-LEFT: 30px}BODY{MARGIN-TOP: 0px; MARGIN-LEFT: 0px; COLOR:
> #000000; FONT-FAMILY: Verdana; BACKGROUND-COLOR: white}P{MARGIN-TOP:
> 0px; MARGIN-BOTTOM: 12px; COLOR: #000000; FONT-FAMILY:
> Verdana}PRE{BORDER-RIGHT: #f0f0e0 1px solid; PADDING-RIGHT: 5px;
> BORDER-TOP: #f0f0e0 1px solid; MARGIN-TOP: -5px; PADDING-LEFT: 5px;
> FONT-SIZE: 1.2em; PADDING-BOTTOM: 5px; BORDER-LEFT: #f0f0e0 1px solid;
> PADDING-TOP: 5px; BORDER-BOTTOM: #f0f0e0 1px solid; FONT-FAMILY:
> Courier New; BACKGROUND-COLOR: #e5e5cc}.heading1{MARGIN-TOP: 0px;
> PADDING-LEFT: 15px; FONT-WEIGHT: normal; FONT-SIZE: 26px;
> MARGIN-BOTTOM: 0px; PADDING-BOTTOM: 3px; MARGIN-LEFT: -30px; WIDTH:
> 100%; COLOR: #ffffff; PADDING-TOP: 10px; FONT-FAMILY: Tahoma;
> BACKGROUND-COLOR: #003366}.intro{MARGIN-LEFT:
> -15px}</STYLE><TITLE>Service</TITLE></HEAD><BODY><DIV id="content"><P
> class="heading1">Service</P><BR/><P class="intro">You have created a
> service.<P class='intro'>To test this service, you will need to create
> a client and use it to call the service. You can do this using the
> svcutil.exe tool from the command line with the following syntax:</P>
> <BR/><PRE>svcutil.exe <A
> HREF="https://cs1pr8401mb1223.namprd84.prod.outlook.com:444/EWS/Services.wsdl">https://cs1pr8401mb1223.namprd84.prod.outlook.com:444/EWS/Services.wsdl</A></PRE></P><P
> class="intro"/>This will generate a configuration file and a code file
> that contains the client class. Add the two files to your client
> application and use the generated client class to call the Service.
> For example:<BR/><P class='intro'><B>C#</B></P><PRE><font
> color="blue">class </font><font color="teal">Test
> </font>{
> <font color="blue">    static void </font>Main()
>     {
>         <font color="teal">HelloClient</font> client = <font
> color="blue">new </font><font color="teal">HelloClient</font>();
>
> <font color="green">        // Use the 'client' variable to call
> operations on the service.
>
> </font><font color="green">        // Always close the client.
> </font>        client.Close();
>     }
> }
> </PRE><BR/><P class='intro'><B>Visual Basic</B></P><PRE><font
> color="blue">Class </font><font color="teal">Test
> </font><font color="blue">    Shared Sub </font>Main()
> <font color="blue">        Dim </font>client As <font
> color="teal">HelloClient</font> = <font color="blue">New </font><font
> color="teal">HelloClient</font>()
> <font color="green">        ' Use the 'client' variable to call
> operations on the service.
>
> </font><font color="green">        ' Always close the client.
> </font>        client.Close()
> <font color="blue">    End Sub
> </font><font color="blue">End Class</font></PRE></DIV></BODY></HTML>
>
> which matches the file saved with wget.
>
> That has to be worth something....
>
> Thanks!
>
> -C
>
> On Fri, Jun 14, 2019 at 4:07 PM tenspd137 . <dcday137 <at> gmail.com> wrote:
> >
> > Just some more things I noticed from the wget log vs. the emacs buffers:
> >
> > emacs: Accept-encoding:gzip, wget:Accept-encoding: identity
> >
> > emacs: GET  https://outlook.office365.com/EWS/Exchange.asmx HTTP/1.1,
> > wget: GET /EWS/Exchange.asmx HTTP/1.1
> >
> > Don't know if any of that is helpful, but there it is.
> >
> > Thanks!
> >
> > -C
> >
> > On Fri, Jun 14, 2019 at 3:48 PM tenspd137 . <dcday137 <at> gmail.com> wrote:
> > >
> > > Sorry - I am not in control of my proxy.  So, before I give up on this
> > > for today, I was going over the wget logs from trying the same thing.
> > > One thing I noticed is:
> > >
> > > Connection: Keep-Alive
> > > Proxy-Connection: Keep-Alive
> > >
> > > I see stuff in the emacs buffers mentioning Connection, and it says
> > > closed (unless marking the connection as busy is the same as keeping
> > > it open), but I don't see anything mentioning the proxy
> > > connection....?  Maybe another place to look?
> > >
> > > Just some ideas...
> > >
> > > Thanks!
> > >
> > > -C
> > >
> > > On Fri, Jun 14, 2019 at 3:02 PM Thomas Fitzsimmons <fitzsim <at> fitzsim.org> wrote:
> > > >
> > > > Hi,
> > > >
> > > > Thanks for following up with these further test results.
> > > >
> > > > "tenspd137 ." <dcday137 <at> gmail.com> writes:
> > > >
> > > > > I was able to try stepping through an Emacs/proxy/Exchange test in an
> > > > > emacs -Q session.  After setting the proxy and configuring the
> > > > > debugger to step through url-http and url-http-async-sentilnel, the
> > > > > only thing I noticed is that it appears url-http-async-sentinel is not
> > > > > being called.
> > > >
> > > > OK, that's probably expected.  I listed it for completeness (all the
> > > > call sites of url-https-proxy-connect), but I guess it would only be
> > > > called under url-retrieve, not under url-retrieve-synchronously.
> > > >
> > > > > I also put the url-https-proxy-connect override you gave me earlier
> > > > > into a file, loaded it and set the debugger to run through that as
> > > > > well as set up proxies.  A broken down list of steps:
> > > > >
> > > > > 1.  Load file containing proxy and altered url-https-proxyconnect, set
> > > > > debugger to run through it when hit.
> > > > > 2.  set up url-http and url-http-async-sentinel to be picked up by debugger
> > > > > 3.  eval (url-retrieve-synchronously
> > > > > "https://outlook.office365.com/EWS/Exchange.asmx"), step through
> > > > > url-http
> > > > > 4.  Input username and password when asked
> > > > > 5.  Continue stepping until end
> > > > >
> > > > > url-http-async-sentinel is never called. " *http* ... -####" has text
> > > > > indicating failure. url-https-proxy-connect is indeed called.
> > > >
> > > > OK, it's good to know that url-http ultimately calls
> > > > url-https-proxy-connect.  Unfortunately, despite this, proxying does not
> > > > work.
> > > >
> > > > > I don't know how to look at the actual <process>, if I try to C-x C-e
> > > > > or M-: "connection", it goes into the debugger.  The url's, etc look
> > > > > good as far as I can tell.  Not sure what else I can do.  If there are
> > > > > certain pieces of url-http you want me to look at, just let me know,
> > > > > but not really knowing what has to happen under the hood, I am not
> > > > > going to be able to do much else.
> > > >
> > > > I think I'll probably have to set up my own test environment.  Are you
> > > > in control of the proxy and its configuration?  If so, can you provide
> > > > rough configuration instructions (proxy software, version, relevant
> > > > config settings)?  If not, that's OK, I can try setting up generic proxy
> > > > software.
> > > >
> > > > Thanks,
> > > > Thomas

Message #70 received at 35969 <at> debbugs.gnu.org (full text, mbox):

From: Thomas Fitzsimmons <fitzsim <at> fitzsim.org>
To: "tenspd137 ." <dcday137 <at> gmail.com>
Cc: 35969 <at> debbugs.gnu.org
Subject: Re: bug#35969: 26.2, Excorporate
Date: Fri, 14 Jun 2019 20:14:21 -0400

Hi,

Yes, this seems really promising.  If you just replace (thefile) with
real-fname (and re-eval your url-http-create-request), does Excorporate
then work through the proxy?

Thomas

"tenspd137 ." <dcday137 <at> gmail.com> writes:

> Sorry - I was working fast:
>
> *I created a function that changed the line
> in  url-http-create-request so that the request looks like:*
>
> In a file I did:
>
> (defun thefile() "/EWS/Exchange.asmx")
>
> (copied from url-http library)
> (defun url-http-create-request (&optional ref-url)
> .....
>    ;; This was done with a call to `format'.  Concatenating parts has
>     ;; the advantage of keeping the parts of each header together and
>     ;; allows us to elide null lines directly, at the cost of making
>     ;; the layout less clear.
>     (setq request
>           (concat
>              ;; The request
>              (or url-http-method "GET") " "
>              (url-http--encode-string
>               (if using-proxy (thefile) real-fname))
> <------------Changed this to "hardcode" the request as proof of
> concept
>              " HTTP/" url-http-version "\r\n"
>              ;; Version of MIME we speak
>              "MIME-Version: 1.0\r\n"
>              ;; (maybe) Try to keep the connection open
>              "Connection: " (if (or using-proxy .....
> .......
> .......
>
> Loaded the file and then ran (url-retriev-synchronously "https://......")
>
> Thanks!
>
> -C
>
> On Fri, Jun 14, 2019 at 5:07 PM tenspd137 . <dcday137 <at> gmail.com> wrote:
>>
>> I think I figured it out.  I created a function that changed the line
>> in  url-http-create-request so that the request looks like:
>> GET /EWS/Exchange.asmx HTTP/1.1 <-----  *no server / host*
>> MIME-Version: 1.0
>> Connection: close
>> Extension: Security/Digest Security/SSL
>> Host: outlook.office365.com
>> Accept-encoding: gzip
>> Accept: */*
>> User-Agent: URL/Emacs Emacs/26.2 (X11; x86_64-pc-linux-gnu)
>> Cookie: OIDC=1; ClientId=8998C5691CD143E784857A0D01537963
>> Authorization: Basic ZGF2aWQuYy5kYXlAaHAuY29tOlMxa3kzbGk3bmUwNzMxJSU=
>>
>> instead of
>>
>> GET https://outlook.office365.com/EWS/Exchange.asmx HTTP/1.1
>> MIME-Version: 1.0
>> Connection: close
>> Extension: Security/Digest Security/SSL
>> Host: outlook.office365.com
>> Accept-encoding: gzip
>> Accept: */*
>> User-Agent: URL/Emacs Emacs/26.2 (X11; x86_64-pc-linux-gnu)
>> Cookie: OIDC=1; ClientId=8998C5691CD143E784857A0D01537963
>> Authorization: Basic ZGF2aWQuYy5kYXlAaHAuY29tOlMxa3kzbGk3bmUwNzMxJSU=
>>
>> and the results buffer gives: *http....*-#####
>> HTTP/1.1 200 OK
>> Cache-Control: private
>> Content-Length: 1213
>> Content-Type: text/html; charset=UTF-8
>> Content-Encoding: gzip
>> Vary: Accept-Encoding
>> Server: Microsoft-IIS/10.0
>> request-id: bcd31568-29fd-44e4-935f-35b54d697f33
>> X-CalculatedFETarget: CY4PR18CU003.internal.outlook.com
>> X-BackEndHttpStatus: 200
>> Set-Cookie: exchangecookie=6c731ced364846f2a979bfaa84496f1e;
>> expires=Sun, 14-Jun-2020 23:02:57 GMT; path=/; secure; HttpOnly
>> X-FEProxyInfo: CY4PR18CA0059.NAMPRD18.PROD.OUTLOOK.COM
>> X-CalculatedBETarget: CS1PR8401MB1223.NAMPRD84.PROD.OUTLOOK.COM
>> X-BackEndHttpStatus: 200
>> X-RUM-Validated: 1
>> X-AspNet-Version: 4.0.30319
>> X-BeSku: Gen9
>> X-DiagInfo: CS1PR8401MB1223
>> X-BEServer: CS1PR8401MB1223
>> X-FEServer: CY4PR18CA0059
>> X-Powered-By: ASP.NET
>> X-FEServer: SN4PR0501CA0003
>> Date: Fri, 14 Jun 2019 23:02:56 GMT
>> Connection: close
>>
>> <HTML><HEAD><link rel="alternate" type="text/xml"
>> href="https://cs1pr8401mb1223.namprd84.prod.outlook.com:444/EWS/Exchange.asmx?disco"/><STYLE
>> type="text/css">#content{ FONT-SIZE: 0.7em; PADDING-BOTTOM: 2em;
>> MARGIN-LEFT: 30px}BODY{MARGIN-TOP: 0px; MARGIN-LEFT: 0px; COLOR:
>> #000000; FONT-FAMILY: Verdana; BACKGROUND-COLOR: white}P{MARGIN-TOP:
>> 0px; MARGIN-BOTTOM: 12px; COLOR: #000000; FONT-FAMILY:
>> Verdana}PRE{BORDER-RIGHT: #f0f0e0 1px solid; PADDING-RIGHT: 5px;
>> BORDER-TOP: #f0f0e0 1px solid; MARGIN-TOP: -5px; PADDING-LEFT: 5px;
>> FONT-SIZE: 1.2em; PADDING-BOTTOM: 5px; BORDER-LEFT: #f0f0e0 1px solid;
>> PADDING-TOP: 5px; BORDER-BOTTOM: #f0f0e0 1px solid; FONT-FAMILY:
>> Courier New; BACKGROUND-COLOR: #e5e5cc}.heading1{MARGIN-TOP: 0px;
>> PADDING-LEFT: 15px; FONT-WEIGHT: normal; FONT-SIZE: 26px;
>> MARGIN-BOTTOM: 0px; PADDING-BOTTOM: 3px; MARGIN-LEFT: -30px; WIDTH:
>> 100%; COLOR: #ffffff; PADDING-TOP: 10px; FONT-FAMILY: Tahoma;
>> BACKGROUND-COLOR: #003366}.intro{MARGIN-LEFT:
>> -15px}</STYLE><TITLE>Service</TITLE></HEAD><BODY><DIV id="content"><P
>> class="heading1">Service</P><BR/><P class="intro">You have created a
>> service.<P class='intro'>To test this service, you will need to create
>> a client and use it to call the service. You can do this using the
>> svcutil.exe tool from the command line with the following syntax:</P>
>> <BR/><PRE>svcutil.exe <A
>> HREF="https://cs1pr8401mb1223.namprd84.prod.outlook.com:444/EWS/Services.wsdl">https://cs1pr8401mb1223.namprd84.prod.outlook.com:444/EWS/Services.wsdl</A></PRE></P><P
>> class="intro"/>This will generate a configuration file and a code file
>> that contains the client class. Add the two files to your client
>> application and use the generated client class to call the Service.
>> For example:<BR/><P class='intro'><B>C#</B></P><PRE><font
>> color="blue">class </font><font color="teal">Test
>> </font>{
>> <font color="blue">    static void </font>Main()
>>     {
>>         <font color="teal">HelloClient</font> client = <font
>> color="blue">new </font><font color="teal">HelloClient</font>();
>>
>> <font color="green">        // Use the 'client' variable to call
>> operations on the service.
>>
>> </font><font color="green">        // Always close the client.
>> </font>        client.Close();
>>     }
>> }
>> </PRE><BR/><P class='intro'><B>Visual Basic</B></P><PRE><font
>> color="blue">Class </font><font color="teal">Test
>> </font><font color="blue">    Shared Sub </font>Main()
>> <font color="blue">        Dim </font>client As <font
>> color="teal">HelloClient</font> = <font color="blue">New </font><font
>> color="teal">HelloClient</font>()
>> <font color="green">        ' Use the 'client' variable to call
>> operations on the service.
>>
>> </font><font color="green">        ' Always close the client.
>> </font>        client.Close()
>> <font color="blue">    End Sub
>> </font><font color="blue">End Class</font></PRE></DIV></BODY></HTML>
>>
>> which matches the file saved with wget.
>>
>> That has to be worth something....
>>
>> Thanks!
>>
>> -C
>>
>> On Fri, Jun 14, 2019 at 4:07 PM tenspd137 . <dcday137 <at> gmail.com> wrote:
>> >
>> > Just some more things I noticed from the wget log vs. the emacs buffers:
>> >
>> > emacs: Accept-encoding:gzip, wget:Accept-encoding: identity
>> >
>> > emacs: GET  https://outlook.office365.com/EWS/Exchange.asmx HTTP/1.1,
>> > wget: GET /EWS/Exchange.asmx HTTP/1.1
>> >
>> > Don't know if any of that is helpful, but there it is.
>> >
>> > Thanks!
>> >
>> > -C
>> >
>> > On Fri, Jun 14, 2019 at 3:48 PM tenspd137 . <dcday137 <at> gmail.com> wrote:
>> > >
>> > > Sorry - I am not in control of my proxy.  So, before I give up on this
>> > > for today, I was going over the wget logs from trying the same thing.
>> > > One thing I noticed is:
>> > >
>> > > Connection: Keep-Alive
>> > > Proxy-Connection: Keep-Alive
>> > >
>> > > I see stuff in the emacs buffers mentioning Connection, and it says
>> > > closed (unless marking the connection as busy is the same as keeping
>> > > it open), but I don't see anything mentioning the proxy
>> > > connection....?  Maybe another place to look?
>> > >
>> > > Just some ideas...
>> > >
>> > > Thanks!
>> > >
>> > > -C
>> > >
>> > > On Fri, Jun 14, 2019 at 3:02 PM Thomas Fitzsimmons <fitzsim <at> fitzsim.org> wrote:
>> > > >
>> > > > Hi,
>> > > >
>> > > > Thanks for following up with these further test results.
>> > > >
>> > > > "tenspd137 ." <dcday137 <at> gmail.com> writes:
>> > > >
>> > > > > I was able to try stepping through an Emacs/proxy/Exchange test in an
>> > > > > emacs -Q session.  After setting the proxy and configuring the
>> > > > > debugger to step through url-http and url-http-async-sentilnel, the
>> > > > > only thing I noticed is that it appears url-http-async-sentinel is not
>> > > > > being called.
>> > > >
>> > > > OK, that's probably expected.  I listed it for completeness (all the
>> > > > call sites of url-https-proxy-connect), but I guess it would only be
>> > > > called under url-retrieve, not under url-retrieve-synchronously.
>> > > >
>> > > > > I also put the url-https-proxy-connect override you gave me earlier
>> > > > > into a file, loaded it and set the debugger to run through that as
>> > > > > well as set up proxies.  A broken down list of steps:
>> > > > >
>> > > > > 1.  Load file containing proxy and altered url-https-proxyconnect, set
>> > > > > debugger to run through it when hit.
>> > > > > 2.  set up url-http and url-http-async-sentinel to be picked up by debugger
>> > > > > 3.  eval (url-retrieve-synchronously
>> > > > > "https://outlook.office365.com/EWS/Exchange.asmx"), step through
>> > > > > url-http
>> > > > > 4.  Input username and password when asked
>> > > > > 5.  Continue stepping until end
>> > > > >
>> > > > > url-http-async-sentinel is never called. " *http* ... -####" has text
>> > > > > indicating failure. url-https-proxy-connect is indeed called.
>> > > >
>> > > > OK, it's good to know that url-http ultimately calls
>> > > > url-https-proxy-connect.  Unfortunately, despite this, proxying does not
>> > > > work.
>> > > >
>> > > > > I don't know how to look at the actual <process>, if I try to C-x C-e
>> > > > > or M-: "connection", it goes into the debugger.  The url's, etc look
>> > > > > good as far as I can tell.  Not sure what else I can do.  If there are
>> > > > > certain pieces of url-http you want me to look at, just let me know,
>> > > > > but not really knowing what has to happen under the hood, I am not
>> > > > > going to be able to do much else.
>> > > >
>> > > > I think I'll probably have to set up my own test environment.  Are you
>> > > > in control of the proxy and its configuration?  If so, can you provide
>> > > > rough configuration instructions (proxy software, version, relevant
>> > > > config settings)?  If not, that's OK, I can try setting up generic proxy
>> > > > software.
>> > > >
>> > > > Thanks,
>> > > > Thomas

Message #73 received at 35969 <at> debbugs.gnu.org (full text, mbox):

From: Thomas Fitzsimmons <fitzsim <at> fitzsim.org>
To: "tenspd137 ." <dcday137 <at> gmail.com>
Cc: 35969 <at> debbugs.gnu.org
Subject: Re: bug#35969: 26.2, Excorporate
Date: Fri, 14 Jun 2019 20:36:24 -0400

Hi,

"tenspd137 ." <dcday137 <at> gmail.com> writes:

> (defun thefile() "/EWS/Exchange.asmx")
> 
> (copied from url-http library)
> (defun url-http-create-request (&optional ref-url)
> .....
>    ;; This was done with a call to `format'.  Concatenating parts has
>     ;; the advantage of keeping the parts of each header together and
>     ;; allows us to elide null lines directly, at the cost of making
>     ;; the layout less clear.
>     (setq request
>           (concat
>              ;; The request
>              (or url-http-method "GET") " "
>              (url-http--encode-string
>               (if using-proxy (thefile) real-fname)) <------------Changed this to "hardcode" the request as proof of concept

Starting from your results I searched the web and turned up a Stack
Overflow post [1] which pointed me to the wget source code:

    if (proxy
#ifdef HAVE_SSL
        /* When using SSL over proxy, CONNECT establishes a direct
           connection to the HTTPS server.  Therefore use the same
           argument as when talking to the server directly. */
        && u->scheme != SCHEME_HTTPS
#endif
        )
      meth_arg = xstrdup (u->url);
    else
      meth_arg = url_full_path (u);

I think that comment explains why the url connection to the HTTPS
Exchange server is failing in your case.  Applying the same logic in the
location you found would give:

diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
index 00803a103a..723d111d58 100644
--- a/lisp/url/url-http.el
+++ b/lisp/url/url-http.el
@@ -329,7 +329,10 @@ url-http-create-request
              ;; The request
              (or url-http-method "GET") " "
              (url-http--encode-string
-              (if using-proxy (url-recreate-url url-http-target-url) real-fname))
+              (if (and using-proxy
+                       (not (equal "https" (url-type url-http-target-url))))
+                  (url-recreate-url url-http-target-url)
+                real-fname))
              " HTTP/" url-http-version "\r\n"
              ;; Version of MIME we speak
              "MIME-Version: 1.0\r\n"

Can you try out that patch and a) see if url-retrieve-synchronously
still works, and if so, b) see if Excorporate works through the proxy
with it?

Thanks,
Thomas

1. https://stackoverflow.com/questions/20050052/what-is-the-correct-url-format-to-send-to-an-http-proxy-server

Message #76 received at 35969 <at> debbugs.gnu.org (full text, mbox):

From: "tenspd137 ." <dcday137 <at> gmail.com>
To: Thomas Fitzsimmons <fitzsim <at> fitzsim.org>
Cc: 35969 <at> debbugs.gnu.org
Subject: Re: bug#35969: 26.2, Excorporate
Date: Fri, 14 Jun 2019 18:47:36 -0600

[Message part 1 (text/plain, inline)]

I'll be able to try on Monday....

Thanks!

C

On Fri, Jun 14, 2019, 6:14 PM Thomas Fitzsimmons <fitzsim <at> fitzsim.org>
wrote:

> Hi,
>
> Yes, this seems really promising.  If you just replace (thefile) with
> real-fname (and re-eval your url-http-create-request), does Excorporate
> then work through the proxy?
>
> Thomas
>
> "tenspd137 ." <dcday137 <at> gmail.com> writes:
>
> > Sorry - I was working fast:
> >
> > *I created a function that changed the line
> > in  url-http-create-request so that the request looks like:*
> >
> > In a file I did:
> >
> > (defun thefile() "/EWS/Exchange.asmx")
> >
> > (copied from url-http library)
> > (defun url-http-create-request (&optional ref-url)
> > .....
> >    ;; This was done with a call to `format'.  Concatenating parts has
> >     ;; the advantage of keeping the parts of each header together and
> >     ;; allows us to elide null lines directly, at the cost of making
> >     ;; the layout less clear.
> >     (setq request
> >           (concat
> >              ;; The request
> >              (or url-http-method "GET") " "
> >              (url-http--encode-string
> >               (if using-proxy (thefile) real-fname))
> > <------------Changed this to "hardcode" the request as proof of
> > concept
> >              " HTTP/" url-http-version "\r\n"
> >              ;; Version of MIME we speak
> >              "MIME-Version: 1.0\r\n"
> >              ;; (maybe) Try to keep the connection open
> >              "Connection: " (if (or using-proxy .....
> > .......
> > .......
> >
> > Loaded the file and then ran (url-retriev-synchronously "https://
> ......")
> >
> > Thanks!
> >
> > -C
> >
> > On Fri, Jun 14, 2019 at 5:07 PM tenspd137 . <dcday137 <at> gmail.com> wrote:
> >>
> >> I think I figured it out.  I created a function that changed the line
> >> in  url-http-create-request so that the request looks like:
> >> GET /EWS/Exchange.asmx HTTP/1.1 <-----  *no server / host*
> >> MIME-Version: 1.0
> >> Connection: close
> >> Extension: Security/Digest Security/SSL
> >> Host: outlook.office365.com
> >> Accept-encoding: gzip
> >> Accept: */*
> >> User-Agent: URL/Emacs Emacs/26.2 (X11; x86_64-pc-linux-gnu)
> >> Cookie: OIDC=1; ClientId=8998C5691CD143E784857A0D01537963
> >> Authorization: Basic ZGF2aWQuYy5kYXlAaHAuY29tOlMxa3kzbGk3bmUwNzMxJSU=
> >>
> >> instead of
> >>
> >> GET https://outlook.office365.com/EWS/Exchange.asmx HTTP/1.1
> >> MIME-Version: 1.0
> >> Connection: close
> >> Extension: Security/Digest Security/SSL
> >> Host: outlook.office365.com
> >> Accept-encoding: gzip
> >> Accept: */*
> >> User-Agent: URL/Emacs Emacs/26.2 (X11; x86_64-pc-linux-gnu)
> >> Cookie: OIDC=1; ClientId=8998C5691CD143E784857A0D01537963
> >> Authorization: Basic ZGF2aWQuYy5kYXlAaHAuY29tOlMxa3kzbGk3bmUwNzMxJSU=
> >>
> >> and the results buffer gives: *http....*-#####
> >> HTTP/1.1 200 OK
> >> Cache-Control: private
> >> Content-Length: 1213
> >> Content-Type: text/html; charset=UTF-8
> >> Content-Encoding: gzip
> >> Vary: Accept-Encoding
> >> Server: Microsoft-IIS/10.0
> >> request-id: bcd31568-29fd-44e4-935f-35b54d697f33
> >> X-CalculatedFETarget: CY4PR18CU003.internal.outlook.com
> >> X-BackEndHttpStatus: 200
> >> Set-Cookie: exchangecookie=6c731ced364846f2a979bfaa84496f1e;
> >> expires=Sun, 14-Jun-2020 23:02:57 GMT; path=/; secure; HttpOnly
> >> X-FEProxyInfo: CY4PR18CA0059.NAMPRD18.PROD.OUTLOOK.COM
> >> X-CalculatedBETarget: CS1PR8401MB1223.NAMPRD84.PROD.OUTLOOK.COM
> >> X-BackEndHttpStatus: 200
> >> X-RUM-Validated: 1
> >> X-AspNet-Version: 4.0.30319
> >> X-BeSku: Gen9
> >> X-DiagInfo: CS1PR8401MB1223
> >> X-BEServer: CS1PR8401MB1223
> >> X-FEServer: CY4PR18CA0059
> >> X-Powered-By: ASP.NET
> >> X-FEServer: SN4PR0501CA0003
> >> Date: Fri, 14 Jun 2019 23:02:56 GMT
> >> Connection: close
> >>
> >> <HTML><HEAD><link rel="alternate" type="text/xml"
> >> href="
> https://cs1pr8401mb1223.namprd84.prod.outlook.com:444/EWS/Exchange.asmx?disco
> "/><STYLE
> >> type="text/css">#content{ FONT-SIZE: 0.7em; PADDING-BOTTOM: 2em;
> >> MARGIN-LEFT: 30px}BODY{MARGIN-TOP: 0px; MARGIN-LEFT: 0px; COLOR:
> >> #000000; FONT-FAMILY: Verdana; BACKGROUND-COLOR: white}P{MARGIN-TOP:
> >> 0px; MARGIN-BOTTOM: 12px; COLOR: #000000; FONT-FAMILY:
> >> Verdana}PRE{BORDER-RIGHT: #f0f0e0 1px solid; PADDING-RIGHT: 5px;
> >> BORDER-TOP: #f0f0e0 1px solid; MARGIN-TOP: -5px; PADDING-LEFT: 5px;
> >> FONT-SIZE: 1.2em; PADDING-BOTTOM: 5px; BORDER-LEFT: #f0f0e0 1px solid;
> >> PADDING-TOP: 5px; BORDER-BOTTOM: #f0f0e0 1px solid; FONT-FAMILY:
> >> Courier New; BACKGROUND-COLOR: #e5e5cc}.heading1{MARGIN-TOP: 0px;
> >> PADDING-LEFT: 15px; FONT-WEIGHT: normal; FONT-SIZE: 26px;
> >> MARGIN-BOTTOM: 0px; PADDING-BOTTOM: 3px; MARGIN-LEFT: -30px; WIDTH:
> >> 100%; COLOR: #ffffff; PADDING-TOP: 10px; FONT-FAMILY: Tahoma;
> >> BACKGROUND-COLOR: #003366}.intro{MARGIN-LEFT:
> >> -15px}</STYLE><TITLE>Service</TITLE></HEAD><BODY><DIV id="content"><P
> >> class="heading1">Service</P><BR/><P class="intro">You have created a
> >> service.<P class='intro'>To test this service, you will need to create
> >> a client and use it to call the service. You can do this using the
> >> svcutil.exe tool from the command line with the following syntax:</P>
> >> <BR/><PRE>svcutil.exe <A
> >> HREF="
> https://cs1pr8401mb1223.namprd84.prod.outlook.com:444/EWS/Services.wsdl">
> https://cs1pr8401mb1223.namprd84.prod.outlook.com:444/EWS/Services.wsdl
> </A></PRE></P><P
> >> class="intro"/>This will generate a configuration file and a code file
> >> that contains the client class. Add the two files to your client
> >> application and use the generated client class to call the Service.
> >> For example:<BR/><P class='intro'><B>C#</B></P><PRE><font
> >> color="blue">class </font><font color="teal">Test
> >> </font>{
> >> <font color="blue">    static void </font>Main()
> >>     {
> >>         <font color="teal">HelloClient</font> client = <font
> >> color="blue">new </font><font color="teal">HelloClient</font>();
> >>
> >> <font color="green">        // Use the 'client' variable to call
> >> operations on the service.
> >>
> >> </font><font color="green">        // Always close the client.
> >> </font>        client.Close();
> >>     }
> >> }
> >> </PRE><BR/><P class='intro'><B>Visual Basic</B></P><PRE><font
> >> color="blue">Class </font><font color="teal">Test
> >> </font><font color="blue">    Shared Sub </font>Main()
> >> <font color="blue">        Dim </font>client As <font
> >> color="teal">HelloClient</font> = <font color="blue">New </font><font
> >> color="teal">HelloClient</font>()
> >> <font color="green">        ' Use the 'client' variable to call
> >> operations on the service.
> >>
> >> </font><font color="green">        ' Always close the client.
> >> </font>        client.Close()
> >> <font color="blue">    End Sub
> >> </font><font color="blue">End Class</font></PRE></DIV></BODY></HTML>
> >>
> >> which matches the file saved with wget.
> >>
> >> That has to be worth something....
> >>
> >> Thanks!
> >>
> >> -C
> >>
> >> On Fri, Jun 14, 2019 at 4:07 PM tenspd137 . <dcday137 <at> gmail.com> wrote:
> >> >
> >> > Just some more things I noticed from the wget log vs. the emacs
> buffers:
> >> >
> >> > emacs: Accept-encoding:gzip, wget:Accept-encoding: identity
> >> >
> >> > emacs: GET  https://outlook.office365.com/EWS/Exchange.asmx HTTP/1.1,
> >> > wget: GET /EWS/Exchange.asmx HTTP/1.1
> >> >
> >> > Don't know if any of that is helpful, but there it is.
> >> >
> >> > Thanks!
> >> >
> >> > -C
> >> >
> >> > On Fri, Jun 14, 2019 at 3:48 PM tenspd137 . <dcday137 <at> gmail.com>
> wrote:
> >> > >
> >> > > Sorry - I am not in control of my proxy.  So, before I give up on
> this
> >> > > for today, I was going over the wget logs from trying the same
> thing.
> >> > > One thing I noticed is:
> >> > >
> >> > > Connection: Keep-Alive
> >> > > Proxy-Connection: Keep-Alive
> >> > >
> >> > > I see stuff in the emacs buffers mentioning Connection, and it says
> >> > > closed (unless marking the connection as busy is the same as keeping
> >> > > it open), but I don't see anything mentioning the proxy
> >> > > connection....?  Maybe another place to look?
> >> > >
> >> > > Just some ideas...
> >> > >
> >> > > Thanks!
> >> > >
> >> > > -C
> >> > >
> >> > > On Fri, Jun 14, 2019 at 3:02 PM Thomas Fitzsimmons <
> fitzsim <at> fitzsim.org> wrote:
> >> > > >
> >> > > > Hi,
> >> > > >
> >> > > > Thanks for following up with these further test results.
> >> > > >
> >> > > > "tenspd137 ." <dcday137 <at> gmail.com> writes:
> >> > > >
> >> > > > > I was able to try stepping through an Emacs/proxy/Exchange test
> in an
> >> > > > > emacs -Q session.  After setting the proxy and configuring the
> >> > > > > debugger to step through url-http and url-http-async-sentilnel,
> the
> >> > > > > only thing I noticed is that it appears url-http-async-sentinel
> is not
> >> > > > > being called.
> >> > > >
> >> > > > OK, that's probably expected.  I listed it for completeness (all
> the
> >> > > > call sites of url-https-proxy-connect), but I guess it would only
> be
> >> > > > called under url-retrieve, not under url-retrieve-synchronously.
> >> > > >
> >> > > > > I also put the url-https-proxy-connect override you gave me
> earlier
> >> > > > > into a file, loaded it and set the debugger to run through that
> as
> >> > > > > well as set up proxies.  A broken down list of steps:
> >> > > > >
> >> > > > > 1.  Load file containing proxy and altered
> url-https-proxyconnect, set
> >> > > > > debugger to run through it when hit.
> >> > > > > 2.  set up url-http and url-http-async-sentinel to be picked up
> by debugger
> >> > > > > 3.  eval (url-retrieve-synchronously
> >> > > > > "https://outlook.office365.com/EWS/Exchange.asmx"), step
> through
> >> > > > > url-http
> >> > > > > 4.  Input username and password when asked
> >> > > > > 5.  Continue stepping until end
> >> > > > >
> >> > > > > url-http-async-sentinel is never called. " *http* ... -####"
> has text
> >> > > > > indicating failure. url-https-proxy-connect is indeed called.
> >> > > >
> >> > > > OK, it's good to know that url-http ultimately calls
> >> > > > url-https-proxy-connect.  Unfortunately, despite this, proxying
> does not
> >> > > > work.
> >> > > >
> >> > > > > I don't know how to look at the actual <process>, if I try to
> C-x C-e
> >> > > > > or M-: "connection", it goes into the debugger.  The url's, etc
> look
> >> > > > > good as far as I can tell.  Not sure what else I can do.  If
> there are
> >> > > > > certain pieces of url-http you want me to look at, just let me
> know,
> >> > > > > but not really knowing what has to happen under the hood, I am
> not
> >> > > > > going to be able to do much else.
> >> > > >
> >> > > > I think I'll probably have to set up my own test environment.
> Are you
> >> > > > in control of the proxy and its configuration?  If so, can you
> provide
> >> > > > rough configuration instructions (proxy software, version,
> relevant
> >> > > > config settings)?  If not, that's OK, I can try setting up
> generic proxy
> >> > > > software.
> >> > > >
> >> > > > Thanks,
> >> > > > Thomas
>

[Message part 2 (text/html, inline)]

Message #79 received at 35969 <at> debbugs.gnu.org (full text, mbox):

From: Andreas Schwab <schwab <at> linux-m68k.org>
To: Thomas Fitzsimmons <fitzsim <at> fitzsim.org>
Cc: "tenspd137 ." <dcday137 <at> gmail.com>, 35969 <at> debbugs.gnu.org
Subject: Re: bug#35969: 26.2, Excorporate
Date: Sat, 15 Jun 2019 09:41:33 +0200

On Jun 14 2019, Thomas Fitzsimmons <fitzsim <at> fitzsim.org> wrote:

> diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
> index 00803a103a..723d111d58 100644
> --- a/lisp/url/url-http.el
> +++ b/lisp/url/url-http.el
> @@ -329,7 +329,10 @@ url-http-create-request
>               ;; The request
>               (or url-http-method "GET") " "
>               (url-http--encode-string
> -              (if using-proxy (url-recreate-url url-http-target-url) real-fname))
> +              (if (and using-proxy
> +                       (not (equal "https" (url-type url-http-target-url))))
> +                  (url-recreate-url url-http-target-url)
> +                real-fname))

That should already be handled by commit 84613dae5c.

Andreas.

-- 
Andreas Schwab, schwab <at> linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Message #82 received at 35969 <at> debbugs.gnu.org (full text, mbox):

From: "tenspd137 ." <dcday137 <at> gmail.com>
To: Andreas Schwab <schwab <at> linux-m68k.org>
Cc: Thomas Fitzsimmons <fitzsim <at> fitzsim.org>, 35969 <at> debbugs.gnu.org
Subject: Re: bug#35969: 26.2, Excorporate
Date: Mon, 17 Jun 2019 10:31:40 -0600

The patch Thomas seems to work from behind the proxy.  My current
emacs version is 26.2, so I would think it would include the commit
Andreas is talking about....  I went and looked it up - is this the
correct commit?

diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
index 53798f7..817c5ce 100644
--- a/lisp/url/url-http.el
+++ b/lisp/url/url-http.el
@@ -1412,7 +1412,9 @@ The return value of this function is the
retrieval buffer."
'url-http-wait-for-headers-change-function)
(set-process-filter tls-connection 'url-http-generic-filter)
(process-send-string tls-connection
- (url-http-create-request)))
+ ;; Use the non-proxy form of the request
+ (let (url-http-proxy)
+ (url-http-create-request))))
(gnutls-error
(url-http-activate-callback)
(error "gnutls-error: %s" e))

Thanks!

-C


On Sat, Jun 15, 2019 at 1:41 AM Andreas Schwab <schwab <at> linux-m68k.org> wrote:
>
> On Jun 14 2019, Thomas Fitzsimmons <fitzsim <at> fitzsim.org> wrote:
>
> > diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
> > index 00803a103a..723d111d58 100644
> > --- a/lisp/url/url-http.el
> > +++ b/lisp/url/url-http.el
> > @@ -329,7 +329,10 @@ url-http-create-request
> >               ;; The request
> >               (or url-http-method "GET") " "
> >               (url-http--encode-string
> > -              (if using-proxy (url-recreate-url url-http-target-url) real-fname))
> > +              (if (and using-proxy
> > +                       (not (equal "https" (url-type url-http-target-url))))
> > +                  (url-recreate-url url-http-target-url)
> > +                real-fname))
>
> That should already be handled by commit 84613dae5c.
>
> Andreas.
>
> --
> Andreas Schwab, schwab <at> linux-m68k.org
> GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
> "And now for something completely different."

Message #85 received at 35969 <at> debbugs.gnu.org (full text, mbox):

From: Thomas Fitzsimmons <fitzsim <at> fitzsim.org>
To: "tenspd137 ." <dcday137 <at> gmail.com>
Cc: Andreas Schwab <schwab <at> linux-m68k.org>, 35969 <at> debbugs.gnu.org
Subject: Re: bug#35969: 26.2, Excorporate
Date: Mon, 17 Jun 2019 18:08:08 -0400

Hi,

Good to hear that the patch I posted worked!

Yes, that's the patch that Andreas's commit
84613dae5c34ea742dd9a3e56f5acb55f604b483 applied.  From what I can tell,
you will not have that in Emacs 26.2.

Can you try reverting my patch and applying Andreas's patch, and see if
Excorporate still works through the proxy?

Thanks,
Thomas

"tenspd137 ." <dcday137 <at> gmail.com> writes:

> The patch Thomas seems to work from behind the proxy.  My current
> emacs version is 26.2, so I would think it would include the commit
> Andreas is talking about....  I went and looked it up - is this the
> correct commit?
>
> diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
> index 53798f7..817c5ce 100644
> --- a/lisp/url/url-http.el
> +++ b/lisp/url/url-http.el
> @@ -1412,7 +1412,9 @@ The return value of this function is the
> retrieval buffer."
> 'url-http-wait-for-headers-change-function)
> (set-process-filter tls-connection 'url-http-generic-filter)
> (process-send-string tls-connection
> - (url-http-create-request)))
> + ;; Use the non-proxy form of the request
> + (let (url-http-proxy)
> + (url-http-create-request))))
> (gnutls-error
> (url-http-activate-callback)
> (error "gnutls-error: %s" e))
>
> Thanks!
>
> -C
>
>
> On Sat, Jun 15, 2019 at 1:41 AM Andreas Schwab <schwab <at> linux-m68k.org> wrote:
>>
>> On Jun 14 2019, Thomas Fitzsimmons <fitzsim <at> fitzsim.org> wrote:
>>
>> > diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
>> > index 00803a103a..723d111d58 100644
>> > --- a/lisp/url/url-http.el
>> > +++ b/lisp/url/url-http.el
>> > @@ -329,7 +329,10 @@ url-http-create-request
>> >               ;; The request
>> >               (or url-http-method "GET") " "
>> >               (url-http--encode-string
>> > -              (if using-proxy (url-recreate-url url-http-target-url) real-fname))
>> > +              (if (and using-proxy
>> > +                       (not (equal "https" (url-type url-http-target-url))))
>> > +                  (url-recreate-url url-http-target-url)
>> > +                real-fname))
>>
>> That should already be handled by commit 84613dae5c.
>>
>> Andreas.
>>
>> --
>> Andreas Schwab, schwab <at> linux-m68k.org
>> GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
>> "And now for something completely different."

Message #88 received at 35969 <at> debbugs.gnu.org (full text, mbox):

From: "tenspd137 ." <dcday137 <at> gmail.com>
To: Thomas Fitzsimmons <fitzsim <at> fitzsim.org>
Cc: Andreas Schwab <schwab <at> linux-m68k.org>, 35969 <at> debbugs.gnu.org
Subject: Re: bug#35969: 26.2, Excorporate
Date: Tue, 18 Jun 2019 10:34:05 -0600

So - I am not sure if I did it correctly, but I copied this function
with Andreas' changes into a file:

(defun url-https-proxy-after-change-function (_st _nd _length)
  (let* ((process-buffer (current-buffer))
         (proc (get-buffer-process process-buffer)))
    (goto-char (point-min))
    (when (re-search-forward "^\r?\n" nil t)
      (backward-char 1)
      ;; Saw the end of the headers
      (setq url-http-end-of-headers (set-marker (make-marker) (point)))
      (url-http-parse-response)
      (cond
       ((null url-http-response-status)
        ;; We got back a headerless malformed response from the
        ;; server.
        (url-http-activate-callback)
        (error "Malformed response from proxy, fail!"))
       ((= url-http-response-status 200)
        (if (gnutls-available-p)
            (condition-case e
                (let ((tls-connection (gnutls-negotiate
                                       :process proc
                                       :hostname (url-host url-current-object)
                                       :verify-error nil)))
                  ;; check certificate validity
                  (setq tls-connection
                        (nsm-verify-connection tls-connection
                                               (url-host url-current-object)
                                               (url-port url-current-object)))
                  (with-current-buffer process-buffer (erase-buffer))
                  (set-process-buffer tls-connection process-buffer)
                  (setq url-http-after-change-function
                        'url-http-wait-for-headers-change-function)
                  (set-process-filter tls-connection 'url-http-generic-filter)
                  (process-send-string tls-connection
                                       ;; Use the non-proxy form of the request
                                       (let (url-http-proxy)
                                         (url-http-create-request))))
              (gnutls-error
               (url-http-activate-callback)
               (error "gnutls-error: %s" e))
              (error
               (url-http-activate-callback)
               (error "error: %s" e)))
          (error "error: gnutls support needed!")))
       (t
        (url-http-debug "error response: %d" url-http-response-status)
        (url-http-activate-callback))))))

and then loaded it before running excorporate.  After that, I did M-x
excorporate, and the minibuffer returns:  error in process filter:
Server response is not an XML document

When I do the similar test by loading the url-http-create-request with
Thiomase's changes, I can get a connection and grab my schedule
through the proxy.

Let me know if I need to try something different.

Thanks!

-C

On Mon, Jun 17, 2019 at 4:08 PM Thomas Fitzsimmons <fitzsim <at> fitzsim.org> wrote:
>
> Hi,
>
> Good to hear that the patch I posted worked!
>
> Yes, that's the patch that Andreas's commit
> 84613dae5c34ea742dd9a3e56f5acb55f604b483 applied.  From what I can tell,
> you will not have that in Emacs 26.2.
>
> Can you try reverting my patch and applying Andreas's patch, and see if
> Excorporate still works through the proxy?
>
> Thanks,
> Thomas
>
> "tenspd137 ." <dcday137 <at> gmail.com> writes:
>
> > The patch Thomas seems to work from behind the proxy.  My current
> > emacs version is 26.2, so I would think it would include the commit
> > Andreas is talking about....  I went and looked it up - is this the
> > correct commit?
> >
> > diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
> > index 53798f7..817c5ce 100644
> > --- a/lisp/url/url-http.el
> > +++ b/lisp/url/url-http.el
> > @@ -1412,7 +1412,9 @@ The return value of this function is the
> > retrieval buffer."
> > 'url-http-wait-for-headers-change-function)
> > (set-process-filter tls-connection 'url-http-generic-filter)
> > (process-send-string tls-connection
> > - (url-http-create-request)))
> > + ;; Use the non-proxy form of the request
> > + (let (url-http-proxy)
> > + (url-http-create-request))))
> > (gnutls-error
> > (url-http-activate-callback)
> > (error "gnutls-error: %s" e))
> >
> > Thanks!
> >
> > -C
> >
> >
> > On Sat, Jun 15, 2019 at 1:41 AM Andreas Schwab <schwab <at> linux-m68k.org> wrote:
> >>
> >> On Jun 14 2019, Thomas Fitzsimmons <fitzsim <at> fitzsim.org> wrote:
> >>
> >> > diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
> >> > index 00803a103a..723d111d58 100644
> >> > --- a/lisp/url/url-http.el
> >> > +++ b/lisp/url/url-http.el
> >> > @@ -329,7 +329,10 @@ url-http-create-request
> >> >               ;; The request
> >> >               (or url-http-method "GET") " "
> >> >               (url-http--encode-string
> >> > -              (if using-proxy (url-recreate-url url-http-target-url) real-fname))
> >> > +              (if (and using-proxy
> >> > +                       (not (equal "https" (url-type url-http-target-url))))
> >> > +                  (url-recreate-url url-http-target-url)
> >> > +                real-fname))
> >>
> >> That should already be handled by commit 84613dae5c.
> >>
> >> Andreas.
> >>
> >> --
> >> Andreas Schwab, schwab <at> linux-m68k.org
> >> GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
> >> "And now for something completely different."

Message #91 received at 35969 <at> debbugs.gnu.org (full text, mbox):

From: Thomas Fitzsimmons <fitzsim <at> fitzsim.org>
To: "tenspd137 ." <dcday137 <at> gmail.com>
Cc: Andreas Schwab <schwab <at> linux-m68k.org>, 35969 <at> debbugs.gnu.org
Subject: Re: bug#35969: 26.2, Excorporate
Date: Wed, 19 Jun 2019 00:26:04 -0400

Hi,

"tenspd137 ." <dcday137 <at> gmail.com> writes:

> So - I am not sure if I did it correctly, but I copied this function
> with Andreas' changes into a file:
>
> (defun url-https-proxy-after-change-function (_st _nd _length)
>   (let* ((process-buffer (current-buffer))
>          (proc (get-buffer-process process-buffer)))
>     (goto-char (point-min))
>     (when (re-search-forward "^\r?\n" nil t)
>       (backward-char 1)
>       ;; Saw the end of the headers
>       (setq url-http-end-of-headers (set-marker (make-marker) (point)))
>       (url-http-parse-response)
>       (cond
>        ((null url-http-response-status)
>         ;; We got back a headerless malformed response from the
>         ;; server.
>         (url-http-activate-callback)
>         (error "Malformed response from proxy, fail!"))
>        ((= url-http-response-status 200)
>         (if (gnutls-available-p)
>             (condition-case e
>                 (let ((tls-connection (gnutls-negotiate
>                                        :process proc
>                                        :hostname (url-host url-current-object)
>                                        :verify-error nil)))
>                   ;; check certificate validity
>                   (setq tls-connection
>                         (nsm-verify-connection tls-connection
>                                                (url-host url-current-object)
>                                                (url-port url-current-object)))
>                   (with-current-buffer process-buffer (erase-buffer))
>                   (set-process-buffer tls-connection process-buffer)
>                   (setq url-http-after-change-function
>                         'url-http-wait-for-headers-change-function)
>                   (set-process-filter tls-connection 'url-http-generic-filter)
>                   (process-send-string tls-connection
>                                        ;; Use the non-proxy form of the request
>                                        (let (url-http-proxy)
>                                          (url-http-create-request))))
>               (gnutls-error
>                (url-http-activate-callback)
>                (error "gnutls-error: %s" e))
>               (error
>                (url-http-activate-callback)
>                (error "error: %s" e)))
>           (error "error: gnutls support needed!")))
>        (t
>         (url-http-debug "error response: %d" url-http-response-status)
>         (url-http-activate-callback))))))
>
> and then loaded it before running excorporate.  After that, I did M-x
> excorporate, and the minibuffer returns:  error in process filter:
> Server response is not an XML document

In this scenario, if you immediately (without restarting Emacs/reloading
anything) re-run M-x excorporate does it still fail?  I just want to
make sure that's not a transient failure.  If it does fail the second
time, can you post the HTTP response from the server?

> When I do the similar test by loading the url-http-create-request with
> Thomas's changes, I can get a connection and grab my schedule
> through the proxy.

OK.

> Let me know if I need to try something different.

Are you in a position to build Emacs master tip and retry the experiment
without patching anything?

Thanks,
Thomas

> On Mon, Jun 17, 2019 at 4:08 PM Thomas Fitzsimmons <fitzsim <at> fitzsim.org> wrote:
>>
>> Hi,
>>
>> Good to hear that the patch I posted worked!
>>
>> Yes, that's the patch that Andreas's commit
>> 84613dae5c34ea742dd9a3e56f5acb55f604b483 applied.  From what I can tell,
>> you will not have that in Emacs 26.2.
>>
>> Can you try reverting my patch and applying Andreas's patch, and see if
>> Excorporate still works through the proxy?
>>
>> Thanks,
>> Thomas
>>
>> "tenspd137 ." <dcday137 <at> gmail.com> writes:
>>
>> > The patch Thomas seems to work from behind the proxy.  My current
>> > emacs version is 26.2, so I would think it would include the commit
>> > Andreas is talking about....  I went and looked it up - is this the
>> > correct commit?
>> >
>> > diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
>> > index 53798f7..817c5ce 100644
>> > --- a/lisp/url/url-http.el
>> > +++ b/lisp/url/url-http.el
>> > @@ -1412,7 +1412,9 @@ The return value of this function is the
>> > retrieval buffer."
>> > 'url-http-wait-for-headers-change-function)
>> > (set-process-filter tls-connection 'url-http-generic-filter)
>> > (process-send-string tls-connection
>> > - (url-http-create-request)))
>> > + ;; Use the non-proxy form of the request
>> > + (let (url-http-proxy)
>> > + (url-http-create-request))))
>> > (gnutls-error
>> > (url-http-activate-callback)
>> > (error "gnutls-error: %s" e))
>> >
>> > Thanks!
>> >
>> > -C
>> >
>> >
>> > On Sat, Jun 15, 2019 at 1:41 AM Andreas Schwab <schwab <at> linux-m68k.org> wrote:
>> >>
>> >> On Jun 14 2019, Thomas Fitzsimmons <fitzsim <at> fitzsim.org> wrote:
>> >>
>> >> > diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
>> >> > index 00803a103a..723d111d58 100644
>> >> > --- a/lisp/url/url-http.el
>> >> > +++ b/lisp/url/url-http.el
>> >> > @@ -329,7 +329,10 @@ url-http-create-request
>> >> >               ;; The request
>> >> >               (or url-http-method "GET") " "
>> >> >               (url-http--encode-string
>> >> > -              (if using-proxy (url-recreate-url url-http-target-url) real-fname))
>> >> > +              (if (and using-proxy
>> >> > +                       (not (equal "https" (url-type url-http-target-url))))
>> >> > +                  (url-recreate-url url-http-target-url)
>> >> > +                real-fname))
>> >>
>> >> That should already be handled by commit 84613dae5c.
>> >>
>> >> Andreas.
>> >>
>> >> --
>> >> Andreas Schwab, schwab <at> linux-m68k.org
>> >> GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
>> >> "And now for something completely different."

Message #94 received at 35969 <at> debbugs.gnu.org (full text, mbox):

From: "tenspd137 ." <dcday137 <at> gmail.com>
To: Thomas Fitzsimmons <fitzsim <at> fitzsim.org>
Cc: Andreas Schwab <schwab <at> linux-m68k.org>, 35969 <at> debbugs.gnu.org
Subject: Re: bug#35969: 26.2, Excorporate
Date: Thu, 20 Jun 2019 14:34:48 -0600

I haven't tried the first item on the list yet - ie reloading Andreas'
function.  i have tried to build emacs from
https://github.com/emacs-mirror/emacs.git because I can't reach the
official repo from my firewall for whatever reason.  I am sure it is
something here.  Anyway, I was having problems because my built
version (27.0.x) was getting seemingly mixed up with my current
installation.  I can try again later, but right now things are pretty
busy at work.

Thanks.

On Tue, Jun 18, 2019 at 10:26 PM Thomas Fitzsimmons <fitzsim <at> fitzsim.org> wrote:
>
> Hi,
>
> "tenspd137 ." <dcday137 <at> gmail.com> writes:
>
> > So - I am not sure if I did it correctly, but I copied this function
> > with Andreas' changes into a file:
> >
> > (defun url-https-proxy-after-change-function (_st _nd _length)
> >   (let* ((process-buffer (current-buffer))
> >          (proc (get-buffer-process process-buffer)))
> >     (goto-char (point-min))
> >     (when (re-search-forward "^\r?\n" nil t)
> >       (backward-char 1)
> >       ;; Saw the end of the headers
> >       (setq url-http-end-of-headers (set-marker (make-marker) (point)))
> >       (url-http-parse-response)
> >       (cond
> >        ((null url-http-response-status)
> >         ;; We got back a headerless malformed response from the
> >         ;; server.
> >         (url-http-activate-callback)
> >         (error "Malformed response from proxy, fail!"))
> >        ((= url-http-response-status 200)
> >         (if (gnutls-available-p)
> >             (condition-case e
> >                 (let ((tls-connection (gnutls-negotiate
> >                                        :process proc
> >                                        :hostname (url-host url-current-object)
> >                                        :verify-error nil)))
> >                   ;; check certificate validity
> >                   (setq tls-connection
> >                         (nsm-verify-connection tls-connection
> >                                                (url-host url-current-object)
> >                                                (url-port url-current-object)))
> >                   (with-current-buffer process-buffer (erase-buffer))
> >                   (set-process-buffer tls-connection process-buffer)
> >                   (setq url-http-after-change-function
> >                         'url-http-wait-for-headers-change-function)
> >                   (set-process-filter tls-connection 'url-http-generic-filter)
> >                   (process-send-string tls-connection
> >                                        ;; Use the non-proxy form of the request
> >                                        (let (url-http-proxy)
> >                                          (url-http-create-request))))
> >               (gnutls-error
> >                (url-http-activate-callback)
> >                (error "gnutls-error: %s" e))
> >               (error
> >                (url-http-activate-callback)
> >                (error "error: %s" e)))
> >           (error "error: gnutls support needed!")))
> >        (t
> >         (url-http-debug "error response: %d" url-http-response-status)
> >         (url-http-activate-callback))))))
> >
> > and then loaded it before running excorporate.  After that, I did M-x
> > excorporate, and the minibuffer returns:  error in process filter:
> > Server response is not an XML document
>
> In this scenario, if you immediately (without restarting Emacs/reloading
> anything) re-run M-x excorporate does it still fail?  I just want to
> make sure that's not a transient failure.  If it does fail the second
> time, can you post the HTTP response from the server?
>
> > When I do the similar test by loading the url-http-create-request with
> > Thomas's changes, I can get a connection and grab my schedule
> > through the proxy.
>
> OK.
>
> > Let me know if I need to try something different.
>
> Are you in a position to build Emacs master tip and retry the experiment
> without patching anything?
>
> Thanks,
> Thomas
>
> > On Mon, Jun 17, 2019 at 4:08 PM Thomas Fitzsimmons <fitzsim <at> fitzsim.org> wrote:
> >>
> >> Hi,
> >>
> >> Good to hear that the patch I posted worked!
> >>
> >> Yes, that's the patch that Andreas's commit
> >> 84613dae5c34ea742dd9a3e56f5acb55f604b483 applied.  From what I can tell,
> >> you will not have that in Emacs 26.2.
> >>
> >> Can you try reverting my patch and applying Andreas's patch, and see if
> >> Excorporate still works through the proxy?
> >>
> >> Thanks,
> >> Thomas
> >>
> >> "tenspd137 ." <dcday137 <at> gmail.com> writes:
> >>
> >> > The patch Thomas seems to work from behind the proxy.  My current
> >> > emacs version is 26.2, so I would think it would include the commit
> >> > Andreas is talking about....  I went and looked it up - is this the
> >> > correct commit?
> >> >
> >> > diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
> >> > index 53798f7..817c5ce 100644
> >> > --- a/lisp/url/url-http.el
> >> > +++ b/lisp/url/url-http.el
> >> > @@ -1412,7 +1412,9 @@ The return value of this function is the
> >> > retrieval buffer."
> >> > 'url-http-wait-for-headers-change-function)
> >> > (set-process-filter tls-connection 'url-http-generic-filter)
> >> > (process-send-string tls-connection
> >> > - (url-http-create-request)))
> >> > + ;; Use the non-proxy form of the request
> >> > + (let (url-http-proxy)
> >> > + (url-http-create-request))))
> >> > (gnutls-error
> >> > (url-http-activate-callback)
> >> > (error "gnutls-error: %s" e))
> >> >
> >> > Thanks!
> >> >
> >> > -C
> >> >
> >> >
> >> > On Sat, Jun 15, 2019 at 1:41 AM Andreas Schwab <schwab <at> linux-m68k.org> wrote:
> >> >>
> >> >> On Jun 14 2019, Thomas Fitzsimmons <fitzsim <at> fitzsim.org> wrote:
> >> >>
> >> >> > diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
> >> >> > index 00803a103a..723d111d58 100644
> >> >> > --- a/lisp/url/url-http.el
> >> >> > +++ b/lisp/url/url-http.el
> >> >> > @@ -329,7 +329,10 @@ url-http-create-request
> >> >> >               ;; The request
> >> >> >               (or url-http-method "GET") " "
> >> >> >               (url-http--encode-string
> >> >> > -              (if using-proxy (url-recreate-url url-http-target-url) real-fname))
> >> >> > +              (if (and using-proxy
> >> >> > +                       (not (equal "https" (url-type url-http-target-url))))
> >> >> > +                  (url-recreate-url url-http-target-url)
> >> >> > +                real-fname))
> >> >>
> >> >> That should already be handled by commit 84613dae5c.
> >> >>
> >> >> Andreas.
> >> >>
> >> >> --
> >> >> Andreas Schwab, schwab <at> linux-m68k.org
> >> >> GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
> >> >> "And now for something completely different."

Message #97 received at 35969 <at> debbugs.gnu.org (full text, mbox):

From: Thomas Fitzsimmons <fitzsim <at> fitzsim.org>
To: "tenspd137 ." <dcday137 <at> gmail.com>
Cc: Andreas Schwab <schwab <at> linux-m68k.org>, 35969 <at> debbugs.gnu.org
Subject: Re: bug#35969: 26.2, Excorporate
Date: Tue, 09 Jul 2019 17:52:58 -0400

Hi,

Were you able to complete building Emacs?  I'd like to know if my patch
is needed on top of Andreas's to make Excorporate work through your
proxy.

I'm not sure why the built version would interfere with the current
installation.  If you were doing "make install" to a common prefix, that
might explain it.  Instead you can try something like this:

cd emacs-master [your github.com checkout]
make
mkdir test-home
HOME=`pwd`/test-home ./src/emacs -Q

That will ensure you're only running the built Emacs, and completely
ignoring the packages installed in your home directory.

Thomas

"tenspd137 ." <dcday137 <at> gmail.com> writes:

> I haven't tried the first item on the list yet - ie reloading Andreas'
> function.  i have tried to build emacs from
> https://github.com/emacs-mirror/emacs.git because I can't reach the
> official repo from my firewall for whatever reason.  I am sure it is
> something here.  Anyway, I was having problems because my built
> version (27.0.x) was getting seemingly mixed up with my current
> installation.  I can try again later, but right now things are pretty
> busy at work.
>
> Thanks.
>
> On Tue, Jun 18, 2019 at 10:26 PM Thomas Fitzsimmons <fitzsim <at> fitzsim.org> wrote:
>>
>> Hi,
>>
>> "tenspd137 ." <dcday137 <at> gmail.com> writes:
>>
>> > So - I am not sure if I did it correctly, but I copied this function
>> > with Andreas' changes into a file:
>> >
>> > (defun url-https-proxy-after-change-function (_st _nd _length)
>> >   (let* ((process-buffer (current-buffer))
>> >          (proc (get-buffer-process process-buffer)))
>> >     (goto-char (point-min))
>> >     (when (re-search-forward "^\r?\n" nil t)
>> >       (backward-char 1)
>> >       ;; Saw the end of the headers
>> >       (setq url-http-end-of-headers (set-marker (make-marker) (point)))
>> >       (url-http-parse-response)
>> >       (cond
>> >        ((null url-http-response-status)
>> >         ;; We got back a headerless malformed response from the
>> >         ;; server.
>> >         (url-http-activate-callback)
>> >         (error "Malformed response from proxy, fail!"))
>> >        ((= url-http-response-status 200)
>> >         (if (gnutls-available-p)
>> >             (condition-case e
>> >                 (let ((tls-connection (gnutls-negotiate
>> >                                        :process proc
>> >                                        :hostname (url-host url-current-object)
>> >                                        :verify-error nil)))
>> >                   ;; check certificate validity
>> >                   (setq tls-connection
>> >                         (nsm-verify-connection tls-connection
>> >                                                (url-host url-current-object)
>> >                                                (url-port url-current-object)))
>> >                   (with-current-buffer process-buffer (erase-buffer))
>> >                   (set-process-buffer tls-connection process-buffer)
>> >                   (setq url-http-after-change-function
>> >                         'url-http-wait-for-headers-change-function)
>> >                   (set-process-filter tls-connection 'url-http-generic-filter)
>> >                   (process-send-string tls-connection
>> >                                        ;; Use the non-proxy form of the request
>> >                                        (let (url-http-proxy)
>> >                                          (url-http-create-request))))
>> >               (gnutls-error
>> >                (url-http-activate-callback)
>> >                (error "gnutls-error: %s" e))
>> >               (error
>> >                (url-http-activate-callback)
>> >                (error "error: %s" e)))
>> >           (error "error: gnutls support needed!")))
>> >        (t
>> >         (url-http-debug "error response: %d" url-http-response-status)
>> >         (url-http-activate-callback))))))
>> >
>> > and then loaded it before running excorporate.  After that, I did M-x
>> > excorporate, and the minibuffer returns:  error in process filter:
>> > Server response is not an XML document
>>
>> In this scenario, if you immediately (without restarting Emacs/reloading
>> anything) re-run M-x excorporate does it still fail?  I just want to
>> make sure that's not a transient failure.  If it does fail the second
>> time, can you post the HTTP response from the server?
>>
>> > When I do the similar test by loading the url-http-create-request with
>> > Thomas's changes, I can get a connection and grab my schedule
>> > through the proxy.
>>
>> OK.
>>
>> > Let me know if I need to try something different.
>>
>> Are you in a position to build Emacs master tip and retry the experiment
>> without patching anything?
>>
>> Thanks,
>> Thomas
>>
>> > On Mon, Jun 17, 2019 at 4:08 PM Thomas Fitzsimmons <fitzsim <at> fitzsim.org> wrote:
>> >>
>> >> Hi,
>> >>
>> >> Good to hear that the patch I posted worked!
>> >>
>> >> Yes, that's the patch that Andreas's commit
>> >> 84613dae5c34ea742dd9a3e56f5acb55f604b483 applied.  From what I can tell,
>> >> you will not have that in Emacs 26.2.
>> >>
>> >> Can you try reverting my patch and applying Andreas's patch, and see if
>> >> Excorporate still works through the proxy?
>> >>
>> >> Thanks,
>> >> Thomas
>> >>
>> >> "tenspd137 ." <dcday137 <at> gmail.com> writes:
>> >>
>> >> > The patch Thomas seems to work from behind the proxy.  My current
>> >> > emacs version is 26.2, so I would think it would include the commit
>> >> > Andreas is talking about....  I went and looked it up - is this the
>> >> > correct commit?
>> >> >
>> >> > diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
>> >> > index 53798f7..817c5ce 100644
>> >> > --- a/lisp/url/url-http.el
>> >> > +++ b/lisp/url/url-http.el
>> >> > @@ -1412,7 +1412,9 @@ The return value of this function is the
>> >> > retrieval buffer."
>> >> > 'url-http-wait-for-headers-change-function)
>> >> > (set-process-filter tls-connection 'url-http-generic-filter)
>> >> > (process-send-string tls-connection
>> >> > - (url-http-create-request)))
>> >> > + ;; Use the non-proxy form of the request
>> >> > + (let (url-http-proxy)
>> >> > + (url-http-create-request))))
>> >> > (gnutls-error
>> >> > (url-http-activate-callback)
>> >> > (error "gnutls-error: %s" e))
>> >> >
>> >> > Thanks!
>> >> >
>> >> > -C
>> >> >
>> >> >
>> >> > On Sat, Jun 15, 2019 at 1:41 AM Andreas Schwab <schwab <at> linux-m68k.org> wrote:
>> >> >>
>> >> >> On Jun 14 2019, Thomas Fitzsimmons <fitzsim <at> fitzsim.org> wrote:
>> >> >>
>> >> >> > diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
>> >> >> > index 00803a103a..723d111d58 100644
>> >> >> > --- a/lisp/url/url-http.el
>> >> >> > +++ b/lisp/url/url-http.el
>> >> >> > @@ -329,7 +329,10 @@ url-http-create-request
>> >> >> >               ;; The request
>> >> >> >               (or url-http-method "GET") " "
>> >> >> >               (url-http--encode-string
>> >> >> > -              (if using-proxy (url-recreate-url url-http-target-url) real-fname))
>> >> >> > +              (if (and using-proxy
>> >> >> > +                       (not (equal "https" (url-type url-http-target-url))))
>> >> >> > +                  (url-recreate-url url-http-target-url)
>> >> >> > +                real-fname))
>> >> >>
>> >> >> That should already be handled by commit 84613dae5c.
>> >> >>
>> >> >> Andreas.
>> >> >>
>> >> >> --
>> >> >> Andreas Schwab, schwab <at> linux-m68k.org
>> >> >> GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
>> >> >> "And now for something completely different."

Message #100 received at 35969 <at> debbugs.gnu.org (full text, mbox):

From: Collin Day <dcday137 <at> gmail.com>
To: Thomas Fitzsimmons <fitzsim <at> fitzsim.org>
Cc: Andreas Schwab <schwab <at> linux-m68k.org>, 35969 <at> debbugs.gnu.org
Subject: Re: bug#35969: 26.2, Excorporate
Date: Tue, 9 Jul 2019 16:09:04 -0600

[Message part 1 (text/plain, inline)]

Sorry, I have not had the chance to.  A lot has been going on at my place
of employment.  It has crossed my mind, and as soon as I have a chance, I
will try what you suggested above.  Thanks, sorry for the inconvenience.

On Tue, Jul 9, 2019, 3:53 PM Thomas Fitzsimmons <fitzsim <at> fitzsim.org> wrote:

> Hi,
>
> Were you able to complete building Emacs?  I'd like to know if my patch
> is needed on top of Andreas's to make Excorporate work through your
> proxy.
>
> I'm not sure why the built version would interfere with the current
> installation.  If you were doing "make install" to a common prefix, that
> might explain it.  Instead you can try something like this:
>
> cd emacs-master [your github.com checkout]
> make
> mkdir test-home
> HOME=`pwd`/test-home ./src/emacs -Q
>
> That will ensure you're only running the built Emacs, and completely
> ignoring the packages installed in your home directory.
>
> Thomas
>
> "tenspd137 ." <dcday137 <at> gmail.com> writes:
>
> > I haven't tried the first item on the list yet - ie reloading Andreas'
> > function.  i have tried to build emacs from
> > https://github.com/emacs-mirror/emacs.git because I can't reach the
> > official repo from my firewall for whatever reason.  I am sure it is
> > something here.  Anyway, I was having problems because my built
> > version (27.0.x) was getting seemingly mixed up with my current
> > installation.  I can try again later, but right now things are pretty
> > busy at work.
> >
> > Thanks.
> >
> > On Tue, Jun 18, 2019 at 10:26 PM Thomas Fitzsimmons <fitzsim <at> fitzsim.org>
> wrote:
> >>
> >> Hi,
> >>
> >> "tenspd137 ." <dcday137 <at> gmail.com> writes:
> >>
> >> > So - I am not sure if I did it correctly, but I copied this function
> >> > with Andreas' changes into a file:
> >> >
> >> > (defun url-https-proxy-after-change-function (_st _nd _length)
> >> >   (let* ((process-buffer (current-buffer))
> >> >          (proc (get-buffer-process process-buffer)))
> >> >     (goto-char (point-min))
> >> >     (when (re-search-forward "^\r?\n" nil t)
> >> >       (backward-char 1)
> >> >       ;; Saw the end of the headers
> >> >       (setq url-http-end-of-headers (set-marker (make-marker)
> (point)))
> >> >       (url-http-parse-response)
> >> >       (cond
> >> >        ((null url-http-response-status)
> >> >         ;; We got back a headerless malformed response from the
> >> >         ;; server.
> >> >         (url-http-activate-callback)
> >> >         (error "Malformed response from proxy, fail!"))
> >> >        ((= url-http-response-status 200)
> >> >         (if (gnutls-available-p)
> >> >             (condition-case e
> >> >                 (let ((tls-connection (gnutls-negotiate
> >> >                                        :process proc
> >> >                                        :hostname (url-host
> url-current-object)
> >> >                                        :verify-error nil)))
> >> >                   ;; check certificate validity
> >> >                   (setq tls-connection
> >> >                         (nsm-verify-connection tls-connection
> >> >                                                (url-host
> url-current-object)
> >> >                                                (url-port
> url-current-object)))
> >> >                   (with-current-buffer process-buffer (erase-buffer))
> >> >                   (set-process-buffer tls-connection process-buffer)
> >> >                   (setq url-http-after-change-function
> >> >                         'url-http-wait-for-headers-change-function)
> >> >                   (set-process-filter tls-connection
> 'url-http-generic-filter)
> >> >                   (process-send-string tls-connection
> >> >                                        ;; Use the non-proxy form of
> the request
> >> >                                        (let (url-http-proxy)
> >> >                                          (url-http-create-request))))
> >> >               (gnutls-error
> >> >                (url-http-activate-callback)
> >> >                (error "gnutls-error: %s" e))
> >> >               (error
> >> >                (url-http-activate-callback)
> >> >                (error "error: %s" e)))
> >> >           (error "error: gnutls support needed!")))
> >> >        (t
> >> >         (url-http-debug "error response: %d" url-http-response-status)
> >> >         (url-http-activate-callback))))))
> >> >
> >> > and then loaded it before running excorporate.  After that, I did M-x
> >> > excorporate, and the minibuffer returns:  error in process filter:
> >> > Server response is not an XML document
> >>
> >> In this scenario, if you immediately (without restarting Emacs/reloading
> >> anything) re-run M-x excorporate does it still fail?  I just want to
> >> make sure that's not a transient failure.  If it does fail the second
> >> time, can you post the HTTP response from the server?
> >>
> >> > When I do the similar test by loading the url-http-create-request with
> >> > Thomas's changes, I can get a connection and grab my schedule
> >> > through the proxy.
> >>
> >> OK.
> >>
> >> > Let me know if I need to try something different.
> >>
> >> Are you in a position to build Emacs master tip and retry the experiment
> >> without patching anything?
> >>
> >> Thanks,
> >> Thomas
> >>
> >> > On Mon, Jun 17, 2019 at 4:08 PM Thomas Fitzsimmons <
> fitzsim <at> fitzsim.org> wrote:
> >> >>
> >> >> Hi,
> >> >>
> >> >> Good to hear that the patch I posted worked!
> >> >>
> >> >> Yes, that's the patch that Andreas's commit
> >> >> 84613dae5c34ea742dd9a3e56f5acb55f604b483 applied.  From what I can
> tell,
> >> >> you will not have that in Emacs 26.2.
> >> >>
> >> >> Can you try reverting my patch and applying Andreas's patch, and see
> if
> >> >> Excorporate still works through the proxy?
> >> >>
> >> >> Thanks,
> >> >> Thomas
> >> >>
> >> >> "tenspd137 ." <dcday137 <at> gmail.com> writes:
> >> >>
> >> >> > The patch Thomas seems to work from behind the proxy.  My current
> >> >> > emacs version is 26.2, so I would think it would include the commit
> >> >> > Andreas is talking about....  I went and looked it up - is this the
> >> >> > correct commit?
> >> >> >
> >> >> > diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
> >> >> > index 53798f7..817c5ce 100644
> >> >> > --- a/lisp/url/url-http.el
> >> >> > +++ b/lisp/url/url-http.el
> >> >> > @@ -1412,7 +1412,9 @@ The return value of this function is the
> >> >> > retrieval buffer."
> >> >> > 'url-http-wait-for-headers-change-function)
> >> >> > (set-process-filter tls-connection 'url-http-generic-filter)
> >> >> > (process-send-string tls-connection
> >> >> > - (url-http-create-request)))
> >> >> > + ;; Use the non-proxy form of the request
> >> >> > + (let (url-http-proxy)
> >> >> > + (url-http-create-request))))
> >> >> > (gnutls-error
> >> >> > (url-http-activate-callback)
> >> >> > (error "gnutls-error: %s" e))
> >> >> >
> >> >> > Thanks!
> >> >> >
> >> >> > -C
> >> >> >
> >> >> >
> >> >> > On Sat, Jun 15, 2019 at 1:41 AM Andreas Schwab <
> schwab <at> linux-m68k.org> wrote:
> >> >> >>
> >> >> >> On Jun 14 2019, Thomas Fitzsimmons <fitzsim <at> fitzsim.org> wrote:
> >> >> >>
> >> >> >> > diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
> >> >> >> > index 00803a103a..723d111d58 100644
> >> >> >> > --- a/lisp/url/url-http.el
> >> >> >> > +++ b/lisp/url/url-http.el
> >> >> >> > @@ -329,7 +329,10 @@ url-http-create-request
> >> >> >> >               ;; The request
> >> >> >> >               (or url-http-method "GET") " "
> >> >> >> >               (url-http--encode-string
> >> >> >> > -              (if using-proxy (url-recreate-url
> url-http-target-url) real-fname))
> >> >> >> > +              (if (and using-proxy
> >> >> >> > +                       (not (equal "https" (url-type
> url-http-target-url))))
> >> >> >> > +                  (url-recreate-url url-http-target-url)
> >> >> >> > +                real-fname))
> >> >> >>
> >> >> >> That should already be handled by commit 84613dae5c.
> >> >> >>
> >> >> >> Andreas.
> >> >> >>
> >> >> >> --
> >> >> >> Andreas Schwab, schwab <at> linux-m68k.org
> >> >> >> GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780
> A9DA AEC1
> >> >> >> "And now for something completely different."
>

[Message part 2 (text/html, inline)]

Message #103 received at 35969 <at> debbugs.gnu.org (full text, mbox):

From: Collin Day <dcday137 <at> gmail.com>
To: Thomas Fitzsimmons <fitzsim <at> fitzsim.org>
Cc: Andreas Schwab <schwab <at> linux-m68k.org>, 35969 <at> debbugs.gnu.org
Subject: Re: bug#35969: 26.2, Excorporate
Date: Tue, 9 Jul 2019 16:53:39 -0600

Did a git pull and followed instructions above.

There are some issues.  First, I need to run M-x package-install <RET>
excorporate <RET>  four times because I see (each line after each
invocation)

package--with-response-buffer-1:
https://elpa.gnu.org/packages/url-http-ntlm-2.0.4.el: Method not
allowed
package--with-response-buffer-1:
https://elpa.gnu.org/packages/fsm-0.2.1.el: Method not allowed
package--with-response-buffer-1:
https://elpa.gnu.org/packages/excorporate-0.8.3.tar: Method not
allowed

After the 4th call it compiles and becomes available.

M-x customize-group <RET> excorporate, set up for no autoconfig right
now, hit apply for current sessions, save not availiable....

M-x excorporate <RET>

enter uname and password

Contacting host: outlook.office365.com:443
error in process filter: exco--parse-xml-in-current-buffer: Server
response is not an XML document
error in process filter: Server response is not an XML documen

*http outlook.office.365.com:443*

HTTP/1.1 401 Unauthorized
Server: Microsoft-IIS/10.0
request-id: f8e80f64-b51d-4d66-88ef-4ee7f09814f3
X-WSSecurity-Enabled: True
X-WSSecurity-For: Logon
X-FederationTrustTokenIssuerUri: urn:federation:MicrosoftOnline
X-WSSecurity-SymmetricKey-Enabled: True
X-WSSecurity-X509Cert-Enabled: True
X-OAuth-Enabled: True
X-Powered-By: ASP.NET
X-FEServer: SN4PR0401CA0031
WWW-Authenticate: Basic Realm=""
Date: Tue, 09 Jul 2019 22:48:22 GMT
Content-Length: 0

URL-DEBUG
http -> Saw end of headers... ( *http outlook.office365.com:443*)
http -> url-http-parse-response called in ( *http outlook.office365.com:443*)
http -> Got a content-length, being smart about document end.
http -> Got 0-length content-length, activating callback immediately.
http -> Marking connection as free: outlook.office365.com:443
#<process outlook.office365.com>
http -> url-http-parse-headers called in ( *http outlook.office365.com:443*)
http -> url-http-parse-response called in ( *http outlook.office365.com:443*)
http -> Parsed HTTP headers: class=4 status=401
http -> Handling normal authentication
http -> Found existing connection: outlook.office365.com:443 #<process
outlook.office365.com>
http -> Reusing existing connection: outlook.office365.com:443
http -> Marking connection as busy: outlook.office365.com:443
#<process outlook.office365.com>
http -> getting referer from buffer: buffer:#<buffer  *http
outlook.office365.com:443*> target-url:#s(url "https" nil nil
"outlook.office365.com" nil "/EWS/Services.wsdl" nil nil t nil t t)
lastloc:nil
http -> Finished parsing HTTP headers: nil
http -> Spinning waiting for headers...
http -> Calling after change function
`url-https-proxy-after-change-function' for `#<process
outlook.office365.com>'
http -> url-http-parse-response called in ( *http
outlook.office365.com:443*-535296)
http -> error response: 400
http -> Marking connection as free: outlook.office365.com:443
#<process outlook.office365.com>
http -> Activating callback in buffer ( *http
outlook.office365.com:443*-535296): #[257
"p\303\304\305\306\307 !\310\"\311$\216
@\312=\203$\0\313\301\314\315\316\302\"#\210\317\300\320\"\202/\0\313\301\321\322
#\210\317\300\323\")\207" [fsm-exco--fsm-0 (:identifier ("my email" .
"https://outlook.office365.com/EWS/Excahnge.asmx") :mail-address "my
email" :retrying nil :autodiscovery-urls nil :service-url
"https://outlook.office365.com/EWS/Excahnge.asmx" :service-xml nil
:service-wsdl nil :next-state-after-success nil :failure-message nil
:server-version nil) "https://outlook.office365.com/EWS/Services.wsdl"
make-byte-code 0 "\301\300!\205    \0\302\300!\207" vconcat vector
[buffer-live-p kill-buffer] 2 :error plist-put :failure-message format
"Failed to retrieve %s" fsm-send :unrecoverable-error :service-xml
exco--parse-xml-in-current-buffer :success] 8 "

(fn STATUS)"] ((:peer (:certificates ((:version 3 :serial-number
"0a:48:28:08:de:bc:a4:10:92:0f:87:6a:63:05:a1:e9" :issuer
"C=US,O=DigiCert Inc,CN=DigiCert Cloud Services CA-1" :valid-from
"2018-11-18" :valid-to "2020-11-18" :subject
"C=US,ST=Washington,L=Redmond,O=Microsoft Corporation,CN=outlook.com"
:public-key-algorithm "RSA" :certificate-security-level "Medium"
:signature-algorithm "RSA-SHA256" :public-key-id
"sha1:ae:b5:9e:3f:4e:c3:72:a8:c0:fc:24:4e:24:b4:d6:8d:99:b0:b9:e1"
:certificate-id
"sha1:05:b8:36:4c:33:e5:63:7e:fd:88:e0:a3:b8:7e:7d:cf:6f:8d:d0:d4")
(:version 3 :serial-number
"01:9e:c1:c6:bd:3f:59:7b:b2:0c:33:38:e5:51:d8:77" :issuer
"C=US,O=DigiCert Inc,OU=www.digicert.com,CN=DigiCert Global Root CA"
:valid-from "2015-08-04" :valid-to "2030-08-04" :subject
"C=US,O=DigiCert Inc,CN=DigiCert Cloud Services CA-1"
:public-key-algorithm "RSA" :certificate-security-level "Medium"
:signature-algorithm "RSA-SHA256" :public-key-id
"sha1:ac:13:d5:79:03:66:f3:cc:ff:fe:7c:23:71:a7:61:5d:39:20:89:6e"
:certificate-id
"sha1:81:b6:8d:6c:d2:f2:21:f8:f5:34:e6:77:52:3b:b2:36:bb:a1:dc:56"))
:certificate (:version 3 :serial-number
"0a:48:28:08:de:bc:a4:10:92:0f:87:6a:63:05:a1:e9" :issuer
"C=US,O=DigiCert Inc,CN=DigiCert Cloud Services CA-1" :valid-from
"2018-11-18" :valid-to "2020-11-18" :subject
"C=US,ST=Washington,L=Redmond,O=Microsoft Corporation,CN=outlook.com"
:public-key-algorithm "RSA" :certificate-security-level "Medium"
:signature-algorithm "RSA-SHA256" :public-key-id
"sha1:ae:b5:9e:3f:4e:c3:72:a8:c0:fc:24:4e:24:b4:d6:8d:99:b0:b9:e1"
:certificate-id
"sha1:05:b8:36:4c:33:e5:63:7e:fd:88:e0:a3:b8:7e:7d:cf:6f:8d:d0:d4")
:key-exchange "ECDHE-RSA" :protocol "TLS1.2" :cipher "AES-256-GCM"
:mac "AEAD")))

So it appears not to work, at least on the current head as of today....

Thanks!

-C

On Tue, Jul 9, 2019 at 4:09 PM Collin Day <dcday137 <at> gmail.com> wrote:
>
> Sorry, I have not had the chance to.  A lot has been going on at my place of employment.  It has crossed my mind, and as soon as I have a chance, I will try what you suggested above.  Thanks, sorry for the inconvenience.
>
> On Tue, Jul 9, 2019, 3:53 PM Thomas Fitzsimmons <fitzsim <at> fitzsim.org> wrote:
>>
>> Hi,
>>
>> Were you able to complete building Emacs?  I'd like to know if my patch
>> is needed on top of Andreas's to make Excorporate work through your
>> proxy.
>>
>> I'm not sure why the built version would interfere with the current
>> installation.  If you were doing "make install" to a common prefix, that
>> might explain it.  Instead you can try something like this:
>>
>> cd emacs-master [your github.com checkout]
>> make
>> mkdir test-home
>> HOME=`pwd`/test-home ./src/emacs -Q
>>
>> That will ensure you're only running the built Emacs, and completely
>> ignoring the packages installed in your home directory.
>>
>> Thomas
>>
>> "tenspd137 ." <dcday137 <at> gmail.com> writes:
>>
>> > I haven't tried the first item on the list yet - ie reloading Andreas'
>> > function.  i have tried to build emacs from
>> > https://github.com/emacs-mirror/emacs.git because I can't reach the
>> > official repo from my firewall for whatever reason.  I am sure it is
>> > something here.  Anyway, I was having problems because my built
>> > version (27.0.x) was getting seemingly mixed up with my current
>> > installation.  I can try again later, but right now things are pretty
>> > busy at work.
>> >
>> > Thanks.
>> >
>> > On Tue, Jun 18, 2019 at 10:26 PM Thomas Fitzsimmons <fitzsim <at> fitzsim.org> wrote:
>> >>
>> >> Hi,
>> >>
>> >> "tenspd137 ." <dcday137 <at> gmail.com> writes:
>> >>
>> >> > So - I am not sure if I did it correctly, but I copied this function
>> >> > with Andreas' changes into a file:
>> >> >
>> >> > (defun url-https-proxy-after-change-function (_st _nd _length)
>> >> >   (let* ((process-buffer (current-buffer))
>> >> >          (proc (get-buffer-process process-buffer)))
>> >> >     (goto-char (point-min))
>> >> >     (when (re-search-forward "^\r?\n" nil t)
>> >> >       (backward-char 1)
>> >> >       ;; Saw the end of the headers
>> >> >       (setq url-http-end-of-headers (set-marker (make-marker) (point)))
>> >> >       (url-http-parse-response)
>> >> >       (cond
>> >> >        ((null url-http-response-status)
>> >> >         ;; We got back a headerless malformed response from the
>> >> >         ;; server.
>> >> >         (url-http-activate-callback)
>> >> >         (error "Malformed response from proxy, fail!"))
>> >> >        ((= url-http-response-status 200)
>> >> >         (if (gnutls-available-p)
>> >> >             (condition-case e
>> >> >                 (let ((tls-connection (gnutls-negotiate
>> >> >                                        :process proc
>> >> >                                        :hostname (url-host url-current-object)
>> >> >                                        :verify-error nil)))
>> >> >                   ;; check certificate validity
>> >> >                   (setq tls-connection
>> >> >                         (nsm-verify-connection tls-connection
>> >> >                                                (url-host url-current-object)
>> >> >                                                (url-port url-current-object)))
>> >> >                   (with-current-buffer process-buffer (erase-buffer))
>> >> >                   (set-process-buffer tls-connection process-buffer)
>> >> >                   (setq url-http-after-change-function
>> >> >                         'url-http-wait-for-headers-change-function)
>> >> >                   (set-process-filter tls-connection 'url-http-generic-filter)
>> >> >                   (process-send-string tls-connection
>> >> >                                        ;; Use the non-proxy form of the request
>> >> >                                        (let (url-http-proxy)
>> >> >                                          (url-http-create-request))))
>> >> >               (gnutls-error
>> >> >                (url-http-activate-callback)
>> >> >                (error "gnutls-error: %s" e))
>> >> >               (error
>> >> >                (url-http-activate-callback)
>> >> >                (error "error: %s" e)))
>> >> >           (error "error: gnutls support needed!")))
>> >> >        (t
>> >> >         (url-http-debug "error response: %d" url-http-response-status)
>> >> >         (url-http-activate-callback))))))
>> >> >
>> >> > and then loaded it before running excorporate.  After that, I did M-x
>> >> > excorporate, and the minibuffer returns:  error in process filter:
>> >> > Server response is not an XML document
>> >>
>> >> In this scenario, if you immediately (without restarting Emacs/reloading
>> >> anything) re-run M-x excorporate does it still fail?  I just want to
>> >> make sure that's not a transient failure.  If it does fail the second
>> >> time, can you post the HTTP response from the server?
>> >>
>> >> > When I do the similar test by loading the url-http-create-request with
>> >> > Thomas's changes, I can get a connection and grab my schedule
>> >> > through the proxy.
>> >>
>> >> OK.
>> >>
>> >> > Let me know if I need to try something different.
>> >>
>> >> Are you in a position to build Emacs master tip and retry the experiment
>> >> without patching anything?
>> >>
>> >> Thanks,
>> >> Thomas
>> >>
>> >> > On Mon, Jun 17, 2019 at 4:08 PM Thomas Fitzsimmons <fitzsim <at> fitzsim.org> wrote:
>> >> >>
>> >> >> Hi,
>> >> >>
>> >> >> Good to hear that the patch I posted worked!
>> >> >>
>> >> >> Yes, that's the patch that Andreas's commit
>> >> >> 84613dae5c34ea742dd9a3e56f5acb55f604b483 applied.  From what I can tell,
>> >> >> you will not have that in Emacs 26.2.
>> >> >>
>> >> >> Can you try reverting my patch and applying Andreas's patch, and see if
>> >> >> Excorporate still works through the proxy?
>> >> >>
>> >> >> Thanks,
>> >> >> Thomas
>> >> >>
>> >> >> "tenspd137 ." <dcday137 <at> gmail.com> writes:
>> >> >>
>> >> >> > The patch Thomas seems to work from behind the proxy.  My current
>> >> >> > emacs version is 26.2, so I would think it would include the commit
>> >> >> > Andreas is talking about....  I went and looked it up - is this the
>> >> >> > correct commit?
>> >> >> >
>> >> >> > diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
>> >> >> > index 53798f7..817c5ce 100644
>> >> >> > --- a/lisp/url/url-http.el
>> >> >> > +++ b/lisp/url/url-http.el
>> >> >> > @@ -1412,7 +1412,9 @@ The return value of this function is the
>> >> >> > retrieval buffer."
>> >> >> > 'url-http-wait-for-headers-change-function)
>> >> >> > (set-process-filter tls-connection 'url-http-generic-filter)
>> >> >> > (process-send-string tls-connection
>> >> >> > - (url-http-create-request)))
>> >> >> > + ;; Use the non-proxy form of the request
>> >> >> > + (let (url-http-proxy)
>> >> >> > + (url-http-create-request))))
>> >> >> > (gnutls-error
>> >> >> > (url-http-activate-callback)
>> >> >> > (error "gnutls-error: %s" e))
>> >> >> >
>> >> >> > Thanks!
>> >> >> >
>> >> >> > -C
>> >> >> >
>> >> >> >
>> >> >> > On Sat, Jun 15, 2019 at 1:41 AM Andreas Schwab <schwab <at> linux-m68k.org> wrote:
>> >> >> >>
>> >> >> >> On Jun 14 2019, Thomas Fitzsimmons <fitzsim <at> fitzsim.org> wrote:
>> >> >> >>
>> >> >> >> > diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
>> >> >> >> > index 00803a103a..723d111d58 100644
>> >> >> >> > --- a/lisp/url/url-http.el
>> >> >> >> > +++ b/lisp/url/url-http.el
>> >> >> >> > @@ -329,7 +329,10 @@ url-http-create-request
>> >> >> >> >               ;; The request
>> >> >> >> >               (or url-http-method "GET") " "
>> >> >> >> >               (url-http--encode-string
>> >> >> >> > -              (if using-proxy (url-recreate-url url-http-target-url) real-fname))
>> >> >> >> > +              (if (and using-proxy
>> >> >> >> > +                       (not (equal "https" (url-type url-http-target-url))))
>> >> >> >> > +                  (url-recreate-url url-http-target-url)
>> >> >> >> > +                real-fname))
>> >> >> >>
>> >> >> >> That should already be handled by commit 84613dae5c.
>> >> >> >>
>> >> >> >> Andreas.
>> >> >> >>
>> >> >> >> --
>> >> >> >> Andreas Schwab, schwab <at> linux-m68k.org
>> >> >> >> GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
>> >> >> >> "And now for something completely different."

Message #106 received at 35969 <at> debbugs.gnu.org (full text, mbox):

From: Thomas Fitzsimmons <fitzsim <at> fitzsim.org>
To: Collin Day <dcday137 <at> gmail.com>
Cc: Andreas Schwab <schwab <at> linux-m68k.org>, 35969 <at> debbugs.gnu.org
Subject: Re: bug#35969: 26.2, Excorporate
Date: Tue, 09 Jul 2019 20:08:50 -0400

OK, can you apply my patch, then run "make" again and confirm it works?

Thanks,
Thomas

Collin Day <dcday137 <at> gmail.com> writes:

> Did a git pull and followed instructions above.
>
> There are some issues.  First, I need to run M-x package-install <RET>
> excorporate <RET>  four times because I see (each line after each
> invocation)
>
> package--with-response-buffer-1:
> https://elpa.gnu.org/packages/url-http-ntlm-2.0.4.el: Method not
> allowed
> package--with-response-buffer-1:
> https://elpa.gnu.org/packages/fsm-0.2.1.el: Method not allowed
> package--with-response-buffer-1:
> https://elpa.gnu.org/packages/excorporate-0.8.3.tar: Method not
> allowed
>
> After the 4th call it compiles and becomes available.
>
> M-x customize-group <RET> excorporate, set up for no autoconfig right
> now, hit apply for current sessions, save not availiable....
>
> M-x excorporate <RET>
>
> enter uname and password
>
> Contacting host: outlook.office365.com:443
> error in process filter: exco--parse-xml-in-current-buffer: Server
> response is not an XML document
> error in process filter: Server response is not an XML documen
>
> *http outlook.office.365.com:443*

[...]

> So it appears not to work, at least on the current head as of today....
>
> Thanks!
>
> -C
>
> On Tue, Jul 9, 2019 at 4:09 PM Collin Day <dcday137 <at> gmail.com> wrote:
>>
>> Sorry, I have not had the chance to.  A lot has been going on at my
>> place of employment.  It has crossed my mind, and as soon as I have
>> a chance, I will try what you suggested above.  Thanks, sorry for
>> the inconvenience.
>>
>> On Tue, Jul 9, 2019, 3:53 PM Thomas Fitzsimmons <fitzsim <at> fitzsim.org> wrote:
>>>
>>> Hi,
>>>
>>> Were you able to complete building Emacs?  I'd like to know if my patch
>>> is needed on top of Andreas's to make Excorporate work through your
>>> proxy.
>>>
>>> I'm not sure why the built version would interfere with the current
>>> installation.  If you were doing "make install" to a common prefix, that
>>> might explain it.  Instead you can try something like this:
>>>
>>> cd emacs-master [your github.com checkout]
>>> make
>>> mkdir test-home
>>> HOME=`pwd`/test-home ./src/emacs -Q
>>>
>>> That will ensure you're only running the built Emacs, and completely
>>> ignoring the packages installed in your home directory.
>>>
>>> Thomas
>>>
>>> "tenspd137 ." <dcday137 <at> gmail.com> writes:
>>>
>>> > I haven't tried the first item on the list yet - ie reloading Andreas'
>>> > function.  i have tried to build emacs from
>>> > https://github.com/emacs-mirror/emacs.git because I can't reach the
>>> > official repo from my firewall for whatever reason.  I am sure it is
>>> > something here.  Anyway, I was having problems because my built
>>> > version (27.0.x) was getting seemingly mixed up with my current
>>> > installation.  I can try again later, but right now things are pretty
>>> > busy at work.
>>> >
>>> > Thanks.
>>> >
>>> > On Tue, Jun 18, 2019 at 10:26 PM Thomas Fitzsimmons <fitzsim <at> fitzsim.org> wrote:
>>> >>
>>> >> Hi,
>>> >>
>>> >> "tenspd137 ." <dcday137 <at> gmail.com> writes:
>>> >>
>>> >> > So - I am not sure if I did it correctly, but I copied this function
>>> >> > with Andreas' changes into a file:
>>> >> >
>>> >> > (defun url-https-proxy-after-change-function (_st _nd _length)
>>> >> >   (let* ((process-buffer (current-buffer))
>>> >> >          (proc (get-buffer-process process-buffer)))
>>> >> >     (goto-char (point-min))
>>> >> >     (when (re-search-forward "^\r?\n" nil t)
>>> >> >       (backward-char 1)
>>> >> >       ;; Saw the end of the headers
>>> >> >       (setq url-http-end-of-headers (set-marker (make-marker) (point)))
>>> >> >       (url-http-parse-response)
>>> >> >       (cond
>>> >> >        ((null url-http-response-status)
>>> >> >         ;; We got back a headerless malformed response from the
>>> >> >         ;; server.
>>> >> >         (url-http-activate-callback)
>>> >> >         (error "Malformed response from proxy, fail!"))
>>> >> >        ((= url-http-response-status 200)
>>> >> >         (if (gnutls-available-p)
>>> >> >             (condition-case e
>>> >> >                 (let ((tls-connection (gnutls-negotiate
>>> >> >                                        :process proc
>>> >> >                                        :hostname (url-host url-current-object)
>>> >> >                                        :verify-error nil)))
>>> >> >                   ;; check certificate validity
>>> >> >                   (setq tls-connection
>>> >> >                         (nsm-verify-connection tls-connection
>>> >> >                                                (url-host url-current-object)
>>> >> >                                                (url-port url-current-object)))
>>> >> >                   (with-current-buffer process-buffer (erase-buffer))
>>> >> >                   (set-process-buffer tls-connection process-buffer)
>>> >> >                   (setq url-http-after-change-function
>>> >> >                         'url-http-wait-for-headers-change-function)
>>> >> >                   (set-process-filter tls-connection 'url-http-generic-filter)
>>> >> >                   (process-send-string tls-connection
>>> >> >                                        ;; Use the non-proxy form of the request
>>> >> >                                        (let (url-http-proxy)
>>> >> >                                          (url-http-create-request))))
>>> >> >               (gnutls-error
>>> >> >                (url-http-activate-callback)
>>> >> >                (error "gnutls-error: %s" e))
>>> >> >               (error
>>> >> >                (url-http-activate-callback)
>>> >> >                (error "error: %s" e)))
>>> >> >           (error "error: gnutls support needed!")))
>>> >> >        (t
>>> >> >         (url-http-debug "error response: %d" url-http-response-status)
>>> >> >         (url-http-activate-callback))))))
>>> >> >
>>> >> > and then loaded it before running excorporate.  After that, I did M-x
>>> >> > excorporate, and the minibuffer returns:  error in process filter:
>>> >> > Server response is not an XML document
>>> >>
>>> >> In this scenario, if you immediately (without restarting Emacs/reloading
>>> >> anything) re-run M-x excorporate does it still fail?  I just want to
>>> >> make sure that's not a transient failure.  If it does fail the second
>>> >> time, can you post the HTTP response from the server?
>>> >>
>>> >> > When I do the similar test by loading the url-http-create-request with
>>> >> > Thomas's changes, I can get a connection and grab my schedule
>>> >> > through the proxy.
>>> >>
>>> >> OK.
>>> >>
>>> >> > Let me know if I need to try something different.
>>> >>
>>> >> Are you in a position to build Emacs master tip and retry the experiment
>>> >> without patching anything?
>>> >>
>>> >> Thanks,
>>> >> Thomas
>>> >>
>>> >> > On Mon, Jun 17, 2019 at 4:08 PM Thomas Fitzsimmons <fitzsim <at> fitzsim.org> wrote:
>>> >> >>
>>> >> >> Hi,
>>> >> >>
>>> >> >> Good to hear that the patch I posted worked!
>>> >> >>
>>> >> >> Yes, that's the patch that Andreas's commit
>>> >> >> 84613dae5c34ea742dd9a3e56f5acb55f604b483 applied.  From what I can tell,
>>> >> >> you will not have that in Emacs 26.2.
>>> >> >>
>>> >> >> Can you try reverting my patch and applying Andreas's patch, and see if
>>> >> >> Excorporate still works through the proxy?
>>> >> >>
>>> >> >> Thanks,
>>> >> >> Thomas
>>> >> >>
>>> >> >> "tenspd137 ." <dcday137 <at> gmail.com> writes:
>>> >> >>
>>> >> >> > The patch Thomas seems to work from behind the proxy.  My current
>>> >> >> > emacs version is 26.2, so I would think it would include the commit
>>> >> >> > Andreas is talking about....  I went and looked it up - is this the
>>> >> >> > correct commit?
>>> >> >> >
>>> >> >> > diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
>>> >> >> > index 53798f7..817c5ce 100644
>>> >> >> > --- a/lisp/url/url-http.el
>>> >> >> > +++ b/lisp/url/url-http.el
>>> >> >> > @@ -1412,7 +1412,9 @@ The return value of this function is the
>>> >> >> > retrieval buffer."
>>> >> >> > 'url-http-wait-for-headers-change-function)
>>> >> >> > (set-process-filter tls-connection 'url-http-generic-filter)
>>> >> >> > (process-send-string tls-connection
>>> >> >> > - (url-http-create-request)))
>>> >> >> > + ;; Use the non-proxy form of the request
>>> >> >> > + (let (url-http-proxy)
>>> >> >> > + (url-http-create-request))))
>>> >> >> > (gnutls-error
>>> >> >> > (url-http-activate-callback)
>>> >> >> > (error "gnutls-error: %s" e))
>>> >> >> >
>>> >> >> > Thanks!
>>> >> >> >
>>> >> >> > -C
>>> >> >> >
>>> >> >> >
>>> >> >> > On Sat, Jun 15, 2019 at 1:41 AM Andreas Schwab <schwab <at> linux-m68k.org> wrote:
>>> >> >> >>
>>> >> >> >> On Jun 14 2019, Thomas Fitzsimmons <fitzsim <at> fitzsim.org> wrote:
>>> >> >> >>
>>> >> >> >> > diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
>>> >> >> >> > index 00803a103a..723d111d58 100644
>>> >> >> >> > --- a/lisp/url/url-http.el
>>> >> >> >> > +++ b/lisp/url/url-http.el
>>> >> >> >> > @@ -329,7 +329,10 @@ url-http-create-request
>>> >> >> >> >               ;; The request
>>> >> >> >> >               (or url-http-method "GET") " "
>>> >> >> >> >               (url-http--encode-string
>>> >> >> >> > -              (if using-proxy (url-recreate-url url-http-target-url) real-fname))
>>> >> >> >> > +              (if (and using-proxy
>>> >> >> >> > +                       (not (equal "https" (url-type url-http-target-url))))
>>> >> >> >> > +                  (url-recreate-url url-http-target-url)
>>> >> >> >> > +                real-fname))
>>> >> >> >>
>>> >> >> >> That should already be handled by commit 84613dae5c.
>>> >> >> >>
>>> >> >> >> Andreas.
>>> >> >> >>
>>> >> >> >> --
>>> >> >> >> Andreas Schwab, schwab <at> linux-m68k.org
>>> >> >> >> GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
>>> >> >> >> "And now for something completely different."

Message #109 received at 35969 <at> debbugs.gnu.org (full text, mbox):

From: Thomas Fitzsimmons <fitzsim <at> fitzsim.org>
To: Collin Day <dcday137 <at> gmail.com>
Cc: Andreas Schwab <schwab <at> linux-m68k.org>, 35969 <at> debbugs.gnu.org
Subject: Re: bug#35969: 26.2, Excorporate
Date: Tue, 09 Jul 2019 20:11:40 -0400

I think there should be no need to remove test-home, so you won't have
to re-run the M-x package-install steps.

Thomas

Thomas Fitzsimmons <fitzsim <at> fitzsim.org> writes:

> OK, can you apply my patch, then run "make" again and confirm it works?
>
> Thanks,
> Thomas
>
> Collin Day <dcday137 <at> gmail.com> writes:
>
>> Did a git pull and followed instructions above.
>>
>> There are some issues.  First, I need to run M-x package-install <RET>
>> excorporate <RET>  four times because I see (each line after each
>> invocation)
>>
>> package--with-response-buffer-1:
>> https://elpa.gnu.org/packages/url-http-ntlm-2.0.4.el: Method not
>> allowed
>> package--with-response-buffer-1:
>> https://elpa.gnu.org/packages/fsm-0.2.1.el: Method not allowed
>> package--with-response-buffer-1:
>> https://elpa.gnu.org/packages/excorporate-0.8.3.tar: Method not
>> allowed
>>
>> After the 4th call it compiles and becomes available.
>>
>> M-x customize-group <RET> excorporate, set up for no autoconfig right
>> now, hit apply for current sessions, save not availiable....
>>
>> M-x excorporate <RET>
>>
>> enter uname and password
>>
>> Contacting host: outlook.office365.com:443
>> error in process filter: exco--parse-xml-in-current-buffer: Server
>> response is not an XML document
>> error in process filter: Server response is not an XML documen
>>
>> *http outlook.office.365.com:443*
>
> [...]
>
>> So it appears not to work, at least on the current head as of today....
>>
>> Thanks!
>>
>> -C
>>
>> On Tue, Jul 9, 2019 at 4:09 PM Collin Day <dcday137 <at> gmail.com> wrote:
>>>
>>> Sorry, I have not had the chance to.  A lot has been going on at my
>>> place of employment.  It has crossed my mind, and as soon as I have
>>> a chance, I will try what you suggested above.  Thanks, sorry for
>>> the inconvenience.
>>>
>>> On Tue, Jul 9, 2019, 3:53 PM Thomas Fitzsimmons <fitzsim <at> fitzsim.org> wrote:
>>>>
>>>> Hi,
>>>>
>>>> Were you able to complete building Emacs?  I'd like to know if my patch
>>>> is needed on top of Andreas's to make Excorporate work through your
>>>> proxy.
>>>>
>>>> I'm not sure why the built version would interfere with the current
>>>> installation.  If you were doing "make install" to a common prefix, that
>>>> might explain it.  Instead you can try something like this:
>>>>
>>>> cd emacs-master [your github.com checkout]
>>>> make
>>>> mkdir test-home
>>>> HOME=`pwd`/test-home ./src/emacs -Q
>>>>
>>>> That will ensure you're only running the built Emacs, and completely
>>>> ignoring the packages installed in your home directory.
>>>>
>>>> Thomas
>>>>
>>>> "tenspd137 ." <dcday137 <at> gmail.com> writes:
>>>>
>>>> > I haven't tried the first item on the list yet - ie reloading Andreas'
>>>> > function.  i have tried to build emacs from
>>>> > https://github.com/emacs-mirror/emacs.git because I can't reach the
>>>> > official repo from my firewall for whatever reason.  I am sure it is
>>>> > something here.  Anyway, I was having problems because my built
>>>> > version (27.0.x) was getting seemingly mixed up with my current
>>>> > installation.  I can try again later, but right now things are pretty
>>>> > busy at work.
>>>> >
>>>> > Thanks.
>>>> >
>>>> > On Tue, Jun 18, 2019 at 10:26 PM Thomas Fitzsimmons <fitzsim <at> fitzsim.org> wrote:
>>>> >>
>>>> >> Hi,
>>>> >>
>>>> >> "tenspd137 ." <dcday137 <at> gmail.com> writes:
>>>> >>
>>>> >> > So - I am not sure if I did it correctly, but I copied this function
>>>> >> > with Andreas' changes into a file:
>>>> >> >
>>>> >> > (defun url-https-proxy-after-change-function (_st _nd _length)
>>>> >> >   (let* ((process-buffer (current-buffer))
>>>> >> >          (proc (get-buffer-process process-buffer)))
>>>> >> >     (goto-char (point-min))
>>>> >> >     (when (re-search-forward "^\r?\n" nil t)
>>>> >> >       (backward-char 1)
>>>> >> >       ;; Saw the end of the headers
>>>> >> >       (setq url-http-end-of-headers (set-marker (make-marker) (point)))
>>>> >> >       (url-http-parse-response)
>>>> >> >       (cond
>>>> >> >        ((null url-http-response-status)
>>>> >> >         ;; We got back a headerless malformed response from the
>>>> >> >         ;; server.
>>>> >> >         (url-http-activate-callback)
>>>> >> >         (error "Malformed response from proxy, fail!"))
>>>> >> >        ((= url-http-response-status 200)
>>>> >> >         (if (gnutls-available-p)
>>>> >> >             (condition-case e
>>>> >> >                 (let ((tls-connection (gnutls-negotiate
>>>> >> >                                        :process proc
>>>> >> >                                        :hostname (url-host url-current-object)
>>>> >> >                                        :verify-error nil)))
>>>> >> >                   ;; check certificate validity
>>>> >> >                   (setq tls-connection
>>>> >> >                         (nsm-verify-connection tls-connection
>>>> >> >                                                (url-host url-current-object)
>>>> >> >                                                (url-port url-current-object)))
>>>> >> >                   (with-current-buffer process-buffer (erase-buffer))
>>>> >> >                   (set-process-buffer tls-connection process-buffer)
>>>> >> >                   (setq url-http-after-change-function
>>>> >> >                         'url-http-wait-for-headers-change-function)
>>>> >> >                   (set-process-filter tls-connection 'url-http-generic-filter)
>>>> >> >                   (process-send-string tls-connection
>>>> >> >                                        ;; Use the non-proxy form of the request
>>>> >> >                                        (let (url-http-proxy)
>>>> >> >                                          (url-http-create-request))))
>>>> >> >               (gnutls-error
>>>> >> >                (url-http-activate-callback)
>>>> >> >                (error "gnutls-error: %s" e))
>>>> >> >               (error
>>>> >> >                (url-http-activate-callback)
>>>> >> >                (error "error: %s" e)))
>>>> >> >           (error "error: gnutls support needed!")))
>>>> >> >        (t
>>>> >> >         (url-http-debug "error response: %d" url-http-response-status)
>>>> >> >         (url-http-activate-callback))))))
>>>> >> >
>>>> >> > and then loaded it before running excorporate.  After that, I did M-x
>>>> >> > excorporate, and the minibuffer returns:  error in process filter:
>>>> >> > Server response is not an XML document
>>>> >>
>>>> >> In this scenario, if you immediately (without restarting Emacs/reloading
>>>> >> anything) re-run M-x excorporate does it still fail?  I just want to
>>>> >> make sure that's not a transient failure.  If it does fail the second
>>>> >> time, can you post the HTTP response from the server?
>>>> >>
>>>> >> > When I do the similar test by loading the url-http-create-request with
>>>> >> > Thomas's changes, I can get a connection and grab my schedule
>>>> >> > through the proxy.
>>>> >>
>>>> >> OK.
>>>> >>
>>>> >> > Let me know if I need to try something different.
>>>> >>
>>>> >> Are you in a position to build Emacs master tip and retry the experiment
>>>> >> without patching anything?
>>>> >>
>>>> >> Thanks,
>>>> >> Thomas
>>>> >>
>>>> >> > On Mon, Jun 17, 2019 at 4:08 PM Thomas Fitzsimmons <fitzsim <at> fitzsim.org> wrote:
>>>> >> >>
>>>> >> >> Hi,
>>>> >> >>
>>>> >> >> Good to hear that the patch I posted worked!
>>>> >> >>
>>>> >> >> Yes, that's the patch that Andreas's commit
>>>> >> >> 84613dae5c34ea742dd9a3e56f5acb55f604b483 applied.  From what I can tell,
>>>> >> >> you will not have that in Emacs 26.2.
>>>> >> >>
>>>> >> >> Can you try reverting my patch and applying Andreas's patch, and see if
>>>> >> >> Excorporate still works through the proxy?
>>>> >> >>
>>>> >> >> Thanks,
>>>> >> >> Thomas
>>>> >> >>
>>>> >> >> "tenspd137 ." <dcday137 <at> gmail.com> writes:
>>>> >> >>
>>>> >> >> > The patch Thomas seems to work from behind the proxy.  My current
>>>> >> >> > emacs version is 26.2, so I would think it would include the commit
>>>> >> >> > Andreas is talking about....  I went and looked it up - is this the
>>>> >> >> > correct commit?
>>>> >> >> >
>>>> >> >> > diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
>>>> >> >> > index 53798f7..817c5ce 100644
>>>> >> >> > --- a/lisp/url/url-http.el
>>>> >> >> > +++ b/lisp/url/url-http.el
>>>> >> >> > @@ -1412,7 +1412,9 @@ The return value of this function is the
>>>> >> >> > retrieval buffer."
>>>> >> >> > 'url-http-wait-for-headers-change-function)
>>>> >> >> > (set-process-filter tls-connection 'url-http-generic-filter)
>>>> >> >> > (process-send-string tls-connection
>>>> >> >> > - (url-http-create-request)))
>>>> >> >> > + ;; Use the non-proxy form of the request
>>>> >> >> > + (let (url-http-proxy)
>>>> >> >> > + (url-http-create-request))))
>>>> >> >> > (gnutls-error
>>>> >> >> > (url-http-activate-callback)
>>>> >> >> > (error "gnutls-error: %s" e))
>>>> >> >> >
>>>> >> >> > Thanks!
>>>> >> >> >
>>>> >> >> > -C
>>>> >> >> >
>>>> >> >> >
>>>> >> >> > On Sat, Jun 15, 2019 at 1:41 AM Andreas Schwab <schwab <at> linux-m68k.org> wrote:
>>>> >> >> >>
>>>> >> >> >> On Jun 14 2019, Thomas Fitzsimmons <fitzsim <at> fitzsim.org> wrote:
>>>> >> >> >>
>>>> >> >> >> > diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
>>>> >> >> >> > index 00803a103a..723d111d58 100644
>>>> >> >> >> > --- a/lisp/url/url-http.el
>>>> >> >> >> > +++ b/lisp/url/url-http.el
>>>> >> >> >> > @@ -329,7 +329,10 @@ url-http-create-request
>>>> >> >> >> >               ;; The request
>>>> >> >> >> >               (or url-http-method "GET") " "
>>>> >> >> >> >               (url-http--encode-string
>>>> >> >> >> > -              (if using-proxy (url-recreate-url url-http-target-url) real-fname))
>>>> >> >> >> > +              (if (and using-proxy
>>>> >> >> >> > +                       (not (equal "https" (url-type url-http-target-url))))
>>>> >> >> >> > +                  (url-recreate-url url-http-target-url)
>>>> >> >> >> > +                real-fname))
>>>> >> >> >>
>>>> >> >> >> That should already be handled by commit 84613dae5c.
>>>> >> >> >>
>>>> >> >> >> Andreas.
>>>> >> >> >>
>>>> >> >> >> --
>>>> >> >> >> Andreas Schwab, schwab <at> linux-m68k.org
>>>> >> >> >> GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
>>>> >> >> >> "And now for something completely different."

Message #112 received at 35969 <at> debbugs.gnu.org (full text, mbox):

From: Collin Day <dcday137 <at> gmail.com>
To: Thomas Fitzsimmons <fitzsim <at> fitzsim.org>
Cc: Andreas Schwab <schwab <at> linux-m68k.org>, 35969 <at> debbugs.gnu.org
Subject: Re: bug#35969: 26.2, Excorporate
Date: Wed, 10 Jul 2019 09:35:34 -0600

When I add your patch, there is still an error:

Contacting host: outlook.office365.com:443
error in process filter: exco--parse-xml-in-current-buffer: Server
response is not an XML document
error in process filter: Server response is not an XML document

* http outlook,office365.com:443*
HTTP/1.1 401 Unauthorized
Server: Microsoft-IIS/10.0
request-id: 33a1707c-bed3-4edd-98c6-bd1443827acf
X-WSSecurity-Enabled: True
X-WSSecurity-For: Logon
X-FederationTrustTokenIssuerUri: urn:federation:MicrosoftOnline
X-WSSecurity-SymmetricKey-Enabled: True
X-WSSecurity-X509Cert-Enabled: True
X-OAuth-Enabled: True
X-Powered-By: ASP.NET
X-FEServer: SN4PR0601CA0013
WWW-Authenticate: Basic Realm=""
Date: Wed, 10 Jul 2019 15:27:49 GMT
Content-Length: 0

URL-DEBUG
http -> Contacting host: outlook.office365.com:443
http -> Marking connection as busy: outlook.office365.com:443
#<process outlook.office365.com>
http -> getting referer from buffer: buffer:#<buffer *scratch*>
target-url:#s(url "https" nil nil "outlook.office365.com" nil
"/EWS/Services.wsdl" nil nil t nil t t) lastloc:nil
http -> Calling after change function
`url-https-proxy-after-change-function' for `#<process
outlook.office365.com>'
http -> url-http-parse-response called in ( *http outlook.office365.com:443*)
http -> Request is:
GET /EWS/Services.wsdl HTTP/1.1
MIME-Version: 1.0
Connection: keep-alive
Extension: Security/Digest Security/SSL
Host: outlook.office365.com
Accept-encoding: gzip
Accept: */*
User-Agent: URL/Emacs Emacs/27.0.50 (X11; x86_64-pc-linux-gnu)


http -> Calling after change function
`url-http-wait-for-headers-change-function' for `#<process
outlook.office365.com>'
http -> url-http-wait-for-headers-change-function ( *http
outlook.office365.com:443*)
http -> Saw end of headers... ( *http outlook.office365.com:443*)
http -> url-http-parse-response called in ( *http outlook.office365.com:443*)
http -> Got a content-length, being smart about document end.
http -> Got 0-length content-length, activating callback immediately.
http -> Marking connection as free: outlook.office365.com:443
#<process outlook.office365.com>
http -> url-http-parse-headers called in ( *http outlook.office365.com:443*)
http -> url-http-parse-response called in ( *http outlook.office365.com:443*)
http -> Parsed HTTP headers: class=4 status=401
http -> Handling normal authentication
http -> Found existing connection: outlook.office365.com:443 #<process
outlook.office365.com>
http -> Reusing existing connection: outlook.office365.com:443
http -> Marking connection as busy: outlook.office365.com:443
#<process outlook.office365.com>
http -> getting referer from buffer: buffer:#<buffer  *http
outlook.office365.com:443*> target-url:#s(url "https" nil nil
"outlook.office365.com" nil "/EWS/Services.wsdl" nil nil t nil t t)
lastloc:nil
http -> Finished parsing HTTP headers: nil
http -> Spinning waiting for headers...
http -> Calling after change function
`url-https-proxy-after-change-function' for `#<process
outlook.office365.com>'
http -> url-http-parse-response called in ( *http
outlook.office365.com:443*-293116)
http -> error response: 400
http -> Marking connection as free: outlook.office365.com:443
#<process outlook.office365.com>
http -> Activating callback in buffer ( *http
outlook.office365.com:443*-293116): #[257
"p\303\304\305\306\307 !\310\"\311$\216
@\312=\203$\0\313\301\314\315\316\302\"#\210\317\300\320\"\202/\0\313\301\321\322
#\210\317\300\323\")\207" [fsm-exco--fsm-0 (:identifier ("my email" .
"https://outlook.office365.com/EWS/Exchange.asmx") :mail-address "my
email" :retrying nil :autodiscovery-urls nil :service-url
"https://outlook.office365.com/EWS/Exchange.asmx" :service-xml nil
:service-wsdl nil :next-state-after-success nil :failure-message nil
:server-version nil) "https://outlook.office365.com/EWS/Services.wsdl"
make-byte-code 0 "\301\300!\205    \0\302\300!\207" vconcat vector
[buffer-live-p kill-buffer] 2 :error plist-put :failure-message format
"Failed to retrieve %s" fsm-send :unrecoverable-error :service-xml
exco--parse-xml-in-current-buffer :success] 8 "

(fn STATUS)"] ((:peer (:certificates ((:version 3 :serial-number
"0b:be:0b:85:09:74:2b:7b:f8:7f:80:dc:14:67:6d:54" :issuer
"C=US,O=DigiCert Inc,CN=DigiCert Cloud Services CA-1" :valid-from
"2018-11-19" :valid-to "2020-11-19" :subject
"C=US,ST=Washington,L=Redmond,O=Microsoft Corporation,CN=outlook.com"
:public-key-algorithm "RSA" :certificate-security-level "Medium"
:signature-algorithm "RSA-SHA256" :public-key-id
"sha1:35:1a:fe:e2:6c:d9:35:08:3c:95:a2:c1:12:a1:2a:8e:7f:87:fa:07"
:certificate-id
"sha1:9a:44:16:e2:84:8e:3d:26:86:f0:23:a2:ff:f2:d0:24:28:2f:18:8f")
(:version 3 :serial-number
"01:9e:c1:c6:bd:3f:59:7b:b2:0c:33:38:e5:51:d8:77" :issuer
"C=US,O=DigiCert Inc,OU=www.digicert.com,CN=DigiCert Global Root CA"
:valid-from "2015-08-04" :valid-to "2030-08-04" :subject
"C=US,O=DigiCert Inc,CN=DigiCert Cloud Services CA-1"
:public-key-algorithm "RSA" :certificate-security-level "Medium"
:signature-algorithm "RSA-SHA256" :public-key-id
"sha1:ac:13:d5:79:03:66:f3:cc:ff:fe:7c:23:71:a7:61:5d:39:20:89:6e"
:certificate-id
"sha1:81:b6:8d:6c:d2:f2:21:f8:f5:34:e6:77:52:3b:b2:36:bb:a1:dc:56"))
:certificate (:version 3 :serial-number
"0b:be:0b:85:09:74:2b:7b:f8:7f:80:dc:14:67:6d:54" :issuer
"C=US,O=DigiCert Inc,CN=DigiCert Cloud Services CA-1" :valid-from
"2018-11-19" :valid-to "2020-11-19" :subject
"C=US,ST=Washington,L=Redmond,O=Microsoft Corporation,CN=outlook.com"
:public-key-algorithm "RSA" :certificate-security-level "Medium"
:signature-algorithm "RSA-SHA256" :public-key-id
"sha1:35:1a:fe:e2:6c:d9:35:08:3c:95:a2:c1:12:a1:2a:8e:7f:87:fa:07"
:certificate-id
"sha1:9a:44:16:e2:84:8e:3d:26:86:f0:23:a2:ff:f2:d0:24:28:2f:18:8f")
:key-exchange "ECDHE-RSA" :protocol "TLS1.2" :cipher "AES-256-GCM"
:mac "AEAD")))

url-http.el with patch from describe function, just to make sure patch
is applied:
   ...
    (setq request
          (concat
             ;; The request
             (or url-http-method "GET") " "
             (url-http--encode-string
             ;; (if using-proxy (url-recreate-url url-http-target-url)
real-fname))
             (if (and using-proxy
                 (not (equal "https" (url-type url-http-target-url))))
                 (url-recreate-url url-http-target-url)
              real-fname))
             " HTTP/" url-http-version "\r\n"
    ...

output from make:
make[2]: Entering directory '/home/dayd/projects/emacs/lisp'
  ELC      url/url-http.elc

Sorry if the extra info is extraneous, just trying to grab as much as
I can for you.  I think there is something else wrong at this point
though - like I mentioned it seems strange I had to run
package-install 4 times to get excorporate to install in the first
place.

Thanks!
-C

On Tue, Jul 9, 2019 at 6:11 PM Thomas Fitzsimmons <fitzsim <at> fitzsim.org> wrote:
>
> I think there should be no need to remove test-home, so you won't have
> to re-run the M-x package-install steps.
>
> Thomas
>
> Thomas Fitzsimmons <fitzsim <at> fitzsim.org> writes:
>
> > OK, can you apply my patch, then run "make" again and confirm it works?
> >
> > Thanks,
> > Thomas
> >
> > Collin Day <dcday137 <at> gmail.com> writes:
> >
> >> Did a git pull and followed instructions above.
> >>
> >> There are some issues.  First, I need to run M-x package-install <RET>
> >> excorporate <RET>  four times because I see (each line after each
> >> invocation)
> >>
> >> package--with-response-buffer-1:
> >> https://elpa.gnu.org/packages/url-http-ntlm-2.0.4.el: Method not
> >> allowed
> >> package--with-response-buffer-1:
> >> https://elpa.gnu.org/packages/fsm-0.2.1.el: Method not allowed
> >> package--with-response-buffer-1:
> >> https://elpa.gnu.org/packages/excorporate-0.8.3.tar: Method not
> >> allowed
> >>
> >> After the 4th call it compiles and becomes available.
> >>
> >> M-x customize-group <RET> excorporate, set up for no autoconfig right
> >> now, hit apply for current sessions, save not availiable....
> >>
> >> M-x excorporate <RET>
> >>
> >> enter uname and password
> >>
> >> Contacting host: outlook.office365.com:443
> >> error in process filter: exco--parse-xml-in-current-buffer: Server
> >> response is not an XML document
> >> error in process filter: Server response is not an XML documen
> >>
> >> *http outlook.office.365.com:443*
> >
> > [...]
> >
> >> So it appears not to work, at least on the current head as of today....
> >>
> >> Thanks!
> >>
> >> -C
> >>
> >> On Tue, Jul 9, 2019 at 4:09 PM Collin Day <dcday137 <at> gmail.com> wrote:
> >>>
> >>> Sorry, I have not had the chance to.  A lot has been going on at my
> >>> place of employment.  It has crossed my mind, and as soon as I have
> >>> a chance, I will try what you suggested above.  Thanks, sorry for
> >>> the inconvenience.
> >>>
> >>> On Tue, Jul 9, 2019, 3:53 PM Thomas Fitzsimmons <fitzsim <at> fitzsim.org> wrote:
> >>>>
> >>>> Hi,
> >>>>
> >>>> Were you able to complete building Emacs?  I'd like to know if my patch
> >>>> is needed on top of Andreas's to make Excorporate work through your
> >>>> proxy.
> >>>>
> >>>> I'm not sure why the built version would interfere with the current
> >>>> installation.  If you were doing "make install" to a common prefix, that
> >>>> might explain it.  Instead you can try something like this:
> >>>>
> >>>> cd emacs-master [your github.com checkout]
> >>>> make
> >>>> mkdir test-home
> >>>> HOME=`pwd`/test-home ./src/emacs -Q
> >>>>
> >>>> That will ensure you're only running the built Emacs, and completely
> >>>> ignoring the packages installed in your home directory.
> >>>>
> >>>> Thomas
> >>>>
> >>>> "tenspd137 ." <dcday137 <at> gmail.com> writes:
> >>>>
> >>>> > I haven't tried the first item on the list yet - ie reloading Andreas'
> >>>> > function.  i have tried to build emacs from
> >>>> > https://github.com/emacs-mirror/emacs.git because I can't reach the
> >>>> > official repo from my firewall for whatever reason.  I am sure it is
> >>>> > something here.  Anyway, I was having problems because my built
> >>>> > version (27.0.x) was getting seemingly mixed up with my current
> >>>> > installation.  I can try again later, but right now things are pretty
> >>>> > busy at work.
> >>>> >
> >>>> > Thanks.
> >>>> >
> >>>> > On Tue, Jun 18, 2019 at 10:26 PM Thomas Fitzsimmons <fitzsim <at> fitzsim.org> wrote:
> >>>> >>
> >>>> >> Hi,
> >>>> >>
> >>>> >> "tenspd137 ." <dcday137 <at> gmail.com> writes:
> >>>> >>
> >>>> >> > So - I am not sure if I did it correctly, but I copied this function
> >>>> >> > with Andreas' changes into a file:
> >>>> >> >
> >>>> >> > (defun url-https-proxy-after-change-function (_st _nd _length)
> >>>> >> >   (let* ((process-buffer (current-buffer))
> >>>> >> >          (proc (get-buffer-process process-buffer)))
> >>>> >> >     (goto-char (point-min))
> >>>> >> >     (when (re-search-forward "^\r?\n" nil t)
> >>>> >> >       (backward-char 1)
> >>>> >> >       ;; Saw the end of the headers
> >>>> >> >       (setq url-http-end-of-headers (set-marker (make-marker) (point)))
> >>>> >> >       (url-http-parse-response)
> >>>> >> >       (cond
> >>>> >> >        ((null url-http-response-status)
> >>>> >> >         ;; We got back a headerless malformed response from the
> >>>> >> >         ;; server.
> >>>> >> >         (url-http-activate-callback)
> >>>> >> >         (error "Malformed response from proxy, fail!"))
> >>>> >> >        ((= url-http-response-status 200)
> >>>> >> >         (if (gnutls-available-p)
> >>>> >> >             (condition-case e
> >>>> >> >                 (let ((tls-connection (gnutls-negotiate
> >>>> >> >                                        :process proc
> >>>> >> >                                        :hostname (url-host url-current-object)
> >>>> >> >                                        :verify-error nil)))
> >>>> >> >                   ;; check certificate validity
> >>>> >> >                   (setq tls-connection
> >>>> >> >                         (nsm-verify-connection tls-connection
> >>>> >> >                                                (url-host url-current-object)
> >>>> >> >                                                (url-port url-current-object)))
> >>>> >> >                   (with-current-buffer process-buffer (erase-buffer))
> >>>> >> >                   (set-process-buffer tls-connection process-buffer)
> >>>> >> >                   (setq url-http-after-change-function
> >>>> >> >                         'url-http-wait-for-headers-change-function)
> >>>> >> >                   (set-process-filter tls-connection 'url-http-generic-filter)
> >>>> >> >                   (process-send-string tls-connection
> >>>> >> >                                        ;; Use the non-proxy form of the request
> >>>> >> >                                        (let (url-http-proxy)
> >>>> >> >                                          (url-http-create-request))))
> >>>> >> >               (gnutls-error
> >>>> >> >                (url-http-activate-callback)
> >>>> >> >                (error "gnutls-error: %s" e))
> >>>> >> >               (error
> >>>> >> >                (url-http-activate-callback)
> >>>> >> >                (error "error: %s" e)))
> >>>> >> >           (error "error: gnutls support needed!")))
> >>>> >> >        (t
> >>>> >> >         (url-http-debug "error response: %d" url-http-response-status)
> >>>> >> >         (url-http-activate-callback))))))
> >>>> >> >
> >>>> >> > and then loaded it before running excorporate.  After that, I did M-x
> >>>> >> > excorporate, and the minibuffer returns:  error in process filter:
> >>>> >> > Server response is not an XML document
> >>>> >>
> >>>> >> In this scenario, if you immediately (without restarting Emacs/reloading
> >>>> >> anything) re-run M-x excorporate does it still fail?  I just want to
> >>>> >> make sure that's not a transient failure.  If it does fail the second
> >>>> >> time, can you post the HTTP response from the server?
> >>>> >>
> >>>> >> > When I do the similar test by loading the url-http-create-request with
> >>>> >> > Thomas's changes, I can get a connection and grab my schedule
> >>>> >> > through the proxy.
> >>>> >>
> >>>> >> OK.
> >>>> >>
> >>>> >> > Let me know if I need to try something different.
> >>>> >>
> >>>> >> Are you in a position to build Emacs master tip and retry the experiment
> >>>> >> without patching anything?
> >>>> >>
> >>>> >> Thanks,
> >>>> >> Thomas
> >>>> >>
> >>>> >> > On Mon, Jun 17, 2019 at 4:08 PM Thomas Fitzsimmons <fitzsim <at> fitzsim.org> wrote:
> >>>> >> >>
> >>>> >> >> Hi,
> >>>> >> >>
> >>>> >> >> Good to hear that the patch I posted worked!
> >>>> >> >>
> >>>> >> >> Yes, that's the patch that Andreas's commit
> >>>> >> >> 84613dae5c34ea742dd9a3e56f5acb55f604b483 applied.  From what I can tell,
> >>>> >> >> you will not have that in Emacs 26.2.
> >>>> >> >>
> >>>> >> >> Can you try reverting my patch and applying Andreas's patch, and see if
> >>>> >> >> Excorporate still works through the proxy?
> >>>> >> >>
> >>>> >> >> Thanks,
> >>>> >> >> Thomas
> >>>> >> >>
> >>>> >> >> "tenspd137 ." <dcday137 <at> gmail.com> writes:
> >>>> >> >>
> >>>> >> >> > The patch Thomas seems to work from behind the proxy.  My current
> >>>> >> >> > emacs version is 26.2, so I would think it would include the commit
> >>>> >> >> > Andreas is talking about....  I went and looked it up - is this the
> >>>> >> >> > correct commit?
> >>>> >> >> >
> >>>> >> >> > diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
> >>>> >> >> > index 53798f7..817c5ce 100644
> >>>> >> >> > --- a/lisp/url/url-http.el
> >>>> >> >> > +++ b/lisp/url/url-http.el
> >>>> >> >> > @@ -1412,7 +1412,9 @@ The return value of this function is the
> >>>> >> >> > retrieval buffer."
> >>>> >> >> > 'url-http-wait-for-headers-change-function)
> >>>> >> >> > (set-process-filter tls-connection 'url-http-generic-filter)
> >>>> >> >> > (process-send-string tls-connection
> >>>> >> >> > - (url-http-create-request)))
> >>>> >> >> > + ;; Use the non-proxy form of the request
> >>>> >> >> > + (let (url-http-proxy)
> >>>> >> >> > + (url-http-create-request))))
> >>>> >> >> > (gnutls-error
> >>>> >> >> > (url-http-activate-callback)
> >>>> >> >> > (error "gnutls-error: %s" e))
> >>>> >> >> >
> >>>> >> >> > Thanks!
> >>>> >> >> >
> >>>> >> >> > -C
> >>>> >> >> >
> >>>> >> >> >
> >>>> >> >> > On Sat, Jun 15, 2019 at 1:41 AM Andreas Schwab <schwab <at> linux-m68k.org> wrote:
> >>>> >> >> >>
> >>>> >> >> >> On Jun 14 2019, Thomas Fitzsimmons <fitzsim <at> fitzsim.org> wrote:
> >>>> >> >> >>
> >>>> >> >> >> > diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
> >>>> >> >> >> > index 00803a103a..723d111d58 100644
> >>>> >> >> >> > --- a/lisp/url/url-http.el
> >>>> >> >> >> > +++ b/lisp/url/url-http.el
> >>>> >> >> >> > @@ -329,7 +329,10 @@ url-http-create-request
> >>>> >> >> >> >               ;; The request
> >>>> >> >> >> >               (or url-http-method "GET") " "
> >>>> >> >> >> >               (url-http--encode-string
> >>>> >> >> >> > -              (if using-proxy (url-recreate-url url-http-target-url) real-fname))
> >>>> >> >> >> > +              (if (and using-proxy
> >>>> >> >> >> > +                       (not (equal "https" (url-type url-http-target-url))))
> >>>> >> >> >> > +                  (url-recreate-url url-http-target-url)
> >>>> >> >> >> > +                real-fname))
> >>>> >> >> >>
> >>>> >> >> >> That should already be handled by commit 84613dae5c.
> >>>> >> >> >>
> >>>> >> >> >> Andreas.
> >>>> >> >> >>
> >>>> >> >> >> --
> >>>> >> >> >> Andreas Schwab, schwab <at> linux-m68k.org
> >>>> >> >> >> GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
> >>>> >> >> >> "And now for something completely different."

Message #115 received at 35969 <at> debbugs.gnu.org (full text, mbox):

From: Thomas Fitzsimmons <fitzsim <at> fitzsim.org>
To: Collin Day <dcday137 <at> gmail.com>
Cc: Andreas Schwab <schwab <at> linux-m68k.org>, 35969 <at> debbugs.gnu.org
Subject: Re: bug#35969: 26.2, Excorporate
Date: Sat, 13 Jul 2019 10:37:56 -0400

Hi Collin,

Collin Day <dcday137 <at> gmail.com> writes:

> Did a git pull and followed instructions above.
>
> There are some issues.  First, I need to run M-x package-install <RET>
> excorporate <RET>  four times because I see (each line after each
> invocation)
>
> package--with-response-buffer-1:
> https://elpa.gnu.org/packages/url-http-ntlm-2.0.4.el: Method not
> allowed
> package--with-response-buffer-1:
> https://elpa.gnu.org/packages/fsm-0.2.1.el: Method not allowed
> package--with-response-buffer-1:
> https://elpa.gnu.org/packages/excorporate-0.8.3.tar: Method not
> allowed
>
> After the 4th call it compiles and becomes available.

Thanks for persisting with this.

Did you configure the proxy settings in Emacs before attempting
package-install?

If so, try this in your newly-built Emacs, emacs -Q, with
HOME=`pwd`/test-home:

(progn
  (setq url-debug t)
  (url-retrieve-synchronously
   "https://httpbin.org/basic-auth/user/passwd")
  (dolist (p (seq-filter
              (lambda (b) (string-match " *http*" (buffer-name b)))
              (buffer-list)))
    (message "HTTP result buffer: \"%s\"\n%s"
             (buffer-name p)
             (with-current-buffer p (buffer-string))))
  "check *Messages*")

Username is user, password is passwd.  That will establish whether even
basic authentication over HTTPS is working through the proxy with your
new Emacs build.

Thomas

Message #118 received at 35969 <at> debbugs.gnu.org (full text, mbox):

From: Thomas Fitzsimmons <fitzsim <at> fitzsim.org>
To: Andreas Schwab <schwab <at> linux-m68k.org>
Cc: Collin Day <dcday137 <at> gmail.com>, 35969 <at> debbugs.gnu.org
Subject: Re: bug#35969: 26.2, Excorporate
Date: Wed, 31 Jul 2019 17:07:38 -0400

Hi,

I found a proxy server to test against.  I've now replicated Collin's
findings.

Andreas Schwab <schwab <at> linux-m68k.org> writes:

> On Jun 14 2019, Thomas Fitzsimmons <fitzsim <at> fitzsim.org> wrote:
>
>> diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
>> index 00803a103a..723d111d58 100644
>> --- a/lisp/url/url-http.el
>> +++ b/lisp/url/url-http.el
>> @@ -329,7 +329,10 @@ url-http-create-request
>>               ;; The request
>>               (or url-http-method "GET") " "
>>               (url-http--encode-string
>> -              (if using-proxy (url-recreate-url url-http-target-url) real-fname))
>> +              (if (and using-proxy
>> +                       (not (equal "https" (url-type url-http-target-url))))
>> +                  (url-recreate-url url-http-target-url)
>> +                real-fname))

For discussion purposes, let's call the above "patch T"...

> That should already be handled by commit 84613dae5c.

... and this commit "patch A", which is:

diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
index 53798f77c3..817c5ce3b3 100644
--- a/lisp/url/url-http.el
+++ b/lisp/url/url-http.el
@@ -1412,7 +1412,9 @@ url-https-proxy-after-change-function
                         'url-http-wait-for-headers-change-function)
                   (set-process-filter tls-connection 'url-http-generic-filter)
                   (process-send-string tls-connection
-                                       (url-http-create-request)))
+                                       ;; Use the non-proxy form of the request
+                                       (let (url-http-proxy)
+                                         (url-http-create-request))))
               (gnutls-error
                (url-http-activate-callback)
                (error "gnutls-error: %s" e))

I tried on Emacs 26.2 and master tip, and in both cases, Excorporate
worked with patch T applied and patch A not present (26.2) or reverted
(master); it failed with any other combination of the patches (A and T,
A only, neither A nor T).

One difference I noticed is that with A applied, the Connection header
is set to keep-alive and the connection is reused, whereas with just T,
Connection is set to close and the connection is re-established.  The
attached patch fixes it.  Andreas, do you have a test case that patch A
fixed, and if so, can you retest with the proposed fix?

Thanks,
Thomas

diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
index 838f0a30c1..eb054cd65a 100644
--- a/lisp/url/url-http.el
+++ b/lisp/url/url-http.el
@@ -1436,7 +1436,8 @@ url-https-proxy-after-change-function
                   (set-process-filter tls-connection 'url-http-generic-filter)
                   (process-send-string tls-connection
                                        ;; Use the non-proxy form of the request
-                                       (let (url-http-proxy)
+                                       (let (url-http-proxy
+                                             url-http-attempt-keepalives)
                                          (url-http-create-request))))
               (gnutls-error
                (url-http-activate-callback)

Message #121 received at 35969 <at> debbugs.gnu.org (full text, mbox):

From: Andreas Schwab <schwab <at> linux-m68k.org>
To: Thomas Fitzsimmons <fitzsim <at> fitzsim.org>
Cc: Collin Day <dcday137 <at> gmail.com>, 35969 <at> debbugs.gnu.org
Subject: Re: bug#35969: 26.2, Excorporate
Date: Wed, 31 Jul 2019 23:20:46 +0200

On Jul 31 2019, Thomas Fitzsimmons <fitzsim <at> fitzsim.org> wrote:

> Andreas, do you have a test case that patch A fixed, and if so, can
> you retest with the proposed fix?

Any https request over proxy will do.

Andreas.

-- 
Andreas Schwab, schwab <at> linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Message #124 received at 35969 <at> debbugs.gnu.org (full text, mbox):

From: Collin Day <dcday137 <at> gmail.com>
To: Andreas Schwab <schwab <at> linux-m68k.org>
Cc: Thomas Fitzsimmons <fitzsim <at> fitzsim.org>, 35969 <at> debbugs.gnu.org
Subject: Re: bug#35969: 26.2, Excorporate
Date: Wed, 31 Jul 2019 16:28:38 -0600

[Message part 1 (text/plain, inline)]

Sorry I have been unresponsive.  I am in the middle of changing jobs....
good news that you can recreate it.

C

On Wed, Jul 31, 2019, 3:20 PM Andreas Schwab <schwab <at> linux-m68k.org> wrote:

> On Jul 31 2019, Thomas Fitzsimmons <fitzsim <at> fitzsim.org> wrote:
>
> > Andreas, do you have a test case that patch A fixed, and if so, can
> > you retest with the proposed fix?
>
> Any https request over proxy will do.
>
> Andreas.
>
> --
> Andreas Schwab, schwab <at> linux-m68k.org
> GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
> "And now for something completely different."
>

[Message part 2 (text/html, inline)]

Message #127 received at 35969 <at> debbugs.gnu.org (full text, mbox):

From: Thomas Fitzsimmons <fitzsim <at> fitzsim.org>
To: Andreas Schwab <schwab <at> linux-m68k.org>
Cc: Collin Day <dcday137 <at> gmail.com>, 35969 <at> debbugs.gnu.org
Subject: Re: bug#35969: 26.2, Excorporate
Date: Wed, 31 Jul 2019 21:58:22 -0400

Andreas Schwab <schwab <at> linux-m68k.org> writes:

> On Jul 31 2019, Thomas Fitzsimmons <fitzsim <at> fitzsim.org> wrote:
>
>> Andreas, do you have a test case that patch A fixed, and if so, can
>> you retest with the proposed fix?
>
> Any https request over proxy will do.

OK, I tested the attached patch; it works for a request to an uncached
HTTPS website, and it works for Excorporate accessing an HTTPS server.

I moved the URL form logic to url-http-create-request because that
function refers to using-proxy (set to url-http-proxy on entry) in
multiple places, not just during URL recreation.  When
url-https-proxy-after-change-function was setting using-proxy to nil for
the entire duration of the url-http-create-request, it was interfering
with "Connection" handling later in the function:

   "Connection: " (if (or using-proxy
                          (not url-http-attempt-keepalives))
                      "close" "keep-alive")

Does this look OK to push to master?

Thanks,
Thomas

diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
index 838f0a30c1..354cc56de4 100644
--- a/lisp/url/url-http.el
+++ b/lisp/url/url-http.el
@@ -329,7 +329,9 @@ url-http-create-request
              ;; The request
              (or url-http-method "GET") " "
              (url-http--encode-string
-              (if using-proxy (url-recreate-url url-http-target-url) real-fname))
+              (if (and using-proxy
+                       (not (equal "https" (url-type url-http-target-url))))
+                  (url-recreate-url url-http-target-url) real-fname))
              " HTTP/" url-http-version "\r\n"
              ;; Version of MIME we speak
              "MIME-Version: 1.0\r\n"
@@ -1435,9 +1437,7 @@ url-https-proxy-after-change-function
                         'url-http-wait-for-headers-change-function)
                   (set-process-filter tls-connection 'url-http-generic-filter)
                   (process-send-string tls-connection
-                                       ;; Use the non-proxy form of the request
-                                       (let (url-http-proxy)
-                                         (url-http-create-request))))
+                                       (url-http-create-request)))
               (gnutls-error
                (url-http-activate-callback)
                (error "gnutls-error: %s" e))

Message #132 received at 35969-done <at> debbugs.gnu.org (full text, mbox):

From: Thomas Fitzsimmons <fitzsim <at> fitzsim.org>
To: Andreas Schwab <schwab <at> linux-m68k.org>
Cc: Collin Day <dcday137 <at> gmail.com>, 35969-done <at> debbugs.gnu.org
Subject: Re: bug#35969: 26.2, Excorporate
Date: Thu, 15 Aug 2019 23:40:55 -0400

Thomas Fitzsimmons <fitzsim <at> fitzsim.org> writes:

> Andreas Schwab <schwab <at> linux-m68k.org> writes:
>
>> On Jul 31 2019, Thomas Fitzsimmons <fitzsim <at> fitzsim.org> wrote:
>>
>>> Andreas, do you have a test case that patch A fixed, and if so, can
>>> you retest with the proposed fix?
>>
>> Any https request over proxy will do.
>
> OK, I tested the attached patch; it works for a request to an uncached
> HTTPS website, and it works for Excorporate accessing an HTTPS server.
>
> I moved the URL form logic to url-http-create-request because that
> function refers to using-proxy (set to url-http-proxy on entry) in
> multiple places, not just during URL recreation.  When
> url-https-proxy-after-change-function was setting using-proxy to nil for
> the entire duration of the url-http-create-request, it was interfering
> with "Connection" handling later in the function:
>
>    "Connection: " (if (or using-proxy
>                           (not url-http-attempt-keepalives))
>                       "close" "keep-alive")
>
> Does this look OK to push to master?

I didn't hear any objections, so I pushed the change.

Thanks,
Thomas

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.