GNU logs - #53941, boring messages

Message sent to bug-gnu-emacs@HIDDEN:

Subject: bug#53941: 27.2; socks + tor dont work with https
Resent-From: Jacobo <gnuhacker@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 11 Feb 2022 14:32:01 +0000
Resent-Message-ID: <handler.53941.B.164458988927358 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: 53941 <at> debbugs.gnu.org
From: Jacobo <gnuhacker@HIDDEN>
Date: Fri, 11 Feb 2022 12:09:52 +0100
Message-ID: <87pmntfym7.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain
Received-SPF: pass client-ip=198.252.153.129;
 envelope-from=gnuhacker@HIDDEN; helo=mx1.riseup.net
X-Spam_score_int: -25
X-Spam_score: -2.6
X-Spam_bar: --
X-Spam_report: (-2.6 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_LOW=-0.7,
 RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001,
 SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-Spam-Score: -2.3 (--)
X-Mailman-Approved-At: Fri, 11 Feb 2022 09:31:26 -0500
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)


Emacs can not resolve domains when it is https if you are using a socks
proxy (socks.el)

Emacs Config:

#+begin_src elisp

(setq socks-override-functions t)
(setq url-gateway-method 'socks) ; same problem without this line in conf
(require 'socks)

(setq socks-noproxy '("localhost"))

(setq socks-server
  '("TOR"
    "localhost"
    9050
    5))
    
#+end_src


Tor is the socks proxy

Im running tor.

When I try:

M-x eww RET gnu.org RET

It works, load http://gnu.org (HTTP in plain)

Also work with .onion domains in HTTP plain

No problems with HTTP but



When I try:

M-x eww RET https://gnu.org RET


Return an error:

   Bad Request

   Your browser sent a request that this server could not understand.
   Reason: You're speaking plain HTTP to an SSL-enabled server port.
   Instead use the HTTPS scheme to access this URL, please.


In GNU Emacs 27.2 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.30, cairo version 1.16.0)
Windowing system distributor 'The X.Org Foundation', version 11.0.12101002
System Description: GNU Guix System

Configured using:
 'configure
 CONFIG_SHELL=/gnu/store/4y5m9lb8k3qkb1y9m02sw9w9a6hacd16-bash-minimal-5.1.8/bin/bash
 SHELL=/gnu/store/4y5m9lb8k3qkb1y9m02sw9w9a6hacd16-bash-minimal-5.1.8/bin/bash
 --prefix=/gnu/store/c4bb68f53mw3sjf6xbyr7ba83csgjdkk-emacs-27.2
 --enable-fast-install --with-modules --with-cairo
 --disable-build-details'

Configured features:
XPM JPEG TIFF GIF PNG RSVG CAIRO SOUND GPM DBUS GSETTINGS GLIB NOTIFY
INOTIFY ACL GNUTLS LIBXML2 FREETYPE HARFBUZZ M17N_FLT LIBOTF ZLIB
TOOLKIT_SCROLL_BARS GTK3 X11 XDBE XIM MODULES THREADS JSON PDUMPER GMP

Message sent:

Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Mailer: MIME-tools 5.505 (Entity 5.505)
Content-Type: text/plain; charset=utf-8
X-Loop: help-debbugs@HIDDEN
From: help-debbugs@HIDDEN (GNU bug Tracking System)
To: Jacobo <gnuhacker@HIDDEN>
Subject: bug#53941: Acknowledgement (27.2; socks + tor dont work with https)
Message-ID: <handler.53941.B.164458988927358.ack <at> debbugs.gnu.org>
References: <87pmntfym7.fsf@HIDDEN>
X-Gnu-PR-Message: ack 53941
X-Gnu-PR-Package: emacs
Reply-To: 53941 <at> debbugs.gnu.org
Date: Fri, 11 Feb 2022 14:32:02 +0000

Thank you for filing a new bug report with debbugs.gnu.org.

This is an automatically generated reply to let you know your message
has been received.

Your message is being forwarded to the package maintainers and other
interested parties for their attention; they will reply in due course.

Your message has been sent to the package maintainer(s):
 bug-gnu-emacs@HIDDEN

If you wish to submit further information on this problem, please
send it to 53941 <at> debbugs.gnu.org.

Please do not send mail to help-debbugs@HIDDEN unless you wish
to report a problem with the Bug-tracking system.

--=20
53941: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D53941
GNU Bug Tracking System
Contact help-debbugs@HIDDEN with problems

Message sent to bug-gnu-emacs@HIDDEN:

Subject: bug#53941: 27.2; socks + tor dont work with https
Resent-From: "J.P." <jp@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Mon, 14 Feb 2022 12:38:02 +0000
Resent-Message-ID: <handler.53941.B53941.164484227823308 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: Jacobo <gnuhacker@HIDDEN>
Cc: 53941 <at> debbugs.gnu.org
From: "J.P." <jp@HIDDEN>
References: <87pmntfym7.fsf@HIDDEN>
Date: Mon, 14 Feb 2022 04:37:39 -0800
In-Reply-To: <87pmntfym7.fsf@HIDDEN> (Jacobo's message of "Fri, 11 Feb
 2022 12:09:52 +0100")
Message-ID: <8735kl1v58.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/29.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="=-=-="
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

--=-=-=
Content-Type: text/plain

Hi Jacobo,

Jacobo <gnuhacker@HIDDEN> writes:

> Emacs can not resolve domains when it is https if you are using a
> socks proxy (socks.el) [...] It works, load http://gnu.org (HTTP in
> plain) Also work with .onion domains in HTTP plain No problems with
> HTTP but When I try: M-x eww RET https://gnu.org RET
>
> Return an error: Bad Request

It's certainly possible (see attached). But can it be done responsibly?

In this day and age, when processes and services resolve host names in
all manner of ways, how can we be confident there won't be any leaks? At
present, the main interfaces to various protocol stacks (for example,
url-gw.el and friends) don't seem geared toward making those kinds of
assurances. (Not that they ought to be.)

That said, providing the building blocks on the SOCKS side doesn't seem
like the crime of the century. I've been sitting on what became the
basis for these patches for a while now, but these here were hastily
adapted and might come with some warts. Still, I believe them
straightforward enough to illustrate a basic means of achieving what
you're after.

> In GNU Emacs 27.2 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.30,

I also have some examples with shims for 27 running periodically in CI.
These include a demo of using ERC to connect to Libera.Chat via SOCKS
over TLS. (But that requires an IRCv3 library, which is still a work in
progress.) If you're interested in experimenting with any of this stuff,
please let me know. That goes for anyone else out there as well. Thanks.


--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
 filename=0001-Set-coding-system-for-SOCKS-connections-to-binary.patch

From 1cf058fe106e01d55e9269503994e2e9b274b07a Mon Sep 17 00:00:00 2001
From: "F. Jason Park" <jp@HIDDEN>
Date: Mon, 14 Feb 2022 02:36:57 -0800
Subject: [PATCH 1/4] Set coding system for SOCKS connections to binary

* lisp/net/socks.el (socks-opens-connection): Don't perform
conversions when receiving and sending text.

* test/lisp/net/socks-tests.el (socks-tests-canned-server-create): Fix
bug in process filter to prevent prepared outgoing responses from
being implicitly encoded as utf-8.
---
 lisp/net/socks.el            | 1 +
 test/lisp/net/socks-tests.el | 3 ++-
 2 files changed, 3 insertions(+), 1 deletion(-)

diff --git a/lisp/net/socks.el b/lisp/net/socks.el
index 8df0773e1d..c15b323c9c 100644
--- a/lisp/net/socks.el
+++ b/lisp/net/socks.el
@@ -340,6 +340,7 @@ socks-open-connection
 	  version)
 
       ;; Initialize process and info about the process
+      (set-process-coding-system proc 'binary 'binary)
       (set-process-filter proc #'socks-filter)
       (set-process-query-on-exit-flag proc nil)
       (process-put proc 'socks t)
diff --git a/test/lisp/net/socks-tests.el b/test/lisp/net/socks-tests.el
index 461796bdf9..708b964020 100644
--- a/test/lisp/net/socks-tests.el
+++ b/test/lisp/net/socks-tests.el
@@ -137,13 +137,14 @@ socks-tests-canned-server-create
          (pats socks-tests-canned-server-patterns)
          (filt (lambda (proc line)
                  (pcase-let ((`(,pat . ,resp) (pop pats)))
+                   (setq resp (apply #'unibyte-string (append resp nil)))
                    (unless (or (and (vectorp pat) (equal pat (vconcat line)))
                                (string-match-p pat line))
                      (error "Unknown request: %s" line))
                    (let ((print-escape-control-characters t))
                      (message "[%s] <- %s" name (prin1-to-string line))
                      (message "[%s] -> %s" name (prin1-to-string resp)))
-                   (process-send-string proc (concat resp)))))
+                   (process-send-string proc resp))))
          (serv (make-network-process :server 1
                                      :buffer (get-buffer-create name)
                                      :filter filt
-- 
2.34.1


--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
 filename=0002-Add-support-for-SOCKS-4a.patch

From 84299e3e9dac1e3620a83cf807b564ee276f0cdf Mon Sep 17 00:00:00 2001
From: "F. Jason Park" <jp@HIDDEN>
Date: Mon, 14 Feb 2022 02:36:57 -0800
Subject: [PATCH 2/4] Add support for SOCKS 4a

* lisp/net/socks.el (socks-server): Add new choice `4a' to version
field of option.  This may appear to change the type of the field from
a number to a union of symbols and numbers.  However,
`socks-send-command' and `socks-filter' already expect a possible
`http' value for this field (also a symbol).
(socks--errors-4): Add new constant containing error messages for
socks version 4.  The semantics are faithful, but the wording is
ad-libbed.
(socks-send-command): Massage existing handling for version 4 to
accommodate 4a.

* test/lisp/net/socks-tests.el (socks-tests-v4a-basic): add test for
4a.
---
 lisp/net/socks.el            | 22 ++++++++++++++++++++--
 test/lisp/net/socks-tests.el | 13 +++++++++++++
 2 files changed, 33 insertions(+), 2 deletions(-)

diff --git a/lisp/net/socks.el b/lisp/net/socks.el
index c15b323c9c..0d5ef307e7 100644
--- a/lisp/net/socks.el
+++ b/lisp/net/socks.el
@@ -162,6 +162,7 @@ socks-server
 	  (radio-button-choice :tag "SOCKS Version"
 			       :format "%t: %v"
 			       (const :tag "SOCKS v4  " :format "%t" :value 4)
+                               (const :tag "SOCKS v4a"  :format "%t" :value 4a)
 			       (const :tag "SOCKS v5"   :format "%t" :value 5))))
 
 
@@ -202,6 +203,12 @@ socks-errors
     "Command not supported"
     "Address type not supported"))
 
+(defconst socks--errors-4
+  '("Granted"
+    "Rejected or failed"
+    "Cannot connect to identd on the client"
+    "Client and identd report differing user IDs"))
+
 ;; The socks v5 address types
 (defconst socks-address-type-v4   1)
 (defconst socks-address-type-name 3)
@@ -401,6 +408,7 @@ socks-send-command
 		(format "%c%s" (length address) address))
 	       (t
 		(error "Unknown address type: %d" atype))))
+        trailing
 	request version)
     (or (process-get proc 'socks)
         (error "socks-send-command called on non-SOCKS connection %S" proc))
@@ -418,6 +426,12 @@ socks-send-command
 			     (t
 			      (error "Unsupported address type for HTTP: %d" atype)))
 			    port)))
+     ((when (eq version '4a)
+        (setf addr "\0\0\0\1"
+              trailing (concat address "\0")
+              version 4 ; done with the "a" part
+              (process-get proc 'socks-server-protocol) 4)
+        nil)) ; fall through
      ((equal version 4)
       (setq request (concat
 		     (unibyte-string
@@ -427,7 +441,8 @@ socks-send-command
 		      (logand port #xff)) ; port, low byte
 		     addr                 ; address
 		     (user-full-name)     ; username
-		     "\0")))              ; terminate username
+                     "\0"                 ; terminate username
+                     trailing)))          ; optional host to look up
      ((equal version 5)
       (setq request (concat
 		     (unibyte-string
@@ -448,7 +463,10 @@ socks-send-command
 	nil				; Sweet sweet success!
       (delete-process proc)
       (error "SOCKS: %s"
-             (nth (or (process-get proc 'socks-reply) 1) socks-errors)))
+             (let ((no (or (process-get proc 'socks-reply) 1)))
+               (if (eq version 5)
+                   (nth no socks-errors)
+                 (nth (+ 90 no) socks--errors-4)))))
     proc))
 
 
diff --git a/test/lisp/net/socks-tests.el b/test/lisp/net/socks-tests.el
index 708b964020..b81923fc56 100644
--- a/test/lisp/net/socks-tests.el
+++ b/test/lisp/net/socks-tests.el
@@ -207,6 +207,19 @@ socks-tests-v4-basic
                  (lambda (&optional _) "foo")))
         (socks-tests-perform-hello-world-http-request)))))
 
+(ert-deftest socks-tests-v4a-basic ()
+  "Show correct preparation of SOCKS4a connect command."
+  (let ((socks-server '("server" "127.0.0.1" 10083 4a))
+        (url-user-agent "Test/4a-basic")
+        (socks-tests-canned-server-patterns
+         `(([4 1 0 80 0 0 0 1 ?f ?o ?o 0 ?e ?x ?a ?m ?p ?l ?e ?. ?c ?o ?m 0]
+            . [0 90 0 0 0 0 0 0])
+           ,socks-tests--hello-world-http-request-pattern)))
+    (ert-info ("Make HTTP request over SOCKS4A")
+      (cl-letf (((symbol-function 'user-full-name)
+                 (lambda (&optional _) "foo")))
+        (socks-tests-perform-hello-world-http-request)))))
+
 ;; Replace first pattern below with ([5 3 0 1 2] . [5 2]) to validate
 ;; against curl 7.71 with the following options:
 ;; $ curl --verbose -U foo:bar --proxy socks5h://127.0.0.1:10080 example.com
-- 
2.34.1


--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
 filename=0003-Support-SOCKS-RESOLVE-extension.patch

From ffda45081444e14ca687a505f1fc697b8ef59e0f Mon Sep 17 00:00:00 2001
From: "F. Jason Park" <jp@HIDDEN>
Date: Mon, 14 Feb 2022 02:36:57 -0800
Subject: [PATCH 3/4] Support SOCKS RESOLVE extension

* lisp/net/socks.el (socks-resolve-command): Add new constant for the
SOCKS command RESOLVE, which comes by way of a nonstandard extension
from the TOR project.  It mirrors CONNECT in most respects but asks
the server to RESOLVE a host name and return its IP.  For details, see
https://github.com/torproject/torspec/blob/master/socks-extensions.txt
This shouldn't be confused with 5h/5-hostname, which is used to by
clients like cURL to allow users to bypass attempts to resolve a name
locally.
(socks--extract-resolve-response, socks-tor-resolve): Add utility
functions to query a SOCKS service supporting the RESOLVE extension.
---
 lisp/net/socks.el            | 58 ++++++++++++++++++++++++++++++++++++
 test/lisp/net/socks-tests.el | 34 +++++++++++++++++++++
 2 files changed, 92 insertions(+)

diff --git a/lisp/net/socks.el b/lisp/net/socks.el
index 0d5ef307e7..7201ed8e06 100644
--- a/lisp/net/socks.el
+++ b/lisp/net/socks.el
@@ -181,6 +181,9 @@ socks-udp-associate-command
 (defconst socks-authentication-null 0)
 (defconst socks-authentication-failure 255)
 
+;; Extensions
+(defconst socks-resolve-command #xf0)
+
 ;; Response codes
 (defconst socks-response-success               0)
 (defconst socks-response-general-failure       1)
@@ -654,6 +657,61 @@ socks-nslookup-host
 	res)
     host))
 
+(defun socks--extract-resolve-response (proc)
+  "Parse response for PROC and maybe return destination IP address."
+  (let ((response (process-get proc 'socks-response)))
+    (cl-assert response) ; otherwise, msg not received in its entirety
+    (pcase (process-get proc 'socks-server-protocol)
+      (4 ; https://www.openssh.com/txt/socks4a.protocol
+       (when-let (((zerop (process-get proc 'socks-reply)))
+                  ((eq (aref response 1) 90)) ; #x5a request granted
+                  (a (substring response 4)) ; ignore port for now
+                  ((not (string-empty-p a)))
+                  ((not (string= a "\0\0\0\0"))))
+         a))
+      (5 ; https://tools.ietf.org/html/rfc1928
+       (cl-assert (eq 5 (aref response 0)) t)
+       (pcase (aref response 3) ; ATYP
+         (1 (and-let* ((a (substring response 4 8))
+                       ((not (string= a "\0\0\0\0")))
+                       a)))
+         ;; No reason to support RESOLVE_PTR [F1] extension, right?
+         (3 (let ((len (1- (aref response 4))))
+              (substring response 5 (+ 5 len))))
+         (4 (substring response 4 20)))))))
+
+(declare-function puny-encode-domain "puny" (domain))
+
+(defun socks-tor-resolve (name &optional _family)
+  "Return list of one vector IPv4 address for domain NAME.
+Or return nil on failure.  See `network-lookup-address-info' for format
+of return value.  Server must support the Tor RESOLVE command."
+  (let ((socks-password (or socks-password ""))
+        host
+        (port 80)  ; unused for now
+        proc
+        ip)
+    (unless (string-suffix-p ".onion" name)
+      (setq host (if (string-match "\\`[[:ascii:]]+\\'" name)
+                     name
+                   (require 'puny)
+                   (puny-encode-domain name)))
+      ;; "Host unreachable" may be raised when the lookup fails
+      (unwind-protect
+          (progn
+            (setq proc (socks-open-connection (socks-find-route host port)))
+            (socks-send-command proc
+                                socks-resolve-command
+                                socks-address-type-name
+                                host
+                                port)
+            (cl-assert (eq (process-get proc 'socks-state)
+                           socks-state-connected))
+            (setq ip (socks--extract-resolve-response proc)))
+        (when proc
+          (delete-process proc)))
+      (list (vconcat ip [0])))))
+
 (provide 'socks)
 
 ;;; socks.el ends here
diff --git a/test/lisp/net/socks-tests.el b/test/lisp/net/socks-tests.el
index b81923fc56..51e2e40631 100644
--- a/test/lisp/net/socks-tests.el
+++ b/test/lisp/net/socks-tests.el
@@ -295,4 +295,38 @@ socks-tests-v5-auth-none
       (socks-tests-perform-hello-world-http-request)))
   (should (assq 2 socks-authentication-methods)))
 
+(ert-deftest tor-resolve-4a ()
+  "Make request to TOR resolve service over SOCKS4a"
+  (let* ((socks-server '("server" "127.0.0.1" 19050 4a))
+         (socks-tests-canned-server-patterns
+          '(([4 #xf0 0 80 0 0 0 1 ?f ?o ?o 0 ?e ?x ?a ?m ?p ?l ?e ?. ?c ?o ?m 0]
+             . [0 90 0 0 93 184 216 34])))
+         (inhibit-message noninteractive)
+         (server (socks-tests-canned-server-create)))
+    (ert-info ("Query TOR RESOLVE service over SOCKS4")
+      (cl-letf (((symbol-function 'user-full-name)
+                 (lambda (&optional _) "foo")))
+        (should (equal '([93 184 216 34 0])
+                       (socks-tor-resolve "example.com")))))
+    (kill-buffer (process-buffer server))
+    (delete-process server)))
+
+(ert-deftest tor-resolve-5 ()
+  "Make request to TOR resolve service over SOCKS5"
+  (let* ((socks-server '("server" "127.0.0.1" 19051 5))
+         (socks-username "foo")
+         (socks-authentication-methods (append socks-authentication-methods
+                                               nil))
+         (inhibit-message noninteractive)
+         (socks-tests-canned-server-patterns
+          '(([5 2 0 2] . [5 2])
+            ([1 3 ?f ?o ?o 0] . [1 0])
+            ([5 #xf0 0 3 11 ?e ?x ?a ?m ?p ?l ?e ?. ?c ?o ?m 0 80]
+             . [5 0 0 1 93 184 216 34 0 0])))
+         (server (socks-tests-canned-server-create)))
+    (ert-info ("Query TOR RESOLVE service over SOCKS5")
+      (should (equal '([93 184 216 34 0]) (socks-tor-resolve "example.com"))))
+    (kill-buffer (process-buffer server))
+    (delete-process server)))
+
 ;;; socks-tests.el ends here
-- 
2.34.1


--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
 filename=0004-POC-Demo-SOCKS-RESOLVE-over-HTTPS.patch

From efe0b1bff206efb6f6559154a560a71239aaa78e Mon Sep 17 00:00:00 2001
From: "F. Jason Park" <jp@HIDDEN>
Date: Mon, 14 Feb 2022 02:36:57 -0800
Subject: [PATCH 4/4] [POC] Demo SOCKS RESOLVE over HTTPS

* test/lisp/net/socks-test.el (test-socks-https-poc): Provide
throwaway test demoing an HTTPS connection over a TOR proxy service.
---
 test/lisp/net/socks-tests.el | 55 +++++++++++++++++++++++++++++++++++-
 1 file changed, 54 insertions(+), 1 deletion(-)

diff --git a/test/lisp/net/socks-tests.el b/test/lisp/net/socks-tests.el
index 51e2e40631..4963dd7b40 100644
--- a/test/lisp/net/socks-tests.el
+++ b/test/lisp/net/socks-tests.el
@@ -21,7 +21,7 @@
 
 ;;; Code:
 
-(require 'ert)
+(require 'ert-x)
 (require 'socks)
 (require 'url-http)
 
@@ -329,4 +329,57 @@ tor-resolve-5
     (kill-buffer (process-buffer server))
     (delete-process server)))
 
+(defvar test-socks-service ; "127.0.0.1:1080" -> ("127.0.0.1", 1080)
+  (when-let ((present (getenv "TEST_SOCKS_SERVICE"))
+             (parts (split-string present ":")))
+    (list (car parts) (string-to-number (cadr parts)))))
+
+(declare-function gnutls-negotiate "gnutls"
+                  (&rest spec
+                         &key process type hostname priority-string
+                         trustfiles crlfiles keylist min-prime-bits
+                         verify-flags verify-error verify-hostname-error
+                         &allow-other-keys))
+
+(ert-deftest test-socks-https-poc ()
+  :tags '(:unstable)
+  (unless test-socks-service (ert-skip "SOCKS service missing"))
+  (unless (gnutls-available-p) (ert-skip "SOCKS resolve test needs GNUTLS"))
+  (ert-with-temp-file tempfile
+    :prefix "emacs-test-socks-network-security-"
+    (let* ((socks-server `("tor" ,@test-socks-service 5))
+           (socks-password "")
+           (nsm-settings-file tempfile)
+           (url-gateway-method 'socks)
+           (id "sha1:df77269389e537fcc9a5fe61667133b5bb97d42e")
+           (host "check.torproject.org")
+           (url (url-generic-parse-url "https://check.torproject.org"))
+           ;;
+           done
+           ;;
+           (cb (lambda (&rest _r)
+                 (goto-char (point-min))
+                 (should (search-forward "Congratulations" nil t))
+                 (setq done t)))
+           (orig (symbol-function #'socks--open-network-stream)))
+      (cl-letf (((symbol-function 'socks--open-network-stream)
+                 (lambda (&rest rest)
+                   (let ((proc (apply orig rest)))
+                     (gnutls-negotiate :process proc :hostname host)
+                     (should (nsm-verify-connection proc host 443 t))))))
+        (ert-info ("Connect to HTTPS endpoint over Tor SOCKS proxy")
+          (unwind-protect
+              (progn
+                (advice-add 'network-lookup-address-info :override
+                            #'socks-tor-resolve)
+                (should-not (nsm-host-settings id))
+                (url-http url cb '(nil))
+                (should (nsm-host-settings id))
+                (ert-info ("Wait for response")
+                  (with-timeout (3 (error "Request timed out"))
+                    (unless done
+                      (sleep-for 0.1)))))
+            (advice-remove 'network-lookup-address-info
+                           #'socks-tor-resolve)))))))
+
 ;;; socks-tests.el ends here
-- 
2.34.1


--=-=-=--

Message received at control <at> debbugs.gnu.org:

Received: (at control) by debbugs.gnu.org; 15 Feb 2022 00:31:34 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Feb 14 19:31:33 2022
Received: from localhost ([127.0.0.1]:43035 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1nJlkj-0003CY-P5
	for submit <at> debbugs.gnu.org; Mon, 14 Feb 2022 19:31:33 -0500
Received: from mail-108-mta109.mxroute.com ([136.175.108.109]:46795)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <jp@HIDDEN>) id 1nJlki-0003CJ-2O
 for control <at> debbugs.gnu.org; Mon, 14 Feb 2022 19:31:33 -0500
Received: from filter006.mxroute.com ([140.82.40.27] 140.82.40.27.vultr.com)
 (Authenticated sender: mN4UYu2MZsgR)
 by mail-108-mta109.mxroute.com (ZoneMTA) with ESMTPSA id
 17efaca7be10005a20.001 for <control <at> debbugs.gnu.org>
 (version=TLSv1/SSLv3 cipher=ECDHE-RSA-AES128-GCM-SHA256);
 Tue, 15 Feb 2022 00:31:21 +0000
X-Zone-Loop: 4498b9e094cbc8251598db5cab4bc4fea80f6824341b
X-Originating-IP: [140.82.40.27]
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=neverwas.me
 ; s=x;
 h=Content-Type:MIME-Version:Message-ID:Date:Subject:To:From:Sender:
 Reply-To:Cc:Content-Transfer-Encoding:Content-ID:Content-Description:
 Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:
 In-Reply-To:References:List-Id:List-Help:List-Unsubscribe:List-Subscribe:
 List-Post:List-Owner:List-Archive;
 bh=/cQYJDSiK61tXOlK2Pm3VU3iCYO7hBnTw8kkMSvJgPA=; b=T3gJo8bukaOjXnxD0BG+j9+Gee
 +2dTXpf7bN3kZE/VZrFpXbcfsp086sBpEqvi9y1u5bsmkl+VQ+eqLeK5XtCuyUb5t9xRDsM/YkOPU
 PqBVzMYMSU7gS7nn72CpjURB8IEMyB2uXCIwRRSKzVeENuQumJDukv9T8XZ3r+h34UFrK9YNS+YLU
 3/PBvQvsV85KLHTOn8qVFaIxxRl2vAf/uU5F0rMVnUxUpFPwu+N+PHy94TBn+BrdmlCColWn1k/XJ
 OZ9Y6hUCfuIW+VpP6nR09zt+7AciQESDLj2qBvJOQCMJ4AKTtEdVDpRvyrRq3RDEAiu95iZ+qy5Br
 UTg/dLWA==;
From: "J.P." <jp@HIDDEN>
To: control <at> debbugs.gnu.org
Subject: control message for bug #53941
Date: Mon, 14 Feb 2022 16:31:18 -0800
Message-ID: <87zgmtynqh.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain
X-AuthUser: masked@HIDDEN
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: control
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

tags 53941 + patch
quit

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#53941: 27.2; socks + tor dont work with https
Resent-From: Jacobo <gnuhacker@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sat, 19 Feb 2022 21:05:02 +0000
Resent-Message-ID: <handler.53941.B53941.16453046841283 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 53941
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: patch
To: "J.P." <jp@HIDDEN>
Cc: 53941 <at> debbugs.gnu.org
Received: via spool by 53941-submit <at> debbugs.gnu.org id=B53941.16453046841283
          (code B ref 53941); Sat, 19 Feb 2022 21:05:02 +0000
Received: (at 53941) by debbugs.gnu.org; 19 Feb 2022 21:04:44 +0000
Received: from localhost ([127.0.0.1]:58813 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1nLWuJ-0000Kd-UO
	for submit <at> debbugs.gnu.org; Sat, 19 Feb 2022 16:04:44 -0500
Received: from mx1.riseup.net ([198.252.153.129]:44756)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <gnuhacker@HIDDEN>) id 1nLWuI-0000KO-E7
 for 53941 <at> debbugs.gnu.org; Sat, 19 Feb 2022 16:04:43 -0500
Received: from fews1.riseup.net (fews1-pn.riseup.net [10.0.1.83])
 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
 key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256
 client-signature RSA-PSS (2048 bits) client-digest SHA256)
 (Client CN "mail.riseup.net", Issuer "R3" (not verified))
 by mx1.riseup.net (Postfix) with ESMTPS id 4K1LfX4FSwzF44M;
 Sat, 19 Feb 2022 13:04:36 -0800 (PST)
X-Riseup-User-ID: C84BA236F5101290351971B5CC63B5D706A0B5CABE282B6EA335C44739BD944D
Received: from [127.0.0.1] (localhost [127.0.0.1])
 by fews1.riseup.net (Postfix) with ESMTPSA id 4K1LfW5r9pz5vMd;
 Sat, 19 Feb 2022 13:04:35 -0800 (PST)
From: Jacobo <gnuhacker@HIDDEN>
References: <87pmntfym7.fsf@HIDDEN> <8735kl1v58.fsf@HIDDEN>
Date: Sat, 19 Feb 2022 22:04:34 +0100
In-Reply-To: <8735kl1v58.fsf@HIDDEN> (J. P.'s message of "Mon, 14 Feb
 2022 04:37:39 -0800")
Message-ID: <87a6emftzx.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

"J.P." <jp@HIDDEN> writes:
> Jacobo <gnuhacker@HIDDEN> writes:
>> Emacs can not resolve domains when it is https if you are using a
>> socks proxy (socks.el) [...] It works, load http://gnu.org (HTTP in
>> plain) Also work with .onion domains in HTTP plain No problems with
>> HTTP but When I try: M-x eww RET https://gnu.org RET
>> Return an error: Bad Request

> It's certainly possible (see attached). But can it be done responsibly?
> In this day and age, when processes and services resolve host names in
> all manner of ways, how can we be confident there won't be any leaks?

oh, nevermind

> At present, the main interfaces to various protocol stacks (for
> example, url-gw.el and friends) don't seem geared toward making those
> kinds of assurances. (Not that they ought to be.)  That said,
> providing the building blocks on the SOCKS side doesn't seem like the
> crime of the century. I've been sitting on what became the basis for
> these patches for a while now, but these here were hastily adapted and
> might come with some warts. Still, I believe them straightforward
> enough to illustrate a basic means of achieving what you're after.

>> In GNU Emacs 27.2 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.30,

> I also have some examples with shims for 27 running periodically in CI.
> These include a demo of using ERC to connect to Libera.Chat via SOCKS
> over TLS. (But that requires an IRCv3 library, which is still a work in
> progress.)

Now Ive tryed same in other computer with Trisquel 10, the problem still
happend

> If you're interested in experimenting with any of this stuff, please
> let me know. That goes for anyone else out there as well. Thanks.

yes, how can I help?

-- 
Emacs Lover.
FSF Member.
Free/Libre Software supporter.
stallmansupport.org - Disinformation succeeds because so many people
care deeply about injustice but do not take the time to check the facts.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#53941: 27.2; socks + tor dont work with https
Resent-From: "J.P." <jp@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Mon, 21 Feb 2022 15:03:01 +0000
Resent-Message-ID: <handler.53941.B53941.164545573521313 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 53941
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: patch
To: Jacobo <gnuhacker@HIDDEN>
Cc: 53941 <at> debbugs.gnu.org
Received: via spool by 53941-submit <at> debbugs.gnu.org id=B53941.164545573521313
          (code B ref 53941); Mon, 21 Feb 2022 15:03:01 +0000
Received: (at 53941) by debbugs.gnu.org; 21 Feb 2022 15:02:15 +0000
Received: from localhost ([127.0.0.1]:38608 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1nMACc-0005XN-KB
	for submit <at> debbugs.gnu.org; Mon, 21 Feb 2022 10:02:15 -0500
Received: from mail-108-mta1.mxroute.com ([136.175.108.1]:40489)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <jp@HIDDEN>) id 1nMACa-0005S1-La
 for 53941 <at> debbugs.gnu.org; Mon, 21 Feb 2022 10:02:13 -0500
Received: from filter006.mxroute.com ([140.82.40.27] 140.82.40.27.vultr.com)
 (Authenticated sender: mN4UYu2MZsgR)
 by mail-108-mta1.mxroute.com (ZoneMTA) with ESMTPSA id 17f1ccdc5b20005a20.001
 for <53941 <at> debbugs.gnu.org>
 (version=TLSv1/SSLv3 cipher=ECDHE-RSA-AES128-GCM-SHA256);
 Mon, 21 Feb 2022 15:02:01 +0000
X-Zone-Loop: 2557600d9f6b113017a2a58ac6ba48e0ece56b75b38c
X-Originating-IP: [140.82.40.27]
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=neverwas.me
 ; s=x;
 h=Content-Type:MIME-Version:Message-ID:In-Reply-To:Date:References:
 Subject:Cc:To:From:Sender:Reply-To:Content-Transfer-Encoding:Content-ID:
 Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc
 :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe:
 List-Post:List-Owner:List-Archive;
 bh=figKuNx6501QgoeDkYL1r+901Iay72/4c65H1ue+nsY=; b=A0zy5VbOV/QnxGnibBrCwcYrc6
 LYEzkkTfKwm0OKxoxgyXX4gFwcUXZtbgRhbNlGX8pp0UweNbCY4xpK7NCn7iR43ws7GYF/88EOU28
 VKEiEP6PFTfelj2g7d2idQE9pgILIsVu0HAoxQ5GqlKPAexG7idjn4lU1OuIWBynDVaWodVhjQ7ZD
 e+eKrn0ZPNqyjzZ2rs6r7ovtfVy/nITG8I76hPR3qsEVw5P1xy6S68889v+CTikCEfLPaBDh2QFtd
 vWnDxO/A/nI5zZufsma7NAU22cBVEQEEpkiGAZW33k3F5VqAiByxyuxfz8azi/m7k7ax6wusyC1Qf
 VDpQ64Uw==;
From: "J.P." <jp@HIDDEN>
References: <87pmntfym7.fsf@HIDDEN> <8735kl1v58.fsf@HIDDEN>
 <87a6emftzx.fsf@HIDDEN>
Date: Mon, 21 Feb 2022 07:01:58 -0800
In-Reply-To: <87a6emftzx.fsf@HIDDEN> (Jacobo's message of "Sat, 19 Feb
 2022 22:04:34 +0100")
Message-ID: <87k0do5km1.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/29.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-AuthUser: masked@HIDDEN
X-Spam-Score: -0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Jacobo <gnuhacker@HIDDEN> writes:

>> In this day and age, when processes and services resolve host names in
>> all manner of ways, how can we be confident there won't be any leaks?
>
> oh, nevermind

I certainly don't want to discourage anyone from trying to solve this.
But DNS leaks aside, predicting what ought to be proxied still seems
like a serious undertaking (at least from my peabrained perspective).
For example, if you connect to an IRC network over Tor and click a
hyperlink in a channel, should the resulting connection also happen over
Tor? What about when the SOCKS service isn't Tor but something else,
like SSH? Should similar follow-on connections also originate from the
proxy host (your VPS or shell provider, for example)?

>> I also have some examples with shims for 27 running periodically in CI.
>> These include a demo of using ERC to connect to Libera.Chat via SOCKS
>> over TLS. (But that requires an IRCv3 library, which is still a work in
>> progress.)
>
> Now Ive tryed same in other computer with Trisquel 10, the problem still
> happend

Sorry, are you saying you repeated the steps in your original post and
got the same result (failure) on another computer? If so, that's to be
expected because Tor over SOCKS with TLS isn't supported OOTB with any
Emacs, not even 29. Apologies if I implied otherwise.

>> If you're interested in experimenting with any of this stuff, please
>> let me know. That goes for anyone else out there as well. Thanks.
>
> yes, how can I help?

When 28 comes out, you can try applying those patches. Or, if you're not
cool with that, I can give you a replacement socks.el to shadow the
original. To check whether it's working, do

  M-: (boundp 'socks--errors-4) RET

or similar and then try mimicking the recipe in that last patch (the one
named POC demo something). Once that works, try adapting it to your
needs for whatever protocol (except for ERC, for which you'll need to
install an unofficial WIP version).

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#53941: 27.2; socks + tor dont work with https
Resent-From: "J.P." <jp@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Tue, 01 Mar 2022 14:31:02 +0000
Resent-Message-ID: <handler.53941.B53941.164614500914568 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 53941
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: patch
To: Jacobo <gnuhacker@HIDDEN>
Cc: 53941 <at> debbugs.gnu.org
Received: via spool by 53941-submit <at> debbugs.gnu.org id=B53941.164614500914568
          (code B ref 53941); Tue, 01 Mar 2022 14:31:02 +0000
Received: (at 53941) by debbugs.gnu.org; 1 Mar 2022 14:30:09 +0000
Received: from localhost ([127.0.0.1]:35918 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1nP3Vu-0003mq-3N
	for submit <at> debbugs.gnu.org; Tue, 01 Mar 2022 09:30:09 -0500
Received: from mail-108-mta15.mxroute.com ([136.175.108.15]:39339)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <jp@HIDDEN>) id 1nP3Vq-0003ly-03
 for 53941 <at> debbugs.gnu.org; Tue, 01 Mar 2022 09:30:05 -0500
Received: from filter006.mxroute.com ([140.82.40.27] 140.82.40.27.vultr.com)
 (Authenticated sender: mN4UYu2MZsgR)
 by mail-108-mta15.mxroute.com (ZoneMTA) with ESMTPSA id 17f45e338e00005a20.001
 for <53941 <at> debbugs.gnu.org>
 (version=TLSv1/SSLv3 cipher=ECDHE-RSA-AES128-GCM-SHA256);
 Tue, 01 Mar 2022 14:29:53 +0000
X-Zone-Loop: 79703327aa60909ab55ae994c6fa01a522f9c1064320
X-Originating-IP: [140.82.40.27]
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=neverwas.me
 ; s=x;
 h=Content-Type:MIME-Version:Message-ID:In-Reply-To:Date:References:
 Subject:Cc:To:From:Sender:Reply-To:Content-Transfer-Encoding:Content-ID:
 Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc
 :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe:
 List-Post:List-Owner:List-Archive;
 bh=woorlDzlTwh0zTEf0re8lScUuF36Ls4pYn9sbPnEJZk=; b=ZtsPCdeDutK87cD+dSAXKFDy3G
 VyQazXoR65gED0ag1WUaEI15UOuHfb1WowRRcRIGDwYJRk6ojrrlG6fHks4ZacDf5IZeCvhT9+Y4g
 X6XG3xbYvypm7/EiUjfjRk2USMTTps+MlbdNwskFpom4dBCHWR+rJn3MBP+UYgNaD62QkfITrI8PK
 csKjTdFEd3OhmkB1SlYhtH1+XLJZlkQksWvoaWM9WZb4TqqElXaYV4vonzJXxGGglOlG7k/XLfuCN
 mtjte7ZWEERFey2x+MC76VPQPXrbHj6AJ5ZuD9P+sKM8aQ7FaZqkDDJq8lLshj4RWNmwCff/AgEnF
 uAmEEc2Q==;
From: "J.P." <jp@HIDDEN>
References: <87pmntfym7.fsf@HIDDEN> <8735kl1v58.fsf@HIDDEN>
 <87a6emftzx.fsf@HIDDEN> <87k0do5km1.fsf@HIDDEN>
Date: Tue, 01 Mar 2022 06:29:49 -0800
In-Reply-To: <87k0do5km1.fsf@HIDDEN> (J. P.'s message of "Mon, 21 Feb
 2022 07:01:58 -0800")
Message-ID: <87pmn5n3tu.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/29.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="=-=-="
X-AuthUser: masked@HIDDEN
X-Spam-Score: -0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

--=-=-=
Content-Type: text/plain

v2. Minor corrections (another bug in existing test, etc.).

--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment; filename=0000-NOT-A-PATCH-v1-v2.diff

From 598e8471789bd6e7eb5a7f3ebc1bbed3cf61f4c6 Mon Sep 17 00:00:00 2001
From: "F. Jason Park" <jp@HIDDEN>
Date: Tue, 1 Mar 2022 06:09:00 -0800
Subject: [PATCH 0/5] NOT A PATCH

*** BLURB HERE ***

F. Jason Park (5):
  Simplify network-stream opener in socks.el
  Fix string encoding bug in socks tests
  Add support for SOCKS 4a
  Support SOCKS RESOLVE extension
  [POC] Demo SOCKS RESOLVE over HTTPS

 lisp/net/socks.el            | 130 +++++++++++++++++++++++++++--------
 test/lisp/net/socks-tests.el | 113 ++++++++++++++++++++++++++++--
 2 files changed, 208 insertions(+), 35 deletions(-)

Interdiff:
diff --git a/lisp/net/socks.el b/lisp/net/socks.el
index 7201ed8e06..cd026fd163 100644
--- a/lisp/net/socks.el
+++ b/lisp/net/socks.el
@@ -333,24 +333,23 @@ socks-filter
 
 (defvar socks-override-functions nil
   "If non-nil, overwrite `open-network-stream' function with SOCKSified version.")
-
-(when socks-override-functions
-  (advice-add 'open-network-stream :around #'socks--open-network-stream))
-
-(defun socks-open-connection (server-info)
+(make-obsolete-variable 'socks-override-functions
+                        "`socks--open-network-stream' now takes a process arg."
+                        "29.1")
+
+(defun socks-open-connection (server-info &optional opener)
+  "Create and initialize a SOCKS process.
+Perform authentication if needed.  SERVER-INFO should resemble
+`socks-server'.  OPENER, when present, should be a substitute for
+`open-network-stream' and take the same arguments."
   (interactive)
   (save-excursion
-    (let ((proc
-           (let ((socks-override-functions nil))
-             (open-network-stream "socks"
-				  nil
-				  (nth 1 server-info)
-				  (nth 2 server-info))))
+    (let ((proc (funcall (or opener #'open-network-stream)
+                         "socks" nil (nth 1 server-info) (nth 2 server-info)))
 	  (authtype nil)
 	  version)
 
       ;; Initialize process and info about the process
-      (set-process-coding-system proc 'binary 'binary)
       (set-process-filter proc #'socks-filter)
       (set-process-query-on-exit-flag proc nil)
       (process-put proc 'socks t)
@@ -530,22 +529,18 @@ socks-find-services-entry
   (gethash (downcase service)
 	      (if udp socks-udp-services socks-tcp-services)))
 
-(defun socks-open-network-stream (name buffer host service)
-  (let ((socks-override-functions t))
-    (socks--open-network-stream
-     (lambda (&rest args)
-       (let ((socks-override-functions nil))
-         (apply #'open-network-stream args)))
-     name buffer host service)))
-
-(defun socks--open-network-stream (orig-fun name buffer host service &rest params)
-  (let ((route (and socks-override-functions
-                    (socks-find-route host service))))
-    (if (not route)
-	(apply orig-fun name buffer host service params)
-      ;; FIXME: Obey `params'!
-      (let* ((proc (socks-open-connection route))
-	     (version (process-get proc 'socks-server-protocol))
+(defun socks-open-network-stream (name buffer host service &rest params)
+  (if-let* ((route (socks-find-route host service))
+            (proc (socks-open-connection route #'open-network-stream)))
+      (socks--open-network-stream proc buffer host service)
+    (message "Warning: no SOCKS route found for %s:%s" host service)
+    ;; Support legacy behavior (likely undesirable in most cases)
+    (apply #'open-network-stream name buffer host service params)))
+
+(defun socks--open-network-stream (proc buffer host service)
+  (progn ; temporarily preserve git blame for easier reviewing
+    (progn ; could rename to something like `socks--initiate-command-sequence'
+      (let* ((version (process-get proc 'socks-server-protocol))
              (atype
               (cond
                ((equal version 4)
diff --git a/test/lisp/net/socks-tests.el b/test/lisp/net/socks-tests.el
index 4963dd7b40..f2600210b0 100644
--- a/test/lisp/net/socks-tests.el
+++ b/test/lisp/net/socks-tests.el
@@ -63,21 +63,21 @@ socks-tests-filter-response-parsing-v4
     (process-put proc 'socks-state socks-state-waiting)
     (process-put proc 'socks-server-protocol 4)
     (ert-info ("Receive initial incomplete segment")
-      (socks-filter proc (concat [0 90 0 0 93 184 216]))
-      ;; From example.com: OK status ^      ^ msg start
+      (socks-filter proc (unibyte-string 0 90 0 0 93 184 216))
+      ;; From example.com: OK status       ^      ^ msg start
       (ert-info ("State still set to waiting")
         (should (eq (process-get proc 'socks-state) socks-state-waiting)))
       (ert-info ("Response field is nil because processing incomplete")
         (should-not (process-get proc 'socks-response)))
       (ert-info ("Scratch field holds stashed partial payload")
-        (should (string= (concat [0 90 0 0 93 184 216])
+        (should (string= (unibyte-string 0 90 0 0 93 184 216)
                          (process-get proc 'socks-scratch)))))
     (ert-info ("Last part arrives")
       (socks-filter proc "\42") ; ?\" 34
       (ert-info ("State transitions to complete (length check passes)")
         (should (eq (process-get proc 'socks-state) socks-state-connected)))
       (ert-info ("Scratch and response fields hold stash w. last chunk")
-        (should (string= (concat [0 90 0 0 93 184 216 34])
+        (should (string= (unibyte-string 0 90 0 0 93 184 216 34)
                          (process-get proc 'socks-response)))
         (should (string= (process-get proc 'socks-response)
                          (process-get proc 'socks-scratch)))))
@@ -137,10 +137,10 @@ socks-tests-canned-server-create
          (pats socks-tests-canned-server-patterns)
          (filt (lambda (proc line)
                  (pcase-let ((`(,pat . ,resp) (pop pats)))
-                   (setq resp (apply #'unibyte-string (append resp nil)))
                    (unless (or (and (vectorp pat) (equal pat (vconcat line)))
                                (string-match-p pat line))
                      (error "Unknown request: %s" line))
+                   (setq resp (apply #'unibyte-string (append resp nil)))
                    (let ((print-escape-control-characters t))
                      (message "[%s] <- %s" name (prin1-to-string line))
                      (message "[%s] -> %s" name (prin1-to-string resp)))
@@ -374,11 +374,11 @@ test-socks-https-poc
                             #'socks-tor-resolve)
                 (should-not (nsm-host-settings id))
                 (url-http url cb '(nil))
-                (should (nsm-host-settings id))
                 (ert-info ("Wait for response")
                   (with-timeout (3 (error "Request timed out"))
                     (unless done
-                      (sleep-for 0.1)))))
+                      (sleep-for 0.1))))
+                (should (nsm-host-settings id)))
             (advice-remove 'network-lookup-address-info
                            #'socks-tor-resolve)))))))
 
-- 
2.35.1


--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
 filename=0001-Simplify-network-stream-opener-in-socks.el.patch

From e1b377ee054f95a4f2064eef6972d350f69767f3 Mon Sep 17 00:00:00 2001
From: "F. Jason Park" <jp@HIDDEN>
Date: Tue, 1 Mar 2022 02:12:02 -0800
Subject: [PATCH 1/5] Simplify network-stream opener in socks.el

* lisp/net/socks.el (socks-override-functions): Make variable
obsolete and remove uses.
(socks-open-connection): Add optional opener arg.
(socks-open-network-stream): Accept additional params for calling
`open-network-stream' as a fallback when a route cannot be found.
(socks--open-network-stream): Reduce role to merely issuing the first
command using an existing process.  Change signature accordingly.
---
 lisp/net/socks.el | 50 ++++++++++++++++++++++-------------------------
 1 file changed, 23 insertions(+), 27 deletions(-)

diff --git a/lisp/net/socks.el b/lisp/net/socks.el
index 8df0773e1d..9bc301618c 100644
--- a/lisp/net/socks.el
+++ b/lisp/net/socks.el
@@ -323,19 +323,19 @@ socks-filter
 
 (defvar socks-override-functions nil
   "If non-nil, overwrite `open-network-stream' function with SOCKSified version.")
-
-(when socks-override-functions
-  (advice-add 'open-network-stream :around #'socks--open-network-stream))
-
-(defun socks-open-connection (server-info)
+(make-obsolete-variable 'socks-override-functions
+                        "`socks--open-network-stream' now takes a process arg."
+                        "29.1")
+
+(defun socks-open-connection (server-info &optional opener)
+  "Create and initialize a SOCKS process.
+Perform authentication if needed.  SERVER-INFO should resemble
+`socks-server'.  OPENER, when present, should be a substitute for
+`open-network-stream' and take the same arguments."
   (interactive)
   (save-excursion
-    (let ((proc
-           (let ((socks-override-functions nil))
-             (open-network-stream "socks"
-				  nil
-				  (nth 1 server-info)
-				  (nth 2 server-info))))
+    (let ((proc (funcall (or opener #'open-network-stream)
+                         "socks" nil (nth 1 server-info) (nth 2 server-info)))
 	  (authtype nil)
 	  version)
 
@@ -508,22 +508,18 @@ socks-find-services-entry
   (gethash (downcase service)
 	      (if udp socks-udp-services socks-tcp-services)))
 
-(defun socks-open-network-stream (name buffer host service)
-  (let ((socks-override-functions t))
-    (socks--open-network-stream
-     (lambda (&rest args)
-       (let ((socks-override-functions nil))
-         (apply #'open-network-stream args)))
-     name buffer host service)))
-
-(defun socks--open-network-stream (orig-fun name buffer host service &rest params)
-  (let ((route (and socks-override-functions
-                    (socks-find-route host service))))
-    (if (not route)
-	(apply orig-fun name buffer host service params)
-      ;; FIXME: Obey `params'!
-      (let* ((proc (socks-open-connection route))
-	     (version (process-get proc 'socks-server-protocol))
+(defun socks-open-network-stream (name buffer host service &rest params)
+  (if-let* ((route (socks-find-route host service))
+            (proc (socks-open-connection route #'open-network-stream)))
+      (socks--open-network-stream proc buffer host service)
+    (message "Warning: no SOCKS route found for %s:%s" host service)
+    ;; Support legacy behavior (likely undesirable in most cases)
+    (apply #'open-network-stream name buffer host service params)))
+
+(defun socks--open-network-stream (proc buffer host service)
+  (progn ; temporarily preserve git blame for easier reviewing
+    (progn ; could rename to something like `socks--initiate-command-sequence'
+      (let* ((version (process-get proc 'socks-server-protocol))
              (atype
               (cond
                ((equal version 4)
-- 
2.35.1


--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
 filename=0002-Fix-string-encoding-bug-in-socks-tests.patch

From 8f33588517c7333d3bd08375c406cd46726b51d6 Mon Sep 17 00:00:00 2001
From: "F. Jason Park" <jp@HIDDEN>
Date: Mon, 14 Feb 2022 02:36:57 -0800
Subject: [PATCH 2/5] Fix string encoding bug in socks tests

* test/lisp/net/socks-tests.el (socks-tests-canned-server-create,
socks-tests-filter-response-parsing-v4): Fix bug in process filter to
prevent prepared outgoing responses from being implicitly encoded as
utf-8.  Fix similar mistake in v4 filter test.
---
 test/lisp/net/socks-tests.el | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/test/lisp/net/socks-tests.el b/test/lisp/net/socks-tests.el
index 461796bdf9..d9ef53ae35 100644
--- a/test/lisp/net/socks-tests.el
+++ b/test/lisp/net/socks-tests.el
@@ -63,21 +63,21 @@ socks-tests-filter-response-parsing-v4
     (process-put proc 'socks-state socks-state-waiting)
     (process-put proc 'socks-server-protocol 4)
     (ert-info ("Receive initial incomplete segment")
-      (socks-filter proc (concat [0 90 0 0 93 184 216]))
-      ;; From example.com: OK status ^      ^ msg start
+      (socks-filter proc (unibyte-string 0 90 0 0 93 184 216))
+      ;; From example.com: OK status       ^      ^ msg start
       (ert-info ("State still set to waiting")
         (should (eq (process-get proc 'socks-state) socks-state-waiting)))
       (ert-info ("Response field is nil because processing incomplete")
         (should-not (process-get proc 'socks-response)))
       (ert-info ("Scratch field holds stashed partial payload")
-        (should (string= (concat [0 90 0 0 93 184 216])
+        (should (string= (unibyte-string 0 90 0 0 93 184 216)
                          (process-get proc 'socks-scratch)))))
     (ert-info ("Last part arrives")
       (socks-filter proc "\42") ; ?\" 34
       (ert-info ("State transitions to complete (length check passes)")
         (should (eq (process-get proc 'socks-state) socks-state-connected)))
       (ert-info ("Scratch and response fields hold stash w. last chunk")
-        (should (string= (concat [0 90 0 0 93 184 216 34])
+        (should (string= (unibyte-string 0 90 0 0 93 184 216 34)
                          (process-get proc 'socks-response)))
         (should (string= (process-get proc 'socks-response)
                          (process-get proc 'socks-scratch)))))
@@ -140,10 +140,11 @@ socks-tests-canned-server-create
                    (unless (or (and (vectorp pat) (equal pat (vconcat line)))
                                (string-match-p pat line))
                      (error "Unknown request: %s" line))
+                   (setq resp (apply #'unibyte-string (append resp nil)))
                    (let ((print-escape-control-characters t))
                      (message "[%s] <- %s" name (prin1-to-string line))
                      (message "[%s] -> %s" name (prin1-to-string resp)))
-                   (process-send-string proc (concat resp)))))
+                   (process-send-string proc resp))))
          (serv (make-network-process :server 1
                                      :buffer (get-buffer-create name)
                                      :filter filt
-- 
2.35.1


--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
 filename=0003-Add-support-for-SOCKS-4a.patch

From b90a6474b6edb4dd33cffa0e05f1a7f1a3e1c9be Mon Sep 17 00:00:00 2001
From: "F. Jason Park" <jp@HIDDEN>
Date: Mon, 14 Feb 2022 02:36:57 -0800
Subject: [PATCH 3/5] Add support for SOCKS 4a

* lisp/net/socks.el (socks-server): Add new choice `4a' to version
field of option.  This may appear to change the type of the field from
a number to a union of symbols and numbers.  However,
`socks-send-command' and `socks-filter' already expect a possible
`http' value for this field (also a symbol).
(socks--errors-4): Add new constant containing error messages for
socks version 4.  The semantics are faithful, but the wording is
ad-libbed.
(socks-send-command): Massage existing handling for version 4 to
accommodate 4a.

* test/lisp/net/socks-tests.el (socks-tests-v4a-basic): add test for
4a.
Bug#53941
---
 lisp/net/socks.el            | 22 ++++++++++++++++++++--
 test/lisp/net/socks-tests.el | 13 +++++++++++++
 2 files changed, 33 insertions(+), 2 deletions(-)

diff --git a/lisp/net/socks.el b/lisp/net/socks.el
index 9bc301618c..0615db8681 100644
--- a/lisp/net/socks.el
+++ b/lisp/net/socks.el
@@ -162,6 +162,7 @@ socks-server
 	  (radio-button-choice :tag "SOCKS Version"
 			       :format "%t: %v"
 			       (const :tag "SOCKS v4  " :format "%t" :value 4)
+                               (const :tag "SOCKS v4a"  :format "%t" :value 4a)
 			       (const :tag "SOCKS v5"   :format "%t" :value 5))))
 
 
@@ -202,6 +203,12 @@ socks-errors
     "Command not supported"
     "Address type not supported"))
 
+(defconst socks--errors-4
+  '("Granted"
+    "Rejected or failed"
+    "Cannot connect to identd on the client"
+    "Client and identd report differing user IDs"))
+
 ;; The socks v5 address types
 (defconst socks-address-type-v4   1)
 (defconst socks-address-type-name 3)
@@ -400,6 +407,7 @@ socks-send-command
 		(format "%c%s" (length address) address))
 	       (t
 		(error "Unknown address type: %d" atype))))
+        trailing
 	request version)
     (or (process-get proc 'socks)
         (error "socks-send-command called on non-SOCKS connection %S" proc))
@@ -417,6 +425,12 @@ socks-send-command
 			     (t
 			      (error "Unsupported address type for HTTP: %d" atype)))
 			    port)))
+     ((when (eq version '4a)
+        (setf addr "\0\0\0\1"
+              trailing (concat address "\0")
+              version 4 ; done with the "a" part
+              (process-get proc 'socks-server-protocol) 4)
+        nil)) ; fall through
      ((equal version 4)
       (setq request (concat
 		     (unibyte-string
@@ -426,7 +440,8 @@ socks-send-command
 		      (logand port #xff)) ; port, low byte
 		     addr                 ; address
 		     (user-full-name)     ; username
-		     "\0")))              ; terminate username
+                     "\0"                 ; terminate username
+                     trailing)))          ; optional host to look up
      ((equal version 5)
       (setq request (concat
 		     (unibyte-string
@@ -447,7 +462,10 @@ socks-send-command
 	nil				; Sweet sweet success!
       (delete-process proc)
       (error "SOCKS: %s"
-             (nth (or (process-get proc 'socks-reply) 1) socks-errors)))
+             (let ((no (or (process-get proc 'socks-reply) 1)))
+               (if (eq version 5)
+                   (nth no socks-errors)
+                 (nth (+ 90 no) socks--errors-4)))))
     proc))
 
 
diff --git a/test/lisp/net/socks-tests.el b/test/lisp/net/socks-tests.el
index d9ef53ae35..4e990ffdba 100644
--- a/test/lisp/net/socks-tests.el
+++ b/test/lisp/net/socks-tests.el
@@ -207,6 +207,19 @@ socks-tests-v4-basic
                  (lambda (&optional _) "foo")))
         (socks-tests-perform-hello-world-http-request)))))
 
+(ert-deftest socks-tests-v4a-basic ()
+  "Show correct preparation of SOCKS4a connect command."
+  (let ((socks-server '("server" "127.0.0.1" 10083 4a))
+        (url-user-agent "Test/4a-basic")
+        (socks-tests-canned-server-patterns
+         `(([4 1 0 80 0 0 0 1 ?f ?o ?o 0 ?e ?x ?a ?m ?p ?l ?e ?. ?c ?o ?m 0]
+            . [0 90 0 0 0 0 0 0])
+           ,socks-tests--hello-world-http-request-pattern)))
+    (ert-info ("Make HTTP request over SOCKS4A")
+      (cl-letf (((symbol-function 'user-full-name)
+                 (lambda (&optional _) "foo")))
+        (socks-tests-perform-hello-world-http-request)))))
+
 ;; Replace first pattern below with ([5 3 0 1 2] . [5 2]) to validate
 ;; against curl 7.71 with the following options:
 ;; $ curl --verbose -U foo:bar --proxy socks5h://127.0.0.1:10080 example.com
-- 
2.35.1


--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
 filename=0004-Support-SOCKS-RESOLVE-extension.patch

From 23a430c6d7fb2707dba7e217f279ba293ae2fce6 Mon Sep 17 00:00:00 2001
From: "F. Jason Park" <jp@HIDDEN>
Date: Mon, 14 Feb 2022 02:36:57 -0800
Subject: [PATCH 4/5] Support SOCKS RESOLVE extension

* lisp/net/socks.el (socks-resolve-command): Add new constant for the
SOCKS command RESOLVE, which comes by way of a nonstandard extension
from the TOR project.  It mirrors CONNECT in most respects but asks
the server to RESOLVE a host name and return its IP.  For details, see
https://github.com/torproject/torspec/blob/master/socks-extensions.txt
This shouldn't be confused with 5h/5-hostname, which is used to by
clients like cURL to allow users to bypass attempts to resolve a name
locally.
(socks--extract-resolve-response, socks-tor-resolve): Add utility
functions to query a SOCKS service supporting the RESOLVE extension.
Bug#53941
---
 lisp/net/socks.el            | 58 ++++++++++++++++++++++++++++++++++++
 test/lisp/net/socks-tests.el | 34 +++++++++++++++++++++
 2 files changed, 92 insertions(+)

diff --git a/lisp/net/socks.el b/lisp/net/socks.el
index 0615db8681..cd026fd163 100644
--- a/lisp/net/socks.el
+++ b/lisp/net/socks.el
@@ -181,6 +181,9 @@ socks-udp-associate-command
 (defconst socks-authentication-null 0)
 (defconst socks-authentication-failure 255)
 
+;; Extensions
+(defconst socks-resolve-command #xf0)
+
 ;; Response codes
 (defconst socks-response-success               0)
 (defconst socks-response-general-failure       1)
@@ -649,6 +652,61 @@ socks-nslookup-host
 	res)
     host))
 
+(defun socks--extract-resolve-response (proc)
+  "Parse response for PROC and maybe return destination IP address."
+  (let ((response (process-get proc 'socks-response)))
+    (cl-assert response) ; otherwise, msg not received in its entirety
+    (pcase (process-get proc 'socks-server-protocol)
+      (4 ; https://www.openssh.com/txt/socks4a.protocol
+       (when-let (((zerop (process-get proc 'socks-reply)))
+                  ((eq (aref response 1) 90)) ; #x5a request granted
+                  (a (substring response 4)) ; ignore port for now
+                  ((not (string-empty-p a)))
+                  ((not (string= a "\0\0\0\0"))))
+         a))
+      (5 ; https://tools.ietf.org/html/rfc1928
+       (cl-assert (eq 5 (aref response 0)) t)
+       (pcase (aref response 3) ; ATYP
+         (1 (and-let* ((a (substring response 4 8))
+                       ((not (string= a "\0\0\0\0")))
+                       a)))
+         ;; No reason to support RESOLVE_PTR [F1] extension, right?
+         (3 (let ((len (1- (aref response 4))))
+              (substring response 5 (+ 5 len))))
+         (4 (substring response 4 20)))))))
+
+(declare-function puny-encode-domain "puny" (domain))
+
+(defun socks-tor-resolve (name &optional _family)
+  "Return list of one vector IPv4 address for domain NAME.
+Or return nil on failure.  See `network-lookup-address-info' for format
+of return value.  Server must support the Tor RESOLVE command."
+  (let ((socks-password (or socks-password ""))
+        host
+        (port 80)  ; unused for now
+        proc
+        ip)
+    (unless (string-suffix-p ".onion" name)
+      (setq host (if (string-match "\\`[[:ascii:]]+\\'" name)
+                     name
+                   (require 'puny)
+                   (puny-encode-domain name)))
+      ;; "Host unreachable" may be raised when the lookup fails
+      (unwind-protect
+          (progn
+            (setq proc (socks-open-connection (socks-find-route host port)))
+            (socks-send-command proc
+                                socks-resolve-command
+                                socks-address-type-name
+                                host
+                                port)
+            (cl-assert (eq (process-get proc 'socks-state)
+                           socks-state-connected))
+            (setq ip (socks--extract-resolve-response proc)))
+        (when proc
+          (delete-process proc)))
+      (list (vconcat ip [0])))))
+
 (provide 'socks)
 
 ;;; socks.el ends here
diff --git a/test/lisp/net/socks-tests.el b/test/lisp/net/socks-tests.el
index 4e990ffdba..3d1aca9af4 100644
--- a/test/lisp/net/socks-tests.el
+++ b/test/lisp/net/socks-tests.el
@@ -295,4 +295,38 @@ socks-tests-v5-auth-none
       (socks-tests-perform-hello-world-http-request)))
   (should (assq 2 socks-authentication-methods)))
 
+(ert-deftest tor-resolve-4a ()
+  "Make request to TOR resolve service over SOCKS4a"
+  (let* ((socks-server '("server" "127.0.0.1" 19050 4a))
+         (socks-tests-canned-server-patterns
+          '(([4 #xf0 0 80 0 0 0 1 ?f ?o ?o 0 ?e ?x ?a ?m ?p ?l ?e ?. ?c ?o ?m 0]
+             . [0 90 0 0 93 184 216 34])))
+         (inhibit-message noninteractive)
+         (server (socks-tests-canned-server-create)))
+    (ert-info ("Query TOR RESOLVE service over SOCKS4")
+      (cl-letf (((symbol-function 'user-full-name)
+                 (lambda (&optional _) "foo")))
+        (should (equal '([93 184 216 34 0])
+                       (socks-tor-resolve "example.com")))))
+    (kill-buffer (process-buffer server))
+    (delete-process server)))
+
+(ert-deftest tor-resolve-5 ()
+  "Make request to TOR resolve service over SOCKS5"
+  (let* ((socks-server '("server" "127.0.0.1" 19051 5))
+         (socks-username "foo")
+         (socks-authentication-methods (append socks-authentication-methods
+                                               nil))
+         (inhibit-message noninteractive)
+         (socks-tests-canned-server-patterns
+          '(([5 2 0 2] . [5 2])
+            ([1 3 ?f ?o ?o 0] . [1 0])
+            ([5 #xf0 0 3 11 ?e ?x ?a ?m ?p ?l ?e ?. ?c ?o ?m 0 80]
+             . [5 0 0 1 93 184 216 34 0 0])))
+         (server (socks-tests-canned-server-create)))
+    (ert-info ("Query TOR RESOLVE service over SOCKS5")
+      (should (equal '([93 184 216 34 0]) (socks-tor-resolve "example.com"))))
+    (kill-buffer (process-buffer server))
+    (delete-process server)))
+
 ;;; socks-tests.el ends here
-- 
2.35.1


--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
 filename=0005-POC-Demo-SOCKS-RESOLVE-over-HTTPS.patch

From 598e8471789bd6e7eb5a7f3ebc1bbed3cf61f4c6 Mon Sep 17 00:00:00 2001
From: "F. Jason Park" <jp@HIDDEN>
Date: Mon, 14 Feb 2022 02:36:57 -0800
Subject: [PATCH 5/5] [POC] Demo SOCKS RESOLVE over HTTPS

* test/lisp/net/socks-test.el (test-socks-https-poc): Provide
throwaway test demoing an HTTPS connection over a TOR proxy service.
---
 test/lisp/net/socks-tests.el | 55 +++++++++++++++++++++++++++++++++++-
 1 file changed, 54 insertions(+), 1 deletion(-)

diff --git a/test/lisp/net/socks-tests.el b/test/lisp/net/socks-tests.el
index 3d1aca9af4..f2600210b0 100644
--- a/test/lisp/net/socks-tests.el
+++ b/test/lisp/net/socks-tests.el
@@ -21,7 +21,7 @@
 
 ;;; Code:
 
-(require 'ert)
+(require 'ert-x)
 (require 'socks)
 (require 'url-http)
 
@@ -329,4 +329,57 @@ tor-resolve-5
     (kill-buffer (process-buffer server))
     (delete-process server)))
 
+(defvar test-socks-service ; "127.0.0.1:1080" -> ("127.0.0.1", 1080)
+  (when-let ((present (getenv "TEST_SOCKS_SERVICE"))
+             (parts (split-string present ":")))
+    (list (car parts) (string-to-number (cadr parts)))))
+
+(declare-function gnutls-negotiate "gnutls"
+                  (&rest spec
+                         &key process type hostname priority-string
+                         trustfiles crlfiles keylist min-prime-bits
+                         verify-flags verify-error verify-hostname-error
+                         &allow-other-keys))
+
+(ert-deftest test-socks-https-poc ()
+  :tags '(:unstable)
+  (unless test-socks-service (ert-skip "SOCKS service missing"))
+  (unless (gnutls-available-p) (ert-skip "SOCKS resolve test needs GNUTLS"))
+  (ert-with-temp-file tempfile
+    :prefix "emacs-test-socks-network-security-"
+    (let* ((socks-server `("tor" ,@test-socks-service 5))
+           (socks-password "")
+           (nsm-settings-file tempfile)
+           (url-gateway-method 'socks)
+           (id "sha1:df77269389e537fcc9a5fe61667133b5bb97d42e")
+           (host "check.torproject.org")
+           (url (url-generic-parse-url "https://check.torproject.org"))
+           ;;
+           done
+           ;;
+           (cb (lambda (&rest _r)
+                 (goto-char (point-min))
+                 (should (search-forward "Congratulations" nil t))
+                 (setq done t)))
+           (orig (symbol-function #'socks--open-network-stream)))
+      (cl-letf (((symbol-function 'socks--open-network-stream)
+                 (lambda (&rest rest)
+                   (let ((proc (apply orig rest)))
+                     (gnutls-negotiate :process proc :hostname host)
+                     (should (nsm-verify-connection proc host 443 t))))))
+        (ert-info ("Connect to HTTPS endpoint over Tor SOCKS proxy")
+          (unwind-protect
+              (progn
+                (advice-add 'network-lookup-address-info :override
+                            #'socks-tor-resolve)
+                (should-not (nsm-host-settings id))
+                (url-http url cb '(nil))
+                (ert-info ("Wait for response")
+                  (with-timeout (3 (error "Request timed out"))
+                    (unless done
+                      (sleep-for 0.1))))
+                (should (nsm-host-settings id)))
+            (advice-remove 'network-lookup-address-info
+                           #'socks-tor-resolve)))))))
+
 ;;; socks-tests.el ends here
-- 
2.35.1


--=-=-=--

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#53941: 27.2; socks + tor dont work with https
Resent-From: "J.P." <jp@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Wed, 02 Mar 2022 02:38:02 +0000
Resent-Message-ID: <handler.53941.B53941.16461886575602 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 53941
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: patch
To: 53941 <at> debbugs.gnu.org
Cc: Jacobo <gnuhacker@HIDDEN>
Received: via spool by 53941-submit <at> debbugs.gnu.org id=B53941.16461886575602
          (code B ref 53941); Wed, 02 Mar 2022 02:38:02 +0000
Received: (at 53941) by debbugs.gnu.org; 2 Mar 2022 02:37:37 +0000
Received: from localhost ([127.0.0.1]:38720 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1nPErt-0001SC-Pe
	for submit <at> debbugs.gnu.org; Tue, 01 Mar 2022 21:37:37 -0500
Received: from mail-108-mta130.mxroute.com ([136.175.108.130]:35873)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <jp@HIDDEN>) id 1nPErr-0001Rx-HI
 for 53941 <at> debbugs.gnu.org; Tue, 01 Mar 2022 21:37:32 -0500
Received: from filter006.mxroute.com ([140.82.40.27] 140.82.40.27.vultr.com)
 (Authenticated sender: mN4UYu2MZsgR)
 by mail-108-mta130.mxroute.com (ZoneMTA) with ESMTPSA id
 17f487d3c7d0005a20.001 for <53941 <at> debbugs.gnu.org>
 (version=TLSv1/SSLv3 cipher=ECDHE-RSA-AES128-GCM-SHA256);
 Wed, 02 Mar 2022 02:37:21 +0000
X-Zone-Loop: f59c6fe4143e200293f5a4d0b1fa812f35b3a50cf18c
X-Originating-IP: [140.82.40.27]
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=neverwas.me
 ; s=x;
 h=Content-Type:MIME-Version:Message-ID:In-Reply-To:Date:References:
 Subject:Cc:To:From:Sender:Reply-To:Content-Transfer-Encoding:Content-ID:
 Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc
 :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe:
 List-Post:List-Owner:List-Archive;
 bh=Z5Z6pLBgpsbZfN4oq1RpXEKLQC+TYkRMfKTvniQYR4s=; b=nJqFmZbBm5htBMzLCsoqPiTNX5
 7vjCP1peDxKOskhXLWmdq32gDGNGXVozJC+riaXJ7tjtl4ndVNjxMDLddzBd2aFA5OD9E82dSTvOY
 YWRypC5OwOsC8n9PHMEpQ6BFKGKQ490SwrDs1kVRDK3JU9vghSPle5T80Rg51n6qBmehUaSFOKYLy
 cLhQwgPpJnqrcs7kUSLp6nXej1TCG12y7kzGl/piYwAYa1O8M0DheP947O9xOF9Yx/sp2aKIv5DZg
 ZBb7iQ0DcqEV8SWVQTnjlxkT+4xBgAXdeW6aCfu7vQhmgrgMxpvW/YgItdtZU2K3XSKtsZ/VxGNP4
 5GxAixRA==;
From: "J.P." <jp@HIDDEN>
References: <87pmntfym7.fsf@HIDDEN> <8735kl1v58.fsf@HIDDEN>
 <87a6emftzx.fsf@HIDDEN> <87k0do5km1.fsf@HIDDEN>
 <87pmn5n3tu.fsf@HIDDEN>
Date: Tue, 01 Mar 2022 18:37:16 -0800
In-Reply-To: <87pmn5n3tu.fsf@HIDDEN> (J. P.'s message of "Tue, 01 Mar
 2022 06:29:49 -0800")
Message-ID: <87mti99j1f.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/29.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="=-=-="
X-AuthUser: masked@HIDDEN
X-Spam-Score: -0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

--=-=-=
Content-Type: text/plain

v3. Passing around an opener function was clunky, so I've opted for
passing around contact params instead. I've also gone back to explicitly
setting the coding to binary because folks may not be using
`url-open-stream' (which does this indirectly by let-binding
`coding-system-for-{read,write}').

--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment; filename=0000-NOT-A-PATCH-v2-v3.diff

From 45be9bbb941e91efe9dacf1b3c34d4d362593d53 Mon Sep 17 00:00:00 2001
From: "F. Jason Park" <jp@HIDDEN>
Date: Tue, 1 Mar 2022 14:45:26 -0800
Subject: [PATCH 0/5] NOT A PATCH

*** BLURB HERE ***

F. Jason Park (5):
  Simplify network-stream opener in socks.el
  Fix string encoding bug in socks tests
  Add support for SOCKS 4a
  Support SOCKS resolve extension
  [POC] Demo SOCKS resolve with HTTPS

 lisp/net/socks.el            | 133 ++++++++++++++++++++++++++++-------
 test/lisp/net/socks-tests.el | 113 +++++++++++++++++++++++++++--
 2 files changed, 213 insertions(+), 33 deletions(-)

Interdiff:
diff --git a/lisp/net/socks.el b/lisp/net/socks.el
index cd026fd163..02edd95328 100644
--- a/lisp/net/socks.el
+++ b/lisp/net/socks.el
@@ -334,18 +334,22 @@ socks-filter
 (defvar socks-override-functions nil
   "If non-nil, overwrite `open-network-stream' function with SOCKSified version.")
 (make-obsolete-variable 'socks-override-functions
-                        "`socks--open-network-stream' now takes a process arg."
+                        "use custom opener with `socks-open-stream-function'."
                         "29.1")
 
-(defun socks-open-connection (server-info &optional opener)
+(defvar socks-open-stream-function #'open-network-stream
+  "Function called to open a network stream connection.")
+
+(defun socks-open-connection (server-info &rest params)
   "Create and initialize a SOCKS process.
 Perform authentication if needed.  SERVER-INFO should resemble
-`socks-server'.  OPENER, when present, should be a substitute for
-`open-network-stream' and take the same arguments."
+`socks-server'.  PARAMS are those accepted by `make-network-process'."
   (interactive)
+  (unless (plist-member params :coding)
+    (setf (plist-get params :coding) '(binary . binary)))
   (save-excursion
-    (let ((proc (funcall (or opener #'open-network-stream)
-                         "socks" nil (nth 1 server-info) (nth 2 server-info)))
+    (let ((proc (apply socks-open-stream-function "socks" nil
+                       (nth 1 server-info) (nth 2 server-info) params))
 	  (authtype nil)
 	  version)
 
@@ -531,11 +535,11 @@ socks-find-services-entry
 
 (defun socks-open-network-stream (name buffer host service &rest params)
   (if-let* ((route (socks-find-route host service))
-            (proc (socks-open-connection route #'open-network-stream)))
+            (proc (apply #'socks-open-connection route params)))
       (socks--open-network-stream proc buffer host service)
     (message "Warning: no SOCKS route found for %s:%s" host service)
     ;; Support legacy behavior (likely undesirable in most cases)
-    (apply #'open-network-stream name buffer host service params)))
+    (apply socks-open-stream-function name buffer host service params)))
 
 (defun socks--open-network-stream (proc buffer host service)
   (progn ; temporarily preserve git blame for easier reviewing
@@ -684,17 +688,20 @@ socks-tor-resolve
   (let ((socks-password (or socks-password ""))
         host
         (port 80)  ; unused for now
+        route
         proc
         ip)
     (unless (string-suffix-p ".onion" name)
       (setq host (if (string-match "\\`[[:ascii:]]+\\'" name)
                      name
                    (require 'puny)
-                   (puny-encode-domain name)))
+                   (puny-encode-domain name))
+            route (socks-find-route host port))
+      (cl-assert route)
       ;; "Host unreachable" may be raised when the lookup fails
       (unwind-protect
           (progn
-            (setq proc (socks-open-connection (socks-find-route host port)))
+            (setq proc (socks-open-connection route))
             (socks-send-command proc
                                 socks-resolve-command
                                 socks-address-type-name
-- 
2.35.1


--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
 filename=0001-Simplify-network-stream-opener-in-socks.el.patch

From 90247189d5fe90619f00ef3319012df0f6f6688e Mon Sep 17 00:00:00 2001
From: "F. Jason Park" <jp@HIDDEN>
Date: Tue, 1 Mar 2022 02:12:02 -0800
Subject: [PATCH 1/5] Simplify network-stream opener in socks.el

* lisp/net/socks.el (socks-override-functions,
socks-open-stream-function): Make first variable obsolete and remove
uses.  Replace somewhat with the second, which holds a network stream
opener that defaults to `open-network-stream'.
(socks-open-connection): Accept additional `make-network-process'
params passed on to opener.
(socks-open-network-stream): Likewise with the additional params.
Call `open-network-stream' as a fallback when a route cannot be found.
(socks--open-network-stream): Reduce role to merely issuing the first
command using an existing process.  This may warrant a renaming.
Change signature accordingly.
---
 lisp/net/socks.el | 50 +++++++++++++++++++++++------------------------
 1 file changed, 25 insertions(+), 25 deletions(-)

diff --git a/lisp/net/socks.el b/lisp/net/socks.el
index 8df0773e1d..5b78eb6e84 100644
--- a/lisp/net/socks.el
+++ b/lisp/net/socks.el
@@ -323,19 +323,23 @@ socks-filter
 
 (defvar socks-override-functions nil
   "If non-nil, overwrite `open-network-stream' function with SOCKSified version.")
+(make-obsolete-variable 'socks-override-functions
+                        "use custom opener with `socks-open-stream-function'."
+                        "29.1")
 
-(when socks-override-functions
-  (advice-add 'open-network-stream :around #'socks--open-network-stream))
+(defvar socks-open-stream-function #'open-network-stream
+  "Function called to open a network stream connection.")
 
-(defun socks-open-connection (server-info)
+(defun socks-open-connection (server-info &rest params)
+  "Create and initialize a SOCKS process.
+Perform authentication if needed.  SERVER-INFO should resemble
+`socks-server'.  PARAMS are those accepted by `make-network-process'."
   (interactive)
+  (unless (plist-member params :coding)
+    (setf (plist-get params :coding) '(binary . binary)))
   (save-excursion
-    (let ((proc
-           (let ((socks-override-functions nil))
-             (open-network-stream "socks"
-				  nil
-				  (nth 1 server-info)
-				  (nth 2 server-info))))
+    (let ((proc (apply socks-open-stream-function "socks" nil
+                       (nth 1 server-info) (nth 2 server-info) params))
 	  (authtype nil)
 	  version)
 
@@ -508,22 +512,18 @@ socks-find-services-entry
   (gethash (downcase service)
 	      (if udp socks-udp-services socks-tcp-services)))
 
-(defun socks-open-network-stream (name buffer host service)
-  (let ((socks-override-functions t))
-    (socks--open-network-stream
-     (lambda (&rest args)
-       (let ((socks-override-functions nil))
-         (apply #'open-network-stream args)))
-     name buffer host service)))
-
-(defun socks--open-network-stream (orig-fun name buffer host service &rest params)
-  (let ((route (and socks-override-functions
-                    (socks-find-route host service))))
-    (if (not route)
-	(apply orig-fun name buffer host service params)
-      ;; FIXME: Obey `params'!
-      (let* ((proc (socks-open-connection route))
-	     (version (process-get proc 'socks-server-protocol))
+(defun socks-open-network-stream (name buffer host service &rest params)
+  (if-let* ((route (socks-find-route host service))
+            (proc (apply #'socks-open-connection route params)))
+      (socks--open-network-stream proc buffer host service)
+    (message "Warning: no SOCKS route found for %s:%s" host service)
+    ;; Support legacy behavior (likely undesirable in most cases)
+    (apply socks-open-stream-function name buffer host service params)))
+
+(defun socks--open-network-stream (proc buffer host service)
+  (progn ; temporarily preserve git blame for easier reviewing
+    (progn ; could rename to something like `socks--initiate-command-sequence'
+      (let* ((version (process-get proc 'socks-server-protocol))
              (atype
               (cond
                ((equal version 4)
-- 
2.35.1


--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
 filename=0002-Fix-string-encoding-bug-in-socks-tests.patch

From 181548ce7f931fedd66e243632c42c5c51af640e Mon Sep 17 00:00:00 2001
From: "F. Jason Park" <jp@HIDDEN>
Date: Mon, 14 Feb 2022 02:36:57 -0800
Subject: [PATCH 2/5] Fix string encoding bug in socks tests

* test/lisp/net/socks-tests.el (socks-tests-canned-server-create,
socks-tests-filter-response-parsing-v4): Fix bug in process filter to
prevent prepared outgoing responses from being implicitly encoded as
utf-8.  Fix similar mistake in v4 filter test.
---
 test/lisp/net/socks-tests.el | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/test/lisp/net/socks-tests.el b/test/lisp/net/socks-tests.el
index 461796bdf9..d9ef53ae35 100644
--- a/test/lisp/net/socks-tests.el
+++ b/test/lisp/net/socks-tests.el
@@ -63,21 +63,21 @@ socks-tests-filter-response-parsing-v4
     (process-put proc 'socks-state socks-state-waiting)
     (process-put proc 'socks-server-protocol 4)
     (ert-info ("Receive initial incomplete segment")
-      (socks-filter proc (concat [0 90 0 0 93 184 216]))
-      ;; From example.com: OK status ^      ^ msg start
+      (socks-filter proc (unibyte-string 0 90 0 0 93 184 216))
+      ;; From example.com: OK status       ^      ^ msg start
       (ert-info ("State still set to waiting")
         (should (eq (process-get proc 'socks-state) socks-state-waiting)))
       (ert-info ("Response field is nil because processing incomplete")
         (should-not (process-get proc 'socks-response)))
       (ert-info ("Scratch field holds stashed partial payload")
-        (should (string= (concat [0 90 0 0 93 184 216])
+        (should (string= (unibyte-string 0 90 0 0 93 184 216)
                          (process-get proc 'socks-scratch)))))
     (ert-info ("Last part arrives")
       (socks-filter proc "\42") ; ?\" 34
       (ert-info ("State transitions to complete (length check passes)")
         (should (eq (process-get proc 'socks-state) socks-state-connected)))
       (ert-info ("Scratch and response fields hold stash w. last chunk")
-        (should (string= (concat [0 90 0 0 93 184 216 34])
+        (should (string= (unibyte-string 0 90 0 0 93 184 216 34)
                          (process-get proc 'socks-response)))
         (should (string= (process-get proc 'socks-response)
                          (process-get proc 'socks-scratch)))))
@@ -140,10 +140,11 @@ socks-tests-canned-server-create
                    (unless (or (and (vectorp pat) (equal pat (vconcat line)))
                                (string-match-p pat line))
                      (error "Unknown request: %s" line))
+                   (setq resp (apply #'unibyte-string (append resp nil)))
                    (let ((print-escape-control-characters t))
                      (message "[%s] <- %s" name (prin1-to-string line))
                      (message "[%s] -> %s" name (prin1-to-string resp)))
-                   (process-send-string proc (concat resp)))))
+                   (process-send-string proc resp))))
          (serv (make-network-process :server 1
                                      :buffer (get-buffer-create name)
                                      :filter filt
-- 
2.35.1


--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
 filename=0003-Add-support-for-SOCKS-4a.patch

From db601f1fcbaf5cf088b280966cbac2808a773ee0 Mon Sep 17 00:00:00 2001
From: "F. Jason Park" <jp@HIDDEN>
Date: Mon, 14 Feb 2022 02:36:57 -0800
Subject: [PATCH 3/5] Add support for SOCKS 4a

* lisp/net/socks.el (socks-server): Add new choice `4a' to version
field of option.  This may appear to change the type of the field from
a number to a union of symbols and numbers.  However,
`socks-send-command' and `socks-filter' already expect a possible
`http' value for this field (also a symbol).
(socks--errors-4): Add new constant containing error messages for
socks version 4.  The semantics are faithful, but the wording is
ad-libbed.
(socks-send-command): Massage existing handling for version 4 to
accommodate 4a.

* test/lisp/net/socks-tests.el (socks-tests-v4a-basic): add test for
4a.
Bug#53941
---
 lisp/net/socks.el            | 22 ++++++++++++++++++++--
 test/lisp/net/socks-tests.el | 13 +++++++++++++
 2 files changed, 33 insertions(+), 2 deletions(-)

diff --git a/lisp/net/socks.el b/lisp/net/socks.el
index 5b78eb6e84..a2198d898a 100644
--- a/lisp/net/socks.el
+++ b/lisp/net/socks.el
@@ -162,6 +162,7 @@ socks-server
 	  (radio-button-choice :tag "SOCKS Version"
 			       :format "%t: %v"
 			       (const :tag "SOCKS v4  " :format "%t" :value 4)
+                               (const :tag "SOCKS v4a"  :format "%t" :value 4a)
 			       (const :tag "SOCKS v5"   :format "%t" :value 5))))
 
 
@@ -202,6 +203,12 @@ socks-errors
     "Command not supported"
     "Address type not supported"))
 
+(defconst socks--errors-4
+  '("Granted"
+    "Rejected or failed"
+    "Cannot connect to identd on the client"
+    "Client and identd report differing user IDs"))
+
 ;; The socks v5 address types
 (defconst socks-address-type-v4   1)
 (defconst socks-address-type-name 3)
@@ -404,6 +411,7 @@ socks-send-command
 		(format "%c%s" (length address) address))
 	       (t
 		(error "Unknown address type: %d" atype))))
+        trailing
 	request version)
     (or (process-get proc 'socks)
         (error "socks-send-command called on non-SOCKS connection %S" proc))
@@ -421,6 +429,12 @@ socks-send-command
 			     (t
 			      (error "Unsupported address type for HTTP: %d" atype)))
 			    port)))
+     ((when (eq version '4a)
+        (setf addr "\0\0\0\1"
+              trailing (concat address "\0")
+              version 4 ; done with the "a" part
+              (process-get proc 'socks-server-protocol) 4)
+        nil)) ; fall through
      ((equal version 4)
       (setq request (concat
 		     (unibyte-string
@@ -430,7 +444,8 @@ socks-send-command
 		      (logand port #xff)) ; port, low byte
 		     addr                 ; address
 		     (user-full-name)     ; username
-		     "\0")))              ; terminate username
+                     "\0"                 ; terminate username
+                     trailing)))          ; optional host to look up
      ((equal version 5)
       (setq request (concat
 		     (unibyte-string
@@ -451,7 +466,10 @@ socks-send-command
 	nil				; Sweet sweet success!
       (delete-process proc)
       (error "SOCKS: %s"
-             (nth (or (process-get proc 'socks-reply) 1) socks-errors)))
+             (let ((no (or (process-get proc 'socks-reply) 1)))
+               (if (eq version 5)
+                   (nth no socks-errors)
+                 (nth (+ 90 no) socks--errors-4)))))
     proc))
 
 
diff --git a/test/lisp/net/socks-tests.el b/test/lisp/net/socks-tests.el
index d9ef53ae35..4e990ffdba 100644
--- a/test/lisp/net/socks-tests.el
+++ b/test/lisp/net/socks-tests.el
@@ -207,6 +207,19 @@ socks-tests-v4-basic
                  (lambda (&optional _) "foo")))
         (socks-tests-perform-hello-world-http-request)))))
 
+(ert-deftest socks-tests-v4a-basic ()
+  "Show correct preparation of SOCKS4a connect command."
+  (let ((socks-server '("server" "127.0.0.1" 10083 4a))
+        (url-user-agent "Test/4a-basic")
+        (socks-tests-canned-server-patterns
+         `(([4 1 0 80 0 0 0 1 ?f ?o ?o 0 ?e ?x ?a ?m ?p ?l ?e ?. ?c ?o ?m 0]
+            . [0 90 0 0 0 0 0 0])
+           ,socks-tests--hello-world-http-request-pattern)))
+    (ert-info ("Make HTTP request over SOCKS4A")
+      (cl-letf (((symbol-function 'user-full-name)
+                 (lambda (&optional _) "foo")))
+        (socks-tests-perform-hello-world-http-request)))))
+
 ;; Replace first pattern below with ([5 3 0 1 2] . [5 2]) to validate
 ;; against curl 7.71 with the following options:
 ;; $ curl --verbose -U foo:bar --proxy socks5h://127.0.0.1:10080 example.com
-- 
2.35.1


--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
 filename=0004-Support-SOCKS-resolve-extension.patch

From 67ba3f6e6fcb12b99757fcc49f86f951ad59c02b Mon Sep 17 00:00:00 2001
From: "F. Jason Park" <jp@HIDDEN>
Date: Mon, 14 Feb 2022 02:36:57 -0800
Subject: [PATCH 4/5] Support SOCKS resolve extension

* lisp/net/socks.el (socks-resolve-command): Add new constant for the
SOCKS command RESOLVE, which comes by way of a nonstandard extension
from the TOR project.  It mirrors CONNECT in most respects but asks
the server to RESOLVE a host name and return its IP.  For details, see
https://github.com/torproject/torspec/blob/master/socks-extensions.txt
This shouldn't be confused with 5h/5-hostname, which is used to by
clients like cURL to allow users to bypass attempts to resolve a name
locally.
(socks--extract-resolve-response, socks-tor-resolve): Add utility
functions to query a SOCKS service supporting the RESOLVE extension.
Bug#53941
---
 lisp/net/socks.el            | 61 ++++++++++++++++++++++++++++++++++++
 test/lisp/net/socks-tests.el | 34 ++++++++++++++++++++
 2 files changed, 95 insertions(+)

diff --git a/lisp/net/socks.el b/lisp/net/socks.el
index a2198d898a..02edd95328 100644
--- a/lisp/net/socks.el
+++ b/lisp/net/socks.el
@@ -181,6 +181,9 @@ socks-udp-associate-command
 (defconst socks-authentication-null 0)
 (defconst socks-authentication-failure 255)
 
+;; Extensions
+(defconst socks-resolve-command #xf0)
+
 ;; Response codes
 (defconst socks-response-success               0)
 (defconst socks-response-general-failure       1)
@@ -653,6 +656,64 @@ socks-nslookup-host
 	res)
     host))
 
+(defun socks--extract-resolve-response (proc)
+  "Parse response for PROC and maybe return destination IP address."
+  (let ((response (process-get proc 'socks-response)))
+    (cl-assert response) ; otherwise, msg not received in its entirety
+    (pcase (process-get proc 'socks-server-protocol)
+      (4 ; https://www.openssh.com/txt/socks4a.protocol
+       (when-let (((zerop (process-get proc 'socks-reply)))
+                  ((eq (aref response 1) 90)) ; #x5a request granted
+                  (a (substring response 4)) ; ignore port for now
+                  ((not (string-empty-p a)))
+                  ((not (string= a "\0\0\0\0"))))
+         a))
+      (5 ; https://tools.ietf.org/html/rfc1928
+       (cl-assert (eq 5 (aref response 0)) t)
+       (pcase (aref response 3) ; ATYP
+         (1 (and-let* ((a (substring response 4 8))
+                       ((not (string= a "\0\0\0\0")))
+                       a)))
+         ;; No reason to support RESOLVE_PTR [F1] extension, right?
+         (3 (let ((len (1- (aref response 4))))
+              (substring response 5 (+ 5 len))))
+         (4 (substring response 4 20)))))))
+
+(declare-function puny-encode-domain "puny" (domain))
+
+(defun socks-tor-resolve (name &optional _family)
+  "Return list of one vector IPv4 address for domain NAME.
+Or return nil on failure.  See `network-lookup-address-info' for format
+of return value.  Server must support the Tor RESOLVE command."
+  (let ((socks-password (or socks-password ""))
+        host
+        (port 80)  ; unused for now
+        route
+        proc
+        ip)
+    (unless (string-suffix-p ".onion" name)
+      (setq host (if (string-match "\\`[[:ascii:]]+\\'" name)
+                     name
+                   (require 'puny)
+                   (puny-encode-domain name))
+            route (socks-find-route host port))
+      (cl-assert route)
+      ;; "Host unreachable" may be raised when the lookup fails
+      (unwind-protect
+          (progn
+            (setq proc (socks-open-connection route))
+            (socks-send-command proc
+                                socks-resolve-command
+                                socks-address-type-name
+                                host
+                                port)
+            (cl-assert (eq (process-get proc 'socks-state)
+                           socks-state-connected))
+            (setq ip (socks--extract-resolve-response proc)))
+        (when proc
+          (delete-process proc)))
+      (list (vconcat ip [0])))))
+
 (provide 'socks)
 
 ;;; socks.el ends here
diff --git a/test/lisp/net/socks-tests.el b/test/lisp/net/socks-tests.el
index 4e990ffdba..3d1aca9af4 100644
--- a/test/lisp/net/socks-tests.el
+++ b/test/lisp/net/socks-tests.el
@@ -295,4 +295,38 @@ socks-tests-v5-auth-none
       (socks-tests-perform-hello-world-http-request)))
   (should (assq 2 socks-authentication-methods)))
 
+(ert-deftest tor-resolve-4a ()
+  "Make request to TOR resolve service over SOCKS4a"
+  (let* ((socks-server '("server" "127.0.0.1" 19050 4a))
+         (socks-tests-canned-server-patterns
+          '(([4 #xf0 0 80 0 0 0 1 ?f ?o ?o 0 ?e ?x ?a ?m ?p ?l ?e ?. ?c ?o ?m 0]
+             . [0 90 0 0 93 184 216 34])))
+         (inhibit-message noninteractive)
+         (server (socks-tests-canned-server-create)))
+    (ert-info ("Query TOR RESOLVE service over SOCKS4")
+      (cl-letf (((symbol-function 'user-full-name)
+                 (lambda (&optional _) "foo")))
+        (should (equal '([93 184 216 34 0])
+                       (socks-tor-resolve "example.com")))))
+    (kill-buffer (process-buffer server))
+    (delete-process server)))
+
+(ert-deftest tor-resolve-5 ()
+  "Make request to TOR resolve service over SOCKS5"
+  (let* ((socks-server '("server" "127.0.0.1" 19051 5))
+         (socks-username "foo")
+         (socks-authentication-methods (append socks-authentication-methods
+                                               nil))
+         (inhibit-message noninteractive)
+         (socks-tests-canned-server-patterns
+          '(([5 2 0 2] . [5 2])
+            ([1 3 ?f ?o ?o 0] . [1 0])
+            ([5 #xf0 0 3 11 ?e ?x ?a ?m ?p ?l ?e ?. ?c ?o ?m 0 80]
+             . [5 0 0 1 93 184 216 34 0 0])))
+         (server (socks-tests-canned-server-create)))
+    (ert-info ("Query TOR RESOLVE service over SOCKS5")
+      (should (equal '([93 184 216 34 0]) (socks-tor-resolve "example.com"))))
+    (kill-buffer (process-buffer server))
+    (delete-process server)))
+
 ;;; socks-tests.el ends here
-- 
2.35.1


--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
 filename=0005-POC-Demo-SOCKS-resolve-with-HTTPS.patch

From 45be9bbb941e91efe9dacf1b3c34d4d362593d53 Mon Sep 17 00:00:00 2001
From: "F. Jason Park" <jp@HIDDEN>
Date: Mon, 14 Feb 2022 02:36:57 -0800
Subject: [PATCH 5/5] [POC] Demo SOCKS resolve with HTTPS

* test/lisp/net/socks-test.el (test-socks-https-poc): Provide
throwaway test demoing an HTTPS connection over a TOR proxy service.
---
 test/lisp/net/socks-tests.el | 55 +++++++++++++++++++++++++++++++++++-
 1 file changed, 54 insertions(+), 1 deletion(-)

diff --git a/test/lisp/net/socks-tests.el b/test/lisp/net/socks-tests.el
index 3d1aca9af4..f2600210b0 100644
--- a/test/lisp/net/socks-tests.el
+++ b/test/lisp/net/socks-tests.el
@@ -21,7 +21,7 @@
 
 ;;; Code:
 
-(require 'ert)
+(require 'ert-x)
 (require 'socks)
 (require 'url-http)
 
@@ -329,4 +329,57 @@ tor-resolve-5
     (kill-buffer (process-buffer server))
     (delete-process server)))
 
+(defvar test-socks-service ; "127.0.0.1:1080" -> ("127.0.0.1", 1080)
+  (when-let ((present (getenv "TEST_SOCKS_SERVICE"))
+             (parts (split-string present ":")))
+    (list (car parts) (string-to-number (cadr parts)))))
+
+(declare-function gnutls-negotiate "gnutls"
+                  (&rest spec
+                         &key process type hostname priority-string
+                         trustfiles crlfiles keylist min-prime-bits
+                         verify-flags verify-error verify-hostname-error
+                         &allow-other-keys))
+
+(ert-deftest test-socks-https-poc ()
+  :tags '(:unstable)
+  (unless test-socks-service (ert-skip "SOCKS service missing"))
+  (unless (gnutls-available-p) (ert-skip "SOCKS resolve test needs GNUTLS"))
+  (ert-with-temp-file tempfile
+    :prefix "emacs-test-socks-network-security-"
+    (let* ((socks-server `("tor" ,@test-socks-service 5))
+           (socks-password "")
+           (nsm-settings-file tempfile)
+           (url-gateway-method 'socks)
+           (id "sha1:df77269389e537fcc9a5fe61667133b5bb97d42e")
+           (host "check.torproject.org")
+           (url (url-generic-parse-url "https://check.torproject.org"))
+           ;;
+           done
+           ;;
+           (cb (lambda (&rest _r)
+                 (goto-char (point-min))
+                 (should (search-forward "Congratulations" nil t))
+                 (setq done t)))
+           (orig (symbol-function #'socks--open-network-stream)))
+      (cl-letf (((symbol-function 'socks--open-network-stream)
+                 (lambda (&rest rest)
+                   (let ((proc (apply orig rest)))
+                     (gnutls-negotiate :process proc :hostname host)
+                     (should (nsm-verify-connection proc host 443 t))))))
+        (ert-info ("Connect to HTTPS endpoint over Tor SOCKS proxy")
+          (unwind-protect
+              (progn
+                (advice-add 'network-lookup-address-info :override
+                            #'socks-tor-resolve)
+                (should-not (nsm-host-settings id))
+                (url-http url cb '(nil))
+                (ert-info ("Wait for response")
+                  (with-timeout (3 (error "Request timed out"))
+                    (unless done
+                      (sleep-for 0.1))))
+                (should (nsm-host-settings id)))
+            (advice-remove 'network-lookup-address-info
+                           #'socks-tor-resolve)))))))
+
 ;;; socks-tests.el ends here
-- 
2.35.1


--=-=-=--

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#53941: 27.2; socks + tor dont work with https
Resent-From: Jacobo <gnuhacker@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sun, 06 Mar 2022 02:41:02 +0000
Resent-Message-ID: <handler.53941.B53941.164653444912888 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 53941
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: patch
To: "J.P." <jp@HIDDEN>
Cc: 53941 <at> debbugs.gnu.org
Received: via spool by 53941-submit <at> debbugs.gnu.org id=B53941.164653444912888
          (code B ref 53941); Sun, 06 Mar 2022 02:41:02 +0000
Received: (at 53941) by debbugs.gnu.org; 6 Mar 2022 02:40:49 +0000
Received: from localhost ([127.0.0.1]:49570 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1nQgpF-0003Lo-B0
	for submit <at> debbugs.gnu.org; Sat, 05 Mar 2022 21:40:49 -0500
Received: from mta763.solicitae.com ([89.35.150.78]:56120
 helo=smtp.gnuhacker.org) by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <gnuhacker@HIDDEN>) id 1nQgpD-0003Lf-MI
 for 53941 <at> debbugs.gnu.org; Sat, 05 Mar 2022 21:40:48 -0500
Received: from hackerlab.gnu.org (localhost [127.0.0.1])
 by smtp.gnuhacker.org (Postfix) with ESMTP id C088C6008CF;
 Sun,  6 Mar 2022 03:40:45 +0100 (CET)
From: Jacobo <gnuhacker@HIDDEN>
References: <87pmntfym7.fsf@HIDDEN> <8735kl1v58.fsf@HIDDEN>
 <87a6emftzx.fsf@HIDDEN> <87k0do5km1.fsf@HIDDEN>
 <87pmn5n3tu.fsf@HIDDEN> <87mti99j1f.fsf@HIDDEN>
Date: Sun, 06 Mar 2022 03:40:45 +0100
In-Reply-To: <87mti99j1f.fsf@HIDDEN> (J. P.'s message of "Tue, 01 Mar
 2022 18:37:16 -0800")
Message-ID: <87wnh7hkgi.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

"J.P." <jp@HIDDEN> writes:

> v3. Passing around an opener function was clunky, so I've opted for
> passing around contact params instead. I've also gone back to explicitly
> setting the coding to binary because folks may not be using
> `url-open-stream' (which does this indirectly by let-binding
> `coding-system-for-{read,write}').

Emacs 28.0.91 compiled with this patches, dont work, connections dont
use the proxy

-- 
Emacs Lover.
FSF Member.
Free/Libre Software supporter.
stallmansupport.org - Disinformation succeeds because so many people
care deeply about injustice but do not take the time to check the facts.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#53941: 27.2; socks + tor dont work with https
Resent-From: "J.P." <jp@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sun, 06 Mar 2022 03:00:01 +0000
Resent-Message-ID: <handler.53941.B53941.164653554923051 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 53941
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: patch
To: Jacobo <gnuhacker@HIDDEN>
Cc: 53941 <at> debbugs.gnu.org
Received: via spool by 53941-submit <at> debbugs.gnu.org id=B53941.164653554923051
          (code B ref 53941); Sun, 06 Mar 2022 03:00:01 +0000
Received: (at 53941) by debbugs.gnu.org; 6 Mar 2022 02:59:09 +0000
Received: from localhost ([127.0.0.1]:49575 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1nQh6y-0005zj-TR
	for submit <at> debbugs.gnu.org; Sat, 05 Mar 2022 21:59:09 -0500
Received: from mail-108-mta94.mxroute.com ([136.175.108.94]:36637)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <jp@HIDDEN>) id 1nQh6w-0005zH-LJ
 for 53941 <at> debbugs.gnu.org; Sat, 05 Mar 2022 21:59:07 -0500
Received: from filter006.mxroute.com ([140.82.40.27] 140.82.40.27.vultr.com)
 (Authenticated sender: mN4UYu2MZsgR)
 by mail-108-mta94.mxroute.com (ZoneMTA) with ESMTPSA id 17f5d2a777f000763e.001
 for <53941 <at> debbugs.gnu.org>
 (version=TLSv1/SSLv3 cipher=ECDHE-RSA-AES128-GCM-SHA256);
 Sun, 06 Mar 2022 02:58:58 +0000
X-Zone-Loop: dd5c5b6a6c2440c4e8621c03dfa438652ae0aa8f21ff
X-Originating-IP: [140.82.40.27]
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=neverwas.me
 ; s=x;
 h=Content-Type:MIME-Version:Message-ID:In-Reply-To:Date:References:
 Subject:Cc:To:From:Sender:Reply-To:Content-Transfer-Encoding:Content-ID:
 Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc
 :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe:
 List-Post:List-Owner:List-Archive;
 bh=XWAJppG7Vb/bbdAKFgpP0XUEsTdHDsx85g2O3i9uLsE=; b=G9eogZNwDSlQa+fKG/EJ25Dxa8
 8Z/eKX2eI8/0E1QoCl34tklDCHkQYaQxBOF0CA1WTe2g0B1fjdOxdqbbwZEIc11i70rz2BT4ojUYr
 vJlL10s0/5IhwU5WQ84aKZmPKdxmJ2p28OavuA0Z0TjS+9d9KR2/THsRQQnaUYtPhIwSP96BWC6vs
 SlwfD4ATJ68+tPya7tLxhGv6HLjt/jZTBzoarNQsrYtZbvW31J6M+Hbu/CTir+CZqNa0sn4fsRFJz
 RVqcaj1wKs2qm76lENP5D/bzwlItsOoXOJ9wyDB4oD8CV28r94deLiUDspcBcR0pMC8TRr5Vnoqc3
 tqv598lg==;
From: "J.P." <jp@HIDDEN>
References: <87pmntfym7.fsf@HIDDEN> <8735kl1v58.fsf@HIDDEN>
 <87a6emftzx.fsf@HIDDEN> <87k0do5km1.fsf@HIDDEN>
 <87pmn5n3tu.fsf@HIDDEN> <87mti99j1f.fsf@HIDDEN>
 <87wnh7hkgi.fsf@HIDDEN>
Date: Sat, 05 Mar 2022 18:58:55 -0800
In-Reply-To: <87wnh7hkgi.fsf@HIDDEN> (Jacobo's message of "Sun, 06 Mar 2022
 03:40:45 +0100")
Message-ID: <87pmmz947k.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/29.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-AuthUser: masked@HIDDEN
X-Spam-Score: -0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Jacobo <gnuhacker@HIDDEN> writes:

> "J.P." <jp@HIDDEN> writes:
>
>> v3. Passing around an opener function was clunky, so I've opted for
>> passing around contact params instead. I've also gone back to explicitly
>> setting the coding to binary because folks may not be using
>> `url-open-stream' (which does this indirectly by let-binding
>> `coding-system-for-{read,write}').
>
> Emacs 28.0.91 compiled with this patches, dont work, connections dont
> use the proxy

As I tried to explain up thread, the patches only get you half way
there. But perhaps that wasn't clear. You still need to do something
like the following, which is what I MemoServ'd you about (but I guess
you didn't get it). Quoting from there:

  ;; This works with eww. Try https://check.torproject.org

  (require 'socks)
  (require 'gnutls)
  (require 'nsm)

  (setq socks-server '("tor" "127.0.0.1" 9050 5)
        socks-username "user"
        socks-password ""
        url-gateway-method 'socks)

  (defun my-socks-open-https (orig name buffer host service &rest params)
    (let ((proc (apply orig name buffer host service params)))
      (advice-add 'network-lookup-address-info :override #'socks-tor-resolve)
      (unwind-protect
          (when (eq service 443)
            (gnutls-negotiate :process proc :hostname host)
            (unless (string-suffix-p ".onion" host)
              (nsm-verify-connection proc host service)))
        (advice-remove 'network-lookup-address-info #'socks-tor-resolve))
      proc))

  (defun my-url-open-stream (args)
    (pcase-let ((`(,name ,buffer ,host ,service ,gateway-method) args))
      (when (and (eq url-gateway-method 'socks)
                 (eq gateway-method 'tls))
        (setq gateway-method nil))
      (list name buffer host service gateway-method)))

  (advice-add 'socks-open-network-stream :around #'my-socks-open-https)
  (advice-add 'url-open-stream :filter-args #'my-url-open-stream)


The above is an example of what I was getting at in my initial reply
about mimicking the recipe in the last patch (the ERT test). If you have
questions about how to use it, I can help you in real time on Libera, as
we did with applying the patches. Also, please try this with emacs -Q.
Thanks.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#53941: 27.2; socks + tor dont work with https
Resent-From: "J.P." <jp@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Mon, 07 Mar 2022 07:11:02 +0000
Resent-Message-ID: <handler.53941.B53941.164663700327982 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 53941
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: patch
To: Jacobo <gnuhacker@HIDDEN>
Cc: 53941 <at> debbugs.gnu.org
Received: via spool by 53941-submit <at> debbugs.gnu.org id=B53941.164663700327982
          (code B ref 53941); Mon, 07 Mar 2022 07:11:02 +0000
Received: (at 53941) by debbugs.gnu.org; 7 Mar 2022 07:10:03 +0000
Received: from localhost ([127.0.0.1]:52309 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1nR7VH-0007Gp-Pd
	for submit <at> debbugs.gnu.org; Mon, 07 Mar 2022 02:10:03 -0500
Received: from mail-108-mta39.mxroute.com ([136.175.108.39]:36187)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <jp@HIDDEN>) id 1nR7VE-0007Gb-J6
 for 53941 <at> debbugs.gnu.org; Mon, 07 Mar 2022 02:09:58 -0500
Received: from filter006.mxroute.com ([140.82.40.27] 140.82.40.27.vultr.com)
 (Authenticated sender: mN4UYu2MZsgR)
 by mail-108-mta39.mxroute.com (ZoneMTA) with ESMTPSA id 17f63367f5e000763e.001
 for <53941 <at> debbugs.gnu.org>
 (version=TLSv1/SSLv3 cipher=ECDHE-RSA-AES128-GCM-SHA256);
 Mon, 07 Mar 2022 07:09:50 +0000
X-Zone-Loop: a370af079bc3f99dec2512fd261ef6f3cf6a4ca08cee
X-Originating-IP: [140.82.40.27]
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=neverwas.me
 ; s=x;
 h=Content-Type:MIME-Version:Message-ID:In-Reply-To:Date:References:
 Subject:Cc:To:From:Sender:Reply-To:Content-Transfer-Encoding:Content-ID:
 Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc
 :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe:
 List-Post:List-Owner:List-Archive;
 bh=tRomcL8oFkO2u8/kq+xXxgM4ohDhRdaBq7krMZKUF5M=; b=IQKhQuZuC6Ex4rX9QvIzWTO8ZK
 /BLUXWQUbrc0hmsKGrON+CJxDmC2dO58bY6N5v5Zjrzxfi7s+FeEDouuHscC8RrxowuH/9IKcIPg9
 Yt5AUvPEAgo3akvNmwqF0dRQsNhEZD1gyznSP7TzOH4klmFcKpUvcTOuNeGjI+sj8LpiAq9GntHcy
 G/ceyCbskKf0avg3uS1Nub6k0dPUjAm6uLLngEzEibk5aizTVIhBsshQvC/k43yxPzg41kaMu+ImB
 Gm5v4LXxbrSTC7DMt07QQMXFosl4aXc8sie/7JDvNbATbMPTxisBOEoSqFh5p+r0Kl8zQeSmHlVjy
 Yk4cYeMA==;
From: "J.P." <jp@HIDDEN>
References: <87pmntfym7.fsf@HIDDEN> <8735kl1v58.fsf@HIDDEN>
 <87a6emftzx.fsf@HIDDEN> <87k0do5km1.fsf@HIDDEN>
 <87pmn5n3tu.fsf@HIDDEN> <87mti99j1f.fsf@HIDDEN>
 <87wnh7hkgi.fsf@HIDDEN> <87pmmz947k.fsf@HIDDEN>
Date: Sun, 06 Mar 2022 23:09:47 -0800
In-Reply-To: <87pmmz947k.fsf@HIDDEN> (J. P.'s message of "Sat, 05 Mar
 2022 18:58:55 -0800")
Message-ID: <8735ju44sk.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/29.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="=-=-="
X-AuthUser: masked@HIDDEN
X-Spam-Score: -0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

--=-=-=
Content-Type: text/plain

v4. Include a minimal (hacky[1]) url-gw integration.

I'm now slightly of the opinion that offering no interface whatsoever is
probably too stark an approach. Ignoring the three Tor-related patches
for now, it seems that even without proper url.el integration[2], we can
still try to ensure that for most use cases, no unnecessary hackery need
apply.

Another issue is whether to address the questionable top-level advising
going on with `open-network-streams', since we're already refactoring
all the functions it affects. Assuming users exist who still have
`socks-override-functions' non-nil at load time, would it make sense to
warn them more fervently than would be done for a normal deprecation?
The thinking is that folks may be relying on this for things like
bypassing firewalls at work (and could therefore get dinged more than
usual just for upgrading Emacs).

As a start, I figured we could try and determine exactly why this
(perhaps somewhat ill-considered) top-level advising was ever instituted
in the first place[3]. AFAICT, it was mainly intended to

1. allow libraries calling `open-network-stream' (and unaware of
   `socks-open-network-stream') to proxy transparently[4]

2. guard the tunneled protocol from being accidentally subject to a
   recommencing of the SOCKS dialog

If anyone has better ideas, please share. Thanks.


Notes
~~~~~

[1] The second patch is new and a bit of an ugly hack. It has to do with
    this change from a while back:

      Do not set `url-gateway-method' in `url-https'
      commit 98c58df832975b01287ef749dd5235199d4cd431
      Sun Sep 28 20:00:54 2014 +0200

    which made it impossible for `url-gateway-method' to be respected by
    `url-open-stream' when called by `url-https'. But rather than
    undoing the offending portions out of hand, it might be nicer to
    first figure out how url-proxy.el is supposed to work and maybe get
    it and `url-retrieve-internal' (and `url-https') more in sync and
    sensitive to `url-gateway-method'.

[2] If we do end up with a proper url.el solution, it might then make
    more sense to emphasize the fact that `socks-open-network-stream' is
    really mostly about catering to url-gw (which it is). If that's
    agreeable, we could rename the following like so:

               socks-open-network-stream -> socks-url-open
        socks-open-network-stream-legacy -> socks-open-network-stream
      socks-open-network-stream-function -> socks-url-open-function

[3] A summary of the advice-based behavior triggered by
    `socks-override-functions', assuming `socks-find-route' returns
    non-nil:

    | topmost function invoked  | o-n-s advised | s-o-f | proxied |
    |---------------------------+---------------+-------+---------|
    | socks-open-network-stream | nil           | t     | yes     |
    | socks-open-network-stream | nil           | nil   | yes     |
    | socks-open-network-stream | t             | t     | yes     |
    | socks-open-network-stream | t             | nil   | yes     |
    | open-network-stream       | nil           | t     | no      |
    | open-network-stream       | nil           | nil   | no      |
    | open-network-stream       | t             | nil   | no      |
    | open-network-stream       | t             | t     | yes     |

    o-n-s: open-network-stream
    s-o-f: socks-override-functions

[4] It could be argued that the 2014 commit in [1] converted gw into one
    such library insofar as `url-https' is concerned.

--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment; filename=0000-NOT-A-PATCH-v3-v4.diff

From 62062472fd14dc9911a105016badcc921d63ae95 Mon Sep 17 00:00:00 2001
From: "F. Jason Park" <jp@HIDDEN>
Date: Sun, 6 Mar 2022 21:21:49 -0800
Subject: [PATCH 0/6] *** NOT A PATCH ***

*** BLURB HERE ***

F. Jason Park (6):
  Simplify network-stream opener in socks.el
  ; * lisp/url/url-gw.el (url-open-stream): Honor socks gateway-method
  Fix string encoding bug in socks tests
  Add support for SOCKS 4a
  Support SOCKS resolve extension
  [POC] Demo SOCKS resolve with HTTPS

 lisp/net/socks.el            | 145 ++++++++++++++++++++++++++++-------
 lisp/url/url-gw.el           |   2 +
 test/lisp/net/socks-tests.el | 113 +++++++++++++++++++++++++--
 3 files changed, 225 insertions(+), 35 deletions(-)

Interdiff:
diff --git a/lisp/net/socks.el b/lisp/net/socks.el
index 02edd95328..9285cbf805 100644
--- a/lisp/net/socks.el
+++ b/lisp/net/socks.el
@@ -334,22 +334,19 @@ socks-filter
 (defvar socks-override-functions nil
   "If non-nil, overwrite `open-network-stream' function with SOCKSified version.")
 (make-obsolete-variable 'socks-override-functions
-                        "use custom opener with `socks-open-stream-function'."
+                        "see `socks-open-network-stream-function'."
                         "29.1")
 
-(defvar socks-open-stream-function #'open-network-stream
-  "Function called to open a network stream connection.")
-
-(defun socks-open-connection (server-info &rest params)
+(defun socks-open-connection (server-info &rest kw-args)
   "Create and initialize a SOCKS process.
 Perform authentication if needed.  SERVER-INFO should resemble
-`socks-server'.  PARAMS are those accepted by `make-network-process'."
+`socks-server'.  KW-ARGS are those accepted by `open-network-stream'."
   (interactive)
-  (unless (plist-member params :coding)
-    (setf (plist-get params :coding) '(binary . binary)))
+  (unless (plist-member kw-args :coding)
+    (setf (plist-get kw-args :coding) '(binary . binary)))
   (save-excursion
-    (let ((proc (apply socks-open-stream-function "socks" nil
-                       (nth 1 server-info) (nth 2 server-info) params))
+    (let ((proc (apply #'open-network-stream "socks" nil
+                       (nth 1 server-info) (nth 2 server-info) kw-args))
 	  (authtype nil)
 	  version)
 
@@ -533,17 +530,31 @@ socks-find-services-entry
   (gethash (downcase service)
 	      (if udp socks-udp-services socks-tcp-services)))
 
-(defun socks-open-network-stream (name buffer host service &rest params)
+(defcustom socks-open-network-stream-function
+  #'socks-open-network-stream-legacy
+  "Function to open a SOCKS connection.
+Called with NAME, BUFFER, HOST, and SERVICE, for compatibility with
+similar functions in the url-gw framework.  May also be passed
+additional keyword args suitable for `make-network-process'."
+  :type '(choice (const :tag "Default fallback-oriented opener.")
+                 (function :tag "User-provided function")))
+
+(defun socks-open-network-stream-legacy (name buffer host service &rest params)
+  "Open a SOCKS connection for a valid route.
+Fall back to non-SOCKS connections for unknown or undesired routes."
   (if-let* ((route (socks-find-route host service))
             (proc (apply #'socks-open-connection route params)))
       (socks--open-network-stream proc buffer host service)
-    (message "Warning: no SOCKS route found for %s:%s" host service)
-    ;; Support legacy behavior (likely undesirable in most cases)
-    (apply socks-open-stream-function name buffer host service params)))
+    ;; Retain legacy behavior and connect anyway without warning
+    (apply #'open-network-stream name buffer host service params)))
+
+(defun socks-open-network-stream (name buffer host service &rest params)
+  "Open a SOCKS connection.  PARAMS are passed to `open-network-stream'."
+  (apply socks-open-network-stream-function name buffer host service params))
 
 (defun socks--open-network-stream (proc buffer host service)
   (progn ; temporarily preserve git blame for easier reviewing
-    (progn ; could rename to something like `socks--initiate-command-sequence'
+    (progn ; could rename to something like `socks--initiate-command-connect'
       (let* ((version (process-get proc 'socks-server-protocol))
              (atype
               (cond
@@ -685,34 +696,31 @@ socks-tor-resolve
   "Return list of one vector IPv4 address for domain NAME.
 Or return nil on failure.  See `network-lookup-address-info' for format
 of return value.  Server must support the Tor RESOLVE command."
-  (let ((socks-password (or socks-password ""))
-        host
-        (port 80)  ; unused for now
-        route
-        proc
-        ip)
-    (unless (string-suffix-p ".onion" name)
-      (setq host (if (string-match "\\`[[:ascii:]]+\\'" name)
-                     name
-                   (require 'puny)
-                   (puny-encode-domain name))
-            route (socks-find-route host port))
-      (cl-assert route)
-      ;; "Host unreachable" may be raised when the lookup fails
-      (unwind-protect
-          (progn
-            (setq proc (socks-open-connection route))
-            (socks-send-command proc
-                                socks-resolve-command
-                                socks-address-type-name
-                                host
-                                port)
-            (cl-assert (eq (process-get proc 'socks-state)
-                           socks-state-connected))
-            (setq ip (socks--extract-resolve-response proc)))
-        (when proc
-          (delete-process proc)))
-      (list (vconcat ip [0])))))
+  (let* ((socks-password (or socks-password ""))
+         (host (if (string-match "\\`[[:ascii:]]+\\'" name)
+                   name
+                 (require 'puny)
+                 (puny-encode-domain name)))
+         (port 80)  ; unused for now
+         (route (socks-find-route host nil))
+         proc
+         ip)
+    (cl-assert route)
+    ;; "Host unreachable" may be raised when the lookup fails
+    (unwind-protect
+        (progn
+          (setq proc (socks-open-connection route))
+          (socks-send-command proc
+                              socks-resolve-command
+                              socks-address-type-name
+                              host
+                              port)
+          (cl-assert (eq (process-get proc 'socks-state)
+                         socks-state-connected))
+          (setq ip (socks--extract-resolve-response proc)))
+      (when proc
+        (delete-process proc)))
+    (list (vconcat ip [0]))))
 
 (provide 'socks)
 
diff --git a/lisp/url/url-gw.el b/lisp/url/url-gw.el
index c4a41f56b3..822cbcb64e 100644
--- a/lisp/url/url-gw.el
+++ b/lisp/url/url-gw.el
@@ -215,6 +215,8 @@ url-open-stream
 Optional arg GATEWAY-METHOD specifies the gateway to be used,
 overriding the value of `url-gateway-method'."
   (unless url-gateway-unplugged
+    (when (eq url-gateway-method 'socks)
+      (setq gateway-method nil))
     (let* ((gwm (or gateway-method url-gateway-method))
            (gw-method (if (and url-gateway-local-host-regexp
                                (not (eq 'tls gwm))
diff --git a/test/lisp/net/socks-tests.el b/test/lisp/net/socks-tests.el
index f2600210b0..402ccf979d 100644
--- a/test/lisp/net/socks-tests.el
+++ b/test/lisp/net/socks-tests.el
@@ -348,6 +348,7 @@ test-socks-https-poc
   (ert-with-temp-file tempfile
     :prefix "emacs-test-socks-network-security-"
     (let* ((socks-server `("tor" ,@test-socks-service 5))
+           (socks-username "user")
            (socks-password "")
            (nsm-settings-file tempfile)
            (url-gateway-method 'socks)
@@ -361,25 +362,24 @@ test-socks-https-poc
                  (goto-char (point-min))
                  (should (search-forward "Congratulations" nil t))
                  (setq done t)))
-           (orig (symbol-function #'socks--open-network-stream)))
-      (cl-letf (((symbol-function 'socks--open-network-stream)
-                 (lambda (&rest rest)
-                   (let ((proc (apply orig rest)))
-                     (gnutls-negotiate :process proc :hostname host)
-                     (should (nsm-verify-connection proc host 443 t))))))
-        (ert-info ("Connect to HTTPS endpoint over Tor SOCKS proxy")
-          (unwind-protect
-              (progn
-                (advice-add 'network-lookup-address-info :override
-                            #'socks-tor-resolve)
-                (should-not (nsm-host-settings id))
-                (url-http url cb '(nil))
-                (ert-info ("Wait for response")
-                  (with-timeout (3 (error "Request timed out"))
-                    (unless done
-                      (sleep-for 0.1))))
-                (should (nsm-host-settings id)))
-            (advice-remove 'network-lookup-address-info
-                           #'socks-tor-resolve)))))))
+           (socks-open-network-stream-function
+            (lambda (&rest rest)
+              (let ((proc (apply #'socks-open-network-stream-legacy rest)))
+                (gnutls-negotiate :process proc :hostname host)
+                (should (nsm-verify-connection proc host 443 t))))))
+      (ert-info ("Connect to HTTPS endpoint over Tor SOCKS proxy")
+        (unwind-protect
+            (progn
+              (advice-add 'network-lookup-address-info :override
+                          #'socks-tor-resolve)
+              (should-not (nsm-host-settings id))
+              (url-https url cb '(nil))
+              (ert-info ("Wait for response")
+                (with-timeout (3 (error "Request timed out"))
+                  (unless done
+                    (sleep-for 0.1))))
+              (should (nsm-host-settings id)))
+          (advice-remove 'network-lookup-address-info
+                         #'socks-tor-resolve))))))
 
 ;;; socks-tests.el ends here
-- 
2.35.1


--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
 filename=0001-Simplify-network-stream-opener-in-socks.el.patch

From ed93ee2fdc8d6b920a44ddaa2b0571948cf77c88 Mon Sep 17 00:00:00 2001
From: "F. Jason Park" <jp@HIDDEN>
Date: Tue, 1 Mar 2022 02:12:02 -0800
Subject: [PATCH 1/6] Simplify network-stream opener in socks.el

* lisp/net/socks.el (socks-override-functions): Make variable obsolete
and remove uses throughout.
(socks-open-connection): Accept additional `make-network-process'
params passed on to opener.
(socks-open-network-stream-function): Add new custom option to hold an
opener function.
(socks-open-network-stream-legacy): Simulate original
`socks-open-network-stream' functionality, only without
`socks-override-functions'.  Call `open-network-stream' as a fallback
when a route cannot be found.
(socks-open-network-stream): Accept additional params.  Delegate to
`socks-open-network-stream-function' for actual work.
(socks--open-network-stream): Reduce role to merely issuing the first
command using an existing process.
---
 lisp/net/socks.el | 65 +++++++++++++++++++++++++++--------------------
 1 file changed, 38 insertions(+), 27 deletions(-)

diff --git a/lisp/net/socks.el b/lisp/net/socks.el
index 8df0773e1d..fe66a94d18 100644
--- a/lisp/net/socks.el
+++ b/lisp/net/socks.el
@@ -323,19 +323,20 @@ socks-filter
 
 (defvar socks-override-functions nil
   "If non-nil, overwrite `open-network-stream' function with SOCKSified version.")
-
-(when socks-override-functions
-  (advice-add 'open-network-stream :around #'socks--open-network-stream))
-
-(defun socks-open-connection (server-info)
+(make-obsolete-variable 'socks-override-functions
+                        "see `socks-open-network-stream-function'."
+                        "29.1")
+
+(defun socks-open-connection (server-info &rest kw-args)
+  "Create and initialize a SOCKS process.
+Perform authentication if needed.  SERVER-INFO should resemble
+`socks-server'.  KW-ARGS are those accepted by `open-network-stream'."
   (interactive)
+  (unless (plist-member kw-args :coding)
+    (setf (plist-get kw-args :coding) '(binary . binary)))
   (save-excursion
-    (let ((proc
-           (let ((socks-override-functions nil))
-             (open-network-stream "socks"
-				  nil
-				  (nth 1 server-info)
-				  (nth 2 server-info))))
+    (let ((proc (apply #'open-network-stream "socks" nil
+                       (nth 1 server-info) (nth 2 server-info) kw-args))
 	  (authtype nil)
 	  version)
 
@@ -508,22 +509,32 @@ socks-find-services-entry
   (gethash (downcase service)
 	      (if udp socks-udp-services socks-tcp-services)))
 
-(defun socks-open-network-stream (name buffer host service)
-  (let ((socks-override-functions t))
-    (socks--open-network-stream
-     (lambda (&rest args)
-       (let ((socks-override-functions nil))
-         (apply #'open-network-stream args)))
-     name buffer host service)))
-
-(defun socks--open-network-stream (orig-fun name buffer host service &rest params)
-  (let ((route (and socks-override-functions
-                    (socks-find-route host service))))
-    (if (not route)
-	(apply orig-fun name buffer host service params)
-      ;; FIXME: Obey `params'!
-      (let* ((proc (socks-open-connection route))
-	     (version (process-get proc 'socks-server-protocol))
+(defcustom socks-open-network-stream-function
+  #'socks-open-network-stream-legacy
+  "Function to open a SOCKS connection.
+Called with NAME, BUFFER, HOST, and SERVICE, for compatibility with
+similar functions in the url-gw framework.  May also be passed
+additional keyword args suitable for `make-network-process'."
+  :type '(choice (const :tag "Default fallback-oriented opener.")
+                 (function :tag "User-provided function")))
+
+(defun socks-open-network-stream-legacy (name buffer host service &rest params)
+  "Open a SOCKS connection for a valid route.
+Fall back to non-SOCKS connections for unknown or undesired routes."
+  (if-let* ((route (socks-find-route host service))
+            (proc (apply #'socks-open-connection route params)))
+      (socks--open-network-stream proc buffer host service)
+    ;; Retain legacy behavior and connect anyway without warning
+    (apply #'open-network-stream name buffer host service params)))
+
+(defun socks-open-network-stream (name buffer host service &rest params)
+  "Open a SOCKS connection.  PARAMS are passed to `open-network-stream'."
+  (apply socks-open-network-stream-function name buffer host service params))
+
+(defun socks--open-network-stream (proc buffer host service)
+  (progn ; temporarily preserve git blame for easier reviewing
+    (progn ; could rename to something like `socks--initiate-command-connect'
+      (let* ((version (process-get proc 'socks-server-protocol))
              (atype
               (cond
                ((equal version 4)
-- 
2.35.1


--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
 filename=0002-lisp-url-url-gw.el-url-open-stream-Honor-socks-gatew.patch

From a8bc5fe336356528dd0ebca86ec18ca541cb4b27 Mon Sep 17 00:00:00 2001
From: "F. Jason Park" <jp@HIDDEN>
Date: Sun, 6 Mar 2022 17:14:50 -0800
Subject: [PATCH 2/6] ; * lisp/url/url-gw.el (url-open-stream): Honor socks
 gateway-method

---
 lisp/url/url-gw.el | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/lisp/url/url-gw.el b/lisp/url/url-gw.el
index c4a41f56b3..822cbcb64e 100644
--- a/lisp/url/url-gw.el
+++ b/lisp/url/url-gw.el
@@ -215,6 +215,8 @@ url-open-stream
 Optional arg GATEWAY-METHOD specifies the gateway to be used,
 overriding the value of `url-gateway-method'."
   (unless url-gateway-unplugged
+    (when (eq url-gateway-method 'socks)
+      (setq gateway-method nil))
     (let* ((gwm (or gateway-method url-gateway-method))
            (gw-method (if (and url-gateway-local-host-regexp
                                (not (eq 'tls gwm))
-- 
2.35.1


--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
 filename=0003-Fix-string-encoding-bug-in-socks-tests.patch

From e365303eeced26d5fc901e623eb44b3f6c2515cb Mon Sep 17 00:00:00 2001
From: "F. Jason Park" <jp@HIDDEN>
Date: Mon, 14 Feb 2022 02:36:57 -0800
Subject: [PATCH 3/6] Fix string encoding bug in socks tests

* test/lisp/net/socks-tests.el (socks-tests-canned-server-create,
socks-tests-filter-response-parsing-v4): Fix bug in process filter to
prevent prepared outgoing responses from being implicitly encoded as
utf-8.  Fix similar mistake in v4 filter test.
---
 test/lisp/net/socks-tests.el | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/test/lisp/net/socks-tests.el b/test/lisp/net/socks-tests.el
index 461796bdf9..d9ef53ae35 100644
--- a/test/lisp/net/socks-tests.el
+++ b/test/lisp/net/socks-tests.el
@@ -63,21 +63,21 @@ socks-tests-filter-response-parsing-v4
     (process-put proc 'socks-state socks-state-waiting)
     (process-put proc 'socks-server-protocol 4)
     (ert-info ("Receive initial incomplete segment")
-      (socks-filter proc (concat [0 90 0 0 93 184 216]))
-      ;; From example.com: OK status ^      ^ msg start
+      (socks-filter proc (unibyte-string 0 90 0 0 93 184 216))
+      ;; From example.com: OK status       ^      ^ msg start
       (ert-info ("State still set to waiting")
         (should (eq (process-get proc 'socks-state) socks-state-waiting)))
       (ert-info ("Response field is nil because processing incomplete")
         (should-not (process-get proc 'socks-response)))
       (ert-info ("Scratch field holds stashed partial payload")
-        (should (string= (concat [0 90 0 0 93 184 216])
+        (should (string= (unibyte-string 0 90 0 0 93 184 216)
                          (process-get proc 'socks-scratch)))))
     (ert-info ("Last part arrives")
       (socks-filter proc "\42") ; ?\" 34
       (ert-info ("State transitions to complete (length check passes)")
         (should (eq (process-get proc 'socks-state) socks-state-connected)))
       (ert-info ("Scratch and response fields hold stash w. last chunk")
-        (should (string= (concat [0 90 0 0 93 184 216 34])
+        (should (string= (unibyte-string 0 90 0 0 93 184 216 34)
                          (process-get proc 'socks-response)))
         (should (string= (process-get proc 'socks-response)
                          (process-get proc 'socks-scratch)))))
@@ -140,10 +140,11 @@ socks-tests-canned-server-create
                    (unless (or (and (vectorp pat) (equal pat (vconcat line)))
                                (string-match-p pat line))
                      (error "Unknown request: %s" line))
+                   (setq resp (apply #'unibyte-string (append resp nil)))
                    (let ((print-escape-control-characters t))
                      (message "[%s] <- %s" name (prin1-to-string line))
                      (message "[%s] -> %s" name (prin1-to-string resp)))
-                   (process-send-string proc (concat resp)))))
+                   (process-send-string proc resp))))
          (serv (make-network-process :server 1
                                      :buffer (get-buffer-create name)
                                      :filter filt
-- 
2.35.1


--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
 filename=0004-Add-support-for-SOCKS-4a.patch

From bb2187da12d88e8b32f9fd005926342e116970c3 Mon Sep 17 00:00:00 2001
From: "F. Jason Park" <jp@HIDDEN>
Date: Mon, 14 Feb 2022 02:36:57 -0800
Subject: [PATCH 4/6] Add support for SOCKS 4a

* lisp/net/socks.el (socks-server): Add new choice `4a' to version
field of option.  This may appear to change the type of the field from
a number to a union of symbols and numbers.  However,
`socks-send-command' and `socks-filter' already expect a possible
`http' value for this field (also a symbol).
(socks--errors-4): Add new constant containing error messages for
socks version 4.  The semantics are faithful, but the wording is
ad-libbed.
(socks-send-command): Massage existing handling for version 4 to
accommodate 4a.

* test/lisp/net/socks-tests.el (socks-tests-v4a-basic): add test for
4a.
Bug#53941
---
 lisp/net/socks.el            | 22 ++++++++++++++++++++--
 test/lisp/net/socks-tests.el | 13 +++++++++++++
 2 files changed, 33 insertions(+), 2 deletions(-)

diff --git a/lisp/net/socks.el b/lisp/net/socks.el
index fe66a94d18..9f60ecbf36 100644
--- a/lisp/net/socks.el
+++ b/lisp/net/socks.el
@@ -162,6 +162,7 @@ socks-server
 	  (radio-button-choice :tag "SOCKS Version"
 			       :format "%t: %v"
 			       (const :tag "SOCKS v4  " :format "%t" :value 4)
+                               (const :tag "SOCKS v4a"  :format "%t" :value 4a)
 			       (const :tag "SOCKS v5"   :format "%t" :value 5))))
 
 
@@ -202,6 +203,12 @@ socks-errors
     "Command not supported"
     "Address type not supported"))
 
+(defconst socks--errors-4
+  '("Granted"
+    "Rejected or failed"
+    "Cannot connect to identd on the client"
+    "Client and identd report differing user IDs"))
+
 ;; The socks v5 address types
 (defconst socks-address-type-v4   1)
 (defconst socks-address-type-name 3)
@@ -401,6 +408,7 @@ socks-send-command
 		(format "%c%s" (length address) address))
 	       (t
 		(error "Unknown address type: %d" atype))))
+        trailing
 	request version)
     (or (process-get proc 'socks)
         (error "socks-send-command called on non-SOCKS connection %S" proc))
@@ -418,6 +426,12 @@ socks-send-command
 			     (t
 			      (error "Unsupported address type for HTTP: %d" atype)))
 			    port)))
+     ((when (eq version '4a)
+        (setf addr "\0\0\0\1"
+              trailing (concat address "\0")
+              version 4 ; done with the "a" part
+              (process-get proc 'socks-server-protocol) 4)
+        nil)) ; fall through
      ((equal version 4)
       (setq request (concat
 		     (unibyte-string
@@ -427,7 +441,8 @@ socks-send-command
 		      (logand port #xff)) ; port, low byte
 		     addr                 ; address
 		     (user-full-name)     ; username
-		     "\0")))              ; terminate username
+                     "\0"                 ; terminate username
+                     trailing)))          ; optional host to look up
      ((equal version 5)
       (setq request (concat
 		     (unibyte-string
@@ -448,7 +463,10 @@ socks-send-command
 	nil				; Sweet sweet success!
       (delete-process proc)
       (error "SOCKS: %s"
-             (nth (or (process-get proc 'socks-reply) 1) socks-errors)))
+             (let ((no (or (process-get proc 'socks-reply) 1)))
+               (if (eq version 5)
+                   (nth no socks-errors)
+                 (nth (+ 90 no) socks--errors-4)))))
     proc))
 
 
diff --git a/test/lisp/net/socks-tests.el b/test/lisp/net/socks-tests.el
index d9ef53ae35..4e990ffdba 100644
--- a/test/lisp/net/socks-tests.el
+++ b/test/lisp/net/socks-tests.el
@@ -207,6 +207,19 @@ socks-tests-v4-basic
                  (lambda (&optional _) "foo")))
         (socks-tests-perform-hello-world-http-request)))))
 
+(ert-deftest socks-tests-v4a-basic ()
+  "Show correct preparation of SOCKS4a connect command."
+  (let ((socks-server '("server" "127.0.0.1" 10083 4a))
+        (url-user-agent "Test/4a-basic")
+        (socks-tests-canned-server-patterns
+         `(([4 1 0 80 0 0 0 1 ?f ?o ?o 0 ?e ?x ?a ?m ?p ?l ?e ?. ?c ?o ?m 0]
+            . [0 90 0 0 0 0 0 0])
+           ,socks-tests--hello-world-http-request-pattern)))
+    (ert-info ("Make HTTP request over SOCKS4A")
+      (cl-letf (((symbol-function 'user-full-name)
+                 (lambda (&optional _) "foo")))
+        (socks-tests-perform-hello-world-http-request)))))
+
 ;; Replace first pattern below with ([5 3 0 1 2] . [5 2]) to validate
 ;; against curl 7.71 with the following options:
 ;; $ curl --verbose -U foo:bar --proxy socks5h://127.0.0.1:10080 example.com
-- 
2.35.1


--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
 filename=0005-Support-SOCKS-resolve-extension.patch

From a33717db1379a661ba8007f924dc937feeb2ad1b Mon Sep 17 00:00:00 2001
From: "F. Jason Park" <jp@HIDDEN>
Date: Mon, 14 Feb 2022 02:36:57 -0800
Subject: [PATCH 5/6] Support SOCKS resolve extension

* lisp/net/socks.el (socks-resolve-command): Add new constant for the
SOCKS command RESOLVE, which comes by way of a nonstandard extension
from the TOR project.  It mirrors CONNECT in most respects but asks
the server to RESOLVE a host name and return its IP.  For details, see
https://github.com/torproject/torspec/blob/master/socks-extensions.txt
This shouldn't be confused with 5h/5-hostname, which is used to by
clients like cURL to allow users to bypass attempts to resolve a name
locally.
(socks--extract-resolve-response, socks-tor-resolve): Add utility
functions to query a SOCKS service supporting the RESOLVE extension.
Bug#53941
---
 lisp/net/socks.el            | 58 ++++++++++++++++++++++++++++++++++++
 test/lisp/net/socks-tests.el | 34 +++++++++++++++++++++
 2 files changed, 92 insertions(+)

diff --git a/lisp/net/socks.el b/lisp/net/socks.el
index 9f60ecbf36..9285cbf805 100644
--- a/lisp/net/socks.el
+++ b/lisp/net/socks.el
@@ -181,6 +181,9 @@ socks-udp-associate-command
 (defconst socks-authentication-null 0)
 (defconst socks-authentication-failure 255)
 
+;; Extensions
+(defconst socks-resolve-command #xf0)
+
 ;; Response codes
 (defconst socks-response-success               0)
 (defconst socks-response-general-failure       1)
@@ -664,6 +667,61 @@ socks-nslookup-host
 	res)
     host))
 
+(defun socks--extract-resolve-response (proc)
+  "Parse response for PROC and maybe return destination IP address."
+  (let ((response (process-get proc 'socks-response)))
+    (cl-assert response) ; otherwise, msg not received in its entirety
+    (pcase (process-get proc 'socks-server-protocol)
+      (4 ; https://www.openssh.com/txt/socks4a.protocol
+       (when-let (((zerop (process-get proc 'socks-reply)))
+                  ((eq (aref response 1) 90)) ; #x5a request granted
+                  (a (substring response 4)) ; ignore port for now
+                  ((not (string-empty-p a)))
+                  ((not (string= a "\0\0\0\0"))))
+         a))
+      (5 ; https://tools.ietf.org/html/rfc1928
+       (cl-assert (eq 5 (aref response 0)) t)
+       (pcase (aref response 3) ; ATYP
+         (1 (and-let* ((a (substring response 4 8))
+                       ((not (string= a "\0\0\0\0")))
+                       a)))
+         ;; No reason to support RESOLVE_PTR [F1] extension, right?
+         (3 (let ((len (1- (aref response 4))))
+              (substring response 5 (+ 5 len))))
+         (4 (substring response 4 20)))))))
+
+(declare-function puny-encode-domain "puny" (domain))
+
+(defun socks-tor-resolve (name &optional _family)
+  "Return list of one vector IPv4 address for domain NAME.
+Or return nil on failure.  See `network-lookup-address-info' for format
+of return value.  Server must support the Tor RESOLVE command."
+  (let* ((socks-password (or socks-password ""))
+         (host (if (string-match "\\`[[:ascii:]]+\\'" name)
+                   name
+                 (require 'puny)
+                 (puny-encode-domain name)))
+         (port 80)  ; unused for now
+         (route (socks-find-route host nil))
+         proc
+         ip)
+    (cl-assert route)
+    ;; "Host unreachable" may be raised when the lookup fails
+    (unwind-protect
+        (progn
+          (setq proc (socks-open-connection route))
+          (socks-send-command proc
+                              socks-resolve-command
+                              socks-address-type-name
+                              host
+                              port)
+          (cl-assert (eq (process-get proc 'socks-state)
+                         socks-state-connected))
+          (setq ip (socks--extract-resolve-response proc)))
+      (when proc
+        (delete-process proc)))
+    (list (vconcat ip [0]))))
+
 (provide 'socks)
 
 ;;; socks.el ends here
diff --git a/test/lisp/net/socks-tests.el b/test/lisp/net/socks-tests.el
index 4e990ffdba..3d1aca9af4 100644
--- a/test/lisp/net/socks-tests.el
+++ b/test/lisp/net/socks-tests.el
@@ -295,4 +295,38 @@ socks-tests-v5-auth-none
       (socks-tests-perform-hello-world-http-request)))
   (should (assq 2 socks-authentication-methods)))
 
+(ert-deftest tor-resolve-4a ()
+  "Make request to TOR resolve service over SOCKS4a"
+  (let* ((socks-server '("server" "127.0.0.1" 19050 4a))
+         (socks-tests-canned-server-patterns
+          '(([4 #xf0 0 80 0 0 0 1 ?f ?o ?o 0 ?e ?x ?a ?m ?p ?l ?e ?. ?c ?o ?m 0]
+             . [0 90 0 0 93 184 216 34])))
+         (inhibit-message noninteractive)
+         (server (socks-tests-canned-server-create)))
+    (ert-info ("Query TOR RESOLVE service over SOCKS4")
+      (cl-letf (((symbol-function 'user-full-name)
+                 (lambda (&optional _) "foo")))
+        (should (equal '([93 184 216 34 0])
+                       (socks-tor-resolve "example.com")))))
+    (kill-buffer (process-buffer server))
+    (delete-process server)))
+
+(ert-deftest tor-resolve-5 ()
+  "Make request to TOR resolve service over SOCKS5"
+  (let* ((socks-server '("server" "127.0.0.1" 19051 5))
+         (socks-username "foo")
+         (socks-authentication-methods (append socks-authentication-methods
+                                               nil))
+         (inhibit-message noninteractive)
+         (socks-tests-canned-server-patterns
+          '(([5 2 0 2] . [5 2])
+            ([1 3 ?f ?o ?o 0] . [1 0])
+            ([5 #xf0 0 3 11 ?e ?x ?a ?m ?p ?l ?e ?. ?c ?o ?m 0 80]
+             . [5 0 0 1 93 184 216 34 0 0])))
+         (server (socks-tests-canned-server-create)))
+    (ert-info ("Query TOR RESOLVE service over SOCKS5")
+      (should (equal '([93 184 216 34 0]) (socks-tor-resolve "example.com"))))
+    (kill-buffer (process-buffer server))
+    (delete-process server)))
+
 ;;; socks-tests.el ends here
-- 
2.35.1


--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
 filename=0006-POC-Demo-SOCKS-resolve-with-HTTPS.patch

From 62062472fd14dc9911a105016badcc921d63ae95 Mon Sep 17 00:00:00 2001
From: "F. Jason Park" <jp@HIDDEN>
Date: Mon, 14 Feb 2022 02:36:57 -0800
Subject: [PATCH 6/6] [POC] Demo SOCKS resolve with HTTPS

* test/lisp/net/socks-test.el (test-socks-https-poc): Provide
throwaway test demoing an HTTPS connection over a TOR proxy service.
---
 test/lisp/net/socks-tests.el | 55 +++++++++++++++++++++++++++++++++++-
 1 file changed, 54 insertions(+), 1 deletion(-)

diff --git a/test/lisp/net/socks-tests.el b/test/lisp/net/socks-tests.el
index 3d1aca9af4..402ccf979d 100644
--- a/test/lisp/net/socks-tests.el
+++ b/test/lisp/net/socks-tests.el
@@ -21,7 +21,7 @@
 
 ;;; Code:
 
-(require 'ert)
+(require 'ert-x)
 (require 'socks)
 (require 'url-http)
 
@@ -329,4 +329,57 @@ tor-resolve-5
     (kill-buffer (process-buffer server))
     (delete-process server)))
 
+(defvar test-socks-service ; "127.0.0.1:1080" -> ("127.0.0.1", 1080)
+  (when-let ((present (getenv "TEST_SOCKS_SERVICE"))
+             (parts (split-string present ":")))
+    (list (car parts) (string-to-number (cadr parts)))))
+
+(declare-function gnutls-negotiate "gnutls"
+                  (&rest spec
+                         &key process type hostname priority-string
+                         trustfiles crlfiles keylist min-prime-bits
+                         verify-flags verify-error verify-hostname-error
+                         &allow-other-keys))
+
+(ert-deftest test-socks-https-poc ()
+  :tags '(:unstable)
+  (unless test-socks-service (ert-skip "SOCKS service missing"))
+  (unless (gnutls-available-p) (ert-skip "SOCKS resolve test needs GNUTLS"))
+  (ert-with-temp-file tempfile
+    :prefix "emacs-test-socks-network-security-"
+    (let* ((socks-server `("tor" ,@test-socks-service 5))
+           (socks-username "user")
+           (socks-password "")
+           (nsm-settings-file tempfile)
+           (url-gateway-method 'socks)
+           (id "sha1:df77269389e537fcc9a5fe61667133b5bb97d42e")
+           (host "check.torproject.org")
+           (url (url-generic-parse-url "https://check.torproject.org"))
+           ;;
+           done
+           ;;
+           (cb (lambda (&rest _r)
+                 (goto-char (point-min))
+                 (should (search-forward "Congratulations" nil t))
+                 (setq done t)))
+           (socks-open-network-stream-function
+            (lambda (&rest rest)
+              (let ((proc (apply #'socks-open-network-stream-legacy rest)))
+                (gnutls-negotiate :process proc :hostname host)
+                (should (nsm-verify-connection proc host 443 t))))))
+      (ert-info ("Connect to HTTPS endpoint over Tor SOCKS proxy")
+        (unwind-protect
+            (progn
+              (advice-add 'network-lookup-address-info :override
+                          #'socks-tor-resolve)
+              (should-not (nsm-host-settings id))
+              (url-https url cb '(nil))
+              (ert-info ("Wait for response")
+                (with-timeout (3 (error "Request timed out"))
+                  (unless done
+                    (sleep-for 0.1))))
+              (should (nsm-host-settings id)))
+          (advice-remove 'network-lookup-address-info
+                         #'socks-tor-resolve))))))
+
 ;;; socks-tests.el ends here
-- 
2.35.1


--=-=-=--

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#53941: 27.2; socks + tor dont work with https
Resent-From: "J.P." <jp@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Thu, 10 Mar 2022 09:00:02 +0000
Resent-Message-ID: <handler.53941.B53941.16469027456056 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 53941
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: patch
To: Jacobo <gnuhacker@HIDDEN>
Cc: 53941 <at> debbugs.gnu.org
Received: via spool by 53941-submit <at> debbugs.gnu.org id=B53941.16469027456056
          (code B ref 53941); Thu, 10 Mar 2022 09:00:02 +0000
Received: (at 53941) by debbugs.gnu.org; 10 Mar 2022 08:59:05 +0000
Received: from localhost ([127.0.0.1]:33165 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1nSEdQ-0001ZD-9z
	for submit <at> debbugs.gnu.org; Thu, 10 Mar 2022 03:59:04 -0500
Received: from mail-108-mta139.mxroute.com ([136.175.108.139]:42317)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <jp@HIDDEN>) id 1nSEdM-0001Yw-E6
 for 53941 <at> debbugs.gnu.org; Thu, 10 Mar 2022 03:58:58 -0500
Received: from filter006.mxroute.com ([140.82.40.27] 140.82.40.27.vultr.com)
 (Authenticated sender: mN4UYu2MZsgR)
 by mail-108-mta139.mxroute.com (ZoneMTA) with ESMTPSA id
 17f730d5008000763e.001 for <53941 <at> debbugs.gnu.org>
 (version=TLSv1/SSLv3 cipher=ECDHE-RSA-AES128-GCM-SHA256);
 Thu, 10 Mar 2022 08:58:46 +0000
X-Zone-Loop: 3f7267d7e8efb810517dd8d06c9dc50dbf74de14a54a
X-Originating-IP: [140.82.40.27]
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=neverwas.me
 ; s=x;
 h=Content-Type:MIME-Version:Message-ID:In-Reply-To:Date:References:
 Subject:Cc:To:From:Sender:Reply-To:Content-Transfer-Encoding:Content-ID:
 Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc
 :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe:
 List-Post:List-Owner:List-Archive;
 bh=sdQJk+Qk2J9phfmQ1MnNo26t/6Nbr6JozG8JkKcWrhw=; b=UrlDkiXKiB9QIcMMULk6yreL0y
 qB9FruNAb5M94n+9vwKCYPeflGigafwIlZh5DbNDjPDhPjv5cwssIsNyrR7BBe5l9qBBZaq7lA+JQ
 7QcxZn2iXoMMjYXg1ReB18u2XmSQ7sHIDZQ5G4cEvul+eJPZH91cPle7OKttINfEa1OZq3MlMv8Ag
 6M32Do5AhbKQzddAuKiUzn6yIhGIoqAkzCaR3wICF0kH0nMtfXus184lZIzdNJTJ1egTRTxNHHd26
 iK4gSBaQdIwr/1Hvn40oh+oWQJ/Q4EhFSgLdBLlKRz/oP+6krFpBBOXccJvmB025sEWSAcQuowjtY
 Y4J9FxSw==;
From: "J.P." <jp@HIDDEN>
References: <87pmntfym7.fsf@HIDDEN> <8735kl1v58.fsf@HIDDEN>
 <87a6emftzx.fsf@HIDDEN> <87k0do5km1.fsf@HIDDEN>
 <87pmn5n3tu.fsf@HIDDEN> <87mti99j1f.fsf@HIDDEN>
 <87wnh7hkgi.fsf@HIDDEN> <87pmmz947k.fsf@HIDDEN>
 <8735ju44sk.fsf@HIDDEN>
Date: Thu, 10 Mar 2022 00:58:42 -0800
In-Reply-To: <8735ju44sk.fsf@HIDDEN> (J. P.'s message of "Sun, 06 Mar
 2022 23:09:47 -0800")
Message-ID: <87lexikwu5.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/29.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="=-=-="
X-AuthUser: masked@HIDDEN
X-Spam-Score: -0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

--=-=-=
Content-Type: text/plain

v5. Forgot to account for look-up failures (shocking not shocking).
Also removed hard-coded port numbers from tests.

The EWW example from earlier needs some adapting:

  ;; M-x eww RET https://check.torproject.org RET

  (require 'socks)
  (require 'gnutls)
  (require 'nsm)

  (defun my-socks-open-https (name buffer host service &rest params)
    (let ((proc (apply #'socks-open-network-stream-legacy
                       name buffer host service params)))
      (advice-add 'network-lookup-address-info :override #'socks-tor-resolve)
      (unwind-protect
          (when (eq service 443)
            (gnutls-negotiate :process proc :hostname host)
            (unless (string-suffix-p ".onion" host)
              (nsm-verify-connection proc host service)))
        (advice-remove 'network-lookup-address-info #'socks-tor-resolve))
      proc))

  (setq socks-server '("tor" "127.0.0.1" 9050 5)
        socks-username ""
        socks-password ""
        url-gateway-method 'socks
        socks-open-network-stream-function #'my-socks-open-https)

Let me know if you need help. Thanks.


--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment; filename=0000-NOT-A-PATCH-v4-v5.diff

From 52a7f3269992166074ebe277f6905c219885d7cf Mon Sep 17 00:00:00 2001
From: "F. Jason Park" <jp@HIDDEN>
Date: Thu, 10 Mar 2022 00:18:09 -0800
Subject: [PATCH 0/6] *** SUBJECT HERE ***

*** BLURB HERE ***

F. Jason Park (6):
  Simplify network-stream opener in socks.el
  ; * lisp/url/url-gw.el (url-open-stream): Honor socks gateway-method
  Fix string encoding bug in socks tests
  Add support for SOCKS 4a
  Support SOCKS resolve extension
  [POC] Demo SOCKS resolve with HTTPS

 lisp/net/socks.el            | 157 +++++++++++++++++++++++++-------
 lisp/url/url-gw.el           |   2 +
 test/lisp/net/socks-tests.el | 168 ++++++++++++++++++++++++++++++++---
 3 files changed, 285 insertions(+), 42 deletions(-)

Interdiff:
diff --git a/lisp/net/socks.el b/lisp/net/socks.el
index 9285cbf805..9ce23b517e 100644
--- a/lisp/net/socks.el
+++ b/lisp/net/socks.el
@@ -319,7 +319,8 @@ socks-filter
 		     ((pred (= socks-address-type-name))
 		      (if (< (length string) 5)
 			  255
-		        (+ 1 (aref string 4)))))))
+                        (+ 1 (aref string 4))))
+                     (0 0))))
 	  (if (< (length string) desired-len)
 	      nil			; Need to spin some more
 	    (process-put proc 'socks-state socks-state-connected)
@@ -469,7 +470,7 @@ socks-send-command
              (let ((no (or (process-get proc 'socks-reply) 1)))
                (if (eq version 5)
                    (nth no socks-errors)
-                 (nth (+ 90 no) socks--errors-4)))))
+                 (nth (- no 90) socks--errors-4)))))
     proc))
 
 
@@ -692,19 +693,11 @@ socks--extract-resolve-response
 
 (declare-function puny-encode-domain "puny" (domain))
 
-(defun socks-tor-resolve (name &optional _family)
-  "Return list of one vector IPv4 address for domain NAME.
-Or return nil on failure.  See `network-lookup-address-info' for format
-of return value.  Server must support the Tor RESOLVE command."
-  (let* ((socks-password (or socks-password ""))
-         (host (if (string-match "\\`[[:ascii:]]+\\'" name)
-                   name
-                 (require 'puny)
-                 (puny-encode-domain name)))
-         (port 80)  ; unused for now
-         (route (socks-find-route host nil))
-         proc
-         ip)
+(defun socks--tor-resolve (host)
+  (let ((socks-password (or socks-password ""))
+        (route (socks-find-route host nil))
+        proc
+        ip)
     (cl-assert route)
     ;; "Host unreachable" may be raised when the lookup fails
     (unwind-protect
@@ -714,13 +707,30 @@ socks-tor-resolve
                               socks-resolve-command
                               socks-address-type-name
                               host
-                              port)
-          (cl-assert (eq (process-get proc 'socks-state)
-                         socks-state-connected))
+                              0)
           (setq ip (socks--extract-resolve-response proc)))
       (when proc
         (delete-process proc)))
-    (list (vconcat ip [0]))))
+    ip))
+
+(defun socks-tor-resolve (name &optional _family)
+  "Return list of one IPv4 address for domain NAME.
+See `network-lookup-address-info' for format of return value.  Return
+nil on failure.
+
+SOCKS server must support the Tor RESOLVE command.  Note that using this
+in place of `network-lookup-address-info' may not be enough to prevent a
+DNS leak.  For example, see `url-gateway-broken-resolution'."
+  (unless (string-match "\\`[[:ascii:]]+\\'" name)
+    (require 'puny)
+    (setq name (puny-encode-domain name)))
+  (condition-case err
+      (when-let ((ip (socks--tor-resolve name)))
+        (list (vconcat ip [0])))
+    (error
+     (unless (member (cadr err)
+                     '("SOCKS: Host unreachable" "SOCKS: Rejected or failed"))
+       (signal (car err) (cdr err))))))
 
 (provide 'socks)
 
diff --git a/test/lisp/net/socks-tests.el b/test/lisp/net/socks-tests.el
index 402ccf979d..0c58fcc863 100644
--- a/test/lisp/net/socks-tests.el
+++ b/test/lisp/net/socks-tests.el
@@ -133,7 +133,8 @@ socks-tests-canned-server-patterns
 (defun socks-tests-canned-server-create ()
   "Create and return a fake SOCKS server."
   (let* ((port (nth 2 socks-server))
-         (name (format "socks-server:%d" port))
+         (name (format "socks-server:%s"
+                       (or (numberp port) (ert-test-name (ert-running-test)))))
          (pats socks-tests-canned-server-patterns)
          (filt (lambda (proc line)
                  (pcase-let ((`(,pat . ,resp) (pop pats)))
@@ -152,8 +153,10 @@ socks-tests-canned-server-create
                                      :family 'ipv4
                                      :host 'local
                                      :coding 'binary
-                                     :service port)))
+                                     :service (or port t))))
     (set-process-query-on-exit-flag serv nil)
+    (unless (numberp (nth 2 socks-server))
+      (setf (nth 2 socks-server) (process-contact serv :service)))
     serv))
 
 (defvar socks-tests--hello-world-http-request-pattern
@@ -192,7 +195,7 @@ socks-tests-perform-hello-world-http-request
 
 (ert-deftest socks-tests-v4-basic ()
   "Show correct preparation of SOCKS4 connect command (Bug#46342)."
-  (let ((socks-server '("server" "127.0.0.1" 10079 4))
+  (let ((socks-server '("server" "127.0.0.1" t 4))
         (url-user-agent "Test/4-basic")
         (socks-tests-canned-server-patterns
          `(([4 1 0 80 93 184 216 34 ?f ?o ?o 0] . [0 90 0 0 0 0 0 0])
@@ -209,7 +212,7 @@ socks-tests-v4-basic
 
 (ert-deftest socks-tests-v4a-basic ()
   "Show correct preparation of SOCKS4a connect command."
-  (let ((socks-server '("server" "127.0.0.1" 10083 4a))
+  (let ((socks-server '("server" "127.0.0.1" t 4a))
         (url-user-agent "Test/4a-basic")
         (socks-tests-canned-server-patterns
          `(([4 1 0 80 0 0 0 1 ?f ?o ?o 0 ?e ?x ?a ?m ?p ?l ?e ?. ?c ?o ?m 0]
@@ -227,7 +230,7 @@ socks-tests-v4a-basic
 (ert-deftest socks-tests-v5-auth-user-pass ()
   "Verify correct handling of SOCKS5 user/pass authentication."
   (should (assq 2 socks-authentication-methods))
-  (let ((socks-server '("server" "127.0.0.1" 10080 5))
+  (let ((socks-server '("server" "127.0.0.1" t 5))
         (socks-username "foo")
         (socks-password "bar")
         (url-user-agent "Test/auth-user-pass")
@@ -261,7 +264,7 @@ socks-tests-v5-auth-user-pass
 (ert-deftest socks-tests-v5-auth-user-pass-blank ()
   "Verify correct SOCKS5 user/pass authentication with empty pass."
   (should (assq 2 socks-authentication-methods))
-  (let ((socks-server '("server" "127.0.0.1" 10081 5))
+  (let ((socks-server '("server" "127.0.0.1" t 5))
         (socks-username "foo") ; defaults to (user-login-name)
         (socks-password "") ; simulate user hitting enter when prompted
         (url-user-agent "Test/auth-user-pass-blank")
@@ -280,7 +283,7 @@ socks-tests-v5-auth-user-pass-blank
 
 (ert-deftest socks-tests-v5-auth-none ()
   "Verify correct handling of SOCKS5 when auth method 0 requested."
-  (let ((socks-server '("server" "127.0.0.1" 10082 5))
+  (let ((socks-server '("server" "127.0.0.1" t 5))
         (socks-authentication-methods (append socks-authentication-methods
                                               nil))
         (url-user-agent "Test/auth-none")
@@ -297,9 +300,9 @@ socks-tests-v5-auth-none
 
 (ert-deftest tor-resolve-4a ()
   "Make request to TOR resolve service over SOCKS4a"
-  (let* ((socks-server '("server" "127.0.0.1" 19050 4a))
+  (let* ((socks-server '("server" "127.0.0.1" t 4a))
          (socks-tests-canned-server-patterns
-          '(([4 #xf0 0 80 0 0 0 1 ?f ?o ?o 0 ?e ?x ?a ?m ?p ?l ?e ?. ?c ?o ?m 0]
+          '(([4 #xf0 0 0 0 0 0 1 ?f ?o ?o 0 ?e ?x ?a ?m ?p ?l ?e ?. ?c ?o ?m 0]
              . [0 90 0 0 93 184 216 34])))
          (inhibit-message noninteractive)
          (server (socks-tests-canned-server-create)))
@@ -311,9 +314,40 @@ tor-resolve-4a
     (kill-buffer (process-buffer server))
     (delete-process server)))
 
+(ert-deftest tor-resolve-4a-fail ()
+  (let* ((socks-server '("server" "127.0.0.1" t 4a))
+         (socks-tests-canned-server-patterns
+          '(([4 #xf0 0 0 0 0 0 1 ?f ?o ?o 0 ?e ?x ?a ?m ?p ?l ?e ?. ?c ?o ?m 0]
+             . [0 91 0 0 0 0 0 0])))
+         (inhibit-message noninteractive)
+         (server (socks-tests-canned-server-create)))
+    (ert-info ("Query TOR RESOLVE service over SOCKS4")
+      (cl-letf (((symbol-function 'user-full-name)
+                 (lambda (&optional _) "foo")))
+        (should-not (socks-tor-resolve "example.com"))))
+    (kill-buffer (process-buffer server))
+    (delete-process server)))
+
+(ert-deftest tor-resolve-5-fail ()
+  (let* ((socks-server '("server" "127.0.0.1" t 5))
+         (socks-username "")
+         (socks-authentication-methods (copy-sequence
+                                        socks-authentication-methods))
+         (inhibit-message noninteractive)
+         (socks-tests-canned-server-patterns
+          '(([5 2 0 2] . [5 2])
+            ([1 0 0] . [1 0])
+            ([5 #xf0 0 3 11 ?e ?x ?a ?m ?p ?l ?e ?. ?c ?o ?m 0 0]
+             . [5 4 0 0 0 0 0 0 0 0])))
+         (server (socks-tests-canned-server-create)))
+    (ert-info ("Query TOR RESOLVE service over SOCKS5")
+      (should-not (socks-tor-resolve "example.com")))
+    (kill-buffer (process-buffer server))
+    (delete-process server)))
+
 (ert-deftest tor-resolve-5 ()
   "Make request to TOR resolve service over SOCKS5"
-  (let* ((socks-server '("server" "127.0.0.1" 19051 5))
+  (let* ((socks-server '("server" "127.0.0.1" t 5))
          (socks-username "foo")
          (socks-authentication-methods (append socks-authentication-methods
                                                nil))
@@ -321,7 +355,7 @@ tor-resolve-5
          (socks-tests-canned-server-patterns
           '(([5 2 0 2] . [5 2])
             ([1 3 ?f ?o ?o 0] . [1 0])
-            ([5 #xf0 0 3 11 ?e ?x ?a ?m ?p ?l ?e ?. ?c ?o ?m 0 80]
+            ([5 #xf0 0 3 11 ?e ?x ?a ?m ?p ?l ?e ?. ?c ?o ?m 0 0]
              . [5 0 0 1 93 184 216 34 0 0])))
          (server (socks-tests-canned-server-create)))
     (ert-info ("Query TOR RESOLVE service over SOCKS5")
@@ -341,6 +375,15 @@ test-socks-service
                          verify-flags verify-error verify-hostname-error
                          &allow-other-keys))
 
+(ert-deftest test-socks-resolve-fail ()
+  :tags '(:unstable)
+  (unless test-socks-service (ert-skip "SOCKS service missing"))
+  (let* ((socks-server `("tor" ,@test-socks-service 5)) ; also try 4a
+         (socks-username "")
+         (socks-password ""))
+    (ert-info ("Connect to HTTP endpoint over Tor SOCKS proxy")
+      (should-not (socks-tor-resolve "test-socks-resolve-fail--fake.com")))))
+
 (ert-deftest test-socks-https-poc ()
   :tags '(:unstable)
   (unless test-socks-service (ert-skip "SOCKS service missing"))
-- 
2.35.1


--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
 filename=0001-Simplify-network-stream-opener-in-socks.el.patch

From dcb7c638d970c0924933ebd83f361298ebccf242 Mon Sep 17 00:00:00 2001
From: "F. Jason Park" <jp@HIDDEN>
Date: Tue, 1 Mar 2022 02:12:02 -0800
Subject: [PATCH 1/6] Simplify network-stream opener in socks.el

* lisp/net/socks.el (socks-override-functions): Make variable obsolete
and remove uses throughout.
(socks-open-connection): Accept additional `make-network-process'
params passed on to opener.
(socks-open-network-stream-function): Add new custom option to hold an
opener function.
(socks-open-network-stream-legacy): Simulate original
`socks-open-network-stream' functionality, only without
`socks-override-functions'.  Call `open-network-stream' as a fallback
when a route cannot be found.
(socks-open-network-stream): Accept additional params.  Delegate to
`socks-open-network-stream-function' for actual work.
(socks--open-network-stream): Reduce role to merely issuing the first
command using an existing process.
---
 lisp/net/socks.el | 65 +++++++++++++++++++++++++++--------------------
 1 file changed, 38 insertions(+), 27 deletions(-)

diff --git a/lisp/net/socks.el b/lisp/net/socks.el
index 8df0773e1d..fe66a94d18 100644
--- a/lisp/net/socks.el
+++ b/lisp/net/socks.el
@@ -323,19 +323,20 @@ socks-filter
 
 (defvar socks-override-functions nil
   "If non-nil, overwrite `open-network-stream' function with SOCKSified version.")
-
-(when socks-override-functions
-  (advice-add 'open-network-stream :around #'socks--open-network-stream))
-
-(defun socks-open-connection (server-info)
+(make-obsolete-variable 'socks-override-functions
+                        "see `socks-open-network-stream-function'."
+                        "29.1")
+
+(defun socks-open-connection (server-info &rest kw-args)
+  "Create and initialize a SOCKS process.
+Perform authentication if needed.  SERVER-INFO should resemble
+`socks-server'.  KW-ARGS are those accepted by `open-network-stream'."
   (interactive)
+  (unless (plist-member kw-args :coding)
+    (setf (plist-get kw-args :coding) '(binary . binary)))
   (save-excursion
-    (let ((proc
-           (let ((socks-override-functions nil))
-             (open-network-stream "socks"
-				  nil
-				  (nth 1 server-info)
-				  (nth 2 server-info))))
+    (let ((proc (apply #'open-network-stream "socks" nil
+                       (nth 1 server-info) (nth 2 server-info) kw-args))
 	  (authtype nil)
 	  version)
 
@@ -508,22 +509,32 @@ socks-find-services-entry
   (gethash (downcase service)
 	      (if udp socks-udp-services socks-tcp-services)))
 
-(defun socks-open-network-stream (name buffer host service)
-  (let ((socks-override-functions t))
-    (socks--open-network-stream
-     (lambda (&rest args)
-       (let ((socks-override-functions nil))
-         (apply #'open-network-stream args)))
-     name buffer host service)))
-
-(defun socks--open-network-stream (orig-fun name buffer host service &rest params)
-  (let ((route (and socks-override-functions
-                    (socks-find-route host service))))
-    (if (not route)
-	(apply orig-fun name buffer host service params)
-      ;; FIXME: Obey `params'!
-      (let* ((proc (socks-open-connection route))
-	     (version (process-get proc 'socks-server-protocol))
+(defcustom socks-open-network-stream-function
+  #'socks-open-network-stream-legacy
+  "Function to open a SOCKS connection.
+Called with NAME, BUFFER, HOST, and SERVICE, for compatibility with
+similar functions in the url-gw framework.  May also be passed
+additional keyword args suitable for `make-network-process'."
+  :type '(choice (const :tag "Default fallback-oriented opener.")
+                 (function :tag "User-provided function")))
+
+(defun socks-open-network-stream-legacy (name buffer host service &rest params)
+  "Open a SOCKS connection for a valid route.
+Fall back to non-SOCKS connections for unknown or undesired routes."
+  (if-let* ((route (socks-find-route host service))
+            (proc (apply #'socks-open-connection route params)))
+      (socks--open-network-stream proc buffer host service)
+    ;; Retain legacy behavior and connect anyway without warning
+    (apply #'open-network-stream name buffer host service params)))
+
+(defun socks-open-network-stream (name buffer host service &rest params)
+  "Open a SOCKS connection.  PARAMS are passed to `open-network-stream'."
+  (apply socks-open-network-stream-function name buffer host service params))
+
+(defun socks--open-network-stream (proc buffer host service)
+  (progn ; temporarily preserve git blame for easier reviewing
+    (progn ; could rename to something like `socks--initiate-command-connect'
+      (let* ((version (process-get proc 'socks-server-protocol))
              (atype
               (cond
                ((equal version 4)
-- 
2.35.1


--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
 filename=0002-lisp-url-url-gw.el-url-open-stream-Honor-socks-gatew.patch

From dde4ed3bfdc5cebd4649534efe04b32c488f7b56 Mon Sep 17 00:00:00 2001
From: "F. Jason Park" <jp@HIDDEN>
Date: Sun, 6 Mar 2022 17:14:50 -0800
Subject: [PATCH 2/6] ; * lisp/url/url-gw.el (url-open-stream): Honor socks
 gateway-method

---
 lisp/url/url-gw.el | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/lisp/url/url-gw.el b/lisp/url/url-gw.el
index c4a41f56b3..822cbcb64e 100644
--- a/lisp/url/url-gw.el
+++ b/lisp/url/url-gw.el
@@ -215,6 +215,8 @@ url-open-stream
 Optional arg GATEWAY-METHOD specifies the gateway to be used,
 overriding the value of `url-gateway-method'."
   (unless url-gateway-unplugged
+    (when (eq url-gateway-method 'socks)
+      (setq gateway-method nil))
     (let* ((gwm (or gateway-method url-gateway-method))
            (gw-method (if (and url-gateway-local-host-regexp
                                (not (eq 'tls gwm))
-- 
2.35.1


--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
 filename=0003-Fix-string-encoding-bug-in-socks-tests.patch

From 1af9240dee9fdd2b112d7e1580f4d2ce4bc66321 Mon Sep 17 00:00:00 2001
From: "F. Jason Park" <jp@HIDDEN>
Date: Mon, 14 Feb 2022 02:36:57 -0800
Subject: [PATCH 3/6] Fix string encoding bug in socks tests

* test/lisp/net/socks-tests.el (socks-tests-canned-server-create,
socks-tests-filter-response-parsing-v4): Fix bug in process filter to
prevent prepared outgoing responses from being implicitly encoded as
utf-8.  Fix similar mistake in v4 filter test.  Also allow system
to choose port instead of hard-coding it.
---
 test/lisp/net/socks-tests.el | 26 +++++++++++++++-----------
 1 file changed, 15 insertions(+), 11 deletions(-)

diff --git a/test/lisp/net/socks-tests.el b/test/lisp/net/socks-tests.el
index 461796bdf9..807c926185 100644
--- a/test/lisp/net/socks-tests.el
+++ b/test/lisp/net/socks-tests.el
@@ -63,21 +63,21 @@ socks-tests-filter-response-parsing-v4
     (process-put proc 'socks-state socks-state-waiting)
     (process-put proc 'socks-server-protocol 4)
     (ert-info ("Receive initial incomplete segment")
-      (socks-filter proc (concat [0 90 0 0 93 184 216]))
-      ;; From example.com: OK status ^      ^ msg start
+      (socks-filter proc (unibyte-string 0 90 0 0 93 184 216))
+      ;; From example.com: OK status       ^      ^ msg start
       (ert-info ("State still set to waiting")
         (should (eq (process-get proc 'socks-state) socks-state-waiting)))
       (ert-info ("Response field is nil because processing incomplete")
         (should-not (process-get proc 'socks-response)))
       (ert-info ("Scratch field holds stashed partial payload")
-        (should (string= (concat [0 90 0 0 93 184 216])
+        (should (string= (unibyte-string 0 90 0 0 93 184 216)
                          (process-get proc 'socks-scratch)))))
     (ert-info ("Last part arrives")
       (socks-filter proc "\42") ; ?\" 34
       (ert-info ("State transitions to complete (length check passes)")
         (should (eq (process-get proc 'socks-state) socks-state-connected)))
       (ert-info ("Scratch and response fields hold stash w. last chunk")
-        (should (string= (concat [0 90 0 0 93 184 216 34])
+        (should (string= (unibyte-string 0 90 0 0 93 184 216 34)
                          (process-get proc 'socks-response)))
         (should (string= (process-get proc 'socks-response)
                          (process-get proc 'socks-scratch)))))
@@ -133,17 +133,19 @@ socks-tests-canned-server-patterns
 (defun socks-tests-canned-server-create ()
   "Create and return a fake SOCKS server."
   (let* ((port (nth 2 socks-server))
-         (name (format "socks-server:%d" port))
+         (name (format "socks-server:%s"
+                       (or (numberp port) (ert-test-name (ert-running-test)))))
          (pats socks-tests-canned-server-patterns)
          (filt (lambda (proc line)
                  (pcase-let ((`(,pat . ,resp) (pop pats)))
                    (unless (or (and (vectorp pat) (equal pat (vconcat line)))
                                (string-match-p pat line))
                      (error "Unknown request: %s" line))
+                   (setq resp (apply #'unibyte-string (append resp nil)))
                    (let ((print-escape-control-characters t))
                      (message "[%s] <- %s" name (prin1-to-string line))
                      (message "[%s] -> %s" name (prin1-to-string resp)))
-                   (process-send-string proc (concat resp)))))
+                   (process-send-string proc resp))))
          (serv (make-network-process :server 1
                                      :buffer (get-buffer-create name)
                                      :filter filt
@@ -151,8 +153,10 @@ socks-tests-canned-server-create
                                      :family 'ipv4
                                      :host 'local
                                      :coding 'binary
-                                     :service port)))
+                                     :service (or port t))))
     (set-process-query-on-exit-flag serv nil)
+    (unless (numberp (nth 2 socks-server))
+      (setf (nth 2 socks-server) (process-contact serv :service)))
     serv))
 
 (defvar socks-tests--hello-world-http-request-pattern
@@ -191,7 +195,7 @@ socks-tests-perform-hello-world-http-request
 
 (ert-deftest socks-tests-v4-basic ()
   "Show correct preparation of SOCKS4 connect command (Bug#46342)."
-  (let ((socks-server '("server" "127.0.0.1" 10079 4))
+  (let ((socks-server '("server" "127.0.0.1" t 4))
         (url-user-agent "Test/4-basic")
         (socks-tests-canned-server-patterns
          `(([4 1 0 80 93 184 216 34 ?f ?o ?o 0] . [0 90 0 0 0 0 0 0])
@@ -213,7 +217,7 @@ socks-tests-v4-basic
 (ert-deftest socks-tests-v5-auth-user-pass ()
   "Verify correct handling of SOCKS5 user/pass authentication."
   (should (assq 2 socks-authentication-methods))
-  (let ((socks-server '("server" "127.0.0.1" 10080 5))
+  (let ((socks-server '("server" "127.0.0.1" t 5))
         (socks-username "foo")
         (socks-password "bar")
         (url-user-agent "Test/auth-user-pass")
@@ -247,7 +251,7 @@ socks-tests-v5-auth-user-pass
 (ert-deftest socks-tests-v5-auth-user-pass-blank ()
   "Verify correct SOCKS5 user/pass authentication with empty pass."
   (should (assq 2 socks-authentication-methods))
-  (let ((socks-server '("server" "127.0.0.1" 10081 5))
+  (let ((socks-server '("server" "127.0.0.1" t 5))
         (socks-username "foo") ; defaults to (user-login-name)
         (socks-password "") ; simulate user hitting enter when prompted
         (url-user-agent "Test/auth-user-pass-blank")
@@ -266,7 +270,7 @@ socks-tests-v5-auth-user-pass-blank
 
 (ert-deftest socks-tests-v5-auth-none ()
   "Verify correct handling of SOCKS5 when auth method 0 requested."
-  (let ((socks-server '("server" "127.0.0.1" 10082 5))
+  (let ((socks-server '("server" "127.0.0.1" t 5))
         (socks-authentication-methods (append socks-authentication-methods
                                               nil))
         (url-user-agent "Test/auth-none")
-- 
2.35.1


--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
 filename=0004-Add-support-for-SOCKS-4a.patch

From 55702321a8b17914ff577b5e7fc426ffb7ff0462 Mon Sep 17 00:00:00 2001
From: "F. Jason Park" <jp@HIDDEN>
Date: Mon, 14 Feb 2022 02:36:57 -0800
Subject: [PATCH 4/6] Add support for SOCKS 4a

* lisp/net/socks.el (socks-server): Add new choice `4a' to version
field of option.  This may appear to change the type of the field from
a number to a union of symbols and numbers.  However,
`socks-send-command' and `socks-filter' already expect a possible
`http' value for this field (also a symbol).
(socks--errors-4): Add new constant containing error messages for
socks version 4.  The semantics are faithful, but the wording is
ad-libbed.
(socks-send-command): Massage existing handling for version 4 to
accommodate 4a.

* test/lisp/net/socks-tests.el (socks-tests-v4a-basic): add test for
4a.
Bug#53941
---
 lisp/net/socks.el            | 22 ++++++++++++++++++++--
 test/lisp/net/socks-tests.el | 13 +++++++++++++
 2 files changed, 33 insertions(+), 2 deletions(-)

diff --git a/lisp/net/socks.el b/lisp/net/socks.el
index fe66a94d18..73afcc38d3 100644
--- a/lisp/net/socks.el
+++ b/lisp/net/socks.el
@@ -162,6 +162,7 @@ socks-server
 	  (radio-button-choice :tag "SOCKS Version"
 			       :format "%t: %v"
 			       (const :tag "SOCKS v4  " :format "%t" :value 4)
+                               (const :tag "SOCKS v4a"  :format "%t" :value 4a)
 			       (const :tag "SOCKS v5"   :format "%t" :value 5))))
 
 
@@ -202,6 +203,12 @@ socks-errors
     "Command not supported"
     "Address type not supported"))
 
+(defconst socks--errors-4
+  '("Granted"
+    "Rejected or failed"
+    "Cannot connect to identd on the client"
+    "Client and identd report differing user IDs"))
+
 ;; The socks v5 address types
 (defconst socks-address-type-v4   1)
 (defconst socks-address-type-name 3)
@@ -401,6 +408,7 @@ socks-send-command
 		(format "%c%s" (length address) address))
 	       (t
 		(error "Unknown address type: %d" atype))))
+        trailing
 	request version)
     (or (process-get proc 'socks)
         (error "socks-send-command called on non-SOCKS connection %S" proc))
@@ -418,6 +426,12 @@ socks-send-command
 			     (t
 			      (error "Unsupported address type for HTTP: %d" atype)))
 			    port)))
+     ((when (eq version '4a)
+        (setf addr "\0\0\0\1"
+              trailing (concat address "\0")
+              version 4 ; done with the "a" part
+              (process-get proc 'socks-server-protocol) 4)
+        nil)) ; fall through
      ((equal version 4)
       (setq request (concat
 		     (unibyte-string
@@ -427,7 +441,8 @@ socks-send-command
 		      (logand port #xff)) ; port, low byte
 		     addr                 ; address
 		     (user-full-name)     ; username
-		     "\0")))              ; terminate username
+                     "\0"                 ; terminate username
+                     trailing)))          ; optional host to look up
      ((equal version 5)
       (setq request (concat
 		     (unibyte-string
@@ -448,7 +463,10 @@ socks-send-command
 	nil				; Sweet sweet success!
       (delete-process proc)
       (error "SOCKS: %s"
-             (nth (or (process-get proc 'socks-reply) 1) socks-errors)))
+             (let ((no (or (process-get proc 'socks-reply) 1)))
+               (if (eq version 5)
+                   (nth no socks-errors)
+                 (nth (- no 90) socks--errors-4)))))
     proc))
 
 
diff --git a/test/lisp/net/socks-tests.el b/test/lisp/net/socks-tests.el
index 807c926185..a0191d9341 100644
--- a/test/lisp/net/socks-tests.el
+++ b/test/lisp/net/socks-tests.el
@@ -210,6 +210,19 @@ socks-tests-v4-basic
                  (lambda (&optional _) "foo")))
         (socks-tests-perform-hello-world-http-request)))))
 
+(ert-deftest socks-tests-v4a-basic ()
+  "Show correct preparation of SOCKS4a connect command."
+  (let ((socks-server '("server" "127.0.0.1" t 4a))
+        (url-user-agent "Test/4a-basic")
+        (socks-tests-canned-server-patterns
+         `(([4 1 0 80 0 0 0 1 ?f ?o ?o 0 ?e ?x ?a ?m ?p ?l ?e ?. ?c ?o ?m 0]
+            . [0 90 0 0 0 0 0 0])
+           ,socks-tests--hello-world-http-request-pattern)))
+    (ert-info ("Make HTTP request over SOCKS4A")
+      (cl-letf (((symbol-function 'user-full-name)
+                 (lambda (&optional _) "foo")))
+        (socks-tests-perform-hello-world-http-request)))))
+
 ;; Replace first pattern below with ([5 3 0 1 2] . [5 2]) to validate
 ;; against curl 7.71 with the following options:
 ;; $ curl --verbose -U foo:bar --proxy socks5h://127.0.0.1:10080 example.com
-- 
2.35.1


--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
 filename=0005-Support-SOCKS-resolve-extension.patch

From a26bf29fb9363d4face0049dcf5ec3d353c799ac Mon Sep 17 00:00:00 2001
From: "F. Jason Park" <jp@HIDDEN>
Date: Mon, 14 Feb 2022 02:36:57 -0800
Subject: [PATCH 5/6] Support SOCKS resolve extension

* lisp/net/socks.el (socks-resolve-command): Add new constant for the
SOCKS command RESOLVE, which comes by way of a nonstandard extension
from the TOR project.  It mirrors CONNECT in most respects but asks
the server to RESOLVE a host name and return its IP.  For details, see
https://github.com/torproject/torspec/blob/master/socks-extensions.txt
This shouldn't be confused with 5h/5-hostname, which is used to by
clients like cURL to allow users to bypass attempts to resolve a name
locally.
(socks--extract-resolve-response, socks-tor-resolve): Add utility
functions to query a SOCKS service supporting the RESOLVE extension.
(socks--tor-resolve, socks-tor-resolve): Provide internal function to
perform resolve command as well as a partial drop-in replacement for
`network-lookup-address-info'.
(socks-filter): Allow for a null type field on error with version 5.
Bug#53941
---
 lisp/net/socks.el            | 70 +++++++++++++++++++++++++++++++++++-
 test/lisp/net/socks-tests.el | 65 +++++++++++++++++++++++++++++++++
 2 files changed, 134 insertions(+), 1 deletion(-)

diff --git a/lisp/net/socks.el b/lisp/net/socks.el
index 73afcc38d3..9ce23b517e 100644
--- a/lisp/net/socks.el
+++ b/lisp/net/socks.el
@@ -181,6 +181,9 @@ socks-udp-associate-command
 (defconst socks-authentication-null 0)
 (defconst socks-authentication-failure 255)
 
+;; Extensions
+(defconst socks-resolve-command #xf0)
+
 ;; Response codes
 (defconst socks-response-success               0)
 (defconst socks-response-general-failure       1)
@@ -316,7 +319,8 @@ socks-filter
 		     ((pred (= socks-address-type-name))
 		      (if (< (length string) 5)
 			  255
-		        (+ 1 (aref string 4)))))))
+                        (+ 1 (aref string 4))))
+                     (0 0))))
 	  (if (< (length string) desired-len)
 	      nil			; Need to spin some more
 	    (process-put proc 'socks-state socks-state-connected)
@@ -664,6 +668,70 @@ socks-nslookup-host
 	res)
     host))
 
+(defun socks--extract-resolve-response (proc)
+  "Parse response for PROC and maybe return destination IP address."
+  (let ((response (process-get proc 'socks-response)))
+    (cl-assert response) ; otherwise, msg not received in its entirety
+    (pcase (process-get proc 'socks-server-protocol)
+      (4 ; https://www.openssh.com/txt/socks4a.protocol
+       (when-let (((zerop (process-get proc 'socks-reply)))
+                  ((eq (aref response 1) 90)) ; #x5a request granted
+                  (a (substring response 4)) ; ignore port for now
+                  ((not (string-empty-p a)))
+                  ((not (string= a "\0\0\0\0"))))
+         a))
+      (5 ; https://tools.ietf.org/html/rfc1928
+       (cl-assert (eq 5 (aref response 0)) t)
+       (pcase (aref response 3) ; ATYP
+         (1 (and-let* ((a (substring response 4 8))
+                       ((not (string= a "\0\0\0\0")))
+                       a)))
+         ;; No reason to support RESOLVE_PTR [F1] extension, right?
+         (3 (let ((len (1- (aref response 4))))
+              (substring response 5 (+ 5 len))))
+         (4 (substring response 4 20)))))))
+
+(declare-function puny-encode-domain "puny" (domain))
+
+(defun socks--tor-resolve (host)
+  (let ((socks-password (or socks-password ""))
+        (route (socks-find-route host nil))
+        proc
+        ip)
+    (cl-assert route)
+    ;; "Host unreachable" may be raised when the lookup fails
+    (unwind-protect
+        (progn
+          (setq proc (socks-open-connection route))
+          (socks-send-command proc
+                              socks-resolve-command
+                              socks-address-type-name
+                              host
+                              0)
+          (setq ip (socks--extract-resolve-response proc)))
+      (when proc
+        (delete-process proc)))
+    ip))
+
+(defun socks-tor-resolve (name &optional _family)
+  "Return list of one IPv4 address for domain NAME.
+See `network-lookup-address-info' for format of return value.  Return
+nil on failure.
+
+SOCKS server must support the Tor RESOLVE command.  Note that using this
+in place of `network-lookup-address-info' may not be enough to prevent a
+DNS leak.  For example, see `url-gateway-broken-resolution'."
+  (unless (string-match "\\`[[:ascii:]]+\\'" name)
+    (require 'puny)
+    (setq name (puny-encode-domain name)))
+  (condition-case err
+      (when-let ((ip (socks--tor-resolve name)))
+        (list (vconcat ip [0])))
+    (error
+     (unless (member (cadr err)
+                     '("SOCKS: Host unreachable" "SOCKS: Rejected or failed"))
+       (signal (car err) (cdr err))))))
+
 (provide 'socks)
 
 ;;; socks.el ends here
diff --git a/test/lisp/net/socks-tests.el b/test/lisp/net/socks-tests.el
index a0191d9341..077b80cb0b 100644
--- a/test/lisp/net/socks-tests.el
+++ b/test/lisp/net/socks-tests.el
@@ -298,4 +298,69 @@ socks-tests-v5-auth-none
       (socks-tests-perform-hello-world-http-request)))
   (should (assq 2 socks-authentication-methods)))
 
+(ert-deftest tor-resolve-4a ()
+  "Make request to TOR resolve service over SOCKS4a"
+  (let* ((socks-server '("server" "127.0.0.1" t 4a))
+         (socks-tests-canned-server-patterns
+          '(([4 #xf0 0 0 0 0 0 1 ?f ?o ?o 0 ?e ?x ?a ?m ?p ?l ?e ?. ?c ?o ?m 0]
+             . [0 90 0 0 93 184 216 34])))
+         (inhibit-message noninteractive)
+         (server (socks-tests-canned-server-create)))
+    (ert-info ("Query TOR RESOLVE service over SOCKS4")
+      (cl-letf (((symbol-function 'user-full-name)
+                 (lambda (&optional _) "foo")))
+        (should (equal '([93 184 216 34 0])
+                       (socks-tor-resolve "example.com")))))
+    (kill-buffer (process-buffer server))
+    (delete-process server)))
+
+(ert-deftest tor-resolve-4a-fail ()
+  (let* ((socks-server '("server" "127.0.0.1" t 4a))
+         (socks-tests-canned-server-patterns
+          '(([4 #xf0 0 0 0 0 0 1 ?f ?o ?o 0 ?e ?x ?a ?m ?p ?l ?e ?. ?c ?o ?m 0]
+             . [0 91 0 0 0 0 0 0])))
+         (inhibit-message noninteractive)
+         (server (socks-tests-canned-server-create)))
+    (ert-info ("Query TOR RESOLVE service over SOCKS4")
+      (cl-letf (((symbol-function 'user-full-name)
+                 (lambda (&optional _) "foo")))
+        (should-not (socks-tor-resolve "example.com"))))
+    (kill-buffer (process-buffer server))
+    (delete-process server)))
+
+(ert-deftest tor-resolve-5-fail ()
+  (let* ((socks-server '("server" "127.0.0.1" t 5))
+         (socks-username "")
+         (socks-authentication-methods (copy-sequence
+                                        socks-authentication-methods))
+         (inhibit-message noninteractive)
+         (socks-tests-canned-server-patterns
+          '(([5 2 0 2] . [5 2])
+            ([1 0 0] . [1 0])
+            ([5 #xf0 0 3 11 ?e ?x ?a ?m ?p ?l ?e ?. ?c ?o ?m 0 0]
+             . [5 4 0 0 0 0 0 0 0 0])))
+         (server (socks-tests-canned-server-create)))
+    (ert-info ("Query TOR RESOLVE service over SOCKS5")
+      (should-not (socks-tor-resolve "example.com")))
+    (kill-buffer (process-buffer server))
+    (delete-process server)))
+
+(ert-deftest tor-resolve-5 ()
+  "Make request to TOR resolve service over SOCKS5"
+  (let* ((socks-server '("server" "127.0.0.1" t 5))
+         (socks-username "foo")
+         (socks-authentication-methods (append socks-authentication-methods
+                                               nil))
+         (inhibit-message noninteractive)
+         (socks-tests-canned-server-patterns
+          '(([5 2 0 2] . [5 2])
+            ([1 3 ?f ?o ?o 0] . [1 0])
+            ([5 #xf0 0 3 11 ?e ?x ?a ?m ?p ?l ?e ?. ?c ?o ?m 0 0]
+             . [5 0 0 1 93 184 216 34 0 0])))
+         (server (socks-tests-canned-server-create)))
+    (ert-info ("Query TOR RESOLVE service over SOCKS5")
+      (should (equal '([93 184 216 34 0]) (socks-tor-resolve "example.com"))))
+    (kill-buffer (process-buffer server))
+    (delete-process server)))
+
 ;;; socks-tests.el ends here
-- 
2.35.1


--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
 filename=0006-POC-Demo-SOCKS-resolve-with-HTTPS.patch

From 52a7f3269992166074ebe277f6905c219885d7cf Mon Sep 17 00:00:00 2001
From: "F. Jason Park" <jp@HIDDEN>
Date: Mon, 14 Feb 2022 02:36:57 -0800
Subject: [PATCH 6/6] [POC] Demo SOCKS resolve with HTTPS

* test/lisp/net/socks-test.el (test-socks-https-poc): Provide
throwaway test demoing an HTTPS connection over a TOR proxy service.
---
 test/lisp/net/socks-tests.el | 64 +++++++++++++++++++++++++++++++++++-
 1 file changed, 63 insertions(+), 1 deletion(-)

diff --git a/test/lisp/net/socks-tests.el b/test/lisp/net/socks-tests.el
index 077b80cb0b..0c58fcc863 100644
--- a/test/lisp/net/socks-tests.el
+++ b/test/lisp/net/socks-tests.el
@@ -21,7 +21,7 @@
 
 ;;; Code:
 
-(require 'ert)
+(require 'ert-x)
 (require 'socks)
 (require 'url-http)
 
@@ -363,4 +363,66 @@ tor-resolve-5
     (kill-buffer (process-buffer server))
     (delete-process server)))
 
+(defvar test-socks-service ; "127.0.0.1:1080" -> ("127.0.0.1", 1080)
+  (when-let ((present (getenv "TEST_SOCKS_SERVICE"))
+             (parts (split-string present ":")))
+    (list (car parts) (string-to-number (cadr parts)))))
+
+(declare-function gnutls-negotiate "gnutls"
+                  (&rest spec
+                         &key process type hostname priority-string
+                         trustfiles crlfiles keylist min-prime-bits
+                         verify-flags verify-error verify-hostname-error
+                         &allow-other-keys))
+
+(ert-deftest test-socks-resolve-fail ()
+  :tags '(:unstable)
+  (unless test-socks-service (ert-skip "SOCKS service missing"))
+  (let* ((socks-server `("tor" ,@test-socks-service 5)) ; also try 4a
+         (socks-username "")
+         (socks-password ""))
+    (ert-info ("Connect to HTTP endpoint over Tor SOCKS proxy")
+      (should-not (socks-tor-resolve "test-socks-resolve-fail--fake.com")))))
+
+(ert-deftest test-socks-https-poc ()
+  :tags '(:unstable)
+  (unless test-socks-service (ert-skip "SOCKS service missing"))
+  (unless (gnutls-available-p) (ert-skip "SOCKS resolve test needs GNUTLS"))
+  (ert-with-temp-file tempfile
+    :prefix "emacs-test-socks-network-security-"
+    (let* ((socks-server `("tor" ,@test-socks-service 5))
+           (socks-username "user")
+           (socks-password "")
+           (nsm-settings-file tempfile)
+           (url-gateway-method 'socks)
+           (id "sha1:df77269389e537fcc9a5fe61667133b5bb97d42e")
+           (host "check.torproject.org")
+           (url (url-generic-parse-url "https://check.torproject.org"))
+           ;;
+           done
+           ;;
+           (cb (lambda (&rest _r)
+                 (goto-char (point-min))
+                 (should (search-forward "Congratulations" nil t))
+                 (setq done t)))
+           (socks-open-network-stream-function
+            (lambda (&rest rest)
+              (let ((proc (apply #'socks-open-network-stream-legacy rest)))
+                (gnutls-negotiate :process proc :hostname host)
+                (should (nsm-verify-connection proc host 443 t))))))
+      (ert-info ("Connect to HTTPS endpoint over Tor SOCKS proxy")
+        (unwind-protect
+            (progn
+              (advice-add 'network-lookup-address-info :override
+                          #'socks-tor-resolve)
+              (should-not (nsm-host-settings id))
+              (url-https url cb '(nil))
+              (ert-info ("Wait for response")
+                (with-timeout (3 (error "Request timed out"))
+                  (unless done
+                    (sleep-for 0.1))))
+              (should (nsm-host-settings id)))
+          (advice-remove 'network-lookup-address-info
+                         #'socks-tor-resolve))))))
+
 ;;; socks-tests.el ends here
-- 
2.35.1


--=-=-=--

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#53941: Last-minute socks.el improvements for Emacs 29?
Resent-From: "J.P." <jp@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Mon, 28 Nov 2022 15:31:01 +0000
Resent-Message-ID: <handler.53941.B53941.166964943320943 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 53941
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: patch
To: 53941 <at> debbugs.gnu.org
Cc: Lars Ingebrigtsen <larsi@HIDDEN>, Eli Zaretskii <eliz@HIDDEN>, Jacobo <gnuhacker@HIDDEN>
Received: via spool by 53941-submit <at> debbugs.gnu.org id=B53941.166964943320943
          (code B ref 53941); Mon, 28 Nov 2022 15:31:01 +0000
Received: (at 53941) by debbugs.gnu.org; 28 Nov 2022 15:30:33 +0000
Received: from localhost ([127.0.0.1]:49248 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1ozg5X-0005Rj-K2
	for submit <at> debbugs.gnu.org; Mon, 28 Nov 2022 10:30:33 -0500
Received: from mail-108-mta178.mxroute.com ([136.175.108.178]:42273)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <jp@HIDDEN>) id 1ozg5U-0005Rd-Mo
 for 53941 <at> debbugs.gnu.org; Mon, 28 Nov 2022 10:30:30 -0500
Received: from mail-111-mta2.mxroute.com ([136.175.111.2]
 filter006.mxroute.com) (Authenticated sender: mN4UYu2MZsgR)
 by mail-108-mta178.mxroute.com (ZoneMTA) with ESMTPSA id
 184bedc4e410006e99.001 for <53941 <at> debbugs.gnu.org>
 (version=TLSv1/SSLv3 cipher=ECDHE-RSA-AES128-GCM-SHA256);
 Mon, 28 Nov 2022 15:30:19 +0000
X-Zone-Loop: 962f21df90783e7c670a4ca4b792a716771bface1572
X-Originating-IP: [136.175.111.2]
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=neverwas.me
 ; s=x;
 h=Content-Type:MIME-Version:Message-ID:Date:References:In-Reply-To:
 Subject:Cc:To:From:Sender:Reply-To:Content-Transfer-Encoding:Content-ID:
 Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc
 :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe:
 List-Post:List-Owner:List-Archive;
 bh=2KoYb1D2qmC8gohDBCPcRH0uPn3l051TjUXTiwULhWk=; b=goex4gYY4fmryC9Swd6RklohLq
 PQXAv1Gv8h1lhdikEsLygY7cQ8VqqqBAtftYScXdod+gN6vKTg8M6AMNNmCu08V/JZTLqlw+wreU3
 pHJPxHusXcPAHgkvhMFQ3t1LK+4y9k/arunSxZsC+m5/I1+ckWp4jkNwrMjjmSf1QkSDPQd0jBpkm
 I/aZE8QYi43EMWfer1KzQjYrM9ol1uSTzdh6ZycU93IUh2WY+zjRRLJtv1QiRyaJKZ2tHXLRUtzO4
 3vqe2e0DzjgsDLju5DoJYW5Gla4RCULbSIS34nAniE8KH2fCFhrfwJd11F4lXGr1WK1XhsGTP9JeU
 SNiqys5Q==;
From: "J.P." <jp@HIDDEN>
In-Reply-To: <87lexikwu5.fsf@HIDDEN> (J. P.'s message of "Thu, 10 Mar
 2022 00:58:42 -0800")
References: <87pmntfym7.fsf@HIDDEN> <8735kl1v58.fsf@HIDDEN>
 <87a6emftzx.fsf@HIDDEN> <87k0do5km1.fsf@HIDDEN>
 <87pmn5n3tu.fsf@HIDDEN> <87mti99j1f.fsf@HIDDEN>
 <87wnh7hkgi.fsf@HIDDEN> <87pmmz947k.fsf@HIDDEN>
 <8735ju44sk.fsf@HIDDEN> <87lexikwu5.fsf@HIDDEN>
Date: Mon, 28 Nov 2022 07:30:16 -0800
Message-ID: <87mt8baygn.fsf_-_@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="=-=-="
X-Authenticated-Id: masked@HIDDEN
X-Spam-Score: 0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

--=-=-=
Content-Type: text/plain

Hi people, maintainers,

I've lifted some fixes and minor enhancements from my POC stuff posted
to this thread earlier this year. Nothing is directly Tor related, so I
can create a new bug report, if necessary.

The second patch fixes a problem involving SOCKS 5 error handling. It
also adds support for SOCKS 4a, which allows tools that don't speak
SOCKS 5, like socat, to resolve host names. The third addresses a couple
FIXMEs but no bugs, strictly speaking. The fourth is just a demo [1].
Happy to explain whatever in detail.

Thanks,
J.P.


[1] The fourth patch demos a possible approach for tightening the
    integration between socks and url-proxy, but it's not fit for
    inclusion in Emacs 29. To try it out with Tor, do something like

      (setq url-proxy-services '(("https" . "socks5h://127.0.0.1:9050"))
            socks-username "foo"
            socks-password "")

    followed by an M-x eww RET https://check.torproject.org RET. (Note
    that this still leaks DNS.)


--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
 filename=0001-Don-t-hard-code-server-ports-in-SOCKS-tests.patch

From 0780339ceee3b0068700f5a3bf6d48aa4023915e Mon Sep 17 00:00:00 2001
From: "F. Jason Park" <jp@HIDDEN>
Date: Mon, 14 Feb 2022 02:36:57 -0800
Subject: [PATCH 1/4] ; Don't hard code server ports in SOCKS tests

* test/lisp/net/socks-tests.el (socks-tests-canned-server-create,
socks-tests-filter-response-parsing-v4, socks-tests-v4-basic,
socks-tests-v5-auth-user-pass, socks-tests-v5-auth-user-blank,
socks-tests-v5-auth-none): Fix bug in process filter to prevent
prepared outgoing responses from being implicitly encoded as utf-8.
Fix similar mistake in v4 filter test.  Also allow system to choose
port instead of hard-coding it.
---
 test/lisp/net/socks-tests.el | 26 +++++++++++++++-----------
 1 file changed, 15 insertions(+), 11 deletions(-)

diff --git a/test/lisp/net/socks-tests.el b/test/lisp/net/socks-tests.el
index 461796bdf9..f1ecf1630f 100644
--- a/test/lisp/net/socks-tests.el
+++ b/test/lisp/net/socks-tests.el
@@ -63,21 +63,21 @@ socks-tests-filter-response-parsing-v4
     (process-put proc 'socks-state socks-state-waiting)
     (process-put proc 'socks-server-protocol 4)
     (ert-info ("Receive initial incomplete segment")
-      (socks-filter proc (concat [0 90 0 0 93 184 216]))
-      ;; From example.com: OK status ^      ^ msg start
+      (socks-filter proc (unibyte-string 0 90 0 0 93 184 216))
+      ;; From example.com: OK status       ^      ^ msg start
       (ert-info ("State still set to waiting")
         (should (eq (process-get proc 'socks-state) socks-state-waiting)))
       (ert-info ("Response field is nil because processing incomplete")
         (should-not (process-get proc 'socks-response)))
       (ert-info ("Scratch field holds stashed partial payload")
-        (should (string= (concat [0 90 0 0 93 184 216])
+        (should (string= (unibyte-string 0 90 0 0 93 184 216)
                          (process-get proc 'socks-scratch)))))
     (ert-info ("Last part arrives")
       (socks-filter proc "\42") ; ?\" 34
       (ert-info ("State transitions to complete (length check passes)")
         (should (eq (process-get proc 'socks-state) socks-state-connected)))
       (ert-info ("Scratch and response fields hold stash w. last chunk")
-        (should (string= (concat [0 90 0 0 93 184 216 34])
+        (should (string= (unibyte-string 0 90 0 0 93 184 216 34)
                          (process-get proc 'socks-response)))
         (should (string= (process-get proc 'socks-response)
                          (process-get proc 'socks-scratch)))))
@@ -133,17 +133,19 @@ socks-tests-canned-server-patterns
 (defun socks-tests-canned-server-create ()
   "Create and return a fake SOCKS server."
   (let* ((port (nth 2 socks-server))
-         (name (format "socks-server:%d" port))
+         (name (format "socks-server:%s"
+                       (if (numberp port) port (ert-test-name (ert-running-test)))))
          (pats socks-tests-canned-server-patterns)
          (filt (lambda (proc line)
                  (pcase-let ((`(,pat . ,resp) (pop pats)))
                    (unless (or (and (vectorp pat) (equal pat (vconcat line)))
                                (string-match-p pat line))
                      (error "Unknown request: %s" line))
+                   (setq resp (apply #'unibyte-string (append resp nil)))
                    (let ((print-escape-control-characters t))
                      (message "[%s] <- %s" name (prin1-to-string line))
                      (message "[%s] -> %s" name (prin1-to-string resp)))
-                   (process-send-string proc (concat resp)))))
+                   (process-send-string proc resp))))
          (serv (make-network-process :server 1
                                      :buffer (get-buffer-create name)
                                      :filter filt
@@ -151,8 +153,10 @@ socks-tests-canned-server-create
                                      :family 'ipv4
                                      :host 'local
                                      :coding 'binary
-                                     :service port)))
+                                     :service (or port t))))
     (set-process-query-on-exit-flag serv nil)
+    (unless (numberp (nth 2 socks-server))
+      (setf (nth 2 socks-server) (process-contact serv :service)))
     serv))
 
 (defvar socks-tests--hello-world-http-request-pattern
@@ -191,7 +195,7 @@ socks-tests-perform-hello-world-http-request
 
 (ert-deftest socks-tests-v4-basic ()
   "Show correct preparation of SOCKS4 connect command (Bug#46342)."
-  (let ((socks-server '("server" "127.0.0.1" 10079 4))
+  (let ((socks-server '("server" "127.0.0.1" t 4))
         (url-user-agent "Test/4-basic")
         (socks-tests-canned-server-patterns
          `(([4 1 0 80 93 184 216 34 ?f ?o ?o 0] . [0 90 0 0 0 0 0 0])
@@ -213,7 +217,7 @@ socks-tests-v4-basic
 (ert-deftest socks-tests-v5-auth-user-pass ()
   "Verify correct handling of SOCKS5 user/pass authentication."
   (should (assq 2 socks-authentication-methods))
-  (let ((socks-server '("server" "127.0.0.1" 10080 5))
+  (let ((socks-server '("server" "127.0.0.1" t 5))
         (socks-username "foo")
         (socks-password "bar")
         (url-user-agent "Test/auth-user-pass")
@@ -247,7 +251,7 @@ socks-tests-v5-auth-user-pass
 (ert-deftest socks-tests-v5-auth-user-pass-blank ()
   "Verify correct SOCKS5 user/pass authentication with empty pass."
   (should (assq 2 socks-authentication-methods))
-  (let ((socks-server '("server" "127.0.0.1" 10081 5))
+  (let ((socks-server '("server" "127.0.0.1" t 5))
         (socks-username "foo") ; defaults to (user-login-name)
         (socks-password "") ; simulate user hitting enter when prompted
         (url-user-agent "Test/auth-user-pass-blank")
@@ -266,7 +270,7 @@ socks-tests-v5-auth-user-pass-blank
 
 (ert-deftest socks-tests-v5-auth-none ()
   "Verify correct handling of SOCKS5 when auth method 0 requested."
-  (let ((socks-server '("server" "127.0.0.1" 10082 5))
+  (let ((socks-server '("server" "127.0.0.1" t 5))
         (socks-authentication-methods (append socks-authentication-methods
                                               nil))
         (url-user-agent "Test/auth-none")
-- 
2.38.1


--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
 filename=0002-Improve-SOCKS-error-handling-and-add-support-for-4a.patch

From 2de287eac55c577001ac5470e57f1a2ed80c5faf Mon Sep 17 00:00:00 2001
From: "F. Jason Park" <jp@HIDDEN>
Date: Mon, 14 Feb 2022 02:36:57 -0800
Subject: [PATCH 2/4] Improve SOCKS error handling and add support for 4a

* lisp/net/socks.el (socks-server): Add new choice `4a' to version
field of option.  This may appear to change the type of the field from
a number to a union of symbols and numbers.  However,
`socks-send-command' and `socks-filter' already expect a possible
`http' value for this field (also a symbol).
(socks--errors-4): Add new constant containing error messages for
version 4.  The semantics are faithful to the spec, but the exact
wording is adapted.
(socks-filter): Allow for a null "type" field on error with version 5.
In some cases, errors from certain servers were inaccessible.
(socks-connect-function): New option for specifying an
`open-network-stream'-like connect function.
(socks-open-connection): Accept additional `open-network-stream'
params passed on to opener.
(socks-send-command): Massage existing handling for version 4 to
accommodate 4a.

* test/lisp/net/socks-tests.el (socks-tests-v4a-basic): Add test for
SOCKS version 4a.  (Bug#53941.)
---
 lisp/net/socks.el            | 50 +++++++++++++++++++++++++++++-------
 test/lisp/net/socks-tests.el | 13 ++++++++++
 2 files changed, 54 insertions(+), 9 deletions(-)

diff --git a/lisp/net/socks.el b/lisp/net/socks.el
index 2ba1c20566..b9af2aa06e 100644
--- a/lisp/net/socks.el
+++ b/lisp/net/socks.el
@@ -162,6 +162,7 @@ socks-server
 	  (radio-button-choice :tag "SOCKS Version"
 			       :format "%t: %v"
 			       (const :tag "SOCKS v4  " :format "%t" :value 4)
+                               (const :tag "SOCKS v4a"  :format "%t" :value 4a)
 			       (const :tag "SOCKS v5"   :format "%t" :value 5))))
 
 
@@ -202,6 +203,12 @@ socks-errors
     "Command not supported"
     "Address type not supported"))
 
+(defconst socks--errors-4
+  '("Granted"
+    "Rejected or failed"
+    "Cannot connect to identd on the client"
+    "Client and identd report differing user IDs"))
+
 ;; The socks v5 address types
 (defconst socks-address-type-v4   1)
 (defconst socks-address-type-name 3)
@@ -309,7 +316,8 @@ socks-filter
 		     ((pred (= socks-address-type-name))
 		      (if (< (length string) 5)
 			  255
-		        (+ 1 (aref string 4)))))))
+                        (+ 1 (aref string 4))))
+                     (0 0))))
 	  (if (< (length string) desired-len)
 	      nil			; Need to spin some more
 	    (process-put proc 'socks-state socks-state-connected)
@@ -327,15 +335,27 @@ socks-override-functions
 (when socks-override-functions
   (advice-add 'open-network-stream :around #'socks--open-network-stream))
 
-(defun socks-open-connection (server-info)
+(defcustom socks-connect-function 'open-network-stream
+  "Function to open a network connection to a SOCKS provider.
+Called with arguments suitable for `open-network-stream'."
+  :version "29.1"
+  :type '(choice (function-item :value open-network-stream)
+                 (function :tag "User-provided function")))
+
+(defun socks-open-connection (server-info &rest stream-params)
+  "Create and initialize a SOCKS process.
+Perform authentication if needed.  Expect SERVER-INFO to take the
+form of `socks-server' and STREAM-PARAMS to be keyword params
+accepted by `open-network-stream'."
   (interactive)
+  (unless (plist-member stream-params :coding)
+    (setf (plist-get stream-params :coding) '(binary . binary)))
   (save-excursion
     (let ((proc
-           (let ((socks-override-functions nil))
-             (open-network-stream "socks"
-				  nil
-				  (nth 1 server-info)
-				  (nth 2 server-info))))
+           (with-suppressed-warnings ((obsolete socks-override-functions))
+             (let ((socks-override-functions nil))
+               (apply socks-connect-function (nth 0 server-info) nil
+                      (nth 1 server-info) (nth 2 server-info) stream-params))))
 	  (authtype nil)
 	  version)
 
@@ -400,6 +420,7 @@ socks-send-command
 		(format "%c%s" (length address) address))
 	       (t
 		(error "Unknown address type: %d" atype))))
+        trailing
 	request version)
     (or (process-get proc 'socks)
         (error "socks-send-command called on non-SOCKS connection %S" proc))
@@ -416,6 +437,12 @@ socks-send-command
 			     (t
 			      (error "Unsupported address type for HTTP: %d" atype)))
 			    port)))
+     ((when (eq version '4a)
+        (setf addr "\0\0\0\1"
+              trailing (concat address "\0")
+              version 4 ; done with the "a" part
+              (process-get proc 'socks-server-protocol) 4)
+        nil)) ; fall through
      ((equal version 4)
       (setq request (concat
 		     (unibyte-string
@@ -425,7 +452,8 @@ socks-send-command
 		      (logand port #xff)) ; port, low byte
 		     addr                 ; address
 		     (user-full-name)     ; username
-		     "\0")))              ; terminate username
+                     "\0"                 ; terminate username
+                     trailing)))          ; optional host to look up
      ((equal version 5)
       (setq request (concat
 		     (unibyte-string
@@ -446,7 +474,11 @@ socks-send-command
 	nil				; Sweet sweet success!
       (delete-process proc)
       (error "SOCKS: %s"
-             (nth (or (process-get proc 'socks-reply) 1) socks-errors)))
+             (let ((no (or (process-get proc 'socks-reply) 99)))
+               (or (if (eq version 5) ; 99 - 90 >= length(errors)
+                       (nth no socks-errors)
+                     (nth (- no 90) socks--errors-4))
+                   "Unknown error"))))
     proc))
 
 
diff --git a/test/lisp/net/socks-tests.el b/test/lisp/net/socks-tests.el
index f1ecf1630f..9e341ebd4d 100644
--- a/test/lisp/net/socks-tests.el
+++ b/test/lisp/net/socks-tests.el
@@ -210,6 +210,19 @@ socks-tests-v4-basic
                  (lambda (&optional _) "foo")))
         (socks-tests-perform-hello-world-http-request)))))
 
+(ert-deftest socks-tests-v4a-basic ()
+  "Show correct preparation of SOCKS4a connect command."
+  (let ((socks-server '("server" "127.0.0.1" t 4a))
+        (url-user-agent "Test/4a-basic")
+        (socks-tests-canned-server-patterns
+         `(([4 1 0 80 0 0 0 1 ?f ?o ?o 0 ?e ?x ?a ?m ?p ?l ?e ?. ?c ?o ?m 0]
+            . [0 90 0 0 0 0 0 0])
+           ,socks-tests--hello-world-http-request-pattern)))
+    (ert-info ("Make HTTP request over SOCKS4A")
+      (cl-letf (((symbol-function 'user-full-name)
+                 (lambda (&optional _) "foo")))
+        (socks-tests-perform-hello-world-http-request)))))
+
 ;; Replace first pattern below with ([5 3 0 1 2] . [5 2]) to validate
 ;; against curl 7.71 with the following options:
 ;; $ curl --verbose -U foo:bar --proxy socks5h://127.0.0.1:10080 example.com
-- 
2.38.1


--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
 filename=0003-Simplify-network-stream-openers-in-socks.el.patch

From e0593f5be91e27541a9a13bad9632696fceb9f2a Mon Sep 17 00:00:00 2001
From: "F. Jason Park" <jp@HIDDEN>
Date: Tue, 1 Mar 2022 02:12:02 -0800
Subject: [PATCH 3/4] Simplify network-stream openers in socks.el

* lisp/net/socks.el (socks-override-functions): Make variable obsolete
and remove uses throughout.
(socks-open-network-stream-fallback): Add new custom option indicating
whether to fall back on non-SOCKS connections.
(socks-open-network-stream-tls-services): Add new custom option for
specifying ports for proxied connections that should be encrypted with
TLS.
(socks-open-network-stream): Recognize additional `url' struct param.
Also prefer parsed URL details when present in `url-using-proxy'.
(socks--open-network-stream): Reduce role to merely issuing the first
command using an existing process.
---
 lisp/net/socks.el | 109 ++++++++++++++++++++++++++++++++++++----------
 1 file changed, 86 insertions(+), 23 deletions(-)

diff --git a/lisp/net/socks.el b/lisp/net/socks.el
index b9af2aa06e..ac732b228b 100644
--- a/lisp/net/socks.el
+++ b/lisp/net/socks.el
@@ -34,7 +34,7 @@
 
 ;;; Code:
 
-(eval-when-compile (require 'cl-lib))
+(eval-when-compile (require 'cl-lib) (require 'url-parse))
 
 ;; ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
 ;; ;;; Custom widgets
@@ -325,15 +325,15 @@ socks-filter
 	    (process-put proc 'socks-response string))))))
      ((= state socks-state-connected)))))
 
-;; FIXME this is a terrible idea.
-;; It is not even compatible with the argument spec of open-network-stream
-;; in 24.1.
-
 (defvar socks-override-functions nil
   "If non-nil, overwrite `open-network-stream' function with SOCKSified version.")
 
-(when socks-override-functions
-  (advice-add 'open-network-stream :around #'socks--open-network-stream))
+;; Libraries typically offer a "stream opener" option, such as ERC's
+;; `erc-server-connect-function'.  These provide a level of
+;; flexibility tantamount to what this variable formerly offered.
+(make-obsolete-variable
+ 'socks-override-functions
+ "see `socks-open-network-stream' and `socks-connect-function'." "29.1")
 
 (defcustom socks-connect-function 'open-network-stream
   "Function to open a network connection to a SOCKS provider.
@@ -539,22 +539,85 @@ socks-find-services-entry
   (gethash (downcase service)
 	      (if udp socks-udp-services socks-tcp-services)))
 
-(defun socks-open-network-stream (name buffer host service)
-  (let ((socks-override-functions t))
-    (socks--open-network-stream
-     (lambda (&rest args)
-       (let ((socks-override-functions nil))
-         (apply #'open-network-stream args)))
-     name buffer host service)))
-
-(defun socks--open-network-stream (orig-fun name buffer host service &rest params)
-  (let ((route (and socks-override-functions
-                    (socks-find-route host service))))
-    (if (not route)
-	(apply orig-fun name buffer host service params)
-      ;; FIXME: Obey `params'!
-      (let* ((proc (socks-open-connection route))
-	     (version (process-get proc 'socks-server-protocol))
+(defcustom socks-open-network-stream-fallback nil
+  "Whether `socks-open-network-stream' should fall back to non-SOCKS."
+  :version "29.1"
+  :type 'boolean)
+
+(defcustom socks-proxied-tls-services '(443 6697)
+  "Ports whose connections should use TLS.
+Note that the system resolver may be consulted to look up host
+names for checking domain validation certs."
+  :version "29.1"
+  :type '(repeat number))
+
+(declare-function gnutls-negotiate "gnutls" (&rest rest))
+(declare-function nsm-verify-connection "nsm"
+                  (process host port &optional
+                           save-fingerprint warn-unencrypted))
+
+;;;###autoload
+(defun socks-open-network-stream (name buffer host service &rest params)
+  "Open and return a connection, possibly proxied over SOCKS.
+Expect PARAMS to contain keyword parameters recognized by
+`open-network-stream'.  Assume HOST and SERVICE refer to the
+proxied remote peer rather than the SOCKS server, but assume the
+opposite for PARAMS.  That is, if PARAMS contains a `:type' of
+`tls', treat the underlying connection to the proxy server as
+destined for encryption rather than the tunneled connection (even
+though `socks-connect-function' has the final say).  For TLS with
+proxied connections, see the option `socks-proxied-tls-services'.
+
+Before connecting, check the host against `socks-noproxy', and on
+rejection either signal an error or fall back to non-SOCKS,
+depending on the value of `socks-open-network-stream-fallback'.
+But, before doing anything, check if `url-using-proxy' is bound
+to a `url' struct object, as defined in `url-parse'.  If so,
+assume it represents the address of the desired SOCKS server
+rather than that of the remote peer, and use its fields instead
+of `socks-server' for all SOCKS connection details."
+  (require 'url-parse)
+  (let* ((url (and (url-p url-using-proxy)
+                   (string-prefix-p "socks" (url-type url-using-proxy))
+                   url-using-proxy))
+         (socks-server (if url
+                           (list name (url-host url) (url-port url)
+                                 (pcase (url-type url)
+                                   ("socks4://" 4)
+                                   ("socks4a://" '4a)
+                                   (_ 5)))
+                         socks-server))
+         (socks-username (or (and url (url-user url))
+                             socks-username))
+         (socks-password (or (and url (url-password url))
+                             socks-password)))
+    (if-let* ((route (socks-find-route host service))
+              (proc (apply #'socks-open-connection route params)))
+        (let ((port (if (numberp service)
+                        service
+                      (process-contact proc :service)))
+              (certs (plist-get params :client-certificate)))
+          (socks--open-network-stream proc buffer host service)
+          (if (and (memq port socks-proxied-tls-services)
+                   (gnutls-available-p)
+                   (require 'gnutls nil t)
+                   (require 'nsm nil t))
+              (progn (gnutls-negotiate :process proc
+                                       :hostname host
+                                       :keylist (and certs (list certs)))
+                     (nsm-verify-connection proc host port))
+            proc))
+      ;; Retain legacy behavior and connect anyway without warning
+      (if socks-open-network-stream-fallback
+          (with-suppressed-warnings ((obsolete socks-override-functions))
+            (let (socks-override-functions)
+              (apply #'open-network-stream name buffer host service params)))
+        (error "Connection rejected by `socks-noproxy'")))))
+
+(defun socks--open-network-stream (proc buffer host service)
+  (progn ; preserve indentation level for git blame / code review
+    (progn ; could rename to something like `socks--initiate-command-connect'
+      (let* ((version (process-get proc 'socks-server-protocol))
              (atype
               (cond
                ((equal version 4)
-- 
2.38.1


--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
 filename=0004-POC-30.0.50-Integrate-the-socks-and-url-libraries.patch

From 5ac2987f3085dede2e20755bb6c9631f7d47380b Mon Sep 17 00:00:00 2001
From: "F. Jason Park" <jp@HIDDEN>
Date: Tue, 1 Mar 2022 01:38:33 -0800
Subject: [PATCH 4/4] [POC/30.0.50] Integrate the socks and url libraries

* lisp/url/url-gw.el (url-open-stream): Use presence and type of
`url-using-proxy' to detect caller and massage input values according
to legacy practices.

* lisp/url/url-http.el: (url-http-find-free-connection): Don't call
`url-open-stream' with host and port from active proxy.
(url-http, url-http-async-sentinel): Only open
`url-https-proxy-connect' for non-SOCKS proxies.

* lisp/url/url-proxy.el (url-proxy--socks-scheme-regexp): Add new
const.
(url-default-find-proxy-for-url): Accommodate SOCKS entries but defy
original design somewhat by requiring a URL scheme in the host value
for detection.
(url-find-proxy-for-url): Recognize modified host/address value for
socks entries of `url-proxy-services' and deal accordingly.
(url-proxy): Handle a SOCKS proxy for http(s) connections only.

* lisp/url/url-vars.el (url-proxy-services): Explain that values for
certain gateways may need a leading scheme:// portion.
(url-using-proxy): Add warning regarding expected type.
---
 lisp/url/url-gw.el    |  8 +++++++-
 lisp/url/url-http.el  | 16 +++++++---------
 lisp/url/url-proxy.el | 18 ++++++++++++++++--
 lisp/url/url-vars.el  | 13 ++++++++++---
 4 files changed, 40 insertions(+), 15 deletions(-)

diff --git a/lisp/url/url-gw.el b/lisp/url/url-gw.el
index e4d1ca72a0..c93edc0d4e 100644
--- a/lisp/url/url-gw.el
+++ b/lisp/url/url-gw.el
@@ -28,7 +28,7 @@
 (require 'url-vars)
 (require 'url-parse)
 
-(autoload 'socks-open-network-stream "socks")
+(autoload 'socks-open-network-stream "socks") ; FIXME remove this
 
 (defgroup url-gateway nil
   "URL gateway variables."
@@ -220,6 +220,12 @@ url-open-stream
 Optional arg GATEWAY-METHOD specifies the gateway to be used,
 overriding the value of `url-gateway-method'."
   (unless url-gateway-unplugged
+    (when (url-p url-using-proxy)
+      (if (or (eq 'socks url-gateway-method)
+              (string-prefix-p "socks" (url-type url-using-proxy)))
+          (setq gateway-method 'socks)
+        (setq host (url-host url-using-proxy)
+              service (url-port url-using-proxy))))
     (let* ((gwm (or gateway-method url-gateway-method))
            (gw-method (if (and url-gateway-local-host-regexp
                                (not (eq 'tls gwm))
diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
index 94ef156108..864048a73c 100644
--- a/lisp/url/url-http.el
+++ b/lisp/url/url-http.el
@@ -195,12 +195,7 @@ url-http-find-free-connection
 	;; like authentication.  But we use another buffer afterwards.
 	(unwind-protect
             (let ((proc (url-open-stream host buf
-                                         (if url-using-proxy
-                                             (url-host url-using-proxy)
-                                           host)
-                                         (if url-using-proxy
-                                             (url-port url-using-proxy)
-                                           port)
+                                         host port
                                          gateway-method)))
 	      ;; url-open-stream might return nil.
 	      (when (processp proc)
@@ -1396,8 +1391,9 @@ url-http
            (error "Could not create connection to %s:%d" (url-host url)
                   (url-port url)))
           (_
-           (if (and url-http-proxy (string= "https"
-                                            (url-type url-current-object)))
+           (if (and url-http-proxy
+                    (not (string-prefix-p "socks" (url-type url-http-proxy)))
+                    (string= "https" (url-type url-current-object)))
                (url-https-proxy-connect connection)
              (set-process-sentinel connection
                                    #'url-http-end-of-document-sentinel)
@@ -1479,7 +1475,9 @@ url-http-async-sentinel
 	(url-http-end-of-document-sentinel proc why))
        ((string= (substring why 0 4) "open")
 	(setq url-http-connection-opened t)
-        (if (and url-http-proxy (string= "https" (url-type url-current-object)))
+        (if (and url-http-proxy
+                 (not (string-prefix-p "socks" (url-type url-http-proxy)))
+                 (string= "https" (url-type url-current-object)))
             (url-https-proxy-connect proc)
           (condition-case error
               (process-send-string proc (url-http-create-request))
diff --git a/lisp/url/url-proxy.el b/lisp/url/url-proxy.el
index c72e459a4e..f4ddd639f6 100644
--- a/lisp/url/url-proxy.el
+++ b/lisp/url/url-proxy.el
@@ -25,6 +25,9 @@
 
 (require 'url-parse)
 
+(defconst url-proxy--socks-scheme-regexp
+  (rx bot "socks" (? (or "4" "4a" "5" "5h")) "://"))
+
 (defun url-default-find-proxy-for-url (urlobj host)
   (cond
    ((or (and (assoc "no_proxy" url-proxy-services)
@@ -35,7 +38,12 @@ url-default-find-proxy-for-url
 	(equal "www" (url-type urlobj)))
     "DIRECT")
    ((cdr (assoc (url-type urlobj) url-proxy-services))
-    (concat "PROXY " (cdr (assoc (url-type urlobj) url-proxy-services))))
+    (let ((found (alist-get (url-type urlobj) url-proxy-services
+                            nil nil #'equal)))
+      (concat (if (string-match url-proxy--socks-scheme-regexp found)
+                  "SOCKS "
+                "PROXY ")
+              found)))
    ;;
    ;; Should check for socks
    ;;
@@ -57,7 +65,10 @@ url-find-proxy-for-url
      ((string-match "^PROXY +" proxy)
       (concat "http://" (substring proxy (match-end 0)) "/"))
      ((string-match "^SOCKS +" proxy)
-      (concat "socks://" (substring proxy (match-end 0))))
+      (if-let* ((m (substring proxy (match-end 0)))
+                ((string-match url-proxy--socks-scheme-regexp m)))
+          m
+        (concat "socks://" m)))
      (t
       (display-warning 'url (format "Unknown proxy directive: %s" proxy) :error)
       nil))))
@@ -72,6 +83,9 @@ url-proxy
   (cond
    ((string= (url-type url-using-proxy) "http")
     (url-http url callback cbargs))
+   ((and (string-prefix-p "socks" (url-type url-using-proxy))
+         (string-prefix-p "http" (url-type url)))
+    (url-http url callback cbargs))
    (t
     (error "Don't know how to use proxy `%s'" url-using-proxy))))
 
diff --git a/lisp/url/url-vars.el b/lisp/url/url-vars.el
index 4cdca05554..209d387ea7 100644
--- a/lisp/url/url-vars.el
+++ b/lisp/url/url-vars.el
@@ -191,11 +191,16 @@ url-mail-command
 
 (defcustom url-proxy-services nil
   "An alist of schemes and proxy servers that gateway them.
-Looks like ((\"http\" . \"hostname:portnumber\") ...).  This is set up
-from the ACCESS_proxy environment variables."
+Looks like ((\"http\" . \"hostname:portnumber\") ...).  This is
+set up from the ACCESS_proxy environment variables.  Depending on
+the gateway type, values may instead be expected to look like
+\"proxyscheme://hostname:portnumber\" where \"proxyscheme\" is
+something like \"socks5\".  As of Emacs 30.1, this only applies
+to SOCKS servers."
   :type '(repeat (cons :format "%v"
 		       (string :tag "Protocol")
 		       (string :tag "Proxy")))
+  :version "30.1"
   :group 'url)
 
 (defcustom url-standalone-mode nil
@@ -310,7 +315,9 @@ url-show-status
 
 (defvar url-using-proxy nil
   "Either nil or the fully qualified proxy URL in use, e.g.
-https://www.example.com/")
+https://www.example.com/.  Beware that some functions, such as
+`url-proxy' and `url-http-end-of-document-sentinel', set this to
+a `url' struct.")
 
 (defcustom url-news-server nil
   "The default news server from which to get newsgroups/articles.
-- 
2.38.1


--=-=-=--

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#53941: Last-minute socks.el improvements for Emacs 29?
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Mon, 28 Nov 2022 17:12:02 +0000
Resent-Message-ID: <handler.53941.B53941.166965552024500 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 53941
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: patch
To: "J.P." <jp@HIDDEN>
Cc: larsi@HIDDEN, gnuhacker@HIDDEN, 53941 <at> debbugs.gnu.org
Received: via spool by 53941-submit <at> debbugs.gnu.org id=B53941.166965552024500
          (code B ref 53941); Mon, 28 Nov 2022 17:12:02 +0000
Received: (at 53941) by debbugs.gnu.org; 28 Nov 2022 17:12:00 +0000
Received: from localhost ([127.0.0.1]:49799 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1ozhfk-0006N6-0M
	for submit <at> debbugs.gnu.org; Mon, 28 Nov 2022 12:12:00 -0500
Received: from eggs.gnu.org ([209.51.188.92]:58436)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1ozhfg-0006N0-E7
 for 53941 <at> debbugs.gnu.org; Mon, 28 Nov 2022 12:11:58 -0500
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1ozhfa-0002KG-P2; Mon, 28 Nov 2022 12:11:50 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=MdHbX+/5Upwy/4NkCznjDlv2wufskl1ZlG4CAi/C+OM=; b=qN7jwXUiFq0z
 H3ABIo3UEWtFxbGootBuoANypArmhIctA/Vw9eXZHKn3pse6+AHysQxMP5cPQumjk0NnmWe35Vug3
 PNqa/bRtrXCRoytpp/L5eALzAKaqN5XGmIvkr+U6srvY2/q6AX8xRgd4VqEATrqVh2z+1SxchxfCh
 jqvchK7FMEYxC0G0yLtftwbGSw7SEcKd2qPZtiEj/ELUvD2ntuo4VpSFtNvpHlP8lpfNcakmjq8C+
 Q5o8L6PYL1HCxbs6+tToPQcKFmSWn57UO+8EjwvEZYTUszuaCsiQr1pEN1kirdy8T6+thNuCUHJkH
 Y39bURPmSBlDD+C6rEP/sw==;
Received: from [87.69.77.57] (helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1ozhfa-0000t9-4p; Mon, 28 Nov 2022 12:11:50 -0500
Date: Mon, 28 Nov 2022 19:12:19 +0200
Message-Id: <8335a3nguk.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <87mt8baygn.fsf_-_@HIDDEN> (jp@HIDDEN)
References: <87pmntfym7.fsf@HIDDEN> <8735kl1v58.fsf@HIDDEN>
 <87a6emftzx.fsf@HIDDEN> <87k0do5km1.fsf@HIDDEN>
 <87pmn5n3tu.fsf@HIDDEN> <87mti99j1f.fsf@HIDDEN>
 <87wnh7hkgi.fsf@HIDDEN> <87pmmz947k.fsf@HIDDEN>
 <8735ju44sk.fsf@HIDDEN> <87lexikwu5.fsf@HIDDEN>
 <87mt8baygn.fsf_-_@HIDDEN>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: "J.P." <jp@HIDDEN>
> Cc: Jacobo <gnuhacker@HIDDEN>, Lars Ingebrigtsen <larsi@HIDDEN>,
>  Eli Zaretskii <eliz@HIDDEN>
> Date: Mon, 28 Nov 2022 07:30:16 -0800
> 
> I've lifted some fixes and minor enhancements from my POC stuff posted
> to this thread earlier this year. Nothing is directly Tor related, so I
> can create a new bug report, if necessary.
> 
> The second patch fixes a problem involving SOCKS 5 error handling. It
> also adds support for SOCKS 4a, which allows tools that don't speak
> SOCKS 5, like socat, to resolve host names. The third addresses a couple
> FIXMEs but no bugs, strictly speaking. The fourth is just a demo [1].
> Happy to explain whatever in detail.

I'm really uncomfortable with installing these changes before the release
branch is cut.  The changes are hardly trivial, some controversial even to
my eyes, even though I'm no expert on network connections.  For example:

> +(defun socks-open-connection (server-info &rest stream-params)
> +  "Create and initialize a SOCKS process.
> +Perform authentication if needed.  Expect SERVER-INFO to take the
> +form of `socks-server' and STREAM-PARAMS to be keyword params
> +accepted by `open-network-stream'."
>    (interactive)
> +  (unless (plist-member stream-params :coding)
> +    (setf (plist-get stream-params :coding) '(binary . binary)))

AFAIU, this constitutes an incompatible change in behavior: the default for
:coding is was never 'binary' before, it was determined from the locale's
preferences.  Why are we making this change here?

> @@ -446,7 +474,11 @@ socks-send-command
>  	nil				; Sweet sweet success!
>        (delete-process proc)
>        (error "SOCKS: %s"
> -             (nth (or (process-get proc 'socks-reply) 1) socks-errors)))
> +             (let ((no (or (process-get proc 'socks-reply) 99)))
> +               (or (if (eq version 5) ; 99 - 90 >= length(errors)
> +                       (nth no socks-errors)
> +                     (nth (- no 90) socks--errors-4))
> +                   "Unknown error"))))

I don't really understand the semantics here (so maybe comments need to be
upgraded), but the old and the new versions don't look to me like equivalent
code -- why the change?

> -(when socks-override-functions
> -  (advice-add 'open-network-stream :around #'socks--open-network-stream))
> +;; Libraries typically offer a "stream opener" option, such as ERC's
> +;; `erc-server-connect-function'.  These provide a level of
> +;; flexibility tantamount to what this variable formerly offered.
> +(make-obsolete-variable
> + 'socks-override-functions
> + "see `socks-open-network-stream' and `socks-connect-function'." "29.1")

Why this last-minute obsolescence?

> +(defun socks-open-network-stream (name buffer host service &rest params)
> +  "Open and return a connection, possibly proxied over SOCKS.

The changes in this public function are so significant that I don't
understand how they can be suggested so close to the branching.

If it is possible to add support for SOCKS 4a without affecting any
previously supported versions, I'm fine.  Adding tests is also fine.  But
for the rest, I think you should wait until after the release branch is cut
and install this on the master branch.  Sorry, it really is too late for
such changes.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#53941: Last-minute socks.el improvements for Emacs 29?
Resent-From: "J.P." <jp@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Tue, 29 Nov 2022 14:25:01 +0000
Resent-Message-ID: <handler.53941.B53941.166973187611647 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 53941
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: patch
To: Eli Zaretskii <eliz@HIDDEN>
Cc: larsi@HIDDEN, gnuhacker@HIDDEN, 53941 <at> debbugs.gnu.org
Received: via spool by 53941-submit <at> debbugs.gnu.org id=B53941.166973187611647
          (code B ref 53941); Tue, 29 Nov 2022 14:25:01 +0000
Received: (at 53941) by debbugs.gnu.org; 29 Nov 2022 14:24:36 +0000
Received: from localhost ([127.0.0.1]:54544 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1p01XG-00031n-G2
	for submit <at> debbugs.gnu.org; Tue, 29 Nov 2022 09:24:36 -0500
Received: from mail-108-mta220.mxroute.com ([136.175.108.220]:35753)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <jp@HIDDEN>) id 1p01XC-00031g-Fi
 for 53941 <at> debbugs.gnu.org; Tue, 29 Nov 2022 09:24:32 -0500
Received: from mail-111-mta2.mxroute.com ([136.175.111.2]
 filter006.mxroute.com) (Authenticated sender: mN4UYu2MZsgR)
 by mail-108-mta220.mxroute.com (ZoneMTA) with ESMTPSA id
 184c3c63cd90006e99.001 for <53941 <at> debbugs.gnu.org>
 (version=TLSv1/SSLv3 cipher=ECDHE-RSA-AES128-GCM-SHA256);
 Tue, 29 Nov 2022 14:24:19 +0000
X-Zone-Loop: fa2069edd9547ebb1cf019439e8831a3a3df0e4a4e44
X-Originating-IP: [136.175.111.2]
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=neverwas.me
 ; s=x;
 h=Content-Type:MIME-Version:Message-ID:Date:References:In-Reply-To:
 Subject:Cc:To:From:Sender:Reply-To:Content-Transfer-Encoding:Content-ID:
 Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc
 :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe:
 List-Post:List-Owner:List-Archive;
 bh=pzDhj02KsXlAwql8Nv1T3s6DwCA5LQbd/+Bc2TD6v7w=; b=QXp0W4akZKybcVgW3KCxOvf6F0
 3qddAQM0IoiX9m6zrwC+ZKiHnSaHLWthMxnUF4AQctmjy5rfBE6w5bFjEzssWt9zzf4eyFGXA8EAM
 Me4TFxll8t9fFoYs3gz05ERMxw9pcB1Harjj+0U4KaxLugTW4HtSupe0SSxpceAG0lWAUaltWdVUO
 gy8KPqtpIkvSOtA4TVdZgjr7sL+/SuzmRMaq3soXdI20MTp9YgZBeFG3bX7gUN1kigdKDyLb3zYIs
 V9S+B/4qmEjxxntC11O9oqdVuDc/8doBYxL7ButlwxeQ2pmzLDwDc1WTQHO8QvAR1lEGAHunYIIcM
 4sh5tRtg==;
From: "J.P." <jp@HIDDEN>
In-Reply-To: <8335a3nguk.fsf@HIDDEN> (Eli Zaretskii's message of "Mon, 28 Nov
 2022 19:12:19 +0200")
References: <87pmntfym7.fsf@HIDDEN> <8735kl1v58.fsf@HIDDEN>
 <87a6emftzx.fsf@HIDDEN> <87k0do5km1.fsf@HIDDEN>
 <87pmn5n3tu.fsf@HIDDEN> <87mti99j1f.fsf@HIDDEN>
 <87wnh7hkgi.fsf@HIDDEN> <87pmmz947k.fsf@HIDDEN>
 <8735ju44sk.fsf@HIDDEN> <87lexikwu5.fsf@HIDDEN>
 <87mt8baygn.fsf_-_@HIDDEN> <8335a3nguk.fsf@HIDDEN>
Date: Tue, 29 Nov 2022 06:24:15 -0800
Message-ID: <87fse1kfe8.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="=-=-="
X-Authenticated-Id: masked@HIDDEN
X-Spam-Score: 0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

--=-=-=
Content-Type: text/plain

Eli Zaretskii <eliz@HIDDEN> writes:

> I'm really uncomfortable with installing these changes before the release
> branch is cut.  The changes are hardly trivial, some controversial even to
> my eyes, even though I'm no expert on network connections.

Well, I myself am just about the furthest thing from (an expert), which
certainly doesn't comport well with dropping rash changes at the
eleventh hour. (That was rather disrespectful on my part, so shame on
me.) As such, if it's easier to revisit this once things settle down,
just ignore this email and I'll re-ping you sometime down the road.

> For example:
>
>> +(defun socks-open-connection (server-info &rest stream-params)
>> +  "Create and initialize a SOCKS process.
>> +Perform authentication if needed.  Expect SERVER-INFO to take the
>> +form of `socks-server' and STREAM-PARAMS to be keyword params
>> +accepted by `open-network-stream'."
>>    (interactive)
>> +  (unless (plist-member stream-params :coding)
>> +    (setf (plist-get stream-params :coding) '(binary . binary)))
>
> AFAIU, this constitutes an incompatible change in behavior: the default for
> :coding is was never 'binary' before, it was determined from the locale's
> preferences.  Why are we making this change here?

Just good old fashioned stupidity, I'm afraid. (And also recklessness in
overly trusting the me from eight months ago, surely.) I guess I somehow
assumed that if the caller didn't set :coding explicitly, they would do
so once handed back the process, which is certifiably dumb.

>> @@ -446,7 +474,11 @@ socks-send-command
>>  	nil				; Sweet sweet success!
>>        (delete-process proc)
>>        (error "SOCKS: %s"
>> -             (nth (or (process-get proc 'socks-reply) 1) socks-errors)))
>> +             (let ((no (or (process-get proc 'socks-reply) 99)))
>> +               (or (if (eq version 5) ; 99 - 90 >= length(errors)
>> +                       (nth no socks-errors)
>> +                     (nth (- no 90) socks--errors-4))
>> +                   "Unknown error"))))
>
> I don't really understand the semantics here (so maybe comments need to be
> upgraded), but the old and the new versions don't look to me like equivalent
> code -- why the change?

This sets the fallback message to "Unknown error" (made up) rather than
"General SOCKS server failure" (an official error code). At first, I
figured the distinction more faithfully conveyed the nature of the
error, but now I see that it just adds clutter because the fallback path
can only be triggered by a protocol mishap, and that's unlikely, given
that the conversation must progress to its third back-and-forth by the
time this runs.

(BTW, the words "error handling" in the patch's title refer to the added
"(0 0)" `pcase' condition in `socks-filter' and not the snippet above.)

>> -(when socks-override-functions
>> -  (advice-add 'open-network-stream :around #'socks--open-network-stream))
>> +;; Libraries typically offer a "stream opener" option, such as ERC's
>> +;; `erc-server-connect-function'.  These provide a level of
>> +;; flexibility tantamount to what this variable formerly offered.
>> +(make-obsolete-variable
>> + 'socks-override-functions
>> + "see `socks-open-network-stream' and `socks-connect-function'." "29.1")
>
> Why this last-minute obsolescence?

Just my being callous. I now see that obsoleting that variable is
problematic, not least because we continue to honor `socks-noproxy'. But
the two complement each other and are closely coupled, usage-wise.
Getting rid of the one and pretending the other still works as intended
was doubly irresponsible.

>> +(defun socks-open-network-stream (name buffer host service &rest params)
>> +  "Open and return a connection, possibly proxied over SOCKS.
>
> The changes in this public function are so significant that I don't
> understand how they can be suggested so close to the branching.

The old signature was

  (name buffer host service) -> process

and the new &rest arguments would be optional. And since the lone
in-tree call site sticks to the four required positionals, I didn't
think a move from (4 . 4) to (4 . many), in `func-arity' terms, stood to
break any advice in the wild. Still, there are side effects in the new
version that could use more thorough exploring, and further attention
could be paid to its treatment of `socks-override-functions' in terms of
preserving old behavior.

> If it is possible to add support for SOCKS 4a without affecting any
> previously supported versions, I'm fine.  Adding tests is also fine.
> But for the rest, I think you should wait until after the release
> branch is cut and install this on the master branch. Sorry, it really
> is too late for such changes.

You're very gracious, but I think I've learned my lesson and will
refrain from pursuing any of these changes for Emacs 29. Apologies for
abusing your time and maintainerly patience (yet again).


--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment; filename=0000-v1-v2.diff

From 96a4de741663672e928fd30af6c93b335b346691 Mon Sep 17 00:00:00 2001
From: "F. Jason Park" <jp@HIDDEN>
Date: Tue, 29 Nov 2022 00:18:42 -0800
Subject: [PATCH 0/4] *** NOT A PATCH ***

*** BLURB HERE ***

F. Jason Park (4):
  ; Don't hard code server ports in SOCKS tests
  [30.0.50] Improve SOCKS error handling and add support for 4a
  [WIP/30.0.50] Simplify network-stream openers in socks.el
  [POC/30.0.50] Integrate the socks and url libraries

 lisp/net/socks.el            | 142 +++++++++++++++++++++++++++++------
 lisp/url/url-gw.el           |   8 +-
 lisp/url/url-http.el         |  16 ++--
 lisp/url/url-proxy.el        |  18 ++++-
 lisp/url/url-vars.el         |  13 +++-
 test/lisp/net/socks-tests.el |  39 +++++++---
 6 files changed, 186 insertions(+), 50 deletions(-)

Interdiff:
diff --git a/lisp/net/socks.el b/lisp/net/socks.el
index ac732b228b..65436ed047 100644
--- a/lisp/net/socks.el
+++ b/lisp/net/socks.el
@@ -325,20 +325,20 @@ socks-filter
 	    (process-put proc 'socks-response string))))))
      ((= state socks-state-connected)))))
 
+;; FIXME this is a terrible idea.
+;; It is not even compatible with the argument spec of open-network-stream
+;; in 24.1.
+
 (defvar socks-override-functions nil
   "If non-nil, overwrite `open-network-stream' function with SOCKSified version.")
 
-;; Libraries typically offer a "stream opener" option, such as ERC's
-;; `erc-server-connect-function'.  These provide a level of
-;; flexibility tantamount to what this variable formerly offered.
-(make-obsolete-variable
- 'socks-override-functions
- "see `socks-open-network-stream' and `socks-connect-function'." "29.1")
+(when socks-override-functions
+  (advice-add 'open-network-stream :around #'socks--open-network-stream))
 
 (defcustom socks-connect-function 'open-network-stream
   "Function to open a network connection to a SOCKS provider.
 Called with arguments suitable for `open-network-stream'."
-  :version "29.1"
+  :version "30.1"
   :type '(choice (function-item :value open-network-stream)
                  (function :tag "User-provided function")))
 
@@ -348,14 +348,11 @@ socks-open-connection
 form of `socks-server' and STREAM-PARAMS to be keyword params
 accepted by `open-network-stream'."
   (interactive)
-  (unless (plist-member stream-params :coding)
-    (setf (plist-get stream-params :coding) '(binary . binary)))
   (save-excursion
     (let ((proc
-           (with-suppressed-warnings ((obsolete socks-override-functions))
-             (let ((socks-override-functions nil))
-               (apply socks-connect-function (nth 0 server-info) nil
-                      (nth 1 server-info) (nth 2 server-info) stream-params))))
+           (let ((socks-override-functions nil))
+             (apply socks-connect-function (nth 0 server-info) nil
+                    (nth 1 server-info) (nth 2 server-info) stream-params)))
 	  (authtype nil)
 	  version)
 
@@ -474,11 +471,11 @@ socks-send-command
 	nil				; Sweet sweet success!
       (delete-process proc)
       (error "SOCKS: %s"
-             (let ((no (or (process-get proc 'socks-reply) 99)))
-               (or (if (eq version 5) ; 99 - 90 >= length(errors)
-                       (nth no socks-errors)
-                     (nth (- no 90) socks--errors-4))
-                   "Unknown error"))))
+             (let ((err (process-get proc 'socks-reply)))
+               (if (eql version 5)
+                   (nth (or err 1) socks-errors)
+                 (nth (- (if (and err (<= 90 err 93)) err 91) 90)
+                      socks--errors-4)))))
     proc))
 
 
@@ -539,16 +536,15 @@ socks-find-services-entry
   (gethash (downcase service)
 	      (if udp socks-udp-services socks-tcp-services)))
 
-(defcustom socks-open-network-stream-fallback nil
-  "Whether `socks-open-network-stream' should fall back to non-SOCKS."
-  :version "29.1"
-  :type 'boolean)
+(defun socks--open-network-stream (orig-fun name buffer host service &rest params)
+  (let ((socks-override-functions orig-fun))
+    (apply #'socks-open-network-stream name buffer host service params)))
 
 (defcustom socks-proxied-tls-services '(443 6697)
   "Ports whose connections should use TLS.
 Note that the system resolver may be consulted to look up host
 names for checking domain validation certs."
-  :version "29.1"
+  :version "30.1"
   :type '(repeat number))
 
 (declare-function gnutls-negotiate "gnutls" (&rest rest))
@@ -568,9 +564,12 @@ socks-open-network-stream
 though `socks-connect-function' has the final say).  For TLS with
 proxied connections, see the option `socks-proxied-tls-services'.
 
-Before connecting, check the host against `socks-noproxy', and on
-rejection either signal an error or fall back to non-SOCKS,
-depending on the value of `socks-open-network-stream-fallback'.
+Before connecting, check the host against `socks-noproxy' and, on
+rejection, fall back to non-SOCKS.  Similarly, when
+`socks-override-functions' is a function, call it directly and
+trust that it's not interested in options defined in this
+library, such as `socks-server'.
+
 But, before doing anything, check if `url-using-proxy' is bound
 to a `url' struct object, as defined in `url-parse'.  If so,
 assume it represents the address of the desired SOCKS server
@@ -591,13 +590,14 @@ socks-open-network-stream
                              socks-username))
          (socks-password (or (and url (url-password url))
                              socks-password)))
-    (if-let* ((route (socks-find-route host service))
+    (if-let* (((booleanp socks-override-functions))
+              (route (socks-find-route host service))
               (proc (apply #'socks-open-connection route params)))
         (let ((port (if (numberp service)
                         service
                       (process-contact proc :service)))
               (certs (plist-get params :client-certificate)))
-          (socks--open-network-stream proc buffer host service)
+          (socks--initiate-command-connect proc buffer host service)
           (if (and (memq port socks-proxied-tls-services)
                    (gnutls-available-p)
                    (require 'gnutls nil t)
@@ -607,16 +607,15 @@ socks-open-network-stream
                                        :keylist (and certs (list certs)))
                      (nsm-verify-connection proc host port))
             proc))
-      ;; Retain legacy behavior and connect anyway without warning
-      (if socks-open-network-stream-fallback
-          (with-suppressed-warnings ((obsolete socks-override-functions))
-            (let (socks-override-functions)
-              (apply #'open-network-stream name buffer host service params)))
-        (error "Connection rejected by `socks-noproxy'")))))
-
-(defun socks--open-network-stream (proc buffer host service)
+      (let ((fn (if (functionp socks-override-functions)
+                    socks-override-functions ; `socks-noproxy' not consulted
+                  #'open-network-stream)) ; `socks-noproxy' is non-nil
+            socks-override-functions)
+        (apply fn name buffer host service params)))))
+
+(defun socks--initiate-command-connect (proc buffer host service)
   (progn ; preserve indentation level for git blame / code review
-    (progn ; could rename to something like `socks--initiate-command-connect'
+    (progn
       (let* ((version (process-get proc 'socks-server-protocol))
              (atype
               (cond
-- 
2.38.1


--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
 filename=0001-Don-t-hard-code-server-ports-in-SOCKS-tests.patch

From 0780339ceee3b0068700f5a3bf6d48aa4023915e Mon Sep 17 00:00:00 2001
From: "F. Jason Park" <jp@HIDDEN>
Date: Mon, 14 Feb 2022 02:36:57 -0800
Subject: [PATCH 1/4] ; Don't hard code server ports in SOCKS tests

* test/lisp/net/socks-tests.el (socks-tests-canned-server-create,
socks-tests-filter-response-parsing-v4, socks-tests-v4-basic,
socks-tests-v5-auth-user-pass, socks-tests-v5-auth-user-blank,
socks-tests-v5-auth-none): Fix bug in process filter to prevent
prepared outgoing responses from being implicitly encoded as utf-8.
Fix similar mistake in v4 filter test.  Also allow system to choose
port instead of hard-coding it.
---
 test/lisp/net/socks-tests.el | 26 +++++++++++++++-----------
 1 file changed, 15 insertions(+), 11 deletions(-)

diff --git a/test/lisp/net/socks-tests.el b/test/lisp/net/socks-tests.el
index 461796bdf9..f1ecf1630f 100644
--- a/test/lisp/net/socks-tests.el
+++ b/test/lisp/net/socks-tests.el
@@ -63,21 +63,21 @@ socks-tests-filter-response-parsing-v4
     (process-put proc 'socks-state socks-state-waiting)
     (process-put proc 'socks-server-protocol 4)
     (ert-info ("Receive initial incomplete segment")
-      (socks-filter proc (concat [0 90 0 0 93 184 216]))
-      ;; From example.com: OK status ^      ^ msg start
+      (socks-filter proc (unibyte-string 0 90 0 0 93 184 216))
+      ;; From example.com: OK status       ^      ^ msg start
       (ert-info ("State still set to waiting")
         (should (eq (process-get proc 'socks-state) socks-state-waiting)))
       (ert-info ("Response field is nil because processing incomplete")
         (should-not (process-get proc 'socks-response)))
       (ert-info ("Scratch field holds stashed partial payload")
-        (should (string= (concat [0 90 0 0 93 184 216])
+        (should (string= (unibyte-string 0 90 0 0 93 184 216)
                          (process-get proc 'socks-scratch)))))
     (ert-info ("Last part arrives")
       (socks-filter proc "\42") ; ?\" 34
       (ert-info ("State transitions to complete (length check passes)")
         (should (eq (process-get proc 'socks-state) socks-state-connected)))
       (ert-info ("Scratch and response fields hold stash w. last chunk")
-        (should (string= (concat [0 90 0 0 93 184 216 34])
+        (should (string= (unibyte-string 0 90 0 0 93 184 216 34)
                          (process-get proc 'socks-response)))
         (should (string= (process-get proc 'socks-response)
                          (process-get proc 'socks-scratch)))))
@@ -133,17 +133,19 @@ socks-tests-canned-server-patterns
 (defun socks-tests-canned-server-create ()
   "Create and return a fake SOCKS server."
   (let* ((port (nth 2 socks-server))
-         (name (format "socks-server:%d" port))
+         (name (format "socks-server:%s"
+                       (if (numberp port) port (ert-test-name (ert-running-test)))))
          (pats socks-tests-canned-server-patterns)
          (filt (lambda (proc line)
                  (pcase-let ((`(,pat . ,resp) (pop pats)))
                    (unless (or (and (vectorp pat) (equal pat (vconcat line)))
                                (string-match-p pat line))
                      (error "Unknown request: %s" line))
+                   (setq resp (apply #'unibyte-string (append resp nil)))
                    (let ((print-escape-control-characters t))
                      (message "[%s] <- %s" name (prin1-to-string line))
                      (message "[%s] -> %s" name (prin1-to-string resp)))
-                   (process-send-string proc (concat resp)))))
+                   (process-send-string proc resp))))
          (serv (make-network-process :server 1
                                      :buffer (get-buffer-create name)
                                      :filter filt
@@ -151,8 +153,10 @@ socks-tests-canned-server-create
                                      :family 'ipv4
                                      :host 'local
                                      :coding 'binary
-                                     :service port)))
+                                     :service (or port t))))
     (set-process-query-on-exit-flag serv nil)
+    (unless (numberp (nth 2 socks-server))
+      (setf (nth 2 socks-server) (process-contact serv :service)))
     serv))
 
 (defvar socks-tests--hello-world-http-request-pattern
@@ -191,7 +195,7 @@ socks-tests-perform-hello-world-http-request
 
 (ert-deftest socks-tests-v4-basic ()
   "Show correct preparation of SOCKS4 connect command (Bug#46342)."
-  (let ((socks-server '("server" "127.0.0.1" 10079 4))
+  (let ((socks-server '("server" "127.0.0.1" t 4))
         (url-user-agent "Test/4-basic")
         (socks-tests-canned-server-patterns
          `(([4 1 0 80 93 184 216 34 ?f ?o ?o 0] . [0 90 0 0 0 0 0 0])
@@ -213,7 +217,7 @@ socks-tests-v4-basic
 (ert-deftest socks-tests-v5-auth-user-pass ()
   "Verify correct handling of SOCKS5 user/pass authentication."
   (should (assq 2 socks-authentication-methods))
-  (let ((socks-server '("server" "127.0.0.1" 10080 5))
+  (let ((socks-server '("server" "127.0.0.1" t 5))
         (socks-username "foo")
         (socks-password "bar")
         (url-user-agent "Test/auth-user-pass")
@@ -247,7 +251,7 @@ socks-tests-v5-auth-user-pass
 (ert-deftest socks-tests-v5-auth-user-pass-blank ()
   "Verify correct SOCKS5 user/pass authentication with empty pass."
   (should (assq 2 socks-authentication-methods))
-  (let ((socks-server '("server" "127.0.0.1" 10081 5))
+  (let ((socks-server '("server" "127.0.0.1" t 5))
         (socks-username "foo") ; defaults to (user-login-name)
         (socks-password "") ; simulate user hitting enter when prompted
         (url-user-agent "Test/auth-user-pass-blank")
@@ -266,7 +270,7 @@ socks-tests-v5-auth-user-pass-blank
 
 (ert-deftest socks-tests-v5-auth-none ()
   "Verify correct handling of SOCKS5 when auth method 0 requested."
-  (let ((socks-server '("server" "127.0.0.1" 10082 5))
+  (let ((socks-server '("server" "127.0.0.1" t 5))
         (socks-authentication-methods (append socks-authentication-methods
                                               nil))
         (url-user-agent "Test/auth-none")
-- 
2.38.1


--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
 filename=0002-30.0.50-Improve-SOCKS-error-handling-and-add-support.patch

From 0f199273a45210ba577e958e9b9205f1e4fcc9a7 Mon Sep 17 00:00:00 2001
From: "F. Jason Park" <jp@HIDDEN>
Date: Mon, 14 Feb 2022 02:36:57 -0800
Subject: [PATCH 2/4] [30.0.50] Improve SOCKS error handling and add support
 for 4a

* lisp/net/socks.el (socks-server): Add new choice `4a' to version
field of option.  This may appear to change the type of the field from
a number to a union of symbols and numbers.  However,
`socks-send-command' and `socks-filter' already expect a possible
`http' value for this field (also a symbol).
(socks--errors-4): Add new constant containing error messages for
version 4.  The semantics are faithful to the spec, but the exact
wording is adapted.
(socks-filter): Allow for a null "type" field on error with version 5.
In some cases, errors from certain servers were inaccessible.
(socks-send-command): Massage existing handling for version 4 to
accommodate 4a.

* test/lisp/net/socks-tests.el (socks-tests-v4a-basic): Add test for
SOCKS version 4a.  (Bug#53941.)
---
 lisp/net/socks.el            | 26 +++++++++++++++++++++++---
 test/lisp/net/socks-tests.el | 13 +++++++++++++
 2 files changed, 36 insertions(+), 3 deletions(-)

diff --git a/lisp/net/socks.el b/lisp/net/socks.el
index 2ba1c20566..0e84a2d594 100644
--- a/lisp/net/socks.el
+++ b/lisp/net/socks.el
@@ -162,6 +162,7 @@ socks-server
 	  (radio-button-choice :tag "SOCKS Version"
 			       :format "%t: %v"
 			       (const :tag "SOCKS v4  " :format "%t" :value 4)
+                               (const :tag "SOCKS v4a"  :format "%t" :value 4a)
 			       (const :tag "SOCKS v5"   :format "%t" :value 5))))
 
 
@@ -202,6 +203,12 @@ socks-errors
     "Command not supported"
     "Address type not supported"))
 
+(defconst socks--errors-4
+  '("Granted"
+    "Rejected or failed"
+    "Cannot connect to identd on the client"
+    "Client and identd report differing user IDs"))
+
 ;; The socks v5 address types
 (defconst socks-address-type-v4   1)
 (defconst socks-address-type-name 3)
@@ -309,7 +316,8 @@ socks-filter
 		     ((pred (= socks-address-type-name))
 		      (if (< (length string) 5)
 			  255
-		        (+ 1 (aref string 4)))))))
+                        (+ 1 (aref string 4))))
+                     (0 0))))
 	  (if (< (length string) desired-len)
 	      nil			; Need to spin some more
 	    (process-put proc 'socks-state socks-state-connected)
@@ -400,6 +408,7 @@ socks-send-command
 		(format "%c%s" (length address) address))
 	       (t
 		(error "Unknown address type: %d" atype))))
+        trailing
 	request version)
     (or (process-get proc 'socks)
         (error "socks-send-command called on non-SOCKS connection %S" proc))
@@ -416,6 +425,12 @@ socks-send-command
 			     (t
 			      (error "Unsupported address type for HTTP: %d" atype)))
 			    port)))
+     ((when (eq version '4a)
+        (setf addr "\0\0\0\1"
+              trailing (concat address "\0")
+              version 4 ; done with the "a" part
+              (process-get proc 'socks-server-protocol) 4)
+        nil)) ; fall through
      ((equal version 4)
       (setq request (concat
 		     (unibyte-string
@@ -425,7 +440,8 @@ socks-send-command
 		      (logand port #xff)) ; port, low byte
 		     addr                 ; address
 		     (user-full-name)     ; username
-		     "\0")))              ; terminate username
+                     "\0"                 ; terminate username
+                     trailing)))          ; optional host to look up
      ((equal version 5)
       (setq request (concat
 		     (unibyte-string
@@ -446,7 +462,11 @@ socks-send-command
 	nil				; Sweet sweet success!
       (delete-process proc)
       (error "SOCKS: %s"
-             (nth (or (process-get proc 'socks-reply) 1) socks-errors)))
+             (let ((err (process-get proc 'socks-reply)))
+               (if (eql version 5)
+                   (nth (or err 1) socks-errors)
+                 (nth (- (if (and err (<= 90 err 93)) err 91) 90)
+                      socks--errors-4)))))
     proc))
 
 
diff --git a/test/lisp/net/socks-tests.el b/test/lisp/net/socks-tests.el
index f1ecf1630f..9e341ebd4d 100644
--- a/test/lisp/net/socks-tests.el
+++ b/test/lisp/net/socks-tests.el
@@ -210,6 +210,19 @@ socks-tests-v4-basic
                  (lambda (&optional _) "foo")))
         (socks-tests-perform-hello-world-http-request)))))
 
+(ert-deftest socks-tests-v4a-basic ()
+  "Show correct preparation of SOCKS4a connect command."
+  (let ((socks-server '("server" "127.0.0.1" t 4a))
+        (url-user-agent "Test/4a-basic")
+        (socks-tests-canned-server-patterns
+         `(([4 1 0 80 0 0 0 1 ?f ?o ?o 0 ?e ?x ?a ?m ?p ?l ?e ?. ?c ?o ?m 0]
+            . [0 90 0 0 0 0 0 0])
+           ,socks-tests--hello-world-http-request-pattern)))
+    (ert-info ("Make HTTP request over SOCKS4A")
+      (cl-letf (((symbol-function 'user-full-name)
+                 (lambda (&optional _) "foo")))
+        (socks-tests-perform-hello-world-http-request)))))
+
 ;; Replace first pattern below with ([5 3 0 1 2] . [5 2]) to validate
 ;; against curl 7.71 with the following options:
 ;; $ curl --verbose -U foo:bar --proxy socks5h://127.0.0.1:10080 example.com
-- 
2.38.1


--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
 filename=0003-WIP-30.0.50-Simplify-network-stream-openers-in-socks.patch

From cee91dc42dd90bce6faab93bcfe85f2889166ebd Mon Sep 17 00:00:00 2001
From: "F. Jason Park" <jp@HIDDEN>
Date: Mon, 28 Nov 2022 22:31:50 -0800
Subject: [PATCH 3/4] [WIP/30.0.50] Simplify network-stream openers in socks.el

* lisp/net/socks.el (socks-open-network-stream-tls-services): Add new
custom option for specifying ports whose proxied connections should
use TLS.
(socks-connect-function): New option for specifying an
`open-network-stream'-like connect function.
(socks-open-connection): Accept additional `open-network-stream'
params passed on to opener, now `socks-connect-function'
in place of `open-network-stream'.
(socks-open-network-stream): Recognize additional `url' struct param.
Also prefer parsed URL details when present in `url-using-proxy'.
(socks--initiate-command-connect): New function to house renamed
latter half of the original `socks--open-network-stream'.  Role now
reduced issuing the first command using an existing process.
(socks--open-network-stream): Serve as thin advice-only wrapper for
`socks-open-network-stream'.
---
 lisp/net/socks.el | 116 +++++++++++++++++++++++++++++++++++++---------
 1 file changed, 95 insertions(+), 21 deletions(-)

diff --git a/lisp/net/socks.el b/lisp/net/socks.el
index 0e84a2d594..65436ed047 100644
--- a/lisp/net/socks.el
+++ b/lisp/net/socks.el
@@ -34,7 +34,7 @@
 
 ;;; Code:
 
-(eval-when-compile (require 'cl-lib))
+(eval-when-compile (require 'cl-lib) (require 'url-parse))
 
 ;; ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
 ;; ;;; Custom widgets
@@ -335,15 +335,24 @@ socks-override-functions
 (when socks-override-functions
   (advice-add 'open-network-stream :around #'socks--open-network-stream))
 
-(defun socks-open-connection (server-info)
+(defcustom socks-connect-function 'open-network-stream
+  "Function to open a network connection to a SOCKS provider.
+Called with arguments suitable for `open-network-stream'."
+  :version "30.1"
+  :type '(choice (function-item :value open-network-stream)
+                 (function :tag "User-provided function")))
+
+(defun socks-open-connection (server-info &rest stream-params)
+  "Create and initialize a SOCKS process.
+Perform authentication if needed.  Expect SERVER-INFO to take the
+form of `socks-server' and STREAM-PARAMS to be keyword params
+accepted by `open-network-stream'."
   (interactive)
   (save-excursion
     (let ((proc
            (let ((socks-override-functions nil))
-             (open-network-stream "socks"
-				  nil
-				  (nth 1 server-info)
-				  (nth 2 server-info))))
+             (apply socks-connect-function (nth 0 server-info) nil
+                    (nth 1 server-info) (nth 2 server-info) stream-params)))
 	  (authtype nil)
 	  version)
 
@@ -527,22 +536,87 @@ socks-find-services-entry
   (gethash (downcase service)
 	      (if udp socks-udp-services socks-tcp-services)))
 
-(defun socks-open-network-stream (name buffer host service)
-  (let ((socks-override-functions t))
-    (socks--open-network-stream
-     (lambda (&rest args)
-       (let ((socks-override-functions nil))
-         (apply #'open-network-stream args)))
-     name buffer host service)))
-
 (defun socks--open-network-stream (orig-fun name buffer host service &rest params)
-  (let ((route (and socks-override-functions
-                    (socks-find-route host service))))
-    (if (not route)
-	(apply orig-fun name buffer host service params)
-      ;; FIXME: Obey `params'!
-      (let* ((proc (socks-open-connection route))
-	     (version (process-get proc 'socks-server-protocol))
+  (let ((socks-override-functions orig-fun))
+    (apply #'socks-open-network-stream name buffer host service params)))
+
+(defcustom socks-proxied-tls-services '(443 6697)
+  "Ports whose connections should use TLS.
+Note that the system resolver may be consulted to look up host
+names for checking domain validation certs."
+  :version "30.1"
+  :type '(repeat number))
+
+(declare-function gnutls-negotiate "gnutls" (&rest rest))
+(declare-function nsm-verify-connection "nsm"
+                  (process host port &optional
+                           save-fingerprint warn-unencrypted))
+
+;;;###autoload
+(defun socks-open-network-stream (name buffer host service &rest params)
+  "Open and return a connection, possibly proxied over SOCKS.
+Expect PARAMS to contain keyword parameters recognized by
+`open-network-stream'.  Assume HOST and SERVICE refer to the
+proxied remote peer rather than the SOCKS server, but assume the
+opposite for PARAMS.  That is, if PARAMS contains a `:type' of
+`tls', treat the underlying connection to the proxy server as
+destined for encryption rather than the tunneled connection (even
+though `socks-connect-function' has the final say).  For TLS with
+proxied connections, see the option `socks-proxied-tls-services'.
+
+Before connecting, check the host against `socks-noproxy' and, on
+rejection, fall back to non-SOCKS.  Similarly, when
+`socks-override-functions' is a function, call it directly and
+trust that it's not interested in options defined in this
+library, such as `socks-server'.
+
+But, before doing anything, check if `url-using-proxy' is bound
+to a `url' struct object, as defined in `url-parse'.  If so,
+assume it represents the address of the desired SOCKS server
+rather than that of the remote peer, and use its fields instead
+of `socks-server' for all SOCKS connection details."
+  (require 'url-parse)
+  (let* ((url (and (url-p url-using-proxy)
+                   (string-prefix-p "socks" (url-type url-using-proxy))
+                   url-using-proxy))
+         (socks-server (if url
+                           (list name (url-host url) (url-port url)
+                                 (pcase (url-type url)
+                                   ("socks4://" 4)
+                                   ("socks4a://" '4a)
+                                   (_ 5)))
+                         socks-server))
+         (socks-username (or (and url (url-user url))
+                             socks-username))
+         (socks-password (or (and url (url-password url))
+                             socks-password)))
+    (if-let* (((booleanp socks-override-functions))
+              (route (socks-find-route host service))
+              (proc (apply #'socks-open-connection route params)))
+        (let ((port (if (numberp service)
+                        service
+                      (process-contact proc :service)))
+              (certs (plist-get params :client-certificate)))
+          (socks--initiate-command-connect proc buffer host service)
+          (if (and (memq port socks-proxied-tls-services)
+                   (gnutls-available-p)
+                   (require 'gnutls nil t)
+                   (require 'nsm nil t))
+              (progn (gnutls-negotiate :process proc
+                                       :hostname host
+                                       :keylist (and certs (list certs)))
+                     (nsm-verify-connection proc host port))
+            proc))
+      (let ((fn (if (functionp socks-override-functions)
+                    socks-override-functions ; `socks-noproxy' not consulted
+                  #'open-network-stream)) ; `socks-noproxy' is non-nil
+            socks-override-functions)
+        (apply fn name buffer host service params)))))
+
+(defun socks--initiate-command-connect (proc buffer host service)
+  (progn ; preserve indentation level for git blame / code review
+    (progn
+      (let* ((version (process-get proc 'socks-server-protocol))
              (atype
               (cond
                ((equal version 4)
-- 
2.38.1


--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
 filename=0004-POC-30.0.50-Integrate-the-socks-and-url-libraries.patch

From 96a4de741663672e928fd30af6c93b335b346691 Mon Sep 17 00:00:00 2001
From: "F. Jason Park" <jp@HIDDEN>
Date: Tue, 1 Mar 2022 01:38:33 -0800
Subject: [PATCH 4/4] [POC/30.0.50] Integrate the socks and url libraries

* lisp/url/url-gw.el (url-open-stream): Use presence and type of
`url-using-proxy' to detect caller and massage input values according
to legacy practices.

* lisp/url/url-http.el: (url-http-find-free-connection): Don't call
`url-open-stream' with host and port from active proxy.
(url-http, url-http-async-sentinel): Only open
`url-https-proxy-connect' for non-SOCKS proxies.

* lisp/url/url-proxy.el (url-proxy--socks-scheme-regexp): Add new
const.
(url-default-find-proxy-for-url): Accommodate SOCKS entries but defy
original design somewhat by requiring a URL scheme in the host value
for detection.
(url-find-proxy-for-url): Recognize modified host/address value for
socks entries of `url-proxy-services' and deal accordingly.
(url-proxy): Handle a SOCKS proxy for http(s) connections only.

* lisp/url/url-vars.el (url-proxy-services): Explain that values for
certain gateways may need a leading scheme:// portion.
(url-using-proxy): Add warning regarding expected type.
---
 lisp/url/url-gw.el    |  8 +++++++-
 lisp/url/url-http.el  | 16 +++++++---------
 lisp/url/url-proxy.el | 18 ++++++++++++++++--
 lisp/url/url-vars.el  | 13 ++++++++++---
 4 files changed, 40 insertions(+), 15 deletions(-)

diff --git a/lisp/url/url-gw.el b/lisp/url/url-gw.el
index e4d1ca72a0..c93edc0d4e 100644
--- a/lisp/url/url-gw.el
+++ b/lisp/url/url-gw.el
@@ -28,7 +28,7 @@
 (require 'url-vars)
 (require 'url-parse)
 
-(autoload 'socks-open-network-stream "socks")
+(autoload 'socks-open-network-stream "socks") ; FIXME remove this
 
 (defgroup url-gateway nil
   "URL gateway variables."
@@ -220,6 +220,12 @@ url-open-stream
 Optional arg GATEWAY-METHOD specifies the gateway to be used,
 overriding the value of `url-gateway-method'."
   (unless url-gateway-unplugged
+    (when (url-p url-using-proxy)
+      (if (or (eq 'socks url-gateway-method)
+              (string-prefix-p "socks" (url-type url-using-proxy)))
+          (setq gateway-method 'socks)
+        (setq host (url-host url-using-proxy)
+              service (url-port url-using-proxy))))
     (let* ((gwm (or gateway-method url-gateway-method))
            (gw-method (if (and url-gateway-local-host-regexp
                                (not (eq 'tls gwm))
diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
index 94ef156108..864048a73c 100644
--- a/lisp/url/url-http.el
+++ b/lisp/url/url-http.el
@@ -195,12 +195,7 @@ url-http-find-free-connection
 	;; like authentication.  But we use another buffer afterwards.
 	(unwind-protect
             (let ((proc (url-open-stream host buf
-                                         (if url-using-proxy
-                                             (url-host url-using-proxy)
-                                           host)
-                                         (if url-using-proxy
-                                             (url-port url-using-proxy)
-                                           port)
+                                         host port
                                          gateway-method)))
 	      ;; url-open-stream might return nil.
 	      (when (processp proc)
@@ -1396,8 +1391,9 @@ url-http
            (error "Could not create connection to %s:%d" (url-host url)
                   (url-port url)))
           (_
-           (if (and url-http-proxy (string= "https"
-                                            (url-type url-current-object)))
+           (if (and url-http-proxy
+                    (not (string-prefix-p "socks" (url-type url-http-proxy)))
+                    (string= "https" (url-type url-current-object)))
                (url-https-proxy-connect connection)
              (set-process-sentinel connection
                                    #'url-http-end-of-document-sentinel)
@@ -1479,7 +1475,9 @@ url-http-async-sentinel
 	(url-http-end-of-document-sentinel proc why))
        ((string= (substring why 0 4) "open")
 	(setq url-http-connection-opened t)
-        (if (and url-http-proxy (string= "https" (url-type url-current-object)))
+        (if (and url-http-proxy
+                 (not (string-prefix-p "socks" (url-type url-http-proxy)))
+                 (string= "https" (url-type url-current-object)))
             (url-https-proxy-connect proc)
           (condition-case error
               (process-send-string proc (url-http-create-request))
diff --git a/lisp/url/url-proxy.el b/lisp/url/url-proxy.el
index c72e459a4e..f4ddd639f6 100644
--- a/lisp/url/url-proxy.el
+++ b/lisp/url/url-proxy.el
@@ -25,6 +25,9 @@
 
 (require 'url-parse)
 
+(defconst url-proxy--socks-scheme-regexp
+  (rx bot "socks" (? (or "4" "4a" "5" "5h")) "://"))
+
 (defun url-default-find-proxy-for-url (urlobj host)
   (cond
    ((or (and (assoc "no_proxy" url-proxy-services)
@@ -35,7 +38,12 @@ url-default-find-proxy-for-url
 	(equal "www" (url-type urlobj)))
     "DIRECT")
    ((cdr (assoc (url-type urlobj) url-proxy-services))
-    (concat "PROXY " (cdr (assoc (url-type urlobj) url-proxy-services))))
+    (let ((found (alist-get (url-type urlobj) url-proxy-services
+                            nil nil #'equal)))
+      (concat (if (string-match url-proxy--socks-scheme-regexp found)
+                  "SOCKS "
+                "PROXY ")
+              found)))
    ;;
    ;; Should check for socks
    ;;
@@ -57,7 +65,10 @@ url-find-proxy-for-url
      ((string-match "^PROXY +" proxy)
       (concat "http://" (substring proxy (match-end 0)) "/"))
      ((string-match "^SOCKS +" proxy)
-      (concat "socks://" (substring proxy (match-end 0))))
+      (if-let* ((m (substring proxy (match-end 0)))
+                ((string-match url-proxy--socks-scheme-regexp m)))
+          m
+        (concat "socks://" m)))
      (t
       (display-warning 'url (format "Unknown proxy directive: %s" proxy) :error)
       nil))))
@@ -72,6 +83,9 @@ url-proxy
   (cond
    ((string= (url-type url-using-proxy) "http")
     (url-http url callback cbargs))
+   ((and (string-prefix-p "socks" (url-type url-using-proxy))
+         (string-prefix-p "http" (url-type url)))
+    (url-http url callback cbargs))
    (t
     (error "Don't know how to use proxy `%s'" url-using-proxy))))
 
diff --git a/lisp/url/url-vars.el b/lisp/url/url-vars.el
index 4cdca05554..209d387ea7 100644
--- a/lisp/url/url-vars.el
+++ b/lisp/url/url-vars.el
@@ -191,11 +191,16 @@ url-mail-command
 
 (defcustom url-proxy-services nil
   "An alist of schemes and proxy servers that gateway them.
-Looks like ((\"http\" . \"hostname:portnumber\") ...).  This is set up
-from the ACCESS_proxy environment variables."
+Looks like ((\"http\" . \"hostname:portnumber\") ...).  This is
+set up from the ACCESS_proxy environment variables.  Depending on
+the gateway type, values may instead be expected to look like
+\"proxyscheme://hostname:portnumber\" where \"proxyscheme\" is
+something like \"socks5\".  As of Emacs 30.1, this only applies
+to SOCKS servers."
   :type '(repeat (cons :format "%v"
 		       (string :tag "Protocol")
 		       (string :tag "Proxy")))
+  :version "30.1"
   :group 'url)
 
 (defcustom url-standalone-mode nil
@@ -310,7 +315,9 @@ url-show-status
 
 (defvar url-using-proxy nil
   "Either nil or the fully qualified proxy URL in use, e.g.
-https://www.example.com/")
+https://www.example.com/.  Beware that some functions, such as
+`url-proxy' and `url-http-end-of-document-sentinel', set this to
+a `url' struct.")
 
 (defcustom url-news-server nil
   "The default news server from which to get newsgroups/articles.
-- 
2.38.1


--=-=-=--

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#53941: Last-minute socks.el improvements for Emacs 29?
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Tue, 29 Nov 2022 14:36:02 +0000
Resent-Message-ID: <handler.53941.B53941.166973254012043 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 53941
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: patch
To: "J.P." <jp@HIDDEN>
Cc: larsi@HIDDEN, gnuhacker@HIDDEN, 53941 <at> debbugs.gnu.org
Received: via spool by 53941-submit <at> debbugs.gnu.org id=B53941.166973254012043
          (code B ref 53941); Tue, 29 Nov 2022 14:36:02 +0000
Received: (at 53941) by debbugs.gnu.org; 29 Nov 2022 14:35:40 +0000
Received: from localhost ([127.0.0.1]:54568 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1p01i0-00038B-Bd
	for submit <at> debbugs.gnu.org; Tue, 29 Nov 2022 09:35:40 -0500
Received: from eggs.gnu.org ([209.51.188.92]:35308)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1p01hy-000384-7k
 for 53941 <at> debbugs.gnu.org; Tue, 29 Nov 2022 09:35:39 -0500
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1p01hs-0000O5-Ao; Tue, 29 Nov 2022 09:35:32 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=fUgPMV1q161m0VpeblH/Po7ymom43IQo2JmHcHpECiA=; b=es4BJOc9AeTG
 JVRcqZQEA7lt1QonrYPgRWCng+ZBo2cnTNag1HgnJCDchl/+KFaoxFGiGbsnbS8zJ0LBEEbCgn1yx
 ZORT4sZrLeji+1jBUCPR9EPxItoVgvdeKDqoMXdR4udbngpSjlq15qjHgqgfEaPszWA5e1FdYedpZ
 NnW1geIqEdXDIG9oA/SagwGj8ltyDZpJFQRW2kZ0TMTE6EZfQk4+UtWMikX2/RI75Njw1TwtEk2zO
 Crf0uq34ZlzI4huy58EwovZPFN6kUpCe54zZwn2bFnFS5lcNOzoOT6AVYGrZ/za76hbV9oCpg/KHb
 wWaNFNauSmWlCLc0ovyMvQ==;
Received: from [87.69.77.57] (helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1p01hr-0006fb-KF; Tue, 29 Nov 2022 09:35:32 -0500
Date: Tue, 29 Nov 2022 16:36:03 +0200
Message-Id: <831qpln7zg.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <87fse1kfe8.fsf@HIDDEN> (jp@HIDDEN)
References: <87pmntfym7.fsf@HIDDEN> <8735kl1v58.fsf@HIDDEN>
 <87a6emftzx.fsf@HIDDEN> <87k0do5km1.fsf@HIDDEN>
 <87pmn5n3tu.fsf@HIDDEN> <87mti99j1f.fsf@HIDDEN>
 <87wnh7hkgi.fsf@HIDDEN> <87pmmz947k.fsf@HIDDEN>
 <8735ju44sk.fsf@HIDDEN> <87lexikwu5.fsf@HIDDEN>
 <87mt8baygn.fsf_-_@HIDDEN> <8335a3nguk.fsf@HIDDEN>
 <87fse1kfe8.fsf@HIDDEN>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: "J.P." <jp@HIDDEN>
> Cc: 53941 <at> debbugs.gnu.org,  gnuhacker@HIDDEN,  larsi@HIDDEN
> Date: Tue, 29 Nov 2022 06:24:15 -0800
> 
> Eli Zaretskii <eliz@HIDDEN> writes:
> 
> > I'm really uncomfortable with installing these changes before the release
> > branch is cut.  The changes are hardly trivial, some controversial even to
> > my eyes, even though I'm no expert on network connections.
> 
> Well, I myself am just about the furthest thing from (an expert), which
> certainly doesn't comport well with dropping rash changes at the
> eleventh hour. (That was rather disrespectful on my part, so shame on
> me.) As such, if it's easier to revisit this once things settle down,
> just ignore this email and I'll re-ping you sometime down the road.

Yes, please do.  The branch is cut now, so if we agree on installing such
changes on master, it's now up to you when to post another version of these
changes with the requisite fixes.

> >> +(defun socks-open-network-stream (name buffer host service &rest params)
> >> +  "Open and return a connection, possibly proxied over SOCKS.
> >
> > The changes in this public function are so significant that I don't
> > understand how they can be suggested so close to the branching.
> 
> The old signature was
> 
>   (name buffer host service) -> process

I didn't mean the signature (which is OK), I meant the body.  It is very
different from the previous version.

> > If it is possible to add support for SOCKS 4a without affecting any
> > previously supported versions, I'm fine.  Adding tests is also fine.
> > But for the rest, I think you should wait until after the release
> > branch is cut and install this on the master branch. Sorry, it really
> > is too late for such changes.
> 
> You're very gracious, but I think I've learned my lesson and will
> refrain from pursuing any of these changes for Emacs 29. Apologies for
> abusing your time and maintainerly patience (yet again).

No need to apologize, this is software development.

Thanks for working on these issues.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#53941: 27.2; socks + tor dont work with https
Resent-From: Stefan Kangas <stefankangas@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Wed, 06 Sep 2023 22:26:01 +0000
Resent-Message-ID: <handler.53941.B53941.169403913326731 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 53941
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: patch
To: Eli Zaretskii <eliz@HIDDEN>
Cc: 53941 <at> debbugs.gnu.org, larsi@HIDDEN, gnuhacker@HIDDEN, "J.P." <jp@HIDDEN>
Received: via spool by 53941-submit <at> debbugs.gnu.org id=B53941.169403913326731
          (code B ref 53941); Wed, 06 Sep 2023 22:26:01 +0000
Received: (at 53941) by debbugs.gnu.org; 6 Sep 2023 22:25:33 +0000
Received: from localhost ([127.0.0.1]:38056 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qe0xp-0006x5-Cg
	for submit <at> debbugs.gnu.org; Wed, 06 Sep 2023 18:25:33 -0400
Received: from mail-lf1-x136.google.com ([2a00:1450:4864:20::136]:59871)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <stefankangas@HIDDEN>) id 1qe0xj-0006wi-HO
 for 53941 <at> debbugs.gnu.org; Wed, 06 Sep 2023 18:25:31 -0400
Received: by mail-lf1-x136.google.com with SMTP id
 2adb3069b0e04-500b66f8b27so465032e87.3
 for <53941 <at> debbugs.gnu.org>; Wed, 06 Sep 2023 15:25:26 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=gmail.com; s=20221208; t=1694039120; x=1694643920; darn=debbugs.gnu.org;
 h=cc:to:subject:message-id:date:mime-version:references:in-reply-to
 :from:from:to:cc:subject:date:message-id:reply-to;
 bh=LlbKSrP8Rj8h2Po0KKD1Gh8MrEZ94BpUrUtYWuDFnJ0=;
 b=EFCIH55K5Bs0ghTBRjLr5Zgjv0wQokjQoisvLOdmdWvZjL1ZnXqN0Xy51Wkd7AJZQT
 VNmjctb1BeIee3wjS86ofl0FdFoRy/cLzKklCGby9j8rX+WxPvp3JZ3mh4HSLpwt0gDq
 wxDwwpsDf4+HkzMJg4vwxrdR0haKS8i83miHx6pqGAsDoJkYd2KrFWELDa5l6h69OODw
 yM45IhOPWlcvlZekG8sFQfkiH70dZoZbJrbUbyIxA5Ml9Oo0bkTx8hkSh0EoUlgMCSyN
 gM6yjYYycmBNOH4aeV7SVUX9SujvjALJY7hjMXGTDiZSu42tmBg3uUgXEie/Xzkl0DGH
 VBlA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20221208; t=1694039120; x=1694643920;
 h=cc:to:subject:message-id:date:mime-version:references:in-reply-to
 :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to;
 bh=LlbKSrP8Rj8h2Po0KKD1Gh8MrEZ94BpUrUtYWuDFnJ0=;
 b=kxg+poqkjn1Zc3IbSbUBZAshVbPECGSTRBz8uyzI5So2sR8wwhZwvW86PYRVFRuyii
 IOBxPDLI18yleaIGGeePLga5Z+WxZYQ9xT6Z0FITDzUcHumDyWEwhuhb9UR3tAJQ5+q4
 BVSLypF0NyiAs0uUWMUhDbc2dimSdXHYYJUhKMfT04ewyCsQ1UX+8eou1NeVBLv2K2yr
 B/ocq8BgQP47VYq9uX9AyAeZYVedDWyriQaSS6UknLykeQfZZSJF/t5hbOkWYX4TA9rj
 ArYQDGgOgN1VUWBiuoLcN3C58HVid7Ts94O/gt/UqrNJGFpr060RHIVhjwFZxel+y969
 iJnw==
X-Gm-Message-State: AOJu0YyZ0pWwdPee8T0Fldn5wiRUMkYl3OUjC9ZbgHDjNHc5+AYK2cKU
 V/hTQxG1ou3hVoBMPEOIPeQdVWTyQKAoKnEL5qc=
X-Google-Smtp-Source: AGHT+IFPjaXuWI2Dt0FcY+5EjZhV1LjKmTXAAw+cwlQJRXxHcZI90rum4kxOi9cShcGXmPUFgW3gwsRpbkRPprcDnu8=
X-Received: by 2002:a19:384c:0:b0:500:b2f6:592 with SMTP id
 d12-20020a19384c000000b00500b2f60592mr2915259lfj.50.1694039120016; Wed, 06
 Sep 2023 15:25:20 -0700 (PDT)
Received: from 753933720722 named unknown by gmailapi.google.com with
 HTTPREST; Wed, 6 Sep 2023 15:25:19 -0700
From: Stefan Kangas <stefankangas@HIDDEN>
In-Reply-To: <831qpln7zg.fsf@HIDDEN> (Eli Zaretskii's message of "Tue, 29 Nov
 2022 16:36:03 +0200")
References: <87pmntfym7.fsf@HIDDEN> <8735kl1v58.fsf@HIDDEN>
 <87a6emftzx.fsf@HIDDEN> <87k0do5km1.fsf@HIDDEN>
 <87pmn5n3tu.fsf@HIDDEN>
 <87mti99j1f.fsf@HIDDEN> <87wnh7hkgi.fsf@HIDDEN>
 <87pmmz947k.fsf@HIDDEN>
 <8735ju44sk.fsf@HIDDEN> <87lexikwu5.fsf@HIDDEN>
 <87mt8baygn.fsf_-_@HIDDEN>
 <8335a3nguk.fsf@HIDDEN> <87fse1kfe8.fsf@HIDDEN> <831qpln7zg.fsf@HIDDEN>
MIME-Version: 1.0
Date: Wed, 6 Sep 2023 15:25:19 -0700
Message-ID: <CADwFkmm0hCO3ek-w0TWkGWOROCU9YfOgDO0ivtcz-Q7feUE7nA@HIDDEN>
Content-Type: text/plain; charset="UTF-8"
X-Spam-Score: 0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Eli Zaretskii <eliz@HIDDEN> writes:

>> > I'm really uncomfortable with installing these changes before the release
>> > branch is cut.  The changes are hardly trivial, some controversial even to
>> > my eyes, even though I'm no expert on network connections.
>>
>> Well, I myself am just about the furthest thing from (an expert), which
>> certainly doesn't comport well with dropping rash changes at the
>> eleventh hour. (That was rather disrespectful on my part, so shame on
>> me.) As such, if it's easier to revisit this once things settle down,
>> just ignore this email and I'll re-ping you sometime down the road.
>
> Yes, please do.  The branch is cut now, so if we agree on installing such
> changes on master, it's now up to you when to post another version of these
> changes with the requisite fixes.

Could we revisit this now?  It sounds like something we'd want to fix.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#53941: 27.2; socks + tor dont work with https
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Thu, 07 Sep 2023 05:55:02 +0000
Resent-Message-ID: <handler.53941.B53941.169406604816374 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 53941
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: patch
To: jp@HIDDEN, Stefan Kangas <stefankangas@HIDDEN>
Cc: larsi@HIDDEN, 53941 <at> debbugs.gnu.org, gnuhacker@HIDDEN
Received: via spool by 53941-submit <at> debbugs.gnu.org id=B53941.169406604816374
          (code B ref 53941); Thu, 07 Sep 2023 05:55:02 +0000
Received: (at 53941) by debbugs.gnu.org; 7 Sep 2023 05:54:08 +0000
Received: from localhost ([127.0.0.1]:38346 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qe7xw-0004G2-27
	for submit <at> debbugs.gnu.org; Thu, 07 Sep 2023 01:54:08 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:58032)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qe7xq-0004FU-A5
 for 53941 <at> debbugs.gnu.org; Thu, 07 Sep 2023 01:54:06 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qe7xi-0005Cw-Dr; Thu, 07 Sep 2023 01:53:54 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=Uq6ScRKM3eBaus8ywrQSq9yXnqVOacME79BzH6mKX3Y=; b=J6UeYt0SzPrj
 KsoIs4s7PLsLB16JxUpXic6fKjQd4yHftO5Xs+aIat/uWo8AVBj9e7MDO8zD+V4EUhTmMEXrQUSxg
 4dBcVCsuMDowuOnZJODLTzUmMFpxy2lBMoJZ4qaweaZbOOkjUH48T7tjMjTs+hj+sUbWj3UBmy8dK
 phekN/bOeMfftwHr4jLlKBFDYiw5AwVEp6/SspRCcJYACFnaZHsg2jMxqVITl9UUh4AKymG5zm+k4
 Mc3H/eL7LlxdbyxUqiLyPk41M9sesiMOkiAL4sfQrJ2GM0mbsAnEzrTq5QfeiKkUa2zbEokwg2H3J
 NGCqP1Xv4PH4QurJpl2B7g==;
Date: Thu, 07 Sep 2023 08:53:41 +0300
Message-Id: <837cp21qca.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <CADwFkmm0hCO3ek-w0TWkGWOROCU9YfOgDO0ivtcz-Q7feUE7nA@HIDDEN>
 (message from Stefan Kangas on Wed, 6 Sep 2023 15:25:19 -0700)
References: <87pmntfym7.fsf@HIDDEN> <8735kl1v58.fsf@HIDDEN>
 <87a6emftzx.fsf@HIDDEN> <87k0do5km1.fsf@HIDDEN>
 <87pmn5n3tu.fsf@HIDDEN>
 <87mti99j1f.fsf@HIDDEN> <87wnh7hkgi.fsf@HIDDEN>
 <87pmmz947k.fsf@HIDDEN>
 <8735ju44sk.fsf@HIDDEN> <87lexikwu5.fsf@HIDDEN>
 <87mt8baygn.fsf_-_@HIDDEN>
 <8335a3nguk.fsf@HIDDEN> <87fse1kfe8.fsf@HIDDEN> <831qpln7zg.fsf@HIDDEN>
 <CADwFkmm0hCO3ek-w0TWkGWOROCU9YfOgDO0ivtcz-Q7feUE7nA@HIDDEN>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: Stefan Kangas <stefankangas@HIDDEN>
> Date: Wed, 6 Sep 2023 15:25:19 -0700
> Cc: "J.P." <jp@HIDDEN>, larsi@HIDDEN, gnuhacker@HIDDEN, 
> 	53941 <at> debbugs.gnu.org
> 
> Eli Zaretskii <eliz@HIDDEN> writes:
> 
> >> > I'm really uncomfortable with installing these changes before the release
> >> > branch is cut.  The changes are hardly trivial, some controversial even to
> >> > my eyes, even though I'm no expert on network connections.
> >>
> >> Well, I myself am just about the furthest thing from (an expert), which
> >> certainly doesn't comport well with dropping rash changes at the
> >> eleventh hour. (That was rather disrespectful on my part, so shame on
> >> me.) As such, if it's easier to revisit this once things settle down,
> >> just ignore this email and I'll re-ping you sometime down the road.
> >
> > Yes, please do.  The branch is cut now, so if we agree on installing such
> > changes on master, it's now up to you when to post another version of these
> > changes with the requisite fixes.
> 
> Could we revisit this now?  It sounds like something we'd want to fix.

No objections from me, but I don't think we saw "another version"
posted.  J.P.?

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#53941: 27.2; socks + tor dont work with https
Resent-From: "J.P." <jp@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Thu, 07 Sep 2023 13:27:01 +0000
Resent-Message-ID: <handler.53941.B53941.169409316816510 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 53941
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: patch
To: Eli Zaretskii <eliz@HIDDEN>
Cc: larsi@HIDDEN, 53941 <at> debbugs.gnu.org, Stefan Kangas <stefankangas@HIDDEN>, gnuhacker@HIDDEN
Received: via spool by 53941-submit <at> debbugs.gnu.org id=B53941.169409316816510
          (code B ref 53941); Thu, 07 Sep 2023 13:27:01 +0000
Received: (at 53941) by debbugs.gnu.org; 7 Sep 2023 13:26:08 +0000
Received: from localhost ([127.0.0.1]:38877 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qeF1K-0004IC-3T
	for submit <at> debbugs.gnu.org; Thu, 07 Sep 2023 09:26:08 -0400
Received: from mail-108-mta206.mxroute.com ([136.175.108.206]:43933)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <jp@HIDDEN>) id 1qeF1E-0004Hr-UB
 for 53941 <at> debbugs.gnu.org; Thu, 07 Sep 2023 09:26:05 -0400
Received: from mail-111-mta2.mxroute.com ([136.175.111.2]
 filter006.mxroute.com) (Authenticated sender: mN4UYu2MZsgR)
 by mail-108-mta206.mxroute.com (ZoneMTA) with ESMTPSA id
 18a6fd2122a000d7b6.001 for <53941 <at> debbugs.gnu.org>
 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384);
 Thu, 07 Sep 2023 13:25:52 +0000
X-Zone-Loop: a27e54a860b1add2f2f520969f220ba03e0658006f36
X-Originating-IP: [136.175.111.2]
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=neverwas.me
 ; s=x;
 h=Content-Type:MIME-Version:Message-ID:Date:References:In-Reply-To:
 Subject:Cc:To:From:Sender:Reply-To:Content-Transfer-Encoding:Content-ID:
 Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc
 :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe:
 List-Post:List-Owner:List-Archive;
 bh=1QmIexs6BNPPG2ia/j+JzhEJ8u1eMxkCR6+YYpcoq3w=; b=bidvr2a+u8poEVAB5pv9MzPQPP
 CZFlpjJ0sZaHoLNKvD5vUXktBqzpMjoQSDesSyeQSQtoF1X3cf2fsyp/uUhMrKe1yMc3XxylCkW4v
 GJfNW2rF2WX4kJPs90OhNGeqKNr9hfIImJ5iyZcsjamzDPbwm9yqdvYAmQBMWOvhEQo3D+io70+bi
 mnsLKmA4pAImfz8QkaSX4Ei0xD64TqYQiZ6pIdgd4lvR877AdSVx8B4c0XhhYJh253iz5QX/qdgX9
 F1WXoB5JBsQKMX9h6JmsdD6d63RuTA3CISDMI1uy+2jZhs2zIg9slFdCpUKbRh+3z9nMazR0igofZ
 sZi2+uTA==;
From: "J.P." <jp@HIDDEN>
In-Reply-To: <837cp21qca.fsf@HIDDEN> (Eli Zaretskii's message of "Thu, 07 Sep
 2023 08:53:41 +0300")
References: <87pmntfym7.fsf@HIDDEN> <8735kl1v58.fsf@HIDDEN>
 <87a6emftzx.fsf@HIDDEN> <87k0do5km1.fsf@HIDDEN>
 <87pmn5n3tu.fsf@HIDDEN> <87mti99j1f.fsf@HIDDEN>
 <87wnh7hkgi.fsf@HIDDEN> <87pmmz947k.fsf@HIDDEN>
 <8735ju44sk.fsf@HIDDEN> <87lexikwu5.fsf@HIDDEN>
 <87mt8baygn.fsf_-_@HIDDEN> <8335a3nguk.fsf@HIDDEN>
 <87fse1kfe8.fsf@HIDDEN> <831qpln7zg.fsf@HIDDEN>
 <CADwFkmm0hCO3ek-w0TWkGWOROCU9YfOgDO0ivtcz-Q7feUE7nA@HIDDEN>
 <837cp21qca.fsf@HIDDEN>
Date: Thu, 07 Sep 2023 06:25:47 -0700
Message-ID: <878r9iktd0.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="=-=-="
X-Authenticated-Id: masked@HIDDEN
X-Spam-Score: 0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

--=-=-=
Content-Type: text/plain

Eli Zaretskii <eliz@HIDDEN> writes:

>> From: Stefan Kangas <stefankangas@HIDDEN>
>> Date: Wed, 6 Sep 2023 15:25:19 -0700
>> Cc: "J.P." <jp@HIDDEN>, larsi@HIDDEN, gnuhacker@HIDDEN, 
>> 	53941 <at> debbugs.gnu.org
>> 
>> Eli Zaretskii <eliz@HIDDEN> writes:
>> 
>> >> > I'm really uncomfortable with installing these changes before the release
>> >> > branch is cut.  The changes are hardly trivial, some controversial even to
>> >> > my eyes, even though I'm no expert on network connections.
>> >>
>> >> Well, I myself am just about the furthest thing from (an expert), which
>> >> certainly doesn't comport well with dropping rash changes at the
>> >> eleventh hour. (That was rather disrespectful on my part, so shame on
>> >> me.) As such, if it's easier to revisit this once things settle down,
>> >> just ignore this email and I'll re-ping you sometime down the road.
>> >
>> > Yes, please do.  The branch is cut now, so if we agree on installing such
>> > changes on master, it's now up to you when to post another version of these
>> > changes with the requisite fixes.
>> 
>> Could we revisit this now?  It sounds like something we'd want to fix.
>
> No objections from me, but I don't think we saw "another version"
> posted.  J.P.?

Hi, unfortunately, there is no other version of any real note, at least
not yet. The attached version is slightly updated but lacks substantial
progress in the areas that matter most. As of now, the only patches I'd
be comfortable offering would be the first two, which aren't even
directly related to this bug.

In addition to providing some cosmetic refactoring [1], the third patch
is mainly just a working placeholder for the socks.el side of a
hypothetical integration with the URL framework (currently imagined as
an overhauled version of `socks-open-connection' made newly compatible
with the `url-proxy' and `open-network-stream' interfaces). A sketch of
the URL side of such an arrangement can be found in the fourth patch,
but it isn't well thought out (or thoroughly researched).

AFAIR, the OP's main complaint concerns the lack of a transparent and
easily configurable experience for proxying built-in Emacs applications
over SOCKS, perhaps compared to what you'd get with a typical web
browser. I think we can probably agree that users shouldn't have to
customize both `socks'- and `url'-owned options or write extra code to
achieve a working setup, which is how things are currently (AFAICT).

If we're to solve this using the `url' library's existing public API, it
may need to be more clearly defined in a few areas. For example, at
least one function (`url-http-find-free-connection') uses the `host' and
`service' parameters of `url-open-stream' for dialing HTTP proxies. But
if other protocols are meant to do the same, they'll need to somehow
encode both proxy- and logical-endpoint addresses into those two params
or find some other means of conveying the same info. The fourth patch
currently uses `url-using-proxy' as a dynamic variable for this purpose,
but perhaps that's unwise. It's quite possible I'm breaking something.

There's also the issue of DNS lookups for verifying domain certs. Last I
looked, we can't run `nsm' checks without involving the system resolver,
which may be a deal breaker for the more privacy minded. If that's true,
we may want to find an acceptable way of cluing folks in to the
situation.

As thing stand, I haven't really invested enough in understanding how
`url' works to take the lead in planning a comprehensive integration
strategy (sorry). But, perhaps others with open bugs in the same area
[2] might be interested in lending some expertise or insight. Thanks.


[1] The third patch also attempts to untangle some of the hairiness
    brought about by the `socks-override-functions' flag. The original
    picture looks like:

    s-o-f:    `socks-override-functions', flag var
    o-n-s:    `open-network-stream', standard (non-SOCKS) opener
    o-n-s*:   `open-network-stream' with s--o-n-s as :around advice
    s-o-c:    `socks-open-connection', proxy (SOCKS) opener
    s-o-n-s:  `socks-open-network-stream', semi o-n-s compatible thin
               wrapper around s--o-n-s
    s--o-n-s: `socks--open-network-stream': workhorse with SOCKS/non
               code paths

    | s-o-f | entry point         | no-route | route          |
    |-------+---------------------+----------+----------------|
    | t     | o-n-s* -> s--o-n-s  | o-n-s    | s-o-c -> o-n-s |
    | t/nil | s-o-n-s -> s--o-n-s | o-n-s    | s-o-c -> o-n-s |

    I've changed the above to:

    s--o-n-s: thin wrapper around `s-o-n-s'
    s-o-n-s:  o-n-s compatible workhorse with SOCKS/non code paths

    | s-o-f | entry point                   | no-route | route          |
    |-------+-------------------------------+----------+----------------|
    | t     | o-n-s* -> s--o-n-s -> s-o-n-s | o-n-s    | s-o-c -> o-n-s |
    | t/nil | s-o-n-s                       | o-n-s    | s-o-c -> o-n-s |


[2]  Bug#62598: "29.0.60; url-https-proxy-connect doesn't support
     multi-stage auth to proxies"

     https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62598


--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment; filename=0000-v2-v3.diff

From 8a54568d3e7d70f23cfeda292aff21b8c2203f49 Mon Sep 17 00:00:00 2001
From: "F. Jason Park" <jp@HIDDEN>
Date: Thu, 7 Sep 2023 06:08:38 -0700
Subject: [PATCH 0/4] *** NOT A PATCH ***

*** BLURB HERE ***

F. Jason Park (4):
  Don't hard code server ports in SOCKS tests
  Improve SOCKS error handling and add support for 4a
  [POC] Simplify network-stream openers in socks.el
  [POC] Integrate the socks and url libraries

 lisp/net/socks.el            | 139 +++++++++++++++++++++++++++++------
 lisp/url/url-gw.el           |   8 +-
 lisp/url/url-http.el         |  16 ++--
 lisp/url/url-proxy.el        |  18 ++++-
 lisp/url/url-vars.el         |  11 ++-
 test/lisp/net/socks-tests.el |  67 +++++++++++++----
 6 files changed, 206 insertions(+), 53 deletions(-)

Interdiff:
diff --git a/lisp/net/socks.el b/lisp/net/socks.el
index f8881ba3990..1bb78113d52 100644
--- a/lisp/net/socks.el
+++ b/lisp/net/socks.el
@@ -335,17 +335,14 @@ socks-override-functions
 (when socks-override-functions
   (advice-add 'open-network-stream :around #'socks--open-network-stream))
 
-(defcustom socks-connect-function 'open-network-stream
+(defvar socks-connect-function #'open-network-stream
   "Function to open a network connection to a SOCKS provider.
-Called with arguments suitable for `open-network-stream'."
-  :version "30.1"
-  :type '(choice (function-item :value open-network-stream)
-                 (function :tag "User-provided function")))
+Called with arguments suitable for `open-network-stream'.")
 
 (defun socks-open-connection (server-info &rest stream-params)
   "Create and initialize a SOCKS process.
-Perform authentication if needed.  Expect SERVER-INFO to take the
-form of `socks-server' and STREAM-PARAMS to be keyword params
+Perform authentication if needed.  Expect SERVER-INFO to resemble
+`socks-server' and STREAM-PARAMS to be keyword parameters
 accepted by `open-network-stream'."
   (save-excursion
     (let ((proc
@@ -473,7 +470,9 @@ socks-send-command
              (let ((err (process-get proc 'socks-reply)))
                (if (eql version 5)
                    (nth (or err 1) socks-errors)
-                 (nth (- (if (and err (<= 90 err 93)) err 91) 90)
+                 ;; The defined error codes for v4 range from
+                 ;; 90-93, but we store them in a simple list.
+                 (nth (pcase err (90 0) (92 2) (93 3) (_ 1))
                       socks--errors-4)))))
     proc))
 
@@ -536,8 +535,12 @@ socks-find-services-entry
 	      (if udp socks-udp-services socks-tcp-services)))
 
 (defun socks--open-network-stream (orig-fun name buffer host service &rest params)
-  (let ((socks-override-functions orig-fun))
-    (apply #'socks-open-network-stream name buffer host service params)))
+  "Call `socks-open-network-stream', falling back to ORIG-FUN.
+Expect NAME, BUFFER, HOST, SERVICE, and PARAMS to be compatible
+with `open-network-stream'."
+  (let ((socks-connect-function orig-fun))
+    (apply (if socks-override-functions #'socks-open-network-stream orig-fun)
+           name buffer host service params)))
 
 (defcustom socks-proxied-tls-services '(443 6697)
   "Ports whose connections should use TLS.
@@ -563,11 +566,9 @@ socks-open-network-stream
 though `socks-connect-function' has the final say).  For TLS with
 proxied connections, see the option `socks-proxied-tls-services'.
 
-Before connecting, check the host against `socks-noproxy' and, on
-rejection, fall back to non-SOCKS.  Similarly, when
-`socks-override-functions' is a function, call it directly and
-trust that it's not interested in options defined in this
-library, such as `socks-server'.
+Before connecting, check the HOST against `socks-noproxy'.  On
+rejection, fall back to a non-SOCKS connection determined by
+the variable `socks-connect-function'.
 
 But, before doing anything, check if `url-using-proxy' is bound
 to a `url' struct object, as defined in `url-parse'.  If so,
@@ -589,9 +590,8 @@ socks-open-network-stream
                              socks-username))
          (socks-password (or (and url (url-password url))
                              socks-password)))
-    (if-let* (((booleanp socks-override-functions))
-              (route (socks-find-route host service))
-              (proc (apply #'socks-open-connection route params)))
+    (if-let ((route (socks-find-route host service))
+             (proc (apply #'socks-open-connection route params)))
         (let ((port (if (numberp service)
                         service
                       (process-contact proc :service)))
@@ -604,13 +604,10 @@ socks-open-network-stream
               (progn (gnutls-negotiate :process proc
                                        :hostname host
                                        :keylist (and certs (list certs)))
+                     ;; FIXME skip when TLD is .onion.
                      (nsm-verify-connection proc host port))
             proc))
-      (let ((fn (if (functionp socks-override-functions)
-                    socks-override-functions ; `socks-noproxy' not consulted
-                  #'open-network-stream)) ; `socks-noproxy' is non-nil
-            socks-override-functions)
-        (apply fn name buffer host service params)))))
+      (apply socks-connect-function name buffer host service params))))
 
 (defun socks--initiate-command-connect (proc buffer host service)
   (progn ; preserve indentation level for git blame / code review
diff --git a/lisp/url/url-vars.el b/lisp/url/url-vars.el
index 73585275a71..d96890db04a 100644
--- a/lisp/url/url-vars.el
+++ b/lisp/url/url-vars.el
@@ -191,12 +191,12 @@ url-mail-command
 
 (defcustom url-proxy-services nil
   "An alist of schemes and proxy servers that gateway them.
-Looks like ((\"http\" . \"hostname:portnumber\") ...).  This is
-set up from the ACCESS_proxy environment variables.  Depending on
-the gateway type, values may instead be expected to look like
-\"proxyscheme://hostname:portnumber\" where \"proxyscheme\" is
-something like \"socks5\".  As of Emacs 30.1, this only applies
-to SOCKS servers."
+Looks like ((\"http\" . \"hostname:portnumber\") ...).  This is set up
+from the ACCESS_proxy environment variables.  Depending on the
+gateway type, Emacs may expect certain server values to specfiy a
+\"scheme\", for example, \"proxyscheme://hostname:portnumber\",
+in which \"proxyscheme\" is something like \"socks5\".  As of
+Emacs 30.1, this only applies to SOCKS servers."
   :type '(repeat (cons :format "%v"
 		       (string :tag "Protocol")
 		       (string :tag "Proxy")))
diff --git a/test/lisp/net/socks-tests.el b/test/lisp/net/socks-tests.el
index 7c7f68eafa2..df69fb2f5cf 100644
--- a/test/lisp/net/socks-tests.el
+++ b/test/lisp/net/socks-tests.el
@@ -165,9 +165,9 @@ socks-tests--hello-world-http-request-pattern
                          "Content-Length: 13\r\n\r\n"
                          "Hello World!\n")))
 
-(defun socks-tests-perform-hello-world-http-request ()
+(defun socks-tests-perform-hello-world-http-request (&optional method)
   "Start canned server, validate hello-world response, and finalize."
-  (let* ((url-gateway-method 'socks)
+  (let* ((url-gateway-method (or method 'socks))
          (url (url-generic-parse-url "http://example.com"))
          (server (socks-tests-canned-server-create))
          ;;
@@ -281,7 +281,7 @@ socks-tests-v5-auth-user-pass-blank
 ;; against curl 7.71 with the following options:
 ;; $ curl --verbose --proxy socks5h://127.0.0.1:10082 example.com
 
-(ert-deftest socks-tests-v5-auth-none ()
+(defun socks-tests-v5-auth-none (method)
   "Verify correct handling of SOCKS5 when auth method 0 requested."
   (let ((socks-server '("server" "127.0.0.1" t 5))
         (socks-authentication-methods (append socks-authentication-methods
@@ -295,7 +295,27 @@ socks-tests-v5-auth-none
     (socks-unregister-authentication-method 2)
     (should-not (assq 2 socks-authentication-methods))
     (ert-info ("Make HTTP request over SOCKS5 with no auth method")
-      (socks-tests-perform-hello-world-http-request)))
+      (socks-tests-perform-hello-world-http-request method)))
   (should (assq 2 socks-authentication-methods)))
 
+(ert-deftest socks-tests-v5-auth-none ()
+  (socks-tests-v5-auth-none 'socks))
+
+;; This simulates the top-level advice around `open-network-stream'
+;; that's applied when loading the library with a non-nil
+;; `socks-override-functions'.
+(ert-deftest socks-override-functions ()
+  (should-not socks-override-functions)
+  (should-not (advice-member-p #'socks--open-network-stream
+                               'open-network-stream))
+  (advice-add 'open-network-stream :around #'socks--open-network-stream)
+  (should (advice-member-p #'socks--open-network-stream 'open-network-stream))
+
+  (unwind-protect (let ((socks-override-functions t))
+                    (socks-tests-v5-auth-none 'native))
+    (advice-remove 'open-network-stream #'socks--open-network-stream))
+
+  (should-not (advice-member-p #'socks--open-network-stream
+                               'open-network-stream)))
+
 ;;; socks-tests.el ends here
-- 
2.41.0


--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
 filename=0001-Don-t-hard-code-server-ports-in-SOCKS-tests.patch

From f76735a6beceb12f2e88befafe67aaef593e0763 Mon Sep 17 00:00:00 2001
From: "F. Jason Park" <jp@HIDDEN>
Date: Mon, 14 Feb 2022 02:36:57 -0800
Subject: [PATCH 1/4] Don't hard code server ports in SOCKS tests

* test/lisp/net/socks-tests.el (socks-tests-canned-server-create,
socks-tests-filter-response-parsing-v4, socks-tests-v4-basic,
socks-tests-v5-auth-user-pass, socks-tests-v5-auth-user-blank,
socks-tests-v5-auth-none): Fix bug in process filter to prevent
prepared outgoing responses from being implicitly encoded as UTF-8.
Fix similar mistake in v4 filter test.  Allow system to choose port
instead of hard-coding it.
(socks-tests-perform-hello-world-http-request):
Add option method parameter to specify a gateway method.
(socks-tests-v5-auth-none): Move body to helper function of the same
name.
(socks-override-functions): New test ensuring top-level advice around
`open-networks-stream' still supported.  (Bug#53941)
---
 test/lisp/net/socks-tests.el | 54 ++++++++++++++++++++++++++----------
 1 file changed, 39 insertions(+), 15 deletions(-)

diff --git a/test/lisp/net/socks-tests.el b/test/lisp/net/socks-tests.el
index 958e2ff44a8..d4c3828df45 100644
--- a/test/lisp/net/socks-tests.el
+++ b/test/lisp/net/socks-tests.el
@@ -63,21 +63,21 @@ socks-tests-filter-response-parsing-v4
     (process-put proc 'socks-state socks-state-waiting)
     (process-put proc 'socks-server-protocol 4)
     (ert-info ("Receive initial incomplete segment")
-      (socks-filter proc (concat [0 90 0 0 93 184 216]))
-      ;; From example.com: OK status ^      ^ msg start
+      (socks-filter proc (unibyte-string 0 90 0 0 93 184 216))
+      ;; From example.com: OK status       ^      ^ msg start
       (ert-info ("State still set to waiting")
         (should (eq (process-get proc 'socks-state) socks-state-waiting)))
       (ert-info ("Response field is nil because processing incomplete")
         (should-not (process-get proc 'socks-response)))
       (ert-info ("Scratch field holds stashed partial payload")
-        (should (string= (concat [0 90 0 0 93 184 216])
+        (should (string= (unibyte-string 0 90 0 0 93 184 216)
                          (process-get proc 'socks-scratch)))))
     (ert-info ("Last part arrives")
       (socks-filter proc "\42") ; ?\" 34
       (ert-info ("State transitions to complete (length check passes)")
         (should (eq (process-get proc 'socks-state) socks-state-connected)))
       (ert-info ("Scratch and response fields hold stash w. last chunk")
-        (should (string= (concat [0 90 0 0 93 184 216 34])
+        (should (string= (unibyte-string 0 90 0 0 93 184 216 34)
                          (process-get proc 'socks-response)))
         (should (string= (process-get proc 'socks-response)
                          (process-get proc 'socks-scratch)))))
@@ -133,17 +133,19 @@ socks-tests-canned-server-patterns
 (defun socks-tests-canned-server-create ()
   "Create and return a fake SOCKS server."
   (let* ((port (nth 2 socks-server))
-         (name (format "socks-server:%d" port))
+         (name (format "socks-server:%s"
+                       (if (numberp port) port (ert-test-name (ert-running-test)))))
          (pats socks-tests-canned-server-patterns)
          (filt (lambda (proc line)
                  (pcase-let ((`(,pat . ,resp) (pop pats)))
                    (unless (or (and (vectorp pat) (equal pat (vconcat line)))
                                (string-match-p pat line))
                      (error "Unknown request: %s" line))
+                   (setq resp (apply #'unibyte-string (append resp nil)))
                    (let ((print-escape-control-characters t))
                      (message "[%s] <- %s" name (prin1-to-string line))
                      (message "[%s] -> %s" name (prin1-to-string resp)))
-                   (process-send-string proc (concat resp)))))
+                   (process-send-string proc resp))))
          (serv (make-network-process :server 1
                                      :buffer (get-buffer-create name)
                                      :filter filt
@@ -151,8 +153,10 @@ socks-tests-canned-server-create
                                      :family 'ipv4
                                      :host 'local
                                      :coding 'binary
-                                     :service port)))
+                                     :service (or port t))))
     (set-process-query-on-exit-flag serv nil)
+    (unless (numberp (nth 2 socks-server))
+      (setf (nth 2 socks-server) (process-contact serv :service)))
     serv))
 
 (defvar socks-tests--hello-world-http-request-pattern
@@ -161,9 +165,9 @@ socks-tests--hello-world-http-request-pattern
                          "Content-Length: 13\r\n\r\n"
                          "Hello World!\n")))
 
-(defun socks-tests-perform-hello-world-http-request ()
+(defun socks-tests-perform-hello-world-http-request (&optional method)
   "Start canned server, validate hello-world response, and finalize."
-  (let* ((url-gateway-method 'socks)
+  (let* ((url-gateway-method (or method 'socks))
          (url (url-generic-parse-url "http://example.com"))
          (server (socks-tests-canned-server-create))
          ;;
@@ -191,7 +195,7 @@ socks-tests-perform-hello-world-http-request
 
 (ert-deftest socks-tests-v4-basic ()
   "Show correct preparation of SOCKS4 connect command (Bug#46342)."
-  (let ((socks-server '("server" "127.0.0.1" 10079 4))
+  (let ((socks-server '("server" "127.0.0.1" t 4))
         (url-user-agent "Test/4-basic")
         (socks-tests-canned-server-patterns
          `(([4 1 0 80 93 184 216 34 ?f ?o ?o 0] . [0 90 0 0 0 0 0 0])
@@ -213,7 +217,7 @@ socks-tests-v4-basic
 (ert-deftest socks-tests-v5-auth-user-pass ()
   "Verify correct handling of SOCKS5 user/pass authentication."
   (should (assq 2 socks-authentication-methods))
-  (let ((socks-server '("server" "127.0.0.1" 10080 5))
+  (let ((socks-server '("server" "127.0.0.1" t 5))
         (socks-username "foo")
         (socks-password "bar")
         (url-user-agent "Test/auth-user-pass")
@@ -247,7 +251,7 @@ socks-tests-v5-auth-user-pass
 (ert-deftest socks-tests-v5-auth-user-pass-blank ()
   "Verify correct SOCKS5 user/pass authentication with empty pass."
   (should (assq 2 socks-authentication-methods))
-  (let ((socks-server '("server" "127.0.0.1" 10081 5))
+  (let ((socks-server '("server" "127.0.0.1" t 5))
         (socks-username "foo") ; defaults to (user-login-name)
         (socks-password "") ; simulate user hitting enter when prompted
         (url-user-agent "Test/auth-user-pass-blank")
@@ -264,9 +268,9 @@ socks-tests-v5-auth-user-pass-blank
 ;; against curl 7.71 with the following options:
 ;; $ curl --verbose --proxy socks5h://127.0.0.1:10082 example.com
 
-(ert-deftest socks-tests-v5-auth-none ()
+(defun socks-tests-v5-auth-none (method)
   "Verify correct handling of SOCKS5 when auth method 0 requested."
-  (let ((socks-server '("server" "127.0.0.1" 10082 5))
+  (let ((socks-server '("server" "127.0.0.1" t 5))
         (socks-authentication-methods (append socks-authentication-methods
                                               nil))
         (url-user-agent "Test/auth-none")
@@ -278,7 +282,27 @@ socks-tests-v5-auth-none
     (socks-unregister-authentication-method 2)
     (should-not (assq 2 socks-authentication-methods))
     (ert-info ("Make HTTP request over SOCKS5 with no auth method")
-      (socks-tests-perform-hello-world-http-request)))
+      (socks-tests-perform-hello-world-http-request method)))
   (should (assq 2 socks-authentication-methods)))
 
+(ert-deftest socks-tests-v5-auth-none ()
+  (socks-tests-v5-auth-none 'socks))
+
+;; This simulates the top-level advice around `open-network-stream'
+;; that's applied when loading the library with a non-nil
+;; `socks-override-functions'.
+(ert-deftest socks-override-functions ()
+  (should-not socks-override-functions)
+  (should-not (advice-member-p #'socks--open-network-stream
+                               'open-network-stream))
+  (advice-add 'open-network-stream :around #'socks--open-network-stream)
+  (should (advice-member-p #'socks--open-network-stream 'open-network-stream))
+
+  (unwind-protect (let ((socks-override-functions t))
+                    (socks-tests-v5-auth-none 'native))
+    (advice-remove 'open-network-stream #'socks--open-network-stream))
+
+  (should-not (advice-member-p #'socks--open-network-stream
+                               'open-network-stream)))
+
 ;;; socks-tests.el ends here
-- 
2.41.0


--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
 filename=0002-Improve-SOCKS-error-handling-and-add-support-for-4a.patch

From 60913c0951ebfe851c23c2d59479fcbc6ed0b80c Mon Sep 17 00:00:00 2001
From: "F. Jason Park" <jp@HIDDEN>
Date: Mon, 14 Feb 2022 02:36:57 -0800
Subject: [PATCH 2/4] Improve SOCKS error handling and add support for 4a

* lisp/net/socks.el (socks-server): Add new Custom choice `4a' for
version field.  This change does not overload the field in terms of
expected type because `socks-send-command' and `socks-filter' already
accommodate the symbol `http'.
(socks--errors-4): Add new constant containing error messages for
version 4.  The semantics are faithful to the spec, but the exact
wording is adapted.
(socks-filter): Allow for a null "type" field on error with version 5.
Previously, certain errors would not propagate because a wrong-type
signal would get in the way.
(socks-send-command): Massage existing version 4 protocol parsing to
accommodate 4a, and add error handling for version 4.
* test/lisp/net/socks-tests.el (socks-tests-v4a-basic): Add test for
SOCKS version 4a.  (Bug#53941)
---
 lisp/net/socks.el            | 28 +++++++++++++++++++++++++---
 test/lisp/net/socks-tests.el | 13 +++++++++++++
 2 files changed, 38 insertions(+), 3 deletions(-)

diff --git a/lisp/net/socks.el b/lisp/net/socks.el
index 968a28d2be8..958d8bf23c9 100644
--- a/lisp/net/socks.el
+++ b/lisp/net/socks.el
@@ -162,6 +162,7 @@ socks-server
 	  (radio-button-choice :tag "SOCKS Version"
 			       :format "%t: %v"
 			       (const :tag "SOCKS v4  " :format "%t" :value 4)
+                               (const :tag "SOCKS v4a"  :format "%t" :value 4a)
 			       (const :tag "SOCKS v5"   :format "%t" :value 5))))
 
 
@@ -202,6 +203,12 @@ socks-errors
     "Command not supported"
     "Address type not supported"))
 
+(defconst socks--errors-4
+  '("Granted"
+    "Rejected or failed"
+    "Cannot connect to identd on the client"
+    "Client and identd report differing user IDs"))
+
 ;; The socks v5 address types
 (defconst socks-address-type-v4   1)
 (defconst socks-address-type-name 3)
@@ -309,7 +316,8 @@ socks-filter
 		     ((pred (= socks-address-type-name))
 		      (if (< (length string) 5)
 			  255
-		        (+ 1 (aref string 4)))))))
+                        (+ 1 (aref string 4))))
+                     (0 0))))
 	  (if (< (length string) desired-len)
 	      nil			; Need to spin some more
 	    (process-put proc 'socks-state socks-state-connected)
@@ -399,6 +407,7 @@ socks-send-command
 		(format "%c%s" (length address) address))
 	       (t
 		(error "Unknown address type: %d" atype))))
+        trailing
 	request version)
     (or (process-get proc 'socks)
         (error "socks-send-command called on non-SOCKS connection %S" proc))
@@ -415,6 +424,12 @@ socks-send-command
 			     (t
 			      (error "Unsupported address type for HTTP: %d" atype)))
 			    port)))
+     ((when (eq version '4a)
+        (setf addr "\0\0\0\1"
+              trailing (concat address "\0")
+              version 4 ; done with the "a" part
+              (process-get proc 'socks-server-protocol) 4)
+        nil)) ; fall through
      ((equal version 4)
       (setq request (concat
 		     (unibyte-string
@@ -424,7 +439,8 @@ socks-send-command
 		      (logand port #xff)) ; port, low byte
 		     addr                 ; address
 		     (user-full-name)     ; username
-		     "\0")))              ; terminate username
+                     "\0"                 ; terminate username
+                     trailing)))          ; optional host to look up
      ((equal version 5)
       (setq request (concat
 		     (unibyte-string
@@ -445,7 +461,13 @@ socks-send-command
 	nil				; Sweet sweet success!
       (delete-process proc)
       (error "SOCKS: %s"
-             (nth (or (process-get proc 'socks-reply) 1) socks-errors)))
+             (let ((err (process-get proc 'socks-reply)))
+               (if (eql version 5)
+                   (nth (or err 1) socks-errors)
+                 ;; The defined error codes for v4 range from
+                 ;; 90-93, but we store them in a simple list.
+                 (nth (pcase err (90 0) (92 2) (93 3) (_ 1))
+                      socks--errors-4)))))
     proc))
 
 
diff --git a/test/lisp/net/socks-tests.el b/test/lisp/net/socks-tests.el
index d4c3828df45..df69fb2f5cf 100644
--- a/test/lisp/net/socks-tests.el
+++ b/test/lisp/net/socks-tests.el
@@ -210,6 +210,19 @@ socks-tests-v4-basic
                  (lambda (&optional _) "foo")))
         (socks-tests-perform-hello-world-http-request)))))
 
+(ert-deftest socks-tests-v4a-basic ()
+  "Show correct preparation of SOCKS4a connect command."
+  (let ((socks-server '("server" "127.0.0.1" t 4a))
+        (url-user-agent "Test/4a-basic")
+        (socks-tests-canned-server-patterns
+         `(([4 1 0 80 0 0 0 1 ?f ?o ?o 0 ?e ?x ?a ?m ?p ?l ?e ?. ?c ?o ?m 0]
+            . [0 90 0 0 0 0 0 0])
+           ,socks-tests--hello-world-http-request-pattern)))
+    (ert-info ("Make HTTP request over SOCKS4A")
+      (cl-letf (((symbol-function 'user-full-name)
+                 (lambda (&optional _) "foo")))
+        (socks-tests-perform-hello-world-http-request)))))
+
 ;; Replace first pattern below with ([5 3 0 1 2] . [5 2]) to validate
 ;; against curl 7.71 with the following options:
 ;; $ curl --verbose -U foo:bar --proxy socks5h://127.0.0.1:10080 example.com
-- 
2.41.0


--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
 filename=0003-POC-Simplify-network-stream-openers-in-socks.el.patch

From a51bee63fb2b74d8eacdd0c3244df1f1a707f61b Mon Sep 17 00:00:00 2001
From: "F. Jason Park" <jp@HIDDEN>
Date: Mon, 28 Nov 2022 22:31:50 -0800
Subject: [PATCH 3/4] [POC] Simplify network-stream openers in socks.el

* lisp/net/socks.el (socks-connect-function): New variable for
specifying an `open-network-stream'-like connect function.
(socks-open-connection): Accept additional `open-network-stream'
params passed on to opener, now `socks-connect-function',
in place of `open-network-stream'.
(socks-proxied-tls-services): Add new option for specifying ports
whose proxied connections should use TLS.
(socks--open-network-stream): Rework to serve as thin wrapper for
`socks-open-network-stream' that now hinges on rather than ignores the
variable `socks-override-functions'.
(socks-open-network-stream): Prefer parsed URL details, when present
in a non-nil `url-using-proxy', for improved compatibility with the gw
framework.
(socks--initiate-command-connect): New function to house renamed
latter half of the original `socks--open-network-stream'.  Role now
reduced to issuing the first command using an existing
process.  (Bug#53941)
---
 lisp/net/socks.el | 111 +++++++++++++++++++++++++++++++++++++---------
 1 file changed, 90 insertions(+), 21 deletions(-)

diff --git a/lisp/net/socks.el b/lisp/net/socks.el
index 958d8bf23c9..1bb78113d52 100644
--- a/lisp/net/socks.el
+++ b/lisp/net/socks.el
@@ -34,7 +34,7 @@
 
 ;;; Code:
 
-(eval-when-compile (require 'cl-lib))
+(eval-when-compile (require 'cl-lib) (require 'url-parse))
 
 ;; ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
 ;; ;;; Custom widgets
@@ -335,14 +335,20 @@ socks-override-functions
 (when socks-override-functions
   (advice-add 'open-network-stream :around #'socks--open-network-stream))
 
-(defun socks-open-connection (server-info)
+(defvar socks-connect-function #'open-network-stream
+  "Function to open a network connection to a SOCKS provider.
+Called with arguments suitable for `open-network-stream'.")
+
+(defun socks-open-connection (server-info &rest stream-params)
+  "Create and initialize a SOCKS process.
+Perform authentication if needed.  Expect SERVER-INFO to resemble
+`socks-server' and STREAM-PARAMS to be keyword parameters
+accepted by `open-network-stream'."
   (save-excursion
     (let ((proc
            (let ((socks-override-functions nil))
-             (open-network-stream "socks"
-				  nil
-				  (nth 1 server-info)
-				  (nth 2 server-info))))
+             (apply socks-connect-function (nth 0 server-info) nil
+                    (nth 1 server-info) (nth 2 server-info) stream-params)))
 	  (authtype nil)
 	  version)
 
@@ -528,22 +534,85 @@ socks-find-services-entry
   (gethash (downcase service)
 	      (if udp socks-udp-services socks-tcp-services)))
 
-(defun socks-open-network-stream (name buffer host service)
-  (let ((socks-override-functions t))
-    (socks--open-network-stream
-     (lambda (&rest args)
-       (let ((socks-override-functions nil))
-         (apply #'open-network-stream args)))
-     name buffer host service)))
-
 (defun socks--open-network-stream (orig-fun name buffer host service &rest params)
-  (let ((route (and socks-override-functions
-                    (socks-find-route host service))))
-    (if (not route)
-	(apply orig-fun name buffer host service params)
-      ;; FIXME: Obey `params'!
-      (let* ((proc (socks-open-connection route))
-	     (version (process-get proc 'socks-server-protocol))
+  "Call `socks-open-network-stream', falling back to ORIG-FUN.
+Expect NAME, BUFFER, HOST, SERVICE, and PARAMS to be compatible
+with `open-network-stream'."
+  (let ((socks-connect-function orig-fun))
+    (apply (if socks-override-functions #'socks-open-network-stream orig-fun)
+           name buffer host service params)))
+
+(defcustom socks-proxied-tls-services '(443 6697)
+  "Ports whose connections should use TLS.
+Note that the system resolver may be consulted to look up host
+names for checking domain validation certs."
+  :version "30.1"
+  :type '(repeat number))
+
+(declare-function gnutls-negotiate "gnutls" (&rest rest))
+(declare-function nsm-verify-connection "nsm"
+                  (process host port &optional
+                           save-fingerprint warn-unencrypted))
+
+;;;###autoload
+(defun socks-open-network-stream (name buffer host service &rest params)
+  "Open and return a connection, possibly proxied over SOCKS.
+Expect PARAMS to contain keyword parameters recognized by
+`open-network-stream'.  Assume HOST and SERVICE refer to the
+proxied remote peer rather than the SOCKS server, but assume the
+opposite for PARAMS.  That is, if PARAMS contains a `:type' of
+`tls', treat the underlying connection to the proxy server as
+destined for encryption rather than the tunneled connection (even
+though `socks-connect-function' has the final say).  For TLS with
+proxied connections, see the option `socks-proxied-tls-services'.
+
+Before connecting, check the HOST against `socks-noproxy'.  On
+rejection, fall back to a non-SOCKS connection determined by
+the variable `socks-connect-function'.
+
+But, before doing anything, check if `url-using-proxy' is bound
+to a `url' struct object, as defined in `url-parse'.  If so,
+assume it represents the address of the desired SOCKS server
+rather than that of the remote peer, and use its fields instead
+of `socks-server' for all SOCKS connection details."
+  (require 'url-parse)
+  (let* ((url (and (url-p url-using-proxy)
+                   (string-prefix-p "socks" (url-type url-using-proxy))
+                   url-using-proxy))
+         (socks-server (if url
+                           (list name (url-host url) (url-port url)
+                                 (pcase (url-type url)
+                                   ("socks4://" 4)
+                                   ("socks4a://" '4a)
+                                   (_ 5)))
+                         socks-server))
+         (socks-username (or (and url (url-user url))
+                             socks-username))
+         (socks-password (or (and url (url-password url))
+                             socks-password)))
+    (if-let ((route (socks-find-route host service))
+             (proc (apply #'socks-open-connection route params)))
+        (let ((port (if (numberp service)
+                        service
+                      (process-contact proc :service)))
+              (certs (plist-get params :client-certificate)))
+          (socks--initiate-command-connect proc buffer host service)
+          (if (and (memq port socks-proxied-tls-services)
+                   (gnutls-available-p)
+                   (require 'gnutls nil t)
+                   (require 'nsm nil t))
+              (progn (gnutls-negotiate :process proc
+                                       :hostname host
+                                       :keylist (and certs (list certs)))
+                     ;; FIXME skip when TLD is .onion.
+                     (nsm-verify-connection proc host port))
+            proc))
+      (apply socks-connect-function name buffer host service params))))
+
+(defun socks--initiate-command-connect (proc buffer host service)
+  (progn ; preserve indentation level for git blame / code review
+    (progn
+      (let* ((version (process-get proc 'socks-server-protocol))
              (atype
               (cond
                ((equal version 4)
-- 
2.41.0


--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
 filename=0004-POC-Integrate-the-socks-and-url-libraries.patch

From 8a54568d3e7d70f23cfeda292aff21b8c2203f49 Mon Sep 17 00:00:00 2001
From: "F. Jason Park" <jp@HIDDEN>
Date: Tue, 1 Mar 2022 01:38:33 -0800
Subject: [PATCH 4/4] [POC] Integrate the socks and url libraries

* lisp/url/url-gw.el (url-open-stream): Use presence and type of
`url-using-proxy' to detect caller and massage input values according
to legacy practices.

* lisp/url/url-http.el: (url-http-find-free-connection): Don't call
`url-open-stream' with host and port from active proxy.
(url-http, url-http-async-sentinel): Only open
`url-https-proxy-connect' for non-SOCKS proxies.

* lisp/url/url-proxy.el (url-proxy--socks-scheme-regexp): Add new
const.
(url-default-find-proxy-for-url): Accommodate SOCKS entries but defy
original design somewhat by requiring a URL scheme in the host value
for detection.
(url-find-proxy-for-url): Recognize modified host/address value for
socks entries of `url-proxy-services' and deal accordingly.
(url-proxy): Handle a SOCKS proxy for http(s) connections only.

* lisp/url/url-vars.el (url-proxy-services): Explain that values for
certain gateways may need a leading scheme:// portion.
(url-using-proxy): Add warning regarding expected type.
---
 lisp/url/url-gw.el    |  8 +++++++-
 lisp/url/url-http.el  | 16 +++++++---------
 lisp/url/url-proxy.el | 18 ++++++++++++++++--
 lisp/url/url-vars.el  | 11 +++++++++--
 4 files changed, 39 insertions(+), 14 deletions(-)

diff --git a/lisp/url/url-gw.el b/lisp/url/url-gw.el
index 4d7297f6f2e..baf94b96819 100644
--- a/lisp/url/url-gw.el
+++ b/lisp/url/url-gw.el
@@ -28,7 +28,7 @@
 (require 'url-vars)
 (require 'url-parse)
 
-(autoload 'socks-open-network-stream "socks")
+(autoload 'socks-open-network-stream "socks") ; FIXME remove this
 
 (defgroup url-gateway nil
   "URL gateway variables."
@@ -220,6 +220,12 @@ url-open-stream
 Optional arg GATEWAY-METHOD specifies the gateway to be used,
 overriding the value of `url-gateway-method'."
   (unless url-gateway-unplugged
+    (when (url-p url-using-proxy)
+      (if (or (eq 'socks url-gateway-method)
+              (string-prefix-p "socks" (url-type url-using-proxy)))
+          (setq gateway-method 'socks)
+        (setq host (url-host url-using-proxy)
+              service (url-port url-using-proxy))))
     (let* ((gwm (or gateway-method url-gateway-method))
            (gw-method (if (and url-gateway-local-host-regexp
                                (not (eq 'tls gwm))
diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
index ada6341ee73..42cfb9959a7 100644
--- a/lisp/url/url-http.el
+++ b/lisp/url/url-http.el
@@ -195,12 +195,7 @@ url-http-find-free-connection
 	;; like authentication.  But we use another buffer afterwards.
 	(unwind-protect
             (let ((proc (url-open-stream host buf
-                                         (if url-using-proxy
-                                             (url-host url-using-proxy)
-                                           host)
-                                         (if url-using-proxy
-                                             (url-port url-using-proxy)
-                                           port)
+                                         host port
                                          gateway-method)))
 	      ;; url-open-stream might return nil.
 	      (when (processp proc)
@@ -1396,8 +1391,9 @@ url-http
            (error "Could not create connection to %s:%d" (url-host url)
                   (url-port url)))
           (_
-           (if (and url-http-proxy (string= "https"
-                                            (url-type url-current-object)))
+           (if (and url-http-proxy
+                    (not (string-prefix-p "socks" (url-type url-http-proxy)))
+                    (string= "https" (url-type url-current-object)))
                (url-https-proxy-connect connection)
              (set-process-sentinel connection
                                    #'url-http-end-of-document-sentinel)
@@ -1479,7 +1475,9 @@ url-http-async-sentinel
 	(url-http-end-of-document-sentinel proc why))
        ((string= (substring why 0 4) "open")
 	(setq url-http-connection-opened t)
-        (if (and url-http-proxy (string= "https" (url-type url-current-object)))
+        (if (and url-http-proxy
+                 (not (string-prefix-p "socks" (url-type url-http-proxy)))
+                 (string= "https" (url-type url-current-object)))
             (url-https-proxy-connect proc)
           (condition-case error
               (process-send-string proc (url-http-create-request))
diff --git a/lisp/url/url-proxy.el b/lisp/url/url-proxy.el
index 0c330069789..1a278bb1673 100644
--- a/lisp/url/url-proxy.el
+++ b/lisp/url/url-proxy.el
@@ -25,6 +25,9 @@
 
 (require 'url-parse)
 
+(defconst url-proxy--socks-scheme-regexp
+  (rx bot "socks" (? (or "4" "4a" "5" "5h")) "://"))
+
 (defun url-default-find-proxy-for-url (urlobj host)
   (cond
    ((or (and (assoc "no_proxy" url-proxy-services)
@@ -35,7 +38,12 @@ url-default-find-proxy-for-url
 	(equal "www" (url-type urlobj)))
     "DIRECT")
    ((cdr (assoc (url-type urlobj) url-proxy-services))
-    (concat "PROXY " (cdr (assoc (url-type urlobj) url-proxy-services))))
+    (let ((found (alist-get (url-type urlobj) url-proxy-services
+                            nil nil #'equal)))
+      (concat (if (string-match url-proxy--socks-scheme-regexp found)
+                  "SOCKS "
+                "PROXY ")
+              found)))
    ;;
    ;; Should check for socks
    ;;
@@ -57,7 +65,10 @@ url-find-proxy-for-url
      ((string-match "^PROXY +" proxy)
       (concat "http://" (substring proxy (match-end 0)) "/"))
      ((string-match "^SOCKS +" proxy)
-      (concat "socks://" (substring proxy (match-end 0))))
+      (if-let* ((m (substring proxy (match-end 0)))
+                ((string-match url-proxy--socks-scheme-regexp m)))
+          m
+        (concat "socks://" m)))
      (t
       (display-warning 'url (format "Unknown proxy directive: %s" proxy) :error)
       nil))))
@@ -72,6 +83,9 @@ url-proxy
   (cond
    ((string= (url-type url-using-proxy) "http")
     (url-http url callback cbargs))
+   ((and (string-prefix-p "socks" (url-type url-using-proxy))
+         (string-prefix-p "http" (url-type url)))
+    (url-http url callback cbargs))
    (t
     (error "Don't know how to use proxy `%s'" url-using-proxy))))
 
diff --git a/lisp/url/url-vars.el b/lisp/url/url-vars.el
index 7e2290217d0..d96890db04a 100644
--- a/lisp/url/url-vars.el
+++ b/lisp/url/url-vars.el
@@ -192,10 +192,15 @@ url-mail-command
 (defcustom url-proxy-services nil
   "An alist of schemes and proxy servers that gateway them.
 Looks like ((\"http\" . \"hostname:portnumber\") ...).  This is set up
-from the ACCESS_proxy environment variables."
+from the ACCESS_proxy environment variables.  Depending on the
+gateway type, Emacs may expect certain server values to specfiy a
+\"scheme\", for example, \"proxyscheme://hostname:portnumber\",
+in which \"proxyscheme\" is something like \"socks5\".  As of
+Emacs 30.1, this only applies to SOCKS servers."
   :type '(repeat (cons :format "%v"
 		       (string :tag "Protocol")
 		       (string :tag "Proxy")))
+  :version "30.1"
   :group 'url)
 
 (defcustom url-standalone-mode nil
@@ -310,7 +315,9 @@ url-show-status
 
 (defvar url-using-proxy nil
   "Either nil or the fully qualified proxy URL in use, e.g.
-https://www.example.com/")
+https://www.example.com/.  Beware that some functions, such as
+`url-proxy' and `url-http-end-of-document-sentinel', set this to
+a `url' struct.")
 
 (defcustom url-news-server nil
   "The default news server from which to get newsgroups/articles.
-- 
2.41.0


--=-=-=--

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#53941: 27.2; socks + tor dont work with https
Resent-From: Stefan Kangas <stefankangas@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Thu, 07 Sep 2023 13:48:01 +0000
Resent-Message-ID: <handler.53941.B53941.169409444428802 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 53941
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: patch
To: "J.P." <jp@HIDDEN>, Eli Zaretskii <eliz@HIDDEN>
Cc: larsi@HIDDEN, 53941 <at> debbugs.gnu.org, gnuhacker@HIDDEN
Received: via spool by 53941-submit <at> debbugs.gnu.org id=B53941.169409444428802
          (code B ref 53941); Thu, 07 Sep 2023 13:48:01 +0000
Received: (at 53941) by debbugs.gnu.org; 7 Sep 2023 13:47:24 +0000
Received: from localhost ([127.0.0.1]:38925 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qeFLr-0007UQ-7p
	for submit <at> debbugs.gnu.org; Thu, 07 Sep 2023 09:47:23 -0400
Received: from mail-lf1-x12a.google.com ([2a00:1450:4864:20::12a]:62573)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <stefankangas@HIDDEN>) id 1qeFLm-0007U8-6u
 for 53941 <at> debbugs.gnu.org; Thu, 07 Sep 2023 09:47:18 -0400
Received: by mail-lf1-x12a.google.com with SMTP id
 2adb3069b0e04-501b9f27eb2so2625595e87.0
 for <53941 <at> debbugs.gnu.org>; Thu, 07 Sep 2023 06:47:12 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=gmail.com; s=20221208; t=1694094426; x=1694699226; darn=debbugs.gnu.org;
 h=cc:to:subject:message-id:date:mime-version:references:in-reply-to
 :from:from:to:cc:subject:date:message-id:reply-to;
 bh=QYKtUNjhzCRdbWmL0C/T92jgSloQodUzIhK3c2cK5Yw=;
 b=OoOUjSrpSt/9zwvsFbaZEuHVWobaYRW46Xq03vhTwju51GsM3mXY0IBVo2vMG1hfiX
 LQKjvwhPM85EgbjB2+LaMM/bkGpFZTntd6k7Eqd/wWB1lTo4EXfqLua902LyZGWd9Rjb
 9HqPAlFNXl+Naqo4ZlhZXJFWAq3vSfUGYW5lGkyfCq4jEkVkBYVKrJ8rPUk5+khjCaY+
 nuWufc84CxqG1OizsKc1or5q5JGpDf8u71kvo3ikckbGKsGAzjdFwWuYry4saFJw5L/W
 8Lf2LDNSgNyLLxPz0FFtKhmODsMh2WV4gYJBlyt5jypYBapiDTj+kW0BgE3eSXgkVdnN
 bvZg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20221208; t=1694094426; x=1694699226;
 h=cc:to:subject:message-id:date:mime-version:references:in-reply-to
 :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to;
 bh=QYKtUNjhzCRdbWmL0C/T92jgSloQodUzIhK3c2cK5Yw=;
 b=VnJUE2fyWvcKYzzwIpjgJpKWvo4FwiEacG+rwrOCC1dGV9NK7ndc/muhwZdqRJrT1Y
 c2Zg7vejI2RldQLMSe0nTD67mSXIjt4FsrYDzaAcAZsPrb0mUXYczZe94LKalUTsZeFb
 vZY6esoXD3pXyGoB8gRU90bhIxOSuNZ+/qvcodItIz+Q7kyw3yqlie3h8f0/WMj8IaLV
 zxGJgWeaSiuOBfkD4i7eptis/nuOghDAaC/MeGxCNAj6/DY7gA5BDjMdrPMRKCQSYj4A
 pO6kUrUPgSbo3sTHZC1hJ374negMtU64V2ag3kwCClPBce15Iu+EwW4nao983Jrck2AC
 UDvQ==
X-Gm-Message-State: AOJu0Yx45y1cSaauas5DQg4y7tlyOF5qOpyfhGbMxgxEhZCdEYxHelGp
 NPrCxzFrLYvNPjlUVzCkSvDjBIOipVq/nYNGhIQ=
X-Google-Smtp-Source: AGHT+IFHvgFRcrEDh+2HPJuKO6ZBOgrCPOeERCT+vapgFfO43bJpJvg6kf1qal1x2eIVUwBrd36bonoL+x3hWRNrWKA=
X-Received: by 2002:a05:6512:12ca:b0:501:ca56:39d8 with SMTP id
 p10-20020a05651212ca00b00501ca5639d8mr1068226lfg.32.1694094426178; Thu, 07
 Sep 2023 06:47:06 -0700 (PDT)
Received: from 753933720722 named unknown by gmailapi.google.com with
 HTTPREST; Thu, 7 Sep 2023 06:47:05 -0700
From: Stefan Kangas <stefankangas@HIDDEN>
In-Reply-To: <878r9iktd0.fsf@HIDDEN>
References: <87pmntfym7.fsf@HIDDEN> <8735kl1v58.fsf@HIDDEN>
 <87a6emftzx.fsf@HIDDEN> <87k0do5km1.fsf@HIDDEN>
 <87pmn5n3tu.fsf@HIDDEN>
 <87mti99j1f.fsf@HIDDEN> <87wnh7hkgi.fsf@HIDDEN>
 <87pmmz947k.fsf@HIDDEN>
 <8735ju44sk.fsf@HIDDEN> <87lexikwu5.fsf@HIDDEN>
 <87mt8baygn.fsf_-_@HIDDEN>
 <8335a3nguk.fsf@HIDDEN> <87fse1kfe8.fsf@HIDDEN> <831qpln7zg.fsf@HIDDEN>
 <CADwFkmm0hCO3ek-w0TWkGWOROCU9YfOgDO0ivtcz-Q7feUE7nA@HIDDEN>
 <837cp21qca.fsf@HIDDEN> <878r9iktd0.fsf@HIDDEN>
MIME-Version: 1.0
Date: Thu, 7 Sep 2023 06:47:05 -0700
Message-ID: <CADwFkmnEG66bLtSrhWcL_w0PDng3EMPE_zyGftAQFVtFKNmg4A@HIDDEN>
Content-Type: text/plain; charset="UTF-8"
X-Spam-Score: 0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

"J.P." <jp@HIDDEN> writes:

> As of now, the only patches I'd be comfortable offering would be the
> first two, which aren't even directly related to this bug.

Thanks.  The first two patches do add tests as well, so I can see some
value in installing them separately, perhaps even right now.  Even more
so if it simplifies your work on the tasks you think are more important.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#53941: 27.2; socks + tor dont work with https
Resent-From: "J.P." <jp@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 08 Sep 2023 02:56:02 +0000
Resent-Message-ID: <handler.53941.B53941.169414173621249 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 53941
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: patch
To: Stefan Kangas <stefankangas@HIDDEN>
Cc: Eli Zaretskii <eliz@HIDDEN>, 53941 <at> debbugs.gnu.org, larsi@HIDDEN, gnuhacker@HIDDEN
Received: via spool by 53941-submit <at> debbugs.gnu.org id=B53941.169414173621249
          (code B ref 53941); Fri, 08 Sep 2023 02:56:02 +0000
Received: (at 53941) by debbugs.gnu.org; 8 Sep 2023 02:55:36 +0000
Received: from localhost ([127.0.0.1]:41816 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qeReg-0005We-3q
	for submit <at> debbugs.gnu.org; Thu, 07 Sep 2023 22:55:35 -0400
Received: from mail-108-mta165.mxroute.com ([136.175.108.165]:39033)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <jp@HIDDEN>) id 1qeRea-0005WQ-MA
 for 53941 <at> debbugs.gnu.org; Thu, 07 Sep 2023 22:55:33 -0400
Received: from mail-111-mta2.mxroute.com ([136.175.111.2]
 filter006.mxroute.com) (Authenticated sender: mN4UYu2MZsgR)
 by mail-108-mta165.mxroute.com (ZoneMTA) with ESMTPSA id
 18a72b73b3a000d7b6.001 for <53941 <at> debbugs.gnu.org>
 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384);
 Fri, 08 Sep 2023 02:55:25 +0000
X-Zone-Loop: 88678204e3ad8793a1c225010059011239c328f3a126
X-Originating-IP: [136.175.111.2]
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=neverwas.me
 ; s=x;
 h=Content-Type:MIME-Version:Message-ID:Date:References:In-Reply-To:
 Subject:Cc:To:From:Sender:Reply-To:Content-Transfer-Encoding:Content-ID:
 Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc
 :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe:
 List-Post:List-Owner:List-Archive;
 bh=GNZQepQO9+dCDTN3+FHf/UFYUuQbBs8aTka9M3c5wZo=; b=H+Og/pgB23TrnhU6E3zGsATqBi
 3fwQVOFcJtcUMrvXWNi+ZyCIgpmO7P2KsSMqLDH7nIhR2XM8nOK3qszx5CAbzrC1dnDMazHhNz+OH
 wC3OSUMyQo1bj9mpOfNAn9XSE8E50zYc/iXYytL7IKcylmgaUkwzXFEWYXoVutDpxWJadtNDigmJ2
 +5EZwSPBVXlGXakJhFLA1Oq7TqDLsGjx9tACYXhznlvQbYlFKcirXrDNOaN19IxcurFth0klGGtke
 LZFVG+oUwQpuPXwlnht3EckuzLx5jOGRuL3hKupWz6Ia3y6ToMkMhFg2hcH+s8facLpOcLnpDgDC3
 ZLhPymmg==;
From: "J.P." <jp@HIDDEN>
In-Reply-To: <CADwFkmnEG66bLtSrhWcL_w0PDng3EMPE_zyGftAQFVtFKNmg4A@HIDDEN>
 (Stefan Kangas's message of "Thu, 7 Sep 2023 06:47:05 -0700")
References: <87pmntfym7.fsf@HIDDEN> <8735kl1v58.fsf@HIDDEN>
 <87a6emftzx.fsf@HIDDEN> <87k0do5km1.fsf@HIDDEN>
 <87pmn5n3tu.fsf@HIDDEN> <87mti99j1f.fsf@HIDDEN>
 <87wnh7hkgi.fsf@HIDDEN> <87pmmz947k.fsf@HIDDEN>
 <8735ju44sk.fsf@HIDDEN> <87lexikwu5.fsf@HIDDEN>
 <87mt8baygn.fsf_-_@HIDDEN> <8335a3nguk.fsf@HIDDEN>
 <87fse1kfe8.fsf@HIDDEN> <831qpln7zg.fsf@HIDDEN>
 <CADwFkmm0hCO3ek-w0TWkGWOROCU9YfOgDO0ivtcz-Q7feUE7nA@HIDDEN>
 <837cp21qca.fsf@HIDDEN> <878r9iktd0.fsf@HIDDEN>
 <CADwFkmnEG66bLtSrhWcL_w0PDng3EMPE_zyGftAQFVtFKNmg4A@HIDDEN>
Date: Thu, 07 Sep 2023 19:55:16 -0700
Message-ID: <87il8lidbf.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
Content-Type: text/plain
X-Authenticated-Id: masked@HIDDEN
X-Spam-Score: 0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Stefan Kangas <stefankangas@HIDDEN> writes:

> "J.P." <jp@HIDDEN> writes:
>
>> As of now, the only patches I'd be comfortable offering would be the
>> first two, which aren't even directly related to this bug.
>
> Thanks.  The first two patches do add tests as well, so I can see some
> value in installing them separately, perhaps even right now.  Even more
> so if it simplifies your work on the tasks you think are more important.

OK, nice. I'd also like to add at least one test case that simulates a
realistic error condition (and maybe also a NEWS item for v4a, if that's
considered a feature). If no one else has thoughts regarding the first
two, I'll install them in a few days.

(Congrats on your appointment, BTW.)

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#53941: 27.2; socks + tor dont work with https
Resent-From: Stefan Kangas <stefankangas@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 08 Sep 2023 11:05:01 +0000
Resent-Message-ID: <handler.53941.B53941.169417106716725 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 53941
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: patch
To: "J.P." <jp@HIDDEN>
Cc: Eli Zaretskii <eliz@HIDDEN>, 53941 <at> debbugs.gnu.org, larsi@HIDDEN, gnuhacker@HIDDEN
Received: via spool by 53941-submit <at> debbugs.gnu.org id=B53941.169417106716725
          (code B ref 53941); Fri, 08 Sep 2023 11:05:01 +0000
Received: (at 53941) by debbugs.gnu.org; 8 Sep 2023 11:04:27 +0000
Received: from localhost ([127.0.0.1]:42316 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qeZHn-0004Lh-Ap
	for submit <at> debbugs.gnu.org; Fri, 08 Sep 2023 07:04:27 -0400
Received: from mail-lf1-x132.google.com ([2a00:1450:4864:20::132]:50185)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <stefankangas@HIDDEN>) id 1qeZHk-0004LS-V0
 for 53941 <at> debbugs.gnu.org; Fri, 08 Sep 2023 07:04:26 -0400
Received: by mail-lf1-x132.google.com with SMTP id
 2adb3069b0e04-5007abb15e9so3299144e87.0
 for <53941 <at> debbugs.gnu.org>; Fri, 08 Sep 2023 04:04:22 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=gmail.com; s=20221208; t=1694171057; x=1694775857; darn=debbugs.gnu.org;
 h=cc:to:subject:message-id:date:mime-version:references:in-reply-to
 :from:from:to:cc:subject:date:message-id:reply-to;
 bh=G9UTk2bGOW6Zmruf3CNEzhPEeDzRblnQIu+zSshwHKg=;
 b=gHU+nfzZH2t6oK4+IEu96SR1iziM7UaoWCFBLv4fcbCewOG2Pzov7VavqIPwsb90k9
 r3kRbOaQxE/kb5jjQCiBJk93YgWS2D4nU2rHhuWZCZNo5DPoGCiwwPlJZ+0NoVlQdY4p
 GD1AZ4M75RcR3Tavk6F5XxQ6oFN4+QJxTzwQopENCZfLxgLprmSkP+UgdrLv06mIwYEn
 Fgwo26cdoUKC0T6jXGtKVRT92y4igj3eC1mA8VVDiUgOLDZ6WnfeYN4u7Wzx6eR2EtmY
 d/Zzo26ue4MF9HL/7OMcGqID56JlwySynbCRghcbuFmvSPpoSkHFPKyGwP0CLSqTl89t
 oIvA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20230601; t=1694171057; x=1694775857;
 h=cc:to:subject:message-id:date:mime-version:references:in-reply-to
 :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to;
 bh=G9UTk2bGOW6Zmruf3CNEzhPEeDzRblnQIu+zSshwHKg=;
 b=Vr75Vy0Wd0fJVbaYvjczjVkaZm0Y5ItZFbD0s/WFlBBxiSDKDLs3t9+h/KHoAZRBd/
 +2dqLLi0aPWBUNof08ORoQqnlsHZJr5ORmWwW44TNJ1ofvGQdaASBXKibH5Qygy5HjdC
 uT+htoKWhcYRQeAVWYrmQo6nuc09LIL3WiZmTVB/uD1zyDVfcufvHVEk4mXnxF5ocyKQ
 H5tumwlq1HUCGyR5OiakJdCYppQfh+Yy3T3DiM/cvEcsxnWXM4+tYxB4dNIYzaGYran+
 0nTAd0gb5VZgMQ+27t3nLmTHuKAVNeqA42p3S8TmDaafTuzxTJae2InJVABNnc990weO
 JA9w==
X-Gm-Message-State: AOJu0YzzQZbtXqPS/6HyX+DxygaA/rc1NwV3mMS8jIxR5RfyKSzbxLXK
 Eboky5zQCF+ok0HegejE7MpmB9MuX6NoaA3J0vAc3yCw
X-Google-Smtp-Source: AGHT+IH4oBwWp11iqBbzJbMV5ETzC1wzU+dQpIwu4q8hsJqsrXXZ+TT3V3a+6+TnnMYskAJ4uAqMWTvjldceyex2M90=
X-Received: by 2002:a05:6512:118d:b0:500:771f:4887 with SMTP id
 g13-20020a056512118d00b00500771f4887mr1982685lfr.55.1694171056682; Fri, 08
 Sep 2023 04:04:16 -0700 (PDT)
Received: from 753933720722 named unknown by gmailapi.google.com with
 HTTPREST; Fri, 8 Sep 2023 04:04:16 -0700
From: Stefan Kangas <stefankangas@HIDDEN>
In-Reply-To: <87il8lidbf.fsf@HIDDEN>
References: <87pmntfym7.fsf@HIDDEN> <8735kl1v58.fsf@HIDDEN>
 <87a6emftzx.fsf@HIDDEN> <87k0do5km1.fsf@HIDDEN>
 <87pmn5n3tu.fsf@HIDDEN>
 <87mti99j1f.fsf@HIDDEN> <87wnh7hkgi.fsf@HIDDEN>
 <87pmmz947k.fsf@HIDDEN>
 <8735ju44sk.fsf@HIDDEN> <87lexikwu5.fsf@HIDDEN>
 <87mt8baygn.fsf_-_@HIDDEN>
 <8335a3nguk.fsf@HIDDEN> <87fse1kfe8.fsf@HIDDEN> <831qpln7zg.fsf@HIDDEN>
 <CADwFkmm0hCO3ek-w0TWkGWOROCU9YfOgDO0ivtcz-Q7feUE7nA@HIDDEN>
 <837cp21qca.fsf@HIDDEN> <878r9iktd0.fsf@HIDDEN>
 <CADwFkmnEG66bLtSrhWcL_w0PDng3EMPE_zyGftAQFVtFKNmg4A@HIDDEN>
 <87il8lidbf.fsf@HIDDEN>
MIME-Version: 1.0
Date: Fri, 8 Sep 2023 04:04:16 -0700
Message-ID: <CADwFkm==rOVxMqaSU0qq42_pjNZ8d=WK3aYnVyndGb-bzz-2Ug@HIDDEN>
Content-Type: text/plain; charset="UTF-8"
X-Spam-Score: 0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

"J.P." <jp@HIDDEN> writes:

> OK, nice. I'd also like to add at least one test case that simulates a
> realistic error condition (and maybe also a NEWS item for v4a, if that's
> considered a feature). If no one else has thoughts regarding the first
> two, I'll install them in a few days.

Sounds good to me.

> (Congrats on your appointment, BTW.)

Thanks!

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#53941: 27.2; socks + tor dont work with https
Resent-From: "J.P." <jp@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 08 Sep 2023 13:30:02 +0000
Resent-Message-ID: <handler.53941.B53941.169417975819164 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 53941
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: patch
To: 53941 <at> debbugs.gnu.org,
Cc: larsi@HIDDEN, gnuhacker@HIDDEN, Eli Zaretskii <eliz@HIDDEN>, Stefan Kangas <stefankangas@HIDDEN>
Received: via spool by 53941-submit <at> debbugs.gnu.org id=B53941.169417975819164
          (code B ref 53941); Fri, 08 Sep 2023 13:30:02 +0000
Received: (at 53941) by debbugs.gnu.org; 8 Sep 2023 13:29:18 +0000
Received: from localhost ([127.0.0.1]:42609 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qebXv-0004yy-MY
	for submit <at> debbugs.gnu.org; Fri, 08 Sep 2023 09:29:18 -0400
Received: from mail-108-mta71.mxroute.com ([136.175.108.71]:34965)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <jp@HIDDEN>) id 1qebXr-0004yo-BA
 for 53941 <at> debbugs.gnu.org; Fri, 08 Sep 2023 09:29:14 -0400
Received: from mail-111-mta2.mxroute.com ([136.175.111.2]
 filter006.mxroute.com) (Authenticated sender: mN4UYu2MZsgR)
 by mail-108-mta71.mxroute.com (ZoneMTA) with ESMTPSA id 18a74fb5775000d7b6.001
 for <53941 <at> debbugs.gnu.org>
 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384);
 Fri, 08 Sep 2023 13:29:03 +0000
X-Zone-Loop: 8d8bb6b8c18792fdea39256344d9c163063b6a1f32a4
X-Originating-IP: [136.175.111.2]
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=neverwas.me
 ; s=x;
 h=Content-Type:MIME-Version:Message-ID:Date:References:In-Reply-To:
 Subject:Cc:To:From:Sender:Reply-To:Content-Transfer-Encoding:Content-ID:
 Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc
 :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe:
 List-Post:List-Owner:List-Archive;
 bh=WUNqtEMooUPHqDyvfPOB6A0gGEMED8jJu1K35Ew9V4E=; b=NNsNgyNuBH+BlwVwU5bSZ6nHb/
 X2o0ZUaM2U6Ce+Lb7okYIPDThL2sabI42OaqCyCkhVy3iVNjMfnJrVaIDuzWixFE1utRSEQ/j82b+
 /tzJ4ShFTWH4Fn/bXU6l4xmy4p0FwHUQYyuDMl2IMm5ppwdTGGYRJ/TpA8QrkvnI6GK6k0MaAA5TR
 kuWy3TnjStm4WXJuqQzYOqRhiObPOtc9Yt5oZzj5EAVmBSzN7H/l+LPit/udNDatUMpO6WMvaKPtx
 12nyE3pF+2vOZunx3vhi+Si2j4fORktsjSQOk0LeP0+JIvKYegTXJT5Qi8UEr9DRTKghFee6K612D
 P6o3juVQ==;
From: "J.P." <jp@HIDDEN>
In-Reply-To: <878r9iktd0.fsf@HIDDEN> (J. P.'s message of "Thu, 07 Sep
 2023 06:25:47 -0700")
References: <87pmntfym7.fsf@HIDDEN> <8735kl1v58.fsf@HIDDEN>
 <87a6emftzx.fsf@HIDDEN> <87k0do5km1.fsf@HIDDEN>
 <87pmn5n3tu.fsf@HIDDEN> <87mti99j1f.fsf@HIDDEN>
 <87wnh7hkgi.fsf@HIDDEN> <87pmmz947k.fsf@HIDDEN>
 <8735ju44sk.fsf@HIDDEN> <87lexikwu5.fsf@HIDDEN>
 <87mt8baygn.fsf_-_@HIDDEN> <8335a3nguk.fsf@HIDDEN>
 <87fse1kfe8.fsf@HIDDEN> <831qpln7zg.fsf@HIDDEN>
 <CADwFkmm0hCO3ek-w0TWkGWOROCU9YfOgDO0ivtcz-Q7feUE7nA@HIDDEN>
 <837cp21qca.fsf@HIDDEN> <878r9iktd0.fsf@HIDDEN>
Date: Fri, 08 Sep 2023 06:28:58 -0700
Message-ID: <878r9ghjz9.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="=-=-="
X-Authenticated-Id: masked@HIDDEN
X-Spam-Score: 0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

--=-=-=
Content-Type: text/plain

"J.P." <jp@HIDDEN> writes:

> There's also the issue of DNS lookups for verifying domain certs. Last I
> looked, we can't run `nsm' checks without involving the system resolver,
> which may be a deal breaker for the more privacy minded. If that's true,
> we may want to find an acceptable way of cluing folks in to the
> situation.

Just a tiny update to the POC `url' integration stuff in case anyone
ever tries it. Previously, when connecting to a .onion domain over TLS,
e.g.,

  https://www.dwnewsgngmhlplxy6o2twtfgjnrnjxbegbwqx6wnotdhkzt562tszfid.onion/en/

you'd get spammed with "name or service unknown" messages in the echo
area with every EWW link clicked (at least on GNU Linux, probably from
something like GAI EAI_NONAME). Anyway, no longer. Also added a test, a
news entry, and some doc tweaks. Thanks.


--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment; filename=0000-v3-v4.diff

From 97f2cb52b73d5c0bd7409044fef5469b914a9ec9 Mon Sep 17 00:00:00 2001
From: "F. Jason Park" <jp@HIDDEN>
Date: Fri, 8 Sep 2023 06:06:35 -0700
Subject: [PATCH 0/4] *** NOT A PATCH ***

*** BLURB HERE ***

F. Jason Park (4):
  Don't hard code server ports in SOCKS tests
  Improve SOCKS error handling and add support for 4a
  [POC] Simplify network-stream openers in socks.el
  [POC] Integrate the socks and url libraries

 doc/misc/url.texi            |   8 +-
 etc/NEWS                     |   7 ++
 lisp/net/socks.el            | 141 ++++++++++++++++++++++++++++-------
 lisp/url/url-gw.el           |   8 +-
 lisp/url/url-http.el         |  16 ++--
 lisp/url/url-proxy.el        |  18 ++++-
 lisp/url/url-vars.el         |  11 ++-
 test/lisp/net/socks-tests.el |  84 ++++++++++++++++-----
 8 files changed, 231 insertions(+), 62 deletions(-)

Interdiff:
diff --git a/doc/misc/url.texi b/doc/misc/url.texi
index e6636e32507..6517f858324 100644
--- a/doc/misc/url.texi
+++ b/doc/misc/url.texi
@@ -1083,16 +1083,18 @@ Gateways in general
 @defopt socks-server
 This specifies the default server, it takes the form
 @w{@code{("Default server" @var{server} @var{port} @var{version})}}
-where @var{version} can be either 4 or 5.
+where @var{version} can be 4, 4a, or 5.
 @end defopt
 @defvar socks-password
 If this is @code{nil} then you will be asked for the password,
 otherwise it will be used as the password for authenticating you to
-the @sc{socks} server.
+the @sc{socks} server.  You can often set this to @code{""} for
+servers on your local network.
 @end defvar
 @defvar socks-username
 This is the username to use when authenticating yourself to the
-@sc{socks} server.  By default this is your login name.
+@sc{socks} server.  By default, this is your login name.  In versions
+4 and 4a, ERC uses this for the @samp{ID} field.
 @end defvar
 @defvar socks-timeout
 This controls how long, in seconds, to wait for responses from the
diff --git a/etc/NEWS b/etc/NEWS
index f6be603294e..55bcf957021 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -756,6 +756,13 @@ neither of which have been supported by Emacs since version 23.1.
 The user option 'url-gateway-nslookup-program' and the function
 'url-gateway-nslookup-host' are consequently also obsolete.
 
+** socks
+
++++
+*** SOCKS supports version 4a.
+The 'socks-server' option now accepts '4a' as a valid value for its
+version field.
+
 
 * New Modes and Packages in Emacs 30.1
 
diff --git a/lisp/net/socks.el b/lisp/net/socks.el
index 1bb78113d52..f5820e7968c 100644
--- a/lisp/net/socks.el
+++ b/lisp/net/socks.el
@@ -444,7 +444,7 @@ socks-send-command
 		      (ash port -8)       ; port, high byte
 		      (logand port #xff)) ; port, low byte
 		     addr                 ; address
-		     (user-full-name)     ; username
+                     socks-username       ; username
                      "\0"                 ; terminate username
                      trailing)))          ; optional host to look up
      ((equal version 5)
@@ -582,8 +582,8 @@ socks-open-network-stream
          (socks-server (if url
                            (list name (url-host url) (url-port url)
                                  (pcase (url-type url)
-                                   ("socks4://" 4)
-                                   ("socks4a://" '4a)
+                                   ("socks4" 4)
+                                   ("socks4a" '4a)
                                    (_ 5)))
                          socks-server))
          (socks-username (or (and url (url-user url))
@@ -604,9 +604,9 @@ socks-open-network-stream
               (progn (gnutls-negotiate :process proc
                                        :hostname host
                                        :keylist (and certs (list certs)))
-                     ;; FIXME skip when TLD is .onion.
-                     (nsm-verify-connection proc host port))
-            proc))
+                     (unless (string-suffix-p ".onion" host)
+                       (nsm-verify-connection proc host port))))
+          proc)
       (apply socks-connect-function name buffer host service params))))
 
 (defun socks--initiate-command-connect (proc buffer host service)
diff --git a/lisp/url/url-proxy.el b/lisp/url/url-proxy.el
index 1a278bb1673..c9c5a7aacac 100644
--- a/lisp/url/url-proxy.el
+++ b/lisp/url/url-proxy.el
@@ -37,13 +37,11 @@ url-default-find-proxy-for-url
 	      host))
 	(equal "www" (url-type urlobj)))
     "DIRECT")
-   ((cdr (assoc (url-type urlobj) url-proxy-services))
-    (let ((found (alist-get (url-type urlobj) url-proxy-services
-                            nil nil #'equal)))
-      (concat (if (string-match url-proxy--socks-scheme-regexp found)
+   ((and-let* ((found (assoc (url-type urlobj) url-proxy-services)))
+      (concat (if (string-match url-proxy--socks-scheme-regexp (cdr found))
                   "SOCKS "
                 "PROXY ")
-              found)))
+              (cdr found))))
    ;;
    ;; Should check for socks
    ;;
@@ -65,8 +63,8 @@ url-find-proxy-for-url
      ((string-match "^PROXY +" proxy)
       (concat "http://" (substring proxy (match-end 0)) "/"))
      ((string-match "^SOCKS +" proxy)
-      (if-let* ((m (substring proxy (match-end 0)))
-                ((string-match url-proxy--socks-scheme-regexp m)))
+      (if-let ((m (substring proxy (match-end 0)))
+               ((string-match url-proxy--socks-scheme-regexp m)))
           m
         (concat "socks://" m)))
      (t
diff --git a/test/lisp/net/socks-tests.el b/test/lisp/net/socks-tests.el
index df69fb2f5cf..1a4bac37bf9 100644
--- a/test/lisp/net/socks-tests.el
+++ b/test/lisp/net/socks-tests.el
@@ -139,7 +139,7 @@ socks-tests-canned-server-create
          (filt (lambda (proc line)
                  (pcase-let ((`(,pat . ,resp) (pop pats)))
                    (unless (or (and (vectorp pat) (equal pat (vconcat line)))
-                               (string-match-p pat line))
+                               (and (stringp pat) (string-match-p pat line)))
                      (error "Unknown request: %s" line))
                    (setq resp (apply #'unibyte-string (append resp nil)))
                    (let ((print-escape-control-characters t))
@@ -197,6 +197,7 @@ socks-tests-v4-basic
   "Show correct preparation of SOCKS4 connect command (Bug#46342)."
   (let ((socks-server '("server" "127.0.0.1" t 4))
         (url-user-agent "Test/4-basic")
+        (socks-username "foo")
         (socks-tests-canned-server-patterns
          `(([4 1 0 80 93 184 216 34 ?f ?o ?o 0] . [0 90 0 0 0 0 0 0])
            ,socks-tests--hello-world-http-request-pattern))
@@ -205,23 +206,34 @@ socks-tests-v4-basic
       (cl-letf (((symbol-function 'socks-nslookup-host)
                  (lambda (host)
                    (should (equal host "example.com"))
-                   (list 93 184 216 34)))
-                ((symbol-function 'user-full-name)
-                 (lambda (&optional _) "foo")))
+                   (list 93 184 216 34))))
         (socks-tests-perform-hello-world-http-request)))))
 
 (ert-deftest socks-tests-v4a-basic ()
   "Show correct preparation of SOCKS4a connect command."
   (let ((socks-server '("server" "127.0.0.1" t 4a))
+        (socks-username "foo")
         (url-user-agent "Test/4a-basic")
         (socks-tests-canned-server-patterns
          `(([4 1 0 80 0 0 0 1 ?f ?o ?o 0 ?e ?x ?a ?m ?p ?l ?e ?. ?c ?o ?m 0]
             . [0 90 0 0 0 0 0 0])
            ,socks-tests--hello-world-http-request-pattern)))
     (ert-info ("Make HTTP request over SOCKS4A")
-      (cl-letf (((symbol-function 'user-full-name)
-                 (lambda (&optional _) "foo")))
-        (socks-tests-perform-hello-world-http-request)))))
+      (socks-tests-perform-hello-world-http-request))))
+
+(ert-deftest socks-tests-v4a-error ()
+  "Show error signaled when destination address rejected."
+  (let ((socks-server '("server" "127.0.0.1" t 4a))
+        (url-user-agent "Test/4a-basic")
+        (socks-username "")
+        (socks-tests-canned-server-patterns
+         `(([4 1 0 80 0 0 0 1 0 ?e ?x ?a ?m ?p ?l ?e ?. ?c ?o ?m 0]
+            . [0 91 0 0 0 0 0 0])
+           ,socks-tests--hello-world-http-request-pattern)))
+    (ert-info ("Make HTTP request over SOCKS4A")
+      (let ((err (should-error
+                  (socks-tests-perform-hello-world-http-request))))
+        (should (equal err '(error "SOCKS: Rejected or failed")))))))
 
 ;; Replace first pattern below with ([5 3 0 1 2] . [5 2]) to validate
 ;; against curl 7.71 with the following options:
@@ -309,12 +321,9 @@ socks-override-functions
   (should-not (advice-member-p #'socks--open-network-stream
                                'open-network-stream))
   (advice-add 'open-network-stream :around #'socks--open-network-stream)
-  (should (advice-member-p #'socks--open-network-stream 'open-network-stream))
-
   (unwind-protect (let ((socks-override-functions t))
                     (socks-tests-v5-auth-none 'native))
     (advice-remove 'open-network-stream #'socks--open-network-stream))
-
   (should-not (advice-member-p #'socks--open-network-stream
                                'open-network-stream)))
 
-- 
2.41.0


--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
 filename=0001-Don-t-hard-code-server-ports-in-SOCKS-tests.patch

From a58a38ff9a599bf93d5e5467a01198444be2cf15 Mon Sep 17 00:00:00 2001
From: "F. Jason Park" <jp@HIDDEN>
Date: Mon, 14 Feb 2022 02:36:57 -0800
Subject: [PATCH 1/4] Don't hard code server ports in SOCKS tests

* test/lisp/net/socks-tests.el (socks-tests-canned-server-create,
socks-tests-filter-response-parsing-v4, socks-tests-v4-basic,
socks-tests-v5-auth-user-pass, socks-tests-v5-auth-user-blank,
socks-tests-v5-auth-none): Fix bug in process filter to prevent
prepared outgoing responses from being implicitly encoded as UTF-8.
Fix similar mistake in v4 filter test.  Allow system to choose port
instead of hard-coding it.
(socks-tests-perform-hello-world-http-request):
Add option method parameter to specify a gateway method.
(socks-tests-v5-auth-none): Move body to helper function of the same
name.
(socks-override-functions): New test ensuring top-level advice around
`open-networks-stream' still supported.  (Bug#53941)
---
 test/lisp/net/socks-tests.el | 53 +++++++++++++++++++++++++-----------
 1 file changed, 37 insertions(+), 16 deletions(-)

diff --git a/test/lisp/net/socks-tests.el b/test/lisp/net/socks-tests.el
index 958e2ff44a8..0890ace826f 100644
--- a/test/lisp/net/socks-tests.el
+++ b/test/lisp/net/socks-tests.el
@@ -63,21 +63,21 @@ socks-tests-filter-response-parsing-v4
     (process-put proc 'socks-state socks-state-waiting)
     (process-put proc 'socks-server-protocol 4)
     (ert-info ("Receive initial incomplete segment")
-      (socks-filter proc (concat [0 90 0 0 93 184 216]))
-      ;; From example.com: OK status ^      ^ msg start
+      (socks-filter proc (unibyte-string 0 90 0 0 93 184 216))
+      ;; From example.com: OK status       ^      ^ msg start
       (ert-info ("State still set to waiting")
         (should (eq (process-get proc 'socks-state) socks-state-waiting)))
       (ert-info ("Response field is nil because processing incomplete")
         (should-not (process-get proc 'socks-response)))
       (ert-info ("Scratch field holds stashed partial payload")
-        (should (string= (concat [0 90 0 0 93 184 216])
+        (should (string= (unibyte-string 0 90 0 0 93 184 216)
                          (process-get proc 'socks-scratch)))))
     (ert-info ("Last part arrives")
       (socks-filter proc "\42") ; ?\" 34
       (ert-info ("State transitions to complete (length check passes)")
         (should (eq (process-get proc 'socks-state) socks-state-connected)))
       (ert-info ("Scratch and response fields hold stash w. last chunk")
-        (should (string= (concat [0 90 0 0 93 184 216 34])
+        (should (string= (unibyte-string 0 90 0 0 93 184 216 34)
                          (process-get proc 'socks-response)))
         (should (string= (process-get proc 'socks-response)
                          (process-get proc 'socks-scratch)))))
@@ -133,17 +133,19 @@ socks-tests-canned-server-patterns
 (defun socks-tests-canned-server-create ()
   "Create and return a fake SOCKS server."
   (let* ((port (nth 2 socks-server))
-         (name (format "socks-server:%d" port))
+         (name (format "socks-server:%s"
+                       (if (numberp port) port (ert-test-name (ert-running-test)))))
          (pats socks-tests-canned-server-patterns)
          (filt (lambda (proc line)
                  (pcase-let ((`(,pat . ,resp) (pop pats)))
                    (unless (or (and (vectorp pat) (equal pat (vconcat line)))
-                               (string-match-p pat line))
+                               (and (stringp pat) (string-match-p pat line)))
                      (error "Unknown request: %s" line))
+                   (setq resp (apply #'unibyte-string (append resp nil)))
                    (let ((print-escape-control-characters t))
                      (message "[%s] <- %s" name (prin1-to-string line))
                      (message "[%s] -> %s" name (prin1-to-string resp)))
-                   (process-send-string proc (concat resp)))))
+                   (process-send-string proc resp))))
          (serv (make-network-process :server 1
                                      :buffer (get-buffer-create name)
                                      :filter filt
@@ -151,8 +153,10 @@ socks-tests-canned-server-create
                                      :family 'ipv4
                                      :host 'local
                                      :coding 'binary
-                                     :service port)))
+                                     :service (or port t))))
     (set-process-query-on-exit-flag serv nil)
+    (unless (numberp (nth 2 socks-server))
+      (setf (nth 2 socks-server) (process-contact serv :service)))
     serv))
 
 (defvar socks-tests--hello-world-http-request-pattern
@@ -161,9 +165,9 @@ socks-tests--hello-world-http-request-pattern
                          "Content-Length: 13\r\n\r\n"
                          "Hello World!\n")))
 
-(defun socks-tests-perform-hello-world-http-request ()
+(defun socks-tests-perform-hello-world-http-request (&optional method)
   "Start canned server, validate hello-world response, and finalize."
-  (let* ((url-gateway-method 'socks)
+  (let* ((url-gateway-method (or method 'socks))
          (url (url-generic-parse-url "http://example.com"))
          (server (socks-tests-canned-server-create))
          ;;
@@ -191,7 +195,7 @@ socks-tests-perform-hello-world-http-request
 
 (ert-deftest socks-tests-v4-basic ()
   "Show correct preparation of SOCKS4 connect command (Bug#46342)."
-  (let ((socks-server '("server" "127.0.0.1" 10079 4))
+  (let ((socks-server '("server" "127.0.0.1" t 4))
         (url-user-agent "Test/4-basic")
         (socks-tests-canned-server-patterns
          `(([4 1 0 80 93 184 216 34 ?f ?o ?o 0] . [0 90 0 0 0 0 0 0])
@@ -213,7 +217,7 @@ socks-tests-v4-basic
 (ert-deftest socks-tests-v5-auth-user-pass ()
   "Verify correct handling of SOCKS5 user/pass authentication."
   (should (assq 2 socks-authentication-methods))
-  (let ((socks-server '("server" "127.0.0.1" 10080 5))
+  (let ((socks-server '("server" "127.0.0.1" t 5))
         (socks-username "foo")
         (socks-password "bar")
         (url-user-agent "Test/auth-user-pass")
@@ -247,7 +251,7 @@ socks-tests-v5-auth-user-pass
 (ert-deftest socks-tests-v5-auth-user-pass-blank ()
   "Verify correct SOCKS5 user/pass authentication with empty pass."
   (should (assq 2 socks-authentication-methods))
-  (let ((socks-server '("server" "127.0.0.1" 10081 5))
+  (let ((socks-server '("server" "127.0.0.1" t 5))
         (socks-username "foo") ; defaults to (user-login-name)
         (socks-password "") ; simulate user hitting enter when prompted
         (url-user-agent "Test/auth-user-pass-blank")
@@ -264,9 +268,9 @@ socks-tests-v5-auth-user-pass-blank
 ;; against curl 7.71 with the following options:
 ;; $ curl --verbose --proxy socks5h://127.0.0.1:10082 example.com
 
-(ert-deftest socks-tests-v5-auth-none ()
+(defun socks-tests-v5-auth-none (method)
   "Verify correct handling of SOCKS5 when auth method 0 requested."
-  (let ((socks-server '("server" "127.0.0.1" 10082 5))
+  (let ((socks-server '("server" "127.0.0.1" t 5))
         (socks-authentication-methods (append socks-authentication-methods
                                               nil))
         (url-user-agent "Test/auth-none")
@@ -278,7 +282,24 @@ socks-tests-v5-auth-none
     (socks-unregister-authentication-method 2)
     (should-not (assq 2 socks-authentication-methods))
     (ert-info ("Make HTTP request over SOCKS5 with no auth method")
-      (socks-tests-perform-hello-world-http-request)))
+      (socks-tests-perform-hello-world-http-request method)))
   (should (assq 2 socks-authentication-methods)))
 
+(ert-deftest socks-tests-v5-auth-none ()
+  (socks-tests-v5-auth-none 'socks))
+
+;; This simulates the top-level advice around `open-network-stream'
+;; that's applied when loading the library with a non-nil
+;; `socks-override-functions'.
+(ert-deftest socks-override-functions ()
+  (should-not socks-override-functions)
+  (should-not (advice-member-p #'socks--open-network-stream
+                               'open-network-stream))
+  (advice-add 'open-network-stream :around #'socks--open-network-stream)
+  (unwind-protect (let ((socks-override-functions t))
+                    (socks-tests-v5-auth-none 'native))
+    (advice-remove 'open-network-stream #'socks--open-network-stream))
+  (should-not (advice-member-p #'socks--open-network-stream
+                               'open-network-stream)))
+
 ;;; socks-tests.el ends here
-- 
2.41.0


--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
 filename=0002-Improve-SOCKS-error-handling-and-add-support-for-4a.patch

From af4f2b326ec5bdceb039e861745dd4e79608181e Mon Sep 17 00:00:00 2001
From: "F. Jason Park" <jp@HIDDEN>
Date: Mon, 14 Feb 2022 02:36:57 -0800
Subject: [PATCH 2/4] Improve SOCKS error handling and add support for 4a

* doc/misc/url.texi: Mention version 4a in SOCKS portion of "Gateways
in general" node.
* etc/NEWS: Mention version 4a support.
* lisp/net/socks.el (socks-server): Add new Custom choice `4a' for
version field.  This change does not overload the field in terms of
expected type because `socks-send-command' and `socks-filter' already
accommodate the symbol `http'.
(socks--errors-4): Add new constant containing error messages for
version 4.  The semantics are faithful to the spec, but the exact
wording is adapted.
(socks-filter): Allow for a null "type" field on error with version 5.
Previously, certain errors would not propagate because a wrong-type
signal would get in the way.
(socks-send-command): Massage existing version 4 protocol parsing to
accommodate 4a, and add error handling for version 4.  Use variable
`socks-username' for v4 variable-length ID field instead of calling
`user-full-name'.
* test/lisp/net/socks-tests.el (socks-tests-v4-basic): Don't mock
`user-full-name' because `socks-send-command' no longer calls it to
determine the id.
(socks-tests-v4a-basic, socks-tests-v4a-error): Add a couple tests for
SOCKS version 4a.  (Bug#53941)
---
 doc/misc/url.texi            |  8 +++++---
 etc/NEWS                     |  7 +++++++
 lisp/net/socks.el            | 30 ++++++++++++++++++++++++++----
 test/lisp/net/socks-tests.el | 31 ++++++++++++++++++++++++++++---
 4 files changed, 66 insertions(+), 10 deletions(-)

diff --git a/doc/misc/url.texi b/doc/misc/url.texi
index e6636e32507..6517f858324 100644
--- a/doc/misc/url.texi
+++ b/doc/misc/url.texi
@@ -1083,16 +1083,18 @@ Gateways in general
 @defopt socks-server
 This specifies the default server, it takes the form
 @w{@code{("Default server" @var{server} @var{port} @var{version})}}
-where @var{version} can be either 4 or 5.
+where @var{version} can be 4, 4a, or 5.
 @end defopt
 @defvar socks-password
 If this is @code{nil} then you will be asked for the password,
 otherwise it will be used as the password for authenticating you to
-the @sc{socks} server.
+the @sc{socks} server.  You can often set this to @code{""} for
+servers on your local network.
 @end defvar
 @defvar socks-username
 This is the username to use when authenticating yourself to the
-@sc{socks} server.  By default this is your login name.
+@sc{socks} server.  By default, this is your login name.  In versions
+4 and 4a, ERC uses this for the @samp{ID} field.
 @end defvar
 @defvar socks-timeout
 This controls how long, in seconds, to wait for responses from the
diff --git a/etc/NEWS b/etc/NEWS
index f6be603294e..55bcf957021 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -756,6 +756,13 @@ neither of which have been supported by Emacs since version 23.1.
 The user option 'url-gateway-nslookup-program' and the function
 'url-gateway-nslookup-host' are consequently also obsolete.
 
+** socks
+
++++
+*** SOCKS supports version 4a.
+The 'socks-server' option now accepts '4a' as a valid value for its
+version field.
+
 
 * New Modes and Packages in Emacs 30.1
 
diff --git a/lisp/net/socks.el b/lisp/net/socks.el
index 968a28d2be8..b781b6a4eab 100644
--- a/lisp/net/socks.el
+++ b/lisp/net/socks.el
@@ -162,6 +162,7 @@ socks-server
 	  (radio-button-choice :tag "SOCKS Version"
 			       :format "%t: %v"
 			       (const :tag "SOCKS v4  " :format "%t" :value 4)
+                               (const :tag "SOCKS v4a"  :format "%t" :value 4a)
 			       (const :tag "SOCKS v5"   :format "%t" :value 5))))
 
 
@@ -202,6 +203,12 @@ socks-errors
     "Command not supported"
     "Address type not supported"))
 
+(defconst socks--errors-4
+  '("Granted"
+    "Rejected or failed"
+    "Cannot connect to identd on the client"
+    "Client and identd report differing user IDs"))
+
 ;; The socks v5 address types
 (defconst socks-address-type-v4   1)
 (defconst socks-address-type-name 3)
@@ -309,7 +316,8 @@ socks-filter
 		     ((pred (= socks-address-type-name))
 		      (if (< (length string) 5)
 			  255
-		        (+ 1 (aref string 4)))))))
+                        (+ 1 (aref string 4))))
+                     (0 0))))
 	  (if (< (length string) desired-len)
 	      nil			; Need to spin some more
 	    (process-put proc 'socks-state socks-state-connected)
@@ -399,6 +407,7 @@ socks-send-command
 		(format "%c%s" (length address) address))
 	       (t
 		(error "Unknown address type: %d" atype))))
+        trailing
 	request version)
     (or (process-get proc 'socks)
         (error "socks-send-command called on non-SOCKS connection %S" proc))
@@ -415,6 +424,12 @@ socks-send-command
 			     (t
 			      (error "Unsupported address type for HTTP: %d" atype)))
 			    port)))
+     ((when (eq version '4a)
+        (setf addr "\0\0\0\1"
+              trailing (concat address "\0")
+              version 4 ; done with the "a" part
+              (process-get proc 'socks-server-protocol) 4)
+        nil)) ; fall through
      ((equal version 4)
       (setq request (concat
 		     (unibyte-string
@@ -423,8 +438,9 @@ socks-send-command
 		      (ash port -8)       ; port, high byte
 		      (logand port #xff)) ; port, low byte
 		     addr                 ; address
-		     (user-full-name)     ; username
-		     "\0")))              ; terminate username
+                     socks-username       ; username
+                     "\0"                 ; terminate username
+                     trailing)))          ; optional host to look up
      ((equal version 5)
       (setq request (concat
 		     (unibyte-string
@@ -445,7 +461,13 @@ socks-send-command
 	nil				; Sweet sweet success!
       (delete-process proc)
       (error "SOCKS: %s"
-             (nth (or (process-get proc 'socks-reply) 1) socks-errors)))
+             (let ((err (process-get proc 'socks-reply)))
+               (if (eql version 5)
+                   (nth (or err 1) socks-errors)
+                 ;; The defined error codes for v4 range from
+                 ;; 90-93, but we store them in a simple list.
+                 (nth (pcase err (90 0) (92 2) (93 3) (_ 1))
+                      socks--errors-4)))))
     proc))
 
 
diff --git a/test/lisp/net/socks-tests.el b/test/lisp/net/socks-tests.el
index 0890ace826f..1a4bac37bf9 100644
--- a/test/lisp/net/socks-tests.el
+++ b/test/lisp/net/socks-tests.el
@@ -197,6 +197,7 @@ socks-tests-v4-basic
   "Show correct preparation of SOCKS4 connect command (Bug#46342)."
   (let ((socks-server '("server" "127.0.0.1" t 4))
         (url-user-agent "Test/4-basic")
+        (socks-username "foo")
         (socks-tests-canned-server-patterns
          `(([4 1 0 80 93 184 216 34 ?f ?o ?o 0] . [0 90 0 0 0 0 0 0])
            ,socks-tests--hello-world-http-request-pattern))
@@ -205,11 +206,35 @@ socks-tests-v4-basic
       (cl-letf (((symbol-function 'socks-nslookup-host)
                  (lambda (host)
                    (should (equal host "example.com"))
-                   (list 93 184 216 34)))
-                ((symbol-function 'user-full-name)
-                 (lambda (&optional _) "foo")))
+                   (list 93 184 216 34))))
         (socks-tests-perform-hello-world-http-request)))))
 
+(ert-deftest socks-tests-v4a-basic ()
+  "Show correct preparation of SOCKS4a connect command."
+  (let ((socks-server '("server" "127.0.0.1" t 4a))
+        (socks-username "foo")
+        (url-user-agent "Test/4a-basic")
+        (socks-tests-canned-server-patterns
+         `(([4 1 0 80 0 0 0 1 ?f ?o ?o 0 ?e ?x ?a ?m ?p ?l ?e ?. ?c ?o ?m 0]
+            . [0 90 0 0 0 0 0 0])
+           ,socks-tests--hello-world-http-request-pattern)))
+    (ert-info ("Make HTTP request over SOCKS4A")
+      (socks-tests-perform-hello-world-http-request))))
+
+(ert-deftest socks-tests-v4a-error ()
+  "Show error signaled when destination address rejected."
+  (let ((socks-server '("server" "127.0.0.1" t 4a))
+        (url-user-agent "Test/4a-basic")
+        (socks-username "")
+        (socks-tests-canned-server-patterns
+         `(([4 1 0 80 0 0 0 1 0 ?e ?x ?a ?m ?p ?l ?e ?. ?c ?o ?m 0]
+            . [0 91 0 0 0 0 0 0])
+           ,socks-tests--hello-world-http-request-pattern)))
+    (ert-info ("Make HTTP request over SOCKS4A")
+      (let ((err (should-error
+                  (socks-tests-perform-hello-world-http-request))))
+        (should (equal err '(error "SOCKS: Rejected or failed")))))))
+
 ;; Replace first pattern below with ([5 3 0 1 2] . [5 2]) to validate
 ;; against curl 7.71 with the following options:
 ;; $ curl --verbose -U foo:bar --proxy socks5h://127.0.0.1:10080 example.com
-- 
2.41.0


--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
 filename=0003-POC-Simplify-network-stream-openers-in-socks.el.patch

From f84edbf90bc249863ca6b8c28d90378ea4f22e9e Mon Sep 17 00:00:00 2001
From: "F. Jason Park" <jp@HIDDEN>
Date: Mon, 28 Nov 2022 22:31:50 -0800
Subject: [PATCH 3/4] [POC] Simplify network-stream openers in socks.el

* lisp/net/socks.el (socks-connect-function): New variable for
specifying an `open-network-stream'-like connect function.
(socks-open-connection): Accept additional `open-network-stream'
params passed on to opener, now `socks-connect-function',
in place of `open-network-stream'.
(socks-proxied-tls-services): Add new option for specifying ports
whose proxied connections should use TLS.
(socks--open-network-stream): Rework to serve as thin wrapper for
`socks-open-network-stream' that now hinges on rather than ignores the
variable `socks-override-functions'.
(socks-open-network-stream): Prefer parsed URL details, when present
in a non-nil `url-using-proxy', for improved compatibility with the gw
framework.
(socks--initiate-command-connect): New function to house renamed
latter half of the original `socks--open-network-stream'.  Role now
reduced to issuing the first command using an existing
process.  (Bug#53941)
---
 lisp/net/socks.el | 111 +++++++++++++++++++++++++++++++++++++---------
 1 file changed, 90 insertions(+), 21 deletions(-)

diff --git a/lisp/net/socks.el b/lisp/net/socks.el
index b781b6a4eab..f5820e7968c 100644
--- a/lisp/net/socks.el
+++ b/lisp/net/socks.el
@@ -34,7 +34,7 @@
 
 ;;; Code:
 
-(eval-when-compile (require 'cl-lib))
+(eval-when-compile (require 'cl-lib) (require 'url-parse))
 
 ;; ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
 ;; ;;; Custom widgets
@@ -335,14 +335,20 @@ socks-override-functions
 (when socks-override-functions
   (advice-add 'open-network-stream :around #'socks--open-network-stream))
 
-(defun socks-open-connection (server-info)
+(defvar socks-connect-function #'open-network-stream
+  "Function to open a network connection to a SOCKS provider.
+Called with arguments suitable for `open-network-stream'.")
+
+(defun socks-open-connection (server-info &rest stream-params)
+  "Create and initialize a SOCKS process.
+Perform authentication if needed.  Expect SERVER-INFO to resemble
+`socks-server' and STREAM-PARAMS to be keyword parameters
+accepted by `open-network-stream'."
   (save-excursion
     (let ((proc
            (let ((socks-override-functions nil))
-             (open-network-stream "socks"
-				  nil
-				  (nth 1 server-info)
-				  (nth 2 server-info))))
+             (apply socks-connect-function (nth 0 server-info) nil
+                    (nth 1 server-info) (nth 2 server-info) stream-params)))
 	  (authtype nil)
 	  version)
 
@@ -528,22 +534,85 @@ socks-find-services-entry
   (gethash (downcase service)
 	      (if udp socks-udp-services socks-tcp-services)))
 
-(defun socks-open-network-stream (name buffer host service)
-  (let ((socks-override-functions t))
-    (socks--open-network-stream
-     (lambda (&rest args)
-       (let ((socks-override-functions nil))
-         (apply #'open-network-stream args)))
-     name buffer host service)))
-
 (defun socks--open-network-stream (orig-fun name buffer host service &rest params)
-  (let ((route (and socks-override-functions
-                    (socks-find-route host service))))
-    (if (not route)
-	(apply orig-fun name buffer host service params)
-      ;; FIXME: Obey `params'!
-      (let* ((proc (socks-open-connection route))
-	     (version (process-get proc 'socks-server-protocol))
+  "Call `socks-open-network-stream', falling back to ORIG-FUN.
+Expect NAME, BUFFER, HOST, SERVICE, and PARAMS to be compatible
+with `open-network-stream'."
+  (let ((socks-connect-function orig-fun))
+    (apply (if socks-override-functions #'socks-open-network-stream orig-fun)
+           name buffer host service params)))
+
+(defcustom socks-proxied-tls-services '(443 6697)
+  "Ports whose connections should use TLS.
+Note that the system resolver may be consulted to look up host
+names for checking domain validation certs."
+  :version "30.1"
+  :type '(repeat number))
+
+(declare-function gnutls-negotiate "gnutls" (&rest rest))
+(declare-function nsm-verify-connection "nsm"
+                  (process host port &optional
+                           save-fingerprint warn-unencrypted))
+
+;;;###autoload
+(defun socks-open-network-stream (name buffer host service &rest params)
+  "Open and return a connection, possibly proxied over SOCKS.
+Expect PARAMS to contain keyword parameters recognized by
+`open-network-stream'.  Assume HOST and SERVICE refer to the
+proxied remote peer rather than the SOCKS server, but assume the
+opposite for PARAMS.  That is, if PARAMS contains a `:type' of
+`tls', treat the underlying connection to the proxy server as
+destined for encryption rather than the tunneled connection (even
+though `socks-connect-function' has the final say).  For TLS with
+proxied connections, see the option `socks-proxied-tls-services'.
+
+Before connecting, check the HOST against `socks-noproxy'.  On
+rejection, fall back to a non-SOCKS connection determined by
+the variable `socks-connect-function'.
+
+But, before doing anything, check if `url-using-proxy' is bound
+to a `url' struct object, as defined in `url-parse'.  If so,
+assume it represents the address of the desired SOCKS server
+rather than that of the remote peer, and use its fields instead
+of `socks-server' for all SOCKS connection details."
+  (require 'url-parse)
+  (let* ((url (and (url-p url-using-proxy)
+                   (string-prefix-p "socks" (url-type url-using-proxy))
+                   url-using-proxy))
+         (socks-server (if url
+                           (list name (url-host url) (url-port url)
+                                 (pcase (url-type url)
+                                   ("socks4" 4)
+                                   ("socks4a" '4a)
+                                   (_ 5)))
+                         socks-server))
+         (socks-username (or (and url (url-user url))
+                             socks-username))
+         (socks-password (or (and url (url-password url))
+                             socks-password)))
+    (if-let ((route (socks-find-route host service))
+             (proc (apply #'socks-open-connection route params)))
+        (let ((port (if (numberp service)
+                        service
+                      (process-contact proc :service)))
+              (certs (plist-get params :client-certificate)))
+          (socks--initiate-command-connect proc buffer host service)
+          (if (and (memq port socks-proxied-tls-services)
+                   (gnutls-available-p)
+                   (require 'gnutls nil t)
+                   (require 'nsm nil t))
+              (progn (gnutls-negotiate :process proc
+                                       :hostname host
+                                       :keylist (and certs (list certs)))
+                     (unless (string-suffix-p ".onion" host)
+                       (nsm-verify-connection proc host port))))
+          proc)
+      (apply socks-connect-function name buffer host service params))))
+
+(defun socks--initiate-command-connect (proc buffer host service)
+  (progn ; preserve indentation level for git blame / code review
+    (progn
+      (let* ((version (process-get proc 'socks-server-protocol))
              (atype
               (cond
                ((equal version 4)
-- 
2.41.0


--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
 filename=0004-POC-Integrate-the-socks-and-url-libraries.patch

From 97f2cb52b73d5c0bd7409044fef5469b914a9ec9 Mon Sep 17 00:00:00 2001
From: "F. Jason Park" <jp@HIDDEN>
Date: Tue, 1 Mar 2022 01:38:33 -0800
Subject: [PATCH 4/4] [POC] Integrate the socks and url libraries

* lisp/url/url-gw.el (url-open-stream): Use presence and type of
`url-using-proxy' to detect caller and massage input values according
to legacy practices.

* lisp/url/url-http.el: (url-http-find-free-connection): Don't call
`url-open-stream' with host and port from active proxy.
(url-http, url-http-async-sentinel): Only open
`url-https-proxy-connect' for non-SOCKS proxies.

* lisp/url/url-proxy.el (url-proxy--socks-scheme-regexp): Add new
const.
(url-default-find-proxy-for-url): Accommodate SOCKS entries but defy
original design somewhat by requiring a URL scheme in the host value
for detection.
(url-find-proxy-for-url): Recognize modified host/address value for
socks entries of `url-proxy-services' and deal accordingly.
(url-proxy): Handle a SOCKS proxy for http(s) connections only.

* lisp/url/url-vars.el (url-proxy-services): Explain that values for
certain gateways may need a leading scheme:// portion.
(url-using-proxy): Add warning regarding expected type.
---
 lisp/url/url-gw.el    |  8 +++++++-
 lisp/url/url-http.el  | 16 +++++++---------
 lisp/url/url-proxy.el | 18 +++++++++++++++---
 lisp/url/url-vars.el  | 11 +++++++++--
 4 files changed, 38 insertions(+), 15 deletions(-)

diff --git a/lisp/url/url-gw.el b/lisp/url/url-gw.el
index 568ce8679f5..a65245a58a3 100644
--- a/lisp/url/url-gw.el
+++ b/lisp/url/url-gw.el
@@ -28,7 +28,7 @@
 (require 'url-vars)
 (require 'url-parse)
 
-(autoload 'socks-open-network-stream "socks")
+(autoload 'socks-open-network-stream "socks") ; FIXME remove this
 
 (defgroup url-gateway nil
   "URL gateway variables."
@@ -226,6 +226,12 @@ url-open-stream
 Optional arg GATEWAY-METHOD specifies the gateway to be used,
 overriding the value of `url-gateway-method'."
   (unless url-gateway-unplugged
+    (when (url-p url-using-proxy)
+      (if (or (eq 'socks url-gateway-method)
+              (string-prefix-p "socks" (url-type url-using-proxy)))
+          (setq gateway-method 'socks)
+        (setq host (url-host url-using-proxy)
+              service (url-port url-using-proxy))))
     (let* ((gwm (or gateway-method url-gateway-method))
            (gw-method (if (and url-gateway-local-host-regexp
                                (not (eq 'tls gwm))
diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
index ada6341ee73..42cfb9959a7 100644
--- a/lisp/url/url-http.el
+++ b/lisp/url/url-http.el
@@ -195,12 +195,7 @@ url-http-find-free-connection
 	;; like authentication.  But we use another buffer afterwards.
 	(unwind-protect
             (let ((proc (url-open-stream host buf
-                                         (if url-using-proxy
-                                             (url-host url-using-proxy)
-                                           host)
-                                         (if url-using-proxy
-                                             (url-port url-using-proxy)
-                                           port)
+                                         host port
                                          gateway-method)))
 	      ;; url-open-stream might return nil.
 	      (when (processp proc)
@@ -1396,8 +1391,9 @@ url-http
            (error "Could not create connection to %s:%d" (url-host url)
                   (url-port url)))
           (_
-           (if (and url-http-proxy (string= "https"
-                                            (url-type url-current-object)))
+           (if (and url-http-proxy
+                    (not (string-prefix-p "socks" (url-type url-http-proxy)))
+                    (string= "https" (url-type url-current-object)))
                (url-https-proxy-connect connection)
              (set-process-sentinel connection
                                    #'url-http-end-of-document-sentinel)
@@ -1479,7 +1475,9 @@ url-http-async-sentinel
 	(url-http-end-of-document-sentinel proc why))
        ((string= (substring why 0 4) "open")
 	(setq url-http-connection-opened t)
-        (if (and url-http-proxy (string= "https" (url-type url-current-object)))
+        (if (and url-http-proxy
+                 (not (string-prefix-p "socks" (url-type url-http-proxy)))
+                 (string= "https" (url-type url-current-object)))
             (url-https-proxy-connect proc)
           (condition-case error
               (process-send-string proc (url-http-create-request))
diff --git a/lisp/url/url-proxy.el b/lisp/url/url-proxy.el
index 0c330069789..c9c5a7aacac 100644
--- a/lisp/url/url-proxy.el
+++ b/lisp/url/url-proxy.el
@@ -25,6 +25,9 @@
 
 (require 'url-parse)
 
+(defconst url-proxy--socks-scheme-regexp
+  (rx bot "socks" (? (or "4" "4a" "5" "5h")) "://"))
+
 (defun url-default-find-proxy-for-url (urlobj host)
   (cond
    ((or (and (assoc "no_proxy" url-proxy-services)
@@ -34,8 +37,11 @@ url-default-find-proxy-for-url
 	      host))
 	(equal "www" (url-type urlobj)))
     "DIRECT")
-   ((cdr (assoc (url-type urlobj) url-proxy-services))
-    (concat "PROXY " (cdr (assoc (url-type urlobj) url-proxy-services))))
+   ((and-let* ((found (assoc (url-type urlobj) url-proxy-services)))
+      (concat (if (string-match url-proxy--socks-scheme-regexp (cdr found))
+                  "SOCKS "
+                "PROXY ")
+              (cdr found))))
    ;;
    ;; Should check for socks
    ;;
@@ -57,7 +63,10 @@ url-find-proxy-for-url
      ((string-match "^PROXY +" proxy)
       (concat "http://" (substring proxy (match-end 0)) "/"))
      ((string-match "^SOCKS +" proxy)
-      (concat "socks://" (substring proxy (match-end 0))))
+      (if-let ((m (substring proxy (match-end 0)))
+               ((string-match url-proxy--socks-scheme-regexp m)))
+          m
+        (concat "socks://" m)))
      (t
       (display-warning 'url (format "Unknown proxy directive: %s" proxy) :error)
       nil))))
@@ -72,6 +81,9 @@ url-proxy
   (cond
    ((string= (url-type url-using-proxy) "http")
     (url-http url callback cbargs))
+   ((and (string-prefix-p "socks" (url-type url-using-proxy))
+         (string-prefix-p "http" (url-type url)))
+    (url-http url callback cbargs))
    (t
     (error "Don't know how to use proxy `%s'" url-using-proxy))))
 
diff --git a/lisp/url/url-vars.el b/lisp/url/url-vars.el
index ef4b8b2841b..87dfdb9916c 100644
--- a/lisp/url/url-vars.el
+++ b/lisp/url/url-vars.el
@@ -192,10 +192,15 @@ url-mail-command
 (defcustom url-proxy-services nil
   "An alist of schemes and proxy servers that gateway them.
 Looks like ((\"http\" . \"hostname:portnumber\") ...).  This is set up
-from the ACCESS_proxy environment variables."
+from the ACCESS_proxy environment variables.  Depending on the
+gateway type, Emacs may expect certain server values to specfiy a
+\"scheme\", for example, \"proxyscheme://hostname:portnumber\",
+in which \"proxyscheme\" is something like \"socks5\".  As of
+Emacs 30.1, this only applies to SOCKS servers."
   :type '(repeat (cons :format "%v"
 		       (string :tag "Protocol")
 		       (string :tag "Proxy")))
+  :version "30.1"
   :group 'url)
 
 (defcustom url-standalone-mode nil
@@ -310,7 +315,9 @@ url-show-status
 
 (defvar url-using-proxy nil
   "Either nil or the fully qualified proxy URL in use, e.g.
-https://www.example.com/")
+https://www.example.com/.  Beware that some functions, such as
+`url-proxy' and `url-http-end-of-document-sentinel', set this to
+a `url' struct.")
 
 (defcustom url-news-server nil
   "The default news server from which to get newsgroups/articles.
-- 
2.41.0


--=-=-=--

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#53941: 27.2; socks + tor dont work with https
Resent-From: "J.P." <jp@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sat, 09 Sep 2023 14:06:02 +0000
Resent-Message-ID: <handler.53941.B53941.169426833112602 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 53941
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: patch
To: 53941 <at> debbugs.gnu.org
Cc: larsi@HIDDEN, gnuhacker@HIDDEN, Eli Zaretskii <eliz@HIDDEN>, Stefan Kangas <stefankangas@HIDDEN>
Received: via spool by 53941-submit <at> debbugs.gnu.org id=B53941.169426833112602
          (code B ref 53941); Sat, 09 Sep 2023 14:06:02 +0000
Received: (at 53941) by debbugs.gnu.org; 9 Sep 2023 14:05:31 +0000
Received: from localhost ([127.0.0.1]:48193 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qeyaX-0003HA-GG
	for submit <at> debbugs.gnu.org; Sat, 09 Sep 2023 10:05:31 -0400
Received: from mail-108-mta248.mxroute.com ([136.175.108.248]:32947)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <jp@HIDDEN>) id 1qeyaU-0003H0-6d
 for 53941 <at> debbugs.gnu.org; Sat, 09 Sep 2023 10:05:29 -0400
Received: from mail-111-mta2.mxroute.com ([136.175.111.2]
 filter006.mxroute.com) (Authenticated sender: mN4UYu2MZsgR)
 by mail-108-mta248.mxroute.com (ZoneMTA) with ESMTPSA id
 18a7a42e1f9000d7b6.001 for <53941 <at> debbugs.gnu.org>
 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384);
 Sat, 09 Sep 2023 14:05:18 +0000
X-Zone-Loop: 39245c3c703e9f5f283ebd37fce9fd5cec016028e1ac
X-Originating-IP: [136.175.111.2]
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=neverwas.me
 ; s=x;
 h=Content-Type:MIME-Version:Message-ID:Date:References:In-Reply-To:
 Subject:Cc:To:From:Sender:Reply-To:Content-Transfer-Encoding:Content-ID:
 Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc
 :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe:
 List-Post:List-Owner:List-Archive;
 bh=1IyR58ZBpgcgMMQZR1fYt3pTaF10r8cY5PwqOil5Xoo=; b=en2TJU3XzyJyVdJ+A2bJ2VWOqI
 pqoervtnKTBbKsjFe3y8IvpNL1J1KNFRQWtGE52KfRbN9GxkKRvU0Z6BYv/JtuaNYub2+L1IungqH
 LIM4nJvsuCi7zc0MzRwfiJgyZc0QLs7o9OzUocCt/HIwVW/iD6uRlhtO7V9sBPt2y3xmDL+TlvBt+
 x7QnGT0qk5L/htxI7aeoFlJBBzi0c9K7jkNr2N+7F/PcnsXacHrDE5ijkv+FgYPNn5Y+mPP4Pdfi4
 SKBSIFjDqhdfV5IAeiIFCv0janmK5BcFRgi/sb/JVzl0BRC8PF7ixiJHGd/+ZuzU0oHSDdeSgQlkD
 3wjWkwKA==;
From: "J.P." <jp@HIDDEN>
In-Reply-To: <878r9ghjz9.fsf@HIDDEN> (J. P.'s message of "Fri, 08 Sep
 2023 06:28:58 -0700")
References: <87pmntfym7.fsf@HIDDEN> <8735kl1v58.fsf@HIDDEN>
 <87a6emftzx.fsf@HIDDEN> <87k0do5km1.fsf@HIDDEN>
 <87pmn5n3tu.fsf@HIDDEN> <87mti99j1f.fsf@HIDDEN>
 <87wnh7hkgi.fsf@HIDDEN> <87pmmz947k.fsf@HIDDEN>
 <8735ju44sk.fsf@HIDDEN> <87lexikwu5.fsf@HIDDEN>
 <87mt8baygn.fsf_-_@HIDDEN> <8335a3nguk.fsf@HIDDEN>
 <87fse1kfe8.fsf@HIDDEN> <831qpln7zg.fsf@HIDDEN>
 <CADwFkmm0hCO3ek-w0TWkGWOROCU9YfOgDO0ivtcz-Q7feUE7nA@HIDDEN>
 <837cp21qca.fsf@HIDDEN> <878r9iktd0.fsf@HIDDEN>
 <878r9ghjz9.fsf@HIDDEN>
Date: Sat, 09 Sep 2023 07:05:12 -0700
Message-ID: <87cyyrbfxj.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="=-=-="
X-Authenticated-Id: masked@HIDDEN
X-Spam-Score: 0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

--=-=-=
Content-Type: text/plain

I seem to have missed the fact that `url-methods' populates the option
`url-proxy-services' from environment variables when initializing
`url-scheme-registry'. As explained in the docs, it actually accepts
full URLs instead of just host:port pairs. I take this to mean it's
probably less disruptive than initially thought to extend this liberty
to `url-proxy-services' itself. I've updated the POC patches to reflect
this, so it should now be possible to do:

    (setenv "HTTPS_PROXY" "socks5h://localhost:9050")
    (eww "https://check.torproject.org/")

Of course, this shouldn't interfere with traditional http proxies, such
as those that provide CONNECT tunneling:

    $ ncat -l --proxy-type http localhost 8888
    (setenv "HTTPS_PROXY" "localhost:8888")
    (eww "https://www.example.com/")


--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment; filename=0000-v4-v5.diff

From 6e0e98f0bc89a2c9a434c9a1e837750a371f6d1e Mon Sep 17 00:00:00 2001
From: "F. Jason Park" <jp@HIDDEN>
Date: Sat, 9 Sep 2023 06:22:33 -0700
Subject: [PATCH 0/4] *** NOT A PATCH ***

*** BLURB HERE ***

F. Jason Park (4):
  Don't hard code server ports in SOCKS tests
  Improve SOCKS error handling and add support for 4a
  [POC] Simplify network-stream openers in socks.el
  [POC] Integrate the socks and url libraries

 doc/misc/url.texi            |   8 +-
 etc/NEWS                     |   7 ++
 lisp/net/socks.el            | 141 ++++++++++++++++++++++++++++-------
 lisp/url/url-gw.el           |   8 +-
 lisp/url/url-http.el         |  19 ++---
 lisp/url/url-methods.el      |   8 +-
 lisp/url/url-proxy.el        |  22 ++++--
 lisp/url/url-vars.el         |  20 ++++-
 test/lisp/net/socks-tests.el |  84 ++++++++++++++++-----
 9 files changed, 248 insertions(+), 69 deletions(-)

Interdiff:
diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
index 42cfb9959a7..47c785a0735 100644
--- a/lisp/url/url-http.el
+++ b/lisp/url/url-http.el
@@ -1392,7 +1392,10 @@ url-http
                   (url-port url)))
           (_
            (if (and url-http-proxy
-                    (not (string-prefix-p "socks" (url-type url-http-proxy)))
+                    ;; Set to "http" by `url-find-proxy-for-url' for
+                    ;; any matching non-blacklisted, non-SOCKS scheme
+                    ;; in `url-proxy-services', including "https".
+                    (equal "http" (url-type url-http-proxy))
                     (string= "https" (url-type url-current-object)))
                (url-https-proxy-connect connection)
              (set-process-sentinel connection
@@ -1476,7 +1479,7 @@ url-http-async-sentinel
        ((string= (substring why 0 4) "open")
 	(setq url-http-connection-opened t)
         (if (and url-http-proxy
-                 (not (string-prefix-p "socks" (url-type url-http-proxy)))
+                 (equal "http" (url-type url-http-proxy))
                  (string= "https" (url-type url-current-object)))
             (url-https-proxy-connect proc)
           (condition-case error
diff --git a/lisp/url/url-methods.el b/lisp/url/url-methods.el
index 9643e992044..9592307aea8 100644
--- a/lisp/url/url-methods.el
+++ b/lisp/url/url-methods.el
@@ -92,7 +92,6 @@ url-scheme-register-proxy
      ;; Then check if its a fully specified URL
      ((string-match url-nonrelative-link env-proxy)
       (setq urlobj (url-generic-parse-url env-proxy))
-      (setf (url-type urlobj) "http")
       (setf (url-target urlobj) nil))
      ;; Finally, fall back on the assumption that its just a hostname
      (t
@@ -103,8 +102,11 @@ url-scheme-register-proxy
      (if (and (not cur-proxy) urlobj)
 	 (progn
 	   (setq url-proxy-services
-		 (cons (cons scheme (format "%s:%d" (url-host urlobj)
-					    (url-port urlobj)))
+                 (cons (cons scheme (if (member (url-type urlobj)
+                                                url-proxy-full-address-types)
+                                        (url-recreate-url urlobj)
+                                      (format "%s:%d" (url-host urlobj)
+                                              (url-port urlobj))))
 		       url-proxy-services))
 	   (message "Using a proxy for %s..." scheme)))))
 
diff --git a/lisp/url/url-proxy.el b/lisp/url/url-proxy.el
index c9c5a7aacac..b1583523cc6 100644
--- a/lisp/url/url-proxy.el
+++ b/lisp/url/url-proxy.el
@@ -25,9 +25,6 @@
 
 (require 'url-parse)
 
-(defconst url-proxy--socks-scheme-regexp
-  (rx bot "socks" (? (or "4" "4a" "5" "5h")) "://"))
-
 (defun url-default-find-proxy-for-url (urlobj host)
   (cond
    ((or (and (assoc "no_proxy" url-proxy-services)
@@ -38,13 +35,12 @@ url-default-find-proxy-for-url
 	(equal "www" (url-type urlobj)))
     "DIRECT")
    ((and-let* ((found (assoc (url-type urlobj) url-proxy-services)))
-      (concat (if (string-match url-proxy--socks-scheme-regexp (cdr found))
-                  "SOCKS "
+      (concat (if-let ((non-scheme (string-search "://" (cdr found)))
+                       (scheme (substring (cdr found) 0 non-scheme))
+                       ((member scheme url-proxy-full-address-types)))
+                  (concat scheme " ")
                 "PROXY ")
               (cdr found))))
-   ;;
-   ;; Should check for socks
-   ;;
    (t
     "DIRECT")))
 
@@ -62,9 +58,9 @@ url-find-proxy-for-url
      ((string-match "^DIRECT" proxy) nil)
      ((string-match "^PROXY +" proxy)
       (concat "http://" (substring proxy (match-end 0)) "/"))
-     ((string-match "^SOCKS +" proxy)
+     ((string-match  (rx bot "SOCKS" (** 0 2 alnum) " ") proxy)
       (if-let ((m (substring proxy (match-end 0)))
-               ((string-match url-proxy--socks-scheme-regexp m)))
+               ((string-search "://" m)))
           m
         (concat "socks://" m)))
      (t
diff --git a/lisp/url/url-vars.el b/lisp/url/url-vars.el
index 87dfdb9916c..f10158d66a1 100644
--- a/lisp/url/url-vars.el
+++ b/lisp/url/url-vars.el
@@ -189,18 +189,27 @@ url-mail-command
   :type 'function
   :group 'url)
 
+(defvar url-proxy-full-address-types
+  '("socks" "socks5" "socks5h" "socks4" "socks4a")
+  "Schemes for URL types preserved in `url-proxy-services' entries.
+When dynamically adding a new `url-proxy-services' entry derived
+from the environment, Emacs only retains the host and port
+portions unless the URL's scheme appears in this variable's
+value.")
+
 (defcustom url-proxy-services nil
   "An alist of schemes and proxy servers that gateway them.
 Looks like ((\"http\" . \"hostname:portnumber\") ...).  This is set up
-from the ACCESS_proxy environment variables.  Depending on the
-gateway type, Emacs may expect certain server values to specfiy a
-\"scheme\", for example, \"proxyscheme://hostname:portnumber\",
-in which \"proxyscheme\" is something like \"socks5\".  As of
-Emacs 30.1, this only applies to SOCKS servers."
+from the ACCESS_proxy environment variables.  Certain gateway
+types need server values to take the form of full URLs in order
+to convey addtional information about for the proxy connection
+itself, for example, SCHEME://USER@HOSTNAME:PORTNUMBER, in which
+SCHEME is something like \"socks5\".  As of Emacs 30.1, this only
+applies to SCHEMEs appearing in the variable
+`url-proxy-full-address-types'."
   :type '(repeat (cons :format "%v"
 		       (string :tag "Protocol")
 		       (string :tag "Proxy")))
-  :version "30.1"
   :group 'url)
 
 (defcustom url-standalone-mode nil
@@ -317,7 +326,7 @@ url-using-proxy
   "Either nil or the fully qualified proxy URL in use, e.g.
 https://www.example.com/.  Beware that some functions, such as
 `url-proxy' and `url-http-end-of-document-sentinel', set this to
-a `url' struct.")
+a `url' struct object.")
 
 (defcustom url-news-server nil
   "The default news server from which to get newsgroups/articles.
-- 
2.41.0


--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
 filename=0001-Don-t-hard-code-server-ports-in-SOCKS-tests.patch

From a58a38ff9a599bf93d5e5467a01198444be2cf15 Mon Sep 17 00:00:00 2001
From: "F. Jason Park" <jp@HIDDEN>
Date: Mon, 14 Feb 2022 02:36:57 -0800
Subject: [PATCH 1/4] Don't hard code server ports in SOCKS tests

* test/lisp/net/socks-tests.el (socks-tests-canned-server-create,
socks-tests-filter-response-parsing-v4, socks-tests-v4-basic,
socks-tests-v5-auth-user-pass, socks-tests-v5-auth-user-blank,
socks-tests-v5-auth-none): Fix bug in process filter to prevent
prepared outgoing responses from being implicitly encoded as UTF-8.
Fix similar mistake in v4 filter test.  Allow system to choose port
instead of hard-coding it.
(socks-tests-perform-hello-world-http-request):
Add option method parameter to specify a gateway method.
(socks-tests-v5-auth-none): Move body to helper function of the same
name.
(socks-override-functions): New test ensuring top-level advice around
`open-networks-stream' still supported.  (Bug#53941)
---
 test/lisp/net/socks-tests.el | 53 +++++++++++++++++++++++++-----------
 1 file changed, 37 insertions(+), 16 deletions(-)

diff --git a/test/lisp/net/socks-tests.el b/test/lisp/net/socks-tests.el
index 958e2ff44a8..0890ace826f 100644
--- a/test/lisp/net/socks-tests.el
+++ b/test/lisp/net/socks-tests.el
@@ -63,21 +63,21 @@ socks-tests-filter-response-parsing-v4
     (process-put proc 'socks-state socks-state-waiting)
     (process-put proc 'socks-server-protocol 4)
     (ert-info ("Receive initial incomplete segment")
-      (socks-filter proc (concat [0 90 0 0 93 184 216]))
-      ;; From example.com: OK status ^      ^ msg start
+      (socks-filter proc (unibyte-string 0 90 0 0 93 184 216))
+      ;; From example.com: OK status       ^      ^ msg start
       (ert-info ("State still set to waiting")
         (should (eq (process-get proc 'socks-state) socks-state-waiting)))
       (ert-info ("Response field is nil because processing incomplete")
         (should-not (process-get proc 'socks-response)))
       (ert-info ("Scratch field holds stashed partial payload")
-        (should (string= (concat [0 90 0 0 93 184 216])
+        (should (string= (unibyte-string 0 90 0 0 93 184 216)
                          (process-get proc 'socks-scratch)))))
     (ert-info ("Last part arrives")
       (socks-filter proc "\42") ; ?\" 34
       (ert-info ("State transitions to complete (length check passes)")
         (should (eq (process-get proc 'socks-state) socks-state-connected)))
       (ert-info ("Scratch and response fields hold stash w. last chunk")
-        (should (string= (concat [0 90 0 0 93 184 216 34])
+        (should (string= (unibyte-string 0 90 0 0 93 184 216 34)
                          (process-get proc 'socks-response)))
         (should (string= (process-get proc 'socks-response)
                          (process-get proc 'socks-scratch)))))
@@ -133,17 +133,19 @@ socks-tests-canned-server-patterns
 (defun socks-tests-canned-server-create ()
   "Create and return a fake SOCKS server."
   (let* ((port (nth 2 socks-server))
-         (name (format "socks-server:%d" port))
+         (name (format "socks-server:%s"
+                       (if (numberp port) port (ert-test-name (ert-running-test)))))
          (pats socks-tests-canned-server-patterns)
          (filt (lambda (proc line)
                  (pcase-let ((`(,pat . ,resp) (pop pats)))
                    (unless (or (and (vectorp pat) (equal pat (vconcat line)))
-                               (string-match-p pat line))
+                               (and (stringp pat) (string-match-p pat line)))
                      (error "Unknown request: %s" line))
+                   (setq resp (apply #'unibyte-string (append resp nil)))
                    (let ((print-escape-control-characters t))
                      (message "[%s] <- %s" name (prin1-to-string line))
                      (message "[%s] -> %s" name (prin1-to-string resp)))
-                   (process-send-string proc (concat resp)))))
+                   (process-send-string proc resp))))
          (serv (make-network-process :server 1
                                      :buffer (get-buffer-create name)
                                      :filter filt
@@ -151,8 +153,10 @@ socks-tests-canned-server-create
                                      :family 'ipv4
                                      :host 'local
                                      :coding 'binary
-                                     :service port)))
+                                     :service (or port t))))
     (set-process-query-on-exit-flag serv nil)
+    (unless (numberp (nth 2 socks-server))
+      (setf (nth 2 socks-server) (process-contact serv :service)))
     serv))
 
 (defvar socks-tests--hello-world-http-request-pattern
@@ -161,9 +165,9 @@ socks-tests--hello-world-http-request-pattern
                          "Content-Length: 13\r\n\r\n"
                          "Hello World!\n")))
 
-(defun socks-tests-perform-hello-world-http-request ()
+(defun socks-tests-perform-hello-world-http-request (&optional method)
   "Start canned server, validate hello-world response, and finalize."
-  (let* ((url-gateway-method 'socks)
+  (let* ((url-gateway-method (or method 'socks))
          (url (url-generic-parse-url "http://example.com"))
          (server (socks-tests-canned-server-create))
          ;;
@@ -191,7 +195,7 @@ socks-tests-perform-hello-world-http-request
 
 (ert-deftest socks-tests-v4-basic ()
   "Show correct preparation of SOCKS4 connect command (Bug#46342)."
-  (let ((socks-server '("server" "127.0.0.1" 10079 4))
+  (let ((socks-server '("server" "127.0.0.1" t 4))
         (url-user-agent "Test/4-basic")
         (socks-tests-canned-server-patterns
          `(([4 1 0 80 93 184 216 34 ?f ?o ?o 0] . [0 90 0 0 0 0 0 0])
@@ -213,7 +217,7 @@ socks-tests-v4-basic
 (ert-deftest socks-tests-v5-auth-user-pass ()
   "Verify correct handling of SOCKS5 user/pass authentication."
   (should (assq 2 socks-authentication-methods))
-  (let ((socks-server '("server" "127.0.0.1" 10080 5))
+  (let ((socks-server '("server" "127.0.0.1" t 5))
         (socks-username "foo")
         (socks-password "bar")
         (url-user-agent "Test/auth-user-pass")
@@ -247,7 +251,7 @@ socks-tests-v5-auth-user-pass
 (ert-deftest socks-tests-v5-auth-user-pass-blank ()
   "Verify correct SOCKS5 user/pass authentication with empty pass."
   (should (assq 2 socks-authentication-methods))
-  (let ((socks-server '("server" "127.0.0.1" 10081 5))
+  (let ((socks-server '("server" "127.0.0.1" t 5))
         (socks-username "foo") ; defaults to (user-login-name)
         (socks-password "") ; simulate user hitting enter when prompted
         (url-user-agent "Test/auth-user-pass-blank")
@@ -264,9 +268,9 @@ socks-tests-v5-auth-user-pass-blank
 ;; against curl 7.71 with the following options:
 ;; $ curl --verbose --proxy socks5h://127.0.0.1:10082 example.com
 
-(ert-deftest socks-tests-v5-auth-none ()
+(defun socks-tests-v5-auth-none (method)
   "Verify correct handling of SOCKS5 when auth method 0 requested."
-  (let ((socks-server '("server" "127.0.0.1" 10082 5))
+  (let ((socks-server '("server" "127.0.0.1" t 5))
         (socks-authentication-methods (append socks-authentication-methods
                                               nil))
         (url-user-agent "Test/auth-none")
@@ -278,7 +282,24 @@ socks-tests-v5-auth-none
     (socks-unregister-authentication-method 2)
     (should-not (assq 2 socks-authentication-methods))
     (ert-info ("Make HTTP request over SOCKS5 with no auth method")
-      (socks-tests-perform-hello-world-http-request)))
+      (socks-tests-perform-hello-world-http-request method)))
   (should (assq 2 socks-authentication-methods)))
 
+(ert-deftest socks-tests-v5-auth-none ()
+  (socks-tests-v5-auth-none 'socks))
+
+;; This simulates the top-level advice around `open-network-stream'
+;; that's applied when loading the library with a non-nil
+;; `socks-override-functions'.
+(ert-deftest socks-override-functions ()
+  (should-not socks-override-functions)
+  (should-not (advice-member-p #'socks--open-network-stream
+                               'open-network-stream))
+  (advice-add 'open-network-stream :around #'socks--open-network-stream)
+  (unwind-protect (let ((socks-override-functions t))
+                    (socks-tests-v5-auth-none 'native))
+    (advice-remove 'open-network-stream #'socks--open-network-stream))
+  (should-not (advice-member-p #'socks--open-network-stream
+                               'open-network-stream)))
+
 ;;; socks-tests.el ends here
-- 
2.41.0


--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
 filename=0002-Improve-SOCKS-error-handling-and-add-support-for-4a.patch

From af4f2b326ec5bdceb039e861745dd4e79608181e Mon Sep 17 00:00:00 2001
From: "F. Jason Park" <jp@HIDDEN>
Date: Mon, 14 Feb 2022 02:36:57 -0800
Subject: [PATCH 2/4] Improve SOCKS error handling and add support for 4a

* doc/misc/url.texi: Mention version 4a in SOCKS portion of "Gateways
in general" node.
* etc/NEWS: Mention version 4a support.
* lisp/net/socks.el (socks-server): Add new Custom choice `4a' for
version field.  This change does not overload the field in terms of
expected type because `socks-send-command' and `socks-filter' already
accommodate the symbol `http'.
(socks--errors-4): Add new constant containing error messages for
version 4.  The semantics are faithful to the spec, but the exact
wording is adapted.
(socks-filter): Allow for a null "type" field on error with version 5.
Previously, certain errors would not propagate because a wrong-type
signal would get in the way.
(socks-send-command): Massage existing version 4 protocol parsing to
accommodate 4a, and add error handling for version 4.  Use variable
`socks-username' for v4 variable-length ID field instead of calling
`user-full-name'.
* test/lisp/net/socks-tests.el (socks-tests-v4-basic): Don't mock
`user-full-name' because `socks-send-command' no longer calls it to
determine the id.
(socks-tests-v4a-basic, socks-tests-v4a-error): Add a couple tests for
SOCKS version 4a.  (Bug#53941)
---
 doc/misc/url.texi            |  8 +++++---
 etc/NEWS                     |  7 +++++++
 lisp/net/socks.el            | 30 ++++++++++++++++++++++++++----
 test/lisp/net/socks-tests.el | 31 ++++++++++++++++++++++++++++---
 4 files changed, 66 insertions(+), 10 deletions(-)

diff --git a/doc/misc/url.texi b/doc/misc/url.texi
index e6636e32507..6517f858324 100644
--- a/doc/misc/url.texi
+++ b/doc/misc/url.texi
@@ -1083,16 +1083,18 @@ Gateways in general
 @defopt socks-server
 This specifies the default server, it takes the form
 @w{@code{("Default server" @var{server} @var{port} @var{version})}}
-where @var{version} can be either 4 or 5.
+where @var{version} can be 4, 4a, or 5.
 @end defopt
 @defvar socks-password
 If this is @code{nil} then you will be asked for the password,
 otherwise it will be used as the password for authenticating you to
-the @sc{socks} server.
+the @sc{socks} server.  You can often set this to @code{""} for
+servers on your local network.
 @end defvar
 @defvar socks-username
 This is the username to use when authenticating yourself to the
-@sc{socks} server.  By default this is your login name.
+@sc{socks} server.  By default, this is your login name.  In versions
+4 and 4a, ERC uses this for the @samp{ID} field.
 @end defvar
 @defvar socks-timeout
 This controls how long, in seconds, to wait for responses from the
diff --git a/etc/NEWS b/etc/NEWS
index f6be603294e..55bcf957021 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -756,6 +756,13 @@ neither of which have been supported by Emacs since version 23.1.
 The user option 'url-gateway-nslookup-program' and the function
 'url-gateway-nslookup-host' are consequently also obsolete.
 
+** socks
+
++++
+*** SOCKS supports version 4a.
+The 'socks-server' option now accepts '4a' as a valid value for its
+version field.
+
 
 * New Modes and Packages in Emacs 30.1
 
diff --git a/lisp/net/socks.el b/lisp/net/socks.el
index 968a28d2be8..b781b6a4eab 100644
--- a/lisp/net/socks.el
+++ b/lisp/net/socks.el
@@ -162,6 +162,7 @@ socks-server
 	  (radio-button-choice :tag "SOCKS Version"
 			       :format "%t: %v"
 			       (const :tag "SOCKS v4  " :format "%t" :value 4)
+                               (const :tag "SOCKS v4a"  :format "%t" :value 4a)
 			       (const :tag "SOCKS v5"   :format "%t" :value 5))))
 
 
@@ -202,6 +203,12 @@ socks-errors
     "Command not supported"
     "Address type not supported"))
 
+(defconst socks--errors-4
+  '("Granted"
+    "Rejected or failed"
+    "Cannot connect to identd on the client"
+    "Client and identd report differing user IDs"))
+
 ;; The socks v5 address types
 (defconst socks-address-type-v4   1)
 (defconst socks-address-type-name 3)
@@ -309,7 +316,8 @@ socks-filter
 		     ((pred (= socks-address-type-name))
 		      (if (< (length string) 5)
 			  255
-		        (+ 1 (aref string 4)))))))
+                        (+ 1 (aref string 4))))
+                     (0 0))))
 	  (if (< (length string) desired-len)
 	      nil			; Need to spin some more
 	    (process-put proc 'socks-state socks-state-connected)
@@ -399,6 +407,7 @@ socks-send-command
 		(format "%c%s" (length address) address))
 	       (t
 		(error "Unknown address type: %d" atype))))
+        trailing
 	request version)
     (or (process-get proc 'socks)
         (error "socks-send-command called on non-SOCKS connection %S" proc))
@@ -415,6 +424,12 @@ socks-send-command
 			     (t
 			      (error "Unsupported address type for HTTP: %d" atype)))
 			    port)))
+     ((when (eq version '4a)
+        (setf addr "\0\0\0\1"
+              trailing (concat address "\0")
+              version 4 ; done with the "a" part
+              (process-get proc 'socks-server-protocol) 4)
+        nil)) ; fall through
      ((equal version 4)
       (setq request (concat
 		     (unibyte-string
@@ -423,8 +438,9 @@ socks-send-command
 		      (ash port -8)       ; port, high byte
 		      (logand port #xff)) ; port, low byte
 		     addr                 ; address
-		     (user-full-name)     ; username
-		     "\0")))              ; terminate username
+                     socks-username       ; username
+                     "\0"                 ; terminate username
+                     trailing)))          ; optional host to look up
      ((equal version 5)
       (setq request (concat
 		     (unibyte-string
@@ -445,7 +461,13 @@ socks-send-command
 	nil				; Sweet sweet success!
       (delete-process proc)
       (error "SOCKS: %s"
-             (nth (or (process-get proc 'socks-reply) 1) socks-errors)))
+             (let ((err (process-get proc 'socks-reply)))
+               (if (eql version 5)
+                   (nth (or err 1) socks-errors)
+                 ;; The defined error codes for v4 range from
+                 ;; 90-93, but we store them in a simple list.
+                 (nth (pcase err (90 0) (92 2) (93 3) (_ 1))
+                      socks--errors-4)))))
     proc))
 
 
diff --git a/test/lisp/net/socks-tests.el b/test/lisp/net/socks-tests.el
index 0890ace826f..1a4bac37bf9 100644
--- a/test/lisp/net/socks-tests.el
+++ b/test/lisp/net/socks-tests.el
@@ -197,6 +197,7 @@ socks-tests-v4-basic
   "Show correct preparation of SOCKS4 connect command (Bug#46342)."
   (let ((socks-server '("server" "127.0.0.1" t 4))
         (url-user-agent "Test/4-basic")
+        (socks-username "foo")
         (socks-tests-canned-server-patterns
          `(([4 1 0 80 93 184 216 34 ?f ?o ?o 0] . [0 90 0 0 0 0 0 0])
            ,socks-tests--hello-world-http-request-pattern))
@@ -205,11 +206,35 @@ socks-tests-v4-basic
       (cl-letf (((symbol-function 'socks-nslookup-host)
                  (lambda (host)
                    (should (equal host "example.com"))
-                   (list 93 184 216 34)))
-                ((symbol-function 'user-full-name)
-                 (lambda (&optional _) "foo")))
+                   (list 93 184 216 34))))
         (socks-tests-perform-hello-world-http-request)))))
 
+(ert-deftest socks-tests-v4a-basic ()
+  "Show correct preparation of SOCKS4a connect command."
+  (let ((socks-server '("server" "127.0.0.1" t 4a))
+        (socks-username "foo")
+        (url-user-agent "Test/4a-basic")
+        (socks-tests-canned-server-patterns
+         `(([4 1 0 80 0 0 0 1 ?f ?o ?o 0 ?e ?x ?a ?m ?p ?l ?e ?. ?c ?o ?m 0]
+            . [0 90 0 0 0 0 0 0])
+           ,socks-tests--hello-world-http-request-pattern)))
+    (ert-info ("Make HTTP request over SOCKS4A")
+      (socks-tests-perform-hello-world-http-request))))
+
+(ert-deftest socks-tests-v4a-error ()
+  "Show error signaled when destination address rejected."
+  (let ((socks-server '("server" "127.0.0.1" t 4a))
+        (url-user-agent "Test/4a-basic")
+        (socks-username "")
+        (socks-tests-canned-server-patterns
+         `(([4 1 0 80 0 0 0 1 0 ?e ?x ?a ?m ?p ?l ?e ?. ?c ?o ?m 0]
+            . [0 91 0 0 0 0 0 0])
+           ,socks-tests--hello-world-http-request-pattern)))
+    (ert-info ("Make HTTP request over SOCKS4A")
+      (let ((err (should-error
+                  (socks-tests-perform-hello-world-http-request))))
+        (should (equal err '(error "SOCKS: Rejected or failed")))))))
+
 ;; Replace first pattern below with ([5 3 0 1 2] . [5 2]) to validate
 ;; against curl 7.71 with the following options:
 ;; $ curl --verbose -U foo:bar --proxy socks5h://127.0.0.1:10080 example.com
-- 
2.41.0


--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
 filename=0003-POC-Simplify-network-stream-openers-in-socks.el.patch

From f84edbf90bc249863ca6b8c28d90378ea4f22e9e Mon Sep 17 00:00:00 2001
From: "F. Jason Park" <jp@HIDDEN>
Date: Mon, 28 Nov 2022 22:31:50 -0800
Subject: [PATCH 3/4] [POC] Simplify network-stream openers in socks.el

* lisp/net/socks.el (socks-connect-function): New variable for
specifying an `open-network-stream'-like connect function.
(socks-open-connection): Accept additional `open-network-stream'
params passed on to opener, now `socks-connect-function',
in place of `open-network-stream'.
(socks-proxied-tls-services): Add new option for specifying ports
whose proxied connections should use TLS.
(socks--open-network-stream): Rework to serve as thin wrapper for
`socks-open-network-stream' that now hinges on rather than ignores the
variable `socks-override-functions'.
(socks-open-network-stream): Prefer parsed URL details, when present
in a non-nil `url-using-proxy', for improved compatibility with the gw
framework.
(socks--initiate-command-connect): New function to house renamed
latter half of the original `socks--open-network-stream'.  Role now
reduced to issuing the first command using an existing
process.  (Bug#53941)
---
 lisp/net/socks.el | 111 +++++++++++++++++++++++++++++++++++++---------
 1 file changed, 90 insertions(+), 21 deletions(-)

diff --git a/lisp/net/socks.el b/lisp/net/socks.el
index b781b6a4eab..f5820e7968c 100644
--- a/lisp/net/socks.el
+++ b/lisp/net/socks.el
@@ -34,7 +34,7 @@
 
 ;;; Code:
 
-(eval-when-compile (require 'cl-lib))
+(eval-when-compile (require 'cl-lib) (require 'url-parse))
 
 ;; ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
 ;; ;;; Custom widgets
@@ -335,14 +335,20 @@ socks-override-functions
 (when socks-override-functions
   (advice-add 'open-network-stream :around #'socks--open-network-stream))
 
-(defun socks-open-connection (server-info)
+(defvar socks-connect-function #'open-network-stream
+  "Function to open a network connection to a SOCKS provider.
+Called with arguments suitable for `open-network-stream'.")
+
+(defun socks-open-connection (server-info &rest stream-params)
+  "Create and initialize a SOCKS process.
+Perform authentication if needed.  Expect SERVER-INFO to resemble
+`socks-server' and STREAM-PARAMS to be keyword parameters
+accepted by `open-network-stream'."
   (save-excursion
     (let ((proc
            (let ((socks-override-functions nil))
-             (open-network-stream "socks"
-				  nil
-				  (nth 1 server-info)
-				  (nth 2 server-info))))
+             (apply socks-connect-function (nth 0 server-info) nil
+                    (nth 1 server-info) (nth 2 server-info) stream-params)))
 	  (authtype nil)
 	  version)
 
@@ -528,22 +534,85 @@ socks-find-services-entry
   (gethash (downcase service)
 	      (if udp socks-udp-services socks-tcp-services)))
 
-(defun socks-open-network-stream (name buffer host service)
-  (let ((socks-override-functions t))
-    (socks--open-network-stream
-     (lambda (&rest args)
-       (let ((socks-override-functions nil))
-         (apply #'open-network-stream args)))
-     name buffer host service)))
-
 (defun socks--open-network-stream (orig-fun name buffer host service &rest params)
-  (let ((route (and socks-override-functions
-                    (socks-find-route host service))))
-    (if (not route)
-	(apply orig-fun name buffer host service params)
-      ;; FIXME: Obey `params'!
-      (let* ((proc (socks-open-connection route))
-	     (version (process-get proc 'socks-server-protocol))
+  "Call `socks-open-network-stream', falling back to ORIG-FUN.
+Expect NAME, BUFFER, HOST, SERVICE, and PARAMS to be compatible
+with `open-network-stream'."
+  (let ((socks-connect-function orig-fun))
+    (apply (if socks-override-functions #'socks-open-network-stream orig-fun)
+           name buffer host service params)))
+
+(defcustom socks-proxied-tls-services '(443 6697)
+  "Ports whose connections should use TLS.
+Note that the system resolver may be consulted to look up host
+names for checking domain validation certs."
+  :version "30.1"
+  :type '(repeat number))
+
+(declare-function gnutls-negotiate "gnutls" (&rest rest))
+(declare-function nsm-verify-connection "nsm"
+                  (process host port &optional
+                           save-fingerprint warn-unencrypted))
+
+;;;###autoload
+(defun socks-open-network-stream (name buffer host service &rest params)
+  "Open and return a connection, possibly proxied over SOCKS.
+Expect PARAMS to contain keyword parameters recognized by
+`open-network-stream'.  Assume HOST and SERVICE refer to the
+proxied remote peer rather than the SOCKS server, but assume the
+opposite for PARAMS.  That is, if PARAMS contains a `:type' of
+`tls', treat the underlying connection to the proxy server as
+destined for encryption rather than the tunneled connection (even
+though `socks-connect-function' has the final say).  For TLS with
+proxied connections, see the option `socks-proxied-tls-services'.
+
+Before connecting, check the HOST against `socks-noproxy'.  On
+rejection, fall back to a non-SOCKS connection determined by
+the variable `socks-connect-function'.
+
+But, before doing anything, check if `url-using-proxy' is bound
+to a `url' struct object, as defined in `url-parse'.  If so,
+assume it represents the address of the desired SOCKS server
+rather than that of the remote peer, and use its fields instead
+of `socks-server' for all SOCKS connection details."
+  (require 'url-parse)
+  (let* ((url (and (url-p url-using-proxy)
+                   (string-prefix-p "socks" (url-type url-using-proxy))
+                   url-using-proxy))
+         (socks-server (if url
+                           (list name (url-host url) (url-port url)
+                                 (pcase (url-type url)
+                                   ("socks4" 4)
+                                   ("socks4a" '4a)
+                                   (_ 5)))
+                         socks-server))
+         (socks-username (or (and url (url-user url))
+                             socks-username))
+         (socks-password (or (and url (url-password url))
+                             socks-password)))
+    (if-let ((route (socks-find-route host service))
+             (proc (apply #'socks-open-connection route params)))
+        (let ((port (if (numberp service)
+                        service
+                      (process-contact proc :service)))
+              (certs (plist-get params :client-certificate)))
+          (socks--initiate-command-connect proc buffer host service)
+          (if (and (memq port socks-proxied-tls-services)
+                   (gnutls-available-p)
+                   (require 'gnutls nil t)
+                   (require 'nsm nil t))
+              (progn (gnutls-negotiate :process proc
+                                       :hostname host
+                                       :keylist (and certs (list certs)))
+                     (unless (string-suffix-p ".onion" host)
+                       (nsm-verify-connection proc host port))))
+          proc)
+      (apply socks-connect-function name buffer host service params))))
+
+(defun socks--initiate-command-connect (proc buffer host service)
+  (progn ; preserve indentation level for git blame / code review
+    (progn
+      (let* ((version (process-get proc 'socks-server-protocol))
              (atype
               (cond
                ((equal version 4)
-- 
2.41.0


--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
 filename=0004-POC-Integrate-the-socks-and-url-libraries.patch

From 6e0e98f0bc89a2c9a434c9a1e837750a371f6d1e Mon Sep 17 00:00:00 2001
From: "F. Jason Park" <jp@HIDDEN>
Date: Tue, 1 Mar 2022 01:38:33 -0800
Subject: [PATCH 4/4] [POC] Integrate the socks and url libraries

FIXME add tests, and mention in doc/misc/url.texi that some
`url-proxy-services' items can have full URLs, much like their env-var
counterparts.

* lisp/url/url-gw.el (url-open-stream): Use presence and type of
`url-using-proxy' to detect caller and massage input values according
to legacy practices.
* lisp/url/url-http.el: (url-http-find-free-connection): Don't call
`url-open-stream' with host and port from active proxy.
(url-http, url-http-async-sentinel): Only run
`url-https-proxy-connect' for http proxies.
* lisp/url/url-methods.el (url-scheme-register-proxy): When an
environment variable's value is a full URL, include the scheme in the
value of the new entry added to the `url-proxy-services' option if it
appears in the variable `url-proxy-full-address-types'.
* lisp/url/url-proxy.el (url-default-find-proxy-for-url): Preserve
`url-proxy-services' entries whose value is a URL containing a scheme
that appears in `url-proxy-full-address-type', and return that URL
prefixed by the upcased scheme.
(url-find-proxy-for-url): Recognize modified host/address value for
socks entries of `url-proxy-services' and deal accordingly.
(url-proxy): Handle a SOCKS proxy for http(s) connections only.
* lisp/url/url-vars.el (url-proxy-full-address-types): New variable to
specify types of URLs that should be preserved in full in the values
of `url-proxy-services' entries.
(url-proxy-services): Explain that values for
certain gateways may need a leading scheme:// portion.
(url-using-proxy): Add warning regarding expected type.
---
 lisp/url/url-gw.el      |  8 +++++++-
 lisp/url/url-http.el    | 19 ++++++++++---------
 lisp/url/url-methods.el |  8 +++++---
 lisp/url/url-proxy.el   | 22 +++++++++++++++-------
 lisp/url/url-vars.el    | 20 ++++++++++++++++++--
 5 files changed, 55 insertions(+), 22 deletions(-)

diff --git a/lisp/url/url-gw.el b/lisp/url/url-gw.el
index 568ce8679f5..a65245a58a3 100644
--- a/lisp/url/url-gw.el
+++ b/lisp/url/url-gw.el
@@ -28,7 +28,7 @@
 (require 'url-vars)
 (require 'url-parse)
 
-(autoload 'socks-open-network-stream "socks")
+(autoload 'socks-open-network-stream "socks") ; FIXME remove this
 
 (defgroup url-gateway nil
   "URL gateway variables."
@@ -226,6 +226,12 @@ url-open-stream
 Optional arg GATEWAY-METHOD specifies the gateway to be used,
 overriding the value of `url-gateway-method'."
   (unless url-gateway-unplugged
+    (when (url-p url-using-proxy)
+      (if (or (eq 'socks url-gateway-method)
+              (string-prefix-p "socks" (url-type url-using-proxy)))
+          (setq gateway-method 'socks)
+        (setq host (url-host url-using-proxy)
+              service (url-port url-using-proxy))))
     (let* ((gwm (or gateway-method url-gateway-method))
            (gw-method (if (and url-gateway-local-host-regexp
                                (not (eq 'tls gwm))
diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
index ada6341ee73..47c785a0735 100644
--- a/lisp/url/url-http.el
+++ b/lisp/url/url-http.el
@@ -195,12 +195,7 @@ url-http-find-free-connection
 	;; like authentication.  But we use another buffer afterwards.
 	(unwind-protect
             (let ((proc (url-open-stream host buf
-                                         (if url-using-proxy
-                                             (url-host url-using-proxy)
-                                           host)
-                                         (if url-using-proxy
-                                             (url-port url-using-proxy)
-                                           port)
+                                         host port
                                          gateway-method)))
 	      ;; url-open-stream might return nil.
 	      (when (processp proc)
@@ -1396,8 +1391,12 @@ url-http
            (error "Could not create connection to %s:%d" (url-host url)
                   (url-port url)))
           (_
-           (if (and url-http-proxy (string= "https"
-                                            (url-type url-current-object)))
+           (if (and url-http-proxy
+                    ;; Set to "http" by `url-find-proxy-for-url' for
+                    ;; any matching non-blacklisted, non-SOCKS scheme
+                    ;; in `url-proxy-services', including "https".
+                    (equal "http" (url-type url-http-proxy))
+                    (string= "https" (url-type url-current-object)))
                (url-https-proxy-connect connection)
              (set-process-sentinel connection
                                    #'url-http-end-of-document-sentinel)
@@ -1479,7 +1478,9 @@ url-http-async-sentinel
 	(url-http-end-of-document-sentinel proc why))
        ((string= (substring why 0 4) "open")
 	(setq url-http-connection-opened t)
-        (if (and url-http-proxy (string= "https" (url-type url-current-object)))
+        (if (and url-http-proxy
+                 (equal "http" (url-type url-http-proxy))
+                 (string= "https" (url-type url-current-object)))
             (url-https-proxy-connect proc)
           (condition-case error
               (process-send-string proc (url-http-create-request))
diff --git a/lisp/url/url-methods.el b/lisp/url/url-methods.el
index 9643e992044..9592307aea8 100644
--- a/lisp/url/url-methods.el
+++ b/lisp/url/url-methods.el
@@ -92,7 +92,6 @@ url-scheme-register-proxy
      ;; Then check if its a fully specified URL
      ((string-match url-nonrelative-link env-proxy)
       (setq urlobj (url-generic-parse-url env-proxy))
-      (setf (url-type urlobj) "http")
       (setf (url-target urlobj) nil))
      ;; Finally, fall back on the assumption that its just a hostname
      (t
@@ -103,8 +102,11 @@ url-scheme-register-proxy
      (if (and (not cur-proxy) urlobj)
 	 (progn
 	   (setq url-proxy-services
-		 (cons (cons scheme (format "%s:%d" (url-host urlobj)
-					    (url-port urlobj)))
+                 (cons (cons scheme (if (member (url-type urlobj)
+                                                url-proxy-full-address-types)
+                                        (url-recreate-url urlobj)
+                                      (format "%s:%d" (url-host urlobj)
+                                              (url-port urlobj))))
 		       url-proxy-services))
 	   (message "Using a proxy for %s..." scheme)))))
 
diff --git a/lisp/url/url-proxy.el b/lisp/url/url-proxy.el
index 0c330069789..b1583523cc6 100644
--- a/lisp/url/url-proxy.el
+++ b/lisp/url/url-proxy.el
@@ -34,11 +34,13 @@ url-default-find-proxy-for-url
 	      host))
 	(equal "www" (url-type urlobj)))
     "DIRECT")
-   ((cdr (assoc (url-type urlobj) url-proxy-services))
-    (concat "PROXY " (cdr (assoc (url-type urlobj) url-proxy-services))))
-   ;;
-   ;; Should check for socks
-   ;;
+   ((and-let* ((found (assoc (url-type urlobj) url-proxy-services)))
+      (concat (if-let ((non-scheme (string-search "://" (cdr found)))
+                       (scheme (substring (cdr found) 0 non-scheme))
+                       ((member scheme url-proxy-full-address-types)))
+                  (concat scheme " ")
+                "PROXY ")
+              (cdr found))))
    (t
     "DIRECT")))
 
@@ -56,8 +58,11 @@ url-find-proxy-for-url
      ((string-match "^DIRECT" proxy) nil)
      ((string-match "^PROXY +" proxy)
       (concat "http://" (substring proxy (match-end 0)) "/"))
-     ((string-match "^SOCKS +" proxy)
-      (concat "socks://" (substring proxy (match-end 0))))
+     ((string-match  (rx bot "SOCKS" (** 0 2 alnum) " ") proxy)
+      (if-let ((m (substring proxy (match-end 0)))
+               ((string-search "://" m)))
+          m
+        (concat "socks://" m)))
      (t
       (display-warning 'url (format "Unknown proxy directive: %s" proxy) :error)
       nil))))
@@ -72,6 +77,9 @@ url-proxy
   (cond
    ((string= (url-type url-using-proxy) "http")
     (url-http url callback cbargs))
+   ((and (string-prefix-p "socks" (url-type url-using-proxy))
+         (string-prefix-p "http" (url-type url)))
+    (url-http url callback cbargs))
    (t
     (error "Don't know how to use proxy `%s'" url-using-proxy))))
 
diff --git a/lisp/url/url-vars.el b/lisp/url/url-vars.el
index ef4b8b2841b..f10158d66a1 100644
--- a/lisp/url/url-vars.el
+++ b/lisp/url/url-vars.el
@@ -189,10 +189,24 @@ url-mail-command
   :type 'function
   :group 'url)
 
+(defvar url-proxy-full-address-types
+  '("socks" "socks5" "socks5h" "socks4" "socks4a")
+  "Schemes for URL types preserved in `url-proxy-services' entries.
+When dynamically adding a new `url-proxy-services' entry derived
+from the environment, Emacs only retains the host and port
+portions unless the URL's scheme appears in this variable's
+value.")
+
 (defcustom url-proxy-services nil
   "An alist of schemes and proxy servers that gateway them.
 Looks like ((\"http\" . \"hostname:portnumber\") ...).  This is set up
-from the ACCESS_proxy environment variables."
+from the ACCESS_proxy environment variables.  Certain gateway
+types need server values to take the form of full URLs in order
+to convey addtional information about for the proxy connection
+itself, for example, SCHEME://USER@HOSTNAME:PORTNUMBER, in which
+SCHEME is something like \"socks5\".  As of Emacs 30.1, this only
+applies to SCHEMEs appearing in the variable
+`url-proxy-full-address-types'."
   :type '(repeat (cons :format "%v"
 		       (string :tag "Protocol")
 		       (string :tag "Proxy")))
@@ -310,7 +324,9 @@ url-show-status
 
 (defvar url-using-proxy nil
   "Either nil or the fully qualified proxy URL in use, e.g.
-https://www.example.com/")
+https://www.example.com/.  Beware that some functions, such as
+`url-proxy' and `url-http-end-of-document-sentinel', set this to
+a `url' struct object.")
 
 (defcustom url-news-server nil
   "The default news server from which to get newsgroups/articles.
-- 
2.41.0


--=-=-=--

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#53941: 27.2; socks + tor dont work with https
Resent-From: "J.P." <jp@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Wed, 18 Oct 2023 13:41:01 +0000
Resent-Message-ID: <handler.53941.B53941.16976364121116 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 53941
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: patch
To: Stefan Kangas <stefankangas@HIDDEN>
Cc: Eli Zaretskii <eliz@HIDDEN>, 53941 <at> debbugs.gnu.org, larsi@HIDDEN, gnuhacker@HIDDEN
Received: via spool by 53941-submit <at> debbugs.gnu.org id=B53941.16976364121116
          (code B ref 53941); Wed, 18 Oct 2023 13:41:01 +0000
Received: (at 53941) by debbugs.gnu.org; 18 Oct 2023 13:40:12 +0000
Received: from localhost ([127.0.0.1]:33481 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qt6mR-0000Hu-72
	for submit <at> debbugs.gnu.org; Wed, 18 Oct 2023 09:40:12 -0400
Received: from mail-108-mta17.mxroute.com ([136.175.108.17]:40501)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <jp@HIDDEN>) id 1qt6lA-0000Dn-Kp
 for 53941 <at> debbugs.gnu.org; Wed, 18 Oct 2023 09:38:53 -0400
Received: from mail-111-mta2.mxroute.com ([136.175.111.2]
 filter006.mxroute.com) (Authenticated sender: mN4UYu2MZsgR)
 by mail-108-mta17.mxroute.com (ZoneMTA) with ESMTPSA id 18b43024b7c000ff68.001
 for <53941 <at> debbugs.gnu.org>
 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384);
 Wed, 18 Oct 2023 13:38:25 +0000
X-Zone-Loop: b06dd4282b10bbb450e84c53a236b5c7b58b2efa0e23
X-Originating-IP: [136.175.111.2]
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=neverwas.me
 ; s=x;
 h=Content-Type:MIME-Version:Message-ID:Date:References:In-Reply-To:
 Subject:Cc:To:From:Sender:Reply-To:Content-Transfer-Encoding:Content-ID:
 Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc
 :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe:
 List-Post:List-Owner:List-Archive;
 bh=5PJ5AR+aJdW5X9qldjleZkD9Ff7eB+0GYUqMW2djTy0=; b=XfrPya+4OhXd2mZhgMUnFEgGpE
 bQFTNCCvaq+d7ymzeGjy30JSaKn5xL5oDYlaRbYb965hssfIYDXLm4nNb63xNQQkpizPIxuIBvpyi
 Nia2TaZLMIp6YvVZBQI8eeJ7LUboB7y2uWeyCN/WNLYTWzD6PvWRmx7scpEk99BEhoWP8vXccZgwQ
 PrEtb2rvalJljQ251kOqJjBjZ1bysm1KGZ/gDIkpyXl48mfcRJ61aHwH4sez9oYLVmi9Cs/IjcwiX
 f/j9w45fZXdHYSoCq7m4MQuB06haEcM2nI/H+TuSNGoq9oN2DDelXlVH0CUAz4EerzwtBU819LxKB
 x961yh2Q==;
From: "J.P." <jp@HIDDEN>
In-Reply-To: <87il8lidbf.fsf@HIDDEN> (J. P.'s message of "Thu, 07 Sep
 2023 19:55:16 -0700")
References: <87pmntfym7.fsf@HIDDEN> <8735kl1v58.fsf@HIDDEN>
 <87a6emftzx.fsf@HIDDEN> <87k0do5km1.fsf@HIDDEN>
 <87pmn5n3tu.fsf@HIDDEN> <87mti99j1f.fsf@HIDDEN>
 <87wnh7hkgi.fsf@HIDDEN> <87pmmz947k.fsf@HIDDEN>
 <8735ju44sk.fsf@HIDDEN> <87lexikwu5.fsf@HIDDEN>
 <87mt8baygn.fsf_-_@HIDDEN> <8335a3nguk.fsf@HIDDEN>
 <87fse1kfe8.fsf@HIDDEN> <831qpln7zg.fsf@HIDDEN>
 <CADwFkmm0hCO3ek-w0TWkGWOROCU9YfOgDO0ivtcz-Q7feUE7nA@HIDDEN>
 <837cp21qca.fsf@HIDDEN> <878r9iktd0.fsf@HIDDEN>
 <CADwFkmnEG66bLtSrhWcL_w0PDng3EMPE_zyGftAQFVtFKNmg4A@HIDDEN>
 <87il8lidbf.fsf@HIDDEN>
Date: Wed, 18 Oct 2023 06:38:21 -0700
Message-ID: <87bkcwcape.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
Content-Type: text/plain
X-Authenticated-Id: masked@HIDDEN
X-Spam-Score: 0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

"J.P." <jp@HIDDEN> writes:

> Stefan Kangas <stefankangas@HIDDEN> writes:
>
>> "J.P." <jp@HIDDEN> writes:
>>
>>> As of now, the only patches I'd be comfortable offering would be the
>>> first two, which aren't even directly related to this bug.
>>
>> Thanks.  The first two patches do add tests as well, so I can see some
>> value in installing them separately, perhaps even right now.  Even more
>> so if it simplifies your work on the tasks you think are more important.
>
> OK, nice. I'd also like to add at least one test case that simulates a
> realistic error condition (and maybe also a NEWS item for v4a, if that's
> considered a feature). If no one else has thoughts regarding the first
> two, I'll install them in a few days.

s/days/fortnights/

(Apologies for the delay. Done now.)

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#53941: 27.2; socks + tor dont work with https
Resent-From: "J.P." <jp@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Tue, 19 Dec 2023 16:31:01 +0000
Resent-Message-ID: <handler.53941.B53941.170300340526546 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 53941
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: patch
To: 53941 <at> debbugs.gnu.org
Cc: larsi@HIDDEN, Eli Zaretskii <eliz@HIDDEN>, Stefan Kangas <stefankangas@HIDDEN>, gnuhacker@HIDDEN
Received: via spool by 53941-submit <at> debbugs.gnu.org id=B53941.170300340526546
          (code B ref 53941); Tue, 19 Dec 2023 16:31:01 +0000
Received: (at 53941) by debbugs.gnu.org; 19 Dec 2023 16:30:05 +0000
Received: from localhost ([127.0.0.1]:37138 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1rFcym-0006qN-F8
	for submit <at> debbugs.gnu.org; Tue, 19 Dec 2023 11:30:04 -0500
Received: from mail-108-mta43.mxroute.com ([136.175.108.43]:34333)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <jp@HIDDEN>) id 1rFcyi-0006qC-CR
 for 53941 <at> debbugs.gnu.org; Tue, 19 Dec 2023 11:29:59 -0500
Received: from filter006.mxroute.com ([136.175.111.2] filter006.mxroute.com)
 (Authenticated sender: mN4UYu2MZsgR)
 by mail-108-mta43.mxroute.com (ZoneMTA) with ESMTPSA id 18c82e9772000065b4.001
 for <53941 <at> debbugs.gnu.org>
 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384);
 Tue, 19 Dec 2023 16:29:47 +0000
X-Zone-Loop: fee1fb34c49c07022707737249a21a371e8d72eaf9ef
X-Originating-IP: [136.175.111.2]
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=neverwas.me
 ; s=x;
 h=Content-Type:MIME-Version:Message-ID:Date:References:In-Reply-To:
 Subject:Cc:To:From:Sender:Reply-To:Content-Transfer-Encoding:Content-ID:
 Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc
 :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe:
 List-Post:List-Owner:List-Archive;
 bh=xy6WvMa9cq8F3AHo2esHctLNUt0Utx8rgJ6Mz2NKT1Y=; b=FCX/r8JhyvPYrUlpgGkZPeH+dG
 Vtz2yOabaGD6VVnai0wEa/RcnGr9wGcV0pj6cXQo81llISA1BsXK+tma2wlOYeQwWog0yHBsHT5++
 WMNx/iaA7gmDI0SPOiv12GSH24zpfUeQ342AHL+J6NO4s2zx6967/aUHJy6oV0BS3k7Hov3+MuB0C
 8UEQmnUq45HuARRwLPkrI9Gq0Ksb2Dbbi1puzfZN3EM+7dDgIuDvJrGMiwJPk2di9/BUAMX4uHlM6
 HlYguuZtetir27f28oLdALloi/pnaItzDCuaCieavsMdsHvI+CLz/659NAVJAEia+EyBVeIW61+cr
 IhHWr3kA==;
From: "J.P." <jp@HIDDEN>
In-Reply-To: <87bkcwcape.fsf@HIDDEN> (J. P.'s message of "Wed, 18 Oct
 2023 06:38:21 -0700")
References: <87pmntfym7.fsf@HIDDEN> <8735kl1v58.fsf@HIDDEN>
 <87a6emftzx.fsf@HIDDEN> <87k0do5km1.fsf@HIDDEN>
 <87pmn5n3tu.fsf@HIDDEN> <87mti99j1f.fsf@HIDDEN>
 <87wnh7hkgi.fsf@HIDDEN> <87pmmz947k.fsf@HIDDEN>
 <8735ju44sk.fsf@HIDDEN> <87lexikwu5.fsf@HIDDEN>
 <87mt8baygn.fsf_-_@HIDDEN> <8335a3nguk.fsf@HIDDEN>
 <87fse1kfe8.fsf@HIDDEN> <831qpln7zg.fsf@HIDDEN>
 <CADwFkmm0hCO3ek-w0TWkGWOROCU9YfOgDO0ivtcz-Q7feUE7nA@HIDDEN>
 <837cp21qca.fsf@HIDDEN> <878r9iktd0.fsf@HIDDEN>
 <CADwFkmnEG66bLtSrhWcL_w0PDng3EMPE_zyGftAQFVtFKNmg4A@HIDDEN>
 <87il8lidbf.fsf@HIDDEN> <87bkcwcape.fsf@HIDDEN>
Date: Tue, 19 Dec 2023 08:29:43 -0800
Message-ID: <87zfy65f60.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="=-=-="
X-Authenticated-Id: masked@HIDDEN
X-Spam-Score: -0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

--=-=-=
Content-Type: text/plain

In light of recent related activity on the development mailing list [1],
I think it's worth summarizing where this bug stands, even though that
discussion seems to favor other approaches that I personally have little
interest in. As a reminder, this bug is about integrating the built-in
`socks' and `url' libraries to make proxying specific connections over
Tor mostly transparent and hassle free.

As it stands, the two biggest blockers I can see are both heavy on the
legwork side of things. The first and more daunting involves surveying
how the `url' library is used in the wild when it comes to proxies: what
interfaces people use and how that compares with what's prescribed in
the manual and what was promised in the original design discussions. The
goal would be to use these findings to enhance or overhaul the existing
interface and behavior and to affirm the result as canon. This will
likely involve reworking parts of the existing documentation and
saturating affected parts of the library in comprehensive test coverage.
It may also require central coordination to ensure interoperability
among consuming libraries and continuity of purpose among future
contributors.

The second and IMO easier (but still labor-intensive) task is fleshing
out and fortifying the actual networking implementation. What I propose
is that we start by leveraging the built-in logging facility of torsocks
itself [2]. By default, the verbose-level output is noisy but tolerable
after minimal modifications (e.g., by commenting out the "close" and
"fclose" statements). We'd need volunteers with access to the various
platforms Emacs supports (capable of running a Tor daemon) to run
through some contrived recipes, such as loading a site with `eww' and
fetching a package from an ELPA endpoint.

  $ LD_PRELOAD=/home/me/torsocks/src/lib/.libs/libtorsocks.so \
    TORSOCKS_LOG_FILE_PATH=/tmp/torsocks.log \
    TORSOCKS_LOG_LEVEL=5 ./src/emacs -Q

What we'd be looking for in the output is activity in those libc "GAI"
functions shadowed by the program.

While this methodology seems hokey, I think it's actually preferable (at
least as a starting point) over more traditional tracing solutions
because with the latter we'd still have to isolate the set of
torsocks-shadowed functions in whatever recording is produced, and then
filter out all the false positives from things like connect(2) calls for
AF_LOCAL/AF_UNIX, which we don't care about. Moreover, these
hand-crafted logs show us other niceties, like parameters of interest
(socket type, hint flags, etc.), which makes sense because this program
comes from the Tor Project itself and is well written. Anyway, once we
have a solid idea of what needs intercepting and/or inhibiting [3],
we'll need volunteers yet again, this time to help run packet traces
against a prototype both for the proxied connection and for DNS leakage.
Obviously, help from those familiar with the Emacs network stack would
go a long way here.

Anyway, the attached POC implementation is basically just a slightly
cleaned up version of the same patch set I've been fiddling with this
entire bug thread. WRT to the `url' side, I've done basically zero
additional research, and the interface you see is just a "best guess"
based on the current documentation and a cursory stroll around the
library. As for the `socks' side, everything is based on observations
from exactly one Emacs installation on one GNU/Linux machine. That it
doesn't leak DNS (for me) should just be a data point.

If others want to work on this or take over, please let me know. I
encourage anyone with a similar setup (GNU/Linux, x86_64, modern
libraries) to try them out. You'll need a Tor service running locally,
preferably with the default configuration. The API is pretty basic and
could be simplified even further. For now, it's

  (require 'socks)
  (require 'url)
  (setopt url-proxy-services '(("https" . "socks5h://localhost:9050")
                               ("http" . "socks5h://localhost:9050"))
          socks-server '("tor" "localhost" 9050 5)
          socks-username ""
          socks-password "")

then something like

  M-x eww RET https://check.torproject.org RET

or

  M-x list-packages RET.


[1] https://lists.gnu.org/archive/html/emacs-devel/2023-12/msg00570.html

[2] https://gitlab.torproject.org/tpo/core/torsocks.git

[3] Just to clarify a couple things I've personally found confusing,
    in case others may find them beneficial:

    If we only decide to support local proxies, we don't have to worry
    about DNS leakage for the proxy service itself. I've heard folks
    express concern about calls to getaddrinfo and, in particular, the
    async `:nowait' variant getaddrinfo_a (because torsocks AFAIK
    doesn't shadow it). But those won't come into play since our only
    connections through `make-network-process' are to the local proxy
    service. IOW, the actual remote lookup is tunneled, so the only
    calls we'll need to care about intercepting WRT DNS are those from
    `nsm-check' to `network-lookup-address-info'. (In the attached POC
    implementation, I simply reroute these through a tor-specific
    resolver.)

    The "onion cookie" bookkeeping business performed by torsocks is of
    no interest to us because we control the means of connection. IOW,
    at no point should an onion address (or that of a normal remote
    endpoint) be exposed to the underlying networking machinery. It's
    all just part of the opaque application data unit (payload).



--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment; filename=0000-v5-v6.diff

From a32e6d440e38b97090c9ae3fbf607ec71a49277e Mon Sep 17 00:00:00 2001
From: "F. Jason Park" <jp@HIDDEN>
Date: Tue, 19 Dec 2023 07:08:36 -0800
Subject: [PATCH 0/3] *** NOT A PATCH ***

*** BLURB HERE ***

F. Jason Park (3):
  [POC] Support SOCKS resolve extension
  [POC] Simplify network-stream openers in socks.el
  [POC] Integrate the socks and url libraries

 lisp/net/nsm.el              |   8 +-
 lisp/net/socks.el            | 189 +++++++++++++++++++++++++++++++----
 lisp/url/url-gw.el           |   8 +-
 lisp/url/url-http.el         |  19 ++--
 lisp/url/url-methods.el      |   8 +-
 lisp/url/url-proxy.el        |  22 ++--
 lisp/url/url-vars.el         |  20 +++-
 test/lisp/net/socks-tests.el |  70 +++++++++++++
 8 files changed, 300 insertions(+), 44 deletions(-)

Interdiff:
diff --git a/lisp/net/nsm.el b/lisp/net/nsm.el
index 09f7ac52537..234a7c5e74a 100644
--- a/lisp/net/nsm.el
+++ b/lisp/net/nsm.el
@@ -220,6 +220,10 @@ nsm-network-same-subnet
                                       (aref mask i))))))
         matches)))
 
+(defvar nsm--network-lookup-address-function nil
+  "Function to replace `network-lookup-address-info' in nsm check.
+It should have the same signature as the original.")
+
 (defun nsm-should-check (host)
   "Determine whether NSM should check for TLS problems for HOST.
 
@@ -227,7 +231,9 @@ nsm-should-check
 host address is a localhost address, or in the same subnet as one
 of the local interfaces, this function returns nil.  Non-nil
 otherwise."
-  (let ((addresses (network-lookup-address-info host))
+  (let ((addresses (if nsm--network-lookup-address-function
+                       (funcall nsm--network-lookup-address-function host)
+                     (network-lookup-address-info host)))
         (network-interface-list (network-interface-list t))
         (off-net t))
     (when
diff --git a/lisp/net/socks.el b/lisp/net/socks.el
index a04f93e0960..8d16db75834 100644
--- a/lisp/net/socks.el
+++ b/lisp/net/socks.el
@@ -181,6 +181,9 @@ socks-udp-associate-command
 (defconst socks-authentication-null 0)
 (defconst socks-authentication-failure 255)
 
+;; Extensions
+(defconst socks-resolve-command #xf0)
+
 ;; Response codes
 (defconst socks-response-success               0)
 (defconst socks-response-general-failure       1)
@@ -554,6 +557,9 @@ socks-proxied-tls-services
                   (process host port &optional
                            save-fingerprint warn-unencrypted))
 
+(defvar socks-server-name-as-tor-service-regexp (rx bow "tor" eow)
+  "Regexp to determine if a `socks-server' entry is TOR service.")
+
 ;;;###autoload
 (defun socks-open-network-stream (name buffer host service &rest params)
   "Open and return a connection, possibly proxied over SOCKS.
@@ -579,17 +585,18 @@ socks-open-network-stream
   (let* ((url (and (url-p url-using-proxy)
                    (string-prefix-p "socks" (url-type url-using-proxy))
                    url-using-proxy))
+         (server-name (and url (string= (nth 1 socks-server) (url-host url))
+                           (= (nth 2 socks-server) (url-port url))
+                           (car socks-server)))
          (socks-server (if url
-                           (list name (url-host url) (url-port url)
+                           (list server-name (url-host url) (url-port url)
                                  (pcase (url-type url)
                                    ("socks4" 4)
                                    ("socks4a" '4a)
                                    (_ 5)))
                          socks-server))
-         (socks-username (or (and url (url-user url))
-                             socks-username))
-         (socks-password (or (and url (url-password url))
-                             socks-password)))
+         (socks-username (or (and url (url-user url)) socks-username))
+         (socks-password (or (and url (url-password url)) socks-password)))
     (if-let ((route (socks-find-route host service))
              (proc (apply #'socks-open-connection route params)))
         (let ((port (if (numberp service)
@@ -597,15 +604,20 @@ socks-open-network-stream
                       (process-contact proc :service)))
               (certs (plist-get params :client-certificate)))
           (socks--initiate-command-connect proc buffer host service)
-          (if (and (memq port socks-proxied-tls-services)
-                   (gnutls-available-p)
-                   (require 'gnutls nil t)
-                   (require 'nsm nil t))
-              (progn (gnutls-negotiate :process proc
-                                       :hostname host
-                                       :keylist (and certs (list certs)))
-                     (unless (string-suffix-p ".onion" host)
-                       (nsm-verify-connection proc host port))))
+          (when (and (memq port socks-proxied-tls-services)
+                     (gnutls-available-p)
+                     (require 'gnutls nil t)
+                     (require 'nsm nil t))
+            (defvar nsm--network-lookup-address-function)
+            (let ((nsm--network-lookup-address-function
+                   (and (string-match socks-server-name-as-tor-service-regexp
+                                      (car socks-server))
+                        #'socks-tor-resolve)))
+              (gnutls-negotiate :process proc
+                                :hostname host
+                                :keylist (and certs (list certs)))
+              (unless (string-suffix-p ".onion" host)
+                (nsm-verify-connection proc host port))))
           proc)
       (apply socks-connect-function name buffer host service params))))
 
@@ -724,6 +736,72 @@ socks-nslookup-host
 	res)
     host))
 
+(defun socks--extract-resolve-response (proc)
+  "Parse response for PROC and maybe return destination IP address."
+  (when-let ((response (process-get proc 'socks-response)))
+    (pcase (process-get proc 'socks-server-protocol)
+      (4 ; https://www.openssh.com/txt/socks4a.protocol
+       (and-let* (((zerop (process-get proc 'socks-reply)))
+                  ((eq (aref response 1) 90)) ; #x5a request granted
+                  (a (substring response 4)) ; ignore port for now
+                  ((not (string-empty-p a)))
+                  ((not (string= a "\0\0\0\0"))))
+         a))
+      (5 ; https://tools.ietf.org/html/rfc1928
+       (cl-assert (eq 5 (aref response 0)) t)
+       (pcase (aref response 3) ; ATYP
+         (1 (and-let* ((a (substring response 4 8))
+                       ((not (string= a "\0\0\0\0")))
+                       a)))
+         ;; No reason to support RESOLVE_PTR [F1] extension, right?
+         (3 (let ((len (1- (aref response 4))))
+              (substring response 5 (+ 5 len))))
+         (4 (substring response 4 20)))))))
+
+(declare-function puny-encode-domain "puny" (domain))
+
+(defun socks--tor-resolve (name &optional _family _flags)
+  (condition-case err
+      (if-let ((socks-password (or socks-password ""))
+               (route (socks-find-route name nil))
+               (proc (socks-send-command (socks-open-connection route)
+                                         socks-resolve-command
+                                         socks-address-type-name
+                                         name
+                                         0))
+               (ip (prog1 (socks--extract-resolve-response proc)
+                     (delete-process proc))))
+          (list (vconcat ip [0]))
+        (error "Failed to resolve %s" name))
+    (error
+     (unless (member (cadr err)
+                     '("SOCKS: Host unreachable" "SOCKS: Rejected or failed"))
+       (signal (car err) (cdr err))))))
+
+(defvar socks--tor-resolve-cache nil)
+
+(defun socks-tor-resolve (name &optional _family _flags)
+  "Return list with a single IPv4 address for domain NAME.
+Return nil on failure.
+
+See `network-lookup-address-info' for format of return value.  As
+of 0.4.8.9, TOR's resolution service does not support IPv6.
+SOCKS server must support the Tor RESOLVE command.  Note that
+this function exists for novelty purposes only.  Using it in
+place of `network-lookup-address-info' or similar may not prevent
+DNS leaks."
+  (unless (string-match (rx bot (+ ascii) eot) name)
+    (require 'puny)
+    (setq name (puny-encode-domain name)))
+  ;; FIXME use some kind of LRU here.  Currently resets at 5 min.
+  (if socks--tor-resolve-cache
+      (when (time-less-p (car socks--tor-resolve-cache) (current-time))
+        (clrhash (cdr socks--tor-resolve-cache)))
+    (setq socks--tor-resolve-cache (cons (time-add (* 60 5) (current-time))
+                                         (make-hash-table :test #'equal))))
+  (with-memoization (gethash name (cdr socks--tor-resolve-cache))
+    (socks--tor-resolve name)))
+
 (provide 'socks)
 
 ;;; socks.el ends here
diff --git a/test/lisp/net/socks-tests.el b/test/lisp/net/socks-tests.el
index 1a4bac37bf9..cc9f5a385d2 100644
--- a/test/lisp/net/socks-tests.el
+++ b/test/lisp/net/socks-tests.el
@@ -327,4 +327,74 @@ socks-override-functions
   (should-not (advice-member-p #'socks--open-network-stream
                                'open-network-stream)))
 
+(ert-deftest tor-resolve-4a ()
+  "Make request to TOR resolve service over SOCKS4a"
+  (let* ((socks-server '("server" "127.0.0.1" t 4a))
+         (socks-username "foo") ; defaults to (user-login-name)
+         (socks-tests-canned-server-patterns
+          '(([4 #xf0 0 0 0 0 0 1 ?f ?o ?o 0 ?e ?x ?a ?m ?p ?l ?e ?. ?c ?o ?m 0]
+             . [0 90 0 0 93 184 216 34])))
+         (inhibit-message noninteractive)
+         (server (socks-tests-canned-server-create))
+         socks--tor-resolve-cache)
+    (ert-info ("Query TOR RESOLVE service over SOCKS4")
+      (cl-letf (((symbol-function 'user-full-name)
+                 (lambda (&optional _) "foo")))
+        (should (equal '([93 184 216 34 0])
+                       (socks-tor-resolve "example.com")))))
+    (kill-buffer (process-buffer server))
+    (delete-process server)))
+
+(ert-deftest tor-resolve-4a-fail ()
+  (let* ((socks-server '("server" "127.0.0.1" t 4a))
+         (socks-username "foo") ; defaults to (user-login-name)
+         (socks-tests-canned-server-patterns
+          '(([4 #xf0 0 0 0 0 0 1 ?f ?o ?o 0 ?e ?x ?a ?m ?p ?l ?e ?. ?c ?o ?m 0]
+             . [0 91 0 0 0 0 0 0])))
+         (inhibit-message noninteractive)
+         (server (socks-tests-canned-server-create))
+         socks--tor-resolve-cache)
+    (ert-info ("Query TOR RESOLVE service over SOCKS4")
+      (cl-letf (((symbol-function 'user-full-name)
+                 (lambda (&optional _) "foo")))
+        (should-not (socks-tor-resolve "example.com"))))
+    (kill-buffer (process-buffer server))
+    (delete-process server)))
+
+(ert-deftest tor-resolve-5-fail ()
+  (let* ((socks-server '("server" "127.0.0.1" t 5))
+         (socks-username "")
+         (socks-authentication-methods (copy-sequence
+                                        socks-authentication-methods))
+         (inhibit-message noninteractive)
+         (socks-tests-canned-server-patterns
+          '(([5 2 0 2] . [5 2])
+            ([1 0 0] . [1 0])
+            ([5 #xf0 0 3 11 ?e ?x ?a ?m ?p ?l ?e ?. ?c ?o ?m 0 0]
+             . [5 4 0 0 0 0 0 0 0 0])))
+         (server (socks-tests-canned-server-create)))
+    (ert-info ("Query TOR RESOLVE service over SOCKS5")
+      (should-not (socks-tor-resolve "example.com")))
+    (kill-buffer (process-buffer server))
+    (delete-process server)))
+
+(ert-deftest tor-resolve-5 ()
+  "Make request to TOR resolve service over SOCKS5"
+  (let* ((socks-server '("server" "127.0.0.1" t 5))
+         (socks-username "foo")
+         (socks-authentication-methods (append socks-authentication-methods
+                                               nil))
+         (inhibit-message noninteractive)
+         (socks-tests-canned-server-patterns
+          '(([5 2 0 2] . [5 2])
+            ([1 3 ?f ?o ?o 0] . [1 0])
+            ([5 #xf0 0 3 11 ?e ?x ?a ?m ?p ?l ?e ?. ?c ?o ?m 0 0]
+             . [5 0 0 1 93 184 216 34 0 0])))
+         (server (socks-tests-canned-server-create))
+         socks--tor-resolve-cache)
+    (ert-info ("Query TOR RESOLVE service over SOCKS5")
+      (should (equal '([93 184 216 34 0]) (socks-tor-resolve "example.com"))))
+    (kill-buffer (process-buffer server))
+    (delete-process server)))
+
 ;;; socks-tests.el ends here
-- 
2.42.0


--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
 filename=0001-POC-Support-SOCKS-resolve-extension.patch

From d7cca6703475b8bcdf523407383a813ec3749fad Mon Sep 17 00:00:00 2001
From: "F. Jason Park" <jp@HIDDEN>
Date: Mon, 14 Feb 2022 02:36:57 -0800
Subject: [PATCH 1/3] [POC] Support SOCKS resolve extension

* lisp/net/socks.el (socks-resolve-command): Add new constant for the
RESOLVE command, a nonstandard SOCKS extension from the Tor project.
It mirrors CONNECT in most respects but asks the server to RESOLVE a
host name and return its IP.  For details, see
doc/socks/socks-extensions.txt in the source tree for torsocks.  This
shouldn't be confused with 5h/5-hostname, which is used to by clients
like cURL to allow users to bypass attempts to resolve a name locally.
(socks--extract-resolve-response, socks-tor-resolve): Add utility
functions to query a SOCKS service supporting the RESOLVE extension.
* test/lisp/net/socks-tests.el (tor-resolve-4a, tor-resolve-4a-fail,
tor-resolve-5-fail, tor-resolve-5):  (Bug#53941.)
---
 lisp/net/socks.el            | 69 +++++++++++++++++++++++++++++++++++
 test/lisp/net/socks-tests.el | 70 ++++++++++++++++++++++++++++++++++++
 2 files changed, 139 insertions(+)

diff --git a/lisp/net/socks.el b/lisp/net/socks.el
index e572e5c9bdf..4c4eb8cf751 100644
--- a/lisp/net/socks.el
+++ b/lisp/net/socks.el
@@ -181,6 +181,9 @@ socks-udp-associate-command
 (defconst socks-authentication-null 0)
 (defconst socks-authentication-failure 255)
 
+;; Extensions
+(defconst socks-resolve-command #xf0)
+
 ;; Response codes
 (defconst socks-response-success               0)
 (defconst socks-response-general-failure       1)
@@ -655,6 +658,72 @@ socks-nslookup-host
 	res)
     host))
 
+(defun socks--extract-resolve-response (proc)
+  "Parse response for PROC and maybe return destination IP address."
+  (when-let ((response (process-get proc 'socks-response)))
+    (pcase (process-get proc 'socks-server-protocol)
+      (4 ; https://www.openssh.com/txt/socks4a.protocol
+       (and-let* (((zerop (process-get proc 'socks-reply)))
+                  ((eq (aref response 1) 90)) ; #x5a request granted
+                  (a (substring response 4)) ; ignore port for now
+                  ((not (string-empty-p a)))
+                  ((not (string= a "\0\0\0\0"))))
+         a))
+      (5 ; https://tools.ietf.org/html/rfc1928
+       (cl-assert (eq 5 (aref response 0)) t)
+       (pcase (aref response 3) ; ATYP
+         (1 (and-let* ((a (substring response 4 8))
+                       ((not (string= a "\0\0\0\0")))
+                       a)))
+         ;; No reason to support RESOLVE_PTR [F1] extension, right?
+         (3 (let ((len (1- (aref response 4))))
+              (substring response 5 (+ 5 len))))
+         (4 (substring response 4 20)))))))
+
+(declare-function puny-encode-domain "puny" (domain))
+
+(defun socks--tor-resolve (name &optional _family _flags)
+  (condition-case err
+      (if-let ((socks-password (or socks-password ""))
+               (route (socks-find-route name nil))
+               (proc (socks-send-command (socks-open-connection route)
+                                         socks-resolve-command
+                                         socks-address-type-name
+                                         name
+                                         0))
+               (ip (prog1 (socks--extract-resolve-response proc)
+                     (delete-process proc))))
+          (list (vconcat ip [0]))
+        (error "Failed to resolve %s" name))
+    (error
+     (unless (member (cadr err)
+                     '("SOCKS: Host unreachable" "SOCKS: Rejected or failed"))
+       (signal (car err) (cdr err))))))
+
+(defvar socks--tor-resolve-cache nil)
+
+(defun socks-tor-resolve (name &optional _family _flags)
+  "Return list with a single IPv4 address for domain NAME.
+Return nil on failure.
+
+See `network-lookup-address-info' for format of return value.  As
+of 0.4.8.9, TOR's resolution service does not support IPv6.
+SOCKS server must support the Tor RESOLVE command.  Note that
+this function exists for novelty purposes only.  Using it in
+place of `network-lookup-address-info' or similar may not prevent
+DNS leaks."
+  (unless (string-match (rx bot (+ ascii) eot) name)
+    (require 'puny)
+    (setq name (puny-encode-domain name)))
+  ;; FIXME use some kind of LRU here.  Currently resets at 5 min.
+  (if socks--tor-resolve-cache
+      (when (time-less-p (car socks--tor-resolve-cache) (current-time))
+        (clrhash (cdr socks--tor-resolve-cache)))
+    (setq socks--tor-resolve-cache (cons (time-add (* 60 5) (current-time))
+                                         (make-hash-table :test #'equal))))
+  (with-memoization (gethash name (cdr socks--tor-resolve-cache))
+    (socks--tor-resolve name)))
+
 (provide 'socks)
 
 ;;; socks.el ends here
diff --git a/test/lisp/net/socks-tests.el b/test/lisp/net/socks-tests.el
index 1a4bac37bf9..cc9f5a385d2 100644
--- a/test/lisp/net/socks-tests.el
+++ b/test/lisp/net/socks-tests.el
@@ -327,4 +327,74 @@ socks-override-functions
   (should-not (advice-member-p #'socks--open-network-stream
                                'open-network-stream)))
 
+(ert-deftest tor-resolve-4a ()
+  "Make request to TOR resolve service over SOCKS4a"
+  (let* ((socks-server '("server" "127.0.0.1" t 4a))
+         (socks-username "foo") ; defaults to (user-login-name)
+         (socks-tests-canned-server-patterns
+          '(([4 #xf0 0 0 0 0 0 1 ?f ?o ?o 0 ?e ?x ?a ?m ?p ?l ?e ?. ?c ?o ?m 0]
+             . [0 90 0 0 93 184 216 34])))
+         (inhibit-message noninteractive)
+         (server (socks-tests-canned-server-create))
+         socks--tor-resolve-cache)
+    (ert-info ("Query TOR RESOLVE service over SOCKS4")
+      (cl-letf (((symbol-function 'user-full-name)
+                 (lambda (&optional _) "foo")))
+        (should (equal '([93 184 216 34 0])
+                       (socks-tor-resolve "example.com")))))
+    (kill-buffer (process-buffer server))
+    (delete-process server)))
+
+(ert-deftest tor-resolve-4a-fail ()
+  (let* ((socks-server '("server" "127.0.0.1" t 4a))
+         (socks-username "foo") ; defaults to (user-login-name)
+         (socks-tests-canned-server-patterns
+          '(([4 #xf0 0 0 0 0 0 1 ?f ?o ?o 0 ?e ?x ?a ?m ?p ?l ?e ?. ?c ?o ?m 0]
+             . [0 91 0 0 0 0 0 0])))
+         (inhibit-message noninteractive)
+         (server (socks-tests-canned-server-create))
+         socks--tor-resolve-cache)
+    (ert-info ("Query TOR RESOLVE service over SOCKS4")
+      (cl-letf (((symbol-function 'user-full-name)
+                 (lambda (&optional _) "foo")))
+        (should-not (socks-tor-resolve "example.com"))))
+    (kill-buffer (process-buffer server))
+    (delete-process server)))
+
+(ert-deftest tor-resolve-5-fail ()
+  (let* ((socks-server '("server" "127.0.0.1" t 5))
+         (socks-username "")
+         (socks-authentication-methods (copy-sequence
+                                        socks-authentication-methods))
+         (inhibit-message noninteractive)
+         (socks-tests-canned-server-patterns
+          '(([5 2 0 2] . [5 2])
+            ([1 0 0] . [1 0])
+            ([5 #xf0 0 3 11 ?e ?x ?a ?m ?p ?l ?e ?. ?c ?o ?m 0 0]
+             . [5 4 0 0 0 0 0 0 0 0])))
+         (server (socks-tests-canned-server-create)))
+    (ert-info ("Query TOR RESOLVE service over SOCKS5")
+      (should-not (socks-tor-resolve "example.com")))
+    (kill-buffer (process-buffer server))
+    (delete-process server)))
+
+(ert-deftest tor-resolve-5 ()
+  "Make request to TOR resolve service over SOCKS5"
+  (let* ((socks-server '("server" "127.0.0.1" t 5))
+         (socks-username "foo")
+         (socks-authentication-methods (append socks-authentication-methods
+                                               nil))
+         (inhibit-message noninteractive)
+         (socks-tests-canned-server-patterns
+          '(([5 2 0 2] . [5 2])
+            ([1 3 ?f ?o ?o 0] . [1 0])
+            ([5 #xf0 0 3 11 ?e ?x ?a ?m ?p ?l ?e ?. ?c ?o ?m 0 0]
+             . [5 0 0 1 93 184 216 34 0 0])))
+         (server (socks-tests-canned-server-create))
+         socks--tor-resolve-cache)
+    (ert-info ("Query TOR RESOLVE service over SOCKS5")
+      (should (equal '([93 184 216 34 0]) (socks-tor-resolve "example.com"))))
+    (kill-buffer (process-buffer server))
+    (delete-process server)))
+
 ;;; socks-tests.el ends here
-- 
2.42.0


--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
 filename=0002-POC-Simplify-network-stream-openers-in-socks.el.patch

From dd72b020f3ceca191f2d18d16253501d16ed582b Mon Sep 17 00:00:00 2001
From: "F. Jason Park" <jp@HIDDEN>
Date: Mon, 28 Nov 2022 22:31:50 -0800
Subject: [PATCH 2/3] [POC] Simplify network-stream openers in socks.el

* lisp/net/nsm.el (nsm--network-lookup-address-function):
New function-valued variable to override `network-lookup-address-info'
during nsm checks.
(nsm-should-check): Defer to `nsm--network-lookup-address-function'
to resolve hosts when non-nil.
* lisp/net/socks.el (socks-connect-function): New variable for
specifying an `open-network-stream'-like connect function.
(socks-open-connection): Accept additional `open-network-stream'
params passed on to opener, now `socks-connect-function',
in place of `open-network-stream'.
(socks-proxied-tls-services): Add new option for specifying ports
whose proxied connections should use TLS.
(socks--open-network-stream): Rework to serve as thin wrapper for
`socks-open-network-stream' that now hinges on rather than ignores the
variable `socks-override-functions'.
(socks-open-network-stream): Prefer parsed URL details, when present
in a non-nil `url-using-proxy', for improved compatibility with the gw
framework.
(socks--initiate-command-connect): New function to house renamed
latter half of the original `socks--open-network-stream'.  Role now
reduced to issuing the first command using an existing
process.  (Bug#53941)
---
 lisp/net/nsm.el   |   8 +++-
 lisp/net/socks.el | 120 ++++++++++++++++++++++++++++++++++++++--------
 2 files changed, 106 insertions(+), 22 deletions(-)

diff --git a/lisp/net/nsm.el b/lisp/net/nsm.el
index 09f7ac52537..234a7c5e74a 100644
--- a/lisp/net/nsm.el
+++ b/lisp/net/nsm.el
@@ -220,6 +220,10 @@ nsm-network-same-subnet
                                       (aref mask i))))))
         matches)))
 
+(defvar nsm--network-lookup-address-function nil
+  "Function to replace `network-lookup-address-info' in nsm check.
+It should have the same signature as the original.")
+
 (defun nsm-should-check (host)
   "Determine whether NSM should check for TLS problems for HOST.
 
@@ -227,7 +231,9 @@ nsm-should-check
 host address is a localhost address, or in the same subnet as one
 of the local interfaces, this function returns nil.  Non-nil
 otherwise."
-  (let ((addresses (network-lookup-address-info host))
+  (let ((addresses (if nsm--network-lookup-address-function
+                       (funcall nsm--network-lookup-address-function host)
+                     (network-lookup-address-info host)))
         (network-interface-list (network-interface-list t))
         (off-net t))
     (when
diff --git a/lisp/net/socks.el b/lisp/net/socks.el
index 4c4eb8cf751..8d16db75834 100644
--- a/lisp/net/socks.el
+++ b/lisp/net/socks.el
@@ -34,7 +34,7 @@
 
 ;;; Code:
 
-(eval-when-compile (require 'cl-lib))
+(eval-when-compile (require 'cl-lib) (require 'url-parse))
 
 ;; ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
 ;; ;;; Custom widgets
@@ -338,14 +338,20 @@ socks-override-functions
 (when socks-override-functions
   (advice-add 'open-network-stream :around #'socks--open-network-stream))
 
-(defun socks-open-connection (server-info)
+(defvar socks-connect-function #'open-network-stream
+  "Function to open a network connection to a SOCKS provider.
+Called with arguments suitable for `open-network-stream'.")
+
+(defun socks-open-connection (server-info &rest stream-params)
+  "Create and initialize a SOCKS process.
+Perform authentication if needed.  Expect SERVER-INFO to resemble
+`socks-server' and STREAM-PARAMS to be keyword parameters
+accepted by `open-network-stream'."
   (save-excursion
     (let ((proc
            (let ((socks-override-functions nil))
-             (open-network-stream "socks"
-				  nil
-				  (nth 1 server-info)
-				  (nth 2 server-info))))
+             (apply socks-connect-function (nth 0 server-info) nil
+                    (nth 1 server-info) (nth 2 server-info) stream-params)))
 	  (authtype nil)
 	  version)
 
@@ -531,22 +537,94 @@ socks-find-services-entry
   (gethash (downcase service)
 	      (if udp socks-udp-services socks-tcp-services)))
 
-(defun socks-open-network-stream (name buffer host service)
-  (let ((socks-override-functions t))
-    (socks--open-network-stream
-     (lambda (&rest args)
-       (let ((socks-override-functions nil))
-         (apply #'open-network-stream args)))
-     name buffer host service)))
-
 (defun socks--open-network-stream (orig-fun name buffer host service &rest params)
-  (let ((route (and socks-override-functions
-                    (socks-find-route host service))))
-    (if (not route)
-	(apply orig-fun name buffer host service params)
-      ;; FIXME: Obey `params'!
-      (let* ((proc (socks-open-connection route))
-	     (version (process-get proc 'socks-server-protocol))
+  "Call `socks-open-network-stream', falling back to ORIG-FUN.
+Expect NAME, BUFFER, HOST, SERVICE, and PARAMS to be compatible
+with `open-network-stream'."
+  (let ((socks-connect-function orig-fun))
+    (apply (if socks-override-functions #'socks-open-network-stream orig-fun)
+           name buffer host service params)))
+
+(defcustom socks-proxied-tls-services '(443 6697)
+  "Ports whose connections should use TLS.
+Note that the system resolver may be consulted to look up host
+names for checking domain validation certs."
+  :version "30.1"
+  :type '(repeat number))
+
+(declare-function gnutls-negotiate "gnutls" (&rest rest))
+(declare-function nsm-verify-connection "nsm"
+                  (process host port &optional
+                           save-fingerprint warn-unencrypted))
+
+(defvar socks-server-name-as-tor-service-regexp (rx bow "tor" eow)
+  "Regexp to determine if a `socks-server' entry is TOR service.")
+
+;;;###autoload
+(defun socks-open-network-stream (name buffer host service &rest params)
+  "Open and return a connection, possibly proxied over SOCKS.
+Expect PARAMS to contain keyword parameters recognized by
+`open-network-stream'.  Assume HOST and SERVICE refer to the
+proxied remote peer rather than the SOCKS server, but assume the
+opposite for PARAMS.  That is, if PARAMS contains a `:type' of
+`tls', treat the underlying connection to the proxy server as
+destined for encryption rather than the tunneled connection (even
+though `socks-connect-function' has the final say).  For TLS with
+proxied connections, see the option `socks-proxied-tls-services'.
+
+Before connecting, check the HOST against `socks-noproxy'.  On
+rejection, fall back to a non-SOCKS connection determined by
+the variable `socks-connect-function'.
+
+But, before doing anything, check if `url-using-proxy' is bound
+to a `url' struct object, as defined in `url-parse'.  If so,
+assume it represents the address of the desired SOCKS server
+rather than that of the remote peer, and use its fields instead
+of `socks-server' for all SOCKS connection details."
+  (require 'url-parse)
+  (let* ((url (and (url-p url-using-proxy)
+                   (string-prefix-p "socks" (url-type url-using-proxy))
+                   url-using-proxy))
+         (server-name (and url (string= (nth 1 socks-server) (url-host url))
+                           (= (nth 2 socks-server) (url-port url))
+                           (car socks-server)))
+         (socks-server (if url
+                           (list server-name (url-host url) (url-port url)
+                                 (pcase (url-type url)
+                                   ("socks4" 4)
+                                   ("socks4a" '4a)
+                                   (_ 5)))
+                         socks-server))
+         (socks-username (or (and url (url-user url)) socks-username))
+         (socks-password (or (and url (url-password url)) socks-password)))
+    (if-let ((route (socks-find-route host service))
+             (proc (apply #'socks-open-connection route params)))
+        (let ((port (if (numberp service)
+                        service
+                      (process-contact proc :service)))
+              (certs (plist-get params :client-certificate)))
+          (socks--initiate-command-connect proc buffer host service)
+          (when (and (memq port socks-proxied-tls-services)
+                     (gnutls-available-p)
+                     (require 'gnutls nil t)
+                     (require 'nsm nil t))
+            (defvar nsm--network-lookup-address-function)
+            (let ((nsm--network-lookup-address-function
+                   (and (string-match socks-server-name-as-tor-service-regexp
+                                      (car socks-server))
+                        #'socks-tor-resolve)))
+              (gnutls-negotiate :process proc
+                                :hostname host
+                                :keylist (and certs (list certs)))
+              (unless (string-suffix-p ".onion" host)
+                (nsm-verify-connection proc host port))))
+          proc)
+      (apply socks-connect-function name buffer host service params))))
+
+(defun socks--initiate-command-connect (proc buffer host service)
+  (progn ; preserve indentation level for git blame / code review
+    (progn
+      (let* ((version (process-get proc 'socks-server-protocol))
              (atype
               (cond
                ((equal version 4)
-- 
2.42.0


--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
 filename=0003-POC-Integrate-the-socks-and-url-libraries.patch

From a32e6d440e38b97090c9ae3fbf607ec71a49277e Mon Sep 17 00:00:00 2001
From: "F. Jason Park" <jp@HIDDEN>
Date: Tue, 1 Mar 2022 01:38:33 -0800
Subject: [PATCH 3/3] [POC] Integrate the socks and url libraries

FIXME add tests, and mention in doc/misc/url.texi that some
`url-proxy-services' items can have full URLs, much like their env-var
counterparts. Current API (only known to work on GNU/Linux):

  (require 'socks)
  (require 'url)
  (setopt url-proxy-services '(("https" . "socks5h://localhost:9050")
                               ("http" . "socks5h://localhost:9050"))
          socks-server '("tor" "localhost" 9050 5)
          socks-username ""
          socks-password "")

Then do something like

  M-x eww https://check.torproject.org RET

or

  M-x list-packages RET.

* lisp/url/url-gw.el (url-open-stream): Use presence and type of
`url-using-proxy' to detect caller and massage input values according
to legacy practices.
* lisp/url/url-http.el: (url-http-find-free-connection): Don't call
`url-open-stream' with host and port from active proxy.
(url-http, url-http-async-sentinel): Only run
`url-https-proxy-connect' for http proxies.
* lisp/url/url-methods.el (url-scheme-register-proxy): When an
environment variable's value is a full URL, include the scheme in the
value of the new entry added to the `url-proxy-services' option if it
appears in the variable `url-proxy-full-address-types'.
* lisp/url/url-proxy.el (url-default-find-proxy-for-url): Preserve
`url-proxy-services' entries whose value is a URL containing a scheme
that appears in `url-proxy-full-address-type', and return that URL
prefixed by the upcased scheme.
(url-find-proxy-for-url): Recognize modified host/address value for
socks entries of `url-proxy-services' and deal accordingly.
(url-proxy): Handle a SOCKS proxy for http(s) connections only.
* lisp/url/url-vars.el (url-proxy-full-address-types): New variable to
specify types of URLs that should be preserved in full in the values
of `url-proxy-services' entries.
(url-proxy-services): Explain that values for
certain gateways may need a leading scheme:// portion.
(url-using-proxy): Add warning regarding expected type.
---
 lisp/url/url-gw.el      |  8 +++++++-
 lisp/url/url-http.el    | 19 ++++++++++---------
 lisp/url/url-methods.el |  8 +++++---
 lisp/url/url-proxy.el   | 22 +++++++++++++++-------
 lisp/url/url-vars.el    | 20 ++++++++++++++++++--
 5 files changed, 55 insertions(+), 22 deletions(-)

diff --git a/lisp/url/url-gw.el b/lisp/url/url-gw.el
index 568ce8679f5..a65245a58a3 100644
--- a/lisp/url/url-gw.el
+++ b/lisp/url/url-gw.el
@@ -28,7 +28,7 @@
 (require 'url-vars)
 (require 'url-parse)
 
-(autoload 'socks-open-network-stream "socks")
+(autoload 'socks-open-network-stream "socks") ; FIXME remove this
 
 (defgroup url-gateway nil
   "URL gateway variables."
@@ -226,6 +226,12 @@ url-open-stream
 Optional arg GATEWAY-METHOD specifies the gateway to be used,
 overriding the value of `url-gateway-method'."
   (unless url-gateway-unplugged
+    (when (url-p url-using-proxy)
+      (if (or (eq 'socks url-gateway-method)
+              (string-prefix-p "socks" (url-type url-using-proxy)))
+          (setq gateway-method 'socks)
+        (setq host (url-host url-using-proxy)
+              service (url-port url-using-proxy))))
     (let* ((gwm (or gateway-method url-gateway-method))
            (gw-method (if (and url-gateway-local-host-regexp
                                (not (eq 'tls gwm))
diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
index 947c6517ed1..abe2649a474 100644
--- a/lisp/url/url-http.el
+++ b/lisp/url/url-http.el
@@ -195,12 +195,7 @@ url-http-find-free-connection
 	;; like authentication.  But we use another buffer afterwards.
 	(unwind-protect
             (let ((proc (url-open-stream host buf
-                                         (if url-using-proxy
-                                             (url-host url-using-proxy)
-                                           host)
-                                         (if url-using-proxy
-                                             (url-port url-using-proxy)
-                                           port)
+                                         host port
                                          gateway-method)))
 	      ;; url-open-stream might return nil.
 	      (when (processp proc)
@@ -1392,8 +1387,12 @@ url-http
            (error "Could not create connection to %s:%d" (url-host url)
                   (url-port url)))
           (_
-           (if (and url-http-proxy (string= "https"
-                                            (url-type url-current-object)))
+           (if (and url-http-proxy
+                    ;; Set to "http" by `url-find-proxy-for-url' for
+                    ;; any matching non-blacklisted, non-SOCKS scheme
+                    ;; in `url-proxy-services', including "https".
+                    (equal "http" (url-type url-http-proxy))
+                    (string= "https" (url-type url-current-object)))
                (url-https-proxy-connect connection)
              (set-process-sentinel connection
                                    #'url-http-end-of-document-sentinel)
@@ -1475,7 +1474,9 @@ url-http-async-sentinel
 	(url-http-end-of-document-sentinel proc why))
        ((string= (substring why 0 4) "open")
 	(setq url-http-connection-opened t)
-        (if (and url-http-proxy (string= "https" (url-type url-current-object)))
+        (if (and url-http-proxy
+                 (equal "http" (url-type url-http-proxy))
+                 (string= "https" (url-type url-current-object)))
             (url-https-proxy-connect proc)
           (condition-case error
               (process-send-string proc (url-http-create-request))
diff --git a/lisp/url/url-methods.el b/lisp/url/url-methods.el
index 9643e992044..9592307aea8 100644
--- a/lisp/url/url-methods.el
+++ b/lisp/url/url-methods.el
@@ -92,7 +92,6 @@ url-scheme-register-proxy
      ;; Then check if its a fully specified URL
      ((string-match url-nonrelative-link env-proxy)
       (setq urlobj (url-generic-parse-url env-proxy))
-      (setf (url-type urlobj) "http")
       (setf (url-target urlobj) nil))
      ;; Finally, fall back on the assumption that its just a hostname
      (t
@@ -103,8 +102,11 @@ url-scheme-register-proxy
      (if (and (not cur-proxy) urlobj)
 	 (progn
 	   (setq url-proxy-services
-		 (cons (cons scheme (format "%s:%d" (url-host urlobj)
-					    (url-port urlobj)))
+                 (cons (cons scheme (if (member (url-type urlobj)
+                                                url-proxy-full-address-types)
+                                        (url-recreate-url urlobj)
+                                      (format "%s:%d" (url-host urlobj)
+                                              (url-port urlobj))))
 		       url-proxy-services))
 	   (message "Using a proxy for %s..." scheme)))))
 
diff --git a/lisp/url/url-proxy.el b/lisp/url/url-proxy.el
index 0c330069789..b1583523cc6 100644
--- a/lisp/url/url-proxy.el
+++ b/lisp/url/url-proxy.el
@@ -34,11 +34,13 @@ url-default-find-proxy-for-url
 	      host))
 	(equal "www" (url-type urlobj)))
     "DIRECT")
-   ((cdr (assoc (url-type urlobj) url-proxy-services))
-    (concat "PROXY " (cdr (assoc (url-type urlobj) url-proxy-services))))
-   ;;
-   ;; Should check for socks
-   ;;
+   ((and-let* ((found (assoc (url-type urlobj) url-proxy-services)))
+      (concat (if-let ((non-scheme (string-search "://" (cdr found)))
+                       (scheme (substring (cdr found) 0 non-scheme))
+                       ((member scheme url-proxy-full-address-types)))
+                  (concat scheme " ")
+                "PROXY ")
+              (cdr found))))
    (t
     "DIRECT")))
 
@@ -56,8 +58,11 @@ url-find-proxy-for-url
      ((string-match "^DIRECT" proxy) nil)
      ((string-match "^PROXY +" proxy)
       (concat "http://" (substring proxy (match-end 0)) "/"))
-     ((string-match "^SOCKS +" proxy)
-      (concat "socks://" (substring proxy (match-end 0))))
+     ((string-match  (rx bot "SOCKS" (** 0 2 alnum) " ") proxy)
+      (if-let ((m (substring proxy (match-end 0)))
+               ((string-search "://" m)))
+          m
+        (concat "socks://" m)))
      (t
       (display-warning 'url (format "Unknown proxy directive: %s" proxy) :error)
       nil))))
@@ -72,6 +77,9 @@ url-proxy
   (cond
    ((string= (url-type url-using-proxy) "http")
     (url-http url callback cbargs))
+   ((and (string-prefix-p "socks" (url-type url-using-proxy))
+         (string-prefix-p "http" (url-type url)))
+    (url-http url callback cbargs))
    (t
     (error "Don't know how to use proxy `%s'" url-using-proxy))))
 
diff --git a/lisp/url/url-vars.el b/lisp/url/url-vars.el
index 6d7d0d3c94c..ba219905fde 100644
--- a/lisp/url/url-vars.el
+++ b/lisp/url/url-vars.el
@@ -194,10 +194,24 @@ url-mail-command
   :type 'function
   :group 'url)
 
+(defvar url-proxy-full-address-types
+  '("socks" "socks5" "socks5h" "socks4" "socks4a")
+  "Schemes for URL types preserved in `url-proxy-services' entries.
+When dynamically adding a new `url-proxy-services' entry derived
+from the environment, Emacs only retains the host and port
+portions unless the URL's scheme appears in this variable's
+value.")
+
 (defcustom url-proxy-services nil
   "An alist of schemes and proxy servers that gateway them.
 Looks like ((\"http\" . \"hostname:portnumber\") ...).  This is set up
-from the ACCESS_proxy environment variables."
+from the ACCESS_proxy environment variables.  Certain gateway
+types need server values to take the form of full URLs in order
+to convey addtional information about for the proxy connection
+itself, for example, SCHEME://USER@HOSTNAME:PORTNUMBER, in which
+SCHEME is something like \"socks5\".  As of Emacs 30.1, this only
+applies to SCHEMEs appearing in the variable
+`url-proxy-full-address-types'."
   :type '(repeat (cons :format "%v"
 		       (string :tag "Protocol")
 		       (string :tag "Proxy")))
@@ -315,7 +329,9 @@ url-show-status
 
 (defvar url-using-proxy nil
   "Either nil or the fully qualified proxy URL in use, e.g.
-https://www.example.com/")
+https://www.example.com/.  Beware that some functions, such as
+`url-proxy' and `url-http-end-of-document-sentinel', set this to
+a `url' struct object.")
 
 (defcustom url-news-server nil
   "The default news server from which to get newsgroups/articles.
-- 
2.42.0


--=-=-=--

Last modified: Sat, 20 Jan 2024 12:30:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.