GNU bug report logs - #45938
28.0.50; python native completion fails with fancier readline settings

Message received at 45938 <at> debbugs.gnu.org:

Received: (at 45938) by debbugs.gnu.org; 7 Jun 2022 14:06:18 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Jun 07 10:06:18 2022
Received: from localhost ([127.0.0.1]:41035 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1nyZqc-0002eo-30
	for submit <at> debbugs.gnu.org; Tue, 07 Jun 2022 10:06:18 -0400
Received: from quimby.gnus.org ([95.216.78.240]:53950)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <larsi@HIDDEN>) id 1nyZqQ-0002dv-90
 for 45938 <at> debbugs.gnu.org; Tue, 07 Jun 2022 10:06:13 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnus.org;
 s=20200322; h=Content-Transfer-Encoding:Content-Type:MIME-Version:Message-ID
 :In-Reply-To:Date:References:Subject:Cc:To:From:Sender:Reply-To:Content-ID:
 Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc
 :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe:
 List-Post:List-Owner:List-Archive;
 bh=bWRsYL323+TjWmFZ1bDTHbLJAUmkPZ4br/uIy4ht4io=; b=HW5WWEJPG4RTMkf9v7CTGgndVu
 tcZPZzxGbmdtGbZLOW5ZQaRXaJ//tGv5kIVy53/q8RQHZ6n3Px+nvdBGBww3BeDulu2oKVr47k9qo
 q5yBYuPoR8adTPk2OpqureqonNl776vD4mWo/Vappfdx21KkdlOmvLSoxPP3OGdy2oA8=;
Received: from [84.212.220.105] (helo=xo)
 by quimby.gnus.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.92) (envelope-from <larsi@HIDDEN>)
 id 1nyZqG-0003Um-GW; Tue, 07 Jun 2022 16:05:59 +0200
From: Lars Ingebrigtsen <larsi@HIDDEN>
To: =?utf-8?Q?Zolt=C3=A1n?= Vandrus <vandrus.zoltan@HIDDEN>
Subject: Re: bug#45938: 28.0.50; python native completion fails with fancier
 readline settings
References: <87v9bvwn63.fsf@HIDDEN>
X-Now-Playing: Blaine L. Reininger's _The Blue Sleep_: "Molecular Landscape"
Date: Tue, 07 Jun 2022 16:05:55 +0200
In-Reply-To: <87v9bvwn63.fsf@HIDDEN> (=?utf-8?Q?=22Zolt=C3=A1n?=
 Vandrus"'s message of "Sun, 17 Jan 2021 17:20:04 +0100")
Message-ID: <87bkv4h8l8.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/29.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Report: Spam detection software, running on the system "quimby.gnus.org",
 has NOT identified this incoming email as spam.  The original
 message has been attached to this so you can view it or label
 similar future email.  If you have any questions, see
 @@CONTACT_ADDRESS@@ for details.
 
 Content preview:  Zoltán Vandrus <vandrus.zoltan@HIDDEN> writes: > Three
    problematic options I found are: > > set colored-stats on > set colored-completion-prefix
    on > set enable-bracketed-paste on > > With any of these M-x run-python warns:
    > > Warning (python): Y [...] 
 
 Content analysis details:   (-2.9 points, 5.0 required)
 
  pts rule name              description
 ---- ---------------------- --------------------------------------------------
 -1.0 ALL_TRUSTED            Passed through trusted hosts only via SMTP
 -1.9 BAYES_00               BODY: Bayes spam probability is 0 to 1%
                             [score: 0.0000]
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 45938
Cc: 45938 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Zolt=C3=A1n Vandrus <vandrus.zoltan@HIDDEN> writes:

> Three problematic options I found are:
>
>    set colored-stats on
>    set colored-completion-prefix on
>    set enable-bracketed-paste on
>
> With any of these M-x run-python warns:
>
>    Warning (python): Your =E2=80=98python-shell-interpreter=E2=80=99 does=
n=E2=80=99t seem to
>    support readline, yet =E2=80=98python-shell-completion-native-enable=
=E2=80=99 was t and
>    "python3" is not part of the
>    =E2=80=98python-shell-completion-native-disabled-interpreters=E2=80=99=
 list.  Native
>    completions have been disabled locally.

(I'm going through old bug reports that unfortunately weren't resolved
at the time.)

There seems to have been some progress here after this was reported.
The only one of these settings that lead to this warning in Emacs 29 is:

set colored-completion-prefix on

But I'm actually not sure what the commit is that fixed the other cases.
Let's see...

commit a0973fa7a6e8162033abe29084f7b255976c74bc
Author:     Carlos Pita <carlosjosepita@HIDDEN>
AuthorDate: Wed Oct 13 21:09:40 2021 -0300
Commit:     Stefan Kangas <stefan@HIDDEN>
CommitDate: Fri Nov 5 08:02:31 2021 +0100

    Match dummy output even with non-default rl config
=20=20=20=20
    * lisp/progmodes/python.el
    (python-shell-completion-native-get-completions): Match dummy output
    even when readline is configured in non-default ways.  (Bug#51010)

Yes, that's the one -- reverting it makes all of those settings fail,
and not just colored-completion-prefix.

And that's because completion returns:

^[[01;35m0^[[0m^[[K__dummy_completion__  ^[[01;35m1^[[0m^[[K__dummy_complet=
ion__=20=20

And without that setting, we get:

0__dummy_completion__  1__dummy_completion__=20=20

So something has to strip ANSI codes here before doing the matching, and
it's not done?

I'm not really that familiar with all this stuff -- anybody know on what
level this should be fixed?  (The code is in
python-shell-completion-native-get-completions.)

--=20
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message received at submit <at> debbugs.gnu.org:

Received: (at submit) by debbugs.gnu.org; 17 Jan 2021 16:20:18 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Jan 17 11:20:18 2021
Received: from localhost ([127.0.0.1]:45539 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1l1Amn-0006kg-FZ
	for submit <at> debbugs.gnu.org; Sun, 17 Jan 2021 11:20:18 -0500
Received: from lists.gnu.org ([209.51.188.17]:33392)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <vandrus.zoltan@HIDDEN>) id 1l1Ami-0006kU-Oz
 for submit <at> debbugs.gnu.org; Sun, 17 Jan 2021 11:20:15 -0500
Received: from eggs.gnu.org ([2001:470:142:3::10]:57374)
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <vandrus.zoltan@HIDDEN>)
 id 1l1Amh-0006Y5-Ib
 for bug-gnu-emacs@HIDDEN; Sun, 17 Jan 2021 11:20:12 -0500
Received: from mail-ed1-x536.google.com ([2a00:1450:4864:20::536]:46504)
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128)
 (Exim 4.90_1) (envelope-from <vandrus.zoltan@HIDDEN>)
 id 1l1Ame-0008Re-J5
 for bug-gnu-emacs@HIDDEN; Sun, 17 Jan 2021 11:20:11 -0500
Received: by mail-ed1-x536.google.com with SMTP id dj23so12311459edb.13
 for <bug-gnu-emacs@HIDDEN>; Sun, 17 Jan 2021 08:20:07 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=from:to:subject:date:message-id:mime-version
 :content-transfer-encoding;
 bh=MmPZIRSx1JZYmZ6sta3PkWYFGE//Q3Iin7uiWzo/KHY=;
 b=Ok7t4tQI3IBxNu6a2i31G/pTluOf7hFF2hVwCaoptYbXU2GkgURiJHX3BAzdhQnmGI
 XRHRJQkJyWeihYI6IVkqW2BWOgglI4VwnyvC9byEJN6DYjai/ZY+6i774pqSsenZs5Ws
 41Yr+LMegH5dgHjgbG1JmM8RuVHlsWG9Kw9QxcfKgG2Z46KZkwstlS6AQU3O4MNKmOUI
 SCBtFw/rU+vogTmkaYZSsGE9hiKioJfCqhux6OrcUylUgUt3glgox4OiBuQ40FOyLMG9
 64c+2+SEeORsz+nenOzYx8UPir+f1OXTq+MC9Cj8UYcpXVrQ6Nsr7GCbnsUjb2xa0Lfi
 excg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:from:to:subject:date:message-id:mime-version
 :content-transfer-encoding;
 bh=MmPZIRSx1JZYmZ6sta3PkWYFGE//Q3Iin7uiWzo/KHY=;
 b=nv0iXb8IQ77eqE841tlM/4JOED66dASzWTObnjoGDZNeEwg+NZUNtxtgckk0Lq9AOf
 snJUBErCey+6zDnD7eq+r8zDD0J5x29GiBviywGIOREHRtvnJSF1RMPs+yPZB7lt23Ue
 xK/XqQ1oGLG/XYrnoTy2DAJwLp2Kg+OSVzzMv63ZUc6471f8q1BWrTQxaZ2RMBdWMdkn
 Fdf4NxqRbzpSrYTdQzGmIE9ZRH/mLJXxHBlD7FAgWN4RDQp7o2bNrPz/x7fo2lEtEAKG
 g3MODLVM8CaS9dsblWWt4ZrGR1+dlbnzjIo3GZ8vYFJfYC38qGYwiV76t0FE/iuqXzeI
 Jmcw==
X-Gm-Message-State: AOAM530+RO4lw16BX2rCOMUN95JgVrwaaGt5j4Layw8vrhpz1DOHRnfW
 mReq2F0MRixuSBBNy6zsJmhksX2k1AA=
X-Google-Smtp-Source: ABdhPJyY0Q0ycFCIKq0tCLjoc3UB2tPcTs8Ck7ZkrvFp/WE8DJYRafvG3UT1qsUkQUnm4lgGTuyvFg==
X-Received: by 2002:aa7:c3cf:: with SMTP id l15mr17122173edr.282.1610900406556; 
 Sun, 17 Jan 2021 08:20:06 -0800 (PST)
Received: from zoli-laptop (fibhost-66-212-245.fibernet.hu. [85.66.212.245])
 by smtp.gmail.com with ESMTPSA id b101sm1300889edf.49.2021.01.17.08.20.05
 for <bug-gnu-emacs@HIDDEN>
 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
 Sun, 17 Jan 2021 08:20:05 -0800 (PST)
From: =?utf-8?Q?Zolt=C3=A1n_Vandrus?= <vandrus.zoltan@HIDDEN>
To: bug-gnu-emacs@HIDDEN
Subject: 28.0.50; python native completion fails with fancier readline settings
Date: Sun, 17 Jan 2021 17:20:04 +0100
Message-ID: <87v9bvwn63.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Received-SPF: pass client-ip=2a00:1450:4864:20::536;
 envelope-from=vandrus.zoltan@HIDDEN; helo=mail-ed1-x536.google.com
X-Spam_score_int: -20
X-Spam_score: -2.1
X-Spam_bar: --
X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,
 DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001,
 RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001,
 SPF_PASS=-0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-Spam-Score: -1.3 (-)
X-Debbugs-Envelope-To: submit
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -2.3 (--)


Three problematic options I found are:

   set colored-stats on
   set colored-completion-prefix on
   set enable-bracketed-paste on

With any of these M-x run-python warns:

   Warning (python): Your =E2=80=98python-shell-interpreter=E2=80=99 doesn=
=E2=80=99t seem to
   support readline, yet =E2=80=98python-shell-completion-native-enable=E2=
=80=99 was t and
   "python3" is not part of the
   =E2=80=98python-shell-completion-native-disabled-interpreters=E2=80=99 l=
ist.  Native
   completions have been disabled locally.
=20=20=20

set colored-stats on
set colored-completion-prefix on
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D

Any of these on and *Python completions redirect* buffer ends with:
0__dummy_completion__^[[0m^[[K  1__dummy_completion__^[[0m^[[K

^[[0m and ^[[K are escape sequences not understood by comint mode I
think

Related bug: #24401 recommends setting colored-stats conditionally only
for bash, because filename coloring is not that useful for the python
shell.

Even then colored-completion-prefix is useful everywhere, emacs does it
by default too.

set enable-bracketed-paste on
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D

Fails like half of the times for me. For a different reason.

First record output:
   (defun keep-output (output)
     (setq kept (cons output kept)))

   (add-hook 'inferior-python-mode-hook
             (lambda ()
               (setq kept nil)
               (add-hook 'comint-output-filter-functions #'keep-output)))

M-x run-python

kept=E2=80=99s value is
("python.el: native completion setup loaded
^[[?2004h>>> " "^[[?2004l^M" "Python 3.9.0 (default, Oct  7 2020, 23:09:01)=
=20
[GCC 10.2.0] on linux
Type \"help\", \"copyright\", \"credits\" or \"license\" for more informati=
on.
^[[?2004h>>> ")

-----------------------

python-shell-completin-native-setup assumes that after running the
python script that sets up completion the first output will be from the scr=
ipt:
      (when (and
             (python-shell-accept-process-output
              process python-shell-completion-native-try-output-timeout)
             (save-excursion
               (re-search-backward
                (regexp-quote "python.el: native completion setup loaded") =
nil t 1)))
        (python-shell-completion-native-try)))))

What can happen sometimes is that, after the first initial prompt:
   "Python 3.9.0 (default, Oct  7 2020, 23:09:01)=20
   [GCC 10.2.0] on linux
   Type \"help\", \"copyright\", \"credits\" or \"license\" for more inform=
ation.
   ^[[?2004h>>> "
The python script is injected.
python-shell-accept-process-output waits for next prompt
Bracketed mode is disabled instead with clearing the line: "^[[?2004l^M"
Empty line is a valid python prompt, so it's accepted, and the regexp
test fails.

#41959 is another problem with trying to parse the readline output.

--------------------------------------------------

Readline version 8.1.0

In GNU Emacs 28.0.50 (build 4, x86_64-pc-linux-gnu, GTK+ Version 3.24.24, c=
airo version 1.17.4)
 of 2021-01-17 built on zoli-laptop
Repository revision: 1773679af3241919a85d6174b1554070a63cca79
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12010000
System Description: Arch Linux

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
JSON LCMS2 LIBOTF LIBSYSTEMD LIBXML2 M17N_FLT MODULES NOTIFY INOTIFY
PDUMPER PNG RSVG SOUND THREADS TIFF TOOLKIT_SCROLL_BARS X11 XDBE XIM XPM
GTK3 ZLIB

Important settings:
  value of $LC_TIME: C
  value of $LANG: hu_HU.utf8
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  shell-dirtrack-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rmc puny dired dired-loaddefs
rfc822 mml mml-sec epa derived epg epg-config gnus-util rmail
rmail-loaddefs mm-decode mm-bodies mm-encode mail-parse rfc2231
mailabbrev gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums
mm-util mail-prsvr mail-utils warnings compile text-property-search
cl-extra help-mode python easymenu tramp-sh tramp tramp-loaddefs
trampver tramp-integration files-x tramp-compat shell pcomplete
parse-time iso8601 time-date ls-lisp format-spec auth-source cl-seq
eieio eieio-core cl-macs eieio-loaddefs password-cache json subr-x map
seq byte-opt gv bytecomp byte-compile cconv comint ring cl-loaddefs
cl-lib ansi-color iso-transl tooltip eldoc electric uniquify ediff-hook
vc-hooks lisp-float-type mwheel term/x-win x-win term/common-win x-dnd
tool-bar dnd fontset image regexp-opt fringe tabulated-list replace
newcomment text-mode elisp-mode lisp-mode prog-mode register page
tab-bar menu-bar rfn-eshadow isearch timer select scroll-bar mouse
jit-lock font-lock syntax facemenu font-core term/tty-colors frame
minibuffer cl-generic cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
composite charscript charprop case-table epa-hook jka-cmpr-hook help
simple abbrev obarray cl-preloaded nadvice button loaddefs faces
cus-face macroexp files window text-properties overlay sha1 md5 base64
format env code-pages mule custom widget hashtable-print-readable
backquote threads dbusbind inotify lcms2 dynamic-setting
system-font-setting font-render-setting cairo move-toolbar gtk x-toolkit
x multi-tty make-network-process emacs)

Memory information:
((conses 16 74431 8365)
 (symbols 48 9112 1)
 (strings 32 27268 1190)
 (string-bytes 1 989045)
 (vectors 16 15840)
 (vector-slots 8 208279 9279)
 (floats 8 47 29)
 (intervals 56 226 0)
 (buffers 984 14))