GNU bug report logs - #76205
31.0.50; 31.0.50; python.el completion broken for non-builtin objects in IPython 8.30.0 & Python 3.13.2

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 76205 in the body.
You can then email your comments to 76205 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Visuwesh <visuweshm <at> gmail.com>
To: bug-gnu-emacs <at> gnu.org
Subject: 31.0.50; 31.0.50; python.el completion broken for non-builtin
 objects in IPython 8.30.0 & Python 3.13.2
Date: Tue, 11 Feb 2025 20:37:22 +0530

I am unsure if this is a me problem but I can no longer complete
non-builtin objects in a python-mode buffer when I'm using the specified
IPython&Python version combination.  However, if I don't use IPython,
then completion does work as expected.  It prints the following message
though (not sure if it is related):

    Python 3.13.2 (main, Feb  5 2025, 01:23:35) [GCC 14.2.0] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    warning: can't use pyrepl: terminal doesn't have the required clear capability; TERM=dumb
    >>>

To reproduce the issue,

    1. emacs -Q
    2. Open a python buffer.
    3. Type in "import numpy as np" in the buffer.
    4. Say C-u C-c C-p, answer ipython3 --simple-prompt -i to the first
       prompt, then say no to the second.
    5. After the *Python* is created, say C-c C-c in the python-mode
       buffer.
    6. Now type "np." and say M-TAB.
    7. Observe how no completion candidates are offered.
    8. Remove "np." and instead type "pr" and say M-TAB.
    9. Observe how two completion candidates (print and property) are
       offered.
    10. Repeat the same without using IPython and observe how
        python-mode offers completion after "np."

I am on Debian sid with an "apt update && apt upgrade" today some 15
mins ago (no luck after updating if I wasn't clear enough) if it
matters.


In GNU Emacs 31.0.50 (build 30, x86_64-pc-linux-gnu, X toolkit, cairo
 version 1.18.2, Xaw scroll bars) of 2025-01-23 built on astatine
Repository revision: 7dcc7605d58165dac233bcc029509f1ad9807f72
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12101015
System Description: Debian GNU/Linux trixie/sid

Configured using:
 'configure --with-sound=alsa --with-x-toolkit=lucid --without-xaw3d
 --without-gconf --without-libsystemd --with-cairo CFLAGS=-O2'

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
LCMS2 LIBOTF LIBSELINUX LIBXML2 MODULES NOTIFY INOTIFY PDUMPER PNG RSVG
SECCOMP SOUND SQLITE3 THREADS TIFF TOOLKIT_SCROLL_BARS WEBP X11 XDBE XIM
XINPUT2 XPM LUCID ZLIB

Important settings:
  value of $LC_MONETARY: ta_IN.UTF-8
  value of $LC_NUMERIC: ta_IN.UTF-8
  value of $LANG: en_GB.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Python

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  minibuffer-regexp-mode: t
  line-number-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message mailcap yank-media puny dired
dired-loaddefs rfc822 mml mml-sec password-cache epa derived epg rfc6068
epg-config gnus-util mm-decode mm-bodies mm-encode mail-parse rfc2231
mailabbrev gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums
mm-util mail-prsvr mail-utils time-date compile text-property-search
cl-extra help-mode files-x shell pcomplete cl-seq python rx project
byte-opt gv bytecomp byte-compile compat pcase treesit comint subr-x
ansi-osc ring cl-loaddefs cl-lib ansi-color rmc iso-transl tooltip cconv
eldoc paren electric uniquify ediff-hook vc-hooks lisp-float-type
elisp-mode mwheel term/x-win x-win term/common-win x-dnd touch-screen
tool-bar dnd fontset image regexp-opt fringe tabulated-list replace
newcomment text-mode lisp-mode prog-mode register page tab-bar menu-bar
rfn-eshadow isearch easymenu timer select scroll-bar mouse jit-lock
font-lock syntax font-core term/tty-colors frame minibuffer nadvice seq
simple cl-generic indonesian philippine cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese composite emoji-zwj charscript charprop case-table
epa-hook jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button
loaddefs theme-loaddefs faces cus-face macroexp files window
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget keymap hashtable-print-readable backquote threads dbusbind
inotify lcms2 dynamic-setting system-font-setting font-render-setting
cairo x-toolkit xinput2 x multi-tty move-toolbar make-network-process
tty-child-frames emacs)

Memory information:
((conses 16 70184 11581) (symbols 48 8249 0) (strings 32 22079 1977)
 (string-bytes 1 572455) (vectors 16 14799)
 (vector-slots 8 156908 8229) (floats 8 36 27) (intervals 56 313 0)
 (buffers 984 15))

Message #8 received at 76205 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Visuwesh <visuweshm <at> gmail.com>, kobarity <kobarity <at> gmail.com>
Cc: 76205 <at> debbugs.gnu.org
Subject: Re: bug#76205: 31.0.50; 31.0.50;
 python.el completion broken for non-builtin objects in IPython 8.30.0
 & Python 3.13.2
Date: Wed, 12 Feb 2025 14:06:37 +0200

> From: Visuwesh <visuweshm <at> gmail.com>
> Date: Tue, 11 Feb 2025 20:37:22 +0530
> 
> I am unsure if this is a me problem but I can no longer complete
> non-builtin objects in a python-mode buffer when I'm using the specified
> IPython&Python version combination.  However, if I don't use IPython,
> then completion does work as expected.  It prints the following message
> though (not sure if it is related):
> 
>     Python 3.13.2 (main, Feb  5 2025, 01:23:35) [GCC 14.2.0] on linux
>     Type "help", "copyright", "credits" or "license" for more information.
>     warning: can't use pyrepl: terminal doesn't have the required clear capability; TERM=dumb
>     >>>
> 
> To reproduce the issue,
> 
>     1. emacs -Q
>     2. Open a python buffer.
>     3. Type in "import numpy as np" in the buffer.
>     4. Say C-u C-c C-p, answer ipython3 --simple-prompt -i to the first
>        prompt, then say no to the second.
>     5. After the *Python* is created, say C-c C-c in the python-mode
>        buffer.
>     6. Now type "np." and say M-TAB.
>     7. Observe how no completion candidates are offered.
>     8. Remove "np." and instead type "pr" and say M-TAB.
>     9. Observe how two completion candidates (print and property) are
>        offered.
>     10. Repeat the same without using IPython and observe how
>         python-mode offers completion after "np."
> 
> I am on Debian sid with an "apt update && apt upgrade" today some 15
> mins ago (no luck after updating if I wasn't clear enough) if it
> matters.

Thanks.

kobarity, any comments or suggestions?

Message #11 received at 76205 <at> debbugs.gnu.org (full text, mbox):

From: kobarity <kobarity <at> gmail.com>
To: Liu Hui <liuhui1610 <at> gmail.com>, Eli Zaretskii <eliz <at> gnu.org>
Cc: 76205 <at> debbugs.gnu.org, Visuwesh <visuweshm <at> gmail.com>
Subject: Re: bug#76205: 31.0.50; 31.0.50;
 python.el completion broken for non-builtin objects in IPython 8.30.0
 & Python 3.13.2
Date: Thu, 13 Feb 2025 00:03:15 +0900

Eli Zaretskii wrote:
> 
> > From: Visuwesh <visuweshm <at> gmail.com>
> > Date: Tue, 11 Feb 2025 20:37:22 +0530
> > 
> > I am unsure if this is a me problem but I can no longer complete
> > non-builtin objects in a python-mode buffer when I'm using the specified
> > IPython&Python version combination.  However, if I don't use IPython,
> > then completion does work as expected.  It prints the following message
> > though (not sure if it is related):
> > 
> >     Python 3.13.2 (main, Feb  5 2025, 01:23:35) [GCC 14.2.0] on linux
> >     Type "help", "copyright", "credits" or "license" for more information.
> >     warning: can't use pyrepl: terminal doesn't have the required clear capability; TERM=dumb
> >     >>>
> > 
> > To reproduce the issue,
> > 
> >     1. emacs -Q
> >     2. Open a python buffer.
> >     3. Type in "import numpy as np" in the buffer.
> >     4. Say C-u C-c C-p, answer ipython3 --simple-prompt -i to the first
> >        prompt, then say no to the second.
> >     5. After the *Python* is created, say C-c C-c in the python-mode
> >        buffer.
> >     6. Now type "np." and say M-TAB.
> >     7. Observe how no completion candidates are offered.
> >     8. Remove "np." and instead type "pr" and say M-TAB.
> >     9. Observe how two completion candidates (print and property) are
> >        offered.
> >     10. Repeat the same without using IPython and observe how
> >         python-mode offers completion after "np."
> > 
> > I am on Debian sid with an "apt update && apt upgrade" today some 15
> > mins ago (no luck after updating if I wasn't clear enough) if it
> > matters.
> 
> Thanks.
> 
> kobarity, any comments or suggestions?

I reproduced the issue with IPython 8.32.0 and Python 3.13.2.  It
seems that there is no problem with IPython on Python 3.12.

One workaround is to disable native completion:
M-x python-shell-completion-native-turn-off
It will be a little more inconvenient than the native completion.

We need to investigate the cause.

Liu Hui, do you have any idea?

Message #14 received at 76205 <at> debbugs.gnu.org (full text, mbox):

From: Liu Hui <liuhui1610 <at> gmail.com>
To: kobarity <kobarity <at> gmail.com>
Cc: 76205 <at> debbugs.gnu.org, Eli Zaretskii <eliz <at> gnu.org>,
 Visuwesh <visuweshm <at> gmail.com>
Subject: Re: bug#76205: 31.0.50; 31.0.50; python.el completion broken for
 non-builtin objects in IPython 8.30.0 & Python 3.13.2
Date: Thu, 13 Feb 2025 11:47:00 +0800

On Wed, Feb 12, 2025 at 11:03 PM kobarity <kobarity <at> gmail.com> wrote:
>
> Eli Zaretskii wrote:
> >
> > > From: Visuwesh <visuweshm <at> gmail.com>
> > > Date: Tue, 11 Feb 2025 20:37:22 +0530
> > >
> > > I am unsure if this is a me problem but I can no longer complete
> > > non-builtin objects in a python-mode buffer when I'm using the specified
> > > IPython&Python version combination.  However, if I don't use IPython,
> > > then completion does work as expected.  It prints the following message
> > > though (not sure if it is related):
> > >
> > >     Python 3.13.2 (main, Feb  5 2025, 01:23:35) [GCC 14.2.0] on linux
> > >     Type "help", "copyright", "credits" or "license" for more information.
> > >     warning: can't use pyrepl: terminal doesn't have the required clear capability; TERM=dumb
> > >     >>>
> > >
> > > To reproduce the issue,
> > >
> > >     1. emacs -Q
> > >     2. Open a python buffer.
> > >     3. Type in "import numpy as np" in the buffer.
> > >     4. Say C-u C-c C-p, answer ipython3 --simple-prompt -i to the first
> > >        prompt, then say no to the second.
> > >     5. After the *Python* is created, say C-c C-c in the python-mode
> > >        buffer.
> > >     6. Now type "np." and say M-TAB.
> > >     7. Observe how no completion candidates are offered.
> > >     8. Remove "np." and instead type "pr" and say M-TAB.
> > >     9. Observe how two completion candidates (print and property) are
> > >        offered.
> > >     10. Repeat the same without using IPython and observe how
> > >         python-mode offers completion after "np."
> > >
> > > I am on Debian sid with an "apt update && apt upgrade" today some 15
> > > mins ago (no luck after updating if I wasn't clear enough) if it
> > > matters.
> >
> > Thanks.
> >
> > kobarity, any comments or suggestions?
>
> I reproduced the issue with IPython 8.32.0 and Python 3.13.2.  It
> seems that there is no problem with IPython on Python 3.12.
>
> One workaround is to disable native completion:
> M-x python-shell-completion-native-turn-off
> It will be a little more inconvenient than the native completion.
>
> We need to investigate the cause.
>
> Liu Hui, do you have any idea?

The problem seems to be not related to emacs, because the completion
of non-builtin objects doesn't work in terminal too (IPython 8.32.0
and Python 3.13.2):

1. run ipython in a terminal: ipython3 --simple-prompt -i

2. type the following code:
   import readline
   import rlcompleter
   readline.parse_and_bind("tab: complete")
   import numpy as np

3. type np.<TAB>, there is no completion


If I change the completer to others (e.g. jedi):
   from jedi.utils import setup_readline
   setup_readline()

then the completion works in both terminal and emacs.

Message #17 received at 76205 <at> debbugs.gnu.org (full text, mbox):

From: Visuwesh <visuweshm <at> gmail.com>
To: Liu Hui <liuhui1610 <at> gmail.com>
Cc: 76205 <at> debbugs.gnu.org, kobarity <kobarity <at> gmail.com>,
 Eli Zaretskii <eliz <at> gnu.org>
Subject: Re: bug#76205: 31.0.50; 31.0.50; python.el completion broken for
 non-builtin objects in IPython 8.30.0 & Python 3.13.2
Date: Thu, 13 Feb 2025 09:33:13 +0530

[வியாழன் பிப்ரவரி 13, 2025] Liu Hui wrote:

> On Wed, Feb 12, 2025 at 11:03 PM kobarity <kobarity <at> gmail.com> wrote:
>>
>> Eli Zaretskii wrote:
>> >
>> > > From: Visuwesh <visuweshm <at> gmail.com>
>> > > Date: Tue, 11 Feb 2025 20:37:22 +0530
>> > >
>> > > I am unsure if this is a me problem but I can no longer complete
>> > > non-builtin objects in a python-mode buffer when I'm using the specified
>> > > IPython&Python version combination.  However, if I don't use IPython,
>> > > then completion does work as expected.  It prints the following message
>> > > though (not sure if it is related):
>> > >
>> > >     Python 3.13.2 (main, Feb  5 2025, 01:23:35) [GCC 14.2.0] on linux
>> > >     Type "help", "copyright", "credits" or "license" for more information.
>> > >     warning: can't use pyrepl: terminal doesn't have the required clear capability; TERM=dumb
>> > >     >>>
>> > >
>> > > To reproduce the issue,
>> > >
>> > >     1. emacs -Q
>> > >     2. Open a python buffer.
>> > >     3. Type in "import numpy as np" in the buffer.
>> > >     4. Say C-u C-c C-p, answer ipython3 --simple-prompt -i to the first
>> > >        prompt, then say no to the second.
>> > >     5. After the *Python* is created, say C-c C-c in the python-mode
>> > >        buffer.
>> > >     6. Now type "np." and say M-TAB.
>> > >     7. Observe how no completion candidates are offered.
>> > >     8. Remove "np." and instead type "pr" and say M-TAB.
>> > >     9. Observe how two completion candidates (print and property) are
>> > >        offered.
>> > >     10. Repeat the same without using IPython and observe how
>> > >         python-mode offers completion after "np."
>> > >
>> > > I am on Debian sid with an "apt update && apt upgrade" today some 15
>> > > mins ago (no luck after updating if I wasn't clear enough) if it
>> > > matters.
>> >
>> > Thanks.
>> >
>> > kobarity, any comments or suggestions?
>>
>> I reproduced the issue with IPython 8.32.0 and Python 3.13.2.  It
>> seems that there is no problem with IPython on Python 3.12.
>>
>> One workaround is to disable native completion:
>> M-x python-shell-completion-native-turn-off
>> It will be a little more inconvenient than the native completion.

Thank you very much, this works good enough for my needs.  I can work in
peace now.

>> We need to investigate the cause.
>>
>> Liu Hui, do you have any idea?
>
> The problem seems to be not related to emacs, because the completion
> of non-builtin objects doesn't work in terminal too (IPython 8.32.0
> and Python 3.13.2):
>
> 1. run ipython in a terminal: ipython3 --simple-prompt -i
>
> 2. type the following code:
>    import readline
>    import rlcompleter
>    readline.parse_and_bind("tab: complete")
>    import numpy as np
>
> 3. type np.<TAB>, there is no completion
>
>
> If I change the completer to others (e.g. jedi):
>    from jedi.utils import setup_readline
>    setup_readline()
>
> then the completion works in both terminal and emacs.

Indeed, the completion is broken in the terminal.  It did not occur to
me to test it from the terminal at all.  :(
Should we take this as a IPython bug then?

Message #20 received at 76205 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Visuwesh <visuweshm <at> gmail.com>
Cc: liuhui1610 <at> gmail.com, kobarity <at> gmail.com, 76205 <at> debbugs.gnu.org
Subject: Re: bug#76205: 31.0.50; 31.0.50; python.el completion broken for
 non-builtin objects in IPython 8.30.0 & Python 3.13.2
Date: Thu, 13 Feb 2025 10:01:23 +0200

> From: Visuwesh <visuweshm <at> gmail.com>
> Cc: kobarity <kobarity <at> gmail.com>,  76205 <at> debbugs.gnu.org,  Eli Zaretskii
>  <eliz <at> gnu.org>
> Date: Thu, 13 Feb 2025 09:33:13 +0530
> 
> Indeed, the completion is broken in the terminal.  It did not occur to
> me to test it from the terminal at all.  :(
> Should we take this as a IPython bug then?

Yes, I think so.

Message #23 received at 76205 <at> debbugs.gnu.org (full text, mbox):

From: kobarity <kobarity <at> gmail.com>
To: Visuwesh <visuweshm <at> gmail.com>,
	liuhui1610 <at> gmail.com
Cc: 76205 <at> debbugs.gnu.org, Eli Zaretskii <eliz <at> gnu.org>
Subject: Re: bug#76205: 31.0.50; 31.0.50;
 python.el completion broken for non-builtin objects in IPython 8.30.0
 & Python 3.13.2
Date: Thu, 13 Feb 2025 23:59:50 +0900

Visuwesh wrote:
> 
> [வியாழன் பிப்ரவரி 13, 2025] Liu Hui wrote:
> 
> > On Wed, Feb 12, 2025 at 11:03 PM kobarity <kobarity <at> gmail.com> wrote:
> >>
> >> Eli Zaretskii wrote:
> >> >
> >> > > From: Visuwesh <visuweshm <at> gmail.com>
> >> > > Date: Tue, 11 Feb 2025 20:37:22 +0530
> >> > >
> >> > > I am unsure if this is a me problem but I can no longer complete
> >> > > non-builtin objects in a python-mode buffer when I'm using the specified
> >> > > IPython&Python version combination.  However, if I don't use IPython,
> >> > > then completion does work as expected.  It prints the following message
> >> > > though (not sure if it is related):
> >> > >
> >> > >     Python 3.13.2 (main, Feb  5 2025, 01:23:35) [GCC 14.2.0] on linux
> >> > >     Type "help", "copyright", "credits" or "license" for more information.
> >> > >     warning: can't use pyrepl: terminal doesn't have the required clear capability; TERM=dumb
> >> > >     >>>
> >> > >
> >> > > To reproduce the issue,
> >> > >
> >> > >     1. emacs -Q
> >> > >     2. Open a python buffer.
> >> > >     3. Type in "import numpy as np" in the buffer.
> >> > >     4. Say C-u C-c C-p, answer ipython3 --simple-prompt -i to the first
> >> > >        prompt, then say no to the second.
> >> > >     5. After the *Python* is created, say C-c C-c in the python-mode
> >> > >        buffer.
> >> > >     6. Now type "np." and say M-TAB.
> >> > >     7. Observe how no completion candidates are offered.
> >> > >     8. Remove "np." and instead type "pr" and say M-TAB.
> >> > >     9. Observe how two completion candidates (print and property) are
> >> > >        offered.
> >> > >     10. Repeat the same without using IPython and observe how
> >> > >         python-mode offers completion after "np."
> >> > >
> >> > > I am on Debian sid with an "apt update && apt upgrade" today some 15
> >> > > mins ago (no luck after updating if I wasn't clear enough) if it
> >> > > matters.
> >> >
> >> > Thanks.
> >> >
> >> > kobarity, any comments or suggestions?
> >>
> >> I reproduced the issue with IPython 8.32.0 and Python 3.13.2.  It
> >> seems that there is no problem with IPython on Python 3.12.
> >>
> >> One workaround is to disable native completion:
> >> M-x python-shell-completion-native-turn-off
> >> It will be a little more inconvenient than the native completion.
> 
> Thank you very much, this works good enough for my needs.  I can work in
> peace now.

Then you might want to customize
`python-shell-completion-native-enable' or
`python-shell-completion-native-disabled-interpreters'.

> > The problem seems to be not related to emacs, because the completion
> > of non-builtin objects doesn't work in terminal too (IPython 8.32.0
> > and Python 3.13.2):
> >
> > 1. run ipython in a terminal: ipython3 --simple-prompt -i
> >
> > 2. type the following code:
> >    import readline
> >    import rlcompleter
> >    readline.parse_and_bind("tab: complete")
> >    import numpy as np
> >
> > 3. type np.<TAB>, there is no completion
> >
> >
> > If I change the completer to others (e.g. jedi):
> >    from jedi.utils import setup_readline
> >    setup_readline()
> >
> > then the completion works in both terminal and emacs.

Thanks for pointing this out.  So I confirmed that the following
workaround is also effective.

PYTHONSTARTUP="$(python -m jedi repl)" emacs -nw -Q

Message #26 received at 76205 <at> debbugs.gnu.org (full text, mbox):

From: Visuwesh <visuweshm <at> gmail.com>
To: kobarity <kobarity <at> gmail.com>
Cc: liuhui1610 <at> gmail.com, Eli Zaretskii <eliz <at> gnu.org>, 76205 <at> debbugs.gnu.org
Subject: Re: bug#76205: 31.0.50; 31.0.50; python.el completion broken for
 non-builtin objects in IPython 8.30.0 & Python 3.13.2
Date: Thu, 13 Feb 2025 22:34:07 +0530

[வியாழன் பிப்ரவரி 13, 2025] kobarity wrote:

> Visuwesh wrote:
>> 
>> >> One workaround is to disable native completion:
>> >> M-x python-shell-completion-native-turn-off
>> >> It will be a little more inconvenient than the native completion.
>> 
>> Thank you very much, this works good enough for my needs.  I can work in
>> peace now.
>
> Then you might want to customize
> `python-shell-completion-native-enable' or
> `python-shell-completion-native-disabled-interpreters'.

Thanks once again for the helpful tip!

>> > The problem seems to be not related to emacs, because the completion
>> > of non-builtin objects doesn't work in terminal too (IPython 8.32.0
>> > and Python 3.13.2):
>> >
>> > 1. run ipython in a terminal: ipython3 --simple-prompt -i
>> >
>> > 2. type the following code:
>> >    import readline
>> >    import rlcompleter
>> >    readline.parse_and_bind("tab: complete")
>> >    import numpy as np
>> >
>> > 3. type np.<TAB>, there is no completion
>> >
>> >
>> > If I change the completer to others (e.g. jedi):
>> >    from jedi.utils import setup_readline
>> >    setup_readline()
>> >
>> > then the completion works in both terminal and emacs.
>
> Thanks for pointing this out.  So I confirmed that the following
> workaround is also effective.
>
> PYTHONSTARTUP="$(python -m jedi repl)" emacs -nw -Q

So...what do we do from here?  Unfortunately, I know next to nothing
about this area to file a sensible bug report myself.

Message #29 received at 76205 <at> debbugs.gnu.org (full text, mbox):

From: kobarity <kobarity <at> gmail.com>
To: Visuwesh <visuweshm <at> gmail.com>, Liu Hui <liuhui1610 <at> gmail.com>
Cc: 76205 <at> debbugs.gnu.org, Eli Zaretskii <eliz <at> gnu.org>
Subject: Re: bug#76205: 31.0.50; 31.0.50;
 python.el completion broken for non-builtin objects in IPython 8.30.0
 & Python 3.13.2
Date: Sat, 15 Feb 2025 01:13:55 +0900

[Message part 1 (text/plain, inline)]

Visuwesh wrote:
> >> > The problem seems to be not related to emacs, because the completion
> >> > of non-builtin objects doesn't work in terminal too (IPython 8.32.0
> >> > and Python 3.13.2):
> >> >
> >> > 1. run ipython in a terminal: ipython3 --simple-prompt -i
> >> >
> >> > 2. type the following code:
> >> >    import readline
> >> >    import rlcompleter
> >> >    readline.parse_and_bind("tab: complete")
> >> >    import numpy as np
> >> >
> >> > 3. type np.<TAB>, there is no completion
> >> >
> >> >
> >> > If I change the completer to others (e.g. jedi):
> >> >    from jedi.utils import setup_readline
> >> >    setup_readline()
> >> >
> >> > then the completion works in both terminal and emacs.
> >
> > Thanks for pointing this out.  So I confirmed that the following
> > workaround is also effective.
> >
> > PYTHONSTARTUP="$(python -m jedi repl)" emacs -nw -Q
> 
> So...what do we do from here?  Unfortunately, I know next to nothing
> about this area to file a sensible bug report myself.

I'm not sure either.  As ipython --simple-prompt does not support
completion, I don't know if using rlcompleter with ipython
--simple-prompt is supported or not.

The direct cause of this problem seems to be that rlcompleter.__main__
is not as expected.  So the following steps enable completion on
ipython --simple-prompt on Python 3.13.

import readline
import rlcompleter
import __main__
rlcompleter.__main__ = __main__
readline.parse_and_bind("tab: complete")

Attached is a test patch that does something similar to this.

[workaround.patch (application/octet-stream, attachment)]

Message #32 received at 76205 <at> debbugs.gnu.org (full text, mbox):

From: Visuwesh <visuweshm <at> gmail.com>
To: kobarity <kobarity <at> gmail.com>
Cc: Liu Hui <liuhui1610 <at> gmail.com>, Eli Zaretskii <eliz <at> gnu.org>,
 76205 <at> debbugs.gnu.org
Subject: Re: bug#76205: 31.0.50; 31.0.50; python.el completion broken for
 non-builtin objects in IPython 8.30.0 & Python 3.13.2
Date: Sat, 15 Feb 2025 18:04:09 +0530

[சனி பிப்ரவரி 15, 2025] kobarity wrote:

>> [...]
>> So...what do we do from here?  Unfortunately, I know next to nothing
>> about this area to file a sensible bug report myself.
>
> I'm not sure either.  As ipython --simple-prompt does not support
> completion, I don't know if using rlcompleter with ipython
> --simple-prompt is supported or not.
>
> The direct cause of this problem seems to be that rlcompleter.__main__
> is not as expected.  So the following steps enable completion on
> ipython --simple-prompt on Python 3.13.
>
> import readline
> import rlcompleter
> import __main__
> rlcompleter.__main__ = __main__
> readline.parse_and_bind("tab: complete")
>
> Attached is a test patch that does something similar to this.

I can confirm that completion works for non-builtin objects with this
patch.

> diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
> index 8a99ff0434d..2563044c0ee 100644
> --- a/lisp/progmodes/python.el
> +++ b/lisp/progmodes/python.el
> @@ -4534,6 +4534,12 @@ python-shell-completion-native-setup
>              if not is_ipython:
>                  readline.set_completer(new_completer)
>              else:
> +                import sys
> +                import __main__
> +                try:
> +                    sys.modules['rlcompleter'].__main__ = __main__
> +                except KeyError:
> +                    pass
>                  # Try both initializations to cope with all IPython versions.
>                  # This works fine for IPython 3.x but not for earlier:
>                  readline.set_completer(new_completer)

Message #35 received at 76205 <at> debbugs.gnu.org (full text, mbox):

From: kobarity <kobarity <at> gmail.com>
To: Visuwesh <visuweshm <at> gmail.com>, Liu Hui <liuhui1610 <at> gmail.com>
Cc: 76205 <at> debbugs.gnu.org, Eli Zaretskii <eliz <at> gnu.org>
Subject: Re: bug#76205: 31.0.50; 31.0.50;
 python.el completion broken for non-builtin objects in IPython 8.30.0
 & Python 3.13.2
Date: Sun, 16 Feb 2025 19:20:38 +0900

[Message part 1 (text/plain, inline)]

Visuwesh wrote:
> 
> [சனி பிப்ரவரி 15, 2025] kobarity wrote:
> 
> >> [...]
> >> So...what do we do from here?  Unfortunately, I know next to nothing
> >> about this area to file a sensible bug report myself.
> >
> > I'm not sure either.  As ipython --simple-prompt does not support
> > completion, I don't know if using rlcompleter with ipython
> > --simple-prompt is supported or not.
> >
> > The direct cause of this problem seems to be that rlcompleter.__main__
> > is not as expected.  So the following steps enable completion on
> > ipython --simple-prompt on Python 3.13.
> >
> > import readline
> > import rlcompleter
> > import __main__
> > rlcompleter.__main__ = __main__
> > readline.parse_and_bind("tab: complete")
> >
> > Attached is a test patch that does something similar to this.
> 
> I can confirm that completion works for non-builtin objects with this
> patch.

Thank you for testing the patch.

Here's what's happening.  ipython --simple-prompt on Python 3.13 is
indirectly importing rlcompleter, which sets its Completer().complete
as the completer.  Completer() without the namespace argument uses
__main__.__dict__ as the source of global symbols to complete.  The
__main__ here is the name in rlcompleter.py namespace, which is bound
at import time of rlcompleter.py.  However, __main__ used in the REPL
of ipython --simple-prompt is different from __main__ used at the
import time of rlcompleter.py.  Therefore, changes in globals()
(i.e. __main__.__dict__) are not reflected in
rlcompleter.__main__.__dict__ and the completion will fail.

Prior to Python 3.12, the problem does not occur because rlcompleter
was not imported indirectly.  When we import rlcompleter in the native
completion setup code, __main__ is already the same as the one used in
the REPL.

Since IPython does not use rlcompleter, it seems difficult to consider
this problem a bug in IPython.  It would be better if rlcompleter
reads sys.modules["__main__"] instead of __main__, but it would not be
easy to change a long standing specification.  So I think it would be
better for python.el to address this issue.

To set up a completer that works correctly, it is sufficient to
specify the namespace argument:

readline.set_completer(rlcompleter.Completer(globals()).complete)

However, we need to make sure that we do not override the customized
completer.  The problem is that it is difficult to distinguish between
the completer that is set up by importing rlcompleter and the
customized completer.

So it seems to me that it would be reasonable to override
rlcompleter.__main__ so that the default completer works properly.  I
think the attached revised patch is slightly better than my previous
patch, since it does not add "__main__" to globals.  This change
affects only IPython.  I added some comments and the commit message.

I would like to hear your opinions.

[0001-Improve-completion-in-IPython-Python-3.13.patch (application/octet-stream, attachment)]

Message #38 received at 76205 <at> debbugs.gnu.org (full text, mbox):

From: Visuwesh <visuweshm <at> gmail.com>
To: kobarity <kobarity <at> gmail.com>
Cc: Liu Hui <liuhui1610 <at> gmail.com>, Eli Zaretskii <eliz <at> gnu.org>,
 76205 <at> debbugs.gnu.org
Subject: Re: bug#76205: 31.0.50; 31.0.50; python.el completion broken for
 non-builtin objects in IPython 8.30.0 & Python 3.13.2
Date: Sun, 16 Feb 2025 19:32:36 +0530

[ஞாயிறு பிப்ரவரி 16, 2025] kobarity wrote:

> Visuwesh wrote:
>> 
>> [சனி பிப்ரவரி 15, 2025] kobarity wrote:
>> 
>> >> [...]
>> >> So...what do we do from here?  Unfortunately, I know next to nothing
>> >> about this area to file a sensible bug report myself.
>> >
>> > I'm not sure either.  As ipython --simple-prompt does not support
>> > completion, I don't know if using rlcompleter with ipython
>> > --simple-prompt is supported or not.
>> >
>> > The direct cause of this problem seems to be that rlcompleter.__main__
>> > is not as expected.  So the following steps enable completion on
>> > ipython --simple-prompt on Python 3.13.
>> >
>> > import readline
>> > import rlcompleter
>> > import __main__
>> > rlcompleter.__main__ = __main__
>> > readline.parse_and_bind("tab: complete")
>> >
>> > Attached is a test patch that does something similar to this.
>> 
>> I can confirm that completion works for non-builtin objects with this
>> patch.
>
> Thank you for testing the patch.
>
> Here's what's happening.  ipython --simple-prompt on Python 3.13 is
> indirectly importing rlcompleter, which sets its Completer().complete
> as the completer.  Completer() without the namespace argument uses
> __main__.__dict__ as the source of global symbols to complete.  The
> __main__ here is the name in rlcompleter.py namespace, which is bound
> at import time of rlcompleter.py.  However, __main__ used in the REPL
> of ipython --simple-prompt is different from __main__ used at the
> import time of rlcompleter.py.  Therefore, changes in globals()
> (i.e. __main__.__dict__) are not reflected in
> rlcompleter.__main__.__dict__ and the completion will fail.
>
> Prior to Python 3.12, the problem does not occur because rlcompleter
> was not imported indirectly.  When we import rlcompleter in the native
> completion setup code, __main__ is already the same as the one used in
> the REPL.
>
> Since IPython does not use rlcompleter, it seems difficult to consider
> this problem a bug in IPython.  It would be better if rlcompleter
> reads sys.modules["__main__"] instead of __main__, but it would not be
> easy to change a long standing specification.  So I think it would be
> better for python.el to address this issue.
>
> To set up a completer that works correctly, it is sufficient to
> specify the namespace argument:
>
> readline.set_completer(rlcompleter.Completer(globals()).complete)
>
> However, we need to make sure that we do not override the customized
> completer.  The problem is that it is difficult to distinguish between
> the completer that is set up by importing rlcompleter and the
> customized completer.
>
> So it seems to me that it would be reasonable to override
> rlcompleter.__main__ so that the default completer works properly.  I
> think the attached revised patch is slightly better than my previous
> patch, since it does not add "__main__" to globals.  This change
> affects only IPython.  I added some comments and the commit message.

Thanks for the explanation.  The attached patch works on my end too.

> I would like to hear your opinions.

I do not have background knowledge on these parts so I hold no technical
opinions.  Please go ahead with a solution that you're satisfied with.
I am happy as long as the completion works as expected!

Message #41 received at 76205 <at> debbugs.gnu.org (full text, mbox):

From: Liu Hui <liuhui1610 <at> gmail.com>
To: kobarity <kobarity <at> gmail.com>
Cc: 76205 <at> debbugs.gnu.org, Eli Zaretskii <eliz <at> gnu.org>,
 Visuwesh <visuweshm <at> gmail.com>
Subject: Re: bug#76205: 31.0.50; 31.0.50; python.el completion broken for
 non-builtin objects in IPython 8.30.0 & Python 3.13.2
Date: Mon, 17 Feb 2025 13:55:56 +0800

On Sun, Feb 16, 2025 at 6:20 PM kobarity <kobarity <at> gmail.com> wrote:

> Here's what's happening.  ipython --simple-prompt on Python 3.13 is
> indirectly importing rlcompleter

Thanks for the investigation. It seems that pdb imports rlcompleter
since python 3.13 and ipython always imports pdb at startup.

> So it seems to me that it would be reasonable to override
> rlcompleter.__main__ so that the default completer works properly.  I
> think the attached revised patch is slightly better than my previous
> patch, since it does not add "__main__" to globals.  This change
> affects only IPython.  I added some comments and the commit message.
>
> I would like to hear your opinions.

Thank you. I think the fix is good.

BTW, python 3.13 introduces pyrepl and leads to the warning "can't use
pyrepl: terminal doesn't have the required clear capability;
TERM=dumb".

Maybe we should set PYTHON_BASIC_REPL=1 to suppress the warning when
starting the inferior python process?

Message #44 received at 76205 <at> debbugs.gnu.org (full text, mbox):

From: kobarity <kobarity <at> gmail.com>
To: Visuwesh <visuweshm <at> gmail.com>,
	Liu Hui <liuhui1610 <at> gmail.com>
Cc: 76205 <at> debbugs.gnu.org, Eli Zaretskii <eliz <at> gnu.org>
Subject: Re: bug#76205: 31.0.50; 31.0.50;
 python.el completion broken for non-builtin objects in IPython 8.30.0
 & Python 3.13.2
Date: Mon, 17 Feb 2025 23:50:36 +0900

Visuwesh wrote:
> 
> [ஞாயிறு பிப்ரவரி 16, 2025] kobarity wrote:
> 
> > Visuwesh wrote:
> >> 
> >> [சனி பிப்ரவரி 15, 2025] kobarity wrote:
> >> 
> >> >> [...]
> >> >> So...what do we do from here?  Unfortunately, I know next to nothing
> >> >> about this area to file a sensible bug report myself.
> >> >
> >> > I'm not sure either.  As ipython --simple-prompt does not support
> >> > completion, I don't know if using rlcompleter with ipython
> >> > --simple-prompt is supported or not.
> >> >
> >> > The direct cause of this problem seems to be that rlcompleter.__main__
> >> > is not as expected.  So the following steps enable completion on
> >> > ipython --simple-prompt on Python 3.13.
> >> >
> >> > import readline
> >> > import rlcompleter
> >> > import __main__
> >> > rlcompleter.__main__ = __main__
> >> > readline.parse_and_bind("tab: complete")
> >> >
> >> > Attached is a test patch that does something similar to this.
> >> 
> >> I can confirm that completion works for non-builtin objects with this
> >> patch.
> >
> > Thank you for testing the patch.
> >
> > Here's what's happening.  ipython --simple-prompt on Python 3.13 is
> > indirectly importing rlcompleter, which sets its Completer().complete
> > as the completer.  Completer() without the namespace argument uses
> > __main__.__dict__ as the source of global symbols to complete.  The
> > __main__ here is the name in rlcompleter.py namespace, which is bound
> > at import time of rlcompleter.py.  However, __main__ used in the REPL
> > of ipython --simple-prompt is different from __main__ used at the
> > import time of rlcompleter.py.  Therefore, changes in globals()
> > (i.e. __main__.__dict__) are not reflected in
> > rlcompleter.__main__.__dict__ and the completion will fail.
> >
> > Prior to Python 3.12, the problem does not occur because rlcompleter
> > was not imported indirectly.  When we import rlcompleter in the native
> > completion setup code, __main__ is already the same as the one used in
> > the REPL.
> >
> > Since IPython does not use rlcompleter, it seems difficult to consider
> > this problem a bug in IPython.  It would be better if rlcompleter
> > reads sys.modules["__main__"] instead of __main__, but it would not be
> > easy to change a long standing specification.  So I think it would be
> > better for python.el to address this issue.
> >
> > To set up a completer that works correctly, it is sufficient to
> > specify the namespace argument:
> >
> > readline.set_completer(rlcompleter.Completer(globals()).complete)
> >
> > However, we need to make sure that we do not override the customized
> > completer.  The problem is that it is difficult to distinguish between
> > the completer that is set up by importing rlcompleter and the
> > customized completer.
> >
> > So it seems to me that it would be reasonable to override
> > rlcompleter.__main__ so that the default completer works properly.  I
> > think the attached revised patch is slightly better than my previous
> > patch, since it does not add "__main__" to globals.  This change
> > affects only IPython.  I added some comments and the commit message.
> 
> Thanks for the explanation.  The attached patch works on my end too.
> 
> > I would like to hear your opinions.
> 
> I do not have background knowledge on these parts so I hold no technical
> opinions.  Please go ahead with a solution that you're satisfied with.
> I am happy as long as the completion works as expected!

Thanks for testing the patch again.

Liu Hui wrote:
> 
> On Sun, Feb 16, 2025 at 6:20 PM kobarity <kobarity <at> gmail.com> wrote:
> 
> > Here's what's happening.  ipython --simple-prompt on Python 3.13 is
> > indirectly importing rlcompleter
> 
> Thanks for the investigation. It seems that pdb imports rlcompleter
> since python 3.13 and ipython always imports pdb at startup.
> 
> > So it seems to me that it would be reasonable to override
> > rlcompleter.__main__ so that the default completer works properly.  I
> > think the attached revised patch is slightly better than my previous
> > patch, since it does not add "__main__" to globals.  This change
> > affects only IPython.  I added some comments and the commit message.
> >
> > I would like to hear your opinions.
> 
> Thank you. I think the fix is good.

Thanks for your comment.

> BTW, python 3.13 introduces pyrepl and leads to the warning "can't use
> pyrepl: terminal doesn't have the required clear capability;
> TERM=dumb".
> 
> Maybe we should set PYTHON_BASIC_REPL=1 to suppress the warning when
> starting the inferior python process?

Yes, I think that is a good idea.  Could you suggest a patch as a new
bug?

Message #47 received at 76205 <at> debbugs.gnu.org (full text, mbox):

From: Liu Hui <liuhui1610 <at> gmail.com>
To: kobarity <kobarity <at> gmail.com>
Cc: 76205 <at> debbugs.gnu.org, Eli Zaretskii <eliz <at> gnu.org>,
 Visuwesh <visuweshm <at> gmail.com>
Subject: Re: bug#76205: 31.0.50; 31.0.50; python.el completion broken for
 non-builtin objects in IPython 8.30.0 & Python 3.13.2
Date: Tue, 18 Feb 2025 18:00:00 +0800

[Message part 1 (text/plain, inline)]

On Mon, Feb 17, 2025 at 10:50 PM kobarity <kobarity <at> gmail.com> wrote:

> > BTW, python 3.13 introduces pyrepl and leads to the warning "can't use
> > pyrepl: terminal doesn't have the required clear capability;
> > TERM=dumb".
> >
> > Maybe we should set PYTHON_BASIC_REPL=1 to suppress the warning when
> > starting the inferior python process?
>
> Yes, I think that is a good idea.  Could you suggest a patch as a new
> bug?

Here is the patch.

[0001-Disable-PyREPL-in-Python-shell.patch (text/x-patch, attachment)]

Message #50 received at 76205 <at> debbugs.gnu.org (full text, mbox):

From: kobarity <kobarity <at> gmail.com>
To: Liu Hui <liuhui1610 <at> gmail.com>,
	Eli Zaretskii <eliz <at> gnu.org>
Cc: 76205 <at> debbugs.gnu.org, Visuwesh <visuweshm <at> gmail.com>
Subject: Re: bug#76205: 31.0.50; 31.0.50;
 python.el completion broken for non-builtin objects in IPython 8.30.0
 & Python 3.13.2
Date: Tue, 18 Feb 2025 23:21:54 +0900

Liu Hui wrote:
> 
> On Mon, Feb 17, 2025 at 10:50 PM kobarity <kobarity <at> gmail.com> wrote:
> 
> > > BTW, python 3.13 introduces pyrepl and leads to the warning "can't use
> > > pyrepl: terminal doesn't have the required clear capability;
> > > TERM=dumb".
> > >
> > > Maybe we should set PYTHON_BASIC_REPL=1 to suppress the warning when
> > > starting the inferior python process?
> >
> > Yes, I think that is a good idea.  Could you suggest a patch as a new
> > bug?
> 
> Here is the patch.

Thanks.  I confirmed that the patch works as expected.

If there are no other objections or comments, I would like both of the
following patches applied:

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=76205;att=1;msg=35;filename=0001-Improve-completion-in-IPython-Python-3.13.patch
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=76205;att=1;msg=47;filename=0001-Disable-PyREPL-in-Python-shell.patch

Message #53 received at 76205 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Kangas <stefankangas <at> gmail.com>
To: kobarity <kobarity <at> gmail.com>, Liu Hui <liuhui1610 <at> gmail.com>, 
 Eli Zaretskii <eliz <at> gnu.org>
Cc: 76205 <at> debbugs.gnu.org, Visuwesh <visuweshm <at> gmail.com>
Subject: Re: bug#76205: 31.0.50; 31.0.50; python.el completion broken for
 non-builtin objects in IPython 8.30.0 & Python 3.13.2
Date: Tue, 18 Feb 2025 11:05:38 -0800

kobarity <kobarity <at> gmail.com> writes:

> Liu Hui wrote:
>>
>> On Mon, Feb 17, 2025 at 10:50 PM kobarity <kobarity <at> gmail.com> wrote:
>>
>> > > BTW, python 3.13 introduces pyrepl and leads to the warning "can't use
>> > > pyrepl: terminal doesn't have the required clear capability;
>> > > TERM=dumb".
>> > >
>> > > Maybe we should set PYTHON_BASIC_REPL=1 to suppress the warning when
>> > > starting the inferior python process?
>> >
>> > Yes, I think that is a good idea.  Could you suggest a patch as a new
>> > bug?
>>
>> Here is the patch.
>
> Thanks.  I confirmed that the patch works as expected.
>
> If there are no other objections or comments, I would like both of the
> following patches applied:

Thanks, installed on master.  Should the bug therefore be closed?

Message #56 received at 76205 <at> debbugs.gnu.org (full text, mbox):

From: kobarity <kobarity <at> gmail.com>
To: Stefan Kangas <stefankangas <at> gmail.com>
Cc: Liu Hui <liuhui1610 <at> gmail.com>, Eli Zaretskii <eliz <at> gnu.org>,
 Visuwesh <visuweshm <at> gmail.com>, 76205 <at> debbugs.gnu.org
Subject: Re: bug#76205: 31.0.50; 31.0.50;
 python.el completion broken for non-builtin objects in IPython 8.30.0
 & Python 3.13.2
Date: Wed, 19 Feb 2025 23:18:32 +0900

Stefan Kangas wrote:
> 
> kobarity <kobarity <at> gmail.com> writes:
> 
> > Liu Hui wrote:
> >>
> >> On Mon, Feb 17, 2025 at 10:50 PM kobarity <kobarity <at> gmail.com> wrote:
> >>
> >> > > BTW, python 3.13 introduces pyrepl and leads to the warning "can't use
> >> > > pyrepl: terminal doesn't have the required clear capability;
> >> > > TERM=dumb".
> >> > >
> >> > > Maybe we should set PYTHON_BASIC_REPL=1 to suppress the warning when
> >> > > starting the inferior python process?
> >> >
> >> > Yes, I think that is a good idea.  Could you suggest a patch as a new
> >> > bug?
> >>
> >> Here is the patch.
> >
> > Thanks.  I confirmed that the patch works as expected.
> >
> > If there are no other objections or comments, I would like both of the
> > following patches applied:
> 
> Thanks, installed on master.  Should the bug therefore be closed?

Thanks.  Yes, I think this bug can be closed.

Message #61 received at 76205-done <at> debbugs.gnu.org (full text, mbox):

From: Stefan Kangas <stefankangas <at> gmail.com>
To: kobarity <kobarity <at> gmail.com>
Cc: Liu Hui <liuhui1610 <at> gmail.com>, Eli Zaretskii <eliz <at> gnu.org>,
 76205-done <at> debbugs.gnu.org, Visuwesh <visuweshm <at> gmail.com>
Subject: Re: bug#76205: 31.0.50; 31.0.50; python.el completion broken for
 non-builtin objects in IPython 8.30.0 & Python 3.13.2
Date: Wed, 19 Feb 2025 23:18:09 +0000

kobarity <kobarity <at> gmail.com> writes:

> Thanks.  Yes, I think this bug can be closed.

Done.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.