GNU bug report logs - #42670
27.1; comint-redirect-* is quite fragile wrt `comint-prompt-regexp'

Previous Next

To reply to this bug, email your comments to 42670 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Phil Sainty <psainty <at> orcon.net.nz>
To: bug-gnu-emacs <at> gnu.org
Cc: Stefan Monnier <monnier <at> iro.umontreal.ca>, Miles Bader <miles <at> gnu.org>
Subject: 27.1; comint-redirect-* is quite fragile wrt `comint-prompt-regexp'
Date: Sun, 2 Aug 2020 21:26:26 +1200

The comint-redirect-* feature lets you send a command to a comint
process and collect its output in a different buffer (or automatically
into a list), and it uses `comint-prompt-regexp' to figure out when
the command has finished.

`comint-redirect-send-command-to-process' passes `comint-prompt-regexp'
as the `finished-regexp' argument to `comint-redirect-setup', and the
latter sets `comint-redirect-finished-regexp' to that same value.

A problem I've observed is that, by default, *anything* matching the
`comint-prompt-regexp' is stripped from the output by
`comint-redirect-preoutput-filter':

 (and (string-match comint-redirect-finished-regexp filtered-input-string)
      (setq filtered-input-string
	    (replace-match "" nil nil filtered-input-string)))

There's a flag `comint-redirect-insert-matching-regexp' for preventing
matching text from being removed, but it's inactive by default, and
it's all fairly non-obvious without reading code -- outside of its own
docstring, this variable is not mentioned.

While testing https://debbugs.gnu.org/cgi/bugreport.cgi?bug=42662
I noticed how easily this default behaviour gets tripped up.  E.g.:

With recent versions of GNU coreutils 'ls', filenames are shown as
'single-quoted' if they would need to be quoted for use in the shell
(disabled by the -N, --literal option), and so auto-save files are
listed as '#foo#' (with single quotes included).

M-x shell

$ ls --version
=> ls (GNU coreutils) 8.28

$ touch '#foo#'
$ ls -1 '#foo#'
=> '#foo#'

M-: comint-redirect-send-command RET ls -1 '#foo#'
=> foo#'
=> $

The leading '# has been removed from the output because it's a match for
the prompt regexp:

M-: comint-prompt-regexp
"^[^#$%>\n]*[#$%>] *"


Even if that doesn't seem like a practical example, it was one I stumbled
on accidentally while trying to do something else, and it took me a while
to figure out what was going on.

I don't know for sure how fixable this is, but at minimum it can be
documented more clearly as a default pitfall that users need to
understand.  The docstrings do not really discuss the issue, and it's
not mentioned in the manual at all; however the *code* has the
following:

;; NOTE: It is EXTREMELY important that `comint-prompt-regexp' be set to the
;; correct prompt for your interpreter, or that you supply a regexp that says
;; when the redirection is finished.  Otherwise, redirection will continue
;; indefinitely.  The code now does a sanity check to ensure that it can find
;; a prompt in the comint buffer; however, it is still important to ensure that
;; this prompt is set correctly.
;;
;; XXX: This doesn't work so well unless `comint-prompt-regexp' is set;
;; perhaps it should prompt for a terminating string (with an
;; appropriate magic default by examining what we think is the prompt)?
;;
;; Fixme: look for appropriate fields, rather than regexp, if
;; `comint-use-prompt-regexp' is true.

The latter two paragraphs were added at different times.  Regarding the
second-to-last paragraph, I suspect such "magic" would be prone to
failures if the command results in the final prompt not being identical
to the initial prompt.  (Maybe it'd still be a better default, though?)

That later "Fixme" paragraph regarding fields sounds promising, but I
note that the "magic" paragraph was actually part of the commit which
implemented the "fields" support in comint.el in the first place, so
maybe there was some reason why it wasn't utilised for this use-case.

I'm not familiar with the use of these field properties, so I'm hoping
that someone can advise (CCing to Miles and Stefan based on commit
history relating to fields in comint.el.)


-Phil




In GNU Emacs 27.1 (build 1, x86_64-pc-linux-gnu, X toolkit, Xaw3d scroll bars)
 of 2020-08-02 built on shodan
Repository revision: f54ddb0198640e38c1d34bf6031ff5117c117c85
Repository branch: emacs-27
Windowing system distributor 'The X.Org Foundation', version 11.0.12008000
System Description: Ubuntu 18.04.4 LTS

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.

Configured using:
 'configure --prefix=/home/phil/emacs/27.0/usr/local
 --with-x-toolkit=lucid --without-sound'

Configured features:
XAW3D XPM JPEG TIFF GIF PNG RSVG DBUS GSETTINGS GLIB NOTIFY INOTIFY
GNUTLS LIBXML2 FREETYPE HARFBUZZ XFT ZLIB TOOLKIT_SCROLL_BARS LUCID X11
XDBE XIM MODULES THREADS JSON PDUMPER LCMS2 GMP

Important settings:
  value of $LANG: en_NZ.UTF-8
  locale-coding-system: utf-8

Major mode: Dired by name

Minor modes in effect:
  show-paren-mode: t
  minibuffer-depth-indicate-mode: t
  winner-mode: t
  global-hl-line-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rmc puny format-spec rfc822 mml
mml-sec password-cache epa derived epg epg-config gnus-util rmail
rmail-loaddefs text-property-search time-date subr-x seq byte-opt gv
bytecomp byte-compile cconv mm-decode mm-bodies mm-encode mail-parse
rfc2231 mailabbrev gmm-utils mailheader cl-loaddefs cl-lib sendmail
rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils dired-x
easymenu paren mb-depth winner ring hl-line dired dired-loaddefs advice
tooltip eldoc electric uniquify ediff-hook vc-hooks lisp-float-type
mwheel term/x-win x-win term/common-win x-dnd tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode elisp-mode
lisp-mode prog-mode register page tab-bar menu-bar rfn-eshadow isearch
timer select scroll-bar mouse jit-lock font-lock syntax facemenu
font-core term/tty-colors frame minibuffer cl-generic cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese composite charscript charprop
case-table epa-hook jka-cmpr-hook help simple abbrev obarray
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote threads dbusbind
inotify lcms2 dynamic-setting system-font-setting font-render-setting
x-toolkit x multi-tty make-network-process emacs)

Memory information:
((conses 16 49752 8602)
 (symbols 48 6423 1)
 (strings 32 17494 1478)
 (string-bytes 1 553565)
 (vectors 16 9982)
 (vector-slots 8 131176 9148)
 (floats 8 29 25)
 (intervals 56 251 0)
 (buffers 1000 13))

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.