GNU bug report logs - #3735
shell-mode editing, movement, and command entry broken by prompt/output changes

Previous Next

Package: emacs;

Reported by: "Daniel B." <dsb <at> smart.net>

Date: Wed, 1 Jul 2009 19:55:05 UTC

Severity: normal

Tags: moreinfo

Merged with 18135

Fixed in version 29.1

Done: Lars Ingebrigtsen <larsi <at> gnus.org>

Bug is archived. No further changes may be made.

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 3735 in the body.
You can then email your comments to 3735 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to bug-submit-list <at> lists.donarmstrong.com, Emacs Bugs <bug-gnu-emacs <at> gnu.org>:
bug#3735; Package emacs. (Wed, 01 Jul 2009 19:55:05 GMT) Full text and rfc822 format available.

Acknowledgement sent to "Daniel B." <dsb <at> smart.net>:
New bug report received and forwarded. Copy sent to Emacs Bugs <bug-gnu-emacs <at> gnu.org>. (Wed, 01 Jul 2009 19:55:05 GMT) Full text and rfc822 format available.

Message #5 received at submit <at> emacsbugs.donarmstrong.com (full text, mbox):

From: "Daniel B." <dsb <at> smart.net>
To: bug-gnu-emacs <at> gnu.org
Subject: shell-mode editing, movement, and command entry broken by prompt/output
 changes
Date: Wed, 01 Jul 2009 15:17:59 -0400
Introduction:

In shell mode, editing of command output and previous command lines into
new command lines and submission of those new command lines are quite
broken.

For example, editing a previous command's output lines to turn them into
a new command line and then typing RET (executing comint-send-input) to
try to submit it to the shell frequently submits only part of the line,
and C-a and C-e frequently stop somewhere in the middle of the line.

These problems severely reduce the usability of shell mode, which,
before the breakage, was great for editing command output into related
command lines (and editing it in general).

Also, note that the partial submission can be very dangerous.  For
example, trying to edit an output line "one_thing" (e.g., from an
"ls /dir" command) into the command line "rm -rf /dir/one_thing" can
easily result in the command line "rm -rf /dir/" instead.

The problem appears to be because of changes made to distinguish
shell-prompt text from command text (and maybe also changes made to
highlight error lines in command output.  Apparently, boundaries
between runs of text with different attributes cause commands to hit
those boundaries instead of their normal boundaries (e.g., line
boundaries for C-e, C-a, RET (comint-send-input)).

Version information: This problem:
- did not exist in Emacs 20.7.2,
- first appeared somewhere in Emacs 21.x,
- exists in Emacs 22.2.21 (NTEmacs and Debian Etch's emacs22 package,
  version_22.2+2-5), and
- exists in Emacs 22.3 (built from the source in
  http://ftp.gnu.org/pub/gnu/emacs/emacs-22.3.tar.gz as of 2009-06-30).


Details:

Most of the problems occur when editing a line that originally was a
command output line and that is being edited into a new command line,
or when typing RET (to run comint-send-input) on such a line to execute
the command line.

However, some of the problems also occur when editing or when typing RET
on a line that was a previously entered command.

The main subproblems are:
- Typing RET on a line does not submit the whole intended command to
  the shell:
  - It tends to submit only manually inserted text, ignoring original
    text (from command output or a previous command line).
  - Text inserted by yanking/pasting sometimes gets treated differently
    than manually inserted text.  (It tends to get ignored.)
  - Treatment of manually inserted and/or pasted text seems to depend on
    the source of text is it inserted next to.
  - Text inserted by command completion sometimes gets ignored or causes
    other text to be ignored.
- Some of Emacs' most basic movement commands do not work normally (how
  they usually work elsewhere in Emacs) and some do not even work
  consistently in shell mode (do not work the same at different places
  in a line), even accounting for presumably intended behavior changes
  related to distinguishing shell prompts from command text:
  - C-a sometimes does not move to the beginning of the line (for a
    command-output line) or of the command line (for a previous-command
    line), and in some places a second consecutive C-a moves further
    than the first one.
  - Mostly similarly, C-e sometimes does not move to the end of the
    line, and in some places a second C-e moves further.
  - M-f and M-b sometimes move less than to the next word boundary.
  - C-n sometimes moves only forward within an line, not to the next
    line.
- C-k does not work normally.
  - C-k sometimes does not kill to the end of the line.
  - C-k sometimes kills the line break.
- (All of those incorrect movements and killing seem to involve
  boundaries between manually inserted text vs. original text on the
  line (command output or a submitted previous command line).)

Representative Test Cases:

- C1: Inserting text causes previous command output text to be ignored:
  1. Generate command output to use as command line and then to edit
     into later, modified command line:
     1.0 At command prompt in shell mode ...
     1.1 Insert "echo ls".
     1.2 Type RET.
     1.3 (Emacs submits whole command line "echo ls".)
     1.4 (Shell gives output line "ls".)
  2. Use command output as command line:
     2.1 Go up to output line "ls" (C-p).
     2.2 Type RET.
     2.3 Note that Emacs submits whole command line "ls" as expected and
         intended (and as in Emacs 20).
  3. Edit command output into command line:
     3.1 Repeat steps 1.x and 2.1, or go back up to original output line
         from 1.4.
     3.2 Go to end of output line (C-e).
     3.3 Insert " -la".
     3.4 Type RET.
     3.5 PROBLEM: Note that Emacs submits just " -la", instead of whole
         command line "ls -la" (as intended, expected, and in Emacs 20).

  (If you need a more realistic example, use something like "ls" for the
  initial command, and then at the beginning of an output line insert
  something like "ls -l " or "emacs " and then type RET.)

- C2: Pasting differs from manual inserting in previous command input:
  1. Execute command line to re-execute and then edit into modified
     command line to be executed:
     1.0 At command prompt in shell mode ...
     1.1 Insert "echo x".
     1.2 Type RET.
  2. Re-execute previous command line:
     2.1 Go up to previous command line "echo x" (C-p C-p).
     2.2 Type RET.
     2.3 Note that Emacs submits the whole command line "echo x" as
         expected and intended.  (Emacs copies the command text to the
         current command-prompt line at the bottom of the buffer, and
	 the shell prints "x".)
  3. Insert to edit previous command line into modified version:
     3.1 Go up to previous command line "echo x" (C-p C-p).
     3.2 Go to end of line (C-e).
     3.3 Insert "y".
     3.4 Type RET.
     3.5 Note that Emacs submits whole command line "echo xy" as
         expected and intended.
  4. Paste to edit previous command line into modified version:
     4.1 Copy some text from somewhere.  (E.g., move to the "e" in
         "echo", type C-@ (C-SP) (set-mark-command), type C-f, type M-w
	 (command kill-ring-save).)
     4.2 Go to previous command line "echo xy".
     4.3 Go to end of line (C-e).
     4.4 Yank/paste text (C-y).  (E.g., resulting in "echo xye".)
     4.5 Type RET.
     4.6 PROBLEM: Note that Emacs submits nothing, instead of whole
         command line (e.g., "echo xye") (as intended, expected, and in
	 Emacs 20).
     4.7 Go to now-previous command line (e.g., "echo xye").
     4.8 Go to beginning of line.
     4.9 Type RET.
     4.10 PROBLEM: Note that Emacs submits just "echo xy", instead of
         whole command line (e.g., "echo xye") (as intended, expected,
	 and in Emacs 20).  (Also, of course, note that Emacs submitted
	 something different for the exact same line as in step 4.5.)

- C3: C-a, C-e, M-f, M-b don't move normally in edited command line.
  1. Run case C2 to set up.
  2. Move to beginning of unedited previous command line:
     2.1 Go to the end of the previous command line from step 2.1.
     2.2 Type C-a (beginning-of-line).
     2.3 Note that Emacs moves left to the beginning of the command line
         (to the right of the shell prompt text), as apparently
	 intended (in Emacs 21+).
  3. Move to beginning of edited previous command line:
     3.1 Go to the end of the previous command line from step 4.7.
     3.2 Type C-a (beginning-of-line).
     3.3 PROBLEM: Note that Emacs moves left only to the beginning of
         the pasted text, instead of to either the beginning of the
	 the command line (as it does for an unedited command)
	 or the beginning of the line (as it does in Emacs 20).
     3.4 Type C-a again.
     3.5 PROBLEM: Note that Emacs moves further left (to the beginning
         of the command line), showing that C-a is not idempotent as
         it usually is in Emacs.
  4. Move forwards in line by word boundaries:
     4.1 Go to the beginning of the previous command line (see step 3.1).
     4.2 Type M-f (forward-word).
     4.3 Note that Emacs moves to the space in "echo xye", as expected.
     4.4 Type M-f again.
     4.5 PROBLEM: Note that Emacs moves to the "e" in "xye", instead of
         to the newline character after "xye", as expected, as it does
	 in Emacs 20, as it usually does elsewhere.
     4.6 Type M-f a third time.
     4.7 Note that Emacs now moves to the newline character after "xye"
         as expected.
  5. Move backwards in line by word boundaries:
     5.1 Go to the end of the previous command line (see step 3.1).
     5.2 Type M-b (backward-word).
     5.3 PROBLEM: Note that Emacs moves to the "e" in "xye" instead of
         to the "x", as expected, as in Emacs 20, and as it usually does
	 elsewhere.
     5.4 Type M-b again.
     5.5 Note that emacs moves to the "x", as expected.
     5.6 Type M-b a third time.
     5.7 Note that emacs moves to the "e" in "echo", as expected.

- C4: C-k not at end of line kills newline character too in edited
      output line
  1. Execute command line to generate command output to use as command
     arguments:
     1.0 At command prompt in shell mode ...
     1.1 Insert some command that gives several output lines (e.g.,
         "ls").
     1.2 Type RET.
     1.3 (Emacs submits whole command line and shell gives output
         lines.)
  2. Edit command output into command with arguments:
     2.1 Move to beginning of an output line (e.g., C-p, C-a).
     2.2 Insert some command to use edited previous output as
         argument(s) (e.g., "mv ").
     2.3 Type C-k to try to kill to end of line (to copy previous
         command's output line to paste twice back into line and edit
	 into line command arguments).
     2.4 PROBLEM: Emacs kills to end of line PLUS the newline character,
         instead of killing just to the end of line, as it does here in
	 shell mode in Emacs 20 and as it usually does elsewhere.

  (For a full example use consider this:
    1. Execute "ls *.PNG".
    2. Get output line "xyz.PNG".
    3. Move up to output line and try to edit into "mv xyz.PNG xyz.png":
       a. Type C-p C-a.
       b. Insert "mv ".
       c. Type C-k
       d. Type C-y SP C-y.
       e. Edit "PNG" before point(?)/cursor to "png". (Type DEL DEL DEL,
          insert "png".)
	  With the correct C-k behavior, you get "mv xyz.PNG xyz.png";
	  with the broken C-k behavior, you get "mv xyz.PNG" NL
	  " xyx.png".
    3. Submit the edited command line.
       a. Type RET.
          With the correct C-k behavior, you execute
	  "mv xyz.PNG xyx.png";
	  with the broken C-k behavior, you execute "mv xyz.PNG".)

   (Example of danger: Trying to compose something like "mv file1 file2
   dir/" could easily result in "mv file1 file2", destroying the
   previous contents of file2.)


- C5: C-k kills less than to end of line in edited output line.
  1. Do steps 1.x of case C4.
  2. Edit command output into command line.
     2.1 Move to beginning of an output line (e.g., C-p, C-a).
     2.2 Insert some command to use edited previous output as
         argument(s) (e.g., "rm -rf ").
  3. Try to kill command line to yank to command-prompt line.
     3.1 Move to the beginning of the line (e.g., C-a).
     3.2 Type C-k.
     3.3 PROBLEM: Emacs kills (and copies) only to the end of the
         inserted text, not killing the rest of the text on the line,
	 instead of killing to the end of line as it does here in
	 shell mode in Emacs 20 and as it usually does elsewhere.
  (For a full example use consider this:
    1. Execute "ls /backups"
    2. Get some output line "dir_to_delete"
    3. Move up to output line and edit to
       "rm -rf /backups/dir_to_delete":
       a. Type C-p C-a.
       b. insert "rm -rf /backups/".
    4. Try to copy line, paste into command-prompt line, and execute:
       a. Type C-a C-k.
       c. Move to command-prompt line (e.g., M->).
       d. Type C-y.
       e. Type RET.
          With the correct C-k behavior, you execute
	  "rm -rf /backups/dir_to_delete"; with the broken C-k b
	  behavior, you execute "rm -rf /backups/"!)

- C6: Command completion prevents resubmission of previous command line:
  1. Execute "ls /e".  (Get error from shell.)
  2. Go to previous command line.
  3. Type TAB to complete.
  4. Get "ls /etc".
  5. Type RET to execute.
  6. Emacs submits "" instead of while "ls /etc".
  7. Go to beginning of line.
  8. Type RET to execute.
  6. Emacs submits "ls /e" instead of while "ls /etc".

- C7: Variation on C1 and C3:
  1. Execute "echo a ; echo b".
  2. Type C-p C-p C-a; insert "echo ".
  3. Type C-e C-d; insert "x".
  4. Type C-e; type RET.
  5. Emacs submits nothing instead of "echo axb".
  6. Do steps 1 - 4.
  7. Type C-a; type RET.
  8. Emacs submits "x" instead of "echo axb".
  9. Do steps 1 - 4.
  7. Type C-a C-b C-a; type RET.
  8. Emacs submits "echo " instead of "echo axb".

- C8: C-n doesn't go to next line in edited command output:
  Note: I'm having trouble reproducing this one.  The sequence is
  _close_ to this:
  1. Execute "echo A ; echo B ; echo C".
  2. Edit to "echo A B C":
     - Type C-p C-p C-p C-a.
     - Insert "echo ".
     - Replace NLs with space with query-replace.
       (Type M-x, "query-replace", RET, C-q C-j, RET, SP, RET, y, y, n.)
  3. Try to move from edited line to next line.
     - Move up to beginning of edited line (C-p from where
       query-replace ends, then C-a).
     - Try to move down to next line with C-n.
     - When Emacs exhibits the buggy behavior (that is, when whatever
       unidentified steps are executed trigger it), Emacs moves only to
       the space in "echo ", instead of moving down to the next line.
       (Additionally, C-e goes only partway to the end of the line


Additional Use Case and Danger Examples:

- Example use case broken by broken RET behavior:
  1. Execute "ls".
  2. Get some output lines listing files (e.g., "a.txt" and "b.txt").
  3. Edit output into a command operating on the files (e.g., C-p C-p
     C-a, insert "wc " to insert command; C-e C-d SP to gather multiple
     filenames into the command line).
  4. Type RET to try to execute the line ("wc a.txt b.txt").
     With the correct RET behavior, you execute "wc a.txt b.txt";
     with the broken RET behavior, you execute nothing.

  Example of danger:
    Trying to edit output lines "source1" and "source2" to compose and
    execute something like "tar -cf x.tar source1 source2" then
    executing "rm -r source1 source2" could easily result in
    destruction of the not-copied data in source1 and source2.


- Example use case broken by broken C-a behavior:
  1. Execute a command and get output.
  2. Move to an output line and insert a command at the beginning of the
     line.
  3. Join following lines to the current line with C-e C-d.
  4. Decide to modify the command further and try to return to the
     beginning of the line by typing C-a.
     With the correct C-a behavior, you end up at the beginning of the
     line; with the broken C-a behavior, you end up somewhere else.

  Example of danger:
    Starting to compose something like "rm a b" from output lines
    including "a" and "b" and then deciding to edit the command into
    "echo rm a b" could easily result in executing "rm a b".


(Note: The following says 22.3.3 apparently because I built 22.3 three
times.)


In GNU Emacs 22.3.3 (i686-pc-linux-gnu)
 of 2009-07-01 on willy
configured using `configure  '--without-x''

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8
  default-enable-multibyte-characters: t

Major mode: Shell

Minor modes in effect:
  shell-dirtrack-mode: t
  encoded-kbd-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  unify-8859-on-encoding-mode: t
  utf-translate-cjk-mode: t
  auto-compression-mode: t
  line-number-mode: t

Recent input:
ESC x s h e l l RET e c h o SPC ; SPC DEL DEL a SPC
; SPC e c h o SPC b RET C-p C-a C-p e c h o SPC C-e
x C-d RET ESC x r e p o e TAB DEL r t TAB b DEL RET
C-z C-x C-f C-g C-x 2 C-x b RET C-x C-f ESC b ESC b
ESC b ESC b C-k RET f C-n C-n C-n C-@ C-e ESC w C-x
o ESC x r e p e o TAB C-p C-p C-p C-p C-n C-n C-n C-n
e c h o SPC a SPC ; SPC e c h o SPC b RET C-p C-p C-a
e c h o SPC C-e C-d x RET r e p o TAB C-_ C-_ ESC x
r e p TAB e DEL o r TAB RET

Recent messages:
find-file-read-args: Command attempted to use minibuffer while in minibuffer
Quit
Loading dired...done
Mark set
Quit
Completing command name...
Partially completed
Undo! [2 times]
Making completion list...
Loading help-mode...done








Reply sent to Chong Yidong <cyd <at> stupidchicken.com>:
You have taken responsibility. (Sun, 13 Sep 2009 22:15:05 GMT) Full text and rfc822 format available.

Notification sent to "Daniel B." <dsb <at> smart.net>:
bug acknowledged by developer. (Sun, 13 Sep 2009 22:15:06 GMT) Full text and rfc822 format available.

Message #10 received at 3735-done <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Chong Yidong <cyd <at> stupidchicken.com>
To: "Daniel B." <dsb <at> smart.net>
Cc: 3735-done <at> debbugs.gnu.org
Subject: Re: shell-mode editing, movement, and command entry broken by prompt/output changes
Date: Sun, 13 Sep 2009 18:10:26 -0400
> In shell mode, editing of command output and previous command lines into
> new command lines and submission of those new command lines are quite
> broken.
>
> This problem:
> - did not exist in Emacs 20.7.2,
> - first appeared somewhere in Emacs 21.x,
> - exists in Emacs 22.2.21 (NTEmacs and Debian Etch's emacs22 package,
>   version_22.2+2-5), and
> - exists in Emacs 22.3 (built from the source in
>   http://ftp.gnu.org/pub/gnu/emacs/emacs-22.3.tar.gz as of 2009-06-30).

I just checked, and it seems to be fixed in Emacs 23 (I do see the error
in 22.3).  I don't recall who fixed it, though.  Thanks for the bug
report.



bug archived. Request was from Debbugs Internal Request <help-debbugs <at> gnu.org> to internal_control <at> emacsbugs.donarmstrong.com. (Mon, 12 Oct 2009 14:24:14 GMT) Full text and rfc822 format available.

bug unarchived. Request was from Don Hopkins <don <at> DonHopkins.com> to control <at> debbugs.gnu.org. (Mon, 28 Jul 2014 22:38:02 GMT) Full text and rfc822 format available.

Did not alter fixed versions and reopened. Request was from Debbugs Internal Request <help-debbugs <at> gnu.org> to internal_control <at> debbugs.gnu.org. (Mon, 28 Jul 2014 22:40:02 GMT) Full text and rfc822 format available.

Merged 3735 18135. Request was from Glenn Morris <rgm <at> gnu.org> to control <at> debbugs.gnu.org. (Mon, 28 Jul 2014 22:47:02 GMT) Full text and rfc822 format available.

Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#3735; Package emacs. (Sat, 04 Dec 2021 21:16:02 GMT) Full text and rfc822 format available.

Message #21 received at 3735 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Chong Yidong <cyd <at> gnu.org>
Cc: 3735 <at> debbugs.gnu.org, dhopkins <at> donhopkins.com, dsb <at> smart.net,
 18135 <at> debbugs.gnu.org, Don Hopkins <don <at> donhopkins.com>
Subject: Re: bug#3735: shell-mode editing, movement, and command entry
 broken by prompt/output changes
Date: Sat, 04 Dec 2021 22:15:13 +0100
Chong Yidong <cyd <at> gnu.org> writes:

> I was going through old emails and just saw this; sorry for the huge
> delay in triaging this bug.
>
> For what it's worth, I can't reproduce the original bug recipe on Emacs 25.2:
>
> - At command prompt in shell mode ...
> - Insert "echo ls".
> - Type RET.
> - (Emacs submits whole command line "echo ls".)
> - (Shell gives output line "ls".)
> - Go up to output line "ls" (C-p).
> - Go to end of output line (C-e).
> - Insert " -la".
> - Type RET.
>
> The original bug report says
>
>      3.5 PROBLEM: Note that Emacs submits just " -la", instead of whole
>          command line "ls -la" (as intended, expected, and in Emacs 20).
>
> Instead, I see the output of ls -la.

(I'm going through old bug reports that unfortunately weren't resolved
at the time.)

I'm not able to reproduce this problem in Emacs 29, either.

Don, are there any additional steps needed to reproduce the problem
(starting from "emacs -Q")?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no




Added tag(s) moreinfo. Request was from Lars Ingebrigtsen <larsi <at> gnus.org> to control <at> debbugs.gnu.org. (Sat, 04 Dec 2021 21:16:02 GMT) Full text and rfc822 format available.

Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#3735; Package emacs. (Sat, 04 Dec 2021 22:20:02 GMT) Full text and rfc822 format available.

Message #26 received at 3735 <at> debbugs.gnu.org (full text, mbox):

From: Don Hopkins <don <at> donhopkins.com>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: 3735 <at> debbugs.gnu.org, dsb <at> smart.net, Chong Yidong <cyd <at> gnu.org>,
 18135 <at> debbugs.gnu.org, Don Hopkins <don <at> donhopkins.com>
Subject: Re: bug#3735: shell-mode editing, movement, and command entry broken
 by prompt/output changes
Date: Sat, 4 Dec 2021 23:18:57 +0100
Thanks for looking into this! 

I can’t reproduce the problem through that recipe, but weird things still happen in other situations. 

Try putting “ -la” in the kill buffer and then yanking it instead of typing it at the end of the output of “echo ls”. 

It totally ignores the whole line, as if you entered an empty line.

But the thing that usually happens to me is that it enters the text that I yanked, but I’m not sure what the exact conditions to trigger it are. 

But at any rate, yanking “ -la” and hitting return should have the exact same behavior as typing “ -la” and hitting return, so it’s probably caused by the same underlying problem.

I’ll let you know if I figure out other test cases that get it to enter the text you yanked but not the text before that. It usually has something to do with yanking text, possibly at the end or also in the middle of the line.

-Don


> On Dec 4, 2021, at 22:15, Lars Ingebrigtsen <larsi <at> gnus.org> wrote:
> 
> Chong Yidong <cyd <at> gnu.org> writes:
> 
>> I was going through old emails and just saw this; sorry for the huge
>> delay in triaging this bug.
>> 
>> For what it's worth, I can't reproduce the original bug recipe on Emacs 25.2:
>> 
>> - At command prompt in shell mode ...
>> - Insert "echo ls".
>> - Type RET.
>> - (Emacs submits whole command line "echo ls".)
>> - (Shell gives output line "ls".)
>> - Go up to output line "ls" (C-p).
>> - Go to end of output line (C-e).
>> - Insert " -la".
>> - Type RET.
>> 
>> The original bug report says
>> 
>>     3.5 PROBLEM: Note that Emacs submits just " -la", instead of whole
>>         command line "ls -la" (as intended, expected, and in Emacs 20).
>> 
>> Instead, I see the output of ls -la.
> 
> (I'm going through old bug reports that unfortunately weren't resolved
> at the time.)
> 
> I'm not able to reproduce this problem in Emacs 29, either.
> 
> Don, are there any additional steps needed to reproduce the problem
> (starting from "emacs -Q")?
> 
> -- 
> (domestic pets only, the antidote for overdose, milk.)
>   bloggy blog: http://lars.ingebrigtsen.no





Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#3735; Package emacs. (Sun, 05 Dec 2021 18:08:04 GMT) Full text and rfc822 format available.

Message #29 received at 3735 <at> debbugs.gnu.org (full text, mbox):

From: miha <at> kamnitnik.top
To: Don Hopkins <don <at> donhopkins.com>, Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: 3735 <at> debbugs.gnu.org, dsb <at> smart.net, Chong Yidong <cyd <at> gnu.org>,
 18135 <at> debbugs.gnu.org, Don Hopkins <don <at> donhopkins.com>
Subject: Re: bug#18135: bug#3735: shell-mode editing, movement, and command
 entry broken by prompt/output changes
Date: Sun, 05 Dec 2021 17:13:06 +0100
[Message part 1 (text/plain, inline)]
Don Hopkins <don <at> donhopkins.com> writes:

> Thanks for looking into this! 
>
> I can’t reproduce the problem through that recipe, but weird things still happen in other situations. 
>
> Try putting “ -la” in the kill buffer and then yanking it instead of typing it at the end of the output of “echo ls”. 
>
> It totally ignores the whole line, as if you entered an empty line.
>
> But the thing that usually happens to me is that it enters the text that I yanked, but I’m not sure what the exact conditions to trigger it are. 
>
> But at any rate, yanking “ -la” and hitting return should have the exact same behavior as typing “ -la” and hitting return, so it’s probably caused by the same underlying problem.
>
> I’ll let you know if I figure out other test cases that get it to enter the text you yanked but not the text before that. It usually has something to do with yanking text, possibly at the end or also in the middle of the line.

Reproducer:

- copy " -la" into kill ring
- type "echo ls" into M-x shell
- (Shell gives output line "ls".)
- Go up to output line "ls" (C-p).
- yank " -la"
- press C-b
- the line now "ls -la" with point after "l"
- press RET

Only " -la" is sent to the process instead of "ls -la"

This is because the 'field' text property of "ls" is 'output' but the
" -la" is inserted with insert-for-yank and doesn't have a 'field' text
property.

A similar thing happens if we insert a space using M-SPC
(just-one-space) in the middle of process output. The space is inserted
with 'insert' and doesn't inherit the 'field' text property of the
surrounding text.

This didn't cause a problem in older Emacs because the field text
property wasn't used yet at that time in comint.

I am willing to write up a patch to fix this. I'm thinking of having
comint-mode register a function in after-change-functions to mark text
inserted in the 'output' field as 'output'. Would this be okay or are
there any obvious flaws with this approach?
[signature.asc (application/pgp-signature, inline)]

Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#3735; Package emacs. (Sun, 05 Dec 2021 19:28:02 GMT) Full text and rfc822 format available.

Message #32 received at 3735 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: miha <at> kamnitnik.top
Cc: 3735 <at> debbugs.gnu.org, dsb <at> smart.net, Chong Yidong <cyd <at> gnu.org>,
 18135 <at> debbugs.gnu.org, Don Hopkins <don <at> donhopkins.com>
Subject: Re: bug#18135: bug#3735: shell-mode editing, movement, and command
 entry broken by prompt/output changes
Date: Sun, 05 Dec 2021 20:27:16 +0100
miha <at> kamnitnik.top writes:

> Reproducer:
>
> - copy " -la" into kill ring
> - type "echo ls" into M-x shell
> - (Shell gives output line "ls".)
> - Go up to output line "ls" (C-p).
> - yank " -la"
> - press C-b
> - the line now "ls -la" with point after "l"
> - press RET
>
> Only " -la" is sent to the process instead of "ls -la"

Thanks; with that I'm also able to reproduce the problem.

> I am willing to write up a patch to fix this. I'm thinking of having
> comint-mode register a function in after-change-functions to mark text
> inserted in the 'output' field as 'output'. Would this be okay or are
> there any obvious flaws with this approach?

I think that sounds like a promising approach, but I wonder whether
something could be done with rear-nonsticky here.  `field' isn't
rear-sticky here for probably good reasons, but perhaps something could
be tweaked there?  (I haven't actually tried, though.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no




Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#3735; Package emacs. (Tue, 07 Dec 2021 16:53:02 GMT) Full text and rfc822 format available.

Message #35 received at 3735 <at> debbugs.gnu.org (full text, mbox):

From: <miha <at> kamnitnik.top>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: 3735 <at> debbugs.gnu.org, dsb <at> smart.net, Chong Yidong <cyd <at> gnu.org>,
 18135 <at> debbugs.gnu.org, Don Hopkins <don <at> donhopkins.com>
Subject: Re: bug#18135: bug#3735: shell-mode editing, movement, and command
 entry broken by prompt/output changes
Date: Tue, 07 Dec 2021 17:57:34 +0100
[Message part 1 (text/plain, inline)]
Lars Ingebrigtsen <larsi <at> gnus.org> writes:

> miha <at> kamnitnik.top writes:
>
>> Reproducer:
>>
>> - copy " -la" into kill ring
>> - type "echo ls" into M-x shell
>> - (Shell gives output line "ls".)
>> - Go up to output line "ls" (C-p).
>> - yank " -la"
>> - press C-b
>> - the line now "ls -la" with point after "l"
>> - press RET
>>
>> Only " -la" is sent to the process instead of "ls -la"
>
> Thanks; with that I'm also able to reproduce the problem.
>
>> I am willing to write up a patch to fix this. I'm thinking of having
>> comint-mode register a function in after-change-functions to mark text
>> inserted in the 'output' field as 'output'. Would this be okay or are
>> there any obvious flaws with this approach?
>
> I think that sounds like a promising approach, but I wonder whether
> something could be done with rear-nonsticky here. `field' isn't
> rear-sticky here for probably good reasons, but perhaps something
> could be tweaked there? (I haven't actually tried, though.)

Yeah I thought about that too. However, commands such as 'yank' or
'just-one-space' use insert rather than insert-and-inherit and thus
always bypass inheritance of surrounding text properties regardless of
it's stickiness. That is why I settled for using the
'insert-in-front-hooks' text property, patch attached.

[0001-Improve-yanking-in-the-middle-of-comint-process-outp.patch (text/x-patch, inline)]
From 7914fe85ba096f93d4b6817bc0d7b7976f3d316a Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Miha=20Rihtar=C5=A1i=C4=8D?= <miha <at> kamnitnik.top>
Date: Tue, 7 Dec 2021 16:59:39 +0100
Subject: [PATCH] Improve yanking in the middle of comint process output

* lisp/comint.el
(comint--unmark-string-as-output): New function to remove unwanted
properties from text yanked from comint buffers.

(comint-mode): Use it as a 'filter-buffer-substring-function'.

(comint-output-filter): Set 'insert-in-front-hooks' text property on
process output such that text yanked by the user in the middle of
process output is marked as process output (Bug#3735).

(comint--mark-as-output): New function.
(comint--mark-yanked-as-output): New function.
---
 lisp/comint.el | 54 +++++++++++++++++++++++++++++++++++++++++---------
 1 file changed, 45 insertions(+), 9 deletions(-)

diff --git a/lisp/comint.el b/lisp/comint.el
index 544f0b8b82..5f99f560cf 100644
--- a/lisp/comint.el
+++ b/lisp/comint.el
@@ -730,6 +730,8 @@ comint-mode
               (or (file-remote-p default-directory) ""))
   (setq-local comint-accum-marker (make-marker))
   (setq-local font-lock-defaults '(nil t))
+  (add-function :filter-return (local 'filter-buffer-substring-function)
+                #'comint--unmark-string-as-output)
   (add-hook 'change-major-mode-hook 'font-lock-defontify nil t)
   (add-hook 'isearch-mode-hook 'comint-history-isearch-setup nil t)
   (add-hook 'completion-at-point-functions 'comint-completion-at-point nil t)
@@ -1815,7 +1817,8 @@ comint-add-to-input-history
     (ring-insert comint-input-ring cmd)))
 
 (defconst comint--prompt-rear-nonsticky
-  '(field inhibit-line-move-field-capture read-only font-lock-face)
+  '( field inhibit-line-move-field-capture read-only font-lock-face
+     insert-in-front-hooks)
   "Text properties we set on the prompt and don't want to leak past it.")
 
 (defun comint-send-input (&optional no-newline artificial)
@@ -2152,14 +2155,7 @@ comint-output-filter
 	    (goto-char (process-mark process)) ; In case a filter moved it.
 
 	    (unless comint-use-prompt-regexp
-              (with-silent-modifications
-                (add-text-properties comint-last-output-start (point)
-                                     `(rear-nonsticky
-				       ,comint--prompt-rear-nonsticky
-				       front-sticky
-				       (field inhibit-line-move-field-capture)
-				       field output
-				       inhibit-line-move-field-capture t))))
+              (comint--mark-as-output comint-last-output-start (point)))
 
 	    ;; Highlight the prompt, where we define `prompt' to mean
 	    ;; the most recent output that doesn't end with a newline.
@@ -2191,6 +2187,46 @@ comint-output-filter
 	                             ,comint--prompt-rear-nonsticky)))
 	    (goto-char saved-point)))))))
 
+(defun comint--mark-as-output (beg end)
+  (with-silent-modifications
+    (add-text-properties
+     beg end
+     `(rear-nonsticky
+       ,comint--prompt-rear-nonsticky
+       front-sticky
+       (field inhibit-line-move-field-capture)
+       field output
+       inhibit-line-move-field-capture t
+       ;; Text inserted by a user in the middle of process output
+       ;; should be marked as output.  This is needed for commands
+       ;; such as `yank' or `just-one-space' which don't use
+       ;; `insert-and-inherit' and thus bypass default text property
+       ;; inheritance.
+       insert-in-front-hooks
+       (,#'comint--mark-as-output ,#'comint--mark-yanked-as-output)))))
+
+(defun comint--mark-yanked-as-output (beg end)
+  ;; `yank' removes the field text property from the text it inserts
+  ;; due to `yank-excluded-properties', so arrange for this text
+  ;; property to be reapplied in the `after-change-functions'.
+  (let (fun)
+    (setq
+     fun
+     (lambda (beg1 end1 _len1)
+       (remove-hook 'after-change-functions fun t)
+       (when (and (= beg beg1)
+                  (= end end1))
+         (comint--mark-as-output beg1 end1))))
+    (add-hook 'after-change-functions fun nil t)))
+
+(defun comint--unmark-string-as-output (string)
+  (remove-list-of-text-properties
+   0 (length string)
+   '( rear-nonsticky front-sticky field
+      inhibit-line-move-field-capture insert-in-front-hooks)
+   string)
+  string)
+
 (defun comint-preinput-scroll-to-bottom ()
   "Go to the end of buffer in all windows showing it.
 Movement occurs if point in the selected window is not after the process mark,
-- 
2.34.1

[signature.asc (application/pgp-signature, inline)]

Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#3735; Package emacs. (Tue, 07 Dec 2021 20:17:02 GMT) Full text and rfc822 format available.

Message #38 received at 3735 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: <miha <at> kamnitnik.top>
Cc: 3735 <at> debbugs.gnu.org, dsb <at> smart.net, Chong Yidong <cyd <at> gnu.org>,
 18135 <at> debbugs.gnu.org, Don Hopkins <don <at> donhopkins.com>
Subject: Re: bug#18135: bug#3735: shell-mode editing, movement, and command
 entry broken by prompt/output changes
Date: Tue, 07 Dec 2021 21:16:43 +0100
<miha <at> kamnitnik.top> writes:

> Yeah I thought about that too. However, commands such as 'yank' or
> 'just-one-space' use insert rather than insert-and-inherit and thus
> always bypass inheritance of surrounding text properties regardless of
> it's stickiness. That is why I settled for using the
> 'insert-in-front-hooks' text property, patch attached.

Thanks; works well here, too, so I've pushed this to Emacs 29 now.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no




bug marked as fixed in version 29.1, send any further explanations to 18135 <at> debbugs.gnu.org and Don Hopkins <don <at> DonHopkins.com> Request was from Lars Ingebrigtsen <larsi <at> gnus.org> to control <at> debbugs.gnu.org. (Tue, 07 Dec 2021 20:18:02 GMT) Full text and rfc822 format available.

Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#3735; Package emacs. (Wed, 08 Dec 2021 00:21:02 GMT) Full text and rfc822 format available.

Message #43 received at 3735 <at> debbugs.gnu.org (full text, mbox):

From: Don Hopkins <don <at> donhopkins.com>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: 3735 <at> debbugs.gnu.org, dsb <at> smart.net, Chong Yidong <cyd <at> gnu.org>,
 18135 <at> debbugs.gnu.org, miha <at> kamnitnik.top
Subject: Re: bug#18135: bug#3735: shell-mode editing, movement, and command
 entry broken by prompt/output changes
Date: Wed, 8 Dec 2021 01:20:46 +0100
Thank you! I’m looking forward to upgrading to emacs 29. 

-Don


> On Dec 7, 2021, at 21:16, Lars Ingebrigtsen <larsi <at> gnus.org> wrote:
> 
> <miha <at> kamnitnik.top> writes:
> 
>> Yeah I thought about that too. However, commands such as 'yank' or
>> 'just-one-space' use insert rather than insert-and-inherit and thus
>> always bypass inheritance of surrounding text properties regardless of
>> it's stickiness. That is why I settled for using the
>> 'insert-in-front-hooks' text property, patch attached.
> 
> Thanks; works well here, too, so I've pushed this to Emacs 29 now.
> 
> -- 
> (domestic pets only, the antidote for overdose, milk.)
>   bloggy blog: http://lars.ingebrigtsen.no





bug archived. Request was from Debbugs Internal Request <help-debbugs <at> gnu.org> to internal_control <at> debbugs.gnu.org. (Wed, 05 Jan 2022 12:24:05 GMT) Full text and rfc822 format available.

This bug report was last modified 2 years and 73 days ago.

Previous Next


GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.