GNU bug report logs - #78519
30.0.91; In Gnus, when yanking an article, all its newlines become soft

Message received at submit <at> debbugs.gnu.org:

Received: (at submit) by debbugs.gnu.org; 20 May 2025 22:51:34 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue May 20 18:51:34 2025
Received: from localhost ([127.0.0.1]:38445 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1uHVo4-00078c-EG
	for submit <at> debbugs.gnu.org; Tue, 20 May 2025 18:51:34 -0400
Received: from lists.gnu.org ([2001:470:142::17]:49352)
 by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.84_2) (envelope-from <mekeor@HIDDEN>) id 1uHVo0-00077R-NR
 for submit <at> debbugs.gnu.org; Tue, 20 May 2025 18:51:29 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10])
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <mekeor@HIDDEN>) id 1uHVnq-0004v1-Ks
 for bug-gnu-emacs@HIDDEN; Tue, 20 May 2025 18:51:19 -0400
Received: from mout01.posteo.de ([185.67.36.65])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <mekeor@HIDDEN>) id 1uHVnn-0001XL-Jf
 for bug-gnu-emacs@HIDDEN; Tue, 20 May 2025 18:51:17 -0400
Received: from submission (posteo.de [185.67.36.169]) 
 by mout01.posteo.de (Postfix) with ESMTPS id D97D3240027
 for <bug-gnu-emacs@HIDDEN>; Wed, 21 May 2025 00:51:09 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.de; s=2017;
 t=1747781469; bh=I/PIEVlWRJgrOtWPzccQHnvRruhgFUzfVPr3ErBod+Q=;
 h=From:To:Subject:Date:Message-ID:MIME-Version:Content-Type:From;
 b=jbxiU1Snn8AhDVtASKnVYU7ScTafwEg7lXzKvKifLGzDmg4wr+9x+avFXt9f73McA
 8Dcvat9BGNDSsG2pYAiaoz9B1+wOlrPHFTG4Gh2up5JmXP5iTj3uP9IpIbaHQv34ln
 N4bupdnqCqn7vErwkz7b58GdsQq13lk0dYcbO0jBw/foCVpzI+JpoK5k/dzvRkLgns
 7AQAG4+7AjxjXt6VWnZOFbJ85JWoGJk4ZCrKIQTwLEB2PDp0zhCJpvowRodXEqrCn+
 0UOVZr0LpELIcOchFb/WlcrxVx5FuWHyIhYXwqrIHGSlrvSG5goDF1PX0q+fzRLhqc
 P+wa41HrVV9Fg==
Received: from customer (localhost [127.0.0.1])
 by submission (posteo.de) with ESMTPSA id 4b28t50K7pz6twK;
 Wed, 21 May 2025 00:51:09 +0200 (CEST)
From: Mekeor Melire <mekeor@HIDDEN>
To: bug-gnu-emacs@HIDDEN
Subject: 30.0.91; In Gnus, when yanking an article, all its newlines become
 soft
Date: Tue, 20 May 2025 22:50:21 +0000
Message-ID: <87h61eew2a.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="=-=-="
Received-SPF: pass client-ip=185.67.36.65; envelope-from=mekeor@HIDDEN;
 helo=mout01.posteo.de
X-Spam_score_int: -43
X-Spam_score: -4.4
X-Spam_bar: ----
X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,
 DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1,
 RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001,
 RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001,
 SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-Spam-Score: 1.0 (+)
X-Debbugs-Envelope-To: submit
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.0 (/)

--=-=-=
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

GNU Emacs and Gnus support RFC 3676 (obsoleting RFC 2646) =E2=80=9CThe
Text/Plain Format and DelSp Parameters=E2=80=9D as described in (info
"(emacs-mime) Flowed text"). But when replying to an article, or
more specifically, when yanking an article (no matter whether it's
Fixed or Flowed) while composing a message, all yanked newlines
become soft newlines, or more specifically, no newline has the
text-property `hard' set to t. As a consequence, when the composed
message is sent with Format=3DFlowed, it will be encoded wrongly and
its readers will see it terribly displayed.


How to arrive at this misbehavior?
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D

1. Open an article of content-type text/plain with Gnus. Gnus
decodes and inlines the article. If the article has the Format
MIME-parameter set to Flowed, the function `mm-inline-text'
inlines the article and refills it to the column specified per
`fill-flowed-display-column' variable by calling
`fill-flowed'. `fill-flowed' itself can tell which newlines are
soft and which are hard because they are respectively present as
either as the sequence SPC CRLF, or just CRLF without preceding
SPC. But after `fill-flowed' is called, both types of newlines are
just CRLF and even without any text-property.

2. Start composing a reply to the article, e.g. by pressing R,
bound `gnus-article-reply-with-original' in
`gnus-article-mode-map'. Gnus will set up a compose buffer and
yank the original article with citation prefix. Yanking happens by
first calling `gnus-copy-article-buffer', which copies the
original article, from the buffer where it is displayed and read,
to the new composing message buffer while removing all
text-properties. (Then, yanking will continue by adding the
citation prefixes.) The resulting buffer has only soft newlines.

A. Note that this behavior is only a bug if the composed reply has
Format parameter set to Flowed, which Emacs decides to do if at
least once in the buffer, the text-property `hard' is set to
t. See =E2=80=9COn encoding text, regardless of =E2=80=98use-hard-newlines=
=E2=80=99, lines
terminated by soft newline characters are filled together and
wrapped=E2=80=9D at (info "(emacs-mime) Flowed text") and the form (setq
use-hard-newlines (text-property-any (point-min) (point-max) 'hard
't)) in definition of function `mml-generate-mime-1' [2].


What would be the correct behavior?
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

X. In my opinion, we should retain the property that it makes no
difference in effect between first turning on `use-hard-newlines'
and then yanking, or the other way around.

Y. In my opinion, we should continue to use the `hard'
text-property on hard newlines in the composing buffer, because
most filling related commands know about it.

Z. In my opinion, when yanking a Fixed formatted article, all
yanked newlines should become hard or rather be interpreted as
hard, i.e. they should get the `hard' text-property set to t. This
is because it's hard to tell if we can interpret a newline as soft
newline, or if it's really meant to be hard, e.g. because it's a
poem. [3]

Thus, I think that yanking hard newlines from Format=3DFlowed
articles should have `hard' set to t in the compose
buffer. Similarly, any newline yanked from a Fixed [4] article
should have `hard' set to t as well. And the detection described
at (A.), whether Format parameter should be set to Fixed for the
composing message, should be changed, so that it only checks
whether `use-hard-newlines' is non-nil rather than looking for any
`hard' property set to t, because this would conflict with (X.).


How can we approach the proposed correct behavior?
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

=CE=B1. Let's fix `fill-flowed' so that it sets `hard' to t on hard
newlines after refilling. Please find attached a patch for this.

=CE=B2. Let's fix `gnus-copy-article-buffer' so that it keeps `hard'
text-properties on newlines. Please find attached a patch for
this. I'm not sure if it suffices to fix this function, because we
want the `hard' propertized hard newlines in many cases:

  - Content-Type text/plain, without any Format specification; or
    with Format specification set to Fixed or Flowed
  - Content-Type multipart/* where for each text/plain part we
    want proper hard newlines.

=CE=B3. Let's fix (Z) but I'm yet sure how.


Footnotes
=3D=3D=3D=3D=3D=3D=3D=3D=3D

[2] Oops, in my init.el, I have configured
    `message-citation-line-format' to contain a `hard-newline. I
    guess, all my sent reply-messages have Format=3DFlowed?

[3] There might be approximating heuristics for this. But what'd
    be more important is to provide users commands so that they
    can convert hard and soft newlines within a region or
    paragraph back and forth, as desired. But that's future work.

[4] Be it Fixed by explicitly setting Format=3DFixed, or by omitting
    the Format parameter completely.


--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment; filename=fill-flowed.diff

diff --git a/lisp/mail/flow-fill.el b/lisp/mail/flow-fill.el
index 683aa981864..2c22d245ee9 100644
--- a/lisp/mail/flow-fill.el
+++ b/lisp/mail/flow-fill.el
@@ -170,7 +170,8 @@ fill-flowed
 		  (fill-region (line-beginning-position)
                                (line-end-position)
 			       'left 'nosqueeze))))))
-        (forward-line 1)))))
+	(forward-line 1)
+        (put-text-property (1- (point)) (point) 'hard t)))))
 
 (make-obsolete-variable 'fill-flowed-encode-tests nil "27.1")
 (defvar fill-flowed-encode-tests)

--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment; filename=gnus-copy-article-buffer.diff

diff --git a/lisp/gnus/gnus-msg.el b/lisp/gnus/gnus-msg.el
index 807c4e17a33..e4370b43eb3 100644
--- a/lisp/gnus/gnus-msg.el
+++ b/lisp/gnus/gnus-msg.el
@@ -851,10 +851,21 @@ gnus-copy-article-buffer
 	    (gnus-remove-text-with-property 'gnus-prev)
 	    (gnus-remove-text-with-property 'gnus-next)
 	    (gnus-remove-text-with-property 'gnus-decoration)
-	    (insert
-	     (prog1
-		 (buffer-substring-no-properties (point-min) (point-max))
-	       (erase-buffer)))
+            ;; Remove all text-properties except for non-nil `hard' on
+            ;; newlines:
+            (let ((pt (point-min)) (end (point-max)))
+	      (goto-char pt)
+	      (while (search-forward "\n" end 'move)
+		;; Remove all text properties before newline.
+		(set-text-properties pt (match-beginning 0) nil)
+		;; Replace newline with hard or soft newline.
+		(replace-match
+		 (if (get-text-property (match-beginning 0) 'hard)
+		     hard-newline "\n"))
+		;; Start next loop with point after newline.
+		(setq pt (goto-char (match-end 0))))
+	      ;; Remove all text properties after last newline.
+	      (set-text-properties pt (point) nil))
 	    ;; Find the original headers.
 	    (set-buffer gnus-original-article-buffer)
 	    (goto-char (point-min))

--=-=-=--