GNU bug report logs - #13949
24.3.50; `fill-paragraph' should not always put the buffer as modified

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 13949 in the body.
You can then email your comments to 13949 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Dani Moncayo <dmoncayo <at> gmail.com>
To: bug-gnu-emacs <at> gnu.org
Subject: 24.3.50; `fill-paragraph' should not always put the buffer as modified
Date: Wed, 13 Mar 2013 23:09:35 +0100

Recipe from "emacs -Q":

1. Visit some plain text file.
2. Move point to some paragraph with more that one line.
3. M-q C-x C-s
4. M-q

After step #3, the buffer is not modified wrt its file (you've just
save it), but step #4 puts the buffer in a modified state ("**" flag
in the modeline).

This seems a bug, since step #4 didn't make any change in the buffer
contents (the paragraph was already filled).


In GNU Emacs 24.3.50.1 (x86_64-unknown-linux-gnu, GTK+ Version 2.24.13)
 of 2013-03-13 on LEG570
Bzr revision: 112040 kfogel <at> red-bean.com-20130313185405-ibq2um8vj55d4x0a
Windowing system distributor `The X.Org Foundation', version 11.0.11300000
System Description:	Ubuntu 12.10

-- 
Dani Moncayo

Message #8 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Dani Moncayo <dmoncayo <at> gmail.com>
Cc: 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.3.50;
	`fill-paragraph' should not always put the buffer as modified
Date: Thu, 14 Mar 2013 05:43:13 +0200

> Date: Wed, 13 Mar 2013 23:09:35 +0100
> From: Dani Moncayo <dmoncayo <at> gmail.com>
> 
> 1. Visit some plain text file.
> 2. Move point to some paragraph with more that one line.
> 3. M-q C-x C-s
> 4. M-q
> 
> After step #3, the buffer is not modified wrt its file (you've just
> save it), but step #4 puts the buffer in a modified state ("**" flag
> in the modeline).
> 
> This seems a bug, since step #4 didn't make any change in the buffer
> contents (the paragraph was already filled).

Step #4 does change the buffer, because M-q doesn't know whether the
paragraph is already filled, so it fills it anew each time.

Emacs always behaved like that.

Message #11 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Dani Moncayo <dmoncayo <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.3.50; `fill-paragraph' should not always put the
	buffer as modified
Date: Thu, 14 Mar 2013 08:57:35 +0100

>> This seems a bug, since step #4 didn't make any change in the buffer
>> contents (the paragraph was already filled).
>
> Step #4 does change the buffer, because M-q doesn't know whether the
> paragraph is already filled, so it fills it anew each time.
>
> Emacs always behaved like that.

Ah, Ok.

Well, since the `fill-paragraph' command at step #4 leaved the buffer
with the same contents, flagging the buffer as modified was
unnecessary in this case.

In general, I think that a command should flag the buffer as modified
only when the buffer contents at the end of the command were different
from the contents at the beginning of that  same command.

But, I don't know complex is that to implement, and perhaps that
complexity outweighs the benefits.

I'll let you (maintainers) to judge this.  Feel free to close this bug
if the change isn't worth the trouble.

Thanks.

-- 
Dani Moncayo

Message #14 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Andreas Röhler <andreas.roehler <at> easy-emacs.de>
To: bug-gnu-emacs <at> gnu.org
Subject: Re: bug#13949: 24.3.50; `fill-paragraph' should not always put the
	buffer as modified
Date: Thu, 14 Mar 2013 11:27:25 +0100

Am 14.03.2013 08:57, schrieb Dani Moncayo:
>>> This seems a bug, since step #4 didn't make any change in the buffer
>>> contents (the paragraph was already filled).
>>
>> Step #4 does change the buffer, because M-q doesn't know whether the
>> paragraph is already filled, so it fills it anew each time.
>>
>> Emacs always behaved like that.
>
> Ah, Ok.
>
> Well, since the `fill-paragraph' command at step #4 leaved the buffer
> with the same contents, flagging the buffer as modified was
> unnecessary in this case.
>
> In general, I think that a command should flag the buffer as modified
> only when the buffer contents at the end of the command were different
> from the contents at the beginning of that  same command.
>
> But, I don't know complex is that to implement, and perhaps that
> complexity outweighs the benefits.

Hi,

don't think it's complex.
AFAIU the changed-flag presently is set by some commands assumed to change
 the buffer without regard to the result - as shown.

Should not be that complicated to copy the buffers contents into a temporary buffer in this case
and check both buffers afterwards if being equal.

IMHO it's worth to do that change wrt fill-commands.

Best,

Andreas





>
> I'll let you (maintainers) to judge this.  Feel free to close this bug
> if the change isn't worth the trouble.
>
> Thanks.
>

Message #17 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Dani Moncayo <dmoncayo <at> gmail.com>
Cc: Eli Zaretskii <eliz <at> gnu.org>, 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.3.50;
	`fill-paragraph' should not always put the buffer as modified
Date: Thu, 14 Mar 2013 09:38:08 -0400

> Well, since the `fill-paragraph' command at step #4 leaved the buffer
> with the same contents, flagging the buffer as modified was
> unnecessary in this case.

AFAIK there are two ways to go about it:
- compare the sha1 of the paragraph before and after filling and reset
  buffer-modified-p if it shows the text hasn't changed.
- change fill.el so that filling paragraph doesn't just "unfill whole
  paragraph + fill whole paragraph" but instead goes line by line, and
  only modifies the text where there's a need to.

The second option has the advantage that it truly doesn't modify the
buffer (hence, less font-lock work, less redisplay work, and also
text-properties, overlays and markers aren't affected, contrary to the
current behavior), but it requires more coding effort.


        Stefan

Message #20 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Dani Moncayo <dmoncayo <at> gmail.com>
Cc: 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.3.50;
	`fill-paragraph' should not always put the buffer as modified
Date: Thu, 14 Mar 2013 19:46:41 +0200

> Date: Thu, 14 Mar 2013 08:57:35 +0100
> From: Dani Moncayo <dmoncayo <at> gmail.com>
> Cc: 13949 <at> debbugs.gnu.org
> 
> In general, I think that a command should flag the buffer as modified
> only when the buffer contents at the end of the command were different
> from the contents at the beginning of that  same command.

Then your wish is much broader than the original bug report says.
E.g., you'd like the following to leave the buffer marked as
unmodified, right?

  emacs -Q
  M-<
  C-d
  ;

Or how about this:

  emacs -Q
  M-x overwrite-mode RET
  M-<
  ;

fill-paragraph first removes all the newlines from the paragraph, and
then inserts only as many as needed to get a filled paragraph.  So the
buffer gets changed at least twice in the process.

> But, I don't know complex is that to implement, and perhaps that
> complexity outweighs the benefits.

Stefan answered that.  I'll write there why I think it might not be a
good idea.

Message #23 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Andreas Röhler <andreas.roehler <at> easy-emacs.de>
Cc: 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.3.50;
	`fill-paragraph' should not always put the buffer as modified
Date: Thu, 14 Mar 2013 19:50:30 +0200

> Date: Thu, 14 Mar 2013 11:27:25 +0100
> From: Andreas Röhler <andreas.roehler <at> easy-emacs.de>
> 
> AFAIU the changed-flag presently is set by some commands assumed to change
>   the buffer without regard to the result - as shown.

The flag is set by functions that are about to insert, delete, or
replace portions of a buffer.

> Should not be that complicated to copy the buffers contents into a temporary buffer in this case
> and check both buffers afterwards if being equal.
> 
> IMHO it's worth to do that change wrt fill-commands.

IMNSHO, this doesn't make sense.  E.g., what if the buffer is very
large, and there won't be enough memory to have another copy of it?
Even if it is not that large, why move large amounts of data in this
case?

Emacs uses the gap buffer paradigm precisely to avoid this kind of
lossage.  It doesn't make sense to throw that out the window for the
benefit of a minor improvement.

Message #26 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: 13949 <at> debbugs.gnu.org, dmoncayo <at> gmail.com
Subject: Re: bug#13949: 24.3.50;
	`fill-paragraph' should not always put the buffer as modified
Date: Thu, 14 Mar 2013 19:53:41 +0200

> From: Stefan Monnier <monnier <at> iro.umontreal.ca>
> Cc: Eli Zaretskii <eliz <at> gnu.org>,  13949 <at> debbugs.gnu.org
> Date: Thu, 14 Mar 2013 09:38:08 -0400
> 
> > Well, since the `fill-paragraph' command at step #4 leaved the buffer
> > with the same contents, flagging the buffer as modified was
> > unnecessary in this case.
> 
> AFAIK there are two ways to go about it:
> - compare the sha1 of the paragraph before and after filling and reset
>   buffer-modified-p if it shows the text hasn't changed.

This has the disadvantage of scanning the entire buffer, which might
increase paging and memory pressure in general.

> - change fill.el so that filling paragraph doesn't just "unfill whole
>   paragraph + fill whole paragraph" but instead goes line by line, and
>   only modifies the text where there's a need to.

But it sounds like Dani wants this behavior not only for
fill-paragraph, but for any command that can potentially modify the
buffer, but actually doesn't.  This would require to compute sha1
before and after every command that might change the buffer.

Message #29 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Andreas Röhler <andreas.roehler <at> easy-emacs.de>
To: bug-gnu-emacs <at> gnu.org
Subject: Re: bug#13949: 24.3.50; `fill-paragraph' should not always put the
	buffer as modified
Date: Thu, 14 Mar 2013 19:34:40 +0100

Am 14.03.2013 18:53, schrieb Eli Zaretskii:
>> From: Stefan Monnier <monnier <at> iro.umontreal.ca>
>> Cc: Eli Zaretskii <eliz <at> gnu.org>,  13949 <at> debbugs.gnu.org
>> Date: Thu, 14 Mar 2013 09:38:08 -0400
>>
>>> Well, since the `fill-paragraph' command at step #4 leaved the buffer
>>> with the same contents, flagging the buffer as modified was
>>> unnecessary in this case.
>>
>> AFAIK there are two ways to go about it:
>> - compare the sha1 of the paragraph before and after filling and reset
>>    buffer-modified-p if it shows the text hasn't changed.
>
> This has the disadvantage of scanning the entire buffer, which might
> increase paging and memory pressure in general.
>
>> - change fill.el so that filling paragraph doesn't just "unfill whole
>>    paragraph + fill whole paragraph" but instead goes line by line, and
>>    only modifies the text where there's a need to.
>
> But it sounds like Dani wants this behavior not only for
> fill-paragraph, but for any command that can potentially modify the
> buffer, but actually doesn't.  This would require to compute sha1
> before and after every command that might change the buffer.
>
>
>
>

If fill-paragraph is at stake only, store paragraph in a string at beginning and
compare the result should be enough to reset the modify flag if justified.

Andreas

Message #32 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Dani Moncayo <dmoncayo <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.3.50; `fill-paragraph' should not always put the
	buffer as modified
Date: Thu, 14 Mar 2013 19:34:31 +0100

>> In general, I think that a command should flag the buffer as modified
>> only when the buffer contents at the end of the command were different
>> from the contents at the beginning of that  same command.
>
> Then your wish is much broader than the original bug report says.
> E.g., you'd like the following to leave the buffer marked as
> unmodified, right?
>
>   emacs -Q
>   M-<
>   C-d
>   ;
>
> Or how about this:
>
>   emacs -Q
>   M-x overwrite-mode RET
>   M-<
>   ;

These two examples are recipes for modifying the buffer with several
different commands whose global effect is void.  In principle, one
could expect that the buffer be flagged as modified only when the
current text is different from the text in the file.  That makes
sense, and I thought about that, but I agree with you that that
behavior, while desirable, would imply a big impact in performance and
memory consumption, which is not justified at all for such a small
feature.

The expected behavior I described above is not exactly this, though,
because I put it in a command-by-command basis, but anyway, I think
that even then the price to pay would be too high in many cases (one
single command may modify a large portion of text, even the whole
buffer).

> fill-paragraph first removes all the newlines from the paragraph, and
> then inserts only as many as needed to get a filled paragraph.  So the
> buffer gets changed at least twice in the process.

Yes I understood that.  I think that a nicer algorithm would be not to
modify the buffer unless the resulting text is going to be different
to the current one.  But if that would entail too much work, I suggest
to spend your valuable energy in more important things :)


-- 
Dani Moncayo

Message #35 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Andreas Röhler <andreas.roehler <at> easy-emacs.de>
Cc: 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.3.50;
	`fill-paragraph' should not always put the buffer as modified
Date: Thu, 14 Mar 2013 20:49:06 +0200

> Date: Thu, 14 Mar 2013 19:34:40 +0100
> From: Andreas Röhler <andreas.roehler <at> easy-emacs.de>
> 
> If fill-paragraph is at stake only, store paragraph in a string at beginning and
> compare the result should be enough to reset the modify flag if justified.

It is not uncommon to have very large buffers with a single
paragraph.  What you suggest is hardly memory-efficient.

Message #38 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Andreas Röhler <andreas.roehler <at> easy-emacs.de>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.3.50;	`fill-paragraph' should not always put the
	buffer as modified
Date: Thu, 14 Mar 2013 20:01:13 +0100

Am 14.03.2013 19:49, schrieb Eli Zaretskii:
>> Date: Thu, 14 Mar 2013 19:34:40 +0100
>> From: Andreas Röhler <andreas.roehler <at> easy-emacs.de>
>>
>> If fill-paragraph is at stake only, store paragraph in a string at beginning and
>> compare the result should be enough to reset the modify flag if justified.
>
> It is not uncommon to have very large buffers with a single
> paragraph.  What you suggest is hardly memory-efficient.
>

In this case, after checking the length, writing it to a temp-file might be an option.

Message #41 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Andreas Röhler <andreas.roehler <at> easy-emacs.de>
Cc: 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.3.50;
	`fill-paragraph' should not always put the buffer as modified
Date: Thu, 14 Mar 2013 21:19:16 +0200

> Date: Thu, 14 Mar 2013 20:01:13 +0100
> From: Andreas Röhler <andreas.roehler <at> easy-emacs.de>
> CC: 13949 <at> debbugs.gnu.org
> 
> Am 14.03.2013 19:49, schrieb Eli Zaretskii:
> >> Date: Thu, 14 Mar 2013 19:34:40 +0100
> >> From: Andreas Röhler <andreas.roehler <at> easy-emacs.de>
> >>
> >> If fill-paragraph is at stake only, store paragraph in a string at beginning and
> >> compare the result should be enough to reset the modify flag if justified.
> >
> > It is not uncommon to have very large buffers with a single
> > paragraph.  What you suggest is hardly memory-efficient.
> >
> 
> In this case, after checking the length, writing it to a temp-file might be an option.

April 1 is still way too far away to enjoy this.

Message #44 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Andreas Röhler <andreas.roehler <at> easy-emacs.de>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.3.50;	`fill-paragraph' should not always put the
	buffer as modified
Date: Thu, 14 Mar 2013 20:32:16 +0100

> April 1 is still way too far away to enjoy this.
>

And until then let's invent the complicated, until even it's creator
can't fiddle out the worm any more?

Message #47 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 13949 <at> debbugs.gnu.org, dmoncayo <at> gmail.com
Subject: Re: bug#13949: 24.3.50;
	`fill-paragraph' should not always put the buffer as modified
Date: Fri, 15 Mar 2013 00:02:00 -0400

>> AFAIK there are two ways to go about it:
>> - compare the sha1 of the paragraph before and after filling and reset
>> buffer-modified-p if it shows the text hasn't changed.
> This has the disadvantage of scanning the entire buffer, which might
> increase paging and memory pressure in general.

I think you can compute the sha1 of only the paragraph, so that should
be cheap enough that it's not a big issue.

>> - change fill.el so that filling paragraph doesn't just "unfill whole
>> paragraph + fill whole paragraph" but instead goes line by line, and
>> only modifies the text where there's a need to.
> But it sounds like Dani wants this behavior not only for
> fill-paragraph, but for any command that can potentially modify the
> buffer, but actually doesn't.

While I think he'd be happy if we could do that, I didn't take his
request to be so extreme.  So even if we can't handle all cases, it
makes sense to try and improve this one case.



        Stefan

Message #50 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Dani Moncayo <dmoncayo <at> gmail.com>
To: Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: Eli Zaretskii <eliz <at> gnu.org>, 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.3.50; `fill-paragraph' should not always put the
	buffer as modified
Date: Fri, 15 Mar 2013 08:27:58 +0100

>>> - change fill.el so that filling paragraph doesn't just "unfill whole
>>> paragraph + fill whole paragraph" but instead goes line by line, and
>>> only modifies the text where there's a need to.
>> But it sounds like Dani wants this behavior not only for
>> fill-paragraph, but for any command that can potentially modify the
>> buffer, but actually doesn't.
>
> While I think he'd be happy if we could do that, I didn't take his
> request to be so extreme.  So even if we can't handle all cases, it
> makes sense to try and improve this one case.

+1

As I said yesterday in a reply to Eli, modifying the fill algorithm as
you suggest above would be an improvement, I think.


-- 
Dani Moncayo

Message #63 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Jaakov <j_k_v <at> ro.ru>
To: control <at> debbugs.gnu.org, 13949 <at> debbugs.gnu.org
Subject: bug#13949: 24.4.1; `fill-paragraph' should not always put the buffer
 as modified
Date: Tue, 22 Mar 2016 11:50:08 +0100

found 13949 24.4.1
severity 13949 normal
thanks

Dani said:
> fill-paragraph first removes all the newlines from the paragraph, and
> then inserts only as many as needed to get a filled paragraph.  So the
> buffer gets changed at least twice in the process.

This is _how_ it is done, not _what_ is done. Then "what" is described 
in the documentation

https://www.gnu.org/software/emacs/manual/html_node/elisp/Buffer-Modification.html 
:

"Emacs keeps a flag called the modified flag for each buffer, to record 
whether you have changed the text of the buffer. This flag is set to t 
whenever you alter the contents of the buffer, and cleared to nil when 
you save it."

The description of fill-paragraph at

http://www.gnu.org/software/emacs/manual/html_node/emacs/Fill-Commands.html

mentions no exception to the above and "Emacs always behaved like that" 
is just saying that the issue is old.

Since fill-paragraph does not heed the above piece of 
"modified"-flag--documentation, it represents a non-compliance with the 
(informal) specification, i.e., a typical bug.

Therefore, I changed the severity from wishlist to normal.

There are two ways to deal with it: to repair fill-paragraph or to 
repair the documentation.

(A non-related personal aside: since recently, I had to rely both on the 
star in the left lower corner /which means modified/ and paragraph 
filling quite a lot. So the issue really, really bothers me. Of course, 
nobody is forced to repair it if it is just extremely hard to do. We are 
all busy. But I would be extremely happy to see the fill-paragraph 
repaired, at least for text-mode and latex-mode with/without installed 
auctex, if it makes any difference.

Btw., I tend to think that hash computing like sha1 could potentially 
lead to rare, hard-to-reproduce hash clashes, where the text has 
changed, but the sha1 says that the text is the same. If so, 
implementing hash-checking would be worse that the current situation.)

Message #70 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Andreas Röhler <andreas.roehler <at> easy-emacs.de>
To: bug-gnu-emacs <at> gnu.org
Subject: Re: bug#13949: 24.4.1; `fill-paragraph' should not always put the
 buffer as modified
Date: Tue, 22 Mar 2016 12:39:17 +0100

On 22.03.2016 11:50, Jaakov wrote:
> found 13949 24.4.1
> severity 13949 normal
> thanks
>
> Dani said:
> > fill-paragraph first removes all the newlines from the paragraph, and
> > then inserts only as many as needed to get a filled paragraph.  So the
> > buffer gets changed at least twice in the process.
>
> This is _how_ it is done, not _what_ is done. Then "what" is described 
> in the documentation
>
> https://www.gnu.org/software/emacs/manual/html_node/elisp/Buffer-Modification.html 
> :
>
> "Emacs keeps a flag called the modified flag for each buffer, to 
> record whether you have changed the text of the buffer. This flag is 
> set to t whenever you alter the contents of the buffer, and cleared to 
> nil when you save it."
>
> The description of fill-paragraph at
>
> http://www.gnu.org/software/emacs/manual/html_node/emacs/Fill-Commands.html 
>
>
> mentions no exception to the above and "Emacs always behaved like 
> that" is just saying that the issue is old.
>
> Since fill-paragraph does not heed the above piece of 
> "modified"-flag--documentation, it represents a non-compliance with 
> the (informal) specification, i.e., a typical bug.
>
> Therefore, I changed the severity from wishlist to normal.
>
> There are two ways to deal with it: to repair fill-paragraph or to 
> repair the documentation.
>
> (A non-related personal aside: since recently, I had to rely both on 
> the star in the left lower corner /which means modified/ and paragraph 
> filling quite a lot. So the issue really, really bothers me. Of 
> course, nobody is forced to repair it if it is just extremely hard to 
> do. We are all busy. But I would be extremely happy to see the 
> fill-paragraph repaired, at least for text-mode and latex-mode 
> with/without installed auctex, if it makes any difference.
>
> Btw., I tend to think that hash computing like sha1 could potentially 
> lead to rare, hard-to-reproduce hash clashes, where the text has 
> changed, but the sha1 says that the text is the same. If so, 
> implementing hash-checking would be worse that the current situation.)
>
>
>

IMHO fill-paragraph deserves a clean-up, a complete re-write. Initial 
relying at return-values of or-clause looks error-prone.

Message #75 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Jaakov <j_k_v <at> ro.ru>
Cc: 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1;
 `fill-paragraph' should not always put the buffer as modified
Date: Tue, 22 Mar 2016 18:15:33 +0200

> From: Jaakov <j_k_v <at> ro.ru>
> Date: Tue, 22 Mar 2016 11:50:08 +0100
> 
> > fill-paragraph first removes all the newlines from the paragraph, and
> > then inserts only as many as needed to get a filled paragraph.  So the
> > buffer gets changed at least twice in the process.
> 
> This is _how_ it is done, not _what_ is done. Then "what" is described in the documentation
> 
> https://www.gnu.org/software/emacs/manual/html_node/elisp/Buffer-Modification.html :
> 
> "Emacs keeps a flag called the modified flag for each buffer, to record whether you have changed the text of the buffer. This flag is set to t whenever you alter the contents of the buffer, and cleared to nil when you save it."
> 
> The description of fill-paragraph at
> 
> http://www.gnu.org/software/emacs/manual/html_node/emacs/Fill-Commands.html
> 
> mentions no exception to the above and "Emacs always behaved like that" is just saying that the issue is old.
> 
> Since fill-paragraph does not heed the above piece of "modified"-flag--documentation, it represents a non-compliance with the (informal) specification, i.e., a typical bug.
> 
> Therefore, I changed the severity from wishlist to normal.

I disagree.  I think Dani is right: the buffer text is changed (at
least twice), which turns on the modified flag.  This situation is
equivalent to inserting a character and then deleting it: the buffer
stays modified, although its text is identical to the original one.

Therefore, this is still a wishlist request for a new feature, not a
bug.  Patches to implement such a feature are welcome.

Thanks.

Message #78 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Jaakov <j_k_v <at> ro.ru>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1; `fill-paragraph' should not always put the
 buffer as modified
Date: Tue, 22 Mar 2016 18:40:13 +0100

On 03/22/2016 05:15 PM, Eli Zaretskii wrote:
>> From: Jaakov <j_k_v <at> ro.ru>
>> Date: Tue, 22 Mar 2016 11:50:08 +0100
>>
>>> fill-paragraph first removes all the newlines from the paragraph, and
>>> then inserts only as many as needed to get a filled paragraph.  So the
>>> buffer gets changed at least twice in the process.
>>
>> This is _how_ it is done, not _what_ is done. Then "what" is described in the documentation
>>
>> https://www.gnu.org/software/emacs/manual/html_node/elisp/Buffer-Modification.html :
>>
>> "Emacs keeps a flag called the modified flag for each buffer, to record whether you have changed the text of the buffer. This flag is set to t whenever you alter the contents of the buffer, and cleared to nil when you save it."
>>
>> The description of fill-paragraph at
>>
>> http://www.gnu.org/software/emacs/manual/html_node/emacs/Fill-Commands.html
>>
>> mentions no exception to the above and "Emacs always behaved like that" is just saying that the issue is old.
>>
>> Since fill-paragraph does not heed the above piece of "modified"-flag--documentation, it represents a non-compliance with the (informal) specification, i.e., a typical bug.
>>
>> Therefore, I changed the severity from wishlist to normal.
>
> I disagree.  I think Dani is right: the buffer text is changed (at
> least twice), which turns on the modified flag.  This situation is
> equivalent to inserting a character and then deleting it: the buffer
> stays modified, although its text is identical to the original one.
>
Objection for the following reason:
- It's a human who types in and deletes a charter.
- fill-paragraph is not a human, but a routine.

A routine is allowed to do all kinds of strange stuff, e.g., to split a 
long line it into chunks such that each chunk fits into a memory-page, 
process the chunks separately, then recombine the results, inserting, 
deleting, and cutting on need. Or to store all the text lines compressed 
and run clever algorithms on the compressed representation. An ingenious 
(though probably currently nonexistant) LISP interpreter could take your 
implementation of fill-paragraph and automatically convert it into an 
equivalent, lazy routine which modifies each cell of the the buffer zero 
times or once---completely transparent to you as the programmer. 
Probably, the user cannot and should not be aware of any such details.

Was I clear on the distinction of
(1) what is done by a command and
(2) how it is done
?

In my view, changing the buffer twice is an implementation decision (2) 
of fill-paragraph, not a specification/documentation decision (1). In my 
view, the modifies-flag, or, at least, the star in the left lower 
corner, also refers to (1).

If one does nevertheless represent the viewpoint that this 
implementation detail (rewriting the buffer twice) is important and 
should be user-visible, one should probably rewrite the documentation of 
fill-paragraph to reflect this issue.

If one represents the viewpoint that the modifies flag should reflect 
the internal implementation, one should probably implement the 
user-visible buffer-modification in some other way than reading 
modifies-flag. And document it properly.

Message #81 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Jaakov <j_k_v <at> ro.ru>
Cc: 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1; `fill-paragraph' should not always put the
 buffer as modified
Date: Tue, 22 Mar 2016 18:52:14 +0100

Another implementation to fill paragraph in text mode could have as well 
scanned through the text linewise as suggested by Stefan Monnier.

I would implement just line splitting along the following idea for 
text-mode (not in LISP - please excuse me):

FOR EACH line l in the current paragraph
  REMARK space = whitespace or TAB
  c := rightmost column of l with space or -1 if none found
  IF c >= fill-column => 0 THEN
    REPEAT
      c' := c
      c := rightmost column of l|_{[0,c'[} with space
           or -1 if none found
    UNTIL c < fill-column
    IF c<0 THEN
      split line l at column c'
    ELSE
      split line l at column c
    ENDIF
    modifies := true
  END IF
END FOR

Of course, fill-paragraph needs to be much more than line splitting, so 
don't take my terrible piece of code too seriously, i.e., for anything 
except the initial idea.

Message #84 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Michael Heerdegen <michael_heerdegen <at> web.de>
To: Jaakov <j_k_v <at> ro.ru>
Cc: Eli Zaretskii <eliz <at> gnu.org>, 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1;
 `fill-paragraph' should not always put the buffer as modified
Date: Tue, 22 Mar 2016 18:56:00 +0100

Jaakov <j_k_v <at> ro.ru> writes:

> Objection for the following reason:
> - It's a human who types in and deletes a charter.
> - fill-paragraph is not a human, but a routine.

I agree.  Executing one command that has the effect of (1) not changing
the buffer and (2) marking the buffer modified makes no sense to the
user.  It makes sense from the viewpoint of the code, but not from the
viewpoint of the user interface.  It is confusing to the user, that's
not good, so if we could change it, it would be progress, no matter how
this issue is classified in technical terms.

Michael.

Message #87 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Drew Adams <drew.adams <at> oracle.com>
To: Michael Heerdegen <michael_heerdegen <at> web.de>, Jaakov <j_k_v <at> ro.ru>
Cc: 13949 <at> debbugs.gnu.org
Subject: RE: bug#13949: 24.4.1; `fill-paragraph' should not always put the
 buffer as modified
Date: Tue, 22 Mar 2016 11:07:12 -0700 (PDT)

> > Objection for the following reason:
> > - It's a human who types in and deletes a charter.
> > - fill-paragraph is not a human, but a routine.
> 
> I agree.  Executing one command that has the effect of (1) not changing
> the buffer and (2) marking the buffer modified makes no sense to the
> user.  It makes sense from the viewpoint of the code, but not from the
> viewpoint of the user interface.  It is confusing to the user, that's
> not good, so if we could change it, it would be progress, no matter how
> this issue is classified in technical terms.

Yes, _IF_ we can guarantee that there are no differences between
the before and after states, that is, no differences that a user
or Lisp program can discern.

A related question is how to handle certain changes that a user
might be able to detect but that s?he might not want to have
associated with the notion of buffer modification - e.g.,
text-property changes.  Currently, all buffer modifications are
handled the same way in terms of reporting/testing whether modified.

Message #90 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Michael Heerdegen <michael_heerdegen <at> web.de>
To: Drew Adams <drew.adams <at> oracle.com>
Cc: Jaakov <j_k_v <at> ro.ru>, 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1;
 `fill-paragraph' should not always put the buffer as modified
Date: Tue, 22 Mar 2016 19:23:07 +0100

Drew Adams <drew.adams <at> oracle.com> writes:

> Yes, _IF_ we can guarantee that there are no differences between the
> before and after states, that is, no differences that a user or Lisp
> program can discern.

Of course!

And if we do that, I guess we should also remove the head of
`buffer-undo-list' which suffers from the same problem.


Michael.

Message #93 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Jaakov <j_k_v <at> ro.ru>
Cc: 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1; `fill-paragraph' should not always put the
 buffer as modified
Date: Tue, 22 Mar 2016 20:31:28 +0200

> Cc: 13949 <at> debbugs.gnu.org
> From: Jaakov <j_k_v <at> ro.ru>
> Date: Tue, 22 Mar 2016 18:40:13 +0100
> 
> > I disagree.  I think Dani is right: the buffer text is changed (at
> > least twice), which turns on the modified flag.  This situation is
> > equivalent to inserting a character and then deleting it: the buffer
> > stays modified, although its text is identical to the original one.
> >
> Objection for the following reason:
> - It's a human who types in and deletes a charter.
> - fill-paragraph is not a human, but a routine.

We obviously disagree.  But it's pointless to continue the argument,
because no matter whether this is a bug or a feature request, it must
be coded to be fixed.

So once again: patches are welcome, thanks in advance.

Message #96 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Jaakov <j_k_v <at> ro.ru>
To: control <at> debbugs.gnu.org, 13949 <at> debbugs.gnu.org
Date: Tue, 22 Mar 2016 19:40:32 +0100

severity 13949 normal
thanks

Regarding severity: I protest to the previous resetting to minor.

I agree with Drew & Michael: I don't see why code-level arguments should 
have any validity at all. The bug is a manifestation of noncompliance of 
the current behavior with the documentation. A typical, normal bug, not 
a wishlist one. There were no new arguments from Eli, just a statement 
of disagreement.

I do see that a correct algorithm behind fill-paragraph would be not 
completely trivial even for the text mode.

Thus, the bug is not 'minor' (= a problem which doesn't affect the 
package's usefulness, AND is presumably trivial to fix).

I do not consider it 'wishlist' (= for any feature request, and also for 
any bugs that are very difficult to fix due to major design 
considerations), since it's
- neither a feature
- nor very difficult to fix due to major design considerations.

Regarding coding: unfortunately, no patches form my side in the next few 
months. After that, unclear.

Message #101 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Jaakov <j_k_v <at> ro.ru>
Cc: 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1; `fill-paragraph' should not always put the
 buffer as modified
Date: Tue, 22 Mar 2016 19:42:11 +0100

On 03/22/2016 07:31 PM, Eli Zaretskii wrote:
>> Cc: 13949 <at> debbugs.gnu.org
>> From: Jaakov <j_k_v <at> ro.ru>
>> Date: Tue, 22 Mar 2016 18:40:13 +0100
>>
>>> I disagree.  I think Dani is right: the buffer text is changed (at
>>> least twice), which turns on the modified flag.  This situation is
>>> equivalent to inserting a character and then deleting it: the buffer
>>> stays modified, although its text is identical to the original one.
>>>
>> Objection for the following reason:
>> - It's a human who types in and deletes a charter.
>> - fill-paragraph is not a human, but a routine.
>
> We obviously disagree.

I don't consider the previous argument for disagreement valid for the 
mentioned reasons.

Message #106 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Jaakov <j_k_v <at> ro.ru>
Cc: 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: (no subject)
Date: Tue, 22 Mar 2016 20:56:02 +0200

> From: Jaakov <j_k_v <at> ro.ru>
> Date: Tue, 22 Mar 2016 19:40:32 +0100
> 
> severity 13949 normal
> thanks
> 
> Regarding severity: I protest to the previous resetting to minor.

Your protest is respectfully noted.  But please stop changing the
severity.

Message #109 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Jaakov <j_k_v <at> ro.ru>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: (no subject)
Date: Tue, 22 Mar 2016 20:07:17 +0100

On 03/22/2016 07:56 PM, Eli Zaretskii wrote:
>> From: Jaakov <j_k_v <at> ro.ru>
>> Date: Tue, 22 Mar 2016 19:40:32 +0100
>>
>> severity 13949 normal
>> thanks
>>
>> Regarding severity: I protest to the previous resetting to minor.
>
> Your protest is respectfully noted.  But please stop changing the
> severity.
>
I changed it since I consider myself right and you wrong. Obviously, you 
think somehow differently.

I think you just didn't get my point.

Am I being unclear on the principal difference between
(1) _what_ a routine should do and
(2) _how_ it should do it?
?

Message #112 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Jaakov <j_k_v <at> ro.ru>
Cc: 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: (no subject)
Date: Tue, 22 Mar 2016 21:10:48 +0200

> Cc: 13949 <at> debbugs.gnu.org
> From: Jaakov <j_k_v <at> ro.ru>
> Date: Tue, 22 Mar 2016 20:07:17 +0100
> 
> On 03/22/2016 07:56 PM, Eli Zaretskii wrote:
> >> From: Jaakov <j_k_v <at> ro.ru>
> >> Date: Tue, 22 Mar 2016 19:40:32 +0100
> >>
> >> severity 13949 normal
> >> thanks
> >>
> >> Regarding severity: I protest to the previous resetting to minor.
> >
> > Your protest is respectfully noted.  But please stop changing the
> > severity.
> >
> I changed it since I consider myself right and you wrong. Obviously, you 
> think somehow differently.
> 
> I think you just didn't get my point.

I'm getting your point, believe me.

> Am I being unclear on the principal difference between
> (1) _what_ a routine should do and
> (2) _how_ it should do it?
> ?

I understand you, I just don't agree.

Message #115 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Jaakov <j_k_v <at> ro.ru>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: (no subject)
Date: Tue, 22 Mar 2016 20:53:24 +0100

On 03/22/2016 08:10 PM, Eli Zaretskii wrote:
>> Cc: 13949 <at> debbugs.gnu.org
>> From: Jaakov <j_k_v <at> ro.ru>
>> Date: Tue, 22 Mar 2016 20:07:17 +0100
>>
>> On 03/22/2016 07:56 PM, Eli Zaretskii wrote:
>>>> From: Jaakov <j_k_v <at> ro.ru>
>>>> Date: Tue, 22 Mar 2016 19:40:32 +0100
>>>>
>>>> severity 13949 normal
>>>> thanks
>>>>
>>>> Regarding severity: I protest to the previous resetting to minor.
>>>
>>> Your protest is respectfully noted.  But please stop changing the
>>> severity.
>>>
>> I changed it since I consider myself right and you wrong. Obviously, you
>> think somehow differently.
>>
>> I think you just didn't get my point.
>
> I'm getting your point, believe me.
>
>> Am I being unclear on the principal difference between
>> (1) _what_ a routine should do and
>> (2) _how_ it should do it?
>> ?
>
> I understand you, I just don't agree.

Your argument for not agreeing was:

"the buffer text is changed (at least twice), which turns on the 
modified flag."

If you do understand me, please observe that from the viewpoint of (1) 
in the described examples the buffer text is NOT changed, neither once, 
nor twice, not at all.
(Some properties may change, but not the buffer text. Also, the user has 
no practical way to look at the intermediate computation.)

Reason:

In our case, in the view of (1) the term "buffer text is changed" is 
defined, somewhat diffusely, as not "the same contents as the 
corresponding file on the disk".

Source:
"The text displayed in the mode line has the following format:
cs:ch-fr  buf      pos line   (major minor)
...
The next element on the mode line is the string indicated by ch. This 
shows two dashes (‘--’) if the buffer displayed in the window has the 
same contents as the corresponding file on the disk; i.e., if the buffer 
is “unmodified”. If the buffer is modified, it shows two stars (‘**’)."
from
https://www.gnu.org/software/emacs/manual/html_node/emacs/Mode-Line.html#Mode-Line

Therefore, the first part of your argument is invalid.

Am I being clear?

Message #118 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Jaakov <j_k_v <at> ro.ru>
Cc: 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: (no subject)
Date: Tue, 22 Mar 2016 22:07:33 +0200

> Cc: 13949 <at> debbugs.gnu.org
> From: Jaakov <j_k_v <at> ro.ru>
> Date: Tue, 22 Mar 2016 20:53:24 +0100
> 
> >> I think you just didn't get my point.
> >
> > I'm getting your point, believe me.
> >
> >> Am I being unclear on the principal difference between
> >> (1) _what_ a routine should do and
> >> (2) _how_ it should do it?
> >> ?
> >
> > I understand you, I just don't agree.
> 
> Your argument for not agreeing was:
> 
> "the buffer text is changed (at least twice), which turns on the 
> modified flag."
> 
> If you do understand me, please observe that from the viewpoint of (1) 
> in the described examples the buffer text is NOT changed, neither once, 
> nor twice, not at all.
> (Some properties may change, but not the buffer text. Also, the user has 
> no practical way to look at the intermediate computation.)
> 
> Reason:
> 
> In our case, in the view of (1) the term "buffer text is changed" is 
> defined, somewhat diffusely, as not "the same contents as the 
> corresponding file on the disk".
> 
> Source:
> "The text displayed in the mode line has the following format:
> cs:ch-fr  buf      pos line   (major minor)
> ...
> The next element on the mode line is the string indicated by ch. This 
> shows two dashes (‘--’) if the buffer displayed in the window has the 
> same contents as the corresponding file on the disk; i.e., if the buffer 
> is “unmodified”. If the buffer is modified, it shows two stars (‘**’)."
> from
> https://www.gnu.org/software/emacs/manual/html_node/emacs/Mode-Line.html#Mode-Line
> 
> Therefore, the first part of your argument is invalid.
> 
> Am I being clear?

Yes.

But you are entirely missing the point.  I'm not saying anything about
the subject of this report, except this: it's an enhancement request.
Why?  Because (a) the code does exactly what it was designed to do,
not something different; and (b) the effect of what the code does in
this case is not a serious problem, like a crash or inability to do
something important, it is just a minor annoyance.

Therefore, the triage of the bug report as an enhancement request
(a.k.a. "wishlist") is correct.

Please note that I said nothing at all about whether the code should
do something else, or whether the documentation should be corrected to
use a different definition of what the "**" indication means.  This
would be a different argument, and I might even agree with you there.
I'm only talking about the severity value, nothing else.

OK?

Message #121 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Jaakov <j_k_v <at> ro.ru>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: (no subject)
Date: Tue, 22 Mar 2016 22:58:20 +0100

On 03/22/2016 09:07 PM, Eli Zaretskii wrote:
>> Cc: 13949 <at> debbugs.gnu.org
>> From: Jaakov <j_k_v <at> ro.ru>
>> Date: Tue, 22 Mar 2016 20:53:24 +0100
>>
>>>> I think you just didn't get my point.
>>>
>>> I'm getting your point, believe me.
>>>
>>>> Am I being unclear on the principal difference between
>>>> (1) _what_ a routine should do and
>>>> (2) _how_ it should do it?
>>>> ?
>>>
>>> I understand you, I just don't agree.
>>
>> Your argument for not agreeing was:
>>
>> "the buffer text is changed (at least twice), which turns on the
>> modified flag."
>>
>> If you do understand me, please observe that from the viewpoint of (1)
>> in the described examples the buffer text is NOT changed, neither once,
>> nor twice, not at all.
>> (Some properties may change, but not the buffer text. Also, the user has
>> no practical way to look at the intermediate computation.)
>>
>> Reason:
>>
>> In our case, in the view of (1) the term "buffer text is changed" is
>> defined, somewhat diffusely, as not "the same contents as the
>> corresponding file on the disk".
>>
>> Source:
>> "The text displayed in the mode line has the following format:
>> cs:ch-fr  buf      pos line   (major minor)
>> ...
>> The next element on the mode line is the string indicated by ch. This
>> shows two dashes (‘--’) if the buffer displayed in the window has the
>> same contents as the corresponding file on the disk; i.e., if the buffer
>> is “unmodified”. If the buffer is modified, it shows two stars (‘**’)."
>> from
>> https://www.gnu.org/software/emacs/manual/html_node/emacs/Mode-Line.html#Mode-Line
>>
>> Therefore, the first part of your argument is invalid.
>>
>> Am I being clear?
>
> Yes.
>
> But you are entirely missing the point.  I'm not saying anything about
> the subject of this report, except this: it's an enhancement request.
> Why?  Because (a) the code does exactly what it was designed to do,
> not something different; and (b) the effect of what the code does in
> this case is not a serious problem, like a crash or inability to do
> something important, it is just a minor annoyance.
>
> Therefore, the triage of the bug report as an enhancement request
> (a.k.a. "wishlist") is correct.
>
> Please note that I said nothing at all about whether the code should
> do something else, or whether the documentation should be corrected to
> use a different definition of what the "**" indication means.  This
> would be a different argument, and I might even agree with you there.
> I'm only talking about the severity value, nothing else.
>
> OK?
>
I'm puzzled that I have to write the following trivialities, but no.

Objections to your first paragraph:

Please note that your (a) is neither usual nor directly usable: there is 
no easy way to check "what it was designed to do", since the original 
design of ** and fill-paragraph need not be
- available to today users
- relevant to the current state of the evolved software.

So, if you do insist on (a) as a way to differentiate on whether some 
behavior is a bug or a feature, I would like to see this definition on 
the GNU pages, accompanied by a proof that it was there yesterday (e.g. 
a reference to the waybackmachine). And with a description of the 
earlier _designed_ behavior of ** and fill-paragraph.

The right thing to check is not "what it was designed to do", but 
whether fill-paragraph produces "an incorrect or unexpected result, or 
[...] behave[s] in unintended ways." (See 
https://en.wikipedia.org/wiki/Software_bug with today's date.) Our only 
_common_ source of correct, expected, intended behavior descriptions is 
here the documentation of the mode line and fill-paragraph. With regard 
to this source of descriptions, ** and fill-paragraph together have a 
bug, not a feature by the Wikipedia definition which I have no reason to 
disbelieve.

Your (b) "the effect ... is not a serious problem" is absolutely 
_unrelated_ to the GNU wishlist definition: "for any feature request, 
and also for any bugs that are very difficult to fix due to major design 
considerations.", see
https://debbugs.gnu.org/Developer.html#severities

Therefore, I consider the above reasons for triaging as "wishlist" 
flawed both in (a) and in (b).

Is that enlightening?

I am retracting the claim that modifies-flag or fill-properties have 
bugs _alone_: this claim is too imprecise. What does have a bug is the 
combination of ** and M-x fill-paragraph, with respect to the online 
documentation mentioned before.

Please note that I am not claiming that the expectations, intentions, 
correctness statements for ** and fill-paragraph were clearly expressed. 
In fact, they are very diffusely expressed. However, if you downgrade 
this bug report based on the fact of the imprecision of the descriptions 
of the results/behavior, I urge you to downgrade virtually all other bug 
reports about routines with a comparable or worse level of 
documentation. I.e., probably every single report in the emacs bug 
database. If you do not do that, I see no reason to keep the bug report 
downgraded and urge you to upgrade this bug report to 'normal'.

Message #124 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Glenn Morris <rgm <at> gnu.org>
To: Jaakov <j_k_v <at> ro.ru>
Cc: Eli Zaretskii <eliz <at> gnu.org>, 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: (no subject)
Date: Tue, 22 Mar 2016 18:38:27 -0400

Please just accept the word of several Emacs developers that this is a
wishlist item. Arguing about the severity just decreases the time
available to do anything productive, and makes the report
noisier and less useful to read.

Message #127 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: "Petros Travioli" <travioli <at> gmx.de>
To: 13949 <at> debbugs.gnu.org
Subject: `fill-paragraph' should not always put the buffer as modified
Date: Wed, 23 Mar 2016 00:44:16 +0100

Funny that the view at GNU's own page 
https://debbugs.gnu.org/Developer.html#severities
really is oblivious with respect to the seriousness of problems when defining wishlists.

Of course, users cannot be blamed for that.

Btw., the modifying-issue of fill-paragraph affects me, too.

Petros

Message #130 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Jaakov <j_k_v <at> ro.ru>
Cc: 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: (no subject)
Date: Wed, 23 Mar 2016 17:57:17 +0200

> Cc: 13949 <at> debbugs.gnu.org
> From: Jaakov <j_k_v <at> ro.ru>
> Date: Tue, 22 Mar 2016 22:58:20 +0100
> 
> > But you are entirely missing the point.  I'm not saying anything about
> > the subject of this report, except this: it's an enhancement request.
> > Why?  Because (a) the code does exactly what it was designed to do,
> > not something different; and (b) the effect of what the code does in
> > this case is not a serious problem, like a crash or inability to do
> > something important, it is just a minor annoyance.
> >
> > Therefore, the triage of the bug report as an enhancement request
> > (a.k.a. "wishlist") is correct.
> >
> > Please note that I said nothing at all about whether the code should
> > do something else, or whether the documentation should be corrected to
> > use a different definition of what the "**" indication means.  This
> > would be a different argument, and I might even agree with you there.
> > I'm only talking about the severity value, nothing else.
> >
> > OK?
> >
> I'm puzzled that I have to write the following trivialities, but no.
> 
> Objections to your first paragraph:
> 
> Please note that your (a) is neither usual nor directly usable: there is 
> no easy way to check "what it was designed to do", since the original 
> design of ** and fill-paragraph need not be
> - available to today users
> - relevant to the current state of the evolved software.

Of course, there's a way: we, the Emacs developers, know very well how
this was designed.  The buffer-modified indication is set upon each
modification of buffer text, and is reset by saving the buffer to a
file or by a direct manual action, as in M-~.  That is how it's
designed to work, and that's what it does.

> So, if you do insist on (a) as a way to differentiate on whether some 
> behavior is a bug or a feature, I would like to see this definition on 
> the GNU pages, accompanied by a proof that it was there yesterday (e.g. 
> a reference to the waybackmachine). And with a description of the 
> earlier _designed_ behavior of ** and fill-paragraph.

You have it above.  If that's not official enough for you, I'm sorry,
but we don't have enough resources to satisfy your requests for such
documentation (and you don't have any real right to demand it to begin
with).

> The right thing to check is not "what it was designed to do", but 
> whether fill-paragraph produces "an incorrect or unexpected result, or 
> [...] behave[s] in unintended ways." (See 
> https://en.wikipedia.org/wiki/Software_bug with today's date.) Our only 
> _common_ source of correct, expected, intended behavior descriptions is 
> here the documentation of the mode line and fill-paragraph. With regard 
> to this source of descriptions, ** and fill-paragraph together have a 
> bug, not a feature by the Wikipedia definition which I have no reason to 
> disbelieve.

I'm not arguing whether or not there's a bug, I'm arguing only about
its severity level.

> Your (b) "the effect ... is not a serious problem" is absolutely 
> _unrelated_ to the GNU wishlist definition: "for any feature request, 
> and also for any bugs that are very difficult to fix due to major design 
> considerations.", see
> https://debbugs.gnu.org/Developer.html#severities

Wishlist is the only level provided by debbugs which means
"enhancement", so that's what we use.  If there were a better named
value, we would probably use it instead.

> Therefore, I consider the above reasons for triaging as "wishlist" 
> flawed both in (a) and in (b).

I'm not surprised.

> Is that enlightening?

Not really.

> Please note that I am not claiming that the expectations, intentions, 
> correctness statements for ** and fill-paragraph were clearly expressed. 
> In fact, they are very diffusely expressed. However, if you downgrade 
> this bug report based on the fact of the imprecision of the descriptions 
> of the results/behavior, I urge you to downgrade virtually all other bug 
> reports about routines with a comparable or worse level of 
> documentation. I.e., probably every single report in the emacs bug 
> database. If you do not do that, I see no reason to keep the bug report 
> downgraded and urge you to upgrade this bug report to 'normal'.

Sorry, I'm not going to upgrade it.  And I don't really understand why
you keep insisting on that, since the severity has no effect
whatsoever on either the probability that the bug will be fixed, nor
on our willingness to accept patches for that if and when submitted.
IOW, this argument is a complete waste of time and energy.

Message #133 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Glenn Morris <rgm <at> gnu.org>
Cc: j_k_v <at> ro.ru, 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: (no subject)
Date: Wed, 23 Mar 2016 17:57:54 +0200

> From: Glenn Morris <rgm <at> gnu.org>
> Cc: Eli Zaretskii <eliz <at> gnu.org>,  13949 <at> debbugs.gnu.org
> Date: Tue, 22 Mar 2016 18:38:27 -0400
> 
> Please just accept the word of several Emacs developers that this is a
> wishlist item. Arguing about the severity just decreases the time
> available to do anything productive, and makes the report
> noisier and less useful to read.

I couldn't agree more.

Message #136 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Jaakov <j_k_v <at> ro.ru>
Cc: control <at> debbugs.gnu.org, 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: (no subject)
Date: Wed, 23 Mar 2016 18:45:21 +0100

severity 13949 normal
thanks

>> Please just accept the word of several Emacs developers that this is a
>> wishlist item. Arguing about the severity just decreases the time
>> available to do anything productive, and makes the report
>> noisier and less useful to read.
>  I couldn't agree more.

In other words, you disregard your own
https://debbugs.gnu.org/Developer.html#severities
(importance or seriousness is not influencing "wishlist") and are asking 
to accept it unconditionally. Of course not!

I am going to upgrade the severity level this last time but not any 
more, because, apparently, you downgraded it before and have more 
resources to do that.

But, you know, I don't have to
- provide patches or
- donate anything to FSF if you repair that bug. I could have done that.

Keep this bug around with you.

Bye-bye!

Message #143 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: John Wiegley <jwiegley <at> gmail.com>
To: Glenn Morris <rgm <at> gnu.org>
Cc: Jaakov <j_k_v <at> ro.ru>, 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: (no subject)
Date: Sat, 26 Mar 2016 16:33:42 -0700

[Message part 1 (text/plain, inline)]

>>>>> Glenn Morris <rgm <at> gnu.org> writes:

> Please just accept the word of several Emacs developers that this is a
> wishlist item. Arguing about the severity just decreases the time available
> to do anything productive, and makes the report noisier and less useful to
> read.

Jaakov, if you disagree about the setting of a bug's priority, you can appeal
the decision to me, as maintainer. In this case, I agree with Eli, who in
general has complete authority to set priorities as he sees fit, since he is
one of the primary developers who addresses these issues.

I appreciate you bringing the matter to our attention, but further discussion
on the topic of severity is not the best way to spend our collective time.

Happy hacking,
-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

[signature.asc (application/pgp-signature, inline)]

Message #146 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: John Wiegley <jwiegley <at> gmail.com>
To: Jaakov <j_k_v <at> ro.ru>
Cc: 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1;
 `fill-paragraph' should not always put the buffer as modified
Date: Sat, 26 Mar 2016 16:46:49 -0700

severity 13949 wishlist
thanks

>>>>> Jaakov  <j_k_v <at> ro.ru> writes:

> I don't consider the previous argument for disagreement valid for the
> mentioned reasons.

Hi Jaakov,

Eli does not have to convince you of anything: If he wants to work on a bug,
he will; if he doesn't, he won't. Setting the priority does not determine what
our developers decide to work on, it only serves as a general classification.
So you are welcome to express your thoughts on the relevance of a bug, but
there is no cause for argument; also, please stop adjusting bug priorities.

As to your point, I like the distinction you're making.  In fact, one could
imagine a guarding form that could be used by functions like `fill-paragraph':

  (modified-only-if-changed
    FORM)

This would save the current buffer-modification flag, and perform some check
at the end to verify changes were actually made before allowing it to be set
(such as checking the textual content of a filled region for real textual
modifications).

However, while great intellectually, this does have it downsides:

  1. The complexity of our code is increased for a problem that is not severe.
  2. There is a performance cost, especially if the fill region is huge.

So we must ask ourselves: What will fixing this issue actually solve? We'd no
longer modify timestamps when unnecessary, and the user wouldn't feel
compelled to save at times when it is not needed. That is all I can think of.
Therefore, this bug is truly a wishlist item.

I've noticed over the past couple of decades that M-q always sets my modified
flag. It never once occurred to me that this should be considered a problem.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Message #149 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Óscar Fuentes <ofv <at> wanadoo.es>
To: John Wiegley <jwiegley <at> gmail.com>
Cc: Jaakov <j_k_v <at> ro.ru>, 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1;
 `fill-paragraph' should not always put the buffer as modified
Date: Sun, 27 Mar 2016 05:31:19 +0200

John Wiegley <jwiegley <at> gmail.com> writes:

> So we must ask ourselves: What will fixing this issue actually solve? We'd no
> longer modify timestamps when unnecessary, and the user wouldn't feel
> compelled to save at times when it is not needed. That is all I can think of.
> Therefore, this bug is truly a wishlist item.
>
> I've noticed over the past couple of decades that M-q always sets my modified
> flag. It never once occurred to me that this should be considered a problem.

Emacs thinks that the buffer is modified when actually it isn't, and
gives this false information to the user and to itself. So we can't no
longer rely on the modeline indicator to know if the file was modified.

Some features and packages (M-x compile, magit) ask the user when they
are invoked and there are buffers with unsaved changes. Saving a buffer
that doesn't change the file's contents (it just updates the file's
timestamp) may cause undesirable effects, like triggering a lengthy
build of a project. And so on. So it is not true that this "wishlist"
issue has no serious effects. Emacs is well below its usual level of
cleverness here.

Computing hashes of the paragraph (or the whole buffer, if you wish)
before and after the operation and comparing them was suggested. Luckily
it is not complex at all. We already have `secure-hash' which can
operate on whole buffers or ranges. I attach a quick and dirty proof of
concept for fill-paragraph, which should be useful for evaluating
worst-case performance impact. This approach is not enough, as there are
other functions (such as lisp-fill-paragraph) that shows the same
problem. The low level functions which are used by *fill-paragraph are
the ones that should be patched.

To Jaakov: I agree with you that this is a bug, and not a minor one at
that. However, the severity associated to this or any other report is
mostly irrelevant. Solving the problem depends on the existence of
someone who is willing to fix the issue. Almost all contributors here
work on what they decide to work on, and utterly ignore those labels.


diff --git a/lisp/textmodes/fill.el b/lisp/textmodes/fill.el
index 100e2a2..9e1f430 100644
--- a/lisp/textmodes/fill.el
+++ b/lisp/textmodes/fill.el
@@ -804,6 +804,7 @@ fill-paragraph
   (interactive (progn
 		 (barf-if-buffer-read-only)
 		 (list (if current-prefix-arg 'full) t)))
+  (setq h (if (buffer-modified-p) "" (secure-hash 'md5 (current-buffer))))
   (or
    ;; 1. Fill the region if it is active when called interactively.
    (and region transient-mark-mode mark-active
@@ -862,7 +863,10 @@ fill-paragraph
                      ;; fill-region.
                      (fill-region beg end justify)
                    (fill-region-as-paragraph beg end justify))))))
-     fill-pfx)))
+     fill-pfx))
+  (when (and (not (string= h ""))
+             (string= h (secure-hash 'md5 (current-buffer))))
+    (set-buffer-modified-p nil)))
 
 (declare-function comment-search-forward "newcomment" (limit &optional noerror))
 (declare-function comment-string-strip "newcomment" (str beforep afterp))

Message #152 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Andreas Röhler <andreas.roehler <at> easy-emacs.de>
To: bug-gnu-emacs <at> gnu.org
Subject: Re: bug#13949: 24.4.1; `fill-paragraph' should not always put the
 buffer as modified
Date: Sun, 27 Mar 2016 09:44:55 +0200

On 27.03.2016 05:31, Óscar Fuentes wrote:
> John Wiegley <jwiegley <at> gmail.com> writes:
>
>> So we must ask ourselves: What will fixing this issue actually solve? We'd no
>> longer modify timestamps when unnecessary, and the user wouldn't feel
>> compelled to save at times when it is not needed. That is all I can think of.
>> Therefore, this bug is truly a wishlist item.
>>
>> I've noticed over the past couple of decades that M-q always sets my modified
>> flag. It never once occurred to me that this should be considered a problem.
> Emacs thinks that the buffer is modified when actually it isn't, and
> gives this false information to the user and to itself. So we can't no
> longer rely on the modeline indicator to know if the file was modified.
>
> Some features and packages (M-x compile, magit) ask the user when they
> are invoked and there are buffers with unsaved changes. Saving a buffer
> that doesn't change the file's contents (it just updates the file's
> timestamp) may cause undesirable effects, like triggering a lengthy
> build of a project. And so on. So it is not true that this "wishlist"
> issue has no serious effects. Emacs is well below its usual level of
> cleverness here.
>
> Computing hashes of the paragraph (or the whole buffer, if you wish)
> before and after the operation and comparing them was suggested. Luckily
> it is not complex at all. We already have `secure-hash' which can
> operate on whole buffers or ranges. I attach a quick and dirty proof of
> concept for fill-paragraph, which should be useful for evaluating
> worst-case performance impact. This approach is not enough, as there are
> other functions (such as lisp-fill-paragraph) that shows the same
> problem. The low level functions which are used by *fill-paragraph are
> the ones that should be patched.
>
> To Jaakov: I agree with you that this is a bug, and not a minor one at
> that. However, the severity associated to this or any other report is
> mostly irrelevant. Solving the problem depends on the existence of
> someone who is willing to fix the issue. Almost all contributors here
> work on what they decide to work on, and utterly ignore those labels.
>
>
> diff --git a/lisp/textmodes/fill.el b/lisp/textmodes/fill.el
> index 100e2a2..9e1f430 100644
> --- a/lisp/textmodes/fill.el
> +++ b/lisp/textmodes/fill.el
> @@ -804,6 +804,7 @@ fill-paragraph
>     (interactive (progn
>   		 (barf-if-buffer-read-only)
>   		 (list (if current-prefix-arg 'full) t)))
> +  (setq h (if (buffer-modified-p) "" (secure-hash 'md5 (current-buffer))))
>     (or
>      ;; 1. Fill the region if it is active when called interactively.
>      (and region transient-mark-mode mark-active
> @@ -862,7 +863,10 @@ fill-paragraph
>                        ;; fill-region.
>                        (fill-region beg end justify)
>                      (fill-region-as-paragraph beg end justify))))))
> -     fill-pfx)))
> +     fill-pfx))
> +  (when (and (not (string= h ""))
> +             (string= h (secure-hash 'md5 (current-buffer))))
> +    (set-buffer-modified-p nil)))
>   
>   (declare-function comment-search-forward "newcomment" (limit &optional noerror))
>   (declare-function comment-string-strip "newcomment" (str beforep afterp))
>
>
>

Another solution would hash only the paragraph in question, re-format it 
in a temp buffer and replace original content only if changed.

Message #155 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Andreas Schwab <schwab <at> linux-m68k.org>
To: Óscar Fuentes <ofv <at> wanadoo.es>
Cc: John Wiegley <jwiegley <at> gmail.com>, Jaakov <j_k_v <at> ro.ru>,
 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1;
 `fill-paragraph' should not always put the buffer as modified
Date: Sun, 27 Mar 2016 10:42:23 +0200

Óscar Fuentes <ofv <at> wanadoo.es> writes:

> Emacs thinks that the buffer is modified when actually it isn't,

Emacs correctly tells you that the buffer has been modified.  That the
buffer's contents now match the file's contents is irrelevant.

Andreas.

-- 
Andreas Schwab, schwab <at> linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Message #158 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Óscar Fuentes <ofv <at> wanadoo.es>
Cc: jwiegley <at> gmail.com, j_k_v <at> ro.ru, 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1;
 `fill-paragraph' should not always put the buffer as modified
Date: Sun, 27 Mar 2016 17:56:26 +0300

> From: Óscar Fuentes <ofv <at> wanadoo.es>
> Date: Sun, 27 Mar 2016 05:31:19 +0200
> Cc: Jaakov <j_k_v <at> ro.ru>, 13949 <at> debbugs.gnu.org
> 
> diff --git a/lisp/textmodes/fill.el b/lisp/textmodes/fill.el
> index 100e2a2..9e1f430 100644
> --- a/lisp/textmodes/fill.el
> +++ b/lisp/textmodes/fill.el
> @@ -804,6 +804,7 @@ fill-paragraph
>    (interactive (progn
>  		 (barf-if-buffer-read-only)
>  		 (list (if current-prefix-arg 'full) t)))
> +  (setq h (if (buffer-modified-p) "" (secure-hash 'md5 (current-buffer))))
>    (or
>     ;; 1. Fill the region if it is active when called interactively.
>     (and region transient-mark-mode mark-active
> @@ -862,7 +863,10 @@ fill-paragraph
>                       ;; fill-region.
>                       (fill-region beg end justify)
>                     (fill-region-as-paragraph beg end justify))))))
> -     fill-pfx)))
> +     fill-pfx))
> +  (when (and (not (string= h ""))
> +             (string= h (secure-hash 'md5 (current-buffer))))
> +    (set-buffer-modified-p nil)))

Thanks, but I'm not sure computing the hash is enough: the functions
involved in refilling can change text properties, so the test should
also account for that.

Message #161 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Óscar Fuentes <ofv <at> wanadoo.es>
To: Andreas Schwab <schwab <at> linux-m68k.org>
Cc: John Wiegley <jwiegley <at> gmail.com>, Jaakov <j_k_v <at> ro.ru>,
 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1;
 `fill-paragraph' should not always put the buffer as modified
Date: Sun, 27 Mar 2016 16:59:03 +0200

Andreas Schwab <schwab <at> linux-m68k.org> writes:

>> Emacs thinks that the buffer is modified when actually it isn't,
>
> Emacs correctly tells you that the buffer has been modified.  That the
> buffer's contents now match the file's contents is irrelevant.

It is very relevant for me and, judging by the comments of others on
this and bug report and its duplicates, it is relevant for them too.

The tools must be adapted to the mental model of their users.

Message #164 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Óscar Fuentes <ofv <at> wanadoo.es>
To: Andreas Röhler <andreas.roehler <at> easy-emacs.de>
Cc: 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1;
 `fill-paragraph' should not always put the buffer as modified
Date: Sun, 27 Mar 2016 17:09:12 +0200

Andreas Röhler <andreas.roehler <at> easy-emacs.de> writes:

> Another solution would hash only the paragraph in question, re-format
> it in a temp buffer and replace original content only if changed.

If we know the paragraph's begin and end points, we can hash just that
range and then there is no need of a temporary buffer. I think that this
condition is met on all cases.

Anyway, I've made some experiments and hashing a 160 MB file on a 8 year
old 2.4 GHz 64 bit workstation takes 2.4 seconds.

Message #167 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Drew Adams <drew.adams <at> oracle.com>
To: Andreas Schwab <schwab <at> linux-m68k.org>, Óscar Fuentes
 <ofv <at> wanadoo.es>
Cc: John Wiegley <jwiegley <at> gmail.com>, Jaakov <j_k_v <at> ro.ru>,
 13949 <at> debbugs.gnu.org
Subject: RE: bug#13949: 24.4.1; `fill-paragraph' should not always put the
 buffer as modified
Date: Sun, 27 Mar 2016 08:13:05 -0700 (PDT)

> > Emacs thinks that the buffer is modified when actually it isn't,
> 
> Emacs correctly tells you that the buffer has been modified.  That the
> buffer's contents now match the file's contents is irrelevant.

"Irrelevant" is in the eye of the beholder.  What might be
relevant to given code or to a given user in a given context
can be irrelevant to another.

I agree that the current, fine-grained indication of whether
any kind of changes have been made to the buffer is important
not to lose.  But it could also be useful to layer on top of
this indications of other levels - particular kinds - of buffer
changes.

IOW, one-size-fits-all is not ideal, but we definitely do not
want to lose the finest-grain indication, which is what we
have now.  Emacs should provide more help - more kinds of
change indication - both for code and interactively, for users.

Message #170 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Drew Adams <drew.adams <at> oracle.com>
To: Óscar Fuentes <ofv <at> wanadoo.es>, Andreas Schwab
 <schwab <at> linux-m68k.org>
Cc: John Wiegley <jwiegley <at> gmail.com>, Jaakov <j_k_v <at> ro.ru>,
 13949 <at> debbugs.gnu.org
Subject: RE: bug#13949: 24.4.1; `fill-paragraph' should not always put the
 buffer as modified
Date: Sun, 27 Mar 2016 08:15:47 -0700 (PDT)

> >> Emacs thinks that the buffer is modified when actually it isn't,
> >
> > Emacs correctly tells you that the buffer has been modified.  That the
> > buffer's contents now match the file's contents is irrelevant.
> 
> It is very relevant for me and, judging by the comments of others on
> this and bug report and its duplicates, it is relevant for them too.
> 
> The tools must be adapted to the mental model of their users.

Yes, and:

1. Different users have different use cases and different mental
   models.

2. The same user has different mental models, depending on the
   current context.

3. Code too, not just users, needs to be able to detect buffer
   changes.

4. There are different kinds of buffer changes.  It would be
   good to have ways to detect these as such.

Message #173 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Óscar Fuentes <ofv <at> wanadoo.es>
To: Drew Adams <drew.adams <at> oracle.com>
Cc: John Wiegley <jwiegley <at> gmail.com>, Jaakov <j_k_v <at> ro.ru>,
 13949 <at> debbugs.gnu.org, Andreas Schwab <schwab <at> linux-m68k.org>
Subject: Re: bug#13949: 24.4.1;
 `fill-paragraph' should not always put the buffer as modified
Date: Sun, 27 Mar 2016 17:21:47 +0200

Drew Adams <drew.adams <at> oracle.com> writes:

>> The tools must be adapted to the mental model of their users.
>
> Yes, and:
>
> 1. Different users have different use cases and different mental
>    models.
>
> 2. The same user has different mental models, depending on the
>    current context.
>
> 3. Code too, not just users, needs to be able to detect buffer
>    changes.
>
> 4. There are different kinds of buffer changes.  It would be
>    good to have ways to detect these as such.

Please provide an example were the user would benefit from marking the
buffer as modified after applying a fill-paragraph operation that
doesn't make a difference on the buffer contents.

Message #176 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Dmitry Gutov <dgutov <at> yandex.ru>
To: Óscar Fuentes <ofv <at> wanadoo.es>,
 John Wiegley <jwiegley <at> gmail.com>
Cc: Jaakov <j_k_v <at> ro.ru>, 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1; `fill-paragraph' should not always put the
 buffer as modified
Date: Sun, 27 Mar 2016 18:28:22 +0300

On 03/27/2016 06:31 AM, Óscar Fuentes wrote:

> +  (when (and (not (string= h ""))
> +             (string= h (secure-hash 'md5 (current-buffer))))
> +    (set-buffer-modified-p nil)))

Hashes have collisions (and md5 is a bit famous for them). So in 
principle, I don't think using a hash is a good choice in this case.

Either way, you'd have to keep the original string around, to compare 
against if the hashes match.

Message #179 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Óscar Fuentes <ofv <at> wanadoo.es>
To: bug-gnu-emacs <at> gnu.org
Subject: Re: bug#13949: 24.4.1;
 `fill-paragraph' should not always put the buffer as modified
Date: Sun, 27 Mar 2016 17:28:42 +0200

Eli Zaretskii <eliz <at> gnu.org> writes:

> Thanks, but I'm not sure computing the hash is enough: the functions
> involved in refilling can change text properties, so the test should
> also account for that.

The docs say:


  This shows two dashes (‘--’) if the buffer displayed in the window has
  the same contents as the corresponding file on the disk; i.e., if the
  buffer is unmodified.



AFAIK, a file-visiting buffer is not marked as modified when text
properties are applied to it. This makes sense, because text properties
are not part of the contents of the corresponding file.

Message #182 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Óscar Fuentes <ofv <at> wanadoo.es>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: jwiegley <at> gmail.com, j_k_v <at> ro.ru, 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1;
 `fill-paragraph' should not always put the buffer as modified
Date: Sun, 27 Mar 2016 17:29:36 +0200

Eli Zaretskii <eliz <at> gnu.org> writes:

> Thanks, but I'm not sure computing the hash is enough: the functions
> involved in refilling can change text properties, so the test should
> also account for that.

The docs say:


  This shows two dashes (‘--’) if the buffer displayed in the window has
  the same contents as the corresponding file on the disk; i.e., if the
  buffer is unmodified.



AFAIK, a file-visiting buffer is not marked as modified when text
properties are applied to it. This makes sense, because text properties
are not part of the contents of the corresponding file.

Message #185 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Óscar Fuentes <ofv <at> wanadoo.es>
To: Dmitry Gutov <dgutov <at> yandex.ru>
Cc: John Wiegley <jwiegley <at> gmail.com>, Jaakov <j_k_v <at> ro.ru>,
 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1;
 `fill-paragraph' should not always put the buffer as modified
Date: Sun, 27 Mar 2016 17:35:23 +0200

Dmitry Gutov <dgutov <at> yandex.ru> writes:

> Hashes have collisions (and md5 is a bit famous for them). So in
> principle, I don't think using a hash is a good choice in this case.
>
> Either way, you'd have to keep the original string around, to compare
> against if the hashes match.

Dmitry, it would be absolutely glorious if we ever find a md5 collision
among a piece of text and the result of applying fill-paragraph to it. I
would demand to be credited by the feat :-)

Message #188 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Lars Magne Ingebrigtsen <larsi <at> gnus.org>
To: Dmitry Gutov <dgutov <at> yandex.ru>
Cc: Óscar Fuentes <ofv <at> wanadoo.es>,
 John Wiegley <jwiegley <at> gmail.com>, Jaakov <j_k_v <at> ro.ru>, 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1;
 `fill-paragraph' should not always put the buffer as modified
Date: Sun, 27 Mar 2016 17:35:25 +0200

Dmitry Gutov <dgutov <at> yandex.ru> writes:

> On 03/27/2016 06:31 AM, Óscar Fuentes wrote:
>
>> +  (when (and (not (string= h ""))
>> +             (string= h (secure-hash 'md5 (current-buffer))))
>> +    (set-buffer-modified-p nil)))
>
> Hashes have collisions (and md5 is a bit famous for them). So in
> principle, I don't think using a hash is a good choice in this case.

md5 is famous for enabling attackers to construct strings with the same
hash, not for arbitrarily making strings hash to the same result.

It's less likely that the before/after `M-q' strings hash to the same
md5 than cosmic rays reprogramming your Emacs into vi, so:

> Either way, you'd have to keep the original string around, to compare
> against if the hashes match.

Not really.

(Ok, I'm exaggerating.  Slightly!)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #191 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Dmitry Gutov <dgutov <at> yandex.ru>
To: Lars Magne Ingebrigtsen <larsi <at> gnus.org>
Cc: Óscar Fuentes <ofv <at> wanadoo.es>,
 John Wiegley <jwiegley <at> gmail.com>, Jaakov <j_k_v <at> ro.ru>, 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1; `fill-paragraph' should not always put the
 buffer as modified
Date: Sun, 27 Mar 2016 18:42:35 +0300

On 03/27/2016 06:35 PM, Lars Magne Ingebrigtsen wrote:

> It's less likely that the before/after `M-q' strings hash to the same
> md5 than cosmic rays reprogramming your Emacs into vi, so:

And yet, why would we allow such possibility? String comparison is 
plenty fast already.

Comparing the contents of xdisp.c to itself takes 0.2ms here.

Try:

(setq s (buffer-string))
(setq ss (copy-sequence s))
(benchmark 1 '(equal s ss))

Message #194 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Dmitry Gutov <dgutov <at> yandex.ru>
To: Óscar Fuentes <ofv <at> wanadoo.es>, 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1; `fill-paragraph' should not always put the
 buffer as modified
Date: Sun, 27 Mar 2016 18:42:43 +0300

On 03/27/2016 06:28 PM, Óscar Fuentes wrote:

> The docs say:
>
>
>   This shows two dashes (‘--’) if the buffer displayed in the window has
>   the same contents as the corresponding file on the disk; i.e., if the
>   buffer is unmodified.

That might be a documentation bug. Switch to an unmodified buffer. Type 
`a', and then backspace. What do you see in the status?

> AFAIK, a file-visiting buffer is not marked as modified when text
> properties are applied to it.

It is. If you're thinking of font-lock, then it uses 
with-silent-modifications.

Message #197 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Dmitry Gutov <dgutov <at> yandex.ru>
To: Óscar Fuentes <ofv <at> wanadoo.es>
Cc: John Wiegley <jwiegley <at> gmail.com>, Jaakov <j_k_v <at> ro.ru>,
 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1; `fill-paragraph' should not always put the
 buffer as modified
Date: Sun, 27 Mar 2016 18:46:44 +0300

On 03/27/2016 06:35 PM, Óscar Fuentes wrote:

> Dmitry, it would be absolutely glorious if we ever find a md5 collision
> among a piece of text and the result of applying fill-paragraph to it. I
> would demand to be credited by the feat :-)

Glorious or not, such an occasion would likely go unnoticed, and could 
result in the user not having their latest changes saved to disk.

The only thing they would remember is that Emacs didn't work as 
expected, and your choice of the algorithm would go uncredited.

Message #200 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Lars Magne Ingebrigtsen <larsi <at> gnus.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: Óscar Fuentes <ofv <at> wanadoo.es>, jwiegley <at> gmail.com,
 j_k_v <at> ro.ru, 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1;
 `fill-paragraph' should not always put the buffer as modified
Date: Sun, 27 Mar 2016 17:46:53 +0200

Eli Zaretskii <eliz <at> gnu.org> writes:

>> +  (when (and (not (string= h ""))
>> +             (string= h (secure-hash 'md5 (current-buffer))))
>> +    (set-buffer-modified-p nil)))
>
> Thanks, but I'm not sure computing the hash is enough: the functions
> involved in refilling can change text properties, so the test should
> also account for that.

True.  Do we have an efficient way to get the text properties, too?  (I
mean, without doing a `buffer-substring'...)

Hm...  looking at `secure-hash', it seems incredibly inefficient.
(Unless I'm misreading the code.)  All the coding system conversion
stuff is completely irrelevant for this usage...  

What we basically need is a fast hashing function for the buffer,
including text properties.  So it would basically do:

1) move the gap out of the way
2) call the hashing function on the buffer contents
c) call the hashing function on the text properties
4) hash them together

This should be really fast, I think?  If the text properties are
available in a fashion where we can do some hashing on them without
copying them around a lot.  And I know nothing about how text properties
are represented.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #203 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Lars Magne Ingebrigtsen <larsi <at> gnus.org>
To: Dmitry Gutov <dgutov <at> yandex.ru>
Cc: Óscar Fuentes <ofv <at> wanadoo.es>,
 John Wiegley <jwiegley <at> gmail.com>, Jaakov <j_k_v <at> ro.ru>, 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1;
 `fill-paragraph' should not always put the buffer as modified
Date: Sun, 27 Mar 2016 17:50:58 +0200

Dmitry Gutov <dgutov <at> yandex.ru> writes:

> And yet, why would we allow such possibility? String comparison is
> plenty fast already.

Comparison is fast, but making a copy of a buffer (or its contents)
isn't.  (If the buffer is large, that is.)  If you've loaded a 2GB file
and hit `M-q' on a line, it would be rather awkward if that made Emacs
allocate an additional 2GB of data.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #206 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Óscar Fuentes <ofv <at> wanadoo.es>
To: Dmitry Gutov <dgutov <at> yandex.ru>
Cc: 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1;
 `fill-paragraph' should not always put the buffer as modified
Date: Sun, 27 Mar 2016 17:52:34 +0200

Dmitry Gutov <dgutov <at> yandex.ru> writes:

> On 03/27/2016 06:28 PM, Óscar Fuentes wrote:
>
>> The docs say:
>>
>>
>>   This shows two dashes (‘--’) if the buffer displayed in the window has
>>   the same contents as the corresponding file on the disk; i.e., if the
>>   buffer is unmodified.
>
> That might be a documentation bug. Switch to an unmodified buffer.
> Type `a', and then backspace. What do you see in the status?

When you type `a', you changed the buffer. Checking that your subsequent
actions gives a result that is identical to the saved file is something
that would be nice to have, but I guess that few users would think that
it is a reasonable requirement.

OTOH, `undo' clears the `modified' flag.

>> AFAIK, a file-visiting buffer is not marked as modified when text
>> properties are applied to it.
>
> It is. If you're thinking of font-lock, then it uses
> with-silent-modifications.

And why it does use with-silent-modifications? Is there a case where the
user is benefited from marking the buffer as modified after applying
text properties?

Message #209 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Lars Magne Ingebrigtsen <larsi <at> gnus.org>
To: Dmitry Gutov <dgutov <at> yandex.ru>
Cc: Óscar Fuentes <ofv <at> wanadoo.es>,
 John Wiegley <jwiegley <at> gmail.com>, Jaakov <j_k_v <at> ro.ru>, 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1;
 `fill-paragraph' should not always put the buffer as modified
Date: Sun, 27 Mar 2016 17:53:23 +0200

Dmitry Gutov <dgutov <at> yandex.ru> writes:

> Glorious or not, such an occasion would likely go unnoticed, and could
> result in the user not having their latest changes saved to disk.

I look forward to seeing you visiting the git mailing list and starting
to agitate for not using sha-1 hashes as object identifiers in git,
because it might obviously lose data if you happen to get collisions.

Have fun.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #212 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Lars Magne Ingebrigtsen <larsi <at> gnus.org>
To: Óscar Fuentes <ofv <at> wanadoo.es>
Cc: 13949 <at> debbugs.gnu.org, Dmitry Gutov <dgutov <at> yandex.ru>
Subject: Re: bug#13949: 24.4.1;
 `fill-paragraph' should not always put the buffer as modified
Date: Sun, 27 Mar 2016 17:57:17 +0200

Óscar Fuentes <ofv <at> wanadoo.es> writes:

> When you type `a', you changed the buffer. Checking that your subsequent
> actions gives a result that is identical to the saved file is something
> that would be nice to have, but I guess that few users would think that
> it is a reasonable requirement.

I think that would be a very nice feature, though.  Like, if Emacs
computed the hash of the buffer when you loaded it, and then checks
again every time you edit something, and uses that for the "buffer
changed" marker.  :-)

It's probably unrealistically slow, though.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #215 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Óscar Fuentes <ofv <at> wanadoo.es>
Cc: jwiegley <at> gmail.com, j_k_v <at> ro.ru, 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1;
 `fill-paragraph' should not always put the buffer as modified
Date: Sun, 27 Mar 2016 18:58:48 +0300

> From: Óscar Fuentes <ofv <at> wanadoo.es>
> Cc: jwiegley <at> gmail.com,  j_k_v <at> ro.ru,  13949 <at> debbugs.gnu.org
> Date: Sun, 27 Mar 2016 17:29:36 +0200
> 
> AFAIK, a file-visiting buffer is not marked as modified when text
> properties are applied to it. This makes sense, because text properties
> are not part of the contents of the corresponding file.

It is, and they are.  Here, try this:

 emacs -Q
 C-x C-f README RET
 M-: (put-text-property 1 10 'myprop 'foo) RET

Then look at the buffer-modified indication.

Message #218 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Óscar Fuentes <ofv <at> wanadoo.es>
To: bug-gnu-emacs <at> gnu.org
Subject: Re: bug#13949: 24.4.1;
 `fill-paragraph' should not always put the buffer as modified
Date: Sun, 27 Mar 2016 18:05:36 +0200

Eli Zaretskii <eliz <at> gnu.org> writes:

>> AFAIK, a file-visiting buffer is not marked as modified when text
>> properties are applied to it. This makes sense, because text properties
>> are not part of the contents of the corresponding file.
>
> It is, and they are.  Here, try this:
>
>  emacs -Q
>  C-x C-f README RET
>  M-: (put-text-property 1 10 'myprop 'foo) RET
>
> Then look at the buffer-modified indication.

Sorry, but AFAIK if some feature does this without a
with-silent-modifications wrapping it, a bug report would is in order,
right?

Again, the question nobody dares to answer: is there a legit case where
the user benefits from marking the buffer as modified after applying or
changing text properties?

From the lack of response so far, I guess that the answer is "no".

Message #221 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Lars Magne Ingebrigtsen <larsi <at> gnus.org>
Cc: ofv <at> wanadoo.es, jwiegley <at> gmail.com, j_k_v <at> ro.ru, 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1;
 `fill-paragraph' should not always put the buffer as modified
Date: Sun, 27 Mar 2016 19:04:34 +0300

> From: Lars Magne Ingebrigtsen <larsi <at> gnus.org>
> Cc: Óscar Fuentes <ofv <at> wanadoo.es>,  jwiegley <at> gmail.com,
>   j_k_v <at> ro.ru,  13949 <at> debbugs.gnu.org
> Date: Sun, 27 Mar 2016 17:46:53 +0200
> 
> 1) move the gap out of the way

This will call memmove to move a potentially large chunk of text.  Why
not just hash the two parts separately?

Message #224 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Jaakov <j_k_v <at> ro.ru>
To: Lars Magne Ingebrigtsen <larsi <at> gnus.org>, Dmitry Gutov <dgutov <at> yandex.ru>
Cc: Óscar Fuentes <ofv <at> wanadoo.es>,
 John Wiegley <jwiegley <at> gmail.com>, 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1; `fill-paragraph' should not always put the
 buffer as modified
Date: Sun, 27 Mar 2016 18:08:16 +0200

> It's less likely that the before/after `M-q' strings hash to the same
> md5 than cosmic rays reprogramming your Emacs into vi, so:

Users' texts are not completely random, but highly correlated. 
Therefore, the argument about unlikeliness (which might hold for 
equidistributed texts, for each length) does not apply.

If the above patch stays upstream, I'm going to switch from emacs to 
something else and advise others to do so. Really.

Please don't cc me any more.

Message #227 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Lars Magne Ingebrigtsen <larsi <at> gnus.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: ofv <at> wanadoo.es, jwiegley <at> gmail.com, 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1;
 `fill-paragraph' should not always put the buffer as modified
Date: Sun, 27 Mar 2016 18:11:47 +0200

Eli Zaretskii <eliz <at> gnu.org> writes:

> This will call memmove to move a potentially large chunk of text.  Why
> not just hash the two parts separately?

Yeah, that's true.

What about the text property representation -- can that easily be
hashed, do you think?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #230 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Óscar Fuentes <ofv <at> wanadoo.es>
Cc: 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1;
 `fill-paragraph' should not always put the buffer as modified
Date: Sun, 27 Mar 2016 19:12:02 +0300

> From: Óscar Fuentes <ofv <at> wanadoo.es>
> Date: Sun, 27 Mar 2016 18:05:36 +0200
> 
> >  emacs -Q
> >  C-x C-f README RET
> >  M-: (put-text-property 1 10 'myprop 'foo) RET
> >
> > Then look at the buffer-modified indication.
> 
> Sorry, but AFAIK if some feature does this without a
> with-silent-modifications wrapping it, a bug report would is in order,
> right?

No, I don't see why it should be a bug.  The text property can be
anything, including something that has a profound effect on how text
is displayed, like 'invisible' or 'display'.

> Again, the question nobody dares to answer: is there a legit case where
> the user benefits from marking the buffer as modified after applying or
> changing text properties?
> 
> >From the lack of response so far, I guess that the answer is "no".

This is the tail wagging the dog: we are not going to overturn a
long-standing way Emacs works with text properties for the sake of a
minor feature.  If you really want this feature to be accepted, please
find a way of doing that without any such notable effects, i.e. count
changes in the text properties, if any, with the text changes.

Message #233 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Lars Magne Ingebrigtsen <larsi <at> gnus.org>
To: Óscar Fuentes <ofv <at> wanadoo.es>
Cc: 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1;
 `fill-paragraph' should not always put the buffer as modified
Date: Sun, 27 Mar 2016 18:14:43 +0200

Óscar Fuentes <ofv <at> wanadoo.es> writes:

> Again, the question nobody dares to answer: is there a legit case where
> the user benefits from marking the buffer as modified after applying or
> changing text properties?
>
> From the lack of response so far, I guess that the answer is "no".

Some modes support generating various "rich text" modes based on text
properties.

Those modes are rare, though.  I think we probably could say that those
modes should "do something special" and make Emacs, by default, not care
about text properties for the "buffer changed" thing.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #236 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Lars Magne Ingebrigtsen <larsi <at> gnus.org>
Cc: ofv <at> wanadoo.es, jwiegley <at> gmail.com, 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1;
 `fill-paragraph' should not always put the buffer as modified
Date: Sun, 27 Mar 2016 19:18:00 +0300

> From: Lars Magne Ingebrigtsen <larsi <at> gnus.org>
> Cc: ofv <at> wanadoo.es,  jwiegley <at> gmail.com,  13949 <at> debbugs.gnu.org
> Date: Sun, 27 Mar 2016 18:11:47 +0200
> 
> What about the text property representation -- can that easily be
> hashed, do you think?

One could hack something together along the lines of
compare_string_intervals and intervals_equal, I think.  That is,
traverse the tree as they do, and compute the hash while at that.

Message #239 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Óscar Fuentes <ofv <at> wanadoo.es>
To: Lars Magne Ingebrigtsen <larsi <at> gnus.org>
Cc: 13949 <at> debbugs.gnu.org, Dmitry Gutov <dgutov <at> yandex.ru>
Subject: Re: bug#13949: 24.4.1;
 `fill-paragraph' should not always put the buffer as modified
Date: Sun, 27 Mar 2016 18:21:55 +0200

Lars Magne Ingebrigtsen <larsi <at> gnus.org> writes:

> Óscar Fuentes <ofv <at> wanadoo.es> writes:
>
>> When you type `a', you changed the buffer. Checking that your subsequent
>> actions gives a result that is identical to the saved file is something
>> that would be nice to have, but I guess that few users would think that
>> it is a reasonable requirement.
>
> I think that would be a very nice feature, though.  Like, if Emacs
> computed the hash of the buffer when you loaded it, and then checks
> again every time you edit something, and uses that for the "buffer
> changed" marker.  :-)

Interesting... :-)

> It's probably unrealistically slow, though.

I don't think so. It could be auto-disabled for large buffers and the
check would only take place when the size of the buffer is the same as
the size of the original file. Then, the hash would be calculated on a
idle timer that is fired after a command that changes the contents...

It is doable, moreover if the current hash function is so inefficient as
you say. MD5 is rated at more than a hundred MB/s on not-so-new
hardware.

Message #242 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Lars Magne Ingebrigtsen <larsi <at> gnus.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: ofv <at> wanadoo.es, jwiegley <at> gmail.com, 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1;
 `fill-paragraph' should not always put the buffer as modified
Date: Sun, 27 Mar 2016 18:28:43 +0200

Eli Zaretskii <eliz <at> gnu.org> writes:

> One could hack something together along the lines of
> compare_string_intervals and intervals_equal, I think.  That is,
> traverse the tree as they do, and compute the hash while at that.

*peruses code*

I see.  Looking at this, and looking at the sha1 reference for all of
two minutes, it looks like we could create a new C-level function called
something like hash_buffer that would look basically like

sha1_init_ctx
sha1_process_bytes(first_part_of_buffer, len)
sha1_process_bytes(last_part_of_buffer, len)
for iterate_over_all_intervals
  sha1_process_bytes(interval, len)
sha1_finish_ctx

and there you have it.  A hashing function that would not allocate
anything much, and it should be fast enough even in huge buffers for
`M-q'.  I think.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #245 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Lars Magne Ingebrigtsen <larsi <at> gnus.org>
To: Óscar Fuentes <ofv <at> wanadoo.es>
Cc: 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1;
 `fill-paragraph' should not always put the buffer as modified
Date: Sun, 27 Mar 2016 18:33:35 +0200

Óscar Fuentes <ofv <at> wanadoo.es> writes:

> I don't think so. It could be auto-disabled for large buffers and the
> check would only take place when the size of the buffer is the same as
> the size of the original file.

Oh yeah, that's true...  Except for the problem with the text
properties.  :-)

> Then, the hash would be calculated on a idle timer that is fired after
> a command that changes the contents...

I think I'd want the "buffer changed" indicator to reflect the state
immediately after doing an edit, though.  It's been a long-standing
annoyance that Emacs claims that the buffer is changed when it "kinda
isn't" (i.e., insert "a" and then delete it).  I want to see that
immediately in the mode line.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #248 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Óscar Fuentes <ofv <at> wanadoo.es>
To: bug-gnu-emacs <at> gnu.org
Subject: Re: bug#13949: 24.4.1;
 `fill-paragraph' should not always put the buffer as modified
Date: Sun, 27 Mar 2016 18:37:10 +0200

Eli Zaretskii <eliz <at> gnu.org> writes:

>> Again, the question nobody dares to answer: is there a legit case where
>> the user benefits from marking the buffer as modified after applying or
>> changing text properties?
>> 
>> >From the lack of response so far, I guess that the answer is "no".
>
> This is the tail wagging the dog: we are not going to overturn a
> long-standing way Emacs works with text properties for the sake of a
> minor feature.

"minor" is your judgement. And so far there is zero evidence that this
change could cause undesired effects. So we have users suffering this
bug (yes, it is obvious to some of us that it is a bug) and, OTOH, we
have FUD.

> If you really want this feature to be accepted,

Sorry Eli, but as much as I appreciate your opinions, I'm not submitting
the feature for *your* approval. You are not the only one that can
approve (or reject) the patch (once we have one.)

> please find a way of doing that without any such notable effects, i.e.
> count changes in the text properties, if any, with the text changes.

You are decided to block a solution for this issue, just because. Got
it.

Message #251 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Óscar Fuentes <ofv <at> wanadoo.es>
To: 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1;
 `fill-paragraph' should not always put the buffer as modified
Date: Sun, 27 Mar 2016 18:38:03 +0200

Eli Zaretskii <eliz <at> gnu.org> writes:

>> Again, the question nobody dares to answer: is there a legit case where
>> the user benefits from marking the buffer as modified after applying or
>> changing text properties?
>> 
>> >From the lack of response so far, I guess that the answer is "no".
>
> This is the tail wagging the dog: we are not going to overturn a
> long-standing way Emacs works with text properties for the sake of a
> minor feature.

"minor" is your judgement. And so far there is zero evidence that this
change could cause undesired effects. So we have users suffering this
bug (yes, it is obvious to some of us that it is a bug) and, OTOH, we
have FUD.

> If you really want this feature to be accepted,

Sorry Eli, but as much as I appreciate your opinions, I'm not submitting
the feature for *your* approval. You are not the only one that can
approve (or reject) the patch (once we have one.)

> please find a way of doing that without any such notable effects, i.e.
> count changes in the text properties, if any, with the text changes.

You are decided to block a solution for this issue, just because. Got
it.

Message #254 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Óscar Fuentes <ofv <at> wanadoo.es>
To: Lars Magne Ingebrigtsen <larsi <at> gnus.org>
Cc: 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1;
 `fill-paragraph' should not always put the buffer as modified
Date: Sun, 27 Mar 2016 18:46:27 +0200

Lars Magne Ingebrigtsen <larsi <at> gnus.org> writes:

> Óscar Fuentes <ofv <at> wanadoo.es> writes:
>
>> I don't think so. It could be auto-disabled for large buffers and the
>> check would only take place when the size of the buffer is the same as
>> the size of the original file.
>
> Oh yeah, that's true...  Except for the problem with the text
> properties.  :-)

Oh yes, apparently the fix for this bug must be bug-compatible with the
feature it is fixing.

>> Then, the hash would be calculated on a idle timer that is fired after
>> a command that changes the contents...
>
> I think I'd want the "buffer changed" indicator to reflect the state
> immediately after doing an edit, though.  It's been a long-standing
> annoyance that Emacs claims that the buffer is changed when it "kinda
> isn't" (i.e., insert "a" and then delete it).  I want to see that
> immediately in the mode line.

Agreed. As long as the file is small enough for the hash function, an
idle timer with a short span would do. We don't want the hashing running
again and again while some command does multiple changes to the buffer.

Message #257 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Óscar Fuentes <ofv <at> wanadoo.es>
Cc: 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1;
 `fill-paragraph' should not always put the buffer as modified
Date: Sun, 27 Mar 2016 19:50:57 +0300

> From: Óscar Fuentes <ofv <at> wanadoo.es>
> Date: Sun, 27 Mar 2016 18:37:10 +0200
> 
> Eli Zaretskii <eliz <at> gnu.org> writes:
> 
> >> Again, the question nobody dares to answer: is there a legit case where
> >> the user benefits from marking the buffer as modified after applying or
> >> changing text properties?
> >> 
> >> >From the lack of response so far, I guess that the answer is "no".
> >
> > This is the tail wagging the dog: we are not going to overturn a
> > long-standing way Emacs works with text properties for the sake of a
> > minor feature.
> 
> "minor" is your judgement.

You could try making a case for it not being minor, maybe you will be
able to convince.  For now, I don't see how it could be anything but
minor, or else we would have changed it long ago.

> And so far there is zero evidence that this change could cause
> undesired effects.

That's irrelevant.  It would be irresponsible for us to change such
basic aspects of Emacs operation at this point in Emacs history.  We
have been burnt with much less significant backward-incompatible
changes.

> > If you really want this feature to be accepted,
> 
> Sorry Eli, but as much as I appreciate your opinions, I'm not submitting
> the feature for *your* approval. You are not the only one that can
> approve (or reject) the patch (once we have one.)

I'm not the only one, but I'm one of those you need to convince.

> > please find a way of doing that without any such notable effects, i.e.
> > count changes in the text properties, if any, with the text changes.
> 
> You are decided to block a solution for this issue, just because. Got
> it.

That's a nasty, unfair way of putting things, and you know it.

Message #260 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Lars Magne Ingebrigtsen <larsi <at> gnus.org>
To: Óscar Fuentes <ofv <at> wanadoo.es>
Cc: 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1;
 `fill-paragraph' should not always put the buffer as modified
Date: Sun, 27 Mar 2016 18:58:21 +0200

Óscar Fuentes <ofv <at> wanadoo.es> writes:

>>> I don't think so. It could be auto-disabled for large buffers and the
>>> check would only take place when the size of the buffer is the same as
>>> the size of the original file.
>>
>> Oh yeah, that's true...  Except for the problem with the text
>> properties.  :-)
>
> Oh yes, apparently the fix for this bug must be bug-compatible with the
> feature it is fixing.

:-)

But, come to think of it, I think it's quite rare in practice to do a
lot of text property related editing without changing the size of the
buffer, so perhaps this doesn't matter much.  I mean, if you have a work
flow that involves you opening a 2GB file, and then placing text
properties (unrelated to font-locking) all over the place without
changing the buffer otherwise, then...  you're probably kinda unusual?

So the "only hash when the buffer size is the same as when you loaded
the file" thing would probably avoid the hashing in more than 99% of the
use cases.

>> I think I'd want the "buffer changed" indicator to reflect the state
>> immediately after doing an edit, though.  It's been a long-standing
>> annoyance that Emacs claims that the buffer is changed when it "kinda
>> isn't" (i.e., insert "a" and then delete it).  I want to see that
>> immediately in the mode line.
>
> Agreed. As long as the file is small enough for the hash function, an
> idle timer with a short span would do. We don't want the hashing running
> again and again while some command does multiple changes to the buffer.

But I really think it has to be immediate and predictable for Emacs to
keep working as it does.

I mean

(progn
  (find-file "~/foo")
  (insert "zot")
  (save-buffer))

must work reliably.

One argument against switching to hash based edition detection is that
we'd have a different method of computing the change when we have a
buffer visiting a file and not...  or...  perhaps not?
`(set-buffer-modified-p nil)' could be the function that computes the
"initial" hash'n'size that would be used later, and `buffer-modified-p'
could just compare with that tuple...

This would allow us to get rid of the...  er...  thing that keeps track
of buffer modification now...  is that the text->modiff thing?

I think this sounds kinda exciting.  :-)  If it's feasible in practice.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #263 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Lars Magne Ingebrigtsen <larsi <at> gnus.org>
To: Óscar Fuentes <ofv <at> wanadoo.es>
Cc: 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1;
 `fill-paragraph' should not always put the buffer as modified
Date: Sun, 27 Mar 2016 19:00:46 +0200

Óscar Fuentes <ofv <at> wanadoo.es> writes:

> "minor" is your judgement. And so far there is zero evidence that this
> change could cause undesired effects. So we have users suffering this
> bug (yes, it is obvious to some of us that it is a bug) and, OTOH, we
> have FUD.

I think you're being somewhat unfair here.  None of this is obvious, and
we're just talking about how to make Emacs slightly better.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #266 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Óscar Fuentes <ofv <at> wanadoo.es>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1;
 `fill-paragraph' should not always put the buffer as modified
Date: Sun, 27 Mar 2016 19:30:36 +0200

Eli Zaretskii <eliz <at> gnu.org> writes:

>> "minor" is your judgement.
>
> You could try making a case for it not being minor, maybe you will be
> able to convince.  For now, I don't see how it could be anything but
> minor, or else we would have changed it long ago.

See this bug report and its duplicateds. It is not a data-loss bug, but
something that can be a constant annoyance to some users.

It occurred to me at least two times to use M-q on comments on some C++
header, see no changes, proceed with other edits elsewhere on the
project, and much later do `C-x s ! M-x compile' and see how the build
compiled files that shouldn't be affected by my edits, which, apart from
the waste of time on the extended build, caused more time to be wasted
on investigating the cause. Since I aware of the problem, if I use M-q
on a source file, I need to use `C-x s d' to see a diff and, if the diff
is empty, use undo to restore the modified flag.

>> And so far there is zero evidence that this change could cause
>> undesired effects.
>
> That's irrelevant.  It would be irresponsible for us to change such
> basic aspects of Emacs operation at this point in Emacs history.  We
> have been burnt with much less significant backward-incompatible
> changes.

This is a recipe for changing *nothing* that is older than some
threshold, isn't it? And we are talking about fill-paragraph here, not
about some core data structure. Apart from the fact that marking the
buffer as modified when text properties are changed is wrong in
principle (otherwise, why don't mark as modified the file-visiting
buffers as soon as some text properties are applied when the major/minor
modes are enabled?)

[snip]

Message #269 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Óscar Fuentes <ofv <at> wanadoo.es>
Cc: 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1;
 `fill-paragraph' should not always put the buffer as modified
Date: Sun, 27 Mar 2016 20:51:35 +0300

> From: Óscar Fuentes <ofv <at> wanadoo.es>
> Cc: 13949 <at> debbugs.gnu.org
> Date: Sun, 27 Mar 2016 19:30:36 +0200
> 
> It occurred to me at least two times to use M-q on comments on some C++
> header, see no changes, proceed with other edits elsewhere on the
> project, and much later do `C-x s ! M-x compile' and see how the build
> compiled files that shouldn't be affected by my edits, which, apart from
> the waste of time on the extended build, caused more time to be wasted
> on investigating the cause. Since I aware of the problem, if I use M-q
> on a source file, I need to use `C-x s d' to see a diff and, if the diff
> is empty, use undo to restore the modified flag.

You are describing what I consider to be a minor annoyance.  I agree
it's an annoyance, and I agree it would be good to have an option to
prevent that, I'm just saying the annoyance is minor.

> >> And so far there is zero evidence that this change could cause
> >> undesired effects.
> >
> > That's irrelevant.  It would be irresponsible for us to change such
> > basic aspects of Emacs operation at this point in Emacs history.  We
> > have been burnt with much less significant backward-incompatible
> > changes.
> 
> This is a recipe for changing *nothing* that is older than some
> threshold, isn't it?

No, only those aspects that are very basic, like text properties being
an integral part of buffer text.

> And we are talking about fill-paragraph here, not about some core
> data structure.

I wasn't talking about fill-paragraph, I was talking about deciding
that changes in text properties aren't considered buffer
modifications.

> Apart from the fact that marking the buffer as modified when text
> properties are changed is wrong in principle (otherwise, why don't
> mark as modified the file-visiting buffers as soon as some text
> properties are applied when the major/minor modes are enabled?)

I think you are only considering face properties.  But text properties
can be something entirely different.  I gave 2 examples before, here's
another, perhaps more relevant one: the 'fill-space' and 'hard'
properties that are directly involved in text filling.

Message #272 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Drew Adams <drew.adams <at> oracle.com>
To: Lars Magne Ingebrigtsen <larsi <at> gnus.org>,
 Óscar Fuentes <ofv <at> wanadoo.es>
Cc: 13949 <at> debbugs.gnu.org
Subject: RE: bug#13949: 24.4.1; `fill-paragraph' should not always put the
 buffer as modified
Date: Sun, 27 Mar 2016 11:22:59 -0700 (PDT)

> But, come to think of it, I think it's quite rare in practice to do a
> lot of text property related editing without changing the size of the
> buffer, so perhaps this doesn't matter much.  I mean, if you have a work
> flow that involves you opening a 2GB file, and then placing text
> properties (unrelated to font-locking) all over the place without
> changing the buffer otherwise, then...  you're probably kinda unusual?

Why do you think that?  Code that uses text properties does not
necessarily change other things in the buffer.  I see no connection
between the two.

The fact that you even added "unrelated to font-locking" is perhaps
a giveaway. Font-locking is the best known use of text properties.
And even it does not typically involve changing other things in the
buffer.  Why would you expect that other code that uses text properties
would be different?  What's so special about font-locking in this regard?

If anything, I would expect other code that uses text properties
to _not_ change the buffer otherwise - just like font-lock.

I say "if anything", because I don't really think there is any
connection between using text properties and other buffer changes.

> So the "only hash when the buffer size is the same as when you loaded
> the file" thing would probably avoid the hashing in more than 99% of
> the use cases.

Why?  Where do you get 99%?  Or even 25%?  On what do you base the
assumption that code that uses text properties also makes other
buffer changes?  What do you see as the use cases of text properties,
of which 99% change the buffer otherwise?

Message #275 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Drew Adams <drew.adams <at> oracle.com>
To: Óscar Fuentes <ofv <at> wanadoo.es>, Andreas Schwab
 <schwab <at> linux-m68k.org>
Cc: John Wiegley <jwiegley <at> gmail.com>, Jaakov <j_k_v <at> ro.ru>,
 13949 <at> debbugs.gnu.org
Subject: RE: bug#13949: 24.4.1; `fill-paragraph' should not always put the
 buffer as modified
Date: Sun, 27 Mar 2016 11:53:41 -0700 (PDT)

I think it is unwise to _replace_ willy-nilly the longstanding
definition of buffer modification - for either users or Lisp
code.  Just because you think you can do something clever using
hashes, that does not mean that you should, if it changes the
longstanding behavior.  No gratuitous loss, please.

On the other hand, as I said, it might be good to expand that
notion to different _kinds_ of buffer modification - not only
for indication to users, but also for code.

IOW, additional flexibility, why not?  But replacement, no thanks.

I said:

  There are different kinds of buffer changes.  It would be
  good to have ways to detect these as such.

The indication to users could perhaps be to use other chars for
this, in addition to `*', in mode-line indicators `**' and `%*'.

Any such change in the indication should be optional, e.g.,
controlled by a user option.

There are two things that a user might want to customize here:
(1) which kinds of buffer change to indicate and (2) how to
indicate them.

Wrt #2, a user could choose, for example, not to use different
markers for different buffer changes, i.e., to always use only
`*', even if s?he uses #1 to define "change" (only text changes,
text and text-property changes,...) for the given buffer.  (The
option could be buffer-local, of course.)

Message #278 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Dmitry Gutov <dgutov <at> yandex.ru>
To: Lars Magne Ingebrigtsen <larsi <at> gnus.org>
Cc: Óscar Fuentes <ofv <at> wanadoo.es>,
 John Wiegley <jwiegley <at> gmail.com>, Jaakov <j_k_v <at> ro.ru>, 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1; `fill-paragraph' should not always put the
 buffer as modified
Date: Sun, 27 Mar 2016 23:24:55 +0300

On 03/27/2016 06:53 PM, Lars Magne Ingebrigtsen wrote:

> I look forward to seeing you visiting the git mailing list and starting
> to agitate for not using sha-1 hashes as object identifiers in git,
> because it might obviously lose data if you happen to get collisions.

a) Why didn't they use md5, I wonder?

b) Git has a global object index. It _can_ detect collisions, or at 
least that detection can be implemented.

Message #281 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Dmitry Gutov <dgutov <at> yandex.ru>
To: Lars Magne Ingebrigtsen <larsi <at> gnus.org>
Cc: Óscar Fuentes <ofv <at> wanadoo.es>,
 John Wiegley <jwiegley <at> gmail.com>, Jaakov <j_k_v <at> ro.ru>, 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1; `fill-paragraph' should not always put the
 buffer as modified
Date: Sun, 27 Mar 2016 23:27:40 +0300

On 03/27/2016 06:50 PM, Lars Magne Ingebrigtsen wrote:

> Comparison is fast, but making a copy of a buffer (or its contents)
> isn't.  (If the buffer is large, that is.)  If you've loaded a 2GB file
> and hit `M-q' on a line, it would be rather awkward if that made Emacs
> allocate an additional 2GB of data.

Compare the "current paragraph", then (its bounds can be saved at the 
beginning of the fill-paragraph).

Or track the affected area via before-change-functions.

buffer-undo-list should also have the necessary information.

Message #284 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Óscar Fuentes <ofv <at> wanadoo.es>
To: Dmitry Gutov <dgutov <at> yandex.ru>
Cc: John Wiegley <jwiegley <at> gmail.com>, Lars Magne Ingebrigtsen <larsi <at> gnus.org>,
 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1;
 `fill-paragraph' should not always put the buffer as modified
Date: Sun, 27 Mar 2016 23:05:42 +0200

Dmitry Gutov <dgutov <at> yandex.ru> writes:

> On 03/27/2016 06:53 PM, Lars Magne Ingebrigtsen wrote:
>
>> I look forward to seeing you visiting the git mailing list and starting
>> to agitate for not using sha-1 hashes as object identifiers in git,
>> because it might obviously lose data if you happen to get collisions.
>
> a) Why didn't they use md5, I wonder?

AFAIK, at first they intended to use the hash as a method for avoiding
malicious tampering of the VC contents, and MD5 was already broken as a
crypto hash algorithm. (It is entirely different to find a collision by
chance and to *fabricate* a collision; MD5 is broken for the later, but
reliable for preventing the former.)

I guess that the extra bits of entropy (160 vs 128) was a "fuzzy-warm"
factor too on using SHA-1 instead of MD5. Git must avoid collisions
among potentially hundreds of millions of objects (repos with that size
already exists or will exist on the near future.) Each and every hash
must be different from all the others and hence avoid the Birthday
Problem. Anyway, 128 bit hashes still would be good enough for those
huge repos. fill-paragraph needs to discriminate only between 2 chunks
of data.

> b) Git has a global object index. It _can_ detect collisions, or at
> least that detection can be implemented.

And what to do when a collision is detected?

Back to the topic, your suggetion about comparing the pre- and post-
contents of the paragraph (and avoiding huge copies of the pre- contents
by restricting the copied area to the paragraph itself) does not work
when the file contains just one paragraph. Try visiting a big CSV dump
or log and press M-q. You can abort the operation with C-g, but if Emacs
starts to swap like crazy or exceeds the process memory limit and it is
killed... We can be confident that this would happen multiple times out
there, the contrary of having the same MD5 for the pre- and post- result
of fill-paragraph.

Message #287 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Dmitry Gutov <dgutov <at> yandex.ru>
To: Óscar Fuentes <ofv <at> wanadoo.es>
Cc: John Wiegley <jwiegley <at> gmail.com>, Lars Magne Ingebrigtsen <larsi <at> gnus.org>,
 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1; `fill-paragraph' should not always put the
 buffer as modified
Date: Mon, 28 Mar 2016 00:20:08 +0300

On 03/28/2016 12:05 AM, Óscar Fuentes wrote:

> I guess that the extra bits of entropy (160 vs 128) was a "fuzzy-warm"
> factor too on using SHA-1 instead of MD5. Git must avoid collisions
> among potentially hundreds of millions of objects (repos with that size
> already exists or will exist on the near future.)

Are there fewer different texts we'd have to be able to discern?

> Each and every hash
> must be different from all the others and hence avoid the Birthday
> Problem. Anyway, 128 bit hashes still would be good enough for those
> huge repos. fill-paragraph needs to discriminate only between 2 chunks
> of data.

I think you mean "2 chunks of data that must only be different in 
positioning and presence of newlines". Then yes, the odds of a collision 
must be slim. Still, I haven't seen (or performed) a sufficient analysis 
to evaluate them.

>> b) Git has a global object index. It _can_ detect collisions, or at
>> least that detection can be implemented.
>
> And what to do when a collision is detected?

Abort the current operation? Wait 50ms and retry creating the commit? 
Not 100% how the file contents are indexed: e.g. whether mtime factors 
into its hash value, too.

> Back to the topic, your suggetion about comparing the pre- and post-
> contents of the paragraph (and avoiding huge copies of the pre- contents
> by restricting the copied area to the paragraph itself) does not work
> when the file contains just one paragraph. Try visiting a big CSV dump
> or log and press M-q. You can abort the operation with C-g, but if Emacs
> starts to swap like crazy or exceeds the process memory limit and it is
> killed...

You can choose to skip the "did it changed" check if the region to check 
is too long. If the dump was one huge line, we can be confident that it 
will be changed upon filling.

Message #290 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Jaakov <j_k_v <at> ro.ru>
Cc: 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1; `fill-paragraph' should not always put the
 buffer as modified
Date: Sun, 27 Mar 2016 23:36:22 +0200

For every hash function X, there is a set m(X) of people who are likely 
to abhor hash-checked-implementation of buffer modification:

m(X)  =  the maintainers of databases of collisions for X.

For each X: today, m(X) may be empty, small, or large, but, as the time 
goes by and hash functions remain, m(X) will be growing.

Message #293 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Óscar Fuentes <ofv <at> wanadoo.es>
To: Dmitry Gutov <dgutov <at> yandex.ru>
Cc: John Wiegley <jwiegley <at> gmail.com>, Lars Magne Ingebrigtsen <larsi <at> gnus.org>,
 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1;
 `fill-paragraph' should not always put the buffer as modified
Date: Mon, 28 Mar 2016 00:03:05 +0200

Dmitry Gutov <dgutov <at> yandex.ru> writes:

> On 03/28/2016 12:05 AM, Óscar Fuentes wrote:
>
>> I guess that the extra bits of entropy (160 vs 128) was a "fuzzy-warm"
>> factor too on using SHA-1 instead of MD5. Git must avoid collisions
>> among potentially hundreds of millions of objects (repos with that size
>> already exists or will exist on the near future.)
>
> Are there fewer different texts we'd have to be able to discern?

As stated on my previous message, statistically it is entirely different
to avoid collisions among pairs of objects than within arbitrarily large
sets. For this case we are on the pair scenario. IIUC, Lars' idea about
using hashes on buffers to test for modification also is the pair case.

>> Each and every hash
>> must be different from all the others and hence avoid the Birthday
>> Problem. Anyway, 128 bit hashes still would be good enough for those
>> huge repos. fill-paragraph needs to discriminate only between 2 chunks
>> of data.
>
> I think you mean "2 chunks of data that must only be different in
> positioning and presence of newlines". Then yes, the odds of a
> collision must be slim. Still, I haven't seen (or performed) a
> sufficient analysis to evaluate them.

For naturally occurring modifications (opposed to specially chosen
modifications with the purpose of creating collisions), inserting
newlines or any string makes little difference to the hash algorithm.

>>> b) Git has a global object index. It _can_ detect collisions, or at
>>> least that detection can be implemented.
>>
>> And what to do when a collision is detected?
>
> Abort the current operation? Wait 50ms and retry creating the commit?
> Not 100% how the file contents are indexed: e.g. whether mtime factors
> into its hash value, too.

This would not work, for several reasons (colliding commits exists
before they are merged or incorporated into a repo where they met; file
and tree objects, whose content is identified by their SHA-1 hashes, can
not be "retried"; etc.) Having a collision is something that Should Not
Happen on Git, and the designers chose a crypto hash precisely because
those algorithms are the best at avoiding collisions.

>> Back to the topic, your suggetion about comparing the pre- and post-
>> contents of the paragraph (and avoiding huge copies of the pre- contents
>> by restricting the copied area to the paragraph itself) does not work
>> when the file contains just one paragraph. Try visiting a big CSV dump
>> or log and press M-q. You can abort the operation with C-g, but if Emacs
>> starts to swap like crazy or exceeds the process memory limit and it is
>> killed...
>
> You can choose to skip the "did it changed" check if the region to
> check is too long. If the dump was one huge line, we can be confident
> that it will be changed upon filling.

What about a file with lots of lines? If you intentionally press M-q on
such a file and see the modified indicator, you either will assume that
the file changed or use `diff-buffer-with-file' to check for
modifications and possibly be greeted with a very long (possibly longer
than the original file) diff that will render Emacs to its feet.

Using the hash approach will put the "too long" threshold on a higher
level (or eliminate it altogether), does not require extra memory and it
is simpler to implement.

Dmitry, if your proposal about comparing the paragraphs is motivated
only by your fear of hash collisions, you are way out off the mark there
:-)

Message #296 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Andreas Röhler <andreas.roehler <at> easy-emacs.de>
To: Óscar Fuentes <ofv <at> wanadoo.es>
Cc: 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1; `fill-paragraph' should not always put the
 buffer as modified
Date: Mon, 28 Mar 2016 10:01:54 +0200

On 27.03.2016 17:09, Óscar Fuentes wrote:
> Andreas Röhler <andreas.roehler <at> easy-emacs.de> writes:
>
>> Another solution would hash only the paragraph in question, re-format
>> it in a temp buffer and replace original content only if changed.
> If we know the paragraph's begin and end points, we can hash just that
> range and then there is no need of a temporary buffer. I think that this
> condition is met on all cases.

Ideally fill-paragraph would know about the borders of action.

> Anyway, I've made some experiments and hashing a 160 MB file on a 8 year
> old 2.4 GHz 64 bit workstation takes 2.4 seconds.

If there is a faster solution, why not use them? Sure such a function 
isn't called somewhere repeatedly, travelling paragraph by paragraph?

Message #299 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Andreas Röhler <andreas.roehler <at> easy-emacs.de>
To: bug-gnu-emacs <at> gnu.org
Cc: Eli Zaretskii <eliz <at> gnu.org>
Subject: Re: bug#13949: 24.4.1; `fill-paragraph' should not always put the
 buffer as modified
Date: Mon, 28 Mar 2016 10:09:58 +0200

On 27.03.2016 16:56, Eli Zaretskii wrote:
>> From: Óscar Fuentes <ofv <at> wanadoo.es>
>> Date: Sun, 27 Mar 2016 05:31:19 +0200
>> Cc: Jaakov <j_k_v <at> ro.ru>, 13949 <at> debbugs.gnu.org
>>
>> diff --git a/lisp/textmodes/fill.el b/lisp/textmodes/fill.el
>> index 100e2a2..9e1f430 100644
>> --- a/lisp/textmodes/fill.el
>> +++ b/lisp/textmodes/fill.el
>> @@ -804,6 +804,7 @@ fill-paragraph
>>     (interactive (progn
>>   		 (barf-if-buffer-read-only)
>>   		 (list (if current-prefix-arg 'full) t)))
>> +  (setq h (if (buffer-modified-p) "" (secure-hash 'md5 (current-buffer))))
>>     (or
>>      ;; 1. Fill the region if it is active when called interactively.
>>      (and region transient-mark-mode mark-active
>> @@ -862,7 +863,10 @@ fill-paragraph
>>                        ;; fill-region.
>>                        (fill-region beg end justify)
>>                      (fill-region-as-paragraph beg end justify))))))
>> -     fill-pfx)))
>> +     fill-pfx))
>> +  (when (and (not (string= h ""))
>> +             (string= h (secure-hash 'md5 (current-buffer))))
>> +    (set-buffer-modified-p nil)))
> Thanks, but I'm not sure computing the hash is enough: the functions
> involved in refilling can change text properties, so the test should
> also account for that.
>
>
>

Maybe restrict the notion of changes here to all values, which an 
auto-save would store?

Message #302 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: "Petros Travioli" <travioli <at> gmx.de>
To: 13949 <at> debbugs.gnu.org
Subject: fill-paragraph is buggy, but using MD5 is even more buggy
Date: Mon, 28 Mar 2016 06:55:58 +0200

Some details on hashes. For eggheads.

Let's assume that you have never typed in some large number, say, 89434823472. And you have never seen it so far. And noone you know has ever seen this number. Neither have your colleagues, though you don't speak to them. On this empirical basis, you declare this number nonexistant for all practical purposes. Encountering this number in the future is less likely than the probability of finding oneself with those
two sweet blond ladies you've always wanted at your home in a horizontal position simultaneously. Since instead of the ladies, you have been accustomed to endure the society of your wife, well, you decide to use the value 89434823472 as an error value when, say, returning the file size.

Is it clever? Of course NO. There is absolutely nothing which precludes the next large file you see to have size exactly 89434823472. There is absolutely nothing which precludes the next two large files you see to have this size. As the time goes by, files get larger, so 89 GB files may be normal in a decade or so. (The ladies will never change their mind, though. Sigh.)

But that's exactly what happens when you are using hash functions to verify buffer equality, just with a more complicated mathematical formulation and at a slightly different scale.

So don't use hash functions to a two-sided correct answer to test buffer equality. For a one-sided answer (if hash(x) != hash(y) then x != y), you are fine.

Petros

Message #305 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Andreas Schwab <schwab <at> linux-m68k.org>
To: "Petros Travioli" <travioli <at> gmx.de>
Cc: 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: fill-paragraph is buggy,
 but using MD5 is even more buggy
Date: Mon, 28 Mar 2016 12:32:26 +0200

"Petros Travioli" <travioli <at> gmx.de> writes:

> But that's exactly what happens when you are using hash functions to
> verify buffer equality, just with a more complicated mathematical
> formulation and at a slightly different scale.
>
> So don't use hash functions to a two-sided correct answer to test buffer
> equality. For a one-sided answer (if hash(x) != hash(y) then x != y), you
> are fine.

There is a difference between a hash function and a cryptographic hash
function.  An inportant property of a cryptographic hash function is the
avalanche effect, that means a small change in the plaintext will result
in a big change in the hash value.  That makes such a hash function
suitable for the reverse condition x != y => hash(x) != hash(y), with a
very high probability of being true.

Andreas.

-- 
Andreas Schwab, schwab <at> linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Message #308 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Lars Magne Ingebrigtsen <larsi <at> gnus.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1;
 `fill-paragraph' should not always put the buffer as modified
Date: Mon, 28 Mar 2016 12:39:54 +0200

Lars Magne Ingebrigtsen <larsi <at> gnus.org> writes:

> for iterate_over_all_intervals
>   sha1_process_bytes(interval, len)

I completely forgot about the distinction between text property changes
that "count" and the ones that don't here.  Font locking, for instance,
runs with `with-silent-modifications' so those changes "don't count",
but there's nothing in the intervals themselves that you can examine
after the fact, as far as I can tell.  Is that correct?

So the question is, I guess: Does `M-q' does something to text
properties that we have to keep track of, or is it sufficient to just
hash the buffer contents to determine whether `M-q' did something?

(Please take any further discussions about how likely it is that an
editing change will end up with the same sha1 to the emacs-tangents
mailing list.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #311 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Lars Magne Ingebrigtsen <larsi <at> gnus.org>
To: 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1;
 `fill-paragraph' should not always put the buffer as modified
Date: Mon, 28 Mar 2016 13:27:49 +0200

I've now made a simple buffer hashing function (that does not take text
properties into account) to see how slow this would be.

The following form

(with-temp-buffer
  (dotimes (i 10000)
    (insert (make-string 100 ?k) "\n"))
  (benchmark-run 1000 (buffer-hash)))

takes 2.7 seconds (that's 1000 1MB buffers), and unsurprisingly

(with-temp-buffer
  (dotimes (i 10000000)
    (insert (make-string 100 ?k) "\n"))
  (benchmark-run 1 (progn (message "%s" (buffer-size) (buffer-hash)))))

(which is 1 1GB buffer) takes the same amount of time.  :-)

(This is on a 2.7GHz i5 from like five years ago.)

So on big buffers this isn't really something you'd want to run a lot,
but it's workable (especially since in the `M-q' case you'd only run it
if the buffer isn't already modified).

So...  

diff --git a/src/fns.c b/src/fns.c
index 0e3fc27..d027058 100644
--- a/src/fns.c
+++ b/src/fns.c
@@ -4737,6 +4737,22 @@ It should be the case that if (eq (funcall HASH x1) (funcall HASH x2))
 #include "sha256.h"
 #include "sha512.h"
 
+static Lisp_Object
+make_digest_string (Lisp_Object digest, int digest_size)
+{
+  unsigned char *p = SDATA (digest);
+  int i;
+
+  for (i = digest_size - 1; i >= 0; i--)
+    {
+      static char const hexdigit[16] = "0123456789abcdef";
+      int p_i = p[i];
+      p[2 * i] = hexdigit[p_i >> 4];
+      p[2 * i + 1] = hexdigit[p_i & 0xf];
+    }
+  return digest;
+}
+
 /* ALGORITHM is a symbol: md5, sha1, sha224 and so on. */
 
 static Lisp_Object
@@ -4936,17 +4952,7 @@ It should be the case that if (eq (funcall HASH x1) (funcall HASH x2))
 	     SSDATA (digest));
 
   if (NILP (binary))
-    {
-      unsigned char *p = SDATA (digest);
-      for (i = digest_size - 1; i >= 0; i--)
-	{
-	  static char const hexdigit[16] = "0123456789abcdef";
-	  int p_i = p[i];
-	  p[2 * i] = hexdigit[p_i >> 4];
-	  p[2 * i + 1] = hexdigit[p_i & 0xf];
-	}
-      return digest;
-    }
+    return make_digest_string (digest, digest_size);
   else
     return make_unibyte_string (SSDATA (digest), digest_size);
 }
@@ -4997,6 +5003,44 @@ It should be the case that if (eq (funcall HASH x1) (funcall HASH x2))
 {
   return secure_hash (algorithm, object, start, end, Qnil, Qnil, binary);
 }
+
+DEFUN ("buffer-hash", Fbuffer_hash, Sbuffer_hash, 0, 1, 0,
+       doc: /* Return a hash of the contents of BUFFER-OR-NAME.
+If nil, use the current buffer." */ )
+     (Lisp_Object buffer_or_name)
+{
+  Lisp_Object buffer;
+  struct buffer *b;
+  struct sha1_ctx ctx;
+  Lisp_Object digest = make_uninit_string (SHA1_DIGEST_SIZE * 2);
+
+  if (NILP (buffer_or_name))
+    buffer = Fcurrent_buffer ();
+  else
+    buffer = Fget_buffer (buffer_or_name);
+  if (NILP (buffer))
+    nsberror (buffer_or_name);
+
+  b = XBUFFER (buffer);
+  sha1_init_ctx (&ctx);
+
+  /* Process the first part of the buffer. */
+  sha1_process_bytes (BUF_BEG_ADDR (b),
+		      BUF_GPT_BYTE (b) - BUF_BEG_BYTE (b),
+		      &ctx);
+
+  /* If the gap is before the end of the buffer, process the last half
+     of the buffer. */
+  if (BUF_GPT_BYTE (b) < BUF_Z_BYTE (b))
+    sha1_process_bytes (BUF_GAP_END_ADDR (b),
+			BUF_Z_BYTE (b) - (BUF_GPT_BYTE (b) + BUF_GAP_SIZE (b)),
+			&ctx);
+
+  sha1_finish_ctx (&ctx, SSDATA (digest));
+  return make_digest_string (digest, SHA1_DIGEST_SIZE);
+}
+
+
 
 void
 syms_of_fns (void)
@@ -5156,6 +5200,7 @@ It should be the case that if (eq (funcall HASH x1) (funcall HASH x2))
   defsubr (&Sbase64_decode_string);
   defsubr (&Smd5);
   defsubr (&Ssecure_hash);
+  defsubr (&Sbuffer_hash);
   defsubr (&Slocale_info);
 
   hashtest_eq.name = Qeq;


-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #314 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Lars Magne Ingebrigtsen <larsi <at> gnus.org>
To: 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1;
 `fill-paragraph' should not always put the buffer as modified
Date: Mon, 28 Mar 2016 13:32:13 +0200

Lars Magne Ingebrigtsen <larsi <at> gnus.org> writes:

> +  /* If the gap is before the end of the buffer, process the last half
> +     of the buffer. */
> +  if (BUF_GPT_BYTE (b) < BUF_Z_BYTE (b))
> +    sha1_process_bytes (BUF_GAP_END_ADDR (b),
> +			BUF_Z_BYTE (b) - (BUF_GPT_BYTE (b) + BUF_GAP_SIZE (b)),
> +			&ctx);

Errr...  this is all wrong.  That's not how the gap thing works...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #317 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Lars Magne Ingebrigtsen <larsi <at> gnus.org>
To: 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1;
 `fill-paragraph' should not always put the buffer as modified
Date: Mon, 28 Mar 2016 13:39:00 +0200

Lars Magne Ingebrigtsen <larsi <at> gnus.org> writes:

> Errr...  this is all wrong.  That's not how the gap thing works...

This seems to do the trick:

  /* If the gap is before the end of the buffer, process the last half
     of the buffer. */
  if (BUF_GPT_BYTE (b) < BUF_Z_BYTE (b))
    sha1_process_bytes (BUF_GAP_END_ADDR (b),
			BUF_Z_ADDR (b) - BUF_GAP_END_ADDR (b),
			&ctx);


-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #320 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Lars Magne Ingebrigtsen <larsi <at> gnus.org>
To: 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1;
 `fill-paragraph' should not always put the buffer as modified
Date: Mon, 28 Mar 2016 15:46:16 +0200

Lars Magne Ingebrigtsen <larsi <at> gnus.org> writes:

> takes 2.7 seconds (that's 1000 1MB buffers), and unsurprisingly

With md5 it's 1.7 seconds, so that might be a better choice, although
we'd have to deal with an endless string of bug reports about md5 being
"broken" that we'd have to mark as "notabug wontfix closed" every three
days, I think.  :-)  So it might not be worth it.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #323 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Andreas Schwab <schwab <at> linux-m68k.org>
To: "Petros Travioli" <travioli <at> gmx.de>
Cc: 13949 <at> debbugs.gnu.org
Subject: Re: Aw: Re: bug#13949: fill-paragraph is buggy,
 but using MD5 is even more buggy
Date: Mon, 28 Mar 2016 16:31:32 +0200

"Petros Travioli" <travioli <at> gmx.de> writes:

>> > But that's exactly what happens when you are using hash functions to
>> > verify buffer equality, just with a more complicated mathematical
>> > formulation and at a slightly different scale.
>> >
>> > So don't use hash functions to a two-sided correct answer to test buffer
>> > equality. For a one-sided answer (if hash(x) != hash(y) then x != y), you
>> > are fine.
>> 
>> There is a difference between a hash function and a cryptographic hash
>> function.  An inportant property of a cryptographic hash function is the
>> avalanche effect, that means a small change in the plaintext will result
>> in a big change in the hash value.  That makes such a hash function
>> suitable for the reverse condition x != y => hash(x) != hash(y), with a
>> very high probability of being true.
>> 
> So far most old crypto functions have been broken. There is no doubt this will happen to any newer one sooner or later. If any single person would lose his work because of a random collision, this is an argument agains crypto hash functions.

This is irrelevant.  See avalanche effect.

Andreas.

-- 
Andreas Schwab, schwab <at> linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Message #326 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Lars Magne Ingebrigtsen <larsi <at> gnus.org>
Cc: 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1;
 `fill-paragraph' should not always put the buffer as modified
Date: Mon, 28 Mar 2016 18:15:32 +0300

> From: Lars Magne Ingebrigtsen <larsi <at> gnus.org>
> Cc: 13949 <at> debbugs.gnu.org
> Date: Mon, 28 Mar 2016 12:39:54 +0200
> 
> Lars Magne Ingebrigtsen <larsi <at> gnus.org> writes:
> 
> > for iterate_over_all_intervals
> >   sha1_process_bytes(interval, len)
> 
> I completely forgot about the distinction between text property changes
> that "count" and the ones that don't here.  Font locking, for instance,
> runs with `with-silent-modifications' so those changes "don't count",
> but there's nothing in the intervals themselves that you can examine
> after the fact, as far as I can tell.  Is that correct?

The intervals do store the property itself, but I actually don't
understand why should you bother discerning between faces and the
other properties.  If the buffer text is really unchanged, the face
properties will be identical as well, right?

> So the question is, I guess: Does `M-q' does something to text
> properties that we have to keep track of, or is it sufficient to just
> hash the buffer contents to determine whether `M-q' did something?

There are a couple of properties that have special meaning for fill.el
functions, see there.

However, I thought you are working on infrastructure that isn't
supposed to be limited to what M-q does.  Was I mistaken?

Message #329 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Lars Magne Ingebrigtsen <larsi <at> gnus.org>
Cc: 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1;
 `fill-paragraph' should not always put the buffer as modified
Date: Mon, 28 Mar 2016 18:29:08 +0300

> From: Lars Magne Ingebrigtsen <larsi <at> gnus.org>
> Date: Mon, 28 Mar 2016 13:27:49 +0200
> 
> I've now made a simple buffer hashing function (that does not take text
> properties into account) to see how slow this would be.
> 
> The following form
> 
> (with-temp-buffer
>   (dotimes (i 10000)
>     (insert (make-string 100 ?k) "\n"))
>   (benchmark-run 1000 (buffer-hash)))
> 
> takes 2.7 seconds (that's 1000 1MB buffers), and unsurprisingly
> 
> (with-temp-buffer
>   (dotimes (i 10000000)
>     (insert (make-string 100 ?k) "\n"))
>   (benchmark-run 1 (progn (message "%s" (buffer-size) (buffer-hash)))))
> 
> (which is 1 1GB buffer) takes the same amount of time.  :-)
> 
> (This is on a 2.7GHz i5 from like five years ago.)

Optimized build or unoptimized?

Message #332 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Lars Magne Ingebrigtsen <larsi <at> gnus.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1;
 `fill-paragraph' should not always put the buffer as modified
Date: Mon, 28 Mar 2016 17:39:18 +0200

Eli Zaretskii <eliz <at> gnu.org> writes:

> The intervals do store the property itself, but I actually don't
> understand why should you bother discerning between faces and the
> other properties.  If the buffer text is really unchanged, the face
> properties will be identical as well, right?

[...]

> There are a couple of properties that have special meaning for fill.el
> functions, see there.
>
> However, I thought you are working on infrastructure that isn't
> supposed to be limited to what M-q does.  Was I mistaken?

Well, this is all kinda exploratory.  What's feasible to do in general,
and if nothing general can be done, can we still do something for `M-q'?

In general: It would be really nice if `buffer-modified-p' really said
whether the buffer was changed or not.  That is, load a file, add an
"a", delete the "a", and Emacs says "unchanged".  If we had that
mechanism, `M-q' would fall out naturally as a result.

But as far as I can tell, this isn't really feasible because of the way
we handle text properties: We consider some of them to change the
buffer, and some of them to not change the buffer.  And it doesn't look
like we actually store that information in the text properties
themselves.  (Please correct me if I'm wrong or you have an idea for how
to make this work.)

So on to the specific problem of `M-q' again: If we think the general
solution is a no go, would it still make sense to do the hash-the-buffer
solution just for `M-q'?  That is, does `M-q' ever change text
properties in a way that we want maintained without changing the text
itself?  I think the answer to the last question is "no", but I'm not
sure.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #335 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Lars Magne Ingebrigtsen <larsi <at> gnus.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1;
 `fill-paragraph' should not always put the buffer as modified
Date: Mon, 28 Mar 2016 17:39:58 +0200

Eli Zaretskii <eliz <at> gnu.org> writes:

> Optimized build or unoptimized?

Let's see...  src/Makefile says "CFLAGS = -g3 -O2".

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #338 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Óscar Fuentes <ofv <at> wanadoo.es>
To: Lars Magne Ingebrigtsen <larsi <at> gnus.org>
Cc: Eli Zaretskii <eliz <at> gnu.org>, 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1;
 `fill-paragraph' should not always put the buffer as modified
Date: Mon, 28 Mar 2016 17:52:26 +0200

Lars Magne Ingebrigtsen <larsi <at> gnus.org> writes:

[snip]

> So on to the specific problem of `M-q' again: If we think the general
> solution is a no go, would it still make sense to do the hash-the-buffer
> solution just for `M-q'?  That is, does `M-q' ever change text
> properties in a way that we want maintained without changing the text
> itself?  I think the answer to the last question is "no", but I'm not
> sure.

To be precise, the question is not about *maintaining* the changed
properties (nobody suggested to throw them away) but about marking the
buffer as modified just because those properties [might have] changed.

Message #341 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Lars Magne Ingebrigtsen <larsi <at> gnus.org>
To: Óscar Fuentes <ofv <at> wanadoo.es>
Cc: Eli Zaretskii <eliz <at> gnu.org>, 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1;
 `fill-paragraph' should not always put the buffer as modified
Date: Mon, 28 Mar 2016 18:29:21 +0200

Óscar Fuentes <ofv <at> wanadoo.es> writes:

> Lars Magne Ingebrigtsen <larsi <at> gnus.org> writes:
>
> [snip]
>
>> So on to the specific problem of `M-q' again: If we think the general
>> solution is a no go, would it still make sense to do the hash-the-buffer
>> solution just for `M-q'?  That is, does `M-q' ever change text
>> properties in a way that we want maintained without changing the text
>> itself?  I think the answer to the last question is "no", but I'm not
>> sure.
>
> To be precise, the question is not about *maintaining* the changed
> properties (nobody suggested to throw them away) but about marking the
> buffer as modified just because those properties [might have] changed.

I've been trying to `M-q' stuff in a handful of different modes, and
there doesn't seem to be any text property changes at all, as far as I
can see.  (When the text doesn't change, that is.)

So I could just commit the changes I've got here (the new `hash-buffer'
function and the `M-q' changes, along with unit tests and doc fixes),
and then we can see how this feels and whether there are any gotchas.

And then ponder the more general issues later.

Does that sound OK to everybody?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #344 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: "Petros Travioli" <travioli <at> gmx.de>
To: "Andreas Schwab" <schwab <at> linux-m68k.org>, 13949 <at> debbugs.gnu.org
Subject: Aw: Re: bug#13949: fill-paragraph is buggy, but using MD5 is even
 more buggy
Date: Mon, 28 Mar 2016 15:29:19 +0200

> > But that's exactly what happens when you are using hash functions to
> > verify buffer equality, just with a more complicated mathematical
> > formulation and at a slightly different scale.
> >
> > So don't use hash functions to a two-sided correct answer to test buffer
> > equality. For a one-sided answer (if hash(x) != hash(y) then x != y), you
> > are fine.
> 
> There is a difference between a hash function and a cryptographic hash
> function.  An inportant property of a cryptographic hash function is the
> avalanche effect, that means a small change in the plaintext will result
> in a big change in the hash value.  That makes such a hash function
> suitable for the reverse condition x != y => hash(x) != hash(y), with a
> very high probability of being true.
> 
So far most old crypto functions have been broken. There is no doubt this will happen to any newer one sooner or later. If any single person would lose his work because of a random collision, this is an argument agains crypto hash functions.

I am citing RFC 6151 (https://tools.ietf.org/html/rfc6151):

"MD5 is no longer acceptable where collision resistance is required..."

If the developers (I think, it was Eli who embraced the patch) are so sure about collision freedom:

Eli, if you are so sure about MD5, please post your password MD5 hash here with login data and a consent that anyone is allowed to hack in. I do not want to go into prison. Then wait for, say, 1 week.

Message #347 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: "Petros Travioli" <travioli <at> gmx.de>
To: 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: fill-paragraph is buggy, but using MD5 is even more
 buggy
Date: Mon, 28 Mar 2016 17:16:18 +0200

To any of the developers (especially Eli, who embraced the MD5 patch) who is so sure about collision freedom:

Please be so kind to post your password MD5 hash here with login data and a consent that anyone is allowed to hack in. I do not want to go into jail. Then wait for, say, 1 week.

Message #350 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Lars Magne Ingebrigtsen <larsi <at> gnus.org>
Cc: ofv <at> wanadoo.es, 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1;
 `fill-paragraph' should not always put the buffer as modified
Date: Mon, 28 Mar 2016 20:04:16 +0300

> From: Lars Magne Ingebrigtsen <larsi <at> gnus.org>
> Cc: 13949 <at> debbugs.gnu.org, Eli Zaretskii <eliz <at> gnu.org>
> Date: Mon, 28 Mar 2016 18:29:21 +0200
> 
> So I could just commit the changes I've got here (the new `hash-buffer'
> function and the `M-q' changes, along with unit tests and doc fixes),
> and then we can see how this feels and whether there are any gotchas.
> 
> And then ponder the more general issues later.
> 
> Does that sound OK to everybody?

As optional behavior, yes.  (Plus documentation, please.)

Message #353 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Lars Magne Ingebrigtsen <larsi <at> gnus.org>
To: "Petros Travioli" <travioli <at> gmx.de>
Cc: 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: fill-paragraph is buggy,
 but using MD5 is even more buggy
Date: Mon, 28 Mar 2016 19:05:37 +0200

"Petros Travioli" <travioli <at> gmx.de> writes:

> To any of the developers (especially Eli, who embraced the MD5 patch)
> who is so sure about collision freedom:
>
> Please be so kind to post your password MD5 hash here with login data
> and a consent that anyone is allowed to hack in. I do not want to go
> into jail. Then wait for, say, 1 week.

This is irrelevant.

Please take discussion of cryptographic hash collisions to the
emacs-tangents mailing list.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #356 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: "Petros Travioli" <travioli <at> gmx.de>
Cc: 13949 <at> debbugs.gnu.org, schwab <at> linux-m68k.org
Subject: Re: bug#13949: Aw: Re: bug#13949: fill-paragraph is buggy,
 but using MD5 is even more buggy
Date: Mon, 28 Mar 2016 20:05:38 +0300

> From: "Petros Travioli" <travioli <at> gmx.de>
> Date: Mon, 28 Mar 2016 15:29:19 +0200
> 
> If the developers (I think, it was Eli who embraced the patch) are so sure about collision freedom:
> 
> Eli, if you are so sure about MD5, please post your password MD5 hash here with login data and a consent that anyone is allowed to hack in. I do not want to go into prison. Then wait for, say, 1 week.

Who is that "Eli" you are addressing this to?  It's not me, I'm sure.

Message #359 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: "Petros Travioli" <travioli <at> gmx.de>
Cc: 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: fill-paragraph is buggy,
 but using MD5 is even more buggy
Date: Mon, 28 Mar 2016 20:06:20 +0300

> From: "Petros Travioli" <travioli <at> gmx.de>
> Date: Mon, 28 Mar 2016 17:16:18 +0200
> 
> Eli, who embraced the MD5 patch

I did?

Message #362 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Lars Magne Ingebrigtsen <larsi <at> gnus.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: ofv <at> wanadoo.es, 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1;
 `fill-paragraph' should not always put the buffer as modified
Date: Mon, 28 Mar 2016 19:07:12 +0200

Eli Zaretskii <eliz <at> gnu.org> writes:

>> From: Lars Magne Ingebrigtsen <larsi <at> gnus.org>
>> Cc: 13949 <at> debbugs.gnu.org, Eli Zaretskii <eliz <at> gnu.org>
>> Date: Mon, 28 Mar 2016 18:29:21 +0200
>> 
>> So I could just commit the changes I've got here (the new `hash-buffer'
>> function and the `M-q' changes, along with unit tests and doc fixes),
>> and then we can see how this feels and whether there are any gotchas.
>> 
>> And then ponder the more general issues later.
>> 
>> Does that sound OK to everybody?
>
> As optional behavior, yes.

With a variable to switch it off or on?  I'm not quite sure I see the
need, but I'm not against that...

> (Plus documentation, please.)

Of course.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #369 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Lars Magne Ingebrigtsen <larsi <at> gnus.org>
Cc: ofv <at> wanadoo.es, 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1;
 `fill-paragraph' should not always put the buffer as modified
Date: Mon, 28 Mar 2016 20:37:39 +0300

> From: Lars Magne Ingebrigtsen <larsi <at> gnus.org>
> Cc: ofv <at> wanadoo.es,  13949 <at> debbugs.gnu.org
> Date: Mon, 28 Mar 2016 19:07:12 +0200
> 
> Eli Zaretskii <eliz <at> gnu.org> writes:
> 
> >> Does that sound OK to everybody?
> >
> > As optional behavior, yes.
> 
> With a variable to switch it off or on?

Yes, I think that'd be good enough.

> I'm not quite sure I see the need, but I'm not against that...

You've heard here that some people think the current behavior is
correct, so we should expect users to ask "how do I get the old
behavior back?"  We should have an answer for that.

> > (Plus documentation, please.)
> 
> Of course.

Thanks.

Message #372 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Lars Magne Ingebrigtsen <larsi <at> gnus.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: ofv <at> wanadoo.es, 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1;
 `fill-paragraph' should not always put the buffer as modified
Date: Mon, 28 Mar 2016 19:45:37 +0200

Eli Zaretskii <eliz <at> gnu.org> writes:

>> With a variable to switch it off or on?
>
> Yes, I think that'd be good enough.
>
>> I'm not quite sure I see the need, but I'm not against that...
>
> You've heard here that some people think the current behavior is
> correct, so we should expect users to ask "how do I get the old
> behavior back?"  We should have an answer for that.

Oh, I think I missed that.  I thought they were only being confused
about how cryptographic hashes worked?  There were somebody who really
wanted the "`M-q' didn't change anything, but I still want Emacs to say
that the buffer was changed" behaviour back?

In that case, I can add a variable for sure, but is that really the
case?  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #375 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Lars Magne Ingebrigtsen <larsi <at> gnus.org>
Cc: ofv <at> wanadoo.es, 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: 24.4.1;
 `fill-paragraph' should not always put the buffer as modified
Date: Mon, 28 Mar 2016 21:17:19 +0300

> From: Lars Magne Ingebrigtsen <larsi <at> gnus.org>
> Cc: ofv <at> wanadoo.es,  13949 <at> debbugs.gnu.org
> Date: Mon, 28 Mar 2016 19:45:37 +0200
> 
> > You've heard here that some people think the current behavior is
> > correct, so we should expect users to ask "how do I get the old
> > behavior back?"  We should have an answer for that.
> 
> Oh, I think I missed that.  I thought they were only being confused
> about how cryptographic hashes worked?  There were somebody who really
> wanted the "`M-q' didn't change anything, but I still want Emacs to say
> that the buffer was changed" behaviour back?
> 
> In that case, I can add a variable for sure, but is that really the
> case?  :-)

That's my understanding.  And in any case, a change such as this one
shouldn't be without a fire escape.

Message #378 received at 13949 <at> debbugs.gnu.org (full text, mbox):

From: "Petros Travioli" <travioli <at> gmx.de>
To: eliz <at> gnu.org, 13949 <at> debbugs.gnu.org
Subject: Re: bug#13949: fill-paragraph is buggy, but using MD5 is even more
 buggy
Date: Mon, 28 Mar 2016 21:03:20 +0200

> > Eli, who embraced the MD5 patch
> 
> I did?
> 
I'm sorry, you are right, you did not. It was the suggestion of Óscar on
Sun, 27 Mar 2016 05:31:19 +0200

Anyhow, whoever introduces hashing is welcome to put his/her password hash and login data here. With a consent that the account is alowed to be hacked. For 1 week.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.