GNU bug report logs -
#44638
[PATCH 1/2] autorevert: don't reuse existing watch descriptors
Previous Next
Reported by: Spencer Baugh <sbaugh <at> catern.com>
Date: Sat, 14 Nov 2020 16:56:02 UTC
Severity: normal
Tags: patch
Fixed in version 28.1
Done: Michael Albinus <michael.albinus <at> gmx.de>
Bug is archived. No further changes may be made.
To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 44638 in the body.
You can then email your comments to 44638 AT debbugs.gnu.org in the normal way.
Toggle the display of automated, internal messages from the tracker.
Report forwarded
to
bug-gnu-emacs <at> gnu.org
:
bug#44638
; Package
emacs
.
(Sat, 14 Nov 2020 16:56:02 GMT)
Full text and
rfc822 format available.
Acknowledgement sent
to
Spencer Baugh <sbaugh <at> catern.com>
:
New bug report received and forwarded. Copy sent to
bug-gnu-emacs <at> gnu.org
.
(Sat, 14 Nov 2020 16:56:02 GMT)
Full text and
rfc822 format available.
Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):
Previously, when enabling autorevert for a new buffer, we would search
the buffers already registered with autorevert to see if any of them
had the same filename.
This is very slow with a large number of buffers - with 1000, it takes
2 seconds on my system. This 2-second overhead is paid for every new
file opened.
But this is an unnecesary optimization; registering the same file
twice with file-notify has minimal or no overhead, depending on the
implementation.
In fact, file-notify has some baked-in overhead to support registering
the same file twice without problems. For example, inotify on Linux
returns the same inotify watch descriptor when the same file is
registered twice; file-notify adds an additional uniquifying id so
that all watch descriptors are unique in Emacs, even with inotify.
We can rely on file-notify's existing support for handling the same
file being registered twice. We don't need this slow and complex
logic.
With this code deleted, enabling autorevert for a new buffer is
essentially instant even with 1000 buffers.
---
lisp/autorevert.el | 29 +++++++----------------------
1 file changed, 7 insertions(+), 22 deletions(-)
diff --git a/lisp/autorevert.el b/lisp/autorevert.el
index 046ea2b5d6..d5bb75c2f1 100644
--- a/lisp/autorevert.el
+++ b/lisp/autorevert.el
@@ -650,30 +650,15 @@ will use an up-to-date value of `auto-revert-interval'."
(string-match auto-revert-notify-exclude-dir-regexp
(expand-file-name default-directory))
(file-symlink-p (or buffer-file-name default-directory)))
- ;; Check, whether this has been activated already.
(let ((file (if buffer-file-name
(expand-file-name buffer-file-name default-directory)
(expand-file-name default-directory))))
- (maphash
- (lambda (key _value)
- (when (and
- (file-notify-valid-p key)
- (equal (file-notify--watch-absolute-filename
- (gethash key file-notify-descriptors))
- (directory-file-name file))
- (equal (file-notify--watch-callback
- (gethash key file-notify-descriptors))
- 'auto-revert-notify-handler))
- (setq auto-revert-notify-watch-descriptor key)))
- auto-revert--buffers-by-watch-descriptor)
- ;; Create a new watch if needed.
- (unless auto-revert-notify-watch-descriptor
- (setq auto-revert-notify-watch-descriptor
- (ignore-errors
- (file-notify-add-watch
- file
- (if buffer-file-name '(change attribute-change) '(change))
- 'auto-revert-notify-handler))))
+ (setq auto-revert-notify-watch-descriptor
+ (ignore-errors
+ (file-notify-add-watch
+ file
+ (if buffer-file-name '(change attribute-change) '(change))
+ 'auto-revert-notify-handler))))
(when auto-revert-notify-watch-descriptor
(setq auto-revert-notify-modified-p t)
(puthash
@@ -682,7 +667,7 @@ will use an up-to-date value of `auto-revert-interval'."
(gethash auto-revert-notify-watch-descriptor
auto-revert--buffers-by-watch-descriptor))
auto-revert--buffers-by-watch-descriptor)
- (add-hook 'kill-buffer-hook #'auto-revert-notify-rm-watch nil t)))))
+ (add-hook 'kill-buffer-hook #'auto-revert-notify-rm-watch nil t))))
;; If we have file notifications, we want to update the auto-revert buffers
;; immediately when a notification occurs. Since file updates can happen very
--
2.28.0
Information forwarded
to
bug-gnu-emacs <at> gnu.org
:
bug#44638
; Package
emacs
.
(Sat, 14 Nov 2020 17:23:02 GMT)
Full text and
rfc822 format available.
Message #8 received at 44638 <at> debbugs.gnu.org (full text, mbox):
> From: Spencer Baugh <sbaugh <at> catern.com>
> Date: Sat, 14 Nov 2020 11:54:58 -0500
> Cc: Spencer Baugh <sbaugh <at> catern.com>
>
> Previously, when enabling autorevert for a new buffer, we would search
> the buffers already registered with autorevert to see if any of them
> had the same filename.
>
> This is very slow with a large number of buffers - with 1000, it takes
> 2 seconds on my system. This 2-second overhead is paid for every new
> file opened.
>
> But this is an unnecesary optimization; registering the same file
> twice with file-notify has minimal or no overhead, depending on the
> implementation.
Emacs actually watches the file's directory, not the file itself. The
directory is what's registered with inotify and other similar
backends.
> In fact, file-notify has some baked-in overhead to support registering
> the same file twice without problems. For example, inotify on Linux
> returns the same inotify watch descriptor when the same file is
> registered twice; file-notify adds an additional uniquifying id so
> that all watch descriptors are unique in Emacs, even with inotify.
Again, we watch the directory of the file, so what inotify does with
files is not really relevant, IMO. I wonder what that means for the
changes you propose.
Information forwarded
to
bug-gnu-emacs <at> gnu.org
:
bug#44638
; Package
emacs
.
(Sat, 14 Nov 2020 21:20:02 GMT)
Full text and
rfc822 format available.
Message #11 received at 44638 <at> debbugs.gnu.org (full text, mbox):
Eli Zaretskii <eliz <at> gnu.org> writes:
>> From: Spencer Baugh <sbaugh <at> catern.com>
>> Date: Sat, 14 Nov 2020 11:54:58 -0500
>> Cc: Spencer Baugh <sbaugh <at> catern.com>
>>
>> Previously, when enabling autorevert for a new buffer, we would search
>> the buffers already registered with autorevert to see if any of them
>> had the same filename.
>>
>> This is very slow with a large number of buffers - with 1000, it takes
>> 2 seconds on my system. This 2-second overhead is paid for every new
>> file opened.
>>
>> But this is an unnecesary optimization; registering the same file
>> twice with file-notify has minimal or no overhead, depending on the
>> implementation.
>
> Emacs actually watches the file's directory, not the file itself. The
> directory is what's registered with inotify and other similar
> backends.
>
>> In fact, file-notify has some baked-in overhead to support registering
>> the same file twice without problems. For example, inotify on Linux
>> returns the same inotify watch descriptor when the same file is
>> registered twice; file-notify adds an additional uniquifying id so
>> that all watch descriptors are unique in Emacs, even with inotify.
>
> Again, we watch the directory of the file, so what inotify does with
> files is not really relevant, IMO. I wonder what that means for the
> changes you propose.
Ah. Well, the inotify comment was just an example. Even given that Emacs
watches directories rather than files, I think this change is for the
best - the overhead of registering 5 files directly instead of 1
underlying directory is minimal.
Information forwarded
to
bug-gnu-emacs <at> gnu.org
:
bug#44638
; Package
emacs
.
(Mon, 30 Nov 2020 18:02:02 GMT)
Full text and
rfc822 format available.
Message #14 received at 44638 <at> debbugs.gnu.org (full text, mbox):
Spencer Baugh <sbaugh <at> catern.com> writes:
> Eli Zaretskii <eliz <at> gnu.org> writes:
>> Again, we watch the directory of the file, so what inotify does with
>> files is not really relevant, IMO. I wonder what that means for the
>> changes you propose.
>
> Ah. Well, the inotify comment was just an example. Even given that Emacs
> watches directories rather than files, I think this change is for the
> best - the overhead of registering 5 files directly instead of 1
> underlying directory is minimal.
So does this kind of change still seem plausible? Again, the overhead
of additional watches for the same file in the operating system is
minimal, and is rare anyway.
The alternative to this change is to add a new hash table to autorevert
which maps filenames to file-notify watches, so that autorevert can look
up duplicate watches efficiently in O(1) instead of iterating over all
watches in O(n) as it does now.
I started doing that originally, but concluded it was unnecessary since
sharing watches is unnecessary. But it's a much smaller change to
behavior while still getting the same performance gain, so if you'd
prefer that, I can do it.
Information forwarded
to
bug-gnu-emacs <at> gnu.org
:
bug#44638
; Package
emacs
.
(Mon, 30 Nov 2020 18:22:02 GMT)
Full text and
rfc822 format available.
Message #17 received at 44638 <at> debbugs.gnu.org (full text, mbox):
> From: Spencer Baugh <sbaugh <at> catern.com>
> Cc: 44638 <at> debbugs.gnu.org
> Date: Mon, 30 Nov 2020 13:01:43 -0500
>
> Spencer Baugh <sbaugh <at> catern.com> writes:
> > Eli Zaretskii <eliz <at> gnu.org> writes:
> >> Again, we watch the directory of the file, so what inotify does with
> >> files is not really relevant, IMO. I wonder what that means for the
> >> changes you propose.
> >
> > Ah. Well, the inotify comment was just an example. Even given that Emacs
> > watches directories rather than files, I think this change is for the
> > best - the overhead of registering 5 files directly instead of 1
> > underlying directory is minimal.
>
> So does this kind of change still seem plausible?
I'd expect Michael Albinus to chime in on this, as autorevert is his
domain. Michael?
Information forwarded
to
bug-gnu-emacs <at> gnu.org
:
bug#44638
; Package
emacs
.
(Mon, 30 Nov 2020 18:33:02 GMT)
Full text and
rfc822 format available.
Message #20 received at 44638 <at> debbugs.gnu.org (full text, mbox):
Eli Zaretskii <eliz <at> gnu.org> writes:
>> So does this kind of change still seem plausible?
>
> I'd expect Michael Albinus to chime in on this, as autorevert is his
> domain. Michael?
It's already on my todo list. Hmm, too much items there. I'll try to
find a sufficient time slot tomorrow.
Sorry for not being as responsive as I should :-(
Best regards, Michael.
Information forwarded
to
bug-gnu-emacs <at> gnu.org
:
bug#44638
; Package
emacs
.
(Tue, 01 Dec 2020 20:17:01 GMT)
Full text and
rfc822 format available.
Message #23 received at 44638 <at> debbugs.gnu.org (full text, mbox):
Michael Albinus <michael.albinus <at> gmx.de> writes:
> It's already on my todo list. Hmm, too much items there. I'll try to
> find a sufficient time slot tomorrow.
As expected, it takes more time. Currently I'm writing a test case which
covers the problem. This must run then for different backends with and
w/o your patch, at least inotify, gfilenotify and kqueue. If possible
also for w32notify.
Best regards, Michael.
Information forwarded
to
bug-gnu-emacs <at> gnu.org
:
bug#44638
; Package
emacs
.
(Wed, 02 Dec 2020 03:21:01 GMT)
Full text and
rfc822 format available.
Message #26 received at 44638 <at> debbugs.gnu.org (full text, mbox):
> From: Michael Albinus <michael.albinus <at> gmx.de>
> Cc: Eli Zaretskii <eliz <at> gnu.org>, 44638 <at> debbugs.gnu.org
> Date: Tue, 01 Dec 2020 21:16:25 +0100
>
> Michael Albinus <michael.albinus <at> gmx.de> writes:
>
> > It's already on my todo list. Hmm, too much items there. I'll try to
> > find a sufficient time slot tomorrow.
>
> As expected, it takes more time. Currently I'm writing a test case which
> covers the problem. This must run then for different backends with and
> w/o your patch, at least inotify, gfilenotify and kqueue. If possible
> also for w32notify.
Thanks, let me know if I can help with the latter.
Information forwarded
to
bug-gnu-emacs <at> gnu.org
:
bug#44638
; Package
emacs
.
(Wed, 02 Dec 2020 15:18:01 GMT)
Full text and
rfc822 format available.
Message #29 received at 44638 <at> debbugs.gnu.org (full text, mbox):
Eli Zaretskii <eliz <at> gnu.org> writes:
>> As expected, it takes more time. Currently I'm writing a test case which
>> covers the problem. This must run then for different backends with and
>> w/o your patch, at least inotify, gfilenotify and kqueue. If possible
>> also for w32notify.
>
> Thanks, let me know if I can help with the latter.
I've pushed a new test of autorevert-tests.el to master,
auto-revert-test07-auto-revert-several-buffers. It ought to cover the
scenario described in this report. It passes for me with the inotify,
gfilenotify and kqueue backends, which is fine. There are some problems
with remote backends (inotifywait and gio), but since there are also
problems with other tests, I wouldn't count on these failures for the
propsed patch.
Eli, could you run autorevert-test on MS Windows, for the w32notify
backend? Either a complete run, or just
$ make -C test autorevert-tests SELECTOR=auto-revert-test07-auto-revert-several-buffers
Thanks, and best regards, Michael.
Information forwarded
to
bug-gnu-emacs <at> gnu.org
:
bug#44638
; Package
emacs
.
(Wed, 02 Dec 2020 15:42:02 GMT)
Full text and
rfc822 format available.
Message #32 received at 44638 <at> debbugs.gnu.org (full text, mbox):
> From: Michael Albinus <michael.albinus <at> gmx.de>
> Cc: sbaugh <at> catern.com, 44638 <at> debbugs.gnu.org
> Date: Wed, 02 Dec 2020 16:17:01 +0100
>
> Eli, could you run autorevert-test on MS Windows, for the w32notify
> backend? Either a complete run, or just
>
> $ make -C test autorevert-tests SELECTOR=auto-revert-test07-auto-revert-several-buffers
auto-revert-test07-auto-revert-several-buffers succeeds. Running the
entire test finds 2 failures:
2 unexpected results:
FAILED auto-revert-test04-auto-revert-mode-dired
FAILED auto-revert-test05-global-notify
The evidence for the first failure is:
Test auto-revert-test04-auto-revert-mode-dired condition:
(ert-test-failed
((should-not
(string-match name
(substring-no-properties ...)))
:form
(string-match "auto-revert-testhf8BmX" " c:/Documents and Settings/Zaretskii/Local Settings/Temp:\12 total used in directory 1178 available 46.5 GiB\12 drwxrwxrwx 1 Zaretskii None 0 2012-04-16 ..\12 drwxrwxrwx 1 Zaretskii None 0 12-02 17:35 .\12 drwxrwxrwx 1 Zaretskii None 0 09-25 21:47 aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\12 drwxrwxrwx 1 Zaretskii None 0 12-01 17:26 acrord32_sbx\12 -rw-rw-rw- 1 Zaretskii None 38296 12...(the rest elided)
Not sure what that means.
The second test fails because:
Test auto-revert-test05-global-notify condition:
(ert-test-failed
((should
(buffer-local-value 'auto-revert-notify-watch-descriptor buf-1))
:form
(buffer-local-value auto-revert-notify-watch-descriptor #<killed buffer>)
:value nil))
Let me know if I can help you more.
Information forwarded
to
bug-gnu-emacs <at> gnu.org
:
bug#44638
; Package
emacs
.
(Wed, 02 Dec 2020 17:06:01 GMT)
Full text and
rfc822 format available.
Message #35 received at 44638 <at> debbugs.gnu.org (full text, mbox):
Eli Zaretskii <eliz <at> gnu.org> writes:
Hi Eli,
>> Eli, could you run autorevert-test on MS Windows, for the w32notify
>> backend? Either a complete run, or just
>>
>> $ make -C test autorevert-tests
>> SELECTOR=auto-revert-test07-auto-revert-several-buffers
>
> auto-revert-test07-auto-revert-several-buffers succeeds. Running the
> entire test finds 2 failures:
>
> 2 unexpected results:
> FAILED auto-revert-test04-auto-revert-mode-dired
> FAILED auto-revert-test05-global-notify
>
> Let me know if I can help you more.
Well, if the same errors happen also w/o that patch, I believe we are
safe to install the patch in master.
In general, I need to debug what happens, the ert reports are not
sufficient (for me) to understand what's going wrong. Do you know how to
get an MS Windows VM instance? I would be even willing to pay for a
license.
I have a spare server which runs already VMs, like the FreeBSD instance
I've just taken for the kqueue tests.
Best regards, Michael.
Information forwarded
to
bug-gnu-emacs <at> gnu.org
:
bug#44638
; Package
emacs
.
(Wed, 02 Dec 2020 17:14:02 GMT)
Full text and
rfc822 format available.
Message #38 received at 44638 <at> debbugs.gnu.org (full text, mbox):
> From: Michael Albinus <michael.albinus <at> gmx.de>
> Cc: sbaugh <at> catern.com, 44638 <at> debbugs.gnu.org
> Date: Wed, 02 Dec 2020 18:05:35 +0100
>
> > auto-revert-test07-auto-revert-several-buffers succeeds. Running the
> > entire test finds 2 failures:
> >
> > 2 unexpected results:
> > FAILED auto-revert-test04-auto-revert-mode-dired
> > FAILED auto-revert-test05-global-notify
> >
> > Let me know if I can help you more.
>
> Well, if the same errors happen also w/o that patch, I believe we are
> safe to install the patch in master.
Yes, I think those tests always failed for me.
> In general, I need to debug what happens, the ert reports are not
> sufficient (for me) to understand what's going wrong. Do you know how to
> get an MS Windows VM instance? I would be even willing to pay for a
> license.
Sorry, I don't know. maybe someone else here does.
Information forwarded
to
bug-gnu-emacs <at> gnu.org
:
bug#44638
; Package
emacs
.
(Wed, 02 Dec 2020 17:21:01 GMT)
Full text and
rfc822 format available.
Message #41 received at 44638 <at> debbugs.gnu.org (full text, mbox):
Eli Zaretskii <eliz <at> gnu.org> writes:
>> > auto-revert-test07-auto-revert-several-buffers succeeds. Running the
>> > entire test finds 2 failures:
>> >
>> > 2 unexpected results:
>> > FAILED auto-revert-test04-auto-revert-mode-dired
>> > FAILED auto-revert-test05-global-notify
>> >
>> > Let me know if I can help you more.
>>
>> Well, if the same errors happen also w/o that patch, I believe we are
>> safe to install the patch in master.
>
> Yes, I think those tests always failed for me.
Good, so I will install patch 1/2 in master. Likely tomorrow.
>> In general, I need to debug what happens, the ert reports are not
>> sufficient (for me) to understand what's going wrong. Do you know how to
>> get an MS Windows VM instance? I would be even willing to pay for a
>> license.
>
> Sorry, I don't know. maybe someone else here does.
Hmm. What would be the canonical address to ask Microsoft directly?
Best regards, Michael.
Information forwarded
to
bug-gnu-emacs <at> gnu.org
:
bug#44638
; Package
emacs
.
(Wed, 02 Dec 2020 17:45:02 GMT)
Full text and
rfc822 format available.
Message #44 received at 44638 <at> debbugs.gnu.org (full text, mbox):
> From: Michael Albinus <michael.albinus <at> gmx.de>
> Cc: sbaugh <at> catern.com, 44638 <at> debbugs.gnu.org
> Date: Wed, 02 Dec 2020 18:20:01 +0100
>
> >> Do you know how to get an MS Windows VM instance? I would be even
> >> willing to pay for a license.
> >
> > Sorry, I don't know. maybe someone else here does.
>
> Hmm. What would be the canonical address to ask Microsoft directly?
Does the below help?
https://developer.microsoft.com/en-us/microsoft-edge/tools/vms/
https://developer.microsoft.com/en-us/windows/downloads/virtual-machines/
https://www.extremetech.com/computing/198427-how-to-install-windows-10-in-a-virtual-machine
https://superuser.com/questions/961322/find-a-windows-10-iso-to-install-it-in-a-virtual-machine
Information forwarded
to
bug-gnu-emacs <at> gnu.org
:
bug#44638
; Package
emacs
.
(Wed, 02 Dec 2020 17:47:01 GMT)
Full text and
rfc822 format available.
Message #47 received at 44638 <at> debbugs.gnu.org (full text, mbox):
On 02.12.2020 19:13, Eli Zaretskii wrote:
>> In general, I need to debug what happens, the ert reports are not
>> sufficient (for me) to understand what's going wrong. Do you know how to
>> get an MS Windows VM instance? I would be even willing to pay for a
>> license.
> Sorry, I don't know. maybe someone else here does.
Check out one of these:
https://developer.microsoft.com/en-us/microsoft-edge/tools/vms/
https://developer.microsoft.com/en-us/windows/downloads/virtual-machines/
These are distributed explicitly for development and testing. They are
free, but with an expiration date that's fairly close.
bug Marked as fixed in versions 28.1.
Request was from
Michael Albinus <michael.albinus <at> gmx.de>
to
control <at> debbugs.gnu.org
.
(Thu, 03 Dec 2020 15:02:02 GMT)
Full text and
rfc822 format available.
Reply sent
to
Michael Albinus <michael.albinus <at> gmx.de>
:
You have taken responsibility.
(Thu, 03 Dec 2020 15:02:02 GMT)
Full text and
rfc822 format available.
Notification sent
to
Spencer Baugh <sbaugh <at> catern.com>
:
bug acknowledged by developer.
(Thu, 03 Dec 2020 15:02:02 GMT)
Full text and
rfc822 format available.
Message #54 received at 44638-done <at> debbugs.gnu.org (full text, mbox):
Michael Albinus <michael.albinus <at> gmx.de> writes:
> Eli Zaretskii <eliz <at> gnu.org> writes:
>
>>> > auto-revert-test07-auto-revert-several-buffers succeeds. Running the
>>> > entire test finds 2 failures:
>>> >
>>> > 2 unexpected results:
>>> > FAILED auto-revert-test04-auto-revert-mode-dired
>>> > FAILED auto-revert-test05-global-notify
>>> >
>>> > Let me know if I can help you more.
>>>
>>> Well, if the same errors happen also w/o that patch, I believe we are
>>> safe to install the patch in master.
>>
>> Yes, I think those tests always failed for me.
>
> Good, so I will install patch 1/2 in master. Likely tomorrow.
Done, closing the bug.
Best regards, Michael.
bug archived.
Request was from
Debbugs Internal Request <help-debbugs <at> gnu.org>
to
internal_control <at> debbugs.gnu.org
.
(Fri, 01 Jan 2021 12:24:09 GMT)
Full text and
rfc822 format available.
This bug report was last modified 4 years and 184 days ago.
Previous Next
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997,2003 nCipher Corporation Ltd,
1994-97 Ian Jackson.