GNU bug report logs - #14820
24.3; elisp manual: How to write good idle timer worker functions?

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 14820 in the body.
You can then email your comments to 14820 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Phil Sainty <psainty <at> orcon.net.nz>
To: bug-gnu-emacs <at> gnu.org
Subject: 24.3; elisp manual: How to write good idle timer worker functions?
Date: Tue, 09 Jul 2013 01:25:58 +1200

Documentation and/or possible enhancement needed.

Sorry, this is a wee bit long; the short version is that I'd like to
see the elisp manual provide a canonical solution to the problems
raised (but not resolved) in (elisp) Idle Timers in the paragraph
beginning "Do not write an idle timer function containing a loop"...

The long version is:


I'm trying to do some work with idle timers, but this seems like one
of the trickier aspects of Emacs to work with, and I'm finding it hard
to determine the correct approach. In particular I'm wishing that the
documentation at "(elisp) Idle Timers" provided some robust example
code which could safely be used as a template for this kind of thing.

I feel that improved documentation would probably help a lot of people.
(I would be happy to draft up some changes to the documentation myself
if only I knew for sure what that example code should look like.)

Frustratingly, the documentation actually tells me what NOT to do, but
does not tell me what I should do instead.


Here's the use case:

There is a substantial amount of work to perform 'in the background',
taking (in total) in excess of 10 seconds to complete, and also
needing to be run on a semi-regular basis, so making the user wait
each time is out of the question. I wish to perform the work in idle
time, as quickly as possible, but without disrupting the user.

The work can be broken down into a queue of small items, each of which
takes a fraction of a second to complete.

When my idle timer triggers, I would like Emacs to start processing
the queue, and continue to process the queue without pausing unless
Emacs ceases to be idle (or should be dealing with something else, at
any rate). It's this last part where I've been struggling with the
documentation.

Essentially I want to iterate over the queue, and after processing
each item I want to check to see if Emacs is waiting to do something
else. If not I would like it to continue processing immediately, not
introducing any unnecessary pauses. If there is some kind of input
or other necessary activity pending, I need to break out of my loop
and wait for Emacs to become idle again before resuming.


At present the documentation says:

> Do not write an idle timer function containing a loop which does a
> certain amount of processing each time around, and exits when
> `(input-pending-p)' is non-`nil'.  This approach seems very natural
> but has two problems:
>
>   * It blocks out all process output (since Emacs accepts process
>     output only while waiting).
>
>   * It blocks out any idle timers that ought to run during that time.

So that helpfully warns me about the wrong approach for my use case,
but doesn't say what I should do instead (the remainder of that info
node discusses intentionally introducing a delay into the processing,
which seems like a different topic -- I don't wish to introduce any
delays if I can avoid it).

The quote above indicates that `input-pending-p' is not sufficient to
detect all kinds of pending activity, but it doesn't suggest other
options. My current impression is that the `sit-for' function seems
like the way to allow Emacs check for other pending activity, and that
(sit-for 0 t) looks like a way to do so without adding unnecessary
delays or activity.

I've tried this in conjunction with `with-timeout' and it does enable
me to make Emacs break out of a loop after a certain duration, which
suggests to me that this resolves the first of the documentation's
stated problems -- blocking other idle timers (given that
`with-timeout' depends upon timers).

So with the `sit-for' call added into the mix, is it then enough to
check `input-pending-p'? Will this combination also deal with the
second stated problem of blocking process output? Or is this still
insufficient in some cases?


In other words, will something like the following be a robust
approach in general?

(defvar my-queue nil "Queue of items to process")
(defvar my-idle-duration 2 "Required idle time before starting")

(defvar my-timer nil)
(defvar my-resume-timer nil)

;; after populating the queue, initialise processing with:
(setq my-timer (run-with-idle-timer
                my-idle-duration t 'my-process-queue))

(defun my-process-queue ()
  "Process the queue until interrupted."
  (while (and my-queue (not (input-pending-p)))
    (my-process-queue-item (pop my-queue))
    (sit-for 0 t))

  (if my-queue
      (let ((idle (current-idle-time)))
        (when idle
          ;; Schedule a one-off resume timer.
          (when (timerp my-resume-timer)
            (cancel-timer my-resume-timer))
          (setq my-resume-timer
                (run-with-idle-timer
                 (time-add idle (seconds-to-time my-idle-duration))
                 nil 'my-process-queue))))
    ;; Queue is empty; cancel the repeating timer.
    (cancel-timer my-timer)
    (when (timerp my-resume-timer)
      (cancel-timer my-resume-timer))))


I also don't know if the sit-for duration of zero would prevent Emacs
from doing things that a larger number would enable? And if so, what
is the best way to ensure that Emacs will be able to deal with any
other processes, while not also causing undue delays in processing the
queue items?

Basically I don't know enough about how Emacs interacts with processes
to be able to answer these kinds of questions myself, and it would be
great if the manual provided enough detail that I could be confident
about making Emacs perform arbitrary work in idle time without
introducing any potential problems.


Perhaps there's a good case to be made for introducing some new
macro into the API to provide a simple standard way of implementing
this kind of idle-time worker function?

There's certainly enough boiler-plate in the code I've come up with
that some kind of wrapper would seem warranted, and I would suspect
that this is true even if I'm barking up the wrong tree with my
approach.


thanks,
-Phil

Message #8 received at 14820 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> IRO.UMontreal.CA>
To: Phil Sainty <psainty <at> orcon.net.nz>
Cc: 14820 <at> debbugs.gnu.org
Subject: Re: bug#14820: 24.3;
 elisp manual: How to write good idle timer worker functions?
Date: Wed, 10 Jul 2013 04:35:01 -0400

> Documentation and/or possible enhancement needed.

Currently, Emacs doesn't support such "background tasks" well.
Documentation could help, but I think it's not really worth
the trouble.  Better come up with a package that provides support
for it.

IIRC someone posted some packages that try to provide support for
related issues, but I can't find the corresponding email.


        Stefan

Message #11 received at 14820 <at> debbugs.gnu.org (full text, mbox):

From: "Phil Sainty" <psainty <at> orcon.net.nz>
To: "Stefan Monnier" <monnier <at> IRO.UMontreal.CA>
Cc: Phil Sainty <psainty <at> orcon.net.nz>, 14820 <at> debbugs.gnu.org
Subject: Re: bug#14820: 24.3;
 elisp manual: How to write good idle timer worker functions?
Date: Thu, 11 Jul 2013 03:31:25 +1200 (NZST)

Stefan Monnier wrote:
> Currently, Emacs doesn't support such "background tasks" well.
> Documentation could help, but I think it's not really worth the
> trouble.  Better come up with a package that provides support
> for it.

Well surely you'd agree that writing such a package is going to be
more much difficult without documentation? That is essentially the
problem I ran into here, after all.

At the absolute minimum, that page definitely needs to mention
`sit-for' / "(elisp) Waiting" and `accept-process-output' /
"(elisp) Accepting Output". After all, when specific pit-falls
are mentioned, I think that the available facilities for working
around them warrant at least a passing mention.

(Honestly, the merest mention of these two functions after the
description of the problems of blocking other timers and processes
would have saved me many hours, so it's worth it just for the
potential to save someone else from being similarly stumped. I'd
simply never had cause to use them before, so it wasn't *at all*
obvious where to look.)

Having found my way to the latter, and run some experiments with
generating output from a shell while running my idle timer code,
I've managed to answer my own question about `sit-for' (which is
that it does not solve the problem of accepting process output), but
determined that (unsurprisingly) calling `(accept-process-output)'
does provide a solution, so I believe I'm on my way to resolving my
original issue.

I do still think that if there were some nice easy-to-use wrappers
around the functionality that Emacs *does* provide, we could say
something a little more positive about the situation -- that
*despite* not supporting "background tasks" well, Emacs makes
it easy to fake them!


> IIRC someone posted some packages that try to provide support
> for related issues, but I can't find the corresponding email.

Ah, that's a shame; but thank you for looking. Maybe someone
else remembers what those were?

Failing that, if I end up with something that seems useful and
generic, I'll submit it for consideration.


-Phil

p.s. Although I *think* I now know what I'm doing here, I don't
want to close this issue just yet. I'm very interested in any and
all comments on this subject, if any other people wish to chip in
with their thoughts?

Message #14 received at 14820 <at> debbugs.gnu.org (full text, mbox):

From: Daimrod <daimrod <at> gmail.com>
To: "Phil Sainty" <psainty <at> orcon.net.nz>
Cc: 14820 <at> debbugs.gnu.org, Stefan Monnier <monnier <at> IRO.UMontreal.CA>
Subject: Re: bug#14820: 24.3;
 elisp manual: How to write good idle timer worker functions?
Date: Thu, 11 Jul 2013 15:39:52 +0200

"Phil Sainty" <psainty <at> orcon.net.nz> writes:

> Stefan Monnier wrote:
>> Currently, Emacs doesn't support such "background tasks" well.
>> Documentation could help, but I think it's not really worth the
>> trouble.  Better come up with a package that provides support
>> for it.
>
> Well surely you'd agree that writing such a package is going to be
> more much difficult without documentation? That is essentially the
> problem I ran into here, after all.
>
> At the absolute minimum, that page definitely needs to mention
> `sit-for' / "(elisp) Waiting" and `accept-process-output' /
> "(elisp) Accepting Output". After all, when specific pit-falls
> are mentioned, I think that the available facilities for working
> around them warrant at least a passing mention.
>
> (Honestly, the merest mention of these two functions after the
> description of the problems of blocking other timers and processes
> would have saved me many hours, so it's worth it just for the
> potential to save someone else from being similarly stumped. I'd
> simply never had cause to use them before, so it wasn't *at all*
> obvious where to look.)
>
> Having found my way to the latter, and run some experiments with
> generating output from a shell while running my idle timer code,
> I've managed to answer my own question about `sit-for' (which is
> that it does not solve the problem of accepting process output), but
> determined that (unsurprisingly) calling `(accept-process-output)'
> does provide a solution, so I believe I'm on my way to resolving my
> original issue.
>
> I do still think that if there were some nice easy-to-use wrappers
> around the functionality that Emacs *does* provide, we could say
> something a little more positive about the situation -- that
> *despite* not supporting "background tasks" well, Emacs makes
> it easy to fake them!
>
>
>> IIRC someone posted some packages that try to provide support
>> for related issues, but I can't find the corresponding email.
>
> Ah, that's a shame; but thank you for looking. Maybe someone
> else remembers what those were?
>
> Failing that, if I end up with something that seems useful and
> generic, I'll submit it for consideration.

A bit of background:
I'm GSoC student working on the XWidget branch and I wanted to add test
to it. But I've encountered two problems while trying to write "classic"
ERT tests. One is that since I'm working on the C side on an
experimental branch, sometimes Emacs crashes. The second problem is that
some tests needs to be run in a graphical Emacs. So I can't run the
tests in batch mode to circumvent the first problem.


I've wrote a package, emacs-parallel[1], (the API isn't very clean but
it's good enough for my current needs) to eval some stuff in another
Emacs process. I've been heavily inspired by emacs-async[2], but I've
made differents choices:
- it can start a graphical Emacs (not Batch Mode);
- it communicates through a Unix socket instead of standard I/O stream
(because, AFAIK, it is not possible to use them without Batch Mode);
- you can send data from the remote process while the process isn't
finished instead of just sending the result of the call;
- the main entry point is a function, not a macro;
- it tells you whether the process terminated normally, and if not, what
happens (exit code or signal number).
- it handles timeout (you can stop the remote instance after X seconds);

I haven't wrote much doc yet (I've started it monday) so maybe
emacs-async is a better choice (ATM) if you don't need an graphical
Emacs or if you don't need to send data without interrupting the remote
process.

(parallel-start (lambda () (parallel-send 42) (sleep-for 3) 12)
                :post-exec (lambda (results _status)
                             (message "%s" results)))

(emacs-async equivalent)
(async-start (lambda () (sleep-for 3) (list 12 42))
             (lambda (result) (message "%s" result)))

=> after 3 sec it displays (12 42)

However, since those packages simulate parallel/async jobs with another
Emacs instance, you cannot execute code with side effects. Well, you
can, but it won't have any effect in your current Emacs instance, only
on the remote. You have to isolate the side effects and to put them in a
function to be executed once the computation (free of side effect) is
done (`finish-func' in `async-start' or `:post-exec' in
`parallel-start').

> -Phil
>
> p.s. Although I *think* I now know what I'm doing here, I don't
> want to close this issue just yet. I'm very interested in any and
> all comments on this subject, if any other people wish to chip in
> with their thoughts?

[1] https://github.com/daimrod/emacs-parallel.git
[2] https://github.com/jwiegley/emacs-async.git

-- 
Daimrod/Greg

Message #17 received at 14820 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> IRO.UMontreal.CA>
To: "Phil Sainty" <psainty <at> orcon.net.nz>
Cc: 14820 <at> debbugs.gnu.org
Subject: Re: bug#14820: 24.3;
 elisp manual: How to write good idle timer worker functions?
Date: Thu, 11 Jul 2013 18:52:11 -0400

>> Currently, Emacs doesn't support such "background tasks" well.
>> Documentation could help, but I think it's not really worth the
>> trouble.  Better come up with a package that provides support
>> for it.
> Well surely you'd agree that writing such a package is going to be
> more much difficult without documentation? That is essentially the
> problem I ran into here, after all.

No, I think that we don't know the solution, so writing the doc is just as
hard because we don't know what the doc should say until we've figured
out how to write the code.


        Stefan

Message #20 received at 14820 <at> debbugs.gnu.org (full text, mbox):

From: Phil Sainty <psainty <at> orcon.net.nz>
To: Daimrod <daimrod <at> gmail.com>
Cc: 14820 <at> debbugs.gnu.org, Stefan Monnier <monnier <at> IRO.UMontreal.CA>
Subject: Re: bug#14820: 24.3; elisp manual: How to write good idle timer worker
 functions?
Date: Sat, 13 Jul 2013 18:38:48 +1200

Thanks Greg, that sounds like good stuff in general.

Sadly it's not applicable in my case -- the processing I'm doing
is very much tied to the current running Emacs instance -- but it's
definitely good to know those packages exist for other use cases.

cheers,
-Phil


On 12/07/2013 01:39, Daimrod wrote:
> A bit of background:
> I'm GSoC student working on the XWidget branch and I wanted to add test
> to it. But I've encountered two problems while trying to write "classic"
> ERT tests. One is that since I'm working on the C side on an
> experimental branch, sometimes Emacs crashes. The second problem is that
> some tests needs to be run in a graphical Emacs. So I can't run the
> tests in batch mode to circumvent the first problem.
>
>
> I've wrote a package, emacs-parallel[1], (the API isn't very clean but
> it's good enough for my current needs) to eval some stuff in another
> Emacs process. I've been heavily inspired by emacs-async[2], but I've
> made differents choices:
> - it can start a graphical Emacs (not Batch Mode);
> - it communicates through a Unix socket instead of standard I/O stream
> (because, AFAIK, it is not possible to use them without Batch Mode);
> - you can send data from the remote process while the process isn't
> finished instead of just sending the result of the call;
> - the main entry point is a function, not a macro;
> - it tells you whether the process terminated normally, and if not, what
> happens (exit code or signal number).
> - it handles timeout (you can stop the remote instance after X seconds);
>
> I haven't wrote much doc yet (I've started it monday) so maybe
> emacs-async is a better choice (ATM) if you don't need an graphical
> Emacs or if you don't need to send data without interrupting the remote
> process.
>
> (parallel-start (lambda () (parallel-send 42) (sleep-for 3) 12)
>                  :post-exec (lambda (results _status)
>                               (message "%s" results)))
>
> (emacs-async equivalent)
> (async-start (lambda () (sleep-for 3) (list 12 42))
>               (lambda (result) (message "%s" result)))
>
> => after 3 sec it displays (12 42)
>
> However, since those packages simulate parallel/async jobs with another
> Emacs instance, you cannot execute code with side effects. Well, you
> can, but it won't have any effect in your current Emacs instance, only
> on the remote. You have to isolate the side effects and to put them in a
> function to be executed once the computation (free of side effect) is
> done (`finish-func' in `async-start' or `:post-exec' in
> `parallel-start').
>
> [1] https://github.com/daimrod/emacs-parallel.git
> [2] https://github.com/jwiegley/emacs-async.git

Message #25 received at 14820 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Phil Sainty <psainty <at> orcon.net.nz>
Cc: 14820 <at> debbugs.gnu.org
Subject: Re: bug#14820: 24.3; elisp manual: How to write good idle timer
 worker functions?
Date: Wed, 26 Jan 2022 18:41:41 +0100

Phil Sainty <psainty <at> orcon.net.nz> writes:

> There is a substantial amount of work to perform 'in the background',
> taking (in total) in excess of 10 seconds to complete, and also
> needing to be run on a semi-regular basis, so making the user wait
> each time is out of the question. I wish to perform the work in idle
> time, as quickly as possible, but without disrupting the user.
>
> The work can be broken down into a queue of small items, each of which
> takes a fraction of a second to complete.

[...]

> The quote above indicates that `input-pending-p' is not sufficient to
> detect all kinds of pending activity, but it doesn't suggest other
> options.

(I'm going through old bug reports that unfortunately weren't resolved
at the time.)

I think that these days, you'd probably implement this using Emacs
threads, but I think the general advice in that node is still sound --
if you're doing something that takes a long time from an idle hook, you
should instead schedule a new idle timer (with a very short timeout)
instead of trying to use `pending-input-p' and `sit-for'.

So I'm not sure whether there's anything more to do in this bug report?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #30 received at 14820 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Phil Sainty <psainty <at> orcon.net.nz>
Cc: 14820 <at> debbugs.gnu.org
Subject: Re: bug#14820: 24.3; elisp manual: How to write good idle timer
 worker functions?
Date: Wed, 23 Feb 2022 21:15:53 +0100

Lars Ingebrigtsen <larsi <at> gnus.org> writes:

> So I'm not sure whether there's anything more to do in this bug report?

Nobody had any opinions here in a month, so I'm closing this bug report.
If there's more to be done here, please respond to the debbugs address
and we'll reopen.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.