GNU bug report logs - #37906
26.2; `call-process' docstring, section DESTINATION

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 37906 in the body.
You can then email your comments to 37906 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: "Florian v. Savigny" <f.savigny <at> mailbox.org>
To: bug-gnu-emacs <at> gnu.org
Subject: 26.2; `call-process' docstring, section DESTINATION
Date: Thu, 24 Oct 2019 17:45:07 +0200 (CEST)

Dear Emacs maintainers,

since this pertains to documentation only, I have left out all the unnecessary debugging information. 

The docstring of `call-process', which is extremely confusing (and partly wrong) as regards the DESTINATION argument, seems to have been in its present form for what must have been at least 15 years. Although the relevant section in the Emacs Lisp Info does a much better job (even though I still find the choice of "REAL-" for standard out puzzling), I would strongly suggest that the docstring be rewritten, too. (That would probably have considerably sped up my understanding of Elisp in particular and *nix systems in general.)

Apart from the confusion, the docstring specifically does not acknowledge that e.g. (nil nil) as the DESTINATION (which might be a style somebody chooses to be more explicit about the the use of stdout and stderr) is perfectly acceptable, and also e.g. (nil "file"). Even '(0 "file") will be accepted and work as expected. 

The following would be my suggestion as to how the relevant section could be rephrased:

-------------------------------------------------------------------------------------------
Third argument DESTINATION specifies how to handle program’s standard
and standard error output. Its general form is a two-element list 
(STDOUT STDERR), but there are shorthand notations for common uses (see below):

STDOUT can be:
- 0: discard it and return immediately, i.e. do not even wait for the program to terminate
- nil: discard it.
- t: insert it into the current buffer before point.
- a buffer name (i.e. a string): insert it into the buffer of that name before point.
  If that buffer does not exist yet, create it (even if there is no output).
- a list (:file "some_file_name"): Write it to a file called "some_file_name".
  If the "some_file_name" exists, overwrite it, otherwise, create it.

STDERR can be:
- nil: discard it.
- t: direct it where stdout goes (mix it with stdout).
- a file name: Write it to a file of that name, overwriting it if it exists,
  otherwise creating it.

As shorthand for the list (STDOUT nil), i. e. when the user wants to
discard STDERR, also accept STDOUT without a list, as detailed above.
I.e. take e.g. '(:file "some_file_name") as '((:file "some_file_name") nil).

Finally, as shorthand for '("buffer" t) (i.e. insert stdout in "buffer"
and stderr, too), accept also the buffer of the name "buffer", without a list. 
I.e. take (get-buffer-create "buffer") to mean '("buffer" t).
-------------------------------------------------------------------------------------------

Best regards,


Florian v. Savigny 
Siebenpfeiffer Str. 25 
66482 Zweibrücken 

0175 - 365 24 17

Message #8 received at 37906 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: "Florian v. Savigny" <f.savigny <at> mailbox.org>
Cc: 37906 <at> debbugs.gnu.org
Subject: Re: bug#37906: 26.2; `call-process' docstring, section DESTINATION
Date: Fri, 25 Oct 2019 11:53:12 +0300

> Date: Thu, 24 Oct 2019 17:45:07 +0200 (CEST)
> From: "Florian v. Savigny" <f.savigny <at> mailbox.org>
> 
> The docstring of `call-process', which is extremely confusing (and partly wrong) as regards the DESTINATION argument, seems to have been in its present form for what must have been at least 15 years. Although the relevant section in the Emacs Lisp Info does a much better job (even though I still find the choice of "REAL-" for standard out puzzling), I would strongly suggest that the docstring be rewritten, too. (That would probably have considerably sped up my understanding of Elisp in particular and *nix systems in general.)
> 
> Apart from the confusion, the docstring specifically does not acknowledge that e.g. (nil nil) as the DESTINATION (which might be a style somebody chooses to be more explicit about the the use of stdout and stderr) is perfectly acceptable, and also e.g. (nil "file"). Even '(0 "file") will be accepted and work as expected. 
> 
> The following would be my suggestion as to how the relevant section could be rephrased:

Thank you for your report.

Could you please elaborate on what are the confusing parts of the
current doc string?  I've re-read it now, and I think it is clear and
describes all the possible options.

E.g., the (nil nil) and (nil "file") forms are described as follows:

  DESTINATION can also have the form (REAL-BUFFER STDERR-FILE); in that case,
   REAL-BUFFER says what to do with standard output, as above,
   while STDERR-FILE says what to do with standard error in the child.
   STDERR-FILE may be nil (discard standard error output),
   t (mix it with ordinary output), or a file name string.

The "as above" part on the second line means that REAL-BUFFER can be
any of the values mentioned above, which includes nil.

> STDOUT can be:
> - 0: discard it and return immediately, i.e. do not even wait for the program to terminate
> - nil: discard it.
> - t: insert it into the current buffer before point.
> - a buffer name (i.e. a string): insert it into the buffer of that name before point.
>   If that buffer does not exist yet, create it (even if there is no output).
> - a list (:file "some_file_name"): Write it to a file called "some_file_name".
>   If the "some_file_name" exists, overwrite it, otherwise, create it.
> 
> STDERR can be:
> - nil: discard it.
> - t: direct it where stdout goes (mix it with stdout).
> - a file name: Write it to a file of that name, overwriting it if it exists,
>   otherwise creating it.
> 
> As shorthand for the list (STDOUT nil), i. e. when the user wants to
> discard STDERR, also accept STDOUT without a list, as detailed above.
> I.e. take e.g. '(:file "some_file_name") as '((:file "some_file_name") nil).
> 
> Finally, as shorthand for '("buffer" t) (i.e. insert stdout in "buffer"
> and stderr, too), accept also the buffer of the name "buffer", without a list. 

I'm not sure this is an improvement.  For starters, it is longer.
More importantly, it reverses the usual case and the rare case: most
uses of call-process use what you call "shorthand", so this order
change causes the reader to have to read more, and then requires
him/her to understand what you mean by "shorthand".

So I suggest to start by describing the confusing parts.  Perhaps we
can then arrive at text that will be clearer, but without the adverse
effects mentioned above.

Message #13 received at 37906 <at> debbugs.gnu.org (full text, mbox):

From: "Florian v. Savigny" <f.savigny <at> mailbox.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 37906 <at> debbugs.gnu.org
Subject: Re: bug#37906: 26.2; `call-process' docstring, section DESTINATION
Date: Wed, 30 Oct 2019 16:42:03 +0100 (CET)

Dear Eli,

I am pasting here my reply to your reply from last Friday. I realised I had replied only to your personal E-mail address, which is probably not the appropriate thing to do -- I would like to apologise for that!

The reply I am pasting is quite lengthy, a fact for which I would like to apologise, but which also underlines and illustrates that I *truly* found the docstring confusing.

In the mean time since, however, I have realised that (I think) I CAN now nail down the root cause for our disparate perceptions (you finding the docstring clear, me finding it very confusing):

Once you approach `call-process' with a different idea of what the "usual case" is, as you have put it, the docstring becomes much harder to interpret. The (actually firm) idea that I had in mind when I did "C-h f call-process" was that normally, a function would keep the two output streams separate and treat them equally, and that led to a slew of misunderstandings. (One of the misunderstandings being that for a long time, I thought that just specifing a single value, or scalar or however it is called in Lisp, meant that STDERR would be discarded.)

I do not want to argue what the "true" usual case is or might be. But I do want to argue that if such an assumption is made, it should be made explicitly (e.g. by saying, "... by default, stdout and stderr are mixed together like on the terminal ... To handle them separately, do XY ...", or the like) because not doing so is tantamount to a firm assumption what the user assumes (and perhaps how she wants to use `call-process'), and excludes those users for whom this assumption is wrong.

Once you KNOW how the function works, it is easy to interpret the docstring correctly. But it should be the other way round: Once you have read the docstring once (without having resorted to the Elisp manual), it is easy to use the function correctly. At least that should, as I understand it, be the goal. As I have specified below, I think I have now found a new solution which should be understandable to more people with different expectations than the current official one (and is even shorter), and is unambiguous. (Or at least I hope so.)

Best regards, Florian



My reply from Sunday follows here:

...

I hope I can be forgiven for having assumed that my suggested
alternative description would speak for itself and make glaringly
obvious what's (in my opinion) amiss with the docstring. I had worked
through the docstring and tested the function's behaviour for
something like six hours before being really confident how it behaves.
And I remembered how I had read this docstring before, about 15 years
ago, more or less as a complete newbie, and thought something like,
"if only I had understood this back then".

In other words, having to explain what I find hard to understand was
about the last thing I expected, and at first, I had no idea of how to
do this in writing. The only idea I was finally able to come up with
is to go through the docstring step by step. I apologise this will be
somewhat lengthy, but then the interface is not exactly trivial.

In fact, the root (and perhaps the bigger part) of the confusion seems
to me the inconsistent interface, which for most things runs counter
to the everyday expectation of consistency. Nil is about the only thing that
has a consistent meaning; it always means "discard". But otherwise:

- t means "current buffer" for stdout, but "mix with stdout" for stderr
- a string means a buffer name for stdout, but a file name for stderr
- as a corollary, :file is used for stdout, but not for a stderr file
- only stdout has 0, such that (0 "file") is possible, but
   ((:file "file) 0) is not.

Obviously nothing much can be done about this interface. But because
of that, the documentation has to make up for that, and be extra clear
and explicit.

I have broken up the docstring into mouthfuls of information, and have
added enumeration and itemisation to that end. I have added a comment
after each mouthful as I chew on it.

||   like this:

On working through this again, I realised (again by testing) that my
understanding of the "buffer"/"buffer name" issue was still not
correct. It's actually much simpler than I thought, and I have been
able to come up with a new suggestion which is surprisingly short,
satisfies my sense of clarity, and keeps the order you prefer. (See
the end of the mail.) My first suggestion was also partly wrong.


--------------------------- begin ordeal -------------------------------
Third argument DESTINATION specifies how to handle program’s output.

1. If DESTINATION is a buffer, or t that stands for the current
  buffer, it means insert output in that buffer before point.

|| - ... or a buffer /name/
||    (That new buffers are created even if there is not output is
||    also something that a newbie won't suspect. In other words, that
||    (s)he may have to clean up afterwards.)
||
|| - It says only "output". Does it mean "standard output", or
||    "standard output mixed with standard error", in other words, "all
||    output, mixed"? 

2. If DESTINATION is nil, it means discard output; 0 means discard
  and don’t wait for the program to terminate.

||   - It should be "all output", if you don't use the two-argument list.
||
||   - The oddly dropped-in feature of making call-process asynchronous
||      via this argument certainly does not help clarity, but that is
||      nothing that the docstring can change. It's a bit like turning
||      the hindmost knob (but only that one) on your electric stove to
||      1/2 (but only to that position) also turns on the central heating
||      for the appartment. (I know, I'm exaggerating, but still.)
||
||      But what the docstring could say is: What happens to the
||      exit status if we don't wait for the program to terminate?
  
3. If DESTINATION is ‘(:file FILE)’, where FILE is a file name
  string, it means that output should be written to that file (if
  the file already exists it is overwritten).

|| - Again, "all output" or "standard output"? (If used alone.)

4. DESTINATION can also have the form (REAL-BUFFER STDERR-FILE); in
  that case,

|| Why these odd placeholders, especially REAL-BUFFER?
|| 
|| - stdout is REAL, why? As opposed to stderr being unreal, or
||    virtual??? And it can also be, as discussed above, a buffer name,
||    or t, or nil, or a file name, not only a BUFFER.
||
|| - STDERR-FILE can not only be a FILE, but also t or nil. (nil is
||    not confusing, but t can mean stderr goes to a buffer rather than
||    a file.)

  
  a) REAL-BUFFER says what to do with standard output, as above, while

|| "As above" is correct if, in what I have numbered 1. to 3., it says
|| "standard output", which would however be incorrect again when 1 to 3 are
|| used alone. (I guess this is why the docstring tries to get away by saying "output"
|| above. But this is definitely too imprecise for "single use".)

  b) STDERR-FILE says what to do with standard error in
  the child. STDERR-FILE may be
  
     - nil (discard standard error output),
     - t (mix it with ordinary output), or
     - a file name string.

|| The file is also overwritten, like in the case of stdout.

-------------------------- end ordeal --------------------------------


So, my new suggestion would be:

----------------------------------------------------------------------
If DESTINATION is a single item, lump the program's standard output
and standard error together: If it is a buffer or buffer name (or t
for current buffer), insert all output in that buffer before point. If
it is nil or 0, discard all output; with 0 also don’t wait for the
program to terminate. If it is ‘(:file FILENAME)’ write all output to
that file (overwrite it if it already exists).

If DESTINATION is a two-element list (STDOUT STDERR), keep the output
streams separate. STDOUT can take the same values as detailed above
(which then apply only to stdout, however). STDERR, in contrast, may
only be nil (discard it), t (mix it with stdout), or the name of a
file to write it to (which also gets overwritten.)
----------------------------------------------------------------------

t for STDERR is, as I just realised, completely redundant, because it
specifies precisely what happens when DESTINATION is a single item.
Perhaps it would be better to leave this out, if nobody needs it?

I dearly hope that at the very least, my own understanding of the
DESTINATION argument is completely correct now.


Best regards, and thank you for bearing with me.

Florian



> Eli Zaretskii <eliz <at> gnu.org> hat am 25. Oktober 2019 um 10:53 geschrieben:
> 
> 
> > Date: Thu, 24 Oct 2019 17:45:07 +0200 (CEST)
> > From: "Florian v. Savigny" <f.savigny <at> mailbox.org>
> > 
> > The docstring of `call-process', which is extremely confusing (and partly wrong) as regards the DESTINATION argument, seems to have been in its present form for what must have been at least 15 years. Although the relevant section in the Emacs Lisp Info does a much better job (even though I still find the choice of "REAL-" for standard out puzzling), I would strongly suggest that the docstring be rewritten, too. (That would probably have considerably sped up my understanding of Elisp in particular and *nix systems in general.)
> > 
> > Apart from the confusion, the docstring specifically does not acknowledge that e.g. (nil nil) as the DESTINATION (which might be a style somebody chooses to be more explicit about the the use of stdout and stderr) is perfectly acceptable, and also e.g. (nil "file"). Even '(0 "file") will be accepted and work as expected. 
> > 
> > The following would be my suggestion as to how the relevant section could be rephrased:
> 
> Thank you for your report.
> 
> Could you please elaborate on what are the confusing parts of the
> current doc string?  I've re-read it now, and I think it is clear and
> describes all the possible options.
> 
> E.g., the (nil nil) and (nil "file") forms are described as follows:
> 
>   DESTINATION can also have the form (REAL-BUFFER STDERR-FILE); in that case,
>    REAL-BUFFER says what to do with standard output, as above,
>    while STDERR-FILE says what to do with standard error in the child.
>    STDERR-FILE may be nil (discard standard error output),
>    t (mix it with ordinary output), or a file name string.
> 
> The "as above" part on the second line means that REAL-BUFFER can be
> any of the values mentioned above, which includes nil.
> 
> > STDOUT can be:
> > - 0: discard it and return immediately, i.e. do not even wait for the program to terminate
> > - nil: discard it.
> > - t: insert it into the current buffer before point.
> > - a buffer name (i.e. a string): insert it into the buffer of that name before point.
> >   If that buffer does not exist yet, create it (even if there is no output).
> > - a list (:file "some_file_name"): Write it to a file called "some_file_name".
> >   If the "some_file_name" exists, overwrite it, otherwise, create it.
> > 
> > STDERR can be:
> > - nil: discard it.
> > - t: direct it where stdout goes (mix it with stdout).
> > - a file name: Write it to a file of that name, overwriting it if it exists,
> >   otherwise creating it.
> > 
> > As shorthand for the list (STDOUT nil), i. e. when the user wants to
> > discard STDERR, also accept STDOUT without a list, as detailed above.
> > I.e. take e.g. '(:file "some_file_name") as '((:file "some_file_name") nil).
> > 
> > Finally, as shorthand for '("buffer" t) (i.e. insert stdout in "buffer"
> > and stderr, too), accept also the buffer of the name "buffer", without a list. 
> 
> I'm not sure this is an improvement.  For starters, it is longer.
> More importantly, it reverses the usual case and the rare case: most
> uses of call-process use what you call "shorthand", so this order
> change causes the reader to have to read more, and then requires
> him/her to understand what you mean by "shorthand".
> 
> So I suggest to start by describing the confusing parts.  Perhaps we
> can then arrive at text that will be clearer, but without the adverse
> effects mentioned above.

Florian v. Savigny
Siebenpfeiffer Str. 25
66482 Zweibrücken

0175 - 365 24 17

Message #18 received at 37906 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: "Florian v. Savigny" <f.savigny <at> mailbox.org>
Cc: Eli Zaretskii <eliz <at> gnu.org>, 37906 <at> debbugs.gnu.org
Subject: Re: bug#37906: 26.2; `call-process' docstring, section DESTINATION
Date: Tue, 31 Aug 2021 04:14:57 +0200

"Florian v. Savigny" <f.savigny <at> mailbox.org> writes:

> I do not want to argue what the "true" usual case is or might be. But
> I do want to argue that if such an assumption is made, it should be
> made explicitly (e.g. by saying, "... by default, stdout and stderr
> are mixed together like on the terminal ...

I've now amended the doc string of call-process in Emacs 28 to
explicitly say that we mean both stdout and stderr when we talk about
"output".

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.