GNU bug report logs - #46388
27.1; emacs -batch does not output messages immediately when invoked outside of the command prompt

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 46388 in the body.
You can then email your comments to 46388 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Ioannis Kappas <ioannis.kappas <at> gmail.com>
To: bug-gnu-emacs <at> gnu.org
Subject: 27.1; emacs -batch does not output messages immediately when invoked
 outside of the command prompt
Date: Mon, 8 Feb 2021 21:20:47 +0000

    There appears to be an issue with the /message/ function when
    emacs is invoked in -batch mode on windows-nt from outside the
    windows console (i.e. not directly from the command
    prompt). Messages are not displayed to the user until emacs exits.

    Consider the following command from example, which suppose to
    print the message "hi" first to the user and then exit after 5 seconds:

    : emacs -Q --batch --eval="(progn (message \"hi\") (sit-for 5))"

    When this command is invoked from within an emacs eshell on
    windows, the message is only displayed to the user after about 5 seconds,
    i.e when emacs is about to exit.

    Here is a small summary running the above under Linux and windows
    native:

    | system              | invoked from   | terminal output
                 |
    |---------------------+----------------+--------------------------------------------|
    | x86_64-pc-linux-gnu | terminal       | prints "hi", then after 5
seconds it exits |
    | x86_64-pc-linux-gnu | emacs eshell   | prints "hi", then after 5
seconds it exits |
    | x86_64-w64-mingw32  | command prompt | prints "hi", then after 5
seconds it exits |
    | x86_64-w64-mingw32  | emacs eshell   | after 5 seconds prints
"hi", then exits    |

    The issue on windows is not exclusive to invoking emacs -batch
    from within emacs (e.g. from eshell), it applies to any invocation
that happens
    outside the command prompt, e.g. from inside the MSYS2 mintty
    terminal.

    It appears as if the issue is caused by a different stderr
    buffering mode used when a program is invoked outside of
    the command prompt. Analysis to follow.

In GNU Emacs 27.1 (build 1, x86_64-w64-mingw32)
 of 2020-11-19 built on fv-az68-340
Repository revision: ec297125a76481c55390d0b329e541907879d6f3
Repository branch: master
Windowing system distributor 'Microsoft Corp.', version 10

Message #8 received at 46388 <at> debbugs.gnu.org (full text, mbox):

From: Ioannis Kappas <ioannis.kappas <at> gmail.com>
To: 46388 <at> debbugs.gnu.org
Subject: 27.1; emacs -batch does not output messages immediately when invoked
 outside of the command prompt
Date: Mon, 8 Feb 2021 21:42:09 +0000

[Message part 1 (text/plain, inline)]

Some analysis follows

    Looking at the code, it appears that /message/s in -batch mode are
    delivered to the process' standard error by the
    [editfnc.c:message()]
      ->[xdisp.c:message3_nolog()]
        ->[xdispl.c:message_to_stderr()]
    function, i.e. messages are written to stderr. The latter function
    uses the standard fwrite and fputc calls to deliver the message to
    stderr.

    The buffering regime used for stderror on windows appears to be different
    depending whether a program was started from the command prompt or
    not. When started from the command prompt stderr is
    unbuffered, while there is a buffer employed otherwise and output
    is only delivered when that buffer is full.

    Below is a sample C program that can demonstrate the behaviour,
    which writes a single character to the standard output and then
exits after 2 seconds:

      #include <stdio.h>
      #include <unistd.h>
      #include <windows.h>

      int main()
      {

fwrite("t", 1, sizeof("t"), stderr);

sleep(2);

return 0;
      }


    | case | invoked from           | result
         |
    |------+------------------------+-------------------------------------------|
    |  1   | command prompt         | prints "t", then after 2 seconds exits |
    |  2   | outside command prompt | after about 2 seconds prints
"t", then exits |

    It appears that the buffer size used in #2 is 2048 bytes (i.e. the
    message is only immediately displayed when is more than 2048 bytes long).

    A solution thus to correct this behavior in emacs -batch on
    windows-nt would be to check if it is connected to the console,
    and when not, set stderr mode to unbuffered.

    A) Likely the win32 api provide GetConsoleMode() that can be used to
    check if a standard HANDLE is attached to the console. Given the
    STD_ERROR_HANDLE as an argument, it will  return non-zero if is
    attached to the console or a 0 otherwise, setting GetLastError()
to "The handle is
    invalid" message.

    B) To set stderr to an unbuffered mode, we can use standard C's
    setvbuf() with _IONBF (no buffering).

    C) The location where the descriptors are prepared for use in emacs
    appear to be at w32.c:init_ntproc().

    If the above analysis is correct, here is an example patch that
puts A/B/C together:

--- a/src/w32.c
+++ b/src/w32.c
@@ -10216,6 +10216,19 @@ init_ntproc (int dumping)
     else
       _open ("nul", O_TEXT | O_NOINHERIT | O_WRONLY);
     _fdopen (2, "w");
+
+    /* When in -batch mode, windows appears to buffer stderr when it
+       is invoked outside of the command prompt. This has the
+       undesired side effect that /message/s are not output unless the
+       buffer is full or emacs exits */
+    DWORD console_mode;
+    if (noninteractive &&
+ // stderr is not attached to the console
+ !GetConsoleMode (GetStdHandle(STD_ERROR_HANDLE), &console_mode))
+      {
+ // set stderr to unbuffered
+ setvbuf (stderr, NULL, _IONBF, 0);
+      }

  I have also attached an ert tests to capture the correct
  behavior. The test invokes an emacs -batch process which prints a
  /message/ and exits after five seconds. The test check immediately
  after two seconds if the batch process has output the message as
  expected. It can be invoked with:

: emacs.exe -batch -l ert -l batch-message-test.el -f
ert-run-tests-batch-and-exit

[batch-message-test.el (application/octet-stream, attachment)]

Message #11 received at 46388 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Ioannis Kappas <ioannis.kappas <at> gmail.com>
Cc: 46388 <at> debbugs.gnu.org
Subject: Re: bug#46388: 27.1;
 emacs -batch does not output messages immediately when invoked
 outside of the command prompt
Date: Tue, 09 Feb 2021 05:38:46 +0200

> From: Ioannis Kappas <ioannis.kappas <at> gmail.com>
> Date: Mon, 8 Feb 2021 21:20:47 +0000
> 
>     There appears to be an issue with the /message/ function when
>     emacs is invoked in -batch mode on windows-nt from outside the
>     windows console (i.e. not directly from the command
>     prompt). Messages are not displayed to the user until emacs exits.
> 
>     Consider the following command from example, which suppose to
>     print the message "hi" first to the user and then exit after 5 seconds:
> 
>     : emacs -Q --batch --eval="(progn (message \"hi\") (sit-for 5))"
> 
>     When this command is invoked from within an emacs eshell on
>     windows, the message is only displayed to the user after about 5 seconds,
>     i.e when emacs is about to exit.

I believe this is because of buffering: Eshell on MS-Windows invokes
subprograms via pipes, which are buffered.

Message #14 received at 46388 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Ioannis Kappas <ioannis.kappas <at> gmail.com>
Cc: 46388 <at> debbugs.gnu.org
Subject: Re: bug#46388: 27.1;
 emacs -batch does not output messages immediately when invoked
 outside of the command prompt
Date: Tue, 09 Feb 2021 00:36:33 -0500

> From: Ioannis Kappas <ioannis.kappas <at> gmail.com>
> Date: Mon, 8 Feb 2021 21:42:09 +0000
> 
>     A solution thus to correct this behavior in emacs -batch on
>     windows-nt would be to check if it is connected to the console,
>     and when not, set stderr mode to unbuffered.

Thanks for the analysis and the patch proposal.  However, I don't
think this is a good idea.  For starters, stderr could be connected to
a file, in which case we do want it to be buffered.  More generally,
stderr could be used for something other than outputting urgent
messages, in which case making it unbuffered will make I/O less
efficient for no good reason.  And this would make Emacs on Windows
behave differently from Posix systems, which is also a downside.

I think a much better way forward in this area is to teach Emacs to
use the Pseudo Consoles introduced in recent Windows versions.  That
would allow us to support subprocess communications via PTYs on
MS-Windows, and thus will solve this and other similar issues.

Patches to support PTYs on Windows are welcome.

Message #17 received at 46388 <at> debbugs.gnu.org (full text, mbox):

From: Ioannis Kappas <ioannis.kappas <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 46388 <at> debbugs.gnu.org
Subject: Re: bug#46388: 27.1; emacs -batch does not output messages
 immediately when invoked outside of the command prompt
Date: Tue, 9 Feb 2021 20:15:34 +0000

Hi Eli,

thanks for taking the time to look into this bug report!

(my apologies for the long reply that follows)

I understand there are two concerns:

1. stderr could be connected to a file, in which case we do want it to
   be buffered for efficiency.
2. If stderr is forced to unbuffered, this will make it behave
   differently from posix systems.

I had a look around #2 in the posix standard. The section under
stderr, stdin, stdout at
https://pubs.opengroup.org/onlinepubs/9699919799/functions/stdin.html
reads:

"""At program start-up, three streams shall be predefined and need not
be opened explicitly: standard input (for reading conventional input),
standard output (for writing conventional output), and standard error
(for writing diagnostic output). When opened, the standard error
stream is not fully buffered; the standard input and standard output
streams are fully buffered if and only if the stream can be determined
not to refer to an interactive device.  """

Thus, at open (I assume this to mean at program invocation), the posix
spec suggests that stderr is not "fully buffered".

The section under setvbuf under the same spec at
https://pubs.opengroup.org/onlinepubs/009695399/functions/setvbuf.html
differentiates between the three different buffering regimes:


"""
 {_IOFBF} shall cause input/output to be fully buffered.

 {_IOLBF} shall cause input/output to be line buffered.

  {_IONBF} shall cause input/output to be unbuffered.
 """

So, if stderr is not "fully buffered", it can be either "line
buffered" or "unbuffered".

(This does make some sense to me, since stderr is usually for urgent
messages and thus any messages are needed to be readily available to
the recipients. Fully buffered defeats the purpose?).

How does Linux conforms to the posix standard? Having a look at
stderr(3) in the foot note, it reads:

""" Notes

The stream stderr is unbuffered. The stream stdout is line-buffered
when it points to a terminal. Partial lines will not appear until
fflush(3) or exit(3) is called, or a newline is printed. This can
produce unexpected results, especially with debugging output. The
buffering mode of the standard streams (or any other stream) can be
changed using the setbuf(3) or setvbuf(3) call. Note that in case
stdin is associated with a terminal, there may also be input buffering
in the terminal driver, entirely unrelated to stdio
buffering. (Indeed, normally terminal input is line buffered in the
kernel.) This kernel input handling can be modified using calls like
tcsetattr(3); see also stty(1), and termios(3).

"""

Which appears to suggests that stderr always starts as "unbuffered"
when a program starts, irrespective of how it was invoked?  If so,
then it conforms to the posix standard of not being "fully buffered"
at open, by always being "unbuffered" (this behavior has also been
seen in all tests mentioned so far). If this is the case, then the
desired buffering efficiency as mentioned in concern #1, does not
exist in Linux.

I have also conducted an additional test to check Linux behavior when
stderr is redirected to a file (using the same c program that I
mentioned in the original bug report):

: ./fwrite-a-character-to-stderr-and-sleep-for-two-seconds 2> io.txt

while at around the same time in another terminal I read from that file:

: cat io.txt

The result is that cat outputs "t", the character the fwrite program
has written to stderr, i.e. the io.txt file, proving that no buffering
has being utilized by the fwrite process. The result is the same when
I try the same experiment from inside an emacs shell.

Back to the windows behavior at hand, the simple test conducted with
fwrite as the bug report mentioned, has a 2048 bytes associated with
it which I will consider it to be categorized as "fully buffered",
which brings it further away to the posix standard and even more further
away from Linux being on the other end (always unbuffered).

(There is also this thread on stackoverlow quoting the following from
comp.lang.c:

"""

Unix convention is that stdin and stdout are line-buffered when
associated with a terminal, and fully-buffered (aka block-buffered)
otherwise. stderr is always unbuffered.

"""

emphasis on *Unix convention ... stderr .. always unbuffered*, though
it will be difficult to confirm this claim for all *nices)

Thus, if we assume the above interpretation given so far to be
correct, then both of
the concerns can be addressed as such:

For #1, we can use Linux as a benchmark of established
behavior. There is no efficiency in Linux with regards to buffering
when stderr is redirected to files, thus having emacs batch mode on
windows being
on par with Linux should be good enough in this respect.

For #2, emacs batch stderr behavior when invoked from outside the
command line is not compatible with the posix standard, since it uses
"fully buffered" mode, which actually makes it behave very differently
from Linux (which is posix compliant by always having stderr as being
unbuffered). Thus, the suggested patch actually addresses this
concern, rather than invalidating it.

(I understand ConPTY is the long term solution looking for a hacker,
though I still do
suppose the current behavior on windows is odd and needs to be
addressed until that time).

Sincerely





On Tue, Feb 9, 2021 at 5:36 AM Eli Zaretskii <eliz <at> gnu.org> wrote:
>
> > From: Ioannis Kappas <ioannis.kappas <at> gmail.com>
> > Date: Mon, 8 Feb 2021 21:42:09 +0000
> >
> >     A solution thus to correct this behavior in emacs -batch on
> >     windows-nt would be to check if it is connected to the console,
> >     and when not, set stderr mode to unbuffered.
>
> Thanks for the analysis and the patch proposal.  However, I don't
> think this is a good idea.  For starters, stderr could be connected to
> a file, in which case we do want it to be buffered.  More generally,
> stderr could be used for something other than outputting urgent
> messages, in which case making it unbuffered will make I/O less
> efficient for no good reason.  And this would make Emacs on Windows
> behave differently from Posix systems, which is also a downside.
>
> I think a much better way forward in this area is to teach Emacs to
> use the Pseudo Consoles introduced in recent Windows versions.  That
> would allow us to support subprocess communications via PTYs on
> MS-Windows, and thus will solve this and other similar issues.
>
> Patches to support PTYs on Windows are welcome.

Message #20 received at 46388 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Ioannis Kappas <ioannis.kappas <at> gmail.com>
Cc: 46388 <at> debbugs.gnu.org
Subject: Re: bug#46388: 27.1; emacs -batch does not output messages
 immediately when invoked outside of the command prompt
Date: Tue, 09 Feb 2021 22:52:13 +0200

> From: Ioannis Kappas <ioannis.kappas <at> gmail.com>
> Date: Tue, 9 Feb 2021 20:15:34 +0000
> Cc: 46388 <at> debbugs.gnu.org
> 
> Which appears to suggests that stderr always starts as "unbuffered"
> when a program starts, irrespective of how it was invoked?

AFAIR, on GNU/Linux stderr is not unbuffered by default.  We had a
long discussion of related issues in June 2019, I think people who
know about the glibc internals told there stderr is not really
completely unbuffered (and Posix allows that).

On MS-Windows it is by default unbuffered.

> Back to the windows behavior at hand, the simple test conducted with
> fwrite as the bug report mentioned, has a 2048 bytes associated with
> it which I will consider it to be categorized as "fully buffered",
> which brings it further away to the posix standard and even more further
> away from Linux being on the other end (always unbuffered).

I'm not sure your conclusion is correct.  I think the buffering is
imposed by the pipes which Emacs uses for communications with
subprocesses.  Or at least this is an additional buffering that is
related to the issue at hand.

> Unix convention is that stdin and stdout are line-buffered when
> associated with a terminal, and fully-buffered (aka block-buffered)
> otherwise. stderr is always unbuffered.
> 
> """
> 
> emphasis on *Unix convention ... stderr .. always unbuffered*, though
> it will be difficult to confirm this claim for all *nices)

I think Windows behaves the same, at least by default.  But GNU/Linux
isn't.

> For #2, emacs batch stderr behavior when invoked from outside the
> command line is not compatible with the posix standard, since it uses
> "fully buffered" mode, which actually makes it behave very differently
> from Linux (which is posix compliant by always having stderr as being
> unbuffered). Thus, the suggested patch actually addresses this
> concern, rather than invalidating it.

Again, are you sure it isn't the pipe that imposes buffering?

> (I understand ConPTY is the long term solution looking for a hacker,
> though I still do
> suppose the current behavior on windows is odd and needs to be
> addressed until that time).

I don't see it as odd.  The behavior of stdout and stdin could be odd
(again, due to pipes being used), but not that of stderr.

Message #23 received at 46388 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: ioannis.kappas <at> gmail.com
Cc: 46388 <at> debbugs.gnu.org
Subject: Re: bug#46388: 27.1;
 emacs -batch does not output messages immediately when invoked
 outside of the command prompt
Date: Tue, 09 Feb 2021 23:14:55 +0200

> Date: Tue, 09 Feb 2021 22:52:13 +0200
> From: Eli Zaretskii <eliz <at> gnu.org>
> Cc: 46388 <at> debbugs.gnu.org
> 
> > For #2, emacs batch stderr behavior when invoked from outside the
> > command line is not compatible with the posix standard, since it uses
> > "fully buffered" mode, which actually makes it behave very differently
> > from Linux (which is posix compliant by always having stderr as being
> > unbuffered). Thus, the suggested patch actually addresses this
> > concern, rather than invalidating it.
> 
> Again, are you sure it isn't the pipe that imposes buffering?

Moreover, even if we did make the change you propose, it will only
affect Emacs as a subprocess, but not when Emacs is the parent process
and some other program (e.g., Python) is the subprocess.  IOW, the
main problem, which is that interactive subprocesses behave on
MS-Windows differently from their behavior when invoked from the
console -- this problem will remain unsolved as long as Emacs can only
use pipes for communications with subprocesses.

Message #26 received at 46388 <at> debbugs.gnu.org (full text, mbox):

From: Ioannis Kappas <ioannis.kappas <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>, 46388 <at> debbugs.gnu.org
Subject: bug#46388: 27.1; emacs -batch does not output messages immediately
 when invoked outside of the command prompt
Date: Wed, 10 Feb 2021 12:48:38 +0000

Hi Eli,

    the reported issue and the draft patch is only concerned with the
    behavior of the MESSAGE function in emacs -batch, not with
    the communication of any emacs subprocess in general.

    I think I now understand where you are coming from.

    Let me summarize once more where we are, introducing buffering in
    the description (assuming MESSAGE length is < 2048):

    | # | System     | emacs -batch invoked from | MESSAGE behavior

        |
    |---+------------+---------------------------+----------------------------------------------------------------------------------------------------|
    | 1 | Linux      | bash                      | any MESSAGE is
immediately printed, i.e. stderr is unbuffered
              |
    | 2 | Linux      | emacs eshell/shell etc    | >>

        |
    | 3 | Windows 10 | command prompt            | >>

        |
    | 4 | Windows 10 | mintty                    | MESSAGEs are only
displayed when emacs -batch is about to exit, i.e. stderr appears to
be buffered |
    | 5 | Windows 10 | emacs eshell              | >>

        |

    The issue here is that if someone invokes emacs -batch with #4 and
    #5, they won't be getting any MESSAGEs/feedback on their terminal until the
    program exits. This, I consider to be unacceptable if the
caller/user is expecting
    to get feedback on their terminal from a long run emacs -batch
script. And thus the bug report.

    I think I've just realized what you were saying from the
    beginning. That the difference in behavior is expected, since
    it is the parent process which decides the buffering regime to be
    used for the subprocess. Thus in #5, it is emacs on windows that
    decided to invoke emacs -batch as a subprocess using pipes, which
    has resulted in emacs -batch's stderr to be buffered.

    From my side, seeing that #4 and #5 behaving similarly, I had made the wild
    assumption that it is was windows that is enforcing the emacs
    -batch's stderr to be buffered, only based on the fact that the subprocess
    was not attached to the console alone. And thus the suggested
    patch to correct this as such.

    I will need to look into how exactly mintty and emacs invoke a
    subprocess and confirm indeed that stderr is buffered because they
are both using
    pipes (or similar methods) as you suggested,  and is not as I have
arbitrarily assumed it to be.

    If the current behavior is indeed the correct expected behavior, how do I
    flush text message to stderr (or even stdout) from an emacs
    -batch script/eval?

    Hope this clarifies things a bit.

On Tue, Feb 9, 2021 at 9:14 PM Eli Zaretskii <eliz <at> gnu.org> wrote:
>
> > Date: Tue, 09 Feb 2021 22:52:13 +0200
> > From: Eli Zaretskii <eliz <at> gnu.org>
> > Cc: 46388 <at> debbugs.gnu.org
> >
> > > For #2, emacs batch stderr behavior when invoked from outside the
> > > command line is not compatible with the posix standard, since it uses
> > > "fully buffered" mode, which actually makes it behave very differently
> > > from Linux (which is posix compliant by always having stderr as being
> > > unbuffered). Thus, the suggested patch actually addresses this
> > > concern, rather than invalidating it.
> >
> > Again, are you sure it isn't the pipe that imposes buffering?
>
> Moreover, even if we did make the change you propose, it will only
> affect Emacs as a subprocess, but not when Emacs is the parent process
> and some other program (e.g., Python) is the subprocess.  IOW, the
> main problem, which is that interactive subprocesses behave on
> MS-Windows differently from their behavior when invoked from the
> console -- this problem will remain unsolved as long as Emacs can only
> use pipes for communications with subprocesses.

Message #29 received at 46388 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Ioannis Kappas <ioannis.kappas <at> gmail.com>
Cc: 46388 <at> debbugs.gnu.org
Subject: Re: bug#46388: 27.1; emacs -batch does not output messages immediately
 when invoked outside of the command prompt
Date: Wed, 10 Feb 2021 17:57:15 +0200

> From: Ioannis Kappas <ioannis.kappas <at> gmail.com>
> Date: Wed, 10 Feb 2021 12:48:38 +0000
> 
>     I think I now understand where you are coming from.
> 
>     Let me summarize once more where we are, introducing buffering in
>     the description (assuming MESSAGE length is < 2048):
> 
>     | # | System     | emacs -batch invoked from | MESSAGE behavior
> 
>         |
>     |---+------------+---------------------------+----------------------------------------------------------------------------------------------------|
>     | 1 | Linux      | bash                      | any MESSAGE is
> immediately printed, i.e. stderr is unbuffered
>               |
>     | 2 | Linux      | emacs eshell/shell etc    | >>

Did you try this 2nd item when the connection type for the subprocess
is 'pipe'?  Because otherwise we are comparing apples to oranges.

>     I think I've just realized what you were saying from the
>     beginning. That the difference in behavior is expected, since
>     it is the parent process which decides the buffering regime to be
>     used for the subprocess. Thus in #5, it is emacs on windows that
>     decided to invoke emacs -batch as a subprocess using pipes, which
>     has resulted in emacs -batch's stderr to be buffered.

That is correct.  As we don't support PTYs on Windows, we can only use
pipes for communicating with subprocesses there.

Btw, did you try to play with the value of w32-pipe-buffer-size,
e.g. setting it to a small value?

>     If the current behavior is indeed the correct expected behavior, how do I
>     flush text message to stderr (or even stdout) from an emacs
>     -batch script/eval?

My reading of the code is that we already fflush stderr after emitting
a message, so this should already happen.  See message_to_stderr.  If
that still doesn't help, then there's some buffering in the OS (for
example, in the pipe machinery itself), which we cannot control.

There were some changes in this area lately (that's the discussion
from 2019 I mentioned before): we now try to make a buffered variant
of stderr, and use that for error messages.  The reason, in a
nutshell, is that when you build Emacs with "make -jN", several copies
of the Emacs process can work in parallel, so it was deemed better to
have their messages be emitted atomically, instead of being
interspersed with one another, which produces an illegible mess.
However, that change makes a line-buffered variant of stderr, and on
Windows line buffering is the same as full buffering.  So maybe we
would need more changes in that area, for example some variable to
control this behavior instead of making it unconditional.

Message #32 received at 46388 <at> debbugs.gnu.org (full text, mbox):

From: Ioannis Kappas <ioannis.kappas <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 46388 <at> debbugs.gnu.org
Subject: Re: bug#46388: 27.1; emacs -batch does not output messages
 immediately when invoked outside of the command prompt
Date: Thu, 11 Feb 2021 08:10:34 +0000

On Wed, Feb 10, 2021 at 3:57 PM Eli Zaretskii <eliz <at> gnu.org> wrote:

> >
> >     | # | System     | emacs -batch invoked from | MESSAGE behavior
> >
> >         |
> >     |---+------------+---------------------------+----------------------------------------------------------------------------------------------------|
> >     | 1 | Linux      | bash                      | any MESSAGE is
> > immediately printed, i.e. stderr is unbuffered
> >               |
> >     | 2 | Linux      | emacs eshell/shell etc    | >>
>
> Did you try this 2nd item when the connection type for the subprocess
> is 'pipe'?  Because otherwise we are comparing apples to oranges.

I haven't, how do I do this? I was merely describing the plain user
experience of running emacs -batch from a shell.

for case #2 for example:

1. On Linux, open emacs
2.  M-x eshell
3. at the eshell prompt type the following command:
3.1 emacs -Q --batch --eval="(progn (message \"hi\") (sit-for 5))"
4. observe the result
4.1 "hi" is printed immediately, emacs -batch exits after five seconds.

In my tests, the second command was ran in all five cases, without applying any
other configuration to the parent command prompt/shell/eshell process.
Apologies if this was not clear.

> >     I think I've just realized what you were saying from the
> >     beginning. That the difference in behavior is expected, since
> >     it is the parent process which decides the buffering regime to be
> >     used for the subprocess. Thus in #5, it is emacs on windows that
> >     decided to invoke emacs -batch as a subprocess using pipes, which
> >     has resulted in emacs -batch's stderr to be buffered.
>
> That is correct.  As we don't support PTYs on Windows, we can only use
> pipes for communicating with subprocesses there.
>
> Btw, did you try to play with the value of w32-pipe-buffer-size,
> e.g. setting it to a small value?

No, I haven't experimented with pipes yet, my next plan is to study
how subprocess
are invoked from within emacs and mintty (since both demonstrate the
same behavior).

> >     If the current behavior is indeed the correct expected behavior, how do I
> >     flush text message to stderr (or even stdout) from an emacs
> >     -batch script/eval?
>
> My reading of the code is that we already fflush stderr after emitting
> a message, so this should already happen.  See message_to_stderr.  If
> that still doesn't help, then there's some buffering in the OS (for
> example, in the pipe machinery itself), which we cannot control.

the xdisp.c:message_to_stderr() is the first function i studied with
gdb when I started the investigation. Unless I've missed something,
it does not seem to lead to calling fflush (under windows at least):

1. it will sysdep.c:errwrite() the message
1.1 errwrite() will call sysdep.c:errstream() to get stderr
1.1.1 errstream() *may* call fflush on stderr in some system (via
buferr static variable), but this does not happen under windows-nt at
least.
1.2 will call fwrite to write the message
2. it may call sysdep.c:errputc() to to append a newline, following
exactly the same logic as for errwrite(), but using fputc() instead

/* Log the message M to stderr.  Log an empty line if M is not a string.  */

static void
message_to_stderr (Lisp_Object m)
{
  if (noninteractive_need_newline)
    {
      noninteractive_need_newline = false;
      errputc ('\n');
    }
  if (STRINGP (m))
    {
      Lisp_Object coding_system = Vlocale_coding_system;
      Lisp_Object s;

      if (!NILP (Vcoding_system_for_write))
coding_system = Vcoding_system_for_write;
      if (!NILP (coding_system))
s = code_convert_string_norecord (m, coding_system, true);
      else
s = m;

      errwrite (SDATA (s), SBYTES (s));
    }
  if (STRINGP (m) || !cursor_in_echo_area)
    errputc ('\n');
}


void
errputc (int c)
{
  fputc_unlocked (c, errstream ());
}

void
errwrite (void const *buf, ptrdiff_t nbuf)
{
  fwrite_unlocked (buf, 1, nbuf, errstream ());
}

/* Return the error output stream.  */
static FILE *
errstream (void)
{
  FILE *err = buferr;
  if (!err)
    return stderr;
  fflush_unlocked (stderr);
  return err;
}

in sysdep.c:init_standard_fds()
...
  /* Set buferr if possible on platforms defining _PC_PIPE_BUF, as
     they support the notion of atomic writes to pipes.  */
  #ifdef _PC_PIPE_BUF
    buferr = fdopen (STDERR_FILENO, "w");
    if (buferr)
      setvbuf (buferr, NULL, _IOLBF, 0);
  #endif
...



> There were some changes in this area lately (that's the discussion
> from 2019 I mentioned before): we now try to make a buffered variant
> of stderr, and use that for error messages.  The reason, in a
> nutshell, is that when you build Emacs with "make -jN", several copies
> of the Emacs process can work in parallel, so it was deemed better to
> have their messages be emitted atomically, instead of being
> interspersed with one another, which produces an illegible mess.
> However, that change makes a line-buffered variant of stderr, and on
> Windows line buffering is the same as full buffering.  So maybe we
> would need more changes in that area, for example some variable to
> control this behavior instead of making it unconditional.

Noted, thanks again!

Message #35 received at 46388 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Ioannis Kappas <ioannis.kappas <at> gmail.com>, Paul Eggert <eggert <at> cs.ucla.edu>
Cc: 46388 <at> debbugs.gnu.org
Subject: Re: bug#46388: 27.1; emacs -batch does not output messages
 immediately when invoked outside of the command prompt
Date: Thu, 11 Feb 2021 16:09:28 +0200

> From: Ioannis Kappas <ioannis.kappas <at> gmail.com>
> Date: Thu, 11 Feb 2021 08:10:34 +0000
> Cc: 46388 <at> debbugs.gnu.org
> 
> > My reading of the code is that we already fflush stderr after emitting
> > a message, so this should already happen.  See message_to_stderr.  If
> > that still doesn't help, then there's some buffering in the OS (for
> > example, in the pipe machinery itself), which we cannot control.
> 
> the xdisp.c:message_to_stderr() is the first function i studied with
> gdb when I started the investigation. Unless I've missed something,
> it does not seem to lead to calling fflush (under windows at least):

Then maybe this:

> /* Return the error output stream.  */
> static FILE *
> errstream (void)
> {
>   FILE *err = buferr;
>   if (!err)
>     return stderr;
>   fflush_unlocked (stderr);  <<<<<<<<<<<<<<<<
>   return err;
> }

should be fixed to fflush 'buferr' instead (or in addition to stderr)?

Paul, isn't that a bug that we fflush stderr here, and not 'buferr'?

Message #38 received at 46388 <at> debbugs.gnu.org (full text, mbox):

From: Ioannis Kappas <ioannis.kappas <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 46388 <at> debbugs.gnu.org, Paul Eggert <eggert <at> cs.ucla.edu>
Subject: Re: bug#46388: 27.1; emacs -batch does not output messages
 immediately when invoked outside of the command prompt
Date: Thu, 11 Feb 2021 19:25:31 +0000

> On 11 Feb 2021, at 14:09, Eli Zaretskii <eliz <at> gnu.org> wrote:
> 
>> From: Ioannis Kappas <ioannis.kappas <at> gmail.com>
>> Date: Thu, 11 Feb 2021 08:10:34 +0000
>> Cc: 46388 <at> debbugs.gnu.org
>> 
>>> My reading of the code is that we already fflush stderr after emitting
>>> a message, so this should already happen.  See message_to_stderr.  If
>>> that still doesn't help, then there's some buffering in the OS (for
>>> example, in the pipe machinery itself), which we cannot control.
>> 
>> the xdisp.c:message_to_stderr() is the first function i studied with
>> gdb when I started the investigation. Unless I've missed something,
>> it does not seem to lead to calling fflush (under windows at least):
> 
> Then maybe this:
> 
>> /* Return the error output stream.  */
>> static FILE *
>> errstream (void)
>> {
>>  FILE *err = buferr;
>>  if (!err)
>>    return stderr;
>>  fflush_unlocked (stderr);  <<<<<<<<<<<<<<<<
>>  return err;
>> }
> 
> should be fixed to fflush 'buferr' instead (or in addition to stderr)?
> 
> Paul, isn't that a bug that we fflush stderr here, and not 'buferr’?

(just a small note, “buffer" is NULL under windows, the fn thus returns without flushing anything. Even if buffer was not NULL, the fflush fn would have flushed the content of what ever has been accumulated on the stderr buffer so far, but not the message just  sent to message_to_stderr that we want to print out. Although, there would be this weird effect; message_to_sderr()  does an fwrite of the message followed by an fputc of a newline. This means that if errstream() was to fflush stderr, it would have flushed only the message written by fwrite, and not the newline written by fputc. I think that, if we are indeed considering to explicitly flush the message to stder, the correct place to do it would be directly inside the message_to_stderr(), thanks)

Message #41 received at 46388 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Ioannis Kappas <ioannis.kappas <at> gmail.com>
Cc: 46388 <at> debbugs.gnu.org, eggert <at> cs.ucla.edu
Subject: Re: bug#46388: 27.1; emacs -batch does not output messages
 immediately when invoked outside of the command prompt
Date: Thu, 11 Feb 2021 21:55:00 +0200

> From: Ioannis Kappas <ioannis.kappas <at> gmail.com>
> Date: Thu, 11 Feb 2021 19:25:31 +0000
> Cc: Paul Eggert <eggert <at> cs.ucla.edu>,
>  46388 <at> debbugs.gnu.org
> 
> (just a small note, “buffer" is NULL under windows

Which is a Good Thing, since otherwise we'd have stderr always fully
buffered on Windows (since _IOLBF on Windows is interpreted the same
as _IOFBF, i.e. fully-buffered).

>, the fn thus returns without flushing anything. Even if buffer was not NULL, the fflush fn would have flushed the content of what ever has been accumulated on the stderr buffer so far, but not the message just  sent to message_to_stderr that we want to print out. Although, there would be this weird effect; message_to_sderr()  does an fwrite of the message followed by an fputc of a newline. This means that if errstream() was to fflush stderr, it would have flushed only the message written by fwrite, and not the newline written by fputc. I think that, if we are indeed considering to explicitly flush the message to stder, the correct place to do it would be directly inside the message_to_stderr(), thanks)

If buferr is NULL, we are using the original stderr, which is supposed
to be unbuffered.

But I think you again are looking at the wrong side of the pipe: the
parent Emacs process reads from a pipe, which as its own buffering.
So whatever we do with Emacs's stderr will only affect subprocesses
when the child process is also Emacs, and will not have any effect on
other programs being run as subprocesses.

The correct solution, one that will seamlessly fix all the aspects of
the buffering, is to add pseudo-console support to Emacs on Windows,
and use that by default on systems that can support it.

Message #44 received at 46388 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 46388 <at> debbugs.gnu.org, Ioannis Kappas <ioannis.kappas <at> gmail.com>
Subject: Re: bug#46388: 27.1; emacs -batch does not output messages
 immediately when invoked outside of the command prompt
Date: Thu, 11 Feb 2021 13:15:42 -0800

On 2/11/21 6:09 AM, Eli Zaretskii wrote:
> Then maybe this:
> 
>> /* Return the error output stream.  */
>> static FILE *
>> errstream (void)
>> {
>>    FILE *err = buferr;
>>    if (!err)
>>      return stderr;
>>    fflush_unlocked (stderr);  <<<<<<<<<<<<<<<<
>>    return err;
>> }
> should be fixed to fflush 'buferr' instead (or in addition to stderr)?
> 
> Paul, isn't that a bug that we fflush stderr here, and not 'buferr'?

No, it's intended. The goal of 'errstream' is to flush out all the stuff 
written to stderr, before outputting anything sent to buferr. There is 
no need to fflush buferr here, since it's gonna be flushed soon anyway 
when a newline is output.

Functions like message_to_stderr etc. are supposed to do a series of 
calls to one or more functions like errwrite that end with outputting a 
newline, without invoking any code that outputs directly to stderr. This 
is explained in the comment before sysdep.c's errputc function, a 
comment that could perhaps be improved.

Message #47 received at 46388 <at> debbugs.gnu.org (full text, mbox):

From: Ioannis Kappas <ioannis.kappas <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 46388 <at> debbugs.gnu.org, Paul Eggert <eggert <at> cs.ucla.edu>
Subject: Re: bug#46388: 27.1; emacs -batch does not output messages
 immediately when invoked outside of the command prompt
Date: Fri, 12 Feb 2021 19:59:02 +0000

Hi Eli

Just to recap (with a touch of drama this time).

The following command

emacs -Q --batch --eval="(progn (message \"The bomb will explode in 10
seconds, cut the red cable to disarm\") (sit-for 3600))"

will behave differently depending on the architecture/prompt it is invoked from:

When it is invoked from the Linux terminal or from a plain emacs
eshell (i.e. M-x eshell) prompt  or from a windows command prompt =>
message is printed out immediately, user cuts the red cable, bomb
doesn't explode.

When it is invoked from msys64 mintty terminal or from emacs eshell on
windows => no message is printed out (until emacs -batch returns), no
cables cut, bomb explodes after ten seconds.

How do you suggest we flush the message out so that it is always
displayed to the user irrespective of the architecture/prompt it is
ran on? Is there perhaps an elisp function to flush messages out (e.g.
(progn (message "...") (message-flush!) (sit-for ...))) ?

So far we said:
1. We don't want to a solution where the emacs -batch's stderrr is
forced to unbuffered mode (original patch proposal)
2. message_to_stderr() does not directly or indirectly invoke stderr
flush (at least in windows).

Thanks

On Thu, Feb 11, 2021 at 7:55 PM Eli Zaretskii <eliz <at> gnu.org> wrote:
>
> > From: Ioannis Kappas <ioannis.kappas <at> gmail.com>
> > Date: Thu, 11 Feb 2021 19:25:31 +0000
> > Cc: Paul Eggert <eggert <at> cs.ucla.edu>,
> >  46388 <at> debbugs.gnu.org
> >
> > (just a small note, “buffer" is NULL under windows
>
> Which is a Good Thing, since otherwise we'd have stderr always fully
> buffered on Windows (since _IOLBF on Windows is interpreted the same
> as _IOFBF, i.e. fully-buffered).
>
> >, the fn thus returns without flushing anything. Even if buffer was not NULL, the fflush fn would have flushed the content of what ever has been accumulated on the stderr buffer so far, but not the message just  sent to message_to_stderr that we want to print out. Although, there would be this weird effect; message_to_sderr()  does an fwrite of the message followed by an fputc of a newline. This means that if errstream() was to fflush stderr, it would have flushed only the message written by fwrite, and not the newline written by fputc. I think that, if we are indeed considering to explicitly flush the message to stder, the correct place to do it would be directly inside the message_to_stderr(), thanks)
>
> If buferr is NULL, we are using the original stderr, which is supposed
> to be unbuffered.
>
> But I think you again are looking at the wrong side of the pipe: the
> parent Emacs process reads from a pipe, which as its own buffering.
> So whatever we do with Emacs's stderr will only affect subprocesses
> when the child process is also Emacs, and will not have any effect on
> other programs being run as subprocesses.
>
> The correct solution, one that will seamlessly fix all the aspects of
> the buffering, is to add pseudo-console support to Emacs on Windows,
> and use that by default on systems that can support it.

Message #50 received at 46388 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Ioannis Kappas <ioannis.kappas <at> gmail.com>
Cc: 46388 <at> debbugs.gnu.org, eggert <at> cs.ucla.edu
Subject: Re: bug#46388: 27.1; emacs -batch does not output messages
 immediately when invoked outside of the command prompt
Date: Fri, 12 Feb 2021 22:03:40 +0200

> From: Ioannis Kappas <ioannis.kappas <at> gmail.com>
> Date: Fri, 12 Feb 2021 19:59:02 +0000
> Cc: Paul Eggert <eggert <at> cs.ucla.edu>, 46388 <at> debbugs.gnu.org
> 
> How do you suggest we flush the message out so that it is always
> displayed to the user irrespective of the architecture/prompt it is
> ran on? Is there perhaps an elisp function to flush messages out (e.g.
> (progn (message "...") (message-flush!) (sit-for ...))) ?
> 
> So far we said:
> 1. We don't want to a solution where the emacs -batch's stderrr is
> forced to unbuffered mode (original patch proposal)
> 2. message_to_stderr() does not directly or indirectly invoke stderr
> flush (at least in windows).

As I said, this cannot be helped, not until we teach Emacs on Windows
to use the pseudo-console.  This is how the platform behaves when we
use pipes.

Sorry, there's no way around this.

Message #53 received at 46388 <at> debbugs.gnu.org (full text, mbox):

From: Ioannis Kappas <ioannis.kappas <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 46388 <at> debbugs.gnu.org, Paul Eggert <eggert <at> cs.ucla.edu>
Subject: Re: bug#46388: 27.1; emacs -batch does not output messages
 immediately when invoked outside of the command prompt
Date: Sat, 6 Mar 2021 15:00:17 +0000

Hi Eli,

(apologies for the long email)

On Fri, Feb 12, 2021 at 8:03 PM Eli Zaretskii <eliz <at> gnu.org> wrote:
>
> > From: Ioannis Kappas <ioannis.kappas <at> gmail.com>
> > Date: Fri, 12 Feb 2021 19:59:02 +0000
> > Cc: Paul Eggert <eggert <at> cs.ucla.edu>, 46388 <at> debbugs.gnu.org
> >
> > How do you suggest we flush the message out so that it is always
> > displayed to the user irrespective of the architecture/prompt it is
> > ran on? Is there perhaps an elisp function to flush messages out (e.g.
> > (progn (message "...") (message-flush!) (sit-for ...))) ?
> >
>
> As I said, this cannot be helped, not until we teach Emacs on Windows
> to use the pseudo-console.  This is how the platform behaves when we
> use pipes.
>
> Sorry, there's no way around this.

although ConPTY support will hopefully solve all issues with regards
to an Emacs parent process interacting with its children, it wont
solve this particular issue when the parent process is different than
Emacs or Emacs is running on a Windows version less than 10 (ConPTY as
I understand is only available on Windows 10).

A have created a little C program and written an analysis on the
buffering behavior of stderr when redirected to a pipe at
https://github.com/ikappaki/standard-streams-test#readme, with the aim
to understanding the technology involved, reach your level of
understanding and hopefully strengthen my argument for a fix.

Just to summarize the results:

1. stderr is unbuffered when attached to the console.
2. A child's stderr stream is fully buffered when redirected to a
   pipe, with a default buffer size of 4096 bytes long. This buffer is
   complementary to the pipe's buffer.
3. The pipe's buffer size, configured by the parent process, defaults
   to a size of 4096 bytes long (the size is controlled by
   `w32-pipe-buffer-size' in Emacs, of which see).
4. The stderr stream mode can be reset to unbuffered or fully buffered
   at process start up when no characters are yet written to it.

(PARENT PROCESS) READER <= [PIPE BUFFER] <= [STDERR STREAM BUFFER] <=
WRITER (CHILD PROCESS)

Given the above results, it seems to me like a proper fix to reset the
stream to unbuffered when Emacs --batch stderr stream has been
redirected to a pipe it, ensuring first of course that we distinguish
between pipe, file and socket redirections. I believe one of the
arguments against the original patch is that it will affect
redirections to files, which we do need to be buffered at the stream
level.

Please note, that resetting the stream's buffer mode does not reset
the pipe's buffer which is still in effect. The child writer (in our
case Emacs --batch) will not block on a write, unless of course the
write is more than 4096 bytes long and the parent process is
unavailable to consume data.

Here is the updated patch:

@@ -10266,6 +10266,37 @@ init_ntproc (int dumping)
     else
       _open ("nul", O_TEXT | O_NOINHERIT | O_WRONLY);
     _fdopen (2, "w");
+
+    /* stderr is unbuffered when attached to the console but fully
+       buffered (4096 bytes) when redirected to a pipe (this buffer is
+       complementary to the pipe buffer).
+
+       Since Emacs --batch does not, at least on Windows, flush stderr
+       it means output (such as that generated with `message') will
+       not reach the parent process, not until at least this process
+       exits or the stream buffer is full, resulting to a very poor
+       interaction with the parent process.
+
+       We thus switch to unbuffered mode and rely only on the pipe's
+       buffer for ensuring this process is not blocked writing data
+       chunks to stderr (the default pipe's buffer size is 4096 bytes,
+       but this is only configurable by the parent process). Setting
+       the stream up to line buffer mode unfortunately does not work
+       because it is the same as fully buffered on win32.
+
+       Note that the below does not affect stderr redirection to a
+       file, since this will come back as `FILE_TYPE_DISK'.
+    */
+    HANDLE seh = (HANDLE)_get_osfhandle(_fileno(stderr));
+    if (noninteractive
+ /* is pipe (anonymous, named or socket)*/
+ && GetFileType(seh) == FILE_TYPE_PIPE
+ /* and is definitely not a socket */
+ && GetNamedPipeInfo(seh, NULL, NULL, NULL, NULL))
+      {
+ setvbuf(stderr, NULL, _IONBF, 0);
+ DebPrint((":stderr-mode-set-to-unbuf\n"));
+      }

Please let me know of any cases you might think removing stderr buffer
might prove problematic and is not covered by the presence of the
pipe's buffer presence alone.

An alternative to the above, and perhaps a little more involved, is to
provide an interface to `message' to be able to flush a
pipe-redirected stderr stream after outputting a message with a
newline. This will make it behave as if the `message's output was line
buffered, which I believe would have been the ideal outcome for stderr
to begin with.

Please also note that as mentioned earlier in this thread, there is a
precedence of flushing stderr "on platforms defining _PC_PIPE_BUF"
(see /sysdep.c:init_standard_fds()/), strengthening the argument of the
eligibility of the following fix.

In the patch below, I tried to maintain what I thought was the
existing writing style and also make it obvious that the decision to
flush the buffer was explicitly taken by /xdisp.c:message_to_stderr()/
rather than perhaps hiding it in /sysdep.c:errputc()/, although I am
not so sure about scattering such a simple login in three different
files:

@@ -2789,6 +2789,30 @@ safe_strsignal (int code)
/* Output to stderr.  */

+/* On windows, stderr is unbuffered when attached to the console but
+   fully buffered (4096 bytes) when redirected to a pipe (this buffer
+   is complementary to the pipe buffer).
+
+   Since Emacs --batch, at least on Windows, does not flush stderr it
+   means output (such as that created with `message') will not reach
+   the parent process, not until at least this process exits or the
+   stream buffer is full, resulting to a very poor interaction with
+   the parent process.
+
+   We thus provide an interface to functions such as `message' to
+   flush stderr after they output a newline, make them in effect
+   behave as if stderr was line buffered (unfortunately resetting the
+   stream to line buffer mode is the same as resetting it to full
+   buffer mode, so this can't be enforced at the stream level).
+*/
+void maybe_flush_stderr_after_newline (void)
+{
+#ifdef WINDOWSNT
+  if (is_stderr_pipe())
+    fflush_unlocked(stderr);
+#endif /* WINDOWSNT */
+}
+
 /* Return the error output stream.  */
 static FILE *
 errstream (void)
modified   src/sysstdio.h
@@ -27,6 +27,7 @@ #define EMACS_SYSSTDIO_H
 #include "unlocked-io.h"

 extern FILE *emacs_fopen (char const *, char const *);
+extern void maybe_flush_stderr_after_newline (void);
 extern void errputc (int);
 extern void errwrite (void const *, ptrdiff_t);
 extern void close_output_streams (void);
modified   src/w32.c
@@ -10190,6 +10190,12 @@ term_ntproc (int ignored)
   term_w32select ();
 }

+static bool _is_stderr_pipe = false;
+bool is_stderr_pipe (void)
+{
+  return _is_stderr_pipe;
+}
+
 void
 init_ntproc (int dumping)
 {
@@ -10268,6 +10274,13 @@ init_ntproc (int dumping)
     _fdopen (2, "w");
   }

+  HANDLE seh = (HANDLE)_get_osfhandle(_fileno(stderr));
+  _is_stderr_pipe =
+    /* is pipe (anonymous, named or socket)*/
+    GetFileType(seh) == FILE_TYPE_PIPE
+    /* and is definitely not a socket */
+    && GetNamedPipeInfo(seh, NULL, NULL, NULL, NULL);
+
   /* unfortunately, atexit depends on implementation of malloc */
   /* atexit (term_ntproc); */
   if (!dumping)
modified   src/w32.h
@@ -170,6 +170,9 @@ #define FILE_SERIAL             0x0800
 extern HANDLE maybe_load_unicows_dll (void);
 extern void globals_of_w32 (void);

+/* return whether standard error is redirected to a pipe. */
+extern bool is_stderr_pipe (void);
+
 extern void term_timers (void);
 extern void init_timers (void);

modified   src/xdisp.c
@@ -10968,6 +10968,7 @@ message_to_stderr (Lisp_Object m)
     {
       noninteractive_need_newline = false;
       errputc ('\n');
+      maybe_flush_stderr_after_newline();
     }
   if (STRINGP (m))
     {
@@ -10984,7 +10985,11 @@ message_to_stderr (Lisp_Object m)
       errwrite (SDATA (s), SBYTES (s));
     }
   if (STRINGP (m) || !cursor_in_echo_area)
-    errputc ('\n');
+    {
+      errputc ('\n');
+      maybe_flush_stderr_after_newline();
+    }
+
 }

 /* The non-logging version of message3.

Thank you!

Message #56 received at 46388 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Ioannis Kappas <ioannis.kappas <at> gmail.com>, Eli Zaretskii <eliz <at> gnu.org>
Cc: 46388 <at> debbugs.gnu.org
Subject: Re: bug#46388: 27.1; emacs -batch does not output messages
 immediately when invoked outside of the command prompt
Date: Sun, 7 Mar 2021 20:05:06 -0800

[Message part 1 (text/plain, inline)]

Wouldn't the attached be a simpler fix?

[0001-On-MS-Windows-fflush-stderr-after-newline.patch (text/x-patch, attachment)]

Message #59 received at 46388 <at> debbugs.gnu.org (full text, mbox):

From: Ioannis Kappas <ioannis.kappas <at> gmail.com>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: Eli Zaretskii <eliz <at> gnu.org>, 46388 <at> debbugs.gnu.org
Subject: Re: bug#46388: 27.1; emacs -batch does not output messages
 immediately when invoked outside of the command prompt
Date: Mon, 8 Mar 2021 07:56:27 +0000

Hi Paul,

On Mon, Mar 8, 2021 at 4:05 AM Paul Eggert <eggert <at> cs.ucla.edu> wrote:
>
> Wouldn't the attached be a simpler fix?

It is fine by me.

Thanks

Message #62 received at 46388 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: ioannis.kappas <at> gmail.com, 46388 <at> debbugs.gnu.org
Subject: Re: bug#46388: 27.1; emacs -batch does not output messages
 immediately when invoked outside of the command prompt
Date: Thu, 11 Mar 2021 16:27:42 +0200

> Cc: 46388 <at> debbugs.gnu.org
> From: Paul Eggert <eggert <at> cs.ucla.edu>
> Date: Sun, 7 Mar 2021 20:05:06 -0800
> 
> Wouldn't the attached be a simpler fix?
> 
> >From ff81c979a9528af8065169f83a78de599e99177b Mon Sep 17 00:00:00 2001
> From: Paul Eggert <eggert <at> cs.ucla.edu>
> Date: Sun, 7 Mar 2021 20:01:37 -0800
> Subject: [PATCH] On MS-Windows, fflush stderr after newline
> 
> Problem reported by Ioannis Kappas (Bug#46388).
> * src/sysdep.c (errputc) [WINDOWSNT]: Flush stderr after newline.
> ---
>  src/sysdep.c | 7 +++++++
>  1 file changed, 7 insertions(+)
> 
> diff --git a/src/sysdep.c b/src/sysdep.c
> index 941b4e2fa2..88dd3e71e4 100644
> --- a/src/sysdep.c
> +++ b/src/sysdep.c
> @@ -2662,6 +2662,13 @@ errstream (void)
>  errputc (int c)
>  {
>    fputc_unlocked (c, errstream ());
> +
> +#ifdef WINDOWSNT
> +  /* Flush stderr after outputting a newline since stderr is fully
> +     buffered when redirected to a pipe, contrary to POSIX.  */
> +  if (c == '\n')
> +    fflush_unlocked (stderr);
> +#endif
>  }
>  

This is fine, please install.

Thanks (and sorry for a late response).

Message #67 received at 46388-done <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: ioannis.kappas <at> gmail.com, 46388-done <at> debbugs.gnu.org
Subject: Re: bug#46388: 27.1; emacs -batch does not output messages
 immediately when invoked outside of the command prompt
Date: Thu, 11 Mar 2021 10:43:40 -0800

On 3/11/21 6:27 AM, Eli Zaretskii wrote:
> This is fine, please install.

OK, done, and closing the bug report.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.