GNU bug report logs - #46388
27.1; emacs -batch does not output messages immediately when invoked outside of the command prompt

Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.

Package: emacs; Reported by: Ioannis Kappas <ioannis.kappas@HIDDEN>; dated Mon, 8 Feb 2021 21:22:02 UTC; Maintainer for emacs is bug-gnu-emacs@HIDDEN.

Message received at 46388 <at> debbugs.gnu.org:


Received: (at 46388) by debbugs.gnu.org; 10 Feb 2021 15:57:24 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Wed Feb 10 10:57:24 2021
Received: from localhost ([127.0.0.1]:56953 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1l9rrn-0005w7-QG
	for submit <at> debbugs.gnu.org; Wed, 10 Feb 2021 10:57:24 -0500
Received: from eggs.gnu.org ([209.51.188.92]:42930)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1l9rrm-0005vu-2K
 for 46388 <at> debbugs.gnu.org; Wed, 10 Feb 2021 10:57:22 -0500
Received: from fencepost.gnu.org ([2001:470:142:3::e]:45923)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <eliz@HIDDEN>)
 id 1l9rrg-00038v-QL; Wed, 10 Feb 2021 10:57:16 -0500
Received: from 84.94.185.95.cable.012.net.il ([84.94.185.95]:2490
 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1l9rrf-0003B7-15; Wed, 10 Feb 2021 10:57:15 -0500
Date: Wed, 10 Feb 2021 17:57:15 +0200
Message-Id: <83pn17j4pw.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Ioannis Kappas <ioannis.kappas@HIDDEN>
In-Reply-To: <CAMRHuGCtVw-qyWLc-2i36NbsGRiqg8LpDn3DN9MYqsTF15k=hA@HIDDEN>
 (message from Ioannis Kappas on Wed, 10 Feb 2021 12:48:38 +0000)
Subject: Re: bug#46388: 27.1; emacs -batch does not output messages immediately
 when invoked outside of the command prompt
References: <CAMRHuGB-z+uiS7F71vaLU2L8X-W9BAwUH0+aSDvGz7HmBYmnqQ@HIDDEN>
 <CAMRHuGDC-XAznW-0YtpnCncFkxBRywx-jDgZ2eQf8UNdebR8Og@HIDDEN>
 <E1l9LhR-0002lF-EV@HIDDEN>
 <CAMRHuGC+8ZP270Erd8q=MTrw7CUS-7+PX7GTFc2JJgkexorAsw@HIDDEN>
 <83h7mlj75u.fsf@HIDDEN> <83eehpj640.fsf@HIDDEN>
 <CAMRHuGC_p59uw_hmCL65Z0F1ZdFbVAf9MHcB-sX88bW6jchC-Q@HIDDEN>
 <CAMRHuGCtVw-qyWLc-2i36NbsGRiqg8LpDn3DN9MYqsTF15k=hA@HIDDEN>
X-Spam-Score: -0.7 (/)
X-Debbugs-Envelope-To: 46388
Cc: 46388 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

> From: Ioannis Kappas <ioannis.kappas@HIDDEN>
> Date: Wed, 10 Feb 2021 12:48:38 +0000
> 
>     I think I now understand where you are coming from.
> 
>     Let me summarize once more where we are, introducing buffering in
>     the description (assuming MESSAGE length is < 2048):
> 
>     | # | System     | emacs -batch invoked from | MESSAGE behavior
> 
>         |
>     |---+------------+---------------------------+----------------------------------------------------------------------------------------------------|
>     | 1 | Linux      | bash                      | any MESSAGE is
> immediately printed, i.e. stderr is unbuffered
>               |
>     | 2 | Linux      | emacs eshell/shell etc    | >>

Did you try this 2nd item when the connection type for the subprocess
is 'pipe'?  Because otherwise we are comparing apples to oranges.

>     I think I've just realized what you were saying from the
>     beginning. That the difference in behavior is expected, since
>     it is the parent process which decides the buffering regime to be
>     used for the subprocess. Thus in #5, it is emacs on windows that
>     decided to invoke emacs -batch as a subprocess using pipes, which
>     has resulted in emacs -batch's stderr to be buffered.

That is correct.  As we don't support PTYs on Windows, we can only use
pipes for communicating with subprocesses there.

Btw, did you try to play with the value of w32-pipe-buffer-size,
e.g. setting it to a small value?

>     If the current behavior is indeed the correct expected behavior, how do I
>     flush text message to stderr (or even stdout) from an emacs
>     -batch script/eval?

My reading of the code is that we already fflush stderr after emitting
a message, so this should already happen.  See message_to_stderr.  If
that still doesn't help, then there's some buffering in the OS (for
example, in the pipe machinery itself), which we cannot control.

There were some changes in this area lately (that's the discussion
from 2019 I mentioned before): we now try to make a buffered variant
of stderr, and use that for error messages.  The reason, in a
nutshell, is that when you build Emacs with "make -jN", several copies
of the Emacs process can work in parallel, so it was deemed better to
have their messages be emitted atomically, instead of being
interspersed with one another, which produces an illegible mess.
However, that change makes a line-buffered variant of stderr, and on
Windows line buffering is the same as full buffering.  So maybe we
would need more changes in that area, for example some variable to
control this behavior instead of making it unconditional.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#46388; Package emacs. Full text available.

Message received at 46388 <at> debbugs.gnu.org:


Received: (at 46388) by debbugs.gnu.org; 10 Feb 2021 12:48:55 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Wed Feb 10 07:48:55 2021
Received: from localhost ([127.0.0.1]:55698 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1l9ovP-0007Pr-2c
	for submit <at> debbugs.gnu.org; Wed, 10 Feb 2021 07:48:55 -0500
Received: from mail-oi1-f178.google.com ([209.85.167.178]:45851)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ioannis.kappas@HIDDEN>) id 1l9ovM-0007Pf-V1
 for 46388 <at> debbugs.gnu.org; Wed, 10 Feb 2021 07:48:54 -0500
Received: by mail-oi1-f178.google.com with SMTP id m7so1810656oiw.12
 for <46388 <at> debbugs.gnu.org>; Wed, 10 Feb 2021 04:48:52 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=mime-version:references:in-reply-to:from:date:message-id:subject:to;
 bh=POLmag0GoWnTB2xcL3yMrkOcD/+RPwpzgqXAoXp8nJI=;
 b=NvsgtXW4uLWwj8eDKrdkgWVjyMI1lxRxQkeZgzehkF84lwZIN7viLgyasB5zEVUIHA
 Nf9Q3yuEZ2dljUsuITFQ5RU70dBCUwKdkT/yYyVr9cnDEiA0SjgxHQ21OgstyujV7Vq0
 bWBKaJ2xF3sA6ocLKM8bHdQA19I8WdeTL+zkPww2aMa6fhJJW/bq92AELH+Y5t8ozsG+
 P7rVT8jnJny3QWm+TE3NsbfzhcwdIMkxn1F5SqOrtrYvazxBZKFidhPXjRWcFCL5u8n9
 xacszm7NoFliMcaaOEyOC+O7mERR3Ql9zHx+W8g6xfTZulEKdBiGw0+umQtjfioNYyeZ
 eWpw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:mime-version:references:in-reply-to:from:date
 :message-id:subject:to;
 bh=POLmag0GoWnTB2xcL3yMrkOcD/+RPwpzgqXAoXp8nJI=;
 b=X8OmLau0lXA+Swl4xp1iFhEiVZm6CLtGKDODozzbhWnL2yg8MhsEnB3GcbCgrGITRq
 W23nLW5FmvB2Lh7yf1WfiswRgeBNczDp9TQ2SjgAYsvqUfEwWalCzLk7T7SY2jgI21T7
 kXz7PP8BntoRSnVYwF0AdHvi/l3R2kpvMDIDu1PEBwq8RGhKQc9nURAeL8eygAu5XfbV
 8ByH4V2TF1xFz692G12Xy1JGXdWlr0KgN3urvsRuAXCouGSVmsluOCGlvJtHWLKS9DNN
 4+HIzT+VKGkpEq3ybqEj1WEBRAcHn0qOJQrorsbKf8zWJHNgBmeWErGjKWOaKdz5xpK9
 jKuA==
X-Gm-Message-State: AOAM533h2gmkELV9P1od3WTKjKK2Sc1A+FVL4pRK7TGgY3i9w0BDwta4
 5OLFwixpBd83GUhZiNiwcxi3Z5/1+CQUta0+wfo=
X-Google-Smtp-Source: ABdhPJzs2FpJuwWuUrRm0b51BwB7r1WFneXlp23CnKikOYX8RtjTDp2CAFdU40hui8UfdHZqpbKGLPWUpKsfVGFx774=
X-Received: by 2002:aca:1a1a:: with SMTP id a26mr1980147oia.78.1612961327124; 
 Wed, 10 Feb 2021 04:48:47 -0800 (PST)
MIME-Version: 1.0
References: <CAMRHuGB-z+uiS7F71vaLU2L8X-W9BAwUH0+aSDvGz7HmBYmnqQ@HIDDEN>
 <CAMRHuGDC-XAznW-0YtpnCncFkxBRywx-jDgZ2eQf8UNdebR8Og@HIDDEN>
 <E1l9LhR-0002lF-EV@HIDDEN>
 <CAMRHuGC+8ZP270Erd8q=MTrw7CUS-7+PX7GTFc2JJgkexorAsw@HIDDEN>
 <83h7mlj75u.fsf@HIDDEN> <83eehpj640.fsf@HIDDEN>
 <CAMRHuGC_p59uw_hmCL65Z0F1ZdFbVAf9MHcB-sX88bW6jchC-Q@HIDDEN>
In-Reply-To: <CAMRHuGC_p59uw_hmCL65Z0F1ZdFbVAf9MHcB-sX88bW6jchC-Q@HIDDEN>
From: Ioannis Kappas <ioannis.kappas@HIDDEN>
Date: Wed, 10 Feb 2021 12:48:38 +0000
Message-ID: <CAMRHuGCtVw-qyWLc-2i36NbsGRiqg8LpDn3DN9MYqsTF15k=hA@HIDDEN>
Subject: bug#46388: 27.1; emacs -batch does not output messages immediately
 when invoked outside of the command prompt
To: Eli Zaretskii <eliz@HIDDEN>, 46388 <at> debbugs.gnu.org
Content-Type: text/plain; charset="UTF-8"
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 46388
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Hi Eli,

    the reported issue and the draft patch is only concerned with the
    behavior of the MESSAGE function in emacs -batch, not with
    the communication of any emacs subprocess in general.

    I think I now understand where you are coming from.

    Let me summarize once more where we are, introducing buffering in
    the description (assuming MESSAGE length is < 2048):

    | # | System     | emacs -batch invoked from | MESSAGE behavior

        |
    |---+------------+---------------------------+----------------------------------------------------------------------------------------------------|
    | 1 | Linux      | bash                      | any MESSAGE is
immediately printed, i.e. stderr is unbuffered
              |
    | 2 | Linux      | emacs eshell/shell etc    | >>

        |
    | 3 | Windows 10 | command prompt            | >>

        |
    | 4 | Windows 10 | mintty                    | MESSAGEs are only
displayed when emacs -batch is about to exit, i.e. stderr appears to
be buffered |
    | 5 | Windows 10 | emacs eshell              | >>

        |

    The issue here is that if someone invokes emacs -batch with #4 and
    #5, they won't be getting any MESSAGEs/feedback on their terminal until the
    program exits. This, I consider to be unacceptable if the
caller/user is expecting
    to get feedback on their terminal from a long run emacs -batch
script. And thus the bug report.

    I think I've just realized what you were saying from the
    beginning. That the difference in behavior is expected, since
    it is the parent process which decides the buffering regime to be
    used for the subprocess. Thus in #5, it is emacs on windows that
    decided to invoke emacs -batch as a subprocess using pipes, which
    has resulted in emacs -batch's stderr to be buffered.

    From my side, seeing that #4 and #5 behaving similarly, I had made the wild
    assumption that it is was windows that is enforcing the emacs
    -batch's stderr to be buffered, only based on the fact that the subprocess
    was not attached to the console alone. And thus the suggested
    patch to correct this as such.

    I will need to look into how exactly mintty and emacs invoke a
    subprocess and confirm indeed that stderr is buffered because they
are both using
    pipes (or similar methods) as you suggested,  and is not as I have
arbitrarily assumed it to be.

    If the current behavior is indeed the correct expected behavior, how do I
    flush text message to stderr (or even stdout) from an emacs
    -batch script/eval?

    Hope this clarifies things a bit.

On Tue, Feb 9, 2021 at 9:14 PM Eli Zaretskii <eliz@HIDDEN> wrote:
>
> > Date: Tue, 09 Feb 2021 22:52:13 +0200
> > From: Eli Zaretskii <eliz@HIDDEN>
> > Cc: 46388 <at> debbugs.gnu.org
> >
> > > For #2, emacs batch stderr behavior when invoked from outside the
> > > command line is not compatible with the posix standard, since it uses
> > > "fully buffered" mode, which actually makes it behave very differently
> > > from Linux (which is posix compliant by always having stderr as being
> > > unbuffered). Thus, the suggested patch actually addresses this
> > > concern, rather than invalidating it.
> >
> > Again, are you sure it isn't the pipe that imposes buffering?
>
> Moreover, even if we did make the change you propose, it will only
> affect Emacs as a subprocess, but not when Emacs is the parent process
> and some other program (e.g., Python) is the subprocess.  IOW, the
> main problem, which is that interactive subprocesses behave on
> MS-Windows differently from their behavior when invoked from the
> console -- this problem will remain unsolved as long as Emacs can only
> use pipes for communications with subprocesses.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#46388; Package emacs. Full text available.

Message received at 46388 <at> debbugs.gnu.org:


Received: (at 46388) by debbugs.gnu.org; 9 Feb 2021 21:15:05 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Feb 09 16:15:05 2021
Received: from localhost ([127.0.0.1]:54846 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1l9aLg-0001LP-NM
	for submit <at> debbugs.gnu.org; Tue, 09 Feb 2021 16:15:04 -0500
Received: from eggs.gnu.org ([209.51.188.92]:41834)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1l9aLe-0001Ke-Tm
 for 46388 <at> debbugs.gnu.org; Tue, 09 Feb 2021 16:15:03 -0500
Received: from fencepost.gnu.org ([2001:470:142:3::e]:56171)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <eliz@HIDDEN>)
 id 1l9aLZ-0004G1-OG; Tue, 09 Feb 2021 16:14:57 -0500
Received: from 84.94.185.95.cable.012.net.il ([84.94.185.95]:1189
 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1l9aLY-0003Ak-NU; Tue, 09 Feb 2021 16:14:57 -0500
Date: Tue, 09 Feb 2021 23:14:55 +0200
Message-Id: <83eehpj640.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: ioannis.kappas@HIDDEN
In-Reply-To: <83h7mlj75u.fsf@HIDDEN> (message from Eli Zaretskii on Tue, 09
 Feb 2021 22:52:13 +0200)
Subject: Re: bug#46388: 27.1;
 emacs -batch does not output messages immediately when invoked
 outside of the command prompt
References: <CAMRHuGB-z+uiS7F71vaLU2L8X-W9BAwUH0+aSDvGz7HmBYmnqQ@HIDDEN>
 <CAMRHuGDC-XAznW-0YtpnCncFkxBRywx-jDgZ2eQf8UNdebR8Og@HIDDEN>
 <E1l9LhR-0002lF-EV@HIDDEN>
 <CAMRHuGC+8ZP270Erd8q=MTrw7CUS-7+PX7GTFc2JJgkexorAsw@HIDDEN>
 <83h7mlj75u.fsf@HIDDEN>
X-Spam-Score: -0.7 (/)
X-Debbugs-Envelope-To: 46388
Cc: 46388 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

> Date: Tue, 09 Feb 2021 22:52:13 +0200
> From: Eli Zaretskii <eliz@HIDDEN>
> Cc: 46388 <at> debbugs.gnu.org
> 
> > For #2, emacs batch stderr behavior when invoked from outside the
> > command line is not compatible with the posix standard, since it uses
> > "fully buffered" mode, which actually makes it behave very differently
> > from Linux (which is posix compliant by always having stderr as being
> > unbuffered). Thus, the suggested patch actually addresses this
> > concern, rather than invalidating it.
> 
> Again, are you sure it isn't the pipe that imposes buffering?

Moreover, even if we did make the change you propose, it will only
affect Emacs as a subprocess, but not when Emacs is the parent process
and some other program (e.g., Python) is the subprocess.  IOW, the
main problem, which is that interactive subprocesses behave on
MS-Windows differently from their behavior when invoked from the
console -- this problem will remain unsolved as long as Emacs can only
use pipes for communications with subprocesses.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#46388; Package emacs. Full text available.

Message received at 46388 <at> debbugs.gnu.org:


Received: (at 46388) by debbugs.gnu.org; 9 Feb 2021 20:52:30 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Feb 09 15:52:30 2021
Received: from localhost ([127.0.0.1]:54814 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1l9Zzp-0000nV-W9
	for submit <at> debbugs.gnu.org; Tue, 09 Feb 2021 15:52:30 -0500
Received: from eggs.gnu.org ([209.51.188.92]:32828)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1l9Zzo-0000nK-Us
 for 46388 <at> debbugs.gnu.org; Tue, 09 Feb 2021 15:52:29 -0500
Received: from fencepost.gnu.org ([2001:470:142:3::e]:55376)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <eliz@HIDDEN>)
 id 1l9Zzh-0002wH-DT; Tue, 09 Feb 2021 15:52:22 -0500
Received: from 84.94.185.95.cable.012.net.il ([84.94.185.95]:3562
 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1l9Zzc-0005IT-ND; Tue, 09 Feb 2021 15:52:19 -0500
Date: Tue, 09 Feb 2021 22:52:13 +0200
Message-Id: <83h7mlj75u.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Ioannis Kappas <ioannis.kappas@HIDDEN>
In-Reply-To: <CAMRHuGC+8ZP270Erd8q=MTrw7CUS-7+PX7GTFc2JJgkexorAsw@HIDDEN>
 (message from Ioannis Kappas on Tue, 9 Feb 2021 20:15:34 +0000)
Subject: Re: bug#46388: 27.1; emacs -batch does not output messages
 immediately when invoked outside of the command prompt
References: <CAMRHuGB-z+uiS7F71vaLU2L8X-W9BAwUH0+aSDvGz7HmBYmnqQ@HIDDEN>
 <CAMRHuGDC-XAznW-0YtpnCncFkxBRywx-jDgZ2eQf8UNdebR8Og@HIDDEN>
 <E1l9LhR-0002lF-EV@HIDDEN>
 <CAMRHuGC+8ZP270Erd8q=MTrw7CUS-7+PX7GTFc2JJgkexorAsw@HIDDEN>
X-Spam-Score: -0.7 (/)
X-Debbugs-Envelope-To: 46388
Cc: 46388 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

> From: Ioannis Kappas <ioannis.kappas@HIDDEN>
> Date: Tue, 9 Feb 2021 20:15:34 +0000
> Cc: 46388 <at> debbugs.gnu.org
> 
> Which appears to suggests that stderr always starts as "unbuffered"
> when a program starts, irrespective of how it was invoked?

AFAIR, on GNU/Linux stderr is not unbuffered by default.  We had a
long discussion of related issues in June 2019, I think people who
know about the glibc internals told there stderr is not really
completely unbuffered (and Posix allows that).

On MS-Windows it is by default unbuffered.

> Back to the windows behavior at hand, the simple test conducted with
> fwrite as the bug report mentioned, has a 2048 bytes associated with
> it which I will consider it to be categorized as "fully buffered",
> which brings it further away to the posix standard and even more further
> away from Linux being on the other end (always unbuffered).

I'm not sure your conclusion is correct.  I think the buffering is
imposed by the pipes which Emacs uses for communications with
subprocesses.  Or at least this is an additional buffering that is
related to the issue at hand.

> Unix convention is that stdin and stdout are line-buffered when
> associated with a terminal, and fully-buffered (aka block-buffered)
> otherwise. stderr is always unbuffered.
> 
> """
> 
> emphasis on *Unix convention ... stderr .. always unbuffered*, though
> it will be difficult to confirm this claim for all *nices)

I think Windows behaves the same, at least by default.  But GNU/Linux
isn't.

> For #2, emacs batch stderr behavior when invoked from outside the
> command line is not compatible with the posix standard, since it uses
> "fully buffered" mode, which actually makes it behave very differently
> from Linux (which is posix compliant by always having stderr as being
> unbuffered). Thus, the suggested patch actually addresses this
> concern, rather than invalidating it.

Again, are you sure it isn't the pipe that imposes buffering?

> (I understand ConPTY is the long term solution looking for a hacker,
> though I still do
> suppose the current behavior on windows is odd and needs to be
> addressed until that time).

I don't see it as odd.  The behavior of stdout and stdin could be odd
(again, due to pipes being used), but not that of stderr.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#46388; Package emacs. Full text available.

Message received at 46388 <at> debbugs.gnu.org:


Received: (at 46388) by debbugs.gnu.org; 9 Feb 2021 20:15:58 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Feb 09 15:15:58 2021
Received: from localhost ([127.0.0.1]:54742 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1l9ZQT-0007As-7C
	for submit <at> debbugs.gnu.org; Tue, 09 Feb 2021 15:15:58 -0500
Received: from mail-ot1-f41.google.com ([209.85.210.41]:35652)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ioannis.kappas@HIDDEN>) id 1l9ZQN-00070b-IZ
 for 46388 <at> debbugs.gnu.org; Tue, 09 Feb 2021 15:15:56 -0500
Received: by mail-ot1-f41.google.com with SMTP id k10so16320600otl.2
 for <46388 <at> debbugs.gnu.org>; Tue, 09 Feb 2021 12:15:51 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=mime-version:references:in-reply-to:from:date:message-id:subject:to
 :cc; bh=GrjDOgnTSryxhZUfd1hXJZ/5xSTLqcl1eEX7IyGuf+s=;
 b=NjnEok8TlEdRRf0po7fxh+Q/rmF3wKW9OLTllZ1Ji3o6LEhqUUlJOZsjss7F3wlvjO
 ST/dKSLPE6iH683P2J6uZOtQRigkameHm3xdDyZ6IDXXt0huT8w29r0ByuK9xU+sSvY4
 0MN4SrTFjaU2oeBx4A42CgBmfrqxHCAHAouK1zvYDy4KrwhE+ecX/qxluifn/6TWSmBG
 1ma5IYe2ps3xFCFE4DVv6iLUvn35ZmvQU7pTaM4bvs6i/CBRFGnU5YZDE5QXssCV0v9e
 6SCxShLrBIToeKCflCShV2eCDaIvaK/bwtcbTqqP0oNvZugzSvvMTIrP9/akEAJz1JVZ
 EyWQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:mime-version:references:in-reply-to:from:date
 :message-id:subject:to:cc;
 bh=GrjDOgnTSryxhZUfd1hXJZ/5xSTLqcl1eEX7IyGuf+s=;
 b=ss2X2sH21v5RWG502XyL90eXZ+nxmEdXVS3m8IsFia64Z4l5PARmxVntet59rk9g/x
 6BWkIyD1+zlwtblGZsQi3dh8qASP+TuUP0A0pDLr4XdracX1jV7bi8n6xSsnKWrmJzFB
 Cg/3HWLmOdW5y5LJxevrksNZ8rjAMN+/TE+crKpiZ2IR9TtwQjlts6USqu66zMfiY8/u
 mA9EaZ3GRqH/AtEboIY+y1oetjAhvIUWm/NN0uq43TTBiRoNDk0FBAmCozFA0bltZP7/
 7NU8hiFOqJIwahttU1/OIsJfkWkUZxzqS0pkJegzde6471ZhxkF1N7ObRgE6sY33tdhQ
 mTmg==
X-Gm-Message-State: AOAM531X/CCKV/ThgE5TqIWipkWfwMf+oOxQsckPXyXbAz5bkI4eDdPJ
 zp0YekMuyV57KeANA5wWMhpAmcBGvghzxp4iJdk=
X-Google-Smtp-Source: ABdhPJypC17Znnvd5CofAw2lIpT7Fio0yQK55P3BIKonNpC+dYacsff0g1IHgmburT4Sr6xKzxgO2RqHsswlJ9plXwo=
X-Received: by 2002:a9d:3b26:: with SMTP id z35mr16731361otb.192.1612901745885; 
 Tue, 09 Feb 2021 12:15:45 -0800 (PST)
MIME-Version: 1.0
References: <CAMRHuGB-z+uiS7F71vaLU2L8X-W9BAwUH0+aSDvGz7HmBYmnqQ@HIDDEN>
 <CAMRHuGDC-XAznW-0YtpnCncFkxBRywx-jDgZ2eQf8UNdebR8Og@HIDDEN>
 <E1l9LhR-0002lF-EV@HIDDEN>
In-Reply-To: <E1l9LhR-0002lF-EV@HIDDEN>
From: Ioannis Kappas <ioannis.kappas@HIDDEN>
Date: Tue, 9 Feb 2021 20:15:34 +0000
Message-ID: <CAMRHuGC+8ZP270Erd8q=MTrw7CUS-7+PX7GTFc2JJgkexorAsw@HIDDEN>
Subject: Re: bug#46388: 27.1; emacs -batch does not output messages
 immediately when invoked outside of the command prompt
To: Eli Zaretskii <eliz@HIDDEN>
Content-Type: text/plain; charset="UTF-8"
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 46388
Cc: 46388 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Hi Eli,

thanks for taking the time to look into this bug report!

(my apologies for the long reply that follows)

I understand there are two concerns:

1. stderr could be connected to a file, in which case we do want it to
   be buffered for efficiency.
2. If stderr is forced to unbuffered, this will make it behave
   differently from posix systems.

I had a look around #2 in the posix standard. The section under
stderr, stdin, stdout at
https://pubs.opengroup.org/onlinepubs/9699919799/functions/stdin.html
reads:

"""At program start-up, three streams shall be predefined and need not
be opened explicitly: standard input (for reading conventional input),
standard output (for writing conventional output), and standard error
(for writing diagnostic output). When opened, the standard error
stream is not fully buffered; the standard input and standard output
streams are fully buffered if and only if the stream can be determined
not to refer to an interactive device.  """

Thus, at open (I assume this to mean at program invocation), the posix
spec suggests that stderr is not "fully buffered".

The section under setvbuf under the same spec at
https://pubs.opengroup.org/onlinepubs/009695399/functions/setvbuf.html
differentiates between the three different buffering regimes:


"""
 {_IOFBF} shall cause input/output to be fully buffered.

 {_IOLBF} shall cause input/output to be line buffered.

  {_IONBF} shall cause input/output to be unbuffered.
 """

So, if stderr is not "fully buffered", it can be either "line
buffered" or "unbuffered".

(This does make some sense to me, since stderr is usually for urgent
messages and thus any messages are needed to be readily available to
the recipients. Fully buffered defeats the purpose?).

How does Linux conforms to the posix standard? Having a look at
stderr(3) in the foot note, it reads:

""" Notes

The stream stderr is unbuffered. The stream stdout is line-buffered
when it points to a terminal. Partial lines will not appear until
fflush(3) or exit(3) is called, or a newline is printed. This can
produce unexpected results, especially with debugging output. The
buffering mode of the standard streams (or any other stream) can be
changed using the setbuf(3) or setvbuf(3) call. Note that in case
stdin is associated with a terminal, there may also be input buffering
in the terminal driver, entirely unrelated to stdio
buffering. (Indeed, normally terminal input is line buffered in the
kernel.) This kernel input handling can be modified using calls like
tcsetattr(3); see also stty(1), and termios(3).

"""

Which appears to suggests that stderr always starts as "unbuffered"
when a program starts, irrespective of how it was invoked?  If so,
then it conforms to the posix standard of not being "fully buffered"
at open, by always being "unbuffered" (this behavior has also been
seen in all tests mentioned so far). If this is the case, then the
desired buffering efficiency as mentioned in concern #1, does not
exist in Linux.

I have also conducted an additional test to check Linux behavior when
stderr is redirected to a file (using the same c program that I
mentioned in the original bug report):

: ./fwrite-a-character-to-stderr-and-sleep-for-two-seconds 2> io.txt

while at around the same time in another terminal I read from that file:

: cat io.txt

The result is that cat outputs "t", the character the fwrite program
has written to stderr, i.e. the io.txt file, proving that no buffering
has being utilized by the fwrite process. The result is the same when
I try the same experiment from inside an emacs shell.

Back to the windows behavior at hand, the simple test conducted with
fwrite as the bug report mentioned, has a 2048 bytes associated with
it which I will consider it to be categorized as "fully buffered",
which brings it further away to the posix standard and even more further
away from Linux being on the other end (always unbuffered).

(There is also this thread on stackoverlow quoting the following from
comp.lang.c:

"""

Unix convention is that stdin and stdout are line-buffered when
associated with a terminal, and fully-buffered (aka block-buffered)
otherwise. stderr is always unbuffered.

"""

emphasis on *Unix convention ... stderr .. always unbuffered*, though
it will be difficult to confirm this claim for all *nices)

Thus, if we assume the above interpretation given so far to be
correct, then both of
the concerns can be addressed as such:

For #1, we can use Linux as a benchmark of established
behavior. There is no efficiency in Linux with regards to buffering
when stderr is redirected to files, thus having emacs batch mode on
windows being
on par with Linux should be good enough in this respect.

For #2, emacs batch stderr behavior when invoked from outside the
command line is not compatible with the posix standard, since it uses
"fully buffered" mode, which actually makes it behave very differently
from Linux (which is posix compliant by always having stderr as being
unbuffered). Thus, the suggested patch actually addresses this
concern, rather than invalidating it.

(I understand ConPTY is the long term solution looking for a hacker,
though I still do
suppose the current behavior on windows is odd and needs to be
addressed until that time).

Sincerely





On Tue, Feb 9, 2021 at 5:36 AM Eli Zaretskii <eliz@HIDDEN> wrote:
>
> > From: Ioannis Kappas <ioannis.kappas@HIDDEN>
> > Date: Mon, 8 Feb 2021 21:42:09 +0000
> >
> >     A solution thus to correct this behavior in emacs -batch on
> >     windows-nt would be to check if it is connected to the console,
> >     and when not, set stderr mode to unbuffered.
>
> Thanks for the analysis and the patch proposal.  However, I don't
> think this is a good idea.  For starters, stderr could be connected to
> a file, in which case we do want it to be buffered.  More generally,
> stderr could be used for something other than outputting urgent
> messages, in which case making it unbuffered will make I/O less
> efficient for no good reason.  And this would make Emacs on Windows
> behave differently from Posix systems, which is also a downside.
>
> I think a much better way forward in this area is to teach Emacs to
> use the Pseudo Consoles introduced in recent Windows versions.  That
> would allow us to support subprocess communications via PTYs on
> MS-Windows, and thus will solve this and other similar issues.
>
> Patches to support PTYs on Windows are welcome.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#46388; Package emacs. Full text available.

Message received at 46388 <at> debbugs.gnu.org:


Received: (at 46388) by debbugs.gnu.org; 9 Feb 2021 05:36:42 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Feb 09 00:36:42 2021
Received: from localhost ([127.0.0.1]:52380 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1l9Lha-0005Yp-Cg
	for submit <at> debbugs.gnu.org; Tue, 09 Feb 2021 00:36:42 -0500
Received: from eggs.gnu.org ([209.51.188.92]:38118)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1l9LhY-0005Yd-0H
 for 46388 <at> debbugs.gnu.org; Tue, 09 Feb 2021 00:36:41 -0500
Received: from fencepost.gnu.org ([2001:470:142:3::e]:39049)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <eliz@HIDDEN>)
 id 1l9LhS-0001fZ-QB; Tue, 09 Feb 2021 00:36:34 -0500
Received: from eliz by fencepost.gnu.org with local (Exim 4.82)
 (envelope-from <eliz@HIDDEN>)
 id 1l9LhR-0002lF-EV; Tue, 09 Feb 2021 00:36:33 -0500
From: Eli Zaretskii <eliz@HIDDEN>
To: Ioannis Kappas <ioannis.kappas@HIDDEN>
In-Reply-To: <CAMRHuGDC-XAznW-0YtpnCncFkxBRywx-jDgZ2eQf8UNdebR8Og@HIDDEN>
 (message from Ioannis Kappas on Mon, 8 Feb 2021 21:42:09 +0000)
Subject: Re: bug#46388: 27.1;
 emacs -batch does not output messages immediately when invoked
 outside of the command prompt
References: <CAMRHuGB-z+uiS7F71vaLU2L8X-W9BAwUH0+aSDvGz7HmBYmnqQ@HIDDEN>
 <CAMRHuGDC-XAznW-0YtpnCncFkxBRywx-jDgZ2eQf8UNdebR8Og@HIDDEN>
Message-Id: <E1l9LhR-0002lF-EV@HIDDEN>
Date: Tue, 09 Feb 2021 00:36:33 -0500
X-Spam-Score: -0.7 (/)
X-Debbugs-Envelope-To: 46388
Cc: 46388 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Reply-To: Eli Zaretskii <eliz@HIDDEN>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

> From: Ioannis Kappas <ioannis.kappas@HIDDEN>
> Date: Mon, 8 Feb 2021 21:42:09 +0000
> 
>     A solution thus to correct this behavior in emacs -batch on
>     windows-nt would be to check if it is connected to the console,
>     and when not, set stderr mode to unbuffered.

Thanks for the analysis and the patch proposal.  However, I don't
think this is a good idea.  For starters, stderr could be connected to
a file, in which case we do want it to be buffered.  More generally,
stderr could be used for something other than outputting urgent
messages, in which case making it unbuffered will make I/O less
efficient for no good reason.  And this would make Emacs on Windows
behave differently from Posix systems, which is also a downside.

I think a much better way forward in this area is to teach Emacs to
use the Pseudo Consoles introduced in recent Windows versions.  That
would allow us to support subprocess communications via PTYs on
MS-Windows, and thus will solve this and other similar issues.

Patches to support PTYs on Windows are welcome.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#46388; Package emacs. Full text available.

Message received at 46388 <at> debbugs.gnu.org:


Received: (at 46388) by debbugs.gnu.org; 9 Feb 2021 03:38:58 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Feb 08 22:38:57 2021
Received: from localhost ([127.0.0.1]:52313 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1l9Jrd-0002oS-J3
	for submit <at> debbugs.gnu.org; Mon, 08 Feb 2021 22:38:57 -0500
Received: from eggs.gnu.org ([209.51.188.92]:48934)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1l9Jra-0002oD-O0
 for 46388 <at> debbugs.gnu.org; Mon, 08 Feb 2021 22:38:55 -0500
Received: from fencepost.gnu.org ([2001:470:142:3::e]:37634)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <eliz@HIDDEN>)
 id 1l9JrV-00085g-Gd; Mon, 08 Feb 2021 22:38:49 -0500
Received: from 84.94.185.95.cable.012.net.il ([84.94.185.95]:3155
 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1l9JrU-0006h3-Qk; Mon, 08 Feb 2021 22:38:49 -0500
Date: Tue, 09 Feb 2021 05:38:46 +0200
Message-Id: <835z31lxkp.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Ioannis Kappas <ioannis.kappas@HIDDEN>
In-Reply-To: <CAMRHuGB-z+uiS7F71vaLU2L8X-W9BAwUH0+aSDvGz7HmBYmnqQ@HIDDEN>
 (message from Ioannis Kappas on Mon, 8 Feb 2021 21:20:47 +0000)
Subject: Re: bug#46388: 27.1;
 emacs -batch does not output messages immediately when invoked
 outside of the command prompt
References: <CAMRHuGB-z+uiS7F71vaLU2L8X-W9BAwUH0+aSDvGz7HmBYmnqQ@HIDDEN>
X-Spam-Score: -0.7 (/)
X-Debbugs-Envelope-To: 46388
Cc: 46388 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

> From: Ioannis Kappas <ioannis.kappas@HIDDEN>
> Date: Mon, 8 Feb 2021 21:20:47 +0000
> 
>     There appears to be an issue with the /message/ function when
>     emacs is invoked in -batch mode on windows-nt from outside the
>     windows console (i.e. not directly from the command
>     prompt). Messages are not displayed to the user until emacs exits.
> 
>     Consider the following command from example, which suppose to
>     print the message "hi" first to the user and then exit after 5 seconds:
> 
>     : emacs -Q --batch --eval="(progn (message \"hi\") (sit-for 5))"
> 
>     When this command is invoked from within an emacs eshell on
>     windows, the message is only displayed to the user after about 5 seconds,
>     i.e when emacs is about to exit.

I believe this is because of buffering: Eshell on MS-Windows invokes
subprograms via pipes, which are buffered.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#46388; Package emacs. Full text available.

Message received at 46388 <at> debbugs.gnu.org:


Received: (at 46388) by debbugs.gnu.org; 8 Feb 2021 21:42:28 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Feb 08 16:42:28 2021
Received: from localhost ([127.0.0.1]:52064 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1l9EIe-0007H3-CA
	for submit <at> debbugs.gnu.org; Mon, 08 Feb 2021 16:42:28 -0500
Received: from mail-oo1-f48.google.com ([209.85.161.48]:39633)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ioannis.kappas@HIDDEN>) id 1l9EIb-0007Gq-Hn
 for 46388 <at> debbugs.gnu.org; Mon, 08 Feb 2021 16:42:27 -0500
Received: by mail-oo1-f48.google.com with SMTP id z36so3800938ooi.6
 for <46388 <at> debbugs.gnu.org>; Mon, 08 Feb 2021 13:42:25 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=mime-version:from:date:message-id:subject:to;
 bh=yHf2sAFP2LMWhXB8Ra91Ac4AHQI+YetmhHNFrZHr9SM=;
 b=hlMcI9ajTf58upgoS5DPfL8+QDN/eQdDGo1IEi/c0ANaZ0128dpBtkXCR8HCF1o3dX
 Z7epxF/O0jFrDhXE54R5MnxPCMbRNMxTIMbkOn5Rny0HJ7DRRTE9YV090TuZ6MJ9tOSo
 13z6tephzPQm6QHg9ot+GbfYT4hHaBN3hsuVofDT4emyK5yaIS5t479eZRwnuXha8CZw
 LHgEs+B3DLGoX0jB9LdJ+ZLm1eSvT4CdIQmRZ7xxW+sUsJfwA67zPed93tjAwyK7rTYI
 fmRi+UyaouRqf2tyOmjKlsSU0tG0PfAn+LFJnuHP5jQXj9TuungwCoU/lEA0RctuyhI1
 8q1w==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:mime-version:from:date:message-id:subject:to;
 bh=yHf2sAFP2LMWhXB8Ra91Ac4AHQI+YetmhHNFrZHr9SM=;
 b=EjV8p5R56TCUnAJ5XMQppfJsCrX3gNgSGA52Fk7+PEdRdeboxjLyMntjWPiTvf95+g
 BLJcvwdfADu6A/kR4xk1PnErzzIU2J5eWVSNNeI8zkoA0n9Y6up08Aoa+0zTO1yR4+rM
 YCdqbLjOZIxXzyWWj8jdKqkdV056YI3bKoPVEn94Ndj7cb3RVdmlJbvDUObM05WGw5AD
 3osaQT3ZQukmIFGSXaVTHSrKddWCXpaOv30PxGE2EEabgX0HTJu3ZOSWk5U0vFTAt3q2
 0/dwmGsEizFug8DVKaHa3NQlQe3k7G5chkSkBMMcNVP8DdnKJlyb6l9L0tkTx8b5CVIU
 rdOQ==
X-Gm-Message-State: AOAM5314V813ibAZ2nqxr7q2M5ay4pd2Y0hGT6CjSgnlk9mevZsI3Njh
 yNm3920st8zOkiUabx49yJkg4OP02YJTQK1MJ+0dnagCe+z+0A==
X-Google-Smtp-Source: ABdhPJwcsgORFcGnfFeXIrxZKN+PghuafeJlxdK1P2Gf+sHPRnau50nyAnjZnO7TaTlqbJVJ8cFSKHCJzDfzpAa8QV0=
X-Received: by 2002:a4a:5787:: with SMTP id u129mr5281631ooa.81.1612820539761; 
 Mon, 08 Feb 2021 13:42:19 -0800 (PST)
MIME-Version: 1.0
From: Ioannis Kappas <ioannis.kappas@HIDDEN>
Date: Mon, 8 Feb 2021 21:42:09 +0000
Message-ID: <CAMRHuGDC-XAznW-0YtpnCncFkxBRywx-jDgZ2eQf8UNdebR8Og@HIDDEN>
Subject: 27.1; emacs -batch does not output messages immediately when invoked
 outside of the command prompt
To: 46388 <at> debbugs.gnu.org
Content-Type: multipart/mixed; boundary="0000000000005c30c605bada0b06"
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 46388
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

--0000000000005c30c605bada0b06
Content-Type: text/plain; charset="UTF-8"

Some analysis follows

    Looking at the code, it appears that /message/s in -batch mode are
    delivered to the process' standard error by the
    [editfnc.c:message()]
      ->[xdisp.c:message3_nolog()]
        ->[xdispl.c:message_to_stderr()]
    function, i.e. messages are written to stderr. The latter function
    uses the standard fwrite and fputc calls to deliver the message to
    stderr.

    The buffering regime used for stderror on windows appears to be different
    depending whether a program was started from the command prompt or
    not. When started from the command prompt stderr is
    unbuffered, while there is a buffer employed otherwise and output
    is only delivered when that buffer is full.

    Below is a sample C program that can demonstrate the behaviour,
    which writes a single character to the standard output and then
exits after 2 seconds:

      #include <stdio.h>
      #include <unistd.h>
      #include <windows.h>

      int main()
      {

fwrite("t", 1, sizeof("t"), stderr);

sleep(2);

return 0;
      }


    | case | invoked from           | result
         |
    |------+------------------------+-------------------------------------------|
    |  1   | command prompt         | prints "t", then after 2 seconds exits |
    |  2   | outside command prompt | after about 2 seconds prints
"t", then exits |

    It appears that the buffer size used in #2 is 2048 bytes (i.e. the
    message is only immediately displayed when is more than 2048 bytes long).

    A solution thus to correct this behavior in emacs -batch on
    windows-nt would be to check if it is connected to the console,
    and when not, set stderr mode to unbuffered.

    A) Likely the win32 api provide GetConsoleMode() that can be used to
    check if a standard HANDLE is attached to the console. Given the
    STD_ERROR_HANDLE as an argument, it will  return non-zero if is
    attached to the console or a 0 otherwise, setting GetLastError()
to "The handle is
    invalid" message.

    B) To set stderr to an unbuffered mode, we can use standard C's
    setvbuf() with _IONBF (no buffering).

    C) The location where the descriptors are prepared for use in emacs
    appear to be at w32.c:init_ntproc().

    If the above analysis is correct, here is an example patch that
puts A/B/C together:

--- a/src/w32.c
+++ b/src/w32.c
@@ -10216,6 +10216,19 @@ init_ntproc (int dumping)
     else
       _open ("nul", O_TEXT | O_NOINHERIT | O_WRONLY);
     _fdopen (2, "w");
+
+    /* When in -batch mode, windows appears to buffer stderr when it
+       is invoked outside of the command prompt. This has the
+       undesired side effect that /message/s are not output unless the
+       buffer is full or emacs exits */
+    DWORD console_mode;
+    if (noninteractive &&
+ // stderr is not attached to the console
+ !GetConsoleMode (GetStdHandle(STD_ERROR_HANDLE), &console_mode))
+      {
+ // set stderr to unbuffered
+ setvbuf (stderr, NULL, _IONBF, 0);
+      }

  I have also attached an ert tests to capture the correct
  behavior. The test invokes an emacs -batch process which prints a
  /message/ and exits after five seconds. The test check immediately
  after two seconds if the batch process has output the message as
  expected. It can be invoked with:

: emacs.exe -batch -l ert -l batch-message-test.el -f
ert-run-tests-batch-and-exit

--0000000000005c30c605bada0b06
Content-Type: application/octet-stream; name="batch-message-test.el"
Content-Disposition: attachment; filename="batch-message-test.el"
Content-Transfer-Encoding: base64
Content-ID: <f_kkx3l92a0>
X-Attachment-Id: f_kkx3l92a0

Ozs7IC0qLSBsZXhpY2FsLWJpbmRpbmc6IHQ7IC0qLQ0KKHJlcXVpcmUgJ2VydCkNCg0KKGVydC1k
ZWZ0ZXN0IGJhdGNoLW1lc3NhZ2UgKCkNCiAgIlRlc3QgdGhhdCB3aGVuIGludm9raW5nIGVtYWNz
IGluIGJhdGNoIG1vZGUgYXMgYQ0Kc3VicHJvY2VzcyAoaS5lLiBub3QgZGlyZWN0bHkgZnJvbSBh
IHRlcm1pbmFsKSwgL21lc3NhZ2UvcyBhcmUNCmZsdXNoZWQgdG8gb3V0cHV0IGltbWVkaWF0ZWx5
LiAiDQogIChtZXNzYWdlICI6c3lzdGVtICVzIDp2ZXJzaW9uICVzIiBzeXN0ZW0tY29uZmlndXJh
dGlvbiBlbWFjcy12ZXJzaW9uKQ0KDQogICh3aXRoLXRlbXAtYnVmZmVyIA0KICAgIChsZXQqICgo
cHJvYy1idWYgKGN1cnJlbnQtYnVmZmVyKSkNCg0KCSAgIDs7IHN0YXJ0IGEgbmV3IGVtYWNzIHBy
b2Nlc3MgdGhhdCB3aWxsIHByaW50IGEgbWVzc2FnZSwNCgkgICA7OyB3YWl0IGZvciBmaXZlIHNl
Y29uZCBhbmQgdGhlbiBleGl0Lg0KCSAgIChjbWQgKGZvcm1hdCAiZW1hY3MgLVEgLS1iYXRjaCAt
LWV2YWw9XCIlc1wiIg0KCQkJJyhwcm9nbiAobWVzc2FnZSBcXFwiaGlcXFwiKSANCgkJCQkoc2l0
LWZvciA1KSkpKQ0KCSAgIChwcm9jIChzdGFydC1maWxlLXByb2Nlc3Mtc2hlbGwtY29tbWFuZA0K
ICAgICAgICAgICAgICAgICAgInRlc3QvYmF0Y2gtbWVzc2FnZSIgcHJvYy1idWYgY21kKSkNCg0K
CSAgIChvdXRwdXRzICcoKSkpDQogICAgICANCiAgICAgIDs7IGNhcHR1cmUgZW1hY3Mgb3V0cHV0
DQogICAgICAoc2V0LXByb2Nlc3MtZmlsdGVyIHByb2MgKGxhbWJkYSAocHJvYyBvdXRwdXQpDQoJ
CQkJIChwdXNoIG91dHB1dCBvdXRwdXRzKSkpDQogICAgICANCiAgICAgIDs7IHdhaXQgZm9yIHRo
ZSBwcm9jZXNzIHRvIHN0YXJ0DQogICAgICAoc2xlZXAtZm9yIDIpDQogICAgICAoc2hvdWxkIChl
cXVhbCAncnVuIChwcm9jZXNzLXN0YXR1cyBwcm9jKSkpDQogICAgICA7OyBwcm9ncmFtIHNob3Vs
ZCBoYXZlIHByaW50ZWQgb3V0ICJoaVxuIg0KICAgICAgKHNob3VsZCAoZXF1YWwgJygiaGlcbiIp
IG91dHB1dHMpKQ0KDQogICAgICA7OyBraWxsIHByb2Nlc3MgYW5kIHdhaXQgZm9yIGl0IHRvIGRp
ZQ0KICAgICAgKGRlbGV0ZS1wcm9jZXNzIHByb2MpDQogICAgICAoc2xlZXAtZm9yIDEpKSkpDQoN
Cg==
--0000000000005c30c605bada0b06--




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#46388; Package emacs. Full text available.

Message received at submit <at> debbugs.gnu.org:


Received: (at submit) by debbugs.gnu.org; 8 Feb 2021 21:21:12 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Feb 08 16:21:12 2021
Received: from localhost ([127.0.0.1]:52037 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1l9Dy3-0004f3-HT
	for submit <at> debbugs.gnu.org; Mon, 08 Feb 2021 16:21:11 -0500
Received: from lists.gnu.org ([209.51.188.17]:33764)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ioannis.kappas@HIDDEN>) id 1l9Dxx-0004es-Rq
 for submit <at> debbugs.gnu.org; Mon, 08 Feb 2021 16:21:09 -0500
Received: from eggs.gnu.org ([2001:470:142:3::10]:35244)
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ioannis.kappas@HIDDEN>)
 id 1l9Dxw-0001Ul-FR
 for bug-gnu-emacs@HIDDEN; Mon, 08 Feb 2021 16:21:04 -0500
Received: from mail-ot1-x334.google.com ([2607:f8b0:4864:20::334]:33840)
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128)
 (Exim 4.90_1) (envelope-from <ioannis.kappas@HIDDEN>)
 id 1l9Dxt-0004kG-Hz
 for bug-gnu-emacs@HIDDEN; Mon, 08 Feb 2021 16:21:04 -0500
Received: by mail-ot1-x334.google.com with SMTP id y11so15588147otq.1
 for <bug-gnu-emacs@HIDDEN>; Mon, 08 Feb 2021 13:20:57 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=mime-version:from:date:message-id:subject:to;
 bh=1nJFGSGOvcoMKoIwe+VBpx4DDYI4g7XQetIaBigsn5M=;
 b=bpKP9I22XQyvIAdkEoBI2CmDeG3XPDb30iscC/DDucBha41UJqQqgESkXaKXTsAcOo
 ScvHZsfyCN3wnTAFw5+o+MlvmAKXBN/QYpt0ubZVaxDM0qJIM6VlYZUeaxg/FF1245J0
 VIDAm3tTlt8ZsPOk4IhqTvixhWYLIGdmT8fzfIVrPL7sGEW8Idi7MkqZDv3Sgkgp5xlD
 YSwgr+VZJ5jozE7chdnaqICKbE1jy5ihRQXjARaytZldhxVHeoUNefu25Jnr9yWxxUyA
 twNeNxrzUJZKznpERBpDjBVst+OgCTYC+cW2XVOP43spC5hHmKEyczsKJFV8gluGJ4HB
 Lt4w==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:mime-version:from:date:message-id:subject:to;
 bh=1nJFGSGOvcoMKoIwe+VBpx4DDYI4g7XQetIaBigsn5M=;
 b=XTkSmNChrtH7PngfgGjxGMwoiy+wwtmTpyUmdvRta6bYXjzsGgGdpNSolBCezP1wDB
 Gdvy9ZdkV+Az/F0J9HeZfl7dY0XhNzZrRrCo0k73gt0Gn1XAOCHI3Pbt/4kUEgV0DIpA
 Ybwrl7I8DAuC7ospiae4Q9+MIDFLRMTteJuxc4n6jfP3gy64VAg2EMH6gtGqgdpBZ3j7
 6IenJD/GEFta9UqpacZfP1wNM+5GPoGFp60iVWJY1Qjhqv++RhJF7oW7DoeSXvM4hEUB
 9nIxlySuMFq0A7Jhdc70obMMdYV8tcdzCfNSFSBHsnxm8Fxln7llBu16nzG1pt/hkMkz
 13/g==
X-Gm-Message-State: AOAM531N66xz0wvKiCah/ODrjHlaQ3lQNavCi/VsTerbLDkqQW8GEitw
 M4/1pNXK5xtq0P9XjPMvzDyC7q2wgINcPkyg5eanjshYp6akJQ==
X-Google-Smtp-Source: ABdhPJwcrSCRhFvb6x4FjjxXLyAxHx3ZTaUnK/ueqRUenKq7KOnqh7TTPHe7+0emewB3/AAAkzZDunb0PAvAqaVHP8Y=
X-Received: by 2002:a9d:3b26:: with SMTP id z35mr13613555otb.192.1612819256991; 
 Mon, 08 Feb 2021 13:20:56 -0800 (PST)
MIME-Version: 1.0
From: Ioannis Kappas <ioannis.kappas@HIDDEN>
Date: Mon, 8 Feb 2021 21:20:47 +0000
Message-ID: <CAMRHuGB-z+uiS7F71vaLU2L8X-W9BAwUH0+aSDvGz7HmBYmnqQ@HIDDEN>
Subject: 27.1; emacs -batch does not output messages immediately when invoked
 outside of the command prompt
To: bug-gnu-emacs@HIDDEN
Content-Type: text/plain; charset="UTF-8"
Received-SPF: pass client-ip=2607:f8b0:4864:20::334;
 envelope-from=ioannis.kappas@HIDDEN; helo=mail-ot1-x334.google.com
X-Spam_score_int: -20
X-Spam_score: -2.1
X-Spam_bar: --
X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,
 DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001,
 RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001,
 SPF_PASS=-0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-Spam-Score: -1.3 (-)
X-Debbugs-Envelope-To: submit
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -2.3 (--)

    There appears to be an issue with the /message/ function when
    emacs is invoked in -batch mode on windows-nt from outside the
    windows console (i.e. not directly from the command
    prompt). Messages are not displayed to the user until emacs exits.

    Consider the following command from example, which suppose to
    print the message "hi" first to the user and then exit after 5 seconds:

    : emacs -Q --batch --eval="(progn (message \"hi\") (sit-for 5))"

    When this command is invoked from within an emacs eshell on
    windows, the message is only displayed to the user after about 5 seconds,
    i.e when emacs is about to exit.

    Here is a small summary running the above under Linux and windows
    native:

    | system              | invoked from   | terminal output
                 |
    |---------------------+----------------+--------------------------------------------|
    | x86_64-pc-linux-gnu | terminal       | prints "hi", then after 5
seconds it exits |
    | x86_64-pc-linux-gnu | emacs eshell   | prints "hi", then after 5
seconds it exits |
    | x86_64-w64-mingw32  | command prompt | prints "hi", then after 5
seconds it exits |
    | x86_64-w64-mingw32  | emacs eshell   | after 5 seconds prints
"hi", then exits    |

    The issue on windows is not exclusive to invoking emacs -batch
    from within emacs (e.g. from eshell), it applies to any invocation
that happens
    outside the command prompt, e.g. from inside the MSYS2 mintty
    terminal.

    It appears as if the issue is caused by a different stderr
    buffering mode used when a program is invoked outside of
    the command prompt. Analysis to follow.

In GNU Emacs 27.1 (build 1, x86_64-w64-mingw32)
 of 2020-11-19 built on fv-az68-340
Repository revision: ec297125a76481c55390d0b329e541907879d6f3
Repository branch: master
Windowing system distributor 'Microsoft Corp.', version 10




Acknowledgement sent to Ioannis Kappas <ioannis.kappas@HIDDEN>:
New bug report received and forwarded. Copy sent to bug-gnu-emacs@HIDDEN. Full text available.
Report forwarded to bug-gnu-emacs@HIDDEN:
bug#46388; Package emacs. Full text available.
Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.
Last modified: Wed, 10 Feb 2021 16:00:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.