GNU bug report logs - #20555
Emacs 24.2 vs 24.4 on Solaris: M-x shell and "tty => not a tty"

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 20555 in the body.
You can then email your comments to 20555 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Georges Ko <gko <at> gko.net>, 
 Emacs bug reports and feature requests <bug-gnu-emacs <at> gnu.org>
Subject: Re: Emacs 24.2 vs 24.4 on Solaris: M-x shell and "tty => not a tty"
Date: Mon, 11 May 2015 23:08:34 -0700

[moving this to bug-gnu-emacs, so that it gets a proper bug number]

Georges Ko wrote:
> What were the changes done on Emacs 24.4 regarding subshells (or
> processes), compared to version 24.2?

Quite a few, I suspect.  You can use 'diff' to find out the details....

> When compiled in the same environment, "tty" in a subshell
> (M-x shell) has different outputs:
>
>    - in Emacs 24.2, "tty" outputs the tty,
>    - in Emacs 24.4, "tty" outputs "not a tty".
>
> That is, if I run the command below to spawn a shell:
>
>    (apply 'start-process "shell" "*shell*" "/bin/ksh" '("-i"))
>
>    - in Emacs 24.2, "ps" shows the shell has a normal tty,
>    - in Emacs 24.4, "ps" shows the shell has a tty with "?".

I don't observe either problem when running Emacs 24.5 under Solaris 11 x86-64 
compiled with the bundled GCC (4.5.2).  However, I do observe them when running 
Emacs 24.5 under Solaris 10 sparc compiled with Sun C 5.13.

> I couldn't compile 24.3 due to some errors, so I don't know its behavior.

Emacs 24.3 is bundled within Solaris 11.2, for what it's worth.

If you're still running into problems, I suggest replying with details of how 
you built Emacs (version of OS and compiler, 'configure' arguments, log from 
'configure' and 'make', etc.).

PS.  This looks a bit like Bug#9150 <http://bugs.gnu.org/9150>, but that bug was 
in an older Emacs, which probably means it's independent.

Message #8 received at 20555 <at> debbugs.gnu.org (full text, mbox):

From: Glenn Morris <rgm <at> gnu.org>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: 20555 <at> debbugs.gnu.org, gko <at> gko.net
Subject: Re: bug#20555: Emacs 24.2 vs 24.4 on Solaris: M-x shell and "tty =>
 not a tty"
Date: Tue, 12 May 2015 02:36:01 -0400

Paul Eggert wrote:

> [moving this to bug-gnu-emacs, so that it gets a proper bug number]

It already has one: http://debbugs.gnu.org/19191

Message #13 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Georges Ko <gko <at> gko.net>
To: bug-gnu-emacs <at> gnu.org
Cc: okshirai <at> gmail.com
Subject: Re: bug#20555: Emacs 24.2 vs 24.4 on Solaris: M-x shell and "tty =>
 not a tty"
Date: Wed, 13 May 2015 15:12:32 +0800

Glenn Morris <rgm <at> gnu.org> writes:

> Paul Eggert wrote:
>
>> [moving this to bug-gnu-emacs, so that it gets a proper bug number]
>
> It already has one: http://debbugs.gnu.org/19191

I have been able to "fix" the problem by trial and error.

On my system, the function allocate_pty (process.c) calls the macro 
PTY_TTY_NAME_SPRINTF, generated in config.h, which is: 

  #define PTY_TTY_NAME_SPRINTF
  {
    char *ptsname (int), *ptyname;
    int grantpt_result;
    sigset_t blocked;
    sigemptyset (&blocked);
    sigaddset (&blocked, SIGCHLD);
    pthread_sigmask (SIG_BLOCK, &blocked, 0);
    grantpt_result = grantpt (fd);
    pthread_sigmask (SIG_UNBLOCK, &blocked, 0);
    if (grantpt_result == -1 || unlockpt (fd) == -1 || !(ptyname = ptsname (fd)))
      {
        emacs_close (fd);
        return -1;
      }
    snprintf (pty_name, PTY_NAME_SIZE, "%s", ptyname);
  } 

It turned out that grantpt(fd) returns -1, with errno = 13 (EACCES), 
so -1 is returned by allocate_pty.

After checking EACCES of grantpt():

  The slave pseudoterminal was opened before grantpt(), or a grantpt() was 
  already issued. In either case, slave pseudoterminal permissions and 
  ownership have already been updated. If you use grantpt() to change slave 
  pseudoterminal permissions, you must issue grantpt() between the master 
  open and the first pseudoterminal open, and grantpt() can only be issued 
  once. 
  
I changed the macro to:

    if ( /* grantpt_result == -1 || */ unlockpt (fd) == -1 || !(ptyname = ptsname (fd)))

and it "fixed" the problem. I'll let the experts figure out the root cause...

The diff of configure (24.5):

$ diff configure.24.5 configure.24.5.ttyfix
17105c17105
<                 $as_echo "#define PTY_TTY_NAME_SPRINTF { char *ptsname (int), *ptyname; int grantpt_result; sigset_t blocked; sigemptyset (&blocked); sigaddset (&blocked, SIGCHLD); pthread_sigmask (SIG_BLOCK, &blocked, 0); grantpt_result = grantpt (fd); pthread_sigmask (SIG_UNBLOCK, &blocked, 0); if (grantpt_result == -1 || unlockpt (fd) == -1 || !(ptyname = ptsname (fd))) { emacs_close (fd); return -1; } snprintf (pty_name, PTY_NAME_SIZE, \"%s\", ptyname); }" >>confdefs.h
---
>                 $as_echo "#define PTY_TTY_NAME_SPRINTF { char *ptsname (int), *ptyname; int grantpt_result; sigset_t blocked; sigemptyset (&blocked); sigaddset (&blocked, SIGCHLD); pthread_sigmask (SIG_BLOCK, &blocked, 0); grantpt_result = grantpt (fd); pthread_sigmask (SIG_UNBLOCK, &blocked, 0); if ( /* grantpt_result == -1 || */ unlockpt (fd) == -1 || !(ptyname = ptsname (fd))) { emacs_close (fd); return -1; } snprintf (pty_name, PTY_NAME_SIZE, \"%s\", ptyname); }" >>confdefs.h

Georges
-- 
 Georges Ko                     gko <at> gko.net                      2015-05-13

Message #20 received at 20555 <at> debbugs.gnu.org (full text, mbox):

From: 白井彰 <okshirai <at> gmail.com>
To: 20555 <at> debbugs.gnu.org
Cc: Akira Shirai <okshirai <at> gmail.com>
Subject: Re: bug#20555: Emacs 24.2 vs 24.4 on Solaris: M-x shell and "tty =>
 not a tty"
Date: Tue, 2 Jun 2015 01:53:12 +0900

I wrote a small program my-emacs-24.5.c, and found that grantpt(fd)
returned -1 after fcntl(fd, F_SETFD, FD_CLOEXEC).
Without fcntl(fd, F_SETFD, FD_CLOEXEC), grantpt(fd) returns 0.

aaa% sh emacs-24.5/build-aux/config.guess
i386-pc-solaris2.10

aaa% ls my-emacs-24.5.c
my-emacs-24.5.c

aaa% make my-emacs-24.5 && ./my-emacs-24.5
cc    -o my-emacs-24.5 my-emacs-24.5.c
call_fcntl_fd_cloexec: 1
fcntl_result: 0
fd: 3
grantpt_result: -1
errno: 13
grantpt: Permission denied

aaa% make my-emacs-24.5 && ./my-emacs-24.5 call_fcntl_fd_cloexec=0
`my-emacs-24.5' is up to date.
call_fcntl_fd_cloexec: 0
fd: 3
grantpt_result: 0

bbb% sh emacs-24.5/build-aux/config.guess
sparc-sun-solaris2.8

bbb% rm my-emacs-24.5
bbb% make my-emacs-24.5 && ./my-emacs-24.5
cc    -o my-emacs-24.5 my-emacs-24.5.c
call_fcntl_fd_cloexec: 1
fcntl_result: 0
fd: 3
grantpt_result: -1
errno: 0
grantpt: Error 0

bbb% make my-emacs-24.5 && ./my-emacs-24.5 call_fcntl_fd_cloexec=0
`my-emacs-24.5' is up to date.
call_fcntl_fd_cloexec: 0
fd: 3
grantpt_result: 0

% cat -n my-emacs-24.5.c
     1  /* 24.5 */
     2
     3  #include <errno.h>
     4  #include <fcntl.h>
     5  #include <stdio.h>
     6  #include <stdlib.h>
     7  #include <string.h>
     8  #include <sys/stat.h>
     9  #include <sys/types.h>
    10  #include <unistd.h>
    11
    12  int call_fcntl_fd_cloexec;
    13
    14  enum { PTY_NAME_SIZE = 24 };
    15
    16  int emacs_open(const char *file, int oflags, int mode) {
    17  #define O_CLOEXEC 0
    18    int fd;
    19    oflags |= O_CLOEXEC;
    20    while ((fd = open(file, oflags, mode)) < 0 && errno == EINTR) abort();
    21    if (! O_CLOEXEC && 0 <= fd) {
    22      fprintf(stderr, "call_fcntl_fd_cloexec: %d\n", call_fcntl_fd_cloexec);
    23      if (call_fcntl_fd_cloexec) {
    24        int fcntl_result = fcntl(fd, F_SETFD, FD_CLOEXEC);
    25        fprintf(stderr, "fcntl_result: %d\n", fcntl_result);
    26      }
    27    }
    28    return fd;
    29  }
    30
    31  static int allocate_pty(char pty_name[PTY_NAME_SIZE]) {
    32    int fd;
    33    strcpy(pty_name, "/dev/ptmx");
    34    fd = emacs_open(pty_name, O_RDWR | O_NONBLOCK, 0);
    35    fprintf(stderr, "fd: %d\n", fd);
    36    if (fd >= 0) {
    37      int grantpt_result = grantpt(fd);
    38      fprintf(stderr, "grantpt_result: %d\n", grantpt_result);
    39      if (grantpt_result != 0) {
    40        fprintf(stderr, "errno: %d\n", errno);
    41        perror("grantpt");
    42      }
    43    }
    44    return fd;
    45  }
    46
    47  int main(int argc, char **argv) {
    48    char pty_name[PTY_NAME_SIZE];
    49    if (argc == 2) call_fcntl_fd_cloexec = 0;
    50    else           call_fcntl_fd_cloexec = 1;
    51    allocate_pty(pty_name);
    52    return 0;
    53  }

Message #23 received at 20555 <at> debbugs.gnu.org (full text, mbox):

From: 白井彰 <okshirai <at> gmail.com>
To: 20555 <at> debbugs.gnu.org
Cc: Akira Shirai <okshirai <at> gmail.com>
Subject: Re: bug#20555: Emacs 24.2 vs 24.4 on Solaris: M-x shell and "tty =>
 not a tty"
Date: Wed, 3 Jun 2015 01:45:21 +0900

Today I found the same problem on Ruby:

| grantpt() doesn't work with CLOEXEC on Solaris 10
| http://svn.ruby-lang.org/cgi-bin/viewvc.cgi?revision=33519&view=revision

They say that Solaris grantpt() internally uses a setuid root program
/usr/lib/pt_chmod which receives an fd and calls ptsname(), chown(),
and chmod().  If FD_CLOEXEC is set for the fd, the fd passed for
/usr/lib/pt_chmod is already closed and /usr/lib/pt_chmod does not
work.

The following is their fix for the problem:

% gzcat ruby-2.2.2.tar.gz | tar tvf - | grep ruby-2.2.2/ext/pty/pty.c
-rw-r--r-- 1044/1044  22152 Nov 15 20:49 2014 ruby-2.2.2/ext/pty/pty.c

% cat -n ruby-2.2.2/ext/pty/pty.c | sed -n '221,223p;234,256p;330,341p;450p'
   221  static int
   222  get_device_once(int *master, int *slave, char SlaveName[DEVICELEN], int nomesg, int fail)
   223  {
   234  #if defined(__sun) || (defined(__FreeBSD__) && __FreeBSD_version < 902000)
   235      /* workaround for Solaris 10: grantpt() doesn't work if FD_CLOEXEC is set.  [ruby-dev:44688] */
   236      /* FreeBSD 9.2 or later supports O_CLOEXEC
   237       * http://www.freebsd.org/cgi/query-pr.cgi?pr=162374 */
   238      if ((masterfd = posix_openpt(O_RDWR|O_NOCTTY)) == -1) goto error;
   239      if (sigaction(SIGCHLD, &dfl, &old) == -1) goto error;
   240      if (grantpt(masterfd) == -1) goto grantpt_error;
   241      rb_fd_fix_cloexec(masterfd);
   242  #else
   243      {
   244          int flags = O_RDWR|O_NOCTTY;
   245  # if defined(O_CLOEXEC)
   246          /* glibc posix_openpt() in GNU/Linux calls open("/dev/ptmx", flags) internally.
   247           * So version dependency on GNU/Linux is same as O_CLOEXEC with open().
   248           * O_CLOEXEC is available since Linux 2.6.23.  Linux 2.6.18 silently ignore it. */
   249          flags |= O_CLOEXEC;
   250  # endif
   251          if ((masterfd = posix_openpt(flags)) == -1) goto error;
   252      }
   253      rb_fd_fix_cloexec(masterfd);
   254      if (sigaction(SIGCHLD, &dfl, &old) == -1) goto error;
   255      if (grantpt(masterfd) == -1) goto grantpt_error;
   256  #endif
   330  #if defined(__sun)
   331      /* workaround for Solaris 10: grantpt() doesn't work if FD_CLOEXEC is set.  [ruby-dev:44688] */
   332      if((masterfd = open("/dev/ptmx", O_RDWR, 0)) == -1) goto error;
   333      s = signal(SIGCHLD, SIG_DFL);
   334      if(grantpt(masterfd) == -1) goto error;
   335      rb_fd_fix_cloexec(masterfd);
   336  #else
   337      if((masterfd = rb_cloexec_open("/dev/ptmx", O_RDWR, 0)) == -1) goto error;
   338      rb_update_max_fd(masterfd);
   339      s = signal(SIGCHLD, SIG_DFL);
   340      if(grantpt(masterfd) == -1) goto error;
   341  #endif
   450  }

Message #26 received at 20555 <at> debbugs.gnu.org (full text, mbox):

From: Glenn Morris <rgm <at> gnu.org>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: okshirai <at> gmail.com, 20555 <at> debbugs.gnu.org
Subject: Re: bug#20555: Emacs 24.2 vs 24.4 on Solaris: M-x shell and "tty =>
 not a tty"
Date: Sun, 07 Jun 2015 14:11:46 -0400

Paul, you are the go-to-guy for these kind of problems.
Any thoughts on

http://debbugs.gnu.org/20555#13
http://debbugs.gnu.org/20555#20
http://debbugs.gnu.org/20555#23


?

Message #31 received at 20555-done <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Akira Shirai <okshirai <at> gmail.com>
Cc: "Warren, John W CIV \(US\)" <john.w.warren.civ <at> mail.mil>,
 20555-done <at> debbugs.gnu.org, Georges Ko <gko <at> gko.net>
Subject: Re: Emacs 24.2 vs 24.4 on Solaris: M-x shell and "tty => not a tty"
Date: Thu, 11 Jun 2015 16:48:10 -0700

[Message part 1 (text/plain, inline)]

Akira Shirai, thanks for the info about grantpt and its internal setuid program; 
that made the problem obvious.  I installed the attached patch into the Emacs 
master sources and am (perhaps optimistically) marking this bug as done.

[0001-Fix-not-a-tty-bug-on-Solaris-10.patch (text/x-diff, attachment)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.