GNU bug report logs - #39164
27.0.60; Intermittent crash on MacOS 10.14 in setup_process_coding_systems

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 39164 in the body.
You can then email your comments to 39164 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Justin Guenther <jguenther <at> gmail.com>
To: bug-gnu-emacs <at> gnu.org
Subject: 27.0.60;
 Intermittent crash on MacOS 10.14 in setup_process_coding_systems
Date: Fri, 17 Jan 2020 13:00:47 -0600

[Message part 1 (text/plain, inline)]

I've been seeing an intermittent crash in setup_process_coding_systems
(crash traces are attached). It seems to most frequently happen from
create_process -> setup_process_coding_systems, but that may just be
due to my frequent usage of `helm-do-ag-project-root'.

For the attached crash_2020-01-15_1_create_process.log, it happened
when using `helm-do-ag-project-root', which takes input from the
minibuffer and runs an external `ag` (the_silver_searcher) process to
search the current project (in this case, a git repo) and
display/refresh the results as I type in the minibuffer.

For the other crash logs, unfortunately I didn't record what I was
doing at the time. But IIRC most of them were from
`helm-do-ag-project-root`.

I have also seen this crash happen in other packages: `magit`, when
opening a magit-status buffer (which runs various git commands, e.g.
`git status`); and when starting up an external language server
process from `lsp-go` (although I've most frequently seen this from
helm).

I've also had at least one crash in `setup_process_coding_systems'
that was called by `connect_network_socket' (see
crash_2020-01-06_2.log attached). I believe this crash happened when I
opened a `go-mode' buffer and `lsp-mode' was starting up an external
language server process, but I'm not 100% sure of that.

Unfortunately it's a similar situation to bug#38822--I started seeing
this at some point after recompiling emacs from source, somewhere
approx. between cbc10ec71e9f189e8d6fd5c6927aec4872e0fd96 and
0bc00cda3cb6e765b0e1f0105edf75425bce13c9 (which is a long range of
commits unfortunately).

I've also recompiled from source from the emacs-27 branch a few times
since then, but it doesn't seem to have affected how often I get this
crash (my most recent build was @
16eaaa07e6ba6a41ccdc562dcb52fc87c38f8083, with a small patch applied
from https://debbugs.gnu.org/cgi/bugreport.cgi?bug=38748#92 to try
fixing a seemingly-unrelated crash to this one--see bug#38822).

I haven't managed to reproduce this from 'emacs -Q' yet, and also
haven't caught it in a debugger unfortunately.

The crash seems to happen somewhat reliably when I open a couple file
buffers from one project, then open a new frame -> another buffer in
another large project that has lots of files (which happens to also
use `go-mode' and `lsp-mode'), and try running
`helm-do-ag-project-root' while point is inside a symbol (which causes
helm to pass a default to the minibuffer). I usually get a crash at
this point.

Sorry, I know that's not a lot of help (and unfortunately I can't
share the contents of either of those projects that's triggering
this). I'll try to get better reproduction steps and/or get a debugger
backtrace if possible.

---

In GNU Emacs 27.0.60 (build 1, x86_64-apple-darwin18.7.0, NS
appkit-1671.60 Version 10.14.6 (Build 18G2022))
 of 2020-01-10 built on jguenther-mbp
Repository revision: ff8996a337377731905a7a3c9bf05e2281df40ef [NB:
this is 16eaaa07e6ba6a41ccdc562dcb52fc87c38f8083 with the above patch
applied]
Repository branch: emacs-27
Windowing system distributor 'Apple', version 10.3.1671
System Description:  Mac OS X 10.14.6

Configured using:
 'configure --disable-silent-rules
 --enable-locallisppath=/usr/local/share/emacs/site-lisp
 --infodir=/usr/local/Cellar/emacs/27.0.60/share/info/emacs
 --prefix=/usr/local/Cellar/emacs/27.0.60 --with-gnutls --without-x
 --with-xml2 --with-dbus --with-imagemagick --with-modules --with-rsvg
 --with-json --with-ns --disable-ns-self-contained 'CFLAGS=-Os -w -pipe
 -march=nehalem -Xclang -target-feature -Xclang -aes
 -mmacosx-version-min=10.14 -isysroot
 /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk'
 'CPPFLAGS=-I/usr/local/opt/icu4c/include
 -I/usr/local/opt/sqlite/include -I/usr/local/opt/readline/include
 -I/usr/local/opt/imagemagick <at> 6/include
 -I/usr/local/opt/openssl <at> 1.1/include -I/usr/local/opt/gettext/include
 -F/usr/local/Frameworks -isysroot
 /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk'
 'LDFLAGS=-L/usr/local/opt/icu4c/lib -L/usr/local/opt/sqlite/lib
 -L/usr/local/opt/readline/lib -L/usr/local/opt/imagemagick <at> 6/lib
 -L/usr/local/opt/openssl <at> 1.1/lib -L/usr/local/opt/libffi/lib
 -L/usr/local/opt/gettext/lib -L/usr/local/lib -F/usr/local/Frameworks
 -Wl,-headerpad_max_install_names -isysroot
 /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk''

Configured features:
RSVG IMAGEMAGICK DBUS GLIB NOTIFY KQUEUE ACL GNUTLS LIBXML2 ZLIB
TOOLKIT_SCROLL_BARS NS MODULES THREADS JSON PDUMPER LCMS2 GMP

Important settings:
  value of $EMACSDATA: /usr/local/opt/emacs/share/emacs/27.0.60/etc
  value of $EMACSDOC: /usr/local/opt/emacs/share/emacs/27.0.60/etc/
  value of $EMACSLOADPATH:
:/usr/local/opt/emacs/share/emacs/site-lisp:/usr/local/share/emacs/site-lisp/cask:/usr/local/opt/emacs/share/emacs/27.0.60/lisp:/usr/local/opt/emacs/share/emacs/27.0.60/site-lisp
  value of $EMACSPATH:
/usr/local/opt/emacs/libexec/emacs/27.0.60/x86_64-apple-darwin18.7.0
  value of $LC_ALL: en_CA.UTF-8
  value of $LANG: en_CA.UTF-8
  locale-coding-system: utf-8

<snip>

[crash_2020-01-15_1_create_process.log (application/octet-stream, attachment)]

[crash_2019-12-30_4.log (application/octet-stream, attachment)]

[crash_2020-01-02_1.log (application/octet-stream, attachment)]

[crash_2019-12-31_2.log (application/octet-stream, attachment)]

[crash_2020-01-06_2.log (application/octet-stream, attachment)]

[crash_2019-12-31_1.log (application/octet-stream, attachment)]

Message #8 received at 39164 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Justin Guenther <jguenther <at> gmail.com>
Cc: 39164 <at> debbugs.gnu.org
Subject: Re: bug#39164: 27.0.60;
 Intermittent crash on MacOS 10.14 in setup_process_coding_systems
Date: Fri, 17 Jan 2020 23:01:10 +0200

> From: Justin Guenther <jguenther <at> gmail.com>
> Date: Fri, 17 Jan 2020 13:00:47 -0600
> 
> I've been seeing an intermittent crash in setup_process_coding_systems
> (crash traces are attached). It seems to most frequently happen from
> create_process -> setup_process_coding_systems, but that may just be
> due to my frequent usage of `helm-do-ag-project-root'.
> 
> For the attached crash_2020-01-15_1_create_process.log, it happened
> when using `helm-do-ag-project-root', which takes input from the
> minibuffer and runs an external `ag` (the_silver_searcher) process to
> search the current project (in this case, a git repo) and
> display/refresh the results as I type in the minibuffer.
> 
> For the other crash logs, unfortunately I didn't record what I was
> doing at the time. But IIRC most of them were from
> `helm-do-ag-project-root`.

Thanks, but the backtraces don't show any source information, and it
is very hard to do anything about this without that.  Can you tell
where in coding.c is setup_process_coding_systems + 152?

Message #11 received at 39164 <at> debbugs.gnu.org (full text, mbox):

From: Robert Pluim <rpluim <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: Justin Guenther <jguenther <at> gmail.com>, 39164 <at> debbugs.gnu.org
Subject: Re: bug#39164: 27.0.60; Intermittent crash on MacOS 10.14 in
 setup_process_coding_systems
Date: Mon, 20 Jan 2020 10:15:28 +0100

>>>>> On Fri, 17 Jan 2020 23:01:10 +0200, Eli Zaretskii <eliz <at> gnu.org> said:

    Eli> Thanks, but the backtraces don't show any source information, and it
    Eli> is very hard to do anything about this without that.  Can you tell
    Eli> where in coding.c is setup_process_coding_systems + 152?

Iʼm on 10.14, and donʼt see this crash. Assuming my emacs-27 build is
similar to Justin's, itʼs this:

7988      setup_coding_system (coding_system, proc_decode_coding_system[inch]);
   0x000000010019645f <+143>:   mov    (%r12,%r15,8),%rsi
   0x0000000100196463 <+147>:   callq  0x10007b610 <setup_coding_system>

7989
7990      if (!proc_encode_coding_system[outch])
   0x0000000100196468 <+152>:   lea    0x4d3061(%rip),%r15        # 0x1006694d0
   0x000000010019646f <+159>:   mov    (%r15,%r14,8),%rsi
   0x0000000100196473 <+163>:   test   %rsi,%rsi
   0x0000000100196476 <+166>:   jne    0x100196489 <setup_process_coding_systems+185>

Message #14 received at 39164 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Robert Pluim <rpluim <at> gmail.com>
Cc: jguenther <at> gmail.com, 39164 <at> debbugs.gnu.org
Subject: Re: bug#39164: 27.0.60; Intermittent crash on MacOS 10.14 in
 setup_process_coding_systems
Date: Mon, 20 Jan 2020 20:00:49 +0200

> From: Robert Pluim <rpluim <at> gmail.com>
> Cc: Justin Guenther <jguenther <at> gmail.com>,  39164 <at> debbugs.gnu.org
> Date: Mon, 20 Jan 2020 10:15:28 +0100
> 
> >>>>> On Fri, 17 Jan 2020 23:01:10 +0200, Eli Zaretskii <eliz <at> gnu.org> said:
> 
>     Eli> Thanks, but the backtraces don't show any source information, and it
>     Eli> is very hard to do anything about this without that.  Can you tell
>     Eli> where in coding.c is setup_process_coding_systems + 152?
> 
> Iʼm on 10.14, and donʼt see this crash. Assuming my emacs-27 build is
> similar to Justin's, itʼs this:
> 
> 7988      setup_coding_system (coding_system, proc_decode_coding_system[inch]);
>    0x000000010019645f <+143>:   mov    (%r12,%r15,8),%rsi
>    0x0000000100196463 <+147>:   callq  0x10007b610 <setup_coding_system>
> 
> 7989
> 7990      if (!proc_encode_coding_system[outch])
>    0x0000000100196468 <+152>:   lea    0x4d3061(%rip),%r15        # 0x1006694d0
>    0x000000010019646f <+159>:   mov    (%r15,%r14,8),%rsi
>    0x0000000100196473 <+163>:   test   %rsi,%rsi
>    0x0000000100196476 <+166>:   jne    0x100196489 <setup_process_coding_systems+185>

Does this mean proc_encode_coding_system is NULL (or garbled), or the
value of outch is outlandish (or garbled)?  Showing the values of
these variables will be appreciated.

Message #17 received at 39164 <at> debbugs.gnu.org (full text, mbox):

From: Robert Pluim <rpluim <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: jguenther <at> gmail.com, 39164 <at> debbugs.gnu.org
Subject: Re: bug#39164: 27.0.60; Intermittent crash on MacOS 10.14 in
 setup_process_coding_systems
Date: Mon, 20 Jan 2020 19:42:10 +0100

>>>>> On Mon, 20 Jan 2020 20:00:49 +0200, Eli Zaretskii <eliz <at> gnu.org> said:

    Eli> Does this mean proc_encode_coding_system is NULL (or garbled), or the
    Eli> value of outch is outlandish (or garbled)?  Showing the values of
    Eli> these variables will be appreciated.

Iʼd show you if I could, but I donʼt see the crash.

Robert

Message #20 received at 39164 <at> debbugs.gnu.org (full text, mbox):

From: Tomasz Kowal <tomekowal <at> gmail.com>
To: 39164 <at> debbugs.gnu.org
Subject: Reproducing SIGSEGV bug
Date: Sun, 26 Jan 2020 18:41:42 +0100

Hi! I have a speculation to share.

In the project that produced SIGSEGV, I was using lsp-mode.
It asks if I want to "watch files" every time I open a project.
I added really high threshold for the number of allowed watched files
in .dir-locals.
Shortly after I encountered the SIGSEGV for the first time.

I now changed it back to not watching the files, and I am not
encountering the problem anymore.
In the Bug#39164 upstream:
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=39164, they also mention
lsp-mode.

I am not sure if that helps but I believe, enabling file watching in
lsp-mode on a big project consumes enough memory to make the bug more
likely to reoccur.

BR,
Tomasz Kowal

Message #23 received at 39164 <at> debbugs.gnu.org (full text, mbox):

From: mituharu <at> math.s.chiba-u.ac.jp
To: "Robert Pluim" <rpluim <at> gmail.com>
Cc: Tomasz Kowal <tomekowal <at> gmail.com>, Eli Zaretskii <eliz <at> gnu.org>,
 39164 <at> debbugs.gnu.org, jguenther <at> gmail.com
Subject: Re: bug#39164: 27.0.60; Intermittent crash on MacOS 10.14 in
 setup_process_coding_systems
Date: Mon, 27 Jan 2020 22:26:10 +0900

>>>>>> On Mon, 20 Jan 2020 20:00:49 +0200, Eli Zaretskii <eliz <at> gnu.org>
>>>>>> said:
>
>     Eli> Does this mean proc_encode_coding_system is NULL (or garbled), or
> the
>     Eli> value of outch is outlandish (or garbled)?  Showing the values of
>     Eli> these variables will be appreciated.
>
> I?d show you if I could, but I don?t see the crash.
>
> Robert

This is a variant of Bug#24325: Crash - fd larger than FD_SETSIZE.
Some functions in the Core Foundation framework on macOS
call setrlimit for RLIMIT_NOFILE in order to increase the limit of
the maximum number of open files for the process:

  https://opensource.apple.com/source/CF/CF-1153.18/CFSocket.c.auto.html

So the fix for Bug#24325 doesn't work in this case.

				     YAMAMOTO Mitsuharu
				mituharu <at> math.s.chiba-u.ac.jp

Message #26 received at 39164 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: mituharu <at> math.s.chiba-u.ac.jp
Cc: tomekowal <at> gmail.com, rpluim <at> gmail.com, 39164 <at> debbugs.gnu.org,
 jguenther <at> gmail.com
Subject: Re: bug#39164: 27.0.60; Intermittent crash on MacOS 10.14 in
 setup_process_coding_systems
Date: Mon, 27 Jan 2020 20:19:07 +0200

> Date: Mon, 27 Jan 2020 22:26:10 +0900
> From: mituharu <at> math.s.chiba-u.ac.jp
> Cc: "Eli Zaretskii" <eliz <at> gnu.org>,
>  jguenther <at> gmail.com,
>  "Tomasz Kowal" <tomekowal <at> gmail.com>,
>  39164 <at> debbugs.gnu.org
> 
> This is a variant of Bug#24325: Crash - fd larger than FD_SETSIZE.
> Some functions in the Core Foundation framework on macOS
> call setrlimit for RLIMIT_NOFILE in order to increase the limit of
> the maximum number of open files for the process:
> 
>   https://opensource.apple.com/source/CF/CF-1153.18/CFSocket.c.auto.html
> 
> So the fix for Bug#24325 doesn't work in this case.

Thanks.

So I guess one possible solution would be to see that fd is beyond
FD_SETSIZE, and if so, call getrlimit to see if the limit was bumped
up, and if so, reallocate the arrays used by process.c?  Would that
make sense?

Message #29 received at 39164 <at> debbugs.gnu.org (full text, mbox):

From: Robert Pluim <rpluim <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: tomekowal <at> gmail.com, jguenther <at> gmail.com, 39164 <at> debbugs.gnu.org,
 mituharu <at> math.s.chiba-u.ac.jp
Subject: Re: bug#39164: 27.0.60; Intermittent crash on MacOS 10.14 in
 setup_process_coding_systems
Date: Tue, 28 Jan 2020 08:59:35 +0100

>>>>> On Mon, 27 Jan 2020 20:19:07 +0200, Eli Zaretskii <eliz <at> gnu.org> said:

    >> Date: Mon, 27 Jan 2020 22:26:10 +0900
    >> From: mituharu <at> math.s.chiba-u.ac.jp
    >> Cc: "Eli Zaretskii" <eliz <at> gnu.org>,
    >> jguenther <at> gmail.com,
    >> "Tomasz Kowal" <tomekowal <at> gmail.com>,
    >> 39164 <at> debbugs.gnu.org
    >> 
    >> This is a variant of Bug#24325: Crash - fd larger than FD_SETSIZE.
    >> Some functions in the Core Foundation framework on macOS
    >> call setrlimit for RLIMIT_NOFILE in order to increase the limit of
    >> the maximum number of open files for the process:
    >> 
    >> https://opensource.apple.com/source/CF/CF-1153.18/CFSocket.c.auto.html
    >> 
    >> So the fix for Bug#24325 doesn't work in this case.

    Eli> Thanks.

    Eli> So I guess one possible solution would be to see that fd is beyond
    Eli> FD_SETSIZE, and if so, call getrlimit to see if the limit was bumped
    Eli> up, and if so, reallocate the arrays used by process.c?  Would that
    Eli> make sense?

This will break {p}select, no? Thatʼs defined to only work up to
FD_SETSIZE.

Robert

Message #32 received at 39164 <at> debbugs.gnu.org (full text, mbox):

From: YAMAMOTO Mitsuharu <mituharu <at> math.s.chiba-u.ac.jp>
To: Robert Pluim <rpluim <at> gmail.com>
Cc: tomekowal <at> gmail.com, Eli Zaretskii <eliz <at> gnu.org>, 39164 <at> debbugs.gnu.org,
 jguenther <at> gmail.com
Subject: Re: bug#39164: 27.0.60;
 Intermittent crash on MacOS 10.14 in setup_process_coding_systems
Date: Tue, 28 Jan 2020 17:23:25 +0900

On Tue, 28 Jan 2020 16:59:35 +0900,
Robert Pluim wrote:
> 
> >>>>> On Mon, 27 Jan 2020 20:19:07 +0200, Eli Zaretskii <eliz <at> gnu.org> said:
> 
>     >> Date: Mon, 27 Jan 2020 22:26:10 +0900
>     >> From: mituharu <at> math.s.chiba-u.ac.jp
>     >> Cc: "Eli Zaretskii" <eliz <at> gnu.org>,
>     >> jguenther <at> gmail.com,
>     >> "Tomasz Kowal" <tomekowal <at> gmail.com>,
>     >> 39164 <at> debbugs.gnu.org
>     >> 
>     >> This is a variant of Bug#24325: Crash - fd larger than FD_SETSIZE.
>     >> Some functions in the Core Foundation framework on macOS
>     >> call setrlimit for RLIMIT_NOFILE in order to increase the limit of
>     >> the maximum number of open files for the process:
>     >> 
>     >> https://opensource.apple.com/source/CF/CF-1153.18/CFSocket.c.auto.html
>     >> 
>     >> So the fix for Bug#24325 doesn't work in this case.
> 
>     Eli> Thanks.
> 
>     Eli> So I guess one possible solution would be to see that fd is beyond
>     Eli> FD_SETSIZE, and if so, call getrlimit to see if the limit was bumped
>     Eli> up, and if so, reallocate the arrays used by process.c?  Would that
>     Eli> make sense?
> 
> This will break {p}select, no? Thatʼs defined to only work up to
> FD_SETSIZE.

Yes.  I'm currently thinking about adding the following workaround to
the next release of the Mac port.  It is not a perfect solution, but
the Mac port is still based on Emacs 26, and I don't want to make a
drastic change there.

			     YAMAMOTO Mitsuharu
			mituharu <at> math.s.chiba-u.ac.jp

diff --git a/src/macappkit.m b/src/macappkit.m
index d2647aff97..08a7e6bc9b 100644
--- a/src/macappkit.m
+++ b/src/macappkit.m
@@ -1337,6 +1337,21 @@ - (void)applicationDidFinishLaunching:(NSNotification *)notification
   setenv ("CAVIEW_USE_GL", "1", 0);
 #endif
 
+  /* Some functions/methods in CoreFoundation/Foundation increase the
+     maximum number of open files for the process in their first call.
+     We make dummy calls to them and then reduce the resource limit
+     here, since pselect cannot handle file descriptors that are
+     greater than or equal to FD_SETSIZE.  */
+  CFSocketGetTypeID ();
+  CFFileDescriptorGetTypeID ();
+  MRC_RELEASE ([[NSFileHandle alloc] init]);
+  struct rlimit rlim;
+  if (getrlimit (RLIMIT_NOFILE, &rlim) == 0 && rlim.rlim_cur > FD_SETSIZE)
+    {
+      rlim.rlim_cur = FD_SETSIZE;
+      setrlimit (RLIMIT_NOFILE, &rlim);
+    }
+
   /* Exit from the main event loop.  */
   [NSApp stop:nil];
   [NSApp postDummyEvent];

Message #35 received at 39164 <at> debbugs.gnu.org (full text, mbox):

From: Robert Pluim <rpluim <at> gmail.com>
To: YAMAMOTO Mitsuharu <mituharu <at> math.s.chiba-u.ac.jp>
Cc: tomekowal <at> gmail.com, Eli Zaretskii <eliz <at> gnu.org>, 39164 <at> debbugs.gnu.org,
 jguenther <at> gmail.com
Subject: Re: bug#39164: 27.0.60; Intermittent crash on MacOS 10.14 in
 setup_process_coding_systems
Date: Tue, 28 Jan 2020 09:41:13 +0100

>>>>> On Tue, 28 Jan 2020 17:23:25 +0900, YAMAMOTO Mitsuharu <mituharu <at> math.s.chiba-u.ac.jp> said:

    YAMAMOTO> Yes.  I'm currently thinking about adding the following workaround to
    YAMAMOTO> the next release of the Mac port.  It is not a perfect solution, but
    YAMAMOTO> the Mac port is still based on Emacs 26, and I don't want to make a
    YAMAMOTO> drastic change there.

I guess the real solution would be to switch to poll(2) or
something. Definitely not a small change. <time passes> Hmm, w32
emulates select, so weʼd have to write an emulation of poll.

Robert

Message #38 received at 39164 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: YAMAMOTO Mitsuharu <mituharu <at> math.s.chiba-u.ac.jp>,
 Robert Pluim <rpluim <at> gmail.com>
Cc: tomekowal <at> gmail.com, jguenther <at> gmail.com, 39164 <at> debbugs.gnu.org
Subject: Re: bug#39164: 27.0.60;
 Intermittent crash on MacOS 10.14 in setup_process_coding_systems
Date: Tue, 28 Jan 2020 11:02:52 +0200

On January 28, 2020 10:23:25 AM GMT+02:00, YAMAMOTO Mitsuharu <mituharu <at> math.s.chiba-u.ac.jp> wrote:
> On Tue, 28 Jan 2020 16:59:35 +0900,
> Robert Pluim wrote:
> > 
> > This will break {p}select, no? Thatʼs defined to only work up to
> > FD_SETSIZE.
> 
> Yes.  I'm currently thinking about adding the following workaround to
> the next release of the Mac port.  It is not a perfect solution, but
> the Mac port is still based on Emacs 26, and I don't want to make a
> drastic change there.

I'm not sure I understand the implication.  How will this workaround ensure starting a  subprocess doesn't produce a file descriptor beyond FD_SETSIZE?  Maybe all I'm missing is under what conditions will the code you propose be called.

Message #41 received at 39164 <at> debbugs.gnu.org (full text, mbox):

From: YAMAMOTO Mitsuharu <mituharu <at> math.s.chiba-u.ac.jp>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: tomekowal <at> gmail.com, Robert Pluim <rpluim <at> gmail.com>, 39164 <at> debbugs.gnu.org,
 jguenther <at> gmail.com
Subject: Re: bug#39164: 27.0.60;
 Intermittent crash on MacOS 10.14 in setup_process_coding_systems
Date: Tue, 28 Jan 2020 18:14:41 +0900

On Tue, 28 Jan 2020 18:02:52 +0900,
Eli Zaretskii wrote:
> 
> On January 28, 2020 10:23:25 AM GMT+02:00, YAMAMOTO Mitsuharu <mituharu <at> math.s.chiba-u.ac.jp> wrote:
> > On Tue, 28 Jan 2020 16:59:35 +0900,
> > Robert Pluim wrote:
> > > 
> > > This will break {p}select, no? Thatʼs defined to only work up to
> > > FD_SETSIZE.
> > 
> > Yes.  I'm currently thinking about adding the following workaround to
> > the next release of the Mac port.  It is not a perfect solution, but
> > the Mac port is still based on Emacs 26, and I don't want to make a
> > drastic change there.
> 
> I'm not sure I understand the implication.  How will this workaround ensure starting a  subprocess doesn't produce a file descriptor beyond FD_SETSIZE?  Maybe all I'm missing is under what conditions will the code you propose be called.

It is called during the Mac terminal initialization.

				     YAMAMOTO Mitsuharu
				mituharu <at> math.s.chiba-u.ac.jp

Message #44 received at 39164 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: YAMAMOTO Mitsuharu <mituharu <at> math.s.chiba-u.ac.jp>
Cc: tomekowal <at> gmail.com, Robert Pluim <rpluim <at> gmail.com>, 39164 <at> debbugs.gnu.org,
 jguenther <at> gmail.com
Subject: Re: bug#39164: 27.0.60;
 Intermittent crash on MacOS 10.14 in setup_process_coding_systems
Date: Tue, 28 Jan 2020 11:42:29 +0200

On January 28, 2020 11:14:41 AM GMT+02:00, YAMAMOTO Mitsuharu <mituharu <at> math.s.chiba-u.ac.jp> wrote:
> On Tue, 28 Jan 2020 18:02:52 +0900,
> Eli Zaretskii wrote:
> > 
> > I'm not sure I understand the implication.  How will this workaround
> ensure starting a  subprocess doesn't produce a file descriptor beyond
> FD_SETSIZE?  Maybe all I'm missing is under what conditions will the
> code you propose be called.
> 
> It is called during the Mac terminal initialization.


And the increase in the number of available descriptors triggered by the system libraries is guaranteed to be called before that?  Then how is this call to setrlimit is dufferent from what we already do at startup?

Message #47 received at 39164 <at> debbugs.gnu.org (full text, mbox):

From: YAMAMOTO Mitsuharu <mituharu <at> math.s.chiba-u.ac.jp>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: tomekowal <at> gmail.com, Robert Pluim <rpluim <at> gmail.com>, 39164 <at> debbugs.gnu.org,
 jguenther <at> gmail.com
Subject: Re: bug#39164: 27.0.60;
 Intermittent crash on MacOS 10.14 in setup_process_coding_systems
Date: Tue, 28 Jan 2020 19:06:09 +0900

On Tue, 28 Jan 2020 18:42:29 +0900,
Eli Zaretskii wrote:
> 
> On January 28, 2020 11:14:41 AM GMT+02:00, YAMAMOTO Mitsuharu <mituharu <at> math.s.chiba-u.ac.jp> wrote:
> > On Tue, 28 Jan 2020 18:02:52 +0900,
> > Eli Zaretskii wrote:
> > > 
> > > I'm not sure I understand the implication.  How will this workaround
> > ensure starting a  subprocess doesn't produce a file descriptor beyond
> > FD_SETSIZE?  Maybe all I'm missing is under what conditions will the
> > code you propose be called.
> > 
> > It is called during the Mac terminal initialization.
> 
> 
> And the increase in the number of available descriptors triggered by the system libraries is guaranteed to be called before that?  Then how is this call to setrlimit is dufferent from what we already do at startup?

The increase happens only once per function in question.  So we make a
dummy call in advance so later calls may not cause the increase in
unpredicable timings.

				     YAMAMOTO Mitsuharu
				mituharu <at> math.s.chiba-u.ac.jp

Message #50 received at 39164 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: YAMAMOTO Mitsuharu <mituharu <at> math.s.chiba-u.ac.jp>
Cc: tomekowal <at> gmail.com, rpluim <at> gmail.com, 39164 <at> debbugs.gnu.org,
 jguenther <at> gmail.com
Subject: Re: bug#39164: 27.0.60;
 Intermittent crash on MacOS 10.14 in setup_process_coding_systems
Date: Tue, 28 Jan 2020 19:37:10 +0200

> Date: Tue, 28 Jan 2020 19:06:09 +0900
> From: YAMAMOTO Mitsuharu <mituharu <at> math.s.chiba-u.ac.jp>
> Cc: Robert Pluim <rpluim <at> gmail.com>,
> 	tomekowal <at> gmail.com,
> 	39164 <at> debbugs.gnu.org,
> 	jguenther <at> gmail.com
> 
> The increase happens only once per function in question.  So we make a
> dummy call in advance so later calls may not cause the increase in
> unpredicable timings.

Can these dummy calls be made just before we call setrlimit in
init_process_emacs, as opposed to in a macOS-specific source file?

Also, could there be Core Foundation functions other than those you
propose to call, that have the same effect?  IOW, how can we be sure
we issued a dummy call for every such function that matters to Emacs?

Thanks.

Message #53 received at 39164 <at> debbugs.gnu.org (full text, mbox):

From: YAMAMOTO Mitsuharu <mituharu <at> math.s.chiba-u.ac.jp>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: tomekowal <at> gmail.com, rpluim <at> gmail.com, 39164 <at> debbugs.gnu.org,
 jguenther <at> gmail.com
Subject: Re: bug#39164: 27.0.60;
 Intermittent crash on MacOS 10.14 in setup_process_coding_systems
Date: Wed, 29 Jan 2020 19:26:23 +0900

On Wed, 29 Jan 2020 02:37:10 +0900,
Eli Zaretskii wrote:
> 
> > The increase happens only once per function in question.  So we make a
> > dummy call in advance so later calls may not cause the increase in
> > unpredicable timings.
> 
> Can these dummy calls be made just before we call setrlimit in
> init_process_emacs, as opposed to in a macOS-specific source file?

The functions in question are used together with Mac-specific GUI
event handing.  I placed the dummy calls inside macOS-specific code so
the -nw case may not change the behavior from the original one.

> Also, could there be Core Foundation functions other than those you
> propose to call, that have the same effect?  IOW, how can we be sure
> we issued a dummy call for every such function that matters to Emacs?

That's why I called it as "workaround" and "not a perfect solution".
Nevertheless it is well-suited for my case, i.e., the Mac port made on
top of established Emacs 26.

				     YAMAMOTO Mitsuharu
				mituharu <at> math.s.chiba-u.ac.jp

Message #56 received at 39164 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: YAMAMOTO Mitsuharu <mituharu <at> math.s.chiba-u.ac.jp>
Cc: tomekowal <at> gmail.com, rpluim <at> gmail.com, 39164 <at> debbugs.gnu.org,
 jguenther <at> gmail.com
Subject: Re: bug#39164: 27.0.60;
 Intermittent crash on MacOS 10.14 in setup_process_coding_systems
Date: Wed, 29 Jan 2020 13:09:16 +0200

On January 29, 2020 12:26:23 PM GMT+02:00, YAMAMOTO Mitsuharu <mituharu <at> math.s.chiba-u.ac.jp> wrote:
> On Wed, 29 Jan 2020 02:37:10 +0900,
> Eli Zaretskii wrote:
> > 
> > > The increase happens only once per function in question.  So we
> make a
> > > dummy call in advance so later calls may not cause the increase in
> > > unpredicable timings.
> > 
> > Can these dummy calls be made just before we call setrlimit in
> > init_process_emacs, as opposed to in a macOS-specific source file?
> 
> The functions in question are used together with Mac-specific GUI
> event handing.  I placed the dummy calls inside macOS-specific code so
> the -nw case may not change the behavior from the original one.

We could condition the code in process.c by the aporopriate NS preprocessor condition to make it take effect only in GUI sessions.

Message #59 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: bug-gnu-emacs <at> gnu.org,YAMAMOTO Mitsuharu <mituharu <at> math.s.chiba-u.ac.jp>
Cc: tomekowal <at> gmail.com, rpluim <at> gmail.com, 39164 <at> debbugs.gnu.org,
 jguenther <at> gmail.com
Subject: Re: bug#39164: 27.0.60;
 Intermittent crash on MacOS 10.14 in setup_process_coding_systems
Date: Wed, 29 Jan 2020 14:38:32 +0200

On January 29, 2020 1:09:16 PM GMT+02:00, Eli Zaretskii <eliz <at> gnu.org> wrote:
> On January 29, 2020 12:26:23 PM GMT+02:00, YAMAMOTO Mitsuharu
> <mituharu <at> math.s.chiba-u.ac.jp> wrote:
> > On Wed, 29 Jan 2020 02:37:10 +0900,
> > Eli Zaretskii wrote:
> > > 
> > > > The increase happens only once per function in question.  So we
> > make a
> > > > dummy call in advance so later calls may not cause the increase
> in
> > > > unpredicable timings.
> > > 
> > > Can these dummy calls be made just before we call setrlimit in
> > > init_process_emacs, as opposed to in a macOS-specific source file?
> > 
> > The functions in question are used together with Mac-specific GUI
> > event handing.  I placed the dummy calls inside macOS-specific code
> so
> > the -nw case may not change the behavior from the original one.
> 
> We could condition the code in process.c by the aporopriate NS
> preprocessor condition to make it take effect only in GUI sessions.

Sorry, preprocessor condition is not TRT, we should have a runtime test that limits these dummy calls to GUI sessions.

My point is that having this code in the same place for all platforms is better for long-range maintenance.

Message #65 received at submit <at> debbugs.gnu.org (full text, mbox):

From: YAMAMOTO Mitsuharu <mituharu <at> math.s.chiba-u.ac.jp>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: tomekowal <at> gmail.com, bug-gnu-emacs <at> gnu.org, 39164 <at> debbugs.gnu.org,
 rpluim <at> gmail.com, jguenther <at> gmail.com
Subject: Re: bug#39164: 27.0.60;
 Intermittent crash on MacOS 10.14 in setup_process_coding_systems
Date: Thu, 30 Jan 2020 18:42:41 +0900

On Wed, 29 Jan 2020 21:38:32 +0900,
Eli Zaretskii wrote:
> 
> On January 29, 2020 1:09:16 PM GMT+02:00, Eli Zaretskii <eliz <at> gnu.org> wrote:
> > On January 29, 2020 12:26:23 PM GMT+02:00, YAMAMOTO Mitsuharu
> > <mituharu <at> math.s.chiba-u.ac.jp> wrote:
> > > On Wed, 29 Jan 2020 02:37:10 +0900,
> > > Eli Zaretskii wrote:
> > > > 
> > > > > The increase happens only once per function in question.  So we
> > > make a
> > > > > dummy call in advance so later calls may not cause the increase
> > in
> > > > > unpredicable timings.
> > > > 
> > > > Can these dummy calls be made just before we call setrlimit in
> > > > init_process_emacs, as opposed to in a macOS-specific source file?
> > > 
> > > The functions in question are used together with Mac-specific GUI
> > > event handing.  I placed the dummy calls inside macOS-specific code
> > so
> > > the -nw case may not change the behavior from the original one.
> > 
> > We could condition the code in process.c by the aporopriate NS
> > preprocessor condition to make it take effect only in GUI sessions.
> 
> Sorry, preprocessor condition is not TRT, we should have a runtime test that limits these dummy calls to GUI sessions.
> 
> My point is that having this code in the same place for all platforms is better for long-range maintenance.

If we make dummy calls before setrlimit in init_process_emacs, then an
artificially increased limit is recorded to the variable nofile_limit,
and that will be used in restore_nofile_limit later.

				     YAMAMOTO Mitsuharu
				mituharu <at> math.s.chiba-u.ac.jp

Message #71 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: YAMAMOTO Mitsuharu <mituharu <at> math.s.chiba-u.ac.jp>
Cc: tomekowal <at> gmail.com, bug-gnu-emacs <at> gnu.org, 39164 <at> debbugs.gnu.org,
 rpluim <at> gmail.com, jguenther <at> gmail.com
Subject: Re: bug#39164: 27.0.60;
 Intermittent crash on MacOS 10.14 in setup_process_coding_systems
Date: Thu, 30 Jan 2020 16:49:48 +0200

> Date: Thu, 30 Jan 2020 18:42:41 +0900
> From: YAMAMOTO Mitsuharu <mituharu <at> math.s.chiba-u.ac.jp>
> Cc: bug-gnu-emacs <at> gnu.org,
> 	tomekowal <at> gmail.com,
> 	rpluim <at> gmail.com,
> 	39164 <at> debbugs.gnu.org,
> 	jguenther <at> gmail.com
> 
> > My point is that having this code in the same place for all platforms is better for long-range maintenance.
> 
> If we make dummy calls before setrlimit in init_process_emacs, then an
> artificially increased limit is recorded to the variable nofile_limit,
> and that will be used in restore_nofile_limit later.

That's true, but restore_nofile_limit is called after the fork in the
context of the child process, so can that call cause any harm?

If it does do harm, maybe we should simply avoid that call on macOS?

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.