GNU bug report logs - #65908
29.1.50; Emacs 29 regresses on macOS

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 65908 in the body.
You can then email your comments to 65908 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Zhang Haijun <ccsmile2008 <at> outlook.com>
To: "bug-gnu-emacs <at> gnu.org" <bug-gnu-emacs <at> gnu.org>
Subject: 29.1.50; Emacs 29 regresses on macOS
Date: Wed, 13 Sep 2023 06:47:12 +0000

[Message part 1 (text/plain, inline)]

  1.  emacs -Q
  2.  paste the following code to scratch buffer and execute it for saveral times

(benchmark-progn
  (setq enable-dir-local-variables nil
            enable-local-variables nil)

  (let* ((test-dir "~/source/emacs-git/lisp")
             (file-list (directory-files test-dir t
                                                       "\\`[fsw].*[.]el\\'" t))
             (start-tm (current-time))
             end-tm buf buf-list)

      (dolist (test-file file-list)
        (setq buf (find-file-noselect test-file))
        (with-current-buffer buf
            (goto-char (point-min))
            (while (not (eobp))
              (end-of-defun)))
        (setq buf-list (cons buf buf-list)))

      (setq end-tm (current-time))

      (dolist (buf buf-list)
        (kill-buffer buf))

      (let ((delta (time-subtract end-tm start-tm)))
        (message "--- files: %d, delta: %s"
                     (length file-list) (float-time delta)))))


======== Result for Emacs 27: ======
--- files: 61, delta: 3.021547
Elapsed time: 3.030485s (0.173305s in 14 GCs)
"--- files: 61, delta: 3.021547"
--- files: 61, delta: 3.1373
Elapsed time: 3.145406s (0.181551s in 14 GCs)
"--- files: 61, delta: 3.1373"
--- files: 61, delta: 3.093033
Elapsed time: 3.100581s (0.162783s in 13 GCs)
"--- files: 61, delta: 3.093033"


======== Result for Emacs 29: =======
--- files: 61, delta: 5.271817
Elapsed time: 5.278284s (0.174266s in 13 GCs)
"--- files: 61, delta: 5.271817"
--- files: 61, delta: 4.919798
Elapsed time: 4.923492s (0.162042s in 13 GCs)
"--- files: 61, delta: 4.919798"
--- files: 61, delta: 5.127965
Elapsed time: 5.131604s (0.154718s in 12 GCs)
"--- files: 61, delta: 5.127965"
--- files: 61, delta: 4.978964
Elapsed time: 4.982757s (0.149401s in 12 GCs)
"--- files: 61, delta: 4.978964"


Time used in Emacs 29 is 1.6x than in Emacs 27. They are all built with "-O2".
I also tested on Linux. There is no such big regress.

[Message part 2 (text/html, inline)]

Message #8 received at 65908 <at> debbugs.gnu.org (full text, mbox):

From: Gerd Möllmann <gerd.moellmann <at> gmail.com>
To: Zhang Haijun <ccsmile2008 <at> outlook.com>
Cc: 65908 <at> debbugs.gnu.org
Subject: Re: bug#65908: 29.1.50; Emacs 29 regresses on macOS
Date: Wed, 04 Oct 2023 07:44:33 +0200

Zhang Haijun <ccsmile2008 <at> outlook.com> writes:

> 1 emacs -Q
> 2 paste the following code to scratch buffer and execute it for saveral times
>
> (benchmark-progn 
>   (setq enable-dir-local-variables nil
>             enable-local-variables nil)
>
>   (let* ((test-dir "~/source/emacs-git/lisp")
>              (file-list (directory-files test-dir t
>                                                        "\\`[fsw].*[.]el\\'" t))
>              (start-tm (current-time))
>              end-tm buf buf-list)
>
>       (dolist (test-file file-list)
>         (setq buf (find-file-noselect test-file))
>         (with-current-buffer buf
>             (goto-char (point-min))
>             (while (not (eobp))
>               (end-of-defun)))
>         (setq buf-list (cons buf buf-list)))
>
>       (setq end-tm (current-time))
>
>       (dolist (buf buf-list)
>         (kill-buffer buf))
>
>       (let ((delta (time-subtract end-tm start-tm)))
>         (message "--- files: %d, delta: %s"
>                      (length file-list) (float-time delta)))))

(Your mailer does strange things with whitespace...)

> Time used in Emacs 29 is 1.6x than in Emacs 27. They are all built with "-O2".
> I also tested on Linux. There is no such big regress.

Thanks for the report.

I can reproduce this on macOS 14.0 (OCLP).

emacs-28, HEAD:
"--- files: 61, delta: 3.086468"
"--- files: 61, delta: 3.106478"
"--- files: 61, delta: 3.325344"

emacs-29, HEAD:
"--- files: 61, delta: 5.052421"
"--- files: 61, delta: 5.068663"
"--- files: 61, delta: 5.110502"

FWIW, master
"--- files: 61, delta: 5.124109"
"--- files: 61, delta: 5.101217"
"--- files: 61, delta: 5.13068"

Message #11 received at 65908 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Gerd Möllmann <gerd.moellmann <at> gmail.com>
Cc: 65908 <at> debbugs.gnu.org, ccsmile2008 <at> outlook.com
Subject: Re: bug#65908: 29.1.50; Emacs 29 regresses on macOS
Date: Wed, 04 Oct 2023 10:38:39 +0300

> Cc: 65908 <at> debbugs.gnu.org
> From: Gerd Möllmann <gerd.moellmann <at> gmail.com>
> Date: Wed, 04 Oct 2023 07:44:33 +0200
> 
> Zhang Haijun <ccsmile2008 <at> outlook.com> writes:
> 
> > 1 emacs -Q
> > 2 paste the following code to scratch buffer and execute it for saveral times
> >
> > (benchmark-progn 
> >   (setq enable-dir-local-variables nil
> >             enable-local-variables nil)
> >
> >   (let* ((test-dir "~/source/emacs-git/lisp")
> >              (file-list (directory-files test-dir t
> >                                                        "\\`[fsw].*[.]el\\'" t))
> >              (start-tm (current-time))
> >              end-tm buf buf-list)
> >
> >       (dolist (test-file file-list)
> >         (setq buf (find-file-noselect test-file))
> >         (with-current-buffer buf
> >             (goto-char (point-min))
> >             (while (not (eobp))
> >               (end-of-defun)))
> >         (setq buf-list (cons buf buf-list)))
> >
> >       (setq end-tm (current-time))
> >
> >       (dolist (buf buf-list)
> >         (kill-buffer buf))
> >
> >       (let ((delta (time-subtract end-tm start-tm)))
> >         (message "--- files: %d, delta: %s"
> >                      (length file-list) (float-time delta)))))
> 
> (Your mailer does strange things with whitespace...)
> 
> > Time used in Emacs 29 is 1.6x than in Emacs 27. They are all built with "-O2".
> > I also tested on Linux. There is no such big regress.
> 
> Thanks for the report.
> 
> I can reproduce this on macOS 14.0 (OCLP).
> 
> emacs-28, HEAD:
> "--- files: 61, delta: 3.086468"
> "--- files: 61, delta: 3.106478"
> "--- files: 61, delta: 3.325344"
> 
> emacs-29, HEAD:
> "--- files: 61, delta: 5.052421"
> "--- files: 61, delta: 5.068663"
> "--- files: 61, delta: 5.110502"
> 
> FWIW, master
> "--- files: 61, delta: 5.124109"
> "--- files: 61, delta: 5.101217"
> "--- files: 61, delta: 5.13068"

A completely expanded Lisp profile, after loading the Lisp files
involved in this as *.el, would be appreciated.

Message #14 received at 65908 <at> debbugs.gnu.org (full text, mbox):

From: Gerd Möllmann <gerd.moellmann <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 65908 <at> debbugs.gnu.org, ccsmile2008 <at> outlook.com
Subject: Re: bug#65908: 29.1.50; Emacs 29 regresses on macOS
Date: Wed, 4 Oct 2023 09:50:18 +0200

On 04.10.23 09:38, Eli Zaretskii wrote:
>> Cc: 65908 <at> debbugs.gnu.org
>> From: Gerd Möllmann <gerd.moellmann <at> gmail.com>
>> Date: Wed, 04 Oct 2023 07:44:33 +0200
>>
>> Zhang Haijun <ccsmile2008 <at> outlook.com> writes:
>>
>>> 1 emacs -Q
>>> 2 paste the following code to scratch buffer and execute it for saveral times
>>>
>>> (benchmark-progn
>>>    (setq enable-dir-local-variables nil
>>>             enable-local-variables nil)
>>>
>>>    (let* ((test-dir "~/source/emacs-git/lisp")
>>>              (file-list (directory-files test-dir t
>>>                                                        "\\`[fsw].*[.]el\\'" t))
>>>              (start-tm (current-time))
>>>              end-tm buf buf-list)
>>>
>>>       (dolist (test-file file-list)
>>>         (setq buf (find-file-noselect test-file))
>>>         (with-current-buffer buf
>>>             (goto-char (point-min))
>>>             (while (not (eobp))
>>>               (end-of-defun)))
>>>         (setq buf-list (cons buf buf-list)))
>>>
>>>       (setq end-tm (current-time))
>>>
>>>       (dolist (buf buf-list)
>>>         (kill-buffer buf))
>>>
>>>       (let ((delta (time-subtract end-tm start-tm)))
>>>         (message "--- files: %d, delta: %s"
>>>                      (length file-list) (float-time delta)))))
>>
>> (Your mailer does strange things with whitespace...)
>>
>>> Time used in Emacs 29 is 1.6x than in Emacs 27. They are all built with "-O2".
>>> I also tested on Linux. There is no such big regress.
>>
>> Thanks for the report.
>>
>> I can reproduce this on macOS 14.0 (OCLP).
>>
>> emacs-28, HEAD:
>> "--- files: 61, delta: 3.086468"
>> "--- files: 61, delta: 3.106478"
>> "--- files: 61, delta: 3.325344"
>>
>> emacs-29, HEAD:
>> "--- files: 61, delta: 5.052421"
>> "--- files: 61, delta: 5.068663"
>> "--- files: 61, delta: 5.110502"
>>
>> FWIW, master
>> "--- files: 61, delta: 5.124109"
>> "--- files: 61, delta: 5.101217"
>> "--- files: 61, delta: 5.13068"
> 
> A completely expanded Lisp profile, after loading the Lisp files
> involved in this as *.el, would be appreciated.

I'm trying to bisect this, which turns out to be a bit tedious because 
many commits on emacs-29 don't build.  I'll come back when I have something.

Message #17 received at 65908 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Kangas <stefankangas <at> gmail.com>
To: Gerd Möllmann <gerd.moellmann <at> gmail.com>, 
 Eli Zaretskii <eliz <at> gnu.org>
Cc: Gregory Heytings <gregory <at> heytings.org>, 65908 <at> debbugs.gnu.org,
 ccsmile2008 <at> outlook.com
Subject: Re: bug#65908: 29.1.50; Emacs 29 regresses on macOS
Date: Wed, 4 Oct 2023 01:29:25 -0700

Gerd Möllmann <gerd.moellmann <at> gmail.com> writes:

> I'm trying to bisect this, which turns out to be a bit tedious because
> many commits on emacs-29 don't build.  I'll come back when I have something.

Does the script admin/git-bisect-start help?

Gregory will correct me if I'm wrong, but I think it's okay to add any
commits that don't build to the list there.

Message #20 received at 65908 <at> debbugs.gnu.org (full text, mbox):

From: Gerd Möllmann <gerd.moellmann <at> gmail.com>
To: Stefan Kangas <stefankangas <at> gmail.com>, Eli Zaretskii <eliz <at> gnu.org>
Cc: Gregory Heytings <gregory <at> heytings.org>, 65908 <at> debbugs.gnu.org,
 ccsmile2008 <at> outlook.com
Subject: Re: bug#65908: 29.1.50; Emacs 29 regresses on macOS
Date: Wed, 4 Oct 2023 10:32:22 +0200

On 04.10.23 10:29, Stefan Kangas wrote:
> Gerd Möllmann <gerd.moellmann <at> gmail.com> writes:
> 
>> I'm trying to bisect this, which turns out to be a bit tedious because
>> many commits on emacs-29 don't build.  I'll come back when I have something.
> 
> Does the script admin/git-bisect-start help?

Yes it helps.  Gregory set me on the right path already.  I had 
completely forgotten.

> 
> Gregory will correct me if I'm wrong, but I think it's okay to add any
> commits that don't build to the list there.

If that's okay, I already have one I could add.

Message #23 received at 65908 <at> debbugs.gnu.org (full text, mbox):

From: Gregory Heytings <gregory <at> heytings.org>
To: Stefan Kangas <stefankangas <at> gmail.com>
Cc: Gerd Möllmann <gerd.moellmann <at> gmail.com>,
 Eli Zaretskii <eliz <at> gnu.org>, 65908 <at> debbugs.gnu.org, ccsmile2008 <at> outlook.com
Subject: Re: bug#65908: 29.1.50; Emacs 29 regresses on macOS
Date: Wed, 04 Oct 2023 09:02:14 +0000

>
> Gregory will correct me if I'm wrong, but I think it's okay to add any 
> commits that don't build to the list there.
>

It's okay in principle, but the idea of that script is to include only 
commits that don't build on GNU/Linux, with GCC, and with the default 
build options (just "make").  I try to update the list of failing commits 
monthly.

Message #26 received at 65908 <at> debbugs.gnu.org (full text, mbox):

From: Gerd Möllmann <gerd.moellmann <at> gmail.com>
To: Gregory Heytings <gregory <at> heytings.org>,
 Stefan Kangas <stefankangas <at> gmail.com>
Cc: Eli Zaretskii <eliz <at> gnu.org>, 65908 <at> debbugs.gnu.org,
 ccsmile2008 <at> outlook.com
Subject: Re: bug#65908: 29.1.50; Emacs 29 regresses on macOS
Date: Wed, 4 Oct 2023 11:12:16 +0200

On 04.10.23 11:02, Gregory Heytings wrote:
> 
>>
>> Gregory will correct me if I'm wrong, but I think it's okay to add any 
>> commits that don't build to the list there.
>>
> 
> It's okay in principle, but the idea of that script is to include only 
> commits that don't build on GNU/Linux, with GCC, and with the default 
> build options (just "make").  I try to update the list of failing 
> commits monthly.
> 
That's okay, I can roll my own.

Message #29 received at 65908 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Kangas <stefankangas <at> gmail.com>
To: Gregory Heytings <gregory <at> heytings.org>
Cc: Gerd Möllmann <gerd.moellmann <at> gmail.com>,
 Eli Zaretskii <eliz <at> gnu.org>, 65908 <at> debbugs.gnu.org, ccsmile2008 <at> outlook.com
Subject: Re: bug#65908: 29.1.50; Emacs 29 regresses on macOS
Date: Wed, 4 Oct 2023 02:12:32 -0700

Gregory Heytings <gregory <at> heytings.org> writes:

>> Gregory will correct me if I'm wrong, but I think it's okay to add any
>> commits that don't build to the list there.
>
> It's okay in principle, but the idea of that script is to include only
> commits that don't build on GNU/Linux, with GCC, and with the default
> build options (just "make").

Thanks, I now see that explained a bit further down in the file.

Perhaps that explanation could be moved to the very top of the file?

> I try to update the list of failing commits monthly.

Thanks for maintaining that script.  We need more people showing that
kind of initiative and drive.

BTW, if you have any scripts that you use to update the list, it might
be useful to add them to emacs.git.  Perhaps just as a comment to
git-bisect-start, depending on how long it is.  That would decrease the
bus factor.

Message #32 received at 65908 <at> debbugs.gnu.org (full text, mbox):

From: Gerd Möllmann <gerd.moellmann <at> gmail.com>
To: Stefan Kangas <stefankangas <at> gmail.com>
Cc: Gregory Heytings <gregory <at> heytings.org>, 65908 <at> debbugs.gnu.org,
 Eli Zaretskii <eliz <at> gnu.org>, Alan Third <alan <at> idiocy.org>,
 ccsmile2008 <at> outlook.com
Subject: Re: bug#65908: 29.1.50; Emacs 29 regresses on macOS
Date: Wed, 04 Oct 2023 12:55:55 +0200

Stefan Kangas <stefankangas <at> gmail.com> writes:

>> I try to update the list of failing commits monthly.
>
> Thanks for maintaining that script.  We need more people showing that
> kind of initiative and drive.

+1

And, three weeks later, the culprit is

commit 058c012f73d4abe014ace44b46c23babd48aebbc (HEAD)
Author: Alan Third <alan <at> idiocy.org>
Date:   Sun Nov 14 15:09:43 2021 +0000

    Only set LANG if the ID is valid
    
    * src/nsterm.m (ns_init_locale): Check the provided locale identifier
    is available before trying to use it.

But don't ask me how that comes into play here because that function
seems to be called from main only.  Maybe it's because LANG is now not
always set, but used somewhere else?

Added Alan Third to the CC.

Message #35 received at 65908 <at> debbugs.gnu.org (full text, mbox):

From: Gerd Möllmann <gerd.moellmann <at> gmail.com>
To: Stefan Kangas <stefankangas <at> gmail.com>
Cc: ccsmile2008 <at> outlook.com, Gregory Heytings <gregory <at> heytings.org>,
 65908 <at> debbugs.gnu.org, Eli Zaretskii <eliz <at> gnu.org>,
 Alan Third <alan <at> idiocy.org>
Subject: Re: bug#65908: 29.1.50; Emacs 29 regresses on macOS
Date: Wed, 04 Oct 2023 14:43:05 +0200

Gerd Möllmann <gerd.moellmann <at> gmail.com> writes:

> Stefan Kangas <stefankangas <at> gmail.com> writes:
>
>>> I try to update the list of failing commits monthly.
>>
>> Thanks for maintaining that script.  We need more people showing that
>> kind of initiative and drive.
>
> +1
>
> And, three weeks later, the culprit is
>
> commit 058c012f73d4abe014ace44b46c23babd48aebbc (HEAD)
> Author: Alan Third <alan <at> idiocy.org>
> Date:   Sun Nov 14 15:09:43 2021 +0000
>
>     Only set LANG if the ID is valid
>     
>     * src/nsterm.m (ns_init_locale): Check the provided locale identifier
>     is available before trying to use it.
>
> But don't ask me how that comes into play here because that function
> seems to be called from main only.  Maybe it's because LANG is now not
> always set, but used somewhere else?
>
> Added Alan Third to the CC.

As an experiment, I evaluated this in *scratch*:

(setenv "LANG" "")
(setenv "LC_COLLATE" "C")
(setenv "LC_CTYPE" "UTF-8")
(setenv "LC_MESSAGES" "C")
(setenv "LC_MONETARY" "C")
(setenv "LC_NUMERIC" "C")
(setenv "LC_TIME" "C")

which is the output of 'locale' in a terminal, translated to Elisp.
With these settings, the slowdown is gone, without changing the code.

Message #38 received at 65908 <at> debbugs.gnu.org (full text, mbox):

From: Alan Third <alan <at> idiocy.org>
To: Gerd Möllmann <gerd.moellmann <at> gmail.com>
Cc: Gregory Heytings <gregory <at> heytings.org>, 65908 <at> debbugs.gnu.org,
 Eli Zaretskii <eliz <at> gnu.org>, Stefan Kangas <stefankangas <at> gmail.com>,
 ccsmile2008 <at> outlook.com
Subject: Re: bug#65908: 29.1.50; Emacs 29 regresses on macOS
Date: Wed, 4 Oct 2023 22:16:22 +0100

On Wed, Oct 04, 2023 at 02:43:05PM +0200, Gerd Möllmann wrote:
> Gerd Möllmann <gerd.moellmann <at> gmail.com> writes:
> 
> > Stefan Kangas <stefankangas <at> gmail.com> writes:
> >
> >>> I try to update the list of failing commits monthly.
> >>
> >> Thanks for maintaining that script.  We need more people showing that
> >> kind of initiative and drive.
> >
> > +1
> >
> > And, three weeks later, the culprit is
> >
> > commit 058c012f73d4abe014ace44b46c23babd48aebbc (HEAD)
> > Author: Alan Third <alan <at> idiocy.org>
> > Date:   Sun Nov 14 15:09:43 2021 +0000
> >
> >     Only set LANG if the ID is valid
> >     
> >     * src/nsterm.m (ns_init_locale): Check the provided locale identifier
> >     is available before trying to use it.
> >
> > But don't ask me how that comes into play here because that function
> > seems to be called from main only.  Maybe it's because LANG is now not
> > always set, but used somewhere else?

AFAIK LANG isn't used by Emacs at all. That code is just there to help
set up a default environment for running shell commands.

> As an experiment, I evaluated this in *scratch*:
> 
> (setenv "LANG" "")
> (setenv "LC_COLLATE" "C")
> (setenv "LC_CTYPE" "UTF-8")
> (setenv "LC_MESSAGES" "C")
> (setenv "LC_MONETARY" "C")
> (setenv "LC_NUMERIC" "C")
> (setenv "LC_TIME" "C")
> 
> which is the output of 'locale' in a terminal, translated to Elisp.
> With these settings, the slowdown is gone, without changing the code.

So is Emacs run in the terminal with a LANG of something like
en_GB.UTF-8 slower too? Because iirc my mac's terminal doesn't default
to 'C' and I don't see anything GUI specific in the test code...

FWIW I don't see any meaningful difference using different locale
settings on my Debian box, which makes me wonder if there is some
low-level darwin code that reads the locale from the environment. Any
idea if it's loading the files or stepping through the defuns that's
slower, or both?

Long story short: I have no idea what's going on here.
-- 
Alan Third

Message #41 received at 65908 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Alan Third <alan <at> idiocy.org>
Cc: gerd.moellmann <at> gmail.com, gregory <at> heytings.org, 65908 <at> debbugs.gnu.org,
 stefankangas <at> gmail.com, ccsmile2008 <at> outlook.com
Subject: Re: bug#65908: 29.1.50; Emacs 29 regresses on macOS
Date: Thu, 05 Oct 2023 08:41:28 +0300

> Date: Wed, 4 Oct 2023 22:16:22 +0100
> From: Alan Third <alan <at> idiocy.org>
> Cc: Stefan Kangas <stefankangas <at> gmail.com>,
> 	Gregory Heytings <gregory <at> heytings.org>, 65908 <at> debbugs.gnu.org,
> 	Eli Zaretskii <eliz <at> gnu.org>, ccsmile2008 <at> outlook.com
> 
> On Wed, Oct 04, 2023 at 02:43:05PM +0200, Gerd Möllmann wrote:
> > Gerd Möllmann <gerd.moellmann <at> gmail.com> writes:
> > 
> > > Stefan Kangas <stefankangas <at> gmail.com> writes:
> > >
> > >>> I try to update the list of failing commits monthly.
> > >>
> > >> Thanks for maintaining that script.  We need more people showing that
> > >> kind of initiative and drive.
> > >
> > > +1
> > >
> > > And, three weeks later, the culprit is
> > >
> > > commit 058c012f73d4abe014ace44b46c23babd48aebbc (HEAD)
> > > Author: Alan Third <alan <at> idiocy.org>
> > > Date:   Sun Nov 14 15:09:43 2021 +0000
> > >
> > >     Only set LANG if the ID is valid
> > >     
> > >     * src/nsterm.m (ns_init_locale): Check the provided locale identifier
> > >     is available before trying to use it.
> > >
> > > But don't ask me how that comes into play here because that function
> > > seems to be called from main only.  Maybe it's because LANG is now not
> > > always set, but used somewhere else?
> 
> AFAIK LANG isn't used by Emacs at all. That code is just there to help
> set up a default environment for running shell commands.
> 
> > As an experiment, I evaluated this in *scratch*:
> > 
> > (setenv "LANG" "")
> > (setenv "LC_COLLATE" "C")
> > (setenv "LC_CTYPE" "UTF-8")
> > (setenv "LC_MESSAGES" "C")
> > (setenv "LC_MONETARY" "C")
> > (setenv "LC_NUMERIC" "C")
> > (setenv "LC_TIME" "C")
> > 
> > which is the output of 'locale' in a terminal, translated to Elisp.
> > With these settings, the slowdown is gone, without changing the code.
> 
> So is Emacs run in the terminal with a LANG of something like
> en_GB.UTF-8 slower too? Because iirc my mac's terminal doesn't default
> to 'C' and I don't see anything GUI specific in the test code...
> 
> FWIW I don't see any meaningful difference using different locale
> settings on my Debian box, which makes me wonder if there is some
> low-level darwin code that reads the locale from the environment. Any
> idea if it's loading the files or stepping through the defuns that's
> slower, or both?
> 
> Long story short: I have no idea what's going on here.

Crystal ball says that the UTF-8 locale turns on some expensive
processing where it doesn't happen with the C locale.  One potential
place is case-insensitive comparisons and such likes.

Selectively setting only some of the LC_* variables limits the
expensive processing only to some operations.

In general, Emacs only needs to locale's settings to decide which
locale-environment and language-environment to turn on, all the rest
we do by ourselves.

Message #44 received at 65908 <at> debbugs.gnu.org (full text, mbox):

From: Gerd Möllmann <gerd.moellmann <at> gmail.com>
To: Alan Third <alan <at> idiocy.org>
Cc: Gregory Heytings <gregory <at> heytings.org>, 65908 <at> debbugs.gnu.org,
 Eli Zaretskii <eliz <at> gnu.org>, Stefan Kangas <stefankangas <at> gmail.com>,
 ccsmile2008 <at> outlook.com
Subject: Re: bug#65908: 29.1.50; Emacs 29 regresses on macOS
Date: Thu, 05 Oct 2023 07:55:55 +0200

Alan Third <alan <at> idiocy.org> writes:

>> which is the output of 'locale' in a terminal, translated to Elisp.
>> With these settings, the slowdown is gone, without changing the code.
>
> So is Emacs run in the terminal with a LANG of something like
> en_GB.UTF-8 slower too? Because iirc my mac's terminal doesn't default
> to 'C' and I don't see anything GUI specific in the test code...
>
> FWIW I don't see any meaningful difference using different locale
> settings on my Debian box, which makes me wonder if there is some
> low-level darwin code that reads the locale from the environment. Any
> idea if it's loading the files or stepping through the defuns that's
> slower, or both?
>
> Long story short: I have no idea what's going on here.

Me neither.  I tried to find something definitive about locales+bundles
Apple's docs, and on the Internet in general, but failed miserably.
What a mess.

Anyway, something is odd here, I think.

When I start 058c012f73d4abe014ace44b46c23babd48aebbc by double-clicking
Emacs.app, then M-x shell, I get

$ locale
LANG=""
LC_COLLATE="C"
LC_CTYPE="C"
LC_MESSAGES="C"
LC_MONETARY="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_ALL=

That can't be right, or is it?

The same, one commit before that:

$ locale
LANG="en_DE.UTF-8"
LC_COLLATE="C"
LC_CTYPE="C"
LC_MESSAGES="C"
LC_MONETARY="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_ALL=

At least LANG looks correct to me (I'm using an English UI, in DE.).
The LC_* are odd, though.

As an aside - vscode sets all of these vars to "en_US.UTF-8" which is
looks wrong, and the Zed editor does the same, except for LANG which it
set to "".

I must admit that I can't come to a conclusion here.  Maybe the right
thing would be LANG="en_DE.UTF-8" plus setting LC_ALL=$LANG like vscode?

Message #47 received at 65908 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Gerd Möllmann <gerd.moellmann <at> gmail.com>
Cc: alan <at> idiocy.org, gregory <at> heytings.org, 65908 <at> debbugs.gnu.org,
 stefankangas <at> gmail.com, ccsmile2008 <at> outlook.com
Subject: Re: bug#65908: 29.1.50; Emacs 29 regresses on macOS
Date: Thu, 05 Oct 2023 09:08:04 +0300

> From: Gerd Möllmann <gerd.moellmann <at> gmail.com>
> Cc: Stefan Kangas <stefankangas <at> gmail.com>,  Gregory Heytings
>  <gregory <at> heytings.org>,  65908 <at> debbugs.gnu.org,  Eli Zaretskii
>  <eliz <at> gnu.org>,  ccsmile2008 <at> outlook.com
> Date: Thu, 05 Oct 2023 07:55:55 +0200
> 
> When I start 058c012f73d4abe014ace44b46c23babd48aebbc by double-clicking
> Emacs.app, then M-x shell, I get
> 
> $ locale
> LANG=""
> LC_COLLATE="C"
> LC_CTYPE="C"
> LC_MESSAGES="C"
> LC_MONETARY="C"
> LC_NUMERIC="C"
> LC_TIME="C"
> LC_ALL=
> 
> That can't be right, or is it?
> 
> The same, one commit before that:
> 
> $ locale
> LANG="en_DE.UTF-8"
> LC_COLLATE="C"
> LC_CTYPE="C"
> LC_MESSAGES="C"
> LC_MONETARY="C"
> LC_NUMERIC="C"
> LC_TIME="C"
> LC_ALL=
> 
> At least LANG looks correct to me (I'm using an English UI, in DE.).
> The LC_* are odd, though.

Emacs forces LC_ALL= and LC_NUMERIC=C, to properly localize the
initial error messages and format numbers as in the "C" locale, see
'main' (which also has an NS-specific code).

Message #50 received at 65908 <at> debbugs.gnu.org (full text, mbox):

From: Gerd Möllmann <gerd.moellmann <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: alan <at> idiocy.org, gregory <at> heytings.org, 65908 <at> debbugs.gnu.org,
 stefankangas <at> gmail.com, ccsmile2008 <at> outlook.com
Subject: Re: bug#65908: 29.1.50; Emacs 29 regresses on macOS
Date: Thu, 05 Oct 2023 10:52:21 +0200

Eli Zaretskii <eliz <at> gnu.org> writes:

>> From: Gerd Möllmann <gerd.moellmann <at> gmail.com>
>> Cc: Stefan Kangas <stefankangas <at> gmail.com>,  Gregory Heytings
>>  <gregory <at> heytings.org>,  65908 <at> debbugs.gnu.org,  Eli Zaretskii
>>  <eliz <at> gnu.org>,  ccsmile2008 <at> outlook.com
>> Date: Thu, 05 Oct 2023 07:55:55 +0200
>>
>> When I start 058c012f73d4abe014ace44b46c23babd48aebbc by double-clicking
>> Emacs.app, then M-x shell, I get
>>
>> $ locale
>> LANG=""
>> LC_COLLATE="C"
>> LC_CTYPE="C"
>> LC_MESSAGES="C"
>> LC_MONETARY="C"
>> LC_NUMERIC="C"
>> LC_TIME="C"
>> LC_ALL=
>>
>> That can't be right, or is it?
>>
>> The same, one commit before that:
>>
>> $ locale
>> LANG="en_DE.UTF-8"
>> LC_COLLATE="C"
>> LC_CTYPE="C"
>> LC_MESSAGES="C"
>> LC_MONETARY="C"
>> LC_NUMERIC="C"
>> LC_TIME="C"
>> LC_ALL=
>>
>> At least LANG looks correct to me (I'm using an English UI, in DE.).
>> The LC_* are odd, though.
>
> Emacs forces LC_ALL= and LC_NUMERIC=C, to properly localize the
> initial error messages and format numbers as in the "C" locale, see
> 'main' (which also has an NS-specific code).

Thanks for the hint, Eli.

I've meanwhile found out something.  Not yet sure what to make of this,
but anyway:

1. In ns_init_locale, [locale localeIdentifier] returns "en_DE" on my
system, which is, I think, a consequence of my system settings, as I
mentioned (English UI in Germany).

2. man setlocale(3) says this

     An argument of "" will
     determine the name of the new locale taking into account the environment
     variables LANG and LC_*.

So, the setlocale (LC_ALL, "") in main should initialize LC_* according
to LANG, if set.  (Another question might be if that is what was
intended here, when I read the comment above that code, but be that as
it may.)

And another thing - any setlocale in ns_init_locale is moot because it
will be undone in main.

4. When I setenv LANG="en_DE.UTF-8" in ns_init_locale, the setlocale
(LC_ALL, "") returns NULL, indicating that LANG is invalid.  And, indeed

  $ locale -a | grep de
  de_CH
  de_DE.UTF-8
  de_AT.ISO8859-1
  de_AT.UTF-8
  de_AT.ISO8859-15
  de_DE.ISO8859-15
  de_CH.UTF-8
  de_CH.ISO8859-15
  de_DE.ISO8859-1
  de_CH.ISO8859-1
  de_AT
  de_DE

  There is no en_DE...

5. When I don't use localeIndentifier, but "en_US.UTF-8", for instance,
   the setlocale (LC_ALL, "") in main works like expected, and in a shell
   buffer:

  $ ~/emacs/master/src/ > locale
  LANG="en_US.UTF-8"
  LC_COLLATE="en_US.UTF-8"
  LC_CTYPE="UTF-8"
  LC_MESSAGES="en_US.UTF-8"
  LC_MONETARY="en_US.UTF-8"
  LC_NUMERIC="en_US.UTF-8"
  LC_TIME="en_US.UTF-8"
  LC_ALL=

That's all not very nice :-).

Conclusion so far, for me,

- don't selocale in ns_init_locale, only setenv LANG, or set LC_CTYPE,
  maybe.
- localeIndentifier is wrong for mixed cases like "en_DE".

Message #53 received at 65908 <at> debbugs.gnu.org (full text, mbox):

From: Gregory Heytings <gregory <at> heytings.org>
To: Stefan Kangas <stefankangas <at> gmail.com>
Cc: Gerd Möllmann <gerd.moellmann <at> gmail.com>,
 Eli Zaretskii <eliz <at> gnu.org>, 65908 <at> debbugs.gnu.org, ccsmile2008 <at> outlook.com
Subject: Re: bug#65908: 29.1.50; Emacs 29 regresses on macOS
Date: Thu, 05 Oct 2023 08:58:04 +0000

>> It's okay in principle, but the idea of that script is to include only 
>> commits that don't build on GNU/Linux, with GCC, and with the default 
>> build options (just "make").
>
> Thanks, I now see that explained a bit further down in the file.
>
> Perhaps that explanation could be moved to the very top of the file?
>

Done (cfcdb6ec2a).

>
> BTW, if you have any scripts that you use to update the list, it might 
> be useful to add them to emacs.git.  Perhaps just as a comment to 
> git-bisect-start, depending on how long it is.  That would decrease the 
> bus factor.
>

These scripts are almost trivial: for each not yet checked commit, do git 
checkout, make, and check the result.  The only slightly complex part is 
"for each not yet checked commit": it uses the list of all commits that I 
already checked (currently 21484, in a 860 KB text file), and I'm not sure 
adding that big file to emacs.git would be appropriate.

Message #56 received at 65908 <at> debbugs.gnu.org (full text, mbox):

From: Gregory Heytings <gregory <at> heytings.org>
To: Stefan Kangas <stefankangas <at> gmail.com>
Cc: Gerd Möllmann <gerd.moellmann <at> gmail.com>,
 Eli Zaretskii <eliz <at> gnu.org>, 65908 <at> debbugs.gnu.org, ccsmile2008 <at> outlook.com
Subject: Re: bug#65908: 29.1.50; Emacs 29 regresses on macOS
Date: Thu, 05 Oct 2023 09:03:09 +0000

>
> it uses the list of all commits that I already checked (currently 21484, 
> in a 860 KB text file)
>

Sorry: (currently 39928, in 1.9 MB text file).

Message #59 received at 65908 <at> debbugs.gnu.org (full text, mbox):

From: Alan Third <alan <at> idiocy.org>
To: Gerd Möllmann <gerd.moellmann <at> gmail.com>
Cc: Eli Zaretskii <eliz <at> gnu.org>, gregory <at> heytings.org, 65908 <at> debbugs.gnu.org,
 stefankangas <at> gmail.com, ccsmile2008 <at> outlook.com
Subject: Re: bug#65908: 29.1.50; Emacs 29 regresses on macOS
Date: Thu, 5 Oct 2023 10:48:23 +0100

On Thu, Oct 05, 2023 at 10:52:21AM +0200, Gerd Möllmann wrote:
> Conclusion so far, for me,
> 
> - don't selocale in ns_init_locale, only setenv LANG, or set LC_CTYPE,
>   maybe.

setlocale in ns_init_locale is only there to check if the locale is
valid, it resets it back to its initial value at the end.

> - localeIndentifier is wrong for mixed cases like "en_DE".

There was some trouble with warning messages in the terminal when
using an invalid locale ID (see bug#51321), but that may only be
because Daniel had changed the code to set LC_ALL instead of LANG.

I preferred setting LANG anyway because it acts as a fallback, and
this is pretty much intended as a fallback, anything else explicitly
set will override it.

Out of interest, when is it slow? Is it when we have an invalid locale
ID?
-- 
Alan Third

Message #62 received at 65908 <at> debbugs.gnu.org (full text, mbox):

From: Gerd Möllmann <gerd.moellmann <at> gmail.com>
To: Alan Third <alan <at> idiocy.org>
Cc: Eli Zaretskii <eliz <at> gnu.org>, gregory <at> heytings.org, 65908 <at> debbugs.gnu.org,
 stefankangas <at> gmail.com, ccsmile2008 <at> outlook.com
Subject: Re: bug#65908: 29.1.50; Emacs 29 regresses on macOS
Date: Thu, 05 Oct 2023 13:12:10 +0200

Alan Third <alan <at> idiocy.org> writes:

> On Thu, Oct 05, 2023 at 10:52:21AM +0200, Gerd Möllmann wrote:
>> Conclusion so far, for me,
>> 
>> - don't selocale in ns_init_locale, only setenv LANG, or set LC_CTYPE,
>>   maybe.
>
> setlocale in ns_init_locale is only there to check if the locale is
> valid, it resets it back to its initial value at the end.

That's okay.  What I meant was setting the old locale at the end.

>> - localeIndentifier is wrong for mixed cases like "en_DE".
>
> There was some trouble with warning messages in the terminal when
> using an invalid locale ID (see bug#51321), but that may only be
> because Daniel had changed the code to set LC_ALL instead of LANG.

Thanks, I'll take a look.

>
> I preferred setting LANG anyway because it acts as a fallback, and
> this is pretty much intended as a fallback, anything else explicitly
> set will override it.

I think that's TRT, indeed.

>
> Out of interest, when is it slow? Is it when we have an invalid locale
> ID?

That's an interesting question--whatever I do on master, it's almost
equally slow :-).

It would be nice if I could get at least the right character encoding on
my system...  On the plus side, I at least know now what's happening,
although I don't know what to do.

BTW, @ccsmile: What are your locale settings?

Aprt from that, I'm currently trying to get profiling working on my old
Macbook Pro, to compare the profiles of the 2 commits on emacs-29 which
show the difference.  That could maybe answer your question.

Buhut... I can't currently get Instruments to work, which might be
related to the fact that I've updated my unsupported by Apple Macbook to
Sonoma with OCLP.  Buhut -- maybe the new thunderbolt/magsafe cable for
my display arrives soon, so that I can use my Mac mini again.  A lot of
construction sites ahead :-).

Message #65 received at 65908 <at> debbugs.gnu.org (full text, mbox):

From: Gerd Möllmann <gerd.moellmann <at> gmail.com>
To: Alan Third <alan <at> idiocy.org>
Cc: ccsmile2008 <at> outlook.com, Eli Zaretskii <eliz <at> gnu.org>,
 65908 <at> debbugs.gnu.org, gregory <at> heytings.org, stefankangas <at> gmail.com
Subject: Re: bug#65908: 29.1.50; Emacs 29 regresses on macOS
Date: Thu, 05 Oct 2023 15:18:42 +0200

[Message part 1 (text/plain, inline)]

Gerd Möllmann <gerd.moellmann <at> gmail.com> writes:

> It would be nice if I could get at least the right character encoding on
> my system...  On the plus side, I at least know now what's happening,
> although I don't know what to do.

I'd like to propose the attached patch for problem of invalid LANG
setting.  This simply falls back to "en_US.UTF-8" if LANG isn't valid,
or something valid can't be deduced from the locale.

I find this acceptable even in my setting ("en_DE") because even Apple's
Terminal doesn't get this right.  A closer looks at what is set in the
locale in Terminal shows that, for instance, date format and thousands
separator are en_US anyway.

[0001-Handle-LANG-on-macOS-differntly-bug-bug-65908.patch (text/x-patch, inline)]

From 93457632cd5f17de01ec291d888d2d2fb0acffdf Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Gerd=20M=C3=B6llmann?= <gerd <at> gnu.org>
Date: Thu, 5 Oct 2023 15:12:40 +0200
Subject: [PATCH] Handle LANG on macOS differntly (bug#bug#65908)

* src/nsterm.m (ns_init_locale): If LANG is set, try to use that,
otherwise try to deduce what LANG should be.  Check is the result is
valid, and use LANG="en_US.UTF-8" if not.
---
 src/nsterm.m | 32 ++++++++++++++------------------
 1 file changed, 14 insertions(+), 18 deletions(-)

diff --git a/src/nsterm.m b/src/nsterm.m
index 4e0dfa58c63..09cb7236b6e 100644
--- a/src/nsterm.m
+++ b/src/nsterm.m
@@ -554,29 +554,25 @@ - (unsigned long)unsignedLong
 /* macOS doesn't set any environment variables for the locale when run
    from the GUI. Get the locale from the OS and set LANG.  */
 {
-  NSLocale *locale = [NSLocale currentLocale];
-
   NSTRACE ("ns_init_locale");
 
-  /* If we were run from a terminal then assume an unset LANG variable
-     is intentional and don't try to "fix" it.  */
-  if (!isatty (STDIN_FILENO))
+  const char *lang = getenv ("LANG");
+  if (lang == NULL)
     {
-      char *oldLocale = setlocale (LC_ALL, NULL);
-      /* It seems macOS should probably use UTF-8 everywhere.
-         'localeIdentifier' does not specify the encoding, and I can't
-         find any way to get the OS to tell us which encoding to use,
-         so hard-code '.UTF-8'.  */
-      NSString *localeID = [NSString stringWithFormat:@"%@.UTF-8",
-                                     [locale localeIdentifier]];
+      const NSLocale *locale = [NSLocale currentLocale];
+      const NSString *localeID = [NSString stringWithFormat:@"%@.UTF-8",
+					   [locale localeIdentifier]];
+      lang = [localeID UTF8String];
+    }
 
-      /* Check the locale ID is valid and if so set LANG, but not if
-         it is already set.  */
-      if (setlocale (LC_ALL, [localeID UTF8String]))
-        setenv("LANG", [localeID UTF8String], 0);
+  /* Check if LANG can be used for initializing the locale.  If not,
+     use a default setting.  Note that Emacs' main will undo the
+     setlocale below, initializing the locale from the
+     environment.  */
+  if (setlocale (LC_ALL, lang) == NULL)
+    lang = "en_US.UTF-8";
 
-      setlocale (LC_ALL, oldLocale);
-    }
+  setenv ("LANG", lang, 1);
 }
 
 
-- 
2.39.3 (Apple Git-145)

Message #68 received at 65908 <at> debbugs.gnu.org (full text, mbox):

From: Alan Third <alan <at> idiocy.org>
To: Gerd Möllmann <gerd.moellmann <at> gmail.com>
Cc: ccsmile2008 <at> outlook.com, Eli Zaretskii <eliz <at> gnu.org>,
 65908 <at> debbugs.gnu.org, gregory <at> heytings.org, stefankangas <at> gmail.com
Subject: Re: bug#65908: 29.1.50; Emacs 29 regresses on macOS
Date: Thu, 5 Oct 2023 14:51:58 +0100

On Thu, Oct 05, 2023 at 01:12:10PM +0200, Gerd Möllmann wrote:
> Alan Third <alan <at> idiocy.org> writes:
> 
> > On Thu, Oct 05, 2023 at 10:52:21AM +0200, Gerd Möllmann wrote:
> >> Conclusion so far, for me,
> >> 
> >> - don't selocale in ns_init_locale, only setenv LANG, or set LC_CTYPE,
> >>   maybe.
> >
> > setlocale in ns_init_locale is only there to check if the locale is
> > valid, it resets it back to its initial value at the end.
> 
> That's okay.  What I meant was setting the old locale at the end.

I've managed to reproduce this, and the time difference on my machine
is ridiculous. From 4 seconds without the setlocale calls to 24
seconds with.

And that's with a legitimate, installed, locale ID in both cases.

I wonder if this is why some people report Emacs 29 on macOS to be
noticeably slower. They, of course, blame the graphics changes, but
with that sort of variation in time...

> > Out of interest, when is it slow? Is it when we have an invalid locale
> > ID?
> 
> That's an interesting question--whatever I do on master, it's almost
> equally slow :-).
> 
> It would be nice if I could get at least the right character encoding on
> my system...  On the plus side, I at least know now what's happening,
> although I don't know what to do.

Indeed. I'm tempted to say we should revert the change since I now
suspect I misunderstood how setlocale works when I wrote this and I'm
not sure it's actually improving anything.

OTOH, it would be good if we could get this to actually work and
generate correct locale ID's for the system... But then, I don't know
what *is* a correct locale ID for a system like yours...

> Aprt from that, I'm currently trying to get profiling working on my old
> Macbook Pro, to compare the profiles of the 2 commits on emacs-29 which
> show the difference.  That could maybe answer your question.
> 
> Buhut... I can't currently get Instruments to work, which might be
> related to the fact that I've updated my unsupported by Apple Macbook to
> Sonoma with OCLP.  Buhut -- maybe the new thunderbolt/magsafe cable for
> my display arrives soon, so that I can use my Mac mini again.  A lot of
> construction sites ahead :-).

I couldn't even get the old commit to build, I had to use the current
master and just revert to see the difference. So best of luck! :)
-- 
Alan Third

Message #71 received at 65908 <at> debbugs.gnu.org (full text, mbox):

From: Gerd Möllmann <gerd.moellmann <at> gmail.com>
To: Alan Third <alan <at> idiocy.org>
Cc: ccsmile2008 <at> outlook.com, Eli Zaretskii <eliz <at> gnu.org>,
 65908 <at> debbugs.gnu.org, gregory <at> heytings.org, stefankangas <at> gmail.com
Subject: Re: bug#65908: 29.1.50; Emacs 29 regresses on macOS
Date: Thu, 05 Oct 2023 15:56:11 +0200

[Message part 1 (text/plain, inline)]

Gerd Möllmann <gerd.moellmann <at> gmail.com> writes:

> Aprt from that, I'm currently trying to get profiling working on my old
> Macbook Pro, to compare the profiles of the 2 commits on emacs-29 which
> show the difference.  That could maybe answer your question.

To the question of profiling.  I've now used the built-in profiler on
the two commits, but I must say that I don't see where the difference
comes from.  These are debug builds, 5 runs of the benchmark.

[profile-slow.txt (text/plain, attachment)]

[profile-fast.txt (text/plain, attachment)]

Message #74 received at 65908 <at> debbugs.gnu.org (full text, mbox):

From: Gerd Möllmann <gerd.moellmann <at> gmail.com>
To: Alan Third <alan <at> idiocy.org>
Cc: ccsmile2008 <at> outlook.com, Eli Zaretskii <eliz <at> gnu.org>,
 65908 <at> debbugs.gnu.org, gregory <at> heytings.org, stefankangas <at> gmail.com
Subject: Re: bug#65908: 29.1.50; Emacs 29 regresses on macOS
Date: Thu, 05 Oct 2023 16:01:52 +0200

Alan Third <alan <at> idiocy.org> writes:

>> That's okay.  What I meant was setting the old locale at the end.
>
> I've managed to reproduce this, and the time difference on my machine
> is ridiculous. From 4 seconds without the setlocale calls to 24
> seconds with.
>

Holy cow!  You mean you removed the setlocale at the end, or what did
you do?

> And that's with a legitimate, installed, locale ID in both cases.
>
> I wonder if this is why some people report Emacs 29 on macOS to be
> noticeably slower. They, of course, blame the graphics changes, but
> with that sort of variation in time...
>
>> > Out of interest, when is it slow? Is it when we have an invalid locale
>> > ID?
>> 
>> That's an interesting question--whatever I do on master, it's almost
>> equally slow :-).
>> 
>> It would be nice if I could get at least the right character encoding on
>> my system...  On the plus side, I at least know now what's happening,
>> although I don't know what to do.
>
> Indeed. I'm tempted to say we should revert the change since I now
> suspect I misunderstood how setlocale works when I wrote this and I'm
> not sure it's actually improving anything.
>
> OTOH, it would be good if we could get this to actually work and
> generate correct locale ID's for the system... But then, I don't know
> what *is* a correct locale ID for a system like yours...

Yeah :-(.  AS I wrote in a different mail, even Terminal doesn't do it
"right".  And I can't even find anywhere what would be Apple's preferred
way of doing this.

>> Aprt from that, I'm currently trying to get profiling working on my old
>> Macbook Pro, to compare the profiles of the 2 commits on emacs-29 which
>> show the difference.  That could maybe answer your question.
>> 
>> Buhut... I can't currently get Instruments to work, which might be
>> related to the fact that I've updated my unsupported by Apple Macbook to
>> Sonoma with OCLP.  Buhut -- maybe the new thunderbolt/magsafe cable for
>> my display arrives soon, so that I can use my Mac mini again.  A lot of
>> construction sites ahead :-).
>
> I couldn't even get the old commit to build, I had to use the current
> master and just revert to see the difference. So best of luck! :)

Thanks :-).

Message #77 received at 65908 <at> debbugs.gnu.org (full text, mbox):

From: Gerd Möllmann <gerd.moellmann <at> gmail.com>
To: Alan Third <alan <at> idiocy.org>
Cc: ccsmile2008 <at> outlook.com, Eli Zaretskii <eliz <at> gnu.org>,
 65908 <at> debbugs.gnu.org, gregory <at> heytings.org, stefankangas <at> gmail.com
Subject: Re: bug#65908: 29.1.50; Emacs 29 regresses on macOS
Date: Thu, 05 Oct 2023 16:09:36 +0200

Alan Third <alan <at> idiocy.org> writes:

> Indeed. I'm tempted to say we should revert the change since I now
> suspect I misunderstood how setlocale works when I wrote this and I'm
> not sure it's actually improving anything.

I hadn't checked this before, but with the patch I sent, the slowness is
indeed gone.  What a bloody mess, this stuff :-).

Message #80 received at 65908 <at> debbugs.gnu.org (full text, mbox):

From: Alan Third <alan <at> idiocy.org>
To: Gerd Möllmann <gerd.moellmann <at> gmail.com>
Cc: ccsmile2008 <at> outlook.com, Eli Zaretskii <eliz <at> gnu.org>,
 65908 <at> debbugs.gnu.org, gregory <at> heytings.org, stefankangas <at> gmail.com
Subject: Re: bug#65908: 29.1.50; Emacs 29 regresses on macOS
Date: Thu, 5 Oct 2023 15:58:54 +0100

On Thu, Oct 05, 2023 at 04:09:36PM +0200, Gerd Möllmann wrote:
> Alan Third <alan <at> idiocy.org> writes:
> 
> > Indeed. I'm tempted to say we should revert the change since I now
> > suspect I misunderstood how setlocale works when I wrote this and I'm
> > not sure it's actually improving anything.
> 
> I hadn't checked this before, but with the patch I sent, the slowness is
> indeed gone.  What a bloody mess, this stuff :-).

Yeah.

To answer your question in the other email, I saw the improvement only
by removing the setlocale in the "if" statement. Your code has
something similar, but it shows the speed improvement anyway, so I'm
not sure what's happening differently.

I've gone over the NSLocale documentation and it appears it explicitly
supports locales like en_DE, but the "UNIX" side of macOS doesn't. I
suppose it should be possible to generate your own en_DE locale, but
that's a bit much, really.

I also prodded at the fallback "system" locale, but it doesn't even
provide a locale ID, so it's useless.


There's a couple of typos in your commit message.

  Handle LANG on macOS differntly (bug#bug#65908)
                             ^e    ^

-- 
Alan Third

Message #83 received at 65908 <at> debbugs.gnu.org (full text, mbox):

From: Gerd Möllmann <gerd.moellmann <at> gmail.com>
To: Alan Third <alan <at> idiocy.org>
Cc: ccsmile2008 <at> outlook.com, Eli Zaretskii <eliz <at> gnu.org>,
 65908 <at> debbugs.gnu.org, gregory <at> heytings.org, stefankangas <at> gmail.com
Subject: Re: bug#65908: 29.1.50; Emacs 29 regresses on macOS
Date: Thu, 05 Oct 2023 17:11:09 +0200

Alan Third <alan <at> idiocy.org> writes:

> On Thu, Oct 05, 2023 at 04:09:36PM +0200, Gerd Möllmann wrote:
>> Alan Third <alan <at> idiocy.org> writes:
>> 
>> > Indeed. I'm tempted to say we should revert the change since I now
>> > suspect I misunderstood how setlocale works when I wrote this and I'm
>> > not sure it's actually improving anything.
>> 
>> I hadn't checked this before, but with the patch I sent, the slowness is
>> indeed gone.  What a bloody mess, this stuff :-).
>
> Yeah.
>
> To answer your question in the other email, I saw the improvement only
> by removing the setlocale in the "if" statement. Your code has
> something similar, but it shows the speed improvement anyway, so I'm
> not sure what's happening differently.

I'm actually no longer sure I want to know :-).

> I've gone over the NSLocale documentation and it appears it explicitly
> supports locales like en_DE, but the "UNIX" side of macOS doesn't. I
> suppose it should be possible to generate your own en_DE locale, but
> that's a bit much, really.

Agree.  If someone wants to do this for some reason, more power to them.
They could then set LANG to it.  I won't do it for my case because I
don't have TUI appls that require it, AFAIK.

> I also prodded at the fallback "system" locale, but it doesn't even
> provide a locale ID, so it's useless.
>
>
> There's a couple of typos in your commit message.
>
>   Handle LANG on macOS differntly (bug#bug#65908)
>                              ^e    ^

Oops, thanks for spotting this.

Remains the question if that should go into emacs-29?  I'm for not
rocking the boat too much.  Although...

Message #86 received at 65908 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Gerd Möllmann <gerd.moellmann <at> gmail.com>
Cc: ccsmile2008 <at> outlook.com, alan <at> idiocy.org, 65908 <at> debbugs.gnu.org,
 gregory <at> heytings.org, stefankangas <at> gmail.com
Subject: Re: bug#65908: 29.1.50; Emacs 29 regresses on macOS
Date: Thu, 05 Oct 2023 19:11:01 +0300

> From: Gerd Möllmann <gerd.moellmann <at> gmail.com>
> Cc: Eli Zaretskii <eliz <at> gnu.org>,  gregory <at> heytings.org,
>   65908 <at> debbugs.gnu.org,  stefankangas <at> gmail.com,  ccsmile2008 <at> outlook.com
> Date: Thu, 05 Oct 2023 15:18:42 +0200
> 
> I'd like to propose the attached patch for problem of invalid LANG
> setting.  This simply falls back to "en_US.UTF-8" if LANG isn't valid,
> or something valid can't be deduced from the locale.

Why is it reasonable to force the UTF-8 codeset as the fallback?

What about having a variable exposed to Lisp that allows the users to
control the fallback value of LANG?

Message #89 received at 65908 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Gerd Möllmann <gerd.moellmann <at> gmail.com>
Cc: ccsmile2008 <at> outlook.com, alan <at> idiocy.org, 65908 <at> debbugs.gnu.org,
 gregory <at> heytings.org, stefankangas <at> gmail.com
Subject: Re: bug#65908: 29.1.50; Emacs 29 regresses on macOS
Date: Thu, 05 Oct 2023 19:19:20 +0300

> From: Gerd Möllmann <gerd.moellmann <at> gmail.com>
> Cc: Eli Zaretskii <eliz <at> gnu.org>,  gregory <at> heytings.org,
>   65908 <at> debbugs.gnu.org,  stefankangas <at> gmail.com,  ccsmile2008 <at> outlook.com
> Date: Thu, 05 Oct 2023 15:56:11 +0200
> 
> To the question of profiling.  I've now used the built-in profiler on
> the two commits, but I must say that I don't see where the difference
> comes from.  These are debug builds, 5 runs of the benchmark.

Are you sure you haven't profiled the same build?  The profiles look
almost identical.

Message #92 received at 65908 <at> debbugs.gnu.org (full text, mbox):

From: Alan Third <alan <at> idiocy.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: Gerd Möllmann <gerd.moellmann <at> gmail.com>,
 gregory <at> heytings.org, 65908 <at> debbugs.gnu.org, stefankangas <at> gmail.com,
 ccsmile2008 <at> outlook.com
Subject: Re: bug#65908: 29.1.50; Emacs 29 regresses on macOS
Date: Thu, 5 Oct 2023 17:26:46 +0100

On Thu, Oct 05, 2023 at 07:19:20PM +0300, Eli Zaretskii wrote:
> > From: Gerd Möllmann <gerd.moellmann <at> gmail.com>
> > Cc: Eli Zaretskii <eliz <at> gnu.org>,  gregory <at> heytings.org,
> >   65908 <at> debbugs.gnu.org,  stefankangas <at> gmail.com,  ccsmile2008 <at> outlook.com
> > Date: Thu, 05 Oct 2023 15:56:11 +0200
> > 
> > To the question of profiling.  I've now used the built-in profiler on
> > the two commits, but I must say that I don't see where the difference
> > comes from.  These are debug builds, 5 runs of the benchmark.
> 
> Are you sure you haven't profiled the same build?  The profiles look
> almost identical.

I'm wondering if that indicates the problem is in an externally called
application, like git. I notice VC is being used, and if I copy the
lisp directory into a non-VC'd directory and run the test there it's
significantly faster.

For some reason I can no longer replicate the problem though, so I
can't check this idea myself.
-- 
Alan Third

Message #95 received at 65908 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Gerd Möllmann <gerd.moellmann <at> gmail.com>
Cc: ccsmile2008 <at> outlook.com, alan <at> idiocy.org, 65908 <at> debbugs.gnu.org,
 gregory <at> heytings.org, stefankangas <at> gmail.com
Subject: Re: bug#65908: 29.1.50; Emacs 29 regresses on macOS
Date: Thu, 05 Oct 2023 19:29:58 +0300

> From: Gerd Möllmann <gerd.moellmann <at> gmail.com>
> Cc: Eli Zaretskii <eliz <at> gnu.org>,  gregory <at> heytings.org,
>   65908 <at> debbugs.gnu.org,  stefankangas <at> gmail.com,  ccsmile2008 <at> outlook.com
> Date: Thu, 05 Oct 2023 17:11:09 +0200
> 
> Remains the question if that should go into emacs-29?  I'm for not
> rocking the boat too much.  Although...

Let's see the final patch, and decide then.

Thanks.

Message #98 received at 65908 <at> debbugs.gnu.org (full text, mbox):

From: Alan Third <alan <at> idiocy.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: Gerd Möllmann <gerd.moellmann <at> gmail.com>,
 gregory <at> heytings.org, 65908 <at> debbugs.gnu.org, stefankangas <at> gmail.com,
 ccsmile2008 <at> outlook.com
Subject: Re: bug#65908: 29.1.50; Emacs 29 regresses on macOS
Date: Thu, 5 Oct 2023 17:33:56 +0100

On Thu, Oct 05, 2023 at 07:11:01PM +0300, Eli Zaretskii wrote:
> > From: Gerd Möllmann <gerd.moellmann <at> gmail.com>
> > Cc: Eli Zaretskii <eliz <at> gnu.org>,  gregory <at> heytings.org,
> >   65908 <at> debbugs.gnu.org,  stefankangas <at> gmail.com,  ccsmile2008 <at> outlook.com
> > Date: Thu, 05 Oct 2023 15:18:42 +0200
> > 
> > I'd like to propose the attached patch for problem of invalid LANG
> > setting.  This simply falls back to "en_US.UTF-8" if LANG isn't valid,
> > or something valid can't be deduced from the locale.
> 
> Why is it reasonable to force the UTF-8 codeset as the fallback?

Both the macOS terminals I have access to use UTF-8 as their default,
and I'm not sure if they even let you change it. Therefore it seemed
reasonable to me that the default for Emacs should also be UTF-8.

> What about having a variable exposed to Lisp that allows the users to
> control the fallback value of LANG?

Is this not too early in startup for that? If not then couldn't the
user just use setenv and avoid this whole thing?
-- 
Alan Third

Message #101 received at 65908 <at> debbugs.gnu.org (full text, mbox):

From: Gerd Möllmann <gerd.moellmann <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: ccsmile2008 <at> outlook.com, alan <at> idiocy.org, 65908 <at> debbugs.gnu.org,
 gregory <at> heytings.org, stefankangas <at> gmail.com
Subject: Re: bug#65908: 29.1.50; Emacs 29 regresses on macOS
Date: Thu, 05 Oct 2023 18:36:15 +0200

Eli Zaretskii <eliz <at> gnu.org> writes:

>> From: Gerd Möllmann <gerd.moellmann <at> gmail.com>
>> Cc: Eli Zaretskii <eliz <at> gnu.org>,  gregory <at> heytings.org,
>>   65908 <at> debbugs.gnu.org,  stefankangas <at> gmail.com,  ccsmile2008 <at> outlook.com
>> Date: Thu, 05 Oct 2023 15:56:11 +0200
>> 
>> To the question of profiling.  I've now used the built-in profiler on
>> the two commits, but I must say that I don't see where the difference
>> comes from.  These are debug builds, 5 runs of the benchmark.
>
> Are you sure you haven't profiled the same build?  The profiles look
> almost identical.

Yes, I'm sure.

Message #104 received at 65908 <at> debbugs.gnu.org (full text, mbox):

From: Gerd Möllmann <gerd.moellmann <at> gmail.com>
To: Alan Third <alan <at> idiocy.org>
Cc: Eli Zaretskii <eliz <at> gnu.org>, 65908 <at> debbugs.gnu.org, stefankangas <at> gmail.com,
 gregory <at> heytings.org, ccsmile2008 <at> outlook.com
Subject: Re: bug#65908: 29.1.50; Emacs 29 regresses on macOS
Date: Thu, 05 Oct 2023 18:38:33 +0200

Alan Third <alan <at> idiocy.org> writes:

> On Thu, Oct 05, 2023 at 07:11:01PM +0300, Eli Zaretskii wrote:
>> > From: Gerd Möllmann <gerd.moellmann <at> gmail.com>
>> > Cc: Eli Zaretskii <eliz <at> gnu.org>,  gregory <at> heytings.org,
>> >   65908 <at> debbugs.gnu.org,  stefankangas <at> gmail.com,  ccsmile2008 <at> outlook.com
>> > Date: Thu, 05 Oct 2023 15:18:42 +0200
>> > 
>> > I'd like to propose the attached patch for problem of invalid LANG
>> > setting.  This simply falls back to "en_US.UTF-8" if LANG isn't valid,
>> > or something valid can't be deduced from the locale.
>> 
>> Why is it reasonable to force the UTF-8 codeset as the fallback?
>
> Both the macOS terminals I have access to use UTF-8 as their default,
> and I'm not sure if they even let you change it. Therefore it seemed
> reasonable to me that the default for Emacs should also be UTF-8.

Agree.

>> What about having a variable exposed to Lisp that allows the users to
>> control the fallback value of LANG?
>
> Is this not too early in startup for that? If not then couldn't the
> user just use setenv and avoid this whole thing?

I wonder that, too.

Message #107 received at 65908 <at> debbugs.gnu.org (full text, mbox):

From: Gerd Möllmann <gerd.moellmann <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: ccsmile2008 <at> outlook.com, alan <at> idiocy.org, 65908 <at> debbugs.gnu.org,
 gregory <at> heytings.org, stefankangas <at> gmail.com
Subject: Re: bug#65908: 29.1.50; Emacs 29 regresses on macOS
Date: Thu, 05 Oct 2023 18:39:56 +0200

[Message part 1 (text/plain, inline)]

Eli Zaretskii <eliz <at> gnu.org> writes:

>> From: Gerd Möllmann <gerd.moellmann <at> gmail.com>
>> Cc: Eli Zaretskii <eliz <at> gnu.org>,  gregory <at> heytings.org,
>>   65908 <at> debbugs.gnu.org,  stefankangas <at> gmail.com,  ccsmile2008 <at> outlook.com
>> Date: Thu, 05 Oct 2023 17:11:09 +0200
>> 
>> Remains the question if that should go into emacs-29?  I'm for not
>> rocking the boat too much.  Although...
>
> Let's see the final patch, and decide then.

Please find attached.  Not much gas changed, though.

[0001-Handle-LANG-on-macOS-differently-bug-65908.patch (text/x-patch, inline)]

From f1ba5bbbd9526197052949525d2ac680337b6d6b Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Gerd=20M=C3=B6llmann?= <gerd <at> gnu.org>
Date: Thu, 5 Oct 2023 17:29:17 +0200
Subject: [PATCH] Handle LANG on macOS differently (bug#65908)

* src/nsterm.m (ns_init_locale): If LANG is set, try to use that,
otherwise try to deduce what LANG should be.  Check is the result is
valid, and use LANG="en_US.UTF-8" if not.
---
 src/nsterm.m | 39 +++++++++++++++++++++------------------
 1 file changed, 21 insertions(+), 18 deletions(-)

diff --git a/src/nsterm.m b/src/nsterm.m
index 4e0dfa58c63..56865987b93 100644
--- a/src/nsterm.m
+++ b/src/nsterm.m
@@ -554,29 +554,32 @@ - (unsigned long)unsignedLong
 /* macOS doesn't set any environment variables for the locale when run
    from the GUI. Get the locale from the OS and set LANG.  */
 {
-  NSLocale *locale = [NSLocale currentLocale];
-
   NSTRACE ("ns_init_locale");
 
-  /* If we were run from a terminal then assume an unset LANG variable
-     is intentional and don't try to "fix" it.  */
-  if (!isatty (STDIN_FILENO))
+  /* Either use LANG, if set, or try to construct LANG from
+     NSLocale.  */
+  const char *lang = getenv ("LANG");
+  if (lang == NULL)
     {
-      char *oldLocale = setlocale (LC_ALL, NULL);
-      /* It seems macOS should probably use UTF-8 everywhere.
-         'localeIdentifier' does not specify the encoding, and I can't
-         find any way to get the OS to tell us which encoding to use,
-         so hard-code '.UTF-8'.  */
-      NSString *localeID = [NSString stringWithFormat:@"%@.UTF-8",
-                                     [locale localeIdentifier]];
-
-      /* Check the locale ID is valid and if so set LANG, but not if
-         it is already set.  */
-      if (setlocale (LC_ALL, [localeID UTF8String]))
-        setenv("LANG", [localeID UTF8String], 0);
+      const NSLocale *locale = [NSLocale currentLocale];
+      const NSString *localeID = [NSString stringWithFormat:@"%@.UTF-8",
+					   [locale localeIdentifier]];
+      lang = [localeID UTF8String];
+    }
 
-      setlocale (LC_ALL, oldLocale);
+  /* Check if LANG can be used for initializing the locale.  If not,
+     use a default setting.  Note that Emacs' main will undo the
+     setlocale below, initializing the locale from the
+     environment.  */
+  if (setlocale (LC_ALL, lang) == NULL)
+    {
+      const char *const default_lang = "en_US.UTF-8";
+      fprintf (stderr, "LANG=%s cannot be used, using %s instead.\n",
+	       lang, default_lang);
+      lang = default_lang;
     }
+
+  setenv ("LANG", lang, 1);
 }
 
 
-- 
2.39.3 (Apple Git-145)

Message #110 received at 65908 <at> debbugs.gnu.org (full text, mbox):

From: Gerd Möllmann <gerd.moellmann <at> gmail.com>
To: Alan Third <alan <at> idiocy.org>
Cc: Eli Zaretskii <eliz <at> gnu.org>, 65908 <at> debbugs.gnu.org, stefankangas <at> gmail.com,
 gregory <at> heytings.org, ccsmile2008 <at> outlook.com
Subject: Re: bug#65908: 29.1.50; Emacs 29 regresses on macOS
Date: Thu, 05 Oct 2023 18:43:48 +0200

Alan Third <alan <at> idiocy.org> writes:

> On Thu, Oct 05, 2023 at 07:19:20PM +0300, Eli Zaretskii wrote:
>> > From: Gerd Möllmann <gerd.moellmann <at> gmail.com>
>> > Cc: Eli Zaretskii <eliz <at> gnu.org>,  gregory <at> heytings.org,
>> >   65908 <at> debbugs.gnu.org,  stefankangas <at> gmail.com,  ccsmile2008 <at> outlook.com
>> > Date: Thu, 05 Oct 2023 15:56:11 +0200
>> > 
>> > To the question of profiling.  I've now used the built-in profiler on
>> > the two commits, but I must say that I don't see where the difference
>> > comes from.  These are debug builds, 5 runs of the benchmark.
>> 
>> Are you sure you haven't profiled the same build?  The profiles look
>> almost identical.
>
> I'm wondering if that indicates the problem is in an externally called
> application, like git. I notice VC is being used, and if I copy the
> lisp directory into a non-VC'd directory and run the test there it's
> significantly faster.
>
> For some reason I can no longer replicate the problem though, so I
> can't check this idea myself.

It's ca. 50% faster when I copy the files somewhere else.

Message #113 received at 65908 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Alan Third <alan <at> idiocy.org>
Cc: gerd.moellmann <at> gmail.com, gregory <at> heytings.org, 65908 <at> debbugs.gnu.org,
 stefankangas <at> gmail.com, ccsmile2008 <at> outlook.com
Subject: Re: bug#65908: 29.1.50; Emacs 29 regresses on macOS
Date: Thu, 05 Oct 2023 20:02:41 +0300

> Date: Thu, 5 Oct 2023 17:33:56 +0100
> From: Alan Third <alan <at> idiocy.org>
> Cc: Gerd Möllmann <gerd.moellmann <at> gmail.com>,
> 	ccsmile2008 <at> outlook.com, 65908 <at> debbugs.gnu.org,
> 	gregory <at> heytings.org, stefankangas <at> gmail.com
> 
> On Thu, Oct 05, 2023 at 07:11:01PM +0300, Eli Zaretskii wrote:
> > > From: Gerd Möllmann <gerd.moellmann <at> gmail.com>
> > > Cc: Eli Zaretskii <eliz <at> gnu.org>,  gregory <at> heytings.org,
> > >   65908 <at> debbugs.gnu.org,  stefankangas <at> gmail.com,  ccsmile2008 <at> outlook.com
> > > Date: Thu, 05 Oct 2023 15:18:42 +0200
> > > 
> > > I'd like to propose the attached patch for problem of invalid LANG
> > > setting.  This simply falls back to "en_US.UTF-8" if LANG isn't valid,
> > > or something valid can't be deduced from the locale.
> > 
> > Why is it reasonable to force the UTF-8 codeset as the fallback?
> 
> Both the macOS terminals I have access to use UTF-8 as their default,
> and I'm not sure if they even let you change it. Therefore it seemed
> reasonable to me that the default for Emacs should also be UTF-8.

Are you saying that macOS doesn't support non-UTF-8 encodings?

> > What about having a variable exposed to Lisp that allows the users to
> > control the fallback value of LANG?
> 
> Is this not too early in startup for that? If not then couldn't the
> user just use setenv and avoid this whole thing?

Maybe that's fine, I just asked a question.

Message #116 received at 65908 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Gerd Möllmann <gerd.moellmann <at> gmail.com>
Cc: ccsmile2008 <at> outlook.com, alan <at> idiocy.org, 65908 <at> debbugs.gnu.org,
 gregory <at> heytings.org, stefankangas <at> gmail.com
Subject: Re: bug#65908: 29.1.50; Emacs 29 regresses on macOS
Date: Thu, 05 Oct 2023 20:04:02 +0300

> From: Gerd Möllmann <gerd.moellmann <at> gmail.com>
> Cc: alan <at> idiocy.org,  gregory <at> heytings.org,  65908 <at> debbugs.gnu.org,
>   stefankangas <at> gmail.com,  ccsmile2008 <at> outlook.com
> Date: Thu, 05 Oct 2023 18:36:15 +0200
> 
> Eli Zaretskii <eliz <at> gnu.org> writes:
> 
> >> From: Gerd Möllmann <gerd.moellmann <at> gmail.com>
> >> Cc: Eli Zaretskii <eliz <at> gnu.org>,  gregory <at> heytings.org,
> >>   65908 <at> debbugs.gnu.org,  stefankangas <at> gmail.com,  ccsmile2008 <at> outlook.com
> >> Date: Thu, 05 Oct 2023 15:56:11 +0200
> >> 
> >> To the question of profiling.  I've now used the built-in profiler on
> >> the two commits, but I must say that I don't see where the difference
> >> comes from.  These are debug builds, 5 runs of the benchmark.
> >
> > Are you sure you haven't profiled the same build?  The profiles look
> > almost identical.
> 
> Yes, I'm sure.

Then I guess the problem is not with the Emacs Lisp code, but either
with the primitives implemented in C or some programs (Git?) invoked
by this recipe.

Message #119 received at 65908 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: stefankangas <at> gmail.com, Gerd Möllmann
 <gerd.moellmann <at> gmail.com>
Cc: alan <at> idiocy.org, 65908 <at> debbugs.gnu.org, gregory <at> heytings.org,
 ccsmile2008 <at> outlook.com
Subject: Re: bug#65908: 29.1.50; Emacs 29 regresses on macOS
Date: Thu, 05 Oct 2023 20:06:51 +0300

> From: Gerd Möllmann <gerd.moellmann <at> gmail.com>
> Cc: alan <at> idiocy.org,  gregory <at> heytings.org,  65908 <at> debbugs.gnu.org,
>   stefankangas <at> gmail.com,  ccsmile2008 <at> outlook.com
> Date: Thu, 05 Oct 2023 18:39:56 +0200
> 
> Eli Zaretskii <eliz <at> gnu.org> writes:
> 
> >> From: Gerd Möllmann <gerd.moellmann <at> gmail.com>
> >> Cc: Eli Zaretskii <eliz <at> gnu.org>,  gregory <at> heytings.org,
> >>   65908 <at> debbugs.gnu.org,  stefankangas <at> gmail.com,  ccsmile2008 <at> outlook.com
> >> Date: Thu, 05 Oct 2023 17:11:09 +0200
> >> 
> >> Remains the question if that should go into emacs-29?  I'm for not
> >> rocking the boat too much.  Although...
> >
> > Let's see the final patch, and decide then.
> 
> Please find attached.  Not much gas changed, though.

If you-two consider this safe for Emacs 29.2, I don't mind.

Stefan?

Message #122 received at 65908 <at> debbugs.gnu.org (full text, mbox):

From: Gerd Möllmann <gerd.moellmann <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: ccsmile2008 <at> outlook.com, alan <at> idiocy.org, 65908 <at> debbugs.gnu.org,
 gregory <at> heytings.org, stefankangas <at> gmail.com
Subject: Re: bug#65908: 29.1.50; Emacs 29 regresses on macOS
Date: Thu, 05 Oct 2023 19:15:49 +0200

Eli Zaretskii <eliz <at> gnu.org> writes:

>> > Are you sure you haven't profiled the same build?  The profiles look
>> > almost identical.
>> 
>> Yes, I'm sure.
>
> Then I guess the problem is not with the Emacs Lisp code, but either
> with the primitives implemented in C or some programs (Git?) invoked
> by this recipe.

I find Alans's idea pretty compelling.  Alas, I don't know enough about
git's internals to tell what it does when locales aren't set up
right.  But it's very plausible, given the profiler output.

Message #125 received at 65908 <at> debbugs.gnu.org (full text, mbox):

From: Alan Third <alan <at> idiocy.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: gerd.moellmann <at> gmail.com, gregory <at> heytings.org, 65908 <at> debbugs.gnu.org,
 stefankangas <at> gmail.com, ccsmile2008 <at> outlook.com
Subject: Re: bug#65908: 29.1.50; Emacs 29 regresses on macOS
Date: Thu, 5 Oct 2023 18:15:57 +0100

On Thu, Oct 05, 2023 at 08:02:41PM +0300, Eli Zaretskii wrote:
> > Date: Thu, 5 Oct 2023 17:33:56 +0100
> > From: Alan Third <alan <at> idiocy.org>
> > Cc: Gerd Möllmann <gerd.moellmann <at> gmail.com>,
> > 	ccsmile2008 <at> outlook.com, 65908 <at> debbugs.gnu.org,
> > 	gregory <at> heytings.org, stefankangas <at> gmail.com
> > 
> > On Thu, Oct 05, 2023 at 07:11:01PM +0300, Eli Zaretskii wrote:
> > > > From: Gerd Möllmann <gerd.moellmann <at> gmail.com>
> > > > Cc: Eli Zaretskii <eliz <at> gnu.org>,  gregory <at> heytings.org,
> > > >   65908 <at> debbugs.gnu.org,  stefankangas <at> gmail.com,  ccsmile2008 <at> outlook.com
> > > > Date: Thu, 05 Oct 2023 15:18:42 +0200
> > > > 
> > > > I'd like to propose the attached patch for problem of invalid LANG
> > > > setting.  This simply falls back to "en_US.UTF-8" if LANG isn't valid,
> > > > or something valid can't be deduced from the locale.
> > > 
> > > Why is it reasonable to force the UTF-8 codeset as the fallback?
> > 
> > Both the macOS terminals I have access to use UTF-8 as their default,
> > and I'm not sure if they even let you change it. Therefore it seemed
> > reasonable to me that the default for Emacs should also be UTF-8.
> 
> Are you saying that macOS doesn't support non-UTF-8 encodings?

No, because of the slightly unusual way macOS works, where any
graphical application doesn't inherit much of an environment the
terminals have to set it up. I was just copying what the terminals do
by default. When I say I don't know if they let you change it, I mean
in their settings. You can change it in the shell and the locale
command here shows, for example:

# locale -a | grep en_GB
en_GB.ISO8859-1
en_GB.UTF-8
en_GB.US-ASCII
en_GB
en_GB.ISO8859-15

So we could skip the UTF-8 part if it's the wrong thing to do, but
since everything else in macOS is unicode aware, I suspect we should
keep it.

-- 
Alan Third

Message #128 received at 65908 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Kangas <stefankangas <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>,
 Gerd Möllmann <gerd.moellmann <at> gmail.com>
Cc: alan <at> idiocy.org, 65908 <at> debbugs.gnu.org, gregory <at> heytings.org,
 ccsmile2008 <at> outlook.com
Subject: Re: bug#65908: 29.1.50; Emacs 29 regresses on macOS
Date: Thu, 5 Oct 2023 18:08:45 +0000

Eli Zaretskii <eliz <at> gnu.org> writes:

> If you-two consider this safe for Emacs 29.2, I don't mind.
>
> Stefan?

No objections here, thanks.

Message #131 received at 65908 <at> debbugs.gnu.org (full text, mbox):

From: Alan Third <alan <at> idiocy.org>
To: Gerd Möllmann <gerd.moellmann <at> gmail.com>
Cc: ccsmile2008 <at> outlook.com, Eli Zaretskii <eliz <at> gnu.org>,
 65908 <at> debbugs.gnu.org, gregory <at> heytings.org, stefankangas <at> gmail.com
Subject: Re: bug#65908: 29.1.50; Emacs 29 regresses on macOS
Date: Thu, 5 Oct 2023 19:25:31 +0100

On Thu, Oct 05, 2023 at 07:15:49PM +0200, Gerd Möllmann wrote:
> Eli Zaretskii <eliz <at> gnu.org> writes:
> 
> >> > Are you sure you haven't profiled the same build?  The profiles look
> >> > almost identical.
> >> 
> >> Yes, I'm sure.
> >
> > Then I guess the problem is not with the Emacs Lisp code, but either
> > with the primitives implemented in C or some programs (Git?) invoked
> > by this recipe.
> 
> I find Alans's idea pretty compelling.  Alas, I don't know enough about
> git's internals to tell what it does when locales aren't set up
> right.  But it's very plausible, given the profiler output.

OK, with Gerd's latest patch installed, I set my shell thus:

    export LANG="en_GB.UTF-8"
    export LC_COLLATE=
    export LC_CTYPE=
    export LC_MESSAGES=
    export LC_MONETARY=
    export LC_NUMERIC=
    export LC_TIME=
    export LC_ALL=

and run:

    LANG=en_DE.UTF-8 src/emacs -Q

which prints:

    LANG=en_DE.UTF-8 cannot be used, using en_US.UTF-8 instead.

and sure enough "C-u M-! locale" gives:

    LANG="en_US.UTF-8"
    LC_COLLATE="en_US.UTF-8"
    LC_CTYPE="en_US.UTF-8"
    LC_MESSAGES="en_US.UTF-8"
    LC_MONETARY="en_US.UTF-8"
    LC_NUMERIC="en_US.UTF-8"
    LC_TIME="en_US.UTF-8"
    LC_ALL="en_US.UTF-8"

I then run the test script against the lisp directory in the emacs git repo:

    --- files: 61, delta: 4.428015
    Elapsed time: 4.432377s (0.161921s in 13 GCs)
    "--- files: 61, delta: 4.428015"
    --- files: 61, delta: 4.416348
    Elapsed time: 4.421116s (0.150580s in 12 GCs)
    "--- files: 61, delta: 4.416348"
    --- files: 61, delta: 4.391077
    Elapsed time: 4.395202s (0.150900s in 12 GCs)
    "--- files: 61, delta: 4.391077"

Then against the same directory copied elsewhere:

    --- files: 61, delta: 0.882561
    Elapsed time: 0.887944s (0.138922s in 11 GCs)
    "--- files: 61, delta: 0.882561"
    --- files: 61, delta: 0.882582
    Elapsed time: 0.887593s (0.139193s in 11 GCs)
    "--- files: 61, delta: 0.882582"
    --- files: 61, delta: 0.866825
    Elapsed time: 0.871124s (0.122224s in 10 GCs)
    "--- files: 61, delta: 0.866825"

Then I run:

    LANG= src/emacs -Q

and run the test against the lisp directory in the repo:

    --- files: 61, delta: 5.87121
    Elapsed time: 5.875535s (0.160997s in 13 GCs)
    "--- files: 61, delta: 5.87121"
    --- files: 61, delta: 5.867494
    Elapsed time: 5.871656s (0.150614s in 12 GCs)
    "--- files: 61, delta: 5.867494"
    --- files: 61, delta: 5.83144
    Elapsed time: 5.862261s (0.163723s in 13 GCs)
    "--- files: 61, delta: 5.83144"

and the lisp directory not in the lisp repo:

    --- files: 61, delta: 0.869081
    Elapsed time: 0.873759s (0.134711s in 11 GCs)
    "--- files: 61, delta: 0.869081"
    --- files: 61, delta: 0.873445
    Elapsed time: 0.878768s (0.133343s in 11 GCs)
    "--- files: 61, delta: 0.873445"
    --- files: 61, delta: 0.870359
    Elapsed time: 0.875465s (0.132767s in 11 GCs)
    "--- files: 61, delta: 0.870359"

So the time with no git repo are near identical, but the times with a
git repo vary by over a second.

When I do "LANG=" it's trying to set the local to the empty string
because we only check if LANG is null, which it's not, it's the empty
string (I think) and that's resulting in a locale like (C-u M-! locale):

    LANG=
    LC_COLLATE="C"
    LC_CTYPE="C"
    LC_MESSAGES="C"
    LC_MONETARY="C"
    LC_NUMERIC="C"
    LC_TIME="C"
    LC_ALL="C"

I don't think this is conclusive proof, but I think that either git or
VC have a problem with a slightly broken locale. I'm not saying
there's a bug here, it could simply be that we call git many times and
it's having to work out the locale each time from the dodgy input I've
given it.

(I tried en_GB without the UTF-8 coding and the times were pretty much
identical to the en_US.UTF-8 times.)

-- 
Alan Third

Message #134 received at 65908 <at> debbugs.gnu.org (full text, mbox):

From: Gerd Möllmann <gerd.moellmann <at> gmail.com>
To: Alan Third <alan <at> idiocy.org>
Cc: ccsmile2008 <at> outlook.com, Eli Zaretskii <eliz <at> gnu.org>,
 65908 <at> debbugs.gnu.org, gregory <at> heytings.org, stefankangas <at> gmail.com
Subject: Re: bug#65908: 29.1.50; Emacs 29 regresses on macOS
Date: Thu, 05 Oct 2023 20:33:02 +0200

Alan Third <alan <at> idiocy.org> writes:

> When I do "LANG=" it's trying to set the local to the empty string
> because we only check if LANG is null, which it's not, it's the empty
> string (I think) and that's resulting in a locale like (C-u M-! locale):
>
>     LANG=
>     LC_COLLATE="C"
>     LC_CTYPE="C"
>     LC_MESSAGES="C"
>     LC_MONETARY="C"
>     LC_NUMERIC="C"
>     LC_TIME="C"
>     LC_ALL="C"

That's a good point.  I should also check if LANG is set but empty,
because "" changes the meaning of setlocale.  I'll commit that in a
second :-).

> I don't think this is conclusive proof, but I think that either git or
> VC have a problem with a slightly broken locale. I'm not saying
> there's a bug here, it could simply be that we call git many times and
> it's having to work out the locale each time from the dodgy input I've
> given it.

Yeah, to me that makes a lot of sense.

Message #137 received at 65908 <at> debbugs.gnu.org (full text, mbox):

From: Gerd Möllmann <gerd.moellmann <at> gmail.com>
To: ccsmile2008 <at> outlook.com
Cc: Alan Third <alan <at> idiocy.org>, 65908 <at> debbugs.gnu.org,
 Eli Zaretskii <eliz <at> gnu.org>, gregory <at> heytings.org, stefankangas <at> gmail.com
Subject: Re: bug#65908: 29.1.50; Emacs 29 regresses on macOS
Date: Thu, 05 Oct 2023 20:46:43 +0200

Committed to emacs-29 as aad8b5d78f306ac9ca0c03734524c9f49585bee8.

I'll close this bug, please reopen if it's not fixed for you in 29.2.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.