Received: (at 61285) by debbugs.gnu.org; 6 Feb 2023 00:08:57 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Sun Feb 05 19:08:57 2023 Received: from localhost ([127.0.0.1]:46758 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1pOp44-0006xw-Kz for submit <at> debbugs.gnu.org; Sun, 05 Feb 2023 19:08:57 -0500 Received: from mail-pj1-f41.google.com ([209.85.216.41]:37837) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <casouri@HIDDEN>) id 1pOp42-0006xh-7S for 61285 <at> debbugs.gnu.org; Sun, 05 Feb 2023 19:08:54 -0500 Received: by mail-pj1-f41.google.com with SMTP id c10-20020a17090a1d0a00b0022e63a94799so13651259pjd.2 for <61285 <at> debbugs.gnu.org>; Sun, 05 Feb 2023 16:08:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=to:cc:date:message-id:subject:mime-version :content-transfer-encoding:from:from:to:cc:subject:date:message-id :reply-to; bh=7nJV3m2659JxdgFr8rMpzKKUXncrIfhrKeT4r7KqLdc=; b=a9Y1V/DR0jT7LUIm+u6T9Ju2ugQCitkwh/z/7WoY+sD1qVFj78qAbMZe/ri1mr7f5c LsrZtmf0rNPt/MxVa9kWDFRm9QbxxMaiBq0rGibaUT28AoV0T7l1hWVy5rR1tdN9QzCb 6nbnDeqQvf5A5sOKVZH7Zte1icXIrBJOIA0PsMAEFxUSUbGaChShcagye4Lw0dOiopCE bfuRIrTUvkKrK8Fnoe3AbEPf7JKsixtLpiIenbYzFiHPRbzmUQ6bvFjX+WjgSnhxdUKE FhHc93TePUJce9QKcvpw6NShSYTqHpetbW0PX51WWh5ezETndmaOfJjVvS3epe1Qh9Jw Uyng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=to:cc:date:message-id:subject:mime-version :content-transfer-encoding:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7nJV3m2659JxdgFr8rMpzKKUXncrIfhrKeT4r7KqLdc=; b=cfuU4Ck5hMqu6ggDGEU1JEcxhfj1HScFqdzovb2gPFGZhEa6PGbYe785Biy1l1mP8q OOaYrcc2I4qXz+7l8f8E7DktR/bC5hLuzQqAwCL95lLv7KXOh3uqppEr44nNwN355Bfa eVi8Xfg3kuqpH1XrH3jac6MYvEAAF2j+Om3b5bC2fMZDoEY3ZC3LGIpjn1LMW9qrtDIS gIJetl4gi7svewADaWlSTvVbRFStHQPQaspvrBX7Tdyh8NvvYVW5kAvVMXDYpC8Sz2oR fSIxcb1IJSY3Onh6gTBpFvgBC2qmAcZ/PaF+9Zl3me7V4b+9JYldQ2UL9fdgEGLM4BSY mEJg== X-Gm-Message-State: AO0yUKXzvcIy2z/vhQlzIEvxYBVlJWC9GkWj2Kt/F9UImKAazRWwuQfV FL2vPmExwGkGUWQC7KQTutA= X-Google-Smtp-Source: AK7set+ZgRVRIV+ob+h6xw0OqsFEKAh7Q5Twu0lqK21gZW/SoaHfO/RhH3lDNHfyQ6C+bY7XyHjEPw== X-Received: by 2002:a17:903:1247:b0:195:e92e:c4d3 with SMTP id u7-20020a170903124700b00195e92ec4d3mr22323133plh.46.1675642128026; Sun, 05 Feb 2023 16:08:48 -0800 (PST) Received: from smtpclient.apple (cpe-172-117-161-177.socal.res.rr.com. [172.117.161.177]) by smtp.gmail.com with ESMTPSA id g24-20020a1709029f9800b0019663238703sm3168204plq.109.2023.02.05.16.08.47 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sun, 05 Feb 2023 16:08:47 -0800 (PST) From: Yuan Fu <casouri@HIDDEN> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.300.101.1.3\)) Subject: Re: bug#61285: (Sometimes very) slow font-lock after %w in ruby-ts-mode Message-Id: <2C6F83F0-DD84-4549-9D91-3D820188674D@HIDDEN> Date: Sun, 5 Feb 2023 16:08:36 -0800 To: Dmitry Gutov <dgutov@HIDDEN> X-Mailer: Apple Mail (2.3731.300.101.1.3) X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 61285 Cc: 61285 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) Dmitry Gutov <dgutov@HIDDEN> writes: > This probably involves a parser bug and/or maybe a tree-sitter one. > But I'm posting this here anyway because this might not be the only > way to trigger this problem. Or it could give us some optimization > insights. > > Also, while it involves a node which is parsed to have a large number > of descendants, the performance depends heavily on whether the node is > at the top level of the program (then it's slow), or not. > > To repro: > > 1. Visit test/lisp/progmodes/ruby-mode-resources/ruby.rb > 2. add 'a =3D %w' (without quotes) as a separate new line before all = of > the existing code. > 3. Notice the delay in redisplay after you type 'w'. > > In you do that with a larger file, BTW, this delay may be on the order > of a minute. Here's an example of such file: > = https://github.com/rails/rails/blob/main/activerecord/lib/active_record/as= sociations.rb > > The superficial reason for this delay is that %w opens a new "array of > strings" literal which parses every separate word in the rest of the > buffer as a separate string. So we get a node with thousands of > children, in the case of associations.rb. Or just ~1000 in the case of > ruby.rb. > > I also tried setting treesit--font-lock-fast-mode to t: no effect. > > But! If we do the same not on top-level -- say, put the 'a =3D %w' = line > after the 'foo' line inside the first 'if' statement (i.e. on line 7), > the delay is much smaller -- not noticeable in ruby.rb, and still > apparent but much more bearable in associations.rb (you can put that > statement right after 'module ActiveRecord') -- even though the size > of the tree is changed minimally, and the number of children nodes for > that "array of strings" still counts in the thousands (e.g. 13319). > > Perf report for the "bad" highlighting delay looks like this: > > 61.19% emacs libtree-sitter.so.0.0 [.] = ts_tree_cursor_current_status > 30.88% emacs libtree-sitter.so.0.0 [.] ts_tree_cursor_parent_node > 7.44% emacs libtree-sitter.so.0.0 [.] ts_language_symbol_metadata > 0.06% emacs libtree-sitter.so.0.0 [.] = ts_tree_cursor_goto_first_child > 0.05% emacs libtree-sitter.so.0.0 [.] = ts_tree_cursor_goto_next_sibling > 0.03% emacs libtree-sitter.so.0.0 [.] ts_node_end_byte > > And like this in the "good" case (with many type-backspace = repetitions): > > 32.10% emacs libtree-sitter.so.0.0 [.] = ts_tree_cursor_current_status > 9.50% emacs libtree-sitter.so.0.0 [.] = ts_tree_cursor_goto_first_child > 7.89% emacs libtree-sitter.so.0.0 [.] = ts_tree_cursor_goto_next_sibling > 7.51% emacs libtree-sitter.so.0.0 [.] ts_language_symbol_metadata > 6.45% emacs libtree-sitter.so.0.0 [.] ts_tree_cursor_parent_node > 1.87% emacs libtree-sitter.so.0.0 [.] ts_node_start_point > 1.85% emacs emacs [.] process_mark_stack > 0.93% emacs libtree-sitter.so.0.0 [.] ts_tree_cursor_current_node Interesting. Perhaps it has to do with how tree-sitter implements the "incremental" part of the parser? But the profile doesn=E2=80=99t look = like it=E2=80=99s spending time parsing, I need to look at what does ts_tree_cursor_current_status actually do (maybe it=E2=80=99s used in = parsing?) Yuan
bug-gnu-emacs@HIDDEN:bug#61285; Package emacs.
Full text available.
Received: (at submit) by debbugs.gnu.org; 5 Feb 2023 00:39:54 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Feb 04 19:39:54 2023
Received: from localhost ([127.0.0.1]:43666 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1pOT4T-00032i-0Z
for submit <at> debbugs.gnu.org; Sat, 04 Feb 2023 19:39:54 -0500
Received: from lists.gnu.org ([209.51.188.17]:56606)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <raaahh@HIDDEN>) id 1pOT4N-00032W-5c
for submit <at> debbugs.gnu.org; Sat, 04 Feb 2023 19:39:51 -0500
Received: from eggs.gnu.org ([2001:470:142:3::10])
by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
(Exim 4.90_1) (envelope-from <raaahh@HIDDEN>) id 1pOT4M-0000Fy-NH
for bug-gnu-emacs@HIDDEN; Sat, 04 Feb 2023 19:39:46 -0500
Received: from mail-ed1-x532.google.com ([2a00:1450:4864:20::532])
by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128)
(Exim 4.90_1) (envelope-from <raaahh@HIDDEN>) id 1pOT4K-0003ad-Bm
for bug-gnu-emacs@HIDDEN; Sat, 04 Feb 2023 19:39:46 -0500
Received: by mail-ed1-x532.google.com with SMTP id z11so8486126ede.1
for <bug-gnu-emacs@HIDDEN>; Sat, 04 Feb 2023 16:39:43 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112;
h=content-transfer-encoding:subject:from:to:content-language
:user-agent:mime-version:date:message-id:sender:from:to:cc:subject
:date:message-id:reply-to;
bh=bcPopwKw/jZVDuD0rMoLu3xscbawXWDzUFhNKS4VcjY=;
b=TKnONITubH6KI1erho0xHgKsCOk2Zuy1DJdtdjHOQVt89jZuZZgvgDSlKLumyGnahg
jaMfPT0PLmOvbyugD7laxQ97iR6CtC9QgM59psvXf/cH3hPhDz645JEpZ7xZ8/TaejVy
wgDAJo3NUYKBuDS/hF9mnBsER5JFZmKKTn0MfKFvL+HSLy435CTjmhW2HQZoBYbUa17T
OBTamQE5m5J/oXnxy8pdTsLm4qw9OQ7o+sQDhbNXrMSsZsHyCv3w8Arh6D08i7dn19sX
JxwuIK+PKdq5gn1wYCX89GbIn6nySdOKrGJ7/vlRZM1+x/7KXvftjisUwEZFxhTGhnxJ
6yog==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20210112;
h=content-transfer-encoding:subject:from:to:content-language
:user-agent:mime-version:date:message-id:sender:x-gm-message-state
:from:to:cc:subject:date:message-id:reply-to;
bh=bcPopwKw/jZVDuD0rMoLu3xscbawXWDzUFhNKS4VcjY=;
b=v00o1niX6FNHlIXOhch0NIyL3I5JBpZPVcGq73lu9CN4SJny6ErfSdXXRwzbpbEEwh
zdGL3OoZ+AsKmz7orXYJ0DpOSbKhefOV0aJkGuoXUcALPhrQsjRPyHjlv0Is/gkTjE4L
hUN3NHjRmEJwcyisHZMSnFDkQrfhrzZrXOzNkU7Pj5rKcdqIoig1mBE94qhSB1pyjbPA
tfWSOMnsuw8+NWx8hvJRLdbzH9KRmw50+hXpjjEJOOVtuKb2sBOWfM+0F6FX3C74J27O
i4F+XksakFrfC7PknjY/kmHNqLh/ERY4Q+FxorCgveli3v345Ajw7XC97UuLiE82G2ui
GhcQ==
X-Gm-Message-State: AO0yUKXTfEgfzllzUSsCjAs7NjzAud4K5MC8/ioquKfPaI+ilKCBQowY
eoK0d8/ouYeQPZP/rgiD1Z+YEjsA++M=
X-Google-Smtp-Source: AK7set+7C5PYHphraYdLQLOdVcp3XyjL8fH2fl9hbEIKPN5yT1CWMmeSrAOwxvs+gprWlOKSMko7mw==
X-Received: by 2002:a50:d7cd:0:b0:4aa:a40a:3bd6 with SMTP id
m13-20020a50d7cd000000b004aaa40a3bd6mr2584266edj.24.1675557580581;
Sat, 04 Feb 2023 16:39:40 -0800 (PST)
Received: from [192.168.0.2] ([46.251.119.176])
by smtp.googlemail.com with ESMTPSA id
b12-20020a056402350c00b0049e210884dasm3150478edd.15.2023.02.04.16.39.39
for <bug-gnu-emacs@HIDDEN>
(version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);
Sat, 04 Feb 2023 16:39:39 -0800 (PST)
Message-ID: <17a29b2e-beb2-f246-fe1c-2c3503c1bfe6@HIDDEN>
Date: Sun, 5 Feb 2023 02:39:38 +0200
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
Thunderbird/102.4.2
Content-Language: en-US
To: bug-gnu-emacs@HIDDEN
From: Dmitry Gutov <dgutov@HIDDEN>
Subject: (Sometimes very) slow font-lock after %w in ruby-ts-mode
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Received-SPF: pass client-ip=2a00:1450:4864:20::532;
envelope-from=raaahh@HIDDEN; helo=mail-ed1-x532.google.com
X-Spam_score_int: -14
X-Spam_score: -1.5
X-Spam_bar: -
X-Spam_report: (-1.5 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,
DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FORGED_FROMDOMAIN=0.25,
FREEMAIL_FROM=0.001, HEADER_FROM_DIFFERENT_DOMAINS=0.25,
RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001,
SPF_PASS=-0.001 autolearn=no autolearn_force=no
X-Spam_action: no action
X-Spam-Score: -1.1 (-)
X-Debbugs-Envelope-To: submit
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -2.1 (--)
This probably involves a parser bug and/or maybe a tree-sitter one. But
I'm posting this here anyway because this might not be the only way to
trigger this problem. Or it could give us some optimization insights.
Also, while it involves a node which is parsed to have a large number of
descendants, the performance depends heavily on whether the node is at
the top level of the program (then it's slow), or not.
To repro:
1. Visit test/lisp/progmodes/ruby-mode-resources/ruby.rb
2. add 'a = %w' (without quotes) as a separate new line before all of
the existing code.
3. Notice the delay in redisplay after you type 'w'.
In you do that with a larger file, BTW, this delay may be on the order
of a minute. Here's an example of such file:
https://github.com/rails/rails/blob/main/activerecord/lib/active_record/associations.rb
The superficial reason for this delay is that %w opens a new "array of
strings" literal which parses every separate word in the rest of the
buffer as a separate string. So we get a node with thousands of
children, in the case of associations.rb. Or just ~1000 in the case of
ruby.rb.
I also tried setting treesit--font-lock-fast-mode to t: no effect.
But! If we do the same not on top-level -- say, put the 'a = %w' line
after the 'foo' line inside the first 'if' statement (i.e. on line 7),
the delay is much smaller -- not noticeable in ruby.rb, and still
apparent but much more bearable in associations.rb (you can put that
statement right after 'module ActiveRecord') -- even though the size of
the tree is changed minimally, and the number of children nodes for that
"array of strings" still counts in the thousands (e.g. 13319).
Perf report for the "bad" highlighting delay looks like this:
61.19% emacs libtree-sitter.so.0.0 [.] ts_tree_cursor_current_status
30.88% emacs libtree-sitter.so.0.0 [.] ts_tree_cursor_parent_node
7.44% emacs libtree-sitter.so.0.0 [.] ts_language_symbol_metadata
0.06% emacs libtree-sitter.so.0.0 [.] ts_tree_cursor_goto_first_child
0.05% emacs libtree-sitter.so.0.0 [.] ts_tree_cursor_goto_next_sibling
0.03% emacs libtree-sitter.so.0.0 [.] ts_node_end_byte
And like this in the "good" case (with many type-backspace repetitions):
32.10% emacs libtree-sitter.so.0.0 [.] ts_tree_cursor_current_status
9.50% emacs libtree-sitter.so.0.0 [.] ts_tree_cursor_goto_first_child
7.89% emacs libtree-sitter.so.0.0 [.] ts_tree_cursor_goto_next_sibling
7.51% emacs libtree-sitter.so.0.0 [.] ts_language_symbol_metadata
6.45% emacs libtree-sitter.so.0.0 [.] ts_tree_cursor_parent_node
1.87% emacs libtree-sitter.so.0.0 [.] ts_node_start_point
1.85% emacs emacs [.] process_mark_stack
0.93% emacs libtree-sitter.so.0.0 [.] ts_tree_cursor_current_node
Dmitry Gutov <dgutov@HIDDEN>:bug-gnu-emacs@HIDDEN.
Full text available.bug-gnu-emacs@HIDDEN:bug#61285; Package emacs.
Full text available.
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997 nCipher Corporation Ltd,
1994-97 Ian Jackson.