Received: (at 75356) by debbugs.gnu.org; 6 Jan 2025 20:31:16 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Mon Jan 06 15:31:16 2025 Received: from localhost ([127.0.0.1]:40319 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1tUtkp-0001Mw-VQ for submit <at> debbugs.gnu.org; Mon, 06 Jan 2025 15:31:16 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:48538) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from <acorallo@HIDDEN>) id 1tUtkn-0001Me-EV for 75356 <at> debbugs.gnu.org; Mon, 06 Jan 2025 15:31:14 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from <acorallo@HIDDEN>) id 1tUtkh-0006R3-T9; Mon, 06 Jan 2025 15:31:08 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:References:In-Reply-To:Subject:To: From; bh=1s7JdsKyh9CPAIgtFx7+E4jwpeBiowkQ9mRqURTlsvE=; b=gc+ur9QfYNbLhKOKWppd WdjiKViUiRznG7Gv9C0yAWGr/uY4cWVgYVfXTp6hf6OiozhDqDIo/1Jhcke5z0d+LHyY7oXrALkdC H45rZIRuR6XWrjxDwUlWPbKHzs3Om2p8ncQqUinkkyYrdDD2eDSjaB1CyaNxZRm4yCdXlIQY/p84b yA5H7ARGPlhA+Dy2qg7KR/nztF2CuPEIpNF+6617PxBEnHB3jskzn89qnlffCojOuQlq57pKG67ZG 4gyKvAozreRAQUgb27Tn57zbesTXo/HsxPmaToZKNuDfxeL6lxPUCsZXAhAkSguTHkZzAkX80QWng dHOj60qgsAlgGw==; Received: from acorallo by fencepost.gnu.org with local (Exim 4.90_1) (envelope-from <acorallo@HIDDEN>) id 1tUtk1-0006zQ-23; Mon, 06 Jan 2025 15:30:33 -0500 From: Andrea Corallo <acorallo@HIDDEN> To: Eli Zaretskii <eliz@HIDDEN> Subject: Re: bug#75356: impossible to benchmark byte-compiled code in a native-comp build In-Reply-To: <86ldvo58br.fsf@HIDDEN> (Eli Zaretskii's message of "Mon, 06 Jan 2025 16:24:24 +0200") References: <87ldvqfsi4.fsf@HIDDEN> <yp1v7us5m77.fsf@HIDDEN> <86ldvo58br.fsf@HIDDEN> Date: Mon, 06 Jan 2025 15:30:24 -0500 Message-ID: <yp1v7ur4rdr.fsf@HIDDEN> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 75356 Cc: pipcet@HIDDEN, 75356 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -3.3 (---) Eli Zaretskii <eliz@HIDDEN> writes: >> Cc: pipcet@HIDDEN >> From: Andrea Corallo <acorallo@HIDDEN> >> Date: Mon, 06 Jan 2025 04:24:44 -0500 >> >> Pip Cet via "Bug reports for GNU Emacs, the Swiss army knife of text >> editors" <bug-gnu-emacs@HIDDEN> writes: >> >> > elisp-benchmarks.el unconditionally forces use of native compilation >> > if run on an Emacs which is compiled with native-comp enabled. This >> > is a minor issue, but it's still important to benchmark this >> > configuration once in a while. >> >> Why is this important? On a native compiled Emacs all code is supposed >> to run native compiled, not only the test, but runtime libraries as >> well, further more even Emacs primitives are different in order to >> accomodate native code execution. My opinion is that results of such a >> mixed tests would make little no sense in general. > > I think the idea is that we don't want to force someone who wants to > benchmark bytecode to build a separate Emacs configured without native > compilation. Okay still the results are not very meaningful IMO for the mentioned reasons. > It is okay to default to native code, but it would be nice to have a > knob to run the benchmarks without compiling them to native code. Okay, should be very easy to add one if the 'elb-speed -1' option is deemed not sufficient, suggestions for the name? "elb-no-native"? >> Anyway if you really want to run bytecode on a native compiled Emacs >> elisp-benchamerks let you do this with: >> >> emacs -batch -l ./elisp-benchmarks.el --eval '(progn (setq elb-speed -1) (elisp-benchmarks-run))' > > Would this refrain from loading *.eln files if they are already there? This way (on a native compiled Emacs) it will produce and load eln files with regular bytecode inside. > Or does the benchmark suite remove all the *.eln files after it > finishes? Nope
bug-gnu-emacs@HIDDEN:bug#75356; Package emacs.
Full text available.Received: (at 75356) by debbugs.gnu.org; 6 Jan 2025 14:25:22 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Mon Jan 06 09:25:22 2025 Received: from localhost ([127.0.0.1]:37447 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1tUo2j-0007E9-M3 for submit <at> debbugs.gnu.org; Mon, 06 Jan 2025 09:25:22 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:50596) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from <eliz@HIDDEN>) id 1tUo2g-00078X-M4 for 75356 <at> debbugs.gnu.org; Mon, 06 Jan 2025 09:25:19 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from <eliz@HIDDEN>) id 1tUo2Z-0001Rs-Sr; Mon, 06 Jan 2025 09:25:13 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date: mime-version; bh=LxXwbHVTlxstnKP1fDjj5NjVNnvJYSczc3V/Lh6vG6U=; b=PlpjWnXfAQJX pjmI49XAeN8066BtVv3K9rXmllmw95JZY7pUszYi6iG/kHqFn6dGTKXXNOBbmeKT+8ge7hVwV+0L+ 77DWUv2DQfmM+1e9FQB7ZDfxv/rUELi1GKqP+OD2nSImn/no9eTiL9o94y93vgUoC/mOdZUSRwUIB 5K7PAoUBEvHWf9Qz2sTw06+929Q18DrSvUXx3AFJUJMX5Oa+56I7YKvDWQWLpnLETsoRXTNssvWda LdS90ljGh0o6INRCyGXWMJhRUoQc0s1FiOEqFh6OsNeF5vxceZ4FF0pJKKAizbkDgRd4a+ksGukkR 6m4WcDej1pZFl0Q3fwIgFg==; Date: Mon, 06 Jan 2025 16:24:24 +0200 Message-Id: <86ldvo58br.fsf@HIDDEN> From: Eli Zaretskii <eliz@HIDDEN> To: Andrea Corallo <acorallo@HIDDEN> In-Reply-To: <yp1v7us5m77.fsf@HIDDEN> (message from Andrea Corallo on Mon, 06 Jan 2025 04:24:44 -0500) Subject: Re: bug#75356: impossible to benchmark byte-compiled code in a native-comp build References: <87ldvqfsi4.fsf@HIDDEN> <yp1v7us5m77.fsf@HIDDEN> X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 75356 Cc: pipcet@HIDDEN, 75356 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -3.3 (---) > Cc: pipcet@HIDDEN > From: Andrea Corallo <acorallo@HIDDEN> > Date: Mon, 06 Jan 2025 04:24:44 -0500 > > Pip Cet via "Bug reports for GNU Emacs, the Swiss army knife of text > editors" <bug-gnu-emacs@HIDDEN> writes: > > > elisp-benchmarks.el unconditionally forces use of native compilation > > if run on an Emacs which is compiled with native-comp enabled. This > > is a minor issue, but it's still important to benchmark this > > configuration once in a while. > > Why is this important? On a native compiled Emacs all code is supposed > to run native compiled, not only the test, but runtime libraries as > well, further more even Emacs primitives are different in order to > accomodate native code execution. My opinion is that results of such a > mixed tests would make little no sense in general. I think the idea is that we don't want to force someone who wants to benchmark bytecode to build a separate Emacs configured without native compilation. It is okay to default to native code, but it would be nice to have a knob to run the benchmarks without compiling them to native code. > Anyway if you really want to run bytecode on a native compiled Emacs > elisp-benchamerks let you do this with: > > emacs -batch -l ./elisp-benchmarks.el --eval '(progn (setq elb-speed -1) (elisp-benchmarks-run))' Would this refrain from loading *.eln files if they are already there? Or does the benchmark suite remove all the *.eln files after it finishes? (Apologies for not taking a closer look at the code.) > > In general, the compilation logic of elisp-benchmarks.el is fragile > > and will lead to unreliabe test results, > > Why do you think so? I suggest to avoid such general remarks, and instead talk about specific issues where you think the logic could easily break.
bug-gnu-emacs@HIDDEN:bug#75356; Package emacs.
Full text available.Received: (at 75356) by debbugs.gnu.org; 6 Jan 2025 09:24:54 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Mon Jan 06 04:24:54 2025 Received: from localhost ([127.0.0.1]:36673 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1tUjLx-0008Ta-Pi for submit <at> debbugs.gnu.org; Mon, 06 Jan 2025 04:24:54 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:36762) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from <acorallo@HIDDEN>) id 1tUjLv-0008TF-O0 for 75356 <at> debbugs.gnu.org; Mon, 06 Jan 2025 04:24:52 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from <acorallo@HIDDEN>) id 1tUjLp-0007DH-R3; Mon, 06 Jan 2025 04:24:46 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:References:In-Reply-To:Subject:To: From; bh=ZI1aPRolKRz86OPedHDSmdRaIoYrKSSll3ov2zel8yE=; b=K52zS39LnEwVUrg6EUsd c47CtSd3wL4O8uZfBAw+hYQ4CAOdZm4ezed43xWkCW3Lq/oCPE7PMpeoRbRVoQoxTqMC0RC7lnpbS 52KzvdK7zqaz3NbvDeC66m8l0bszmn92PnE3nCgTWf2qNFNK1C+2FNELwyYgfaGsqWg0Ldo77yJ36 2Jk0+FP1bt0pEJ3JCXWi1RRI8Wo5VoSU8GFnRRYFrsXKVWr3Q7rDvy7MGYau5l2J3iBdPcjcRSySi EBhxmKi6JhRP8J6CHRA0mm9vCbUyyZuSEWWxhel6I5FjgMmCy7Yl6OEIg4PpjezKV5IEnBM5iH+HK SrCNttu3zJFtTg==; Received: from acorallo by fencepost.gnu.org with local (Exim 4.90_1) (envelope-from <acorallo@HIDDEN>) id 1tUjLo-0002Ja-Mj; Mon, 06 Jan 2025 04:24:45 -0500 From: Andrea Corallo <acorallo@HIDDEN> To: Pip Cet via "Bug reports for GNU Emacs, the Swiss army knife of text editors" <bug-gnu-emacs@HIDDEN> Subject: Re: bug#75356: impossible to benchmark byte-compiled code in a native-comp build In-Reply-To: <87ldvqfsi4.fsf@HIDDEN> (Pip Cet via's message of "Sat, 04 Jan 2025 16:35:01 +0000") References: <87ldvqfsi4.fsf@HIDDEN> Date: Mon, 06 Jan 2025 04:24:44 -0500 Message-ID: <yp1v7us5m77.fsf@HIDDEN> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 75356 Cc: Pip Cet <pipcet@HIDDEN>, 75356 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -3.3 (---) Pip Cet via "Bug reports for GNU Emacs, the Swiss army knife of text editors" <bug-gnu-emacs@HIDDEN> writes: > elisp-benchmarks.el unconditionally forces use of native compilation > if run on an Emacs which is compiled with native-comp enabled. This > is a minor issue, but it's still important to benchmark this > configuration once in a while. Why is this important? On a native compiled Emacs all code is supposed to run native compiled, not only the test, but runtime libraries as well, further more even Emacs primitives are different in order to accomodate native code execution. My opinion is that results of such a mixed tests would make little no sense in general. > For example, running > > ./src/emacs -Q --batch --load elisp-benchmarks/elisp-benchmarks.el --eval '(progn (setq no-native-compile t) (elisp-benchmarks-run))' For the resons above I don't think was never supported. Anyway if you really want to run bytecode on a native compiled Emacs elisp-benchamerks let you do this with: emacs -batch -l ./elisp-benchmarks.el --eval '(progn (setq elb-speed -1) (elisp-benchmarks-run))' Alternativelly if you are interested in working with a function granularity you can set manually function speeds to -1. > produces error messages and an "empty" results table: > > | test | non-gc (s) | gc (s) | gcs | total (s) | err (s) | > |-------+------------+--------+-----+-----------+---------| > |-------+------------+--------+-----+-----------+---------| > | total | 0.00 | 0.00 | 0 | 0.00 | 0.00 | > > In particular, I believe it is legitimate to use a native-comp build > on a system which no longer supports compiling additional code. > > In general, the compilation logic of elisp-benchmarks.el is fragile > and will lead to unreliabe test results, Why do you think so?
bug-gnu-emacs@HIDDEN:bug#75356; Package emacs.
Full text available.Received: (at submit) by debbugs.gnu.org; 6 Jan 2025 09:24:54 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Mon Jan 06 04:24:54 2025 Received: from localhost ([127.0.0.1]:36675 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1tUjLy-0008Td-6S for submit <at> debbugs.gnu.org; Mon, 06 Jan 2025 04:24:54 -0500 Received: from lists.gnu.org ([2001:470:142::17]:36020) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from <acorallo@HIDDEN>) id 1tUjLw-0008TH-LI for submit <at> debbugs.gnu.org; Mon, 06 Jan 2025 04:24:53 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from <acorallo@HIDDEN>) id 1tUjLq-0007Tq-Me for bug-gnu-emacs@HIDDEN; Mon, 06 Jan 2025 04:24:46 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from <acorallo@HIDDEN>) id 1tUjLp-0007DH-R3; Mon, 06 Jan 2025 04:24:46 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:References:In-Reply-To:Subject:To: From; bh=ZI1aPRolKRz86OPedHDSmdRaIoYrKSSll3ov2zel8yE=; b=K52zS39LnEwVUrg6EUsd c47CtSd3wL4O8uZfBAw+hYQ4CAOdZm4ezed43xWkCW3Lq/oCPE7PMpeoRbRVoQoxTqMC0RC7lnpbS 52KzvdK7zqaz3NbvDeC66m8l0bszmn92PnE3nCgTWf2qNFNK1C+2FNELwyYgfaGsqWg0Ldo77yJ36 2Jk0+FP1bt0pEJ3JCXWi1RRI8Wo5VoSU8GFnRRYFrsXKVWr3Q7rDvy7MGYau5l2J3iBdPcjcRSySi EBhxmKi6JhRP8J6CHRA0mm9vCbUyyZuSEWWxhel6I5FjgMmCy7Yl6OEIg4PpjezKV5IEnBM5iH+HK SrCNttu3zJFtTg==; Received: from acorallo by fencepost.gnu.org with local (Exim 4.90_1) (envelope-from <acorallo@HIDDEN>) id 1tUjLo-0002Ja-Mj; Mon, 06 Jan 2025 04:24:45 -0500 From: Andrea Corallo <acorallo@HIDDEN> To: Pip Cet via "Bug reports for GNU Emacs, the Swiss army knife of text editors" <bug-gnu-emacs@HIDDEN> Subject: Re: bug#75356: impossible to benchmark byte-compiled code in a native-comp build In-Reply-To: <87ldvqfsi4.fsf@HIDDEN> (Pip Cet via's message of "Sat, 04 Jan 2025 16:35:01 +0000") References: <87ldvqfsi4.fsf@HIDDEN> Date: Mon, 06 Jan 2025 04:24:44 -0500 Message-ID: <yp1v7us5m77.fsf@HIDDEN> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: submit Cc: Pip Cet <pipcet@HIDDEN>, 75356 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) Pip Cet via "Bug reports for GNU Emacs, the Swiss army knife of text editors" <bug-gnu-emacs@HIDDEN> writes: > elisp-benchmarks.el unconditionally forces use of native compilation > if run on an Emacs which is compiled with native-comp enabled. This > is a minor issue, but it's still important to benchmark this > configuration once in a while. Why is this important? On a native compiled Emacs all code is supposed to run native compiled, not only the test, but runtime libraries as well, further more even Emacs primitives are different in order to accomodate native code execution. My opinion is that results of such a mixed tests would make little no sense in general. > For example, running > > ./src/emacs -Q --batch --load elisp-benchmarks/elisp-benchmarks.el --eval '(progn (setq no-native-compile t) (elisp-benchmarks-run))' For the resons above I don't think was never supported. Anyway if you really want to run bytecode on a native compiled Emacs elisp-benchamerks let you do this with: emacs -batch -l ./elisp-benchmarks.el --eval '(progn (setq elb-speed -1) (elisp-benchmarks-run))' Alternativelly if you are interested in working with a function granularity you can set manually function speeds to -1. > produces error messages and an "empty" results table: > > | test | non-gc (s) | gc (s) | gcs | total (s) | err (s) | > |-------+------------+--------+-----+-----------+---------| > |-------+------------+--------+-----+-----------+---------| > | total | 0.00 | 0.00 | 0 | 0.00 | 0.00 | > > In particular, I believe it is legitimate to use a native-comp build > on a system which no longer supports compiling additional code. > > In general, the compilation logic of elisp-benchmarks.el is fragile > and will lead to unreliabe test results, Why do you think so?
bug-gnu-emacs@HIDDEN:bug#75356; Package emacs.
Full text available.Received: (at submit) by debbugs.gnu.org; 4 Jan 2025 16:35:17 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Sat Jan 04 11:35:17 2025 Received: from localhost ([127.0.0.1]:56872 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1tU77N-0000zi-9a for submit <at> debbugs.gnu.org; Sat, 04 Jan 2025 11:35:17 -0500 Received: from lists.gnu.org ([2001:470:142::17]:35034) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from <pipcet@HIDDEN>) id 1tU77L-0000zR-KZ for submit <at> debbugs.gnu.org; Sat, 04 Jan 2025 11:35:16 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from <pipcet@HIDDEN>) id 1tU77G-0006nf-1y for bug-gnu-emacs@HIDDEN; Sat, 04 Jan 2025 11:35:10 -0500 Received: from mail-4322.protonmail.ch ([185.70.43.22]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from <pipcet@HIDDEN>) id 1tU77D-0004nx-Bm for bug-gnu-emacs@HIDDEN; Sat, 04 Jan 2025 11:35:08 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=protonmail.com; s=protonmail3; t=1736008505; x=1736267705; bh=xQhBJHHZ5Fj315av/lHceKKL5X6xWq4SlYwEbHVHwhQ=; h=Date:To:From:Subject:Message-ID:Feedback-ID:From:To:Cc:Date: Subject:Reply-To:Feedback-ID:Message-ID:BIMI-Selector: List-Unsubscribe:List-Unsubscribe-Post; b=JDonMsJOL2sWRBuE5ioPp7R95BYfi72Gz/DQ3xVzC2oAOBLzUc1KsJnQDPvHP6Xa9 yA6azKrRAFUK8qJtqB6ofQ3yCTUFKnoiWrDNplcL5sv4v7E7GJIj7MT9fN4YIY2wYj h+mtiwAWM27P2NBkptEPkzB1bu0CqFsuQxNUZev6ValmJg/h59EXB2W2uXAxJYYwGc /xEi9m+7hQWH2W4KECNePEAO7/wk9v7mkmmiaSnimo+hXo0IqLWzzO+6/MZBZNoHS0 xozkQgWJPiPeL/SK93JZmJb08d0bPKL4Rwa5nGKIODlRfZMwqJGDrUqP8hm0Pifuu8 fmEDAQ/LHES2Q== Date: Sat, 04 Jan 2025 16:35:01 +0000 To: bug-gnu-emacs@HIDDEN From: Pip Cet <pipcet@HIDDEN> Subject: impossible to benchmark byte-compiled code in a native-comp build Message-ID: <87ldvqfsi4.fsf@HIDDEN> Feedback-ID: 112775352:user:proton X-Pm-Message-ID: 275d7ee628b864b7edbc5cb42454b2d8c6071d0a MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Received-SPF: pass client-ip=185.70.43.22; envelope-from=pipcet@HIDDEN; helo=mail-4322.protonmail.ch X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Spam-Score: 1.0 (+) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -0.0 (/) elisp-benchmarks.el unconditionally forces use of native compilation if run on an Emacs which is compiled with native-comp enabled. This is a minor issue, but it's still important to benchmark this configuration once in a while. For example, running ./src/emacs -Q --batch --load elisp-benchmarks/elisp-benchmarks.el --eval '= (progn (setq no-native-compile t) (elisp-benchmarks-run))' produces error messages and an "empty" results table: | test | non-gc (s) | gc (s) | gcs | total (s) | err (s) | |-------+------------+--------+-----+-----------+---------| |-------+------------+--------+-----+-----------+---------| | total | 0.00 | 0.00 | 0 | 0.00 | 0.00 | In particular, I believe it is legitimate to use a native-comp build on a system which no longer supports compiling additional code. In general, the compilation logic of elisp-benchmarks.el is fragile and will lead to unreliabe test results, which is the worst possible outcome for a benchmark (in some cases, we simply fail to produce data, which is better but still not good).
Pip Cet <pipcet@HIDDEN>:bug-gnu-emacs@HIDDEN.
Full text available.bug-gnu-emacs@HIDDEN:bug#75356; Package emacs.
Full text available.
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997 nCipher Corporation Ltd,
1994-97 Ian Jackson.