GNU bug report logs - #78416
[PATCH] * src/lread.c (get-load-suffixes): Avoid module suffix with compressed suffix

Previous Next

To reply to this bug, email your comments to 78416 AT debbugs.gnu.org.
There is no need to reopen the bug first.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Lin Sun <sunlin7 <at> hotmail.com>
To: "bug-gnu-emacs <at> gnu.org" <bug-gnu-emacs <at> gnu.org>
Subject: [PATCH] * src/lread.c (get-load-suffixes): Avoid module suffix with
 compressed suffix
Date: Wed, 14 May 2025 06:21:09 +0000

[Message part 1 (text/plain, inline)]

The Emacs (with module) will try load module with compressed suffix ".gz", for example, (require 'X) will trigger emacs try load file X.dll, X.dll.gz... on windows, but the "X.dll.gz" does not loadable, it affects performance especially on windows.

This patch make the (get-load-suffixes) function return result contains the ".dll" but avoid the ".dll.gz". 

Please help review the patch. Thank you.

[0001-src-lread.c-get-load-suffixes-Avoid-module-suffix-wi.patch (text/x-patch, attachment)]

Message #8 received at 78416 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Lin Sun <sunlin7 <at> hotmail.com>, Paul Eggert <eggert <at> cs.ucla.edu>,
 Po Lu <luangruo <at> yahoo.com>
Cc: 78416 <at> debbugs.gnu.org
Subject: Re: bug#78416: [PATCH] * src/lread.c (get-load-suffixes): Avoid module
 suffix with compressed suffix
Date: Wed, 14 May 2025 15:11:40 +0300

> From: Lin Sun <sunlin7 <at> hotmail.com>
> Date: Wed, 14 May 2025 06:21:09 +0000
> 
> The Emacs (with module) will try load module with compressed suffix ".gz", for example, (require 'X) will trigger emacs try load file X.dll, X.dll.gz... on windows, but the "X.dll.gz" does not loadable, it affects performance especially on windows.
> 
> This patch make the (get-load-suffixes) function return result contains the ".dll" but avoid the ".dll.gz". 

The issue is not specific to Windows, so I don't quite understand why
you are talking about Windows and *.dll files.

Paul and Po Lu, are there any systems we know about where Emacs can
load a compressed shared library whose file-name ends in ".gz" or any
other suffix jka-compr supports?  If there are no such systems, then
we should never consider compressed module files for loading.

However, the patch as it is isn't right, IMO: it's too general.

>      {
>        Lisp_Object exts = Vload_file_rep_suffixes;
>        Lisp_Object suffix = XCAR (suffixes);
> +#ifdef HAVE_MODULES
> +      if (suffix_p(suffix, MODULES_SUFFIX))
> +	continue;
> +#ifdef MODULES_SECONDARY_SUFFIX
> +      else if (suffix_p (suffix, MODULES_SECONDARY_SUFFIX))
> +	continue;
> +#endif
> +#endif
>        FOR_EACH_TAIL (exts)
>  	lst = Fcons (concat2 (suffix, XCAR (exts)), lst);

This assumes that load-file-rep-suffixes holds _only_
compression-related suffixes.  This is indeed what we use it for in
Emacs by default, but the variable's potential uses are more general,
and can support any other suffixes.  So we cannot bluntly ignore any
suffix in this variable.  Assuming that indeed no platform we care
about supports loading compressed shared libraries, the change should
ignore _only_ ".gz" (and maybe other compression-related suffixes),
but not others.

Thanks.

P.S. Did you measure the effect of the proposed change, and if so, can
you show the respective timings?  The amount of speedup is an
important consideration whenever we complicate our code, so it is
important to know what kind of bang we get for the buck.

Message #11 received at 78416 <at> debbugs.gnu.org (full text, mbox):

From: Lin Sun <sunlin7 <at> hotmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>, Paul Eggert <eggert <at> cs.ucla.edu>, Po Lu
 <luangruo <at> yahoo.com>
Cc: "78416 <at> debbugs.gnu.org" <78416 <at> debbugs.gnu.org>
Subject: Re: bug#78416: [PATCH] * src/lread.c (get-load-suffixes): Avoid
 module suffix with compressed suffix
Date: Wed, 14 May 2025 13:40:15 +0000

From: Eli Zaretskii <eliz <at> gnu.org>
Sent: Wednesday, May 14, 2025 12:11 PM
> The issue is not specific to Windows, so I don't quite understand why
> you are talking about Windows and *.dll files.

It's not specific to Windows, but it's notable on Windows. 
My local build reduced calling CreateFile("X.dll.gz") for ~11000 time on startup, and startup time reduced from 4.122s  to 3.869s.

> ...are there any systems we know about where Emacs can
> load a compressed shared library whose file-name ends in ".gz" or any
> other suffix jka-compr supports? 
> ...
> However, the patch as it is isn't right, IMO: it's too general.

The patch corresponds to the lines in the Fload() function, it detects the "is_module" by module suffix, then call Fmodule_load(), that imply other suffix won't be considered as a module.

 1512 #ifdef HAVE_MODULES
 1513   bool is_module =
 1514     suffix_p (found, MODULES_SUFFIX)
 1515 #ifdef MODULES_SECONDARY_SUFFIX
 1516     || suffix_p (found, MODULES_SECONDARY_SUFFIX)
 1517 #endif
 1518     ;
...
 1726   if (is_module)
 1727     {
 1728 #ifdef HAVE_MODULES
 1729       loadhist_initialize (found);
 1730       Fmodule_load (found);
 1731       build_load_history (found, true);
 1732 #else
 1733       /* This cannot happen.  */
 1734       emacs_abort ();
 1735 #endif
 1736     }

Message #14 received at 78416 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Lin Sun <sunlin7 <at> hotmail.com>
Cc: luangruo <at> yahoo.com, 78416 <at> debbugs.gnu.org, eggert <at> cs.ucla.edu
Subject: Re: bug#78416: [PATCH] * src/lread.c (get-load-suffixes): Avoid
 module suffix with compressed suffix
Date: Wed, 14 May 2025 17:28:12 +0300

> From: Lin Sun <sunlin7 <at> hotmail.com>
> CC: "78416 <at> debbugs.gnu.org" <78416 <at> debbugs.gnu.org>
> Date: Wed, 14 May 2025 13:40:15 +0000
> msip_labels:
> 
> From: Eli Zaretskii <eliz <at> gnu.org>
> Sent: Wednesday, May 14, 2025 12:11 PM
> > The issue is not specific to Windows, so I don't quite understand why
> > you are talking about Windows and *.dll files.
> 
> It's not specific to Windows, but it's notable on Windows. 
> My local build reduced calling CreateFile("X.dll.gz") for ~11000 time on startup, and startup time reduced from 4.122s  to 3.869s.

That's about 7% reduction in startup time.

> > ...are there any systems we know about where Emacs can
> > load a compressed shared library whose file-name ends in ".gz" or any
> > other suffix jka-compr supports? 
> > ...
> > However, the patch as it is isn't right, IMO: it's too general.
> 
> The patch corresponds to the lines in the Fload() function, it detects the "is_module" by module suffix, then call Fmodule_load(), that imply other suffix won't be considered as a module.

Yes, I know.  What I meant was that load-file-rep-suffixes aren't
necessarily about compression, they could be anything.  So if we want
to install such a change, it should filter out compression suffixes
like ".gz", not any suffix that happens to be in
load-file-rep-suffixes.

Message #17 received at 78416 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: Lin Sun <sunlin7 <at> hotmail.com>, 78416 <at> debbugs.gnu.org,
 Po Lu <luangruo <at> yahoo.com>
Subject: Re: bug#78416: [PATCH] * src/lread.c (get-load-suffixes): Avoid
 module suffix with compressed suffix
Date: Wed, 14 May 2025 13:01:57 -0700

On 2025-05-14 05:11, Eli Zaretskii wrote:
> Paul and Po Lu, are there any systems we know about where Emacs can
> load a compressed shared library whose file-name ends in ".gz" or any
> other suffix jka-compr supports?

I don't know of any. A quick Google search suggests that software that 
plays this game (e.g., [1], [2]) doesn't add a ".gz"-like extension to 
names of compressed shared libraries. That being said, I'm no expert in 
this area as I think compressing a shared library is generally a mistake.

[1]: 
https://chromium.googlesource.com/chromium/src/+/lkgr/tools/android/elf_compression/
[2]: https://upx.github.io/

Message #20 received at 78416 <at> debbugs.gnu.org (full text, mbox):

From: Lin Sun <sunlin7 <at> hotmail.com>
To: Paul Eggert <eggert <at> cs.ucla.edu>, Eli Zaretskii <eliz <at> gnu.org>
Cc: Po Lu <luangruo <at> yahoo.com>, "78416 <at> debbugs.gnu.org" <78416 <at> debbugs.gnu.org>
Subject: Re: bug#78416: [PATCH] * src/lread.c (get-load-suffixes): Avoid
 module suffix with compressed suffix
Date: Sat, 17 May 2025 00:01:49 +0000

[Message part 1 (text/plain, inline)]

Update the patch towards the previous review comments. 

[0001-src-lread.c-get-load-suffixes-Avoid-module-suffix-wi.patch (text/x-patch, attachment)]

Message #23 received at 78416 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Lin Sun <sunlin7 <at> hotmail.com>
Cc: luangruo <at> yahoo.com, 78416 <at> debbugs.gnu.org, eggert <at> cs.ucla.edu
Subject: Re: bug#78416: [PATCH] * src/lread.c (get-load-suffixes): Avoid
 module suffix with compressed suffix
Date: Sat, 17 May 2025 09:57:47 +0300

> From: Lin Sun <sunlin7 <at> hotmail.com>
> CC: "78416 <at> debbugs.gnu.org" <78416 <at> debbugs.gnu.org>, Po Lu
> 	<luangruo <at> yahoo.com>
> Date: Sat, 17 May 2025 00:01:49 +0000
> 
> Update the patch towards the previous review comments. 

Thanks, but I think you've misunderstood the comments.  We should only
ignore the ".gz" suffix.  Ignoring other suffixes would be a potential
regression, because there's nothing wrong in trying to load a file
called, say, foobar.so.abc.  The current code supports that, and I see
no reason to drop that support.

The ".gz" suffix gets in the way because dynamic loaders cannot load
such compressed shared libraries.

So the condition for not appending a suffix from
load-file-rep-suffixes should also test the value of 'ext', and only
refuse to append it if it compares equal with ".gz".

Message #26 received at 78416 <at> debbugs.gnu.org (full text, mbox):

From: Lin Sun <sunlin7 <at> hotmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: "luangruo <at> yahoo.com" <luangruo <at> yahoo.com>,
 "78416 <at> debbugs.gnu.org" <78416 <at> debbugs.gnu.org>,
 "eggert <at> cs.ucla.edu" <eggert <at> cs.ucla.edu>
Subject: Re: bug#78416: [PATCH] * src/lread.c (get-load-suffixes): Avoid
 module suffix with compressed suffix
Date: Wed, 21 May 2025 20:43:15 +0000

[Message part 1 (text/plain, inline)]

From: Eli Zaretskii <eliz <at> gnu.org>
Sent: Saturday, May 17, 2025 06:57 AM

> Thanks, but I think you've misunderstood the comments.  We should only
> ignore the ".gz" suffix.  Ignoring other suffixes would be a potential
> regression, because there's nothing wrong in trying to load a file
> called, say, foobar.so.abc.  The current code supports that, and I see
> no reason to drop that support.

> The ".gz" suffix gets in the way because dynamic loaders cannot load
> such compressed shared libraries.

> So the condition for not appending a suffix from
> load-file-rep-suffixes should also test the value of 'ext', and only
> refuse to append it if it compares equal with ".gz".

Yes, you're right.

According to the document of "load-file-rep-suffixes", the "jka-compr-load-suffixes" is used for the compressed suffixes.
So I attach a new patch, the patch will seek the "jka-compr-load-suffixes" to determine the compressed suffixes for emacs module and skip it.
Please check the attached patch for details. Thank you.

[0001-src-lread.c-get-load-suffixes-Avoid-module-suffix-wi.patch (text/x-patch, attachment)]

Message #31 received at 78416-done <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Lin Sun <sunlin7 <at> hotmail.com>
Cc: 78416-done <at> debbugs.gnu.org, luangruo <at> yahoo.com, eggert <at> cs.ucla.edu
Subject: Re: bug#78416: [PATCH] * src/lread.c (get-load-suffixes): Avoid
 module suffix with compressed suffix
Date: Sat, 24 May 2025 12:46:51 +0300

> From: Lin Sun <sunlin7 <at> hotmail.com>
> CC: "eggert <at> cs.ucla.edu" <eggert <at> cs.ucla.edu>, "78416 <at> debbugs.gnu.org"
> 	<78416 <at> debbugs.gnu.org>, "luangruo <at> yahoo.com" <luangruo <at> yahoo.com>
> Date: Wed, 21 May 2025 20:43:15 +0000
> 
> From: Eli Zaretskii <eliz <at> gnu.org>
> Sent: Saturday, May 17, 2025 06:57 AM
> 
> > Thanks, but I think you've misunderstood the comments.  We should only
> > ignore the ".gz" suffix.  Ignoring other suffixes would be a potential
> > regression, because there's nothing wrong in trying to load a file
> > called, say, foobar.so.abc.  The current code supports that, and I see
> > no reason to drop that support.
> 
> > The ".gz" suffix gets in the way because dynamic loaders cannot load
> > such compressed shared libraries.
> 
> > So the condition for not appending a suffix from
> > load-file-rep-suffixes should also test the value of 'ext', and only
> > refuse to append it if it compares equal with ".gz".
> 
> Yes, you're right.
> 
> According to the document of "load-file-rep-suffixes", the "jka-compr-load-suffixes" is used for the compressed suffixes.
> So I attach a new patch, the patch will seek the "jka-compr-load-suffixes" to determine the compressed suffixes for emacs module and skip it.
> Please check the attached patch for details. Thank you.

Thanks, I installed this on the master branch, and I'm closing the
bug.

P.S. Please see how I modified the commit log message, and try to
follow this style in the future.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.