GNU bug report logs - #65206
29.1; [windows][patch] build-deps-zips.py is broken and hard to maintain

Previous Next

To reply to this bug, email your comments to 65206 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Corwin Brust <corwin <at> bru.st>
To: bug-gnu-emacs <at> gnu.org
Subject: 29.1;
 [windows][patch] build-deps-zips.py is broken and hard to maintain
Date: Thu, 10 Aug 2023 07:40:48 -0500

[Message part 1 (text/plain, inline)]

The script nt/admin/dist-build/build-deps-zips.py needs help.  This is
the script that I use to collect and package dependencies and sources
for dependencies on Microsoft Windows, as part of releasing Emacs
binaries for Windows.  It is a python script that runs under MSYS2
MSYS console (not MinGW).

Neither the version currently in the emacs-29 nor in the master
branches will work for the given Emacs version without changes.  The
attached patch would make emacs-29 match what I am using locally.

In addition to other changes, the patch reflects my current "transformation map"
approach to deal with MSYS source package paths change, which seems to
be happening quite a bit upstream.

In case it may not be clear, my process is to run the script
after updating local MSYS packages that are dependencies (optional or
no), or edit and run it when Emacs' dependencies have changed.

The patch reflects the script as I have been using it during the Emacs
29 release process.  I'm sure there's general room for improvement
(editing this script is literally my only python coding credit), I'm
opening this bug report because bug#65188 (a packaging error preventing
WEBP from working for people using the Windows binaries) has called
attention to the importance of having additional eyes on build tooling
(especially when it so far contains hard-coded lists of upstream deps).

In GNU Emacs 29.1 (build 2, x86_64-w64-mingw32) of 2023-08-02 built on
 AVALON
Windowing system distributor 'Microsoft Corp.', version 10.0.19045
System Description: Microsoft Windows 10 Home (v10.0.2009.19045.3324)

Configured using:
 'configure --with-modules --without-dbus --with-native-compilation=aot
 --without-compress-install --with-tree-sitter CFLAGS=-O2'

Configured features:
ACL GIF GMP GNUTLS HARFBUZZ JPEG JSON LCMS2 LIBXML2 MODULES NATIVE_COMP
NOTIFY W32NOTIFY PDUMPER PNG RSVG SOUND SQLITE3 THREADS TIFF
TOOLKIT_SCROLL_BARS TREE_SITTER WEBP XPM ZLIB

Important settings:
  value of $LANG: ENU
  locale-coding-system: cp1252

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message mailcap yank-media puny dired
dired-loaddefs rfc822 mml mml-sec password-cache epa derived epg rfc6068
epg-config gnus-util text-property-search time-date mm-decode mm-bodies
mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader cl-loaddefs
comp comp-cstr warnings icons subr-x rx cl-seq cl-macs gv cl-extra
help-mode bytecomp byte-compile cl-lib sendmail rfc2047 rfc2045
ietf-drums mm-util mail-prsvr mail-utils rmc iso-transl tooltip cconv
eldoc paren electric uniquify ediff-hook vc-hooks lisp-float-type
elisp-mode mwheel dos-w32 ls-lisp disp-table term/w32-win w32-win
w32-vars term/common-win tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode lisp-mode prog-mode register
page tab-bar menu-bar rfn-eshadow isearch easymenu timer select
scroll-bar mouse jit-lock font-lock syntax font-core term/tty-colors
frame minibuffer nadvice seq simple cl-generic indonesian philippine
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese composite emoji-zwj charscript
charprop case-table epa-hook jka-cmpr-hook help abbrev obarray oclosure
cl-preloaded button loaddefs theme-loaddefs faces cus-face macroexp
files window text-properties overlay sha1 md5 base64 format env
code-pages mule custom widget keymap hashtable-print-readable backquote
threads w32notify w32 lcms2 multi-tty make-network-process
native-compile emacs)

Memory information:
((conses 16 80415 13861)
 (symbols 48 7175 0)
 (strings 32 21269 1764)
 (string-bytes 1 617449)
 (vectors 16 16398)
 (vector-slots 8 334731 15794)
 (floats 8 29 46)
 (intervals 56 238 0)
 (buffers 984 10))

[0001-Fix-Windows-build-dependancy-packaging-for-Emacs-29-.patch (application/octet-stream, attachment)]

Message #8 received at 65206 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Corwin Brust <corwin <at> bru.st>
Cc: 65206 <at> debbugs.gnu.org
Subject: Re: bug#65206: 29.1;
 [windows][patch] build-deps-zips.py is broken and hard to maintain
Date: Thu, 10 Aug 2023 16:29:12 +0300

> From: Corwin Brust <corwin <at> bru.st>
> Date: Thu, 10 Aug 2023 07:40:48 -0500
> 
> The script nt/admin/dist-build/build-deps-zips.py needs help.  This is
> the script that I use to collect and package dependencies and sources
> for dependencies on Microsoft Windows, as part of releasing Emacs
> binaries for Windows.  It is a python script that runs under MSYS2
> MSYS console (not MinGW).
> 
> Neither the version currently in the emacs-29 nor in the master
> branches will work for the given Emacs version without changes.  The
> attached patch would make emacs-29 match what I am using locally.
> 
> In addition to other changes, the patch reflects my current "transformation map"
> approach to deal with MSYS source package paths change, which seems to
> be happening quite a bit upstream.
> 
> In case it may not be clear, my process is to run the script
> after updating local MSYS packages that are dependencies (optional or
> no), or edit and run it when Emacs' dependencies have changed.
> 
> The patch reflects the script as I have been using it during the Emacs
> 29 release process.  I'm sure there's general room for improvement
> (editing this script is literally my only python coding credit), I'm
> opening this bug report because bug#65188 (a packaging error preventing
> WEBP from working for people using the Windows binaries) has called
> attention to the importance of having additional eyes on build tooling
> (especially when it so far contains hard-coded lists of upstream deps).

Would you mind providing an overview of the process by which the
script (and maybe some additional measures) collect(s) the list of the
dependency packages for the binary distro, including the main ideas
and information sources?  It is hard to glean all that from just a
patch, or even by reading the script, and I think that, given the
pains this gives, perhaps some new ideas are in order.

Thanks.

Message #11 received at 65206 <at> debbugs.gnu.org (full text, mbox):

From: Corwin Brust <corwin <at> bru.st>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 65206 <at> debbugs.gnu.org
Subject: Re: bug#65206: 29.1; [windows][patch] build-deps-zips.py is broken
 and hard to maintain
Date: Thu, 10 Aug 2023 16:09:15 -0500

On Thu, Aug 10, 2023 at 8:29 AM Eli Zaretskii <eliz <at> gnu.org> wrote:
>
>
> Would you mind providing an overview of the process by which the
> script (and maybe some additional measures) collect(s) the list of the
> dependency packages for the binary distro, including the main ideas
> and information sources?  It is hard to glean all that from just a
> patch, or even by reading the script, and I think that, given the
> pains this gives, perhaps some new ideas are in order.

I will happily try my best; sorry that it cannot be immediately/today.
In any case, I completely agree with you.

Message #14 received at 65206 <at> debbugs.gnu.org (full text, mbox):

From: Corwin Brust <corwin <at> bru.st>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 65206 <at> debbugs.gnu.org
Subject: Re: bug#65206: 29.1; [windows][patch] build-deps-zips.py is broken
 and hard to maintain
Date: Tue, 15 Aug 2023 02:39:45 -0500

On Thu, Aug 10, 2023 at 8:29 AM Eli Zaretskii <eliz <at> gnu.org> wrote:
>
> [Provide] an overview of the process by which

nt/admin/dist-build/build-dep-zips.py - it's really just that, as far
as how I collect the deps presently

To be clear, by "deps", I'm referring to DLLs that are not built with
Emacs but which we should (optionally) include when distributing
binary versions of Emacs for Windows.

Similarly pedantically, here are the files I (might) include for a
given release of Windows binaries for Emacs:

FULL ZIP: emacs-${VER}.zip - the "full zip", includes the deps DLLs
this script packages
NO DEPS ZIP: emacs-${VER}-no-deps.zip - specifically without the deps
DLLs this script packages
INSTALLER EXE: emacs-${VERION}-installer.exe - a self-installer
including the deps DLLs (and it compresses stuff even more than zip
-9. nice)
DEPS ZIP: emacs-${MAJOR_VERSION}-${DEPS_CREATE_DATE_IF_NOT_FIRST_AND_ONLY_FOR_MAJOR_VERSION}.zip
- contains only the deps DLLs. Created when changes are needed, more
to come on this.
DEPS SOURCES ZIP: the MSYS2/MINGW64 source code for all DLLs in DEPS
ZIP. Created only when a new DEPS ZIP is created

Finally (I swear, I'm going to answer your actual question), to place
the script in context of my present "use-case" (release and
release-similar builds specifically, here):

0. PREREQ: I have various folders setup, a working MSYS/MINGW64 that
has recently built me a working Emacs, I've been watching mailing
lists, I have ACLs going for me, etc.

1. A (pre)release tarball is pushed to GNU (alpha) FTP.

2. Skip this step if a "last good" deps and deps source file exist and
it is current (i.e. when there are not known new or updated
--including optional-- dependendencies for Emacs to include for the
first time),  Otherwise, update and run the script (more on this
below, obviously) in question to create (new) deps and deps source
archives containing, respectively, a bunch of DLLs we plan to
distribute (nominally) with Emacs (this is the deps archive), and the
MSYS2/MINGW64 sources for these DLLs and their compile time
dependencies (the deps sources archive).

3. Download and unpack the tarball manually

4. Configure && make install  to ${PREFIX}. Note, I have a tedious
pipeline command I like for this; I'm not currently using
"build-zips.sh" from the same folder as the build-dep-zips.sh script
in question.

5. Create a no-deps archive, essentially:

  zip -r9 ${PREFIX} emacs-${TARGET_VERSION_NAME}-no-deps.zip

6. Unpack the deps archive created in step #2 (or the "last good", if
that step was skipped), something like:

  unzip -d ${PREFIX}/bin ${LAST_GOOD_DEPS_ZIP_FOR_EMACS_MAJOR_VERSION}

7. Create a "full" zip of Emacs (now including those "extra" DLLs),
essentially repeating step #5 with a different archive file name  (in
fact, I copy no-deps and re-add bin to it)

8. Create the installer

9. Perform certain rights and incantations to cause files to appear on
GNU (alpha) FTP servers.  Notably, include new deps files ONLY when
they have been newly created along with the release being published.
These files have historically changed only rarely within a given Emacs
major version.

> collect(s) the list of the dependency packages for the binary distro,

Ignoring my patch for the moment, the script contains two hard-coded
lists that represent our first likely source of breakage:

DLL_REQ lists specific DLLs that should be copied into ${PREFIX}/bin,
after make install, to get a "complete" Emacs installation.  During
the Emacs 29 pre-release cycle I added sqlite3 and tree-sitter to this
list, enabling those features to work "out of the box".  In some
cases, such as these two in fact, Emacs would likely function
correctly under Windows if we chose not to distribute a particular
DLL; however, I believe this is not the case for all of DLLs included
and, moreover (in my opinion) would tend to make Emacs to less viable
as a means of drawing Windows users closer to Free Software by virtue
of Just Being Better, which would be a bit sad.

PKG_REQ lists the mingw-w64-x86_64 source package name for each item
in the first list.

These two variables are coordinated lists and so obviously could be
refactored (e.g. into an associative array) that I suspect historical
raisins (meaning, perhaps once these lists were not so coordinated; I
didn't research this so far).

In addition to these "main lists" there are a few other vars/lines to
study (now taking from my patched version):

SKIP_SRC_PKGS=["mingw-w64-gcc-libs"]
SKIP_DEP_PKGS=["mingw-w64-glib2" "mingw-w64-ca-certificates-20211016-3"]

We'll come to the logic -where all of these are used-- next.

> including the main ideas and information sources[.  ...]

Ignoring as much complexity as possible by focusing on the
release(like) use-case, we can ignore the arguments and conditions
and, finally, we can reduce the the script to:

2.1 evaluate above mentioned vars
2.2 collect all DLLs mentioned in DLL_REQ
2.3 collect the DLLs that are unique dependencies the DLLs collected
in 2.2 skipping any which appear in SKIP_DEP_PKGS
2.4 collect the source for all DLLs from 2.2 and 2.3 unless the source
package is listed in SKIP_SRC_PKGS

I hope I did cover information sources well enough, but to put a fine
point on it: I watch Emacs devel, bug reports, IRC, reddit, ..., for
information leading to updates to this script. Developers and others
using Emacs on Windows are the main "information sources", at least
speaking from developing that patch :)

>
> Thanks.
>

Message #17 received at 65206 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Corwin Brust <corwin <at> bru.st>
Cc: 65206 <at> debbugs.gnu.org
Subject: Re: bug#65206: 29.1; [windows][patch] build-deps-zips.py is broken
 and hard to maintain
Date: Tue, 15 Aug 2023 18:43:45 +0300

> From: Corwin Brust <corwin <at> bru.st>
> Date: Tue, 15 Aug 2023 02:39:45 -0500
> Cc: 65206 <at> debbugs.gnu.org
> 
> 5. Create a no-deps archive, essentially:
> 
>   zip -r9 ${PREFIX} emacs-${TARGET_VERSION_NAME}-no-deps.zip
> 
> 6. Unpack the deps archive created in step #2 (or the "last good", if
> that step was skipped), something like:
> 
>   unzip -d ${PREFIX}/bin ${LAST_GOOD_DEPS_ZIP_FOR_EMACS_MAJOR_VERSION}
> 
> 7. Create a "full" zip of Emacs (now including those "extra" DLLs),
> essentially repeating step #5 with a different archive file name  (in
> fact, I copy no-deps and re-add bin to it)
> 
> 8. Create the installer
> 
> 9. Perform certain rights and incantations to cause files to appear on
> GNU (alpha) FTP servers.  Notably, include new deps files ONLY when
> they have been newly created along with the release being published.
> These files have historically changed only rarely within a given Emacs
> major version.
> 
> > collect(s) the list of the dependency packages for the binary distro,
> 
> Ignoring my patch for the moment, the script contains two hard-coded
> lists that represent our first likely source of breakage:
> 
> DLL_REQ lists specific DLLs that should be copied into ${PREFIX}/bin,
> after make install, to get a "complete" Emacs installation.  During
> the Emacs 29 pre-release cycle I added sqlite3 and tree-sitter to this
> list, enabling those features to work "out of the box".  In some
> cases, such as these two in fact, Emacs would likely function
> correctly under Windows if we chose not to distribute a particular
> DLL; however, I believe this is not the case for all of DLLs included
> and, moreover (in my opinion) would tend to make Emacs to less viable
> as a means of drawing Windows users closer to Free Software by virtue
> of Just Being Better, which would be a bit sad.
> 
> PKG_REQ lists the mingw-w64-x86_64 source package name for each item
> in the first list.
> 
> These two variables are coordinated lists and so obviously could be
> refactored (e.g. into an associative array) that I suspect historical
> raisins (meaning, perhaps once these lists were not so coordinated; I
> didn't research this so far).
> 
> In addition to these "main lists" there are a few other vars/lines to
> study (now taking from my patched version):
> 
> SKIP_SRC_PKGS=["mingw-w64-gcc-libs"]
> SKIP_DEP_PKGS=["mingw-w64-glib2" "mingw-w64-ca-certificates-20211016-3"]
> 
> We'll come to the logic -where all of these are used-- next.
> 
> > including the main ideas and information sources[.  ...]
> 
> Ignoring as much complexity as possible by focusing on the
> release(like) use-case, we can ignore the arguments and conditions
> and, finally, we can reduce the the script to:
> 
> 2.1 evaluate above mentioned vars
> 2.2 collect all DLLs mentioned in DLL_REQ
> 2.3 collect the DLLs that are unique dependencies the DLLs collected
> in 2.2 skipping any which appear in SKIP_DEP_PKGS
> 2.4 collect the source for all DLLs from 2.2 and 2.3 unless the source
> package is listed in SKIP_SRC_PKGS
> 
> I hope I did cover information sources well enough, but to put a fine
> point on it: I watch Emacs devel, bug reports, IRC, reddit, ..., for
> information leading to updates to this script. Developers and others
> using Emacs on Windows are the main "information sources", at least
> speaking from developing that patch :)

Thanks.  What I still don't think I understand is how do you make sure
you have a full list of first-order dependencies?  I understand that
you mostly build on the "last good" list from previous release, but
since the list grows from time to time, what is the procedure for
finding the new dependencies, adding them to the list, and making sure
they all are there?

I'm asking because this is exactly where the procedure broke down when
we added WebP image support in Emacs 29.

Message #20 received at 65206 <at> debbugs.gnu.org (full text, mbox):

From: Corwin Brust <corwin <at> bru.st>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 65206 <at> debbugs.gnu.org
Subject: Re: bug#65206: 29.1; [windows][patch] build-deps-zips.py is broken
 and hard to maintain
Date: Tue, 15 Aug 2023 10:53:55 -0500

On Tue, Aug 15, 2023 at 10:43 AM Eli Zaretskii <eliz <at> gnu.org> wrote:
>
> > From: Corwin Brust <corwin <at> bru.st>
> >
> > I hope I did cover information sources well enough, but to put a fine
> > point on it: I watch Emacs devel, bug reports, IRC, reddit, ..., for
> > information leading to updates to this script. Developers and others
> > using Emacs on Windows are the main "information sources", at least
> > speaking from developing that patch :)
>
> Thanks.  What I still don't think I understand is how do you make sure
> you have a full list of first-order dependencies?  I understand that
> you mostly build on the "last good" list from previous release, but
> since the list grows from time to time, what is the procedure for
> finding the new dependencies, adding them to the list, and making sure
> they all are there?
>
> I'm asking because this is exactly where the procedure broke down when
> we added WebP image support in Emacs 29.

The above quote from myself I retained is my best/only answer:

I update the script as issues are reported or when I somehow otherwise
learn that changes are needed.  I have no real process for this, and
nothing in the tooling is helping me with it, so far.

Message #23 received at 65206 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Corwin Brust <corwin <at> bru.st>
Cc: 65206 <at> debbugs.gnu.org
Subject: Re: bug#65206: 29.1; [windows][patch] build-deps-zips.py is broken
 and hard to maintain
Date: Tue, 15 Aug 2023 19:01:53 +0300

> From: Corwin Brust <corwin <at> bru.st>
> Date: Tue, 15 Aug 2023 10:53:55 -0500
> Cc: 65206 <at> debbugs.gnu.org
> 
> > Thanks.  What I still don't think I understand is how do you make sure
> > you have a full list of first-order dependencies?  I understand that
> > you mostly build on the "last good" list from previous release, but
> > since the list grows from time to time, what is the procedure for
> > finding the new dependencies, adding them to the list, and making sure
> > they all are there?
> >
> > I'm asking because this is exactly where the procedure broke down when
> > we added WebP image support in Emacs 29.
> 
> The above quote from myself I retained is my best/only answer:
> 
> I update the script as issues are reported or when I somehow otherwise
> learn that changes are needed.  I have no real process for this, and
> nothing in the tooling is helping me with it, so far.

OK, so here's a suggestion which might improve that crucial part: scan
the list in dynamic-library-alist, on lisp/term/w32-win.el.  Every
dependency that is loaded dynamically (i.e., Emacs is not linked
against it when it is built) must be in that list.  So when we add
dependencies, we add them there.

I believe that given a full and complete lest of first-order
dependencies, those which Emacs actually loads, all the higher-order
dependencies can be found by following the MSYS2 pacman etc., is that
right?

Message #26 received at 65206 <at> debbugs.gnu.org (full text, mbox):

From: Corwin Brust <corwin <at> bru.st>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 65206 <at> debbugs.gnu.org
Subject: Re: bug#65206: 29.1; [windows][patch] build-deps-zips.py is broken
 and hard to maintain
Date: Tue, 15 Aug 2023 20:23:44 -0500

On Tue, Aug 15, 2023 at 11:01 AM Eli Zaretskii <eliz <at> gnu.org> wrote:
>
> >
> > I update the script as issues are reported or when I somehow otherwise
> > learn that changes are needed.  I have no real process for this, and
> > nothing in the tooling is helping me with it, so far.
>
> OK, so here's a suggestion which might improve that crucial part: scan
> the list in dynamic-library-alist, on lisp/term/w32-win.el.  Every
> dependency that is loaded dynamically (i.e., Emacs is not linked
> against it when it is built) must be in that list.  So when we add
> dependencies, we add them there.

I looked at the variable.   OT1H, it serves a very different use-case
("what are valid DLL names for a given library?" in the run-time, vs
"what DLLs should be sent along with Emacs?" in the build-time).
This means that meaningful hackery would likely be needed to
contemplate removing the hard-coded list completely, or even that we
would not be able to device any means of parsing this and choosing the
correct sent among DLLs present on the build system, to include.

OTOH, and more directly to the point of this bug report:

>
> I believe that given a full and complete lest of first-order
> dependencies, those which Emacs actually loads, all the higher-order
> dependencies can be found by following the MSYS2 pacman etc., is that
> right?
>

Right, that is how the script presently works:

    package_info = check_output(["pacman", "-Si",
pkg]).decode("utf-8").split("\n")

Thus, if we are content to have the script detect, and error demanding
correction when out of sync wrt `dynamic-library-alist', I believe it
can be done.  Moreover, IMO this will definitely help.

One process note is that I would likely switch to (maybe) generating
new DEPS right after creating NO DEPS, so the script can count on
invoking the freshly compiled and locally installed Emacs.  This also
seems much easier and also more proper vs anything to parse it "the
hard way".  (Especially so, considering complexities such as format
strings and the libpng version comment on the source setting up the
default value of dynamic-module-alist.)

Does a "invokes Emacs now and errors out if stuff is missing" approach
sound right/good?
Is it too soon for me to try making a new patch that explores this idea further?

Message #29 received at 65206 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Corwin Brust <corwin <at> bru.st>
Cc: 65206 <at> debbugs.gnu.org
Subject: Re: bug#65206: 29.1; [windows][patch] build-deps-zips.py is broken
 and hard to maintain
Date: Wed, 16 Aug 2023 15:08:44 +0300

> From: Corwin Brust <corwin <at> bru.st>
> Date: Tue, 15 Aug 2023 20:23:44 -0500
> Cc: 65206 <at> debbugs.gnu.org
> 
> On Tue, Aug 15, 2023 at 11:01 AM Eli Zaretskii <eliz <at> gnu.org> wrote:
> >
> > OK, so here's a suggestion which might improve that crucial part: scan
> > the list in dynamic-library-alist, on lisp/term/w32-win.el.  Every
> > dependency that is loaded dynamically (i.e., Emacs is not linked
> > against it when it is built) must be in that list.  So when we add
> > dependencies, we add them there.
> 
> I looked at the variable.   OT1H, it serves a very different use-case
> ("what are valid DLL names for a given library?" in the run-time, vs
> "what DLLs should be sent along with Emacs?" in the build-time).
> This means that meaningful hackery would likely be needed to
> contemplate removing the hard-coded list completely, or even that we
> would not be able to device any means of parsing this and choosing the
> correct sent among DLLs present on the build system, to include.

I'm not sure I understand the reservation.  That list mentions every
single DLL that we know can be used for each optional feature.  If a
feature has more than one DLL listed, the first one is usually the
most popular, and should be tried first.  Given this, what problems do
you envision with using that list?

> Thus, if we are content to have the script detect, and error demanding
> correction when out of sync wrt `dynamic-library-alist', I believe it
> can be done.  Moreover, IMO this will definitely help.

Great, that's what I hoped to achieve: a way of verifying that your
list of first-order dependencies is complete.

> Does a "invokes Emacs now and errors out if stuff is missing" approach
> sound right/good?

I'm not sure I understand how would you force Emacs to "error out"
when we are talking about optional dependencies.  They are optional so
that Emacs could run even if they are not present.

Message #32 received at 65206 <at> debbugs.gnu.org (full text, mbox):

From: Corwin Brust <corwin <at> bru.st>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 65206 <at> debbugs.gnu.org
Subject: Re: bug#65206: 29.1; [windows][patch] build-deps-zips.py is broken
 and hard to maintain
Date: Wed, 16 Aug 2023 08:41:25 -0500

On Wed, Aug 16, 2023 at 7:08 AM Eli Zaretskii <eliz <at> gnu.org> wrote:
>
> > From: Corwin Brust <corwin <at> bru.st>
> > Date: Tue, 15 Aug 2023 20:23:44 -0500
> > Cc: 65206 <at> debbugs.gnu.org
> >
> > On Tue, Aug 15, 2023 at 11:01 AM Eli Zaretskii <eliz <at> gnu.org> wrote:
> > >
> > > the list in dynamic-library-alist, on lisp/term/w32-win.el.
>
> I'm not sure I understand the reservation.  That list mentions every
> single DLL that we know can be used for each optional feature.  If a
> feature has more than one DLL listed, the first one is usually the
> most popular, and should be tried first.

This solves my worry completely, or nearly so.

To confirm: when walking the list, I will want to take the first DLL
mentioned that actually exists for each entry. Is that right?

> Given this, what problems do you envision with using that list?

There might not be a problem (except the one we are trying to fix).
The alist contains 22 entries, while var DLL_REQ contains 14 entries.
The five on the alist but on mentioned in the script (so far) are:

gdiplus
shlwapi
gobject
gio
webpdemux - this is pretty obviously a miss in the script; it does get
however because it's required by webp which is listed in DLL_REQ

Are all of these errors with the script (so, the corresponding DLLs
should be included)?  If not, I think we will need a way for the
script to know which alist entries to skip/ignore.

> > Does a "invokes Emacs now and errors out if stuff is missing" approach
> > sound right/good?
>
> I'm not sure I understand how would you force Emacs to "error out"
> when we are talking about optional dependencies.  They are optional so
> that Emacs could run even if they are not present.
>

Oops, badly said:  I mean that my build and packaging process should
stop and report an error if it cannot create a "complete" DEPS ZIP.
Nothing affecting the Emacs run-time.

Message #35 received at 65206 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Corwin Brust <corwin <at> bru.st>
Cc: 65206 <at> debbugs.gnu.org
Subject: Re: bug#65206: 29.1; [windows][patch] build-deps-zips.py is broken
 and hard to maintain
Date: Wed, 16 Aug 2023 17:49:28 +0300

> From: Corwin Brust <corwin <at> bru.st>
> Date: Wed, 16 Aug 2023 08:41:25 -0500
> Cc: 65206 <at> debbugs.gnu.org
> 
> On Wed, Aug 16, 2023 at 7:08 AM Eli Zaretskii <eliz <at> gnu.org> wrote:
> >
> > I'm not sure I understand the reservation.  That list mentions every
> > single DLL that we know can be used for each optional feature.  If a
> > feature has more than one DLL listed, the first one is usually the
> > most popular, and should be tried first.
> 
> This solves my worry completely, or nearly so.
> 
> To confirm: when walking the list, I will want to take the first DLL
> mentioned that actually exists for each entry. Is that right?

Yes.

> There might not be a problem (except the one we are trying to fix).
> The alist contains 22 entries, while var DLL_REQ contains 14 entries.
> The five on the alist but on mentioned in the script (so far) are:
> 
> gdiplus
> shlwapi

You can ignore those: they are for Windows 9X, and they are system
DLLs.

> gobject
> gio

These are needed for librsvg.  You might get away with them because
librsvg pulls them in as second-order dependencies.

> webpdemux - this is pretty obviously a miss in the script; it does get
> however because it's required by webp which is listed in DLL_REQ

Yes, this is required by libwebp, since some of the library functions
are in lobwebpdemux.

> Are all of these errors with the script (so, the corresponding DLLs
> should be included)?  If not, I think we will need a way for the
> script to know which alist entries to skip/ignore.

You should only skip the first two, which are Windows system DLLs.

> > > Does a "invokes Emacs now and errors out if stuff is missing" approach
> > > sound right/good?
> >
> > I'm not sure I understand how would you force Emacs to "error out"
> > when we are talking about optional dependencies.  They are optional so
> > that Emacs could run even if they are not present.
> >
> 
> Oops, badly said:  I mean that my build and packaging process should
> stop and report an error if it cannot create a "complete" DEPS ZIP.
> Nothing affecting the Emacs run-time.

OK.

Message #38 received at 65206 <at> debbugs.gnu.org (full text, mbox):

From: Corwin Brust <corwin <at> bru.st>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 65206 <at> debbugs.gnu.org
Subject: Re: bug#65206: 29.1; [windows][patch] build-deps-zips.py is broken
 and hard to maintain
Date: Thu, 17 Aug 2023 02:25:27 -0500

[Message part 1 (text/plain, inline)]

On Wed, Aug 16, 2023 at 9:49 AM Eli Zaretskii <eliz <at> gnu.org> wrote:
> >
> > To confirm: when walking the list, I will want to take the first DLL
> > mentioned that actually exists for each entry. Is that right?
>
> Yes.
>

The attached patch replaces DLL_REQ (the var holding the "starting"
list of DLLs) with an invocation of emacs batch, as described above.
Once this patch is installed, the script will require the (path to
the) nominally newly created Emacs binary as an argument.

I'm still looking at the differences between outputs from this and
from the script as from my prior patch; however, this runs without
error which will improve the situation if it is applied.  (The patch
is for emacs-29 because I expect Emacs 29.2 will be the next release
packaged.)

>
> You should only skip the first two, which are Windows system DLLs.
>
> > > > Does a "invokes Emacs now and errors out if stuff is missing" approach
> > > > sound right/good?
> > >
> > > I'm not sure I understand how would you force Emacs to "error out"
> > > when we are talking about optional dependencies.  They are optional so
> > > that Emacs could run even if they are not present.
> > >
> >
> > Oops, badly said:  I mean that my build and packaging process should
> > stop and report an error if it cannot create a "complete" DEPS ZIP.
> > Nothing affecting the Emacs run-time.
>
> OK.
>

The patch does not attempt this: I simply remove nil from the "first
present DLL" sweep.

I did verify, currently, only gdiplus and shlwapi return nil, thus the
rest mentioned here are being included (except libgccjig which the
script --still-- expressly omits).

(mapcar (lambda(lib)
(list (car lib)
       (seq-find
(lambda(file)
  (file-exists-p
   (file-name-concat "c:/msys2/mingw64/bin"
     file)))
(cdr lib))))
       dynamic-library-alist)

((gdiplus nil) (shlwapi nil) (xpm "libXpm-nox4.dll")
 (png "libpng16-16.dll") (tiff "libtiff-5.dll") (jpeg "libjpeg-8.dll")
 (gif "libgif-7.dll") (svg "librsvg-2-2.dll") (webp "libwebp-7.dll")
 (webpdemux "libwebpdemux-2.dll") (sqlite3 "libsqlite3-0.dll")
 (gdk-pixbuf "libgdk_pixbuf-2.0-0.dll") (glib "libglib-2.0-0.dll")
 (gio "libgio-2.0-0.dll") (gobject "libgobject-2.0-0.dll")
 (gnutls "libgnutls-30.dll") (libxml2 "libxml2-2.dll")
 (zlib "zlib1.dll") (lcms2 "liblcms2-2.dll") (json "libjansson-4.dll")
 (gccjit "libgccjit-0.dll") (tree-sitter "libtree-sitter.dll"))

[0001-Replace-hard-coded-DLL-list-with-emacs-batch-for-nt.patch (application/octet-stream, attachment)]

Message #41 received at 65206 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Corwin Brust <corwin <at> bru.st>
Cc: 65206 <at> debbugs.gnu.org
Subject: Re: bug#65206: 29.1; [windows][patch] build-deps-zips.py is broken
 and hard to maintain
Date: Thu, 17 Aug 2023 12:55:55 +0300

> From: Corwin Brust <corwin <at> bru.st>
> Date: Thu, 17 Aug 2023 02:25:27 -0500
> Cc: 65206 <at> debbugs.gnu.org
> 
> The attached patch replaces DLL_REQ (the var holding the "starting"
> list of DLLs) with an invocation of emacs batch, as described above.
> Once this patch is installed, the script will require the (path to
> the) nominally newly created Emacs binary as an argument.
> 
> I'm still looking at the differences between outputs from this and
> from the script as from my prior patch; however, this runs without
> error which will improve the situation if it is applied.  (The patch
> is for emacs-29 because I expect Emacs 29.2 will be the next release
> packaged.)

Hmm... what is the value of libjpeg-version?  It's 90 here, so the
library for JPEG is libjpeg-9.dll, not libjpeg-8.dll.

Otherwise, LGTM, thanks.

Message #44 received at 65206 <at> debbugs.gnu.org (full text, mbox):

From: Corwin Brust <corwin <at> bru.st>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 65206 <at> debbugs.gnu.org
Subject: Re: bug#65206: 29.1; [windows][patch] build-deps-zips.py is broken
 and hard to maintain
Date: Thu, 17 Aug 2023 08:31:12 -0500

On Thu, Aug 17, 2023 at 4:55 AM Eli Zaretskii <eliz <at> gnu.org> wrote:
>
>
> Hmm... what is the value of libjpeg-version?  It's 90 here, so the
> library for JPEG is libjpeg-9.dll, not libjpeg-8.dll.

I have 80

>
> Otherwise, LGTM, thanks.
>

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.