Package: emacs;
Reported by: Eli Zaretskii <eliz <at> gnu.org>
Date: Sat, 7 Jan 2023 13:20:01 UTC
Severity: normal
Found in version 29.0.60
To reply to this bug, email your comments to 60628 AT debbugs.gnu.org.
Toggle the display of automated, internal messages from the tracker.
View this report as an mbox folder, status mbox, maintainer mbox
bug-gnu-emacs <at> gnu.org
:bug#60628
; Package emacs
.
(Sat, 07 Jan 2023 13:20:01 GMT) Full text and rfc822 format available.Eli Zaretskii <eliz <at> gnu.org>
:bug-gnu-emacs <at> gnu.org
.
(Sat, 07 Jan 2023 13:20:01 GMT) Full text and rfc822 format available.Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):
From: Eli Zaretskii <eliz <at> gnu.org> To: bug-gnu-emacs <at> gnu.org, Stefan Monnier <monnier <at> iro.umontreal.ca> Subject: 29.0.60; apropos-documentation does not work for preloaded symbols Date: Sat, 07 Jan 2023 15:20:07 +0200
To reproduce: emacs -Q C-h d list.*directory RET Note that the resulting *Apropos* buffer only lists built-in functions and variables, i.e. those defined in C. In particular, missing are functions/variables from files.el, such as find-directory-functions, parse-colon-path, and directory-files-recursively, whose doc strings all match the regexp. This seems to be due to commit 59732a83c8875c8986d2221600d559a24d8309cc Author: Stefan Monnier <monnier <at> iro.umontreal.ca> Date: Thu Dec 30 23:17:45 2021 -0500 Don't store docstrings of preloaded .el files in etc/DOC The doc string of apropos-documentation does say: Note that by default this command only searches in the file specified by ‘internal-doc-file-name’; i.e., the etc/DOC file. With C-u prefix, or if ‘apropos-do-all’ is non-nil, it searches all currently defined documentation strings. However, since we no longer write the doc strings of preloaded Lisp files to DOC, that default no longer makes sense: Emacs should not distinguish between built-in primitives and preloaded stuff, and it didn't up to and including Emacs 28. Moreover, "C-u C-h d" also doesn't work for preloaded functions, because: (apropos-documentation-internal 'directory-files-recursively) => nil This happens because: (apropos-safe-documentation 'directory-files-recursively) => ("files.elc" . 39940) and in that case apropos-documentation-internal calls apropos-documentation-check-elc-file, which does something very strange: it only looks inside strings that begin with "\n#@", and otherwise simply ignores everything else. Is this also a remnant from back when preloaded files had their doc strings in etc/DOC? Or else why does it only look for "#@" comments preceded by a newline? Basically, apropos-documentation is currently completely broken on the release branch. The following simple change seems to at least repair "C-u C-h d", but I don't think it's sufficient, as we need to resurrect the ability to find preloaded symbols without C-u. diff --git a/lisp/apropos.el b/lisp/apropos.el index b260d88..cff6f1c 100644 --- a/lisp/apropos.el +++ b/lisp/apropos.el @@ -1064,7 +1064,9 @@ apropos-documentation-check-elc-file (setq apropos-files-scanned (cons file apropos-files-scanned)) (erase-buffer) (insert-file-contents file) - (while (search-forward "\n#@" nil t) + (while (search-forward "#@" nil t) + (or (not (looking-at-p "[0-9]+")) + (progn ;; Read the comment length, and advance over it. (setq end (read) beg (1+ (point)) @@ -1110,7 +1112,7 @@ apropos-documentation-check-elc-file 'face apropos-match-face doc))) (setcar (nthcdr (if this-is-a-variable 3 2) apropos-item) - doc)))))))))) + doc)))))))))))) Ideas? patches? comments? In GNU Emacs 29.0.60 (build 197, i686-pc-mingw32) of 2023-01-07 built on HOME-C4E4A596F7 Repository revision: 9d410f8de64e91d16999a9bb5dd884d6d06d22bd Repository branch: emacs-29 Windowing system distributor 'Microsoft Corp.', version 5.1.2600 System Description: Microsoft Windows XP Service Pack 3 (v5.1.0.2600) Configured using: 'configure -C --prefix=/d/usr --with-wide-int --enable-checking=yes,glyphs 'CFLAGS=-O0 -gdwarf-4 -g3'' Configured features: ACL GIF GMP GNUTLS HARFBUZZ JPEG JSON LCMS2 LIBXML2 MODULES NOTIFY W32NOTIFY PDUMPER PNG RSVG SOUND SQLITE3 THREADS TIFF TOOLKIT_SCROLL_BARS TREE_SITTER WEBP XPM ZLIB Important settings: value of $LANG: ENU locale-coding-system: cp1255 Major mode: Lisp Interaction Minor modes in effect: tooltip-mode: t global-eldoc-mode: t eldoc-mode: t show-paren-mode: t electric-indent-mode: t mouse-wheel-mode: t tool-bar-mode: t menu-bar-mode: t file-name-shadow-mode: t global-font-lock-mode: t font-lock-mode: t blink-cursor-mode: t line-number-mode: t indent-tabs-mode: t transient-mark-mode: t auto-composition-mode: t auto-encryption-mode: t auto-compression-mode: t Load-path shadows: None found. Features: (shadow sort mail-extr dabbrev emacsbug message mailcap yank-media puny dired dired-loaddefs rfc822 mml mml-sec password-cache epa derived epg rfc6068 epg-config gnus-util text-property-search time-date subr-x mm-decode mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils cl-extra pp cl-print byte-opt gv bytecomp byte-compile help-fns radix-tree mule-util help-mode cl-loaddefs cl-lib apropos rmc iso-transl tooltip cconv eldoc paren electric uniquify ediff-hook vc-hooks lisp-float-type elisp-mode mwheel dos-w32 ls-lisp disp-table term/w32-win w32-win w32-vars term/common-win tool-bar dnd fontset image regexp-opt fringe tabulated-list replace newcomment text-mode lisp-mode prog-mode register page tab-bar menu-bar rfn-eshadow isearch easymenu timer select scroll-bar mouse jit-lock font-lock syntax font-core term/tty-colors frame minibuffer nadvice seq simple cl-generic indonesian philippine cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european ethiopic indian cyrillic chinese composite emoji-zwj charscript charprop case-table epa-hook jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button loaddefs theme-loaddefs faces cus-face macroexp files window text-properties overlay sha1 md5 base64 format env code-pages mule custom widget keymap hashtable-print-readable backquote threads w32notify w32 lcms2 multi-tty make-network-process emacs) Memory information: ((conses 16 54328 11479) (symbols 48 7273 0) (strings 16 20447 2422) (string-bytes 1 523245) (vectors 16 11853) (vector-slots 8 170903 17702) (floats 8 34 42) (intervals 40 449 141) (buffers 888 13))
bug-gnu-emacs <at> gnu.org
:bug#60628
; Package emacs
.
(Thu, 12 Jan 2023 09:37:01 GMT) Full text and rfc822 format available.Message #8 received at 60628 <at> debbugs.gnu.org (full text, mbox):
From: Eli Zaretskii <eliz <at> gnu.org> To: Stefan Monnier <monnier <at> iro.umontreal.ca> Cc: 60628 <at> debbugs.gnu.org Subject: Re: bug#60628: 29.0.60; apropos-documentation does not work for preloaded symbols Date: Thu, 12 Jan 2023 11:36:42 +0200
Ping! > Date: Sat, 07 Jan 2023 15:20:07 +0200 > From: Eli Zaretskii <eliz <at> gnu.org> > > To reproduce: > > emacs -Q > C-h d list.*directory RET > > Note that the resulting *Apropos* buffer only lists built-in functions > and variables, i.e. those defined in C. In particular, missing are > functions/variables from files.el, such as find-directory-functions, > parse-colon-path, and directory-files-recursively, whose doc strings > all match the regexp. > > This seems to be due to > > commit 59732a83c8875c8986d2221600d559a24d8309cc > Author: Stefan Monnier <monnier <at> iro.umontreal.ca> > Date: Thu Dec 30 23:17:45 2021 -0500 > > Don't store docstrings of preloaded .el files in etc/DOC > > The doc string of apropos-documentation does say: > > Note that by default this command only searches in the file specified by > ‘internal-doc-file-name’; i.e., the etc/DOC file. With C-u prefix, > or if ‘apropos-do-all’ is non-nil, it searches all currently defined > documentation strings. > > However, since we no longer write the doc strings of preloaded Lisp > files to DOC, that default no longer makes sense: Emacs should not > distinguish between built-in primitives and preloaded stuff, and it > didn't up to and including Emacs 28. > > Moreover, "C-u C-h d" also doesn't work for preloaded functions, > because: > > (apropos-documentation-internal 'directory-files-recursively) > => nil > > This happens because: > > (apropos-safe-documentation 'directory-files-recursively) > => ("files.elc" . 39940) > > and in that case apropos-documentation-internal calls > apropos-documentation-check-elc-file, which does something very > strange: it only looks inside strings that begin with "\n#@", and > otherwise simply ignores everything else. Is this also a remnant from > back when preloaded files had their doc strings in etc/DOC? Or else > why does it only look for "#@" comments preceded by a newline? > > Basically, apropos-documentation is currently completely broken on the > release branch. The following simple change seems to at least repair > "C-u C-h d", but I don't think it's sufficient, as we need to > resurrect the ability to find preloaded symbols without C-u. > > diff --git a/lisp/apropos.el b/lisp/apropos.el > index b260d88..cff6f1c 100644 > --- a/lisp/apropos.el > +++ b/lisp/apropos.el > @@ -1064,7 +1064,9 @@ apropos-documentation-check-elc-file > (setq apropos-files-scanned (cons file apropos-files-scanned)) > (erase-buffer) > (insert-file-contents file) > - (while (search-forward "\n#@" nil t) > + (while (search-forward "#@" nil t) > + (or (not (looking-at-p "[0-9]+")) > + (progn > ;; Read the comment length, and advance over it. > (setq end (read) > beg (1+ (point)) > @@ -1110,7 +1112,7 @@ apropos-documentation-check-elc-file > 'face apropos-match-face doc))) > (setcar (nthcdr (if this-is-a-variable 3 2) > apropos-item) > - doc)))))))))) > + doc)))))))))))) > > > Ideas? patches? comments? > > > > In GNU Emacs 29.0.60 (build 197, i686-pc-mingw32) of 2023-01-07 built on > HOME-C4E4A596F7 > Repository revision: 9d410f8de64e91d16999a9bb5dd884d6d06d22bd > Repository branch: emacs-29 > Windowing system distributor 'Microsoft Corp.', version 5.1.2600 > System Description: Microsoft Windows XP Service Pack 3 (v5.1.0.2600) > > Configured using: > 'configure -C --prefix=/d/usr --with-wide-int > --enable-checking=yes,glyphs 'CFLAGS=-O0 -gdwarf-4 -g3'' > > Configured features: > ACL GIF GMP GNUTLS HARFBUZZ JPEG JSON LCMS2 LIBXML2 MODULES NOTIFY > W32NOTIFY PDUMPER PNG RSVG SOUND SQLITE3 THREADS TIFF > TOOLKIT_SCROLL_BARS TREE_SITTER WEBP XPM ZLIB > > Important settings: > value of $LANG: ENU > locale-coding-system: cp1255 > > Major mode: Lisp Interaction > > Minor modes in effect: > tooltip-mode: t > global-eldoc-mode: t > eldoc-mode: t > show-paren-mode: t > electric-indent-mode: t > mouse-wheel-mode: t > tool-bar-mode: t > menu-bar-mode: t > file-name-shadow-mode: t > global-font-lock-mode: t > font-lock-mode: t > blink-cursor-mode: t > line-number-mode: t > indent-tabs-mode: t > transient-mark-mode: t > auto-composition-mode: t > auto-encryption-mode: t > auto-compression-mode: t > > Load-path shadows: > None found. > > Features: > (shadow sort mail-extr dabbrev emacsbug message mailcap yank-media puny > dired dired-loaddefs rfc822 mml mml-sec password-cache epa derived epg > rfc6068 epg-config gnus-util text-property-search time-date subr-x > mm-decode mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils > mailheader sendmail rfc2047 rfc2045 ietf-drums mm-util mail-prsvr > mail-utils cl-extra pp cl-print byte-opt gv bytecomp byte-compile > help-fns radix-tree mule-util help-mode cl-loaddefs cl-lib apropos rmc > iso-transl tooltip cconv eldoc paren electric uniquify ediff-hook > vc-hooks lisp-float-type elisp-mode mwheel dos-w32 ls-lisp disp-table > term/w32-win w32-win w32-vars term/common-win tool-bar dnd fontset image > regexp-opt fringe tabulated-list replace newcomment text-mode lisp-mode > prog-mode register page tab-bar menu-bar rfn-eshadow isearch easymenu > timer select scroll-bar mouse jit-lock font-lock syntax font-core > term/tty-colors frame minibuffer nadvice seq simple cl-generic > indonesian philippine cham georgian utf-8-lang misc-lang vietnamese > tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek > romanian slovak czech european ethiopic indian cyrillic chinese > composite emoji-zwj charscript charprop case-table epa-hook > jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button loaddefs > theme-loaddefs faces cus-face macroexp files window text-properties > overlay sha1 md5 base64 format env code-pages mule custom widget keymap > hashtable-print-readable backquote threads w32notify w32 lcms2 multi-tty > make-network-process emacs) > > Memory information: > ((conses 16 54328 11479) > (symbols 48 7273 0) > (strings 16 20447 2422) > (string-bytes 1 523245) > (vectors 16 11853) > (vector-slots 8 170903 17702) > (floats 8 34 42) > (intervals 40 449 141) > (buffers 888 13)) > > > >
bug-gnu-emacs <at> gnu.org
:bug#60628
; Package emacs
.
(Fri, 13 Jan 2023 15:47:01 GMT) Full text and rfc822 format available.Message #11 received at submit <at> debbugs.gnu.org (full text, mbox):
From: Stefan Monnier <monnier <at> iro.umontreal.ca> To: Eli Zaretskii <eliz <at> gnu.org> Cc: bug-gnu-emacs <at> gnu.org Subject: Re: 29.0.60; apropos-documentation does not work for preloaded symbols Date: Fri, 13 Jan 2023 10:46:21 -0500
> The doc string of apropos-documentation does say: > > Note that by default this command only searches in the file specified by > ‘internal-doc-file-name’; i.e., the etc/DOC file. With C-u prefix, > or if ‘apropos-do-all’ is non-nil, it searches all currently defined > documentation strings. > > However, since we no longer write the doc strings of preloaded Lisp > files to DOC, that default no longer makes sense: Emacs should not > distinguish between built-in primitives and preloaded stuff, and it > didn't up to and including Emacs 28. Well, we still do as documented. It's just that what is covered by ‘internal-doc-file-name’ is different now :-) IOW, we should also fix the doc so it doesn't define the behavior based on internal details such as ‘internal-doc-file-name’. We should probably say that by default we only search through the predefined functions or something like that. And as for actually (re)implementing that behavior, we can probably do it by replacing `mapatoms` with a loop through the elements of `load-history` corresponding to the `preloaded-file-list`. > and in that case apropos-documentation-internal calls > apropos-documentation-check-elc-file, which does something very > strange: it only looks inside strings that begin with "\n#@", and > otherwise simply ignores everything else. Is this also a remnant from > back when preloaded files had their doc strings in etc/DOC? Hmm... oh I see, it's because of that Stefan dude: commit 900b09c0235d54d56ef5e88d04cca61bc71cbbb7 Author: Stefan Monnier <monnier <at> iro.umontreal.ca> Date: Fri Aug 5 08:18:04 2022 -0400 bytecomp.el: Further simplifications enabled by commit 59732a83c8875c * lisp/emacs-lisp/bytecomp.el (byte-compile-output-docform): Don't insert a \n before the #@ docstrings since make-docfile doesn't scan .elc files any more. But searching for "#@" (with or without a \n) is not a good idea anyway, since there can be false positives and because once we've found such a "docstring" in a .elc file the problem of finding who this docstring belongs to again depends on details of how `bytecomp.el` decides to print out the code. For `emacs-29` the better option is probably to just revert the above commit. And for `master` we should probably rewrite the code so it uses the POS info in the (FILE . POS) instead of searching for #@, but that requires a significant rewrite. Basically, we should loop through all the functions, collecting their (FILE . POS) data, grouping them by FILE, so we end up for every FILE with a list of (FUNCTION . POS) so we can just jump to POS without having to search for #@ (we can still check the presence of #@ as sanity check). Stefan
bug-gnu-emacs <at> gnu.org
:bug#60628
; Package emacs
.
(Fri, 13 Jan 2023 20:11:02 GMT) Full text and rfc822 format available.Message #14 received at submit <at> debbugs.gnu.org (full text, mbox):
From: Stefan Monnier <monnier <at> iro.umontreal.ca> To: Eli Zaretskii <eliz <at> gnu.org> Cc: bug-gnu-emacs <at> gnu.org Subject: Re: 29.0.60; apropos-documentation does not work for preloaded symbols Date: Fri, 13 Jan 2023 15:10:02 -0500
> For `emacs-29` the better option is probably to just revert the > above commit. Then again, since \n#@ can also match false positives, we may as well keep the above commit and just search for #@ instead. The patch below should fix both problems in a kind of "minimalist" way (i.e. meant for `emacs-29`). Can someone confirm it also works for them while I clean it up? Stefan diff --git a/lisp/apropos.el b/lisp/apropos.el index b260d889955..b66f27b8a9e 100644 --- a/lisp/apropos.el +++ b/lisp/apropos.el @@ -913,8 +913,19 @@ (apropos-sort-by-scores apropos-documentation-sort-by-scores) f v sf sv) (apropos-documentation-check-doc-file) - (if do-all - (mapatoms + (funcall + (if do-all #'mapatoms + (lambda (f) + (let ((preloaded-regexp + (concat "\\`" + (regexp-quote lisp-directory) + (regexp-opt preloaded-file-list) + "\\.elc?\\'"))) + (dolist (x load-history) + (when (string-match preloaded-regexp (car x)) + (dolist (def (cdr x)) + (when (eq 'defun (car-safe def)) + (funcall f (cdr def))))))))) (lambda (symbol) (setq f (apropos-safe-documentation symbol) v (get symbol 'variable-documentation)) @@ -939,7 +950,7 @@ (cons (list symbol (+ (apropos-score-symbol symbol 2) sf sv) f v) - apropos-accumulator))))))) + apropos-accumulator)))))) (apropos-print nil "\n----------------\n" nil t)))) @@ -1064,10 +1064,12 @@ (setq apropos-files-scanned (cons file apropos-files-scanned)) (erase-buffer) (insert-file-contents file) - (while (search-forward "\n#@" nil t) + (while (search-forward "#@" nil t) ;; Read the comment length, and advance over it. - (setq end (read) - beg (1+ (point)) + ;; This #@ may be a false positive, so don't get upset if + ;; it's not followed by the expected number of bytes to skip. + (when (and (setq end (ignore-errors (read))) (natnump end)) + (setq beg (1+ (point)) end (+ (point) end -1)) (forward-char) (if (save-restriction @@ -1110,7 +1112,7 @@ 'face apropos-match-face doc))) (setcar (nthcdr (if this-is-a-variable 3 2) apropos-item) - doc)))))))))) + doc)))))))))))
bug-gnu-emacs <at> gnu.org
:bug#60628
; Package emacs
.
(Sat, 14 Jan 2023 09:30:02 GMT) Full text and rfc822 format available.Message #17 received at 60628 <at> debbugs.gnu.org (full text, mbox):
From: Eli Zaretskii <eliz <at> gnu.org> To: Stefan Monnier <monnier <at> iro.umontreal.ca> Cc: 60628 <at> debbugs.gnu.org Subject: Re: 29.0.60; apropos-documentation does not work for preloaded symbols Date: Sat, 14 Jan 2023 11:29:31 +0200
> From: Stefan Monnier <monnier <at> iro.umontreal.ca> > Cc: bug-gnu-emacs <at> gnu.org > Date: Fri, 13 Jan 2023 15:10:02 -0500 > > > For `emacs-29` the better option is probably to just revert the > > above commit. > > Then again, since \n#@ can also match false positives, we may as well > keep the above commit and just search for #@ instead. > > The patch below should fix both problems in a kind of "minimalist" way > (i.e. meant for `emacs-29`). > Can someone confirm it also works for them while I clean it up? It works much better, thanks. However, some symbols are still not shown, for some reason. For example, find-directory-functions and file-name-at-point-functions (both are defcustoms). I indeed don't see their doc strings in files.elc. Using "C-u C-h d" does produce an *Apropos* buffer with find-directory-functions in it, but file-name-at-point-functions are still not mentioned. I guess this is a separate issue, since Emacs 28 seems to have similar problems with those 2 defcustoms? Thanks.
bug-gnu-emacs <at> gnu.org
:bug#60628
; Package emacs
.
(Sat, 14 Jan 2023 14:28:02 GMT) Full text and rfc822 format available.Message #20 received at 60628 <at> debbugs.gnu.org (full text, mbox):
From: Stefan Monnier <monnier <at> iro.umontreal.ca> To: Eli Zaretskii <eliz <at> gnu.org> Cc: 60628 <at> debbugs.gnu.org Subject: Re: 29.0.60; apropos-documentation does not work for preloaded symbols Date: Sat, 14 Jan 2023 09:27:34 -0500
>> The patch below should fix both problems in a kind of "minimalist" way >> (i.e. meant for `emacs-29`). >> Can someone confirm it also works for them while I clean it up? > > It works much better, thanks. OK, thanks. I pushed a slightly better version of it to `emacs-29`. > However, some symbols are still not shown, for some reason. For > example, find-directory-functions and file-name-at-point-functions > (both are defcustoms). I indeed don't see their doc strings in > files.elc. Using "C-u C-h d" does produce an *Apropos* buffer with > find-directory-functions in it, but file-name-at-point-functions are > still not mentioned. I guess this is a separate issue, since Emacs 28 > seems to have similar problems with those 2 defcustoms? I pushed to master a rewrite of the .elc scanning code which may help (it wasn't its purpose, but it might be a side benefit). Stefan
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997,2003 nCipher Corporation Ltd,
1994-97 Ian Jackson.