GNU bug report logs - #55645
src/print.c; print_object changes make it impossible to compare elisp code across versions

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 55645 in the body.
You can then email your comments to 55645 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Tom Gillespie <tgbugs <at> gmail.com>
To: Emacs Bug Report <bug-gnu-emacs <at> gnu.org>
Subject: src/print.c; print_object changes make it impossible to compare elisp
 code across versions
Date: Wed, 25 May 2022 16:12:02 -0700

The changes made in 637dde4aba921435f78d0de769ad74c4f3230aa6
make it impossible to use prin1-to-string as the basis for hashing elisp
code in a way that is independent of the emacs version.

The change completely breaks my orgstrap library across the 28 -> 29
boundary because the checksum computed on <= 28's prin1-to-string
and the checksum computed by 29's prin1-to-string no long match in
cases where there is a . or a ? in a symbol.

As implemented, there is no way to work around the issue, and I
suspect that many other users will be surprised and rather upset
by this change when they have the misfortune to encounter it.

At the very least this change in behavior needs to be configurable so that
the old behavior can be retained, preferably this change would be
reverted entirely as it does not address the original bug (bug#23130) and
is NOT low impact for users that actually use such functionality, whereas
the change is completely invisible to those that do not use such functionality.

Message #8 received at 55645 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Tom Gillespie <tgbugs <at> gmail.com>
Cc: 55645 <at> debbugs.gnu.org
Subject: Re: bug#55645: src/print.c; print_object changes make it impossible
 to compare elisp code across versions
Date: Thu, 26 May 2022 14:03:36 +0200

Tom Gillespie <tgbugs <at> gmail.com> writes:

> The changes made in 637dde4aba921435f78d0de769ad74c4f3230aa6
> make it impossible to use prin1-to-string as the basis for hashing elisp
> code in a way that is independent of the emacs version.

prin1 has never guaranteed that the printed representation would be
stable across versions, so if it's worked for you, that's by accident.

> At the very least this change in behavior needs to be configurable so that
> the old behavior can be retained, preferably this change would be
> reverted entirely as it does not address the original bug (bug#23130) and
> is NOT low impact for users that actually use such functionality, whereas
> the change is completely invisible to those that do not use such functionality.

It would be possible to add something to control this details (via the
new OVERRIDES parameter), but it doesn't feel like it'd be worth it,
because, well, the representation isn't meant to be stable.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #13 received at 55645 <at> debbugs.gnu.org (full text, mbox):

From: Tom Gillespie <tgbugs <at> gmail.com>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: 55645 <at> debbugs.gnu.org
Subject: Re: bug#55645: src/print.c; print_object changes make it impossible
 to compare elisp code across versions
Date: Thu, 26 May 2022 08:56:19 -0700

> prin1 has never guaranteed that the printed representation would be
> stable across versions, so if it's worked for you, that's by accident.

It may not be meant to be stable but this behavior is unchanged since
at least emacs-23, and given that xemacs has the same behavior, I'm
guessing that this behavior has been stable for nearly 30 years.

If prin1 is not stable, is there a print that is expected to be stable?

> It would be possible to add something to control this details (via the
> new OVERRIDES parameter), but it doesn't feel like it'd be worth it,
> because, well, the representation isn't meant to be stable.

Whether it is meant to be stable or not it has been stable for at least
30 years. If there is no print variant that does have stability guarantees
then I think we do need a dynamic variable that would have the old
behavior when it was nil, like those listed in Variables Affecting Output
https://www.gnu.org/software/emacs/manual/html_node/elisp/Output-Variables.html

Message #16 received at 55645 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Tom Gillespie <tgbugs <at> gmail.com>
Cc: 55645 <at> debbugs.gnu.org
Subject: Re: bug#55645: src/print.c; print_object changes make it impossible
 to compare elisp code across versions
Date: Fri, 27 May 2022 12:33:16 +0200

Tom Gillespie <tgbugs <at> gmail.com> writes:

> If prin1 is not stable, is there a print that is expected to be stable?

I don't think so.  We tweak the Lisp printers from time to time (for
instance, circular objects changed their representation the other day
(again)), but admittedly, these changes are usually for more obscure
objects than symbols.

> Whether it is meant to be stable or not it has been stable for at least
> 30 years.

prin1?  Stable for 30 years?  Nope.  (But some objects have had a stable
printed representation that long, I'm sure.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #19 received at 55645 <at> debbugs.gnu.org (full text, mbox):

From: Tom Gillespie <tgbugs <at> gmail.com>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: 55645 <at> debbugs.gnu.org
Subject: Re: bug#55645: src/print.c; print_object changes make it impossible
 to compare elisp code across versions
Date: Fri, 27 May 2022 12:19:27 -0700

> but admittedly, these changes are usually for more obscure
> objects than symbols

I was thinking the same. I realize that for many objects where
there is no guarantee of homoiconicity the prin1 representation
definitely changes all the time. However, I think that symbols
fall into a more homoiconic category (namely the category
of objects that can appear literally in code).

Some time between emacs 18 and xemacs/22 the reader
DID change with regard to periods in symbols. In 18 you had
to escape the period or the reader throws a syntax error.

Thus your point about there being a mismatch between read
behavior and prin1 representation is accurate.

That being said, I think this is be the first time in the history
of emacs that prin1 for symbols has changed, so having a
way to preserve the old behavior for things that depend on it
would be greatly appreciated.

> prin1?  Stable for 30 years?  Nope.  (But some objects have had a stable
> printed representation that long, I'm sure.)

Er, indeed, I was being overly general, I meant symbols specifically.
I went back and checked in emacs-18 and this is what I get:

(prin1-to-string '(a b\.c d))
"(a b\\.c d)"
(emacs-version)
"GNU Emacs 18.59.1 of Sat Jul 31 2021 on localhost (linux)"

Message #22 received at 55645 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Tom Gillespie <tgbugs <at> gmail.com>
Cc: 55645 <at> debbugs.gnu.org
Subject: Re: bug#55645: src/print.c; print_object changes make it impossible
 to compare elisp code across versions
Date: Sat, 28 May 2022 12:39:58 +0200

Tom Gillespie <tgbugs <at> gmail.com> writes:

> That being said, I think this is be the first time in the history
> of emacs that prin1 for symbols has changed, so having a
> way to preserve the old behavior for things that depend on it
> would be greatly appreciated.

That's makes sense, but:

>> prin1?  Stable for 30 years?  Nope.  (But some objects have had a stable
>> printed representation that long, I'm sure.)
>
> Er, indeed, I was being overly general, I meant symbols specifically.

If you're using prin1 as a way to compare stably differences between
objects, this only works by accident: You have to be very lucky in which
objects you have included in the output so far if you've managed to make
this code work over time.

We can indeed add something that lets you control how symbols in
specific are output by prin1, but I don't really see the utility in
general -- because we have changed, and will change, how other objects
are output by prin1, and if the objects you're outputting happens to
contain these, your algo will break again.

So I think this is one of those "don't do that" things, unfortunately.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #25 received at 55645 <at> debbugs.gnu.org (full text, mbox):

From: Tom Gillespie <tgbugs <at> gmail.com>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: 55645 <at> debbugs.gnu.org
Subject: Re: bug#55645: src/print.c; print_object changes make it impossible
 to compare elisp code across versions
Date: Sat, 28 May 2022 10:54:25 -0700

> If you're using prin1 as a way to compare stably differences between
> objects, this only works by accident: You have to be very lucky in which
> objects you have included in the output so far if you've managed to make
> this code work over time.

Ah I think I know what is missing here. I'm not trying to compare arbitrary
elisp objects. I'm explicitly only trying to compare literal emacs lisp source
code and that has been extremely easy since at least emacs 24 (iirc it is
also the same in 23). The only issue I have encountered was a change in
iirc 27 of the default value for print-escape-newlines or something like that.

> We can indeed add something that lets you control how symbols in
> specific are output by prin1, but I don't really see the utility in
> general -- because we have changed, and will change, how other objects
> are output by prin1, and if the objects you're outputting happens to
> contain these, your algo will break again.

These aren't arbitrary elisp objects, they are only objects that appear in
quoted elisp source code. There has been exactly one change in that
since emacs 24 (and possibly before), and it was a change where there
was already a variable to control the behavior.

Message #28 received at 55645 <at> debbugs.gnu.org (full text, mbox):

From: Richard Stallman <rms <at> gnu.org>
To: Tom Gillespie <tgbugs <at> gmail.com>
Cc: larsi <at> gnus.org, 55645 <at> debbugs.gnu.org
Subject: Re: bug#55645: src/print.c;
 print_object changes make it impossible to compare elisp code across
 versions
Date: Sat, 28 May 2022 19:14:15 -0400

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I was thinking the same. I realize that for many objects where
  > there is no guarantee of homoiconicity the prin1 representation
  > definitely changes all the time. However, I think that symbols
  > fall into a more homoiconic category (namely the category
  > of objects that can appear literally in code).

I agree.

We have no absolute rule against changing the print syntax of symbols,
but we see that it this change is likely to cause a real practical
disadvantage for some users.  We should treat that as a significant
problem, and do what it takes to spare users that practical problem.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Message #31 received at 55645 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Tom Gillespie <tgbugs <at> gmail.com>
Cc: Eli Zaretskii <eliz <at> gnu.org>, 55645 <at> debbugs.gnu.org
Subject: Re: bug#55645: src/print.c; print_object changes make it impossible
 to compare elisp code across versions
Date: Sun, 29 May 2022 15:15:38 +0200

Tom Gillespie <tgbugs <at> gmail.com> writes:

> Ah I think I know what is missing here. I'm not trying to compare arbitrary
> elisp objects. I'm explicitly only trying to compare literal emacs lisp source
> code and that has been extremely easy since at least emacs 24 (iirc it is
> also the same in 23). The only issue I have encountered was a change in
> iirc 27 of the default value for print-escape-newlines or something like that.

I'm just not convinced that there's a compelling use case here that
would support the added maintenance burden of adding Yet Another
Printing Variable here -- you're using prin1 to implement something that
prin1 was never meant to support.  If you want to see whether some code
is "the same", then I think you really have to read the forms in and
compare the forms.

But perhaps Eli has an opinion; added to the CCs.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #34 received at 55645 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: tgbugs <at> gmail.com, 55645 <at> debbugs.gnu.org
Subject: Re: bug#55645: src/print.c; print_object changes make it impossible
 to compare elisp code across versions
Date: Sun, 29 May 2022 18:14:06 +0300

> From: Lars Ingebrigtsen <larsi <at> gnus.org>
> Cc: 55645 <at> debbugs.gnu.org, Eli Zaretskii <eliz <at> gnu.org>
> Date: Sun, 29 May 2022 15:15:38 +0200
> 
> Tom Gillespie <tgbugs <at> gmail.com> writes:
> 
> > Ah I think I know what is missing here. I'm not trying to compare arbitrary
> > elisp objects. I'm explicitly only trying to compare literal emacs lisp source
> > code and that has been extremely easy since at least emacs 24 (iirc it is
> > also the same in 23). The only issue I have encountered was a change in
> > iirc 27 of the default value for print-escape-newlines or something like that.
> 
> I'm just not convinced that there's a compelling use case here that
> would support the added maintenance burden of adding Yet Another
> Printing Variable here -- you're using prin1 to implement something that
> prin1 was never meant to support.  If you want to see whether some code
> is "the same", then I think you really have to read the forms in and
> compare the forms.
> 
> But perhaps Eli has an opinion; added to the CCs.

I think I'd need an example that demonstrates the issue, to make up my
mind.  Tom, can you describe the issue in more detail, and provide an
example or two?

Thanks.

Message #37 received at 55645 <at> debbugs.gnu.org (full text, mbox):

From: Tom Gillespie <tgbugs <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: Lars Ingebrigtsen <larsi <at> gnus.org>, 55645 <at> debbugs.gnu.org
Subject: Re: bug#55645: src/print.c; print_object changes make it impossible
 to compare elisp code across versions
Date: Sun, 29 May 2022 14:03:11 -0700

Hi Eli,
  I have included an example below, with some additional context. Best!
Tom

I'm using my own use case as an example, but I suspect that there are
other users out there who may have similar use cases where stability
of the printed representation of read code is important.

Some examples from a quick search on github that will or could be
affected by this (beyond my own). I only searched for cases matching
my own involving secure-hash and prin1-to-string, but there are surely
other cases that would be affected that I have not imagined.
https://github.com/skeeto/elfeed/blob/162d7d545ed41c27967d108c04aa31f5a61c8e16/web/elfeed-web.el#L73-L75
https://github.com/mrmekon/snitch-el/blob/3b3e7f1bf612c4624764d1ec4b1a96e4d2850b05/snitch-timer.el#L164-L181
https://github.com/radian-software/straight.el/blob/af5437f2afd00936c883124d6d3098721c2d306c/straight.el#L5627

https://github.com/search?q=prin1-to-string+secure-hash&type=Code
You can filter out some of the noise by adding the follow to the search
-elfeed -orgstrap -quelpa -litable -xah-elisp-mode -subr.el

The use case that I have is to store the checksum of a code block to
make sure that it has not changed. The checksum needs to be invariant
to changes in formatting e.g. whitespace and needs to be backward and
forward compatible across Emacs versions. In order to compute the
checksum I need a serialized representation of the code. Note when I
say "compare" two pieces of elisp code, one of them may no longer be
available to be read, because only a checksum was retained, so direct
comparison of the two structures in memory is not possible and defeats
the point of having something that is simple to audit and store.

This is discussed in more detail in the orgstrap readme.
https://github.com/tgbugs/orgstrap/blob/master/README.org#normalization-functions
https://github.com/tgbugs/orgstrap/blob/master/README.org#specification

From Emacs 18 through 28 prin1-to-string and the existing print
variables have been able to provide the needed stability.

There is currently no way to compensate for the change introduced in
637dde4aba921435f78d0de769ad74c4f3230aa6 short of reimplementing the
old behavior of prin1-to-string from scratch, which would ultimately
increase the maintenance load across the whole community.

The example below works on emacs-18 (had to remove the number 1.0
example because emacs 18 does not have support for reading floats).

The output of this code is unchanged from emacs-18 through emacs-28,
however it is now different in emacs-29.
Emacs 18-28:
"(progn (+ 1 2) (a b\\.c d) (defun hello nil world)) (progn (some
elisp code I want to normalize\\. That has strings \"1.0\" and symbols
a\\.b))"
Emacs 29:
"(progn (+ 1 2) (a b.c d) (defun hello nil world)) (progn (some elisp
code I want to normalize. That has strings \"1.0\" and symbols a.b))"

#+begin_src elisp :tangle /tmp/example.el
(defun normalize-elisp-code (body)
  (let (print-quoted print-length print-level) ; proposed variable
would be added here
    (prin1-to-string (read (concat "(progn\n" body "\n)")))))

(defvar example-body-1 "(+ 1 2) (a b\\.c d) (defun hello () world)")
(defvar example-body-2 (prin1-to-string '(some elisp code I want to
normalize\. That has strings "1.0" and symbols a\.b)))
(message "%s %s"
         (normalize-elisp-code example-body-1)
         (normalize-elisp-code example-body-2))
#+end_src

The additional step in my use case is the checksum, which cannot be
read back in, and changes from 28 -> 29 due to the differences in the
output of prin1-to-string seen above.

#+begin_src elisp
(defun checksum-elisp-code (body)
  (secure-hash 'sha256 (normalize-elisp-code body)))

(message "%s %s"
         (checksum-elisp-code example-body-1)
         (checksum-elisp-code example-body-2))
#+end_src

Message #40 received at 55645 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Tom Gillespie <tgbugs <at> gmail.com>
Cc: larsi <at> gnus.org, 55645 <at> debbugs.gnu.org
Subject: Re: bug#55645: src/print.c; print_object changes make it impossible
 to compare elisp code across versions
Date: Sat, 04 Jun 2022 12:15:26 +0300

> From: Tom Gillespie <tgbugs <at> gmail.com>
> Date: Sun, 29 May 2022 14:03:11 -0700
> Cc: Lars Ingebrigtsen <larsi <at> gnus.org>, 55645 <at> debbugs.gnu.org
> 
> Hi Eli,
>   I have included an example below, with some additional context. Best!
> Tom

OK, thanks (and sorry for the delay in responding).  I think I
understand the issue now.

You say in your OP that there's no way to work around the issue, but I
wonder if this is really so.  It seems to me that by adding a
backslash before '.' and '?', before computing the hash, would be one
workaround?

> The example below works on emacs-18 (had to remove the number 1.0
> example because emacs 18 does not have support for reading floats).
> 
> The output of this code is unchanged from emacs-18 through emacs-28,
> however it is now different in emacs-29.
> Emacs 18-28:
> "(progn (+ 1 2) (a b\\.c d) (defun hello nil world)) (progn (some
> elisp code I want to normalize\\. That has strings \"1.0\" and symbols
> a\\.b))"
> Emacs 29:
> "(progn (+ 1 2) (a b.c d) (defun hello nil world)) (progn (some elisp
> code I want to normalize. That has strings \"1.0\" and symbols a.b))"
> 
> #+begin_src elisp :tangle /tmp/example.el
> (defun normalize-elisp-code (body)
>   (let (print-quoted print-length print-level) ; proposed variable
> would be added here
>     (prin1-to-string (read (concat "(progn\n" body "\n)")))))
> 
> (defvar example-body-1 "(+ 1 2) (a b\\.c d) (defun hello () world)")
> (defvar example-body-2 (prin1-to-string '(some elisp code I want to
> normalize\. That has strings "1.0" and symbols a\.b)))
> (message "%s %s"
>          (normalize-elisp-code example-body-1)
>          (normalize-elisp-code example-body-2))
> #+end_src
> 
> The additional step in my use case is the checksum, which cannot be
> read back in, and changes from 28 -> 29 due to the differences in the
> output of prin1-to-string seen above.
> 
> #+begin_src elisp
> (defun checksum-elisp-code (body)
>   (secure-hash 'sha256 (normalize-elisp-code body)))
> 
> (message "%s %s"
>          (checksum-elisp-code example-body-1)
>          (checksum-elisp-code example-body-2))
> #+end_src

Frankly, I'm amazed that someone could have such faith into
immutability of Emacs Lisp.  Why would you assume that the above
produces the same literal string across versions, especially given
everything that happens lately in Emacs, including changes in the
byte-compiler, the introduction of native-compilation, etc.  If
someone would show me a design that is based on this, I'd tell them
they place their bet on a dubious horse, and suggest to find a better
design.  To me, this is taking some Emacs feature that just happened
to be stable, and assuming it will forever be stable.  There's no
reason and no real basis for such assumptions.

Thus, I agree with Lars that it is strange to hear that prin1 is used
as something that's supposed to produce a canonical representation of
Lisp code; it's definitely isn't its purpose.

Anyway, one way forward is to add a new API specifically for that
purpose, and then guarantee that the output of that new API will be
stable.  This will also take care of the issue with the design that
relies on prin1.

Another way forward is to revert the change.  And here I'm asking Lars
what are the reasons for the change, except some aesthetics-related
considerations, whereby we basically didn't see why the escapes are
needed.  If the only problems are that we didn't see a good reason for
keeping the escapes, perhaps now we do have a good reason?

Thanks.

Message #43 received at 55645 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: Tom Gillespie <tgbugs <at> gmail.com>, 55645 <at> debbugs.gnu.org
Subject: Re: bug#55645: src/print.c; print_object changes make it impossible
 to compare elisp code across versions
Date: Sat, 04 Jun 2022 13:35:05 +0200

Eli Zaretskii <eliz <at> gnu.org> writes:

> Another way forward is to revert the change.  And here I'm asking Lars
> what are the reasons for the change, except some aesthetics-related
> considerations, whereby we basically didn't see why the escapes are
> needed.  If the only problems are that we didn't see a good reason for
> keeping the escapes, perhaps now we do have a good reason?

It was mostly an aesthetic change, but sometimes aesthetics are
important.  People do use dot characters in symbols, but it's confusing
to see Emacs display these in a weird way -- so they may be discouraged
from doing so when that's a natural thing to do.

So I'm not very enthusiastic about reverting the change, especially (as
you say):

> If someone would show me a design that is based on this, I'd tell them
> they place their bet on a dubious horse, and suggest to find a better
> design.

The reason for reverting it would be to support a use case that we don't
support anyway.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #46 received at 55645 <at> debbugs.gnu.org (full text, mbox):

From: Po Lu <luangruo <at> yahoo.com>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: Tom Gillespie <tgbugs <at> gmail.com>, Eli Zaretskii <eliz <at> gnu.org>,
 55645 <at> debbugs.gnu.org
Subject: Re: bug#55645: src/print.c; print_object changes make it impossible
 to compare elisp code across versions
Date: Sat, 04 Jun 2022 19:53:08 +0800

Lars Ingebrigtsen <larsi <at> gnus.org> writes:

> It was mostly an aesthetic change, but sometimes aesthetics are
> important.  People do use dot characters in symbols, but it's confusing
> to see Emacs display these in a weird way -- so they may be discouraged
> from doing so when that's a natural thing to do.

FWIW, I don't see how it's natural to use dots in symbols.  It certainly
feels weird to me.

Symbol names should ideally consist of the punctuation characters
"-+=*/", letters and digits.

Message #49 received at 55645 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Po Lu <luangruo <at> yahoo.com>
Cc: Tom Gillespie <tgbugs <at> gmail.com>, Eli Zaretskii <eliz <at> gnu.org>,
 55645 <at> debbugs.gnu.org
Subject: Re: bug#55645: src/print.c; print_object changes make it impossible
 to compare elisp code across versions
Date: Sat, 04 Jun 2022 14:09:06 +0200

Po Lu <luangruo <at> yahoo.com> writes:

> FWIW, I don't see how it's natural to use dots in symbols.  It certainly
> feels weird to me.

It's mostly natural when dealing with data that arrives from other
programming languages that we then have turned into symbols.  So you
have symbols like `vnd.ms-excel' etc.  (We don't use dots in
function/variable names.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #52 received at 55645 <at> debbugs.gnu.org (full text, mbox):

From: Po Lu <luangruo <at> yahoo.com>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: Tom Gillespie <tgbugs <at> gmail.com>, Eli Zaretskii <eliz <at> gnu.org>,
 55645 <at> debbugs.gnu.org
Subject: Re: bug#55645: src/print.c; print_object changes make it impossible
 to compare elisp code across versions
Date: Sat, 04 Jun 2022 20:18:03 +0800

Lars Ingebrigtsen <larsi <at> gnus.org> writes:

> It's mostly natural when dealing with data that arrives from other
> programming languages that we then have turned into symbols.  So you
> have symbols like `vnd.ms-excel' etc.  (We don't use dots in
> function/variable names.)

But that's not something which happens very often, is it?

IOW, this looks like a relatively unimportant asthetic issue that
doesn't justify strong opinions in either direction.  Even though
there's no guarantee that the printed representation of symbols will
remain stable, we might as well do Tom the courtesy of reverting that
particular change, since it caused him problems with code problem-free
since Emacs 18.

Message #55 received at 55645 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Po Lu <luangruo <at> yahoo.com>
Cc: Tom Gillespie <tgbugs <at> gmail.com>, Eli Zaretskii <eliz <at> gnu.org>,
 55645 <at> debbugs.gnu.org
Subject: Re: bug#55645: src/print.c; print_object changes make it impossible
 to compare elisp code across versions
Date: Sat, 04 Jun 2022 15:14:15 +0200

Po Lu <luangruo <at> yahoo.com> writes:

>> It's mostly natural when dealing with data that arrives from other
>> programming languages that we then have turned into symbols.  So you
>> have symbols like `vnd.ms-excel' etc.  (We don't use dots in
>> function/variable names.)
>
> But that's not something which happens very often, is it?

I think it might happen more often if the printed representation wasn't
as ugly.

But we do have a bunch of symbols in-tree that uses dots, too:

(mapatoms (lambda (k) (when (string-match "[.]" (symbol-name k))
                          (insert (format "%s\n" k)))))
Lint Directory...
iso10646.indian-1
gb2312.1980-0
cns11643.1992-7
cns11643.1992-6
cns11643.1992-5
cns11643.1992-4
cns11643.1992-3
cns11643.1992-2
cns11643.1992-1
Reset Counters for Function...
Instrument Package...
Lint File...

(etc.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #58 received at 55645 <at> debbugs.gnu.org (full text, mbox):

From: Po Lu <luangruo <at> yahoo.com>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: Tom Gillespie <tgbugs <at> gmail.com>, Eli Zaretskii <eliz <at> gnu.org>,
 55645 <at> debbugs.gnu.org
Subject: Re: bug#55645: src/print.c; print_object changes make it impossible
 to compare elisp code across versions
Date: Sat, 04 Jun 2022 21:30:35 +0800

Lars Ingebrigtsen <larsi <at> gnus.org> writes:

> I think it might happen more often if the printed representation wasn't
> as ugly.

I find it hard to believe: why would changing the print syntax of
symbols with dots make it more likely for people to use data types that
have type names with those characters?

And if you mean in general, then I don't see the point of using dots in
keywords and function or variable names.

> But we do have a bunch of symbols in-tree that uses dots, too:
>
> (mapatoms (lambda (k) (when (string-match "[.]" (symbol-name k))
>                           (insert (format "%s\n" k)))))
> Lint Directory...
> iso10646.indian-1
> gb2312.1980-0
> cns11643.1992-7
> cns11643.1992-6
> cns11643.1992-5
> cns11643.1992-4
> cns11643.1992-3
> cns11643.1992-2
> cns11643.1992-1
> Reset Counters for Function...
> Instrument Package...
> Lint File...

Most of those are interned from strings (see fc_charset_table in
ftfont.c), and not likely to either be printed or to be written by
users.

Thanks.

Message #61 received at 55645 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Po Lu <luangruo <at> yahoo.com>
Cc: Tom Gillespie <tgbugs <at> gmail.com>, Eli Zaretskii <eliz <at> gnu.org>,
 55645 <at> debbugs.gnu.org
Subject: Re: bug#55645: src/print.c; print_object changes make it impossible
 to compare elisp code across versions
Date: Sat, 04 Jun 2022 15:35:53 +0200

Po Lu <luangruo <at> yahoo.com> writes:

> I find it hard to believe: why would changing the print syntax of
> symbols with dots make it more likely for people to use data types that
> have type names with those characters?

I've certainly been puzzled over the printed representation of these
symbols, and I've wondered whether I should use a transliteration scheme
(which isn't necessary), so I think it's likely that others have been,
too.

> Most of those are interned from strings (see fc_charset_table in
> ftfont.c), and not likely to either be printed or to be written by
> users.

Users will inevitably see symbols like this at some point or another.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #64 received at 55645 <at> debbugs.gnu.org (full text, mbox):

From: Po Lu <luangruo <at> yahoo.com>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: Tom Gillespie <tgbugs <at> gmail.com>, Eli Zaretskii <eliz <at> gnu.org>,
 55645 <at> debbugs.gnu.org
Subject: Re: bug#55645: src/print.c; print_object changes make it impossible
 to compare elisp code across versions
Date: Sat, 04 Jun 2022 21:47:55 +0800

Lars Ingebrigtsen <larsi <at> gnus.org> writes:

> I've certainly been puzzled over the printed representation of these
> symbols, and I've wondered whether I should use a transliteration scheme
> (which isn't necessary), so I think it's likely that others have been,
> too.

Fulfilling a niche is not really a good enough reason to make such a
change, especially once it causes actual problems for someone.

Most programmers will not see a symbol with a dot inside every day, or
even every month (the ones you sent are the first ones I've seen this
year.)  Programmers probably come across reasons to use them even less,
and the amount of those reasons is reduced even further if you rule out
unambiguous interfaces such as `intern'.

> Users will inevitably see symbols like this at some point or another.

Chances are (if it actually happens) those symbols will not be printed,
but displayed to the user as the product of something similar to:

  (message "%s" 'symbol.here)

Which does not escape the dot.

That aside, why is it escaping a rarely encountered character inside a
symbol a "problem" important enough to fix, especially when the fix
breaks someone's code badly enough for him to complain about it?

Message #67 received at 55645 <at> debbugs.gnu.org (full text, mbox):

From: Tom Gillespie <tgbugs <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: Lars Ingebrigtsen <larsi <at> gnus.org>, 55645 <at> debbugs.gnu.org
Subject: Re: bug#55645: src/print.c; print_object changes make it impossible
 to compare elisp code across versions
Date: Mon, 6 Jun 2022 16:30:22 -0700

Hi Eli,
   Replies in line. Best!
Tom

> You say in your OP that there's no way to work around the issue, but I
> wonder if this is really so.  It seems to me that by adding a
> backslash before '.' and '?', before computing the hash, would be one
> workaround?

Unfortunately I don't think there is an easy way to do such a thing
from elisp. The primary issue with such a workaround is that while
it is safe to do that inside strings and symbols, you have to know
the type of the object that is being printed, otherwise floats and
characters will be inadvertently converted to symbols.

As far as I can tell the only place to make such a change is inside
the c implementation of prin1 or some other c function, thus my request
for a dynamic variable that would preserve the old behavior by allowing
me to reach into the c runtime.

> Frankly, I'm amazed that someone could have such faith into
> immutability of Emacs Lisp.  Why would you assume that the above
> produces the same literal string across versions, especially given
> everything that happens lately in Emacs, including changes in the
> byte-compiler, the introduction of native-compilation, etc.

I wouldn't call it faith so much as testing, and perhaps an assumption
that there wasn't any reason to change something so fundamental
that affected so few people and had been stable for so long.

Also, it does not seem such an unfounded assumption because for code
the prin1 representation (or at least the pp-buffer representation) must
produce valid elisp that can be read, and the reader is extremely stable.
In particular the representation also ideally needs to be read by older
versions of emacs because some libraries would like to be able to
support multiple versions of emacs with a single.el file.

As I note below the particular relaxation in the behavior of the reader
was made long ago, the change in the prin1 behavior only affects very
old versions of emacs, though emacs-18 is installable and usable on
gentoo, I have no idea whether anyone actually shares code between
emacs-18 and emacs-29, so that is likely a moot point.

> If someone would show me a design that is based on this, I'd tell them
> they place their bet on a dubious horse, and suggest to find a better
> design.  To me, this is taking some Emacs feature that just happened
> to be stable, and assuming it will forever be stable.  There's no
> reason and no real basis for such assumptions.

As mentioned above, there is a strong basis for such assumptions,
which is that prin1 or at least pp-buffer must produce valid elisp code
that can be read.

Thus the stability of prin1 is intrinsically tied to what the reader
will accept. So the change that we are discussing here is due to
the fact that there was a change in the reader to relax what it would
accept with regard to '.' and '?' in symbols.

>
> Thus, I agree with Lars that it is strange to hear that prin1 is used
> as something that's supposed to produce a canonical representation of
> Lisp code; it's definitely isn't its purpose.

Am I wrong in my assumption about pp-buffer and read? I mean, yes
in principle prin1 could start escaping every single character in every
symbol since it will ultimately read in the same way, but that seems
somewhat unlikely.

> Anyway, one way forward is to add a new API specifically for that
> purpose, and then guarantee that the output of that new API will be
> stable.  This will also take care of the issue with the design that
> relies on prin1.

This seems like the "right" solution long term. It allows us to skip
the aesthetics discussion going on at the moment, and it provides
an explicitly stable representation for readable forms, allowing prin1
to be free of any such constraints (complaints?) in the future.
It would be possible to use fboundp to check whether the new stable
prin1 was present and use that going forward, if it is not found then
prin1 itself should be fine through 28. This seems preferable to having
an implicit subset of readable objects that are mostly stable in prin1
but can sometimes change to match current reader behavior.

Message #70 received at 55645 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Tom Gillespie <tgbugs <at> gmail.com>
Cc: larsi <at> gnus.org, 55645 <at> debbugs.gnu.org
Subject: Re: bug#55645: src/print.c; print_object changes make it impossible
 to compare elisp code across versions
Date: Tue, 07 Jun 2022 19:35:05 +0300

> From: Tom Gillespie <tgbugs <at> gmail.com>
> Date: Mon, 6 Jun 2022 16:30:22 -0700
> Cc: Lars Ingebrigtsen <larsi <at> gnus.org>, 55645 <at> debbugs.gnu.org
> 
> > Thus, I agree with Lars that it is strange to hear that prin1 is used
> > as something that's supposed to produce a canonical representation of
> > Lisp code; it's definitely isn't its purpose.
> 
> Am I wrong in my assumption about pp-buffer and read?

Of course you are wrong, since we changed prin1 without affecting
read.  We could also change both prin1 and read such that what the
former produces the latter can consume, as long as the "usual" Lisp
code can still be read without any problem.

> I mean, yes in principle prin1 could start escaping every single
> character in every symbol since it will ultimately read in the same
> way, but that seems somewhat unlikely.

Unlikely for every character, but we could decide to do that for a few
special ones, for example.

Message #73 received at 55645 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: Tom Gillespie <tgbugs <at> gmail.com>, 55645 <at> debbugs.gnu.org
Subject: Re: bug#55645: src/print.c; print_object changes make it impossible
 to compare elisp code across versions
Date: Mon, 11 Jul 2022 13:22:45 +0200

Eli Zaretskii <eliz <at> gnu.org> writes:

> Of course you are wrong, since we changed prin1 without affecting
> read.  We could also change both prin1 and read such that what the
> former produces the latter can consume, as long as the "usual" Lisp
> code can still be read without any problem.

This was a month ago, and I think the conclusion here is that we don't
want to add an option to tweak this printing behaviour (or guarantee
that we won't change printing in general), so I'm closing this bug
report.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #82 received at 55645 <at> debbugs.gnu.org (full text, mbox):

From: Tom Gillespie <tgbugs <at> gmail.com>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: Eli Zaretskii <eliz <at> gnu.org>, 55645 <at> debbugs.gnu.org
Subject: Re: bug#55645: src/print.c; print_object changes make it impossible
 to compare elisp code across versions
Date: Mon, 11 Jul 2022 11:43:23 -0700

> This was a month ago, and I think the conclusion here is that we don't
> want to add an option to tweak this printing behaviour (or guarantee
> that we won't change printing in general), so I'm closing this bug
> report.

I still need an actual solution here. Will a new stable print function be
implemented in 29 that matches the old behavior?

Message #85 received at 55645 <at> debbugs.gnu.org (full text, mbox):

From: Richard Stallman <rms <at> gnu.org>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: tgbugs <at> gmail.com, eliz <at> gnu.org, 55645 <at> debbugs.gnu.org
Subject: Re: bug#55645: src/print.c;
 print_object changes make it impossible to compare elisp code across
 versions
Date: Mon, 11 Jul 2022 23:04:29 -0400

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > This was a month ago, and I think the conclusion here is that we don't
  > want to add an option to tweak this printing behaviour (or guarantee
  > that we won't change printing in general), so I'm closing this bug
  > report.

Did we implement something else to make it easier to compare
structures including hash tables, for equal contents?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Message #88 received at 55645 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Tom Gillespie <tgbugs <at> gmail.com>
Cc: Eli Zaretskii <eliz <at> gnu.org>, 55645 <at> debbugs.gnu.org
Subject: Re: bug#55645: src/print.c; print_object changes make it impossible
 to compare elisp code across versions
Date: Tue, 12 Jul 2022 14:44:51 +0200

Tom Gillespie <tgbugs <at> gmail.com> writes:

> I still need an actual solution here. Will a new stable print function be
> implemented in 29 that matches the old behavior?

Nope.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #91 received at 55645 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Richard Stallman <rms <at> gnu.org>
Cc: tgbugs <at> gmail.com, eliz <at> gnu.org, 55645 <at> debbugs.gnu.org
Subject: Re: bug#55645: src/print.c; print_object changes make it impossible
 to compare elisp code across versions
Date: Tue, 12 Jul 2022 14:45:11 +0200

Richard Stallman <rms <at> gnu.org> writes:

> Did we implement something else to make it easier to compare
> structures including hash tables, for equal contents?

Nope.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #94 received at 55645 <at> debbugs.gnu.org (full text, mbox):

From: Richard Stallman <rms <at> gnu.org>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: tgbugs <at> gmail.com, eliz <at> gnu.org, 55645 <at> debbugs.gnu.org
Subject: Re: bug#55645: src/print.c; print_object changes make it impossible
 to compare elisp code across versions
Date: Tue, 12 Jul 2022 23:16:15 -0400

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Did we implement something else to make it easier to compare
  > > structures including hash tables, for equal contents?

  > Nope.

I hope someone will do that.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Message #97 received at 55645 <at> debbugs.gnu.org (full text, mbox):

From: Tom Gillespie <tgbugs <at> gmail.com>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: Eli Zaretskii <eliz <at> gnu.org>, 55645 <at> debbugs.gnu.org
Subject: Re: bug#55645: src/print.c; print_object changes make it impossible
 to compare elisp code across versions
Date: Wed, 13 Jul 2022 16:26:07 -0700

> > I still need an actual solution here. Will a new stable print function be
> > implemented in 29 that matches the old behavior?
>
> Nope.

If I were to send a patch with such functionality and stability
guarantees would it be considered?

Message #100 received at 55645 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Tom Gillespie <tgbugs <at> gmail.com>
Cc: Eli Zaretskii <eliz <at> gnu.org>, 55645 <at> debbugs.gnu.org
Subject: Re: bug#55645: src/print.c; print_object changes make it impossible
 to compare elisp code across versions
Date: Thu, 14 Jul 2022 19:22:18 +0200

Tom Gillespie <tgbugs <at> gmail.com> writes:

> If I were to send a patch with such functionality and stability
> guarantees would it be considered?

Like I've said before, we (the Emacs maintainers) would not guarantee
that Emacs' `prin1' will remain the same.  If you want to create a
package that does printing in a different way, you're free to do so, of
course -- we have the `cl-prin1' framework in Emacs already, and you can
use that to print things any way you want o.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #103 received at 55645 <at> debbugs.gnu.org (full text, mbox):

From: Tom Gillespie <tgbugs <at> gmail.com>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: Eli Zaretskii <eliz <at> gnu.org>, 55645 <at> debbugs.gnu.org
Subject: Re: bug#55645: src/print.c; print_object changes make it impossible
 to compare elisp code across versions
Date: Thu, 14 Jul 2022 14:36:08 -0700

> Like I've said before, we (the Emacs maintainers) would not guarantee
> that Emacs' `prin1' will remain the same.  If you want to create a
> package that does printing in a different way, you're free to do so, of
> course -- we have the `cl-prin1' framework in Emacs already, and you can
> use that to print things any way you want

Unfortunately none of these work for my use case because cl-prin1
is not available before emacs 26 and it differs too much from prin1
to be useful without requiring additional code. I have to fit all the code
under 3000 chars because it is in an eval local variable and the changes
for cl-prin1 are too extensive.

I'm not 100% sure how to use cl-prin1 but as far as I can tell defining
new cl-print-object methods (e.g. for symbol) is highly invasive and
makes changes to the entirety of the user's runtime.

Would a patch that leveraged cl-prin1 to create a function that matches
the old behavior of prin1 be more viable?

Without that my only solution is to restrict symbol names so that they
do not include . and ? by raising an error if they are detected, and hoping
that prin1 doesn't undergo another change in the future in the readable
subset of objects.

Message #106 received at 55645 <at> debbugs.gnu.org (full text, mbox):

From: Tom Gillespie <tgbugs <at> gmail.com>
Cc: 55645 <at> debbugs.gnu.org
Subject: Re: bug#55645: src/print.c; print_object changes make it impossible
 to compare elisp code across versions
Date: Mon, 18 Jul 2022 11:13:10 -0700

Ok. I went with the band non-portable symbols solution and it works.
Here's to hoping nothing changes for another 30 years! Will close.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.