GNU bug report logs - #42545
json-encode-key can't handle boolean values

Previous Next

Package: emacs;

Reported by: Marcelo Muñoz <ma.munoz.araya <at> gmail.com>

Date: Sun, 26 Jul 2020 08:16:01 UTC

Severity: normal

Tags: fixed, patch

Merged with 45032, 46174, 46811

Found in versions 24.5, 26.3, 27.1, 28.0.50

Fixed in version 28.1

Done: "Basil L. Contovounesios" <contovob <at> tcd.ie>

Bug is archived. No further changes may be made.

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 42545 in the body.
You can then email your comments to 42545 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to bug-gnu-emacs <at> gnu.org:
bug#42545; Package emacs. (Sun, 26 Jul 2020 08:16:02 GMT) Full text and rfc822 format available.

Acknowledgement sent to Marcelo Muñoz <ma.munoz.araya <at> gmail.com>:
New bug report received and forwarded. Copy sent to bug-gnu-emacs <at> gnu.org. (Sun, 26 Jul 2020 08:16:02 GMT) Full text and rfc822 format available.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Marcelo Muñoz <ma.munoz.araya <at> gmail.com>
To: bug-gnu-emacs <at> gnu.org
Subject: 28.0.50; json-pretty-print cant't handle json having "t" as a key
Date: Sun, 26 Jul 2020 02:13:09 -0400
[Message part 1 (text/plain, inline)]
Try to apply json-pretty-print to follow json:

{"t": 1, "key":2}

fail with the message:  json-pretty-print: Bad JSON object key: t

Some research point me the problem is in the function json-encode-key,
its  implementation is:

(defun json-encode-key (object)
  "Return a JSON representation of OBJECT.
If the resulting JSON object isn't a valid JSON object key,
this signals `json-key-format'."
  (let ((encoded (json-encode object)))
    (unless (stringp (json-read-from-string encoded))
      (signal 'json-key-format (list object)))
    encoded))


The follow change seems to  solve the problem, but I haven't had time
understand it in depth:

(defun json-encode-key (object)
   "Return s string representation of a key"
   (json-encode-string (symbol-name object)))
[Message part 2 (text/html, inline)]

Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#42545; Package emacs. (Sun, 26 Jul 2020 10:40:01 GMT) Full text and rfc822 format available.

Message #8 received at 42545 <at> debbugs.gnu.org (full text, mbox):

From: "Basil L. Contovounesios" <contovob <at> tcd.ie>
To: Marcelo Muñoz <ma.munoz.araya <at> gmail.com>
Cc: 42545 <at> debbugs.gnu.org
Subject: Re: bug#42545: 28.0.50; json-pretty-print cant't handle json having
 "t" as a key
Date: Sun, 26 Jul 2020 13:38:59 +0300
retitle 42545 json-encode-key can't handle boolean values
found 42545 24.5
quit

Marcelo Muñoz <ma.munoz.araya <at> gmail.com> writes:

> Try to apply json-pretty-print to follow json:
>
> {"t": 1, "key":2}
>
> fail with the message:  json-pretty-print: Bad JSON object key: t

Here are some simpler repros:

(json-encode '((nil . 0)))
(json-encode '((t . 0)))
(json-encode-key nil)
(json-encode-key t)

All of these fail with json-key-format since at least as far back as
Emacs 24.5.

> Some research point me the problem is in the function json-encode-key,
> its  implementation is:
>
> (defun json-encode-key (object)
>   "Return a JSON representation of OBJECT.
> If the resulting JSON object isn't a valid JSON object key,
> this signals `json-key-format'."
>   (let ((encoded (json-encode object)))
>     (unless (stringp (json-read-from-string encoded))
>       (signal 'json-key-format (list object)))
>     encoded))
>
> The follow change seems to  solve the problem, but I haven't had time
> understand it in depth:
>
> (defun json-encode-key (object)
>    "Return s string representation of a key"
>    (json-encode-string (symbol-name object)))

This can't be TRT, because OBJECT is not always a symbol.

See also https://debbugs.gnu.org/24252#26 for some precedent in
rewriting json-encode-key without relying on json-encode.

I'm AFK until start of August, but I'll try to have a better look at
this when I get the chance if no-one beats me to it.

Thanks,

-- 
Basil




Changed bug title to 'json-encode-key can't handle boolean values' from '28.0.50; json-pretty-print cant't handle json having "t" as a key' Request was from "Basil L. Contovounesios" <contovob <at> tcd.ie> to control <at> debbugs.gnu.org. (Sun, 26 Jul 2020 10:40:02 GMT) Full text and rfc822 format available.

bug Marked as found in versions 24.5. Request was from "Basil L. Contovounesios" <contovob <at> tcd.ie> to control <at> debbugs.gnu.org. (Sun, 26 Jul 2020 10:40:02 GMT) Full text and rfc822 format available.

Forcibly Merged 42545 45032. Request was from "Basil L. Contovounesios" <contovob <at> tcd.ie> to control <at> debbugs.gnu.org. (Sat, 05 Dec 2020 16:18:02 GMT) Full text and rfc822 format available.

Forcibly Merged 42545 45032 46174. Request was from "Basil L. Contovounesios" <contovob <at> tcd.ie> to control <at> debbugs.gnu.org. (Fri, 29 Jan 2021 16:46:02 GMT) Full text and rfc822 format available.

Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#42545; Package emacs. (Thu, 11 Feb 2021 14:21:01 GMT) Full text and rfc822 format available.

Message #19 received at 42545 <at> debbugs.gnu.org (full text, mbox):

From: "Basil L. Contovounesios" <contovob <at> tcd.ie>
To: Marcelo Muñoz <ma.munoz.araya <at> gmail.com>
Cc: 42545 <at> debbugs.gnu.org
Subject: Re: bug#42545: 28.0.50; json-pretty-print cant't handle json having
 "t" as a key
Date: Thu, 11 Feb 2021 14:20:11 +0000
[Message part 1 (text/plain, inline)]
"Basil L. Contovounesios" <contovob <at> tcd.ie> writes:

> Marcelo Muñoz <ma.munoz.araya <at> gmail.com> writes:
>
>> Try to apply json-pretty-print to follow json:
>>
>> {"t": 1, "key":2}
>>
>> fail with the message:  json-pretty-print: Bad JSON object key: t
>
> Here are some simpler repros:
>
> (json-encode '((nil . 0)))
> (json-encode '((t . 0)))
> (json-encode-key nil)
> (json-encode-key t)
>
> All of these fail with json-key-format since at least as far back as
> Emacs 24.5.

[...]

> See also https://debbugs.gnu.org/24252#26 for some precedent in
> rewriting json-encode-key without relying on json-encode.
>
> I'm AFK until start of August, but I'll try to have a better look at
> this when I get the chance if no-one beats me to it.

Sorry, finally got around to this.  The attached patch should fix this
issue while also speeding up encoding in a backward compatible way.
Using the same benchmark as https://bugs.gnu.org/40693#89 I get the
following:

  encode canada.json
  old
  (1.409496598 96 0.7710352720000002)
  (1.406660968 96 0.7707586369999997)
  (1.406515696 96 0.7698804519999998)
  (1.4098724120000001 96 0.7712946)
  new
  (1.452364951 96 0.7682001569999999)
  (1.451790854 96 0.7712237389999999)
  (1.452158289 96 0.7710199420000006)
  (1.4520665160000001 96 0.7707500029999999)

This shows that the two extra cases of funcall+cond in json-encode
slightly slow down encoding of large numbers of numbers, but I doubt
it's significant enough.  If it is, we can probably just tweak the
dispatch order in json-encode.

  encode citm_catalog.json
  old
  (2.7812737399999996 272 2.1942181940000003)
  (2.77954628 272 2.1904517840000004)
  (2.779567506 272 2.1901039010000005)
  (2.778913438 272 2.189370834)
  new
  (0.7056556740000001 68 0.55314481)
  (0.704577043 68 0.5515927839999994)
  (0.702683784 68 0.5491281600000004)
  (0.703850623 68 0.5503691039999996)

  encode twitter.json
  old
  (1.427292653 148 1.1098771399999983)
  (1.428440774 148 1.109535473000001)
  (1.4265714 148 1.1097104909999977)
  (1.426152699 148 1.110347719)
  new
  (0.365952034 40 0.29652698499999985)
  (0.366947621 40 0.29772050399999905)
  (0.36731820000000004 40 0.29776995099999937)
  (0.366228327 40 0.29696426200000126)

These show that examples with more realistic objects are encoded far
faster.

Decoding performance is not affected.

This change fixes some errors in json.el and brings it a tiny bit closer
to json.c in handling confusable keys, but doesn't go all the way, for
backward compatibility.  For example, json-encode and json-serialize
still disagree on ((:t . 1)), (("t" . 1)), (t 1), ("t" 1), hash tables
with non-string keys, etc.

WDYT?

-- 
Basil

[0001-Fix-json.el-encoding-of-confusable-object-keys.patch (text/x-diff, attachment)]

Added tag(s) patch. Request was from "Basil L. Contovounesios" <contovob <at> tcd.ie> to control <at> debbugs.gnu.org. (Thu, 11 Feb 2021 14:23:02 GMT) Full text and rfc822 format available.

Added tag(s) fixed. Request was from "Basil L. Contovounesios" <contovob <at> tcd.ie> to control <at> debbugs.gnu.org. (Sun, 21 Feb 2021 13:03:02 GMT) Full text and rfc822 format available.

bug marked as fixed in version 28.1, send any further explanations to 42545 <at> debbugs.gnu.org and Marcelo Muñoz <ma.munoz.araya <at> gmail.com> Request was from "Basil L. Contovounesios" <contovob <at> tcd.ie> to control <at> debbugs.gnu.org. (Sun, 21 Feb 2021 13:03:02 GMT) Full text and rfc822 format available.

Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#42545; Package emacs. (Sun, 21 Feb 2021 13:03:03 GMT) Full text and rfc822 format available.

Message #28 received at 42545-done <at> debbugs.gnu.org (full text, mbox):

From: "Basil L. Contovounesios" <contovob <at> tcd.ie>
To: Marcelo Muñoz <ma.munoz.araya <at> gmail.com>
Cc: 42545-done <at> debbugs.gnu.org
Subject: Re: bug#42545: 28.0.50; json-pretty-print cant't handle json having
 "t" as a key
Date: Sun, 21 Feb 2021 13:01:57 +0000
tags 42545 fixed
close 42545 28.1
quit

"Basil L. Contovounesios" <contovob <at> tcd.ie> writes:

> The attached patch should fix this issue while also speeding up
> encoding in a backward compatible way.

I've now pushed it to master, and am therefore closing this report.

Fix json.el encoding of confusable object keys
908f251e19 2021-02-21 12:53:45 +0000
https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=908f251e19dc64c75000f87bc6db4e9a8852d1ad

Thanks,

-- 
Basil




Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#42545; Package emacs. (Tue, 23 Feb 2021 01:24:02 GMT) Full text and rfc822 format available.

Message #31 received at 42545-done <at> debbugs.gnu.org (full text, mbox):

From: Dmitry Gutov <dgutov <at> yandex.ru>
To: "Basil L. Contovounesios" <contovob <at> tcd.ie>,
 Marcelo Muñoz <ma.munoz.araya <at> gmail.com>
Cc: 42545-done <at> debbugs.gnu.org
Subject: Re: bug#42545: 28.0.50; json-pretty-print cant't handle json having
 "t" as a key
Date: Tue, 23 Feb 2021 03:23:19 +0200
Hi Basil,

On 21.02.2021 15:01, Basil L. Contovounesios wrote:
>> The attached patch should fix this issue while also speeding up
>> encoding in a backward compatible way.
> I've now pushed it to master, and am therefore closing this report.

Looking good, thank you!




Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#42545; Package emacs. (Tue, 23 Feb 2021 01:42:01 GMT) Full text and rfc822 format available.

Message #34 received at 42545 <at> debbugs.gnu.org (full text, mbox):

From: "Basil L. Contovounesios" <contovob <at> tcd.ie>
To: Dmitry Gutov <dgutov <at> yandex.ru>
Cc: 42545 <at> debbugs.gnu.org
Subject: Re: bug#42545: 28.0.50; json-pretty-print cant't handle json having
 "t" as a key
Date: Tue, 23 Feb 2021 01:41:41 +0000
Hi Dmitry,

Dmitry Gutov <dgutov <at> yandex.ru> writes:

> On 21.02.2021 15:01, Basil L. Contovounesios wrote:
>>> The attached patch should fix this issue while also speeding up
>>> encoding in a backward compatible way.
>> I've now pushed it to master, and am therefore closing this report.
>
> Looking good, thank you!

Thanks, you're welcome!  But I'm not yet done speeding up json.el
encoding backward compatibly, so stay tuned ;).

-- 
Basil




Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#42545; Package emacs. (Tue, 23 Feb 2021 12:12:01 GMT) Full text and rfc822 format available.

Message #37 received at 42545 <at> debbugs.gnu.org (full text, mbox):

From: Dmitry Gutov <dgutov <at> yandex.ru>
To: "Basil L. Contovounesios" <contovob <at> tcd.ie>
Cc: 42545 <at> debbugs.gnu.org
Subject: Re: bug#42545: 28.0.50; json-pretty-print cant't handle json having
 "t" as a key
Date: Tue, 23 Feb 2021 14:10:58 +0200
On 23.02.2021 03:41, Basil L. Contovounesios wrote:
> Thanks, you're welcome!  But I'm not yet done speeding up json.el
> encoding backward compatibly, so stay tuned;).

I'm more partial to faster decoding myself ;-). but correctness 
improvements are welcome either way!




Forcibly Merged 42545 45032 46174 46811. Request was from "Basil L. Contovounesios" <contovob <at> tcd.ie> to control <at> debbugs.gnu.org. (Sat, 27 Feb 2021 14:08:01 GMT) Full text and rfc822 format available.

bug archived. Request was from Debbugs Internal Request <help-debbugs <at> gnu.org> to internal_control <at> debbugs.gnu.org. (Sun, 28 Mar 2021 11:24:05 GMT) Full text and rfc822 format available.

This bug report was last modified 3 years and 1 day ago.

Previous Next


GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.