GNU bug report logs - #42545
json-encode-key can't handle boolean values

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 42545 in the body.
You can then email your comments to 42545 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Marcelo Muñoz <ma.munoz.araya <at> gmail.com>
To: bug-gnu-emacs <at> gnu.org
Subject: 28.0.50; json-pretty-print cant't handle json having "t" as a key
Date: Sun, 26 Jul 2020 02:13:09 -0400

[Message part 1 (text/plain, inline)]

Try to apply json-pretty-print to follow json:

{"t": 1, "key":2}

fail with the message:  json-pretty-print: Bad JSON object key: t

Some research point me the problem is in the function json-encode-key,
its  implementation is:

(defun json-encode-key (object)
  "Return a JSON representation of OBJECT.
If the resulting JSON object isn't a valid JSON object key,
this signals `json-key-format'."
  (let ((encoded (json-encode object)))
    (unless (stringp (json-read-from-string encoded))
      (signal 'json-key-format (list object)))
    encoded))


The follow change seems to  solve the problem, but I haven't had time
understand it in depth:

(defun json-encode-key (object)
   "Return s string representation of a key"
   (json-encode-string (symbol-name object)))

[Message part 2 (text/html, inline)]

Message #8 received at 42545 <at> debbugs.gnu.org (full text, mbox):

From: "Basil L. Contovounesios" <contovob <at> tcd.ie>
To: Marcelo Muñoz <ma.munoz.araya <at> gmail.com>
Cc: 42545 <at> debbugs.gnu.org
Subject: Re: bug#42545: 28.0.50; json-pretty-print cant't handle json having
 "t" as a key
Date: Sun, 26 Jul 2020 13:38:59 +0300

retitle 42545 json-encode-key can't handle boolean values
found 42545 24.5
quit

Marcelo Muñoz <ma.munoz.araya <at> gmail.com> writes:

> Try to apply json-pretty-print to follow json:
>
> {"t": 1, "key":2}
>
> fail with the message:  json-pretty-print: Bad JSON object key: t

Here are some simpler repros:

(json-encode '((nil . 0)))
(json-encode '((t . 0)))
(json-encode-key nil)
(json-encode-key t)

All of these fail with json-key-format since at least as far back as
Emacs 24.5.

> Some research point me the problem is in the function json-encode-key,
> its  implementation is:
>
> (defun json-encode-key (object)
>   "Return a JSON representation of OBJECT.
> If the resulting JSON object isn't a valid JSON object key,
> this signals `json-key-format'."
>   (let ((encoded (json-encode object)))
>     (unless (stringp (json-read-from-string encoded))
>       (signal 'json-key-format (list object)))
>     encoded))
>
> The follow change seems to  solve the problem, but I haven't had time
> understand it in depth:
>
> (defun json-encode-key (object)
>    "Return s string representation of a key"
>    (json-encode-string (symbol-name object)))

This can't be TRT, because OBJECT is not always a symbol.

See also https://debbugs.gnu.org/24252#26 for some precedent in
rewriting json-encode-key without relying on json-encode.

I'm AFK until start of August, but I'll try to have a better look at
this when I get the chance if no-one beats me to it.

Thanks,

-- 
Basil

Message #19 received at 42545 <at> debbugs.gnu.org (full text, mbox):

From: "Basil L. Contovounesios" <contovob <at> tcd.ie>
To: Marcelo Muñoz <ma.munoz.araya <at> gmail.com>
Cc: 42545 <at> debbugs.gnu.org
Subject: Re: bug#42545: 28.0.50; json-pretty-print cant't handle json having
 "t" as a key
Date: Thu, 11 Feb 2021 14:20:11 +0000

[Message part 1 (text/plain, inline)]

"Basil L. Contovounesios" <contovob <at> tcd.ie> writes:

> Marcelo Muñoz <ma.munoz.araya <at> gmail.com> writes:
>
>> Try to apply json-pretty-print to follow json:
>>
>> {"t": 1, "key":2}
>>
>> fail with the message:  json-pretty-print: Bad JSON object key: t
>
> Here are some simpler repros:
>
> (json-encode '((nil . 0)))
> (json-encode '((t . 0)))
> (json-encode-key nil)
> (json-encode-key t)
>
> All of these fail with json-key-format since at least as far back as
> Emacs 24.5.

[...]

> See also https://debbugs.gnu.org/24252#26 for some precedent in
> rewriting json-encode-key without relying on json-encode.
>
> I'm AFK until start of August, but I'll try to have a better look at
> this when I get the chance if no-one beats me to it.

Sorry, finally got around to this.  The attached patch should fix this
issue while also speeding up encoding in a backward compatible way.
Using the same benchmark as https://bugs.gnu.org/40693#89 I get the
following:

  encode canada.json
  old
  (1.409496598 96 0.7710352720000002)
  (1.406660968 96 0.7707586369999997)
  (1.406515696 96 0.7698804519999998)
  (1.4098724120000001 96 0.7712946)
  new
  (1.452364951 96 0.7682001569999999)
  (1.451790854 96 0.7712237389999999)
  (1.452158289 96 0.7710199420000006)
  (1.4520665160000001 96 0.7707500029999999)

This shows that the two extra cases of funcall+cond in json-encode
slightly slow down encoding of large numbers of numbers, but I doubt
it's significant enough.  If it is, we can probably just tweak the
dispatch order in json-encode.

  encode citm_catalog.json
  old
  (2.7812737399999996 272 2.1942181940000003)
  (2.77954628 272 2.1904517840000004)
  (2.779567506 272 2.1901039010000005)
  (2.778913438 272 2.189370834)
  new
  (0.7056556740000001 68 0.55314481)
  (0.704577043 68 0.5515927839999994)
  (0.702683784 68 0.5491281600000004)
  (0.703850623 68 0.5503691039999996)

  encode twitter.json
  old
  (1.427292653 148 1.1098771399999983)
  (1.428440774 148 1.109535473000001)
  (1.4265714 148 1.1097104909999977)
  (1.426152699 148 1.110347719)
  new
  (0.365952034 40 0.29652698499999985)
  (0.366947621 40 0.29772050399999905)
  (0.36731820000000004 40 0.29776995099999937)
  (0.366228327 40 0.29696426200000126)

These show that examples with more realistic objects are encoded far
faster.

Decoding performance is not affected.

This change fixes some errors in json.el and brings it a tiny bit closer
to json.c in handling confusable keys, but doesn't go all the way, for
backward compatibility.  For example, json-encode and json-serialize
still disagree on ((:t . 1)), (("t" . 1)), (t 1), ("t" 1), hash tables
with non-string keys, etc.

WDYT?

-- 
Basil

[0001-Fix-json.el-encoding-of-confusable-object-keys.patch (text/x-diff, attachment)]

Message #28 received at 42545-done <at> debbugs.gnu.org (full text, mbox):

From: "Basil L. Contovounesios" <contovob <at> tcd.ie>
To: Marcelo Muñoz <ma.munoz.araya <at> gmail.com>
Cc: 42545-done <at> debbugs.gnu.org
Subject: Re: bug#42545: 28.0.50; json-pretty-print cant't handle json having
 "t" as a key
Date: Sun, 21 Feb 2021 13:01:57 +0000

tags 42545 fixed
close 42545 28.1
quit

"Basil L. Contovounesios" <contovob <at> tcd.ie> writes:

> The attached patch should fix this issue while also speeding up
> encoding in a backward compatible way.

I've now pushed it to master, and am therefore closing this report.

Fix json.el encoding of confusable object keys
908f251e19 2021-02-21 12:53:45 +0000
https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=908f251e19dc64c75000f87bc6db4e9a8852d1ad

Thanks,

-- 
Basil

Message #31 received at 42545-done <at> debbugs.gnu.org (full text, mbox):

From: Dmitry Gutov <dgutov <at> yandex.ru>
To: "Basil L. Contovounesios" <contovob <at> tcd.ie>,
 Marcelo Muñoz <ma.munoz.araya <at> gmail.com>
Cc: 42545-done <at> debbugs.gnu.org
Subject: Re: bug#42545: 28.0.50; json-pretty-print cant't handle json having
 "t" as a key
Date: Tue, 23 Feb 2021 03:23:19 +0200

Hi Basil,

On 21.02.2021 15:01, Basil L. Contovounesios wrote:
>> The attached patch should fix this issue while also speeding up
>> encoding in a backward compatible way.
> I've now pushed it to master, and am therefore closing this report.

Looking good, thank you!

Message #34 received at 42545 <at> debbugs.gnu.org (full text, mbox):

From: "Basil L. Contovounesios" <contovob <at> tcd.ie>
To: Dmitry Gutov <dgutov <at> yandex.ru>
Cc: 42545 <at> debbugs.gnu.org
Subject: Re: bug#42545: 28.0.50; json-pretty-print cant't handle json having
 "t" as a key
Date: Tue, 23 Feb 2021 01:41:41 +0000

Hi Dmitry,

Dmitry Gutov <dgutov <at> yandex.ru> writes:

> On 21.02.2021 15:01, Basil L. Contovounesios wrote:
>>> The attached patch should fix this issue while also speeding up
>>> encoding in a backward compatible way.
>> I've now pushed it to master, and am therefore closing this report.
>
> Looking good, thank you!

Thanks, you're welcome!  But I'm not yet done speeding up json.el
encoding backward compatibly, so stay tuned ;).

-- 
Basil

Message #37 received at 42545 <at> debbugs.gnu.org (full text, mbox):

From: Dmitry Gutov <dgutov <at> yandex.ru>
To: "Basil L. Contovounesios" <contovob <at> tcd.ie>
Cc: 42545 <at> debbugs.gnu.org
Subject: Re: bug#42545: 28.0.50; json-pretty-print cant't handle json having
 "t" as a key
Date: Tue, 23 Feb 2021 14:10:58 +0200

On 23.02.2021 03:41, Basil L. Contovounesios wrote:
> Thanks, you're welcome!  But I'm not yet done speeding up json.el
> encoding backward compatibly, so stay tuned;).

I'm more partial to faster decoding myself ;-). but correctness 
improvements are welcome either way!

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.