GNU bug report logs - #48125
28.0.50; Display garbled by emoji variation sequence

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 48125 in the body.
You can then email your comments to 48125 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Filipp Gunbin <fgunbin <at> fastmail.fm>
To: bug-gnu-emacs <at> gnu.org
Subject: 28.0.50; Display garbled by emoji variation sequence
Date: Fri, 30 Apr 2021 19:20:12 +0300

emacs -Q

Enter a base emoji char and composing variation selector right after it:

C-x 8 RET SOCCER BALL
C-x 8 RET VARIATION SELECTOR-16

Then, for example:
C-a a s d f

And the display becomes garbled.

When auto-composition is turned off (M-x toggle-auto-composition),
everything is fine.

The above sequence is a real text which I got in an email header.  It
seems to be a valid variation sequence:

http://www.unicode.org/Public/emoji/5.0/emoji-variation-sequences.txt

...
26BD FE0E  ; text style;  # (5.2) SOCCER BALL
26BD FE0F  ; emoji style; # (5.2) SOCCER BALL
...

This is in tty emacs, built from latest master, on macOS.

Thanks.


In GNU Emacs 28.0.50 (build 4, x86_64-apple-darwin20.3.0, NS appkit-2022.30 Version 11.2.3 (Build 20D91))
 of 2021-04-30 built on fgunbin.local
System Description:  macOS 11.2.3

Configured features:
ACL GLIB GNUTLS LCMS2 LIBXML2 MODULES NOTIFY KQUEUE NS PDUMPER PNG RSVG
THREADS TOOLKIT_SCROLL_BARS XIM ZLIB

Message #8 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Filipp Gunbin <fgunbin <at> fastmail.fm>
To: bug-gnu-emacs <at> gnu.org
Subject: Re: bug#48125: 28.0.50; Display garbled by emoji variation sequence
Date: Fri, 30 Apr 2021 20:26:00 +0300

I'm using standard macOS Terminal app, with font DejaVu Sans Mono
(latest release 2.37 from 2016, see https://dejavu-fonts.github.io).

If I start gui Emacs with the same font, then the soccer ball doesn't
draw (C-u C-x = says "no font").

So Terminal must be using glyphs from somewhere else to draw the soccer
ball…  But how can I find out what exactly?

Message #11 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Filipp Gunbin <fgunbin <at> fastmail.fm>
Cc: bug-gnu-emacs <at> gnu.org
Subject: Re: bug#48125: 28.0.50; Display garbled by emoji variation sequence
Date: Fri, 30 Apr 2021 20:57:45 +0300

> From: Filipp Gunbin <fgunbin <at> fastmail.fm>
> Date: Fri, 30 Apr 2021 19:20:12 +0300
> 
> emacs -Q
> 
> Enter a base emoji char and composing variation selector right after it:
> 
> C-x 8 RET SOCCER BALL
> C-x 8 RET VARIATION SELECTOR-16
> 
> Then, for example:
> C-a a s d f
> 
> And the display becomes garbled.

It isn't garbled here.

> This is in tty emacs, built from latest master, on macOS.

In TTY Emacs, the composition is handled by the terminal emulator.  So
I guess this bug should be reported to them.

Message #14 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Filipp Gunbin <fgunbin <at> fastmail.fm>
Cc: bug-gnu-emacs <at> gnu.org
Subject: Re: bug#48125: 28.0.50; Display garbled by emoji variation sequence
Date: Fri, 30 Apr 2021 21:01:41 +0300

> From: Filipp Gunbin <fgunbin <at> fastmail.fm>
> Date: Fri, 30 Apr 2021 20:26:00 +0300
> 
> I'm using standard macOS Terminal app, with font DejaVu Sans Mono
> (latest release 2.37 from 2016, see https://dejavu-fonts.github.io).
> 
> If I start gui Emacs with the same font, then the soccer ball doesn't
> draw (C-u C-x = says "no font").
> 
> So Terminal must be using glyphs from somewhere else to draw the soccer
> ball…  But how can I find out what exactly?

Ask the Terminal's developers?

Message #17 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Filipp Gunbin <fgunbin <at> fastmail.fm>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: bug-gnu-emacs <at> gnu.org
Subject: Re: bug#48125: 28.0.50; Display garbled by emoji variation sequence
Date: Fri, 30 Apr 2021 22:22:41 +0300

On 30/04/2021 21:01 +0300, Eli Zaretskii wrote:

>> From: Filipp Gunbin <fgunbin <at> fastmail.fm>
>> Date: Fri, 30 Apr 2021 20:26:00 +0300
>>
>> I'm using standard macOS Terminal app, with font DejaVu Sans Mono
>> (latest release 2.37 from 2016, see https://dejavu-fonts.github.io).
>>
>> If I start gui Emacs with the same font, then the soccer ball doesn't
>> draw (C-u C-x = says "no font").
>>
>> So Terminal must be using glyphs from somewhere else to draw the soccer
>> ball…  But how can I find out what exactly?
>
> Ask the Terminal's developers?

Looks like it uses the "Apple Color Emoji" font.

Yes, I should probably report the bug to Apple, so this may be closed.

Thanks.

Message #20 received at 48125 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: 48125 <at> debbugs.gnu.org
Subject: Re: bug#48125: 28.0.50; Display garbled by emoji variation sequence
Date: Sun, 02 May 2021 09:21:13 +0200

Filipp Gunbin <fgunbin <at> fastmail.fm> writes:

> Yes, I should probably report the bug to Apple, so this may be closed.

OK; closing.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #25 received at 48125 <at> debbugs.gnu.org (full text, mbox):

From: Filipp Gunbin <fgunbin <at> fastmail.fm>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 48125 <at> debbugs.gnu.org
Subject: Re: bug#48125: 28.0.50; Display garbled by emoji variation sequence
Date: Tue, 04 May 2021 18:34:09 +0300

On 30/04/2021 20:57 +0300, Eli Zaretskii wrote:

>> From: Filipp Gunbin <fgunbin <at> fastmail.fm>
>> Date: Fri, 30 Apr 2021 19:20:12 +0300
>> 
>> emacs -Q
>> 
>> Enter a base emoji char and composing variation selector right after it:
>> 
>> C-x 8 RET SOCCER BALL
>> C-x 8 RET VARIATION SELECTOR-16
>> 
>> Then, for example:
>> C-a a s d f
>> 
>> And the display becomes garbled.
>
> It isn't garbled here.
>
>> This is in tty emacs, built from latest master, on macOS.
>
> In TTY Emacs, the composition is handled by the terminal emulator.  So
> I guess this bug should be reported to them.

BTW, I see that emoji, as well as other characters from fonts different
from my selected one (Terminal has some font fallback logic), have
different width.  Is it ok for Emacs if a terminal emulator uses
different widths for different chars?

Message #28 received at 48125 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Filipp Gunbin <fgunbin <at> fastmail.fm>
Cc: 48125 <at> debbugs.gnu.org
Subject: Re: bug#48125: 28.0.50; Display garbled by emoji variation sequence
Date: Tue, 04 May 2021 19:05:02 +0300

> From: Filipp Gunbin <fgunbin <at> fastmail.fm>
> Cc: 48125 <at> debbugs.gnu.org
> Date: Tue, 04 May 2021 18:34:09 +0300
> 
> BTW, I see that emoji, as well as other characters from fonts different
> from my selected one (Terminal has some font fallback logic), have
> different width.  Is it ok for Emacs if a terminal emulator uses
> different widths for different chars?

OK in what sense?  On TTY frames layout and rendering are left to the
terminal emulator; if it can handle this situation, Emacs won't care.

Message #31 received at 48125 <at> debbugs.gnu.org (full text, mbox):

From: Filipp Gunbin <fgunbin <at> fastmail.fm>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 48125 <at> debbugs.gnu.org
Subject: Re: bug#48125: 28.0.50; Display garbled by emoji variation sequence
Date: Tue, 04 May 2021 19:20:27 +0300

On 04/05/2021 19:05 +0300, Eli Zaretskii wrote:

>> From: Filipp Gunbin <fgunbin <at> fastmail.fm>
>> Cc: 48125 <at> debbugs.gnu.org
>> Date: Tue, 04 May 2021 18:34:09 +0300
>>
>> BTW, I see that emoji, as well as other characters from fonts different
>> from my selected one (Terminal has some font fallback logic), have
>> different width.  Is it ok for Emacs if a terminal emulator uses
>> different widths for different chars?
>
> OK in what sense?  On TTY frames layout and rendering are left to the
> terminal emulator; if it can handle this situation, Emacs won't care.

The effect of different widths is, of course, different number of chars
per line (easily seen in Terminal with emoji or Asian scripts).  Does
Emacs make assumptions about number of columns per line?  That's what I
meant.

Thanks.

Message #34 received at 48125 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Filipp Gunbin <fgunbin <at> fastmail.fm>
Cc: 48125 <at> debbugs.gnu.org
Subject: Re: bug#48125: 28.0.50; Display garbled by emoji variation sequence
Date: Tue, 04 May 2021 19:51:04 +0300

> From: Filipp Gunbin <fgunbin <at> fastmail.fm>
> Cc: 48125 <at> debbugs.gnu.org
> Date: Tue, 04 May 2021 19:20:27 +0300
> 
> > OK in what sense?  On TTY frames layout and rendering are left to the
> > terminal emulator; if it can handle this situation, Emacs won't care.
> 
> The effect of different widths is, of course, different number of chars
> per line (easily seen in Terminal with emoji or Asian scripts).  Does
> Emacs make assumptions about number of columns per line?

It does, but the number of columns doesn't change, only the width
changes.  Right?

What Emacs expects is that the terminal emulator will space characters
according to the width in columns (either 1 or 2) of each character,
as defined in the Unicode Character database.  You can see what Emacs
thinks about the width of each character by using 'char-width'.
AFAIR, all emoji should have width of 2, is that right?

Message #37 received at 48125 <at> debbugs.gnu.org (full text, mbox):

From: Filipp Gunbin <fgunbin <at> fastmail.fm>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 48125 <at> debbugs.gnu.org
Subject: Re: bug#48125: 28.0.50; Display garbled by emoji variation sequence
Date: Tue, 04 May 2021 22:06:36 +0300

[Message part 1 (text/plain, inline)]

On 04/05/2021 19:51 +0300, Eli Zaretskii wrote:

>> From: Filipp Gunbin <fgunbin <at> fastmail.fm>
>> Cc: 48125 <at> debbugs.gnu.org
>> Date: Tue, 04 May 2021 19:20:27 +0300
>>
>> > OK in what sense?  On TTY frames layout and rendering are left to the
>> > terminal emulator; if it can handle this situation, Emacs won't care.
>>
>> The effect of different widths is, of course, different number of chars
>> per line (easily seen in Terminal with emoji or Asian scripts).  Does
>> Emacs make assumptions about number of columns per line?
>
> It does, but the number of columns doesn't change, only the width
> changes.  Right?
>
> What Emacs expects is that the terminal emulator will space characters
> according to the width in columns (either 1 or 2) of each character,
> as defined in the Unicode Character database.  You can see what Emacs
> thinks about the width of each character by using 'char-width'.
> AFAIR, all emoji should have width of 2, is that right?

I see now!

One more question.  In the attached screenshot there're two lines, one
with soccer balls (width 2), and another with letter "a" (width 1).
Both do not fit on screen, and are continued to the next (screen) line.
When at EOL of either, C-x = reports column=142, but the first is longer
(by 1 column).  That is because it's continued with two backslashes, not
one.  Is this the right behavior?

[Screenshot 2021-05-04 at 21.55.48.png (image/png, attachment)]

Message #40 received at 48125 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Filipp Gunbin <fgunbin <at> fastmail.fm>
Cc: 48125 <at> debbugs.gnu.org
Subject: Re: bug#48125: 28.0.50; Display garbled by emoji variation sequence
Date: Tue, 04 May 2021 22:17:18 +0300

> From: Filipp Gunbin <fgunbin <at> fastmail.fm>
> Cc: 48125 <at> debbugs.gnu.org
> Date: Tue, 04 May 2021 22:06:36 +0300
> 
> One more question.  In the attached screenshot there're two lines, one
> with soccer balls (width 2), and another with letter "a" (width 1).
> Both do not fit on screen, and are continued to the next (screen) line.
> When at EOL of either, C-x = reports column=142, but the first is longer
> (by 1 column).  That is because it's continued with two backslashes, not
> one.  Is this the right behavior?

I don't see the continuation backslashes in the screenshot, but from
your description the behavior sounds right.  There could even be more
than 2 backslashes in some rare cases.

Message #43 received at 48125 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: fgunbin <at> fastmail.fm, 48125 <at> debbugs.gnu.org
Subject: Re: bug#48125: 28.0.50; Display garbled by emoji variation sequence
Date: Tue, 04 May 2021 22:29:23 +0300

> Date: Tue, 04 May 2021 22:17:18 +0300
> From: Eli Zaretskii <eliz <at> gnu.org>
> Cc: 48125 <at> debbugs.gnu.org
> 
> > From: Filipp Gunbin <fgunbin <at> fastmail.fm>
> > Cc: 48125 <at> debbugs.gnu.org
> > Date: Tue, 04 May 2021 22:06:36 +0300
> > 
> > One more question.  In the attached screenshot there're two lines, one
> > with soccer balls (width 2), and another with letter "a" (width 1).
> > Both do not fit on screen, and are continued to the next (screen) line.
> > When at EOL of either, C-x = reports column=142, but the first is longer
> > (by 1 column).  That is because it's continued with two backslashes, not
> > one.  Is this the right behavior?
> 
> I don't see the continuation backslashes in the screenshot, but from
> your description the behavior sounds right.  There could even be more
> than 2 backslashes in some rare cases.

Oh, and of course the backslashes aren't counted as part of the column
number, only the displayed text is counted.

Message #46 received at 48125 <at> debbugs.gnu.org (full text, mbox):

From: Filipp Gunbin <fgunbin <at> fastmail.fm>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 48125 <at> debbugs.gnu.org
Subject: Re: bug#48125: 28.0.50; Display garbled by emoji variation sequence
Date: Tue, 04 May 2021 23:36:19 +0300

[Message part 1 (text/plain, inline)]

On 04/05/2021 22:29 +0300, Eli Zaretskii wrote:

>> Date: Tue, 04 May 2021 22:17:18 +0300
>> From: Eli Zaretskii <eliz <at> gnu.org>
>> Cc: 48125 <at> debbugs.gnu.org
>>
>> > From: Filipp Gunbin <fgunbin <at> fastmail.fm>
>> > Cc: 48125 <at> debbugs.gnu.org
>> > Date: Tue, 04 May 2021 22:06:36 +0300
>> >
>> > One more question.  In the attached screenshot there're two lines, one
>> > with soccer balls (width 2), and another with letter "a" (width 1).
>> > Both do not fit on screen, and are continued to the next (screen) line.
>> > When at EOL of either, C-x = reports column=142, but the first is longer
>> > (by 1 column).  That is because it's continued with two backslashes, not
>> > one.  Is this the right behavior?
>>
>> I don't see the continuation backslashes in the screenshot, but from
>> your description the behavior sounds right.  There could even be more
>> than 2 backslashes in some rare cases.
>
> Oh, and of course the backslashes aren't counted as part of the column
> number, only the displayed text is counted.

Re-checked - they should be visible in the screenshot (maybe you didn't
scroll the image to the right?)

Ok, I have two continuation backslashes in the end of the screen line
with soccer balls.  But what I wanted to note is that on continuation
lines, I get equal column numbers at different screen positions.

I've now made two screenshots with point at each EOL, to show that
screen positions are different.

[Screenshot 2021-05-04 at 23.30.07.png (image/png, attachment)]

[Screenshot 2021-05-04 at 23.30.18.png (image/png, attachment)]

Message #49 received at 48125 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Filipp Gunbin <fgunbin <at> fastmail.fm>
Cc: 48125 <at> debbugs.gnu.org
Subject: Re: bug#48125: 28.0.50; Display garbled by emoji variation sequence
Date: Wed, 05 May 2021 14:49:54 +0300

> From: Filipp Gunbin <fgunbin <at> fastmail.fm>
> Cc: 48125 <at> debbugs.gnu.org
> Date: Tue, 04 May 2021 23:36:19 +0300
> 
> Ok, I have two continuation backslashes in the end of the screen line
> with soccer balls.  But what I wanted to note is that on continuation
> lines, I get equal column numbers at different screen positions.
> 
> I've now made two screenshots with point at each EOL, to show that
> screen positions are different.

OK, but what is the point you want to make?  current-column doesn't
measure screen coordinates, it measures the sum of character-widths.
A single-width character increments the column count by 1, a
double-width character increments it by 2.  Given this fact, is there
something wrong that you see with the reported column in each case?

Message #52 received at 48125 <at> debbugs.gnu.org (full text, mbox):

From: Filipp Gunbin <fgunbin <at> fastmail.fm>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 48125 <at> debbugs.gnu.org
Subject: Re: bug#48125: 28.0.50; Display garbled by emoji variation sequence
Date: Wed, 05 May 2021 16:41:19 +0300

On 05/05/2021 14:49 +0300, Eli Zaretskii wrote:

>> From: Filipp Gunbin <fgunbin <at> fastmail.fm>
>> Cc: 48125 <at> debbugs.gnu.org
>> Date: Tue, 04 May 2021 23:36:19 +0300
>>
>> Ok, I have two continuation backslashes in the end of the screen line
>> with soccer balls.  But what I wanted to note is that on continuation
>> lines, I get equal column numbers at different screen positions.
>>
>> I've now made two screenshots with point at each EOL, to show that
>> screen positions are different.
>
> OK, but what is the point you want to make?  current-column doesn't
> measure screen coordinates, it measures the sum of character-widths.
> A single-width character increments the column count by 1, a
> double-width character increments it by 2.  Given this fact, is there
> something wrong that you see with the reported column in each case?

I wanted to confirm that this all is ok, now you say just that, thanks.

For the record, from what I can see, Terminal has problems when
composing 2-column chars with that VARIATION SELECTOR-16 (and maybe
others).

Filipp

Message #55 received at 48125 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Filipp Gunbin <fgunbin <at> fastmail.fm>
Cc: 48125 <at> debbugs.gnu.org
Subject: Re: bug#48125: 28.0.50; Display garbled by emoji variation sequence
Date: Wed, 05 May 2021 17:05:44 +0300

> From: Filipp Gunbin <fgunbin <at> fastmail.fm>
> Cc: 48125 <at> debbugs.gnu.org
> Date: Wed, 05 May 2021 16:41:19 +0300
> 
> For the record, from what I can see, Terminal has problems when
> composing 2-column chars with that VARIATION SELECTOR-16 (and maybe
> others).

Yes, well.  Emacs doesn't yet support variation selectors, either.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.