GNU bug report logs - #33729
27.0.50; Partial glyphs not rendered for Gujarati with Harfbuzz enabled (renders fine using m17n)

Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.

Package: emacs; Reported by: Kaushal Modi <kaushal.modi@HIDDEN>; dated Thu, 13 Dec 2018 20:22:02 UTC; Maintainer for emacs is bug-gnu-emacs@HIDDEN.

Message received at 33729 <at> debbugs.gnu.org:


Received: (at 33729) by debbugs.gnu.org; 29 Jan 2019 22:34:09 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Jan 29 17:34:09 2019
Received: from localhost ([127.0.0.1]:50931 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gobxI-0000qH-Rw
	for submit <at> debbugs.gnu.org; Tue, 29 Jan 2019 17:34:09 -0500
Received: from mail-wm1-f54.google.com ([209.85.128.54]:54701)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dr.khaled.hosny@HIDDEN>) id 1gobxG-0000pm-SS
 for 33729 <at> debbugs.gnu.org; Tue, 29 Jan 2019 17:34:07 -0500
Received: by mail-wm1-f54.google.com with SMTP id a62so19626379wmh.4
 for <33729 <at> debbugs.gnu.org>; Tue, 29 Jan 2019 14:34:06 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=date:from:to:cc:subject:message-id:references:mime-version
 :content-disposition:content-transfer-encoding:in-reply-to
 :user-agent; bh=sjpl2j4h3lwodH5ZNN8SBKjW0MscgxcsPGHfxezSapQ=;
 b=AwtGu5L9LkJqX4Rm6bRKXiUt5rVrAmmd8glIgG/+SBJIhyg799KP4CySc0g8og/bhP
 2WhOe+xurnOZIu13WzNPWrVnJweBYGGAkvgHv6U5T9p//+TvmLgiBjsJrNW9U60WHW+W
 1c+HLiWn7x00BhkrXa8GOZiJ+7DV8g9YIL6GSFZxYn5kV21bHmxO0rZgFAEebTzeM+5d
 D6LXo8uwho+L/P5Qn371RwZv1C16K6BAHS5E0VwXfzlst6wzaNGsZDLimdi1UOIrmAfG
 9NvMx3IdMC2i+GQgWw+DYVJxrzLNRnO7Iw27i/r2d+ua7J2znxt4IfeDzWG5S4A8SGvv
 pwEw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:date:from:to:cc:subject:message-id:references
 :mime-version:content-disposition:content-transfer-encoding
 :in-reply-to:user-agent;
 bh=sjpl2j4h3lwodH5ZNN8SBKjW0MscgxcsPGHfxezSapQ=;
 b=TNVbZe7ANHIJflA/KQuKhGj+VJk8TY1qXvInXqS3hkk/7dWws/wrmkq1uyHCOHFrR1
 WYVWfywy3EzUL/I+boUkH/xRrkPGQ3FqZMM+l/6e6nbUox3PnCVLbDjJYFboKUE4byjj
 7zD+DHTw6J1bdXlv1Fnm/bRbtcoQahLwFwGolBm6HHetd/sN93LHQsqBuGtC6olR0MBQ
 Qy3S2sYSNC6kUEX7aBsKd6yq46xDzGsQzKqQa5M1i2YhS2lbUEzg6uUylgwctWtwWS/A
 R3PpyTNel84eoVLZiYO7p5kilYh/cMGMqpVci/pR2aJAJoV2Iw0PH8V0VgdD9PGeRgyS
 MrAw==
X-Gm-Message-State: AJcUukdXtInfkVPy/uJhyJBqh/2LzXaRqNHK7YM6j0wTxnOfDDbHA4N9
 5nM4TMcC1XXZ+hCsDxXsz8ywR6W0
X-Google-Smtp-Source: ALg8bN7SXuo5p1oRpVjHHG2k6mQboFN1AKlHfHrqMCl430Tz5Jp2ywCgWz/kn7tHtpoV0Zv5rqccAA==
X-Received: by 2002:a1c:d00d:: with SMTP id h13mr22308758wmg.13.1548801240587; 
 Tue, 29 Jan 2019 14:34:00 -0800 (PST)
Received: from macbook.localdomain ([197.58.129.117])
 by smtp.gmail.com with ESMTPSA id g188sm96124wmf.32.2019.01.29.14.33.49
 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256);
 Tue, 29 Jan 2019 14:33:59 -0800 (PST)
Date: Wed, 30 Jan 2019 00:33:30 +0200
From: Khaled Hosny <dr.khaled.hosny@HIDDEN>
To: Eli Zaretskii <eliz@HIDDEN>
Subject: Re: bug#33729: 27.0.50; Partial glyphs not rendered for Gujarati
 with Harfbuzz enabled (renders fine using m17n)
Message-ID: <20190129223330.GC20784@HIDDEN>
References: <838t0gapcj.fsf@HIDDEN>
 <20181223135109.GA6568@HIDDEN>
 <83va3k8c79.fsf@HIDDEN>
 <20181224020847.GC6568@HIDDEN>
 <83lg4e9a7q.fsf@HIDDEN>
 <20181224173723.GH6568@HIDDEN>
 <838t085qx8.fsf@HIDDEN>
 <20190105205314.GA28761@HIDDEN>
 <20190105210420.GA31354@HIDDEN>
 <83pnt9zn73.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <83pnt9zn73.fsf@HIDDEN>
User-Agent: Mutt/1.11.2 (2019-01-07)
X-Spam-Score: 1.0 (+)
X-Debbugs-Envelope-To: 33729
Cc: behdad@HIDDEN, far.nasiri.m@HIDDEN, 33729 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

On Sun, Jan 06, 2019 at 07:54:24PM +0200, Eli Zaretskii wrote:
> > Date: Sat, 5 Jan 2019 23:04:20 +0200
> > From: Khaled Hosny <dr.khaled.hosny@HIDDEN>
> > Cc: far.nasiri.m@HIDDEN, behdad@HIDDEN, 33729 <at> debbugs.gnu.org
> > 
> > I pushed a couple of commits that does this based on my limited
> > understanding of Emacs code, please check.
> 
> Can you explain why you moved the call to
> hb_buffer_guess_segment_properties _after_ the code which sets some of
> the properties?  I cannot find anything about that in the HarfBuzz
> documentation.  Is this because guessing the unset properties can
> benefit from knowing the properties which _are_ set, such as the
> direction?

hb_buffer_guess_segment_properties() won’t guess set properties, so moving
it last was to avoid wasting time guessing properties that we will
override later anyway.

> I did it the other way around, because my mental model was: first set
> the defaults, then override them where better info is available.

hb_buffer_guess_segment_properties() is not for setting defaults (there
is no such thing as default buffer properties in HarfBuzz working
model), it is a kind of quick and dirty hack and production code should
not use it.

Regards,
Khaled




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33729; Package emacs. Full text available.

Message received at 33729 <at> debbugs.gnu.org:


Received: (at 33729) by debbugs.gnu.org; 29 Jan 2019 22:29:38 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Jan 29 17:29:38 2019
Received: from localhost ([127.0.0.1]:50915 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gobsw-0000hQ-2q
	for submit <at> debbugs.gnu.org; Tue, 29 Jan 2019 17:29:38 -0500
Received: from mail-wm1-f53.google.com ([209.85.128.53]:35595)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dr.khaled.hosny@HIDDEN>) id 1gobst-0000h9-Sn
 for 33729 <at> debbugs.gnu.org; Tue, 29 Jan 2019 17:29:36 -0500
Received: by mail-wm1-f53.google.com with SMTP id t200so19690963wmt.0
 for <33729 <at> debbugs.gnu.org>; Tue, 29 Jan 2019 14:29:35 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=date:from:to:cc:subject:message-id:references:mime-version
 :content-disposition:in-reply-to:user-agent;
 bh=0m8cwCuQ82+6gzIKdbxcnClPxSla5t4Ch5niiQJNV7A=;
 b=E8vn8VWZ1bIDJS+qhp710BFht/OarRPDSPmMkmPCu6/GxxUnqW5r3WMJ91M66KRZio
 SpWUcyE9AeD7PjAPOiCGTxk/oZXBY8rKdRYjiIepCGCZThGc0D3Ymq1piVKzl5tU7bHu
 bKZ7omWCDWd2ZVyjbjnU6bfSTAnn8F2ijjoYcd18olUSPrjOQr96UQfn0YqkPx3bo//l
 Iu4AesD9N83Fn0SiVwKlCqDpz9R+057kKEoHbufcUXIBPvcx9lQl+tKApEMmrcZmutPb
 AdvmDj3FQAwJv1sq7DQ3kvCLuYc89ZvZ61GuZcjZN1H0pBuyreXdGYK/9/3DDWP7H2oH
 jqUA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:date:from:to:cc:subject:message-id:references
 :mime-version:content-disposition:in-reply-to:user-agent;
 bh=0m8cwCuQ82+6gzIKdbxcnClPxSla5t4Ch5niiQJNV7A=;
 b=NvGH7GJpIEb7Z01qYeRvw+KJ3MKG8qqGg1aQidjvr6x8fhZn9ge+gnvTev+Y3xKcuz
 WFSEiO7ayHqQa76BNFldOlc+xF4I3Y9pH02XYFf4wTOyKLw/N13PRpC2QNo5AnJ6itfG
 hpvMYwUfaVnB3+iKzjSs20yZ7AfeiI53enq1KCcaNM8GmUch3IupUm8LHsg0LyNnVwhe
 YOTzluGdm/8VuqMWG5rlzc8v0p6JgsADp8Nj/0b7T2F/dqGxxxj4GBUGdlLyYmWzCCqv
 VdgP0dbtLf3/yt0yoVE9+GS5pP0/wa4wgZ8FAzgNjAzilAjOPDz2qSKb4+wYQfy2CDLm
 yMFw==
X-Gm-Message-State: AJcUukfaVsN/3aeq9d/Rtb2p9IlWGv99eqH3p7LhyM9Un8V15YSCMD7X
 5hcDo1HTWXQzyokthnWXs9o=
X-Google-Smtp-Source: ALg8bN6rUSaf1OAGZTpGQSg5VCu113BOaAEd5NlzZAiTB1SlqaS/0mLj2z/Fu2AGw158BDFdJxq8HQ==
X-Received: by 2002:a1c:f916:: with SMTP id x22mr23900309wmh.87.1548800970083; 
 Tue, 29 Jan 2019 14:29:30 -0800 (PST)
Received: from macbook.localdomain ([197.58.129.117])
 by smtp.gmail.com with ESMTPSA id m21sm71776wmi.43.2019.01.29.14.29.20
 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256);
 Tue, 29 Jan 2019 14:29:29 -0800 (PST)
Date: Wed, 30 Jan 2019 00:29:03 +0200
From: Khaled Hosny <dr.khaled.hosny@HIDDEN>
To: Eli Zaretskii <eliz@HIDDEN>
Subject: Re: bug#33729: 27.0.50; Partial glyphs not rendered for Gujarati
 with Harfbuzz enabled (renders fine using m17n)
Message-ID: <20190129222903.GB20784@HIDDEN>
References: <20181222205948.GF2244@HIDDEN>
 <838t0gapcj.fsf@HIDDEN>
 <20181223135109.GA6568@HIDDEN>
 <83va3k8c79.fsf@HIDDEN>
 <20181224020847.GC6568@HIDDEN>
 <83lg4e9a7q.fsf@HIDDEN>
 <20181224173723.GH6568@HIDDEN>
 <838t085qx8.fsf@HIDDEN>
 <20190105205314.GA28761@HIDDEN>
 <83y37xzswx.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
In-Reply-To: <83y37xzswx.fsf@HIDDEN>
User-Agent: Mutt/1.11.2 (2019-01-07)
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 33729
Cc: behdad@HIDDEN, far.nasiri.m@HIDDEN, 33729 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

On Sun, Jan 06, 2019 at 05:50:54PM +0200, Eli Zaretskii wrote:
> > Date: Sat, 5 Jan 2019 22:53:14 +0200
> > From: Khaled Hosny <dr.khaled.hosny@HIDDEN>
> > I pushed a couple of commits that does this based on my limited
> > understanding of Emacs code, please check.
> 
> Thanks.  Do you see any difference in the results?

Strings with forced direction (e.g. Arabic with LRO) showed difference.
Without my change they were shaped RTL by drawn LTR, with my change
shaping and drawing used LTR direction. Please feel free to revert that
change if you think it is incorrect.

Regards,
Khaled




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33729; Package emacs. Full text available.

Message received at 33729 <at> debbugs.gnu.org:


Received: (at 33729) by debbugs.gnu.org; 29 Jan 2019 22:26:15 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Jan 29 17:26:15 2019
Received: from localhost ([127.0.0.1]:50911 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gobpf-0000cn-Ii
	for submit <at> debbugs.gnu.org; Tue, 29 Jan 2019 17:26:15 -0500
Received: from mail-wm1-f51.google.com ([209.85.128.51]:33016)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dr.khaled.hosny@HIDDEN>) id 1gobpd-0000ca-6K
 for 33729 <at> debbugs.gnu.org; Tue, 29 Jan 2019 17:26:13 -0500
Received: by mail-wm1-f51.google.com with SMTP id r24so14030798wmh.0
 for <33729 <at> debbugs.gnu.org>; Tue, 29 Jan 2019 14:26:13 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=date:from:to:cc:subject:message-id:references:mime-version
 :content-disposition:in-reply-to:user-agent;
 bh=nnOqJtG2W1l+4FHp29MoNuVGkq+tymuiswByMiEvwLY=;
 b=StrGEur11uHFON75mAVhpCYMnqlnEFyOFEsnBHhf/MLLIja5SrwJAsqw6c+V4dhhyo
 sUz1Pu3E4usEk/XXbKrA3tzs5zv5MOLVYUeMoRUIUKs0Etpi+9ZUt+ROjsGoOCaFFmrK
 2Cej6X3YWr324UQlya8kGSI2gLTK7fg7k24+rtVZxUPMN7URXTVdRv66kfVypOc0/U3c
 +r1x0H4gEugmtrxGD/2Rm4Kn4YESzjZlGjks2sn18lgr0UHu896YPSjH2jl9ZAAqo3aj
 hvjO0+XgFiCNRYsPHPMWqu0Wk8hqIUQ+tg7MPqVbV7w6wIptMFv0iZBBMHUm+4VXwhbH
 N4gw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:date:from:to:cc:subject:message-id:references
 :mime-version:content-disposition:in-reply-to:user-agent;
 bh=nnOqJtG2W1l+4FHp29MoNuVGkq+tymuiswByMiEvwLY=;
 b=TSJ5vOQMzPIqCKrRCG9BMJbxhnnFFC31A6IhELUR/oNYBsNFw0Ld3vQOrZMPfd6OxB
 +WynF/n5H60E53tSiVB7AzZDS6bbdLuynJ1hT9DuWYUUg2KvixEURxGjDsjkPMkdBUTG
 hbVBqAIugZJXkaDOVvAZ16qQMXsJN2hYcEFSkVLbLxCAtDbCG22V1eNRrTca+6OAScOx
 yG00yy+PaNUhoGJoDm5v9BnuYBYNggoPbSnBUyOOHYBOhPHBTZrycbAncndBggINCel3
 WOnB5+1PIAir186NffPFabR9yUxMxVuEOusSkcugJ1RHP4Lw6NHFuC5UcoR2gm9ySW5m
 Zueg==
X-Gm-Message-State: AJcUukd6/1Tc7hCo9TD5Y1FkVtTUx3Q/N6Hvs4y4Gkp8WytycBe3dUCI
 CqRHGf8jdKh6QB9h0Bbnevs=
X-Google-Smtp-Source: ALg8bN5wDrgLsKF/0Bf6nDkpm81Jp1dLZlzNpM+rnQsGElIKuYyLWJ75Q6jOX95JyJZQExaoZHFmYw==
X-Received: by 2002:a1c:a913:: with SMTP id s19mr21811394wme.4.1548800767254; 
 Tue, 29 Jan 2019 14:26:07 -0800 (PST)
Received: from macbook.localdomain ([197.58.129.117])
 by smtp.gmail.com with ESMTPSA id f130sm72862wme.41.2019.01.29.14.25.55
 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256);
 Tue, 29 Jan 2019 14:26:05 -0800 (PST)
Date: Wed, 30 Jan 2019 00:25:36 +0200
From: Khaled Hosny <dr.khaled.hosny@HIDDEN>
To: Eli Zaretskii <eliz@HIDDEN>
Subject: Re: bug#33729: 27.0.50; Partial glyphs not rendered for Gujarati
 with Harfbuzz enabled (renders fine using m17n)
Message-ID: <20190129222536.GA20784@HIDDEN>
References: <838t0gapcj.fsf@HIDDEN>
 <20181223135109.GA6568@HIDDEN>
 <83va3k8c79.fsf@HIDDEN>
 <20181224020847.GC6568@HIDDEN>
 <83lg4e9a7q.fsf@HIDDEN>
 <20181224173723.GH6568@HIDDEN>
 <83imzi94tz.fsf@HIDDEN>
 <20190105211514.GB28761@HIDDEN>
 <83wonhzsb8.fsf@HIDDEN> <83h8du3tzv.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
In-Reply-To: <83h8du3tzv.fsf@HIDDEN>
User-Agent: Mutt/1.11.2 (2019-01-07)
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 33729
Cc: 33729 <at> debbugs.gnu.org, handa@HIDDEN, far.nasiri.m@HIDDEN,
 behdad@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

On Sun, Jan 27, 2019 at 07:12:04PM +0200, Eli Zaretskii wrote:
> Could you please respond to the below as well?

I have no time for angering these questions any more, sorry. Please feel
free to do what you find sensible.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33729; Package emacs. Full text available.

Message received at 33729 <at> debbugs.gnu.org:


Received: (at 33729) by debbugs.gnu.org; 27 Jan 2019 17:13:08 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Jan 27 12:13:08 2019
Received: from localhost ([127.0.0.1]:47840 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gnnzY-0001v1-2j
	for submit <at> debbugs.gnu.org; Sun, 27 Jan 2019 12:13:08 -0500
Received: from eggs.gnu.org ([209.51.188.92]:37368)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1gnnzW-0001uU-LN
 for 33729 <at> debbugs.gnu.org; Sun, 27 Jan 2019 12:13:06 -0500
Received: from fencepost.gnu.org ([2001:470:142:3::e]:36682)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>)
 id 1gnnzR-0001uk-Hq; Sun, 27 Jan 2019 12:13:01 -0500
Received: from [176.228.60.248] (port=1412 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1gnnzR-0000az-5G; Sun, 27 Jan 2019 12:13:01 -0500
Date: Sun, 27 Jan 2019 19:12:45 +0200
Message-Id: <83ftte3tyq.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: dr.khaled.hosny@HIDDEN
In-reply-to: <83pnt9zn73.fsf@HIDDEN> (message from Eli Zaretskii on Sun, 06
 Jan 2019 19:54:24 +0200)
Subject: Re: bug#33729: 27.0.50;
 Partial glyphs not rendered for Gujarati with Harfbuzz enabled
 (renders fine using m17n)
References: <83bm5d9wsc.fsf@HIDDEN>
 <20181222205948.GF2244@HIDDEN>
 <838t0gapcj.fsf@HIDDEN>
 <20181223135109.GA6568@HIDDEN>
 <83va3k8c79.fsf@HIDDEN>
 <20181224020847.GC6568@HIDDEN>
 <83lg4e9a7q.fsf@HIDDEN>
 <20181224173723.GH6568@HIDDEN>
 <838t085qx8.fsf@HIDDEN>
 <20190105205314.GA28761@HIDDEN>
 <20190105210420.GA31354@HIDDEN> <83pnt9zn73.fsf@HIDDEN>
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 33729
Cc: behdad@HIDDEN, far.nasiri.m@HIDDEN, 33729 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

> Date: Sun, 06 Jan 2019 19:54:24 +0200
> From: Eli Zaretskii <eliz@HIDDEN>
> Cc: behdad@HIDDEN, far.nasiri.m@HIDDEN, 33729 <at> debbugs.gnu.org
> 
> > Date: Sat, 5 Jan 2019 23:04:20 +0200
> > From: Khaled Hosny <dr.khaled.hosny@HIDDEN>
> > Cc: far.nasiri.m@HIDDEN, behdad@HIDDEN, 33729 <at> debbugs.gnu.org
> > 
> > I pushed a couple of commits that does this based on my limited
> > understanding of Emacs code, please check.
> 
> Can you explain why you moved the call to
> hb_buffer_guess_segment_properties _after_ the code which sets some of
> the properties?  I cannot find anything about that in the HarfBuzz
> documentation.  Is this because guessing the unset properties can
> benefit from knowing the properties which _are_ set, such as the
> direction?
> 
> I did it the other way around, because my mental model was: first set
> the defaults, then override them where better info is available.
> 
> Thanks.

Please respond.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33729; Package emacs. Full text available.

Message received at 33729 <at> debbugs.gnu.org:


Received: (at 33729) by debbugs.gnu.org; 27 Jan 2019 17:12:44 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Jan 27 12:12:44 2019
Received: from localhost ([127.0.0.1]:47836 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gnnz9-0001tu-Hq
	for submit <at> debbugs.gnu.org; Sun, 27 Jan 2019 12:12:43 -0500
Received: from eggs.gnu.org ([209.51.188.92]:37195)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1gnnz4-0001te-GT
 for 33729 <at> debbugs.gnu.org; Sun, 27 Jan 2019 12:12:39 -0500
Received: from fencepost.gnu.org ([2001:470:142:3::e]:36644)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>)
 id 1gnnyp-0001Q9-Gy; Sun, 27 Jan 2019 12:12:27 -0500
Received: from [176.228.60.248] (port=1367 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1gnnyn-0000We-QA; Sun, 27 Jan 2019 12:12:23 -0500
Date: Sun, 27 Jan 2019 19:12:04 +0200
Message-Id: <83h8du3tzv.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: dr.khaled.hosny@HIDDEN
In-reply-to: <83wonhzsb8.fsf@HIDDEN> (message from Eli Zaretskii on Sun, 06
 Jan 2019 18:03:55 +0200)
Subject: Re: bug#33729: 27.0.50;
 Partial glyphs not rendered for Gujarati with Harfbuzz enabled
 (renders fine using m17n)
References: <20181222154945.GE2244@HIDDEN>
 <83bm5d9wsc.fsf@HIDDEN>
 <20181222205948.GF2244@HIDDEN>
 <838t0gapcj.fsf@HIDDEN>
 <20181223135109.GA6568@HIDDEN>
 <83va3k8c79.fsf@HIDDEN>
 <20181224020847.GC6568@HIDDEN>
 <83lg4e9a7q.fsf@HIDDEN>
 <20181224173723.GH6568@HIDDEN>
 <83imzi94tz.fsf@HIDDEN> <20190105211514.GB28761@HIDDEN>
 <83wonhzsb8.fsf@HIDDEN>
MIME-version: 1.0
Content-type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 33729
Cc: 33729 <at> debbugs.gnu.org, handa@HIDDEN, far.nasiri.m@HIDDEN,
 behdad@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Could you please respond to the below as well?

> Date: Sun, 06 Jan 2019 18:03:55 +0200
> From: Eli Zaretskii <eliz@HIDDEN>
> Cc: behdad@HIDDEN, far.nasiri.m@HIDDEN, 33729 <at> debbugs.gnu.org
> 
> > Date: Sat, 5 Jan 2019 23:15:14 +0200
> > From: Khaled Hosny <dr.khaled.hosny@HIDDEN>
> > Cc: rgm@HIDDEN, far.nasiri.m@HIDDEN, behdad@HIDDEN,
> > 	33729 <at> debbugs.gnu.org, kaushal.modi@HIDDEN
> > 
> > > > The built-in HarfBuzz code is for getting the script for a given
> > > > character, but resolving characters with Common script is left to the
> > > > client. Suppose you have this string (upper case for RTL) ABC 123 DEF,
> > > > what HarfBuzz sees during shaping is three separate chunks of text ABC,
> > > > 123, DEF. The 123 part is all Common script characters and thus
> > > > hb_buffer_guess_segment_properties won’t be able to guess anything (and
> > > > based on the font and the script, this can cause rendering differences).
> > > > Emacs will have to resolve the script of Common characters before
> > > > applying bidi algorithm and pass that down to HarfBuzz.
> > > 
> > > I'm not sure I understand: why does HarfBuzz care that 123 was in the
> > > middle if RTL text.
> > 
> > It doesn’t. What it cares about here is the correct script. Because 123
> > are in the middle of RTL text they will be shaped separately, and thus
> > hb_buffer_guess_segment_properties() will only see 123 and won’t to be
> > able to guess the correct script for them (Arabic, Hebrew, etc.,
> > whatever the script for the surrounding RTL text is).
> 
> That's what I was asking: why it's important for HarfBuzz to know that
> 123 should be shaped for the Arabic script?
> 
> > Depending on the font, the digits might be shaped differently if the
> > script is, say Arabic, by e.g. applying script-specific substitutions to
> > forms more suitable for a given script.
> 
> I guess this is what I'm missing, then: these script-specific
> substitutions.  Can you elaborate on that, or point to some place
> where these substitutions are described in detail?
> 
> > > (In general, AFAIK simple characters like 123 will not even go through
> > > HarfBuzz, as Emacs doesn't call the shaper for characters whose entry
> > > in composition-function-table is nil.  So I guess 123 here should
> > > stand for some other characters, not for literal digits?  IOW, I don't
> > > think I understand the example very well.)
> > 
> > This is a bug then and needs to be fixed. All text should go through
> > HarfBuzz since even so-called “simple” character often require shaping
> > depending on the text and the font. If this is done for optimization,
> > then it should be revised to see if shaping with HarfBuzz is actually
> > significantly slower and if it is, find more proper ways to optimize it.
> 
> (Adding Handa-san to the discussion, in the hope that he could comment
> on the issue.)
> 
> I think running all text through a shaper might be prohibitively
> expensive, because the shaper is called through Lisp code (see
> composite.el), and we decide which chunk of text to pass to the shaper
> using regexp search.  See the various files under lisp/language/ which
> set up portions of composition-function-table as appropriate for each
> language that needs it.
> 
> So I think we should identify all the cases where "simple" characters
> surrounded by, or adjacent to, "non-simple" ones need to be passed to
> a shaper, and add the necessary regular expressions to the data
> structures in lisp/languages/.  Can you describe these cases, or point
> me to a place where I can find the relevant info?
> 
> Thanks.
> 
> 
> 
> 




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33729; Package emacs. Full text available.

Message received at 33729 <at> debbugs.gnu.org:


Received: (at 33729) by debbugs.gnu.org; 27 Jan 2019 17:10:18 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Jan 27 12:10:18 2019
Received: from localhost ([127.0.0.1]:47831 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gnnwn-0001qQ-Va
	for submit <at> debbugs.gnu.org; Sun, 27 Jan 2019 12:10:18 -0500
Received: from eggs.gnu.org ([209.51.188.92]:36780)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1gnnwm-0001qB-HD
 for 33729 <at> debbugs.gnu.org; Sun, 27 Jan 2019 12:10:16 -0500
Received: from fencepost.gnu.org ([2001:470:142:3::e]:36518)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>)
 id 1gnnwh-0007oL-7P; Sun, 27 Jan 2019 12:10:11 -0500
Received: from [176.228.60.248] (port=1227 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1gnnwg-0000GT-Am; Sun, 27 Jan 2019 12:10:11 -0500
Date: Sun, 27 Jan 2019 19:09:53 +0200
Message-Id: <83lg363u3i.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Khaled Hosny <dr.khaled.hosny@HIDDEN>
In-reply-to: <20190105205314.GA28761@HIDDEN> (message from Khaled
 Hosny on Sat, 5 Jan 2019 22:53:14 +0200)
Subject: Re: bug#33729: 27.0.50; Partial glyphs not rendered for Gujarati
 with Harfbuzz enabled (renders fine using m17n)
References: <20181222154945.GE2244@HIDDEN>
 <83bm5d9wsc.fsf@HIDDEN>
 <20181222205948.GF2244@HIDDEN>
 <838t0gapcj.fsf@HIDDEN>
 <20181223135109.GA6568@HIDDEN>
 <83va3k8c79.fsf@HIDDEN>
 <20181224020847.GC6568@HIDDEN>
 <83lg4e9a7q.fsf@HIDDEN>
 <20181224173723.GH6568@HIDDEN>
 <838t085qx8.fsf@HIDDEN> <20190105205314.GA28761@HIDDEN>
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 33729
Cc: behdad@HIDDEN, far.nasiri.m@HIDDEN, 33729 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

> Date: Sat, 5 Jan 2019 22:53:14 +0200
> From: Khaled Hosny <dr.khaled.hosny@HIDDEN>
> Cc: far.nasiri.m@HIDDEN, behdad@HIDDEN, 33729 <at> debbugs.gnu.org
> 
> > Done.  Please test.  I made sure it compiles, but I couldn't actually
> > test the results, as I don't have access to a GNU/Linux system with
> > GUI display.  So it could be that I misunderstood the Harfbuzz APIs,
> > as I was essentially flying blind, guided only by the Harfbuzz docs.
> 
> It seems to work, but still not quite right. You seem to be passing the
> paragraph direction, but what HarfBuzz needs is resolved direction of
> the text (i.e. the bidi embedding level of the run). In other words, if
> Emacs is going to draw this text from right to left, then HarfBuzz must
> shape it in right to left direction. Both should use the same direction
> all the time and HarfBuzz direction guessing should never be used (i.e.
> always pass to it an explicit direction).

In response to that, I wrote:

   It isn't the paragraph direction; at least it wasn't supposed to be
   that.  The code is (or was before your changes):

	 if (charpos < endpos)
	   {
	     if (pdir == L2R)
	       direction = QL2R;
	     else if (pdir == R2L)
	       direction = QR2L;
	     [...]
	     cmp_it->reversed_p = 0;
	   }
	 else
	   {
	     [...]
	     cmp_it->reversed_p = 1;
	     [...]
	     if (pdir == L2R)
	       direction = QR2L;
	     else if (pdir == R2L)
	       direction = QL2R;
	     [...]
	   }

   So, as you see, when the paragraph direction is L2R, normal text gets
   L2R direction, while test reversed for display gets R2L, and the other
   way around when the paragraph direction is R2L.  Which AFAIU is what
   HarfBuzz needs, but maybe I'm missing something.

   Did you actually see incorrect display with the code I wrote?  If so,
   could you please show the recipes for reproducing that, preferably
   with screenshots of correct and incorrect display?  I'd like to look
   into that, to understand what I missed.

   > HarfBuzz direction guessing should never be used (i.e.  always pass
   > to it an explicit direction).

   This is in general impossible (or at least very hard), since the
   shaper is sometimes called from Lisp without any display context.  See
   the Lisp callers of the function font-shape-gstring.  One use case is
   when we want to display the composition information for a grapheme
   cluster to the user, see descr-text.el (used by the "C-u C-x ="
   command).  In these cases, the UBA is not invoked, and so we don't
   have the direction information.

   I could provide the direction information in this case by using the
   directionality of the base character of the grapheme cluster, but I
   figured out that HarfBuzz already does this as part of its guessing.
   Doesn't it?

   > I pushed a couple of commits that does this based on my limited
   > understanding of Emacs code, please check.

   Thanks.  Do you see any difference in the results?  If so, can you
   please show the text you used and the results of shaping it with both
   versions.  AFAIU, your code should produce exactly the same results,
   unless I'm missing something.  (I didn't want to use the
   resolved_level attribute because it is ephemeral, and might not
   provide the correct value where we are using it.)

   Btw, did you test both paragraph directions (controlled by the
   bidi-paragraph-direction variable), and also text inside directional
   override which changes its natural direction?

Could you please respond and answer the few questions I asked?  I'd
like us to continue working on the branch.

TIA




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33729; Package emacs. Full text available.

Message received at 33729 <at> debbugs.gnu.org:


Received: (at 33729) by debbugs.gnu.org; 6 Jan 2019 17:54:52 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Jan 06 12:54:51 2019
Received: from localhost ([127.0.0.1]:48977 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1ggCdP-0002Wk-ME
	for submit <at> debbugs.gnu.org; Sun, 06 Jan 2019 12:54:51 -0500
Received: from eggsout.gnu.org ([209.51.188.92]:41930)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1ggCdL-0002Wb-QQ
 for 33729 <at> debbugs.gnu.org; Sun, 06 Jan 2019 12:54:48 -0500
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <eliz@HIDDEN>) id 1ggCdI-0001s1-4F
 for 33729 <at> debbugs.gnu.org; Sun, 06 Jan 2019 12:54:47 -0500
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=-0.0 required=5.0 tests=BAYES_40 autolearn=disabled
 version=3.3.2
Received: from fencepost.gnu.org ([2001:4830:134:3::e]:41702)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>)
 id 1ggCdH-0001rM-0P; Sun, 06 Jan 2019 12:54:43 -0500
Received: from [176.228.60.248] (port=4362 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1ggCdF-0001hm-Tw; Sun, 06 Jan 2019 12:54:42 -0500
Date: Sun, 06 Jan 2019 19:54:24 +0200
Message-Id: <83pnt9zn73.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Khaled Hosny <dr.khaled.hosny@HIDDEN>
In-reply-to: <20190105210420.GA31354@HIDDEN> (message from Khaled
 Hosny on Sat, 5 Jan 2019 23:04:20 +0200)
Subject: Re: bug#33729: 27.0.50; Partial glyphs not rendered for Gujarati
 with Harfbuzz enabled (renders fine using m17n)
References: <83bm5d9wsc.fsf@HIDDEN>
 <20181222205948.GF2244@HIDDEN>
 <838t0gapcj.fsf@HIDDEN>
 <20181223135109.GA6568@HIDDEN>
 <83va3k8c79.fsf@HIDDEN>
 <20181224020847.GC6568@HIDDEN>
 <83lg4e9a7q.fsf@HIDDEN>
 <20181224173723.GH6568@HIDDEN>
 <838t085qx8.fsf@HIDDEN>
 <20190105205314.GA28761@HIDDEN>
 <20190105210420.GA31354@HIDDEN>
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2001:4830:134:3::e
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 33729
Cc: behdad@HIDDEN, far.nasiri.m@HIDDEN, 33729 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

> Date: Sat, 5 Jan 2019 23:04:20 +0200
> From: Khaled Hosny <dr.khaled.hosny@HIDDEN>
> Cc: far.nasiri.m@HIDDEN, behdad@HIDDEN, 33729 <at> debbugs.gnu.org
> 
> I pushed a couple of commits that does this based on my limited
> understanding of Emacs code, please check.

Can you explain why you moved the call to
hb_buffer_guess_segment_properties _after_ the code which sets some of
the properties?  I cannot find anything about that in the HarfBuzz
documentation.  Is this because guessing the unset properties can
benefit from knowing the properties which _are_ set, such as the
direction?

I did it the other way around, because my mental model was: first set
the defaults, then override them where better info is available.

Thanks.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33729; Package emacs. Full text available.

Message received at 33729 <at> debbugs.gnu.org:


Received: (at 33729) by debbugs.gnu.org; 6 Jan 2019 16:04:25 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Jan 06 11:04:25 2019
Received: from localhost ([127.0.0.1]:48962 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1ggAuW-00087a-TB
	for submit <at> debbugs.gnu.org; Sun, 06 Jan 2019 11:04:25 -0500
Received: from eggs.gnu.org ([208.118.235.92]:47948)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1ggAuU-00087U-IW
 for 33729 <at> debbugs.gnu.org; Sun, 06 Jan 2019 11:04:23 -0500
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <eliz@HIDDEN>) id 1ggAuS-00024d-64
 for 33729 <at> debbugs.gnu.org; Sun, 06 Jan 2019 11:04:22 -0500
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50 autolearn=disabled
 version=3.3.2
Received: from fencepost.gnu.org ([2001:4830:134:3::e]:38821)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>)
 id 1ggAuK-0001qw-Oi; Sun, 06 Jan 2019 11:04:12 -0500
Received: from [176.228.60.248] (port=1170 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1ggAuJ-0002Rt-C9; Sun, 06 Jan 2019 11:04:11 -0500
Date: Sun, 06 Jan 2019 18:03:55 +0200
Message-Id: <83wonhzsb8.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Khaled Hosny <dr.khaled.hosny@HIDDEN>, Kenichi Handa <handa@HIDDEN>
In-reply-to: <20190105211514.GB28761@HIDDEN> (message from Khaled
 Hosny on Sat, 5 Jan 2019 23:15:14 +0200)
Subject: Re: bug#33729: 27.0.50; Partial glyphs not rendered for Gujarati
 with Harfbuzz enabled (renders fine using m17n)
References: <20181222154945.GE2244@HIDDEN>
 <83bm5d9wsc.fsf@HIDDEN>
 <20181222205948.GF2244@HIDDEN>
 <838t0gapcj.fsf@HIDDEN>
 <20181223135109.GA6568@HIDDEN>
 <83va3k8c79.fsf@HIDDEN>
 <20181224020847.GC6568@HIDDEN>
 <83lg4e9a7q.fsf@HIDDEN>
 <20181224173723.GH6568@HIDDEN>
 <83imzi94tz.fsf@HIDDEN> <20190105211514.GB28761@HIDDEN>
MIME-version: 1.0
Content-type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2001:4830:134:3::e
X-Spam-Score: -5.0 (-----)
X-Debbugs-Envelope-To: 33729
Cc: behdad@HIDDEN, far.nasiri.m@HIDDEN, 33729 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -6.0 (------)

> Date: Sat, 5 Jan 2019 23:15:14 +0200
> From: Khaled Hosny <dr.khaled.hosny@HIDDEN>
> Cc: rgm@HIDDEN, far.nasiri.m@HIDDEN, behdad@HIDDEN,
> 	33729 <at> debbugs.gnu.org, kaushal.modi@HIDDEN
> 
> > > The built-in HarfBuzz code is for getting the script for a given
> > > character, but resolving characters with Common script is left to the
> > > client. Suppose you have this string (upper case for RTL) ABC 123 DEF,
> > > what HarfBuzz sees during shaping is three separate chunks of text ABC,
> > > 123, DEF. The 123 part is all Common script characters and thus
> > > hb_buffer_guess_segment_properties won’t be able to guess anything (and
> > > based on the font and the script, this can cause rendering differences).
> > > Emacs will have to resolve the script of Common characters before
> > > applying bidi algorithm and pass that down to HarfBuzz.
> > 
> > I'm not sure I understand: why does HarfBuzz care that 123 was in the
> > middle if RTL text.
> 
> It doesn’t. What it cares about here is the correct script. Because 123
> are in the middle of RTL text they will be shaped separately, and thus
> hb_buffer_guess_segment_properties() will only see 123 and won’t to be
> able to guess the correct script for them (Arabic, Hebrew, etc.,
> whatever the script for the surrounding RTL text is).

That's what I was asking: why it's important for HarfBuzz to know that
123 should be shaped for the Arabic script?

> Depending on the font, the digits might be shaped differently if the
> script is, say Arabic, by e.g. applying script-specific substitutions to
> forms more suitable for a given script.

I guess this is what I'm missing, then: these script-specific
substitutions.  Can you elaborate on that, or point to some place
where these substitutions are described in detail?

> > (In general, AFAIK simple characters like 123 will not even go through
> > HarfBuzz, as Emacs doesn't call the shaper for characters whose entry
> > in composition-function-table is nil.  So I guess 123 here should
> > stand for some other characters, not for literal digits?  IOW, I don't
> > think I understand the example very well.)
> 
> This is a bug then and needs to be fixed. All text should go through
> HarfBuzz since even so-called “simple” character often require shaping
> depending on the text and the font. If this is done for optimization,
> then it should be revised to see if shaping with HarfBuzz is actually
> significantly slower and if it is, find more proper ways to optimize it.

(Adding Handa-san to the discussion, in the hope that he could comment
on the issue.)

I think running all text through a shaper might be prohibitively
expensive, because the shaper is called through Lisp code (see
composite.el), and we decide which chunk of text to pass to the shaper
using regexp search.  See the various files under lisp/language/ which
set up portions of composition-function-table as appropriate for each
language that needs it.

So I think we should identify all the cases where "simple" characters
surrounded by, or adjacent to, "non-simple" ones need to be passed to
a shaper, and add the necessary regular expressions to the data
structures in lisp/languages/.  Can you describe these cases, or point
me to a place where I can find the relevant info?

Thanks.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33729; Package emacs. Full text available.

Message received at 33729 <at> debbugs.gnu.org:


Received: (at 33729) by debbugs.gnu.org; 6 Jan 2019 15:51:18 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Jan 06 10:51:17 2019
Received: from localhost ([127.0.0.1]:48959 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1ggAhp-0007kt-Jg
	for submit <at> debbugs.gnu.org; Sun, 06 Jan 2019 10:51:17 -0500
Received: from eggsout.gnu.org ([209.51.188.92]:57199)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1ggAho-0007km-Nr
 for 33729 <at> debbugs.gnu.org; Sun, 06 Jan 2019 10:51:17 -0500
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <eliz@HIDDEN>) id 1ggAhi-00007v-T8
 for 33729 <at> debbugs.gnu.org; Sun, 06 Jan 2019 10:51:16 -0500
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50 autolearn=disabled
 version=3.3.2
Received: from fencepost.gnu.org ([2001:4830:134:3::e]:38546)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>)
 id 1ggAhi-00007m-Q1; Sun, 06 Jan 2019 10:51:10 -0500
Received: from [176.228.60.248] (port=4350 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1ggAhi-0003Lw-A4; Sun, 06 Jan 2019 10:51:10 -0500
Date: Sun, 06 Jan 2019 17:50:54 +0200
Message-Id: <83y37xzswx.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Khaled Hosny <dr.khaled.hosny@HIDDEN>
In-reply-to: <20190105205314.GA28761@HIDDEN> (message from Khaled
 Hosny on Sat, 5 Jan 2019 22:53:14 +0200)
Subject: Re: bug#33729: 27.0.50; Partial glyphs not rendered for Gujarati
 with Harfbuzz enabled (renders fine using m17n)
References: <20181222154945.GE2244@HIDDEN>
 <83bm5d9wsc.fsf@HIDDEN>
 <20181222205948.GF2244@HIDDEN>
 <838t0gapcj.fsf@HIDDEN>
 <20181223135109.GA6568@HIDDEN>
 <83va3k8c79.fsf@HIDDEN>
 <20181224020847.GC6568@HIDDEN>
 <83lg4e9a7q.fsf@HIDDEN>
 <20181224173723.GH6568@HIDDEN>
 <838t085qx8.fsf@HIDDEN> <20190105205314.GA28761@HIDDEN>
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2001:4830:134:3::e
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 33729
Cc: behdad@HIDDEN, far.nasiri.m@HIDDEN, 33729 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

> Date: Sat, 5 Jan 2019 22:53:14 +0200
> From: Khaled Hosny <dr.khaled.hosny@HIDDEN>
> Cc: far.nasiri.m@HIDDEN, behdad@HIDDEN, 33729 <at> debbugs.gnu.org
> 
> > Done.  Please test.  I made sure it compiles, but I couldn't actually
> > test the results, as I don't have access to a GNU/Linux system with
> > GUI display.  So it could be that I misunderstood the Harfbuzz APIs,
> > as I was essentially flying blind, guided only by the Harfbuzz docs.
> 
> It seems to work, but still not quite right. You seem to be passing the
> paragraph direction, but what HarfBuzz needs is resolved direction of
> the text (i.e. the bidi embedding level of the run).

It isn't the paragraph direction; at least it wasn't supposed to be
that.  The code is (or was before your changes):

      if (charpos < endpos)
	{
	  if (pdir == L2R)
	    direction = QL2R;
	  else if (pdir == R2L)
	    direction = QR2L;
	  [...]
	  cmp_it->reversed_p = 0;
	}
      else
	{
	  [...]
	  cmp_it->reversed_p = 1;
	  [...]
	  if (pdir == L2R)
	    direction = QR2L;
	  else if (pdir == R2L)
	    direction = QL2R;
	  [...]
	}

So, as you see, when the paragraph direction is L2R, normal text gets
L2R direction, while test reversed for display gets R2L, and the other
way around when the paragraph direction is R2L.  Which AFAIU is what
HarfBuzz needs, but maybe I'm missing something.

Did you actually see incorrect display with the code I wrote?  If so,
could you please show the recipes for reproducing that, preferably
with screenshots of correct and incorrect display?  I'd like to look
into that, to understand what I missed.

> HarfBuzz direction guessing should never be used (i.e.  always pass
> to it an explicit direction).

This is in general impossible (or at least very hard), since the
shaper is sometimes called from Lisp without any display context.  See
the Lisp callers of the function font-shape-gstring.  One use case is
when we want to display the composition information for a grapheme
cluster to the user, see descr-text.el (used by the "C-u C-x ="
command).  In these cases, the UBA is not invoked, and so we don't
have the direction information.

I could provide the direction information in this case by using the
directionality of the base character of the grapheme cluster, but I
figured out that HarfBuzz already does this as part of its guessing.
Doesn't it?

> I pushed a couple of commits that does this based on my limited
> understanding of Emacs code, please check.

Thanks.  Do you see any difference in the results?  If so, can you
please show the text you used and the results of shaping it with both
versions.  AFAIU, your code should produce exactly the same results,
unless I'm missing something.  (I didn't want to use the
resolved_level attribute because it is ephemeral, and might not
provide the correct value where we are using it.)

Btw, did you test both paragraph directions (controlled by the
bidi-paragraph-direction variable), and also text inside directional
override which changes its natural direction?




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33729; Package emacs. Full text available.

Message received at 33729 <at> debbugs.gnu.org:


Received: (at 33729) by debbugs.gnu.org; 5 Jan 2019 21:15:26 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Jan 05 16:15:26 2019
Received: from localhost ([127.0.0.1]:48094 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gftHx-0002PY-KT
	for submit <at> debbugs.gnu.org; Sat, 05 Jan 2019 16:15:25 -0500
Received: from mail-ed1-f45.google.com ([209.85.208.45]:37290)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dr.khaled.hosny@HIDDEN>) id 1gftHw-0002PK-EP
 for 33729 <at> debbugs.gnu.org; Sat, 05 Jan 2019 16:15:24 -0500
Received: by mail-ed1-f45.google.com with SMTP id h15so34636573edb.4
 for <33729 <at> debbugs.gnu.org>; Sat, 05 Jan 2019 13:15:24 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=date:from:to:cc:subject:message-id:references:mime-version
 :content-disposition:content-transfer-encoding:in-reply-to
 :user-agent; bh=AelQR6wUNPHqIZgJf54xZ43Kxa1MlKefsIXkKACjdI8=;
 b=l5geqCA/orlispZkOSC0D/LBQRYMr1iw2YRZFsDCrVWnsEgVJnE2QYUvN8I/p7M4a0
 0WapYVeYpT/xVm57XyhTzJrEgK8gZBTMdri447LHx1gEyTIKyB5uBxafpsJkvUJuiaiE
 fV/ZO6kGQ+kQbqe9rkHoU1WyG7CFj4rU/nVyCbJZuFdOatxpa51aPOGJKhUH2zhwPjSY
 KQFPqFd6TUAf8slCk4G2fsNplA1c23jk+6x5MrAXC2Cs/YJLFTvy8nGRjqDgBlMvEu/x
 dSftiarKU/9PKwYjRc6k+SdfdD2LCTVgr4T2D4ILxZKScLuSpWg0/riR3ZWuRmjT34Uz
 VGWg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:date:from:to:cc:subject:message-id:references
 :mime-version:content-disposition:content-transfer-encoding
 :in-reply-to:user-agent;
 bh=AelQR6wUNPHqIZgJf54xZ43Kxa1MlKefsIXkKACjdI8=;
 b=sDkU4xay2hzBSAJuAn93hlif4eeWQe0PJwKavzxO1r8BEiKG4s4YdQsGD6NrbC/1FM
 8gfXu+MiWahrohGhkzGHlOZGcHXrchPUsAwYOIevE0RB2cMBler34vWKdivDC+Ua24Tz
 ej1Yd19bB0cKSQh3ddoOpuD1crvwxRJCPuchM+fVw+fgSszX7GpLFRGaQhGXJqNOXBU2
 2xI1oe8KlU1H1E2/P/HFOruTmhRbKFL5dLWtgY5kPD3eIFcLxdrR8mR0nZoVoZm4mvLc
 8YnCgCgK9p+jbaaK34X09kDt/3IemaS98W2qD95EbzIIMTEWTwqsxwV+bKv8/0+V7iLA
 UxfA==
X-Gm-Message-State: AA+aEWa0uXyhl05RDdnC9g3bjVm9oxau/Y7SYzp/eO6WwSYU3+n48Pla
 BdszCB0Csi6j5K2mupgjkoo=
X-Google-Smtp-Source: AFSGD/WdWq/ON9zxUF/kCo5pxgF05GviqTI452/OEbv77o2wYz9A1RIsbSIc7tB2lvI/oLuu22Swcw==
X-Received: by 2002:a50:9724:: with SMTP id c33mr50788146edb.288.1546722918570; 
 Sat, 05 Jan 2019 13:15:18 -0800 (PST)
Received: from macbook.localdomain ([41.237.67.158])
 by smtp.gmail.com with ESMTPSA id t9sm27606596edd.25.2019.01.05.13.15.16
 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256);
 Sat, 05 Jan 2019 13:15:17 -0800 (PST)
Date: Sat, 5 Jan 2019 23:15:14 +0200
From: Khaled Hosny <dr.khaled.hosny@HIDDEN>
To: Eli Zaretskii <eliz@HIDDEN>
Subject: Re: bug#33729: 27.0.50; Partial glyphs not rendered for Gujarati
 with Harfbuzz enabled (renders fine using m17n)
Message-ID: <20190105211514.GB28761@HIDDEN>
References: <20181222154945.GE2244@HIDDEN>
 <83bm5d9wsc.fsf@HIDDEN>
 <20181222205948.GF2244@HIDDEN>
 <838t0gapcj.fsf@HIDDEN>
 <20181223135109.GA6568@HIDDEN>
 <83va3k8c79.fsf@HIDDEN>
 <20181224020847.GC6568@HIDDEN>
 <83lg4e9a7q.fsf@HIDDEN>
 <20181224173723.GH6568@HIDDEN>
 <83imzi94tz.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <83imzi94tz.fsf@HIDDEN>
User-Agent: Mutt/1.11.1 (2018-12-01)
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 33729
Cc: 33729 <at> debbugs.gnu.org, rgm@HIDDEN, kaushal.modi@HIDDEN,
 far.nasiri.m@HIDDEN, behdad@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

On Mon, Dec 24, 2018 at 08:07:04PM +0200, Eli Zaretskii wrote:
> > Date: Mon, 24 Dec 2018 19:37:23 +0200
> > From: Khaled Hosny <dr.khaled.hosny@HIDDEN>
> > Cc: rgm@HIDDEN, far.nasiri.m@HIDDEN, behdad@HIDDEN,
> > 	33729 <at> debbugs.gnu.org, kaushal.modi@HIDDEN
> > 
> > > Per previous discussions, we decided to use the Harfbuzz built-in
> > > methods for determining the script, since Emacs doesn't have this
> > > information, and adding it will just do the same as Harfbuzz does,
> > > i.e. find the first character whose script is not Common etc., using
> > > the UCD database.  I think it was you who suggested to use the
> > > Harfbuzz built-ins in this case.
> > 
> > The built-in HarfBuzz code is for getting the script for a given
> > character, but resolving characters with Common script is left to the
> > client. Suppose you have this string (upper case for RTL) ABC 123 DEF,
> > what HarfBuzz sees during shaping is three separate chunks of text ABC,
> > 123, DEF. The 123 part is all Common script characters and thus
> > hb_buffer_guess_segment_properties won’t be able to guess anything (and
> > based on the font and the script, this can cause rendering differences).
> > Emacs will have to resolve the script of Common characters before
> > applying bidi algorithm and pass that down to HarfBuzz.
> 
> I'm not sure I understand: why does HarfBuzz care that 123 was in the
> middle if RTL text.

It doesn’t. What it cares about here is the correct script. Because 123
are in the middle of RTL text they will be shaped separately, and thus
hb_buffer_guess_segment_properties() will only see 123 and won’t to be
able to guess the correct script for them (Arabic, Hebrew, etc.,
whatever the script for the surrounding RTL text is).

The point I’m trying to make is that script detection, even in its
simplest form, needs to be done on the text as a whole not just the
portion being shaped, which makes hb_buffer_guess_segment_properties()
ill equipped for doing this as it only sees a small portion of the text
at a time.

> Does it need to shape 123 specially in this case?

Depending on the font, the digits might be shaped differently if the
script is, say Arabic, by e.g. applying script-specific substitutions to
forms more suitable for a given script.
 
> (In general, AFAIK simple characters like 123 will not even go through
> HarfBuzz, as Emacs doesn't call the shaper for characters whose entry
> in composition-function-table is nil.  So I guess 123 here should
> stand for some other characters, not for literal digits?  IOW, I don't
> think I understand the example very well.)

This is a bug then and needs to be fixed. All text should go through
HarfBuzz since even so-called “simple” character often require shaping
depending on the text and the font. If this is done for optimization,
then it should be revised to see if shaping with HarfBuzz is actually
significantly slower and if it is, find more proper ways to optimize it.

Regards,
Khaled




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33729; Package emacs. Full text available.

Message received at 33729 <at> debbugs.gnu.org:


Received: (at 33729) by debbugs.gnu.org; 5 Jan 2019 21:04:32 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Jan 05 16:04:32 2019
Received: from localhost ([127.0.0.1]:48088 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gft7Q-00028a-GU
	for submit <at> debbugs.gnu.org; Sat, 05 Jan 2019 16:04:32 -0500
Received: from mail-ed1-f46.google.com ([209.85.208.46]:40023)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dr.khaled.hosny@HIDDEN>) id 1gft7O-00028M-PF
 for 33729 <at> debbugs.gnu.org; Sat, 05 Jan 2019 16:04:31 -0500
Received: by mail-ed1-f46.google.com with SMTP id g22so34593160edr.7
 for <33729 <at> debbugs.gnu.org>; Sat, 05 Jan 2019 13:04:30 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=date:from:to:cc:subject:message-id:references:mime-version
 :content-disposition:in-reply-to:user-agent;
 bh=uQBWwMNoXQXaUx2Yo2mSWIgduhZeZIo/BlHYfo7UUHs=;
 b=X0JTy63ZEhc+ol9dQNh0N98cZ1PGSc8AnqnaPZ0OQST5lMwqrluzyFokyfswx5t40P
 CE/zOKfoeYHLjIYwVtjYM3QQR0Lb1qVLX6UVM7wlWAIz/c/0eT/bsnjqdTtLU1n4QcUc
 gNeUQqsutK2zqv5mkbwwn1spyu7tjIT2EOsbjq9ibDYdlkdiUtRyhix/uEvIc+7BpzWh
 YMTu+Rlp0dxrljgmJq/RaxM06gfj6MB8igP6p0FwmENlUhyQ/Dmd7DkNQIoPjHd5442w
 eXihJMs/4tzwRskm1lItY9qX6p+ACAly0EdMUK8Kg1tB40xZ1sXOWy8JPecCcndlfuzX
 gdEg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:date:from:to:cc:subject:message-id:references
 :mime-version:content-disposition:in-reply-to:user-agent;
 bh=uQBWwMNoXQXaUx2Yo2mSWIgduhZeZIo/BlHYfo7UUHs=;
 b=GEFbCUUHPvGlpdJTYdF7nX9msKYVAs3Wr/SrWX4pxyc1zCkD39iEmZajLJ7jfTypxZ
 zez4FhF/YFnkc43rFiqOGVZNj5wA+DpU3raMP8FJ5/fGtsVsHztX2mG4St5aJFplQJt9
 XsOMc+PsL606y8DyXWPqJOQEVb46K6/pDjiGfNDjgawj4VO8oOj3nwxvGkcq2YfYSWue
 buf7Pn+wGmbSl1pl/t8TN9PTLkPIEhExYBeX+ux4PfoLHTnsxTROpmK+gIuIoYl/x9z1
 30AO96b6ziOizyYyKceAM2sovMP3A2bgZ5joOoerPC8hrb2FNfVPuB9tGGvFJHuhEM/d
 +Pww==
X-Gm-Message-State: AA+aEWZJret9jXoWhmy83ESnl9JMu+3Z5yFu6gBdypxbk24xkiYkrt19
 l280pGDTNisysEh67MdxaD8=
X-Google-Smtp-Source: AFSGD/WiVnPPvmZz2xj0uw0DCoVmVI88vlAh+L9uG30s2MLNHhjtCbzbtdXj7PulIZQOHC0W3OBVZA==
X-Received: by 2002:a50:880d:: with SMTP id b13mr49717222edb.68.1546722264968; 
 Sat, 05 Jan 2019 13:04:24 -0800 (PST)
Received: from macbook.localdomain ([41.237.67.158])
 by smtp.gmail.com with ESMTPSA id c23-v6sm16886587ejb.62.2019.01.05.13.04.23
 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256);
 Sat, 05 Jan 2019 13:04:24 -0800 (PST)
Date: Sat, 5 Jan 2019 23:04:20 +0200
From: Khaled Hosny <dr.khaled.hosny@HIDDEN>
To: Eli Zaretskii <eliz@HIDDEN>
Subject: Re: bug#33729: 27.0.50; Partial glyphs not rendered for Gujarati
 with Harfbuzz enabled (renders fine using m17n)
Message-ID: <20190105210420.GA31354@HIDDEN>
References: <83bm5d9wsc.fsf@HIDDEN>
 <20181222205948.GF2244@HIDDEN>
 <838t0gapcj.fsf@HIDDEN>
 <20181223135109.GA6568@HIDDEN>
 <83va3k8c79.fsf@HIDDEN>
 <20181224020847.GC6568@HIDDEN>
 <83lg4e9a7q.fsf@HIDDEN>
 <20181224173723.GH6568@HIDDEN>
 <838t085qx8.fsf@HIDDEN>
 <20190105205314.GA28761@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
In-Reply-To: <20190105205314.GA28761@HIDDEN>
User-Agent: Mutt/1.11.1 (2018-12-01)
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 33729
Cc: behdad@HIDDEN, far.nasiri.m@HIDDEN, 33729 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

On Sat, Jan 05, 2019 at 10:53:14PM +0200, Khaled Hosny wrote:
> On Sat, Dec 29, 2018 at 04:49:23PM +0200, Eli Zaretskii wrote:
> > > Date: Mon, 24 Dec 2018 19:37:23 +0200
> > > From: Khaled Hosny <dr.khaled.hosny@HIDDEN>
> > > Cc: rgm@HIDDEN, far.nasiri.m@HIDDEN, behdad@HIDDEN,
> > > 	33729 <at> debbugs.gnu.org, kaushal.modi@HIDDEN
> > > 
> > > > > We need to know, for a given lgstring we are shaping:
> > > > > * Its direction (from applying bidi algorithm). Each lgstring we are
> > > > >   shaping must be of a single direction.
> > > > 
> > > > Communicating this to ftfont_shape_by_hb will need changes in a couple
> > > > of interfaces (the existing shaping engines didn't need this
> > > > information).  I will work on this soon.
> > > 
> > > Great.
> > 
> > Done.  Please test.  I made sure it compiles, but I couldn't actually
> > test the results, as I don't have access to a GNU/Linux system with
> > GUI display.  So it could be that I misunderstood the Harfbuzz APIs,
> > as I was essentially flying blind, guided only by the Harfbuzz docs.
> 
> It seems to work, but still not quite right. You seem to be passing the
> paragraph direction, but what HarfBuzz needs is resolved direction of
> the text (i.e. the bidi embedding level of the run). In other words, if
> Emacs is going to draw this text from right to left, then HarfBuzz must
> shape it in right to left direction. Both should use the same direction
> all the time and HarfBuzz direction guessing should never be used (i.e.
> always pass to it an explicit direction).

I pushed a couple of commits that does this based on my limited
understanding of Emacs code, please check.

Regards,
Khaled




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33729; Package emacs. Full text available.

Message received at 33729 <at> debbugs.gnu.org:


Received: (at 33729) by debbugs.gnu.org; 5 Jan 2019 20:53:27 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Jan 05 15:53:27 2019
Received: from localhost ([127.0.0.1]:48085 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gfswh-0001qQ-D8
	for submit <at> debbugs.gnu.org; Sat, 05 Jan 2019 15:53:27 -0500
Received: from mail-ed1-f51.google.com ([209.85.208.51]:34022)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dr.khaled.hosny@HIDDEN>) id 1gfswe-0001qB-QB
 for 33729 <at> debbugs.gnu.org; Sat, 05 Jan 2019 15:53:25 -0500
Received: by mail-ed1-f51.google.com with SMTP id b3so34717762ede.1
 for <33729 <at> debbugs.gnu.org>; Sat, 05 Jan 2019 12:53:24 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=date:from:to:cc:subject:message-id:references:mime-version
 :content-disposition:in-reply-to:user-agent;
 bh=1mAV/ZoaEMJ5L5q3DxakmcuPtg7cHPy33uxHU0PLEqM=;
 b=HmkiNK8oLQtzsbO/8SaPbikTuLxWNaZZhYPQNShqBm8wHGQ+ruQ1rwHAYVH9+fSsvh
 koMXWMPGRGEd6a2FeaL9G3TC2rmeO56pYVoubqmPSIFHdRxmZhewEJU3JxDs5onYFhod
 JBoolcBYaB0+RbOWTssRG5Gh9WNtAue4/SglPnSF0tNhrXHApNNgPAYqE7fM1VimvxuG
 WlJxCadj5k3FH0sYnQRQbXvxDkJo7X4CPSiQz0m+SgrkSZWSkxYRInY14R6rtys/RZHm
 1rLxEVsSl8+NCdNwcstVJUms6H/nxJjggDH89o3Wsow8aX4V9xvvITNTjNA+NvqRW+ax
 Y3MQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:date:from:to:cc:subject:message-id:references
 :mime-version:content-disposition:in-reply-to:user-agent;
 bh=1mAV/ZoaEMJ5L5q3DxakmcuPtg7cHPy33uxHU0PLEqM=;
 b=FI8zUxt8OimVFsZesPVvPnH7k0tQhZ5ZGub+GM/SrNJ9roZSiIJaYi27rZ21PAYmBS
 z2c5sK1bnZ60/d6MKh53xnbPcERmHYmrzT7cfNJcJ/XZ7v/DoPbOgUobnY/mgcAAHVjC
 3a8NvGF1qteQINY0gHI8oUAygwU1U4ummjxyY8qOUixmfn/fxpsksACrtCQCRpwAhxKk
 wKD0muATLqGkOgX597T8WzFPjR0jlxSLlzOHMugd+igUGOpFJuozDCBw2QcKg8oZeo0t
 HOF+jPwxKSwaBotlh/Kiy4NgfOsGWQr7gX/HXw5BmgQWlYnV3i2X2ko3RHvoR6vij+m2
 crCw==
X-Gm-Message-State: AA+aEWYFfgqnBP/Bj95f50mw8AiZ1yoGm50vgpgzEPYxvXIeAzgawdWg
 PAsWbSmp9FyuKRgmh2yLbRM=
X-Google-Smtp-Source: AFSGD/XS++ozfv2IhzaKjP1fd76uFuf4Yem8uSNN81caLgd8wn64hfgBAALgGCK1fv/qn4QmG6VZ9g==
X-Received: by 2002:aa7:d684:: with SMTP id d4mr48434493edr.59.1546721598952; 
 Sat, 05 Jan 2019 12:53:18 -0800 (PST)
Received: from macbook.localdomain ([41.237.67.158])
 by smtp.gmail.com with ESMTPSA id j31sm27098306eda.46.2019.01.05.12.53.16
 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256);
 Sat, 05 Jan 2019 12:53:17 -0800 (PST)
Date: Sat, 5 Jan 2019 22:53:14 +0200
From: Khaled Hosny <dr.khaled.hosny@HIDDEN>
To: Eli Zaretskii <eliz@HIDDEN>
Subject: Re: bug#33729: 27.0.50; Partial glyphs not rendered for Gujarati
 with Harfbuzz enabled (renders fine using m17n)
Message-ID: <20190105205314.GA28761@HIDDEN>
References: <20181222154945.GE2244@HIDDEN>
 <83bm5d9wsc.fsf@HIDDEN>
 <20181222205948.GF2244@HIDDEN>
 <838t0gapcj.fsf@HIDDEN>
 <20181223135109.GA6568@HIDDEN>
 <83va3k8c79.fsf@HIDDEN>
 <20181224020847.GC6568@HIDDEN>
 <83lg4e9a7q.fsf@HIDDEN>
 <20181224173723.GH6568@HIDDEN>
 <838t085qx8.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
In-Reply-To: <838t085qx8.fsf@HIDDEN>
User-Agent: Mutt/1.11.1 (2018-12-01)
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 33729
Cc: behdad@HIDDEN, far.nasiri.m@HIDDEN, 33729 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

On Sat, Dec 29, 2018 at 04:49:23PM +0200, Eli Zaretskii wrote:
> > Date: Mon, 24 Dec 2018 19:37:23 +0200
> > From: Khaled Hosny <dr.khaled.hosny@HIDDEN>
> > Cc: rgm@HIDDEN, far.nasiri.m@HIDDEN, behdad@HIDDEN,
> > 	33729 <at> debbugs.gnu.org, kaushal.modi@HIDDEN
> > 
> > > > We need to know, for a given lgstring we are shaping:
> > > > * Its direction (from applying bidi algorithm). Each lgstring we are
> > > >   shaping must be of a single direction.
> > > 
> > > Communicating this to ftfont_shape_by_hb will need changes in a couple
> > > of interfaces (the existing shaping engines didn't need this
> > > information).  I will work on this soon.
> > 
> > Great.
> 
> Done.  Please test.  I made sure it compiles, but I couldn't actually
> test the results, as I don't have access to a GNU/Linux system with
> GUI display.  So it could be that I misunderstood the Harfbuzz APIs,
> as I was essentially flying blind, guided only by the Harfbuzz docs.

It seems to work, but still not quite right. You seem to be passing the
paragraph direction, but what HarfBuzz needs is resolved direction of
the text (i.e. the bidi embedding level of the run). In other words, if
Emacs is going to draw this text from right to left, then HarfBuzz must
shape it in right to left direction. Both should use the same direction
all the time and HarfBuzz direction guessing should never be used (i.e.
always pass to it an explicit direction).

Regards,
Khaled




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33729; Package emacs. Full text available.

Message received at 33729 <at> debbugs.gnu.org:


Received: (at 33729) by debbugs.gnu.org; 29 Dec 2018 14:49:52 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Dec 29 09:49:52 2018
Received: from localhost ([127.0.0.1]:40940 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gdFw0-0000yA-Cq
	for submit <at> debbugs.gnu.org; Sat, 29 Dec 2018 09:49:52 -0500
Received: from eggs.gnu.org ([208.118.235.92]:34081)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1gdFvz-0000xw-9G
 for 33729 <at> debbugs.gnu.org; Sat, 29 Dec 2018 09:49:51 -0500
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <eliz@HIDDEN>) id 1gdFvq-00085v-A9
 for 33729 <at> debbugs.gnu.org; Sat, 29 Dec 2018 09:49:45 -0500
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50 autolearn=disabled
 version=3.3.2
Received: from fencepost.gnu.org ([2001:4830:134:3::e]:48587)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>)
 id 1gdFvq-00085i-55; Sat, 29 Dec 2018 09:49:42 -0500
Received: from [176.228.60.248] (port=3863 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1gdFvp-0002Gl-PR; Sat, 29 Dec 2018 09:49:42 -0500
Date: Sat, 29 Dec 2018 16:49:23 +0200
Message-Id: <838t085qx8.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Khaled Hosny <dr.khaled.hosny@HIDDEN>
In-reply-to: <20181224173723.GH6568@HIDDEN> (message from Khaled
 Hosny on Mon, 24 Dec 2018 19:37:23 +0200)
Subject: Re: bug#33729: 27.0.50; Partial glyphs not rendered for Gujarati
 with Harfbuzz enabled (renders fine using m17n)
References: <20181222151509.GC2244@HIDDEN>
 <83h8f5a7po.fsf@HIDDEN>
 <20181222154945.GE2244@HIDDEN>
 <83bm5d9wsc.fsf@HIDDEN>
 <20181222205948.GF2244@HIDDEN>
 <838t0gapcj.fsf@HIDDEN>
 <20181223135109.GA6568@HIDDEN>
 <83va3k8c79.fsf@HIDDEN>
 <20181224020847.GC6568@HIDDEN>
 <83lg4e9a7q.fsf@HIDDEN> <20181224173723.GH6568@HIDDEN>
MIME-version: 1.0
Content-type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2001:4830:134:3::e
X-Spam-Score: -5.0 (-----)
X-Debbugs-Envelope-To: 33729
Cc: behdad@HIDDEN, far.nasiri.m@HIDDEN, 33729 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -6.0 (------)

> Date: Mon, 24 Dec 2018 19:37:23 +0200
> From: Khaled Hosny <dr.khaled.hosny@HIDDEN>
> Cc: rgm@HIDDEN, far.nasiri.m@HIDDEN, behdad@HIDDEN,
> 	33729 <at> debbugs.gnu.org, kaushal.modi@HIDDEN
> 
> On Mon, Dec 24, 2018 at 06:10:49PM +0200, Eli Zaretskii wrote:
> > > Date: Mon, 24 Dec 2018 04:08:47 +0200
> > > From: Khaled Hosny <dr.khaled.hosny@HIDDEN>
> > > Cc: rgm@HIDDEN, far.nasiri.m@HIDDEN, behdad@HIDDEN,
> > > 	33729 <at> debbugs.gnu.org, kaushal.modi@HIDDEN
> > > 
> > > I think we are almost good now. There is only one serious FIXME left:
> > > 
> > >   /* FIXME: guess_segment_properties is BAD BAD BAD.
> > >    * we need to get these properties with the LGSTRING. */
> > > #if 1
> > >   hb_buffer_guess_segment_properties (hb_buffer);
> > > #else
> > >   hb_buffer_set_direction (hb_buffer, XXX);
> > >   hb_buffer_set_script (hb_buffer, XXX);
> > >   hb_buffer_set_language (hb_buffer, XXX);
> > > #endif
> > > 
> > > We need to know, for a given lgstring we are shaping:
> > > * Its direction (from applying bidi algorithm). Each lgstring we are
> > >   shaping must be of a single direction.
> > 
> > Communicating this to ftfont_shape_by_hb will need changes in a couple
> > of interfaces (the existing shaping engines didn't need this
> > information).  I will work on this soon.
> 
> Great.

Done.  Please test.  I made sure it compiles, but I couldn't actually
test the results, as I don't have access to a GNU/Linux system with
GUI display.  So it could be that I misunderstood the Harfbuzz APIs,
as I was essentially flying blind, guided only by the Harfbuzz docs.

In particularly, I hope I understood correctly the way we should leave
to Harfbuzz guess the properties not explicitly provided by the Emacs
context, both for the direction of the text and its script.

> > > * Its script, possibly after applying something like:
> > >   http://unicode.org/reports/tr24/#Common
> > 
> > Per previous discussions, we decided to use the Harfbuzz built-in
> > methods for determining the script, since Emacs doesn't have this
> > information, and adding it will just do the same as Harfbuzz does,
> > i.e. find the first character whose script is not Common etc., using
> > the UCD database.  I think it was you who suggested to use the
> > Harfbuzz built-ins in this case.
> 
> The built-in HarfBuzz code is for getting the script for a given
> character, but resolving characters with Common script is left to the
> client. Suppose you have this string (upper case for RTL) ABC 123 DEF,
> what HarfBuzz sees during shaping is three separate chunks of text ABC,
> 123, DEF. The 123 part is all Common script characters and thus
> hb_buffer_guess_segment_properties won’t be able to guess anything (and
> based on the font and the script, this can cause rendering differences).
> Emacs will have to resolve the script of Common characters before
> applying bidi algorithm and pass that down to HarfBuzz.

See my followup questions about this.  For now, I left this aspect to
HarfBuzz.

> > > * Its language, is Emacs allows setting text language (my understand is
> > >   that it doesn’t). Some languages really need this for applying
> > >   language-specfic features (Urdu digits, Serbian alternate glyphs, etc.).
> > 
> > We don't currently have a language property for chunks of text, we
> > only have the current global language setting determined from the
> > locale (and there's a command to change that for Emacs, should the
> > user want it).  This is not really appropriate for multilingual
> > buffers, but we will have to use that for now, and hope that in the
> > future, infrastructure will be added to allow more flexible
> > determination of the language of each run of text.  (I see that
> > Harfbuzz already looks a the locale for its default language, but
> > since Emacs allows user control of this, however unlikely, I think
> > it's best to use the value Emacs uses.)  I will work on this as well.
> 
> Yes, better pass that from Emacs to HarfBuzz.

Done, but please see the FIXME I left behind.  For testing purposes,
you can change the current language like this:

  M-x set-locale-environment RET xx_YY.CODESET RET

For example:

  M-x set-locale-environment RET sr_RS.UTF-8 RET

for the Cyrillic Serbian locale.  This should change the value of
current-iso639-language to the symbol 'sr'.

Please tell if you encounter any difficulties with the code I added,
or if you need any further help.

Thanks.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33729; Package emacs. Full text available.

Message received at 33729 <at> debbugs.gnu.org:


Received: (at 33729) by debbugs.gnu.org; 24 Dec 2018 18:07:26 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Dec 24 13:07:26 2018
Received: from localhost ([127.0.0.1]:35614 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gbUdR-0002HP-Vn
	for submit <at> debbugs.gnu.org; Mon, 24 Dec 2018 13:07:26 -0500
Received: from eggs.gnu.org ([208.118.235.92]:49911)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1gbUdQ-0002HD-Bf
 for 33729 <at> debbugs.gnu.org; Mon, 24 Dec 2018 13:07:24 -0500
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <eliz@HIDDEN>) id 1gbUdK-00024A-6o
 for 33729 <at> debbugs.gnu.org; Mon, 24 Dec 2018 13:07:18 -0500
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50 autolearn=disabled
 version=3.3.2
Received: from fencepost.gnu.org ([2001:4830:134:3::e]:35505)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>)
 id 1gbUdG-00021m-6e; Mon, 24 Dec 2018 13:07:14 -0500
Received: from [176.228.60.248] (port=3591 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1gbUdF-0006l0-Ml; Mon, 24 Dec 2018 13:07:14 -0500
Date: Mon, 24 Dec 2018 20:07:04 +0200
Message-Id: <83imzi94tz.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Khaled Hosny <dr.khaled.hosny@HIDDEN>
In-reply-to: <20181224173723.GH6568@HIDDEN> (message from Khaled
 Hosny on Mon, 24 Dec 2018 19:37:23 +0200)
Subject: Re: bug#33729: 27.0.50; Partial glyphs not rendered for Gujarati
 with Harfbuzz enabled (renders fine using m17n)
References: <20181222151509.GC2244@HIDDEN>
 <83h8f5a7po.fsf@HIDDEN>
 <20181222154945.GE2244@HIDDEN>
 <83bm5d9wsc.fsf@HIDDEN>
 <20181222205948.GF2244@HIDDEN>
 <838t0gapcj.fsf@HIDDEN>
 <20181223135109.GA6568@HIDDEN>
 <83va3k8c79.fsf@HIDDEN>
 <20181224020847.GC6568@HIDDEN>
 <83lg4e9a7q.fsf@HIDDEN> <20181224173723.GH6568@HIDDEN>
MIME-version: 1.0
Content-type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2001:4830:134:3::e
X-Spam-Score: -5.0 (-----)
X-Debbugs-Envelope-To: 33729
Cc: 33729 <at> debbugs.gnu.org, rgm@HIDDEN, kaushal.modi@HIDDEN,
 far.nasiri.m@HIDDEN, behdad@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -6.0 (------)

> Date: Mon, 24 Dec 2018 19:37:23 +0200
> From: Khaled Hosny <dr.khaled.hosny@HIDDEN>
> Cc: rgm@HIDDEN, far.nasiri.m@HIDDEN, behdad@HIDDEN,
> 	33729 <at> debbugs.gnu.org, kaushal.modi@HIDDEN
> 
> > Per previous discussions, we decided to use the Harfbuzz built-in
> > methods for determining the script, since Emacs doesn't have this
> > information, and adding it will just do the same as Harfbuzz does,
> > i.e. find the first character whose script is not Common etc., using
> > the UCD database.  I think it was you who suggested to use the
> > Harfbuzz built-ins in this case.
> 
> The built-in HarfBuzz code is for getting the script for a given
> character, but resolving characters with Common script is left to the
> client. Suppose you have this string (upper case for RTL) ABC 123 DEF,
> what HarfBuzz sees during shaping is three separate chunks of text ABC,
> 123, DEF. The 123 part is all Common script characters and thus
> hb_buffer_guess_segment_properties won’t be able to guess anything (and
> based on the font and the script, this can cause rendering differences).
> Emacs will have to resolve the script of Common characters before
> applying bidi algorithm and pass that down to HarfBuzz.

I'm not sure I understand: why does HarfBuzz care that 123 was in the
middle if RTL text.  Does it need to shape 123 specially in this case?

(In general, AFAIK simple characters like 123 will not even go through
HarfBuzz, as Emacs doesn't call the shaper for characters whose entry
in composition-function-table is nil.  So I guess 123 here should
stand for some other characters, not for literal digits?  IOW, I don't
think I understand the example very well.)




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33729; Package emacs. Full text available.

Message received at 33729 <at> debbugs.gnu.org:


Received: (at 33729) by debbugs.gnu.org; 24 Dec 2018 17:38:56 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Dec 24 12:38:56 2018
Received: from localhost ([127.0.0.1]:35605 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gbUBs-0001Pw-C9
	for submit <at> debbugs.gnu.org; Mon, 24 Dec 2018 12:38:56 -0500
Received: from odoacer.turtle-trading.net ([93.241.193.16]:46243)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <benny@HIDDEN>) id 1gbUBq-0001Pi-K3
 for 33729 <at> debbugs.gnu.org; Mon, 24 Dec 2018 12:38:55 -0500
Received: from justinian.turtle-trading.net ([192.168.2.118])
 by odoacer.turtle-trading.net with esmtp (Exim 4.80)
 (envelope-from <benny@HIDDEN>)
 id 1gbUBj-0004P5-7q; Mon, 24 Dec 2018 18:38:47 +0100
Received: from benny by justinian.turtle-trading.net with local (Exim 4.89)
 (envelope-from <benny@HIDDEN>)
 id 1gbUBi-0003FN-TY; Mon, 24 Dec 2018 18:38:46 +0100
From: Benjamin Riefenstahl <b.riefenstahl@HIDDEN>
To: Khaled Hosny <dr.khaled.hosny@HIDDEN>
Subject: Re: bug#33729: 27.0.50;
 Partial glyphs not rendered for Gujarati with Harfbuzz enabled
 (renders fine using m17n)
References: <CAFyQvY0pznh0yM9XJ9y_cEft47SXCmFp+O5rLkzb6o0By-MQuA@HIDDEN>
 <83h8fghcpo.fsf@HIDDEN> <20181214075056.GI2244@HIDDEN>
 <8336r0h1cb.fsf@HIDDEN> <20181214110316.GK2244@HIDDEN>
 <83y38sfcme.fsf@HIDDEN> <83tvjgf7ux.fsf@HIDDEN>
 <xk4lbd9erb.fsf@HIDDEN> <83mup4du5z.fsf@HIDDEN>
 <20181222085448.GA2244@HIDDEN>
 <20181222090644.GB2244@HIDDEN>
Date: Mon, 24 Dec 2018 18:38:46 +0100
In-Reply-To: <20181222090644.GB2244@HIDDEN> (Khaled Hosny's
 message of "Sat, 22 Dec 2018 11:06:44 +0200")
Message-ID: <874lb2j049.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 33729
Cc: Glenn Morris <rgm@HIDDEN>, far.nasiri.m@HIDDEN, behdad@HIDDEN,
 33729 <at> debbugs.gnu.org, kaushal.modi@HIDDEN, Eli Zaretskii <eliz@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Khaled Hosny writes:
> I pushed a commit to harfbuzz branch that I think fixes this issue now.

I can confirm that this fixes the issue with Syriac.

Thanks,
benny




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33729; Package emacs. Full text available.

Message received at 33729 <at> debbugs.gnu.org:


Received: (at 33729) by debbugs.gnu.org; 24 Dec 2018 17:37:35 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Dec 24 12:37:35 2018
Received: from localhost ([127.0.0.1]:35601 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gbUAY-0001Nw-Ri
	for submit <at> debbugs.gnu.org; Mon, 24 Dec 2018 12:37:35 -0500
Received: from mail-wr1-f50.google.com ([209.85.221.50]:34310)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dr.khaled.hosny@HIDDEN>) id 1gbUAX-0001Ni-A7
 for 33729 <at> debbugs.gnu.org; Mon, 24 Dec 2018 12:37:33 -0500
Received: by mail-wr1-f50.google.com with SMTP id j2so12126499wrw.1
 for <33729 <at> debbugs.gnu.org>; Mon, 24 Dec 2018 09:37:33 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=date:from:to:cc:subject:message-id:references:mime-version
 :content-disposition:content-transfer-encoding:in-reply-to
 :user-agent; bh=Gana4c+e6HGNisJXyoKZtAkhPEAXnBq/NCm0M7y9dF0=;
 b=DyUZ/k5C5UThWt0BWaHJ+0/mY5j7jqe9wbRtUdKErJZALC9I+ZwhimHzdnoVIRezm8
 Us8WjhZDEX957vZ/KhzC23CD5hjAgQAAP/9PydvZizNzkrtQc878bGRXWLWxOKIx2wc6
 L0yeJ9CyDJGDeWjuZPubf5MpmIPgxqYN7CtsYkokFcA8Ql1L4mgnP+LB6Uf/Ykj0RB7b
 BtFQkJe3IcwjlFT67uVOWHvEhgtR6S5aSJ89Z6Oo680kKd+co0l9icdQHbA5W3EPz//6
 pi6pZdU9E+N7t7OPNWOakHC5+XzoDs4xn57uDVilQvtOtDqVDJVFgubv+ukY0on+uTL1
 PWcQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:date:from:to:cc:subject:message-id:references
 :mime-version:content-disposition:content-transfer-encoding
 :in-reply-to:user-agent;
 bh=Gana4c+e6HGNisJXyoKZtAkhPEAXnBq/NCm0M7y9dF0=;
 b=Okk+e3icq7VEarMkse7U6cGG3+MMRuHaZv8cB9dArP6hjVWks9+15bb9kgUkXRLO7p
 VYmZf3z+TUwnNOQQ2TstkuuMXNeT8Nyhv92zKw54+ZVgqDydihhpIwb9vUODAKtiQoZZ
 FInHZsiHSZERr4H5IhQuw48iGHQLvxPkaXDgq+N4Jto8kS7nGVy8qEcz2MYoAyBQsraH
 u7O8b5sqX5vuXeMcf+KwIjUrvDxAvd9oXUmmwRmTVlztGnFH6STGdS+fwHl4wC6O+A26
 q8xlJ8kKQNn5vH3NkuhiSdTsxFBbUKsVgjkOlRfMa0RbSyI1vA0nx5ZTJz7BKuq2Hq55
 rJNw==
X-Gm-Message-State: AJcUukdXPA3BB1x3l+KiaogL66V3DG4D4ZluVwB6tOD40BYT4CQlvGgV
 rs+mvyU8vuttJ1R0OUejNfU=
X-Google-Smtp-Source: ALg8bN5WEdfwyLaBA5FUy4nvqKraDwg73YonM4xaKazRA5CFuWmPI4xAAKTf6j0F8zH9y/Etc12IkQ==
X-Received: by 2002:a5d:694d:: with SMTP id r13mr12059510wrw.323.1545673047517; 
 Mon, 24 Dec 2018 09:37:27 -0800 (PST)
Received: from macbook.localdomain ([41.237.113.27])
 by smtp.gmail.com with ESMTPSA id t63sm16609962wmt.8.2018.12.24.09.37.25
 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256);
 Mon, 24 Dec 2018 09:37:26 -0800 (PST)
Date: Mon, 24 Dec 2018 19:37:23 +0200
From: Khaled Hosny <dr.khaled.hosny@HIDDEN>
To: Eli Zaretskii <eliz@HIDDEN>
Subject: Re: bug#33729: 27.0.50; Partial glyphs not rendered for Gujarati
 with Harfbuzz enabled (renders fine using m17n)
Message-ID: <20181224173723.GH6568@HIDDEN>
References: <20181222151509.GC2244@HIDDEN>
 <83h8f5a7po.fsf@HIDDEN>
 <20181222154945.GE2244@HIDDEN>
 <83bm5d9wsc.fsf@HIDDEN>
 <20181222205948.GF2244@HIDDEN>
 <838t0gapcj.fsf@HIDDEN>
 <20181223135109.GA6568@HIDDEN>
 <83va3k8c79.fsf@HIDDEN>
 <20181224020847.GC6568@HIDDEN>
 <83lg4e9a7q.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <83lg4e9a7q.fsf@HIDDEN>
User-Agent: Mutt/1.11.1 (2018-12-01)
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 33729
Cc: 33729 <at> debbugs.gnu.org, rgm@HIDDEN, kaushal.modi@HIDDEN,
 far.nasiri.m@HIDDEN, behdad@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

On Mon, Dec 24, 2018 at 06:10:49PM +0200, Eli Zaretskii wrote:
> > Date: Mon, 24 Dec 2018 04:08:47 +0200
> > From: Khaled Hosny <dr.khaled.hosny@HIDDEN>
> > Cc: rgm@HIDDEN, far.nasiri.m@HIDDEN, behdad@HIDDEN,
> > 	33729 <at> debbugs.gnu.org, kaushal.modi@HIDDEN
> > 
> > I think we are almost good now. There is only one serious FIXME left:
> > 
> >   /* FIXME: guess_segment_properties is BAD BAD BAD.
> >    * we need to get these properties with the LGSTRING. */
> > #if 1
> >   hb_buffer_guess_segment_properties (hb_buffer);
> > #else
> >   hb_buffer_set_direction (hb_buffer, XXX);
> >   hb_buffer_set_script (hb_buffer, XXX);
> >   hb_buffer_set_language (hb_buffer, XXX);
> > #endif
> > 
> > We need to know, for a given lgstring we are shaping:
> > * Its direction (from applying bidi algorithm). Each lgstring we are
> >   shaping must be of a single direction.
> 
> Communicating this to ftfont_shape_by_hb will need changes in a couple
> of interfaces (the existing shaping engines didn't need this
> information).  I will work on this soon.

Great.

> > * Its script, possibly after applying something like:
> >   http://unicode.org/reports/tr24/#Common
> 
> Per previous discussions, we decided to use the Harfbuzz built-in
> methods for determining the script, since Emacs doesn't have this
> information, and adding it will just do the same as Harfbuzz does,
> i.e. find the first character whose script is not Common etc., using
> the UCD database.  I think it was you who suggested to use the
> Harfbuzz built-ins in this case.

The built-in HarfBuzz code is for getting the script for a given
character, but resolving characters with Common script is left to the
client. Suppose you have this string (upper case for RTL) ABC 123 DEF,
what HarfBuzz sees during shaping is three separate chunks of text ABC,
123, DEF. The 123 part is all Common script characters and thus
hb_buffer_guess_segment_properties won’t be able to guess anything (and
based on the font and the script, this can cause rendering differences).
Emacs will have to resolve the script of Common characters before
applying bidi algorithm and pass that down to HarfBuzz.

> > * Its language, is Emacs allows setting text language (my understand is
> >   that it doesn’t). Some languages really need this for applying
> >   language-specfic features (Urdu digits, Serbian alternate glyphs, etc.).
> 
> We don't currently have a language property for chunks of text, we
> only have the current global language setting determined from the
> locale (and there's a command to change that for Emacs, should the
> user want it).  This is not really appropriate for multilingual
> buffers, but we will have to use that for now, and hope that in the
> future, infrastructure will be added to allow more flexible
> determination of the language of each run of text.  (I see that
> Harfbuzz already looks a the locale for its default language, but
> since Emacs allows user control of this, however unlikely, I think
> it's best to use the value Emacs uses.)  I will work on this as well.

Yes, better pass that from Emacs to HarfBuzz.

Regards,
Khaled




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33729; Package emacs. Full text available.

Message received at 33729 <at> debbugs.gnu.org:


Received: (at 33729) by debbugs.gnu.org; 24 Dec 2018 16:11:12 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Dec 24 11:11:12 2018
Received: from localhost ([127.0.0.1]:35576 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gbSox-0007hw-RK
	for submit <at> debbugs.gnu.org; Mon, 24 Dec 2018 11:11:12 -0500
Received: from eggs.gnu.org ([208.118.235.92]:46648)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1gbSow-0007hi-HF
 for 33729 <at> debbugs.gnu.org; Mon, 24 Dec 2018 11:11:10 -0500
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <eliz@HIDDEN>) id 1gbSoq-0004k6-D1
 for 33729 <at> debbugs.gnu.org; Mon, 24 Dec 2018 11:11:05 -0500
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50 autolearn=disabled
 version=3.3.2
Received: from fencepost.gnu.org ([2001:4830:134:3::e]:33913)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>)
 id 1gbSol-0004g0-Pw; Mon, 24 Dec 2018 11:10:59 -0500
Received: from [176.228.60.248] (port=4294 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1gbSol-0005Pf-B7; Mon, 24 Dec 2018 11:10:59 -0500
Date: Mon, 24 Dec 2018 18:10:49 +0200
Message-Id: <83lg4e9a7q.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Khaled Hosny <dr.khaled.hosny@HIDDEN>
In-reply-to: <20181224020847.GC6568@HIDDEN> (message from Khaled
 Hosny on Mon, 24 Dec 2018 04:08:47 +0200)
Subject: Re: bug#33729: 27.0.50; Partial glyphs not rendered for Gujarati
 with Harfbuzz enabled (renders fine using m17n)
References: <20181222090644.GB2244@HIDDEN>
 <83va3lan24.fsf@HIDDEN>
 <20181222151509.GC2244@HIDDEN>
 <83h8f5a7po.fsf@HIDDEN>
 <20181222154945.GE2244@HIDDEN>
 <83bm5d9wsc.fsf@HIDDEN>
 <20181222205948.GF2244@HIDDEN>
 <838t0gapcj.fsf@HIDDEN>
 <20181223135109.GA6568@HIDDEN>
 <83va3k8c79.fsf@HIDDEN> <20181224020847.GC6568@HIDDEN>
MIME-version: 1.0
Content-type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2001:4830:134:3::e
X-Spam-Score: -5.0 (-----)
X-Debbugs-Envelope-To: 33729
Cc: 33729 <at> debbugs.gnu.org, rgm@HIDDEN, kaushal.modi@HIDDEN,
 far.nasiri.m@HIDDEN, behdad@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -6.0 (------)

> Date: Mon, 24 Dec 2018 04:08:47 +0200
> From: Khaled Hosny <dr.khaled.hosny@HIDDEN>
> Cc: rgm@HIDDEN, far.nasiri.m@HIDDEN, behdad@HIDDEN,
> 	33729 <at> debbugs.gnu.org, kaushal.modi@HIDDEN
> 
> I think we are almost good now. There is only one serious FIXME left:
> 
>   /* FIXME: guess_segment_properties is BAD BAD BAD.
>    * we need to get these properties with the LGSTRING. */
> #if 1
>   hb_buffer_guess_segment_properties (hb_buffer);
> #else
>   hb_buffer_set_direction (hb_buffer, XXX);
>   hb_buffer_set_script (hb_buffer, XXX);
>   hb_buffer_set_language (hb_buffer, XXX);
> #endif
> 
> We need to know, for a given lgstring we are shaping:
> * Its direction (from applying bidi algorithm). Each lgstring we are
>   shaping must be of a single direction.

Communicating this to ftfont_shape_by_hb will need changes in a couple
of interfaces (the existing shaping engines didn't need this
information).  I will work on this soon.

> * Its script, possibly after applying something like:
>   http://unicode.org/reports/tr24/#Common

Per previous discussions, we decided to use the Harfbuzz built-in
methods for determining the script, since Emacs doesn't have this
information, and adding it will just do the same as Harfbuzz does,
i.e. find the first character whose script is not Common etc., using
the UCD database.  I think it was you who suggested to use the
Harfbuzz built-ins in this case.

> * Its language, is Emacs allows setting text language (my understand is
>   that it doesn’t). Some languages really need this for applying
>   language-specfic features (Urdu digits, Serbian alternate glyphs, etc.).

We don't currently have a language property for chunks of text, we
only have the current global language setting determined from the
locale (and there's a command to change that for Emacs, should the
user want it).  This is not really appropriate for multilingual
buffers, but we will have to use that for now, and hope that in the
future, infrastructure will be added to allow more flexible
determination of the language of each run of text.  (I see that
Harfbuzz already looks a the locale for its default language, but
since Emacs allows user control of this, however unlikely, I think
it's best to use the value Emacs uses.)  I will work on this as well.

Thanks.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33729; Package emacs. Full text available.

Message received at 33729 <at> debbugs.gnu.org:


Received: (at 33729) by debbugs.gnu.org; 24 Dec 2018 04:13:45 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Dec 23 23:13:45 2018
Received: from localhost ([127.0.0.1]:34192 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gbHcb-0006Ec-DM
	for submit <at> debbugs.gnu.org; Sun, 23 Dec 2018 23:13:45 -0500
Received: from mail-lj1-f177.google.com ([209.85.208.177]:34350)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <kaushal.modi@HIDDEN>) id 1gbHcV-0006EJ-Cv
 for 33729 <at> debbugs.gnu.org; Sun, 23 Dec 2018 23:13:39 -0500
Received: by mail-lj1-f177.google.com with SMTP id u89-v6so9487643lje.1
 for <33729 <at> debbugs.gnu.org>; Sun, 23 Dec 2018 20:13:35 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=mime-version:references:in-reply-to:from:date:message-id:subject:to
 :cc; bh=dhqe8jZgTCOn+XB5CqJhnYG3zn3alCqofZDga6aa8AM=;
 b=Bm5e3RUpPRnaWocijGpq3y04GMOQCMek6Vb7lR/DqTS3phmYUeRYnLhEV3NvbeHIG8
 Kvrlgy/+yi5i75rum8dBDXqUGHWJ1g6KpTUndRU97O0zCEglSzHBPzytJimQ2PsSITR8
 bCrUxxacbe46XTI7tYkT1hRw3AICQZPBSOl0aYzYVQHvPwqsQ9EsKsUvuK7B/BHl+r8F
 I/tIkO6aV4i9IAY2OHpL5nfzWxcdJnSryFDweIWyxT2AY2cdgYDlfeUDEHOAaW2VLCGn
 mqTCUK/wQiMkgUW3gGHJgGxrFO6LrQdra8jG6JwH22e8oKkOlV3T9lkzLV1lIPN2Bk8o
 PKTA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:mime-version:references:in-reply-to:from:date
 :message-id:subject:to:cc;
 bh=dhqe8jZgTCOn+XB5CqJhnYG3zn3alCqofZDga6aa8AM=;
 b=pHzq18VeGdnUm5xRKkNfdBuI8VxL3v6uZGMFMWVUmhQDn0jcGX8z1Hob9vz85BU+Q3
 MC9LTyRhmVXa/HLVuI5AdAjyaN2EvpAQEgpIuzKNqq3J+fo6OeWbo3aQlpcRIUuN0irP
 jR+8s6kpjKOuTPg5+642sXJgRbHH/RV2iuxHDegvHR8PditJ2T9vkBM0IZfeZ1qe3FHd
 eKeckrG2iyyhHDPzYAmM6ara7Dd5gdiv+aHcP5Xbu8l00LfLLlvmaa63bSTObk0w3hJ+
 BQrFkmSqjT1nGxJJWM1gtqGAiGO9nE5cB7Q72Bur+ZA+qqjDc0kWagNt+9izyFndR/EZ
 PD2w==
X-Gm-Message-State: AJcUukcdEfDoYBUMeZEEJ+0ZxBXGIOEbWrZvLEQbFFHxLnlVw8VlPppW
 uUL0rGsIVr7nQHsgMXFJbCi2QR3tnUMoNdkRLjM=
X-Google-Smtp-Source: ALg8bN7EDZdrVmy2SPC9VsoFDr0aFNq5vseXTI//bta8mP64eAnxadO6IFq7NFNbWPc3ku7Ktxl5wyi/ktHDG9pQSHg=
X-Received: by 2002:a2e:1f01:: with SMTP id
 f1-v6mr5870496ljf.129.1545624809332; 
 Sun, 23 Dec 2018 20:13:29 -0800 (PST)
MIME-Version: 1.0
References: <20181222090644.GB2244@HIDDEN>
 <83va3lan24.fsf@HIDDEN>
 <20181222151509.GC2244@HIDDEN> <83h8f5a7po.fsf@HIDDEN>
 <20181222154945.GE2244@HIDDEN> <83bm5d9wsc.fsf@HIDDEN>
 <20181222205948.GF2244@HIDDEN> <838t0gapcj.fsf@HIDDEN>
 <20181223135109.GA6568@HIDDEN> <83va3k8c79.fsf@HIDDEN>
 <20181224020847.GC6568@HIDDEN>
In-Reply-To: <20181224020847.GC6568@HIDDEN>
From: Kaushal Modi <kaushal.modi@HIDDEN>
Date: Sun, 23 Dec 2018 23:12:52 -0500
Message-ID: <CAFyQvY3dcjhOqFcYER7GdQ7z+Vc=rHX8o_moM9ZztvjuKZSwmA@HIDDEN>
Subject: Re: bug#33729: 27.0.50; Partial glyphs not rendered for Gujarati with
 Harfbuzz enabled (renders fine using m17n)
To: Khaled Hosny <dr.khaled.hosny@HIDDEN>
Content-Type: multipart/alternative; boundary="000000000000b76e6e057dbcd14d"
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 33729
Cc: 33729 <at> debbugs.gnu.org, Glenn Morris <rgm@HIDDEN>,
 Eli Zaretskii <eliz@HIDDEN>, Mohammad Nasirifar <far.nasiri.m@HIDDEN>,
 Behdad Esfahbod <behdad@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

--000000000000b76e6e057dbcd14d
Content-Type: text/plain; charset="UTF-8"

On Sun, Dec 23, 2018 at 9:08 PM Khaled Hosny <dr.khaled.hosny@HIDDEN>
wrote:

> I think we are almost good now.
>

Thanks for working on this!

I confirm that this particular issue related to rendering compound glyphs
in Gujarati script is fixed.

Proof: https://i.imgtc.com/XYSM5fE.png


@Eli I see that discussion related to this fix is on-going. So feel free to
mark this bug as DONE when you see fit.

Thanks all!

--000000000000b76e6e057dbcd14d
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div dir=3D"ltr"><br><div class=3D"gmail_quote"><div dir=
=3D"ltr">On Sun, Dec 23, 2018 at 9:08 PM Khaled Hosny &lt;<a href=3D"mailto=
:dr.khaled.hosny@HIDDEN">dr.khaled.hosny@HIDDEN</a>&gt; wrote:<br></d=
iv><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;bord=
er-left:1px solid rgb(204,204,204);padding-left:1ex">I think we are almost =
good now. <br></blockquote><div><br></div><div>Thanks for working on this!<=
/div><div><br></div><div>I confirm that this particular issue related to re=
ndering compound glyphs in Gujarati script is fixed.</div><div><br></div><d=
iv>Proof: <a href=3D"https://i.imgtc.com/XYSM5fE.png">https://i.imgtc.com/X=
YSM5fE.png</a></div><div><br></div><br><div>@Eli I see that discussion rela=
ted to this fix is on-going. So feel free to mark this bug as DONE when you=
 see fit.</div><div><br></div><div>
<div>Thanks all!</div><div><br></div>

</div></div></div></div>

--000000000000b76e6e057dbcd14d--




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33729; Package emacs. Full text available.

Message received at 33729 <at> debbugs.gnu.org:


Received: (at 33729) by debbugs.gnu.org; 24 Dec 2018 02:08:59 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Dec 23 21:08:59 2018
Received: from localhost ([127.0.0.1]:34165 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gbFfv-0002xZ-21
	for submit <at> debbugs.gnu.org; Sun, 23 Dec 2018 21:08:59 -0500
Received: from mail-wr1-f47.google.com ([209.85.221.47]:42337)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dr.khaled.hosny@HIDDEN>) id 1gbFft-0002xK-Bb
 for 33729 <at> debbugs.gnu.org; Sun, 23 Dec 2018 21:08:57 -0500
Received: by mail-wr1-f47.google.com with SMTP id q18so10410188wrx.9
 for <33729 <at> debbugs.gnu.org>; Sun, 23 Dec 2018 18:08:57 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=date:from:to:cc:subject:message-id:references:mime-version
 :content-disposition:content-transfer-encoding:in-reply-to
 :user-agent; bh=nZOV9QV+WhHVBttOHPNwez+nlyT/V5avzsnJt4DlvTg=;
 b=QnQnf/noCVHWwkrTqhQ54Cby2obBToIBrWsb7CScPrlm0HzvAM0HJARNc14KmOmn6b
 d0yrh+eO0pFe4ZbaMYe9WjCtJgGoqgWa7DVPgqYLb4pSYxP+Y0unNVw9edWrGwIbAm2A
 gUyH7JYWakjspsshiVYw9qnvNzadnSgLqJGEYuE9YdIAs1H8+zQohC3ouae5BfCS6hRK
 RGaUBc82ZjzTIQBrkAvqorJaTGazNlFPaG+87rY1Yv6OI3cCzl7nJOTT1EoDSw1GtcJl
 EhFPPhs4pzur+6A4j/hcysQFIvb2d1RZIWxjEa+3jF8TfOL8y++b4TmgvDAWdG4kAJ7L
 kCDQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:date:from:to:cc:subject:message-id:references
 :mime-version:content-disposition:content-transfer-encoding
 :in-reply-to:user-agent;
 bh=nZOV9QV+WhHVBttOHPNwez+nlyT/V5avzsnJt4DlvTg=;
 b=Zri9TIwV1hjZPDdUhMWXpiwZhbvo4KN0kOzYACR0abKTaOPFHde/RQJT9U0d3r/Qy4
 QnrUlgKiN8y5Cc40SIXVezfWrPlT63tkFi8co+towqavJSPAE9vpX7xWl6EAnp49spVJ
 zKB4L/9bq1gOUpGuVJjlB6Ij7qLv7sAa7TbIoOYQr24i+ddcSnjea7AhQmYwkACKA8FB
 REwqJZ0g//pv4hzOuevuOF3z2oST6vSRpdn87zymqL9ocWcK07HSpdOjm0Kyf9AUxlak
 SEfmTAUfvQ2f7ekjZ0QZBL4Zjq91htSXX9fHRqy2grDD/IiyWDM3AWQ+qv7QjY3qtRfH
 3VKg==
X-Gm-Message-State: AJcUukciJt66ZpxtBOI5lvSD5PSJjJ5kLSFVE/gPr1ZsiXwAvvazrkXe
 6C2ZyrfhPIEgdXrw/mdVKUM=
X-Google-Smtp-Source: ALg8bN7+6WZkVnAkfPppEyeh+YV663s4ryyFPyC0uPHZ0fhCgkASRQ7ay95gXDBDFTy7z/jh2Ddj7g==
X-Received: by 2002:adf:a393:: with SMTP id l19mr10315183wrb.110.1545617331697; 
 Sun, 23 Dec 2018 18:08:51 -0800 (PST)
Received: from macbook.localdomain ([41.237.113.27])
 by smtp.gmail.com with ESMTPSA id 199sm25002083wmh.21.2018.12.23.18.08.50
 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256);
 Sun, 23 Dec 2018 18:08:50 -0800 (PST)
Date: Mon, 24 Dec 2018 04:08:47 +0200
From: Khaled Hosny <dr.khaled.hosny@HIDDEN>
To: Eli Zaretskii <eliz@HIDDEN>
Subject: Re: bug#33729: 27.0.50; Partial glyphs not rendered for Gujarati
 with Harfbuzz enabled (renders fine using m17n)
Message-ID: <20181224020847.GC6568@HIDDEN>
References: <20181222090644.GB2244@HIDDEN>
 <83va3lan24.fsf@HIDDEN>
 <20181222151509.GC2244@HIDDEN>
 <83h8f5a7po.fsf@HIDDEN>
 <20181222154945.GE2244@HIDDEN>
 <83bm5d9wsc.fsf@HIDDEN>
 <20181222205948.GF2244@HIDDEN>
 <838t0gapcj.fsf@HIDDEN>
 <20181223135109.GA6568@HIDDEN>
 <83va3k8c79.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <83va3k8c79.fsf@HIDDEN>
User-Agent: Mutt/1.11.1 (2018-12-01)
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 33729
Cc: 33729 <at> debbugs.gnu.org, rgm@HIDDEN, kaushal.modi@HIDDEN,
 far.nasiri.m@HIDDEN, behdad@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

On Sun, Dec 23, 2018 at 06:00:58PM +0200, Eli Zaretskii wrote:
> > Date: Sun, 23 Dec 2018 15:51:09 +0200
> > From: Khaled Hosny <dr.khaled.hosny@HIDDEN>
> > Cc: rgm@HIDDEN, far.nasiri.m@HIDDEN, behdad@HIDDEN,
> > 	33729 <at> debbugs.gnu.org, kaushal.modi@HIDDEN
> > 
> > > Are there any disadvantages in using the built-in support?  I mean,
> > > why did you envision an Emacs-specific implementation in the first
> > > place?
> > 
> > I thought, but I might be mistaken, that Emacs allow changing these
> > character properties at runtime and someone might possibly want to use
> > that to change some character property (e.g. make some PUA character a
> > combining mark) and it would then be nice if HarfBuzz respected that. I
> > admit that is very niche thing if possible at all, and I’m more than
> > happy to let HarfBuzz use it default Unicode functions and simplify the
> > Emacs integration code.
> 
> Right, I agree that we should for now leave that to HarfBuzz.  It
> could be added later as an optional feature.  (I don't expect many
> users to want to modify the Unicode character properties.)

I think we are almost good now. There is only one serious FIXME left:

  /* FIXME: guess_segment_properties is BAD BAD BAD.
   * we need to get these properties with the LGSTRING. */
#if 1
  hb_buffer_guess_segment_properties (hb_buffer);
#else
  hb_buffer_set_direction (hb_buffer, XXX);
  hb_buffer_set_script (hb_buffer, XXX);
  hb_buffer_set_language (hb_buffer, XXX);
#endif

We need to know, for a given lgstring we are shaping:
* Its direction (from applying bidi algorithm). Each lgstring we are
  shaping must be of a single direction.
* Its script, possibly after applying something like:
  http://unicode.org/reports/tr24/#Common
* Its language, is Emacs allows setting text language (my understand is
  that it doesn’t). Some languages really need this for applying
  language-specfic features (Urdu digits, Serbian alternate glyphs, etc.).




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33729; Package emacs. Full text available.

Message received at 33729 <at> debbugs.gnu.org:


Received: (at 33729) by debbugs.gnu.org; 23 Dec 2018 16:01:36 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Dec 23 11:01:35 2018
Received: from localhost ([127.0.0.1]:33922 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gb6C7-0004Gu-Le
	for submit <at> debbugs.gnu.org; Sun, 23 Dec 2018 11:01:35 -0500
Received: from eggs.gnu.org ([208.118.235.92]:40421)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1gb6C5-0004Gh-VV
 for 33729 <at> debbugs.gnu.org; Sun, 23 Dec 2018 11:01:34 -0500
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <eliz@HIDDEN>) id 1gb6C0-0001A1-3E
 for 33729 <at> debbugs.gnu.org; Sun, 23 Dec 2018 11:01:28 -0500
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50 autolearn=disabled
 version=3.3.2
Received: from fencepost.gnu.org ([2001:4830:134:3::e]:55839)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>)
 id 1gb6Bp-0000y1-OW; Sun, 23 Dec 2018 11:01:19 -0500
Received: from [176.228.60.248] (port=2445 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1gb6Bi-0003t6-BM; Sun, 23 Dec 2018 11:01:11 -0500
Date: Sun, 23 Dec 2018 18:00:58 +0200
Message-Id: <83va3k8c79.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Khaled Hosny <dr.khaled.hosny@HIDDEN>
In-reply-to: <20181223135109.GA6568@HIDDEN> (message from Khaled
 Hosny on Sun, 23 Dec 2018 15:51:09 +0200)
Subject: Re: bug#33729: 27.0.50; Partial glyphs not rendered for Gujarati
 with Harfbuzz enabled (renders fine using m17n)
References: <83mup4du5z.fsf@HIDDEN>
 <20181222085448.GA2244@HIDDEN>
 <20181222090644.GB2244@HIDDEN>
 <83va3lan24.fsf@HIDDEN>
 <20181222151509.GC2244@HIDDEN>
 <83h8f5a7po.fsf@HIDDEN>
 <20181222154945.GE2244@HIDDEN>
 <83bm5d9wsc.fsf@HIDDEN>
 <20181222205948.GF2244@HIDDEN>
 <838t0gapcj.fsf@HIDDEN> <20181223135109.GA6568@HIDDEN>
MIME-version: 1.0
Content-type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2001:4830:134:3::e
X-Spam-Score: -5.0 (-----)
X-Debbugs-Envelope-To: 33729
Cc: 33729 <at> debbugs.gnu.org, rgm@HIDDEN, kaushal.modi@HIDDEN,
 far.nasiri.m@HIDDEN, behdad@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -6.0 (------)

> Date: Sun, 23 Dec 2018 15:51:09 +0200
> From: Khaled Hosny <dr.khaled.hosny@HIDDEN>
> Cc: rgm@HIDDEN, far.nasiri.m@HIDDEN, behdad@HIDDEN,
> 	33729 <at> debbugs.gnu.org, kaushal.modi@HIDDEN
> 
> > Are there any disadvantages in using the built-in support?  I mean,
> > why did you envision an Emacs-specific implementation in the first
> > place?
> 
> I thought, but I might be mistaken, that Emacs allow changing these
> character properties at runtime and someone might possibly want to use
> that to change some character property (e.g. make some PUA character a
> combining mark) and it would then be nice if HarfBuzz respected that. I
> admit that is very niche thing if possible at all, and I’m more than
> happy to let HarfBuzz use it default Unicode functions and simplify the
> Emacs integration code.

Right, I agree that we should for now leave that to HarfBuzz.  It
could be added later as an optional feature.  (I don't expect many
users to want to modify the Unicode character properties.)

Thanks.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33729; Package emacs. Full text available.

Message received at 33729 <at> debbugs.gnu.org:


Received: (at 33729) by debbugs.gnu.org; 23 Dec 2018 13:51:21 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Dec 23 08:51:21 2018
Received: from localhost ([127.0.0.1]:60611 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gb4A5-0000jr-7p
	for submit <at> debbugs.gnu.org; Sun, 23 Dec 2018 08:51:21 -0500
Received: from mail-wm1-f52.google.com ([209.85.128.52]:37036)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dr.khaled.hosny@HIDDEN>) id 1gb4A4-0000ja-0j
 for 33729 <at> debbugs.gnu.org; Sun, 23 Dec 2018 08:51:20 -0500
Received: by mail-wm1-f52.google.com with SMTP id g67so9876886wmd.2
 for <33729 <at> debbugs.gnu.org>; Sun, 23 Dec 2018 05:51:19 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=date:from:to:cc:subject:message-id:references:mime-version
 :content-disposition:content-transfer-encoding:in-reply-to
 :user-agent; bh=oHqB464v2DcOygKq/1ULYmQb8jGmH7avhL9st7Y1wUM=;
 b=uiBnUO36mRbYXD/UX7P6FG3DuzBj0qdTzbB74W4cYW0Y1Y/5pniOMjuWXn6/UlMFny
 p4uz8ge6QXDhKc7yqEMBDdr0dldEX/y2dLcz9uua9OdOOQ01LqupNFR4Jx3aqlnIprqs
 AqdH9ERtcLfvUZlp2EsqV3CNEYptp0sqA46Y6Fk9Pr2LaoQJOtTn4OZ03RW5fjFsk8CI
 eiCY1vszJAaA9wpVLuWtQfoAwNFxdcsA2hbA3b2kGBc4890O0ltgLHQlqIRlN8692YgQ
 SxkAesK4Hfs5Fy9pXzdabqfo7zr2wxgxgYwRr81uCVprMJX8QtfurmIg7kFnDPqgOWRI
 h14g==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:date:from:to:cc:subject:message-id:references
 :mime-version:content-disposition:content-transfer-encoding
 :in-reply-to:user-agent;
 bh=oHqB464v2DcOygKq/1ULYmQb8jGmH7avhL9st7Y1wUM=;
 b=r9/3675aqdOC2FDP7YY3yQ3QgBjvZ0n9/4TbaQdj30eSNm6XURdU0EVLoL6S+iG0MZ
 AqNGY5TyPfpuRBzF+cqei52Fzwh5+m/XF07hUyJE5NFOOeKbnMxyqMffo+kBUIv+iABA
 t2lq8y99xE1SlO6k99Dtd3daz2Y/omMClNpmAFq24zZI/HT9l8bdLOruQUEDDL0uxG6B
 Gi37DrAlJY01MyQMlAUSgv/eko7ogAmWz9oFk8MyTJ8E8A5aGFdiHS9EUw7t8aEajbz7
 rMkrWcnFMiGWkkHhMh/UJq4rsEYxwVIZ4xQm9wbufWuV/Bn1zr+k31Tmct6Jhn8BPh8+
 ewyg==
X-Gm-Message-State: AA+aEWYA2wxRLHzLU788Z+BxiaXa3+Jf7xDvq6rSZ9HIl5FudS52JXV+
 wCDzJHu9MwWYVX4TvAl2LxUNzF23fvU=
X-Google-Smtp-Source: AFSGD/V3rAMYu7OwSwHrIwLNknMjUVlDnZ/PlmCP44V82bYOwcUgN/klT4c171wqbGwzpagXhHKs/w==
X-Received: by 2002:a7b:c156:: with SMTP id z22mr9235362wmi.24.1545573074238; 
 Sun, 23 Dec 2018 05:51:14 -0800 (PST)
Received: from macbook.localdomain ([41.237.113.27])
 by smtp.gmail.com with ESMTPSA id t5sm16754031wmd.15.2018.12.23.05.51.11
 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256);
 Sun, 23 Dec 2018 05:51:13 -0800 (PST)
Date: Sun, 23 Dec 2018 15:51:09 +0200
From: Khaled Hosny <dr.khaled.hosny@HIDDEN>
To: Eli Zaretskii <eliz@HIDDEN>
Subject: Re: bug#33729: 27.0.50; Partial glyphs not rendered for Gujarati
 with Harfbuzz enabled (renders fine using m17n)
Message-ID: <20181223135109.GA6568@HIDDEN>
References: <83mup4du5z.fsf@HIDDEN>
 <20181222085448.GA2244@HIDDEN>
 <20181222090644.GB2244@HIDDEN>
 <83va3lan24.fsf@HIDDEN>
 <20181222151509.GC2244@HIDDEN>
 <83h8f5a7po.fsf@HIDDEN>
 <20181222154945.GE2244@HIDDEN>
 <83bm5d9wsc.fsf@HIDDEN>
 <20181222205948.GF2244@HIDDEN>
 <838t0gapcj.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <838t0gapcj.fsf@HIDDEN>
User-Agent: Mutt/1.11.1 (2018-12-01)
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 33729
Cc: 33729 <at> debbugs.gnu.org, rgm@HIDDEN, kaushal.modi@HIDDEN,
 far.nasiri.m@HIDDEN, behdad@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

On Sun, Dec 23, 2018 at 05:34:04AM +0200, Eli Zaretskii wrote:
> > Date: Sat, 22 Dec 2018 22:59:48 +0200
> > From: Khaled Hosny <dr.khaled.hosny@HIDDEN>
> > Cc: rgm@HIDDEN, far.nasiri.m@HIDDEN, behdad@HIDDEN,
> > 	33729 <at> debbugs.gnu.org, kaushal.modi@HIDDEN
> > 
> > On Sat, Dec 22, 2018 at 09:38:43PM +0200, Eli Zaretskii wrote:
> > > > Date: Sat, 22 Dec 2018 17:49:45 +0200
> > > > From: Khaled Hosny <dr.khaled.hosny@HIDDEN>
> > > > Cc: rgm@HIDDEN, far.nasiri.m@HIDDEN, behdad@HIDDEN,
> > > > 	33729 <at> debbugs.gnu.org, kaushal.modi@HIDDEN
> > > > 
> > > > Yes, the standard Unicode composition and decomposition. HarfBuzz uses
> > > > these during shaping (it prefers composed form for a given sequence if
> > > > supported by the font, and falls back to decomposed form otherwise).
> > > 
> > > Btw, how is this problem solved in the other projects that use
> > > Harfuzz?  Does every project need to provide this functionality, or
> > > does Harfuzz have it built-in, like with the script tags?  If there's
> > > built-in support for this, perhaps Emacs could just use that?
> > 
> > There is built-in support, and currently we are using that. I can just
> > remove the FIXME.
> 
> Are there any disadvantages in using the built-in support?  I mean,
> why did you envision an Emacs-specific implementation in the first
> place?

I thought, but I might be mistaken, that Emacs allow changing these
character properties at runtime and someone might possibly want to use
that to change some character property (e.g. make some PUA character a
combining mark) and it would then be nice if HarfBuzz respected that. I
admit that is very niche thing if possible at all, and I’m more than
happy to let HarfBuzz use it default Unicode functions and simplify the
Emacs integration code.

Regards,
Khaled




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33729; Package emacs. Full text available.

Message received at 33729 <at> debbugs.gnu.org:


Received: (at 33729) by debbugs.gnu.org; 23 Dec 2018 03:34:32 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Dec 22 22:34:32 2018
Received: from localhost ([127.0.0.1]:60388 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gauX9-0000Jd-OP
	for submit <at> debbugs.gnu.org; Sat, 22 Dec 2018 22:34:31 -0500
Received: from eggs.gnu.org ([208.118.235.92]:47846)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1gauX7-0000JQ-QZ
 for 33729 <at> debbugs.gnu.org; Sat, 22 Dec 2018 22:34:30 -0500
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <eliz@HIDDEN>) id 1gauX1-0006pt-MH
 for 33729 <at> debbugs.gnu.org; Sat, 22 Dec 2018 22:34:24 -0500
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=-0.0 required=5.0 tests=BAYES_40 autolearn=disabled
 version=3.3.2
Received: from fencepost.gnu.org ([2001:4830:134:3::e]:33628)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>)
 id 1gauWv-0006k9-Uu; Sat, 22 Dec 2018 22:34:17 -0500
Received: from [176.228.60.248] (port=4169 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1gauWv-0005Od-Fr; Sat, 22 Dec 2018 22:34:17 -0500
Date: Sun, 23 Dec 2018 05:34:04 +0200
Message-Id: <838t0gapcj.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Khaled Hosny <dr.khaled.hosny@HIDDEN>
In-reply-to: <20181222205948.GF2244@HIDDEN> (message from Khaled
 Hosny on Sat, 22 Dec 2018 22:59:48 +0200)
Subject: Re: bug#33729: 27.0.50; Partial glyphs not rendered for Gujarati
 with Harfbuzz enabled (renders fine using m17n)
References: <83tvjgf7ux.fsf@HIDDEN> <xk4lbd9erb.fsf@HIDDEN>
 <83mup4du5z.fsf@HIDDEN>
 <20181222085448.GA2244@HIDDEN>
 <20181222090644.GB2244@HIDDEN>
 <83va3lan24.fsf@HIDDEN>
 <20181222151509.GC2244@HIDDEN>
 <83h8f5a7po.fsf@HIDDEN>
 <20181222154945.GE2244@HIDDEN>
 <83bm5d9wsc.fsf@HIDDEN> <20181222205948.GF2244@HIDDEN>
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2001:4830:134:3::e
X-Spam-Score: -5.0 (-----)
X-Debbugs-Envelope-To: 33729
Cc: 33729 <at> debbugs.gnu.org, rgm@HIDDEN, kaushal.modi@HIDDEN,
 far.nasiri.m@HIDDEN, behdad@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -6.0 (------)

> Date: Sat, 22 Dec 2018 22:59:48 +0200
> From: Khaled Hosny <dr.khaled.hosny@HIDDEN>
> Cc: rgm@HIDDEN, far.nasiri.m@HIDDEN, behdad@HIDDEN,
> 	33729 <at> debbugs.gnu.org, kaushal.modi@HIDDEN
> 
> On Sat, Dec 22, 2018 at 09:38:43PM +0200, Eli Zaretskii wrote:
> > > Date: Sat, 22 Dec 2018 17:49:45 +0200
> > > From: Khaled Hosny <dr.khaled.hosny@HIDDEN>
> > > Cc: rgm@HIDDEN, far.nasiri.m@HIDDEN, behdad@HIDDEN,
> > > 	33729 <at> debbugs.gnu.org, kaushal.modi@HIDDEN
> > > 
> > > Yes, the standard Unicode composition and decomposition. HarfBuzz uses
> > > these during shaping (it prefers composed form for a given sequence if
> > > supported by the font, and falls back to decomposed form otherwise).
> > 
> > Btw, how is this problem solved in the other projects that use
> > Harfuzz?  Does every project need to provide this functionality, or
> > does Harfuzz have it built-in, like with the script tags?  If there's
> > built-in support for this, perhaps Emacs could just use that?
> 
> There is built-in support, and currently we are using that. I can just
> remove the FIXME.

Are there any disadvantages in using the built-in support?  I mean,
why did you envision an Emacs-specific implementation in the first
place?

Thanks.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33729; Package emacs. Full text available.

Message received at 33729 <at> debbugs.gnu.org:


Received: (at 33729) by debbugs.gnu.org; 22 Dec 2018 21:00:14 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Dec 22 16:00:14 2018
Received: from localhost ([127.0.0.1]:60255 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gaoNZ-0003e4-W3
	for submit <at> debbugs.gnu.org; Sat, 22 Dec 2018 16:00:14 -0500
Received: from mail-wr1-f50.google.com ([209.85.221.50]:44816)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dr.khaled.hosny@HIDDEN>) id 1gaoNY-0003d9-EC
 for 33729 <at> debbugs.gnu.org; Sat, 22 Dec 2018 16:00:12 -0500
Received: by mail-wr1-f50.google.com with SMTP id z5so8458695wrt.11
 for <33729 <at> debbugs.gnu.org>; Sat, 22 Dec 2018 13:00:12 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=date:from:to:cc:subject:message-id:references:mime-version
 :content-disposition:in-reply-to:user-agent;
 bh=83Jp+mUfRfb0L2jJcFNrs4u30VELLK5Of5HGV3dZK30=;
 b=Uqw9Qgaz2Lf/rzK+GcbOayZ2CW4M0vKZSa07sDk5gPjmWC3VJ0GS7FMpu06wJPYapg
 C5tpJytqDMczP+4m9VmZuaM9CT+W7a6M5U549fHPkllTNsPrpIZC8iVuxuvSemSpYhrT
 8hMe5vzuWF7YD2ZC/2/1GgGQteIyBHBxWoI0lGjLrLGex2IewOk6S7qeQtr4+O8qQ9NJ
 FGiV/kaO66MHbtQaj6d7Hbb/Nf/6YjvfMNumfNoLmBOmA1CxWfc5X/TYUdTaMIiEM89j
 pTIa/iBQe0BoZy7jL3k3gl9IpHqJxDT58wEOED604UUIlQ2fXCBnTTOMEezPyd9M30dD
 UZKw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:date:from:to:cc:subject:message-id:references
 :mime-version:content-disposition:in-reply-to:user-agent;
 bh=83Jp+mUfRfb0L2jJcFNrs4u30VELLK5Of5HGV3dZK30=;
 b=KubyR0gQsPPWb1x+ciEv9uOAz/YptNUb91bAg7PVV5aKGMjs7b7mjAKmCiyhigChPX
 Ieh1FllWbzbHTw2PnSFuBwn9REOUkCuMp6Yq5aD8DC6Ka+LDW7I3n1vN+mfNMAYvkV41
 EvS/KldvBtvM8ify4LajHuEgWyQqLAHUe77mf2R1KuW5nqhvkTFApMQXeMou63LxdhCw
 htK2o8daweI72Fs1463F4H6Ol2pepE6XUqJGY8wzlH6oQ4B6WeDOh6D2+zT+jvpfj7qI
 Vpyc7KfqIbTgWcl0d/nbx2DHE/Nbin1ZDT3bj7wzi0jRLF6YgEcpDyBAyErLMn9L8MhF
 1//g==
X-Gm-Message-State: AJcUukd22fLxLdYT5iSs375YWSrK03HUbI6DUmPcxDVzUtgs0z2rFUhP
 45wl+cVYscqcNck4bvPyWpc=
X-Google-Smtp-Source: ALg8bN42HsM13oKnlzrfWpvUk6Y8d8/Ow+WkqaFOiJKILWEWCXLGE1uiWj5hcq8BM3p24V45zHtnEQ==
X-Received: by 2002:adf:aa9c:: with SMTP id h28mr6937626wrc.216.1545512406534; 
 Sat, 22 Dec 2018 13:00:06 -0800 (PST)
Received: from macbook.localdomain ([41.237.113.27])
 by smtp.gmail.com with ESMTPSA id v19sm26904844wrd.46.2018.12.22.13.00.02
 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256);
 Sat, 22 Dec 2018 13:00:05 -0800 (PST)
Date: Sat, 22 Dec 2018 22:59:48 +0200
From: Khaled Hosny <dr.khaled.hosny@HIDDEN>
To: Eli Zaretskii <eliz@HIDDEN>
Subject: Re: bug#33729: 27.0.50; Partial glyphs not rendered for Gujarati
 with Harfbuzz enabled (renders fine using m17n)
Message-ID: <20181222205948.GF2244@HIDDEN>
References: <83tvjgf7ux.fsf@HIDDEN> <xk4lbd9erb.fsf@HIDDEN>
 <83mup4du5z.fsf@HIDDEN>
 <20181222085448.GA2244@HIDDEN>
 <20181222090644.GB2244@HIDDEN>
 <83va3lan24.fsf@HIDDEN>
 <20181222151509.GC2244@HIDDEN>
 <83h8f5a7po.fsf@HIDDEN>
 <20181222154945.GE2244@HIDDEN>
 <83bm5d9wsc.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
In-Reply-To: <83bm5d9wsc.fsf@HIDDEN>
User-Agent: Mutt/1.11.1 (2018-12-01)
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 33729
Cc: 33729 <at> debbugs.gnu.org, rgm@HIDDEN, kaushal.modi@HIDDEN,
 far.nasiri.m@HIDDEN, behdad@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

On Sat, Dec 22, 2018 at 09:38:43PM +0200, Eli Zaretskii wrote:
> > Date: Sat, 22 Dec 2018 17:49:45 +0200
> > From: Khaled Hosny <dr.khaled.hosny@HIDDEN>
> > Cc: rgm@HIDDEN, far.nasiri.m@HIDDEN, behdad@HIDDEN,
> > 	33729 <at> debbugs.gnu.org, kaushal.modi@HIDDEN
> > 
> > Yes, the standard Unicode composition and decomposition. HarfBuzz uses
> > these during shaping (it prefers composed form for a given sequence if
> > supported by the font, and falls back to decomposed form otherwise).
> 
> Btw, how is this problem solved in the other projects that use
> Harfuzz?  Does every project need to provide this functionality, or
> does Harfuzz have it built-in, like with the script tags?  If there's
> built-in support for this, perhaps Emacs could just use that?

There is built-in support, and currently we are using that. I can just
remove the FIXME.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33729; Package emacs. Full text available.

Message received at 33729 <at> debbugs.gnu.org:


Received: (at 33729) by debbugs.gnu.org; 22 Dec 2018 19:39:30 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Dec 22 14:39:30 2018
Received: from localhost ([127.0.0.1]:60230 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gan7S-0001eO-D8
	for submit <at> debbugs.gnu.org; Sat, 22 Dec 2018 14:39:30 -0500
Received: from eggs.gnu.org ([208.118.235.92]:34170)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1gan7Q-0001eB-Vk
 for 33729 <at> debbugs.gnu.org; Sat, 22 Dec 2018 14:39:29 -0500
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <eliz@HIDDEN>) id 1gan7G-0007pR-Qp
 for 33729 <at> debbugs.gnu.org; Sat, 22 Dec 2018 14:39:23 -0500
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=-0.0 required=5.0 tests=BAYES_40 autolearn=disabled
 version=3.3.2
Received: from fencepost.gnu.org ([2001:4830:134:3::e]:48612)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>)
 id 1gan75-0007hR-AR; Sat, 22 Dec 2018 14:39:07 -0500
Received: from [176.228.60.248] (port=2851 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1gan73-0000wr-MG; Sat, 22 Dec 2018 14:39:07 -0500
Date: Sat, 22 Dec 2018 21:38:43 +0200
Message-Id: <83bm5d9wsc.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Khaled Hosny <dr.khaled.hosny@HIDDEN>
In-reply-to: <20181222154945.GE2244@HIDDEN> (message from Khaled
 Hosny on Sat, 22 Dec 2018 17:49:45 +0200)
Subject: Re: bug#33729: 27.0.50; Partial glyphs not rendered for Gujarati
 with Harfbuzz enabled (renders fine using m17n)
References: <20181214110316.GK2244@HIDDEN>
 <83y38sfcme.fsf@HIDDEN> <83tvjgf7ux.fsf@HIDDEN>
 <xk4lbd9erb.fsf@HIDDEN> <83mup4du5z.fsf@HIDDEN>
 <20181222085448.GA2244@HIDDEN>
 <20181222090644.GB2244@HIDDEN>
 <83va3lan24.fsf@HIDDEN>
 <20181222151509.GC2244@HIDDEN>
 <83h8f5a7po.fsf@HIDDEN> <20181222154945.GE2244@HIDDEN>
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2001:4830:134:3::e
X-Spam-Score: -5.0 (-----)
X-Debbugs-Envelope-To: 33729
Cc: 33729 <at> debbugs.gnu.org, rgm@HIDDEN, kaushal.modi@HIDDEN,
 far.nasiri.m@HIDDEN, behdad@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -6.0 (------)

> Date: Sat, 22 Dec 2018 17:49:45 +0200
> From: Khaled Hosny <dr.khaled.hosny@HIDDEN>
> Cc: rgm@HIDDEN, far.nasiri.m@HIDDEN, behdad@HIDDEN,
> 	33729 <at> debbugs.gnu.org, kaushal.modi@HIDDEN
> 
> Yes, the standard Unicode composition and decomposition. HarfBuzz uses
> these during shaping (it prefers composed form for a given sequence if
> supported by the font, and falls back to decomposed form otherwise).

Btw, how is this problem solved in the other projects that use
Harfuzz?  Does every project need to provide this functionality, or
does Harfuzz have it built-in, like with the script tags?  If there's
built-in support for this, perhaps Emacs could just use that?




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33729; Package emacs. Full text available.

Message received at 33729 <at> debbugs.gnu.org:


Received: (at 33729) by debbugs.gnu.org; 22 Dec 2018 16:34:35 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Dec 22 11:34:35 2018
Received: from localhost ([127.0.0.1]:60179 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gakEV-0003Wv-LM
	for submit <at> debbugs.gnu.org; Sat, 22 Dec 2018 11:34:35 -0500
Received: from eggs.gnu.org ([208.118.235.92]:43932)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1gakEU-0003Wj-CE
 for 33729 <at> debbugs.gnu.org; Sat, 22 Dec 2018 11:34:34 -0500
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <eliz@HIDDEN>) id 1gakEO-0003KL-DS
 for 33729 <at> debbugs.gnu.org; Sat, 22 Dec 2018 11:34:29 -0500
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50 autolearn=disabled
 version=3.3.2
Received: from fencepost.gnu.org ([2001:4830:134:3::e]:43615)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>)
 id 1gakEL-0003GF-3v; Sat, 22 Dec 2018 11:34:25 -0500
Received: from [176.228.60.248] (port=3240 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1gakEF-0007DF-0r; Sat, 22 Dec 2018 11:34:23 -0500
Date: Sat, 22 Dec 2018 18:33:52 +0200
Message-Id: <83efa9a5cf.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Khaled Hosny <dr.khaled.hosny@HIDDEN>
In-reply-to: <20181222154945.GE2244@HIDDEN> (message from Khaled
 Hosny on Sat, 22 Dec 2018 17:49:45 +0200)
Subject: Re: bug#33729: 27.0.50; Partial glyphs not rendered for Gujarati
 with Harfbuzz enabled (renders fine using m17n)
References: <20181214110316.GK2244@HIDDEN>
 <83y38sfcme.fsf@HIDDEN> <83tvjgf7ux.fsf@HIDDEN>
 <xk4lbd9erb.fsf@HIDDEN> <83mup4du5z.fsf@HIDDEN>
 <20181222085448.GA2244@HIDDEN>
 <20181222090644.GB2244@HIDDEN>
 <83va3lan24.fsf@HIDDEN>
 <20181222151509.GC2244@HIDDEN>
 <83h8f5a7po.fsf@HIDDEN> <20181222154945.GE2244@HIDDEN>
MIME-version: 1.0
Content-type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2001:4830:134:3::e
X-Spam-Score: -5.0 (-----)
X-Debbugs-Envelope-To: 33729
Cc: 33729 <at> debbugs.gnu.org, rgm@HIDDEN, kaushal.modi@HIDDEN,
 far.nasiri.m@HIDDEN, behdad@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -6.0 (------)

> Date: Sat, 22 Dec 2018 17:49:45 +0200
> From: Khaled Hosny <dr.khaled.hosny@HIDDEN>
> Cc: rgm@HIDDEN, far.nasiri.m@HIDDEN, behdad@HIDDEN,
> 	33729 <at> debbugs.gnu.org, kaushal.modi@HIDDEN
> 
> Yes, the standard Unicode composition and decomposition. HarfBuzz uses
> these during shaping (it prefers composed form for a given sequence if
> supported by the font, and falls back to decomposed form otherwise).
> 
> > > I recall you suggested something earlier that I tried but couldn’t
> > > get to work, the exact detail escapes me.
> > 
> > I probably suggested using the 'decomposition' property of a
> > character, and perhaps als the facilities in ucs-normalize.el.
> 
> How can this be done from C.

There are several examples in the sources of calling Lisp from C.  As
just a random example:

	if (STRINGP (curdir))
	  val = call1 (intern ("file-remote-p"), curdir);

This calls the Lisp function file-remote-p with one argument, curdir.

If you tell me more about the arguments and the expected effects of
calling uni_compose and uni_decompose, maybe I could propose a
specific implementation.  The Harfbuzz documentation doesn't seem to
tell enough, or maybe I didn't find the right text.

Thanks.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33729; Package emacs. Full text available.

Message received at 33729 <at> debbugs.gnu.org:


Received: (at 33729) by debbugs.gnu.org; 22 Dec 2018 15:49:57 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Dec 22 10:49:57 2018
Received: from localhost ([127.0.0.1]:60154 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gajXJ-0000Qg-1h
	for submit <at> debbugs.gnu.org; Sat, 22 Dec 2018 10:49:57 -0500
Received: from mail-wr1-f48.google.com ([209.85.221.48]:46746)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dr.khaled.hosny@HIDDEN>) id 1gajXH-0000QR-RT
 for 33729 <at> debbugs.gnu.org; Sat, 22 Dec 2018 10:49:56 -0500
Received: by mail-wr1-f48.google.com with SMTP id l9so8013838wrt.13
 for <33729 <at> debbugs.gnu.org>; Sat, 22 Dec 2018 07:49:55 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=date:from:to:cc:subject:message-id:references:mime-version
 :content-disposition:content-transfer-encoding:in-reply-to
 :user-agent; bh=1K5JLOIUUfWCkBSSly4IU8p75v2teXxSFhcXrPVjtLE=;
 b=t/PSPscNvcxKVjepv6zx98h+xvIOGImei/68MZ23eeJVJjy5S3OrkL6GKDt03PuYNy
 rNtEqPnxIA4Fq41JYa4V3R1+kNEpWO/NYLQZXQXh8E+z7bVafqMpEeRrOTjOiacvuWIt
 Kls2WN992c5iuluH0q/g34/ddhC0i36/CnnRAEGgrUQqHPvxsidThU6pMSeBKWstz1Vs
 KIZjHK4ZohpxxRjHz87kDv4CQe8NyB4Ve8Z4NJX9d5AyAE8izr6r0f6VIqzWPtVoNeQ5
 JNdHDE5pk8HIXyp1s7TSQrHQBvsSJY3idj+4Wh4C4dDMp7KH9H6ODLnESpCi4MqkACyM
 C9Cg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:date:from:to:cc:subject:message-id:references
 :mime-version:content-disposition:content-transfer-encoding
 :in-reply-to:user-agent;
 bh=1K5JLOIUUfWCkBSSly4IU8p75v2teXxSFhcXrPVjtLE=;
 b=APr/MbDPRLlNHBQ36AL0RLx5TkiuGGBPQok6VvNPHtOhaxjNFHaS27agTTrBLeW1No
 UMzp3Zk3l3tyE0Vd3YciyhWPJZN2w/ZZ5DTZMrs/f7am6x1PKDeedOLLOJhBzp9wcLMs
 ep4i/vf2TTP1D6CD8CTKJoRFM3Vy+zMwNTrHHAdXVYRkr+uTh/S/fPce6f6CVo/Li5B1
 hD61rSCW0MbtbCscSv6DZdwJcUVWtF7bAefDd/1SZ+//XqzOJxnhs7mjVBeW7yfXcSRm
 mdKiTld4mijMRBv+0T+pT2mBYE0NUUCYCS0OOsOvtNkhP8zktAmnjXjPD2uyrUzPtF8O
 g5AA==
X-Gm-Message-State: AJcUukclDMKjXbtEU+WFrJwyUheehyvU5QxoLxVF9X2/+MFWz8C6Ciwl
 Ks8KFpflmajsW870QwqqTy4=
X-Google-Smtp-Source: ALg8bN5TL6HupD5aAxchA5RkaWr3x/N/Zo2GGAz1xi29gdUBQ145AIKNSwLliIGOzoPYy0b8WAujwQ==
X-Received: by 2002:adf:e08c:: with SMTP id c12mr6159100wri.199.1545493790051; 
 Sat, 22 Dec 2018 07:49:50 -0800 (PST)
Received: from macbook.localdomain ([41.237.113.27])
 by smtp.gmail.com with ESMTPSA id e16sm17561378wrn.72.2018.12.22.07.49.47
 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256);
 Sat, 22 Dec 2018 07:49:49 -0800 (PST)
Date: Sat, 22 Dec 2018 17:49:45 +0200
From: Khaled Hosny <dr.khaled.hosny@HIDDEN>
To: Eli Zaretskii <eliz@HIDDEN>
Subject: Re: bug#33729: 27.0.50; Partial glyphs not rendered for Gujarati
 with Harfbuzz enabled (renders fine using m17n)
Message-ID: <20181222154945.GE2244@HIDDEN>
References: <20181214110316.GK2244@HIDDEN>
 <83y38sfcme.fsf@HIDDEN> <83tvjgf7ux.fsf@HIDDEN>
 <xk4lbd9erb.fsf@HIDDEN> <83mup4du5z.fsf@HIDDEN>
 <20181222085448.GA2244@HIDDEN>
 <20181222090644.GB2244@HIDDEN>
 <83va3lan24.fsf@HIDDEN>
 <20181222151509.GC2244@HIDDEN>
 <83h8f5a7po.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <83h8f5a7po.fsf@HIDDEN>
User-Agent: Mutt/1.11.1 (2018-12-01)
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 33729
Cc: 33729 <at> debbugs.gnu.org, rgm@HIDDEN, kaushal.modi@HIDDEN,
 far.nasiri.m@HIDDEN, behdad@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

On Sat, Dec 22, 2018 at 05:42:43PM +0200, Eli Zaretskii wrote:
> > Date: Sat, 22 Dec 2018 17:15:09 +0200
> > From: Khaled Hosny <dr.khaled.hosny@HIDDEN>
> > Cc: rgm@HIDDEN, far.nasiri.m@HIDDEN, behdad@HIDDEN,
> > 	33729 <at> debbugs.gnu.org, kaushal.modi@HIDDEN
> > 
> > > There's a FIXME in the change you pushed (which I believe just
> > > repeats what was already in the previous version).  Ca you tell more
> > > about the problem we need to fix there?
> > 
> > We need a way to get Unicode composition and decomposition for the
> > a given character (implementing the uni_compose and uni_decompose
> > functions I deleted).
> 
> Yes, but what does that entail?  Are these compositions and
> decompositions defined by the Unicode UCD?  And how does Harfbuzz use
> the results for a given character?

Yes, the standard Unicode composition and decomposition. HarfBuzz uses
these during shaping (it prefers composed form for a given sequence if
supported by the font, and falls back to decomposed form otherwise).

> > I recall you suggested something earlier that I tried but couldn’t
> > get to work, the exact detail escapes me.
> 
> I probably suggested using the 'decomposition' property of a
> character, and perhaps als the facilities in ucs-normalize.el.

How can this be done from C.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33729; Package emacs. Full text available.

Message received at 33729 <at> debbugs.gnu.org:


Received: (at 33729) by debbugs.gnu.org; 22 Dec 2018 15:43:15 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Dec 22 10:43:15 2018
Received: from localhost ([127.0.0.1]:60145 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gajQo-0000FQ-T4
	for submit <at> debbugs.gnu.org; Sat, 22 Dec 2018 10:43:15 -0500
Received: from eggs.gnu.org ([208.118.235.92]:32841)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1gajQm-0000FB-P5
 for 33729 <at> debbugs.gnu.org; Sat, 22 Dec 2018 10:43:13 -0500
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <eliz@HIDDEN>) id 1gajQg-0006q5-Qu
 for 33729 <at> debbugs.gnu.org; Sat, 22 Dec 2018 10:43:07 -0500
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50 autolearn=disabled
 version=3.3.2
Received: from fencepost.gnu.org ([2001:4830:134:3::e]:42125)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>)
 id 1gajQd-0006eN-Lk; Sat, 22 Dec 2018 10:43:03 -0500
Received: from [176.228.60.248] (port=4079 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1gajQc-0007EU-GN; Sat, 22 Dec 2018 10:43:03 -0500
Date: Sat, 22 Dec 2018 17:42:43 +0200
Message-Id: <83h8f5a7po.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Khaled Hosny <dr.khaled.hosny@HIDDEN>
In-reply-to: <20181222151509.GC2244@HIDDEN> (message from Khaled
 Hosny on Sat, 22 Dec 2018 17:15:09 +0200)
Subject: Re: bug#33729: 27.0.50; Partial glyphs not rendered for Gujarati
 with Harfbuzz enabled (renders fine using m17n)
References: <20181214075056.GI2244@HIDDEN>
 <8336r0h1cb.fsf@HIDDEN>
 <20181214110316.GK2244@HIDDEN>
 <83y38sfcme.fsf@HIDDEN> <83tvjgf7ux.fsf@HIDDEN>
 <xk4lbd9erb.fsf@HIDDEN> <83mup4du5z.fsf@HIDDEN>
 <20181222085448.GA2244@HIDDEN>
 <20181222090644.GB2244@HIDDEN>
 <83va3lan24.fsf@HIDDEN> <20181222151509.GC2244@HIDDEN>
MIME-version: 1.0
Content-type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2001:4830:134:3::e
X-Spam-Score: -5.0 (-----)
X-Debbugs-Envelope-To: 33729
Cc: 33729 <at> debbugs.gnu.org, rgm@HIDDEN, kaushal.modi@HIDDEN,
 far.nasiri.m@HIDDEN, behdad@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -6.0 (------)

> Date: Sat, 22 Dec 2018 17:15:09 +0200
> From: Khaled Hosny <dr.khaled.hosny@HIDDEN>
> Cc: rgm@HIDDEN, far.nasiri.m@HIDDEN, behdad@HIDDEN,
> 	33729 <at> debbugs.gnu.org, kaushal.modi@HIDDEN
> 
> > There's a FIXME in the change you pushed (which I believe just
> > repeats what was already in the previous version).  Ca you tell more
> > about the problem we need to fix there?
> 
> We need a way to get Unicode composition and decomposition for the
> a given character (implementing the uni_compose and uni_decompose
> functions I deleted).

Yes, but what does that entail?  Are these compositions and
decompositions defined by the Unicode UCD?  And how does Harfbuzz use
the results for a given character?

> I recall you suggested something earlier that I tried but couldn’t
> get to work, the exact detail escapes me.

I probably suggested using the 'decomposition' property of a
character, and perhaps als the facilities in ucs-normalize.el.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33729; Package emacs. Full text available.

Message received at 33729 <at> debbugs.gnu.org:


Received: (at 33729) by debbugs.gnu.org; 22 Dec 2018 15:43:02 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Dec 22 10:43:02 2018
Received: from localhost ([127.0.0.1]:60142 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gajQa-0000Eg-E6
	for submit <at> debbugs.gnu.org; Sat, 22 Dec 2018 10:43:02 -0500
Received: from mail-wm1-f54.google.com ([209.85.128.54]:33402)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dr.khaled.hosny@HIDDEN>) id 1gajQY-0000ER-1n
 for 33729 <at> debbugs.gnu.org; Sat, 22 Dec 2018 10:42:58 -0500
Received: by mail-wm1-f54.google.com with SMTP id r24so16261713wmh.0
 for <33729 <at> debbugs.gnu.org>; Sat, 22 Dec 2018 07:42:57 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=date:from:to:cc:subject:message-id:references:mime-version
 :content-disposition:content-transfer-encoding:in-reply-to
 :user-agent; bh=nJnHhLFuY382kjGKA/qEXxBl/HR+fJyjYgQk1K1NYDM=;
 b=AAGZnlh6xYpwoTyIYZxSQvqC0RFMRzYRAWccz4HwCeQVchCoA19GNyoeUjRGTqgfhF
 TYV3fNxnE1T4g0EMs1e7Dt/82EA3fPW6TBQGCXZPouvQ97QthBJBrt5xcMdBfu15DeSk
 go3Vno8v3xAAIHolH5yXnbjgTjKrNP/hCEFSE5RQuANuEqAKJJN02Jd9hwdZp1l6CLWB
 nbykLnLcTjaxFpRMOQ6ahA3H7vIYTMLkpvx+AR6CR+w6Wt/uhJurvqJmKNIZeLB8dp6o
 cwnn4VufxDbiaTk3jkGdT78bwZHS7L4qsM4AdkrjXUoe6W2jhmerJcKxFg/gxMA1k4dL
 ZusA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:date:from:to:cc:subject:message-id:references
 :mime-version:content-disposition:content-transfer-encoding
 :in-reply-to:user-agent;
 bh=nJnHhLFuY382kjGKA/qEXxBl/HR+fJyjYgQk1K1NYDM=;
 b=BCWDoDYLUe9vDme/N0PR8E0ZCYnbMovlod3yMPnN4Z4WNzdXP3w+VI+jK//MeiApGZ
 Sklaqlf5Gtgi0V+GDjbeFwJ+kuOYBeSV0jUK7J8dFnrEtB/NQg1FVjDZ8Rz4BYqDIhTT
 ka1yT/ZG2gTHvjG8cWDKrzuDWxXgoxYeYhDdw1CMAGO1F/t3h406oPlJOGAAOwZicnro
 rNpmSKU1eA0WLQGi9RBfnXCdzwnxsyQQ5qwIVdaH9LqqiD/LyPeltWDnDAzPwPoAVhqU
 MiBwTNFpatARTmbOp8zCUsR73DEh6CyEnmrqPl2cCsRU8efFfRH5TXyZZARMKUDnvEnl
 qWQQ==
X-Gm-Message-State: AJcUukf4dWlIWPK93BogbQMPIox1NKnZxMLnAJrxZ0Ti1ierSk/Txtd1
 zNeBmrZ5Fya9/cIc+5F2zSY=
X-Google-Smtp-Source: ALg8bN5N2+RsLJH7rj/jK/z6DM/PBYiFt/FInkHcaIxFW7tiYmF/QKPHJcbMdpCANpCEbGCA4J9nvA==
X-Received: by 2002:a1c:5dce:: with SMTP id r197mr6363901wmb.130.1545493372134; 
 Sat, 22 Dec 2018 07:42:52 -0800 (PST)
Received: from macbook.localdomain ([41.237.113.27])
 by smtp.gmail.com with ESMTPSA id t5sm13026704wmg.43.2018.12.22.07.42.50
 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256);
 Sat, 22 Dec 2018 07:42:51 -0800 (PST)
Date: Sat, 22 Dec 2018 17:42:47 +0200
From: Khaled Hosny <dr.khaled.hosny@HIDDEN>
To: Behdad Esfahbod <behdad@HIDDEN>
Subject: Re: bug#33729: 27.0.50; Partial glyphs not rendered for Gujarati
 with Harfbuzz enabled (renders fine using m17n)
Message-ID: <20181222154247.GD2244@HIDDEN>
References: <20181214110316.GK2244@HIDDEN>
 <83y38sfcme.fsf@HIDDEN> <83tvjgf7ux.fsf@HIDDEN>
 <xk4lbd9erb.fsf@HIDDEN> <83mup4du5z.fsf@HIDDEN>
 <20181222085448.GA2244@HIDDEN>
 <20181222090644.GB2244@HIDDEN>
 <83va3lan24.fsf@HIDDEN>
 <20181222151509.GC2244@HIDDEN>
 <CAF63+7UadspWrA2qY_S3G8eX_YQwVrjo6ZWA_48U8G_fDAgUUw@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <CAF63+7UadspWrA2qY_S3G8eX_YQwVrjo6ZWA_48U8G_fDAgUUw@HIDDEN>
User-Agent: Mutt/1.11.1 (2018-12-01)
X-Spam-Score: 1.0 (+)
X-Debbugs-Envelope-To: 33729
Cc: rgm@HIDDEN, Eli Zaretskii <eliz@HIDDEN>,
 Kaushal Modi <kaushal.modi@HIDDEN>,
 Mohammad Nasirifar <far.nasiri.m@HIDDEN>, 33729 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

I’m sub-classing the default Unicode functions, so for the callback we
don’t implement the default implementation will be used already.

On Sat, Dec 22, 2018 at 10:27:06AM -0500, Behdad Esfahbod wrote:
> I suggest you enabled UCDN.
> 
> On Sat, Dec 22, 2018 at 10:15 AM Khaled Hosny <dr.khaled.hosny@HIDDEN>
> wrote:
> 
> > On Sat, Dec 22, 2018 at 12:11:15PM +0200, Eli Zaretskii wrote:
> > > > Date: Sat, 22 Dec 2018 11:06:44 +0200
> > > > From: Khaled Hosny <dr.khaled.hosny@HIDDEN>
> > > > Cc: Glenn Morris <rgm@HIDDEN>, far.nasiri.m@HIDDEN,
> > behdad@HIDDEN,
> > > >     33729 <at> debbugs.gnu.org, kaushal.modi@HIDDEN
> > > >
> > > > > Alternatively, we could just use HarfBuzz’s own built in ucdn-based
> > > > > Unicode function for this. The only reason for overriding this in
> > Emacs
> > > > > was to keep HarfBuzz and Emacs Unicode support in sync, but if we are
> > > > > going to duplicate the Unicode script data then better use what
> > HarfBuzz
> > > > > has.
> > > > >
> > > > > I’m going to try this now.
> > > >
> > > > I pushed a commit to harfbuzz branch that I think fixes this issue now.
> > >
> > > Thanks.
> > >
> > > There's a FIXME in the change you pushed (which I believe just
> > > repeats what was already in the previous version).  Ca you tell more
> > > about the problem we need to fix there?
> >
> > We need a way to get Unicode composition and decomposition for the
> > a given character (implementing the uni_compose and uni_decompose
> > functions I deleted). I recall you suggested something earlier that I
> > tried but couldn’t get to work, the exact detail escapes me.
> >
> 
> 
> -- 
> behdad
> http://behdad.org/




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33729; Package emacs. Full text available.

Message received at 33729 <at> debbugs.gnu.org:


Received: (at 33729) by debbugs.gnu.org; 22 Dec 2018 15:27:28 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Dec 22 10:27:28 2018
Received: from localhost ([127.0.0.1]:60137 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gajBX-0008Hg-UC
	for submit <at> debbugs.gnu.org; Sat, 22 Dec 2018 10:27:28 -0500
Received: from mail-lf1-f54.google.com ([209.85.167.54]:41951)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <behdad.esfahbod@HIDDEN>) id 1gajBW-0008HN-Ie
 for 33729 <at> debbugs.gnu.org; Sat, 22 Dec 2018 10:27:27 -0500
Received: by mail-lf1-f54.google.com with SMTP id c16so5887986lfj.8
 for <33729 <at> debbugs.gnu.org>; Sat, 22 Dec 2018 07:27:26 -0800 (PST)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:mime-version:references:in-reply-to:from:date
 :message-id:subject:to:cc;
 bh=gvhgE1f2i0MXn9vr1uuf2mM1ML0wOw3R/mhcd4CaT/I=;
 b=j0Hw+t1l4MJ2yGGcBy5phKB7UEAS+4eh29x0xZJS/8idDrXpeGN3mGY6dY7CQk48Jp
 8vz+Z3wjrjMH6Vwxw52UaEK4oqaCzukWB8sMEzltviZhPvAap2bemWJ7wvv3+PcP3XHN
 ggF35v2dt0rUzZXgFrFX78K/Xor03x6RAwo8+zmEq/E6T9rXKiF+Z14IHVVF83u2f5VO
 /govw4+Mr5AKZclqfUe8Qw9xbZVFxinAQHUST/h2iO6iRkiSMlJvJok1WZkQXakxZFEa
 qGnDel4fK7rmYEuWIguZZAHu4AjtlL2AQQ/N36ddl6jBKlAmh/2082j585VQjO23XR+q
 oerg==
X-Gm-Message-State: AA+aEWYGd5cHIC0Vb5Cgqz2xF8wBpQNW5oKPKgPCCyfckTchptFsebqS
 dHKQX4kT3xcaC63nQB/lEWye1CBRi0FP2kLRiiw=
X-Google-Smtp-Source: AFSGD/XzRZsTFCF6zIIpYsSysfcPgBHBH5iL/rprmIND9DMYTnQYRmziRxUoitr4bHoN6lGxnzWd1r6WESjhGJAFAAQ=
X-Received: by 2002:a19:9fcd:: with SMTP id i196mr3661715lfe.82.1545492438962; 
 Sat, 22 Dec 2018 07:27:18 -0800 (PST)
MIME-Version: 1.0
References: <20181214075056.GI2244@HIDDEN>
 <8336r0h1cb.fsf@HIDDEN>
 <20181214110316.GK2244@HIDDEN> <83y38sfcme.fsf@HIDDEN>
 <83tvjgf7ux.fsf@HIDDEN> <xk4lbd9erb.fsf@HIDDEN>
 <83mup4du5z.fsf@HIDDEN> <20181222085448.GA2244@HIDDEN>
 <20181222090644.GB2244@HIDDEN> <83va3lan24.fsf@HIDDEN>
 <20181222151509.GC2244@HIDDEN>
In-Reply-To: <20181222151509.GC2244@HIDDEN>
From: Behdad Esfahbod <behdad@HIDDEN>
Date: Sat, 22 Dec 2018 10:27:06 -0500
Message-ID: <CAF63+7UadspWrA2qY_S3G8eX_YQwVrjo6ZWA_48U8G_fDAgUUw@HIDDEN>
Subject: Re: bug#33729: 27.0.50; Partial glyphs not rendered for Gujarati with
 Harfbuzz enabled (renders fine using m17n)
To: Khaled Hosny <dr.khaled.hosny@HIDDEN>
Content-Type: multipart/alternative; boundary="000000000000d3d646057d9dffc4"
X-Spam-Score: 0.3 (/)
X-Debbugs-Envelope-To: 33729
Cc: rgm@HIDDEN, Eli Zaretskii <eliz@HIDDEN>,
 Kaushal Modi <kaushal.modi@HIDDEN>,
 Mohammad Nasirifar <far.nasiri.m@HIDDEN>, 33729 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.7 (/)

--000000000000d3d646057d9dffc4
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

I suggest you enabled UCDN.

On Sat, Dec 22, 2018 at 10:15 AM Khaled Hosny <dr.khaled.hosny@HIDDEN>
wrote:

> On Sat, Dec 22, 2018 at 12:11:15PM +0200, Eli Zaretskii wrote:
> > > Date: Sat, 22 Dec 2018 11:06:44 +0200
> > > From: Khaled Hosny <dr.khaled.hosny@HIDDEN>
> > > Cc: Glenn Morris <rgm@HIDDEN>, far.nasiri.m@HIDDEN,
> behdad@HIDDEN,
> > >     33729 <at> debbugs.gnu.org, kaushal.modi@HIDDEN
> > >
> > > > Alternatively, we could just use HarfBuzz=E2=80=99s own built in uc=
dn-based
> > > > Unicode function for this. The only reason for overriding this in
> Emacs
> > > > was to keep HarfBuzz and Emacs Unicode support in sync, but if we a=
re
> > > > going to duplicate the Unicode script data then better use what
> HarfBuzz
> > > > has.
> > > >
> > > > I=E2=80=99m going to try this now.
> > >
> > > I pushed a commit to harfbuzz branch that I think fixes this issue no=
w.
> >
> > Thanks.
> >
> > There's a FIXME in the change you pushed (which I believe just
> > repeats what was already in the previous version).  Ca you tell more
> > about the problem we need to fix there?
>
> We need a way to get Unicode composition and decomposition for the
> a given character (implementing the uni_compose and uni_decompose
> functions I deleted). I recall you suggested something earlier that I
> tried but couldn=E2=80=99t get to work, the exact detail escapes me.
>


--=20
behdad
http://behdad.org/

--000000000000d3d646057d9dffc4
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">I suggest you enabled UCDN.</div><br><div class=3D"gmail_q=
uote"><div dir=3D"ltr">On Sat, Dec 22, 2018 at 10:15 AM Khaled Hosny &lt;<a=
 href=3D"mailto:dr.khaled.hosny@HIDDEN">dr.khaled.hosny@HIDDEN</a>&gt=
; wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px=
 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Sat,=
 Dec 22, 2018 at 12:11:15PM +0200, Eli Zaretskii wrote:<br>
&gt; &gt; Date: Sat, 22 Dec 2018 11:06:44 +0200<br>
&gt; &gt; From: Khaled Hosny &lt;<a href=3D"mailto:dr.khaled.hosny@HIDDEN=
m" target=3D"_blank">dr.khaled.hosny@HIDDEN</a>&gt;<br>
&gt; &gt; Cc: Glenn Morris &lt;<a href=3D"mailto:rgm@HIDDEN" target=3D"_bl=
ank">rgm@HIDDEN</a>&gt;, <a href=3D"mailto:far.nasiri.m@HIDDEN" target=
=3D"_blank">far.nasiri.m@HIDDEN</a>, <a href=3D"mailto:behdad@HIDDEN=
" target=3D"_blank">behdad@HIDDEN</a>,<br>
&gt; &gt;=C2=A0 =C2=A0 =C2=A0<a href=3D"mailto:33729 <at> debbugs.gnu.org" targe=
t=3D"_blank">33729 <at> debbugs.gnu.org</a>, <a href=3D"mailto:kaushal.modi@gmai=
l.com" target=3D"_blank">kaushal.modi@HIDDEN</a><br>
&gt; &gt; <br>
&gt; &gt; &gt; Alternatively, we could just use HarfBuzz=E2=80=99s own buil=
t in ucdn-based<br>
&gt; &gt; &gt; Unicode function for this. The only reason for overriding th=
is in Emacs<br>
&gt; &gt; &gt; was to keep HarfBuzz and Emacs Unicode support in sync, but =
if we are<br>
&gt; &gt; &gt; going to duplicate the Unicode script data then better use w=
hat HarfBuzz<br>
&gt; &gt; &gt; has.<br>
&gt; &gt; &gt; <br>
&gt; &gt; &gt; I=E2=80=99m going to try this now.<br>
&gt; &gt; <br>
&gt; &gt; I pushed a commit to harfbuzz branch that I think fixes this issu=
e now.<br>
&gt; <br>
&gt; Thanks.<br>
&gt; <br>
&gt; There&#39;s a FIXME in the change you pushed (which I believe just<br>
&gt; repeats what was already in the previous version).=C2=A0 Ca you tell m=
ore<br>
&gt; about the problem we need to fix there?<br>
<br>
We need a way to get Unicode composition and decomposition for the<br>
a given character (implementing the uni_compose and uni_decompose<br>
functions I deleted). I recall you suggested something earlier that I<br>
tried but couldn=E2=80=99t get to work, the exact detail escapes me.<br>
</blockquote></div><br clear=3D"all"><div><br></div>-- <br><div dir=3D"ltr"=
 class=3D"gmail_signature">behdad<br><a href=3D"http://behdad.org/" target=
=3D"_blank">http://behdad.org/</a></div>

--000000000000d3d646057d9dffc4--




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33729; Package emacs. Full text available.

Message received at 33729 <at> debbugs.gnu.org:


Received: (at 33729) by debbugs.gnu.org; 22 Dec 2018 15:15:22 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Dec 22 10:15:22 2018
Received: from localhost ([127.0.0.1]:60129 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gaizq-0007y4-39
	for submit <at> debbugs.gnu.org; Sat, 22 Dec 2018 10:15:22 -0500
Received: from mail-wm1-f46.google.com ([209.85.128.46]:55774)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dr.khaled.hosny@HIDDEN>) id 1gaizo-0007xn-LN
 for 33729 <at> debbugs.gnu.org; Sat, 22 Dec 2018 10:15:21 -0500
Received: by mail-wm1-f46.google.com with SMTP id y139so7788077wmc.5
 for <33729 <at> debbugs.gnu.org>; Sat, 22 Dec 2018 07:15:20 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=date:from:to:cc:subject:message-id:references:mime-version
 :content-disposition:content-transfer-encoding:in-reply-to
 :user-agent; bh=FzSop95AfBQMCxG+TGKBaRMY54mqX2F9//fu2gbW0OE=;
 b=jSnaPYC6GXjyBkU1t2nOvCO68GY5ToFmKnk6ZZ1Hew1uFjCB/0Z/uLgjM3hW6mKszM
 /QZogQ40lRTAMOEDk/6uwqZm/NQQAcFFI7TekYnB82lsTFi2YWHqvq3QNZEpQTRTuXWl
 JTaV4y+H4PpX7WU9lcOXd7QAWRpMZd0Q90AtRMGwPtt74+eLI9Xh2jn+h9iq3VQCJjWW
 cmqQ9eQhxGhkYy5QmL15OjRNiPNFqijnNVy0+4cjAElOxNLr4+dn811DcSvTjcVBBpAG
 atFRP6ioklZTs7XzAgxZEggZebgr7ICxWtEiw0prHE7vCV1go6nojz93TgUSWj6+Ayec
 C53A==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:date:from:to:cc:subject:message-id:references
 :mime-version:content-disposition:content-transfer-encoding
 :in-reply-to:user-agent;
 bh=FzSop95AfBQMCxG+TGKBaRMY54mqX2F9//fu2gbW0OE=;
 b=LTlkO2+dp+8H21jHO9RKSvV4Fucoalfaac3svW4XTAinFQQzTcdx4XdntbmmssOx5Z
 0ye5bE1URyq6NOEr7M8iDkPtFgPtCeL4eqS0nzP/6u58D+X7Y0yPPMK5kEX6jmtf4NQJ
 xU4CVDyO660ctesAbR29I/uFyCbDDcA869Vl9Y+fpMlaqglOQB3BumCULtdngs489JlF
 QeoaUXKjy4W+g9eeFilZDpiqgR1ERC5Ukv4ktiN2U9+OO8TGd6k5hxwUrKhX4q1Qrz9R
 As+ACneEPLBnjgDjgQSUnfFls2I3Xji6dSZA1jTFQnbEZze6dgXdrKSdOANPeg+2MB8M
 XHLw==
X-Gm-Message-State: AA+aEWa8Ac8tdxgSGmtCaKXOG1VviRcYPwqMxlFBS8kyahbaiZTdZR9c
 EViq/TYFbS7un5ZouIO6uUw=
X-Google-Smtp-Source: AFSGD/XCqMPRmxTkUgGXu9TJu+t9b7UCcB/QkNUnVx4T4t8QkZw6EqrsEAlh2GMcNUX90vxUZNhHGg==
X-Received: by 2002:a1c:6508:: with SMTP id z8mr6800796wmb.28.1545491714902;
 Sat, 22 Dec 2018 07:15:14 -0800 (PST)
Received: from macbook.localdomain ([41.237.113.27])
 by smtp.gmail.com with ESMTPSA id z12sm12993079wrh.35.2018.12.22.07.15.12
 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256);
 Sat, 22 Dec 2018 07:15:13 -0800 (PST)
Date: Sat, 22 Dec 2018 17:15:09 +0200
From: Khaled Hosny <dr.khaled.hosny@HIDDEN>
To: Eli Zaretskii <eliz@HIDDEN>
Subject: Re: bug#33729: 27.0.50; Partial glyphs not rendered for Gujarati
 with Harfbuzz enabled (renders fine using m17n)
Message-ID: <20181222151509.GC2244@HIDDEN>
References: <20181214075056.GI2244@HIDDEN>
 <8336r0h1cb.fsf@HIDDEN>
 <20181214110316.GK2244@HIDDEN>
 <83y38sfcme.fsf@HIDDEN> <83tvjgf7ux.fsf@HIDDEN>
 <xk4lbd9erb.fsf@HIDDEN> <83mup4du5z.fsf@HIDDEN>
 <20181222085448.GA2244@HIDDEN>
 <20181222090644.GB2244@HIDDEN>
 <83va3lan24.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <83va3lan24.fsf@HIDDEN>
User-Agent: Mutt/1.11.1 (2018-12-01)
X-Spam-Score: 1.0 (+)
X-Debbugs-Envelope-To: 33729
Cc: 33729 <at> debbugs.gnu.org, rgm@HIDDEN, kaushal.modi@HIDDEN,
 far.nasiri.m@HIDDEN, behdad@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

On Sat, Dec 22, 2018 at 12:11:15PM +0200, Eli Zaretskii wrote:
> > Date: Sat, 22 Dec 2018 11:06:44 +0200
> > From: Khaled Hosny <dr.khaled.hosny@HIDDEN>
> > Cc: Glenn Morris <rgm@HIDDEN>, far.nasiri.m@HIDDEN, behdad@HIDDEN,
> > 	33729 <at> debbugs.gnu.org, kaushal.modi@HIDDEN
> > 
> > > Alternatively, we could just use HarfBuzz’s own built in ucdn-based
> > > Unicode function for this. The only reason for overriding this in Emacs
> > > was to keep HarfBuzz and Emacs Unicode support in sync, but if we are
> > > going to duplicate the Unicode script data then better use what HarfBuzz
> > > has.
> > > 
> > > I’m going to try this now.
> > 
> > I pushed a commit to harfbuzz branch that I think fixes this issue now.
> 
> Thanks.
> 
> There's a FIXME in the change you pushed (which I believe just
> repeats what was already in the previous version).  Ca you tell more
> about the problem we need to fix there?

We need a way to get Unicode composition and decomposition for the
a given character (implementing the uni_compose and uni_decompose
functions I deleted). I recall you suggested something earlier that I
tried but couldn’t get to work, the exact detail escapes me.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33729; Package emacs. Full text available.

Message received at 33729 <at> debbugs.gnu.org:


Received: (at 33729) by debbugs.gnu.org; 22 Dec 2018 10:11:44 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Dec 22 05:11:44 2018
Received: from localhost ([127.0.0.1]:59169 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gaeFz-0006vp-Rp
	for submit <at> debbugs.gnu.org; Sat, 22 Dec 2018 05:11:44 -0500
Received: from eggs.gnu.org ([208.118.235.92]:35315)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1gaeFy-0006vd-JY
 for 33729 <at> debbugs.gnu.org; Sat, 22 Dec 2018 05:11:42 -0500
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <eliz@HIDDEN>) id 1gaeFs-0000O9-P7
 for 33729 <at> debbugs.gnu.org; Sat, 22 Dec 2018 05:11:37 -0500
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=-0.0 required=5.0 tests=BAYES_20 autolearn=disabled
 version=3.3.2
Received: from fencepost.gnu.org ([2001:4830:134:3::e]:36070)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>)
 id 1gaeFn-0000IT-Gh; Sat, 22 Dec 2018 05:11:31 -0500
Received: from [176.228.60.248] (port=2954 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1gaeFn-0003NC-2Y; Sat, 22 Dec 2018 05:11:31 -0500
Date: Sat, 22 Dec 2018 12:11:15 +0200
Message-Id: <83va3lan24.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Khaled Hosny <dr.khaled.hosny@HIDDEN>
In-reply-to: <20181222090644.GB2244@HIDDEN> (message from Khaled
 Hosny on Sat, 22 Dec 2018 11:06:44 +0200)
Subject: Re: bug#33729: 27.0.50; Partial glyphs not rendered for Gujarati
 with Harfbuzz enabled (renders fine using m17n)
References: <CAFyQvY0pznh0yM9XJ9y_cEft47SXCmFp+O5rLkzb6o0By-MQuA@HIDDEN>
 <83h8fghcpo.fsf@HIDDEN>
 <20181214075056.GI2244@HIDDEN>
 <8336r0h1cb.fsf@HIDDEN>
 <20181214110316.GK2244@HIDDEN>
 <83y38sfcme.fsf@HIDDEN> <83tvjgf7ux.fsf@HIDDEN>
 <xk4lbd9erb.fsf@HIDDEN> <83mup4du5z.fsf@HIDDEN>
 <20181222085448.GA2244@HIDDEN>
 <20181222090644.GB2244@HIDDEN>
MIME-version: 1.0
Content-type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2001:4830:134:3::e
X-Spam-Score: -5.0 (-----)
X-Debbugs-Envelope-To: 33729
Cc: 33729 <at> debbugs.gnu.org, rgm@HIDDEN, kaushal.modi@HIDDEN,
 far.nasiri.m@HIDDEN, behdad@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -6.0 (------)

> Date: Sat, 22 Dec 2018 11:06:44 +0200
> From: Khaled Hosny <dr.khaled.hosny@HIDDEN>
> Cc: Glenn Morris <rgm@HIDDEN>, far.nasiri.m@HIDDEN, behdad@HIDDEN,
> 	33729 <at> debbugs.gnu.org, kaushal.modi@HIDDEN
> 
> > Alternatively, we could just use HarfBuzz’s own built in ucdn-based
> > Unicode function for this. The only reason for overriding this in Emacs
> > was to keep HarfBuzz and Emacs Unicode support in sync, but if we are
> > going to duplicate the Unicode script data then better use what HarfBuzz
> > has.
> > 
> > I’m going to try this now.
> 
> I pushed a commit to harfbuzz branch that I think fixes this issue now.

Thanks.

There's a FIXME in the change you pushed (which I believe just
repeats what was already in the previous version).  Ca you tell more
about the problem we need to fix there?




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33729; Package emacs. Full text available.

Message received at 33729 <at> debbugs.gnu.org:


Received: (at 33729) by debbugs.gnu.org; 22 Dec 2018 09:06:58 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Dec 22 04:06:58 2018
Received: from localhost ([127.0.0.1]:59123 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gadFJ-0003Kt-O2
	for submit <at> debbugs.gnu.org; Sat, 22 Dec 2018 04:06:58 -0500
Received: from mail-wm1-f54.google.com ([209.85.128.54]:32970)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dr.khaled.hosny@HIDDEN>) id 1gadFH-0003Kg-4V
 for 33729 <at> debbugs.gnu.org; Sat, 22 Dec 2018 04:06:55 -0500
Received: by mail-wm1-f54.google.com with SMTP id r24so15772214wmh.0
 for <33729 <at> debbugs.gnu.org>; Sat, 22 Dec 2018 01:06:55 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=date:from:to:cc:subject:message-id:references:mime-version
 :content-disposition:content-transfer-encoding:in-reply-to
 :user-agent; bh=qcVpSqdJ7le6hJGh1xq2XYove73A1hkwlqmQilswBlg=;
 b=IqdSXy7NYYfTJf6JVpXXQfMzClq47agW0S6D4RGKHraKF2hJwLXX8cBqUyhrmVRJXx
 eQKerx6JYZ6yZHk8HSBAYxS5XS5mEwmsmZsdAtJNa/URJE2NiqatQeB90AdBLxlOWPIy
 S6ScewS6zuF3VNOpnlEvxeviqQZqTSzoML//x+iap0YSX2hewAJLjeK8g3cKFEpT1t/V
 YtnQ9BR+pPIAAVuWj9mtr0CWdzz15nHb0Y9+Iz3i1cV1bi3z4yb3mmpNhwZp9KOQdRge
 pUtcRMyZ/Is6dJUI326kPklBzqb+GR4CRGE8Ox/nxsBhZIwrbCZCuYwp+sdO2j9koyf1
 BMAA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:date:from:to:cc:subject:message-id:references
 :mime-version:content-disposition:content-transfer-encoding
 :in-reply-to:user-agent;
 bh=qcVpSqdJ7le6hJGh1xq2XYove73A1hkwlqmQilswBlg=;
 b=t8VEV0Obds3tVm6xW2CMUIm0Fc76rozaMVkGnpw6SgcoHdW2VhtD+Wm/Y/7KtNBeub
 aFVBfi37aAm7WJBp3Kf+gGoQ5RqBT99BUKHFg1BvnaalAm0GVyaOScpodSJFuNbcI7Kp
 3EY+38ptRevGGKJ4cruLaPhHZSpkYt2WstD+3quXGi8p+wpOn8TlDw5yFxn03W2Fbjdn
 972nrEdFHJBEUk0sKt19zs7yO5lkPF36Q1/FbjNfi7xRKSh+hZxOq5Q8u7gCU6syRbKX
 BCdeGCyP1jzfvA+MyauHUbc9e1xgKNEYfyG8OlkXp7k31DB/gdPPy0lLZ1yZSNH5uPOC
 iVww==
X-Gm-Message-State: AA+aEWYPAy/HGtTC+YtA8y87aBgIs8cUGwVJct7wpfN+DVnr1pa1HPUx
 tEGV+CZi7+Po9ti4C0kq/EY=
X-Google-Smtp-Source: ALg8bN7NCM1gVRooZKeSJbI1hoZxNuxkcFb/pev7neADnmX+XmJ+bFAEZeAaoqZH/gDK8Pjyv7XFnA==
X-Received: by 2002:a1c:2787:: with SMTP id n129mr5997486wmn.128.1545469609316; 
 Sat, 22 Dec 2018 01:06:49 -0800 (PST)
Received: from macbook.localdomain ([41.237.113.27])
 by smtp.gmail.com with ESMTPSA id f66sm16252507wmd.28.2018.12.22.01.06.47
 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256);
 Sat, 22 Dec 2018 01:06:48 -0800 (PST)
Date: Sat, 22 Dec 2018 11:06:44 +0200
From: Khaled Hosny <dr.khaled.hosny@HIDDEN>
To: Eli Zaretskii <eliz@HIDDEN>
Subject: Re: bug#33729: 27.0.50; Partial glyphs not rendered for Gujarati
 with Harfbuzz enabled (renders fine using m17n)
Message-ID: <20181222090644.GB2244@HIDDEN>
References: <CAFyQvY0pznh0yM9XJ9y_cEft47SXCmFp+O5rLkzb6o0By-MQuA@HIDDEN>
 <83h8fghcpo.fsf@HIDDEN>
 <20181214075056.GI2244@HIDDEN>
 <8336r0h1cb.fsf@HIDDEN>
 <20181214110316.GK2244@HIDDEN>
 <83y38sfcme.fsf@HIDDEN> <83tvjgf7ux.fsf@HIDDEN>
 <xk4lbd9erb.fsf@HIDDEN> <83mup4du5z.fsf@HIDDEN>
 <20181222085448.GA2244@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <20181222085448.GA2244@HIDDEN>
User-Agent: Mutt/1.11.1 (2018-12-01)
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 33729
Cc: 33729 <at> debbugs.gnu.org, Glenn Morris <rgm@HIDDEN>, kaushal.modi@HIDDEN,
 far.nasiri.m@HIDDEN, behdad@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

On Sat, Dec 22, 2018 at 10:54:48AM +0200, Khaled Hosny wrote:
> On Mon, Dec 17, 2018 at 05:55:52PM +0200, Eli Zaretskii wrote:
> > > From: Glenn Morris <rgm@HIDDEN>
> > > Cc: far.nasiri.m@HIDDEN,  dr.khaled.hosny@HIDDEN,  behdad@HIDDEN,  33729 <at> debbugs.gnu.org,  kaushal.modi@HIDDEN
> > > Date: Sun, 16 Dec 2018 19:30:00 -0500
> > > 
> > > > After some thinking, my conclusion is that we should import the
> > > > ISO 15924 database from https://unicode.org/iso15924/, use a script
> > > > similar to admin/unidata/blocks.awk to generate an alist from it that
> > > > maps Emacs script names to ISO 15924 tags, and then access that alist
> > > > from uni_script to get the correct script information to Harfbuzz.
> > > >
> > > > Patches implementing that are welcome.
> > > 
> > > I live to write awk scripts. I'm not 100% sure what you want, but as a
> > > first example, the following takes
> > > http://www.unicode.org/Public/UCD/latest/ucd/PropertyValueAliases.txt
> > > as input and outputs lines of the form "(gujr . gujarati)".
> > > 
> > > The aliases are so that the RHS matches charscript.el.
> > > 
> > > If this is not right, please clarify exactly what the inputs and output
> > > should be.
> > 
> > Thanks.
> > 
> > It turns out I didn't have this figured out completely, and your
> > proposal forced me to dig some more into the relevant parts of Unicode
> > and Emacs.  I found a few additional issues and considerations; for at
> > least some of them I'd like to hear the opinions of the Harfbuzz
> > developers.
> > 
> > Here are the issues:
> > 
> >  . Contrary to my original thoughts, I now tend to think that a
> >    separate char-table, say char-iso159240tag-table, that maps
> >    character codepoints directly to the script tags, is a better
> >    solution:
> >     - it will allow a faster look up, obviously
> >     - the subdivision of characters into scripts, as shown in
> >       Unicode's Scripts.txt, is slightly different from what
> >       char-script-table does, so a simple mapping from Emacs scripts
> >       to ISO 15924 script tag will not do.  For example, many
> >       characters Emacs puts into 'latin' or 'symbol' scripts are in
> >       the Common script according to Scripts.txt, and similarly for
> >       the Inherited script.  I imagine this is important for
> >       Harfbuzz.
> 
> Alternatively, we could just use HarfBuzz’s own built in ucdn-based
> Unicode function for this. The only reason for overriding this in Emacs
> was to keep HarfBuzz and Emacs Unicode support in sync, but if we are
> going to duplicate the Unicode script data then better use what HarfBuzz
> has.
> 
> I’m going to try this now.

I pushed a commit to harfbuzz branch that I think fixes this issue now.

Regards,
Khaled




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33729; Package emacs. Full text available.

Message received at 33729 <at> debbugs.gnu.org:


Received: (at 33729) by debbugs.gnu.org; 22 Dec 2018 08:55:01 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Dec 22 03:55:01 2018
Received: from localhost ([127.0.0.1]:59115 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gad3l-00031p-BA
	for submit <at> debbugs.gnu.org; Sat, 22 Dec 2018 03:55:01 -0500
Received: from mail-wr1-f43.google.com ([209.85.221.43]:43291)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dr.khaled.hosny@HIDDEN>) id 1gad3j-00031b-VR
 for 33729 <at> debbugs.gnu.org; Sat, 22 Dec 2018 03:55:00 -0500
Received: by mail-wr1-f43.google.com with SMTP id r10so7419539wrs.10
 for <33729 <at> debbugs.gnu.org>; Sat, 22 Dec 2018 00:54:59 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=date:from:to:cc:subject:message-id:references:mime-version
 :content-disposition:content-transfer-encoding:in-reply-to
 :user-agent; bh=rliGVy5bZ70psuVdS/D/r1p74GeUq82RaemItH03TYI=;
 b=YuEzu+sIfleHp5GNHEqHqh642SjR+XmFLP1PHOHmtt8qhVZf1Fl7jZoo85KYbaPuks
 ZKsXKSXtLqX56dWyFH7/ycmiAZDNqN5ostkOjpMQKY0XZTApfj5QkFKy65arp8SFZ0jw
 QgInCNBWGSSVOVnI1FeEFu4vvQRnYqSGkIBjM6G6dN8P7ODAiG6lr2Y6CUTqgWZJp/0l
 odw/N31zfWasGj2v13bcAueI1u4dbNq5Myi7deY2TBddA2dpO5fWpb0OZbSJhUdBDxLy
 8SmvO+PKykfNxFwKAH+WMPOsTmiWR5n607C5n30wjHdPgN8GfWQQ7Xk0HBTqH3E4LHiL
 oPWw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:date:from:to:cc:subject:message-id:references
 :mime-version:content-disposition:content-transfer-encoding
 :in-reply-to:user-agent;
 bh=rliGVy5bZ70psuVdS/D/r1p74GeUq82RaemItH03TYI=;
 b=CMJnk0wMlrjd+ntx3B1fLHwXXstWMDitJ7N3RG3qRBtDcv99PXQeRYNItiXUSxMZkx
 dVEYAEjTelC6LCwSvOFbb0YROyKTZAxzuWmj0LCuuW0Ahv4SFZNLXWqK9UiCxnBuhMRi
 7bd81eloqfpkGBE7KjLt3kIpnFgi2LFRVTYbpN7lE7hF16dIcbSpt3hoYgY/TcNvqysL
 ubqE9opRDCQfA0vOTVp9TR0xthWiIvtXW6EDSPTeqm5WpNxHPrKfIrFjEs8VSpALQ75j
 aZl2Kxy8uk61qUp+NT5XrLGPokQiI7YN7C5qDFrp1IHqIbQqTarUWRuj49Y+lFSFiCNP
 KJuQ==
X-Gm-Message-State: AJcUukc6XPwtRuPW55qAhC74oH4ZN9cw3lywIJprOY4vYR5HMkc8Zsgz
 n5gUIIf8gsfa+f/esSPNn1o=
X-Google-Smtp-Source: ALg8bN5R62WqAdCQ89223V0P/hfCbZBZSsyZ5DSOFOOmb9oHtu8seaovPZF+aEaLPEfFTPB+cvU+pg==
X-Received: by 2002:adf:c452:: with SMTP id a18mr5644838wrg.145.1545468894122; 
 Sat, 22 Dec 2018 00:54:54 -0800 (PST)
Received: from macbook.localdomain ([41.237.113.27])
 by smtp.gmail.com with ESMTPSA id c77sm16246250wmh.12.2018.12.22.00.54.51
 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256);
 Sat, 22 Dec 2018 00:54:52 -0800 (PST)
Date: Sat, 22 Dec 2018 10:54:48 +0200
From: Khaled Hosny <dr.khaled.hosny@HIDDEN>
To: Eli Zaretskii <eliz@HIDDEN>
Subject: Re: bug#33729: 27.0.50; Partial glyphs not rendered for Gujarati
 with Harfbuzz enabled (renders fine using m17n)
Message-ID: <20181222085448.GA2244@HIDDEN>
References: <20181213203102.GF2244@HIDDEN>
 <CAFyQvY0pznh0yM9XJ9y_cEft47SXCmFp+O5rLkzb6o0By-MQuA@HIDDEN>
 <83h8fghcpo.fsf@HIDDEN>
 <20181214075056.GI2244@HIDDEN>
 <8336r0h1cb.fsf@HIDDEN>
 <20181214110316.GK2244@HIDDEN>
 <83y38sfcme.fsf@HIDDEN> <83tvjgf7ux.fsf@HIDDEN>
 <xk4lbd9erb.fsf@HIDDEN> <83mup4du5z.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <83mup4du5z.fsf@HIDDEN>
User-Agent: Mutt/1.11.1 (2018-12-01)
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 33729
Cc: 33729 <at> debbugs.gnu.org, Glenn Morris <rgm@HIDDEN>, kaushal.modi@HIDDEN,
 far.nasiri.m@HIDDEN, behdad@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

On Mon, Dec 17, 2018 at 05:55:52PM +0200, Eli Zaretskii wrote:
> > From: Glenn Morris <rgm@HIDDEN>
> > Cc: far.nasiri.m@HIDDEN,  dr.khaled.hosny@HIDDEN,  behdad@HIDDEN,  33729 <at> debbugs.gnu.org,  kaushal.modi@HIDDEN
> > Date: Sun, 16 Dec 2018 19:30:00 -0500
> > 
> > > After some thinking, my conclusion is that we should import the
> > > ISO 15924 database from https://unicode.org/iso15924/, use a script
> > > similar to admin/unidata/blocks.awk to generate an alist from it that
> > > maps Emacs script names to ISO 15924 tags, and then access that alist
> > > from uni_script to get the correct script information to Harfbuzz.
> > >
> > > Patches implementing that are welcome.
> > 
> > I live to write awk scripts. I'm not 100% sure what you want, but as a
> > first example, the following takes
> > http://www.unicode.org/Public/UCD/latest/ucd/PropertyValueAliases.txt
> > as input and outputs lines of the form "(gujr . gujarati)".
> > 
> > The aliases are so that the RHS matches charscript.el.
> > 
> > If this is not right, please clarify exactly what the inputs and output
> > should be.
> 
> Thanks.
> 
> It turns out I didn't have this figured out completely, and your
> proposal forced me to dig some more into the relevant parts of Unicode
> and Emacs.  I found a few additional issues and considerations; for at
> least some of them I'd like to hear the opinions of the Harfbuzz
> developers.
> 
> Here are the issues:
> 
>  . Contrary to my original thoughts, I now tend to think that a
>    separate char-table, say char-iso159240tag-table, that maps
>    character codepoints directly to the script tags, is a better
>    solution:
>     - it will allow a faster look up, obviously
>     - the subdivision of characters into scripts, as shown in
>       Unicode's Scripts.txt, is slightly different from what
>       char-script-table does, so a simple mapping from Emacs scripts
>       to ISO 15924 script tag will not do.  For example, many
>       characters Emacs puts into 'latin' or 'symbol' scripts are in
>       the Common script according to Scripts.txt, and similarly for
>       the Inherited script.  I imagine this is important for
>       Harfbuzz.

Alternatively, we could just use HarfBuzz’s own built in ucdn-based
Unicode function for this. The only reason for overriding this in Emacs
was to keep HarfBuzz and Emacs Unicode support in sync, but if we are
going to duplicate the Unicode script data then better use what HarfBuzz
has.

I’m going to try this now.

>  . Whether to produce the character-to-script-tag mapping using the
>    UCD files, such as Scripts.txt and PropertyValueAliases.txt, or the
>    canonical ISO 15924 tags from https://unicode.org/iso15924/,
>    depends on whether the slight differences mentioned in
>    https://www.unicode.org/reports/tr24/#Relation_To_ISO15924 matter
>    for Harfbuzz.  For example, ISO 15924 has separate tags for the
>    Fraktur and Gaelic varieties of the Latin script: does this
>    distinction matter for Harfbuzz?

We want the UCD data.

Regards,
Khaled




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33729; Package emacs. Full text available.

Message received at 33729 <at> debbugs.gnu.org:


Received: (at 33729) by debbugs.gnu.org; 20 Dec 2018 20:46:12 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Dec 20 15:46:12 2018
Received: from localhost ([127.0.0.1]:56771 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1ga5Ct-0001yE-Ub
	for submit <at> debbugs.gnu.org; Thu, 20 Dec 2018 15:46:12 -0500
Received: from mail-lj1-f171.google.com ([209.85.208.171]:36175)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <behdad.esfahbod@HIDDEN>) id 1ga5Cr-0001y0-29
 for 33729 <at> debbugs.gnu.org; Thu, 20 Dec 2018 15:46:09 -0500
Received: by mail-lj1-f171.google.com with SMTP id g11-v6so2804269ljk.3
 for <33729 <at> debbugs.gnu.org>; Thu, 20 Dec 2018 12:46:08 -0800 (PST)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:mime-version:references:in-reply-to:from:date
 :message-id:subject:to:cc;
 bh=qiztYbbWTjmkNotaTEfjLPxecDdh6Y8MJ72zjmNgeRE=;
 b=nEH9heIpZvf8cfrdXOZBRHEYORYRLMSRc7eACZ/DLs8N540Lzbbj5nZ4txqr3ueihp
 c28pYsGIccDLCYWxZiGy73pE2WrKbfRloWOKhSFWqPNiBmWIYRHC0PWzUfbXv6wHdmBx
 KPqJ4UwmAEmNLl6kcKHmsw2h47ZrkXLmdWXeW9q8L/5YT2PRoetv6dcOoQoP01pVa3j1
 XQEDXSk/FrM2R/htoBuO9U4W0E4sJ9yflDwQgaSSH1CnRKiF8/2y+7D7JqY1w8p5ZBSz
 XkTd/s9jM6UEErcySpEtmGxH8c+m43K+rBEQavAKwxH7hHdIrUsYG1UTNO32H0vEPtZg
 DEDQ==
X-Gm-Message-State: AA+aEWZoQ7HzKL6+J2ue/j+uZTQvQonSnVHff9HyIVLwnGosvz0Ua8lo
 KcqjciybTchixVTAk+80kqHsGZpvlv8Ldw4Lp+c=
X-Google-Smtp-Source: AFSGD/WO7fqX0kO5W6YJY5W9DHMJBNd6iAnhQbvfdyNZuixlZh1fULf9t2jcDNaTogaRaokY/PVBn4JkDaTjfRjWQNI=
X-Received: by 2002:a2e:1b47:: with SMTP id
 b68-v6mr13847877ljb.104.1545338762795; 
 Thu, 20 Dec 2018 12:46:02 -0800 (PST)
MIME-Version: 1.0
References: <CAFyQvY3vfV9wqVWo3hAvawRAqpWUSGGf3tebwveM60OCtFkCvA@HIDDEN>
 <CAFyQvY2ZW38s4aMpkkfXLmTw_YrYJ8PRfaRPLugQt03tw0u7Zg@HIDDEN>
 <20181213203102.GF2244@HIDDEN>
 <CAFyQvY0pznh0yM9XJ9y_cEft47SXCmFp+O5rLkzb6o0By-MQuA@HIDDEN>
 <83h8fghcpo.fsf@HIDDEN> <20181214075056.GI2244@HIDDEN>
 <8336r0h1cb.fsf@HIDDEN> <20181214110316.GK2244@HIDDEN>
 <83y38sfcme.fsf@HIDDEN> <83tvjgf7ux.fsf@HIDDEN>
 <xk4lbd9erb.fsf@HIDDEN>
 <83mup4du5z.fsf@HIDDEN> <8336qsc9fl.fsf@HIDDEN>
In-Reply-To: <8336qsc9fl.fsf@HIDDEN>
From: Behdad Esfahbod <behdad@HIDDEN>
Date: Thu, 20 Dec 2018 15:45:50 -0500
Message-ID: <CAF63+7VBMXYrnhp9R9ALshoQuYtnsMeScqgzHg3NDgs2qzCDeA@HIDDEN>
Subject: Re: bug#33729: 27.0.50; Partial glyphs not rendered for Gujarati with
 Harfbuzz enabled (renders fine using m17n)
To: Eli Zaretskii <eliz@HIDDEN>
Content-Type: multipart/alternative; boundary="000000000000039de7057d7a3814"
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 33729
Cc: Khaled Hosny <dr.khaled.hosny@HIDDEN>, rgm@HIDDEN,
 kaushal.modi@HIDDEN, Mohammad Nasirifar <far.nasiri.m@HIDDEN>,
 33729 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

--000000000000039de7057d7a3814
Content-Type: text/plain; charset="UTF-8"

Sounds good to me.

On Thu, Dec 20, 2018 at 1:58 PM Eli Zaretskii <eliz@HIDDEN> wrote:

> Ping!  Could someone on the Harfbuzz team please comment on the
> thoughts below?  Khaled, Mohammad, Behdad?
>
> > Date: Mon, 17 Dec 2018 17:55:52 +0200
> > From: Eli Zaretskii <eliz@HIDDEN>
> > Cc: dr.khaled.hosny@HIDDEN, behdad@HIDDEN, 33729 <at> debbugs.gnu.org,
> >       far.nasiri.m@HIDDEN, kaushal.modi@HIDDEN
> >
> > > From: Glenn Morris <rgm@HIDDEN>
> > > Cc: far.nasiri.m@HIDDEN,  dr.khaled.hosny@HIDDEN,
> behdad@HIDDEN,  33729 <at> debbugs.gnu.org,  kaushal.modi@HIDDEN
> > > Date: Sun, 16 Dec 2018 19:30:00 -0500
> > >
> > > > After some thinking, my conclusion is that we should import the
> > > > ISO 15924 database from https://unicode.org/iso15924/, use a script
> > > > similar to admin/unidata/blocks.awk to generate an alist from it that
> > > > maps Emacs script names to ISO 15924 tags, and then access that alist
> > > > from uni_script to get the correct script information to Harfbuzz.
> > > >
> > > > Patches implementing that are welcome.
> > >
> > > I live to write awk scripts. I'm not 100% sure what you want, but as a
> > > first example, the following takes
> > > http://www.unicode.org/Public/UCD/latest/ucd/PropertyValueAliases.txt
> > > as input and outputs lines of the form "(gujr . gujarati)".
> > >
> > > The aliases are so that the RHS matches charscript.el.
> > >
> > > If this is not right, please clarify exactly what the inputs and output
> > > should be.
> >
> > Thanks.
> >
> > It turns out I didn't have this figured out completely, and your
> > proposal forced me to dig some more into the relevant parts of Unicode
> > and Emacs.  I found a few additional issues and considerations; for at
> > least some of them I'd like to hear the opinions of the Harfbuzz
> > developers.
> >
> > Here are the issues:
> >
> >  . Contrary to my original thoughts, I now tend to think that a
> >    separate char-table, say char-iso159240tag-table, that maps
> >    character codepoints directly to the script tags, is a better
> >    solution:
> >     - it will allow a faster look up, obviously
> >     - the subdivision of characters into scripts, as shown in
> >       Unicode's Scripts.txt, is slightly different from what
> >       char-script-table does, so a simple mapping from Emacs scripts
> >       to ISO 15924 script tag will not do.  For example, many
> >       characters Emacs puts into 'latin' or 'symbol' scripts are in
> >       the Common script according to Scripts.txt, and similarly for
> >       the Inherited script.  I imagine this is important for
> >       Harfbuzz.
> >
> >  . Whether to produce the character-to-script-tag mapping using the
> >    UCD files, such as Scripts.txt and PropertyValueAliases.txt, or the
> >    canonical ISO 15924 tags from https://unicode.org/iso15924/,
> >    depends on whether the slight differences mentioned in
> >    https://www.unicode.org/reports/tr24/#Relation_To_ISO15924 matter
> >    for Harfbuzz.  For example, ISO 15924 has separate tags for the
> >    Fraktur and Gaelic varieties of the Latin script: does this
> >    distinction matter for Harfbuzz?
> >
> >  . Does Harfbuzz handle the issues mentioned in
> >    https://www.unicode.org/reports/tr24/#Script_Anomalies, and in
> >    particular the use case of decomposed characters which yield a
> >    different script than their precomposed variants?  This use case is
> >    quite common in handling of character compositions, so it's
> >    important to understand its implications before we decide on the
> >    implementation.
> >
> > To summarize, unless the Harfbuzz guys advise differently, I'd prefer
> > processing Scripts.txt and PropertyValueAliases.txt into a list
> > similar to the one we produce in charscript.el, then generate a
> > char-table from that list.
> >
> > Thanks again for working on this.
> >
> >
> >
> >
>


-- 
behdad
http://behdad.org/

--000000000000039de7057d7a3814
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Sounds good to me.</div><br><div class=3D"gmail_quote"><di=
v dir=3D"ltr">On Thu, Dec 20, 2018 at 1:58 PM Eli Zaretskii &lt;<a href=3D"=
mailto:eliz@HIDDEN">eliz@HIDDEN</a>&gt; wrote:<br></div><blockquote class=
=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rg=
b(204,204,204);padding-left:1ex">Ping!=C2=A0 Could someone on the Harfbuzz =
team please comment on the<br>
thoughts below?=C2=A0 Khaled, Mohammad, Behdad?<br>
<br>
&gt; Date: Mon, 17 Dec 2018 17:55:52 +0200<br>
&gt; From: Eli Zaretskii &lt;<a href=3D"mailto:eliz@HIDDEN" target=3D"_bla=
nk">eliz@HIDDEN</a>&gt;<br>
&gt; Cc: <a href=3D"mailto:dr.khaled.hosny@HIDDEN" target=3D"_blank">dr.=
khaled.hosny@HIDDEN</a>, <a href=3D"mailto:behdad@HIDDEN" target=3D"=
_blank">behdad@HIDDEN</a>, <a href=3D"mailto:33729 <at> debbugs.gnu.org" tar=
get=3D"_blank">33729 <at> debbugs.gnu.org</a>,<br>
&gt;=C2=A0 =C2=A0 =C2=A0 =C2=A0<a href=3D"mailto:far.nasiri.m@HIDDEN" ta=
rget=3D"_blank">far.nasiri.m@HIDDEN</a>, <a href=3D"mailto:kaushal.modi@=
gmail.com" target=3D"_blank">kaushal.modi@HIDDEN</a><br>
&gt; <br>
&gt; &gt; From: Glenn Morris &lt;<a href=3D"mailto:rgm@HIDDEN" target=3D"_=
blank">rgm@HIDDEN</a>&gt;<br>
&gt; &gt; Cc: <a href=3D"mailto:far.nasiri.m@HIDDEN" target=3D"_blank">f=
ar.nasiri.m@HIDDEN</a>,=C2=A0 <a href=3D"mailto:dr.khaled.hosny@HIDDEN=
m" target=3D"_blank">dr.khaled.hosny@HIDDEN</a>,=C2=A0 <a href=3D"mailto=
:behdad@HIDDEN" target=3D"_blank">behdad@HIDDEN</a>,=C2=A0 <a href=
=3D"mailto:33729 <at> debbugs.gnu.org" target=3D"_blank">33729 <at> debbugs.gnu.org</=
a>,=C2=A0 <a href=3D"mailto:kaushal.modi@HIDDEN" target=3D"_blank">kaush=
al.modi@HIDDEN</a><br>
&gt; &gt; Date: Sun, 16 Dec 2018 19:30:00 -0500<br>
&gt; &gt; <br>
&gt; &gt; &gt; After some thinking, my conclusion is that we should import =
the<br>
&gt; &gt; &gt; ISO 15924 database from <a href=3D"https://unicode.org/iso15=
924/" rel=3D"noreferrer" target=3D"_blank">https://unicode.org/iso15924/</a=
>, use a script<br>
&gt; &gt; &gt; similar to admin/unidata/blocks.awk to generate an alist fro=
m it that<br>
&gt; &gt; &gt; maps Emacs script names to ISO 15924 tags, and then access t=
hat alist<br>
&gt; &gt; &gt; from uni_script to get the correct script information to Har=
fbuzz.<br>
&gt; &gt; &gt;<br>
&gt; &gt; &gt; Patches implementing that are welcome.<br>
&gt; &gt; <br>
&gt; &gt; I live to write awk scripts. I&#39;m not 100% sure what you want,=
 but as a<br>
&gt; &gt; first example, the following takes<br>
&gt; &gt; <a href=3D"http://www.unicode.org/Public/UCD/latest/ucd/PropertyV=
alueAliases.txt" rel=3D"noreferrer" target=3D"_blank">http://www.unicode.or=
g/Public/UCD/latest/ucd/PropertyValueAliases.txt</a><br>
&gt; &gt; as input and outputs lines of the form &quot;(gujr . gujarati)&qu=
ot;.<br>
&gt; &gt; <br>
&gt; &gt; The aliases are so that the RHS matches charscript.el.<br>
&gt; &gt; <br>
&gt; &gt; If this is not right, please clarify exactly what the inputs and =
output<br>
&gt; &gt; should be.<br>
&gt; <br>
&gt; Thanks.<br>
&gt; <br>
&gt; It turns out I didn&#39;t have this figured out completely, and your<b=
r>
&gt; proposal forced me to dig some more into the relevant parts of Unicode=
<br>
&gt; and Emacs.=C2=A0 I found a few additional issues and considerations; f=
or at<br>
&gt; least some of them I&#39;d like to hear the opinions of the Harfbuzz<b=
r>
&gt; developers.<br>
&gt; <br>
&gt; Here are the issues:<br>
&gt; <br>
&gt;=C2=A0 . Contrary to my original thoughts, I now tend to think that a<b=
r>
&gt;=C2=A0 =C2=A0 separate char-table, say char-iso159240tag-table, that ma=
ps<br>
&gt;=C2=A0 =C2=A0 character codepoints directly to the script tags, is a be=
tter<br>
&gt;=C2=A0 =C2=A0 solution:<br>
&gt;=C2=A0 =C2=A0 =C2=A0- it will allow a faster look up, obviously<br>
&gt;=C2=A0 =C2=A0 =C2=A0- the subdivision of characters into scripts, as sh=
own in<br>
&gt;=C2=A0 =C2=A0 =C2=A0 =C2=A0Unicode&#39;s Scripts.txt, is slightly diffe=
rent from what<br>
&gt;=C2=A0 =C2=A0 =C2=A0 =C2=A0char-script-table does, so a simple mapping =
from Emacs scripts<br>
&gt;=C2=A0 =C2=A0 =C2=A0 =C2=A0to ISO 15924 script tag will not do.=C2=A0 F=
or example, many<br>
&gt;=C2=A0 =C2=A0 =C2=A0 =C2=A0characters Emacs puts into &#39;latin&#39; o=
r &#39;symbol&#39; scripts are in<br>
&gt;=C2=A0 =C2=A0 =C2=A0 =C2=A0the Common script according to Scripts.txt, =
and similarly for<br>
&gt;=C2=A0 =C2=A0 =C2=A0 =C2=A0the Inherited script.=C2=A0 I imagine this i=
s important for<br>
&gt;=C2=A0 =C2=A0 =C2=A0 =C2=A0Harfbuzz.<br>
&gt; <br>
&gt;=C2=A0 . Whether to produce the character-to-script-tag mapping using t=
he<br>
&gt;=C2=A0 =C2=A0 UCD files, such as Scripts.txt and PropertyValueAliases.t=
xt, or the<br>
&gt;=C2=A0 =C2=A0 canonical ISO 15924 tags from <a href=3D"https://unicode.=
org/iso15924/" rel=3D"noreferrer" target=3D"_blank">https://unicode.org/iso=
15924/</a>,<br>
&gt;=C2=A0 =C2=A0 depends on whether the slight differences mentioned in<br=
>
&gt;=C2=A0 =C2=A0 <a href=3D"https://www.unicode.org/reports/tr24/#Relation=
_To_ISO15924" rel=3D"noreferrer" target=3D"_blank">https://www.unicode.org/=
reports/tr24/#Relation_To_ISO15924</a> matter<br>
&gt;=C2=A0 =C2=A0 for Harfbuzz.=C2=A0 For example, ISO 15924 has separate t=
ags for the<br>
&gt;=C2=A0 =C2=A0 Fraktur and Gaelic varieties of the Latin script: does th=
is<br>
&gt;=C2=A0 =C2=A0 distinction matter for Harfbuzz?<br>
&gt; <br>
&gt;=C2=A0 . Does Harfbuzz handle the issues mentioned in<br>
&gt;=C2=A0 =C2=A0 <a href=3D"https://www.unicode.org/reports/tr24/#Script_A=
nomalies" rel=3D"noreferrer" target=3D"_blank">https://www.unicode.org/repo=
rts/tr24/#Script_Anomalies</a>, and in<br>
&gt;=C2=A0 =C2=A0 particular the use case of decomposed characters which yi=
eld a<br>
&gt;=C2=A0 =C2=A0 different script than their precomposed variants?=C2=A0 T=
his use case is<br>
&gt;=C2=A0 =C2=A0 quite common in handling of character compositions, so it=
&#39;s<br>
&gt;=C2=A0 =C2=A0 important to understand its implications before we decide=
 on the<br>
&gt;=C2=A0 =C2=A0 implementation.<br>
&gt; <br>
&gt; To summarize, unless the Harfbuzz guys advise differently, I&#39;d pre=
fer<br>
&gt; processing Scripts.txt and PropertyValueAliases.txt into a list<br>
&gt; similar to the one we produce in charscript.el, then generate a<br>
&gt; char-table from that list.<br>
&gt; <br>
&gt; Thanks again for working on this.<br>
&gt; <br>
&gt; <br>
&gt; <br>
&gt; <br>
</blockquote></div><br clear=3D"all"><div><br></div>-- <br><div dir=3D"ltr"=
 class=3D"gmail_signature">behdad<br><a href=3D"http://behdad.org/" target=
=3D"_blank">http://behdad.org/</a></div>

--000000000000039de7057d7a3814--




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33729; Package emacs. Full text available.

Message received at 33729 <at> debbugs.gnu.org:


Received: (at 33729) by debbugs.gnu.org; 20 Dec 2018 18:58:21 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Dec 20 13:58:21 2018
Received: from localhost ([127.0.0.1]:56737 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1ga3WX-0003nc-64
	for submit <at> debbugs.gnu.org; Thu, 20 Dec 2018 13:58:21 -0500
Received: from eggs.gnu.org ([208.118.235.92]:57740)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1ga3WU-0003nO-Tr
 for 33729 <at> debbugs.gnu.org; Thu, 20 Dec 2018 13:58:19 -0500
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <eliz@HIDDEN>) id 1ga3WO-00013D-Jm
 for 33729 <at> debbugs.gnu.org; Thu, 20 Dec 2018 13:58:13 -0500
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50 autolearn=disabled
 version=3.3.2
Received: from fencepost.gnu.org ([2001:4830:134:3::e]:42745)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>)
 id 1ga3WJ-0000yC-Mq; Thu, 20 Dec 2018 13:58:07 -0500
Received: from [176.228.60.248] (port=3934 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1ga3WH-0000tp-Ox; Thu, 20 Dec 2018 13:58:07 -0500
Date: Thu, 20 Dec 2018 20:58:06 +0200
Message-Id: <8336qsc9fl.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: dr.khaled.hosny@HIDDEN, behdad@HIDDEN,, far.nasiri.m@HIDDEN
In-reply-to: <83mup4du5z.fsf@HIDDEN> (message from Eli Zaretskii on Mon, 17
 Dec 2018 17:55:52 +0200)
Subject: Re: bug#33729: 27.0.50;
 Partial glyphs not rendered for Gujarati with Harfbuzz enabled
 (renders fine using m17n)
References: <CAFyQvY3vfV9wqVWo3hAvawRAqpWUSGGf3tebwveM60OCtFkCvA@HIDDEN>
 <CAFyQvY2ZW38s4aMpkkfXLmTw_YrYJ8PRfaRPLugQt03tw0u7Zg@HIDDEN>
 <20181213203102.GF2244@HIDDEN>
 <CAFyQvY0pznh0yM9XJ9y_cEft47SXCmFp+O5rLkzb6o0By-MQuA@HIDDEN>
 <83h8fghcpo.fsf@HIDDEN> <20181214075056.GI2244@HIDDEN>
 <8336r0h1cb.fsf@HIDDEN> <20181214110316.GK2244@HIDDEN>
 <83y38sfcme.fsf@HIDDEN> <83tvjgf7ux.fsf@HIDDEN>
 <xk4lbd9erb.fsf@HIDDEN> <83mup4du5z.fsf@HIDDEN>
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2001:4830:134:3::e
X-Spam-Score: -5.0 (-----)
X-Debbugs-Envelope-To: 33729
Cc: rgm@HIDDEN, 33729 <at> debbugs.gnu.org, kaushal.modi@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -6.0 (------)

Ping!  Could someone on the Harfbuzz team please comment on the
thoughts below?  Khaled, Mohammad, Behdad?

> Date: Mon, 17 Dec 2018 17:55:52 +0200
> From: Eli Zaretskii <eliz@HIDDEN>
> Cc: dr.khaled.hosny@HIDDEN, behdad@HIDDEN, 33729 <at> debbugs.gnu.org,
> 	far.nasiri.m@HIDDEN, kaushal.modi@HIDDEN
> 
> > From: Glenn Morris <rgm@HIDDEN>
> > Cc: far.nasiri.m@HIDDEN,  dr.khaled.hosny@HIDDEN,  behdad@HIDDEN,  33729 <at> debbugs.gnu.org,  kaushal.modi@HIDDEN
> > Date: Sun, 16 Dec 2018 19:30:00 -0500
> > 
> > > After some thinking, my conclusion is that we should import the
> > > ISO 15924 database from https://unicode.org/iso15924/, use a script
> > > similar to admin/unidata/blocks.awk to generate an alist from it that
> > > maps Emacs script names to ISO 15924 tags, and then access that alist
> > > from uni_script to get the correct script information to Harfbuzz.
> > >
> > > Patches implementing that are welcome.
> > 
> > I live to write awk scripts. I'm not 100% sure what you want, but as a
> > first example, the following takes
> > http://www.unicode.org/Public/UCD/latest/ucd/PropertyValueAliases.txt
> > as input and outputs lines of the form "(gujr . gujarati)".
> > 
> > The aliases are so that the RHS matches charscript.el.
> > 
> > If this is not right, please clarify exactly what the inputs and output
> > should be.
> 
> Thanks.
> 
> It turns out I didn't have this figured out completely, and your
> proposal forced me to dig some more into the relevant parts of Unicode
> and Emacs.  I found a few additional issues and considerations; for at
> least some of them I'd like to hear the opinions of the Harfbuzz
> developers.
> 
> Here are the issues:
> 
>  . Contrary to my original thoughts, I now tend to think that a
>    separate char-table, say char-iso159240tag-table, that maps
>    character codepoints directly to the script tags, is a better
>    solution:
>     - it will allow a faster look up, obviously
>     - the subdivision of characters into scripts, as shown in
>       Unicode's Scripts.txt, is slightly different from what
>       char-script-table does, so a simple mapping from Emacs scripts
>       to ISO 15924 script tag will not do.  For example, many
>       characters Emacs puts into 'latin' or 'symbol' scripts are in
>       the Common script according to Scripts.txt, and similarly for
>       the Inherited script.  I imagine this is important for
>       Harfbuzz.
> 
>  . Whether to produce the character-to-script-tag mapping using the
>    UCD files, such as Scripts.txt and PropertyValueAliases.txt, or the
>    canonical ISO 15924 tags from https://unicode.org/iso15924/,
>    depends on whether the slight differences mentioned in
>    https://www.unicode.org/reports/tr24/#Relation_To_ISO15924 matter
>    for Harfbuzz.  For example, ISO 15924 has separate tags for the
>    Fraktur and Gaelic varieties of the Latin script: does this
>    distinction matter for Harfbuzz?
> 
>  . Does Harfbuzz handle the issues mentioned in
>    https://www.unicode.org/reports/tr24/#Script_Anomalies, and in
>    particular the use case of decomposed characters which yield a
>    different script than their precomposed variants?  This use case is
>    quite common in handling of character compositions, so it's
>    important to understand its implications before we decide on the
>    implementation.
> 
> To summarize, unless the Harfbuzz guys advise differently, I'd prefer
> processing Scripts.txt and PropertyValueAliases.txt into a list
> similar to the one we produce in charscript.el, then generate a
> char-table from that list.
> 
> Thanks again for working on this.
> 
> 
> 
> 




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33729; Package emacs. Full text available.

Message received at 33729 <at> debbugs.gnu.org:


Received: (at 33729) by debbugs.gnu.org; 17 Dec 2018 15:56:04 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Dec 17 10:56:04 2018
Received: from localhost ([127.0.0.1]:52004 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gYvFT-00011K-Mr
	for submit <at> debbugs.gnu.org; Mon, 17 Dec 2018 10:56:04 -0500
Received: from eggs.gnu.org ([208.118.235.92]:60549)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1gYvFR-00010o-OE
 for 33729 <at> debbugs.gnu.org; Mon, 17 Dec 2018 10:56:02 -0500
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <eliz@HIDDEN>) id 1gYvFK-0005tV-NZ
 for 33729 <at> debbugs.gnu.org; Mon, 17 Dec 2018 10:55:56 -0500
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50 autolearn=disabled
 version=3.3.2
Received: from fencepost.gnu.org ([2001:4830:134:3::e]:38770)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>)
 id 1gYvFD-0005nl-GS; Mon, 17 Dec 2018 10:55:47 -0500
Received: from [176.228.60.248] (port=3863 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1gYvFB-0005hs-UW; Mon, 17 Dec 2018 10:55:46 -0500
Date: Mon, 17 Dec 2018 17:55:52 +0200
Message-Id: <83mup4du5z.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Glenn Morris <rgm@HIDDEN>
In-reply-to: <xk4lbd9erb.fsf@HIDDEN> (message from Glenn Morris on
 Sun, 16 Dec 2018 19:30:00 -0500)
Subject: Re: bug#33729: 27.0.50;
 Partial glyphs not rendered for Gujarati with Harfbuzz enabled
 (renders fine using m17n)
References: <CAFyQvY3vfV9wqVWo3hAvawRAqpWUSGGf3tebwveM60OCtFkCvA@HIDDEN>
 <CAFyQvY2ZW38s4aMpkkfXLmTw_YrYJ8PRfaRPLugQt03tw0u7Zg@HIDDEN>
 <20181213203102.GF2244@HIDDEN>
 <CAFyQvY0pznh0yM9XJ9y_cEft47SXCmFp+O5rLkzb6o0By-MQuA@HIDDEN>
 <83h8fghcpo.fsf@HIDDEN> <20181214075056.GI2244@HIDDEN>
 <8336r0h1cb.fsf@HIDDEN> <20181214110316.GK2244@HIDDEN>
 <83y38sfcme.fsf@HIDDEN> <83tvjgf7ux.fsf@HIDDEN>
 <xk4lbd9erb.fsf@HIDDEN>
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2001:4830:134:3::e
X-Spam-Score: -5.0 (-----)
X-Debbugs-Envelope-To: 33729
Cc: dr.khaled.hosny@HIDDEN, behdad@HIDDEN, kaushal.modi@HIDDEN,
 far.nasiri.m@HIDDEN, 33729 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -6.0 (------)

> From: Glenn Morris <rgm@HIDDEN>
> Cc: far.nasiri.m@HIDDEN,  dr.khaled.hosny@HIDDEN,  behdad@HIDDEN,  33729 <at> debbugs.gnu.org,  kaushal.modi@HIDDEN
> Date: Sun, 16 Dec 2018 19:30:00 -0500
> 
> > After some thinking, my conclusion is that we should import the
> > ISO 15924 database from https://unicode.org/iso15924/, use a script
> > similar to admin/unidata/blocks.awk to generate an alist from it that
> > maps Emacs script names to ISO 15924 tags, and then access that alist
> > from uni_script to get the correct script information to Harfbuzz.
> >
> > Patches implementing that are welcome.
> 
> I live to write awk scripts. I'm not 100% sure what you want, but as a
> first example, the following takes
> http://www.unicode.org/Public/UCD/latest/ucd/PropertyValueAliases.txt
> as input and outputs lines of the form "(gujr . gujarati)".
> 
> The aliases are so that the RHS matches charscript.el.
> 
> If this is not right, please clarify exactly what the inputs and output
> should be.

Thanks.

It turns out I didn't have this figured out completely, and your
proposal forced me to dig some more into the relevant parts of Unicode
and Emacs.  I found a few additional issues and considerations; for at
least some of them I'd like to hear the opinions of the Harfbuzz
developers.

Here are the issues:

 . Contrary to my original thoughts, I now tend to think that a
   separate char-table, say char-iso159240tag-table, that maps
   character codepoints directly to the script tags, is a better
   solution:
    - it will allow a faster look up, obviously
    - the subdivision of characters into scripts, as shown in
      Unicode's Scripts.txt, is slightly different from what
      char-script-table does, so a simple mapping from Emacs scripts
      to ISO 15924 script tag will not do.  For example, many
      characters Emacs puts into 'latin' or 'symbol' scripts are in
      the Common script according to Scripts.txt, and similarly for
      the Inherited script.  I imagine this is important for
      Harfbuzz.

 . Whether to produce the character-to-script-tag mapping using the
   UCD files, such as Scripts.txt and PropertyValueAliases.txt, or the
   canonical ISO 15924 tags from https://unicode.org/iso15924/,
   depends on whether the slight differences mentioned in
   https://www.unicode.org/reports/tr24/#Relation_To_ISO15924 matter
   for Harfbuzz.  For example, ISO 15924 has separate tags for the
   Fraktur and Gaelic varieties of the Latin script: does this
   distinction matter for Harfbuzz?

 . Does Harfbuzz handle the issues mentioned in
   https://www.unicode.org/reports/tr24/#Script_Anomalies, and in
   particular the use case of decomposed characters which yield a
   different script than their precomposed variants?  This use case is
   quite common in handling of character compositions, so it's
   important to understand its implications before we decide on the
   implementation.

To summarize, unless the Harfbuzz guys advise differently, I'd prefer
processing Scripts.txt and PropertyValueAliases.txt into a list
similar to the one we produce in charscript.el, then generate a
char-table from that list.

Thanks again for working on this.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33729; Package emacs. Full text available.

Message received at 33729 <at> debbugs.gnu.org:


Received: (at 33729) by debbugs.gnu.org; 17 Dec 2018 00:30:17 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Dec 16 19:30:17 2018
Received: from localhost ([127.0.0.1]:50698 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gYgnY-00073t-QY
	for submit <at> debbugs.gnu.org; Sun, 16 Dec 2018 19:30:17 -0500
Received: from eggs.gnu.org ([208.118.235.92]:35860)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <rgm@HIDDEN>) id 1gYgnW-00073e-Dv
 for 33729 <at> debbugs.gnu.org; Sun, 16 Dec 2018 19:30:14 -0500
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <rgm@HIDDEN>) id 1gYgnQ-00038r-B9
 for 33729 <at> debbugs.gnu.org; Sun, 16 Dec 2018 19:30:09 -0500
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=-0.0 required=5.0 tests=BAYES_40 autolearn=disabled
 version=3.3.2
Received: from fencepost.gnu.org ([2001:4830:134:3::e]:52049)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <rgm@HIDDEN>)
 id 1gYgnJ-0002vx-G7; Sun, 16 Dec 2018 19:30:01 -0500
Received: from rgm by fencepost.gnu.org with local (Exim 4.82)
 (envelope-from <rgm@HIDDEN>)
 id 1gYgnI-0000qx-7A; Sun, 16 Dec 2018 19:30:00 -0500
From: Glenn Morris <rgm@HIDDEN>
To: Eli Zaretskii <eliz@HIDDEN>
Subject: Re: bug#33729: 27.0.50;
 Partial glyphs not rendered for Gujarati with Harfbuzz enabled
 (renders fine using m17n)
References: <CAFyQvY3vfV9wqVWo3hAvawRAqpWUSGGf3tebwveM60OCtFkCvA@HIDDEN>
 <CAFyQvY2ZW38s4aMpkkfXLmTw_YrYJ8PRfaRPLugQt03tw0u7Zg@HIDDEN>
 <20181213203102.GF2244@HIDDEN>
 <CAFyQvY0pznh0yM9XJ9y_cEft47SXCmFp+O5rLkzb6o0By-MQuA@HIDDEN>
 <83h8fghcpo.fsf@HIDDEN> <20181214075056.GI2244@HIDDEN>
 <8336r0h1cb.fsf@HIDDEN> <20181214110316.GK2244@HIDDEN>
 <83y38sfcme.fsf@HIDDEN> <83tvjgf7ux.fsf@HIDDEN>
X-Spook: H5N1 SBI Abu Sayyaf Cyber Command Emergency Relief
X-Ran: Ca%JpP]j<g[iGb"$x<D%|V3&E;E\I@N;<w[v40iYi%*|Ssly+;p,4Lo4,v-U{]O?iik3{$
X-Hue: yellow
X-Debbugs-No-Ack: yes
X-Attribution: GM
Date: Sun, 16 Dec 2018 19:30:00 -0500
In-Reply-To: <83tvjgf7ux.fsf@HIDDEN> (Eli Zaretskii's message of "Fri, 14 Dec
 2018 17:25:42 +0200")
Message-ID: <xk4lbd9erb.fsf@HIDDEN>
User-Agent: Gnus (www.gnus.org), GNU Emacs (www.gnu.org/software/emacs/)
MIME-Version: 1.0
Content-Type: text/plain
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2001:4830:134:3::e
X-Spam-Score: -5.0 (-----)
X-Debbugs-Envelope-To: 33729
Cc: dr.khaled.hosny@HIDDEN, behdad@HIDDEN, kaushal.modi@HIDDEN,
 far.nasiri.m@HIDDEN, 33729 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -6.0 (------)

Eli Zaretskii wrote:

> After some thinking, my conclusion is that we should import the
> ISO 15924 database from https://unicode.org/iso15924/, use a script
> similar to admin/unidata/blocks.awk to generate an alist from it that
> maps Emacs script names to ISO 15924 tags, and then access that alist
> from uni_script to get the correct script information to Harfbuzz.
>
> Patches implementing that are welcome.

I live to write awk scripts. I'm not 100% sure what you want, but as a
first example, the following takes
http://www.unicode.org/Public/UCD/latest/ucd/PropertyValueAliases.txt
as input and outputs lines of the form "(gujr . gujarati)".

The aliases are so that the RHS matches charscript.el.

If this is not right, please clarify exactly what the inputs and output
should be.

#!/usr/bin/awk -f

function name2alias(name) {
    name = tolower(name)
    if (name ~ /arabic/) return "arabic"
    else if (name ~ /aramaic/) return "aramaic"
    else if (name ~ /cypriot/) return "cypriot-syllabary"
    else if (name ~ /katakana|hiragana/) return "kana"
    else if (name ~ /myanmar/) return "burmese"
    else if (name ~ /duployan|shorthand/) return "duployan-shorthand"
    else if (name ~ /signwriting/) return "sutton-sign-writing"
    sub(/^new_/, "", name)
    sub(/_(hieroglyphs|cursive)$/, "", name)
    gsub(/_/,"-",name)
    return name
}


$1 == "sc" {
    abbrev = tolower($3)
    alias = name2alias($5)
    if (alias ~ /^inherited|common|unknown/) next
    print "(" abbrev, ".", alias ")"
}




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33729; Package emacs. Full text available.

Message received at 33729 <at> debbugs.gnu.org:


Received: (at 33729) by debbugs.gnu.org; 16 Dec 2018 14:48:14 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Dec 16 09:48:14 2018
Received: from localhost ([127.0.0.1]:49993 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gYXiI-0004wO-Ew
	for submit <at> debbugs.gnu.org; Sun, 16 Dec 2018 09:48:14 -0500
Received: from odoacer.turtle-trading.net ([93.241.193.16]:37145)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <benny@HIDDEN>) id 1gYXiG-0004w9-GN
 for 33729 <at> debbugs.gnu.org; Sun, 16 Dec 2018 09:48:13 -0500
Received: from justinian.turtle-trading.net ([192.168.2.118])
 by odoacer.turtle-trading.net with esmtp (Exim 4.80)
 (envelope-from <benny@HIDDEN>)
 id 1gYXi1-0007Vm-B0; Sun, 16 Dec 2018 15:47:57 +0100
Received: from benny by justinian.turtle-trading.net with local (Exim 4.84_2)
 (envelope-from <benny@HIDDEN>)
 id 1gYXi1-0005NA-6V; Sun, 16 Dec 2018 15:47:57 +0100
From: Benjamin Riefenstahl <b.riefenstahl@HIDDEN>
To: Khaled Hosny <dr.khaled.hosny@HIDDEN>
Subject: Re: bug#33729: 27.0.50;
 Partial glyphs not rendered for Gujarati with Harfbuzz enabled
 (renders fine using m17n)
References: <CAFyQvY3vfV9wqVWo3hAvawRAqpWUSGGf3tebwveM60OCtFkCvA@HIDDEN>
 <CAFyQvY2ZW38s4aMpkkfXLmTw_YrYJ8PRfaRPLugQt03tw0u7Zg@HIDDEN>
 <20181213203102.GF2244@HIDDEN>
 <CAFyQvY0pznh0yM9XJ9y_cEft47SXCmFp+O5rLkzb6o0By-MQuA@HIDDEN>
 <83h8fghcpo.fsf@HIDDEN> <20181214075056.GI2244@HIDDEN>
 <8336r0h1cb.fsf@HIDDEN> <20181214110316.GK2244@HIDDEN>
Date: Sun, 16 Dec 2018 15:47:56 +0100
In-Reply-To: <20181214110316.GK2244@HIDDEN> (Khaled Hosny's
 message of "Fri, 14 Dec 2018 13:03:16 +0200")
Message-ID: <87y38pcyub.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 33729
Cc: behdad@HIDDEN, Eli Zaretskii <eliz@HIDDEN>, 33729 <at> debbugs.gnu.org,
 far.nasiri.m@HIDDEN, kaushal.modi@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Khaled Hosny writes:
> /* FIXME: from_string wants an ISO 15924 script tag here. */
>
> As we discussed earlier, hb_script_from_string() expects ISO 15924
> script tags, but char_script_table does not provide such tags (I don=E2=
=80=99t
> recall what it does provide exactly). We need a way to get ISO 15924
> script tags from Emacs.

The same mismatch also prevents Syriac text from actually shaping.
Syriac shaping works in m17n with the required setup in
composition-function-table and using the Meltho fonts.  With Harfbuzz it
doesn't work, unless I change "syriac" to "syrc" in charscript.el, just
for testing of course.

As a success story OTOH, Mandaic, using the Noto font, works OOTB ;-)

benny




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33729; Package emacs. Full text available.

Message received at 33729 <at> debbugs.gnu.org:


Received: (at 33729) by debbugs.gnu.org; 14 Dec 2018 22:47:58 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Dec 14 17:47:58 2018
Received: from localhost ([127.0.0.1]:48643 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gXwFR-0004Ht-MT
	for submit <at> debbugs.gnu.org; Fri, 14 Dec 2018 17:47:57 -0500
Received: from mail-wm1-f41.google.com ([209.85.128.41]:39020)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dr.khaled.hosny@HIDDEN>) id 1gXwFP-0004Hg-Ix
 for 33729 <at> debbugs.gnu.org; Fri, 14 Dec 2018 17:47:55 -0500
Received: by mail-wm1-f41.google.com with SMTP id f81so7221267wmd.4
 for <33729 <at> debbugs.gnu.org>; Fri, 14 Dec 2018 14:47:55 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=date:from:to:cc:subject:message-id:references:mime-version
 :content-disposition:content-transfer-encoding:in-reply-to
 :user-agent; bh=mDRiAB2kztbHc7Sm7QlEeC7GrjyvMYRKWiz5jLIFjJs=;
 b=Zn+doqxF6R/Awrsw02Y9qvFHgXVXOp2SqTbmtVWvTxrSzj+PwWcIPp9DFkQdo8axil
 eHCkJVkuSe/vsagrzF81HhIOUgMvjuzRQ2FRQi17x0Wj7KiNxIwpbtvqJ4VKQdN/CGYI
 QJjQRKbhkFUqFQbaMd4YXktYntyOG6iukicFiW2fHFMYdORFuNy1tnVfMqQicLcMIWC5
 oSpZnK0h6KtFqNL30G/XZQc3e0JAlq/ZT6juKgAG+9/scsP+zh77qfJCd8UE1K39swSw
 QRfYzusSf/tQqlnuy2CfnYCp+lqM9oFyvwBv0RKz+HuvQ7sQnzpM8FqIFMv+EQ/XrUn4
 iyqg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:date:from:to:cc:subject:message-id:references
 :mime-version:content-disposition:content-transfer-encoding
 :in-reply-to:user-agent;
 bh=mDRiAB2kztbHc7Sm7QlEeC7GrjyvMYRKWiz5jLIFjJs=;
 b=ZWl5vU/NFqoSHC+V0HJalfAbpEfz3P/Mi0yZ719fhohXUrBxy/0sH0s16elVCGwqyl
 2y38BsuIBY7zmdTWpWJxTcuoJHJc/vIdjGnslg3ptmd3EhI4Qgzs3IPCDe/1fstrQVtN
 2F+qQYBvgFmtBT2LsVSDryZqG2V03QHX5EWbsFOZWkoqHyGxA1m4ZTvR725kuVncuIO9
 Dnu5rml2flf/irceOebcklOrVuNrz66GdM52IjGODWPZMFLe1uycY1CVjzYH6FYq6Y9Y
 Roe1S7QjXzqmzdqdvTywQPYnInFclQZOsamhf5ugV6bSSzYa7jh1F2GHOfyZNHTM0NBs
 cH+Q==
X-Gm-Message-State: AA+aEWYAu4pF+XKucM5TfTGV7jNJ8zpC+VL0UCz4SOZhRl9pYIMm2mi3
 kw6SQ/36+tEe1Qsa5GYGUjo=
X-Google-Smtp-Source: AFSGD/Vzj+x+GNDov4vJDBgdedfXMuNr9PmsWe9lwRML+d531Wt9gLPrh8xeine/7rVzqB3ux1STsw==
X-Received: by 2002:a1c:a401:: with SMTP id n1mr4952069wme.114.1544827669746; 
 Fri, 14 Dec 2018 14:47:49 -0800 (PST)
Received: from macbook.localdomain ([197.58.120.254])
 by smtp.gmail.com with ESMTPSA id c12sm6847240wrs.82.2018.12.14.14.47.46
 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256);
 Fri, 14 Dec 2018 14:47:48 -0800 (PST)
Date: Sat, 15 Dec 2018 00:47:43 +0200
From: Khaled Hosny <dr.khaled.hosny@HIDDEN>
To: Eli Zaretskii <eliz@HIDDEN>
Subject: Re: bug#33729: 27.0.50; Partial glyphs not rendered for Gujarati
 with Harfbuzz enabled (renders fine using m17n)
Message-ID: <20181214224743.GL2244@HIDDEN>
References: <CAFyQvY3vfV9wqVWo3hAvawRAqpWUSGGf3tebwveM60OCtFkCvA@HIDDEN>
 <CAFyQvY2ZW38s4aMpkkfXLmTw_YrYJ8PRfaRPLugQt03tw0u7Zg@HIDDEN>
 <20181213203102.GF2244@HIDDEN>
 <CAFyQvY0pznh0yM9XJ9y_cEft47SXCmFp+O5rLkzb6o0By-MQuA@HIDDEN>
 <83h8fghcpo.fsf@HIDDEN>
 <20181214075056.GI2244@HIDDEN>
 <8336r0h1cb.fsf@HIDDEN>
 <20181214110316.GK2244@HIDDEN>
 <83y38sfcme.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <83y38sfcme.fsf@HIDDEN>
User-Agent: Mutt/1.11.1 (2018-12-01)
X-Spam-Score: 1.0 (+)
X-Debbugs-Envelope-To: 33729
Cc: behdad@HIDDEN, 33729 <at> debbugs.gnu.org, far.nasiri.m@HIDDEN,
 kaushal.modi@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

On Fri, Dec 14, 2018 at 03:42:49PM +0200, Eli Zaretskii wrote:
> > Date: Fri, 14 Dec 2018 13:03:16 +0200
> > From: Khaled Hosny <dr.khaled.hosny@HIDDEN>
> > Cc: kaushal.modi@HIDDEN, behdad@HIDDEN, 33729 <at> debbugs.gnu.org,
> > 	far.nasiri.m@HIDDEN
> > 
> > > FWIW, I looked at ftfont.c:uni_script, and I cannot find a problem
> > > with it; in particular looking up in char-script-table each character
> > > of the Gujarati welcome in HELLO yields 'gujarati', so I couldn't see
> > > any evident Emacs issue.  Or are you saying that hb_script_from_string
> > > doesn't DTRT?  Or maybe Kaushal should upgrade to a newer version of
> > > HarfBuzz?
> > 
> > There is this FIXME:
> > 
> > /* FIXME: from_string wants an ISO 15924 script tag here. */
> > 
> > As we discussed earlier, hb_script_from_string() expects ISO 15924
> > script tags, but char_script_table does not provide such tags (I don’t
> > recall what it does provide exactly). We need a way to get ISO 15924
> > script tags from Emacs.
> 
> Right, I forgot about that.
> 
> So you are saying that we need to generate Gujr instead of gujarati,
> is that right?

Yes (and the equivalent for all other scripts, of course).

Regards,
Khaled




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33729; Package emacs. Full text available.

Message received at 33729 <at> debbugs.gnu.org:


Received: (at 33729) by debbugs.gnu.org; 14 Dec 2018 15:25:59 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Dec 14 10:25:59 2018
Received: from localhost ([127.0.0.1]:48435 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gXpLj-0001dL-Ft
	for submit <at> debbugs.gnu.org; Fri, 14 Dec 2018 10:25:59 -0500
Received: from eggs.gnu.org ([208.118.235.92]:43639)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1gXpLi-0001d9-1H
 for 33729 <at> debbugs.gnu.org; Fri, 14 Dec 2018 10:25:58 -0500
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <eliz@HIDDEN>) id 1gXpLW-0008VN-PY
 for 33729 <at> debbugs.gnu.org; Fri, 14 Dec 2018 10:25:52 -0500
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=-0.0 required=5.0 tests=BAYES_20 autolearn=disabled
 version=3.3.2
Received: from fencepost.gnu.org ([2001:4830:134:3::e]:47041)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>)
 id 1gXpLW-0008UH-Hf; Fri, 14 Dec 2018 10:25:46 -0500
Received: from [176.228.60.248] (port=4231 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1gXpLW-0000sY-2c; Fri, 14 Dec 2018 10:25:46 -0500
Date: Fri, 14 Dec 2018 17:25:42 +0200
Message-Id: <83tvjgf7ux.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: far.nasiri.m@HIDDEN
In-reply-to: <83y38sfcme.fsf@HIDDEN> (message from Eli Zaretskii on Fri, 14
 Dec 2018 15:42:49 +0200)
Subject: Re: bug#33729: 27.0.50;
 Partial glyphs not rendered for Gujarati with Harfbuzz enabled
 (renders fine using m17n)
References: <CAFyQvY3vfV9wqVWo3hAvawRAqpWUSGGf3tebwveM60OCtFkCvA@HIDDEN>
 <CAFyQvY2ZW38s4aMpkkfXLmTw_YrYJ8PRfaRPLugQt03tw0u7Zg@HIDDEN>
 <20181213203102.GF2244@HIDDEN>
 <CAFyQvY0pznh0yM9XJ9y_cEft47SXCmFp+O5rLkzb6o0By-MQuA@HIDDEN>
 <83h8fghcpo.fsf@HIDDEN>
 <20181214075056.GI2244@HIDDEN>
 <8336r0h1cb.fsf@HIDDEN> <20181214110316.GK2244@HIDDEN>
 <83y38sfcme.fsf@HIDDEN>
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2001:4830:134:3::e
X-Spam-Score: -5.0 (-----)
X-Debbugs-Envelope-To: 33729
Cc: dr.khaled.hosny@HIDDEN, behdad@HIDDEN, 33729 <at> debbugs.gnu.org,
 kaushal.modi@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -6.0 (------)

> Date: Fri, 14 Dec 2018 15:42:49 +0200
> From: Eli Zaretskii <eliz@HIDDEN>
> Cc: behdad@HIDDEN, kaushal.modi@HIDDEN, 33729 <at> debbugs.gnu.org,
> 	far.nasiri.m@HIDDEN
> 
> Mohammad, do you need help in comping up with a solution?  There's
> otf-script-alist (see fontest.el), but it goes in the opposite
> direction.  We could use rassq (Frassq in C) to find the OTF script
> tag by its Emacs symbol (which is returned by indexing into
> Vchar_script_table), by looking in otf-script-alist.
> 
> Or maybe you prefer a separate data structure, not limited to the OTF
> tags?

After some thinking, my conclusion is that we should import the
ISO 15924 database from https://unicode.org/iso15924/, use a script
similar to admin/unidata/blocks.awk to generate an alist from it that
maps Emacs script names to ISO 15924 tags, and then access that alist
from uni_script to get the correct script information to Harfbuzz.

Patches implementing that are welcome.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33729; Package emacs. Full text available.

Message received at 33729 <at> debbugs.gnu.org:


Received: (at 33729) by debbugs.gnu.org; 14 Dec 2018 13:43:06 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Dec 14 08:43:06 2018
Received: from localhost ([127.0.0.1]:47431 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gXnkA-00077i-5X
	for submit <at> debbugs.gnu.org; Fri, 14 Dec 2018 08:43:06 -0500
Received: from eggs.gnu.org ([208.118.235.92]:35934)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1gXnk9-00077B-2o
 for 33729 <at> debbugs.gnu.org; Fri, 14 Dec 2018 08:43:05 -0500
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <eliz@HIDDEN>) id 1gXnk2-0005nx-TB
 for 33729 <at> debbugs.gnu.org; Fri, 14 Dec 2018 08:42:59 -0500
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50 autolearn=disabled
 version=3.3.2
Received: from fencepost.gnu.org ([2001:4830:134:3::e]:44702)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>)
 id 1gXnjz-0005m6-5s; Fri, 14 Dec 2018 08:42:55 -0500
Received: from [176.228.60.248] (port=1640 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1gXnjy-0001de-JB; Fri, 14 Dec 2018 08:42:55 -0500
Date: Fri, 14 Dec 2018 15:42:49 +0200
Message-Id: <83y38sfcme.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Khaled Hosny <dr.khaled.hosny@HIDDEN>
In-reply-to: <20181214110316.GK2244@HIDDEN> (message from Khaled
 Hosny on Fri, 14 Dec 2018 13:03:16 +0200)
Subject: Re: bug#33729: 27.0.50; Partial glyphs not rendered for Gujarati
 with Harfbuzz enabled (renders fine using m17n)
References: <CAFyQvY3vfV9wqVWo3hAvawRAqpWUSGGf3tebwveM60OCtFkCvA@HIDDEN>
 <CAFyQvY2ZW38s4aMpkkfXLmTw_YrYJ8PRfaRPLugQt03tw0u7Zg@HIDDEN>
 <20181213203102.GF2244@HIDDEN>
 <CAFyQvY0pznh0yM9XJ9y_cEft47SXCmFp+O5rLkzb6o0By-MQuA@HIDDEN>
 <83h8fghcpo.fsf@HIDDEN>
 <20181214075056.GI2244@HIDDEN>
 <8336r0h1cb.fsf@HIDDEN> <20181214110316.GK2244@HIDDEN>
MIME-version: 1.0
Content-type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2001:4830:134:3::e
X-Spam-Score: -5.0 (-----)
X-Debbugs-Envelope-To: 33729
Cc: behdad@HIDDEN, 33729 <at> debbugs.gnu.org, far.nasiri.m@HIDDEN,
 kaushal.modi@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -6.0 (------)

> Date: Fri, 14 Dec 2018 13:03:16 +0200
> From: Khaled Hosny <dr.khaled.hosny@HIDDEN>
> Cc: kaushal.modi@HIDDEN, behdad@HIDDEN, 33729 <at> debbugs.gnu.org,
> 	far.nasiri.m@HIDDEN
> 
> > FWIW, I looked at ftfont.c:uni_script, and I cannot find a problem
> > with it; in particular looking up in char-script-table each character
> > of the Gujarati welcome in HELLO yields 'gujarati', so I couldn't see
> > any evident Emacs issue.  Or are you saying that hb_script_from_string
> > doesn't DTRT?  Or maybe Kaushal should upgrade to a newer version of
> > HarfBuzz?
> 
> There is this FIXME:
> 
> /* FIXME: from_string wants an ISO 15924 script tag here. */
> 
> As we discussed earlier, hb_script_from_string() expects ISO 15924
> script tags, but char_script_table does not provide such tags (I don’t
> recall what it does provide exactly). We need a way to get ISO 15924
> script tags from Emacs.

Right, I forgot about that.

So you are saying that we need to generate Gujr instead of gujarati,
is that right?

Mohammad, do you need help in comping up with a solution?  There's
otf-script-alist (see fontest.el), but it goes in the opposite
direction.  We could use rassq (Frassq in C) to find the OTF script
tag by its Emacs symbol (which is returned by indexing into
Vchar_script_table), by looking in otf-script-alist.

Or maybe you prefer a seperat data structure, not limited to the OTF
tags?

Let me know if you need more help.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33729; Package emacs. Full text available.

Message received at 33729 <at> debbugs.gnu.org:


Received: (at 33729) by debbugs.gnu.org; 14 Dec 2018 11:03:28 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Dec 14 06:03:28 2018
Received: from localhost ([127.0.0.1]:47380 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gXlFg-0000v8-2u
	for submit <at> debbugs.gnu.org; Fri, 14 Dec 2018 06:03:28 -0500
Received: from mail-wm1-f49.google.com ([209.85.128.49]:33831)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dr.khaled.hosny@HIDDEN>) id 1gXlFe-0000ut-Iz
 for 33729 <at> debbugs.gnu.org; Fri, 14 Dec 2018 06:03:27 -0500
Received: by mail-wm1-f49.google.com with SMTP id y185so15969436wmd.1
 for <33729 <at> debbugs.gnu.org>; Fri, 14 Dec 2018 03:03:26 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=date:from:to:cc:subject:message-id:references:mime-version
 :content-disposition:content-transfer-encoding:in-reply-to
 :user-agent; bh=8k+AVQH5MB7DJfbVezJtjbDqtv3OFog7S/ao1HL+SRE=;
 b=WlEeO9fZk4mmoWuJcLfnyyAiNwAu7wolckVijYRy5xW7RDFLfLrNqthteyhKMIdihl
 +sMcOwMZ7fhREEMlEwJvGK+h0Eb5qBxJ7T9RdLTjwE9wTOvhrYOuAl7AGjj3U3exwgrI
 LmnPQRCukoUqbY1KsGAyHTUp6hY0578Cn7eEDk8Z96BqbL+galpVkGGalyQndlgCe8MG
 dnBYc3wjPQIx6L7GfvE8MVaw910k0ua2dtxNArcQub+O/KIYpOxf1mVtrM+4afKEUKZ4
 CjPcUNKnc7etaorsv312Ly+o2qixQmePtDqTxWPIKmaKWGdbwUhhP+OAC3UgEPZAd7t3
 4kuQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:date:from:to:cc:subject:message-id:references
 :mime-version:content-disposition:content-transfer-encoding
 :in-reply-to:user-agent;
 bh=8k+AVQH5MB7DJfbVezJtjbDqtv3OFog7S/ao1HL+SRE=;
 b=OzA8ckWfjILi5UnMXKS1x+tOBGkUyLrU36YdlSQw/ZjROjB6zBYB8lhfTRP1qlgEIO
 93RBHPjQu2nGr85cYlZ6YgEl2eJXsncDErpvl03K1Rlpll+73TU4/noSbgdM9MGDLSXt
 kGjjH0JvmRmemmJq4JRqRj+a0DZWE0rnji4oEnFXiH2VgZFgheIaPzesF91fejrZK0C6
 8y6CCjqU18aLzh2IPJqJgdIMiIncNqB4KVhQnbCINADBM1TBepgx7lpVYJszU09Ftbmn
 XWeCjAphqUh+TVu4KgA6NZCl2YDlOLezr4g7noPLJ1V0j3xe06je7kmuBl2WRTt1CFi5
 XTgQ==
X-Gm-Message-State: AA+aEWYJUt+BJ6qn6Q93Ca1G8fWujRl23rA7Hvc+mmj/jwhtWGdj0N6N
 QUnORzDBBJSg1nENmCv2M8o=
X-Google-Smtp-Source: AFSGD/W9CHYec424oiAJccQx38NkJQ36ksH+v1QEMpm3NDKPDeMSrOskQ9pAHgmEQRQFEOSMfcfMFw==
X-Received: by 2002:a7b:c191:: with SMTP id y17mr2885711wmi.10.1544785400716; 
 Fri, 14 Dec 2018 03:03:20 -0800 (PST)
Received: from macbook.localdomain ([197.58.120.254])
 by smtp.gmail.com with ESMTPSA id 199sm4913398wmh.21.2018.12.14.03.03.18
 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256);
 Fri, 14 Dec 2018 03:03:20 -0800 (PST)
Date: Fri, 14 Dec 2018 13:03:16 +0200
From: Khaled Hosny <dr.khaled.hosny@HIDDEN>
To: Eli Zaretskii <eliz@HIDDEN>
Subject: Re: bug#33729: 27.0.50; Partial glyphs not rendered for Gujarati
 with Harfbuzz enabled (renders fine using m17n)
Message-ID: <20181214110316.GK2244@HIDDEN>
References: <CAFyQvY3vfV9wqVWo3hAvawRAqpWUSGGf3tebwveM60OCtFkCvA@HIDDEN>
 <CAFyQvY2ZW38s4aMpkkfXLmTw_YrYJ8PRfaRPLugQt03tw0u7Zg@HIDDEN>
 <20181213203102.GF2244@HIDDEN>
 <CAFyQvY0pznh0yM9XJ9y_cEft47SXCmFp+O5rLkzb6o0By-MQuA@HIDDEN>
 <83h8fghcpo.fsf@HIDDEN>
 <20181214075056.GI2244@HIDDEN>
 <8336r0h1cb.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <8336r0h1cb.fsf@HIDDEN>
User-Agent: Mutt/1.11.1 (2018-12-01)
X-Spam-Score: 1.0 (+)
X-Debbugs-Envelope-To: 33729
Cc: behdad@HIDDEN, 33729 <at> debbugs.gnu.org, far.nasiri.m@HIDDEN,
 kaushal.modi@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

On Fri, Dec 14, 2018 at 12:03:32PM +0200, Eli Zaretskii wrote:
> > Date: Fri, 14 Dec 2018 09:50:56 +0200
> > From: Khaled Hosny <dr.khaled.hosny@HIDDEN>
> > Cc: Kaushal Modi <kaushal.modi@HIDDEN>, behdad@HIDDEN,
> > 	33729 <at> debbugs.gnu.org, far.nasiri.m@HIDDEN
> > 
> > > Your data indicates that the m17n build performs character composition
> > > at buffer position 34, whereas the harfbuzz build does not.  The
> > > question is why.
> > 
> > See my earlier email, most likely the culprit is the broken Emacs to
> > HarfBuzz script code mapping that we discussed earlier. HarfBuzz needs
> > to know the correct script of the text to perform shaping, and it looks
> > like we are passing nonsense values for certain scripts (or rather for
> > certain scripts we are lucky that the mapping is not broken).
> 
> Thanks.
> 
> I don't yet have access to a GNU/Linux system with HarfBuzz installed,
> so I cannot myself debug it.
> 
> I hope Mohammad will be able to look into this and either fix it or
> provide more focused and detailed analysis of what is wrong, so we
> could fix it.  Or maybe you could point to the problematic code and
> tell more details.
> 
> FWIW, I looked at ftfont.c:uni_script, and I cannot find a problem
> with it; in particular looking up in char-script-table each character
> of the Gujarati welcome in HELLO yields 'gujarati', so I couldn't see
> any evident Emacs issue.  Or are you saying that hb_script_from_string
> doesn't DTRT?  Or maybe Kaushal should upgrade to a newer version of
> HarfBuzz?

There is this FIXME:

/* FIXME: from_string wants an ISO 15924 script tag here. */

As we discussed earlier, hb_script_from_string() expects ISO 15924
script tags, but char_script_table does not provide such tags (I don’t
recall what it does provide exactly). We need a way to get ISO 15924
script tags from Emacs.

Regards,
Khaled




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33729; Package emacs. Full text available.

Message received at 33729 <at> debbugs.gnu.org:


Received: (at 33729) by debbugs.gnu.org; 14 Dec 2018 10:03:49 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Dec 14 05:03:49 2018
Received: from localhost ([127.0.0.1]:47337 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gXkJx-0007si-2S
	for submit <at> debbugs.gnu.org; Fri, 14 Dec 2018 05:03:49 -0500
Received: from eggs.gnu.org ([208.118.235.92]:40589)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1gXkJv-0007sX-5v
 for 33729 <at> debbugs.gnu.org; Fri, 14 Dec 2018 05:03:47 -0500
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <eliz@HIDDEN>) id 1gXkJo-0005zO-Sx
 for 33729 <at> debbugs.gnu.org; Fri, 14 Dec 2018 05:03:41 -0500
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50 autolearn=disabled
 version=3.3.2
Received: from fencepost.gnu.org ([2001:4830:134:3::e]:57034)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>)
 id 1gXkJj-0005vJ-QU; Fri, 14 Dec 2018 05:03:35 -0500
Received: from [176.228.60.248] (port=3980 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1gXkJj-0007bb-Cu; Fri, 14 Dec 2018 05:03:35 -0500
Date: Fri, 14 Dec 2018 12:03:32 +0200
Message-Id: <8336r0h1cb.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Khaled Hosny <dr.khaled.hosny@HIDDEN>
In-reply-to: <20181214075056.GI2244@HIDDEN> (message from Khaled
 Hosny on Fri, 14 Dec 2018 09:50:56 +0200)
Subject: Re: bug#33729: 27.0.50; Partial glyphs not rendered for Gujarati
 with Harfbuzz enabled (renders fine using m17n)
References: <CAFyQvY3vfV9wqVWo3hAvawRAqpWUSGGf3tebwveM60OCtFkCvA@HIDDEN>
 <CAFyQvY2ZW38s4aMpkkfXLmTw_YrYJ8PRfaRPLugQt03tw0u7Zg@HIDDEN>
 <20181213203102.GF2244@HIDDEN>
 <CAFyQvY0pznh0yM9XJ9y_cEft47SXCmFp+O5rLkzb6o0By-MQuA@HIDDEN>
 <83h8fghcpo.fsf@HIDDEN> <20181214075056.GI2244@HIDDEN>
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2001:4830:134:3::e
X-Spam-Score: -5.0 (-----)
X-Debbugs-Envelope-To: 33729
Cc: behdad@HIDDEN, 33729 <at> debbugs.gnu.org, far.nasiri.m@HIDDEN,
 kaushal.modi@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -6.0 (------)

> Date: Fri, 14 Dec 2018 09:50:56 +0200
> From: Khaled Hosny <dr.khaled.hosny@HIDDEN>
> Cc: Kaushal Modi <kaushal.modi@HIDDEN>, behdad@HIDDEN,
> 	33729 <at> debbugs.gnu.org, far.nasiri.m@HIDDEN
> 
> > Your data indicates that the m17n build performs character composition
> > at buffer position 34, whereas the harfbuzz build does not.  The
> > question is why.
> 
> See my earlier email, most likely the culprit is the broken Emacs to
> HarfBuzz script code mapping that we discussed earlier. HarfBuzz needs
> to know the correct script of the text to perform shaping, and it looks
> like we are passing nonsense values for certain scripts (or rather for
> certain scripts we are lucky that the mapping is not broken).

Thanks.

I don't yet have access to a GNU/Linux system with HarfBuzz installed,
so I cannot myself debug it.

I hope Mohammad will be able to look into this and either fix it or
provide more focused and detailed analysis of what is wrong, so we
could fix it.  Or maybe you could point to the problematic code and
tell more details.

FWIW, I looked at ftfont.c:uni_script, and I cannot find a problem
with it; in particular looking up in char-script-table each character
of the Gujarati welcome in HELLO yields 'gujarati', so I couldn't see
any evident Emacs issue.  Or are you saying that hb_script_from_string
doesn't DTRT?  Or maybe Kaushal should upgrade to a newer version of
HarfBuzz?

Thanks.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33729; Package emacs. Full text available.

Message received at 33729 <at> debbugs.gnu.org:


Received: (at 33729) by debbugs.gnu.org; 14 Dec 2018 07:51:09 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Dec 14 02:51:09 2018
Received: from localhost ([127.0.0.1]:47253 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gXiFZ-0004YA-1i
	for submit <at> debbugs.gnu.org; Fri, 14 Dec 2018 02:51:09 -0500
Received: from mail-wr1-f51.google.com ([209.85.221.51]:41019)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dr.khaled.hosny@HIDDEN>) id 1gXiFW-0004Xd-Jq
 for 33729 <at> debbugs.gnu.org; Fri, 14 Dec 2018 02:51:07 -0500
Received: by mail-wr1-f51.google.com with SMTP id x10so4433370wrs.8
 for <33729 <at> debbugs.gnu.org>; Thu, 13 Dec 2018 23:51:06 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=date:from:to:cc:subject:message-id:references:mime-version
 :content-disposition:in-reply-to:user-agent;
 bh=2EhnsLXf16PIk49rid2dop6LbhX1stDWTDO7cqfCidU=;
 b=WJdEX+N6EtOgXlPbG44mBRHgK18y6GfAz6RhM3wd9Er7mfojrZXUmI0lYzrIyV0in7
 r3hIHHhOOzdaqG9psqBMJnwGIni1wjDLf2HmkxLjeYdm/JlU/ici1FLh2hpQJJt7OR7p
 o+Of4VX1mELRqKav/lq1/Le7yuNCyBFRJWSGTLumHbITjI3X4ZYx+dKs1ZGCWNg1AvPS
 egCDydmAiYxxtGGVKmsPlL4sGSOemAFKHCfl7k4/OjpFuFiTKJhgOuuQQ56zTuJ12lYO
 6w2rBqECRknscAMPcsZehAUKAXht2O0rHKs/t+KUdF8ZBkFA9TkwJULpyR2zGIUfI3jz
 HQCQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:date:from:to:cc:subject:message-id:references
 :mime-version:content-disposition:in-reply-to:user-agent;
 bh=2EhnsLXf16PIk49rid2dop6LbhX1stDWTDO7cqfCidU=;
 b=K5UiLfL50Hoe5tJEpFBArKuejGviLbiYBF4gLCIhL16L+lY3i44LAeqIoCPOOA+KHb
 ZhNi7Z0upTTD30v1xzlhmP/JJZB7dFpQH+Q6sVXL1hcPOpzO5AZB+Rs0nJya9cEhqsLP
 dFJaBIO9vPmAnDItdItRl6NsSVXNP8uljD3TCi5THVcguwCgYFmQAgOgW+ZUecHROnkd
 dchq+kBILc11ZzsKyYBNfWITa2Kb65+LtWLGtpZIZF7ESzzhNTRIocAKurscbbaowVvn
 YHC+QzVPLXec6SwVB2bbdTF3ea4TCdgG/Nbn6VHb9UQK7PSkzg62HXCYzfCejv9nY4xv
 0Oyg==
X-Gm-Message-State: AA+aEWa8FbuXjeyl/nNd7GRAiu0HNge5pz+0X4jQyOpFAX0yxs6ZDSIP
 Lhyyrd/utGcsltepWzCX4p4=
X-Google-Smtp-Source: AFSGD/WC8IJ97o2R9Lb0JYk/iUpgwdvbrie8LXocOOIZ4r57oRW4m6iPkcre8YI8DQJhS/ITI7T/nw==
X-Received: by 2002:adf:c711:: with SMTP id k17mr1553771wrg.197.1544773860672; 
 Thu, 13 Dec 2018 23:51:00 -0800 (PST)
Received: from macbook.localdomain ([197.58.120.254])
 by smtp.gmail.com with ESMTPSA id f66sm4312371wmd.28.2018.12.13.23.50.59
 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256);
 Thu, 13 Dec 2018 23:50:59 -0800 (PST)
Date: Fri, 14 Dec 2018 09:50:56 +0200
From: Khaled Hosny <dr.khaled.hosny@HIDDEN>
To: Eli Zaretskii <eliz@HIDDEN>
Subject: Re: bug#33729: 27.0.50; Partial glyphs not rendered for Gujarati
 with Harfbuzz enabled (renders fine using m17n)
Message-ID: <20181214075056.GI2244@HIDDEN>
References: <CAFyQvY3vfV9wqVWo3hAvawRAqpWUSGGf3tebwveM60OCtFkCvA@HIDDEN>
 <CAFyQvY2ZW38s4aMpkkfXLmTw_YrYJ8PRfaRPLugQt03tw0u7Zg@HIDDEN>
 <20181213203102.GF2244@HIDDEN>
 <CAFyQvY0pznh0yM9XJ9y_cEft47SXCmFp+O5rLkzb6o0By-MQuA@HIDDEN>
 <83h8fghcpo.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
In-Reply-To: <83h8fghcpo.fsf@HIDDEN>
User-Agent: Mutt/1.11.1 (2018-12-01)
X-Spam-Score: 1.0 (+)
X-Debbugs-Envelope-To: 33729
Cc: behdad@HIDDEN, 33729 <at> debbugs.gnu.org, far.nasiri.m@HIDDEN,
 Kaushal Modi <kaushal.modi@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

On Fri, Dec 14, 2018 at 07:57:55AM +0200, Eli Zaretskii wrote:
> > From: Kaushal Modi <kaushal.modi@HIDDEN>
> > Date: Thu, 13 Dec 2018 15:43:50 -0500
> > Cc: behdad@HIDDEN, 33729 <at> debbugs.gnu.org, far.nasiri.m@HIDDEN
> > 
> >  For debugging the such rendering differences, the actual font used by
> >  Emacs for a given part of the text need to be known,
> > 
> > I am using Mukta Vaani font for Gujarati. It is a free font and be downloaded from
> > https://ektype.in/mukta-vaani.html.
> 
> Your data indicates that the m17n build performs character composition
> at buffer position 34, whereas the harfbuzz build does not.  The
> question is why.

See my earlier email, most likely the culprit is the broken Emacs to
HarfBuzz script code mapping that we discussed earlier. HarfBuzz needs
to know the correct script of the text to perform shaping, and it looks
like we are passing nonsense values for certain scripts (or rather for
certain scripts we are lucky that the mapping is not broken).




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33729; Package emacs. Full text available.

Message received at 33729 <at> debbugs.gnu.org:


Received: (at 33729) by debbugs.gnu.org; 14 Dec 2018 07:49:10 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Dec 14 02:49:09 2018
Received: from localhost ([127.0.0.1]:47245 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gXiDd-0004Ue-MD
	for submit <at> debbugs.gnu.org; Fri, 14 Dec 2018 02:49:09 -0500
Received: from eggs.gnu.org ([208.118.235.92]:50258)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1gXiDb-0004UQ-IV
 for 33729 <at> debbugs.gnu.org; Fri, 14 Dec 2018 02:49:07 -0500
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <eliz@HIDDEN>) id 1gXiDV-0004J5-Q6
 for 33729 <at> debbugs.gnu.org; Fri, 14 Dec 2018 02:49:02 -0500
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00 autolearn=disabled
 version=3.3.2
Received: from fencepost.gnu.org ([2001:4830:134:3::e]:54040)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>)
 id 1gXiDR-0004HE-53; Fri, 14 Dec 2018 02:48:57 -0500
Received: from [176.228.60.248] (port=3401 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1gXiDQ-0003GH-NE; Fri, 14 Dec 2018 02:48:57 -0500
Date: Fri, 14 Dec 2018 09:48:54 +0200
Message-Id: <838t0sh7kp.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: kaushal.modi@HIDDEN
In-reply-to: <83h8fghcpo.fsf@HIDDEN> (message from Eli Zaretskii on Fri, 14
 Dec 2018 07:57:55 +0200)
Subject: Re: bug#33729: 27.0.50;
 Partial glyphs not rendered for Gujarati with Harfbuzz enabled
 (renders fine using m17n)
References: <CAFyQvY3vfV9wqVWo3hAvawRAqpWUSGGf3tebwveM60OCtFkCvA@HIDDEN>
 <CAFyQvY2ZW38s4aMpkkfXLmTw_YrYJ8PRfaRPLugQt03tw0u7Zg@HIDDEN>
 <20181213203102.GF2244@HIDDEN>
 <CAFyQvY0pznh0yM9XJ9y_cEft47SXCmFp+O5rLkzb6o0By-MQuA@HIDDEN>
 <83h8fghcpo.fsf@HIDDEN>
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2001:4830:134:3::e
X-Spam-Score: -5.0 (-----)
X-Debbugs-Envelope-To: 33729
Cc: dr.khaled.hosny@HIDDEN, behdad@HIDDEN, 33729 <at> debbugs.gnu.org,
 far.nasiri.m@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -6.0 (------)

> Date: Fri, 14 Dec 2018 07:57:55 +0200
> From: Eli Zaretskii <eliz@HIDDEN>
> Cc: dr.khaled.hosny@HIDDEN, behdad@HIDDEN, 33729 <at> debbugs.gnu.org,
> 	far.nasiri.m@HIDDEN
> 
> Your data indicates that the m17n build performs character composition
> at buffer position 34

Sorry, wrong number: I meant buffer position 1612.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33729; Package emacs. Full text available.

Message received at 33729 <at> debbugs.gnu.org:


Received: (at 33729) by debbugs.gnu.org; 14 Dec 2018 06:46:03 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Dec 14 01:46:03 2018
Received: from localhost ([127.0.0.1]:47215 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gXhEZ-0002yP-6l
	for submit <at> debbugs.gnu.org; Fri, 14 Dec 2018 01:46:03 -0500
Received: from zimbra.cs.ucla.edu ([131.179.128.68]:60386)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eggert@HIDDEN>) id 1gXhEV-0002xd-AT
 for 33729 <at> debbugs.gnu.org; Fri, 14 Dec 2018 01:46:02 -0500
Received: from localhost (localhost [127.0.0.1])
 by zimbra.cs.ucla.edu (Postfix) with ESMTP id 3AB5516094D;
 Thu, 13 Dec 2018 22:45:53 -0800 (PST)
Received: from zimbra.cs.ucla.edu ([127.0.0.1])
 by localhost (zimbra.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10032)
 with ESMTP id MSTROIeOh4LC; Thu, 13 Dec 2018 22:45:52 -0800 (PST)
Received: from localhost (localhost [127.0.0.1])
 by zimbra.cs.ucla.edu (Postfix) with ESMTP id 3B05E16092C;
 Thu, 13 Dec 2018 22:45:52 -0800 (PST)
X-Virus-Scanned: amavisd-new at zimbra.cs.ucla.edu
Received: from zimbra.cs.ucla.edu ([127.0.0.1])
 by localhost (zimbra.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10026)
 with ESMTP id dEfXmBtFqzvu; Thu, 13 Dec 2018 22:45:52 -0800 (PST)
Received: from [192.168.1.9] (cpe-23-242-74-103.socal.res.rr.com
 [23.242.74.103])
 by zimbra.cs.ucla.edu (Postfix) with ESMTPSA id 03DA31608FF;
 Thu, 13 Dec 2018 22:45:51 -0800 (PST)
To: 33729 <at> debbugs.gnu.org
From: Paul Eggert <eggert@HIDDEN>
Subject: Re: bug#33729: 27.0.50; Partial glyphs not rendered for Gujarati with
 Harfbuzz enabled (renders fine using m17n)
Organization: UCLA Computer Science Department
Message-ID: <7cf3b43b-ab7c-1bac-48b3-b27e61570aeb@HIDDEN>
Date: Thu, 13 Dec 2018 22:45:48 -0800
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101
 Thunderbird/60.2.1
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 33729
Cc: dr.khaled.hosny@HIDDEN, behdad@HIDDEN, Florian Beck <fb@HIDDEN>,
 far.nasiri.m@HIDDEN, Kaushal Modi <kaushal.modi@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Florian Beck pointed out some examples of possible related problems when=20
rendering Emacs's etc/HELLO file; see:

https://lists.gnu.org/r/emacs-devel/2018-12/msg00271.html

For the names of the languages in the languages, Harfbuzz seems to be bet=
ter for=20
Burmese (=E1=80=99=E1=80=BC=E1=80=94=E1=80=BA=E1=80=99=E1=80=AC) (where m=
aster is wrong); conversely Harfbuzz seems to be wrong=20
for Maldivian (=DE=8B=DE=A8=DE=88=DE=AC=DE=80=DE=A8) (where master is bet=
ter). Please see the following for=20
what these should look like:

https://en.wikipedia.org/wiki/File:Dhivehiscript.svg

https://en.wikipedia.org/wiki/File:Burmese_script_sample.svg




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33729; Package emacs. Full text available.

Message received at 33729 <at> debbugs.gnu.org:


Received: (at 33729) by debbugs.gnu.org; 14 Dec 2018 05:58:11 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Dec 14 00:58:11 2018
Received: from localhost ([127.0.0.1]:47206 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gXgUF-0001OE-BR
	for submit <at> debbugs.gnu.org; Fri, 14 Dec 2018 00:58:11 -0500
Received: from eggs.gnu.org ([208.118.235.92]:53812)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1gXgUD-0001O0-RC
 for 33729 <at> debbugs.gnu.org; Fri, 14 Dec 2018 00:58:10 -0500
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <eliz@HIDDEN>) id 1gXgU7-0004rx-N7
 for 33729 <at> debbugs.gnu.org; Fri, 14 Dec 2018 00:58:04 -0500
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=-0.0 required=5.0 tests=BAYES_40 autolearn=disabled
 version=3.3.2
Received: from fencepost.gnu.org ([2001:4830:134:3::e]:52459)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>)
 id 1gXgU2-0004pB-PA; Fri, 14 Dec 2018 00:57:58 -0500
Received: from [176.228.60.248] (port=4536 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1gXgU2-0003kN-Af; Fri, 14 Dec 2018 00:57:58 -0500
Date: Fri, 14 Dec 2018 07:57:55 +0200
Message-Id: <83h8fghcpo.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Kaushal Modi <kaushal.modi@HIDDEN>
In-reply-to: <CAFyQvY0pznh0yM9XJ9y_cEft47SXCmFp+O5rLkzb6o0By-MQuA@HIDDEN>
 (message from Kaushal Modi on Thu, 13 Dec 2018 15:43:50 -0500)
Subject: Re: bug#33729: 27.0.50;
 Partial glyphs not rendered for Gujarati with Harfbuzz enabled
 (renders fine using m17n)
References: <CAFyQvY3vfV9wqVWo3hAvawRAqpWUSGGf3tebwveM60OCtFkCvA@HIDDEN>
 <CAFyQvY2ZW38s4aMpkkfXLmTw_YrYJ8PRfaRPLugQt03tw0u7Zg@HIDDEN>
 <20181213203102.GF2244@HIDDEN>
 <CAFyQvY0pznh0yM9XJ9y_cEft47SXCmFp+O5rLkzb6o0By-MQuA@HIDDEN>
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2001:4830:134:3::e
X-Spam-Score: -5.0 (-----)
X-Debbugs-Envelope-To: 33729
Cc: dr.khaled.hosny@HIDDEN, behdad@HIDDEN, 33729 <at> debbugs.gnu.org,
 far.nasiri.m@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -6.0 (------)

> From: Kaushal Modi <kaushal.modi@HIDDEN>
> Date: Thu, 13 Dec 2018 15:43:50 -0500
> Cc: behdad@HIDDEN, 33729 <at> debbugs.gnu.org, far.nasiri.m@HIDDEN
> 
>  For debugging the such rendering differences, the actual font used by
>  Emacs for a given part of the text need to be known,
> 
> I am using Mukta Vaani font for Gujarati. It is a free font and be downloaded from
> https://ektype.in/mukta-vaani.html.

Your data indicates that the m17n build performs character composition
at buffer position 34, whereas the harfbuzz build does not.  The
question is why.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33729; Package emacs. Full text available.

Message received at 33729 <at> debbugs.gnu.org:


Received: (at 33729) by debbugs.gnu.org; 13 Dec 2018 21:05:11 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Dec 13 16:05:11 2018
Received: from localhost ([127.0.0.1]:46945 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gXYAR-0005nf-G3
	for submit <at> debbugs.gnu.org; Thu, 13 Dec 2018 16:05:11 -0500
Received: from mail-lj1-f170.google.com ([209.85.208.170]:41965)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <kaushal.modi@HIDDEN>) id 1gXYAP-0005nN-Hg
 for 33729 <at> debbugs.gnu.org; Thu, 13 Dec 2018 16:05:10 -0500
Received: by mail-lj1-f170.google.com with SMTP id k15-v6so3027254ljc.8
 for <33729 <at> debbugs.gnu.org>; Thu, 13 Dec 2018 13:05:09 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=mime-version:references:in-reply-to:from:date:message-id:subject:to
 :cc; bh=30xQwH9KIBHuF/FxGaY4SOpxucDrnQC2CXHg13t1td0=;
 b=jJju5qJ4sVp1UwaCEcGcJldumPQTC7sGgbsrKhfJkdC9xRzDPRO2AnSuxSKZegOzSL
 aUTmO6QqGG1fO4YyXgqnFXW52v4Rvf0/NY1d+vheR1I6qiD6LJgsyQCdC0Xcy7NaxIMA
 5e+PoXfFqp/NoF1YvfUAtQd5bMZjnNyuFgjWFz9cloHHLq8aTYXYEPPYOrY6p34i16f7
 nT21p9KvuIwf9o6OA/tTHa6iQrln6PFZMNp691hSbf4sLRHdz4zTGxXAHSzQ8ClSMRzf
 jGUO1h97111OtQTaUV3zAc2mRi7faDmPCz9qZncoOQkqt8FRk9zcdUV7RBOv6V0OEJZB
 obYQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:mime-version:references:in-reply-to:from:date
 :message-id:subject:to:cc;
 bh=30xQwH9KIBHuF/FxGaY4SOpxucDrnQC2CXHg13t1td0=;
 b=RKnvcZlgtqY8s5UJBGrbmpHGv6WaTxXJxR9LujiQQ8QlQ4gHRZmHjj2cX3GZcunHlV
 +rIwQtI0FafbkTsfpeiDR/AjP01hcCRWVtASN4AYphRT7/H3nmN2VUuwwEG1c9tuI+76
 qJ9o+pb0+mu3xmh+hyE2bHjOITtR5OWoZbxeGZYCU5b1orWeZEcVmlLAww0po2EhL3s2
 nvfFp+wPxjNhWpGIReM52NZus7fHhex48soDtPIpw/lSTytLdQt8r21lpwa8VDTMptx9
 8pVb7GuC4arlcq6p8sdcyTqICoXHVHl2t6cJq5HC3MX//+44i/9vhP86Fnp9L5yqd7MU
 mS9Q==
X-Gm-Message-State: AA+aEWYxpvgH/tNoUU9a1NZVn3eB9kqDtsGo8Ng+81XpTGNmUWGX98/f
 C7w4n6UshgjvhVjbMkri4F2sh+6C4zQzYqnq22Q=
X-Google-Smtp-Source: AFSGD/X5hD0ZU9dhhJFLyYaCr02P3wrA0CZnJGXJZOm841+h78z/lG2HNSI9hhDHG6B8EBhbov8GgyY9zkclImHOYsg=
X-Received: by 2002:a2e:5109:: with SMTP id f9-v6mr216621ljb.52.1544735103303; 
 Thu, 13 Dec 2018 13:05:03 -0800 (PST)
MIME-Version: 1.0
References: <CAFyQvY3vfV9wqVWo3hAvawRAqpWUSGGf3tebwveM60OCtFkCvA@HIDDEN>
 <CAFyQvY2ZW38s4aMpkkfXLmTw_YrYJ8PRfaRPLugQt03tw0u7Zg@HIDDEN>
 <20181213203102.GF2244@HIDDEN>
 <CAFyQvY0pznh0yM9XJ9y_cEft47SXCmFp+O5rLkzb6o0By-MQuA@HIDDEN>
 <20181213205336.GG2244@HIDDEN>
In-Reply-To: <20181213205336.GG2244@HIDDEN>
From: Kaushal Modi <kaushal.modi@HIDDEN>
Date: Thu, 13 Dec 2018 16:04:25 -0500
Message-ID: <CAFyQvY3tcRYTLmwBGTzi82=terknxAqEiA5OSSK18Eh+PNt5FA@HIDDEN>
Subject: Re: 27.0.50; Partial glyphs not rendered for Gujarati with Harfbuzz
 enabled (renders fine using m17n)
To: dr.khaled.hosny@HIDDEN
Content-Type: multipart/alternative; boundary="0000000000001ac933057cedab9c"
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 33729
Cc: behdad@HIDDEN, 33729 <at> debbugs.gnu.org, far.nasiri.m@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

--0000000000001ac933057cedab9c
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Thu, Dec 13, 2018 at 3:53 PM Khaled Hosny <dr.khaled.hosny@HIDDEN>
wrote:

>
> I tried that font and text with hb-view and the output I get is
> identical to m17b.


hb-view is nifty! I wasn't sure if it would work for me (because I haven't
set my terminal to show unicode, etc.). But even with the older Harfbuzz
1.0.3 that I have, hb-view gave this: https://i.imgtc.com/d1N177Z.png

I am impressed. That shows the correct rendering of =E0=AA=A8=E0=AA=AE=E0=
=AA=B8=E0=AB=8D=E0=AA=A4=E0=AB=87. (I just blindly
pasted  =E0=AA=A8=E0=AA=AE=E0=AA=B8=E0=AB=8D=E0=AA=A4=E0=AB=87 as the secon=
d argument and hit enter, my terminal doesn't
even show the pasted text. But the hb-view rendering is correct.)


> If I pass a wrong script to HarfBuzz (e.g.
> --script=3Dlatn), I get the same broken output you see in Emacs. So I=E2=
=80=99m
> guessing something is not correctly working in script itemization. Most
> likely the FIXME in uni_script(), or the FIXME above the call to
> hb_buffer_guess_segment_properties().
>

I am not a C developer. But hopefully this information would help you to
fix the Harfbuzz integration with Emacs.

I am surprised that the rendering of Hindi =E0=A4=A8=E0=A4=AE=E0=A4=B8=E0=
=A5=8D=E0=A4=A4=E0=A5=87 using Harfbuzz in Emacs
is correct, while the  rendering of Gujarati =E0=AA=A8=E0=AA=AE=E0=AA=B8=E0=
=AB=8D=E0=AA=A4=E0=AB=87 is not, when in fact
the two scripts are so similar to each other. [Fun fact: Most of Gujarati
script if superimposed with a line at the top will look like valid Hindi.
You can see that in the case of  =E0=AA=A8=E0=AA=AE=E0=AA=B8=E0=AB=8D=E0=AA=
=A4=E0=AB=87 vs  =E0=A4=A8=E0=A4=AE=E0=A4=B8=E0=A5=8D=E0=A4=A4=E0=A5=87 :) =
]

--0000000000001ac933057cedab9c
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div dir=3D"ltr"><div dir=3D"ltr"><div class=3D"gmail_quot=
e"><div dir=3D"ltr">On Thu, Dec 13, 2018 at 3:53 PM Khaled Hosny &lt;<a hre=
f=3D"mailto:dr.khaled.hosny@HIDDEN">dr.khaled.hosny@HIDDEN</a>&gt; wr=
ote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px=
 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
I tried that font and text with hb-view and the output I get is<br>
identical to m17b.</blockquote><div><br></div><div>hb-view is nifty! I wasn=
&#39;t sure if it would work for me (because I haven&#39;t set my terminal =
to show unicode, etc.). But even with the older Harfbuzz 1.0.3 that I have,=
 hb-view gave this: <a href=3D"https://i.imgtc.com/d1N177Z.png">https://i.i=
mgtc.com/d1N177Z.png</a></div><div><br></div><div>I am impressed. That show=
s the correct rendering of=20
=E0=AA=A8=E0=AA=AE=E0=AA=B8=E0=AB=8D=E0=AA=A4=E0=AB=87. (I just blindly pas=
ted=C2=A0
=E0=AA=A8=E0=AA=AE=E0=AA=B8=E0=AB=8D=E0=AA=A4=E0=AB=87 as the second argume=
nt and hit enter, my terminal doesn&#39;t even show the pasted text. But th=
e hb-view rendering is correct.)

</div><div>=C2=A0</div><blockquote class=3D"gmail_quote" style=3D"margin:0p=
x 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> I=
f I pass a wrong script to HarfBuzz (e.g.<br>
--script=3Dlatn), I get the same broken output you see in Emacs. So I=E2=80=
=99m<br>
guessing something is not correctly working in script itemization. Most<br>
likely the FIXME in uni_script(), or the FIXME above the call to<br>
hb_buffer_guess_segment_properties().<br></blockquote><div><br></div><div>I=
 am not a C developer. But hopefully this information would help you to fix=
 the Harfbuzz integration with Emacs.=C2=A0</div><div><br></div><div>I am s=
urprised that the rendering of Hindi =E0=A4=A8=E0=A4=AE=E0=A4=B8=E0=A5=8D=
=E0=A4=A4=E0=A5=87 using Harfbuzz in Emacs is correct, while the=C2=A0 rend=
ering of Gujarati=20
=E0=AA=A8=E0=AA=AE=E0=AA=B8=E0=AB=8D=E0=AA=A4=E0=AB=87 is not, when in fact=
 the two scripts are so similar to each other. [Fun fact: Most of Gujarati =
script if superimposed with a line at the top will look like valid Hindi. Y=
ou can see that in the case of=C2=A0

=E0=AA=A8=E0=AA=AE=E0=AA=B8=E0=AB=8D=E0=AA=A4=E0=AB=87

vs=C2=A0
=E0=A4=A8=E0=A4=AE=E0=A4=B8=E0=A5=8D=E0=A4=A4=E0=A5=87

 :) ]</div></div></div></div></div>

--0000000000001ac933057cedab9c--




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33729; Package emacs. Full text available.

Message received at 33729 <at> debbugs.gnu.org:


Received: (at 33729) by debbugs.gnu.org; 13 Dec 2018 20:53:47 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Dec 13 15:53:47 2018
Received: from localhost ([127.0.0.1]:46936 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gXXzP-0003YM-2O
	for submit <at> debbugs.gnu.org; Thu, 13 Dec 2018 15:53:47 -0500
Received: from mail-wm1-f47.google.com ([209.85.128.47]:40367)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dr.khaled.hosny@HIDDEN>) id 1gXXzO-0003Y9-0p
 for 33729 <at> debbugs.gnu.org; Thu, 13 Dec 2018 15:53:46 -0500
Received: by mail-wm1-f47.google.com with SMTP id q26so3756001wmf.5
 for <33729 <at> debbugs.gnu.org>; Thu, 13 Dec 2018 12:53:45 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=date:from:to:cc:subject:message-id:references:mime-version
 :content-disposition:content-transfer-encoding:in-reply-to
 :user-agent; bh=4xwe32xkFcAwGc7g/aZIKI5v5ZroCBBo+J8S7AX1XHY=;
 b=V8F7N2sI0IZJv1GYTT/g8h/R4QrFOn+dzsUZ7x8xv1EmeyJuVtzXm4uBhDtI8ORDXM
 ti1cqC4iWReboEPyHRa7b2i7rzCWCbHVt3SeRMf2FSCtoLv5Hm1Q26juQZHEHvU8SZJf
 dVpzPkUCAwRq8OtEFlNHSEcC93ZGr8sZ6/5a5Hv9FxAWmbGWEC9TmcnuVrFx39ewLXzv
 j1xQlPOrzli8v43rMWNsM1ldRFzKO1GQ0ddnz8mtLg9JIQ5UfO6fG4nUin34jH/TtJQw
 aPe79iCQvxqenUqKpr9DYsdkYTS7AtaHUSOw/UxLh0ylFPtJXJRed6LDb6Dg3QRRks4H
 mydQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:date:from:to:cc:subject:message-id:references
 :mime-version:content-disposition:content-transfer-encoding
 :in-reply-to:user-agent;
 bh=4xwe32xkFcAwGc7g/aZIKI5v5ZroCBBo+J8S7AX1XHY=;
 b=QAtgHunXdvUMnCsKRv+n93uEQ365aub+e/GhmnsN5MbzMyWsNlMzN8GXMp8TgonqiT
 d9Wtsa3gz7+PaVZS64L30I8j2dBr9e/vhMYEYHbjc4W8ISoV+EC0/zwcQZ519wvQHgFo
 kUAB4LMOwbQpQ+PZuabZy3nZ0yq404PQ4y9UxcQ5+fGWpGeonKgENZxA9ayJ2osXlH/R
 UypdU38eajZyBPPU+yPQokFnCA6+0BPAqqwQ7VOPBF9qK9iDiGuFmxC5/oOwHVVz5hXV
 YGLm5kKTQemXvKbYMY2RNT5YaXxZflcr5f3A6MMBYKDsIrpLzVz1WGSeVhldHaCSzpud
 5i9w==
X-Gm-Message-State: AA+aEWb1ClfULgd8Y6GE9nUhpg3pEVDz/WNkRUi51gvA9EzFr+MfN9Mv
 Plhk3VvFP0+ENk977G1PvFY=
X-Google-Smtp-Source: AFSGD/VFCJGkZQjV7BQ46oJLGVHwATFC+ybTsIuw53RO20nDy4zvfAJ912/CEZ29tPucikFDtTTByw==
X-Received: by 2002:a1c:864f:: with SMTP id i76mr835269wmd.83.1544734420065;
 Thu, 13 Dec 2018 12:53:40 -0800 (PST)
Received: from macbook.localdomain ([197.58.120.254])
 by smtp.gmail.com with ESMTPSA id r69sm4773376wmd.4.2018.12.13.12.53.38
 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256);
 Thu, 13 Dec 2018 12:53:39 -0800 (PST)
Date: Thu, 13 Dec 2018 22:53:36 +0200
From: Khaled Hosny <dr.khaled.hosny@HIDDEN>
To: Kaushal Modi <kaushal.modi@HIDDEN>
Subject: Re: 27.0.50; Partial glyphs not rendered for Gujarati with Harfbuzz
 enabled (renders fine using m17n)
Message-ID: <20181213205336.GG2244@HIDDEN>
References: <CAFyQvY3vfV9wqVWo3hAvawRAqpWUSGGf3tebwveM60OCtFkCvA@HIDDEN>
 <CAFyQvY2ZW38s4aMpkkfXLmTw_YrYJ8PRfaRPLugQt03tw0u7Zg@HIDDEN>
 <20181213203102.GF2244@HIDDEN>
 <CAFyQvY0pznh0yM9XJ9y_cEft47SXCmFp+O5rLkzb6o0By-MQuA@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <CAFyQvY0pznh0yM9XJ9y_cEft47SXCmFp+O5rLkzb6o0By-MQuA@HIDDEN>
User-Agent: Mutt/1.11.1 (2018-12-01)
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 33729
Cc: behdad@HIDDEN, 33729 <at> debbugs.gnu.org, far.nasiri.m@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

On Thu, Dec 13, 2018 at 03:43:50PM -0500, Kaushal Modi wrote:
> On Thu, Dec 13, 2018 at 3:31 PM Khaled Hosny <dr.khaled.hosny@HIDDEN>
> wrote:
> 
> >
> > The HarfBuzz rendering of Arabic is the correct one in this screenshot.
> >
> 
> Thanks. So here's the status so far:
> 
> Rendering of Namaste as seen in C-h h (M-x view-hello-file):
> 
> |          | harfbuzz | m17b    |
> |----------+----------+---------|
> | Hindi    | correct  | correct |
> | Gujarati | wrong    | correct |
> | Arabic   | correct  | wrong   |
> 
> 
> 
> > For debugging the such rendering differences, the actual font used by
> > Emacs for a given part of the text need to be known,
> 
> 
> I am using Mukta Vaani font for Gujarati. It is a free font and be
> downloaded from https://ektype.in/mukta-vaani.html.
> 
> The string being rendered is "નમસ્તે".

I tried that font and text with hb-view and the output I get is
identical to m17b. If I pass a wrong script to HarfBuzz (e.g.
--script=latn), I get the same broken output you see in Emacs. So I’m
guessing something is not correctly working in script itemization. Most
likely the FIXME in uni_script(), or the FIXME above the call to
hb_buffer_guess_segment_properties().

Regards,
Khaled




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33729; Package emacs. Full text available.

Message received at 33729 <at> debbugs.gnu.org:


Received: (at 33729) by debbugs.gnu.org; 13 Dec 2018 20:44:37 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Dec 13 15:44:37 2018
Received: from localhost ([127.0.0.1]:46910 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gXXqW-0001Jw-Fn
	for submit <at> debbugs.gnu.org; Thu, 13 Dec 2018 15:44:37 -0500
Received: from mail-lj1-f179.google.com ([209.85.208.179]:44653)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <kaushal.modi@HIDDEN>) id 1gXXqU-0001Jj-99
 for 33729 <at> debbugs.gnu.org; Thu, 13 Dec 2018 15:44:35 -0500
Received: by mail-lj1-f179.google.com with SMTP id k19-v6so2960112lji.11
 for <33729 <at> debbugs.gnu.org>; Thu, 13 Dec 2018 12:44:34 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=mime-version:references:in-reply-to:from:date:message-id:subject:to
 :cc; bh=N6hXOEaMk68qB7NM/pjswjlDhIvzZ4KiUpxCPmJLBBU=;
 b=QrUfAppicL0sTimwaXwSl4l6CTogDONpMYe96EXB/Ll/lAM6VXNCOXmBmWowkOo+Lx
 PmA3XBf71RTACH0VoRdeUr5s1NgbjFVyXU5MmKBwvs/VgzoSQObedEWvQ3z7yCxrKCgf
 tEd34AOPebZiWT1PGcrTKHmgGlmbESF67dHLIfDtDPeqf7dia9WgzKXfNQi+cbUtXWEG
 LXHx2fYHU0ftQ11Zv1WsJL7bZZCgm5F+seN0IjX+qLP5GRpb4FtexINpjiXR0P8bKhnc
 1LKm/a1r4MkQ9GMrEIG7OvDmCpvuR2BCBA83hcAuCALgazJtsaEAMdF1JO1eSYkum3+N
 D/4g==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:mime-version:references:in-reply-to:from:date
 :message-id:subject:to:cc;
 bh=N6hXOEaMk68qB7NM/pjswjlDhIvzZ4KiUpxCPmJLBBU=;
 b=HmQgMNemn1AOmicOwxUI8nZAiCT1RBtIeygzcMpbu2j0InfHai9C02lQa6hMhnBP/p
 qdsiIv2fuejCwqSxbkFd1nE72SGN03eHb9mP1UNsbofXj63fCKZmb9wLGLfFUZfGXJTe
 ZansnxDjuOOZ3L+NEmVZEd3jZuWul5nBE8YG52T8bJhcRc4cJrj/Zy9dpVUpzkDI9DcY
 0YUtVnlOggN85pDw1Ty8GVQl+L/fFtz9kxenkj0Grubg8yJzJxnFOrDjdtgqDtIyEWl/
 z6HFr6txlBKTPZux8/Fto7tDyKAKuNkEM87CFRjxv309+XFWdhXxxk/2c9iBHl7ZH6b3
 T9ZQ==
X-Gm-Message-State: AA+aEWaZxvAv9x3gR5f2X2x7P5ErDcAHysEPgwrYchg6XHlmA2ZR4SWc
 uaAJjvwd/YqJ0VczX0XrugORDq1QXUxiU65r8ro=
X-Google-Smtp-Source: AFSGD/Wo3PXAIoSHuyH8oYaPquyZW91d/WkypGdoCfS91z0DKH0Q3NFt2qlb0585sEMji1kvwy030J11EAVnjDTmrYE=
X-Received: by 2002:a2e:96c6:: with SMTP id d6-v6mr166285ljj.35.1544733867730; 
 Thu, 13 Dec 2018 12:44:27 -0800 (PST)
MIME-Version: 1.0
References: <CAFyQvY3vfV9wqVWo3hAvawRAqpWUSGGf3tebwveM60OCtFkCvA@HIDDEN>
 <CAFyQvY2ZW38s4aMpkkfXLmTw_YrYJ8PRfaRPLugQt03tw0u7Zg@HIDDEN>
 <20181213203102.GF2244@HIDDEN>
In-Reply-To: <20181213203102.GF2244@HIDDEN>
From: Kaushal Modi <kaushal.modi@HIDDEN>
Date: Thu, 13 Dec 2018 15:43:50 -0500
Message-ID: <CAFyQvY0pznh0yM9XJ9y_cEft47SXCmFp+O5rLkzb6o0By-MQuA@HIDDEN>
Subject: Re: 27.0.50; Partial glyphs not rendered for Gujarati with Harfbuzz
 enabled (renders fine using m17n)
To: dr.khaled.hosny@HIDDEN
Content-Type: multipart/alternative; boundary="000000000000756f12057ced615b"
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 33729
Cc: behdad@HIDDEN, 33729 <at> debbugs.gnu.org, far.nasiri.m@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

--000000000000756f12057ced615b
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Thu, Dec 13, 2018 at 3:31 PM Khaled Hosny <dr.khaled.hosny@HIDDEN>
wrote:

>
> The HarfBuzz rendering of Arabic is the correct one in this screenshot.
>

Thanks. So here's the status so far:

Rendering of Namaste as seen in C-h h (M-x view-hello-file):

|          | harfbuzz | m17b    |
|----------+----------+---------|
| Hindi    | correct  | correct |
| Gujarati | wrong    | correct |
| Arabic   | correct  | wrong   |



> For debugging the such rendering differences, the actual font used by
> Emacs for a given part of the text need to be known,


I am using Mukta Vaani font for Gujarati. It is a free font and be
downloaded from https://ektype.in/mukta-vaani.html.

The string being rendered is "=E0=AA=A8=E0=AA=AE=E0=AA=B8=E0=AB=8D=E0=AA=A4=
=E0=AB=87".
By placing the cursor on each of those characters and doing C-u x =3D (on t=
he
m17n build), I get:

(1) =E0=AA=A8

             position: 1610 of 3509 (46%), column: 32
            character: =E0=AA=A8 (displayed as =E0=AA=A8) (codepoint 2728, =
#o5250, #xaa8)
              charset: mule-unicode-0100-24ff (Unicode characters of the
range U+0100..U+24FF.)
code point in charset: 0x3968
               script: gujarati
               syntax: w     which means: word
             category: .:Base, L:Left-to-right (strong)
             to input: type "C-x 8 RET aa8" or "C-x 8 RET GUJARATI LETTER
NA"
          buffer code: #xE0 #xAA #xA8
            file code: #xE0 #xAA #xA8 (encoded by coding system utf-8-unix)
              display: by this font (glyph code)
    xft:-unknown-Mukta Vaani-normal-normal-normal-*-18-*-*-*-*-0-iso10646-1
(#x234)

Character code properties: customize what to show
  name: GUJARATI LETTER NA
  general-category: Lo (Letter, Other)
  decomposition: (2728) ('=E0=AA=A8')

There are text properties here:
  charset              mule-unicode-0100-24ff

(2) =E0=AA=AE

             position: 1611 of 3509 (46%), column: 33
            character: =E0=AA=AE (displayed as =E0=AA=AE) (codepoint 2734, =
#o5256, #xaae)
              charset: mule-unicode-0100-24ff (Unicode characters of the
range U+0100..U+24FF.)
code point in charset: 0x396E
               script: gujarati
               syntax: w     which means: word
             category: .:Base, L:Left-to-right (strong)
             to input: type "C-x 8 RET aae" or "C-x 8 RET GUJARATI LETTER
MA"
          buffer code: #xE0 #xAA #xAE
            file code: #xE0 #xAA #xAE (encoded by coding system utf-8-unix)
              display: by this font (glyph code)
    xft:-unknown-Mukta Vaani-normal-normal-normal-*-18-*-*-*-*-0-iso10646-1
(#x239)

Character code properties: customize what to show
  name: GUJARATI LETTER MA
  general-category: Lo (Letter, Other)
  decomposition: (2734) ('=E0=AA=AE')

There are text properties here:
  charset              mule-unicode-0100-24ff

(3) =E0=AA=B8=E0=AB=8D=E0=AA=A4=E0=AB=87

             position: 1612 of 3509 (46%), column: 34
            character: =E0=AA=B8 (displayed as =E0=AA=B8) (codepoint 2744, =
#o5270, #xab8)
              charset: mule-unicode-0100-24ff (Unicode characters of the
range U+0100..U+24FF.)
code point in charset: 0x3978
               script: gujarati
               syntax: w     which means: word
             category: .:Base, L:Left-to-right (strong)
             to input: type "C-x 8 RET ab8" or "C-x 8 RET GUJARATI LETTER
SA"
          buffer code: #xE0 #xAA #xB8
            file code: #xE0 #xAA #xB8 (encoded by coding system utf-8-unix)
              display: composed to form "=E0=AA=B8=E0=AB=8D=E0=AA=A4=E0=AB=
=87" (see below)

Composed with the following character(s) "=E0=AB=8D=E0=AA=A4=E0=AB=87" usin=
g this font:
  xft:-unknown-Mukta Vaani-normal-normal-normal-*-18-*-*-*-*-0-iso10646-1
by these glyphs:
  [0 3 0 645 8 0 11 11 0 [0 0 8]]
  [0 3 2724 560 11 1 11 11 1 nil]
  [0 3 2759 589 0 -9 -2 16 -11 [-1 0 0]]

Character code properties: customize what to show
  name: GUJARATI LETTER SA
  general-category: Lo (Letter, Other)
  decomposition: (2744) ('=E0=AA=B8')

There are text properties here:
  charset              mule-unicode-0100-24ff


=3D=3D=3D=3D=3D


On harfbuzz build, the "=E0=AA=B8=E0=AB=8D=E0=AA=A4=E0=AB=87" part is diffe=
rent.. I can place the cursor
separately on =E0=AA=B8=E0=AB=8D and =E0=AA=A4=E0=AB=87, do C-u x =3D and I=
 get:

(3.1) =E0=AA=B8=E0=AB=8D
             position: 1612 of 3509 (46%), column: 34
            character: =E0=AA=B8 (displayed as =E0=AA=B8) (codepoint 2744, =
#o5270, #xab8)
              charset: mule-unicode-0100-24ff (Unicode characters of the
range U+0100..U+24FF.)
code point in charset: 0x3978
               script: gujarati
               syntax: w     which means: word
             category: .:Base, L:Left-to-right (strong)
             to input: type "C-x 8 RET ab8" or "C-x 8 RET GUJARATI LETTER
SA"
          buffer code: #xE0 #xAA #xB8
            file code: #xE0 #xAA #xB8 (encoded by coding system utf-8-unix)
              display: by this font (glyph code)
    xft:-unknown-Mukta Vaani-normal-normal-normal-*-18-*-*-*-*-0-iso10646-1
(#x241)

Character code properties: customize what to show
  name: GUJARATI LETTER SA
  general-category: Lo (Letter, Other)
  decomposition: (2744) ('=E0=AA=B8')

There are text properties here:
  charset              mule-unicode-0100-24ff

(3.2) =E0=AA=A4=E0=AB=87

             position: 1614 of 3509 (46%), column: 35
            character: =E0=AA=A4 (displayed as =E0=AA=A4) (codepoint 2724, =
#o5244, #xaa4)
              charset: mule-unicode-0100-24ff (Unicode characters of the
range U+0100..U+24FF.)
code point in charset: 0x3964
               script: gujarati
               syntax: w     which means: word
             category: .:Base, L:Left-to-right (strong)
             to input: type "C-x 8 RET aa4" or "C-x 8 RET GUJARATI LETTER
TA"
          buffer code: #xE0 #xAA #xA4
            file code: #xE0 #xAA #xA4 (encoded by coding system utf-8-unix)
              display: by this font (glyph code)
    xft:-unknown-Mukta Vaani-normal-normal-normal-*-18-*-*-*-*-0-iso10646-1
(#x230)

Character code properties: customize what to show
  name: GUJARATI LETTER TA
  general-category: Lo (Letter, Other)
  decomposition: (2724) ('=E0=AA=A4')

There are text properties here:
  charset              mule-unicode-0100-24ff



then the text and
> the font can be checked against vanilla HarfBuzz (e.g. using the hb-view
> command line tool); if it gives the same rendering then it is either a
> HarfBuzz or font issue, if not then it is a bug in the HarfBuzz
> integration code in Emacs.
>

--000000000000756f12057ced615b
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: base64

PGRpdiBkaXI9Imx0ciI+PGRpdiBkaXI9Imx0ciI+PGRpdiBkaXI9Imx0ciI+PGRpdiBkaXI9Imx0
ciI+PGRpdiBkaXI9Imx0ciI+PGRpdiBkaXI9Imx0ciI+PGRpdiBkaXI9Imx0ciI+PGRpdiBkaXI9
Imx0ciI+PGRpdiBkaXI9Imx0ciI+PGRpdiBkaXI9Imx0ciI+PGRpdiBkaXI9Imx0ciI+PGRpdiBk
aXI9Imx0ciI+PGRpdiBjbGFzcz0iZ21haWxfcXVvdGUiPjxkaXYgZGlyPSJsdHIiPk9uIFRodSwg
RGVjIDEzLCAyMDE4IGF0IDM6MzEgUE0gS2hhbGVkIEhvc255ICZsdDs8YSBocmVmPSJtYWlsdG86
ZHIua2hhbGVkLmhvc255QGdtYWlsLmNvbSI+ZHIua2hhbGVkLmhvc255QGdtYWlsLmNvbTwvYT4m
Z3Q7IHdyb3RlOjxicj48L2Rpdj48YmxvY2txdW90ZSBjbGFzcz0iZ21haWxfcXVvdGUiIHN0eWxl
PSJtYXJnaW46MHB4IDBweCAwcHggMC44ZXg7Ym9yZGVyLWxlZnQ6MXB4IHNvbGlkIHJnYigyMDQs
MjA0LDIwNCk7cGFkZGluZy1sZWZ0OjFleCI+PGJyPg0KVGhlIEhhcmZCdXp6IHJlbmRlcmluZyBv
ZiBBcmFiaWMgaXMgdGhlIGNvcnJlY3Qgb25lIGluIHRoaXMgc2NyZWVuc2hvdC48YnI+PC9ibG9j
a3F1b3RlPjxkaXY+PGJyPjwvZGl2PjxkaXY+VGhhbmtzLiBTbyBoZXJlJiMzOTtzIHRoZSBzdGF0
dXMgc28gZmFyOjwvZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+UmVuZGVyaW5nIG9mIE5hbWFzdGUg
YXMgc2VlbiBpbiBDLWggaCAoTS14IHZpZXctaGVsbG8tZmlsZSk6PGJyPjxicj58wqDCoMKgwqDC
oMKgwqDCoMKgIHwgaGFyZmJ1enogfCBtMTdiwqDCoMKgIHw8YnI+fC0tLS0tLS0tLS0rLS0tLS0t
LS0tLSstLS0tLS0tLS18PGJyPnwgSGluZGnCoMKgwqAgfCBjb3JyZWN0wqAgfCBjb3JyZWN0IHw8
YnI+fCBHdWphcmF0aSB8IHdyb25nwqDCoMKgIHwgY29ycmVjdCB8PGJyPnwgQXJhYmljwqDCoCB8
IGNvcnJlY3TCoCB8IHdyb25nwqDCoCB8PGJyPjxicj48L2Rpdj48ZGl2PsKgPC9kaXY+PGJsb2Nr
cXVvdGUgY2xhc3M9ImdtYWlsX3F1b3RlIiBzdHlsZT0ibWFyZ2luOjBweCAwcHggMHB4IDAuOGV4
O2JvcmRlci1sZWZ0OjFweCBzb2xpZCByZ2IoMjA0LDIwNCwyMDQpO3BhZGRpbmctbGVmdDoxZXgi
Pg0KRm9yIGRlYnVnZ2luZyB0aGUgc3VjaCByZW5kZXJpbmcgZGlmZmVyZW5jZXMsIHRoZSBhY3R1
YWwgZm9udCB1c2VkIGJ5PGJyPg0KRW1hY3MgZm9yIGEgZ2l2ZW4gcGFydCBvZiB0aGUgdGV4dCBu
ZWVkIHRvIGJlIGtub3duLDwvYmxvY2txdW90ZT48ZGl2Pjxicj48L2Rpdj48ZGl2PkkgYW0gdXNp
bmcgTXVrdGEgVmFhbmkgZm9udCBmb3IgR3VqYXJhdGkuIEl0IGlzIGEgZnJlZSBmb250IGFuZCBi
ZSBkb3dubG9hZGVkIGZyb20gPGEgaHJlZj0iaHR0cHM6Ly9la3R5cGUuaW4vbXVrdGEtdmFhbmku
aHRtbCI+aHR0cHM6Ly9la3R5cGUuaW4vbXVrdGEtdmFhbmkuaHRtbDwvYT4uPGJyPjwvZGl2Pjxk
aXY+wqA8L2Rpdj48ZGl2PlRoZSBzdHJpbmcgYmVpbmcgcmVuZGVyZWQgaXMgJnF1b3Q74Kqo4Kqu
4Kq44KuN4Kqk4KuHJnF1b3Q7LiANCjxkaXY+QnkgcGxhY2luZyB0aGUgY3Vyc29yIG9uIGVhY2gg
b2YgdGhvc2UgY2hhcmFjdGVycyBhbmQgZG9pbmcgQy11IHggPSAob24gdGhlIG0xN24gYnVpbGQp
LCBJIGdldDo8L2Rpdj48ZGl2PjwvZGl2Pg0KDQo8L2Rpdj48ZGl2Pjxicj48L2Rpdj48ZGl2Pigx
KSANCuCqqA0KDQo8L2Rpdj48ZGl2Pjxicj48L2Rpdj48ZGl2PsKgwqDCoMKgwqDCoMKgwqDCoMKg
wqDCoCBwb3NpdGlvbjogMTYxMCBvZiAzNTA5ICg0NiUpLCBjb2x1bW46IDMyPGJyPsKgwqDCoMKg
wqDCoMKgwqDCoMKgwqAgY2hhcmFjdGVyOiDgqqggKGRpc3BsYXllZCBhcyDgqqgpIChjb2RlcG9p
bnQgMjcyOCwgI281MjUwLCAjeGFhOCk8YnI+wqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqAgY2hh
cnNldDogbXVsZS11bmljb2RlLTAxMDAtMjRmZiAoVW5pY29kZSBjaGFyYWN0ZXJzIG9mIHRoZSBy
YW5nZSBVKzAxMDAuLlUrMjRGRi4pPGJyPmNvZGUgcG9pbnQgaW4gY2hhcnNldDogMHgzOTY4PGJy
PsKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqAgc2NyaXB0OiBndWphcmF0aTxicj7CoMKgwqDC
oMKgwqDCoMKgwqDCoMKgwqDCoMKgIHN5bnRheDogdyDCoMKgwqAgd2hpY2ggbWVhbnM6IHdvcmQ8
YnI+wqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgIGNhdGVnb3J5OiAuOkJhc2UsIEw6TGVmdC10by1y
aWdodCAoc3Ryb25nKTxicj7CoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqAgdG8gaW5wdXQ6IHR5cGUg
JnF1b3Q7Qy14IDggUkVUIGFhOCZxdW90OyBvciAmcXVvdDtDLXggOCBSRVQgR1VKQVJBVEkgTEVU
VEVSIE5BJnF1b3Q7PGJyPsKgwqDCoMKgwqDCoMKgwqDCoCBidWZmZXIgY29kZTogI3hFMCAjeEFB
ICN4QTg8YnI+wqDCoMKgwqDCoMKgwqDCoMKgwqDCoCBmaWxlIGNvZGU6ICN4RTAgI3hBQSAjeEE4
IChlbmNvZGVkIGJ5IGNvZGluZyBzeXN0ZW0gdXRmLTgtdW5peCk8YnI+wqDCoMKgwqDCoMKgwqDC
oMKgwqDCoMKgwqAgZGlzcGxheTogYnkgdGhpcyBmb250IChnbHlwaCBjb2RlKTxicj7CoMKgwqAg
eGZ0Oi11bmtub3duLU11a3RhIFZhYW5pLW5vcm1hbC1ub3JtYWwtbm9ybWFsLSotMTgtKi0qLSot
Ki0wLWlzbzEwNjQ2LTEgKCN4MjM0KTxicj48YnI+Q2hhcmFjdGVyIGNvZGUgcHJvcGVydGllczog
Y3VzdG9taXplIHdoYXQgdG8gc2hvdzxicj7CoCBuYW1lOiBHVUpBUkFUSSBMRVRURVIgTkE8YnI+
wqAgZ2VuZXJhbC1jYXRlZ29yeTogTG8gKExldHRlciwgT3RoZXIpPGJyPsKgIGRlY29tcG9zaXRp
b246ICgyNzI4KSAoJiMzOTvgqqgmIzM5Oyk8YnI+PGJyPlRoZXJlIGFyZSB0ZXh0IHByb3BlcnRp
ZXMgaGVyZTo8YnI+wqAgY2hhcnNldMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgIG11bGUtdW5p
Y29kZS0wMTAwLTI0ZmY8YnI+PC9kaXY+PGJyPjxkaXY+KDIpIOCqrjwvZGl2PjxkaXY+PGJyPjwv
ZGl2PjxkaXY+wqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgIHBvc2l0aW9uOiAxNjExIG9mIDM1MDkg
KDQ2JSksIGNvbHVtbjogMzM8YnI+wqDCoMKgwqDCoMKgwqDCoMKgwqDCoCBjaGFyYWN0ZXI6IOCq
riAoZGlzcGxheWVkIGFzIOCqrikgKGNvZGVwb2ludCAyNzM0LCAjbzUyNTYsICN4YWFlKTxicj7C
oMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoCBjaGFyc2V0OiBtdWxlLXVuaWNvZGUtMDEwMC0yNGZm
IChVbmljb2RlIGNoYXJhY3RlcnMgb2YgdGhlIHJhbmdlIFUrMDEwMC4uVSsyNEZGLik8YnI+Y29k
ZSBwb2ludCBpbiBjaGFyc2V0OiAweDM5NkU8YnI+wqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDC
oCBzY3JpcHQ6IGd1amFyYXRpPGJyPsKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqAgc3ludGF4
OiB3IMKgwqDCoCB3aGljaCBtZWFuczogd29yZDxicj7CoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqAg
Y2F0ZWdvcnk6IC46QmFzZSwgTDpMZWZ0LXRvLXJpZ2h0IChzdHJvbmcpPGJyPsKgwqDCoMKgwqDC
oMKgwqDCoMKgwqDCoCB0byBpbnB1dDogdHlwZSAmcXVvdDtDLXggOCBSRVQgYWFlJnF1b3Q7IG9y
ICZxdW90O0MteCA4IFJFVCBHVUpBUkFUSSBMRVRURVIgTUEmcXVvdDs8YnI+wqDCoMKgwqDCoMKg
wqDCoMKgIGJ1ZmZlciBjb2RlOiAjeEUwICN4QUEgI3hBRTxicj7CoMKgwqDCoMKgwqDCoMKgwqDC
oMKgIGZpbGUgY29kZTogI3hFMCAjeEFBICN4QUUgKGVuY29kZWQgYnkgY29kaW5nIHN5c3RlbSB1
dGYtOC11bml4KTxicj7CoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoCBkaXNwbGF5OiBieSB0aGlz
IGZvbnQgKGdseXBoIGNvZGUpPGJyPsKgwqDCoCB4ZnQ6LXVua25vd24tTXVrdGEgVmFhbmktbm9y
bWFsLW5vcm1hbC1ub3JtYWwtKi0xOC0qLSotKi0qLTAtaXNvMTA2NDYtMSAoI3gyMzkpPGJyPjxi
cj5DaGFyYWN0ZXIgY29kZSBwcm9wZXJ0aWVzOiBjdXN0b21pemUgd2hhdCB0byBzaG93PGJyPsKg
IG5hbWU6IEdVSkFSQVRJIExFVFRFUiBNQTxicj7CoCBnZW5lcmFsLWNhdGVnb3J5OiBMbyAoTGV0
dGVyLCBPdGhlcik8YnI+wqAgZGVjb21wb3NpdGlvbjogKDI3MzQpICgmIzM5O+CqriYjMzk7KTxi
cj48YnI+VGhlcmUgYXJlIHRleHQgcHJvcGVydGllcyBoZXJlOjxicj7CoCBjaGFyc2V0wqDCoMKg
wqDCoMKgwqDCoMKgwqDCoMKgwqAgbXVsZS11bmljb2RlLTAxMDAtMjRmZjxicj48YnI+PC9kaXY+
PGRpdj4oMykgDQoNCuCquOCrjeCqpOCrhyA8YnI+PC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj7C
oMKgwqDCoMKgwqDCoMKgwqDCoMKgwqAgcG9zaXRpb246IDE2MTIgb2YgMzUwOSAoNDYlKSwgY29s
dW1uOiAzNDxicj7CoMKgwqDCoMKgwqDCoMKgwqDCoMKgIGNoYXJhY3Rlcjog4Kq4IChkaXNwbGF5
ZWQgYXMg4Kq4KSAoY29kZXBvaW50IDI3NDQsICNvNTI3MCwgI3hhYjgpPGJyPsKgwqDCoMKgwqDC
oMKgwqDCoMKgwqDCoMKgIGNoYXJzZXQ6IG11bGUtdW5pY29kZS0wMTAwLTI0ZmYgKFVuaWNvZGUg
Y2hhcmFjdGVycyBvZiB0aGUgcmFuZ2UgVSswMTAwLi5VKzI0RkYuKTxicj5jb2RlIHBvaW50IGlu
IGNoYXJzZXQ6IDB4Mzk3ODxicj7CoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgIHNjcmlwdDog
Z3VqYXJhdGk8YnI+wqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoCBzeW50YXg6IHcgwqDCoMKg
IHdoaWNoIG1lYW5zOiB3b3JkPGJyPsKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoCBjYXRlZ29yeTog
LjpCYXNlLCBMOkxlZnQtdG8tcmlnaHQgKHN0cm9uZyk8YnI+wqDCoMKgwqDCoMKgwqDCoMKgwqDC
oMKgIHRvIGlucHV0OiB0eXBlICZxdW90O0MteCA4IFJFVCBhYjgmcXVvdDsgb3IgJnF1b3Q7Qy14
IDggUkVUIEdVSkFSQVRJIExFVFRFUiBTQSZxdW90Ozxicj7CoMKgwqDCoMKgwqDCoMKgwqAgYnVm
ZmVyIGNvZGU6ICN4RTAgI3hBQSAjeEI4PGJyPsKgwqDCoMKgwqDCoMKgwqDCoMKgwqAgZmlsZSBj
b2RlOiAjeEUwICN4QUEgI3hCOCAoZW5jb2RlZCBieSBjb2Rpbmcgc3lzdGVtIHV0Zi04LXVuaXgp
PGJyPsKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgIGRpc3BsYXk6IGNvbXBvc2VkIHRvIGZvcm0g
JnF1b3Q74Kq44KuN4Kqk4KuHJnF1b3Q7IChzZWUgYmVsb3cpPGJyPjxicj5Db21wb3NlZCB3aXRo
IHRoZSBmb2xsb3dpbmcgY2hhcmFjdGVyKHMpICZxdW90O+CrjeCqpOCrhyZxdW90OyB1c2luZyB0
aGlzIGZvbnQ6PGJyPsKgIHhmdDotdW5rbm93bi1NdWt0YSBWYWFuaS1ub3JtYWwtbm9ybWFsLW5v
cm1hbC0qLTE4LSotKi0qLSotMC1pc28xMDY0Ni0xPGJyPmJ5IHRoZXNlIGdseXBoczo8YnI+wqAg
WzAgMyAwIDY0NSA4IDAgMTEgMTEgMCBbMCAwIDhdXTxicj7CoCBbMCAzIDI3MjQgNTYwIDExIDEg
MTEgMTEgMSBuaWxdPGJyPsKgIFswIDMgMjc1OSA1ODkgMCAtOSAtMiAxNiAtMTEgWy0xIDAgMF1d
PGJyPjxicj5DaGFyYWN0ZXIgY29kZSBwcm9wZXJ0aWVzOiBjdXN0b21pemUgd2hhdCB0byBzaG93
PGJyPsKgIG5hbWU6IEdVSkFSQVRJIExFVFRFUiBTQTxicj7CoCBnZW5lcmFsLWNhdGVnb3J5OiBM
byAoTGV0dGVyLCBPdGhlcik8YnI+wqAgZGVjb21wb3NpdGlvbjogKDI3NDQpICgmIzM5O+CquCYj
Mzk7KTxicj48YnI+VGhlcmUgYXJlIHRleHQgcHJvcGVydGllcyBoZXJlOjxicj7CoCBjaGFyc2V0
wqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqAgbXVsZS11bmljb2RlLTAxMDAtMjRmZjxicj48YnI+
PC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj49PT09PTwvZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+
PGJyPjwvZGl2PjxkaXY+T24gaGFyZmJ1enogYnVpbGQsIHRoZSAmcXVvdDvgqrjgq43gqqTgq4cm
cXVvdDsgcGFydCBpcyBkaWZmZXJlbnQuLiBJIGNhbiBwbGFjZSB0aGUgY3Vyc29yIHNlcGFyYXRl
bHkgb24g4Kq44KuNIGFuZCDgqqTgq4csIGRvIEMtdSB4ID0gYW5kIEkgZ2V0OjwvZGl2PjxkaXY+
PGJyPjwvZGl2PjxkaXY+KDMuMSkg4Kq44KuNwqANCg0KPC9kaXY+PGRpdj7CoMKgwqDCoMKgwqDC
oMKgwqDCoMKgwqAgcG9zaXRpb246IDE2MTIgb2YgMzUwOSAoNDYlKSwgY29sdW1uOiAzNDxicj7C
oMKgwqDCoMKgwqDCoMKgwqDCoMKgIGNoYXJhY3Rlcjog4Kq4IChkaXNwbGF5ZWQgYXMg4Kq4KSAo
Y29kZXBvaW50IDI3NDQsICNvNTI3MCwgI3hhYjgpPGJyPsKgwqDCoMKgwqDCoMKgwqDCoMKgwqDC
oMKgIGNoYXJzZXQ6IG11bGUtdW5pY29kZS0wMTAwLTI0ZmYgKFVuaWNvZGUgY2hhcmFjdGVycyBv
ZiB0aGUgcmFuZ2UgVSswMTAwLi5VKzI0RkYuKTxicj5jb2RlIHBvaW50IGluIGNoYXJzZXQ6IDB4
Mzk3ODxicj7CoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgIHNjcmlwdDogZ3VqYXJhdGk8YnI+
wqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoCBzeW50YXg6IHcgwqDCoMKgIHdoaWNoIG1lYW5z
OiB3b3JkPGJyPsKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoCBjYXRlZ29yeTogLjpCYXNlLCBMOkxl
ZnQtdG8tcmlnaHQgKHN0cm9uZyk8YnI+wqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgIHRvIGlucHV0
OiB0eXBlICZxdW90O0MteCA4IFJFVCBhYjgmcXVvdDsgb3IgJnF1b3Q7Qy14IDggUkVUIEdVSkFS
QVRJIExFVFRFUiBTQSZxdW90Ozxicj7CoMKgwqDCoMKgwqDCoMKgwqAgYnVmZmVyIGNvZGU6ICN4
RTAgI3hBQSAjeEI4PGJyPsKgwqDCoMKgwqDCoMKgwqDCoMKgwqAgZmlsZSBjb2RlOiAjeEUwICN4
QUEgI3hCOCAoZW5jb2RlZCBieSBjb2Rpbmcgc3lzdGVtIHV0Zi04LXVuaXgpPGJyPsKgwqDCoMKg
wqDCoMKgwqDCoMKgwqDCoMKgIGRpc3BsYXk6IGJ5IHRoaXMgZm9udCAoZ2x5cGggY29kZSk8YnI+
wqDCoMKgIHhmdDotdW5rbm93bi1NdWt0YSBWYWFuaS1ub3JtYWwtbm9ybWFsLW5vcm1hbC0qLTE4
LSotKi0qLSotMC1pc28xMDY0Ni0xICgjeDI0MSk8YnI+PGJyPkNoYXJhY3RlciBjb2RlIHByb3Bl
cnRpZXM6IGN1c3RvbWl6ZSB3aGF0IHRvIHNob3c8YnI+wqAgbmFtZTogR1VKQVJBVEkgTEVUVEVS
IFNBPGJyPsKgIGdlbmVyYWwtY2F0ZWdvcnk6IExvIChMZXR0ZXIsIE90aGVyKTxicj7CoCBkZWNv
bXBvc2l0aW9uOiAoMjc0NCkgKCYjMzk74Kq4JiMzOTspPGJyPjxicj5UaGVyZSBhcmUgdGV4dCBw
cm9wZXJ0aWVzIGhlcmU6PGJyPsKgIGNoYXJzZXTCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoCBt
dWxlLXVuaWNvZGUtMDEwMC0yNGZmPGJyPjxicj48L2Rpdj4NCigzLjIpIOCqpOCrhw0KDQoNCg0K
PGRpdj48YnI+PC9kaXY+PGRpdj7CoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqAgcG9zaXRpb246IDE2
MTQgb2YgMzUwOSAoNDYlKSwgY29sdW1uOiAzNTxicj7CoMKgwqDCoMKgwqDCoMKgwqDCoMKgIGNo
YXJhY3Rlcjog4KqkIChkaXNwbGF5ZWQgYXMg4KqkKSAoY29kZXBvaW50IDI3MjQsICNvNTI0NCwg
I3hhYTQpPGJyPsKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgIGNoYXJzZXQ6IG11bGUtdW5pY29k
ZS0wMTAwLTI0ZmYgKFVuaWNvZGUgY2hhcmFjdGVycyBvZiB0aGUgcmFuZ2UgVSswMTAwLi5VKzI0
RkYuKTxicj5jb2RlIHBvaW50IGluIGNoYXJzZXQ6IDB4Mzk2NDxicj7CoMKgwqDCoMKgwqDCoMKg
wqDCoMKgwqDCoMKgIHNjcmlwdDogZ3VqYXJhdGk8YnI+wqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKg
wqDCoCBzeW50YXg6IHcgwqDCoMKgIHdoaWNoIG1lYW5zOiB3b3JkPGJyPsKgwqDCoMKgwqDCoMKg
wqDCoMKgwqDCoCBjYXRlZ29yeTogLjpCYXNlLCBMOkxlZnQtdG8tcmlnaHQgKHN0cm9uZyk8YnI+
wqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgIHRvIGlucHV0OiB0eXBlICZxdW90O0MteCA4IFJFVCBh
YTQmcXVvdDsgb3IgJnF1b3Q7Qy14IDggUkVUIEdVSkFSQVRJIExFVFRFUiBUQSZxdW90Ozxicj7C
oMKgwqDCoMKgwqDCoMKgwqAgYnVmZmVyIGNvZGU6ICN4RTAgI3hBQSAjeEE0PGJyPsKgwqDCoMKg
wqDCoMKgwqDCoMKgwqAgZmlsZSBjb2RlOiAjeEUwICN4QUEgI3hBNCAoZW5jb2RlZCBieSBjb2Rp
bmcgc3lzdGVtIHV0Zi04LXVuaXgpPGJyPsKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgIGRpc3Bs
YXk6IGJ5IHRoaXMgZm9udCAoZ2x5cGggY29kZSk8YnI+wqDCoMKgIHhmdDotdW5rbm93bi1NdWt0
YSBWYWFuaS1ub3JtYWwtbm9ybWFsLW5vcm1hbC0qLTE4LSotKi0qLSotMC1pc28xMDY0Ni0xICgj
eDIzMCk8YnI+PGJyPkNoYXJhY3RlciBjb2RlIHByb3BlcnRpZXM6IGN1c3RvbWl6ZSB3aGF0IHRv
IHNob3c8YnI+wqAgbmFtZTogR1VKQVJBVEkgTEVUVEVSIFRBPGJyPsKgIGdlbmVyYWwtY2F0ZWdv
cnk6IExvIChMZXR0ZXIsIE90aGVyKTxicj7CoCBkZWNvbXBvc2l0aW9uOiAoMjcyNCkgKCYjMzk7
4KqkJiMzOTspPGJyPjxicj5UaGVyZSBhcmUgdGV4dCBwcm9wZXJ0aWVzIGhlcmU6PGJyPsKgIGNo
YXJzZXTCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoCBtdWxlLXVuaWNvZGUtMDEwMC0yNGZmPGJy
Pjxicj48YnI+PC9kaXY+PGRpdj48YnI+PC9kaXY+PGJsb2NrcXVvdGUgY2xhc3M9ImdtYWlsX3F1
b3RlIiBzdHlsZT0ibWFyZ2luOjBweCAwcHggMHB4IDAuOGV4O2JvcmRlci1sZWZ0OjFweCBzb2xp
ZCByZ2IoMjA0LDIwNCwyMDQpO3BhZGRpbmctbGVmdDoxZXgiPiB0aGVuIHRoZSB0ZXh0IGFuZDxi
cj4NCnRoZSBmb250IGNhbiBiZSBjaGVja2VkIGFnYWluc3QgdmFuaWxsYSBIYXJmQnV6eiAoZS5n
LiB1c2luZyB0aGUgaGItdmlldzxicj4NCmNvbW1hbmQgbGluZSB0b29sKTsgaWYgaXQgZ2l2ZXMg
dGhlIHNhbWUgcmVuZGVyaW5nIHRoZW4gaXQgaXMgZWl0aGVyIGE8YnI+DQpIYXJmQnV6eiBvciBm
b250IGlzc3VlLCBpZiBub3QgdGhlbiBpdCBpcyBhIGJ1ZyBpbiB0aGUgSGFyZkJ1eno8YnI+DQpp
bnRlZ3JhdGlvbiBjb2RlIGluIEVtYWNzLjxicj4NCjwvYmxvY2txdW90ZT48L2Rpdj48L2Rpdj48
L2Rpdj48L2Rpdj48L2Rpdj48L2Rpdj48L2Rpdj48L2Rpdj48L2Rpdj48L2Rpdj48L2Rpdj48L2Rp
dj48L2Rpdj4NCg==
--000000000000756f12057ced615b--




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33729; Package emacs. Full text available.

Message received at 33729 <at> debbugs.gnu.org:


Received: (at 33729) by debbugs.gnu.org; 13 Dec 2018 20:34:01 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Dec 13 15:34:01 2018
Received: from localhost ([127.0.0.1]:46867 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gXXgG-0000vZ-QA
	for submit <at> debbugs.gnu.org; Thu, 13 Dec 2018 15:34:01 -0500
Received: from mail-wm1-f50.google.com ([209.85.128.50]:35118)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dr.khaled.hosny@HIDDEN>) id 1gXXdY-0000oo-5g
 for 33729 <at> debbugs.gnu.org; Thu, 13 Dec 2018 15:31:12 -0500
Received: by mail-wm1-f50.google.com with SMTP id c126so3734219wmh.0
 for <33729 <at> debbugs.gnu.org>; Thu, 13 Dec 2018 12:31:12 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=date:from:to:cc:subject:message-id:references:mime-version
 :content-disposition:in-reply-to:user-agent;
 bh=gzqAOn5n3m6E+bfjb5C6g5rwvxzYnUzUUFhhiST//10=;
 b=kkFlKQ9TyxoV+iuvizMdkPWWeQZNyQOCaIyvsA/cl94zl0yQg1fZFW85Mmw1DEj0Ra
 RRRNJjrL3vYZyBcDAvdywxeh/ZL0uwkmar37EN2rX4qPXgZytgJt641VERxFuketZo2w
 0+KhQKdf4U6N8JxzyifgfXbxsJrZKsjueOLDtk150hDecfQOY/nmKraKDrxXwe738G0C
 ogiLxZ62ZAMNpW/d1xhnQW6OX7/bysnFRgDRbl7sX30Kh9By/Tqhqt1fCtQPh99CDCdO
 llnjxw7OeiKuOrJE+8bbh0FekD1vyzKELS4lzj1Co7ClC5FddvoH8PZ/0MRbSGVFvuAn
 tdYQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:date:from:to:cc:subject:message-id:references
 :mime-version:content-disposition:in-reply-to:user-agent;
 bh=gzqAOn5n3m6E+bfjb5C6g5rwvxzYnUzUUFhhiST//10=;
 b=kC1IUGl1btQ1wWczUx//N6WrqegexE1ray5VtEFUkragW6nzMWDCUz3vfRBwA5dJmf
 XoDOMHsAcpHh7GWRlbYteEyG+XodBMrOQ9LrTOkw88nT1uiYRsyfbTHxylAc/mM0ZfKd
 G7V86srWlExwlFAS69n2qez7vNYOxU+kXo58RkQeY0zk9r2rR1fwbpj2fIcRCh3oY0dU
 r516oC5vzkwCtlEPYyKfwAnla5+jmTLyvAPlqcEIJ4b5BFpsGJkckQ1TO2lMr9WdIkr4
 fga10Iuw3uda2WOy5wbXZZvlUnI5LkCxY968tHlDHnMhJGd/hvdvL6JP8gTYAYegATAU
 RjSA==
X-Gm-Message-State: AA+aEWb4Jf2D3rTHPWJT42fIt9pnK8WbiDGBK8CZrxPWSaNoTjHro+2E
 a0UOcp1PlXq0wrGFAhvWwwk=
X-Google-Smtp-Source: AFSGD/Wf33wZ3dc1SNUo4/rRI0mdEQGKROQJ+p8Nx0P9iHyQWzd5Q0Wa7JjEJEoDwpy1nDLWRA2siA==
X-Received: by 2002:a1c:aad2:: with SMTP id t201mr798795wme.148.1544733066299; 
 Thu, 13 Dec 2018 12:31:06 -0800 (PST)
Received: from macbook.localdomain ([197.58.120.254])
 by smtp.gmail.com with ESMTPSA id u10sm2441415wrr.33.2018.12.13.12.31.04
 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256);
 Thu, 13 Dec 2018 12:31:05 -0800 (PST)
Date: Thu, 13 Dec 2018 22:31:02 +0200
From: Khaled Hosny <dr.khaled.hosny@HIDDEN>
To: Kaushal Modi <kaushal.modi@HIDDEN>
Subject: Re: 27.0.50; Partial glyphs not rendered for Gujarati with Harfbuzz
 enabled (renders fine using m17n)
Message-ID: <20181213203102.GF2244@HIDDEN>
References: <CAFyQvY3vfV9wqVWo3hAvawRAqpWUSGGf3tebwveM60OCtFkCvA@HIDDEN>
 <CAFyQvY2ZW38s4aMpkkfXLmTw_YrYJ8PRfaRPLugQt03tw0u7Zg@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
In-Reply-To: <CAFyQvY2ZW38s4aMpkkfXLmTw_YrYJ8PRfaRPLugQt03tw0u7Zg@HIDDEN>
User-Agent: Mutt/1.11.1 (2018-12-01)
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 33729
X-Mailman-Approved-At: Thu, 13 Dec 2018 15:33:59 -0500
Cc: behdad@HIDDEN, 33729 <at> debbugs.gnu.org, far.nasiri.m@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

On Thu, Dec 13, 2018 at 03:25:16PM -0500, Kaushal Modi wrote:
> >
> > Screenshot to explain this issue: https://i.imgtc.com/md9Yz7X.png
> >
> 
> I don't know Arabic. But from that same screenshot, it's evident that the
> rendering of that same text is quite different between m17n and harfbuzz.

The HarfBuzz rendering of Arabic is the correct one in this screenshot.
For debugging the such rendering differences, the actual font used by
Emacs for a given part of the text need to be known, then the text and
the font can be checked against vanilla HarfBuzz (e.g. using the hb-view
command line tool); if it gives the same rendering then it is either a
HarfBuzz or font issue, if not then it is a bug in the HarfBuzz
integration code in Emacs.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33729; Package emacs. Full text available.

Message received at 33729 <at> debbugs.gnu.org:


Received: (at 33729) by debbugs.gnu.org; 13 Dec 2018 20:26:03 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Dec 13 15:26:03 2018
Received: from localhost ([127.0.0.1]:46816 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gXXYY-0000bV-Ok
	for submit <at> debbugs.gnu.org; Thu, 13 Dec 2018 15:26:03 -0500
Received: from mail-lf1-f43.google.com ([209.85.167.43]:40242)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <kaushal.modi@HIDDEN>) id 1gXXYW-0000au-BX
 for 33729 <at> debbugs.gnu.org; Thu, 13 Dec 2018 15:26:01 -0500
Received: by mail-lf1-f43.google.com with SMTP id v5so2569234lfe.7
 for <33729 <at> debbugs.gnu.org>; Thu, 13 Dec 2018 12:26:00 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=mime-version:references:in-reply-to:from:date:message-id:subject:to
 :cc; bh=YFDrQBZFIsJn6DKuyqNnw/+ye/KcvCgvo7oZLX6JQYs=;
 b=gbvH05I+22OlGPq36tWVzMsVF4nd8wnuGMM8ipizBawqZoz8kE4mhAAutLYfGfgpIU
 5UlJGK6HaSF1bcoq10wibxF4VF5VtLn7RIfdUNps80oGkP6nEhYuyVIu7oNPR0s/BnFE
 gRoU0qbGUG83rejsu1Pe89F+v6KjUgXL674O9rf4I2LeGEYDtZfwVqgrrgcoRZZRrPto
 VbtO2Mrc9cjAhhS/lVvaczWY7DxaQij/wQ1l9iT3U9L+76rT14ADwiT7Ebt5BIYg4PCB
 3KsAkpKayV2CLgPMwTB2CLU4kvox0LzUuVW0CW34bd+2k6IQleOQglnCLjWQ5s6lmVau
 N7IA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:mime-version:references:in-reply-to:from:date
 :message-id:subject:to:cc;
 bh=YFDrQBZFIsJn6DKuyqNnw/+ye/KcvCgvo7oZLX6JQYs=;
 b=NOhF6gJxXG4yINBZ+UFuQQ2yBGMqIuYibJJ5ifDTbYiahtDdytAfw3ARtL97K/Bcwf
 gIhhqGzuZdnsidikg7pgx039P1Ca0/DPSsBankhktteXJuWwcwByaj5v63ENgZUgwTKh
 4UyE47D5GfdAq6MS4NF8uBJcoVzZjU9pXtmMXS2xqunaZpqCfi8UZyPjXY6oCnf2bDwM
 7M4boxTawvmznLQ+0lZbQ9ThM/DTZiJ5HG1OTMl40mGZJ7jlnFrOvPvUAnLlYNYgnAHO
 jvJGojj4bGGjPcknLlGk3ntll+0m71DuzsZGJLsPymAhRLN7VrPMsWByR99gFeBYdu24
 khPw==
X-Gm-Message-State: AA+aEWa4llvU8o5vyNo3uDwBZHI9omX/doy5McUtLgOooPhNg2Mbovjv
 i3aDC66y/KJ51U+jjNNFYS70wRJfq4ntYMxEPixdQThH
X-Google-Smtp-Source: AFSGD/VIFDHk7GYZbtv63RFHJ4GdnFAks3xz7yraBr7PldM6c186Bm4Jd6LL48gbwRiOszdtIONfc2xOZQlw1NTRMss=
X-Received: by 2002:a19:2395:: with SMTP id j143mr109508lfj.107.1544732753596; 
 Thu, 13 Dec 2018 12:25:53 -0800 (PST)
MIME-Version: 1.0
References: <CAFyQvY3vfV9wqVWo3hAvawRAqpWUSGGf3tebwveM60OCtFkCvA@HIDDEN>
In-Reply-To: <CAFyQvY3vfV9wqVWo3hAvawRAqpWUSGGf3tebwveM60OCtFkCvA@HIDDEN>
From: Kaushal Modi <kaushal.modi@HIDDEN>
Date: Thu, 13 Dec 2018 15:25:16 -0500
Message-ID: <CAFyQvY2ZW38s4aMpkkfXLmTw_YrYJ8PRfaRPLugQt03tw0u7Zg@HIDDEN>
Subject: Re: 27.0.50; Partial glyphs not rendered for Gujarati with Harfbuzz
 enabled (renders fine using m17n)
To: 33729 <at> debbugs.gnu.org
Content-Type: multipart/alternative; boundary="0000000000000d170f057ced1f94"
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 33729
Cc: dr.khaled.hosny@HIDDEN, behdad@HIDDEN, far.nasiri.m@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

--0000000000000d170f057ced1f94
Content-Type: text/plain; charset="UTF-8"

>
> Screenshot to explain this issue: https://i.imgtc.com/md9Yz7X.png
>

I don't know Arabic. But from that same screenshot, it's evident that the
rendering of that same text is quite different between m17n and harfbuzz.

--0000000000000d170f057ced1f94
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div class=3D"gmail_quote"><blockquote class=3D"gmail_quot=
e" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204)=
;padding-left:1ex"><div dir=3D"ltr"><div dir=3D"ltr"><div>Screenshot to exp=
lain this issue: <a href=3D"https://i.imgtc.com/md9Yz7X.png" target=3D"_bla=
nk">https://i.imgtc.com/md9Yz7X.png</a></div></div></div></blockquote><div>=
<br></div><div>I don&#39;t know Arabic. But from that same screenshot, it&#=
39;s evident that the rendering of that same text is quite different betwee=
n m17n and harfbuzz.</div><div><br></div></div></div>

--0000000000000d170f057ced1f94--




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33729; Package emacs. Full text available.

Message received at submit <at> debbugs.gnu.org:


Received: (at submit) by debbugs.gnu.org; 13 Dec 2018 20:21:20 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Dec 13 15:21:20 2018
Received: from localhost ([127.0.0.1]:46787 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gXXU0-0000Rj-0j
	for submit <at> debbugs.gnu.org; Thu, 13 Dec 2018 15:21:20 -0500
Received: from eggs.gnu.org ([208.118.235.92]:44958)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <kaushal.modi@HIDDEN>) id 1gXXTw-0000RU-Fs
 for submit <at> debbugs.gnu.org; Thu, 13 Dec 2018 15:21:17 -0500
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <kaushal.modi@HIDDEN>) id 1gXXTp-0002a2-S1
 for submit <at> debbugs.gnu.org; Thu, 13 Dec 2018 15:21:11 -0500
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50,FREEMAIL_FROM,
 HTML_MESSAGE autolearn=disabled version=3.3.2
Received: from lists.gnu.org ([2001:4830:134:3::11]:42732)
 by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32)
 (Exim 4.71) (envelope-from <kaushal.modi@HIDDEN>)
 id 1gXXTn-0002XI-4h
 for submit <at> debbugs.gnu.org; Thu, 13 Dec 2018 15:21:08 -0500
Received: from eggs.gnu.org ([2001:4830:134:3::10]:54521)
 by lists.gnu.org with esmtp (Exim 4.71)
 (envelope-from <kaushal.modi@HIDDEN>) id 1gXXTl-000535-M7
 for bug-gnu-emacs@HIDDEN; Thu, 13 Dec 2018 15:21:06 -0500
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <kaushal.modi@HIDDEN>) id 1gXXTk-0002Tu-G9
 for bug-gnu-emacs@HIDDEN; Thu, 13 Dec 2018 15:21:05 -0500
Received: from mail-lf1-x12e.google.com ([2a00:1450:4864:20::12e]:39835)
 by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16)
 (Exim 4.71) (envelope-from <kaushal.modi@HIDDEN>)
 id 1gXXTk-0002Rs-7Y
 for bug-gnu-emacs@HIDDEN; Thu, 13 Dec 2018 15:21:04 -0500
Received: by mail-lf1-x12e.google.com with SMTP id n18so2555582lfh.6
 for <bug-gnu-emacs@HIDDEN>; Thu, 13 Dec 2018 12:21:02 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=mime-version:from:date:message-id:subject:to:cc;
 bh=oDp1LZwmbyT6V1HUfXWhOarj9kkCDbckMwjzncSKgMc=;
 b=jO9iEzvoxo2blLCspWK+OEfFm2uUrEogq+07fNNvvp77xFliwGjp/NHRpgIE5iHLfR
 ZRczRwY9iWEJjdwkZoFVdjCuKwcjI+v6uSPxg3NXOX2mA+C4sRoPun3qta2f8/ABOZhR
 0jISp9DVCA7ohX0xPJ9Cq1we5dII/vPTDlcdvRcc0iVs3DMLXsrktdpzDmPSQLKPdG3m
 /lBLxPJoF5kP7ESQiMyjFQtJkSububf9M6iNFfr7sdZCjmXXgYxUC9dIVpsNM24n5WML
 jlPHyBkViYNCrtT1izkyOMDqqLTpKen3c5AZH5ozATCtNXYyp3/YKONApNG9DYSIpS2G
 7EQA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:mime-version:from:date:message-id:subject:to:cc;
 bh=oDp1LZwmbyT6V1HUfXWhOarj9kkCDbckMwjzncSKgMc=;
 b=hXuAIilt3y3bI1gbcu9pdOv9DXIp5E2sdj22dE76R9QrT7R/wJw5uDfiIHO3U2i3Gx
 z70fhuhTfbjQz+sJ7lTzsYqo+9xmrLwbDr20HEQ05TkIXE6qM5SmfUo5GvbaTR1FeTRi
 CfGyyc1r107ityYzwpMrxpRUWFU9ICNOtJQNdv++EA8qXkqBNeqAkB1wb2qMx3Z4NISK
 lbUqEW/ngvSTnNNYYweCdlynJMICruRG+JAeNCqTPl3WWOUlRr0w+0FpSpccNlUnaZXM
 9sZV7aT0xEagNL5nAUtctXq2CKwbve6pb72UA3wOgKWzbJZ5N9Hbltgji9Far/6G+oDK
 jD9A==
X-Gm-Message-State: AA+aEWblc8Y0jvrKB927W07oT4vJBrEM3w9t33SYDbmxmt10h9O4Wep3
 4rC243WGUYirUZHxvUKMp+VhhmQHrmLqR+W0rOg4VBGoXrE=
X-Google-Smtp-Source: AFSGD/X2yJE+xDKIsECUnqwj2YpO6LAqSPMSPDLk/w0GR5QSsCrfMYrs4Wo9+bLzpu10XE+DybzUWN1DgdqJhEnuxAU=
X-Received: by 2002:a19:a86:: with SMTP id 128mr97784lfk.139.1544732460792;
 Thu, 13 Dec 2018 12:21:00 -0800 (PST)
MIME-Version: 1.0
From: Kaushal Modi <kaushal.modi@HIDDEN>
Date: Thu, 13 Dec 2018 15:20:23 -0500
Message-ID: <CAFyQvY3vfV9wqVWo3hAvawRAqpWUSGGf3tebwveM60OCtFkCvA@HIDDEN>
Subject: 27.0.50; Partial glyphs not rendered for Gujarati with Harfbuzz
 enabled (renders fine using m17n)
To: bug-gnu-emacs@HIDDEN
Content-Type: multipart/alternative; boundary="000000000000994640057ced0d2e"
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
 recognized.
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x
X-Received-From: 2001:4830:134:3::11
X-Spam-Score: -4.0 (----)
X-Debbugs-Envelope-To: submit
Cc: dr.khaled.hosny@HIDDEN, behdad@HIDDEN, far.nasiri.m@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -5.0 (-----)

--000000000000994640057ced0d2e
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

 Hello,

I built emacs from harfbuzz branch with harfbuzz 1.0.3 installed (RHEL 6.8)=
.

I quickly compared Hindi and Gujarati rendering difference between emacs
built with m17n vs the new harfbuzz branch build.

With harfbuzz, it does not render the partial glyphs for Gujarati, but does
it fine for Hindi. But on the build with m17n, both Hindi and Gujarati show
that partial glyph rendered fine.

Screenshot to explain this issue: https://i.imgtc.com/md9Yz7X.png



In GNU Emacs 27.0.50 (build 2, x86_64-pc-linux-gnu, GTK+ Version 2.24.23)
 of 2018-12-13
Repository revision: 981b3d292aff49452c2b5f0217b57ec1a2829a8b
Repository branch: harfbuzz
Windowing system distributor 'The X.Org Foundation', version 11.0.60900000
System Description: Red Hat Enterprise Linux Workstation release 6.8
(Santiago)

Recent messages:
Emacs version: GNU Emacs 27.0.50 (build 2, x86_64-pc-linux-gnu, GTK+
Version 2.24.23)
 of 2018-12-13, built using commit 981b3d292aff49452c2b5f0217b57ec1a2829a8b=
.

./configure options:
  --with-modules --prefix=3D/home/kmodi/usr_local/apps/6/emacs/harfbuzz
'--program-transform-name=3Ds/^ctags$/ctags_emacs/' --with-harfbuzz
'CPPFLAGS=3D-I/home/kmodi/stowed/include -I/home/kmodi/usr_local/6/include
-I/usr/include/freetype2 -I/usr/include' 'CFLAGS=3D-O2 -march=3Dnative'
'LDFLAGS=3D-L/home/kmodi/stowed/lib -L/home/kmodi/stowed/lib64
-L/home/kmodi/usr_local/6/lib -L/home/kmodi/usr_local/6/lib64'
PKG_CONFIG_PATH=3D/home/kmodi/usr_local/6/lib/pkgconfig:/home/kmodi/usr_loc=
al/6/lib64/pkgconfig:/cad/adi/apps/gnu/linux/x86_64/6/lib/pkgconfig:/cad/ad=
i/apps/gnu/linux/x86_64/6/lib64/pkgconfig:/home/kmodi/stowed/lib/pkgconfig:=
/usr/lib/pkgconfig:/usr/lib64/pkgconfig:/usr/share/pkgconfig:/lib/pkgconfig=
:/lib64/pkgconfig

Features:
  XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS GSETTINGS GLIB
NOTIFY INOTIFY ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE HARFBUZZ M17N_FLT
LIBOTF XFT ZLIB TOOLKIT_SCROLL_BARS GTK2 X11 XDBE XIM MODULES THREADS GMP

Important settings:
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=3Dnone
  locale-coding-system: utf-8-unix


--
Kaushal Modi

--000000000000994640057ced0d2e
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div dir=3D"ltr">
<div>Hello,</div><div><br></div><div>I built emacs from harfbuzz branch wit=
h harfbuzz 1.0.3 installed (RHEL 6.8).</div><div><br></div><div>I quickly c=
ompared Hindi and Gujarati rendering difference between emacs built with m1=
7n vs the new harfbuzz branch build.</div><div><br></div><div>With
 harfbuzz, it does not render the partial glyphs for Gujarati, but does=20
it fine for Hindi. But on the build with m17n, both Hindi and Gujarati=20
show that partial glyph rendered fine.</div><div><br></div><div>Screenshot =
to explain this issue: <a href=3D"https://i.imgtc.com/md9Yz7X.png">https://=
i.imgtc.com/md9Yz7X.png</a></div><div></div>

<br><br><br>In GNU Emacs 27.0.50 (build 2, x86_64-pc-linux-gnu, GTK+ Versio=
n 2.24.23)<br>=C2=A0of 2018-12-13<br>Repository revision: 981b3d292aff49452=
c2b5f0217b57ec1a2829a8b<br>Repository branch: harfbuzz<br>Windowing system =
distributor &#39;The X.Org Foundation&#39;, version 11.0.60900000<br>System=
 Description: Red Hat Enterprise Linux Workstation release 6.8 (Santiago)<b=
r><br>Recent messages:<br>Emacs version: GNU Emacs 27.0.50 (build 2, x86_64=
-pc-linux-gnu, GTK+ Version 2.24.23)<br>=C2=A0of 2018-12-13, built using co=
mmit 981b3d292aff49452c2b5f0217b57ec1a2829a8b.<br><br>./configure options:<=
br>=C2=A0 --with-modules --prefix=3D/home/kmodi/usr_local/apps/6/emacs/harf=
buzz &#39;--program-transform-name=3Ds/^ctags$/ctags_emacs/&#39; --with-har=
fbuzz &#39;CPPFLAGS=3D-I/home/kmodi/stowed/include -I/home/kmodi/usr_local/=
6/include -I/usr/include/freetype2 -I/usr/include&#39; &#39;CFLAGS=3D-O2 -m=
arch=3Dnative&#39; &#39;LDFLAGS=3D-L/home/kmodi/stowed/lib -L/home/kmodi/st=
owed/lib64 -L/home/kmodi/usr_local/6/lib -L/home/kmodi/usr_local/6/lib64&#3=
9; PKG_CONFIG_PATH=3D/home/kmodi/usr_local/6/lib/pkgconfig:/home/kmodi/usr_=
local/6/lib64/pkgconfig:/cad/adi/apps/gnu/linux/x86_64/6/lib/pkgconfig:/cad=
/adi/apps/gnu/linux/x86_64/6/lib64/pkgconfig:/home/kmodi/stowed/lib/pkgconf=
ig:/usr/lib/pkgconfig:/usr/lib64/pkgconfig:/usr/share/pkgconfig:/lib/pkgcon=
fig:/lib64/pkgconfig<br><br>Features:<br>=C2=A0 XPM JPEG TIFF GIF PNG RSVG =
IMAGEMAGICK SOUND GPM DBUS GSETTINGS GLIB NOTIFY INOTIFY ACL LIBSELINUX GNU=
TLS LIBXML2 FREETYPE HARFBUZZ M17N_FLT LIBOTF XFT ZLIB TOOLKIT_SCROLL_BARS =
GTK2 X11 XDBE XIM MODULES THREADS GMP<br><br>Important settings:<br>=C2=A0 =
value of $LANG: en_US.UTF-8<br>=C2=A0 value of $XMODIFIERS: @im=3Dnone<br>=
=C2=A0 locale-coding-system: utf-8-unix<br><br clear=3D"all"><div><div dir=
=3D"ltr" class=3D"gmail_signature"><div dir=3D"ltr"><br>--<br>Kaushal Modi<=
/div></div></div></div></div>

--000000000000994640057ced0d2e--




Acknowledgement sent to Kaushal Modi <kaushal.modi@HIDDEN>:
New bug report received and forwarded. Copy sent to bug-gnu-emacs@HIDDEN. Full text available.
Report forwarded to bug-gnu-emacs@HIDDEN:
bug#33729; Package emacs. Full text available.
Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.
Last modified: Tue, 29 Jan 2019 22:45:01 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.