Bug 1409011

Summary: wrong character displayed for zhi2; correct character not available in ibus (harfbuzz decompose of 0x2F940 to 0x76F4)
Product: [Fedora] Fedora Reporter: Bill <mattison.computer>
Component: harfbuzzAssignee: Parag Nemade <pnemade>
Status: CLOSED NOTABUG QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 24CC: caolanm, dtardon, erack, i18n-bugs, mattison.computer, mstahl, pnemade, psatpute, sbergman
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-01-11 13:37:26 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
this shows the desired simplified Chinese character for pinyin "zhi2". none

Description Bill 2016-12-28 22:38:16 UTC 
Created attachment 1235746 [details]
this shows the desired simplified Chinese character for pinyin "zhi2".

Description of problem: When entering simplified Chinese text into Libre Office or vi, the desired character (see attachment) for the pinyin "zhi" (second tone)is not available.  When I display a web page in Firefox that should be displaying the correct character, a different character is displayed.


Version-Release number of selected component (if applicable):


How reproducible:
Using ibus, try entering the character "zhi" (second tone) into a Libre Office document.
In Firefox, display a web page that should have the "zhi" character in it.
In Firefox, use a Chinese-English web page that displays characters as GIFs to search for "zhi2".

Steps to Reproduce:
1. In Libre Office, using ibus, enter "zhi".  Go through all the choices of characters and try to find one matching the character in this Bugzilla's attachment.  There is no match.

2. In Firefox, load "http://www.mandarintools.com/worddict.html".  Enter "zhi2" in the "Search" field, set the "as" field to "Pinyin", set the "Match" to "as whole dictionary field", make sure the "Display Chinese using GIFs" is ***not*** checked, and click the "Look It Up!" button.  The correct character will not be shown in the results.  It should be the character meaning "straight; vertical; frank; directly; straightly; upright".

3. In Firefox, load "http://www.mandarintools.com/worddict.html".  Enter "zhi2" in the "Search" field, set the "as" field to "Pinyin", set the "Match" to "as whole dictionary field", make sure the "Display Chinese using GIFs" ***is *** checked, and click the "Look It Up!" button.  The correct character will be shown in the results.  It is the character meaning "straight; vertical; frank; directly; straightly; upright".

Actual results: (see above)


Expected results: The character that shows in test case 3 above should also show in test case 2, and should be available in ibus in test case 1


Additional info:
The problem shows in Libre Office, Firefox, and vi(m).  It might also show in other applications; I didn't try.  I don't think this is an ibus issue.  Might this be a font issue?
Comment 1 Caolan McNamara 2017-01-11 11:50:36 UTC 
This character is 0x2F940 CJK COMPATIBILITY IDEOGRAPH-2F940
Comment 2 Caolan McNamara 2017-01-11 13:34:56 UTC 
If I paste the text from firefox into LibreOffice then if I change the font to fonts like "WenQuanYi Zen Hei", or "Source Han Sans CN" I get your expected glyph

http://www.fileformat.info/info/unicode/char/2f940/index.htm states that 0x2F940 can be decomposed to "straight, erect, vertical (U+76F4)" http://www.fileformat.info/info/unicode/char/76f4/index.htm

If I change the font to e.g. VL Gothic then I get a glyph that looks like CJK UNIFIED IDEOGRAPH-76F4

So if the font doesn't have 0x2F940 but it does have 0x76F4 then it will show 0x76F4. For LibreOffice at least, and presumably everything else, its harfbuzz that does this substitution. (hb_unicode_funcs_set_decompose_func is relevent here).

If the font doesn't have either 0x2F940 or 0x76F4 then fontconfig would be asked for a replacement, in which case if anything has 0x2F940 then that will be used.

So, everthing is apparently working by design on that front.
Comment 3 Caolan McNamara 2017-01-11 13:37:26 UTC 
its possible on a single harfbuzz using app level to disable this with a custom hb_unicode_funcs_set_decompose_func handler that returns false, but we want everything to work the same, so presumably this decomposition is "a good thing" in general and we won't override the upstream harfbuzz defaults
Comment 4 Bill 2017-01-11 22:42:18 UTC 
If I understand the comments correctly, the problem is that the Kai font(s) are missing the desired character.

I see the same problem in ibus's libpinyin.  Based on my experience with ibus libpinyin in the past couple weeks, there are other "gaps" in the Kai font(s).  As an example. when I enter the pinyin "kuai" into ibus, most of the characters matching that are displayed using the desired Kai font.  But some are displayed using some other simplified Chinese ming, wen quanyi, song, or sans-serif font.  This should not be.

So how do I get the gaps in the Kai font(s) properly filled?  What should the bugzilla say, and against what component should the bugzilla be submitted?
Comment 5 Caolan McNamara 2017-01-12 10:20:18 UTC 
The app where I see the problem is firefox, and there with the character selected in the webpage, ctrl+shift+c to launch the introspector then clicking the "fonts" tab I see WenQuanYi Zen Hei and VL Gothic listed. Testing those fonts in LibreOffice I see VL Gothic has 0x76F4 and not 0x2F940, so 0x76F4 is used in its place, so vlgothic is the package in my case.