Bug 1513343 - policytool text rendering problem in Simplified Chinese locale
policytool text rendering problem in Simplified Chinese locale
Status: NEW
Product: Fedora
Classification: Fedora
Component: java-1.8.0-openjdk (Show other bugs)
27
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: jiri vanek
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-11-15 04:18 EST by Peng Wu
Modified: 2017-12-11 05:59 EST (History)
9 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Screenshot of policytool (21.21 KB, image/png)
2017-11-15 04:21 EST, Peng Wu
no flags Details
Simplified Chinese characters (30.42 KB, image/png)
2017-11-15 04:25 EST, Peng Wu
no flags Details
AWT example (386 bytes, text/plain)
2017-11-20 23:00 EST, Peng Wu
no flags Details
Swing example (413 bytes, text/plain)
2017-11-20 23:02 EST, Peng Wu
no flags Details
fc-match output (239.50 KB, text/plain)
2017-11-23 23:26 EST, Peng Wu
no flags Details
JButtonExample output (1.07 MB, application/octet-stream)
2017-11-23 23:30 EST, Peng Wu
no flags Details
ButtonExample output (1.07 MB, application/octet-stream)
2017-11-23 23:34 EST, Peng Wu
no flags Details
JButtonExample output (1.16 MB, application/octet-stream)
2017-11-24 04:50 EST, Peng Wu
no flags Details
ButtonExample output (1.16 MB, application/octet-stream)
2017-11-24 04:54 EST, Peng Wu
no flags Details
Screenshot of policytool with new font cache (25.18 KB, image/png)
2017-11-24 07:09 EST, Peng Wu
no flags Details

  None (edit)
Description Peng Wu 2017-11-15 04:18:25 EST
Description of problem:

On Fedora 25 with latest openjdk installed,
when launching policytool in Simplified Chinese locale.

Using the following command:
$ export LANG=zh_CN.UTF-8
$ export LC_ALL=zh_CN.UTF-8
$ policytool

Some text can't be rendered in the screenshot.

And the "添" character is rendered using Traditional Chinese character.
In the Chinese.png, it shows the Simplified Chinese character "添" .


Version-Release number of selected component (if applicable):
java-1.8.0-openjdk-1.8.0.151-1.b12.fc25.x86_64

How reproducible:
Launch policytool in Simplified Chinese locale.

Steps to Reproduce:
1. export LANG=zh_CN.UTF-8
2. export LC_ALL=zh_CN.UTF-8
3. policytool


Actual results:
Some text can't be rendered or rendered in Traditional Chinese characters.

Expected results:
The text are rendered in Simplified Chinese characters correctly.

Additional info:
Comment 1 Peng Wu 2017-11-15 04:21 EST
Created attachment 1352448 [details]
Screenshot of policytool
Comment 2 Peng Wu 2017-11-15 04:25 EST
Created attachment 1352449 [details]
Simplified Chinese characters
Comment 3 jiri vanek 2017-11-15 09:57:08 EST
Is this regression? How it looked in older versions? was it correct? Both cases? Both wrongly rendered character and the not rendered characters? Can you write simple java pogram printing those three characters? The correct one, the incorrect one, and the not rendered one?
When I copy pasted "添" to source code or properties file, it  seemed correct to me...

There was one patch to fonts recently: https://bugzilla.redhat.com/show_bug.cgi?id=1484079  But I doubt this was an consequentness. 

Openjdk is using system fonts - so maybe there was an change? What about f26 or openjdk9? 

Sorry for asking many information, but debugging Chinese fonts isnot easy task fot Latin alphabets users. TY!!
Comment 4 Peng Wu 2017-11-15 20:26:06 EST
Okay, I will try to write some simple java program.

Actually I think this problem is caused by Chinese fonts changes in Fedora.

We changed the default Chinese fonts from TrueType to OpenType/CFF.

In the past, we use UMing and WQY TrueType fonts.
Now we use Adobe Source Han fonts, which is OpenType/CFF fonts.

Could you tell me where I can find the properties file for translation in policytool?
Maybe I can check if it is related in translation.

Thanks!
Comment 5 Severin Gehwolf 2017-11-16 07:39:05 EST
(In reply to Peng Wu from comment #4)
> Could you tell me where I can find the properties file for translation in
> policytool?
> Maybe I can check if it is related in translation.

jdk/src/share/classes/sun/security/tools/policytool/Resources_zh_CN.java should be it:
http://hg.openjdk.java.net/jdk8u/jdk8u/jdk/file/07011844584f/src/share/classes/sun/security/tools/policytool/Resources_zh_CN.java
Comment 6 Fedora End Of Life 2017-11-16 14:01:39 EST
This message is a reminder that Fedora 25 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 25. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '25'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not
able to fix it before Fedora 25 is end of life. If you would still like
to see this bug fixed and are able to reproduce it against a later version
of Fedora, you are encouraged  change the 'version' to a later Fedora
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.
Comment 7 Severin Gehwolf 2017-11-16 14:08:02 EST
Re-targeting to F27. The JDK is the same there.
Comment 8 Peng Wu 2017-11-20 23:00 EST
Created attachment 1356356 [details]
AWT example
Comment 9 Peng Wu 2017-11-20 23:02 EST
Created attachment 1356357 [details]
Swing example
Comment 10 Peng Wu 2017-11-20 23:09:33 EST
I copied the string from translation and uploaded simple java program for both AWT and Swing.

Please notice the top stroke of the "添" character.

In Simplified Chinese, the top stroke is straight.

But in the java program, the top stroke is a curve.
It seems from Traditional Chinese characters.

Adobe Source Han fonts contains both Simplified Chinese
and Traditional Chinese characters.
Comment 11 Mario Torre 2017-11-22 05:57:33 EST
(In reply to Peng Wu from comment #4)
> Okay, I will try to write some simple java program.
> 
> Actually I think this problem is caused by Chinese fonts changes in Fedora.
> 
> We changed the default Chinese fonts from TrueType to OpenType/CFF.

When this change happened in Fedora? Before the latest JDK update OpenJDK was not picking up CFF fonts at all, only True Type and Type 1. If there was a change in the font file in Fedora it means that before the update OpenJDK would only pick a True Type version of the Chinese font, but after the CFF may be the one selected, depending on what fc-match feels like returning.

Unfortunately the result depends only of the font you have installed, but it may narrow down to a broken/incorrect font, we should be checking this first (identify what font is actually used and then debug the font to see if it's correct).

> In the past, we use UMing and WQY TrueType fonts.
> Now we use Adobe Source Han fonts, which is OpenType/CFF fonts.

Can you please link the output of this:

fc-match -f "sans:regular:roman" -sv

Also, can you please run the test application with this flag:

-Dsun.java2d.debugfonts=true

And attach the whole output too?
 
> Could you tell me where I can find the properties file for translation in
> policytool?
> Maybe I can check if it is related in translation.

This seems unlikely if it was working before, but if you can check this out it would helps us a lot.

Cheers,
Mario
Comment 12 Peng Wu 2017-11-23 23:26 EST
Created attachment 1358483 [details]
fc-match output
Comment 13 Peng Wu 2017-11-23 23:30 EST
Created attachment 1358484 [details]
JButtonExample output
Comment 14 Peng Wu 2017-11-23 23:34 EST
Created attachment 1358485 [details]
ButtonExample output
Comment 15 Peng Wu 2017-11-23 23:37:03 EST
Thanks, I uploaded the output files.

The translation is correct I think.
Comment 16 Mario Torre 2017-11-24 04:22:54 EST
My bad, I forgot to add, can you please run the same command with -Dsun.java2d.debugfonts=true but before delete your $HOME/.java/fonts directory each time?

The JDK caches the fonts to use, so I want to be sure the JDK is using a fully new font set at each run.

Thanks!

Cheers,
Mario
Comment 17 Peng Wu 2017-11-24 04:50 EST
Created attachment 1358577 [details]
JButtonExample output
Comment 18 Peng Wu 2017-11-24 04:54 EST
Created attachment 1358580 [details]
ButtonExample output
Comment 19 Peng Wu 2017-11-24 04:57:17 EST
No problem, I uploaded new output.
Comment 20 Peng Wu 2017-11-24 07:09 EST
Created attachment 1358647 [details]
Screenshot of policytool with new font cache

After remove $HOME/.java/fonts directory,
policytool can render all characters now.

But it still renders some wrong glyphs like "添".
Comment 21 Mario Torre 2017-11-28 10:57:45 EST
I did some more debugging, so far I think the issue is that it's not using Source Han Sans, but instead either NanumGothic or VL Gothic.

If you force your sample application to use Source Han, it displays the correct font:

Font font = Font.createFont(Font.TRUETYPE_FONT, new File("/usr/share/fonts/adobe-source-han-sans-cn/SourceHanSansCN-Normal.otf"));
b.setFont(font.deriveFont(40.0f));

I need to debug a bit more to understand why the font is not loaded by default, that maybe fontconfig returning a different match, or something else entirely.
Comment 22 Mario Torre 2017-12-08 08:29:45 EST
I think I got it, debugging was quite interesting since the toString used by the debugger to see the actual font had a side effect thus hiding the point where the font gets somehow discarded and replaced.

Anyway, the font contains a list of family names and full names:

family:
"Source Han Sans CN"(s)
"思源黑体 CN"(s)
"Source Han Sans CN Regular"(s)
"思源黑体 CN Regular"(s)

fullname:
"Source Han Sans CN Regular"(s)
"思源黑体 CN Regular"(s)

But the JDK doesn't see those, it doesn't seem like FontConfig return all the information, it only returns:

family: Source Han Sans CN
fullname: Source Han Sans CN Regular

There's some code that checks that those matches, but they don't of course (Source Han Sans CN != Source Han Sans CN Regular). It looks to me like a bit bogus check to be honest and I'm not entirely sure what is the proper intention (check the validity of the font, yes, but why comparing those two fields? Doesn't seem correct to me). This font (as well as the TW counterpart) seems the only one that has a multitude of names of all the ones I tested, so they are the only ones to fail the validation. I'll need to check out with upstream why this check is there or if it's correct at all, but the code is quite delicate so even if this is fixed I'm not sure we will be able to back port the fix in the existing releases.

An option could be to scan the file if the match fails to double check if there is another name that identifies the font.

BTW, There's a discrepancy between what FcPatternPrint(fontPattern) does and what querying the pattern directly returns, i.e.:

FcPatternGetString(fontPattern, FC_FILE, 0, &file[i]);
FcPatternGetString(fontPattern, FC_FAMILY, 0, &family[i]);

I wonder if I can make FontConfig return all the names in the first place, then I can match them during initialisation, which is the fastest and safest, but I hadn't find a way yet.

Cheers,
Mario
Comment 23 Akira TAGOH 2017-12-11 00:54:03 EST
You can get all of values in certain property by:
for (i = 0; ; i++) {
  if (FcPatternGetString(fontPattern, FC_FAMILY, i, &file) != FcResultMatch)
    break;
  ...
}
Comment 24 Peng Wu 2017-12-11 01:31:56 EST
I think to get the translated strings.

Maybe need to change FcObjectSetBuild function call to include FC_FAMILYLANG.

Like: https://git.gnome.org/browse/gnome-font-viewer/commit/?id=e512b0bd53636ebad6ce2428c44bfb02816ed7f2
Comment 25 Peng Wu 2017-12-11 01:35:07 EST
Sorry, maybe also need to include FC_FULLNAMELANG.
Comment 26 Mario Torre 2017-12-11 05:59:59 EST
(In reply to Akira TAGOH from comment #23)
> You can get all of values in certain property by:
> for (i = 0; ; i++) {
>   if (FcPatternGetString(fontPattern, FC_FAMILY, i, &file) != FcResultMatch)
>     break;
>   ...
> }

Thanks, this is very useful!

Cheers,
Mario

Note You need to log in before you can comment on or make changes to this bug.