Bug 1513343 - policytool text rendering problem in Simplified Chinese locale
Summary: policytool text rendering problem in Simplified Chinese locale
Keywords:
Status: NEW
Alias: None
Product: Fedora
Classification: Fedora
Component: java-1.8.0-openjdk
Version: 29
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: jiri vanek
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-11-15 09:18 UTC by Peng Wu
Modified: 2019-05-10 12:38 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:


Attachments (Terms of Use)
Screenshot of policytool (21.21 KB, image/png)
2017-11-15 09:21 UTC, Peng Wu
no flags Details
Simplified Chinese characters (30.42 KB, image/png)
2017-11-15 09:25 UTC, Peng Wu
no flags Details
AWT example (386 bytes, text/plain)
2017-11-21 04:00 UTC, Peng Wu
no flags Details
Swing example (413 bytes, text/plain)
2017-11-21 04:02 UTC, Peng Wu
no flags Details
fc-match output (239.50 KB, text/plain)
2017-11-24 04:26 UTC, Peng Wu
no flags Details
JButtonExample output (1.07 MB, application/octet-stream)
2017-11-24 04:30 UTC, Peng Wu
no flags Details
ButtonExample output (1.07 MB, application/octet-stream)
2017-11-24 04:34 UTC, Peng Wu
no flags Details
JButtonExample output (1.16 MB, application/octet-stream)
2017-11-24 09:50 UTC, Peng Wu
no flags Details
ButtonExample output (1.16 MB, application/octet-stream)
2017-11-24 09:54 UTC, Peng Wu
no flags Details
Screenshot of policytool with new font cache (25.18 KB, image/png)
2017-11-24 12:09 UTC, Peng Wu
no flags Details

Description Peng Wu 2017-11-15 09:18:25 UTC
Description of problem:

On Fedora 25 with latest openjdk installed,
when launching policytool in Simplified Chinese locale.

Using the following command:
$ export LANG=zh_CN.UTF-8
$ export LC_ALL=zh_CN.UTF-8
$ policytool

Some text can't be rendered in the screenshot.

And the "添" character is rendered using Traditional Chinese character.
In the Chinese.png, it shows the Simplified Chinese character "添" .


Version-Release number of selected component (if applicable):
java-1.8.0-openjdk-1.8.0.151-1.b12.fc25.x86_64

How reproducible:
Launch policytool in Simplified Chinese locale.

Steps to Reproduce:
1. export LANG=zh_CN.UTF-8
2. export LC_ALL=zh_CN.UTF-8
3. policytool


Actual results:
Some text can't be rendered or rendered in Traditional Chinese characters.

Expected results:
The text are rendered in Simplified Chinese characters correctly.

Additional info:

Comment 1 Peng Wu 2017-11-15 09:21:46 UTC
Created attachment 1352448 [details]
Screenshot of policytool

Comment 2 Peng Wu 2017-11-15 09:25:24 UTC
Created attachment 1352449 [details]
Simplified Chinese characters

Comment 3 jiri vanek 2017-11-15 14:57:08 UTC
Is this regression? How it looked in older versions? was it correct? Both cases? Both wrongly rendered character and the not rendered characters? Can you write simple java pogram printing those three characters? The correct one, the incorrect one, and the not rendered one?
When I copy pasted "添" to source code or properties file, it  seemed correct to me...

There was one patch to fonts recently: https://bugzilla.redhat.com/show_bug.cgi?id=1484079  But I doubt this was an consequentness. 

Openjdk is using system fonts - so maybe there was an change? What about f26 or openjdk9? 

Sorry for asking many information, but debugging Chinese fonts isnot easy task fot Latin alphabets users. TY!!

Comment 4 Peng Wu 2017-11-16 01:26:06 UTC
Okay, I will try to write some simple java program.

Actually I think this problem is caused by Chinese fonts changes in Fedora.

We changed the default Chinese fonts from TrueType to OpenType/CFF.

In the past, we use UMing and WQY TrueType fonts.
Now we use Adobe Source Han fonts, which is OpenType/CFF fonts.

Could you tell me where I can find the properties file for translation in policytool?
Maybe I can check if it is related in translation.

Thanks!

Comment 5 Severin Gehwolf 2017-11-16 12:39:05 UTC
(In reply to Peng Wu from comment #4)
> Could you tell me where I can find the properties file for translation in
> policytool?
> Maybe I can check if it is related in translation.

jdk/src/share/classes/sun/security/tools/policytool/Resources_zh_CN.java should be it:
http://hg.openjdk.java.net/jdk8u/jdk8u/jdk/file/07011844584f/src/share/classes/sun/security/tools/policytool/Resources_zh_CN.java

Comment 6 Fedora End Of Life 2017-11-16 19:01:39 UTC
This message is a reminder that Fedora 25 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 25. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '25'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not
able to fix it before Fedora 25 is end of life. If you would still like
to see this bug fixed and are able to reproduce it against a later version
of Fedora, you are encouraged  change the 'version' to a later Fedora
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

Comment 7 Severin Gehwolf 2017-11-16 19:08:02 UTC
Re-targeting to F27. The JDK is the same there.

Comment 8 Peng Wu 2017-11-21 04:00:35 UTC
Created attachment 1356356 [details]
AWT example

Comment 9 Peng Wu 2017-11-21 04:02:39 UTC
Created attachment 1356357 [details]
Swing example

Comment 10 Peng Wu 2017-11-21 04:09:33 UTC
I copied the string from translation and uploaded simple java program for both AWT and Swing.

Please notice the top stroke of the "添" character.

In Simplified Chinese, the top stroke is straight.

But in the java program, the top stroke is a curve.
It seems from Traditional Chinese characters.

Adobe Source Han fonts contains both Simplified Chinese
and Traditional Chinese characters.

Comment 11 Mario Torre 2017-11-22 10:57:33 UTC
(In reply to Peng Wu from comment #4)
> Okay, I will try to write some simple java program.
> 
> Actually I think this problem is caused by Chinese fonts changes in Fedora.
> 
> We changed the default Chinese fonts from TrueType to OpenType/CFF.

When this change happened in Fedora? Before the latest JDK update OpenJDK was not picking up CFF fonts at all, only True Type and Type 1. If there was a change in the font file in Fedora it means that before the update OpenJDK would only pick a True Type version of the Chinese font, but after the CFF may be the one selected, depending on what fc-match feels like returning.

Unfortunately the result depends only of the font you have installed, but it may narrow down to a broken/incorrect font, we should be checking this first (identify what font is actually used and then debug the font to see if it's correct).

> In the past, we use UMing and WQY TrueType fonts.
> Now we use Adobe Source Han fonts, which is OpenType/CFF fonts.

Can you please link the output of this:

fc-match -f "sans:regular:roman" -sv

Also, can you please run the test application with this flag:

-Dsun.java2d.debugfonts=true

And attach the whole output too?
 
> Could you tell me where I can find the properties file for translation in
> policytool?
> Maybe I can check if it is related in translation.

This seems unlikely if it was working before, but if you can check this out it would helps us a lot.

Cheers,
Mario

Comment 12 Peng Wu 2017-11-24 04:26:53 UTC
Created attachment 1358483 [details]
fc-match output

Comment 13 Peng Wu 2017-11-24 04:30:39 UTC
Created attachment 1358484 [details]
JButtonExample output

Comment 14 Peng Wu 2017-11-24 04:34:23 UTC
Created attachment 1358485 [details]
ButtonExample output

Comment 15 Peng Wu 2017-11-24 04:37:03 UTC
Thanks, I uploaded the output files.

The translation is correct I think.

Comment 16 Mario Torre 2017-11-24 09:22:54 UTC
My bad, I forgot to add, can you please run the same command with -Dsun.java2d.debugfonts=true but before delete your $HOME/.java/fonts directory each time?

The JDK caches the fonts to use, so I want to be sure the JDK is using a fully new font set at each run.

Thanks!

Cheers,
Mario

Comment 17 Peng Wu 2017-11-24 09:50:52 UTC
Created attachment 1358577 [details]
JButtonExample output

Comment 18 Peng Wu 2017-11-24 09:54:24 UTC
Created attachment 1358580 [details]
ButtonExample output

Comment 19 Peng Wu 2017-11-24 09:57:17 UTC
No problem, I uploaded new output.

Comment 20 Peng Wu 2017-11-24 12:09:36 UTC
Created attachment 1358647 [details]
Screenshot of policytool with new font cache

After remove $HOME/.java/fonts directory,
policytool can render all characters now.

But it still renders some wrong glyphs like "添".

Comment 21 Mario Torre 2017-11-28 15:57:45 UTC
I did some more debugging, so far I think the issue is that it's not using Source Han Sans, but instead either NanumGothic or VL Gothic.

If you force your sample application to use Source Han, it displays the correct font:

Font font = Font.createFont(Font.TRUETYPE_FONT, new File("/usr/share/fonts/adobe-source-han-sans-cn/SourceHanSansCN-Normal.otf"));
b.setFont(font.deriveFont(40.0f));

I need to debug a bit more to understand why the font is not loaded by default, that maybe fontconfig returning a different match, or something else entirely.

Comment 22 Mario Torre 2017-12-08 13:29:45 UTC
I think I got it, debugging was quite interesting since the toString used by the debugger to see the actual font had a side effect thus hiding the point where the font gets somehow discarded and replaced.

Anyway, the font contains a list of family names and full names:

family:
"Source Han Sans CN"(s)
"思源黑体 CN"(s)
"Source Han Sans CN Regular"(s)
"思源黑体 CN Regular"(s)

fullname:
"Source Han Sans CN Regular"(s)
"思源黑体 CN Regular"(s)

But the JDK doesn't see those, it doesn't seem like FontConfig return all the information, it only returns:

family: Source Han Sans CN
fullname: Source Han Sans CN Regular

There's some code that checks that those matches, but they don't of course (Source Han Sans CN != Source Han Sans CN Regular). It looks to me like a bit bogus check to be honest and I'm not entirely sure what is the proper intention (check the validity of the font, yes, but why comparing those two fields? Doesn't seem correct to me). This font (as well as the TW counterpart) seems the only one that has a multitude of names of all the ones I tested, so they are the only ones to fail the validation. I'll need to check out with upstream why this check is there or if it's correct at all, but the code is quite delicate so even if this is fixed I'm not sure we will be able to back port the fix in the existing releases.

An option could be to scan the file if the match fails to double check if there is another name that identifies the font.

BTW, There's a discrepancy between what FcPatternPrint(fontPattern) does and what querying the pattern directly returns, i.e.:

FcPatternGetString(fontPattern, FC_FILE, 0, &file[i]);
FcPatternGetString(fontPattern, FC_FAMILY, 0, &family[i]);

I wonder if I can make FontConfig return all the names in the first place, then I can match them during initialisation, which is the fastest and safest, but I hadn't find a way yet.

Cheers,
Mario

Comment 23 Akira TAGOH 2017-12-11 05:54:03 UTC
You can get all of values in certain property by:
for (i = 0; ; i++) {
  if (FcPatternGetString(fontPattern, FC_FAMILY, i, &file) != FcResultMatch)
    break;
  ...
}

Comment 24 Peng Wu 2017-12-11 06:31:56 UTC
I think to get the translated strings.

Maybe need to change FcObjectSetBuild function call to include FC_FAMILYLANG.

Like: https://git.gnome.org/browse/gnome-font-viewer/commit/?id=e512b0bd53636ebad6ce2428c44bfb02816ed7f2

Comment 25 Peng Wu 2017-12-11 06:35:07 UTC
Sorry, maybe also need to include FC_FULLNAMELANG.

Comment 26 Mario Torre 2017-12-11 10:59:59 UTC
(In reply to Akira TAGOH from comment #23)
> You can get all of values in certain property by:
> for (i = 0; ; i++) {
>   if (FcPatternGetString(fontPattern, FC_FAMILY, i, &file) != FcResultMatch)
>     break;
>   ...
> }

Thanks, this is very useful!

Cheers,
Mario

Comment 27 Ben Cotton 2018-11-27 17:52:40 UTC
This message is a reminder that Fedora 27 is nearing its end of life.
On 2018-Nov-30  Fedora will stop maintaining and issuing updates for
Fedora 27. It is Fedora's policy to close all bug reports from releases
that are no longer maintained. At that time this bug will be closed as
EOL if it remains open with a Fedora  'version' of '27'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 27 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 28 Peng Wu 2018-11-28 06:47:15 UTC
This bug still exists in Fedora 29.

Changed version to Fedora 29.

Comment 30 Akira TAGOH 2018-12-13 07:53:14 UTC
There should be some way. but depends how they use fontconfig API in the code and that should be fixed there. so I don't recommend and no guarantees because that may breaks a lot but it still can be done in the config like this. just for deminstration purpose:

<fontconfig>
  <match target="font">
    <test name="prgname" compare="eq">
      <string>java</string>
    </test>
    <edit name="family" mode="assign">
      <name target="font">fullname</name>
    </edit>
  </match>
</fontconfig>


Note You need to log in before you can comment on or make changes to this bug.