Bug 475743

Summary: Japanese desktop defaulting to Chinese fonts
Product: [Fedora] Fedora Reporter: Mamoru TASAKA <mtasaka>
Component: fontconfigAssignee: Behdad Esfahbod <behdad>
Status: CLOSED RAWHIDE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: medium    
Version: rawhideCC: behdad, dcantrell, fonts-bugs, i18n-bugs, me, petersen, ryo-dairiki, tagoh
Target Milestone: ---Keywords: i18n, Regression, Reopened
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 485562 (view as bug list) Environment:
Last Closed: 2009-03-16 22:09:20 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 446452, 476774    
Attachments:
Description Flags
screenshot with VLGothic 20081203
none
screenshot with VLGothic 20081029
none
glyphs of arabic numerals
none
screenshot again
none
another screenshot
none
screenshot with old fontconfig
none
screenshot with new fontconfig none

Description Mamoru TASAKA 2008-12-10 08:34:25 UTC
Description of problem:
Since VLGothic-fonts is upgraded to 20081203, I see many 
chinese glyphs on Japanese environment.
Downloading to 20081029 works good.

Version-Release number of selected component (if applicable):
VLGothic-fonts-20081203-2.fc11

(perhaps?)
cjkunifonts-ukai-0.2.20080216.1-10.fc11.noarch.rpm
cjkunifonts-uming-0.2.20080216.1-10.fc11.noarch.rpm


How reproducible:
100%

Steps to Reproduce:
1. For example, the following URL:
   http://cvs.fedoraproject.org/viewvc/policycoreutils/po/ja.po?root=elvis&r1=1.24&r2=1.25
 
Actual results:
See attached

Comment 1 Mamoru TASAKA 2008-12-10 08:36:20 UTC
Created attachment 326463 [details]
screenshot with VLGothic 20081203

Comment 2 Mamoru TASAKA 2008-12-10 08:37:06 UTC
Created attachment 326465 [details]
screenshot with VLGothic 20081029

Comment 3 Mamoru TASAKA 2008-12-10 08:39:27 UTC
Also with 20081203 the glyphs of arabic numerals frequently changes.

Comment 4 Akira TAGOH 2008-12-10 10:59:25 UTC
my rawhide box is broken now so I can't do any testing... so just guessing, is this issue gone if you do remove /etc/fonts/conf.d/64-ttf-arphic-uming.conf or add testing for Chinese like:

<match>
  <test name="lang" compare="contains">
    <string>zh</string>
  </test>
  <test name="family">
    <string>sans-serif</string>
  </test>
  <edit name="family" mode="prepend" binding="same">
    <string>AR PL UMing HK</string>
    <string>AR PL UMing CN</string>
  </edit>
</match>

instead of preference alias?

Comment 5 Mamoru TASAKA 2008-12-10 11:37:34 UTC
(In reply to comment #4)
> is
> this issue gone if you do remove /etc/fonts/conf.d/64-ttf-arphic-uming.conf 

It seems this works

> or
> add testing for Chinese like:
> <snip>
> 
> instead of preference alias?

Would you tell me to what file?

Comment 6 Nicolas Mailhot 2008-12-10 20:00:50 UTC
Behdad recently wanted to experiment with new syntax to deal with fonts that need locale-specific ordering. You should try to ping him to check if he hasn't a better fontconfig recipe.

Comment 7 Akira TAGOH 2008-12-11 01:13:50 UTC
(In reply to comment #5)
> Would you tell me to what file?

Try to replace:

        <alias>
                <family>sans-serif</family>
                <prefer>
                        <family>AR PL UMing HK</family>
                        <family>AR PL UMing CN</family>
                </prefer>
        </alias>

in 64-ttf-arphic-uming.conf with the above for sans-serif.

Comment 8 Akira TAGOH 2008-12-11 01:19:55 UTC
(In reply to comment #6)
> Behdad recently wanted to experiment with new syntax to deal with fonts that
> need locale-specific ordering. You should try to ping him to check if he hasn't
> a better fontconfig recipe.

Aha. Cc'ing him.

Behdad, do you have any idea to resolve a kind of locale-specific ordering issue in fontconfig?

Comment 9 Mamoru TASAKA 2008-12-12 06:43:41 UTC
(In reply to comment #7)
> (In reply to comment #5)
> > Would you tell me to what file?
> 
> Try to replace:
> 
>         <alias>
>                 <family>sans-serif</family>
>                 <prefer>
>                         <family>AR PL UMing HK</family>
>                         <family>AR PL UMing CN</family>
>                 </prefer>
>         </alias>
> 
> in 64-ttf-arphic-uming.conf with the above for sans-serif.

Thanks. This seems to work for Japanese locale (I cannot test
for Chinese locale because I don't use it and I don't know
where to check...)

Comment 10 Mamoru TASAKA 2008-12-12 06:54:38 UTC
Created attachment 326711 [details]
glyphs of arabic numerals

By the way is it a different issue that the glyphs (especially
the widths) of arabic numerals change according to contexts?

Comment 11 Behdad Esfahbod 2008-12-17 07:00:44 UTC
I had an idea but I tested quickly and it didn't work.  I'll think about it again.  I have some other ideas about how to fix the CJK issues in fontconfig.  I'm currently working on fontconfig.  I'll see if I get to those.

Comment 12 Caius Chance 2009-01-30 00:30:38 UTC
I could lower the ranking of uming (Chinese fonts) by increasing the number of the .conf file.

Which side would you prefer to make the changes? VL Gothic, Uming, or fontconfig?

Comment 13 Jens Petersen 2009-01-30 01:32:39 UTC
The Japanese problem seems to be with Sans: eg if one switches the desktop
application font to Monospace then VLGothic is used correctly.

Comment 14 Jens Petersen 2009-01-30 02:37:50 UTC
Reverting to fontconfig-2.6.0-3.fc10 also fixes the problem, so this really does look like a fontconfig regression or change of behaviour.

Comment 16 Jens Petersen 2009-01-30 07:55:01 UTC
Thanks fixes the Japanese desktop problem for me. :)

I still don't understand the discrepancy between monospace and sans... Behdad, any idea?

Comment 17 Caius Chance 2009-01-30 09:09:59 UTC
If it didn't break some other things, then I'm going to build to rawhide.

Comment 18 Akira TAGOH 2009-01-30 11:42:18 UTC
(In reply to comment #12)
> I could lower the ranking of uming (Chinese fonts) by increasing the number of
> the .conf file.
> 
> Which side would you prefer to make the changes? VL Gothic, Uming, or
> fontconfig?

No options for VLGothic-fonts at least because this issue doesn't happen without cjkuni-fonts.

Comment 19 Mamoru TASAKA 2009-01-30 13:50:25 UTC
Created attachment 330464 [details]
screenshot again

Still seeing many Chinese glyphs.

Note that this screenshot is taken with fontconfig-2.6.95-1.git.66.gb162bfb.fc11
With reverting to fontconfig-2.6.0-3.fc10 it goes _much_ better,
however it does not remove this problem entirely.

Comment 20 Mamoru TASAKA 2009-01-30 14:21:32 UTC
For record:

# rpm -qf /etc/fonts/conf.d/* | sort | uniq
VLGothic-fonts-20081203-2.fc11.noarch
VLGothic-fonts-proportional-20081203-2.fc11.noarch
baekmuk-ttf-batang-fonts-2.2-17.fc11.noarch
baekmuk-ttf-dotum-fonts-2.2-17.fc11.noarch
baekmuk-ttf-gulim-fonts-2.2-17.fc11.noarch
baekmuk-ttf-hline-fonts-2.2-17.fc11.noarch
cjkuni-uming-fonts-0.2.20080216.1-18.fc11.noarch
dejavu-lgc-sans-fonts-2.28-3.fc11.noarch
dejavu-lgc-sans-mono-fonts-2.28-3.fc11.noarch
dejavu-lgc-serif-fonts-2.28-3.fc11.noarch
dejavu-sans-fonts-2.28-3.fc11.noarch
dejavu-sans-mono-fonts-2.28-3.fc11.noarch
dejavu-serif-fonts-2.28-3.fc11.noarch
fontconfig-2.6.95-1.git.66.gb162bfb.fc11.i386
sazanami-gothic-fonts-0.20040629-6.20061016.fc11.noarch
sazanami-mincho-fonts-0.20040629-6.20061016.fc11.noarch

Comment 21 Jens Petersen 2009-01-31 04:53:48 UTC
(In reply to comment #19)
> Created an attachment (id=330464) [details]
> screenshot again
> Still seeing many Chinese glyphs.

What is the URL for that page?

I could be wrong but it could be a Unihan issue with firefox not knowing what language the page is in?

Comment 22 Mamoru TASAKA 2009-01-31 05:01:36 UTC
(In reply to comment #21)
> (In reply to comment #19)
> > Created an attachment (id=330464) [details] [details]
> > screenshot again
> > Still seeing many Chinese glyphs.
> 
> What is the URL for that page?

Note that the glyph is wrong also on the other place
(please look at gnome-panel in the picture: especially the glyph
 of "場所" is apparently not Japanese)

The URL is the same as the one in my comment 0.

Comment 23 Jens Petersen 2009-01-31 06:22:12 UTC
Created attachment 330518 [details]
another screenshot

This is what it currently looks like for me, but I will try with a fresh rawhide install soon. :)

Comment 24 Jens Petersen 2009-01-31 07:24:10 UTC
I just tried on a fresh rawhide install and it still looks good to me.

Tasaka-san, perhaps could you try again?

(Anyway unfortunately this fix/workaround won't be in F11Alpha anyway...)

Comment 25 Mamoru TASAKA 2009-01-31 08:23:32 UTC
Well, after some investigation, it seems that
when I hide (i.e. add 'dot' to the file name)
65-baekmuk-ttf-dotum.conf (in baekmuk-ttf-dotum-fonts-2.2-17.fc11.noarch)
it looks good now (even with fontconfig-2.6.95-1.git.66.gb162bfb.fc11)

Comment 26 Jens Petersen 2009-01-31 10:14:30 UTC
Ah I see, thank you.  Yes, baekmuk dotum is no longer installed by default, but sounds like a similar workaround is needed there too.

Comment 27 Jens Petersen 2009-01-31 10:15:03 UTC
would still appreciate some comment from Behdad...

Comment 28 Caius Chance 2009-01-31 12:07:16 UTC
(In reply to comment #24)
> (Anyway unfortunately this fix/workaround won't be in F11Alpha anyway...)

I will push to F11 immediately when it is released.

(In reply to comment #25)
> Well, after some investigation, it seems that
> when I hide (i.e. add 'dot' to the file name)
> 65-baekmuk-ttf-dotum.conf (in baekmuk-ttf-dotum-fonts-2.2-17.fc11.noarch)
> it looks good now (even with fontconfig-2.6.95-1.git.66.gb162bfb.fc11)

Hmm, then I should also work on baekmuk then.

Comment 29 Mamoru TASAKA 2009-02-03 10:20:22 UTC
It seems that with baekmuk-ttf-XXX-2.2-19.fc11,
cjkuni-XXXX-0.2.20080216.1-20.fc11 Japanese glyphs are shown
correctly.

Comment 30 Caius Chance 2009-02-04 00:51:38 UTC
There are .conf templates from package fontpackages. I applied from that.

cjkuni-fonts is patched:
http://koji.fedoraproject.org/koji/buildinfo?buildID=81269

baekmuk-ttf-fonts is patched:
http://koji.fedoraproject.org/koji/buildinfo?buildID=81270

Comment 31 Jens Petersen 2009-02-04 03:52:37 UTC
No comment from Behdad so can probably close this for now.

Dunno if it is worth discussing the change of behaviour upstream...

Comment 32 Ryo Dairiki 2009-02-14 09:22:14 UTC
On Fedora-10, the same bug appears after usual update.

VLGothic fonts:
vlgothic-p-fonts-20090204-2.fc10.noarch
vlgothic-fonts-20090204-2.fc10.noarch
vlgothic-fonts-common-20090204-2.fc10.noarch

Baekmuk fonts:
baekmuk-ttf-fonts-gulim-2.2-9.fc10.noarch
baekmuk-ttf-fonts-dotum-2.2-9.fc10.noarch
baekmuk-ttf-fonts-batang-2.2-9.fc10.noarch
baekmuk-ttf-fonts-common-2.2-9.fc10.noarch
baekmuk-ttf-fonts-hline-2.2-9.fc10.noarch
baekmuk-bdf-fonts-2.2-5.fc9.noarch

Is this also related to this report?

Comment 33 Jens Petersen 2009-02-16 02:51:36 UTC
(In reply to comment #32)
> On Fedora-10, the same bug appears after usual update.
> baekmuk-ttf-fonts-gulim-2.2-9.fc10.noarch
> baekmuk-ttf-fonts-dotum-2.2-9.fc10.noarch
> baekmuk-ttf-fonts-batang-2.2-9.fc10.noarch
> baekmuk-ttf-fonts-common-2.2-9.fc10.noarch
> baekmuk-ttf-fonts-hline-2.2-9.fc10.noarch
> baekmuk-bdf-fonts-2.2-5.fc9.noarch

It sounds like it could be.
What if you remove the baekmuk fonts?
(They are not default for Hangul anymore anyway.)

Comment 34 Mamoru TASAKA 2009-02-16 14:52:55 UTC
fontconfig-2.6.97-3.g945d6a4.fc11 again reproduces this problem......

Comment 35 Caius Chance 2009-02-16 15:54:14 UTC
I have only built on rawhide. Let me backport that.

Comment 36 Jens Petersen 2009-02-17 03:43:47 UTC
(In reply to comment #35)
> I have only built on rawhide. Let me backport that.

May be better to test first on f10.

Comment 37 Jens Petersen 2009-02-17 04:05:25 UTC
Behdad, could you please update on this important issue?

Comment 38 Mamoru TASAKA 2009-02-17 07:57:31 UTC
Created attachment 332187 [details]
screenshot with old fontconfig

(In reply to comment #34)
> fontconfig-2.6.97-3.g945d6a4.fc11 again reproduces this problem......

To clarify this, screenshot again.
This one with (the old) fontconfig-2.6.95-1.git.66.gb162bfb.fc11.i386 .
Looks fine.

Comment 39 Mamoru TASAKA 2009-02-17 07:58:48 UTC
Created attachment 332188 [details]
screenshot with new fontconfig

And screenshot with new fontconfig fontconfig-2.6.97-3.g945d6a4.fc11
Japanese font glyphs are not used.

Comment 40 Jens Petersen 2009-02-17 08:19:03 UTC
Yes, I see un-core-fonts being used on the ja desktop - I will try to get the fontconfig files there updated soon.

Comment 41 Ryo Dairiki 2009-02-20 17:15:40 UTC
At last, I've found that cjkunifonts-uming package is the cause of this problem in my environment.

It seems like gryphs from that package are chosen in some cases, even in Japanese environments.
Japanese period followed by an alphabet is the one example.

After uninstalling that package, the problem goes away.
And I've confirmed that reinstalling this package caused the problem again.

I'll send you a detailed report later.

Comment 42 Ryo Dairiki 2009-02-24 04:59:14 UTC
Problem:
- Chinese gryphs are shown when you use Monospace even in Japanese locale.
- Some characters seems different in the same context, as different gryphs are selected according to the following characters.
(For example, the width of the space character followed by "A (alphabet)" has different width with the one followed by "a (hiragana)".)

Reason:

Fontconfig choose proper gryph for virtual fonts, according to the current locale, and the context.
If you write Japanese in Japanese locale, fontconfig seeks to fonts which has Japanese gryphs.
It seems like some Chinese fonts also has Japanese gryphs by some reason.
Maybe, they are in CJ unified regions, so we cannot fix this problem simply by eliminating them.
Talking about the second problem, both Japanese fonts and DejaVu fonts have space characters, 
so fontconfig sometimes choose gryphs from Japanese fonts, and sometimes from DejaVu fonts.

Solution:

Change the gryph selection order of Fontconfig.
In Japanese environment, gryphs from Japanese gryphs should be chosen first.
In Chinese environment, gryphs from Chinese gryphs should be chosen first.
Talking about DejaVu fonts, either locale specific gryphs or DejaVu gryphs should be chosen 
independently with the context.
This change maybe require huge change in fonts.conf and might need some patches in the source of fontconfig too.

Comment 43 Behdad Esfahbod 2009-03-11 15:51:10 UTC
I'm still confused.  From what I understand, an alias I added in recent fontconfig updates is causing this.  If that's true, which alias, and if you can move it further down to not cause the bug, what's the patch?  Thanks.

Comment 44 Jens Petersen 2009-03-12 00:32:28 UTC
Ok - I think we have to wait for an installable rawhide or F11 Beta
for further testing.  With various font .conf files added to CJK fonts
I think things are a bit better now but there may still be some
more finer adjusts needed for F11?

Comment 45 Jesse Keating 2009-03-16 18:02:59 UTC
Rawhide should be installable, what I need is a clear idea if this is really a beta blocker issue or not.

Comment 46 Mamoru TASAKA 2009-03-16 18:14:57 UTC
(In reply to comment #45)
> Rawhide should be installable, what I need is a clear idea if this is really a
> beta blocker issue or not.  

Well, I tried the current fontconfig-2.6.99.behdad-3.fc11 and it seems
be working for me. So tagging this fontconfig as f11-beta may be
a good idea.

Comment 47 Jesse Keating 2009-03-16 22:09:20 UTC
We just tagged so I'll close this rawhide.