Bug 499902 - [CJK] 65-nonlatin.conf needs updating and lang tags for CJK etc
Summary: [CJK] 65-nonlatin.conf needs updating and lang tags for CJK etc
Keywords:
Status: CLOSED UPSTREAM
Alias: None
Product: Fedora
Classification: Fedora
Component: fontconfig
Version: rawhide
Hardware: All
OS: Linux
low
medium
Target Milestone: ---
Assignee: Behdad Esfahbod
QA Contact: Fedora Extras Quality Assurance
URL: http://picasaweb.google.com/fangqq/Pr...
Whiteboard:
Depends On:
Blocks: F13Blocker, F13FinalBlocker 554911 572841
TreeView+ depends on / blocked
 
Reported: 2009-05-08 19:16 UTC by Qianqian Fang
Modified: 2010-07-04 05:29 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: Enhancement
Doc Text:
Clone Of:
: 572841 (view as bug list)
Environment:
Last Closed: 2010-03-12 07:03:46 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)
language specific fontconfig settings for Japanese (2.60 KB, text/plain)
2009-05-08 19:16 UTC, Qianqian Fang
no flags Details
language specific fontconfig settings for Chinese (6.73 KB, text/plain)
2009-05-08 19:17 UTC, Qianqian Fang
no flags Details


Links
System ID Private Priority Status Summary Last Updated
FreeDesktop.org 20911 0 None None None Never

Description Qianqian Fang 2009-05-08 19:16:26 UTC
Created attachment 343146 [details]
language specific fontconfig settings for Japanese

The CJK language fontconfig settings has been a long-standing issue ever since the beginning of using fontconfig. There hasn't been a good solution yet. The major difficulties for causing this problem, IMO, include

1. the default 65-nonlatin.conf was badly outdated with unoptimized font orders
2. there are no CJK language-specific fontconfig files, therefore, to override the default orders, one has to add font prefer list in the fontconfig settings shipped with individual CJK font package.
3. because these settings were scattered into many font packages, this further makes the problem complex, and the conflict between different CJK font packages became more and more frequent.

To solve this issue, I propose the following solution
1. update 65-nonlatin.conf and setup an optimized and up-to-date font list
2. add language specific fontconfig settings, defining font preferred orders with for a given lang tag, and assign these files a lower number than 65
3. remove all font order settings from all CJK related font packages

For solution step 1, I proposed a completely updated 65-nonlatin.conf at http://bugs.freedesktop.org/show_bug.cgi?id=20911 . My rationale of the font orders was also explained in details.

For solution step 2, I created two default settings for ja and zh locales (that for ko can be done similarly) as examples, and you can find them in the attachments. Because I used lang tag matching in the rules, these files can be installed concurrently (in this sense, it is better than the language-selector in Ubuntu, which needs to run fontconfig-voodoo to link a set of active config under a given language).

I did the following to test this proposal. From what I saw for zh-*, ja and en locales, all the wanted features were working properly.

Here are the details of my test:

First, I did this test with rawhide updated this morning. On the system, I installed chinese-support and japanese-support, including wqy-bitmap-fonts, wqy-zenhei-fonts, arphic-uming, vlgothic-fonts and some mincho Japanese font. 

To clean up all the font config settings, I "su" to root, and run the following
  cd /etc/fonts/conf.d/
  mkdir cjk
  mv *wqy* cjk
  mv *vlgo* cjk
  mv *arphic* cjk
  mv *gothic* cjk

This is just a quick way to get rid of all the font-wise preference settings. In the future, most of these files can stay, as long as they remove the blocks to set <prefer> or prepend family names.

The second step is to update (of course, make a backup first) the 65-nonlatin.conf by downloading from the attachment at http://bugs.freedesktop.org/show_bug.cgi?id=20911

Then, download the 55-language_fonts_ja-jp.conf and 55-language_fonts_zh-cn.conf from this thread and save them under /etc/fonts/conf.d/ . This completes the settings.

For Chinese users, some of them prefer bitmap glyphs for Han characters (35%), some prefer vectors (65%). This is controlled by installing wqy-bitmap-fonts or uninstalling the bitmap fonts.

I tested my proposed settings for English, Japanese and Chinese languages, with bitmap preferred settings and vector preferred settings. My screenshots for all combination can be found at this online album:
http://picasaweb.google.com/fangqq/ProposalForFontconfigSettingsForCJKLanguages

When bitmap font is installed, for en and zh desktops, fonts at sizes 9pt ~ 12pt will be rendered as bitmaps for all font aliases (sans,serif, mono); otherwise, it will be rendered by the respective vector fonts. Particularly, for en desktop, no more font-mosaic problems caused by high priority of Japanese fonts (such as http://picasaweb.google.com/fangqq/ConfigScreenshot#5333302146968242258) . When one remove the bitmap fonts, all Hanzi glyphs were rendered by the preferred vector fonts, as expected (some remaining ones are from UMing, which I did not disable).

For ja desktop, all fonts were rendered by their preferred vector fonts, no matter bitmap Chinese fonts installed or not (please ignore "驿" and "阵" as they are not defined in JIS).



I believe this approach greatly simplifies the CJK font settings by centralizing the related settings to language specific files. All the expected basic rendering order were achieved with the current proposed config files. Of course one can fine-tune them further. For Korean users, a 55-language-fonts-ko.conf file can be made similarly.

It will be really great if this proposal can be tested and agreed by all CJK font package maintainers.

Comment 1 Qianqian Fang 2009-05-08 19:17:13 UTC
Created attachment 343148 [details]
language specific fontconfig settings for Chinese

Comment 2 Bug Zapper 2009-06-09 15:29:05 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 11 development cycle.
Changing version to '11'.

More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

Comment 3 Jens Petersen 2010-03-03 09:20:46 UTC
Closing since this issue has been under discussion upstream.

Comment 4 Jens Petersen 2010-03-04 10:17:53 UTC
Reopening since WQY Zenhei is still overriding vlgothic in Japanese
webpages in firefox.

Comment 5 Qianqian Fang 2010-03-10 07:10:02 UTC
I don't see how this can still happen. Please attach your FC_DEBUG output of a test command, for example
 pango-view  --waterfall --text='与返骨直' --language=ja

Comment 6 Akira TAGOH 2010-03-11 13:41:30 UTC
(In reply to comment #4)
> Reopening since WQY Zenhei is still overriding vlgothic in Japanese
> webpages in firefox.    

That's the "ll v.s. ll-cc" issue. I've tried to run firefox with FC_DEBUG=4 and have a look at the logs. I see they requested both of lang="ja-jp" and lang="ja" in the queries of the font. ideally it would be nice to fix this issue in fontconfig though, we may want to think about a workaround in firefox for that?

Or add a rule for "ll" in every fonts... dunno.

Comment 7 Qianqian Fang 2010-03-12 00:58:24 UTC
replied from gmail, but never show up, repost.

(In reply to comment #6)
> That's the "ll v.s. ll-cc" issue. I've tried to run firefox with FC_DEBUG=4 and
> have a look at the logs. I see they requested both of lang="ja-jp" and
> lang="ja" in the queries of the font. ideally it would be nice to fix this
> issue in fontconfig though, we may want to think about a workaround in firefox
> for that?
> 
> Or add a rule for "ll" in every fonts... dunno.    

If I were you, I would use the following construct
   <test name="lang" compare="contains"><string>ja</string></test>
to get around this. It is not as dangerous as some of you might think, the language tags are rather stable in Linux.

Of course, the ideal solution would be to ask fontconfig to support regular-expression in compare, something like
   <test name="lang" compare="regex"><string>^ja(-\w+)*</string></test>

Comment 8 fujiwara 2010-03-12 02:04:33 UTC
(In reply to comment #7)
> If I were you, I would use the following construct
>    <test name="lang" compare="contains"><string>ja</string></test>
> to get around this. It is not as dangerous as some of you might think, the
> language tags are rather stable in Linux.

Me either.

Comment 9 Akira TAGOH 2010-03-12 07:00:36 UTC
(In reply to comment #7)
> If I were you, I would use the following construct
>    <test name="lang" compare="contains"><string>ja</string></test>
> to get around this. It is not as dangerous as some of you might think, the
> language tags are rather stable in Linux.

Sure. it looks good in this case but it may depends. we had experienced a bug like http://bugs.freedesktop.org/show_bug.cgi?id=23419. it may be too early saying that is stable in _Linux_.

> Of course, the ideal solution would be to ask fontconfig to support
> regular-expression in compare, something like
>    <test name="lang" compare="regex"><string>^ja(-\w+)*</string></test>    

FWIW "contains" isn't supposed to check the language name comparison only. it tries to check the code coverage with orth files first and then compare the name if it fails.

Comment 10 Akira TAGOH 2010-03-12 07:03:46 UTC
hmm, it may be good to keep this AS IS and close upstream again. I'll make a clone for vlgothic-fonts specific issue.


Note You need to log in before you can comment on or make changes to this bug.