This service will be undergoing maintenance at 00:00 UTC, 2016-08-01. It is expected to last about 1 hours
Bug 107952 - Kochi Gothic, Mincho aren't used in English Unicode locales
Kochi Gothic, Mincho aren't used in English Unicode locales
Status: CLOSED RAWHIDE
Product: Fedora
Classification: Fedora
Component: fontconfig (Show other bugs)
rawhide
All Linux
medium Severity medium
: ---
: ---
Assigned To: Owen Taylor
http://ftp.dulug.duke.edu/pub/redhat/...
: i18n
: 107617 138783 (view as bug list)
Depends On: 111781 150430
Blocks: FC5Target
  Show dependency treegraph
 
Reported: 2003-10-24 16:48 EDT by John Thacker
Modified: 2007-11-30 17:10 EST (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2005-10-23 13:00:12 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
Screenshot of kana not using the same font as kanji (200.69 KB, image/png)
2004-10-06 08:05 EDT, Giovanni Glass
no flags Details
After the change suggested to the local.conf file (303.21 KB, image/png)
2004-10-07 01:48 EDT, Giovanni Glass
no flags Details

  None (edit)
Description John Thacker 2003-10-24 16:48:38 EDT
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.5) Gecko/20031016

Description of problem:
If the locale is set to, say, en_US.UTF-8, then Kochi Gothic and Kochi Mincho
are never used to display Unicode documents.  On systems without
bitmap-fonts-cjk installed, this means something like MiscFixed will be used for
hiragana and katakana, which is extremely ugly.  It works fine with
LANG=ja_JP.UTF-8 is set, however.  The problem occurs when viewing files both in
Mozilla and in other programs, like gedit or gucharmap, when the default "Sans"
alias is selected as the font.  (Mozilla also works fine when viewing Shift-JIS
or EUC-JP files.)

I'm not sure if this is a fontconfig problem or a font problem, but I do know
that the fonts.cache-1 file for Kochi Gothic and Kochi Mincho doesn't list en
(among others) as a supported language, whereas Microsoft's MS Gothic and MS
Mincho files list lots of supported languages and work fine as well.  The
fangsong ti font in bitmap-fonts-cjk also lists lots of supported languages, and
works well also.

As long as your locale is set to some variation of en_US, including en_US.UTF-8,
fontconfig will refuse to use Kochi Gothic and Kochi Mincho to display Japanese
characters.  This is bad, since those are the best Japanese fonts, and it ruins
the point of using Unicode locales in the first place.

Version-Release number of selected component (if applicable):
ttfonts-ja-1.2-29

How reproducible:
Always

Steps to Reproduce:
1. export LANG=en_US.UTF-8
2. Set default font to Sans or Serif
3. View a webpage containing Unicode Japanese with mozilla, open a file with
Unicode encoded Japanese in gedit, or run gucharmap and look at what font is
used for Hiragana and Katakana for Sans   

Actual Results:  Kochi Gothic and Kochi Mincho aren't used to display Japanese
characters.  Fangsong ti, if installed from bitmap-fonts-cjk, MS Gothic or MS
Mincho, if installed, or MiscFixed will be used instead.

Expected Results:  Kochi Gothic and Kochi Mincho, as the preferred fonts for
Japanese display, should be used.

Additional info:
Comment 1 John Thacker 2003-10-25 00:29:47 EDT
To be more precise, fontconfig will select Kochi Gothic and Kochi Mincho, but
only if no other fonts that can provide the glyphs and claim to support language
"en" are available.  However, since the bitmap-fonts RPM must always be
installed for vte and hence for gnome-terminal, this means that the terrible
(for Japanese) MiscFixed font will be preferred for katakana and hiragana over
Kochi Gothic and Kochi Mincho.  (Similarly, Fangsong ti will be preferred, as
will the MS Japanese fonts if installed.)
Comment 2 John Thacker 2003-10-25 00:33:07 EDT
That is, most applications will eventually fallback to Kochi Gothic and Kochi
Mincho, if no other fonts (including MiscFixed) are available.  Mozilla is even
worse-- it always completely refuses to use an "out of language" font,
regardless of glyph presence, and thus won't use Kochi Gothic and Kochi Mincho
for Unicode at all.
Comment 3 Leon Ho 2003-12-07 19:44:41 EST
Maybe it is because Kochi Gothic's lang support does not listed en_US
as one of the supported locale. MiscFixed on the other hand, support
most of the langs (includes en_US). Is it the main reason, Owen?
Comment 4 John Thacker 2003-12-10 01:07:38 EST
That is why, yes.  There's been some discussion on the fontconfig list
about the right way to handle this.  The problem is that it's very
natural to want to have a preferred font used by the Sans alias for
viewing Japanese (and similarly for other languages), and there seems
to be no particular reason to insist that the font used for viewing
Japanese be able to handle English as a supported locale, if it's only
used for Japanese characters not provided in English fonts.

The problem is figuring out the right way for fontconfig to handle it.
 It's even worse when dealing with files in Unicode, which doesn't
have a language associated with the encoding.  Owen and Keith Packard
have discussed it some.

In the short run, adding broader coverage to Kochi Gothic and Kochi
Mincho would help.  In the long run, fontconfig's behavior is clearly
broken.  (It's unreasonble to expect that all fonts for all languages
contain all necessary Japanese glyphs so that fontconfig will be happy
when selecting them for users in Japanese locales, for example.)
Comment 5 John Thacker 2004-01-29 12:41:11 EST
pango-1.3.2, freetype-2.1.7 and other updates from rawhide/development
seem to go a long way towards resolving this problem.  I've also been
running fontconfig-2.2.92, but that didn't seem to really resolve it
until the pango and freetype updates.  In any case, closing as
RAWHIDE, since it looks resolved using development for FC2 right now.
Comment 6 Giovanni Glass 2004-10-06 08:05:17 EDT
Created attachment 104832 [details]
Screenshot of kana not using the same font as kanji

This is from Firefox PR10 and Fedora Core 3 Test2 (patched to current as of
Oct. 6, 2004)

This is a screenshot of google using English unicode with Japanese search
results. I increased the font size to emphasize how all the kana are not using
the same font as the kanji and how they don't scale or become anti-aliased.
This is NOT fixed and it has been happening since Fedora Core 1 (I run this at
work).

SJIS and ISO-2022-JP doesn't seem to have this problem. I can click into any of
these Japaense search results and they don't have the messed up kana like in
the screenshot.
Comment 7 John Thacker 2004-10-06 09:28:04 EDT
Try putting the following in your /etc/fonts/local.conf file.  It will
cause fontconfig (at least on 2.2.96, but probably on the FC included
version as well) to match on Kochi Gothic and Kochi Mincho ignoring
language.  ("Strong" binding matches ignoring language.)

        <match target="pattern">
                <test name="family">
                        <string>sans-serif</string>
                </test>
                <edit name="family" binding="strong" mode="prepend">
                        <string>Luxi Sans</string>
                        <string>Bitstream Vera Sans</string>
                        <string>Verdana</string>
                        <string>Kochi Gothic</string>
                </edit>
        </match>
 
        <match target="pattern">
                <test name="family">
                        <string>serif</string>
                </test>
                <edit name="family" binding="strong" mode="prepend">
                        <string>Bitstream Vera Serif</string>
                        <string>Times New Roman</string>
                        <string>Kochi Mincho</string>
                </edit>
        </match>
                                                                     
          
        <match target="pattern">
                <test name="family">
                        <string>monospace</string>
                </test>
                <edit name="family" binding="strong" mode="prepend">
                        <string>Luxi Mono</string>
                        <string>Bitstream Vera Sans Mono</string>
                        <string>Andale Mono</string>
                        <string>Kochi Gothic</string>
                </edit>
        </match>
Comment 8 Giovanni Glass 2004-10-07 01:39:02 EDT
I tried that out. It works! Is this a hack or something that can be
commited into RAWHIDE CVS?
Comment 9 Giovanni Glass 2004-10-07 01:48:54 EDT
Created attachment 104882 [details]
After the change suggested to the local.conf file

Looks good and it scales correctly.
Comment 10 John Thacker 2004-10-07 09:04:14 EDT
Well, it is a bit of a hack.

It's very much a "Western user who also uses Japanese"-centric
solution, and it also means that the language of one's locale (or the
language associated with a charset in Mozilla) will not be used to
choose the proper font.

It will result in the Japanese Kochi fonts always being preferred over
Chinese (and Korean) fonts for displaying Han characters.  For
Unicode, especially for English-based Unicode locales, there's no way
around this.  This results in Japanese fonts being used for
specifically Chinese webpages and Chinese charsets as well, from what
I remember.  This does create problems with the glyphs which have
different representations in the two languages.  Also, since
characters which are rare in Japanese and don't exist in the Kochi
fonts will be selected from the Chinese fonts, a somewhat inconsistent
character style sometimes results as well.

Also, this results in purely Japanese pages with Japanese charsets
using Luxi Sans (for Sans, and so on for Serif) for the
(non-doublewidth) Roman characters and Kochi fonts for other things,
rather than using Kochi fonts for everything when possible. 
Furthermore, regardless of one's locale, Luxi Sans will be used for
all characters it contains, rather than a language-specific font. 
When language-specific fonts have Latin characters designed to blend
in with their other glyphs better, this will mean slightly worse
looking characters.
Comment 11 Owen Taylor 2004-10-07 11:59:13 EDT
Certainly not something that would go into the default fonts.conf.
It's  conceivable we could tighten up the set of required characters to
support 'en' if someone wanted to investigate what the Kochi
fonts are missing.

This problem will go away with the change of Mozilla to use
Pango, since Pango is smarter about selecting fonts. (It will
realize that an 'en' language makes no sense for Han/Kana
characters and subsitute it with 'ja' for Kana or 'xx' for
Han characters)
Comment 12 John Thacker 2004-10-08 10:27:34 EDT
The required characters to support 'en' according to fontconfig which
are missing are all accented characters sometimes used (but sometimes
dropped) for foreign words which have been adopted by English.  They
are (thanks to fontconfig source and gucharmap):

0x00C0 (Capital Letter A with Grave)
0x00C7 (Capital Letter C with Cedilla)
0x00C8 (Capital Letter E with Grave)
0x00C9 (Capital Letter E with Acute)
0x00CA (Capital Letter E with Circumflex)
0x00CB (Capital Letter E with Diaeresis)
0x00CF (Capital Letter I with Diaeresis)
0x00D1 (Capital Letter N with Tilde)
0x00D4 (Capital Letter O with Circumflex)
0x00D6 (Capital Letter O with Diaeresis)
0x00E0 (Small Letter A with Grave)
0x00E7 (Small Letter C with Cedilla)
0x00E8 (Small Letter E with Grave)
0x00E9 (Small Letter E with Acute)
0x00EA (Small Letter E with Circumflex)
0x00EB (Small Letter E with Diaeresis)
0x00EF (Small Letter I with Diaeresis)
0x00F1 (Small Letter N with Tilde)
0x00F4 (Small Letter O with Circumflex)
0x00F6 (Small Letter O with Diaeresis)

If these characters could somehow be added to the Kochi fonts, the
fonts would support English according to fontconfig.
Comment 13 Owen Taylor 2004-11-18 13:14:06 EST
*** Bug 138783 has been marked as a duplicate of this bug. ***
Comment 14 Need Real Name 2004-12-13 19:09:31 EST
I've hacked at fontconfig today to try and solve this bug.
What I did was to ammend the strict sort order used by FcFontSort so
that it 'satisfies' the language specified in the pattern by locating
the best matching font supporting each pattern language and then
ignores language in the remaining fonts for purposes of matching.

So, when asking for 'sans:lang=en', you'll get an English font first,
and then the remaining fonts sorted with respect to the 'sans' alias
alone -- pushing Kochi fonts ahead of other English-supporting Han fonts.

This change is in Fontconfig HEAD CVS; if someone would like to play
and see if it fixes the problems for them, that would be great.
Comment 15 John Thacker 2004-12-28 17:06:21 EST
I've been home for the holidays since the change was made, and have finally got
a chance to test it.  IMO, this solves all the problems that can be rationally
and consistently solved in a default configuration, and solves everything that
is fontconfig's problem.

Most remaining issues I can think of are clearly issues between pango, user
applications, documents, and individual configuration.  (For example, in an
English Unicode locale, view a UTF-8 document that isn't labeled for language
but is mostly Japanese.  Should the Latin characters be rendered with the
Japanese font used for everything else, or with the preferred English-supporting
font?  Similarly problems of preferring a Japanese vs. Chinese font for Han
characters in an unspecified document when in a non-CJK locale are up to
individual users.)

Now we just need a fontconfig release, and inclusion of that release into Fedora
Core and RHEL.  I know RH prefers to ship stable versions over development
versions, so even though I find the 2.2.9x series very stable, I'd be surprised
if RH included it.  What's preventing a 2.3.0 release?
Comment 16 Jens Petersen 2005-03-01 06:05:39 EST
I tried fontconfig-2.2.99 and it seems to fix bug 138783 too . :)
Comment 17 John Thacker 2005-03-03 11:43:31 EST
So, since 2.3.0 is released, I assume that Owen is working hard on editing the
various packages and default configuration, and we'll see it in RawHide soon,
right? ;)
Comment 18 John Thacker 2005-03-10 11:28:31 EST
Can we *please* update to 2.3.0 or 2.3.1 ( http://fontconfig.org/release/ ) in
time for FC4?  It fixes this (and some other important i18n and font rendering
problems).
Comment 19 Jens Petersen 2005-04-19 03:35:01 EDT
Still appears to be fontconfig-2.2.3 in fc devel...
Comment 20 David Zeuthen 2005-04-19 16:44:53 EDT
Owen told me he wanted to do this upgrade so I'm reassigning the bug to him.
Comment 21 Warren Togami 2005-05-16 05:14:59 EDT
If Sopwith does not accept the version upgrade (which is highly unlikely now
given that regressions in fontconfig will affect many people and we have little
time left before final), can the fix in 2.3.x be backported into our current
fontconfig?  Then that'll be our short-term solution.

Adding FC5Blocker for the long term solution, upgrade fontconfig.
Comment 22 Owen Taylor 2005-05-16 15:52:23 EDT
The code portion is big enough that a backport, in my opinion, would be
riskier than a straight upgrade, and create significant deviation from
upstream, something that we want to avoid in Fedora.
Comment 23 Mike A. Harris 2005-06-02 17:28:15 EDT
Moving to FC5Target, as FC4 is 'done', barring release blockers that
pop up.
Comment 24 Akira TAGOH 2005-09-28 04:58:17 EDT
This bug could be changed to MODIFIED since we got fontconfig 2.3.2-1 on rawhide
now.
Comment 25 Jens Petersen 2005-09-28 05:09:13 EDT
Light testing looks good to me.
Comment 26 John Thacker 2005-10-23 13:00:12 EDT
Yes, this works for me too.  I'll close it as fixed in Rawhide.
Comment 27 John Thacker 2006-10-28 17:00:38 EDT
*** Bug 107617 has been marked as a duplicate of this bug. ***

Note You need to log in before you can comment on or make changes to this bug.