This service will be undergoing maintenance at 00:00 UTC, 2016-08-01. It is expected to last about 1 hours

Bug 485566

Summary: [CJK] Fontconfig should select DejaVu font for rendering ASCII glyphs
Product: [Fedora] Fedora Reporter: Ryo Dairiki <ryo-dairiki>
Component: fontconfigAssignee: Akira TAGOH <tagoh>
Status: NEW --- QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: medium    
Version: rawhideCC: behdad, fonts-bugs, i18n-bugs, mfabian, mwu, petersen, pnemade, pwu, sacredfox, tagoh, tfujiwar, yshao
Target Milestone: ---Keywords: FutureFeature
Target Release: ---   
Hardware: All   
OS: Linux   
See Also: https://bugzilla.redhat.com/show_bug.cgi?id=919861
Whiteboard:
Fixed In Version: Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Attachments:
Description Flags
A new configuration file to force fontconfig to prefer DejaVu glyphs to Japanese glyphs.
none
A new configuration file to force fontconfig to prefer DejaVu glyphs to Japanese glyphs. none

Description Ryo Dairiki 2009-02-14 09:27:47 EST
Description of problem:

As you know, fontconfig finds proper fonts for text rendering, 
depending on locales of process and text to render.
It works properly in the most cases, but there are some sever cases.

For example, if you type only "@" or "}" in gedit, 
it uses glyph of VLGothic in Japanese locale.
However, if you type "a" after the first character, 
it changes one's mind to choose glyphs of DejaVu fonts.
It's very annoying for Japanese users.

In the worst cases, the width of the indentations of sourcecode seems different.
(As the width of the " " glyphs are differ between VLGothic and DejaVu)

How reproducible:


Steps to Reproduce:
1. Login X11 as a Japanese.
2. Open gedit and change the configuration to use "Monospace".
3. Type "@" on the first line, and "@a" on the first line.
  
Actual results:
"@" seems differ between the two lines.

Expected results:
"@" seems the same between the two lines.
The same glyph should be used for rendering the two "@".
Comment 1 Ryo Dairiki 2009-02-14 09:34:23 EST
Created attachment 331924 [details]
A new configuration file to force fontconfig to prefer DejaVu glyphs to Japanese glyphs.

This configuration make fontconfig prefer DejaVu glyphs to Japanese glyphs in rendering English.
When you apply this, fontconfig uses DejaVu glyphs for rendering "@" in Japanese locale.
Note, this configuration will not affect on font selection order for Japanese characters.
Comment 2 Ryo Dairiki 2009-02-14 10:24:22 EST
Created attachment 331925 [details]
A new configuration file to force fontconfig to prefer DejaVu glyphs to Japanese glyphs.

A new configuration file to force fontconfig to prefer DejaVu glyphs to Japanese glyphs.
After installing this configuration, fontconfig uses DejaVu glyph as far as possible 
even in Japanese locale.
Comment 3 Akihiro Nomura 2009-04-28 23:44:11 EDT
I tried this patch to vlgothic-* fonts, into /usr/share/fontconfig/conf.avail/66-vlgothic-*.conf.
This patch seems to resolve following issues (or make them to original state),
which have been occurred since new fontconfig rules applied to VLGothic*.
1. In monospace, "@" glyph is rendered sometimes in vlgothic or sometimes in dejavu mono and
   their width is different. (This bug)
2. Some English alphabet and numbers in the applets in gnome-panel looks like monospace and hard to read.
   I don't know why this occur and is resolved by this patch.
3. Some ambiguous width character (like □) in gnome-terminal is rendered as fill-width but treated as half-width,
   I mentioned this issue in Bug #487061.
Comment 4 Jens Petersen 2009-07-13 19:02:35 EDT
I think we need someone in fedora-i18n to help get these kinds of fixes in?
Comment 5 Nicolas Mailhot 2009-07-14 04:40:44 EDT
This is not really a fix, this is an ugly hack to workaround fontconfig default behaviour. It only works in very specific instances, when specific fonts are installed, and will probably have nasty side-effects somewhere.

The correct fix is either to get upstream to drop latin glyphs from VLGothic, or enhance fontconfig so you can blacklist parts of a font (as people have been requesting for years).

I'm sure Behdad would appreciate help in doing 2. though
Comment 6 Nicolas Mailhot 2009-07-14 04:52:16 EDT
Also, apart from those local fixes, the core of the problem is pango has usually no idea what language you're trying to type, and tries to guess from what you're typing. Sometimes it guesses wrong, or changes its mind when you type more text that gaves it more info.

To put an end to this class of bugs (which are not limited to VLGothic or Japanese) you need a language switcher applet that, in addition to managing input methods, tells explicitely apps what's being typed.
Comment 7 Akira TAGOH 2009-07-14 05:44:35 EDT
(In reply to comment #5)
> The correct fix is either to get upstream to drop latin glyphs from VLGothic,

I'm opposed to this idea. nothing wrong in VLGothic and it introduces another bug in applications using any other rendering/multilingual framework libraries doesn't supports a fallback fonts. like Bug #434753.

(In reply to comment #6)
> Also, apart from those local fixes, the core of the problem is pango has
> usually no idea what language you're trying to type, and tries to guess from
> what you're typing.

Does it? though Pango sets a default PangoLanguage from current locale when it's starting. why doesn't Pango just has a priority to pick up the font according to that?

> To put an end to this class of bugs (which are not limited to VLGothic or
> Japanese) you need a language switcher applet that, in addition to managing
> input methods, tells explicitely apps what's being typed.  

If we want any framework other than locale system to determine the input language etc, I think it may be better adding a kind of feature to the input method right rather than having new one and IM running at the same time.
Comment 8 Nicolas Mailhot 2009-07-14 06:46:39 EDT
(In reply to comment #7)
> (In reply to comment #5)
> > The correct fix is either to get upstream to drop latin glyphs from VLGothic,
> 
> I'm opposed to this idea. nothing wrong in VLGothic and it introduces another
> bug in applications using any other rendering/multilingual framework libraries
> doesn't supports a fallback fonts. like Bug #434753.

Anything that does not use fontconfig nowadays with its built-in font substitution is a bug period. We barely have the energy to get our main font system sort of working without having to work with others.

> (In reply to comment #6)
> > Also, apart from those local fixes, the core of the problem is pango has
> > usually no idea what language you're trying to type, and tries to guess from
> > what you're typing.
> 
> Does it? though Pango sets a default PangoLanguage from current locale when
> it's starting. why doesn't Pango just has a priority to pick up the font
> according to that?

Because there is no strict equivalence between the language of your desktop (the locale) and the language you are typing (what pango needs to process). It is very common to type in a foreign language, and anyway in lots of regions people use the en_US locale because theirs is not available or the translations are incomplete or bad

> > To put an end to this class of bugs (which are not limited to VLGothic or
> > Japanese) you need a language switcher applet that, in addition to managing
> > input methods, tells explicitely apps what's being typed.  
> 
> If we want any framework other than locale system to determine the input
> language etc, I think it may be better adding a kind of feature to the input
> method right rather than having new one and IM running at the same time.  

That assumes there is one true IM system that works for all languages. So far that hasn't been the case. And even if there was one system, you need to distinguish between the input method selection and the language selection because there is no 1:1 relashionship between them (any latin layout will be able to type English even if it's not an English layout, that's probably also true for russian and cyrillic layouts, etc)

Accurate language selection gets you more accurate text rendering, correct spellchecker selection, etc
Comment 9 fujiwara 2009-07-14 22:55:43 EDT
> A new configuration file to force fontconfig to prefer DejaVu glyphs to
Japanese glyphs.

Since the Japanese font has the latin script, should it always use the Japanese glyphs instead of the DejaVu glyphs?
So I have thought the reverse way to fix this.

The following patch fixes this issue and I integrated it in the previous company.
http://src.opensolaris.org/source/xref/jds/spec-files/trunk/patches/pango-03-solaris-cjk-font-table.diff
Comment 10 Jens Petersen 2009-10-19 04:09:59 EDT
*** Bug 529594 has been marked as a duplicate of this bug. ***
Comment 11 Jens Petersen 2009-10-19 04:15:22 EDT
Some simple testcases from bug 529594:

LANG=zh_CN.UTF-8 pango-view --font="Monospace" <(echo -e "my@ip-1-2-3 ~$\n  @@
1 2 3")

LANG=ja_JP.UTF-8 pango-view --font="Monospace" <(echo -e "my@ip-1-2-3 ~$\n  @@
1 2 3")

LANG=ko_KR.UTF-8 pango-view --font="Monospace" <(echo -e "my@ip-1-2-3 ~$\n  @@
1 2 3")

also good without --font option.
Comment 12 Bug Zapper 2009-11-16 04:48:06 EST
This bug appears to have been reported against 'rawhide' during the Fedora 12 development cycle.
Changing version to '12'.

More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 13 Bug Zapper 2010-11-04 07:31:03 EDT
This message is a reminder that Fedora 12 is nearing its end of life.
Approximately 30 (thirty) days from now Fedora will stop maintaining
and issuing updates for Fedora 12.  It is Fedora's policy to close all
bug reports from releases that are no longer maintained.  At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '12'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 12's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 12 is end of life.  If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora please change the 'version' of this 
bug to the applicable version.  If you are unable to change the version, 
please add a comment here and someone will do it for you.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events.  Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 14 Akira TAGOH 2010-11-18 08:35:46 EST
This is still present in even f14.

I'm a bit thinking of a feature to get rid of the specific code point from the charset tables for the certain font cache like:

<match target="scan">
  <test name="family">
    <string>VL Gothic</string>
  </test>
  <edit name="charset" mode="remove">
    <range>
      <string>U+0021</string>
      <string>U+00FF</string>
    </range>
  </edit>
</match>
Comment 15 Akira TAGOH 2010-12-06 01:51:35 EST
proposed changes for this issue:
http://bugs.freedesktop.org/show_bug.cgi?id=31969

we also need to update the config file for vlgothic-fonts.

<match target="scan">
  <test name="family">
    <string>VL Gothic</string>
  </test>
  <edit name="charset" mode="assign">
    <minus>
      <name>charset</name>
      <range>
        <int>0x0021</int>
        <int>0x00FF</int>
      </range>
    </minus>
  </edit>
</match>
Comment 16 Akira TAGOH 2011-01-07 05:16:16 EST
Any chance to get a snapshot or new release of fontconfig for the above upstream bug?

I could remove ASCII characters from charset in vlgothic-fonts perhaps then.
Comment 17 Behdad Esfahbod 2011-01-07 20:28:20 EST
I'll try.
Comment 18 Akira TAGOH 2011-01-27 02:51:35 EST
That would be nice if we can get new version of fontconfig in f15 alpha or earlier. then we can work for some fixes in the font packages with it by alpha or beta.
Comment 19 Akira TAGOH 2011-11-21 03:06:19 EST
still waiting for new feature in fontconfig not yet released.
Comment 20 Akira TAGOH 2011-12-09 00:44:04 EST
*** Bug 765675 has been marked as a duplicate of this bug. ***
Comment 21 Fedora Admin XMLRPC Client 2012-01-10 10:42:39 EST
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.
Comment 22 Fedora Admin XMLRPC Client 2012-03-21 21:37:51 EDT
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.
Comment 23 Peng Wu 2012-10-24 00:06:17 EDT
Recently I also en-countered this bug in Chinese locale.

And I find a work around that if I customize the font to some English font in gedit preference dialog, the problems are gone.
For more details in http://pwu.fedorapeople.org/pango/README.

And I tried several tweaks in Chinese fontconfig confs to simulate the above changes.
See http://pwu.fedorapeople.org/pango/.

Then I spend a long time on investigating it, and finally find its reasons.
See http://pwu.fedorapeople.org/pango/REASONS.

How about we restrict this change only for monospace font.

PS:
The problem is shown in http://pwu.fedorapeople.org/pango/before%20change.png.
And the fixes is shown in http://pwu.fedorapeople.org/pango/after%20change.png.
Comment 24 Akira TAGOH 2012-10-24 00:28:00 EDT
Sure. that is actually same to what being proposed at comment#2 and disadvantages on this way is:

 * if applications doesn't use any rendering library which doesn't support the fallback mechanism, you'll miss Chinese characters on display.

 * it may still gives you different looks when the default Latin font will not be DejaVu anymore and need to fix it yourself.


just to clarify, my suggestion at comment#15 also has a disadvantage...

 * if applications doesn't use any rendering library which doesn't support the fallback mechanism, one will miss ASCII characters on display as it can be seen with Lohit now.
Comment 25 Peng Wu 2012-11-06 22:39:10 EST
As this bug can be work around by manually choose "DejaVu Sans Mono" font in gedit preference dialog, I wrote a small fontconfig file to simulate this change, see 32-dejavu.conf.

How about we add one fontconfig file to the default Latin font?

URL: http://pwu.fedorapeople.org/pango/32-dejavu.conf
Comment 26 Peng Wu 2012-11-11 22:49:32 EST
I also write a small script to allow user to disable this feature, if he meets some problems with this new fontconfig conf file.

See http://pwu.fedorapeople.org/pango/dejavuset
Comment 27 Fedora End Of Life 2013-04-03 14:23:27 EDT
This bug appears to have been reported against 'rawhide' during the Fedora 19 development cycle.
Changing version to '19'.

(As we did not run this process for some time, it could affect also pre-Fedora 19 development
cycle bugs. We are very sorry. It will help us with cleanup during Fedora 19 End Of Life. Thank you.)

More information and reason for this action is here:
https://fedoraproject.org/wiki/BugZappers/HouseKeeping/Fedora19
Comment 28 Fedora End Of Life 2015-01-09 11:12:29 EST
This message is a notice that Fedora 19 is now at end of life. Fedora 
has stopped maintaining and issuing updates for Fedora 19. It is 
Fedora's policy to close all bug reports from releases that are no 
longer maintained. Approximately 4 (four) weeks from now this bug will
be closed as EOL if it remains open with a Fedora 'version' of '19'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 19 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.