Bug 57818

Summary: cyrillic characters in ttfonts-ja are of extremely bad quality
Product: [Retired] Red Hat Linux Reporter: Alexei Podtelezhnikov <apodtele>
Component: ttfonts-jaAssignee: Akira TAGOH <tagoh>
Status: CLOSED RAWHIDE QA Contact: David Lawrence <dkl>
Severity: medium Docs Contact:
Priority: medium    
Version: 7.2CC: akkzilla, eng-asia-list, jshin, pbrown
Target Milestone: ---   
Target Release: ---   
Hardware: i386   
OS: Linux   
URL: www.rian.ru
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2002-03-25 08:14:59 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 54087    
Bug Blocks:    
Attachments:
Description Flags
check out the latest mandrake's version of ttmkfdir
none
Italics in mozilla with ttfonts-ja installed none

Description Alexei Podtelezhnikov 2001-12-24 20:39:52 UTC
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:0.9.4.1) Gecko/20011105

Description of problem:
ttfonts-ja, I think unlawfully, provides microsoft-cp1251 character set of
extremely bad quality. Some characters are missing. Spacing is too wide.
Fonts are too small. The problem is that this is the only font in Red Hat
that provides this encoding. Thus it gets chosen once you encounter a web
page which uses it. And this page gets rendered so poorly in Konqueror.

Uninstalling this font fixes the behaviour by forcing recoding of the
cp1251 page. I think the problem was fixed recently for latin alphabet in
ttfonts-ja. Why not fix it for cyrillic too? 

Please, please, please.... remove all but japanese encodings from this
font, or remove this awful dependency of ghostscript on this font.

Version-Release number of selected component (if applicable):
all ttfonts-ja versions up to 1.1-1 in rawhide.


How reproducible:
Always

Steps to Reproduce:
1. install ttfonts-ja-1.1-1 (or 1.0-7)
2. restart xfs
3. look at www.rian.ru or www.izvestia.ru using Konqueror

	

Actual Results:  Page looks bad... unreadable in fact

Expected Results:  Unistall ttfonts-ja and see how the pages are supposed
to look like

Additional info:

I am talking about unfortunately very common encoding for cyrillic.
ttfonts-ja intrusion into this business is unwelcome. XFree86 provides very
decent fonts, don't mess those up.

Comment 1 Akira TAGOH 2001-12-25 03:23:13 UTC
ttfonts-ja does NOT have some encoding without iso8859-1 and jisx0208*. a cause
of this problem is that xfs remake fonts.dir and fonts.scale. ttmkfdir also
generate the wrong entries.



Comment 2 Alexei Podtelezhnikov 2001-12-26 00:53:53 UTC
Yes, they do. They provide characters and encoding maps for Latin-1, -2, Greek, 
Cyrillic, Symbol, and others, according MS font tool. Freetype tools and xfs 
reveals them all. That's ttfonts-ja--provided fonts.dir, what misses them. This 
fonts are in fact ISO-10646-2 unicode fonts. Regretably, all but japanese 
characters are of extremely bad quality. Yes, even Latin-1 has bad spacings. 
The fonts should be stripped of those. Please leave only japanese support in 
them. *.ttf files contain a lot of informationm by the way. Look at my comment 
to bug 56309 to find out about some other issues in this fonts.

Okay. here I propose two different solutions to the problem:
1) Provide a good unicode TrueType font instead of ttfonts-ja. As for cyrillic 
subset in it use a different set, e.g., from here: 
http://www.kiarchive.ru/pub/misc/fonts/cyrillic/xwindows/ or here:
http://koi8.pp.ru/main.html

2) Never mix different encodings in one font - remove physically (not just from 
fonts.dir) all but japanese characters from this fonts.

Comment 3 Akira TAGOH 2001-12-26 10:41:40 UTC
I can't understand what you have said. ttfonts-ja is TrueType font for Japanese.
if you don't need Japanese and you also need other TrueType font, you should be
able to uninstall ttfonts-ja and you can install ttfonts. it seems that an
upstream author of kochi font is planning Cyrillic support, but current kochi
font is supported Japanese only(exactly the encodings which is used by most of
Japanese. basically it's iso8859-1 and jisx0208*).
Also kochi fonts are made by unicode. so that it's right that fonts.dir and
fonts.scale which are generated by ttmkfdir has iso8859-1 and iso10646-1.
However it's wrong that those files has other encodings. I think that it's bug.
ttmkfdir is generated only using the encoding which it itself has. for Japanese,
ttmkfdir should support JIS X entries.

Comment 4 Alexei Podtelezhnikov 2001-12-26 19:54:21 UTC
We have some misunderstanding here.
You believe that the ttf files contain only Basic Latin and Japanese characters.
I believe that they actually contain a lot more: Cyrillic, Greek, Symbols, etc.

Here is how you can actually see that:
1) install the font.
2) restart xfs and X
3) using xfontsel, chose one of the kochi's and set encoding (the last one) to
cp1251. Yes it's possible.
4) in the bottom line you'll see cyrillic characters that was found in this font.

Alternatively, examine the font on in Windows 2000 character map tool. Man,
you'll see a lot of new characters there.

Please, do not argue that it's an xfs fault that it found the characters there.
It did the right thing. My only complain actually is the quality and the use of
illegal encodings. You are right the sure way to fix the problem is uninstalling
ttfonts-ja. Too bad, cratcher put up a dependency og ghostscript on it.

I just checked ttfonts-ja-1.1-2. I'm glad to see that the licensing issue was
cleared out. I hope the other issues will be creared too.

Comment 5 Akira TAGOH 2002-01-07 04:55:23 UTC
I see. I talked upstream about this problem, he will removes invalid encoding
entries from table. so that this problem will be fixed next release.

Comment 6 Akira TAGOH 2002-01-07 05:08:43 UTC
Reassigned to ttfonts-ja again.
However other Japanese specific problem will appear, but I don't say anymore.

Comment 7 Alexei Podtelezhnikov 2002-01-08 18:48:50 UTC
I just checked http://www.on.cs.keio.ac.jp/~yasu/linux/fonts/ , the invalid 
encodings appear to be removed from table as of 20020108 release. Thanks, 
looking forward to seeing it in rawhide.

Comment 8 Akira TAGOH 2002-01-11 07:02:38 UTC
I tried again with new upstream fonts. However this problem is not still fixed.
I have looked at ttmkfdir's source code. it is checking the fonts with X's
encoding files.
Anyway I need to confirm him whether removes empty glyph or not.
BTW even if encodings.dir has for Japanese, it seems that CJK fonts has too much
the missing glyphs. so that current ttmkfdir can't generate fonts.dir correctly
for CJK.
I suggest we use ttmkfdir which changes to as see CodePages from OS/2 table. How
do you think that, Mike?


Comment 9 Alexei Podtelezhnikov 2002-01-16 02:46:38 UTC
Summary: VFlib2's ttindex is involved in ttf initialization problem

I tried with new upstream fonts too. They are a lot better. Please include 
them in your RPM. At least I could get them to work without interference with 
cyrillic script. Here is what I did:

- Installed ttfonts-ja-1.1-3           (this baby includes old *.ttf files)
- Replaced the old *.ttf files with the new ones from 20020108 release
- Removed all *.tti indeces  (this drastically improved xfs init times back to
                                          normal as without ttfonts-ja)
- Restart xfs and X

Boom! No more cp1251 available! Nothing prohibits me from viewing cyr-pages.
The strange part of it all is that recreation of tti's with ttindex reverts me 
to the bad behavior - cp1251 is back in fonts.dir. Damn it!

My hypothesis now is that it's all ttindex (from VFlib2) fault. I don't know 
if updating that might help. There is definately some strange interaction 
between freetype's mkttfdir and ttindex. Go figure.

Comment 10 Akira TAGOH 2002-01-16 07:56:49 UTC
a reason which I don't make new RPM yet is, ttmkfdir can't still generate the
entries for CJK. Also about cp1251 entries, if encodings.dir which exists on
/usr/share/fonts/ja/TrueType doesn't have cp1251 entry and so on, even if those
fonts is old stuff, cp1251 entries should be generated. no ttindex files is
related because ttmkfdir don't see ttindex files.


Comment 11 Akira TAGOH 2002-01-16 07:57:58 UTC
s/should be/should not be/

Comment 12 Alexei Podtelezhnikov 2002-01-16 21:09:52 UTC
You are right: ttindex and tti files are not to blame. It is ttmkfdir which is 
too smart. Here comes my log reproducing xfs init activity in the directory 
with kochi-mincho. Note how ttmkfdir fails on the second run only in the 
presence of encodings dir! First time it actually works better.

[user@localhost fonts]$ ls -l
total 5820
-rw-r--r--    1 user       users      5925220 Jan  7 17:32 kochi-mincho.ttf
-rw-r--r--    1 user       users        17318 Jan 16 12:06 kochi-mincho.tti
[user@localhost fonts]$ ttmkfdir .
13
kochi-mincho.ttf -misc-Kochi Mincho-medium-r-normal--0-0-0-0-p-0-ascii-0
kochi-mincho.ttf -misc-Kochi Mincho-medium-r-normal--0-0-0-0-p-0-fcd8859-15
kochi-mincho.ttf -misc-Kochi Mincho-medium-r-normal--0-0-0-0-p-0-iso8859-1
kochi-mincho.ttf -misc-Kochi Mincho-medium-r-normal--0-0-0-0-p-0-iso8859-10
kochi-mincho.ttf -misc-Kochi Mincho-medium-r-normal--0-0-0-0-p-0-iso8859-15
kochi-mincho.ttf -misc-Kochi Mincho-medium-r-normal--0-0-0-0-p-0-iso8859-2
kochi-mincho.ttf -misc-Kochi Mincho-medium-r-normal--0-0-0-0-p-0-iso8859-3
kochi-mincho.ttf -misc-Kochi Mincho-medium-r-normal--0-0-0-0-p-0-iso8859-4
kochi-mincho.ttf -misc-Kochi Mincho-medium-r-normal--0-0-0-0-p-0-iso8859-5
kochi-mincho.ttf -misc-Kochi Mincho-medium-r-normal--0-0-0-0-p-0-iso8859-7
kochi-mincho.ttf -misc-Kochi Mincho-medium-r-normal--0-0-0-0-p-0-iso8859-9
kochi-mincho.ttf -misc-Kochi Mincho-medium-r-normal--0-0-0-0-p-0-koi8-r
kochi-mincho.ttf -misc-Kochi Mincho-medium-r-normal--0-0-0-0-p-0-koi8-ru
[user@localhost fonts]$ ttmkfdir . >fonts.scale
[user@localhost fonts]$ mkfontdir -e /usr/X11R6/lib/X11/fonts/encodings -e 
/usr/X11R6/lib/X11/fonts/encodings/large .
[user@localhost fonts]$ ls -l
total 5832
-rw-r--r--    1 user       users         2877 Jan 16 12:45 encodings.dir
-rw-r--r--    1 user       users          974 Jan 16 12:45 fonts.dir
-rw-r--r--    1 user       users          974 Jan 16 12:45 fonts.scale
-rw-r--r--    1 user       users      5925220 Jan  7 17:32 kochi-mincho.ttf
-rw-r--r--    1 user       users        17318 Jan 16 12:06 kochi-mincho.tti
[user@localhost fonts]$ ttmkfdir .
19
kochi-mincho.ttf -misc-Kochi 
Mincho-medium-r-normal--0-0-0-0-p-0-adobe-standard kochi-mincho.ttf 
-misc-Kochi Mincho-medium-r-normal--0-0-0-0-p-0-ascii-0
kochi-mincho.ttf -misc-Kochi Mincho-medium-r-normal--0-0-0-0-p-0-fcd8859-15
kochi-mincho.ttf -misc-Kochi Mincho-medium-r-normal--0-0-0-0-p-0-ibm-cp437
kochi-mincho.ttf -misc-Kochi Mincho-medium-r-normal--0-0-0-0-p-0-ibm-cp850
kochi-mincho.ttf -misc-Kochi Mincho-medium-r-normal--0-0-0-0-p-0-ibm-cp852
kochi-mincho.ttf -misc-Kochi Mincho-medium-r-normal--0-0-0-0-p-0-ibm-cp866
kochi-mincho.ttf -misc-Kochi Mincho-medium-r-normal--0-0-0-0-p-0-iso8859-1
kochi-mincho.ttf -misc-Kochi Mincho-medium-r-normal--0-0-0-0-p-0-iso8859-10
kochi-mincho.ttf -misc-Kochi Mincho-medium-r-normal--0-0-0-0-p-0-iso8859-15
kochi-mincho.ttf -misc-Kochi Mincho-medium-r-normal--0-0-0-0-p-0-iso8859-2
kochi-mincho.ttf -misc-Kochi Mincho-medium-r-normal--0-0-0-0-p-0-iso8859-3
kochi-mincho.ttf -misc-Kochi Mincho-medium-r-normal--0-0-0-0-p-0-iso8859-4
kochi-mincho.ttf -misc-Kochi Mincho-medium-r-normal--0-0-0-0-p-0-iso8859-5
kochi-mincho.ttf -misc-Kochi Mincho-medium-r-normal--0-0-0-0-p-0-iso8859-7
kochi-mincho.ttf -misc-Kochi Mincho-medium-r-normal--0-0-0-0-p-0-iso8859-9
kochi-mincho.ttf -misc-Kochi Mincho-medium-r-normal--0-0-0-0-p-0-koi8-r
kochi-mincho.ttf -misc-Kochi Mincho-medium-r-normal--0-0-0-0-p-0-koi8-ru
kochi-mincho.ttf -misc-Kochi 
Mincho-medium-r-normal--0-0-0-0-p-0-microsoft-cp1251
[user@localhost fonts]$  ttmkfdir . >fonts.scale
[user@localhost fonts]$ mkfontdir -e /usr/X11R6/lib/X11/fonts/encodings -e 
/usr/X11R6/lib/X11/fonts/encodings/large .
[user@localhost fonts]$ ls -l
total 5832
-rw-r--r--    1 user       users         2877 Jan 16 12:48 encodings.dir
-rw-r--r--    1 user       users         1436 Jan 16 12:48 fonts.dir
-rw-r--r--    1 user       users         1436 Jan 16 12:47 fonts.scale
-rw-r--r--    1 user       users      5925220 Jan  7 17:32 kochi-mincho.ttf
-rw-r--r--    1 user       users        17318 Jan 16 12:06 kochi-mincho.tti
[user@localhost fonts]$ ttmkfdir .
19
kochi-mincho.ttf -misc-Kochi Mincho-medium-r-normal--0-0-0-0-p-0-adobe-standard
kochi-mincho.ttf -misc-Kochi Mincho-medium-r-normal--0-0-0-0-p-0-ascii-0
kochi-mincho.ttf -misc-Kochi Mincho-medium-r-normal--0-0-0-0-p-0-fcd8859-15
kochi-mincho.ttf -misc-Kochi Mincho-medium-r-normal--0-0-0-0-p-0-ibm-cp437
kochi-mincho.ttf -misc-Kochi Mincho-medium-r-normal--0-0-0-0-p-0-ibm-cp850
kochi-mincho.ttf -misc-Kochi Mincho-medium-r-normal--0-0-0-0-p-0-ibm-cp852
kochi-mincho.ttf -misc-Kochi Mincho-medium-r-normal--0-0-0-0-p-0-ibm-cp866
kochi-mincho.ttf -misc-Kochi Mincho-medium-r-normal--0-0-0-0-p-0-iso8859-1
kochi-mincho.ttf -misc-Kochi Mincho-medium-r-normal--0-0-0-0-p-0-iso8859-10
kochi-mincho.ttf -misc-Kochi Mincho-medium-r-normal--0-0-0-0-p-0-iso8859-15
kochi-mincho.ttf -misc-Kochi Mincho-medium-r-normal--0-0-0-0-p-0-iso8859-2
kochi-mincho.ttf -misc-Kochi Mincho-medium-r-normal--0-0-0-0-p-0-iso8859-3
kochi-mincho.ttf -misc-Kochi Mincho-medium-r-normal--0-0-0-0-p-0-iso8859-4
kochi-mincho.ttf -misc-Kochi Mincho-medium-r-normal--0-0-0-0-p-0-iso8859-5
kochi-mincho.ttf -misc-Kochi Mincho-medium-r-normal--0-0-0-0-p-0-iso8859-7
kochi-mincho.ttf -misc-Kochi Mincho-medium-r-normal--0-0-0-0-p-0-iso8859-9
kochi-mincho.ttf -misc-Kochi Mincho-medium-r-normal--0-0-0-0-p-0-koi8-r
kochi-mincho.ttf -misc-Kochi Mincho-medium-r-normal--0-0-0-0-p-0-koi8-ru
kochi-mincho.ttf -misc-Kochi 
Mincho-medium-r-normal--0-0-0-0-p-0-microsoft-cp1251

I checked other tt fonts. only kochi-mincho and kochi-gothic display this 
behavior. I guess I'll hit Mike with a bug report.

Comment 13 Alexei Podtelezhnikov 2002-01-17 03:46:23 UTC
Created attachment 42654 [details]
check out the latest mandrake's version of ttmkfdir

Comment 14 Akkana 2002-01-19 19:38:07 UTC
Created attachment 42914 [details]
Italics in mozilla with ttfonts-ja installed

Comment 15 Akira TAGOH 2002-01-24 06:29:09 UTC
kochi fonts has no italic and bold font. AFAIK when you use xtt module only, you
can do that.

Comment 16 Akira TAGOH 2002-03-11 14:38:22 UTC
the latest version of ttfonts-ja no longer contains fonts.dir and fonts.scale.
in addition it's explicit that this problem is related ttmkfdir.

Comment 17 Jungshik Shin 2002-03-24 09:59:31 UTC
Japanese, Chinese and Korean truetype fonts contain
glyphs for a subset of Cyrillic and Greek alphabets
because JIS X 0208, GB 2312-1980, Big5 (or CNS) and
KS X 1001:1997 have them in their character repertoire.
If you can't believe, just pick any X11 bdf font
for any of character sets above and examine it
with 'xfd'. 

> 2) Never mix different encodings in one font - remove physically (not just 
> from 
> fonts.dir) all but japanese characters from this fonts.

  For the reason I gave above, this cannot be done. 
If you remove Cyrillic and Greek letters from them, it doesn't
have the full repertoire of JIS X 0208 any more. 



Their quality may be bad, but it could be pretty good
depending on fonts. A much more serious problem is 
that their metric/width/spacing is not suitable 
for Russian and Greek text rendering. When used as a
fixed-width font, glyphs for Russian and Greek letters 
are twice as wide as the 'normal width' expected by 
Russian and Greek users. That is because in EastAsian
terminal environment, the number of octets taken by
characters (1 for characters in US-ASCII and 2
for characters in local character sets such as 
JIS X 0208, GB 2312-1980, KS C 5601-1987) is proportional
to the width of characters. That is, the width
of US-ASCII character is twice as narrow as 
that of characters in JIS X 0208, GB 2312-1980,
and KS C 5601-1987. Obviously
Cyrillic and Greek alphabets are not a part of
US-ASCII BUT a part of JIS X 0208, GB 2312,
KS C 5601 and Big5.

A workaround may be to filter out the output of
ttmkfdir for ttfonts-{ja,ko,cn,tw} to keep
only entry for locale-specific character sets,
iso10646-1 and us-ascii. This filtering can be
done by initscript, I believe. 

As for iso10646-1, this presents a different kind of
problem. Cyrillic and Greek letters in CJK ttf 
have a 'wrong' width (and possibly bad quality). If
applications like Mozilla hit upon CJK ttf fonts
in iso10646-1 before hitting upon European ttfs,
they'd be happily use them making Russian and
Greek users frustrated. (at the moment, for this
very reason, Mozilla does not use fonts specified
for Unicode in Preference|Appearence|Font).
 
One possible solution is to fix ttmkfdir to
put lang-code in 'add-style' field of XLFD (as shown
below)
and expect applications and KDE/Gnome to make
use of lang-code when selecting fonts and searching
for glyphs. It can be made that
for a certain Unicode code range
(e.g. < U+1100 ), any font with add-style field
'ja', 'ko', 'zh_tw', 'zh_cn' be avoided.   

-baekmuk-batang-medium-r-normal-ko-0-0-0-0-c-0-iso10646-1
    
It has to be noted that using 'add-style' field of XLFD
for lang-code is not my invention but has been already
established by Markus Kuhn and his iso10646-1 fonts(bdf) 
included in XFree86 4.0 and above. 
  
In the long run, moving to Xft and leaving X11 core font
behind will solve this problem.

Comment 18 Akira TAGOH 2002-08-21 14:57:42 UTC
I've changed fonts.scale in 1.2-15 and mkfontdir is called on %%post now because
of avoiding generating the wrong fonts.dir. Unless you change something on
/usr/share/fonts/ja/TrueType, ttfonts-ja doesn't provide any encoding except
Japanese encoding(I mean iso8859-1 and JISX...)

Comment 19 Jungshik Shin 2002-08-21 17:59:44 UTC
> I've changed fonts.scale in 1.2-15 and mkfontdir is called on %%post now because
> of avoiding generating the wrong fonts.dir. Unless you change something on
> /usr/share/fonts/ja/TrueType, ttfonts-ja doesn't provide any encoding except
> Japanese encoding(I mean iso8859-1 and JISX...)

  How about iso10646-1? fonts.scale for ttfonts-ja should have iso10646-1
as well. It may be also a good idea to fill 'add-style' field with
'ja' for iso10646-1 entries. 
 


Comment 20 Akira TAGOH 2002-08-22 13:09:49 UTC
Hmm, almost applications are set 'add-style' as null and requesting such XLFD.
if add 'ja' to it, those applications won't find the kochi font. we don't have
*full* iso10646 font right now. so I don't understand why only CJK fonts should
has 'add-style'.
Anyway the original purpose of this bug should be fixed and it's not iso10646-1
issue. if you still think 'add-style' issue, please submit it as an another bug.