Bug 891457

Summary: uneven emulated Bold/Slant for subsetted large fonts
Product: [Fedora] Fedora Reporter: Hin-Tak Leung <htl10>
Component: freetypeAssignee: Marek Kašík <mkasik>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 18CC: ajohnson, behdad, fonts-bugs, i18n-bugs, kevin, mkasik, otte, pnemade, tagoh
Target Milestone: ---   
Target Release: ---   
Hardware: i686   
OS: Linux   
Whiteboard:
Fixed In Version: freetype-2.4.10-4.fc18 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-01-15 01:30:42 EST Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
Attachments:
Description Flags
the test.Rnw file mentioned
none
find /etc/fonts -ls listing
none
"find /usr/share/fontconfig -ls" listing
none
test-002.pdf generated by a clean account
none
test-002.pdf from an account which has a font.conf which avoid the Droid fonts.
none
the code to reproduce
none
PDF output
none
Updated test case to generate PDF, PS, PNG output
none
broken font
none
force fallback fonts
none
good font
none
from the small test program in comment 10, 32-bit, freetype 2.4.11
none
from the small test program in comment 10, 64-bit, freetype 2.4.11
none
part of a fix
none
back ported fix to 2.4.11
none
screenshot of broken gnome terminal, vs correct one.
none
two broken ftview - just different requested font size none

Description Hin-Tak Leung 2013-01-02 20:06:06 EST
Created attachment 671807 [details]
the test.Rnw file mentioned

Description of problem:
R has a way of generating pdf plots with Chinese annotations via cairo. The plot titles are in sans:style=Bold, I believe. In F17 WenQuanYi Zen Hei is subsetted and embedded as truetype. But in F18, an unknown bitmap is embedded as CairoFont-?-? .

Version-Release number of selected component (if applicable):
fontconfig-2.10.2-1.fc18.x86_64
cairo-1.12.8-1.fc18.x86_64
freetype-2.4.10-2.fc18.x86_64

How reproducible:
Always.

Steps to Reproduce:
1. run "R --vanilla CMD Sweave test.Rnw" (test.Rnw will be attached)
2. look at the test-002.pdf generated.
3.
  
Actual results:
Uneven boldening title

Expected results:
even boldening of the title

Additional info:
You need to run R under R to do this ; xvfb-run could be used for headless system.
Comment 1 Hin-Tak Leung 2013-01-02 20:09:02 EST
Created attachment 671809 [details]
find /etc/fonts -ls listing

find /etc/fonts -ls

listing  - I have carefully done

find /usr/share/fontconfig -type f | xargs rpm -qf | sort | uniq | xargs rpm -V

and removed all local customizations
Comment 2 Hin-Tak Leung 2013-01-02 20:11:05 EST
Created attachment 671810 [details]
"find /usr/share/fontconfig -ls" listing

"find /usr/share/fontconfig -ls" listing

I think this should show all the fonts I have installed (the last listing shows which are active).
Comment 3 Hin-Tak Leung 2013-01-02 20:13:35 EST
Created attachment 671811 [details]
test-002.pdf generated by a clean account

test-002.pdf generated by a clean account, and stock configs.

generated with

xvfb-run R CMD Sweave test.Rnw

from an account which does not have its own .fonts.conf . You can see two or three of the glyphs are bolden more than others.
Comment 4 Hin-Tak Leung 2013-01-02 20:17:15 EST
Created attachment 671812 [details]
test-002.pdf from an account which has a font.conf which avoid the Droid fonts.

test-002.pdf from an account which has a font.conf which avoid the Droid fonts.

this is how I first noticed it - it is more serious if you have a font.conf which avoid the droid fonts:

-----------
<selectfont>
     <rejectfont>
            <pattern>
                 <patelt name="family"><string>Droid Sans</string></patelt>
            </pattern>
     </rejectfont>
     <rejectfont>
            <pattern>
                 <patelt name="family"><string>Droid Serif</string></patelt>
            </pattern>
     </rejectfont>
     <rejectfont>
            <pattern>
                 <patelt name="family"><string>Droid Sans Mono</string></patelt>
            </pattern>
     </rejectfont>
</selectfont>
-----------
Comment 5 Hin-Tak Leung 2013-01-02 20:25:09 EST
I have already tried a few things:

- setting FC_DEBUG (cannot make sense of the outcoming info - some help instruction would be appreciated),

- strace -f ("R CMD SWeave" do fork - in any case, only the sans ascii fonts and [Droid or WQY] are opened).

but none produces acceptable result.

please feel free to assign it as cairo or freetype problem as appropriate. Since removing .fonts.conf (and switching between Droid and WQY) produces bad results from a clean account with headless xvfb-run, I assume this is a system-wide problem not due to customization; I suspect the problem is probably somewhere between cairo and freetype.

Sorry I don't have a simplier non-R way of showing the same problem - pango-view seems to do okay/correct.

Also the CairoFont-?-? font names were never seen under F17.
Comment 6 Akira TAGOH 2013-01-04 01:08:44 EST
It seems working fine on f18 TC4. please try again.
Comment 7 Akira TAGOH 2013-01-04 01:11:03 EST
and it works fine on f17 + new fontconfig. I don't think it is a big in fontconfig anyway.
Comment 8 Hin-Tak Leung 2013-01-04 19:15:12 EST
(In reply to comment #6)
> It seems working fine on f18 TC4. please try again.

(In reply to comment #7)
> and it works fine on f17 + new fontconfig. I don't think it is a big in
> fontconfig anyway.

Argh, I forgot to mention that I have always been running 32-bit R on a 64-bit system (there is an advantage on memory footprint, and there used to be a substantial speed advantage but I am told it isn't anymore). This seems to be an important detail to see this bug - if I run the 64-bit R, then the problem disappears.

Is it possible to see if you can reproduce this on a 32-bit F18 system?

In any case, this seems to apply only to 32-bit R (on a 64-bit system). Also, now that I can get the desired outcome with 64-bit R, I can see that the "CairoFont" font name issue is a red-herring (it just seems that newer cairo behaves differently), as both the correct and icorrect pdf have that as the emboldened font name, but that the "correct" pdf and the "incorrect" pdf differ in the glyph stream data of that font; the other two fonts have the same glyph stream data and only differ slightly in the subset prefix and rounding in fontbox/ascent, etc.

If you can reproduce this on 32-bit F18 system, probably should reassign this to cairo/freetype or somebody who knows about glyph-level font data.
Comment 9 Akira TAGOH 2013-01-08 01:12:38 EST
Indeed I can confirm this issue on i686 box and even with C code and it works on x86_64. so reassigning to cairo.
Comment 10 Akira TAGOH 2013-01-08 01:13:51 EST
Created attachment 674513 [details]
the code to reproduce
Comment 11 Hin-Tak Leung 2013-01-08 14:35:03 EST
(In reply to comment #10)
> Created attachment 674513 [details]
> the code to reproduce

Just a couple of comments - the last working cairo was the one in F17, cairo-1.10.2-7.fc17 ; also the problem is independent of the font used - "WenQuanYi Zen Hei" or "Droid Sans" also shows problem.
Comment 12 Hin-Tak Leung 2013-01-12 21:28:03 EST
Correcting the title after git-bisect. It is not a regression, but a new feature that does not work for some unknown reason on 32-bit cairo.
Comment 13 Hin-Tak Leung 2013-01-12 21:57:19 EST
git bisect'ed, and found the first appearance of the "uneven" look is this:

---------
From: Adrian Johnson <ajohnson@redneon.com>
Date: Mon, 22 Nov 2010 22:46:54 +1030
Subject: [PATCH] Use fallback font for synthetic fonts

If the font has been synthesized we can't use the native subsetters as
the outlines won't be the same. Instead force the use of the fallback
subsetters so the synthesized outlines will used to generate the font.
---------
(this was branched off 1.10.0 and leading onto 1.11.2 then onto 1.12.x).

I also realized that the initial description of the problem was wrong: the problem is that new feature of emulated bold (and slant) for large fonts somehow does not work correctly on 32-bit cairo; before that commit, request for bold style for a typeface which does not provide that style was silently dropped, and substituted with the regular style. With that commit, emulated bold is provided by freetype but somehow only works for 64-bit platforms.

The smallest change (which basically partially revert the above commit) is this: 

--- a/src/cairo-truetype-subset.c
+++ b/src/cairo-truetype-subset.c
@@ -159,6 +159,6 @@ _cairo_truetype_font_create (cairo_scaled_font_subset_t  *scaled_font_subset,
      *   return CAIRO_INT_STATUS_UNSUPPORTED;
      */
 
     /* We need to use a fallback font generated from the synthesized outlines. */
-    if (backend->is_synthetic && backend->is_synthetic (scaled_font_subset->scaled_font))
+    if (0 && (backend->is_synthetic && backend->is_synthetic (scaled_font_subset->scaled_font)))
        return CAIRO_INT_STATUS_UNSUPPORTED;


The same effect can be achieved by:

--- a/src/cairo-ft-font.c
+++ b/src/cairo-ft-font.c
@@ -2172,7 +2172,7 @@ _cairo_ft_scaled_glyph_init (void                 *abstract_font,
      * synthesize glyphs if requested
      */
 #if HAVE_FT_GLYPHSLOT_EMBOLDEN
-    if (scaled_font->ft_options.synth_flags & CAIRO_FT_SYNTHESIZE_BOLD)
+    if (0 && (scaled_font->ft_options.synth_flags & CAIRO_FT_SYNTHESIZE_BOLD))
        FT_GlyphSlot_Embolden (glyph);
 #endif
 
Anyway, it almost appears as if FT_GlyphSlot_Embolden() does not work correctly for 32-bit platform, which is wierd, since freetype is much more well-tested on 32-bit platforms than 64-bit; so my current thought is that something leading up to FT_GlyphSlot_Embolden (glyph) or some post-processing after is wrong.
Comment 14 Adrian Johnson 2013-01-13 02:48:38 EST
Created attachment 677638 [details]
PDF output

This is the PDF output I get from the test case. The 6th character is not bold. In fact it is thinner than the output I get when I set the font weight to normal. This is run on Debian Testing 64-bit using cairo master. I get the same output on Debian Testing 32-bit.

The CairoFont-?-? names are because the font was created using the FreeType outlines instead of subsetting the original font.
Comment 15 Adrian Johnson 2013-01-13 02:54:54 EST
Created attachment 677639 [details]
Updated test case to generate PDF, PS, PNG output

This is an updated test case that prints just the single character that is not bold. It generates output in PDF, PS, and PNG.

I compiled with:

gcc -Wall test.c -o test `pkg-config --cflags --libs cairo`

The PNG output is correct. The PDF and PS is not bold.
Comment 16 Adrian Johnson 2013-01-13 02:58:33 EST
Created attachment 677640 [details]
broken font

I copied the font from the PS output into a separate the file font-broken.ps. This can be opened in fontforge.
Comment 17 Adrian Johnson 2013-01-13 03:02:58 EST
Created attachment 677641 [details]
force fallback fonts

Applying this patch to cairo will force the use of fallback fonts (fonts created from the FreeType outlines instead of using the original font with the unused glyphs stripped out).

I used this patch and ran the test case with the font weight set to CAIRO_FONT_WEIGHT_NORMAL. This allowed me to extract the font from the PS output that was generated using the same code as the font in comment 16.
Comment 18 Adrian Johnson 2013-01-13 03:15:09 EST
Created attachment 677643 [details]
good font

This is the font extracted from the PS file generated in comment 17.

You can open both the good and broken font in fontforge and view the characters side by side. The "bold" character has thinner strokes. The horizontal strokes have almost zero width.

It is hard to see how cairo could cause this sort of output. It looks more like a FreeType bug. I'm not sure how to debug this any further without writing some FreeType test code to check the output from FreeType. And that is something I don't have time to do.

Note: the png output from the testcase is using the glyph bitmaps generated by FreeType while the PDF/PS output is using the outlines from FreeType.
Comment 19 Hin-Tak Leung 2013-01-13 03:27:45 EST
    (In reply to comment #14)
    >... This is run on Debian Testing 64-bit using cairo master. I get
    > the same output on Debian Testing 32-bit.

    plot thickens... so 64-bit cairo also suffers occasionally.

    > The CairoFont-?-? names are because the font was created using the FreeType
    > outlines instead of subsetting the original font.

    Yes, I see from grep'ing through the code - it was the first difference I noticed from the upgrade... related but somewhat a red-herring.

    (In reply to comment #15)
    >...
    > The PNG output is correct. The PDF and PS is not bold.

    R's X11 display is also somewhat cairo-based - and the display is correct; I think it doesn't affect bitmap devices.

    (In reply to comment #16)
    ...
    > broken font

    One possibility crossed my mind is the font-writing part of cairo (and bad-interaction with freetype emulated fonts). Hence I wrote to Werner (of freetype) of both correctly embolden output & wrong ouput and asked if he could tell how and in what way the glyph data is corrected. fontforge could work now that we know the PS output is also broken, and a "reproducible" way of breaking it...
Comment 20 Hin-Tak Leung 2013-01-13 03:33:47 EST
(In reply to comment #18)
...
> It is hard to see how cairo could cause this sort of output. It looks more
> like a FreeType bug. I'm not sure how to debug this any further without
> writing some FreeType test code to check the output from FreeType. And that
> is something I don't have time to do.

Werner suggested trying to reproduce similar problems if possible with ftview, which can do font effects on display. But I am not hopeful in that since it looks like the problem does not affect bitmap-based output.

> Note: the png output from the testcase is using the glyph bitmaps generated
> by FreeType while the PDF/PS output is using the outlines from FreeType.

I haven't figured out where/how cairo write fonts yet...
Comment 21 Adrian Johnson 2013-01-13 04:20:21 EST
(In reply to comment #20)
> Werner suggested trying to reproduce similar problems if possible with
> ftview, which can do font effects on display. But I am not hopeful in that
> since it looks like the problem does not affect bitmap-based output.

I tried ftview but it didn't seem to provide any way to view outlines.

I had another idea. Replace the cairo_show_text with cairo_text_path followed by cairo_fill. This will create paths from the text then fill the paths. This results in correct output.

> I haven't figured out where/how cairo write fonts yet...

It has a number of subsetters as well as the ability to create CFF or type 1 fonts. The easiest to understand is cairo-type1-fallback.c which generates fallback fonts for PostScript.

Since cairo_text_path produced the correct output the next idea I had was to try changing the size of the font outlines that cairo-type1-fallback requests. At line 105 the matrix size is set to 1000 since Type 1 fonts have an EM grid size of 1000. I changed this to 500 and the output was smaller but correct. I tried some different values. 800 works. 900 fails. So somewhere between a font size of 800 and 900 FreeType starts producing incorrect output for some glyphs with synthetic bold enabled.
Comment 22 Hin-Tak Leung 2013-01-13 20:11:00 EST
BTW, while looking at the cairo code surrounding FT_GlyphSlot_Embolden(), I noted that the types of at least one the parameters passed to FT_* is wrong. load_flags is 'unsigned int' on the freetype side but occasionally just int on the cairo side. That may be harmless but might just be important when doing signed-extended operations like "~ mask" or "<<".

freetype is much more well-tested for 32-bit (and embedded/16-bit(?) environments), and in fact some of the internals are written to do fixed-precision maths with integer operations, assuming floating point maths are too expensive or not available. So it is curious the problem is more serious on 32-bit (and isn't as noticeable on 64-bit platforms).
Comment 23 Hin-Tak Leung 2013-01-14 00:10:44 EST
(In reply to comment #21)
> (In reply to comment #20)
...
> Since cairo_text_path produced the correct output the next idea I had was to
> try changing the size of the font outlines that cairo-type1-fallback
> requests. At line 105 the matrix size is set to 1000 since Type 1 fonts have
> an EM grid size of 1000. I changed this to 500 and the output was smaller
> but correct. I tried some different values. 800 works. 900 fails. So
> somewhere between a font size of 800 and 900 FreeType starts producing
> incorrect output for some glyphs with synthetic bold enabled.

That looks like a magic number that can be changed everywhere, for cleanliness, at least:

$ grep -n 1000 src/*fallback*
src/cairo-type1-fallback.c:105:    cairo_matrix_init_scale (&font_matrix, 1000, -1000);
src/cairo-type1-fallback.c:752:	type1_subset->widths[i] = (double)font->widths[i]/1000;
src/cairo-type1-fallback.c:754:    type1_subset->x_min   = (double)font->x_min/1000;
src/cairo-type1-fallback.c:755:    type1_subset->y_min   = (double)font->y_min/1000;
src/cairo-type1-fallback.c:756:    type1_subset->x_max   = (double)font->x_max/1000;
src/cairo-type1-fallback.c:757:    type1_subset->y_max   = (double)font->y_max/1000;
src/cairo-type1-fallback.c:758:    type1_subset->ascent  = (double)font->y_max/1000;
src/cairo-type1-fallback.c:759:    type1_subset->descent = (double)font->y_min/1000;

I Haven't tried, but I think it should work by replacing every "1000" with "800" - since the code basically scale glyph co-ordinates up to x1000 then round to integer. That does not answer why 1000 fails but 800 works though.
Comment 24 Adrian Johnson 2013-01-14 03:55:37 EST
(In reply to comment #23)
> I Haven't tried, but I think it should work by replacing every "1000" with
> "800" - since the code basically scale glyph co-ordinates up to x1000 then
> round to integer. That does not answer why 1000 fails but 800 works though.

I compiled the latest freetype (2.4.11) and the test case now works correctly.

There seems to have been a few fixes to the emboldening code since 2.4.10:
http://git.savannah.gnu.org/cgit/freetype/freetype2.git/log/src/base/ftoutln.c
Comment 25 Hin-Tak Leung 2013-01-14 04:03:25 EST
(In reply to comment #24)
> (In reply to comment #23)
> > I Haven't tried, but I think it should work by replacing every "1000" with
> > "800" - since the code basically scale glyph co-ordinates up to x1000 then
> > round to integer. That does not answer why 1000 fails but 800 works though.
> 
> I compiled the latest freetype (2.4.11) and the test case now works
> correctly.
> 
> There seems to have been a few fixes to the emboldening code since 2.4.10:
> http://git.savannah.gnu.org/cgit/freetype/freetype2.git/log/src/base/ftoutln.
> c

upgrading to freetype 2.4.11 was the first thing I tried as well - it got worse (some of the glyph has gone "reverse video"!). I'll upload that to show you.

The 64-bit version continue to work well, strangely.
Comment 26 Hin-Tak Leung 2013-01-14 04:05:11 EST
Created attachment 678118 [details]
from the small test program in comment 10, 32-bit, freetype 2.4.11

from the small test program in comment 10, 32-bit, freetype 2.4.11

It has gotten worse - two of the glyphs now looks "reversed".
Comment 27 Hin-Tak Leung 2013-01-14 04:06:38 EST
Created attachment 678121 [details]
from the small test program in comment 10, 64-bit, freetype 2.4.11

from the small test program in comment 10, 64-bit, freetype 2.4.11,

just to make sure that 64-bit continues to work correctly...
Comment 28 Adrian Johnson 2013-01-14 04:07:53 EST
Looks like it is definitely a freetype problem. I suggest filing a freetype bug.
Comment 29 Hin-Tak Leung 2013-01-14 04:20:05 EST
(In reply to comment #24)
...
> I compiled the latest freetype (2.4.11) and the test case now works
> correctly.
...

It has gone worse for me (see comment 26) - it is probably some subtle issue that depends on compiler options, etc.

FWIW, I grab the binary first:
http://koji.fedoraproject.org/koji/buildinfo?buildID=375798
then do an rpmbuild --rebuild (with or without --target=i686).

You don't have to install them but it would be interesting say, if you do "rpm2cpio | cpio --extract -m -d ", then do LD_LIBRARY_PATH, if you can get a broken pdf :-).

Is there a debian binary of 32-bit freetype 2.4.11 I can download from a public place?

(In reply to comment #28)
> Looks like it is definitely a freetype problem. I suggest filing a freetype
> bug.

I would probably say so as well, except (1) so far we haven't found a way of doing it without the higher-level libraries of cairo, fontconfig, (I thought about trying the original R sample under windows R; that would possibly be without fontconfig or freetype or both)... (2) freetype is much more well-tested on 32-bit than 64-bit - in fact Werner mentioned that he does not have a 64-bit machine!
Comment 30 Hin-Tak Leung 2013-01-14 07:17:33 EST
(In reply to comment #28)
> Looks like it is definitely a freetype problem. I suggest filing a freetype
> bug.

I tried LD_LIBRARY_PATH both
http://ftp.uk.debian.org/debian/pool/main/f/freetype/libfreetype6_2.4.9-1.1_i386.deb
http://ftp.uk.debian.org/debian/pool/main/f/freetype/libfreetype6_2.4.9-1.1_amd64.deb
then
http://koji.fedoraproject.org/koji/buildinfo?buildID=310621

and indeed I got the 6th glyph and only the 6th glypth wrong in all 4 versions.
then I just did

git checkout VER-2-4-11
./autogen.sh
CFLAGS=-m32 LDFLAGS=-m32 ./configure

and LD_LIB... still got the partially reversed-video look like I had with rpm-driven built 32-bit freetype.

So we agree perfectly with freetype 2.4.9 32-bit or 64-bit outcome, but not 2.4.11 32-bit outcome. How did you build yours?

This does not rule out cairo passing indetermined parameters to freetype (like the signed/unsigned load_flags which are ~mask'ed, I mentioned earlier) though.
Comment 31 Adrian Johnson 2013-01-14 07:29:05 EST
(In reply to comment #30)
> So we agree perfectly with freetype 2.4.9 32-bit or 64-bit outcome, but not
> 2.4.11 32-bit outcome. How did you build yours?

I only tested with 64-bit. I don't have my 32-bit machine here right now and gcc -m32 doesn't seem to work on Debian.

> This does not rule out cairo passing indetermined parameters to freetype
> (like the signed/unsigned load_flags which are ~mask'ed, I mentioned
> earlier) though.

Does it make any difference if you change the signed to unsigned and run the testcase?
Comment 32 Hin-Tak Leung 2013-01-14 07:48:38 EST
(In reply to comment #31)
> (In reply to comment #30)
> > So we agree perfectly with freetype 2.4.9 32-bit or 64-bit outcome, but not
> > 2.4.11 32-bit outcome. How did you build yours?
> 
> I only tested with 64-bit. I don't have my 32-bit machine here right now and
> gcc -m32 doesn't seem to work on Debian.

Oh, but I never did have a problem with 64-bit of everything (freetype 2.4.10/2.4.11) - and only with 32-bit of 2.4.10 where it was first seen, and a different reversed-video look with 32-bit 2.4.11. So for me, it has always been *only* 32-bit being broken, until I tried the earlier 64-bit 2.4.9 .

(fedora 18 ships 2.4.10, the 2.4.9 url is in the compile-farm alpha/beta-build archive, koji, from 9 months ago).

> > This does not rule out cairo passing indetermined parameters to freetype
> > (like the signed/unsigned load_flags which are ~mask'ed, I mentioned
> > earlier) though.
> 
> Does it make any difference if you change the signed to unsigned and run the
> testcase?

It did not - but that's just one mistake I noticed. There are others places where cairo's internals does not work in the same philosophy as freetype (I am not saying which is right/wrong - just different). freetype is usually very clear about what size of paramaters are, and *no bigger*, because at places it does fixed-precision maths without floating-point operations (all those FIXED..  macros). e.g. I see places in cairo - in the bits where it interacts with freetype, such as the cairo-type1-fallback.c, where it uses 'unsigned long' types... now, 'long' is 32-bit on 32-bit platforms but 64-bit on 64-bit platforms...

So we are still back to square one, 32-bit run path does not work correctly.
Comment 33 Hin-Tak Leung 2013-01-14 13:29:43 EST
Okay, I think I have got a handle on it... reassigning to freetype & I'll take it up with Werner... possibly will result in a new freetype version.

What happened is this: newer cairo started to use a feature in freetype which is still being changed (rather a lot!). older cairo just doesn't do emboldening so isn't afffected by freetype's changes; 5 commits in reverse-order from a freetype developer who isn't Werner (and therefore my faith about 32-bit correctness doesn't apply):

commit 0690d3d7b59ccb54a9938ff99c8621350780fc6e
commit dd5718c7d67a5b6d7dfc2546c9bda734b246142e
commit 48ce226ae3c2ccbe3230417d118ada4fea8508e3
commit 2bdd0949762acea68b4335f7c73ae14c5a34a2cc
commit f875fc71179b1e742d6f782b8c99b0cc031c9467

The 4th make 64-bit freetype work well but simultaneously make 32-bit freetype dramatically poor (this happened after 2.4.9). The first 3 have little or no effect on 64-bit build, but introduces various overflow problems. commit 1 "Fix integer overflows in dd5718c7d67a" in commit 2, but does not fully, so the two of them introduced the reverse-video look I see from 2.4.10 to 2.4.11.
Comment 34 Hin-Tak Leung 2013-01-20 09:27:56 EST
Created attachment 683564 [details]
part of a fix

Here is a patch from Alexei Podtelezhnikov (the author of the 4 commits in the last comment) which fixes some of the problems. There will be another one.
Comment 35 Hin-Tak Leung 2013-01-28 23:17:26 EST
Created attachment 689537 [details]
back ported fix to 2.4.11

backported full fix to 2.4.11 (obsoleting the last attachment).

Deal with overflows and underflows.

The equivalent were committed to freetype head as

commit 610ee58e07090ead529849b2a454bb6c503b4995
    [base] Fix broken emboldening at small sizes.
commit da11e5e7647b668dee46fd0418ea5ecbc33ae3b2
    [base] Fix integer overflow. 
commit e1a2ac1900f2f16ec48fb4840a6b7965a8373c2b
    [base] Fix integer overflow.

and part of 
commit 869fb8c49ddf292d6daf4826172a308973d3e11f
    [base] Split out MSB function.

Thanks Alexei and Werner (and me for part of da11e5e7647b668dee46fd0418ea5ecbc33ae3b2) for providing the fixes.
Comment 36 Hin-Tak Leung 2013-01-28 23:20:55 EST
Please close when either 2.4.12 or a patched 2.4.11 with the attachement in the last comment is released.
Comment 37 Kevin Kofler 2013-01-29 12:34:32 EST
Can we please have this fix backported to the Fedora 18 freetype?
Comment 38 Marek Kašík 2013-01-31 06:41:33 EST
Hi, I'm going to look at this today.
Comment 39 Hin-Tak Leung 2013-01-31 09:25:18 EST
(In reply to comment #38)
> Hi, I'm going to look at this today.

Good luck. I would suggest rolling the patch in comment 35 into the f19 2.4.11 rpm. (this is actually what I am running and why/how the patch was/is tested). If you insist on patching the older 2.4.10 or 2.4.9, you'll need to look at the 5 commits listed in comment 33.

To test that you have done so correctly, use *both* the test program in comment 10, and the embolden mode in ftview (press "2"), and under both 32-bit and 64-bit. (the pdf backend of cairo checks one thing, the ftview's embolden mode checks another, and some of the relevant code behaves differently under 32-bit vs 64-bit mode). 'WenQuanYi Zen Hei' (wqy-zenhei-fonts) seems to be worst affected font, but you should see improvements with "Droid Sans" and others.
Comment 40 Marek Kašík 2013-02-01 11:46:36 EST
I'll continue on this after weekend, I have to perform some tests yet.
Comment 41 Marek Kašík 2013-02-04 06:23:48 EST
(In reply to comment #39)
> (In reply to comment #38)
> To test that you have done so correctly, use *both* the test program in
> comment 10, and the embolden mode in ftview (press "2"), and under both
> 32-bit and 64-bit. (the pdf backend of cairo checks one thing, the ftview's
> embolden mode checks another, and some of the relevant code behaves
> differently under 32-bit vs 64-bit mode). 'WenQuanYi Zen Hei'
> (wqy-zenhei-fonts) seems to be worst affected font, but you should see
> improvements with "Droid Sans" and others.

Hi,

could you point me to specific glyph index in a specific font which shows the problem in ftview in the embolden mode? I can not find a glyph showing the problem there.

Marek
Comment 42 Hin-Tak Leung 2013-02-04 12:25:19 EST
(In reply to comment #41)
> Hi,
> 
> could you point me to specific glyph index in a specific font which shows
> the problem in ftview in the embolden mode? I can not find a glyph showing
> the problem there.
> 
> Marek

Even just the bottom page of glyph indices of 'WenQuanYi Zen Hei' (wqy-zenhei-fonts) are quite bad.

The 4 glyphs "因用图圖" (utf8, hopes it shows) are quite bad in 2.4.11 and shows a reversed look (some outlines wound the wrong way due to overflow/underflow) via the cairo pdf backend. File name "【普羅米修斯】打造機器人--"大衛"-0m_gMnzG8w0.mp4" since it is an mp4 file, and "ls --color=auto" is on so embolden in gnome termnial, and can cause gnome-terminal to go blank. Also in the course of testing, various fontconfig-dependent gtk applications - mozilla seamonkey, go blank when trying to to render web pages with embolden texts.
Comment 43 Marek Kašík 2013-02-05 10:23:19 EST
(In reply to comment #42)
> (In reply to comment #41)
> > Hi,
> > 
> > could you point me to specific glyph index in a specific font which shows
> > the problem in ftview in the embolden mode? I can not find a glyph showing
> > the problem there.
> > 
> > Marek
> 
> Even just the bottom page of glyph indices of 'WenQuanYi Zen Hei'
> (wqy-zenhei-fonts) are quite bad.

What is the starting index of those glyphs in ftview? Could you make a screenshot of the page?
Comment 44 Hin-Tak Leung 2013-02-14 09:26:05 EST
(In reply to comment #43)
> What is the starting index of those glyphs in ftview? Could you make a
> screenshot of the page?

ftdump -v provides glyph indices to unicode mapping... I have some "obviously" broken" screenshots of ftview while testing intemediate versions of the fix but don't have the corresponding "good" one at the moment. Also a pair of broken and good screenshot of gnome-terminal ("ls --color=auto" listings of files).

Just a vaguely related note for testing: it seems that gnome-terminal and seamonkey - and possibly other gtk-based applications - do some sort of double-buffering so when some glyphs fail in some way from underflow within freetype, the whole window stays blank. This is different from ftview which shows very interesting results...

The last of the fix is
http://git.savannah.gnu.org/cgit/freetype/freetype2.git/commit/?id=d56e544d653b09c657911629557ffc5277a503e3
but somewhat irrelevant as far as backport is concerned, since it is just a comment explaining a non-obvious point to prevent future rewrites breaking this code.

I'll attach some screenshots when I find the time.
Comment 45 Hin-Tak Leung 2013-03-17 22:18:53 EDT
Created attachment 711682 [details]
screenshot of broken gnome terminal, vs correct one.

screenshot of broken gnome terminal, vs correct one.

These two (gimp'ed to get them side-by-side but otherwise unmod'ed) were among some of the e-mails with the freetype people while testing changes. I think the left hand-side was what prompted the last or the 2nd last of the commits in comment 35. i.e. if you don't apply the whole series but skip the last or possibly the last two you can get this look with gnome terminal.
Comment 46 Hin-Tak Leung 2013-03-17 22:28:04 EDT
Created attachment 711683 [details]
two broken ftview - just different requested font size

screenshots of two brokenness with ftview - just different requested font size

These screenshots were what led to the prototype of
da11e5e7647b668dee46fd0418ea5ecbc33ae3b2 being revised to its final committed form, and the later accompanying comment commit d56e544d653b09c657911629557ffc5277a503e3 .
Comment 47 Hin-Tak Leung 2013-03-17 23:04:02 EDT
The bottom line of the issue is that cairo used freetype in an unanticipated way - until this report -, and the anticipated usage - i.e. usage exercised by ft-demos (ftview/ftdump) is rarely broken.

But then cairo is used by fontconfig and in turn used by gtk so the only sensible way to test fixes is to put freetype out system-wise and see what breaks. gnome terminal broken was unexpected - OTOH, the screenshot shown was the "nicer" brokenness: the nastier brokenness - my first hand experiences - was when the whole window of the web-browser or that of the terminal stopped refreshing and went blank just because either of them had trouble showing a small portion of its window, which involves some texts in bold.

usage under ftview was only broken in the middle of trying to fix usage under cairo.

I am just think what is a good test/validation plan - generally this involves confirming reported issues are fixed without causing serious regressions; "confirming reported issues are fixed" is easy, but considering how widely cairo and freetype are used, and that the freetype's demos (ftview in particular) does not exercise the part of freetype in the way cairo uses it, trying to confirm proposed patch does not cause regressions on dependent software etc - which in this case means any and every GUI-based application which have some texts displayed - is hard.

(FWIW, I have the patched freetype 2.4.11 system-wide, both 32-bit & 64-bit, just to make sure nothing breaks)
Comment 48 Fedora Update System 2013-03-19 12:26:17 EDT
freetype-2.4.10-4.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/freetype-2.4.10-4.fc18
Comment 49 Marek Kašík 2013-03-19 12:27:31 EDT
Hi,

thank you for the info. I've committed the set of backported patches to F18, F19 and rawhide.

Regards

Marek
Comment 50 Kevin Kofler 2013-03-19 13:41:12 EDT
Thanks!

I also applied your backported patches in freetype-freeworld-2.4.11-2.fc20 (not built yet because the build failed, the RPM Fusion builder is hitting checksum errors in the Rawhide repo on dl.fedoraproject.org), freetype-freeworld-2.4.11-2.fc19 and freetype-freeworld-2.4.10-3.fc18.
Comment 51 Fedora Update System 2013-03-20 17:42:11 EDT
Package freetype-2.4.10-4.fc18:
* should fix your issue,
* was pushed to the Fedora 18 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing freetype-2.4.10-4.fc18'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-4093/freetype-2.4.10-4.fc18
then log in and leave karma (feedback).
Comment 52 Fedora End Of Life 2013-12-21 10:49:59 EST
This message is a reminder that Fedora 18 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 18. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '18'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 18's end of life.

Thank you for reporting this issue and we are sorry that we may not be 
able to fix it before Fedora 18 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior to Fedora 18's end of life.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.