Bug 368561

Created attachment 249451 [details]
image of ff and fi shifted by a pixel.

Oops, wrong gtk.

I've seen this in F8 with Subpixel smoothing and Medium hinting. Looks fixed in
rawhide? ("fi" is still 1 pixel shorter than "f" and "i" apart, but no longer 1
pixel dropped.)

Someone pointed out to me that part of what is happening here is that there is a
ligature going on, so updating the title. I can't comment on whether or not it
is fixed in rawhide.

Actually, I've looked again. It is not fixed in rawhide. Wishful thinking!

If anything, this is a pango issue. 

Note that this also happens with default fonts/theme/etc. in F9alpha1.

Updating bug to note that it happens in F9 still, and cc'ing pango guy per mclasen's last comment. (This is sooo minor, and yet soooo irritating ;)

That's most probably a freetype hinting bug.  It knows about basic Latin letters, but not about fancy ligatures, so doesn't hint those.

Reproduced using:

pango-view --font 'dejavu sans 10' --dpi 96 --text 'f f ff fi f f'

Lets see if reassigning to DejaVu helps.

> Lets see if reassigning to DejaVu helps.

I doubt, I'm quite sure this is a Freetype autohinter bug, and we can do nothing about that in DejaVu, that's just Freetype deciding on its own how to render things. Turn on the bytecode interpreter to get hinting from the font itself and the bug is gone.

Thanks Ben.  One thing that may help is to use two-component ligatures.  That is, map the 'fi' to two glyphs that together form the 'fi' ligature.  Not sure though.

(In reply to comment #12)
> Thanks Ben.  One thing that may help is to use two-component ligatures. That
> is, map the 'fi' to two glyphs that together form the 'fi' ligature.  Not
> sure though.

But then Freetype should know how to autohint those ligature components, and I think that's even harder than the ligatures which in this case are at the assigned code points. With components it needs to analyze the OpenType features first. (OTOH, the ligatures don't have to use those code points, so it may need the proper way of handling them anyway)

So I would expect Freetype to have a glyph list containing all Latin letters to get its autohinter metrics from, and that list doesn't contain the fi ligatures. If that's the case, it should be as easy as just adding them to that list. But of course, I have no idea how Freetype's autohinter works :-)

*** Bug 470704 has been marked as a duplicate of this bug. ***

Note that there are also fl and ffi ligatures which have the same problem.

Should we be tweaking Pango and Qt 4 to kill the use of those useless ligatures entirely?

(In reply to comment #15)
> Note that there are also fl and ffi ligatures which have the same problem.
> 
> Should we be tweaking Pango and Qt 4 to kill the use of those useless ligatures
> entirely?

They're not useless, when the software supports them properly the result is better than no ligature (and some fonts rely hugely on them)

Yet:
* Qt/KDE 3 never used those ligatures, nobody ever complained about that.
* I remember reading about complaints from users of languages with dotless i (mainly Turkish) about the fi and ffi ligatures which hide the dot on the i in several fonts.
* Why can't a letter just be a letter? (It's not just ligatures, kerning for letter pairs like To where the o gets moved into the empty space of the T is similarly funky.) All those special cases make the fonts look irregular and inconsistent.

Basically, those ligatures cause this bug without bringing any tangible benefit. Even with this bug fixed, they may still get noticed (as in: hey, there's some weird ligature here which jumps to the eyes) and fail in their intent at getting read as the letter pairs/triplets they're supposed to represent.

</RANT>

(In reply to comment #17)
> Yet:
> * Qt/KDE 3 never used those ligatures, nobody ever complained about that.

People did, you just didn't notice it

> * I remember reading about complaints from users of languages with dotless i
> (mainly Turkish) about the fi and ffi ligatures which hide the dot on the i in
> several fonts.

If the fonts are buggy they should be fixed. If the fonts are ok but freetype makes a mess of them freetype should be fixed

> * Why can't a letter just be a letter?

That's a naïve western view. An awful lot of "letters" in unicode and opentype are not just "letters" but are rendered via various magic of which ligatures are a very small part. The OO.o issue tracker is choke-full of people begging SUN to support this kind of magic (because OO.o right now is even less capable than QT3 was and QT4/pango would be if we followed your suggestion). So users do care about this. (and the OO.o people are working to fix their bugs)

DejaVu ligatures are light and not as spectacular than in other fonts. But DejaVu is not the only ligature user. Libertine has them, and so have an increasing number of other fonts (any self-respecting font designer adds them nowadays, because they're one of the key differentiator with legacy fonts)

http://www.josbuivenga.demon.nl/tallys.html

This is just a freetype autohinter bug. We can't ask font designers to ignore the patented bytecode engine and at the same time not support stuff that work with it. We can't hide your head in the sand while font technology progresses and Microsoft/Adobe invest massively in OpenType to fix its remaining typographical problems. We can't refuse to support stuff that works in other OSes and hope users don't notice.

> (It's not just ligatures, kerning for
> letter pairs like To where the o gets moved into the empty space of the T is
> similarly funky.)

This is overtight kerning in the font of which upstream is aware of (and on the TODO fix list). Not ligatures at all.

> All those special cases make the fonts look irregular and
> inconsistent.

For the ligature bit, complain at freetype
For the kerning bit, complain at dejavu

That's two different problems. Modern fonts are complex beasts. We've progressed a lot in past years but there's still a lot of work to do freetype and free fonts side. Your proposal would not help things at all.

> That's a naïve western view. An awful lot of "letters" in unicode and opentype
> are not just "letters" but are rendered via various magic of which ligatures
> are a very small part.

Sorry, I expressed myself badly. I know ligatures and similar constructs are commonplace in some alphabets (Arabic and most of the Indic scripts, for example).

So let me correct myself:
-> Why can't a Latin letter just be a letter? <-

To me all this ligature stuff for Latin letters looks like a solution looking for a problem, i.e. "hey, we have all this complex text layout framework for Arabic and Indic scripts, let's also use it for Latin just to be fancy".

> http://www.josbuivenga.demon.nl/tallys.html

Those are very good examples of the fi ligature gobbling up the dot on the i.

> This is just a freetype autohinter bug. We can't ask font designers to ignore
> the patented bytecode engine and at the same time not support stuff that work
> with it.

They wouldn't have to do anything at all. My proposal is to disable ligatures for Latin letters systemwide in Pango and Qt 4 Scribe, any ligatures provided by fonts would just be ignored (as they were in the past, and still are in e.g. Qt 3 apps).

As for the kerning, I know it's a separate issue, I only brought it up here because it's part of the same philosophy of special-casing letter pairs. But let's limit ourselves to ligatures in this bug report.

(In reply to comment #19)
> > That's a naïve western view. An awful lot of "letters" in unicode and opentype
> > are not just "letters" but are rendered via various magic of which ligatures
> > are a very small part.
> 
> Sorry, I expressed myself badly. I know ligatures and similar constructs are
> commonplace in some alphabets (Arabic and most of the Indic scripts, for
> example).
> 
> So let me correct myself:
> -> Why can't a Latin letter just be a letter? <-

Because even latin scripts have exceptions that need magic (you wouldn't believe what the africans do with latin letters).

> To me all this ligature stuff for Latin letters looks like a solution looking
> for a problem, i.e. "hey, we have all this complex text layout framework for
> Arabic and Indic scripts, let's also use it for Latin just to be fancy".
> 
> > http://www.josbuivenga.demon.nl/tallys.html
> 
> Those are very good examples of the fi ligature gobbling up the dot on the i.

Which is ok for non-turkish users ie turkish needs a special locl rule for ligatures in fonts or even some special magic at pango/harfbuzz level.

> > This is just a freetype autohinter bug. We can't ask font designers to ignore
> > the patented bytecode engine and at the same time not support stuff that work
> > with it.
> 
> They wouldn't have to do anything at all. My proposal is to disable ligatures
> for Latin letters systemwide in Pango and Qt 4 Scribe, any ligatures provided
> by fonts would just be ignored (as they were in the past, and still are in e.g.
> Qt 3 apps).

Experience shows this won't lead to any fix freetype-side and we'll just have to enable ligature support under user pressure in a few months, with the bug still unfixed. (hiding ligature problems was already tried firefox-side with no good result)

Sorry but ligatures exist and ignoring them in the hope our bugs will magically go away is not a solution. Please spend your energy getting freetype people to fix their stuff.

> To me all this ligature stuff for Latin letters looks like a solution
> looking for a problem, i.e. "hey, we have all this complex text layout
> framework for Arabic and Indic scripts, let's also use it for Latin just to
> be fancy".

Ligatures for Latin already existed in Medieval times with the first printing devices. It's not like they suddenly appeared out of nowhere the past decade just because we created the OpenType system.

It would be nice though if features can be manually turned on and off by a user...

Still a problem in F10; updating appropriately.

I have reported these issues upstream to both DejaVu and to FreeType.

FreeType insists that this is not their bug and closed my bug within hours.  That link is here:
https://savannah.nongnu.org/bugs/?func=detailitem&item_id=25013

DejaVu insists it's a FreeType bug and closed my bug within hours.  Their link is here:
http://bugs.freedesktop.org/show_bug.cgi?id=15909

I hope this can be resolved somehow.
It's rather ugly and embarrassing to see it in Fedora 10.

Not embarrassing to see in F10; embarrassing to see in F8, F9, *and* F10. ;)

Here are the corrected links to the upstream discussions:

FreeType insists that this is not their bug and closed my bug within hours. 
That link is here:
http://savannah.nongnu.org/bugs/?21190

DejaVu insists it's a FreeType bug and closed my bug within hours.  Their link
is here:
http://bugs.freedesktop.org/show_bug.cgi?id=15909

/me takes another look at this.

(In reply to comment #23)
> I have reported these issues upstream to both DejaVu and to FreeType.

No. You asked two *different* questions to both projects. Your problem is freetype does not render ligatures consistently with other glyphs. So ask freetype to fix their rendering of ligatures. Do not ask them to disable ligatures. Which as they noted is not their problem.

Incidentally, the same bug also appears on my nokia N810 with some Debian-based distro which also uses the Freetype autohinter. At least in GTK apps, I don't have a Qt4 application on it to test it with in there, but I remember seeing it on my normal computer when trying out Fedora 10 with KDE4 a few days ago.

(In reply to comment #27)
> (In reply to comment #23)
> > I have reported these issues upstream to both DejaVu and to FreeType.
> 
> No. You asked two *different* questions to both projects. Your problem is
> freetype does not render ligatures consistently with other glyphs. So ask
> freetype to fix their rendering of ligatures. Do not ask them to disable
> ligatures. Which as they noted is not their problem.

Actually, it is ONE question to DejaVu, and TWO questions to Freetype.
Are you interested in fixing this symptom, or are you just trying to show us how smart you think you are?
Stop it.

Folks, please calm down! :-)
@Nicolas: Please see Kriston's corrected link (comment #25).
@Kriston: Nicolas simply missed your correction, no need to flame him for that. ;-)

FreeType's bug has a constructive comment that it could be higher up in the rendering chain. 
If I were to take a wild guess, I would say that when FreeType is presented with the ligature glyph it might actually be rendering it without any hinting instructions at all when the autohinter is enabled.
The FreeType bug report is justified because fftype/ffdiff cannot reproduce it.  Like the FreeType bug report indicates, the symptions are not reproduceable when using fftype/ffdiff Freetype test programs, but the Red Hat bug report reproduces it from the Pango level in comment #10.
Additionally, this bug manifests itself in Ubuntu when Ubuntu is instructed to use the FreeType autohinter (it has the native hinter enabled by default).
I don't want the use of the autohinter to be a red herring when tracking down this problem.

From one of the FreeType bug reports at http://savannah.nongnu.org/bugs/?25013:

>There are high chances that the autohinter doesn't render these ligature
>glyphs well.  However, this problem is not special to ligatures per se but can
>happen with any glyph (well, it shouldn't happen with any glyph...).

This is discouraging, and it's rather why I suggested that the option to disabling ligatures ought to be provided somewhere in the rendering chain.
For purity's sake, the argument that ligatures should "always" be accepted is not backed up by precedent.  The user already can optionally disable deltas, hinting, and kerning.  It naturally follows that he must also be allowed to optionally disable ligatures if ligatures look awful.
Right now, installing a clean copy of Fedora 10 (and 9,8,7, etc) you get awful ligatures.

Might this be a HarfBuzz bug? Is Pango using HarfBuzz yet? Qt 4 definitely is.

(In reply to comment #32)
> This is discouraging, and it's rather why I suggested that the option to
> disabling ligatures ought to be provided somewhere in the rendering chain.

I don't consider a method to disable ligatures a solution for this bug, as it doesn't do anything to fix it. I'm not saying it's not nice to have, but it should be independent from this bug report.

But if we make the option mandatory in Fedora, the bug will be gone for good.

Upstream has thoughtful comment about ligature suppression and is also soliciting help to fix autohinting of ligatures:

http://savannah.nongnu.org/bugs/?25013

If you guys want to see *any* resolution, please stop wild guessing where the problem is.  Or suggest random resolutions...

The bug is clear: the freetype autohinter is performing badly.  Either go dig the code and fix it, or don't make noise as my inbox is full enough already.

Thanks,

And here is how you reproduce it with FreeType:

ftstring -m 'f ff fi fl ffi ffl f' 13.3 /usr/share/fonts/dejavu/DejaVuSans.ttf

(copy/paste this)

(In reply to comment #37)
> If you guys want to see *any* resolution, please stop wild guessing where the
> problem is.  Or suggest random resolutions...
> 
> The bug is clear: the freetype autohinter is performing badly.  Either go dig
> the code and fix it, or don't make noise as my inbox is full enough already.
> 
> Thanks,

It wasn't clear before, but it is clear now to you.  I *cannot* reproduce it on my system using the ft2demos utilities but I can inside Mozilla and other applications.
This discussion is very helpful.  Investigate your email filters.

You can reproduce, if you copy/paste the command I gave in comment #38.  Your comments mostly just confuse people.  Nicolas, Ben, and I know what we are talking about.  Really.

If anyone wants to know how the autohinter works, here is a detailed paper:
http://www.tug.org/TUGboat/Articles/tb24-3/lemberg.pdf

The code lives in freetype2/src/autofit/aflatin.c and aflatin2.c.

Took me 2min to fix the bug:

diff -u -p -d -r1.65 aflatin.c
--- src/autofit/aflatin.c	7 Jul 2007 07:30:39 -0000	1.65
+++ src/autofit/aflatin.c	5 Dec 2008 21:12:08 -0000
@@ -2144,6 +2144,7 @@
     { 0x20A0, 0x20CF },  /* Currency Symbols */
     { 0x2150, 0x218F },  /* Number Forms */
     { 0x2460, 0x24FF },  /* Enclosed Alphanumerics */
+    { 0xFB00, 0xFB06 },  /* Latin Ligatures */
     { 0     , 0      }
   };

(In reply to comment #40)
> Your comments mostly just confuse people.  Nicolas, Ben, and I know what we are
> talking about.  Really.

If this is true then you would have already convinced upstream that it is a FreeType problem, which did not happen until I engaged the two upstream entities with my comments and efforts.
I want this problem to be resolved.  Your comments about other people are wholly unnecessary and we still sit here, years later, with this problem.

Already building fixed package for rawhide and F-10...

I just tested your fix and it has indeed fixed it for Mozilla and the other applications I have seen it in.

Also fixed with:
pango-view --font 'dejavu sans 10' --dpi 96 --text 'f f ff fi f f'

Are you submitting to upstream after building the RPMS?

Submitted upstream already.

> (In reply to comment #12)
> So I would expect Freetype to have a glyph list containing all Latin letters to
> get its autohinter metrics from, and that list doesn't contain the fi
> ligatures. If that's the case, it should be as easy as just adding them to that
> list. But of course, I have no idea how Freetype's autohinter works :-)

Looks like I knew what I was talking about above :-)

Anyway, I'd like to see how the autohinter would handle it when the ligatures are not encoded at those code points, but out of bounds (i.e. without code points, only reachable with opentype features), as is done in some fonts...

In rawhide already.  Koji is down.  I'll push to F10 updates-testing after it comes back.

It's not going to happen if the glyphs are not bound to the ligature characters.  If any font doesn't have those, that's a font bug then.

What about F8 and F9?

Awesome, thanks, Behdad!

(In reply to comment #49)
> It's not going to happen if the glyphs are not bound to the ligature
> characters.  If any font doesn't have those, that's a font bug then.

Not really. The U+FB00-... code points aren't mandatory for the fi ligatures (just like it's not mandatory to use the Arabic presentation forms for contextual forms of Arabic letters). These code points are there for compatibility reasons with legacy encodings.

Also, there are much more possible Latin ligatures that don't have reserved code points, like fb, fh, fĺ etc.

Ben, I know that.  What I'm saying is that it's impossibly hard to do it without those in FreeType.  So, in that case we can blame the font because there's an easy fix for it.

Luis, 'welcome.

Kevin, feel free to build for those.  Anyone else should feel free too.

freetype-2.3.7-2.fc10 has been submitted as an update for Fedora 10.
http://admin.fedoraproject.org/updates/freetype-2.3.7-2.fc10

freetype-2.3.5-7.fc9 has been submitted as an update for Fedora 9.
http://admin.fedoraproject.org/updates/freetype-2.3.5-7.fc9

freetype-2.3.5-5.fc8 has been submitted as an update for Fedora 8.
http://admin.fedoraproject.org/updates/freetype-2.3.5-5.fc8

Be advised that the HEAD version in FreeType's CVS extended the fix to some other ranges.
I have confirmed the bug is fixed in the latest FreeType CVS for the offending Latin ligatures in the DejaVu font family.  Thank you for your effort!

Created attachment 326146 [details]
Diff for ligatures fix

Your patch is reversed and it also contains whitespace changes...

These are the ranges which are added in FreeType's CVS and not in Fedora's CVS:
{  0x2C60  ,  0x2C7F   },  /* Latin Extended-C */
{  0x2DE0  ,  0x2DFF   },  /* Cyrillic Extended-A */
{  0xA640U ,  0xA69FU  },  /* Cyrillic Extended-B */
{  0xA720U ,  0xA7FFU  },  /* Latin Extended-D */
{ 0x1D400UL, 0x1D7FFUL },  /* Mathematical Alphanumeric Symbols */

Comment on attachment 326146 [details]
Diff for ligatures fix

Obsoleted

Created attachment 326180 [details]
Diff between aflatin.c 1.65 to 1.66 from FreeType CVS

Corrected unified diff between aflatin.c 1.65 to 1.66 from FreeType CVS.

*** Bug 476556 has been marked as a duplicate of this bug. ***

If this is in testing, why does "yum update freetype --enablerepo=updates-testing" not find it?

On F10 I pushed it to updates yesterday.

(In reply to comment #64)
> On F10 I pushed it to updates yesterday.

"yum update freetype" gives me "No Packages marked for Update," yet I do not have this updated version.

The updates for F8, F9 and F10 should all be in stable after the push which is currently ongoing.

freetype-2.3.7-2.fc10 has been pushed to the Fedora 10 stable repository.  If problems still persist, please make note of it in this bug report.

freetype-2.3.5-7.fc9 has been pushed to the Fedora 9 stable repository.  If problems still persist, please make note of it in this bug report.

freetype-2.3.5-5.fc8 has been pushed to the Fedora 8 stable repository.  If problems still persist, please make note of it in this bug report.

Got the update today, looks perfect.