Bug 1753295 - Pango no longer supports bitmap fonts
Summary: Pango no longer supports bitmap fonts
Keywords:
Status: NEW
Alias: None
Product: Fedora
Classification: Fedora
Component: fonttools
Version: 31
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Parag Nemade
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 1766800 1779295 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-09-18 14:31 UTC by Jonathan Billings
Modified: 2019-12-04 16:32 UTC (History)
32 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:


Attachments (Terms of Use)
screenshot of GNOME Terminal font chooser (57.91 KB, image/png)
2019-09-20 18:01 UTC, Jonathan Billings
no flags Details
Screenshot of GNOME Terminal compared to xterm (26.25 KB, image/png)
2019-10-29 15:21 UTC, Jonathan Billings
no flags Details
Tahoma as rendered by Pango 1.44 in Fedora 31 (27.86 KB, image/png)
2019-10-31 14:29 UTC, Artem S. Tashkinov
no flags Details
bitmap terminus font with downgraded pango (103.99 KB, image/png)
2019-11-19 17:19 UTC, Sergey Bostandzhyan
no flags Details
converted terminus font with up to date F31 pango (97.16 KB, image/png)
2019-11-19 17:21 UTC, Sergey Bostandzhyan
no flags Details
Tahoma from Windows 10 with pango 1.44.7 (19.06 KB, image/png)
2019-11-24 17:15 UTC, Artem S. Tashkinov
no flags Details

Description Jonathan Billings 2019-09-18 14:31:40 UTC
Description of problem:
In pango 1.44, pango dropped support for bitmap fonts, so font packages like terminus-fonts and ucs-miscfixed-fonts no longer appear in GNOME applications that use pango, such as Terminal.  If you did a dnf system-upgrade from Fedora 30 and were using a bitmap font like Terminus, all your terminals will show the rectangular placeholders the first time you start a terminal.

Bitmap fonts like terminus and ucs-miscfixed are much easier to read in a terminal since they are pixel perfect.  If Fedora 31 is going to disable support for bitmap fonts, it needs to be announced and the package maintainers of those bitmap font packages are going to need to find a way to convert their fonts.

Version-Release number of selected component (if applicable):
pango-1.44.6-1.fc31.x86_64

How reproducible:
always

Steps to Reproduce:
1. install 'terminus-fonts'
2. Start GNOME terminal
3. Open Terminal Preferences
4. Attempt to change the font to Terminus font

Actual results:
If you upgraded from Fedora 30 and had Terminus fonts as the default font, the first time you launch Terminal, all the text will be the rectangular placeholders.  If you search for the Terminus font in the preferences, you won't be able to find it.  

Expected results:
Use the Terminus font as usual

Additional info:
It appears that pango switched from using FreeType to HarfBuzz, which only supports truetype.  I rebuilt the Fedora 30 pango package (1.43) for Fedora 31, downgraded it, and now my terminals are fine showing my bitmap fonts.

Comment 1 Jonathan Billings 2019-09-18 14:33:40 UTC
This blog post mentions the change to HarfBuzz and dropping support for bitmap fonts: https://blogs.gnome.org/mclasen/2019/05/25/pango-future-directions/

Comment 2 Peng Wu 2019-09-20 07:59:42 UTC
Upstream discussion:
https://gitlab.gnome.org/GNOME/pango/issues/386

Khaled Hosny mentioned that the following tool for bitmap fonts conversion.
https://gitlab.freedesktop.org/xorg/app/fonttosfnt

Comment 3 Jonathan Billings 2019-09-20 18:01:21 UTC
Created attachment 1617283 [details]
screenshot of GNOME Terminal font chooser

I've had some minor success with fonttosfnt but in the GNOME font viewer, you can search for the new font you created, but they use the rectangular placeholders as the label, so you just choose an entry and hope it works.  I've attached a screenshot.

Unfortunately, when you convert a bitmap font, it has only one size, the size of the original bitmap font, and if you try to create other ttf fonts with fonttosfnt, the only one file wins, since they all share the same truetype metadata.

I also tried generating the OpenType fonts with the 'fontforge' package, and instead of creating font entries with rectangular placeholders, they're just blank.

Comment 4 Peng Wu 2019-09-26 07:56:54 UTC
I just tried with fonttosfnt recently, and it seems work now.

Here are the steps to make the OpenType Bitmap fonts.

1. clone the fonttosfnt project from freedesktop;
URL: https://gitlab.freedesktop.org/xorg/app/fonttosfnt

2. apply three merge request in the upstream project;
URL: https://gitlab.freedesktop.org/xorg/app/fonttosfnt/merge_requests
from merge request !2 to !4.

3. generate the regular and bold style of fonts;
$./fonttosfnt -b -g 2 -m 2 -o terminusn.otb ../terminus-fonts/ter-*n.pcf.gz
$./fonttosfnt -b -g 2 -m 2 -o terminusb.otb ../terminus-fonts/ter-*b.pcf.gz

4. remove the terminus-fonts package and install the terminusn.otb and terminusn.otb fonts;

5. re-login the desktop and use the "Terminus" font;

Comment 5 Matthias Clasen 2019-09-26 13:23:34 UTC
Good to know, thanks. Is that the best tool to use?

Comment 6 Peng Wu 2019-09-27 04:59:06 UTC
I think so, and plan to create one Fedora copr for fonttosfnt.

Comment 7 Peng Wu 2019-09-29 07:14:29 UTC
Created Fedora copr for fonttosfnt tool.

URL: https://copr.fedorainfracloud.org/coprs/pwu/fonttosfnt/

Comment 8 Hans Ulrich Niedermann 2019-10-18 11:29:21 UTC
After building fonttosfnt from gitlab.freedesktop.org with the
three merge requests, I was able to build the two files
terminusn.otb and terminusb.otb.

After placing those two terminus?.otb files into a new directory
/usr/share/fonts/terminus-otb/ (not the /usr/share/fonts/terminus/
directory which contains the fonts.dir file and the ter-*.pcf.gz
files), gnome-terminal allows me to choose Terminus as the font
again, and shows the text in Terminus.

FTR: On Fedora, the fonttosfnt utility is shipped as part of the
xorg-x11-font-utils package, which contains a number of font related
upstream utilities in one Fedora package. The last fonttosfnt release
was mid-2018, and we definitively need the above mentioned three
pull requests to be merged into fonttosfnt before we can use fonttosfnt
to build *.otb files.

Then there is still the issue that while gnome-terminal now
allows chosing Terminus as the font again, something about
the character spacing looks wrong in some sizes.

Comment 9 Peng Wu 2019-10-22 05:23:50 UTC
> Then there is still the issue that while gnome-terminal now
> allows chosing Terminus as the font again, something about
> the character spacing looks wrong in some sizes.

Does the font sizes with problem exist in the original bitmap font?

Comment 10 Peng Wu 2019-10-22 05:25:57 UTC
I just created another merge request for ProFont bitmap font.

URL: https://gitlab.freedesktop.org/xorg/app/fonttosfnt/merge_requests/5

Comment 11 Jan Pazdziora 2019-10-29 13:47:04 UTC
I've built the fonttosfnt with the patches from pull requests 2 to 4 as fonttosfnt-x in https://copr.fedorainfracloud.org/coprs/adelton/fedora-fixes/. If the approach going forward is to build the otb variants for bitmap fonts, should xorg-x11-font-utils be respin with those patches as well?

For people looking for quick relief, I've now rebuilt terminus-fonts using the fonttosfnt approach from comment 4 into the same copr repository.

Comment 12 Jonathan Billings 2019-10-29 15:17:35 UTC
fonttosfnt (and the newly patched fonttosfnt-x) still produces the Misc Fixed fonts (ucs-miscfixed-fonts) with extra spacing between characters.  I also tried converting the fonts in the xorg-x11-fonts-misc, with similar results.

Basically, I want the same spacing as I'd get if I ran xterm.

Comment 13 Jonathan Billings 2019-10-29 15:21:53 UTC
Created attachment 1630205 [details]
Screenshot of GNOME Terminal compared to xterm

A comparison of the Fixed SemiCondensed font from ucs-miscfixed-fonts, after being converted with fonttosfnt-x to a .otb.  The font height is fine, but the spacing between characters is wrong.

Comment 14 Artem S. Tashkinov 2019-10-29 22:44:38 UTC
*** Bug 1766800 has been marked as a duplicate of this bug. ***

Comment 15 Adam Williamson 2019-10-30 00:01:36 UTC
what is the actual purpose of this bug report, since this was an intentional upstream change whose effect was known and accepted? what does the reporter expect to happen before the report is CLOSED?

Comment 16 Artem S. Tashkinov 2019-10-30 00:15:59 UTC
(In reply to Adam Williamson from comment #15)

> what is the actual purpose of this bug report, since this was an intentional upstream change whose effect was known and accepted? what does the reporter expect to happen before the report is CLOSED?

* Proper support for bitmaps fonts in F31 maybe? No? It's been fixed and you don't need to use COPR's, various utilities, etc?
* A Pango downgrade maybe?
* Respect towards the users who can't live without bitmap fonts maybe?
* A fix maybe since as of now you can't use bitmap fonts in F31 unless you 1) know how to search for/file bugs 2) know how to use various utilities outlined here 3) not be afraid to do all of the above

Again, F31 has been released with BROKEN BITMAP FONTS SUPPORT which many would consider a major feature of a desktop operating system.

I don't understand your stance/comment but this whole Pango debacle looks like a deliberate attempt to alienate even more users.

But then Gnome 3 has been exactly that (we don't need users) from the very beginning. Oh and KDE5 closely follows. XFCE so far has remained the only sane featureful DE in Fedora.

Comment 17 Matthias Clasen 2019-10-30 02:14:23 UTC
I suggest to close this as not actionable.

Comment 18 Jens Petersen 2019-10-30 04:34:23 UTC
Should we move this to fonttosfnt?  What is the cause of the spacing issue?

Comment 19 Tomasz Torcz 2019-10-30 05:10:30 UTC
* entry for CommonBugsPage, so users aren't suprised/var/lib/libvirt/images/ssd

Comment 20 Patryk Obara 2019-10-30 05:27:10 UTC
I am very glad, that Pango adopted HarfBuzz, but this update was half-baked and this breaking change was not announced properly: I see no mention of it in changes accepted for F31 nor in release notes for Gnome 3.34 - it took me completely by surprise. This is a shame because otherwise, F31 seems to be a stellar release.

To be a bit more productive here:

    $ dnf repoquery --whatprovides "*.pcf*" | wc -l
    69
    $ dnf repoquery --whatprovides "*.bdf*" | wc -l
    6

Most of these packages contain fonts, but probably not all of them. Are there any other extensions for bitmap fonts, that are no longer supported by Pango?

Comment 21 Patryk Obara 2019-10-30 05:35:44 UTC
(In reply to Adam Williamson from comment #15)
> what is the actual purpose of this bug report, since this was an intentional
> upstream change whose effect was known and accepted? what does the reporter
> expect to happen before the report is CLOSED?

The reporter didn't bother to actually answer the question, but I'll try:

- identifying fix/workaround for spacing issue (comments in https://gitlab.gnome.org/GNOME/pango/issues/386 indicate, that this might Pango issue, but I am not sure - correct me if I'm wrong, please) and re-assigning it to the appropriate package
- identifying all fonts affected, perhaps prioritizing them (I guess repackaging Terminus only would solve >50% complaints about this change)
- mass-update to affected packages (?)

Comment 22 Adam Williamson 2019-10-30 06:07:29 UTC
All of those are things we can do, but a bug against Pango titled 'Pango no longer supports bitmap fonts' isn't really going to achieve (m)any of them.

If there's a spacing bug - file a bug report on that. If we can change packages that currently provide bitmap fonts to provide them in some form that the new Pango can render - we'll probably want a bug against each individual package, perhaps with a tracker bug, probably assigned to the Fonts SIG.

Comment 23 Peng Wu 2019-10-30 06:35:47 UTC
I think Adobe Type 1 font format is also dropped,
just not requested as much as bitmap fonts.

Comment 24 Arkadiusz Hiler 2019-10-30 12:13:17 UTC
(In reply to Peng Wu from comment #4)
> I just tried with fonttosfnt recently, and it seems work now.
> 
> Here are the steps to make the OpenType Bitmap fonts.
> 
> 1. clone the fonttosfnt project from freedesktop;
> URL: https://gitlab.freedesktop.org/xorg/app/fonttosfnt
> 
> 2. apply three merge request in the upstream project;
> URL: https://gitlab.freedesktop.org/xorg/app/fonttosfnt/merge_requests
> from merge request !2 to !4.
> 
> 3. generate the regular and bold style of fonts;
> $./fonttosfnt -b -g 2 -m 2 -o terminusn.otb ../terminus-fonts/ter-*n.pcf.gz
> $./fonttosfnt -b -g 2 -m 2 -o terminusb.otb ../terminus-fonts/ter-*b.pcf.gz
> 
> 4. remove the terminus-fonts package and install the terminusn.otb and
> terminusn.otb fonts;
> 
> 5. re-login the desktop and use the "Terminus" font;

Or you can use fontforge which is already packaged in Fedora.

See what Arch is doing:
https://git.archlinux.org/svntogit/community.git/tree/trunk?h=packages/terminus-font-otb

cat > otbconvert.pe <<EOF
#!/usr/bin/fontforge
i=1
while ( i<$argc )
  Open($argv[i])
  Generate($argv[i]:r + ".otb")
  i = i+1
endloop
EOF

otbconvert.pe *.bdf

for i in *.otb; do
    install -D -m0644 $i "$pkgdir/usr/share/fonts/misc/$i"
done

(In reply to Adam Williamson from comment #22)
> If we can change packages that currently provide bitmap fonts to provide
> them in some form that the new Pango can render - we'll probably want
> a bug against each individual package, perhaps with a tracker bug, probably 
> assigned to the Fonts SIG.

This sounds sensible - packaging the converted version. Who's up for some bug filing? :-)

Comment 25 Jonathan Billings 2019-10-30 12:39:06 UTC
(In reply to Adam Williamson from comment #15)
> what is the actual purpose of this bug report, since this was an intentional
> upstream change whose effect was known and accepted? what does the reporter
> expect to happen before the report is CLOSED?

Many others have replied with good advice.  I had hoped that this would be fixed before Fedora 31 was released.  When I did a dnf system-upgrade from Fedora 30, I had the ucs-miscfixed-fonts Fixed Semicondensed font for all my GNOME Terminals, as well as Terminus for another Terminal profile.  The upgrade resulted in me having an unusable system until I downgraded pango, because every terminal was filled with the unknown character rectangles.  It was an bad UI experience, but I figured, it's still the beta, it'll be fixed soon.

But due to a bunch of finger-pointing, we ended up with a release that'll break anyone's environment if they like bitmap fonts.

I'm fine moving this to the font provider's tickets, since the upstream pango maintainers intended this breakage (even if it was poorly announced)

Comment 26 Adam Goode 2019-10-30 13:27:21 UTC
Seems reasonable to revert to pango 1.43 until all the fonts (and font packaging guidelines) are updated in Fedora.

Comment 27 Jan Pazdziora 2019-10-31 10:51:29 UTC
For the record, the Terminus-specific bugzilla is bug 1748495, and I've now filed one for LucidaTypewriter as bug 1767384.

Comment 28 Jan Pazdziora 2019-10-31 11:07:42 UTC
There currently does not seem to be a fix for the spacing issue with OpenType Bitmap fonts. So even if the font maintainers wanted to provide the OTB format in their packages, the result will be broken for some of them.

Therefore I'd also prefer moving this bugzilla back to pango and reverting Pango to the older version in Fedora 31. That way users don't have to workaround with things like

  dnf downgrade -y https://kojipkgs.fedoraproject.org//packages/pango/1.43.0/4.fc30/x86_64/pango-1.43.0-4.fc30.x86_64.rpm
  echo exclude=pango >> /etc/dnf/dnf.conf

Since the spacing issue of OTB fonts is also present with pango-1.43.0-4.fc30.x86_64, it will give reasonable platform for coming up with proper and tested solution, hopefully for Fedora 32 or earlier.

Comment 29 Matthias Clasen 2019-10-31 12:29:09 UTC
F31 has been released. The default fonts are not bitmap fonts.

Comment 30 Jonathan Billings 2019-10-31 12:42:46 UTC
bug 1766201 is the ucs-miscfixed-fonts ticket (https://bugzilla.redhat.com/show_bug.cgi?id=1766201)

Comment 31 Matthias Clasen 2019-10-31 12:45:09 UTC
Not trying to be difficult here, but 1.44 supports new attributes. Are you volunteering to do the necessary testing that none of the pango users have started using them? It is one thing for users to downgrade pango and be happy with the result on their system, its another to put a downgrade into the repository for everybody, after all the GA testing has been done.

Comment 32 Tomasz Torcz 2019-10-31 12:48:48 UTC
Let me add that writing "bitmap fonts in F31 are broken" in release notes may not be enough. Users may not *know* they are using bitmap fonts. I did not know I was! I've chosen good looking font (Terminus) and only after it broke on dist-upgrade I've learned (from bugzilla) that it is bitmap; and that the Pango change affects me.

Comment 33 Tomasz Torcz 2019-10-31 12:49:49 UTC
(In reply to Matthias Clasen from comment #29)
> F31 has been released. 

We are not Debian, we do fix things after a release.

Comment 34 Jonathan Billings 2019-10-31 12:52:18 UTC
It doesn't even appear that the pango developers are using harfbuzz correctly to get the hinting right in pango v1.44, which is why we're seeing streched fonts (https://github.com/harfbuzz/harfbuzz/issues/1892).

Comment 35 Matthias Clasen 2019-10-31 12:54:36 UTC
I happen to be one of the two remaining pango developers. If you have insight into this, don't hold back. The best place to share your knowledge is here: https://gitlab.gnome.org/GNOME/pango

Comment 36 Artem S. Tashkinov 2019-10-31 14:29:59 UTC
Created attachment 1631084 [details]
Tahoma as rendered by Pango 1.44 in Fedora 31

I wonder if I should open another bug report against Pango 1.44 as it *totally* breaks the rendering of core TTF fonts, to be precise Tahoma.

See the attached screenshot.

I've downgraded to pango-1.43.0-4.fc30.x86_64.rpm to have a peace of mind.

Comment 37 Adam Williamson 2019-10-31 14:49:37 UTC
Artem: please file a separate bug for that, as it clearly has nothing to do with bitmap fonts. Thanks.

Comment 38 Adam Goode 2019-10-31 17:01:47 UTC
Sounds like it would be tricky to roll this back in F31. Please don't upgrade F30 past 1.43 until the issues are worked out.

Also, I would request that pango changes like this go through the Changes policy in the future to ensure better communication and risk assessment:
https://docs.fedoraproject.org/en-US/program_management/changes_policy/

Comment 39 Artem S. Tashkinov 2019-10-31 20:35:16 UTC
(In reply to Adam Goode from comment #38)

> Sounds like it would be tricky to roll this back in F31.

Here's a very quick workaround which won't break anything and requires almost no tricks/hacks/etc.

Build a new Pango release using *1.43* sources and call it e.g. 1.44.6-2. Some minor shenanigans will be required for a spec file to build the package properly.

Then, when Pango developers finally get their act together, you can replace the sources to their actual version.

Problem solved.

Comment 41 Jan Pazdziora 2019-11-04 12:35:18 UTC
(In reply to Jens Petersen from comment #40)
> https://fedoraproject.org/wiki/
> Common_F31_bugs#Pango_no_longer_supports_bitmap_format_fonts

That page does not seem to have any content about this issue.

Comment 42 Jan Pazdziora 2019-11-04 12:37:45 UTC
Returning to the previous component, fonttools. The bugzilla for terminus-fonts is bug 1748495.

Comment 43 Artem S. Tashkinov 2019-11-04 12:45:09 UTC
Releasing special versions of fonts for a Pango release which doesn't even get font kerning right is _not_ the right modus operandi. Frankly speaking I'm appalled by this whole situation.

Fedora maintainers can can perfectly downgrade Pango for F31 (see comment 39) while Pango guys are fixing crucial bugs.

Comment 44 Jan Pazdziora 2019-11-04 13:34:41 UTC
If the package was downgraded, it would likely be done with the help of epoch, not by mismatching the Fedora package version and what upstream it's built from.

Matthias mentions in comment 31 that 1.44 supports new attributes. It's not clear to me if the new attributes would break pango 1.43 after the downgrade.

I don't see a bit problem with font packages shipping their content in OpenType Bitmap format, in addition to the existing formats. But to enable font package maintainers to do that, we need a guidance about the supported tools and recommended approach to do that, be it with fonttosfnt, fontforge, or something else.

Comment 45 Jens Petersen 2019-11-06 09:55:52 UTC
(In reply to Jan Pazdziora from comment #41)
> > https://fedoraproject.org/wiki/Common_F31_bugs#Pango_no_longer_supports_bitmap_format_fonts
> 
> That page does not seem to have any content about this issue.

Could you check again?

Comment 46 Jan Pazdziora 2019-11-06 10:14:17 UTC
Now I see it.

However, I don't think the workaround listed is correct. As mentioned in comment 12 for ucs-miscfixed-font and in bug 1767384 for bitmap-lucida-typewriter-fonts, the spacing is wrong with the OTB fonts produced using the fonttosfnt approach. So the workaround really only seems to work for terminus-fonts.

The only reliable approach seems to be downgrade to pango from Fedora 30 and prevent its upgrade, shown in comment 28.

Comment 47 Nicolas Mailhot 2019-11-09 09:28:27 UTC
The next edition of Fedora font packaging guidelines will forbid packaging of bitmap fonts in non-opentype format for this reason

https://pagure.io/fork/nim/packaging-committee/raw/7357efb01b0333a57eb064eacc1f0d17bcbd47ff/f/guidelines/modules/ROOT/pages/FontsPolicy.adoc

Sorry that it takes me so long to finish wrapping them up

(You need the Asciidoctor.js Live Preview extention to see a pretty-formatted version of the guidelines)

 Fedora packaging guidelines have been asking font packagers and authors to convert to OpenType for more than a decade, so the only change is that SHOULD becomes a MUST

(The harfbuzz-ng & pango raodmap were defined circa 2006 in a series of Free Desktop text summits, getting them finaly done should not surprise anyone)

Comment 48 Nicolas Mailhot 2019-11-09 09:30:15 UTC
The other big straggler is ghostsctipt, that has STILL not switched to OpenType for its default font set, even though TEX Gyre fonts have been published for this purpose for more than a decade

Comment 49 Jan Pazdziora 2019-11-09 14:53:45 UTC
Is that change to the guidelines already in, or is it still only a proposal that will need to be turned to pull request?

I don't think that forbidding something without giving the maintainers reasonable and working tools to achieve whatever needs to be achieved is the way to go. The guidelines text talks about fonttosfnt for bitmap fonts, but IIUIC, it needs to be patched to give some decent result, and even then there are issues with spacing with some fonts. We need working fonttosfnt (and working pango, if the spacing issue is issue in pango), not guidelines that forbid something.

Comment 50 Nicolas Mailhot 2019-11-09 17:33:26 UTC
That is going to be a proposal but given I'm the one who wrote the previous version 12 years ago and I've been working on this refresh for 2 years now and no one else bothered to work seriously on the subject I'm not going to change it now. Anyone who objects can take up the thankless task of squaring font tech, font standards, font uses, and fedora app (mis)behaviours

The depreciation of legacy formats, the depreciation of X11, the move to fontconfig and harfbuzz ng were already known and announced loud and clear when I wrote the previous version of the guidelines.
(they were decided in the first Text Layout summit in 2006 https://www.freedesktop.org/wiki/TextLayout/)

> Font packages in a legacy format (not TTF or OTF) MUST be registered in the legacy-font group
https://fedoraproject.org/wiki/Packaging:FontsPolicy (old Fedora font packaging policy)

Anyone interested in bitmap fonts had more than twelve years to work on an upgrade path.

No one bothered, and waiting more means no one will bother some more.

So now things will break. That's the ultimate result when depreciation warnings are not heeded.

Don't tell me that 12 years (13 starting with the text layout summit) is to short a wait.

Comment 51 Nicolas Mailhot 2019-11-09 17:35:15 UTC
(and, besides, LibreOffice gave a first warning shot when it made the same change a couple years ago)

Comment 52 Jan Pazdziora 2019-11-10 07:22:40 UTC
What *exactly* should the LucidaTypewriter (from bitmap-fonts) and ucs-miscfixed-fonts maintainers change today in their packages so that the result in Fedora 31 pango matches the visual output in Fedora 30?

Comment 53 Nicolas Mailhot 2019-11-10 08:07:18 UTC
Jan, the maintainers need to convert their files, working with upstream.

The decision to move to Harfbuzz and OpenType was taken in 2006.
(and Harfbuzz is named after OpenType so its reliance on OpenType is not exactly new or hidden https://www.freedesktop.org/wiki/HarfBuzz/)
 https://www.freedesktop.org/wiki/TextLayout/

fonttosfnt was provided at the same date to help bitmap font maintainers migrate (2005)
https://gitlab.freedesktop.org/xorg/app/fonttosfnt/commits/master/COPYING 

The Fedora font packaging guidelines marked non-OpenType formats as legacy and deprecated around the same date (IIRC in 2007, the history on
https://fedoraproject.org/w/index.php?title=Packaging:FontsPolicy&action=history states it was imported from the previous wiki in 2008)

Notice how the dates match? It was all done in coordination by the people working on fonts and text problems on Fedora and Free desktops in general.

By and large people interested in bitmap fonts chose to ignore all this till the 2006 process reached its final stages and things started breaking hard. And now they’re between a rock and a hard place.

There was a *long* window of time to convert font files, proof the result, fix fonttosfnt if necessary, etc (and in that time all major upstream apps, from Libre Office to GNOME to KDE to TEX engines to browsers to openjdk, made the hard work of converting their codebase to OpenType).

That the 13 years of migration time were wasted bitmap side won’t pull the clock back. Compressing 13 years of migration work in a few months because time is up is no one’s preferred outcome.

So, do we regret this situation? Yes we all regret this situation. Do we understand it is painful? Yes we all understand it is painful. But, time is really up now.

Comment 54 Patryk Obara 2019-11-10 18:44:13 UTC
@Nicolas Mailhot
All that you describe should have been accompanied by announcements in Fedora features - probably for every Fedora release since 2006. Just because you make up a policy, it does not mean that it will be magically implemented. And blaming users? Really?

And where does FontsPolicy EXACTLY say, that non-OpenType fonts are deprecated? OLD policy only mentions, that old formats should be grouped in legacy-fonts group, nothing more.

This is probably the most tone-deaf and corporate-like handling of userspace breaking change I've ever witnessed while using Fedora (I use it since ~Fedora Core 5). Instead of fixing the technical issue, we see attempts at corporate-like blame games.

A responsible thing to do would be to revert to the previous Pango release and properly implement and test transition for F32.

Comment 55 Michael J Gruber 2019-11-11 16:26:42 UTC
While most people here are talking about bitmap fonts, dropping type1 fonts is a *major* issue for everyone working with TeX fonts.

Before, it was perfectly possible to mix and match documents created from several programs which use pango/cairo output to pdf and include that in your TeXnical files. Now, the former cannot use type1 fonts any more while the latter cannot work with otf, at lest in their standard incarnation.

While I'm all for otf - the world just is not otf-only yet.

Comment 56 Nicolas Mailhot 2019-11-11 17:49:59 UTC
@Michael J Gruber 

I'm pretty sure some TEX engines have been converted to OpenType. The others will go the way of the dodo with PS1 fonts – they’ve been put in the same depreciation bag as legacy bitmap font formats in 2006.

You need to convert the fonts you care about to OpenType.

You need to tell your app upstreams the death knoll for PS1 and legacy fonts formats is sounding right now (Libre Office opened the ball in 2016 with its 5.3 release but every one else is following because there are no modern text lib alternatives to harfbuzz-ng).

It‘s the result of years of development of better text layouting engines. It won't be rolled back now. Especially for projects that ask for it because they don’t want to participate in improvements of the common text commons.

Comment 57 Michael J Gruber 2019-11-11 20:05:32 UTC
> I'm pretty sure some TEX engines have been converted to OpenType. The others
> will go the way of the dodo with PS1 fonts – they’ve been put in the same
> depreciation bag as legacy bitmap font formats in 2006.

pdfTeX deprecated? This is silly.

Sure TeX will fully go OpenType some day, but not now. XeTeX and LUATeX are not exactly mainstream.

> You need to convert the fonts you care about to OpenType.

I don't need pango forcing this down my throat while many other apps still require type1.
 
> You need to tell your app upstreams the death knoll for PS1 and legacy fonts
> formats is sounding right now (Libre Office opened the ball in 2016 with its
> 5.3 release but every one else is following because there are no modern text
> lib alternatives to harfbuzz-ng).
> 
> It‘s the result of years of development of better text layouting engines. It
> won't be rolled back now. Especially for projects that ask for it because
> they don’t want to participate in improvements of the common text commons.

It's a matter of choice and freedom, and that is what this pango update is cutting down. Let people use both type1 and opentype, it's as simple as that.

For the record: Downgrading to pango-1.43.0-4.fc31 restores my Fedora to a system which gives me the freedom to use both type1 and opentype fonts.

Comment 58 Nicolas Mailhot 2019-11-12 05:22:24 UTC
@Michael J Gruber this is not "just" pango, this is everyone that works on text rendering, because everyone agreed in 2006 to consolidate development on a single OpenType-only engine, and now there is only a single OpenType-only engine for complex text layouting.

Libre Office, openjdk, etc already made the switch, now it's pango's turn, it won't be alone or the last one. You won't be able to downgrade the whole stack of apps that do text just because you don't want to convert fonts to the format everyone agreed on.

>  Let people use both type1 and opentype, it's as simple as that.

So you will do the corresponding dev text layouting side? I though not. You should thank pango dev because they could have made the switch years ago, they kept old legacy redundant and bug-ridden code paths working years past their due date to give you the time to convert fonts.

Comment 59 Nicolas Mailhot 2019-11-12 07:28:12 UTC
To give some perspective:

1. end of 90's: X11 is reaching the end of its useful shelf life. Its codebase is badly outdated, its text rendering sucks loads

2. end of 90’s → 2003 attempts to shove modern text rendering in the X11 codebase, using freetype. It’s a failure. After several false starts (Xft…) decision to separate text rendering from X11, and keep freetype only for low-level rasterization. Other text handling functions will be moved to dedicated libs (starting with fontconfig for font selection). No one is willing to work on fixing the corresponding vestigial and known-broken code freetype and X11 side. The new architecture forms the basis of X12 (that you know under the Wayland name today).

3. 2003-2006. Free text handling projects (Gnome, KDE, X11, Open Office, Firefox, SIL, etc) have all been working separately on writing modern text layouting libs, starting with forks of the previous in-freetype attempts. They are all hitting the same problems, fixing the same bugs, and OpenType keeps adding new twists. They’re sick to death of this situation. They decide to convey a conference to try to find a solution. That’s the first text layout summit. https://www.freedesktop.org/wiki/TextLayout/

4. 2006. Participants agree to drop their separate dev attempts and consolidate on a single lib. They choose to base it on the Harfbuzz GTK component. Harfbuzz is an OpenType-only lib (Harfbuzz is the transliteration of OpenType in Persian).

5. 2006-2016 To mark harfbuzz is now a common codebase, it’s renamed harfbuzz-ng. Everyone from KDE to Open Office contribute its layouting fixes to this lib.

Legacy text codebases get rewritten to use harfbuzz-ng (in Gnome, KDE, Firefox, *Office, OpenKDK, TEX engines, browsers, I probably forget some of them). If you look at who is doing the work, you find the same small number of people contributing in all projects.

The process is not without bumps: Oracle gives Open Office to Apache, and Apache starts by trying to rip anything free software-ish from the codebase. Google poaches lots of free desktop text devs to work on its own text rendering stack (Chrome…). At some time Text Layout summits peter out, because consensus has been achieved and the only remaining part is implementation work.

6. 2016. Libre Office completes its migration to harfbuzz-ng in the 5.3 release. That results in the dropping of legacy font format support. Bitmap and PS1 users grumble, but they have no alternative solution to propose, and the harfbuzz-ng migration fixes in one go years of text layouting problems.

7. 2019. Pango completes its migration to harfbuzz-ng. Same result. That’s what prompted this issue ticket.

So, no use complaining converting font formats is work and OpenType is not mainstream yet.

OpenType is getting mainstream NOW. That’s why it's curtain time for other font formats in all the apps that matter (matter because they have an active maintenance team, which has been doing the harfbuzz-ng migration work in the past decade).

Comment 60 Jan Pazdziora 2019-11-12 09:06:28 UTC
(In reply to Nicolas Mailhot from comment #58)
> stack of apps that do text just because you don't want to convert fonts to
> the format everyone agreed on.

Could you please stop this throwing blame around? People are actually trying to help and work here to get Fedora users back on track.

I've built the patched fonttosfnt in copr repo, I've built OTB Terminus there -- comment 11. It seems to work for me so we could push those patches to the real Fedora packages, however, others have reported the spacing issues with the Terminus OTB fonts -- bug 1748495 comment 17.
And the that same approach does not seem to give reasonable results for Lucida -- bug 1767384, and 
ucs-miscfixed-fonts -- bug 1766201, with Pango. 

I've asked you what should be changed in comment 52, you've so far provided no guidance. If you have some *technical* information about fixing Pango (if there really is an outstanding issue in Pango), please help the developers as requested in comment 35. If you have some *technical* information about changes needed to fonttosfnt, please provide them. If you know how better to change the .spec files so that the OTB fonts don't have the spacing issues, we are all ears, the respective bugzillas have been filed.

If all you have to provide is blame and politics, well, that won't fix the code and .specs.

Comment 61 Nicolas Mailhot 2019-11-12 11:27:59 UTC
(In reply to Jan Pazdziora from comment #60)
> (In reply to Nicolas Mailhot from comment #58)
> > stack of apps that do text just because you don't want to convert fonts to
> > the format everyone agreed on.
> 
> Could you please stop this throwing blame around?

I’m not throwing blame around. I’m explaining why bitmap fonts and PS1 stopped being supported by Pango, and why it is the result of a multi-year dev effort that can not be easily reverted.

You want a magical instant fix-all solution. I don't have one to propose. No one has one to propose. There is no magic instant fix-all solution except converting fonts using existing tools, and adjusting the result by hand when it’s wrong (and improving tools so there is less need to do manual adjustments).

Michael J Gruber messages are quite representative, on why this work got delayed till the rug was pulled under legacy formats.

Comment 62 Jan Pazdziora 2019-11-12 12:17:21 UTC
For audience awareness -- that Fonts packaging policy change is now proposed via pull request: https://pagure.io/packaging-committee/pull-request/934. I've voiced my opinion there.

Comment 63 Kevin Kofler 2019-11-13 10:16:25 UTC
Yet, Qt has been using HarfBuzz for a few years now, and has no trouble loading legacy bitmap fonts. HarfBuzz is mainly a layouting engine, it can be used just fine together with FreeType's font loader. The Pango developers decided against doing that due to some locking issues with FT_Face that would be better addressed in FreeType, rather than switching to a much inferior font loader as they did.

Comment 64 Kevin Kofler 2019-11-13 10:32:23 UTC
I also wonder what Pango dumping FreeType means for features such as hinting, subpixel rendering, etc.

Comment 65 Nicolas Mailhot 2019-11-13 12:13:33 UTC
(In reply to Kevin Kofler from comment #63)
> Yet, Qt has been using HarfBuzz for a few years now, and has no trouble
> loading legacy bitmap fonts. HarfBuzz is mainly a layouting engine, it can
> be used just fine together with FreeType's font loader.

That's what pango (and Libre Office) did at first, before giving up on the legacy path

> The Pango developers decided against doing that due to some locking issues with FT_Face that
> would be better addressed in FreeType,

Because harfbuzz was created and written to replace this buggy code. By the people who used to maintain this buggy code freetype-side, before deciding fixing it properly required creating new lib.

You may as well ask why wayland does not depend on good old X11 code. Replacing the good old bug-ridden code is the whole point of both projects.

Comment 66 Kevin Kofler 2019-11-13 12:18:25 UTC
I think we would be better off without Wayland, too (and in fact, I still use X11 exclusively).

Comment 67 Kevin Kofler 2019-11-13 12:20:04 UTC
(In reply to Nicolas Mailhot from comment #61)
> You want a magical instant fix-all solution. I don't have one to propose. No
> one has one to propose.

I have one to propose: short-term (instant fix), downgrade and Epoch-bump Pango in F31, long-term, fork it. They should use FreeType to load fonts.

Comment 68 Matthias Clasen 2019-11-17 17:04:46 UTC
And I assume you volunteer to develop and maintain the forked pango, Kevin?

I'm quite happy to see more volunteers show up for work on the text rendering stack.

Comment 69 Jan Pazdziora 2019-11-18 07:54:04 UTC
Matthias, what if your proposed course of action for Fedora 31 and users who would like to use bitmap fonts on their terminals? We've tried patched fonttosfnt to convert the existing fonts to OTB format, it only works partially because the spacing is broken. What should we try next?

Comment 70 Sergey Bostandzhyan 2019-11-18 09:41:08 UTC
Just wanted to add that...

> dnf downgrade -y https://kojipkgs.fedoraproject.org//packages/pango/1.43.0/4.fc30/x86_64/pango-1.43.0-4.fc30.x86_64.rpm
> echo exclude=pango >> /etc/dnf/dnf.conf

...really saved my day after the upgrade, thank you for posting this workaround!
For someone who codes in vim those terminal fonts (I am also using terminus) are essential, I hope we can find a permanent solution for this issue.

Comment 71 Jan Pazdziora 2019-11-19 16:54:49 UTC
(In reply to Sergey Bostandzhyan from comment #70)
> For someone who codes in vim those terminal fonts (I am also using terminus)

I've built Terminus in the OTB format and it seems to work for me even with the new Pango, although others have reported spacing issues with the result. If you are OK with trying package from copr repo, could you please check bug 1748495 comment 12 and upgrade terminus-fonts from copr, to see if it is usable for you?

Comment 72 Sergey Bostandzhyan 2019-11-19 17:18:17 UTC
(In reply to Jan Pazdziora from comment #71)
> (In reply to Sergey Bostandzhyan from comment #70)
> > For someone who codes in vim those terminal fonts (I am also using terminus)
> 
> I've built Terminus in the OTB format and it seems to work for me even with
> the new Pango, although others have reported spacing issues with the result.
> If you are OK with trying package from copr repo, could you please check bug
> 1748495 comment 12 and upgrade terminus-fonts from copr, to see if it is
> usable for you?

Sure thing, unfortunately I have those spacing issues as well, it seems like the font is rendered monospaced. I'll attach screenshots containing the downgraded pango version with the bitmap terminus font and up to date pango version with your converted terminus font package.

Comment 73 Sergey Bostandzhyan 2019-11-19 17:19:54 UTC
Created attachment 1637804 [details]
bitmap terminus font with downgraded pango

Comment 74 Sergey Bostandzhyan 2019-11-19 17:21:22 UTC
Created attachment 1637805 [details]
converted terminus font with up to date F31 pango

This is a screenshot that is using a converted version of the terminus font from https://copr-be.cloud.fedoraproject.org/results/adelton/fedora-fixes/fedora-31-x86_64/01081531-terminus-fonts/

Comment 75 Peng Wu 2019-11-20 08:25:13 UTC
Could you try the following command with terminus-fonts?

$fonttosfnt -b -g 2 -m 2 -o terminusn.otb /usr/share/fonts/terminus/ter-*n.pcf.gz

$fonttosfnt -b -g 2 -m 2 -o terminusb.otb /usr/share/fonts/terminus/ter-*b.pcf.gz

Comment 77 Artem S. Tashkinov 2019-11-24 16:53:29 UTC
Here's a new finding:

Nicolas Mailhot wrote ( https://bugzilla.redhat.com/show_bug.cgi?id=1767499#c7 )

> Tahoma and other parts of Microsoft’s “fonts for the web” are very early display fonts with lots of technical mistakes.

> So, they are challenging to display correctly, because they are, basically, incorrect (they were built around the bugs of the Windows text engine, and the screen pixel resolutions, of the time, both of which do not apply on a 2019 Linux system).

> The version Microsoft ships in Windows has been fixed a long time ago (or they may special-case it, I don’t remember).

> The version people keep installing on Linux is the same 1990’s Microsoft dump. Because they do not have access to the fixed font files for legal reasons.

So, bitmaps fonts are no longer supported but it's in theory possible to convert them to OpenType fonts, OK, I'll live with that.

It turns out older TTF fonts which worked near PERFECTLY in the past are also NOT supported.

At this point I'm confused by the direction Fedora and Pango developers are taking.

Is Linux so special you think you can simply throw away perfectly working fonts? That doesn't exactly make Linux a great or enticing OS.

Comment 78 Artem S. Tashkinov 2019-11-24 17:15:57 UTC
Created attachment 1639250 [details]
Tahoma from Windows 10 with pango 1.44.7

Comment 79 Artem S. Tashkinov 2019-11-24 17:17:45 UTC
Oh, Windows 10 TTF fonts are not rendered by Pango 1.44.7 either - kerning is completely borked up.

Comment 80 Artem S. Tashkinov 2019-11-24 17:18:46 UTC
With Pango 1.43.0-4.fc30 everything is perfect.

Comment 81 Nathan Bibb 2019-11-29 22:52:04 UTC
I use the Dina bitmap font, and so far have been fine using the downgrade commands from Comment 28.  However, after my last update I am not able to open Nautilus.  I get the following when opening from the command line:

    $ nautilus
    nautilus: symbol lookup error: nautilus: undefined symbol: pango_attr_insert_hyphens_new

...so I guess I will look into converting Dina to Opentype?  Not sure what other path forward I have to keep my bitmap font and stay on Fedora.  If anyone has suggestions, please let me know.

Comment 82 Nathan Bibb 2019-11-29 23:28:12 UTC
I attempted to use fonttosfnt as directed in this document: https://fedoraproject.org/wiki/BitmapFontConversion.

However, for my font I got an error:

    $ fonttosfnt -b -g 2 -m 2 -o DinaMedium10.otb DinaMedium10.pcf 
    Couldn't select character map: 6.

I am out of my depth here.  I guess I give up on the font I prefer to use?  Or switch distros?

Comment 83 spamulousbastard+redhat 2019-11-30 12:54:22 UTC
Having had some success with Terminus, I tried to convert TamzenForPowerline (https://github.com/sunaku/tamzen-font) to otb, with the method described in Bug 1748495 comment 24. Even using the patched fonttosfnt from pwu's copr (https://copr.fedorainfracloud.org/coprs/pwu/fonttosfnt/), the result fails to be detected as a monospace font, so cannot be selected in gvim or terminal emulators, although it works fine in gedit.

I was able to get it to work, eventually, by opening all the .bdfs at once with fontforge (fontforge TamzenForPowerline*.bdf; this opens over 9000 fontforge windows), manually setting the width for each bitmap strike through the GUI (encoding -> compact, drag select all, right click, set width, accept suggestion), and then generating a TTC.

The manual width-setting step is necessary because otherwise the produced font has zero spacing between characters.  According to the discussion on this fontforge bug: https://github.com/fontforge/fontforge/issues/3853, the cause is that fontforge itself relies on pango to load .bdf fonts.  I.e., the conversion of fonts to make them compatible with pango 1.44 does not work properly under pango 1.44.

It is clear that inflicting this level of breakage in a minor version was a gross error. The time to make this compatibility break, if it could be made, was in 2006, when the automatic conversion program was fresh and maintained, and the  reliance on support for bitmap fonts was less. 13 years later, the only option is to get off the pot.

I have taken the liberty of cursing the pango developers to only be able to use < 100 physical PPI screens forevermore, and if you feel the same I encourage you to donate your voodoo energies to reinforce it.

Comment 84 Nathan Bibb 2019-12-01 19:12:57 UTC
Just an update on my particular journey here, in case anyone is having the same issue and looking for a fix.  General series of event is as follows:

1. On upgrading to Fedora 31, I was hit with the Pango issue, as I use the Dina bitmap font for Terminal, GEdit and other text programs.
2. I reformatted my machine with a fresh install, and immediately downgraded pango as indicated in Comment 28.
3. Things were fine until earlier this week, when Nautilus failed to launch as I reported in Comment 81.
4. I tried every combination of fonttosfnt (Comment 4), fonttosfnt-x (Comment 11), the otb terminus-fonts (Comment 11), and fontforge (Comment 83).  In every case, I got blank fonts.  I cannot figure out why this is happening.

I began to suspect that there was something broken on my system, so I spun up a new Fedora 31 installation in Gnome Boxes, and got the following results:
1. fonttosfnt (Comment 4) still gave me the error "Couldn't select character map: 6"
2. The otb terminus-fonts from the fedora-fixes copr (Comment 11) worked fine
3. fonttosfnt-x from the fedora-fixes copr (Comment 11) worked to convert the Dina font, which satisfied my requirement

I didn't try fontforge since #3 got me where I wanted.

So it looks like some combination of downgrades and maybe other packages/repo installations broke something on my system.  Unfortunately, I think my next step is a clean re-installation of my system.

Hope this helps somebody.  Sorry if this bug report is the wrong place for this, but I wasn't sure where else to post this info.

Comment 85 Peng Wu 2019-12-03 07:10:20 UTC
To convert the Bitmap font to Monospace font,
please use the following repo:
https://copr.fedorainfracloud.org/coprs/pwu/fonttosfnt/

And add the "-c" parameter, it may help.
$fonttosfnt -b -c -g 2 -m 2 -o tamzenr.otb TamzenForPowerline*r.pcf

Comment 86 Michael J Gruber 2019-12-03 12:25:01 UTC
If you have any helpful information about the transition from type1 fonts please comment in bug 1779123 .

Comment 87 Daniel Gonçalves 2019-12-04 16:32:42 UTC
*** Bug 1779295 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.