Bug 1004717 - Some characters are displayed as boxes in Urdu on the first page of the installation
Some characters are displayed as boxes in Urdu on the first page of the insta...
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: lorax (Show other bugs)
22
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Brian Lane
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-09-05 06:46 EDT by Mike FABIAN
Modified: 2015-10-22 09:22 EDT (History)
15 users (show)

See Also:
Fixed In Version: lorax-23.18-1 lorax-23.18-1.fc23
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-10-16 11:19:45 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
boxes-in-urdu-in-language-selection-page.png (100.42 KB, image/png)
2013-09-05 06:46 EDT, Mike FABIAN
no flags Details
urdu-translation-looks-correct-in-gedit-on-f19.png (39.90 KB, image/png)
2013-09-05 06:49 EDT, Mike FABIAN
no flags Details
fonts-dejavu-sans-scheherazade.png (82.73 KB, image/png)
2013-09-05 07:08 EDT, Mike FABIAN
no flags Details
kate-urdu-dejavu-sans.png (30.96 KB, image/png)
2013-09-05 07:11 EDT, Mike FABIAN
no flags Details
boxes-in-urdu-in-language-selection-page-TC6.png (81.08 KB, image/png)
2013-10-28 11:11 EDT, Mike FABIAN
no flags Details
boxes-in-urdu-in-language-selection-page-fedora-23-beta-rc1.png (120.55 KB, image/png)
2015-09-16 05:37 EDT, Mike FABIAN
no flags Details
font-mixture-in-urdu-in-language-selection-page-fedora-23-tc10.png (122.71 KB, image/png)
2015-10-18 14:50 EDT, Mike FABIAN
no flags Details
urdu-anaconda-f23-root-password-possibly-font-mixture.png (58.92 KB, image/png)
2015-10-22 04:47 EDT, Mike FABIAN
no flags Details

  None (edit)
Description Mike FABIAN 2013-09-05 06:46:33 EDT
Created attachment 794162 [details]
boxes-in-urdu-in-language-selection-page.png

Tested with Fedora-20-Alpha-TC3-x86_64-netinst.iso.

See attached screenshot.
Comment 1 Mike FABIAN 2013-09-05 06:48:48 EDT
This is the translation:

#: pyanaconda/ui/gui/spokes/welcome.glade:395
msgid "What language would you like to use during the installation process?"
msgstr "کون سی زبان کا اپ استعمال کر نا چاہتے ہیں؟"

The translation looks correct in gedit tested on f19, see screenshot.
Comment 2 Mike FABIAN 2013-09-05 06:49:41 EDT
Created attachment 794165 [details]
urdu-translation-looks-correct-in-gedit-on-f19.png

Here in gedit, the translation is rendered correctly.
Comment 3 Mike FABIAN 2013-09-05 07:06:34 EDT
Probably a font problem.
Comment 4 Mike FABIAN 2013-09-05 07:08:43 EDT
Created attachment 794204 [details]
fonts-dejavu-sans-scheherazade.png

Screenshot showing that "DejaVu Sans" does not have the glyphs
displayed as boxes but "Scheherazade" for example has them.

The font looks very much like "DejaVu Sans" in Anaconda.

The font in my gedit screenshot is also "DejaVu Sans" but apparently gedit
manages to fallback to something else for the missing glyphs.
Comment 5 Mike FABIAN 2013-09-05 07:11:00 EDT
Created attachment 794205 [details]
kate-urdu-dejavu-sans.png

screenshot of kate, showing that contrary to gedit, kate cannot
fall back to other fonts for the Urdu glyphs missing in "DejaVu Sans".
Comment 6 Mike FABIAN 2013-09-05 07:12:49 EDT
Fonts we have in Fedora 19 which do support Urdu:

mfabian@ari:~/bugs/redhat
$ fc-list :lang=ur | grep  usr/share
/usr/share/fonts/nafees-web-naskh/NafeesWeb.ttf: Nafees Web Naskh:style=Regular
/usr/share/fonts/paktype-naskh-basic/PakTypeNaskhBasicSindhi.ttf: PakType Naskh Basic Sindhi:style=Regular
/usr/share/fonts/sil-scheherazade/ScheherazadeRegOT.ttf: Scheherazade:style=Regular
/usr/share/fonts/paktype-tehreer/PakType_Tehreer.ttf: PakType Tehreer:style=Regular
/usr/share/fonts/paktype-naqsh/PakType_Naqsh.ttf: PakType Naqsh:style=Regular
/usr/share/fonts/nafees-nastaleeq/NafeesNastaleeq.ttf: Nafees Nastaleeq:style=Regular
mfabian@ari:~/bugs/redhat

DejaVu Sans does not support Urdu, only Arabic and Persian:

mfabian@ari:~/bugs/redhat
$ fc-list "DejaVu Sans:lang=ur"
mfabian@ari:~/bugs/redhat
$ fc-list "DejaVu Sans:lang=ar"
/usr/share/fonts/dejavu/DejaVuSansCondensed-Bold.ttf: DejaVu Sans,DejaVu Sans Condensed:style=Condensed Bold,Bold
/usr/share/fonts/dejavu/DejaVuSans.ttf: DejaVu Sans:style=Book
/usr/share/fonts/DejaVuSans.ttf: DejaVu Sans:style=Book
/usr/share/fonts/dejavu/DejaVuSans-Bold.ttf: DejaVu Sans:style=Bold
/home/mfabian/.fonts/nokia/dejavu/DejaVuSans.ttf: DejaVu Sans:style=Book
/usr/share/fonts/dejavu/DejaVuSansCondensed.ttf: DejaVu Sans,DejaVu Sans Condensed:style=Condensed,Book
mfabian@ari:~/bugs/redhat
$ fc-list "DejaVu Sans:lang=fa"
/usr/share/fonts/dejavu/DejaVuSansCondensed-Bold.ttf: DejaVu Sans,DejaVu Sans Condensed:style=Condensed Bold,Bold
/usr/share/fonts/dejavu/DejaVuSans.ttf: DejaVu Sans:style=Book
/usr/share/fonts/DejaVuSans.ttf: DejaVu Sans:style=Book
/usr/share/fonts/dejavu/DejaVuSans-Bold.ttf: DejaVu Sans:style=Bold
/home/mfabian/.fonts/nokia/dejavu/DejaVuSans.ttf: DejaVu Sans:style=Book
/usr/share/fonts/dejavu/DejaVuSansCondensed.ttf: DejaVu Sans,DejaVu Sans Condensed:style=Condensed,Book
mfabian@ari:~/bugs/redhat
$
Comment 7 Mike FABIAN 2013-10-28 11:11:24 EDT
Created attachment 816840 [details]
boxes-in-urdu-in-language-selection-page-TC6.png

The problem still exists in Fedora-20-Beta-TC6-x86_64-DVD.iso
Comment 8 Vratislav Podzimek 2013-11-01 14:17:17 EDT
DejaVu Sans font should support Urdu characters. Reassigning
Comment 9 Mike FABIAN 2015-04-21 01:02:50 EDT
dejavu-sans-fonts-2.34-4.fc22.noarch on Fedora 2 Beta still does not support Urdu.
Comment 10 Fedora End Of Life 2015-05-29 05:22:05 EDT
This message is a reminder that Fedora 20 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 20. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '20'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 20 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.
Comment 11 Paul Flo Williams 2015-05-29 09:30:06 EDT
Updating version to 22 based on comment 9
Comment 12 Nicolas Mailhot 2015-09-06 12:09:27 EDT
Sure, however to adding a whole new script can only happen upstream

In the meanwhile to actually fix the bug an urdu font needs to be added in anaconda (in default fonts) Scheherazade is probably a good choice – SIL fonts are high quality
Comment 13 Brian Lane 2015-09-08 14:18:47 EDT
Will simply adding sil-scheherazade-fonts.noarch to the boot.iso fix this? Or does something more need to be done to make sure it is used instead of dejavu-sans-fonts?

spin-kickstarts will need to add it as well so you should clone this bug for that after answering the above question.
Comment 14 Nicolas Mailhot 2015-09-08 14:56:28 EDT
I'm pretty sure it is sufficient for fontconfig substitutions to kick in and have something displayed in urdu

Will it be perfectly satisfactory ? Only urdu people can tell
Comment 15 Fedora Update System 2015-09-12 01:01:49 EDT
lorax-23.18-1.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2015-15675
Comment 16 Fedora Update System 2015-09-12 17:24:53 EDT
lorax-23.18-1.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.\nIf you want to test the update, you can install it with \n su -c 'yum --enablerepo=updates-testing update lorax'. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-15675
Comment 17 Mike FABIAN 2015-09-16 05:37:09 EDT
Created attachment 1073925 [details]
boxes-in-urdu-in-language-selection-page-fedora-23-beta-rc1.png

The problem still exists in Fedora-Workstation-netinst-x86_64-23_Beta.iso

Maybe Beta RC1 does not yet have the updated version of lorax?
Comment 18 Fedora Update System 2015-10-16 11:19:39 EDT
lorax-23.18-1.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.
Comment 19 Mike FABIAN 2015-10-18 14:50 EDT
Created attachment 1084177 [details]
font-mixture-in-urdu-in-language-selection-page-fedora-23-tc10.png

Mixture of "DejaVu Sans" and "Scheherazade" fonts. Ugly.
Comment 20 Mike FABIAN 2015-10-18 15:04:55 EDT
(In reply to Nicolas Mailhot from comment #14)
> I'm pretty sure it is sufficient for fontconfig substitutions to kick in and
> have something displayed in urdu
> 
> Will it be perfectly satisfactory ? Only urdu people can tell

What we see in Fedora 23 TC10 is now an ugly mixture of
Dejavu Sans and Scheherazade, see screenshot attached to:

https://bugzilla.redhat.com/show_bug.cgi?id=1004717#c19

DejaVu Sans is used first, all glyphs which are available in that font
come from DejaVu Sans. Scherazade is only used for the missing glyphs.

That surely looks quite ugly.

I am not sure whether it is readable, in Arabic script neighboring
glyphs "flow" together, probably that won't work well with two
different fonts.
Comment 21 Brian Lane 2015-10-19 12:11:33 EDT
I have no idea how to fix that, or even how it is expected to work. But it isn't a lorax problem. It is either going to be Gtk or Anaconda.
Comment 22 Mike FABIAN 2015-10-20 05:46:28 EDT
(In reply to Brian Lane from comment #21)
> I have no idea how to fix that, or even how it is expected to work. But it
> isn't a lorax problem. 

Yes.

> It is either going to be Gtk or Anaconda.

If the Gtk UI were started in Urdu locale, the fontconfig substitutions
would work to select a good font for Urdu. But the UI is already
running when one switches to Urdu in the language selection.
So Anaconda would need to switch locales somehow and re-initialize
the fonts.
Comment 23 Nicolas Mailhot 2015-10-20 14:59:02 EDT
(In reply to Mike FABIAN from comment #22)

> > It is either going to be Gtk or Anaconda.
> 
> If the Gtk UI were started in Urdu locale, the fontconfig substitutions
> would work to select a good font for Urdu. 

That's a simplistic solution. Of course it would work but only because system locale is the last-ditch fallback when the app is too dumb to tag text runs with the correct locale. The text stack is perfectly capable to mix text runs in different locales, it's a hard requirement to be able to edit multilingual documents (very common in a globalized world). Unicode does not solve everything, human scripts overlap, there is no way a system can reliably display unicode text without some form of locale tagging by the text author.

(this BTW is why windows i18n works, and *nix i18n does not, the windows input switcher was written for office. Those guys did manage complex human text and understood switching input means switching locales, with keyboard layout a locale option. While *nix people still think locale can be deduced from layout or text encoding, 20 years later, X11 mistake grandfathered in Wayland)
Comment 24 Mike FABIAN 2015-10-21 08:04:18 EDT
(In reply to Nicolas Mailhot from comment #23)
> (In reply to Mike FABIAN from comment #22)
> 
> > > It is either going to be Gtk or Anaconda.
> > 
> > If the Gtk UI were started in Urdu locale, the fontconfig substitutions
> > would work to select a good font for Urdu. 
> 
> That's a simplistic solution.

It is enough for Anaconda though.

> Of course it would work but only because
> system locale is the last-ditch fallback when the app is too dumb to tag
> text runs with the correct locale. The text stack is perfectly capable to
> mix text runs in different locales, it's a hard requirement to be able to
> edit multilingual documents (very common in a globalized world). Unicode
> does not solve everything, human scripts overlap, there is no way a system
> can reliably display unicode text without some form of locale tagging by the
> text author.

If you are writing a web page, you can of course mark text with
stuff like

<span lang="ja">直</span>

to make sure that a Japanese font is used for 直. At least in firefox
and probably most other browsers this works.

But for an installation in Urdu using Anaconda this is kind of
overdoing it.

What else except Urdu (Arabic Script) and some ASCII do you expect
to appear during an installation using Urdu?

If there is really some place during the installation where truly
multilingual stuff is displayed, we could think of using such markup
there. The only such place I can currently think of is the language
selection at the beginning where all language names are shown in their
native script. The fonts used for this might not be optimal for each
language at the moment.  Although no really serious problem has been
found in that page so far.
Comment 25 David Shea 2015-10-21 17:11:24 EDT
Is this a problem on other screens or just the welcome screen?
Comment 26 Mike FABIAN 2015-10-22 04:45:46 EDT
(In reply to David Shea from comment #25)
> Is this a problem on other screens or just the welcome screen?

I tried again and looked at the pages coming after the language selection.
Unfortunately there is so little translated text in Urdu there that it 
is hard to tell. There is some translated text in "Software Selection"and I 
found some where the root password can be entered.
I think we see a mixture of "DejaVu Sans" and "Scheherazade" there
as well. I’ll attach a screen shot in the next comment.

Anaconda does not change locales and reinitialize the font stuff
when proceeding from the welcome screen, or does it?
Comment 27 Mike FABIAN 2015-10-22 04:47 EDT
Created attachment 1085442 [details]
urdu-anaconda-f23-root-password-possibly-font-mixture.png
Comment 28 David Shea 2015-10-22 09:22:03 EDT
(In reply to Mike FABIAN from comment #26)

> Anaconda does not change locales and reinitialize the font stuff
> when proceeding from the welcome screen, or does it?

It resets $LANG and calls setlocale, but it does not do anything to re-initialize gtk's understanding of the locale because that is not possible.

Note You need to log in before you can comment on or make changes to this bug.