Bug 497984 - [qt] [pa_IN] Lohit Punjabi font showing empty square boxes in kde whereas same font works fine in gnome
[qt] [pa_IN] Lohit Punjabi font showing empty square boxes in kde whereas sam...
Status: CLOSED CURRENTRELEASE
Product: Fedora
Classification: Fedora
Component: fontconfig (Show other bugs)
rawhide
All Linux
medium Severity high
: ---
: ---
Assigned To: Behdad Esfahbod
Fedora Extras Quality Assurance
: i18n
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2009-04-28 05:07 EDT by A S Alam
Modified: 2013-07-02 20:51 EDT (History)
11 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2009-05-25 21:43:47 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
fontconfig-2.6.99-pa-in.patch (2.84 KB, patch)
2009-05-06 00:13 EDT, Kevin Kofler
no flags Details | Diff

  None (edit)
Description A S Alam 2009-04-28 05:07:47 EDT
Description of problem:
installed gnome desktop in maithili (mai_IN), later install punjabi-support (pa_IN).
and  tried to login gnome-desktop Punjabi, kde applications are not showing fonts, but square box

Version-Release number of selected component (if applicable):
kdebase-4.2.2-2.fc11.i586
qt-4.5.0-14.fc11.i586
fontconfig-2.6.99.behdad.20090318-1.fc11.i586
lohit-punjabi-fonts-2.3.8-1.fc11.noarch
kde-l10n-Punjabi-4.2.2-1.fc11.noarch

How reproducible:
Evertime

Steps to Reproduce:
1. add punjabi support (yum groupinstall punjabi-support)
2. logout and login Punjabi locale
3. run kate or kwrite
  
Actual results:
actually showing square box instead of font glyph

Expected results:
proper font glyph should shown

Additional info:
Comment 1 A S Alam 2009-04-29 03:53:38 EDT
Fresh install of rawhide 20090428 with kde desktop, then bug exist with kde
Comment 2 A S Alam 2009-04-30 04:13:06 EDT
All Applications for KDE is not working!
Comment 3 Rex Dieter 2009-04-30 07:42:27 EDT
OK, let's pull those over to qt then, hopefully someone there can take a look.
Comment 4 A S Alam 2009-05-01 22:09:17 EDT
1) while using only qt application (qtconfig-qt4) for check font, it also have problem.
1) Interface (GUI) can be used in english, but input is more important, which is also square box only
Comment 5 A S Alam 2009-05-04 02:48:05 EDT
More information:
qt-4.5.0-14.fc11 - Not working  
qt-4.5.0-14.fc10 - Working

where only Build tag is different (fc11 or fc10).
Can it be depends upon something else?

2) qt-4.5.0-12.fc11 also has problem

3) various language tested (zh_CN, or_IN, ja_JP, ml_IN, ur, kn_IN, ta_IN), all those working fine.
Comment 6 Kevin Kofler 2009-05-04 06:47:35 EDT
Suspecting a miscompilation by GCC 4.4.
Comment 7 Parag Nemade 2009-05-04 07:19:40 EDT
sorry but I didn't get meaning of miscompilation here.
Comment 8 Ngo Than 2009-05-04 08:28:11 EDT
we have new fontconfig in rawhide (F-11), it causes this issue
Comment 9 Kevin Kofler 2009-05-04 17:06:22 EDT
Are we sure it's not a graphics driver issue, e.g. https://bugzilla.redhat.com/show_bug.cgi?id=495323 ?
Comment 10 A S Alam 2009-05-04 21:23:25 EDT
I tested those on 3 machines with Intel, ATI and nvidia graphics and issue remains same. So it may not be graphcis driver. will update about fontconfig today
Comment 11 Jens Petersen 2009-05-05 02:35:40 EDT
(In reply to comment #10)
> will update about fontconfig today  

Thanks - if you could also test F11 Beta and Preview that would be helpful to know when this regression started.
Comment 12 A S Alam 2009-05-05 02:47:10 EDT
ok, Beta Test.
Issue was present in F11-Beta with following  version:
qt-4.5.0-3.fc11
fontconfig-2.6.99.behdad.20090318-1.fc11
Comment 13 Jens Petersen 2009-05-05 03:39:21 EDT
Is someone could check 11-Alpha too might be good - we don't seem to have a copy here, otherwise I can download F11Alpha Kde Live.
Comment 14 A S Alam 2009-05-05 05:44:53 EDT
ok, Alpha-11 Results are here (working fine):
qt-4.4.3-14.fc11
fontconfig-2.6.95-1.git.66.gb162bfb.fc11
lohit-fonts-punjabi-2.3.1-1.fc10
Comment 15 Kevin Kofler 2009-05-05 09:15:29 EDT
Hmmm, a lot of things different there: Qt was 4.4.3, it's 4.5.0 in Beta, fontconfig was 2.6.95, was 2.6.99 in Beta and it's also before the mass rebuild with GCC 4.4.

> sorry but I didn't get meaning of miscompilation here.

Miscompilation = the C/C++ code gets compiled into invalid assembly due to a GCC bug. If that's indeed what's happening, it'd explain why the same build works on F10.

Has anybody tried using the F10 qt-x11-4.5.0-14.fc10 build on F11 (in particular, with the F11 fontconfig)? That said, I think that won't easily work because OpenSSL has a different soname in F11 than F10.
Comment 16 Ngo Than 2009-05-05 10:44:54 EDT
It's neither graphics driver nor miscompilation by GCC 4.4. It's a regression in fontconfig in F11 that causes this problem. I fixed the pa orthography file and the problem is gone.

The new package was built in F11-scratch, you can download from the link.
http://koji.fedoraproject.org/koji/taskinfo?taskID=1336937
Comment 17 Parag Nemade 2009-05-05 11:45:37 EDT
yes. fontconfig got many orth files splitted and added new orth files in recent fontconfig releases.
so does Punjabi also got splitted from pa.orth to pa_IN.orth and pa_PK.orth.
Not sure what where goes wrong in pa orthography. But pa.orth from fontconfig-2.6.95 is same as http://cgit.freedesktop.org/~behdad/fontconfig/tree/fc-lang/pa_in.orth. And also from  http://cgit.freedesktop.org/~behdad/fontconfig/commit/?id=4d7412a28b834830d0d1749852115846b3554932, it looks nothing got changed in pa.orth.
Will check scratch build rpms tomorrow.
Comment 18 Behdad Esfahbod 2009-05-05 11:49:18 EDT
This is ridiculous.  You first say everything other than English renders to squares.  Then it turns out it's just one language not working.

Someone please give me an update of what the real issue is, before I can take any action.  The fact that GNOME works, but not KDE makes it hard to qualify it as a fontconfig bug, even if the issue was introduced by a fontconfig change.

Finally, please do a "fc-cache -f" as root and test again.
Comment 19 Parag Nemade 2009-05-05 12:04:20 EDT
AFAICS, This bug exists for Punjabi language in KDE only.
Comment 20 Jens Petersen 2009-05-05 19:00:40 EDT
Aalam and Parag, can you please test Than's build.
Comment 21 Parag Nemade 2009-05-05 23:26:04 EDT
I have tested Than's build and looks Punjabi is working fine. But still question for me remains what change required in orthography file?
Comment 22 A S Alam 2009-05-05 23:51:26 EDT
(In reply to comment #18)
> This is ridiculous.  You first say everything other than English renders to
> squares.  Then it turns out it's just one language not working.
> 
have you read description? Punjabi (pa_IN) is ONLY mentioned language 
with Problem.

> Someone please give me an update of what the real issue is, before I can take
> any action.  The fact that GNOME works, but not KDE makes it hard to qualify it
> as a fontconfig bug, even if the issue was introduced by a fontconfig change.
> 
Than's build working fine!
Thanks Than for help.

> Finally, please do a "fc-cache -f" as root and test again.  

This did not work. forgot to mention before, sorry.
Comment 23 Kevin Kofler 2009-05-06 00:12:40 EDT
You can get the SRPM with Than's fix here:
http://koji.fedoraproject.org/koji/getfile?taskID=1336938&name=fontconfig-2.6.99.behdad.20090318-2.fc11.src.rpm

See the fontconfig-2.6.99-pa-in.patch in that SRPM. Looking at it, he simply replaced the new pa-in.orth by the original 2004 version of pa.orth.
Comment 24 Kevin Kofler 2009-05-06 00:13:37 EDT
Created attachment 342592 [details]
fontconfig-2.6.99-pa-in.patch

Here's the fontconfig-2.6.99-pa-in.patch from Than's SRPM.
Comment 25 Jens Petersen 2009-05-06 00:16:26 EDT
I see, thanks - however that still doesn't really answer Behdad's question on why his current fontconfig is breaking pa in KDE.
Comment 26 Jens Petersen 2009-05-06 00:17:50 EDT
Rahul, could you review the orth changes to see why it would be breaking in qt?
Comment 27 Kevin Kofler 2009-05-06 00:25:46 EDT
Actually, I compared the file Than's patch added with the new one:
http://cgit.freedesktop.org/~behdad/fontconfig/tree/fc-lang/pa_in.orth
They're identical. (I got confused by the copyright notice, but it's still 2004 even in the new file.)

So Than's patch doesn't actually revert any changes to the .orth file itself, all it does is rename pa_in.orth to pa_orth. The file name appears to be the magic.
Comment 28 Kevin Kofler 2009-05-06 00:30:35 EDT
Also look at the upstream commit introducing the change:
http://cgit.freedesktop.org/~behdad/fontconfig/commit/?id=4d7412a28b834830d0d1749852115846b3554932

There were no semantic changes between the old pa.orth and the new pa_in.orth, it's the rename which Qt chokes on.
Comment 29 Kevin Kofler 2009-05-06 00:41:45 EDT
See the languageForWritingSystem hack in this file:
http://websvn.kde.org/trunk/qt-copy/src/gui/text/qfontdatabase_x11.cpp?revision=934499&view=markup
for what I suspect to be the source of the breakage.

In short, AFAICT (and if I'm not mistaken), this tries to look up "pa" in fontconfig, and if the lookup fails, it concludes Gurmukhi script is not supported.

Should this be patched to use "pa_IN", "pa_in" or "pa-in" instead of "pa" in that lookup table?
Comment 30 Kevin Kofler 2009-05-06 00:42:48 EDT
Setting NEEDINFO on Behdad, but maybe one of the i18n folks is also qualified to answer this.
Comment 31 Parag Nemade 2009-05-06 00:49:26 EDT
(In reply to comment #28)
> Also look at the upstream commit introducing the change:
> http://cgit.freedesktop.org/~behdad/fontconfig/commit/?id=4d7412a28b834830d0d1749852115846b3554932
> 
> There were no semantic changes between the old pa.orth and the new pa_in.orth,
> it's the rename which Qt chokes on.  

Kevin,
  Thanks. Thought its scratch build and I will not able to see SRPM. Actually,
I already got that thing that the only way to have Than got working his build
is to revert back to pa.orth from pa_IN.orth.
As I already given my analysis on comment #17, pa.orth and pa_IN.orth are
identical. So question remains now where the problem exists??
Comment 32 Parag Nemade 2009-05-06 01:04:44 EDT
(In reply to comment #29)
> See the languageForWritingSystem hack in this file:
> http://websvn.kde.org/trunk/qt-copy/src/gui/text/qfontdatabase_x11.cpp?revision=934499&view=markup
> for what I suspect to be the source of the breakage.
> 
> In short, AFAICT (and if I'm not mistaken), this tries to look up "pa" in
> fontconfig, and if the lookup fails, it concludes Gurmukhi script is not
> supported.

I think it should succeed as
[parag@rawhide ~]$rpm -q --provides lohit-punjabi-fonts 
font(:lang=pa)  
font(lohitpunjabi)  
lohit-fonts-punjabi = 2.3.8-1.fc11
lohit-punjabi-fonts = 2.3.8-1.fc11

> 
> Should this be patched to use "pa_IN", "pa_in" or "pa-in" instead of "pa" in
> that lookup table?
Comment 33 Kevin Kofler 2009-05-06 01:07:42 EDT
> So question remains now where the problem exists??

Please see comment #29: search for languageForWritingSystem in the linked file (which is src/gui/text/qfontdatabase_x11.cpp from Qt 4.5.1, but that code has been in Qt for ages) and I think you'll realize what's going on... At least I'm fairly sure the problem is there (it looks for "pa", so if fontconfig only knows "pa_IN", it won't find it and conclude the Gurmukhi script is not available).

The RPM Provides are irrelevant, it's what fontconfig finds on lookups which is important. (And rebuilding the font package might actually change the Provides.)
Comment 34 Parag Nemade 2009-05-06 01:15:04 EDT
(In reply to comment #33)
> > So question remains now where the problem exists??
> 
> Please see comment #29: search for languageForWritingSystem in the linked file
> (which is src/gui/text/qfontdatabase_x11.cpp from Qt 4.5.1, but that code has
> been in Qt for ages) and I think you'll realize what's going on... At least I'm
> fairly sure the problem is there (it looks for "pa", so if fontconfig only
> knows "pa_IN", it won't find it and conclude the Gurmukhi script is not
> available).
   Yes. I see that code. And you are right that code has been in Qt for ages. I even tested few new Indic languages based on Devanagari script successfully in QT where that code file has no mention for those new Indic languages like maithili language.
   Ok. your point looks also right that it looks for "pa" and fontconfig currently knows only pa_IN.

> 
> The RPM Provides are irrelevant, it's what fontconfig finds on lookups which is
> important. (And rebuilding the font package might actually change the
> Provides.)  

Ok. Lets wait for Than and Behdad then.
Comment 35 Behdad Esfahbod 2009-05-06 19:39:00 EDT
Yep,  that's it.  Very ugly.  I suggest patching Qt and filing a bug upstream.
Comment 36 Kevin Kofler 2009-05-06 19:42:06 EDT
What's the correct way to look up the language? pa_IN (the locale's official name)? pa_in (the name of the .orth file)? Or pa-in (like zh-cn and zh-tw which appear to work)?
Comment 37 Behdad Esfahbod 2009-05-06 20:07:02 EDT
"pa-in" is fine.  I'll think about this later and see if renaming it back in fontconfig makes sense.  Here's the bug that introduced the change:

http://bugs.freedesktop.org/show_bug.cgi?id=19890
Comment 38 Jens Petersen 2009-05-06 20:29:13 EDT
So pa_IN.orth should be renamed to pa-in.orth?
Comment 39 Behdad Esfahbod 2009-05-06 20:35:34 EDT
No.  Either Qt patched to use "pa-in" instead of "pa", or pa-in.orth renamed back to pa.orth.
Comment 40 Kevin Kofler 2009-05-06 20:43:28 EDT
Maybe renaming it back in fontconfig is the best solution? This is a compatibility issue, and while I can easily patch our Qt, it won't help for third-party stuff statically linked against upstream Qt.
Comment 41 Kevin Kofler 2009-05-06 21:59:37 EDT
FYI, Qt 3 appears not affected, no hardcoded languages there:
http://websvn.kde.org/branches/qt/3.3/qt-copy/src/kernel/qfontdatabase_x11.cpp?revision=763238&view=markup
But that's not surprising, as Qt 3 doesn't properly support Indic and other hard-to-render languages at all.

So this is only about Qt 4. (But most of the Qt stuff nowadays is Qt 4, of course.)
Comment 42 Parag Nemade 2009-05-06 23:27:14 EDT
(In reply to comment #39)
> No.  Either Qt patched to use "pa-in" instead of "pa", or pa-in.orth renamed
> back to pa.orth.  

I see its pa_in.ort in fontconfig. Correct me if I am wrong.
Comment 43 Parag Nemade 2009-05-06 23:34:46 EDT
(In reply to comment #42)
> (In reply to comment #39)
> > No.  Either Qt patched to use "pa-in" instead of "pa", or pa-in.orth renamed
> > back to pa.orth.  
> 
> I see its pa_in.ort in fontconfig. Correct me if I am wrong.  

sorry should be read as pa_in.orth.
Comment 44 Kevin Kofler 2009-05-07 00:04:13 EDT
Patched qt here (4.5.0-15.fc11):
http://koji.fedoraproject.org/koji/buildinfo?buildID=101155

Should we get this tagged into dist-f11 (if it's not too late yet)?
Comment 45 Parag Nemade 2009-05-07 00:11:37 EDT
Ahh! I already have updated qt-4.5.1 from build http://koji.fedoraproject.org/koji/buildinfo?buildID=99514

Do I need to forcefully remove qt rpms to have your build installed in f11?
Comment 46 Kevin Kofler 2009-05-07 00:14:23 EDT
You can use rpm -Uvh --oldpackage to downgrade.

I could also build a new 4.5.1-based build, but 4.5.0-15.fc11 is the most likely candidate to make it into dist-f11.
Comment 47 Kevin Kofler 2009-05-07 00:23:37 EDT
I submitted a qt-4.5.1-3.fc11.1 build too: http://koji.fedoraproject.org/koji/taskinfo?taskID=1340046
(still running).

Most likely we'll also want to sync the changes from Rawhide, but at this point I wanted to get a build with what's currently in F-11 CVS first.
Comment 48 A S Alam 2009-05-07 00:54:45 EDT
Bug fixed with 4.5.0-15.fc11 package.
Thanks
Comment 49 Kevin Kofler 2009-05-07 01:22:34 EDT
Tag request for qt-4.5.0-15.fc11 here:
https://fedorahosted.org/rel-eng/ticket/1754

So, should we treat this as fixed or is a fontconfig change (back to pa.orth) still being considered?
Comment 50 A S Alam 2009-05-07 01:29:59 EDT
fontconfig (which is build  by than) need to rollback. I used rawhide fontconfig
(fontconfig-2.6.99.behdada.20090318-1.fc11) then only this build package
(qt-4.5.0-15.fc11) was working.
Comment 51 Parag Nemade 2009-05-07 01:44:46 EDT
(In reply to comment #46)
> You can use rpm -Uvh --oldpackage to downgrade.
> 
> I could also build a new 4.5.1-based build, but 4.5.0-15.fc11 is the most
> likely candidate to make it into dist-f11.  


Thanks. That worked. Tested successfully fix for this bug using qt-4.5.0-15.fc11 and fontconfig-2.6.99.behdada.20090318-1.fc11

(In reply to comment #49)
> Tag request for qt-4.5.0-15.fc11 here:
> https://fedorahosted.org/rel-eng/ticket/1754
> 
> So, should we treat this as fixed or is a fontconfig change (back to pa.orth)
> still being considered?  

Fixed in qt. no change needed to fontconfig for now.

Thanks for tag request. Added comment there for tagging F11-final.
Comment 52 Kevin Kofler 2009-05-07 01:47:27 EDT
Re comment #50: I'm not sure I understand you. Do you mean that the patched Qt will NOT work with a fontconfig which uses pa.orth? (I need to know this because it means I have to conditionally apply the patch only on F11 and above.)
Comment 53 Parag Nemade 2009-05-07 02:02:19 EDT
1)The fontconfig build I tested is fontconfig-2.6.99.behdad.20090318-1.fc11.x86_64

2) I checkout its source and looked into fontconfig-2.6.99.behdad.20090318/fc-lang/

3) I found following files starting pa_*
pa_in.orth  pa_pk.orth  

4) so with this new qt-4.5.0-15.fc11 build we are using pa_in.orth to see Punjabi font.

And yes the patched Qt will NOT work with a fontconfig which uses pa.orth.
Actually when you gave this new qt build, I was already using Than's fontconfig build. Then I found after updating qt to 4.5.0-15.fc11, I still see empty square boxes not only while giving input in Punjabi but also its translated strings in kate.
Comment 54 Kevin Kofler 2009-05-07 02:08:32 EDT
Ugh, this means there's no way that patch can be upstreamed. Qt needs to support older fontconfig as well. Maybe a safer solution would be to use the "sample character" trick Qt is using for some other languages?

At this stage I'm not sure tagging that patched Qt build into dist-f11 is a good idea at all, I wonder if I shouldn't withdraw my tag request.
Comment 55 Kevin Kofler 2009-05-07 02:11:52 EDT
(I think fontconfig should change to pa.orth instead, and ASAP, to prevent stuff from starting to expect the new name and breaking when it's changed back.)
Comment 56 Kevin Kofler 2009-05-07 04:26:31 EDT
What further confuses these matters is that most speakers of Punjabi are in Pakistan, but making pa_pk.orth the default pa.orth is also going to break Qt, as Qt expects a lookup of "pa" in fontconfig to find the Gurmukhi script, whereas pa_PK is written in the Arabic script.
Comment 57 Parag Nemade 2009-05-07 04:46:15 EDT
Thats what I already got and dunno what to say. But afaik pa is actually pa_IN and not pa_PK. I need someone to confirm me.
Comment 58 Ngo Than 2009-05-07 05:48:42 EDT
the patch breaks the compatibility with old fontconfig. I don't think it's good idea to have it in qt. IMO it's a regression in fontconfig and should be fixed in fontconfig.

Kevin could you please withdraw your tag request?
Comment 59 Kevin Kofler 2009-05-07 06:00:02 EDT
Tag request withdrawn. Bouncing back to fontconfig.
Comment 60 A S Alam 2009-05-07 08:31:26 EDT
Unicode Bug:
http://www.unicode.org/cldr/bugs/locale-bugs?findid=1300

said 
"Change the default content in the metadata so that pa_IN is the default content
for pa."

so 'pa' can be just link or copy of 'pa-IN' safely.
Comment 61 Behdad Esfahbod 2009-05-08 14:32:33 EDT
Update on this.  No qt changes needed.  I'll fix in fontconfig.
Comment 62 Behdad Esfahbod 2009-05-08 16:39:47 EDT
Built fontconfig-2.6.99.behdad.20090508-1.fc11.  Can someone request tagging it please.
Comment 63 Matthias Clasen 2009-05-08 18:04:12 EDT
Tag request is here: https://fedorahosted.org/rel-eng/ticket/1767
Comment 64 A S Alam 2009-05-25 21:42:36 EDT
fontconfig-2.6.99.behdad.20090508-1.fc11.x86_64 Package is available and FIXED the issue.

Note You need to log in before you can comment on or make changes to this bug.