Created attachment 155867 [details]
Screenshot of malaylam page in firefox

have you same problem in konqueror and gedit?

Created attachment 155868 [details]
Screenshot of malayalam page in Konqueror and Gedit

When compared with konqueror, firefox has less errors. But it's interesting to
see that chillu errors are solved in konqueror but there are lots of
koottaksharam errors.

ok, so can you please open another bug for qt (rendering) for koottaksharam?

and yes, are you please add unicode for those combinations like

0d0c + od4d+????

you can use Malayalam Unicode chart from:
http://www.unicode.org/charts/PDF/U0D00.pdf

I am not an expert in unicode and has less knowledge about how the glyphs are
stored. However I will try to get help from some experts about this issue.

Lets look at the issues one by one. 

1) Not rendering chillu properly
This has got to do with wrong input and not a problem with rendering
Earlier pango used to render consonent+chandrakala without ZWJ as chillu now
when it is corrected text entered earlier has also need to be corrected.

ള+്+ZWJ - ള്‍
2) കീഴ്വഴക്കള്‍ is due to a wrong sequence selected in Microsoft Open Type
specification
It can be fixed using suruma patch for pango ( http://suruma.sarovar.org ) and a
font modified to work with it.

3) ഉള്‍പ്പെടുന്ന This is bug in pango that need to be fixed
http://bugzilla.gnome.org/show_bug.cgi?id=441654

For now it can be worked around by adding a ZWNJ after ള്‍
ള+്+ZWJ+ZWNJ+പ+്+പ

4) ണു് samvruthokaram is not yet supported. I guess it has to be supported at font
level.

5)  പ്രത്യേകത refer 3

6) ന്രോ the problem is bug introduced by Fedora developers to fix rendering of
lohit_ml new lipi font by adding an exception and which broke the traditional
lipi rendering.

http://bugzilla.gnome.org/show_bug.cgi?id=357790

It could be solved by including popular traditional lipi fonts like Rachana and
Anjali to Fedora

Most of the issues except for 3 can be solved by using suruma patched pango.

I have some more screenshots using the rachana and anjali fonts. The anjali
seems to have less errors in firefox. The anjali fonts can be downloaded from :

https://savannah.nongnu.org/forum/forum.php?forum_id=4730
http://varamozhi.wikia.com/wiki/Varamozhi

Created attachment 155873 [details]
Screenshots of firefox and konqueror using Rachana and Anjali

The upstream bug:
http://bugzilla.gnome.org/show_bug.cgi?id=481198

*** Bug 379431 has been marked as a duplicate of this bug. ***

*** Bug 382121 has been marked as a duplicate of this bug. ***

LingNing, can you take a look at this, please?

Jens, ok, I will look at it.

I have tested a suruma patch for pango
(http://suruma.sarovar.org/patch4pango-1.16.4.tar.bz2) in Fedora 8 with
lohit-malayalam fonts and Rachana_g02 fonts. There are considerable issues when
using lohit-fonts even with suruma patch. Please compare the two screenshots
attached. 

Created attachment 265721 [details]
Malayalam in firefox using suruma-pango-patch and lohit-malayalam font

Created attachment 265731 [details]
Malayalam in firefox using suruma-pango-patch and Rachana-g02 font

I have discussed the problems with Ani and tend to agree with most of the points
made by Pravin in Comment #8. 
1. Chillu rendering: It used to work in fedora even with  ZWJ. There seems to be
some changes in Pango that has broke the rendering. Chillu problems are also
being addressed in Unicode 5.1 .0. Thus we have to wait for the final verdict on
that.

2. ഴ + ്  + വ: This is rendered wrong since this is a special case where post
base Va is not used. With most other consonants we need to have post base form
of Va. This can be hanlded in the font, without changing the post base flag of
Va in Pango.

3. ഉള്‍പ്പെടുന്ന: This  problem exists due to some reorderings done by pango which
may be neccessary in the rendering of chillu characters. For time being (and may
be even forever) we can use ZWNJ as suggested by Pravin. This is already being
used in the gnome and fedora translations.

4. ണു് : This cannot be supported in font since pango will not allow use of halant
after vowel. 

5. consonant +  ് + യ + ക: This  is most probably a font issue and  can be
solved. Ideally font should have pstf flag for half ya and so should be in the
pango. Additionally I see one 'akhn' combination of ka and ya which may be
affecting this.

6. ന്രോ: This issue is still under discussion. This will be done in both font and
 pango. 
Suruma pathes and Rachana font supports only old lipi in this case. We are
working on a dual solution where both old and new lipi will be supoprted.

1. Chillu Rendering: Whatever be the outcome of Unicode 5.1 we need to support
data which is already using ZWJ for chillu. I broke it in Pango by mistake when
I reverted your changes (ZWJ is ignored now which should not be the case). We
should be able to incorporate this back when we propose a patch to solve all the
existing issues. Even now if you can add this back, I can support the patch
saying it was my mistake in the first place to have removed that flag.
3. I would rather fix the issue than use ZWNJ in this case. My earlier comment
to use ZWNJ was before seeing icu handling this correctly. Also it is not the
correct way to use wrong encoding to work around rendering bugs.
4. This has to be supported in pango and suruma patch gives a solution to this.
If you can review it and if you like it we can submit it upstream (we don't want
to cut our language to fit the rendering engine anymore - and windows renders
this correctly).
5. I don't understand the requirement for supporting the typewriter script
anymore. Can you suggest any logical reasons to continue with typewriter script
now that the technology can handle the tradition script without needing to
cutting down the script? This is a language and cultural issue rather than a
technical issue. If you look around the Malayalam users you can see everyone
following the traditional script. 

Rendering is such a fundamental requirement to have a localized system and we
have been sitting on this for a long time even with a solution to most of these
problems already. We have to address this quickly.

Even while we debate on the typewriter/traditional script issue we can push
patch for adding back ZWJ processing and samvruthokaram that does not affect the
script difference.

Its possible to support both ZWJ and unicode 5.1. We will have to  include the
chillu glyphs at the designated code points and apply GSUB on the same glyph in
such a way that it can be produced as a combination with ZWJ as well.

Please correct the ZWJ mistake, I will certainly support that and also the
samvruthokaram patch looks ok.

Rahul, Manilal,

I have submitted the patch to render chillus and samvruthokaram upstream.

http://bugzilla.gnome.org/show_bug.cgi?id=504810

Please add your comments there and support it.

Rahul,

Lets stop fighting over typewriter script and traditional script. Lets find a
way that is agreeable to both of us and that will solve most important rendering
issues we have now.

AFAIK, you don't agree with suruma patch because it does not render conjuncts
with ra like പ്ര (pra) in typewriter script. We don't agree with you changing the
order in pango for making this work in typewriter script while breaking
traditional script.

Can we find a middle way in which suruma patch can render 'pra' correctly in
typewriter script as well?

I remember you mentioning a way to do it by changing fonts so that both scripts
work. Can you get us that?

Samvruthokaram patch for icu. Please comment here too.
http://bugs.icu-project.org/trac/ticket/6108

Good work Praveen. I will surely test it.

Manilal, Rahul,

Please test the Open Office Version from
http://download.savannah.nongnu.org/releases/smc/fedora/8/smc.repo

and see if any of the issues still remain. We still need some more fixes to
perfect pango rendering but Open Office is perfect with all known issues resolved.

Suresh is working on a way to solve the issue with typewriter script, he has
already solved it for qt. I will post it here after testing it.

Praveen,
I am having some issues with the Fedora desktop at my office. I will fix it
today and test the patch ASAP.

I have installed the pango and libicu patch from smc repository in Fedora 8. I
think there are still issues. However I tested it with lohit-malayalam-fonts.
I'm not sure whether I need to test it with any other fonts. I have included the
screenshots of firefox and openoffice, before and after applying the updated
packages from smc. 

Please let me know if I need to consider any other test cases. I am using Xen
virtual guest, so that I need not worry about any package corruption.

Created attachment 290346 [details]
Screenshots of openoffice and firefox  using tha updates from smc

No, it won't work with Lohit fonts without modification. Test with Rachana_g02
and MalOtf.ttf (from
http://ftp.de.debian.org/debian/pool/main/t/ttf-indic-fonts/ttf-indic-fonts_0.5.0.tar.gz)

I have tested Openoffice.org and Firefox with ttf-indic-fonts and also
Rachana_g02. It seems that different fonts has different isues. I have attached
a detailed list of screenshots of various fonts.

Created attachment 290403 [details]
Screenshots of openoffice and firefox using updates from smc

First of all thanks for testing it.

How did you create the tar ball? It does not open normally, you have gzipped it
twice, so first you have to run gunzip, change the .tar file to .tar.gz and then
do a tar -zxvf

Firstly, the fonts which you have tested are different

1) Rachana_w01 is designed to work with Microsoft Uniscribe rendering engine and
it will not render correctly in pango or icu.
2) To fix the rendering issues Suresh developed patches for pango and icu called
suruma and required changes in fonts also, which means these becomes
incompatible with Uniscribe/Windows and does not work well for typewriter script
(conjuncts of ra). These are marked g01
3) Rahul Bhalerao has made changes to icu and pango to make Lohit render
correctly which breaks Rachana and other traditional scripts
4) Now Suresh has made the fonts compatible with with both suruma and uniscribe,
these are the g02 fonts. He has fixed the MalOtf to render correctly with suruma
patch (I should have given this font instead of the one from debian package - I
have attached the fixed MalOtf)

So for fixing all the rendering problems you have to follow these steps

a) update the icu and pango packages to the latest version in smc repository
b) Install Rachana_g02 and MalOtf (both attached here).

Now if Lohit follow the scheme of MalOtf then it will work with suruma patch.
Once we have both the traditional and typewriter scripts work correctly with
suruma patch, I think Rahul will agree to push these upstream.

Created attachment 290545 [details]
MalOtf made to work with suruma patch

Created attachment 290546 [details]
Rachana_g02

Praveen,
 Sorry, I was in a hurry and created the tarball using the Archive Manager and
not sure how it gone bad.. Anyway I will attach a new file. I will test it
tomorrow morning and post the results.

Created attachment 290569 [details]
Screenshots of openoffice and firefox using updates from smc

Thanks Praveen and Suresh for your efforts. I have tested the updated packages
from SMC repository with Mal0tf and rachana_g02 fonts in Fedora 8 and it works
fine. However I noticed an issue when using the Mal0tf fonts in FireFox. The
word അന്താരാഷ്ട്ര is not rendered correctly. Please see the attachment. I think
these patches should be applied upstream ASAP.

Created attachment 290646 [details]
Screenshots of openoffice and firefox using pango and icu patches from SMC.

The fonts used are Mal0tf and Rachana_g02

Pravin, your message in comment #23 confused me a bit. In the second paragraph,
you say you don't agree to my solution, while in the last line, you are asking
me my suggestion on the changes required in font to support the old script.

As of now, I have only one tested solution for this. Make the reordering in
pango to support typewriter script and make changes in old script font to suit
the reordering. 

There is one more solution hopping in my mind but yet to be tested. On uniscibe,
the typewriter script font uses contextual chain subs (instead of reordering) to
implement 'pra' kind of ligatures. Although OpenType specs for
typewriter(reformed) malayalam script suggest 'reordering at the start of
syllable' for this 'post base' ligature, I expect the old script font to work
out of box for this arrangement, but not sure (I have not tested any windows
machine for this). 

I think pango still has problem handling contextual chain subs for indic
scripts. If this can be implemented, all the fonts would work seamlessly on all
the platforms. I think Behdad can inform and help more on this.   

Btw, supporting both the scripts is a high priority for me too :)

So lets do it the right way. If contextual chain subs are the ways to lets do
it. I will see if I can find someone ready to work on this if Behdad is ready to
mentor.

The important thing is fixing the rendering issues.

1) suruma patch fixes most of the issues
2) If you provide both suruma style and uniscribe style conjuncts for va and ya,
it works  with Vista, suruma patched pango and qt. That would mean we can
maintain one font that works in Vista and pango (if Microsoft is ready to fix
XP, then there also, but we can't do anything there).
3) At least that gives a one-direction compatibility it is better than
incompatibility 

So my choices/priority

1) fix pango  to render fonts the way uniscribe does - difficult and we don't
know how much work is needed. If contextual chain subs are the way to I'm all
for it.
2) suruma way - we know it works - one way compatibility
3) break away from uniscribe and come up with a standard for Free Software
implementations and fix the issue once and forever.

The important thing is we have to get it out sooner than later and the fastest
way is suruma approach.

For breaking away from Uniscribe, we need to find the real problems in
uniscribe. I have already documented few ambiguities and skipped reorderings in
uniscribe. 

About suruma, I am not confident about the way it handles Ya and Va. If it needs
to use 'akhn' tag, then I am not with it. Both of these are having post-base
forms and it should be retained.

Problem is, I don't know the Indic shaping rules or model at all.

What I do know is that we really, really should stay as close as possible to
Uniscribe.  There's no point in implementing OpenType shaping afterall if we
have different requirements.

The way suruma handles Ya and Va works well with Uniscribe version in Vista. We
have a solution with 'akhn' tag and we know it fixes many issues. We are not
very particular about it. Unless someone commits to fix it without going the
'akhn' we need to use this as it is much worse to leave it broken.

1) Use 'akhn' as in suruma patch - we know it fixes many serious rendering bugs
and it is used by everyone in the Swathanthra Malayalam Computing Community.
2) Some one has to commit to fix it without using 'akhn'. We can't leave it
broken as it is now. So if you are not with 'akhn' you have to fix it with
post-base or get someone to commit it. If Red Hat or someone else who wants
uniscribe compatibility can commit resources we have no objections to it.

I am repeating our priority here again. WE CANNOT LEAVE IT BROKEN.

I started with suruma font with a patch for pango which then had problems with
chillu and post base forms.The chillu issues is solved by turning on ZWJ
processing for Malayalam in the pango and by adding  ZWJ in the chillu GSUB
entry in the font.I used 'akhn' for for the post base forms which solved the
issues with the post base forms.

After that, I implemented my idea in the Rachana font for solving the shaping
issues.In the g02 and later versions of the font I retained the uniscribe
features too which means the font can also be used with the qt, unpatched.

I think it is better not to use post/below base forms of RA and LA for xRA and
xLA ligatures.Because there are some consonants whch can't take the isolated
post/below base forms of LA and RA.Eg. YA + RA, YA + LA. RA+RA etc.So for the
reformed script we may have to design some glyphs for xRA and xLA ligaures.It is
worth the effort because we get better and desired rendering.Moreover, AFAIK,
none of the non-uniscribe shaping mechanisms make use of the Language System
Tags from the font.(MAL for Malayalam traditional script and MLR for Malayalam
reformed script)

Now the issues with post base YA and VA forms still persist with pango(like the
'consonant +  ് + യ + ക' shaping issue).Recently I have found a workaround for
this at the font level which I will be using in the next versions of
Rachana/Meera fonts.

This is really exciting. As now only issue 3 from Comment #8 is left unresolved
(as per Suresh's last comment).

Behdad, g03 fonts (designed for suruma patch with 'akhn' tag for Ya and Va) work
with Uniscribe version in Vista and QT.

Rahul, from Suresh's last comment we can solve the issue with xRa by adding some
more conjuncts in Lohit font (MalOtf, another popular MLR font, already uses
this method). What do you think?

Behdad,

I really appreciate your view that we should try to be as close to uniscribe as
possible. Things do not have so many problems there.

Only way to achieve that would be to follow the OpenType specs themselves. 
Considering the issues around Ya (0D2F), Ra (0D30), La (0D32) and Va (0D35) I
would again refer to the Appendix B about the consonant forms at:
http://www.microsoft.com/typography/otfntdev/indicot/appen.aspx

It is clearly mentioned that Ya and Va both have Post Base forms in both
old(MAL) and new(MLR) scripts. Whereas La has Below Base form in both the scripts.
If we follow this strictly, I do not think there would be anything else breaking
around it. (IMO the 'akhn' tag has completely different objective and should not
be mixed here even if it works for few cases.)

About 'Ra', I still find the description in the above document to be ambiguous.
Yet, I am open to learn the trick used in Uniscribe for this. If no one succeeds
in doing that, then and only then we should follow a different approach.

Praveen,
My observations about 'akhn' for Ya and Va vary a lot. Firstly, they did not
work on the uniscribe in XP.
As far as I know, there aren't any changes about these conjuncts in Vista. In
any way, 'akhn' tag used on anything that is reordered internally will override
the natural sequence of the characters in other cases. e.g. try the plain
sequence of 0D4D+0D2F (്യ) without any initial consonant. On pango with
Rachana_g02, this resulted in formation of the post-base Ya glyph which is
wrong. Try testing something like (Ka+Halant+Va)+Ya. I would expect 'akhn' Ya
will break the initial cluster of KaHalantVa. (These are only examples of what
may go wrong, I have not tested it in real on Vista yet.)

About Ra, I assume by 'xRa' you mean the conjuncts of type
'Consonant+halant+Ra'. Here using reformed script has completely different
objective. Reformed script IMO should be used to have only minimal set of glyphs
present in font. Your suggestion does not help that. More, I am also concerned
about the conjunct glyphs that will also need this post-base form of Ra but may
not have a related glyph in font. Its always better to make this happen
automatically. This is happening in uniscribe currently. 

Behdad, sorry for such long descriptions, I understand it would be difficult for
you to follow the script mechanisms being discussed here. But lets hope we at
least manage to implement what is already standardized and working well :)

Thanks.

Rahul says "Reformed script IMO should be used to have only minimal set of glyphs
present in font."

We can't place a below base form LA every time we encounter a 'VIRAMA, LA'
sequence.This is against the orthographic rules in Malayalam.The same reason
holds true for post base form of RA ,too.

Vis-à-vis the the VA and LA forms issues, the use of 'akhn' was for overcoming
the some shaping issues in pango.Now I clould sovle this issue at the font 
level using the 'pstf' tag itself and using the 'VA/LA, VIRAMA' sequence in the
GSUB.I will make available this font for testing.

-suresh

What typically puzzles me and makes me very hesitant to commit patches to the
Pango indic module is when people say "works with Qt".  Guess what, Pango's
indic shaper was quite like Qt's a couple years ago.  Then more and more "fixes"
where committed to Pango, each breaking a dozen other combinations...  And we've
got into the spaghetti that we have now...

What we need first and foremost is a test suite.

Suresh,
Thanks that you realized using 'pstf' tag :). This is  what I have been asking
for so long. Keep post-base forms post-base, below-base to below-base, both in
font and rendering engine and avoid using 'akhn'. Whenever (in rare cases) you
do not want to use them in those forms try some other remedies in font and do
not touch rendering engine for everything.

Thanks.

There is correction in my earlier comment.I messed up with LA and YA.
The line Vis-à-vis the the VA and LA forms issues... should be
Vis-à-vis the the VA and YA forms issues...  

and ... 'pstf' tag itself and using the 'VA/LA, VIRAMA' sequence ... should be 
... 'pstf' tag itself and using the 'VA/YA, VIRAMA' sequence ...

I had to add some glyphs for the 'pstf' stuff to work.So it was a workaround.

For the RA and LA,  I have specified the reason for the non-usage of post base
forms and it is entirely different.It is related to orthography.

When the rendering go haywire with only one particular system we, naturely, go
for the fix in that specific rendering system.

qt4.4/harfbuzz patch is submitted upstream

please test and comment
https://bugs.kde.org/show_bug.cgi?id=160218

Behdad, any chance we can have the patch in the upstream bug included in F9?

This issue seems to be giving considerable pain to the Malayalam community.

It is indeed painful to see the patch sitting in the bugzilla unattended for
more than 5 weeks now.

Which patch?  Which version of which product?

http://bugzilla.gnome.org/attachment.cgi?id=106776

Rahul and Ani, have we tested this patch for the pango in current pre-F9 rawhide?

Rahul, would you please review the upstream patch?

Recently I have updated lohit-fonts-malayalam in F8(version 2.1.9), and it seems
that still there are issues. The Fedora community is on the brim of a brand new
release and it would be really nice to see all the issues resolved. 

Basically the fonts looks ugly(see the items 5 and 4 in the attached image).
There are also critical rendering errors(items 1, 2 and 3 in the image). I have
also attached the screenshot of the same page with Meera fonts(developed by
SMC). The system is installed with the latest pango(pango-1.18.4). I am sure
that the above issues are due to the fonts, because the issues in pango are now
completely fixed.

I believe that the traditional fonts, which is very much familiar to the
Malayalees makes more sense in computers. The typewriter fonts were developed in
an effort to ease the typesetting in printing press and it may not be relevant
in computers. 

If the bugs are not resolved within the timeframe, at least consider making the
Meera fonts(it's already in F9 repo) mandatory for Malayalam Language support in
F9 and remove lohit-fonts-malayalam as a prerequisite for applications such as
Openoffice.org. 

I wish F9 would be the first GNU/Linux distribution with error-free Malayalam
rendering and interface(80% translation of GNOME is completed)

Created attachment 303741 [details]
Screenshot of firefox with lohit-fonts-malayalam-2.1.9

Screenshot of firefox with lohit-fonts-malayalam-2.1.9

Created attachment 303742 [details]
Screenshot of firefox with Meera Font(SMC)

Screenshot of firefox with Meera Font(SMC)

I see there are basically three kind of problems visible here:
1. the positioning of (Consonant+Halant+Ra) glyphs
2. Substitutions of few Conjunct+HalfRa glyphs, due to the wrong ligatures.  
3. The particular combination  involving Ya and KaHalantKa.

The first kind of problem happened, since I copied the glyphs given by Hiren.
Unfortunately he had used a different scaling Ascent-Descent values than the
original default Lohit. This caused the positions to distort with respect to
other glyphs. 

The second problem, I am not sure how it got in, since I had only copy pasted
the fixes given to me. Now after spotting the problem, ligature rules are
certainly incorrect for the conjuncts plus Ra glyphs.

The third problem, is a left behind of the earlier implementation of pango, that
did not use post-base Ya properly. Only reported problem related to this was
post-base Ya followed by Ka, but not the KKa. So this is not a new, but
ignored-for-long-time bug.This is probably present in Rachana as well.

The font in F9, was released quite a long time back and ensured testing by Ani.
So we had ample time to test and correct these kind of last minute fixes.
Unfortunately they were not reported until recently, even now they are not
reported as separate bugs. Said that these are not very complex issues, they
could have been solved on time provided they were reported in time. Anyways, I
would still try to get the fixes right into f9-final. 

One more probable problem I think is the ligature for samvruthokaram that was
present in Hiren's fixes. After comparing it with other fonts, I am not sure if
thats the correct glyph. Appreciate inputs on that.

The ligature for samvrathokaram is working correctly, what I did was adding a
new glyph for samvrathokaram, as what it is done in Raghu font by
Suresh|Suruma.For that glyph too, the right bearing is wrong. My fixes are done
just to make a better font and I still believes the font (with my fixes) is buggy.

Considering the font as a font designer and as a language user/l10n volunteer I
feel the font to be worthless, unattractive and unreadable. The design of some
glyphs are done in wrong manner, proportion of glyphs are not correct and the
font contains more irrelevant glyphs. Being a default font for a language, it is
supposed to be good (at least readable) at the screen and paper, which the lohit
malayalam fails on both. This font requires a heavy modifcation, that too from
its scratch.
point out the fact that this version of fedora is the one which contain many
malayalam localisation efforts and pacakges than any old fedoras. Hence it is
disappointing to have such a fedora with  'more than a buggy' font. Its a fact
that the lohit malayalam is unstable font and suggestions for solutions are : 

1. Make the lohit malayalam bugfree in every sence : which includes redesigning
of existing glyphs, making of more than 600 more glpyhs to get transformed into
traditional font pattern and some technical things (I suggest an year and a half
for that.).Lohit is now is neither an typewriter font or traditional font, it is
a mix of both. Even to follow the same idea, certain more glyphs are required
for the font.
OR
2. Freeze lohit malayalam, let it be so there or in trash. Make any other Free
Software fonts as the default font for F9. Suggested font is Meera of smc-font
package. And allow lohit malayalam designers to work on the developements.

Bad fonts results in  bad computing.
Hiran Venugopalan
http://hiraneffects.blogspot.com

Hiran, thank you for your comments and feedback on Lohit Malayalam.

We need to separate the rendering and fonts issues as far as possible.
It would be very helpful if you could open a separate bug against
lohit-fonts (or multiple bugs) detailing the problems and issues with it.
It sounds like we probably need a tracking bug for it too.

Changing version to '9' as part of upcoming Fedora 9 GA.
More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

requested by Jens Petersen (#27995)

Fedora 10 renders Malayalam perfectly and the default font package is also changed. This ticket may be closed ASAP.

I guess we should have closed this bug long time back. :-)