Bug 825081

Test Case from Attachment (to easy copy/paste):
---
Tunga:
ಕ್ಷಾ ಕ್ಷಿ ಕ್ಷೀ ಕ್ಷು ಕ್ಷೂ ಕ್ಷೃ ಕ್ಷೄ ಕ್ಷೆ ಕ್ಷೇ ಕ್ಷೈ ಕ್ಷೊ ಕ್ಷೋ ಕ್ಷೌ ಕ್ಷ್
ಚ್ಚಾ ಚ್ಚಿ ಚ್ಚೀ ಚ್ಚೃ ಚ್ಚೄ ಚ್ಚು ಚ್ಚೂ ಚ್ಚೆ ಚ್ಚೇ ಚ್ಚೈ ಚ್ಚೊ ಚ್ಚೋ ಚ್ಚ್
ಡ್ಡಾ ಡ್ಡಿ ಡ್ಡೀ ಡ್ಡು ಡ್ಡೂ ಡ್ಡೃ ಡ್ಡೆ ಡ್ಡೇ ಡ್ಡೈ ಡ್ಡೊ ಡ್ಡೋ ಡ್ಡೌ ಡ್ಡ್
---

Can we get Unicode points for such characters?

UniView (http://rishida.net/scripts/uniview/) or BabelMap (http://www.babelstone.co.uk/software/babelmap.html) is your friend. 

(I have also made small correction to the sample text.)

ಕ್ಷಾ ಕ್ಷಿ ಕ್ಷೀ ಕ್ಷು ಕ್ಷೂ ಕ್ಷೃ ಕ್ಷೄ ಕ್ಷೆ ಕ್ಷೇ ಕ್ಷೈ ಕ್ಷೊ ಕ್ಷೋ ಕ್ಷೌ ಕ್ಷ್
ಚ್ಚಾ ಚ್ಚಿ ಚ್ಚೀ ಚ್ಚು ಚ್ಚೂ ಚ್ಚೃ ಚ್ಚೄ ಚ್ಚೆ ಚ್ಚೇ ಚ್ಚೈ ಚ್ಚೊ ಚ್ಚೋ ಚ್ಚೌ ಚ್ಚ್
ಡ್ಬಾ ಡ್ಬಿ ಡ್ಬೀ ಡ್ಬು ಡ್ಬೂ ಡ್ಬೃ ಡ್ಬೄ ಡ್ಬೆ ಡ್ಬೇ ಡ್ಬೈ ಡ್ಬೊ ಡ್ಬೋ ಡ್ಬೌ ಡ್ಬ್

\u0C95\u0CCD\u0CB7\u0CBE \u0C95\u0CCD\u0CB7\u0CBF \u0C95\u0CCD\u0CB7\u0CC0 \u0C95\u0CCD\u0CB7\u0CC1 \u0C95\u0CCD\u0CB7\u0CC2 \u0C95\u0CCD\u0CB7\u0CC3 \u0C95\u0CCD\u0CB7\u0CC4 \u0C95\u0CCD\u0CB7\u0CC6 \u0C95\u0CCD\u0CB7\u0CC7 \u0C95\u0CCD\u0CB7\u0CC8 \u0C95\u0CCD\u0CB7\u0CCA \u0C95\u0CCD\u0CB7\u0CCB \u0C95\u0CCD\u0CB7\u0CCC \u0C95\u0CCD\u0CB7\u0CCD

\u0C9A\u0CCD\u0C9A\u0CBE \u0C9A\u0CCD\u0C9A\u0CBF \u0C9A\u0CCD\u0C9A\u0CC0 \u0C9A\u0CCD\u0C9A\u0CC1 \u0C9A\u0CCD\u0C9A\u0CC2 \u0C9A\u0CCD\u0C9A\u0CC3 \u0C9A\u0CCD\u0C9A\u0CC4 \u0C9A\u0CCD\u0C9A\u0CC6 \u0C9A\u0CCD\u0C9A\u0CC7 \u0C9A\u0CCD\u0C9A\u0CC8 \u0C9A\u0CCD\u0C9A\u0CCA \u0C9A\u0CCD\u0C9A\u0CCB \u0C9A\u0CCD\u0C9A\u0CCC \u0C9A\u0CCD\u0C9A\u0CCD

\u0CA1\u0CCD\u0CAC\u0CBE \u0CA1\u0CCD\u0CAC\u0CBF \u0CA1\u0CCD\u0CAC\u0CC0 \u0CA1\u0CCD\u0CAC\u0CC1 \u0CA1\u0CCD\u0CAC\u0CC2 \u0CA1\u0CCD\u0CAC\u0CC3 \u0CA1\u0CCD\u0CAC\u0CC4 \u0CA1\u0CCD\u0CAC\u0CC6 \u0CA1\u0CCD\u0CAC\u0CC7 \u0CA1\u0CCD\u0CAC\u0CC8 \u0CA1\u0CCD\u0CAC\u0CCA \u0CA1\u0CCD\u0CAC\u0CCB \u0CA1\u0CCD\u0CAC\u0CCC \u0CA1\u0CCD\u0CAC\u0CCD

BTW what will you do with the codepoints?

http://pravins.fedorapeople.org/Lohit-Kannada.ttf
Just tried for ಕ್ಷೃ it works fine in Gnome with pango. Please test it and let me know if its work on Windows.
So i can fix other combination in same way.

Thanks.

(In reply to comment #3)
> Just tried for ಕ್ಷೃ 

I *told* you it works for ಕ್ಷೃ because for ಕ್ಷೃ ಜ್ಞ and two other consonant clusters you have included separate glyphs with all vowel sign combinations in the font. The point was for combinations *other* than these which can very well occur when writing especially Sanskrit-based words in Kannada script.

But even with ಕ್ಷೃ the rendering is not consistent in Linux. See for ಕ್ಷು ಕ್ಷೂ the SSA-vattu is pushed to the right, whereas it is right below the KA in other vowels due to the existence of pre-composed glyphs with vowel sign. And for ಕ್ಷೌ it is also pushed to the right despite the existence of the precomposed-glyph.

(Maybe your rendering in Pango/Fedora is different from what I am seeing in LibreOffice/Kubuntu -- please post a PDF.)

> it works fine in Gnome with pango. Please test it and
> let me know if its work on Windows.

LOL I'm not only using it on Windows but also on my Kubuntu Linux installation. 

On LibreOffice 3.5 on Kubuntu 12.04, the sequences still do NOT render properly. The PDF I had given you was produced by LibreOffice on Linux only and not from Windows.

> So i can fix other combination in same way.

Let me just list the required rendering so you can take care of how it is to be achieved:

Nasically you have four consonant clusters with precomposed glyphs i.e. K.SSA, J.NYA, D.DA and L.LA. Of these, except for vowel signs U, UU and vocalic R, RR, L, LL, combinations with all other vowel signs are precomposed.

1) In the case of these precomposed glyphs:

1a) If VS = vowel sign of U/UU or one of the vocalic vowels,

    C1 + VIR + C2 + VS --> C1_WITH_C2_VATTU VS

1b) If VS is another vowel sign or the virama

    C1 + VIR + C2 + VS --> C1_WITH_C2_VATTU_WITH_VS

2) In the case of the consonant clusters *without* precomposed glyphs

2a) If VS = vowel sign of U/UU or one of the vocalic vowels,

    C1 + VIR + C2 + VS --> C1 VS C2_VATTU

2b) If VS is another vowel sign or the virama

    C1 + VIR + C2 + VS --> C1_WITH_VS C2_VATTU

(In reply to comment #2)
> 
> BTW what will you do with the codepoints?

Sometime we do get bugs consisting string in pdf/image and it takes some time to to decode basic composition.

So it helps non language person like me to select characters in Fontforge for testing with Unicode value. 

Thanks for link from Richard Ishida's very useful. I use python to decode text to Unicode.

I have fixed the first row and committed in upstream.
ಕ್ಷಾ ಕ್ಷಿ ಕ್ಷೀ ಕ್ಷು ಕ್ಷೂ ಕ್ಷೃ ಕ್ಷೄ ಕ್ಷೆ ಕ್ಷೇ ಕ್ಷೈ ಕ್ಷೊ ಕ್ಷೋ ಕ್ಷೌ ಕ್ಷ್

Tunga also renders same way like lohit for below test cases. What is ideal rendering should below base part sit exactly below the base consonant?

2. ಚ್ಚಾ ಚ್ಚಿ ಚ್ಚೀ ಚ್ಚೃ ಚ್ಚೄ ಚ್ಚು ಚ್ಚೂ ಚ್ಚೆ ಚ್ಚೇ ಚ್ಚೈ ಚ್ಚೊ ಚ್ಚೋ ಚ್ಚ್
3. ಡ್ಡಾ ಡ್ಡಿ ಡ್ಡೀ ಡ್ಡು ಡ್ಡೂ ಡ್ಡೃ ಡ್ಡೆ ಡ್ಡೇ ಡ್ಡೈ ಡ್ಡೊ ಡ್ಡೋ ಡ್ಡೌ ಡ್ಡ್
---

Created attachment 673323 [details]
Updated ODT and PDF for test-case

Hello Pravin. Sorry I couldn't reply earlier. 

I have now revisited the problem with the latest[*] Lohit-Kannada using LibreOffice 3.6.3 on Windows XP and 3.5.4 on Kubuntu Precise, but the problem remains the same. [* = At the present instant it says "up-to-date" -- I'm sorry I'm not sure how to ask git for the commit id.]

I hope the updated ODT and PDFs which I am attaching now should make the problem in question clearer as I have marked the problematic clusters from Lohit Kannada in red as against the correct Tunga rendering in green:

1) In the case of consonant cluster + vowel sign vocalic R/RR / vowel sign AI the descender of the vowel sign is overlapping with the below-base conjunct consonant.

2) In case of consonant cluster + virama, the virama is not properly joined to the above base consonant. 

3) In Linux, only in the case of K.SSAI and K.SS.VIRAMA the problem is not occuring (blue in Linux PDF) but in the case of vowel sign of vocalic vowels problem remains.

Sorry for extra post:

As clarification to point 3 in my previous comment:

In Linux, 

1) In the case of the four clusters with separate conjunct glyphs in the font i.e. K.SSA, J.NYA, D.DA and L.LA, AI and Virama are attaching properly (without overlap) but vocalic R/RR are still overlapping with the below-base consonant.

2) In the case of all other clusters, AI and virama are also not attaching properly.

On Windows XP (sorry I am not able to test on Windows 7):

None of the clusters have AI and virama or vocalic R/RR attaching properly.

Thanks Shriramana for more details.

committed some more fixes in git. Latest lohit ttf available at http://pravins.fedorapeople.org/Lohit-Kannada.ttf , please test and let me know if there are more combinations.


From sample file remaining are ಚ್ಚ್  & ಡ್ಡ್  bit tricky but will fix them soon.

Created attachment 674575 [details]
Lohit Kannada on W8

Created attachment 674589 [details]
Lohit from master on Fedora 18

 ಚ್ಚ್  & ಡ್ಡ್  works fine in Windows but not on Harfbuzz, reported bug for https://bugs.freedesktop.org/show_bug.cgi?id=59118 lets see if we can get same rendering with Harfbuzz as well.

Great, Behdad fixed this issues in harfbuzz. Now ಚ್ಚ್  & ಡ್ಡ್   works well with other Kannada fonts as well.

So AFA i see no more combination need fix in list provided.

Created attachment 675746 [details]
Script to produce mostly valid test examples

Hello Pravin and Behdad and thanks for the quick work. I now am attaching a small Python3 script to produce any required number of test cases which are phonologically valid (and to a great extent can really occur) for Indic languages, esp. Sanskrit. You can use it to produce more test cases if you want.

@Pravin: Even now the vowel sign for vocalic R/RR is slightly overlapping in case of K.SSA. Can you do a contextual kerning of the vowel sign when the below-base consonant will collide with it?

(In reply to comment #14)
> Created attachment 675746 [details]
> Script to produce mostly valid test examples
> 
> Hello Pravin and Behdad and thanks for the quick work. I now am attaching a
> small Python3 script to produce any required number of test cases which are
> phonologically valid (and to a great extent can really occur) for Indic
> languages, esp. Sanskrit. You can use it to produce more test cases if you
> want.

ಶ್ರು	ಕ್ತೄ	ಠ್ಳೊ	ಚ್ಥೇ	ಟ್ವೀ	ಪ್ರೃ	ಷ್ಳೀ	ಸ್ರಾ	ನ್ಠಾ	ಬ್ೞ್	ಱ್ೞೌ	ನ್ಖ್	ಯ್ಱಾ	ಗ್ಧೊ	ಣ್ಕೄ	ಫ್ಲು	ಟ್ಚೊ	ಭ್ಱೀ	ದ್ಜಾ	ಹ್ಶೌ	ದ್ಝೀ	ಯ್ಳೃ	ಗ್ದೄ	ೞ್ಲಿ	ಪ್ಚೄ	ಘ್ರಿ	ಮ್ನಾ	ಙ್ಖೈ	ಲ್ರೌ	ನ್ತಾ	ತ್ಕೀ	ಟ್ಕೈ	ಳ್ಲೊ	ರ್ಯೇ	ಮ್ದೃ	ೞ್ಳ್	ಷ್ಯೃ	ಸ್ಸೇ	ಜ್ಬೄ	ಡ್ಝೃ	ತ್ಛೇ	ಯ್ಯೂ	ಶ್ಳೈ	ಡ್ಗೌ	ಗ್ಝೃ	ಸ್ಶಾ	ಬ್ಭೆ	ಢ್ಳೂ	ಸ್ಸೇ	ಜ್ಭ್	ಣ್ಯೊ	ಗ್ಬೋ	ಜ್ಝೈ	ಸ್ಷೆ	ದ್ಜೋ	ಞ್ಗೇ	ಪ್ಖೌ	ಱ್ಳೆ	ಶ್ಲೋ	ವ್ೞೄ	ಚ್ಟೃ	ಗ್ಜೇ	ಟ್ಛಾ	ಷ್ರೃ	ಷ್ಳೌ	ಙ್ಥಾ	ಲ್ವೈ	ಚ್ವೀ	ಲ್ಱೃ	ಷ್ಶೈ	ಶ್ಯೂ	ಱ್ಳಾ	ಹ್ೞು	ಹ್ಸೀ	ಷ್ಳೇ	ಷ್ಸೊ	ಮ್ಡಾ	ಣ್ಚೆ	ಷ್ಷೇ	ಕ್ತೇ	ರ್ರೊ	ಸ್ಸೈ	ಜ್ೞೆ	ಫ್ಱಾ	ಜ್ಬೀ	ಡ್ಘೊ	ದ್ರೇ	ಛ್ರೇ	ಙ್ಱ್	ಳ್ಱ್	ತ್ತೂ	ಹ್ಶೄ	ಡ್ಗಾ	ಹ್ಹೆ	ತ್ತೇ	ಚ್ಪ್	ಚ್ತೇ	ಞ್ಣೂ	ಪ್ತು	ಟ್ಛೌ	

These are the combinations generated by your script. Working well with Lohit Kannada, let me know if you find any problem. Test with http://pravins.fedorapeople.org/Lohit-Kannada.ttf

> 
> @Pravin: Even now the vowel sign for vocalic R/RR is slightly overlapping in
> case of K.SSA. Can you do a contextual kerning of the vowel sign when the
> below-base consonant will collide with it?

Fixed it.

Created attachment 679704 [details]
ODT and PDF of random 2-consonant clusters

Hi Pravin -- every time you run the script it will produce new examples. And right now I have tested on my Windows XP LibO 3.6.4 and Kubuntu Precise LibO 3.5.4 and still many clusters are not rendering correctly. I have colour-coded them in the attached doc. Is it that in HB trunk they are rendering correctly?

Yes, i am testing with Harfbuzz latest releases not trunk. I will check again for new test cases provided by you.

This message is a reminder that Fedora 16 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 16. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '16'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 16's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 16 is end of life. If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora, you are encouraged to click on 
"Clone This Bug" and open it against that version of Fedora.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

This message is a reminder that Fedora 18 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 18. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '18'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 18's end of life.

Thank you for reporting this issue and we are sorry that we may not be 
able to fix it before Fedora 18 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior to Fedora 18's end of life.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

This bug appears to have been reported against 'rawhide' during the Fedora 23 development cycle.
Changing version to '23'.

(As we did not run this process for some time, it could affect also pre-Fedora 23 development
cycle bugs. We are very sorry. It will help us with cleanup during Fedora 23 End Of Life. Thank you.)

More information and reason for this action is here:
https://fedoraproject.org/wiki/BugZappers/HouseKeeping/Fedora23

This message is a reminder that Fedora 23 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 23. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '23'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 23 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

This message is a reminder that Fedora 25 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 25. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '25'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not
able to fix it before Fedora 25 is end of life. If you would still like
to see this bug fixed and are able to reproduce it against a later version
of Fedora, you are encouraged  change the 'version' to a later Fedora
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

Fedora 25 changed to end-of-life (EOL) status on 2017-12-12. Fedora 25 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.