Attachment 826608 Details for Bug 1032504 – 0001-Don-t-strip-characters-with-Unicode-category-Cf-Othe.patch

[patch] 0001-Don-t-strip-characters-with-Unicode-category-Cf-Othe.patch

0001-Don-t-strip-characters-with-Unicode-category-Cf-Othe.patch (text/plain), 2.34 KB, created by Mike FABIAN on 2013-11-20 12:43:26 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Mike FABIAN

Created: 2013-11-20 12:43:26 UTC

Size: 2.34 KB

patch

obsolete

>From b2c49163dd549fe5c600d4ff429ac3aca4e87612 Mon Sep 17 00:00:00 2001
>From: Mike FABIAN <mfabian@redhat.com>
>Date: Wed, 20 Nov 2013 13:26:08 +0100
>Subject: [PATCH] =?UTF-8?q?Don=E2=80=99t=20strip=20characters=20with=20Uni?=
> =?UTF-8?q?code=20category=20=E2=80=9CCf=E2=80=9D=20(Other,=20format)=20fr?=
> =?UTF-8?q?om=20tokens?=
>MIME-Version: 1.0
>Content-Type: text/plain; charset=UTF-8
>Content-Transfer-Encoding: 8bit
>
>See: https://bugzilla.redhat.com/show_bug.cgi?id=1032504
>(Bug 1032504 - [ml_IN][ibus-typing-booster][F20] - 200D does not get committed in user db when it comes at the end)
>
>With ml-inscript transliteration:
>
>    Hfnd]mCd] â à´«à´¿à´²àµâà´¸à´£àµâ
>    Hfnd]mCd  â à´«à´¿à´²àµâà´¸à´£àµ
>
>    mfabian@ari:~
>    $ echo -n à´«à´¿à´²àµâà´¸à´£àµâ | iconv -f utf-8 -t utf16be | od -t x1
>    echo -n à´«à´¿à´²àµâà´¸à´£àµâ | iconv -f utf-8 -t utf16be | od -t x1
>    0000000 0d 2b 0d 3f 0d 32 0d 4d 20 0d 0d 38 0d 23 0d 4d
>    0000020 20 0d
>    0000022
>    mfabian@ari:~
>    $ echo -n à´«à´¿à´²àµâà´¸à´£àµ | iconv -f utf-8 -t utf16be | od -t x1
>    echo -n à´«à´¿à´²àµâà´¸à´£àµ | iconv -f utf-8 -t utf16be | od -t x1
>    0000000 0d 2b 0d 3f 0d 32 0d 4d 20 0d 0d 38 0d 23 0d 4d
>    0000020
>    mfabian@ari:~
>    $
>
>â]â inserts U+200D ZERO WIDTH JOINER. As the example shows,
>stripping this off from the tokens to be saved in the user database is
>a bad idea.
>---
> ibus-typing-booster/engine/itb_util.py | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
>
>diff --git a/ibus-typing-booster/engine/itb_util.py b/ibus-typing-booster/engine/itb_util.py
>index d017fd3..4940983 100755
>--- a/ibus-typing-booster/engine/itb_util.py
>+++ b/ibus-typing-booster/engine/itb_util.py
>@@ -33,9 +33,9 @@ import unicodedata
> # prevents learning such words from user input. I.e. the list of
> # categories to trigger immediate commit should contain only categories
> # which are very unlikely to appear as parts of words.
>-categories_to_trigger_immediate_commit = ['Po', 'Pi', 'Pf', 'Ps', 'Pe', 'Pc', 'Sm', 'Sc', 'Cf']
>+categories_to_trigger_immediate_commit = ['Po', 'Pi', 'Pf', 'Ps', 'Pe', 'Pc', 'Sm', 'Sc']
> 
>-categories_to_strip_from_tokens = ['Po', 'Pi', 'Pf', 'Ps', 'Pe', 'Pc', 'Pd', 'Sm', 'Sc', 'Cf']
>+categories_to_strip_from_tokens = ['Po', 'Pi', 'Pf', 'Ps', 'Pe', 'Pc', 'Pd', 'Sm', 'Sc']
> 
> def lstrip_token(token):
>     token = token.lstrip()
>-- 
>1.8.4.2
>

Actions: View | Diff

Attachments on bug 1032504: 826522 | 826523 | 826608