Bug 855250

+++ This bug was initially created as a clone of Bug #834971 +++

This was originally brought up by Mathieu Bridon and other Hong Kong Fedora users.  The text below is adopted from his mail.


Here's a proposal to fix IBus Table for Hong Kong users.

Both Cangjie and Quick can be used to type Simplified and Traditional
Chinese, however, given their design, there isn't any combination of keys that
would conflict between those languages. In other words, any given
combination of character can only lead to results in one of those
languages, never more than one.

Given all that, it would make sense to simply remove altogether the IBus
filter for the Cangjie and Quick input methods in IBus.

Let's take an example.

In Cangjie, the combination "rji" can only return results in Traditional
Chinese. That means if a user types this combination of keys, he is
expecting results in Traditional Chinese because that's the language
he/she wants to type.

But with the current IBus filter, if the filter is set to only let
Simplified Chinese characters pass, he/she would not get any results.

In the same way, the combination "yri" can only return results in
Simplified Chinese, and the combination "fji" can only return results in
Japanese.

[ed: I have never heard of people using ibus-table for Japanese input]

This is by design of those two input methods: they were designed to
avoid conflicts.

As such, the filter just makes no sense for Cangjie and Quick, and it
should be simply removed for those two input methods.

Now, in the above I claimed that Cangjie and Quick were designed to have
absolutely zero conflicts, which was a little exaggeration. 

In reality, conflicts happen. However, Cangjie and Quick were really
designed with the goal of minimizing conflicts, and they do it so well
that the actual rate of conflicts is 8.04% [1]. This is such a small
number, and it happens in so rare occasions, that it can just be
ignored.

It is also important to note that if 90% of Hong Kong people [2] use
Cangjie and Quick, many people (but much less than in HK) use them in
Taiwan, and almost no one use them in Mainland China or in Japan.

Out of those three, Hong Kong and Taiwan write Traditional Chinese,
Mainland Chinese write Simplified Chinese, and Japanese obviously write
Japanese. So those two input methods really are used almost exclusively
to write Traditional Chinese, which makes the aforementioned 8.04%
figure completely negligible.

As such, it doesn't change the argument at all: the current IBus filter
should be removed for the Cangjie and Quick input methods.

This is an absolute show-stopper for Hong Kong users at the moment
(well, not me, I can't write Chinese , and the simple act of removing
this filter for those two input methods would basically fix 90% of the
problems for 90% of the Hong Kong people.

Do you think you guys could fix this issue before the release of GNOME
3.6?

Of course, we'd be happy to provide a patch if you agree on the solution
and if you can provide some guidance. 

[1] There's a scientific paper published on this, but it's unfortunately
only available in Chinese:
    http://zh.wikipedia.org/wiki/倉頡輸入法

[2] Not just Linux users, actual **people**, as this is how everyone
learns to type at school in Hong Kong.

--- Additional comment from petersen on 2012-06-25 13:53:12 EST ---

Hi Matthieu

Forgive me for "pasting" your mail into bugzilla,
but I am also keen to see this issue get fixed.
I encourage you report any issues and problems
you Hong Kong users are facing directly into bugzilla
so that they can attention. :)

> As such, the filter just makes no sense for Cangjie and Quick, and it
> should be simply removed for those two input methods.
:
> As such, it doesn't change the argument at all: the current IBus filter
> should be removed for the Cangjie and Quick input methods.
> 
> This is an absolute show-stopper for Hong Kong users at the moment
> (well, not me, I can't write Chinese , and the simple act of removing
> this filter for those two input methods would basically fix 90% of the
> problems for 90% of the Hong Kong people.

Sounds reasonable enough

> Do you think you guys could fix this issue before the release of GNOME
> 3.6?

I think so and for Fedora 18 - and of course the fix should be backported
to Fedora 16 and 17.  I am just surprised noone has reported this issue
before...

> Of course, we'd be happy to provide a patch if you agree on the solution
> and if you can provide some guidance. 

That would be helpful if you can.

I am not clear if it is the allowed characters in the tables themselves
that need fixing (or does ibus-table ignore those?) or ibus-table itself?

--- Additional comment from bochecha on 2012-06-25 14:30:19 EST ---

(In reply to comment #1)
> Hi Matthieu

Mathieu. ;)

> I am just surprised noone has reported this issue before...

Let's be honest, there basically aren't any Fedora users in Hong Kong. (whether this fact is because of this IBus issue or because we haven't been good enough at advocacy is not the subject of this bug report).

There's me, but I'm French and can't write Chinese, so I don't really count here.

There seem to be 3 other local Fedora ambassadors, but all my attempts at contacting them basically failed, so they might not be active any more.

The people I have been discussing this issue with are mostly Ubuntu and Debian users though.

Also, it seems to me that the local Linux community (which is very small) is mostly made of geeks/tinkerers, which are very quick to just drop IBus entirely and replace it with something else which is more geared towards the local needs (I think I heard fcitx was more widely praised in wider China).

However, this is becoming an issue with GNOME 3.6 adopting IBus and tightly integrating to it.

I want to advocate GNOME and Fedora in Hong Kong, and my (relatively short, as I've been here for less than two years) experience, as well as discussions I've had with local long time users who can actually type Chinese, indicate that IBus is just unsuitable in Hong Kong at the moment, because of this bug.

> > Of course, we'd be happy to provide a patch if you agree on the solution
> > and if you can provide some guidance. 
> 
> That would be helpful if you can.
> 
> I am not clear if it is the allowed characters in the tables themselves
> that need fixing (or does ibus-table ignore those?) or ibus-table itself?

The problem is the filter.

There is currently a filter to let only characters from a given language pass.

With the Candgie and Quick input methods, this filter makes little to no sense, by design of the Candgie and Quick input methods themselves (see my explanation) and so it should just be dropped.

I don't think it is reasonable, especially for those two input methods given their design, to ask the user to select what the filter should let pass. Also consider that the current GNOME design seems to agree with me on this as it doesn't expose those settings anywhere that I could see.

However, there probably are other input methods where this filter makes a lot of sense, so I'm only asking for it to be dropped for Candgie and Quick, not for the whole of ibus/ibus-table.

There obviously are different methods to impleemnt this, the two that come to my mind being:

1. something like:
    if im_name not in ("candgie", "quick"):
        do_filter()

2. allow each input method to provide their own set of directives to the framework, so that Candgie and Quick could disable the filter themselves

Which one of these, or any better implementation, is more suitable for upstream is of course left for upstream to decide. :)

--- Additional comment from me on 2012-06-26 11:59:28 EST ---

I am a Fedora user from HK but not in HK. Counting my brother who uses Fedora in his workplace as server, it won't be just you. :)

* Introduction *

This is not the problem only of IBus on Fedora, but I think is IBus itself. There are a few things I need to mention separately about this negative user experiences to HK / Traditional Chinese users, that I hope could be documented permanently somewhere on the Internet:

1. Default value.
2. User interface.
3. Filter.
4. Cangjie & Quick.
5. Design of logic.

* Default Value *

Cangjie & Quick are mostly living in ibus-table on Fedora. Since the main developers of ibus and ibus-table are from Simplified Chinese background, the default values are set for Simplified Chinese users they expected to behave. This made the filters of Cangjie and Quick "Simplified Chinese only". When there are no candidate characters will come up that users expected, users with novice level of computing knowledge probably recognized that as a broken input method software.

I remembered some patches have been done by acevery (dev of ibus-table) and others, to check up the system locale when the engine starts for a default value of "Traditional Chinese only" filter. Sadly, this fix did not resolve the problem on my box, which I am using English locale + UI with Cangjie input method. And I think at the end of the day my use case was ignored as "not common scenario".

* User Interface *

Okay, I can confidently say - no one from Hong Kong will select Chinese (Hong Kong) on regional settings during installation. The reason of this partly because there had been no substantial Cantonese localization happened in Fedora, past experience on Windows where no Chinese input methods were provided in such locale, and other problems on fonts/font-config/input methods on IBus, etc. This forced most of users who preferred Traditional Chinese UI to pick zh_TW as locale, no matter they are from HK / TW / Chinese communities around the world. This is somehow should not let it be.

The next thing is the broken UI of IBus on Gnome 3. The toolbar / properties bar / floating bar was gone in Gnome 3. And right click on the IBus tray icon (whatever it is in "zh" or a little keyboard) when Cangjie / Quick is the active input method, the setting items are all blank. This should be some color defaults when the default menu backgroud colors were light in Gnome 2. Having this gives inconvenience to "better than beginners" users to have filter value be changed from "simplified" to "traditional" or anything. I once have read a consumer computing magazine / web site published in HK, the author commented Fedora as "Linux which is not capable to support Chinese users". I think the reputation can get worse along time.

* Filter *
My first Fedora for daily purpose was Fedora Core 5. Its default input method engine / framework was SCIM. In SCIM, it demonstrated the true purposes of Chinese filter (on Cangjie / Quick / anything) - users can input in either Traditional or Simplified Chinese key combinations, and able to output the characters of in each other's standards. This aim the traditional Chinese user who needs to communicate with simplified Chinese user who cannot read the characters of each other, and vice versa. The ibus-table has not implemented that before SCIM was replaced by IBus. FYI, instant messaging software aliwangwang / taobaowangwang developed by alibaba.com for e-business have this function built in, when I was using that years ago.

I understand about the filter Mathieu mentioned and IBus had been using - the specific candidates of the tables are not listed for commit. As Mathieu explained about the design of the Cangjie and Quick, a given key combination of Cangjie should yield 1 Chinese character only, despite of acceptable duplicate %. A key combination of a traditional Chinese should get only tradition Chinese character, except when the table maker put a mapping of traditional Chinese key combination with a simplified Chinese character, and vice versa. FYI, many input method software adopted this behavior, such as "Cangjie Friends" on Windows (made by developers from Malaysia) and KeyKey / Kimo Input on OS X developed by Yahoo!. They put that as a switch in preferences - "Show only big-5 characters", where big-5 is a charset mostly cover traditional Chinese characters even fewer than Unicode defined.

There should be an "agreed" design which resulted the filter used by IBus now. I personally am not against this, but I guess the workload on table maintenance will be heavy. I still wish the filter in SCIM was implemented than the way of IBus has taken. I am not sure how to improve this yet.

* Cangjie & Quick *

About Cangjie, it is originally a design for all Chinese characters by breaking them down into key combinations in regard if it is traditional or simplified. Although during the simplification some strokes were made "not too complying" to the Cangjie design, I have seen so many input method engine / table came up some key mapping of character parts. Cangjie is not so appropriate for daily use of simplified characters, because relatively less strokes on simplified Chinese increased the duplication of candidates results. That's why Smart Pinyin may have higher productivity/typing-speed than Cangjie for simplified characters. Hope Cangjie on Fedora can focus on traditional Chinese first.

About Quick, you take first and last key from key combinations of a character. All quick Quick typists memorized the positions of candidates for frequent characters. The influence of Microsoft Windows again - users who uses Quick will complain "order of candidates and number of candidates on each page, are different from the orders on Windows", until they see the candidates on same place as they are on Windows. AFAIK I disabled the dynamic order on Quick, but changing the default number of candidates on each candidate-window page was not changed.

* Design of Logic *

The ibus-table has so may things hard-coded. It also consists of so many concepts suitable only for China users, like goucima (keys of character combination), pinyin value. The punctuation on ibus-table was hard-coded and was no so good, considering there are some more foreign tables such as Cyrillic, Latin, Thai, Vietnamese that may not using Chinese punctuation symbols. 

I received comments from people who tried ibus and ibus-table, that having sqlite and python slowed down the speed too much. I heard Ding Yi Chen has rewritten that into C but seems it has not been used but the original ibus-table in Python is still the default.

I do believe if no improvements significantly on ibus-table, it won't help the Fedora / Linux promotions in HK. For Taiwan people they just change to gcin/hime after installation, and for China people I heard fcitx is much more welcomed by them with some people did switch to fcitx because it is better to them.

* Conclusion *

I have the best user experience on SCIM (besides re-login was required when it crashed): accurate inputting, filter worked, informative & nice toolbar icons, responsive. If there is a goal of making an input method framework better than SCIM, changes are required on not just ibus-table but ibus from the bottom. There are not enough contributors / someone with resources to do that I personally observed.

--- Additional comment from bochecha on 2012-06-26 13:04:02 EST ---

(In reply to comment #3)
> * Default Value *
> 
> Cangjie & Quick are mostly living in ibus-table on Fedora. Since the main
> developers of ibus and ibus-table are from Simplified Chinese background,
> the default values are set for Simplified Chinese users they expected to
> behave. This made the filters of Cangjie and Quick "Simplified Chinese
> only". When there are no candidate characters will come up that users
> expected, users with novice level of computing knowledge probably recognized
> that as a broken input method software.
> 
> I remembered some patches have been done by acevery (dev of ibus-table) and
> others, to check up the system locale when the engine starts for a default
> value of "Traditional Chinese only" filter. Sadly, this fix did not resolve
> the problem on my box, which I am using English locale + UI with Cangjie
> input method. And I think at the end of the day my use case was ignored as
> "not common scenario".

This is exactly the filter issue I've described from the start, although I had (stupidly) forgotten to mention that the default value of the filter was based on the locale.

See also:
https://code.google.com/p/ibus/issues/detail?id=1472#c8

> * User Interface *

This is all being completely reworked for GNOME 3.6 anyway, so I'm not sure it makes sense talking about it here.

Plus, the UI is probably more related to the implementation of each Desktop Environment rather than IBus in itself?

As the original reporter, I'd like us to try and keep this bug report focused on the one issue I reported in the first place: the filter.

> * Cangjie & Quick *
> 
> About Quick, you take first and last key from key combinations of a
> character. All quick Quick typists memorized the positions of candidates for
> frequent characters. The influence of Microsoft Windows again - users who
> uses Quick will complain "order of candidates and number of candidates on
> each page, are different from the orders on Windows",

That's also a different issue from the one I reported originally.

First, I don't agree that it's a problem, as I stated in the upstream bug report: lots of other things are different between Linux and Windows anyway. We shouldn't aim to be **the same**, we should aim to be **better**.

If users have to relearn when they move from Windows to Linux but once learnt they can type more comfortably, so be it. Also, we should stop only aiming at "converting the masses". Many people are using Linux as their first computing experience, and that's definitely something we should focus on. Those people won't have to relearn anything, they will learn only once whatever the ordering in IBus is.

If the ordering is actually bad (which I'm not qualified to judge), it would indeed deserve a bug report. But could you open a separate bug report, so we can focus this one only on one issue? (the issue at hand being the filter)

> * Design of Logic *

Again, those all seem to me to be improvements which would deserve their own bug reports?

--- Additional comment from petersen on 2012-06-27 16:12:01 EST ---

Thanks guys for the helpful comments.

So perhaps the best solution would be to add ibus-setup-table
with some config UI to allow users to chose what if any
language filtering they prefer?

Anyway I agree that ibus-table seems weaker than scim-tables
so it would be good to improve it, at least to get to parity
in usability.

It would be helpful if you could file separate bugs about
other issues such as those you mentioned above.

Caius: if there is a performance issue with ibus-table
compared to scim-tables, it would be useful if you could
report that too with examples in a bug report.

--- Additional comment from bochecha on 2012-06-27 17:05:58 EST ---

(In reply to comment #5)
> Thanks guys for the helpful comments.
> 
> So perhaps the best solution would be to add ibus-setup-table
> with some config UI to allow users to chose what if any
> language filtering they prefer?

That might help, yes.

And I guess that Desktop Environments which will provide their own UI (e.g GNOME >= 3.6) could show that in some form or other.

For example, they could have the following in their list of "input sources":
[... snip ...]
- Simplified Chinese (Candgie)
- Simplified Chinese (Pinyin)
- Traditional Chinese (Bopomofo)
- Traditional Chinese (Candgie)
- Traditional Chinese (Quick)
[... snip ...]

And if the user chose "Traditional Chinese (Candgie)", then underneath, ibus-table would pick the "Candgie" input method with the filter set on suggesting only "Traditional Chinese" characters.

I think that could work really well. :)

--- Additional comment from jni on 2012-07-05 21:00:18 EST ---

Created attachment 596377 [details]
The screenshot of ibus-table ui

--- Additional comment from jni on 2012-07-05 21:16:06 EST ---

Hi Mathieu, Jens

As caius mentioned in his comment: 
"In SCIM, it demonstrated the true purposes of Chinese filter (on Cangjie / Quick / anything) - users can input in either Traditional or Simplified Chinese key combinations, and able to output the characters of in each other's standards." 

I just investige a bit of ibus-table, I found that actually ibus-table had provide such kind of filter. Indeed, the ibus-table have filters such as "Simplified Chinese only" and "Traditional Chinese only", but it also provide a filter allow both simplified chinese and tranditional chinese passed. 

In upstream, https://github.com/acevery/ibus-table/blob/master/engine/table.py, from line 96 to line 101, five candidate filter mode have defined for ibus-table

# self._chinese_mode: the candidate filter mode,
# 0 is simplify Chinese
# 1 is traditional Chinese
# 2 is Big charset mode, but simplify Chinese first
# 3 is Big charset mode, but traditional Chinese first
# 4 is Big charset mode.

So, mode 4 could be used by hong kong user if they want to input Simplified Chinese and Traditional Chinese at same time, or they can choose mode 2 and 3, both should be ok. I test the input rji and yri in mode 4, the output is correct. 

The attachment is the screenshot, which you can see, user could switch these 5 modes by clicking the second button in UI.  

The test enviroment: 
ibus-table-chinese-cangjie-1.3.5-1.fc16
ibus-table-1.3.9.20110827-1.fc16.noarch
OS: Fedora 16
Locale: en_US.utf8

For tranditional chinese user, the default value of mode is 1. so if the hong kong user wanted, we can change default value of mode to 4, then i think the issue should be resolved. 

Please correct me if i understand the issue wrong. Thanks

--- Additional comment from me on 2012-07-05 21:45:59 EST ---

I think as per Mathieu's comment, having mode 4 by default for cangjie/quick tables may be the answer. It may be already possible to have such configured in tables for such requirement. If not, a patch is the answer.

--- Additional comment from bochecha on 2012-07-09 15:16:44 EST ---

(In reply to comment #8)
> Hi Mathieu, Jens
> 
> As caius mentioned in his comment: 
> "In SCIM, it demonstrated the true purposes of Chinese filter (on Cangjie /
> Quick / anything) - users can input in either Traditional or Simplified
> Chinese key combinations, and able to output the characters of in each
> other's standards." 
> 
> I just investige a bit of ibus-table, I found that actually ibus-table had
> provide such kind of filter.

I know, that's the filter we've been talking about from the beginning, when in my initial comment I said:
> As such, the filter just makes no sense for Cangjie and Quick, and it
> should be simply removed for those two input methods.

> # self._chinese_mode: the candidate filter mode,
> # 0 is simplify Chinese
> # 1 is traditional Chinese
> # 2 is Big charset mode, but simplify Chinese first
> # 3 is Big charset mode, but traditional Chinese first
> # 4 is Big charset mode.
> 
> So, mode 4 could be used by hong kong user if they want to input Simplified
> Chinese and Traditional Chinese at same time, or they can choose mode 2 and
> 3, both should be ok. I test the input rji and yri in mode 4, the output is
> correct. 

Ok, I didn't know that "Big charset" stood for "don't filter at all".

Had I known that, I wouldn't have asked for the removal of the filter in the case of Candgie, I would have asked for the default value to be 4.


> The attachment is the screenshot, which you can see, user could switch these
> 5 modes by clicking the second button in UI.  

That screenshot represents what you get when installing a GNOME Shell extension, it is not the default experience in current GNOME, and probably not the default experience in current Fedora.

Also, it is not what will be the default experience in future GNOME (starting with 3.6):


> For tranditional chinese user, the default value of mode is 1.

Not really: the default value depends on the locale.

It is a fact that lots of Hong Kong people do not use any of the en_HK or zh_HK locale, yet they still want to use the Candgie or Quick input methods with a correct value for the filter.

> so if the
> hong kong user wanted, we can change default value of mode to 4, then i
> think the issue should be resolved. 

For all of IBus-Table? Or just for Candgie?

The former might be a very drastic change with dire consequences for users of other tables. :-/

The latter is what has been asked from the beginning, although I asked for it in different terms ("drop the filter") because I didn't know what "Big charset" meant, but that seems to be absolutely equivalent.

Note though that the default value should be based first on the input method, and only then on the locale.

Part of this seems to be worked on:
https://code.google.com/p/ibus/issues/detail?id=1188#c4

--- Additional comment from dchen on 2012-09-07 16:06:50 EST ---

I suggest we split this bug in to 2: one for ibus-table and the other for ibus-table-chinese for better tracking.