Bug 1202022

There are actually two issues mariadb reported for Fedora: bug #1187469 that non-ASCII characters don't work in interactive mode and bug #1180403 that backward search doesn't work -- both caused by missing features of libedit's readline() function, in comparison to readline's readline() :)

The problem is that mysql binary uses just readline() function from libedit library and this function only offers basic functionality. So the possible fix is either to use el_wgets() function from libedit and configure the backward search feature in mysql binary or (imho better for much more components) fix libedit's readline() function to offer this by default (or as another function, but just offer a simpler usage, as it does readline library).

+++ This bug was initially created as a clone of Bug #1187469 +++

Description of problem:
When I type or paste Chinese characters in mysql client, the Chinese characters are not inputed.

My `locale` result:
    LANG=zh_CN.utf8
    LC_CTYPE="zh_CN.utf8"
    LC_NUMERIC="zh_CN.utf8"
    LC_TIME="zh_CN.utf8"
    LC_COLLATE="zh_CN.utf8"
    LC_MONETARY="zh_CN.utf8"
    LC_MESSAGES="zh_CN.utf8"
    LC_PAPER="zh_CN.utf8"
    LC_NAME="zh_CN.utf8"
    LC_ADDRESS="zh_CN.utf8"
    LC_TELEPHONE="zh_CN.utf8"
    LC_MEASUREMENT="zh_CN.utf8"
    LC_IDENTIFICATION="zh_CN.utf8"
    LC_ALL=
     

My `localectl` result
       System Locale: LANG=zh_CN.UTF-8
           VC Keymap: us
          X11 Layout: us,cn
         X11 Variant: ,
     

Version-Release number of selected component (if applicable):
10.0.x


How reproducible:
Always.

Steps to Reproduce:
1. run mysql client
2. copy & paste the following clauses in mysql client:
   SELECT 'Chinese characters <汉字> are stripped';
3. '汉字' are stripped out in mysql client

Actual results:
MariaDB [(none)]> SELECT 'Chinese characters <> are stripped';
+------------------------------------+
| Chinese characters <> are stripped |
+------------------------------------+
| Chinese characters <> are stripped |
+------------------------------------+


Expected results:
MariaDB [(none)]> SELECT 'Chinese characters <汉字> are stripped';
+------------------------------------------+
| Chinese characters <汉字> are stripped   |
+------------------------------------------+
| Chinese characters <汉字> are stripped   |
+------------------------------------------+


Additional info:
The binaries from mariadb.org works ok:
1. win32 version works ok
2. Linux x86_64 works ok
3. Linux glibc_2.14+ x86_64 works ok

and a user in #mariadb IRC channel report it works in Ubuntu.

The 5.5.x version in Fedora 20 also works.

But, I can choose the history command (which is issued in Linux x86_64 version from mariadb.org, and contains Chinese characters) and the history command works ok.

--- Additional comment from Jan Staněk on 2015-02-03 08:03:44 EST ---

Hi, the trouble seems to originate from the fact that unlike the upstream-provided binaries, fedora's mariadb packages uses libedit for cli client input handling (you could verify that by invoking mysql -V; the output should end with "using EditLine wrapper" if using libedit). I also tried to input some czech chracters not in ASCII range (č, š, etc.) and was not able to input them, so this issue is not limited to Chinese.

According to quick google search, this library had in the past issues with handling UTF-8 input. The official page [1] however claims UTF-8 support, if compiled with correct configure options (which the version included in Fedora is).
So this might be fault of either libedit itself, or it could be that mariadb has wrapped it badly. Asking for input from Boris Ranto, maintainer of libedit.

Boris, do you know if current version of libedit truly supports UTF-8 input?

[1] http://thrysoee.dk/editline/

--- Additional comment from Boris Ranto on 2015-02-26 10:59:35 EST ---

Hi, it does support utf-8 but the support is not as straightforward as you might expect. The source code needs to be modified in order to support utf-8. Please see file examples/wtc1.c provided in dist tarball [1] for more details on how to benefit from the utf-8 support.

[1] http://thrysoee.dk/editline/libedit-20141030-3.1.tar.gz

--- Additional comment from LiuYan on 2015-03-09 03:18:27 EDT ---

I found this feature request of gnuplot project from search engine:
https://sourceforge.net/p/gnuplot/feature-requests/265/#112c
https://sourceforge.net/p/gnuplot/feature-requests/265/#c343

>> Feeding it a UTF8 string causes a segfault.
So far as I can make out, it does not even try to support UTF8. Instead it supports MS-style wide characters (16-bit fixed width), which is not very useful in a UTF8 environment. 

>> I see no mention of UTF encodings in the editline source code. I think there is a typo or misunderstanding in the description that you saw. It would in any case do us no good to convert UTF-8 on input to widechar, because the internal gnuplot code and the support libraries (at least on linux) do not use widechar. We really do want the UTF-8 byte sequence passed through without any conversion.

I'm not sure if it's useful, because the reply date is 2010-09-09 (about 4.5 years ago). maybe editline had added utf-8 support in these years, I don't know.

But if those reply messages are still be true, then maybe mariadb need to be build against readline like the official mariadb binaries did to support UTF-8 characters.

--- Additional comment from Honza Horak on 2015-03-14 06:45:23 EDT ---

I think it may be fixed directly in mariadb or in libedit, we just need to find the best way. Just reported: https://mariadb.atlassian.net/browse/MDEV-7777