Bug 1368765 - hunspell has problems with input above the BMP, causes this: [abrt] ibus-typing-booster: SuggestMgr::leftcommonsubstring(char*, char const*)(): python3.5 killed by SIGSEGV
Summary: hunspell has problems with input above the BMP, causes this: [abrt] ibus-typi...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: hunspell
Version: 24
Hardware: x86_64
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Caolan McNamara
QA Contact: Fedora Extras Quality Assurance
URL: https://retrace.fedoraproject.org/faf...
Whiteboard: abrt_hash:ce549929093186e3290aca1694c...
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-08-21 09:09 UTC by Mike FABIAN
Modified: 2016-09-01 16:53 UTC (History)
6 users (show)

Fixed In Version: hunspell-1.3.3-10.fc24
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-09-01 16:53:11 UTC
Type: ---


Attachments (Terms of Use)
File: backtrace (24.91 KB, text/plain)
2016-08-21 09:09 UTC, Mike FABIAN
no flags Details
File: cgroup (244 bytes, text/plain)
2016-08-21 09:09 UTC, Mike FABIAN
no flags Details
File: core_backtrace (16.65 KB, text/plain)
2016-08-21 09:09 UTC, Mike FABIAN
no flags Details
File: dso_list (8.18 KB, text/plain)
2016-08-21 09:09 UTC, Mike FABIAN
no flags Details
File: environ (2.74 KB, text/plain)
2016-08-21 09:09 UTC, Mike FABIAN
no flags Details
File: exploitable (82 bytes, text/plain)
2016-08-21 09:09 UTC, Mike FABIAN
no flags Details
File: limits (1.29 KB, text/plain)
2016-08-21 09:09 UTC, Mike FABIAN
no flags Details
File: maps (32.96 KB, text/plain)
2016-08-21 09:09 UTC, Mike FABIAN
no flags Details
File: mountinfo (4.20 KB, text/plain)
2016-08-21 09:09 UTC, Mike FABIAN
no flags Details
File: namespaces (102 bytes, text/plain)
2016-08-21 09:09 UTC, Mike FABIAN
no flags Details
File: open_fds (1.44 KB, text/plain)
2016-08-21 09:09 UTC, Mike FABIAN
no flags Details
File: proc_pid_status (1.12 KB, text/plain)
2016-08-21 09:09 UTC, Mike FABIAN
no flags Details
File: var_log_messages (28 bytes, text/plain)
2016-08-21 09:09 UTC, Mike FABIAN
no flags Details
python-enchant-crash.py (463 bytes, text/plain)
2016-08-21 09:56 UTC, Mike FABIAN
no flags Details
hunspell-conversion-problem.txt (19 bytes, text/plain)
2016-08-21 10:00 UTC, Mike FABIAN
no flags Details

Description Mike FABIAN 2016-08-21 09:09:17 UTC
Description of problem:
I tried to use hu-rovas-post.mim with the hu-HU engine of ibus-typing-booster

Version-Release number of selected component:
ibus-typing-booster-1.5.0-1.fc24

Additional info:
reporter:       libreport-2.7.2
backtrace_rating: 4
cmdline:        /usr/bin/python3 /usr/share/ibus-typing-booster/engine/main.py --profile --ibus
crash_function: SuggestMgr::leftcommonsubstring(char*, char const*)
executable:     /usr/bin/python3.5
global_pid:     9190
kernel:         4.6.4-301.fc24.x86_64
pkg_fingerprint: 73BD E983 81B4 6521
pkg_vendor:     Fedora Project
runlevel:       N 5
type:           CCpp
uid:            10030

Truncated backtrace:
Thread no. 0 (10 frames)
 #0 SuggestMgr::leftcommonsubstring(char*, char const*) at suggestmgr.cxx:1859
 #1 SuggestMgr::ngsuggest(char**, char*, int, HashMgr**, int) at suggestmgr.cxx:1106
 #2 Hunspell::suggest(char***, char const*) at hunspell.cxx:889
 #3 MySpellChecker::suggestWord(char const*, unsigned long, unsigned long*) at myspell_checker.cpp:197
 #4 enchant_dict_suggest at enchant.c:943
 #5 ffi_call_unix64 at ../src/x86/unix64.S:76
 #6 ffi_call at ../src/x86/ffi64.c:525
 #7 _ctypes_callproc at /usr/src/debug/Python-3.5.1/Modules/_ctypes/callproc.c:811
 #9 PyCFuncPtr_call at /usr/src/debug/Python-3.5.1/Modules/_ctypes/_ctypes.c:3869
 #10 PyObject_Call at /usr/src/debug/Python-3.5.1/Objects/abstract.c:2165

Comment 1 Mike FABIAN 2016-08-21 09:09:23 UTC
Created attachment 1192545 [details]
File: backtrace

Comment 2 Mike FABIAN 2016-08-21 09:09:25 UTC
Created attachment 1192546 [details]
File: cgroup

Comment 3 Mike FABIAN 2016-08-21 09:09:27 UTC
Created attachment 1192547 [details]
File: core_backtrace

Comment 4 Mike FABIAN 2016-08-21 09:09:28 UTC
Created attachment 1192548 [details]
File: dso_list

Comment 5 Mike FABIAN 2016-08-21 09:09:30 UTC
Created attachment 1192549 [details]
File: environ

Comment 6 Mike FABIAN 2016-08-21 09:09:31 UTC
Created attachment 1192550 [details]
File: exploitable

Comment 7 Mike FABIAN 2016-08-21 09:09:33 UTC
Created attachment 1192551 [details]
File: limits

Comment 8 Mike FABIAN 2016-08-21 09:09:35 UTC
Created attachment 1192552 [details]
File: maps

Comment 9 Mike FABIAN 2016-08-21 09:09:36 UTC
Created attachment 1192553 [details]
File: mountinfo

Comment 10 Mike FABIAN 2016-08-21 09:09:38 UTC
Created attachment 1192554 [details]
File: namespaces

Comment 11 Mike FABIAN 2016-08-21 09:09:40 UTC
Created attachment 1192555 [details]
File: open_fds

Comment 12 Mike FABIAN 2016-08-21 09:09:41 UTC
Created attachment 1192556 [details]
File: proc_pid_status

Comment 13 Mike FABIAN 2016-08-21 09:09:43 UTC
Created attachment 1192557 [details]
File: var_log_messages

Comment 14 Mike FABIAN 2016-08-21 09:56:04 UTC
Created attachment 1192563 [details]
python-enchant-crash.py

It crashes, because python3-enchant crashes:

$ python3 python-enchant-crash.py 
['Budapest', 'Budapesti', 'Budapesté', 'Budapestű']
[b'Budapest', b'Budapesti', b'Budapest\xc3\xa9', b'Budapest\xc5\xb1']
This UTF-8 encoding can't convert to UTF-16:
𐲂𐳪𐳇𐳀𐳠𐳉𐳤𐳦
This UTF-8 encoding can't convert to UTF-16:
𐲂𐳪𐳇𐳀𐳠𐳉𐳤𐳦
This UTF-8 encoding can't convert to UTF-16:
𐲂𐳪𐳇𐳀𐳠𐳉𐳤𐳦
Segmentation fault (コアダンプ)
mfabian@ari:~
$

Comment 15 Mike FABIAN 2016-08-21 10:00:52 UTC
Created attachment 1192564 [details]
hunspell-conversion-problem.txt

python3-enchant probably crashes because of this problem in hunspell:

$ hunspell -d hu_HU -i utf-8 -l hunspell-conversion-problem.txt 
hBudapxst
This UTF-8 encoding can't convert to UTF-16:
😇
This UTF-8 encoding can't convert to UTF-16:
😇
This UTF-8 encoding can't convert to UTF-16:
𐳠
This UTF-8 encoding can't convert to UTF-16:
𐳠
mfabian@ari:~
$ 

Of course the file converts to UTF-16 just fine:

$ iconv -f utf-8 -t utf-16 < hunspell-conversion-problem.txt | iconv -f utf-16 -t utf-8 
Budapxst
😇
𐳠
mfabian@ari:~
$ 

It looks like hunspell has problems with characters above the BMP (Basic Multilingual Plane).

Comment 16 Mike FABIAN 2016-08-21 11:46:31 UTC
It works on current rawhide with hunspell-1.4.1:

[mfabian@Fedora-Workstation-netinst-x86_6 ~]$ cat /etc/fedora-release 
Fedora release 26 (Rawhide)
[mfabian@Fedora-Workstation-netinst-x86_6 ~]$ python3 python-enchant-crash.py 
['Budapest', 'Budapesti', 'Budapesté', 'Budapestű']
[b'Budapest', b'Budapesti', b'Budapest\xc3\xa9', b'Budapest\xc5\xb1']
[]
[]
[mfabian@Fedora-Workstation-netinst-x86_6 ~]$ rpm -q hunspell
hunspell-1.4.1-1.fc25.x86_64
[mfabian@Fedora-Workstation-netinst-x86_6 ~]$ hunspell -d hu_HU -i utf-8 -l hunspell-conversion-problem.txt 
Budapxst
[mfabian@Fedora-Workstation-netinst-x86_6 ~]$

Comment 17 Mike FABIAN 2016-08-21 11:47:01 UTC
Can the fix be backported to f24?

Comment 18 Caolan McNamara 2016-08-22 16:37:15 UTC
Good to know the big rework of stuff in hunspell had a practical worthwhile effect. I'll have to bisect to find when it started working to see what exactly was the cause to see if its backportable in isolation.

Comment 19 Fedora Update System 2016-08-29 11:47:24 UTC
hunspell-1.3.3-10.fc24 has been submitted as an update to Fedora 24. https://bodhi.fedoraproject.org/updates/FEDORA-2016-1a8b18ee44

Comment 20 Fedora Update System 2016-08-29 22:52:32 UTC
hunspell-1.3.3-10.fc24 has been pushed to the Fedora 24 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-1a8b18ee44

Comment 21 Fedora Update System 2016-09-01 16:53:06 UTC
hunspell-1.3.3-10.fc24 has been pushed to the Fedora 24 stable repository. If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.