Bug 876293

Summary: UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 56: ordinal not in range(128)
Product: [Fedora] Fedora Reporter: Steve Tyler <stephent98>
Component: pykickstartAssignee: Vratislav Podzimek <vpodzime>
Status: CLOSED CURRENTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 18CC: anaconda-maint-list, bcl, clumens, g.kaviyarasu, jonathan, sbueno, stephent98, vanmeeuwen+fedora, vpodzime
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Unspecified   
Whiteboard: abrt_hash:15cf01368343ae149bb9964133a35f6d8fd2dd50bc3212a01c284e23fa2447aa
Fixed In Version: 1.99.23-1 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-04-10 14:37:48 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
File: anaconda-tb
none
File: environ
none
File: type
none
File: ifcfg.log
none
File: storage.log
none
File: version
none
File: program.log
none
File: executable
none
File: product
none
File: anaconda.log
none
File: hashmarkername
none
File: packaging.log
none
File: cmdline_file
none
File: release
none
File: messages
none
File: other involved packages
none
screenshot showing NTP configuration dialog after adding '中文 (台湾)' as a server name none

Description Steve Tyler 2012-11-13 18:36:23 UTC
Description of problem:
Test install with zh_TW.

Version-Release number of selected component:
anaconda-18.29-1.fc18.x86_64

Additional info:
libreport version: 2.0.17
cmdline:        /usr/bin/python  /sbin/anaconda --liveinst --method=livecd:///dev/mapper/live-osimg-min --lang en_US.UTF-8
kernel:         3.6.6-3.fc18.x86_64

description:
:The following was filed automatically by anaconda:
:anaconda 18.29 exception report
:Traceback (most recent call first):
:  File "/usr/lib/python2.7/site-packages/pykickstart/base.py", line 284, in __str__
:    retval += obj.__str__()
:  File "/usr/lib64/python2.7/site-packages/pyanaconda/install.py", line 47, in _writeKS
:    f.write(str(ksdata))
:  File "/usr/lib64/python2.7/site-packages/pyanaconda/install.py", line 86, in doConfiguration
:    _writeKS(ksdata)
:  File "/usr/lib64/python2.7/threading.py", line 504, in run
:    self.__target(*self.__args, **self.__kwargs)
:  File "/usr/lib64/python2.7/site-packages/pyanaconda/threads.py", line 91, in run
:    threading.Thread.run(self, *args, **kwargs)
:UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 56: ordinal not in range(128)

Comment 1 Steve Tyler 2012-11-13 18:36:31 UTC
Created attachment 644304 [details]
File: anaconda-tb

Comment 2 Steve Tyler 2012-11-13 18:36:33 UTC
Created attachment 644305 [details]
File: environ

Comment 3 Steve Tyler 2012-11-13 18:36:34 UTC
Created attachment 644306 [details]
File: type

Comment 4 Steve Tyler 2012-11-13 18:36:35 UTC
Created attachment 644307 [details]
File: ifcfg.log

Comment 5 Steve Tyler 2012-11-13 18:36:42 UTC
Created attachment 644308 [details]
File: storage.log

Comment 6 Steve Tyler 2012-11-13 18:36:44 UTC
Created attachment 644309 [details]
File: version

Comment 7 Steve Tyler 2012-11-13 18:36:48 UTC
Created attachment 644310 [details]
File: program.log

Comment 8 Steve Tyler 2012-11-13 18:36:49 UTC
Created attachment 644311 [details]
File: executable

Comment 9 Steve Tyler 2012-11-13 18:36:51 UTC
Created attachment 644312 [details]
File: product

Comment 10 Steve Tyler 2012-11-13 18:36:53 UTC
Created attachment 644313 [details]
File: anaconda.log

Comment 11 Steve Tyler 2012-11-13 18:36:58 UTC
Created attachment 644315 [details]
File: hashmarkername

Comment 12 Steve Tyler 2012-11-13 18:37:00 UTC
Created attachment 644316 [details]
File: packaging.log

Comment 13 Steve Tyler 2012-11-13 18:37:02 UTC
Created attachment 644317 [details]
File: cmdline_file

Comment 14 Steve Tyler 2012-11-13 18:37:03 UTC
Created attachment 644318 [details]
File: release

Comment 15 Steve Tyler 2012-11-13 18:37:07 UTC
Created attachment 644319 [details]
File: messages

Comment 16 Steve Tyler 2012-11-13 18:37:09 UTC
Created attachment 644320 [details]
File: other involved packages

Comment 17 Steve Tyler 2012-11-14 08:49:05 UTC
Attempt to reproduce Bug 876293.

Package: anaconda-18.29-1.fc18.x86_64
OS Release: Fedora release 18

Comment 18 Steve Tyler 2012-11-14 09:01:15 UTC
Created attachment 644677 [details]
screenshot showing NTP configuration dialog after adding '中文 (台湾)' as a server name

Here is a detailed reproducer:

Choose 'Chinese (Taiwan)' for the install language.
Click Date & Time.
Click gear icon to configure NTP.
Enter '中文 (台湾)' in the text box.[1]
Click '+' to add.
Uncheck the ntp.org entries. (See attached screenshot.)
Click 'O' to save.
Click Done.
Configure disk:
    Delete all preexisting partitions.
    Auto-create partitions and filesystems.
Click Begin Installation.
Set root password.
Exception occurs while running post install scripts.
(The installer may appear to hang.)

[1] The Chinese characters are copied from:
>>> import babel
>>> print babel.Locale.parse('zh_TW').display_name
中文 (台湾)

Comment 19 Steve Tyler 2012-11-14 09:28:34 UTC
This can also be reproduced by installing in English and entering '中文 (台湾)' as an NTP server.

Comment 20 Steve Tyler 2012-11-14 17:44:00 UTC
See also Bug 865869 re input validation.

Comment 21 Steve Tyler 2012-11-14 19:06:35 UTC
I couldn't find an actual NTP server with non-ASCII characters in its name, so I made one up that is a reproducer: :-)

españa.ntp.org

Stratum One Time Servers
http://support.ntp.org/bin/view/Servers/StratumOneTimeServers

Stratum Two Time Servers
http://support.ntp.org/bin/view/Servers/StratumTwoTimeServers

Comment 22 Vratislav Podzimek 2012-11-15 12:56:15 UTC
This is not only an issue of NTP servers and should be resolved in a general way.

Comment 23 Vratislav Podzimek 2012-11-30 11:29:12 UTC
(In reply to comment #22)
> This is not only an issue of NTP servers and should be resolved in a general
> way.
This is a problem in pykickstart itself. I've changed the component and sent a patch to anaconda-patches.

Comment 24 Steve Tyler 2012-11-30 18:28:46 UTC
[pykickstart] [PATCH] Beware of possibly unicode strings (#876293)
https://lists.fedorahosted.org/pipermail/anaconda-patches/2012-November/002272.html

Thanks.

This is basically a type-conversion problem. Isn't there a way to get Python to do the type conversion automatically, so we don't have to keep finding and fixing bugs like this?

Comment 25 Vratislav Podzimek 2012-12-03 07:18:09 UTC
(In reply to comment #24)
> [pykickstart] [PATCH] Beware of possibly unicode strings (#876293)
> https://lists.fedorahosted.org/pipermail/anaconda-patches/2012-November/
> 002272.html
> 
> Thanks.
> 
> This is basically a type-conversion problem. Isn't there a way to get Python
> to do the type conversion automatically, so we don't have to keep finding
> and fixing bugs like this?
I don't know about any such way. Python3 helps with that [1], but in Python 2.X one has to do the conversions and use unicode strings or utf8-encoded byte strings consistently. This is a bit difficult when Anaconda uses so many different tools returning us strings. Thus I'm afraid we are just going to keep fixing bugs like this, at least until we switch to Python3.

Comment 26 Vratislav Podzimek 2012-12-03 07:18:52 UTC
[1] http://docs.python.org/3.0/howto/unicode.html#the-string-type