Bug 285631 - optparse isn't friendly to i18n/unicode
optparse isn't friendly to i18n/unicode
Status: CLOSED CURRENTRELEASE
Product: Fedora
Classification: Fedora
Component: python (Show other bugs)
10
All Linux
medium Severity medium
: ---
: ---
Assigned To: Dave Malcolm
Fedora Extras Quality Assurance
bzcl34nup
: Triaged
: 314061 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2007-09-11 02:08 EDT by Satoru SATOH
Modified: 2009-12-02 13:57 EST (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2009-12-02 13:57:47 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)

  None (edit)
Description Satoru SATOH 2007-09-11 02:08:17 EDT
Description of problem: ksvalidator doesn't show help and UnicodeDecodeError
exception raised.


Version-Release number of selected component (if applicable):
pykickstart-1.12-1.fc8

How reproducible: always


Steps to Reproduce:
1. set LANG to any (not C, en_US.UTF-8) UTF-8 lang; e.g. ja_JP.UTF-8, sv_SV.UTF-8.
2. run ksvalidator.

  
Actual results:

Here it is.

ssato@gescom% ksvalidator
Traceback (most recent call last):
  File "/usr/bin/ksvalidator", line 62, in <module>
    op.print_help()
  File "/usr/lib/python2.5/optparse.py", line 1648, in print_help
    file.write(self.format_help().encode(encoding, "replace"))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe6 in position 130:
ordinal not in range(128)
ssato@gescom% rpm -qf =ksvalidator
pykickstart-1.12-1.fc8
ssato@gescom% echo $LANG
ja_JP.UTF-8
ssato@gescom%
ssato@gescom% LANG=C ksvalidator
Usage: ksvalidator [options] ksfile|url

Options:
  -h, --help            show this help message and exit
  -e, --firsterror      halt after the first error or warning
  -i, --followincludes  parse include files when %include is seen
  -v VERSION, --version=VERSION
                        version of kickstart syntax to validate against
ssato@gescom% LANG=en_US.UTF-8 ksvalidator
Usage: ksvalidator [options] ksfile|url

Options:
  -h, --help            show this help message and exit
  -e, --firsterror      halt after the first error or warning
  -i, --followincludes  parse include files when %include is seen
  -v VERSION, --version=VERSION
                        version of kickstart syntax to validate against
ssato@gescom% 


Expected results: Show help.


Additional info:

A workaround fix (dirty hack :P) is below.

--- /usr/bin/ksvalidator        2007-09-05 01:29:32.000000000 +0900
+++ ksvalidator 2007-09-11 14:52:01.000000000 +0900
@@ -59,6 +59,11 @@
 (opts, extra) = op.parse_args(sys.argv[1:])
 
 if len(extra) != 1:
+    # Hack.
+    def help():
+        sys.stdout.write(unicode(op.format_help(),'utf-8').encode('utf-8',
"replace"))
+
+    setattr(op, 'print_help', help)
     op.print_help()
     os._exit(1)
 else:


It should not be a bug of pykickstart but optparse in python in actual.

It seems the cause came from the difference between unicode and str.

>>> s = 'テキスト表示'
>>> s.encode('utf-8')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe3 in position 0: ordinal
not in range(128)
>>> s = u'テキスト表示'
>>> s.encode('utf-8')
'\xe3\x83\x86\xe3\x82\xad\xe3\x82\xb9\xe3\x83\x88\xe8\xa1\xa8\xe7\xa4\xba'
>>>
Comment 1 Andy Shevchenko 2007-10-01 11:33:38 EDT
*** Bug 314061 has been marked as a duplicate of this bug. ***
Comment 2 Andy Shevchenko 2007-10-01 11:48:40 EDT
I thin is duplicate to bug #237956
Comment 3 Bug Zapper 2008-04-04 09:49:02 EDT
Based on the date this bug was created, it appears to have been reported
during the development of Fedora 8. In order to refocus our efforts as
a project we are changing the version of this bug to '8'.

If this bug still exists in rawhide, please change the version back to
rawhide.
(If you're unable to change the bug's version, add a comment to the bug
and someone will change it for you.)

Thanks for your help and we apologize for the interruption.

The process we're following is outlined here:
http://fedoraproject.org/wiki/BugZappers/F9CleanUp

We will be following the process here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping to ensure this
doesn't happen again.
Comment 4 Satoru SATOH 2008-06-09 22:18:50 EDT
It seems the problem still exists in the latest one (rawhide).

bash-3.2$ rpm -qf $(which ksvalidator)
pykickstart-1.37-1.fc10.noarch
bash-3.2$ rpm -qf $(which python)
python-2.5.1-25.fc9.i386
bash-3.2$ env | grep LANG
LANG=ja_JP.utf8
GDM_LANG=ja_JP.utf8
bash-3.2$ ksvalidator 
Traceback (most recent call last):
  File "/usr/bin/ksvalidator", line 62, in <module>
    op.print_help()
  File "/usr/lib/python2.5/optparse.py", line 1648, in print_help
    file.write(self.format_help().encode(encoding, "replace"))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe6 in position 130:
ordinal not in range(128)
bash-3.2$

Comment 5 James Antill 2008-06-09 23:45:25 EDT
 I've worked around similar bugs in yum, which is what I'd also recommend for
pykickstart.
 This might be helped along when we move to 2.6.x.
Comment 6 Bug Zapper 2008-11-25 20:59:09 EST
This bug appears to have been reported against 'rawhide' during the Fedora 10 development cycle.
Changing version to '10'.

More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 7 Bug Zapper 2009-11-18 03:12:32 EST
This message is a reminder that Fedora 10 is nearing its end of life.
Approximately 30 (thirty) days from now Fedora will stop maintaining
and issuing updates for Fedora 10.  It is Fedora's policy to close all
bug reports from releases that are no longer maintained.  At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '10'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 10's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 10 is end of life.  If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora please change the 'version' of this 
bug to the applicable version.  If you are unable to change the version, 
please add a comment here and someone will do it for you.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events.  Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 8 Dave Malcolm 2009-12-02 13:48:30 EST
Tested on F-12 with python-2.6.2 (and pykickstart-1.64-1.fc12.noarch) and it seems to be working OK (or, at least, not raising exceptions):

$ LANG=ja_JP.UTF-8 ksvalidator
Usage: ksvalidator [options] ksfile|url

Options:
  -h, --help            show this help message and exit
  -e, --firsterror      最初のエラーまたは警告で停止する
  -i, --followincludes  %include が表示された時に include ファイルを構文解析する
  -l, --listversions    list the available versions of kickstart syntax
  -v VERSION, --version=VERSION
                        確証対象のキックスタート構文のバージョン

$ LANG=sv_SV.UTF-8 ksvalidator
Usage: ksvalidator [options] ksfile|url

Options:
  -h, --help            show this help message and exit
  -e, --firsterror      stanna efter det f?rsta felet eller den f?rsta
                        varningen
  -i, --followincludes  tolka inkluderingsfiler d? %include hittas
  -l, --listversions    list the available versions of kickstart syntax
  -v VERSION, --version=VERSION
                        version av kickstart-syntax att validera mot
Comment 9 Dave Malcolm 2009-12-02 13:57:47 EST
Looks like this was http://bugs.python.org/issue1498146
which was fixed in http://svn.python.org/view?view=rev&revision=46861

(appears to have been released in Python 2.5 beta 2)

Note You need to log in before you can comment on or make changes to this bug.