Bug 1235061 - smbios-utils-ctl fails to run due to UnicodeDecodeError error
Summary: smbios-utils-ctl fails to run due to UnicodeDecodeError error
Keywords:
Status: CLOSED EOL
Alias: None
Product: Fedora
Classification: Fedora
Component: libsmbios
Version: 22
Hardware: x86_64
OS: All
unspecified
unspecified
Target Milestone: ---
Assignee: srinivas gowda
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-06-23 22:14 UTC by Scott Williams
Modified: 2016-07-19 15:00 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-07-19 15:00:37 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
dmidecode output (28.71 KB, text/plain)
2015-06-23 22:14 UTC, Scott Williams
no flags Details

Description Scott Williams 2015-06-23 22:14:51 UTC
Created attachment 1042551 [details]
dmidecode output

Description of problem:

smbios-token-ctl from the smbios-utils python package, doesn't run in Fedora due to an UnicodeDecodeError

Version-Release number of selected component (if applicable):
smbios-utils-python-2.2.28-11.fc22.x86_64

How reproducible:
Every time

Steps to Reproduce:

# smbios-token-ctl 
================================================================================
  Token: 0x0005 - Serial Port 1 (COM2)
  value: bool = false
   Desc: Configure the systemTraceback (most recent call last):
  File "/sbin/smbios-token-ctl", line 478, in <module>
    sys.exit( main() )
  File "/sbin/smbios-token-ctl", line 385, in main
    dumpTokens(tokenTable, tokenXlator, options)
  File "/sbin/smbios-token-ctl", line 243, in dumpTokens
    cli.wrap(translatedToken.description, indent=len(desc), first_line_start=len(desc))
  File "/usr/share/smbios-utils/cli.py", line 111, in wrap
    sys.stdout.write(c)
  File "/usr/lib64/python2.7/codecs.py", line 357, in write
    data, consumed = self.encode(object, self.errors)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 0: ordinal not in range(128)



Actual results:
Running command immediately yields a python trace.

Expected results:
Able to run command or error gracefully.

Additional info:

Attached dmidecode output for associated hardware reference.

Comment 1 Scott Williams 2015-06-23 22:23:13 UTC
Adding this line before appears to fix it at /usr/share/smbios-utils/cli.py:111

        if not isinstance(c,unicode): c = unicode(c,errors='ignore')
        sys.stdout.write(c)

Comment 2 Scott Williams 2015-07-09 21:21:59 UTC
Since it's just one line I didn't do a full patch.  Is it necessary for me to do so?  It's still working fine for me here with that line added.  It just validates that it's passing the type that it's expecting (unicode).

Scott

Comment 3 Scott Williams 2015-07-09 21:35:00 UTC
It looks like this bug goes back to Fedora 17 with the same issue and that bug was closed EOL.  It's still a bug in Fedora 22.

ref. https://bugzilla.redhat.com/show_bug.cgi?id=990296

Comment 4 srinivas gowda 2015-07-14 15:34:04 UTC
(In reply to Scott Williams from comment #1)
> Adding this line before appears to fix it at
> /usr/share/smbios-utils/cli.py:111
> 
>         if not isinstance(c,unicode): c = unicode(c,errors='ignore')
>         sys.stdout.write(c)


This patch doesnt work for me, I still get UnicodeEncodeError!!! 
I am using a Fedora22 docker image for testing. But that shouldn't matter 

git diff src/py-cli/cli.py 
diff --git a/src/py-cli/cli.py b/src/py-cli/cli.py
index f0a1776..0a06b85 100644
--- a/src/py-cli/cli.py
+++ b/src/py-cli/cli.py
@@ -108,6 +108,7 @@ def wrap(s, line_len=80, indent=0, first_line_indent=0, first_line_start=0):
     sys.stdout.write(" "*first_line_indent)
     chars_printed = first_line_start
     for c in s:
+        if not isinstance(c,unicode): c = unicode(c,errors='ignore')
         sys.stdout.write(c)
         chars_printed = chars_printed + 1
         if chars_printed >= line_len:
bash-4.3# 
bash-4.3# 
bash-4.3# ./src/bin/smbios-token-ctl 
================================================================================
  Token: 0x0005 - Serial Port 1 (COM2)
  value: bool = false
   Desc: Configure the systemTraceback (most recent call last):
  File "./src/bin/smbios-token-ctl", line 486, in <module>
    sys.exit( main() )
  File "./src/bin/smbios-token-ctl", line 393, in main
    dumpTokens(tokenTable, tokenXlator, options)
  File "./src/bin/smbios-token-ctl", line 251, in dumpTokens
    cli.wrap(translatedToken.description, indent=len(desc), first_line_start=len(desc))
  File "/libsmbios/src/bin/../py-cli/cli.py", line 112, in wrap
    sys.stdout.write(c)
  File "/usr/lib64/python2.7/codecs.py", line 351, in write
    data, consumed = self.encode(object, self.errors)
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2019' in position 0: ordinal not in range(128)

Comment 5 srinivas gowda 2015-07-14 15:40:44 UTC
I believe variable "c" will always be unicode, so "if not isinstance(c,unicode):"  doesnt get hit at all.. 

From what I have tested "c = unicode(c,errors='ignore')" gives me UnicodeEncodeError:  . That`s weird.  



Can you please try with this patch. 


diff --git a/src/py-cli/cli.py b/src/py-cli/cli.py
index f0a1776..32d0055 100644
--- a/src/py-cli/cli.py
+++ b/src/py-cli/cli.py
@@ -108,6 +108,9 @@ def wrap(s, line_len=80, indent=0, first_line_indent=0, first_line_start=0):
     sys.stdout.write(" "*first_line_indent)
     chars_printed = first_line_start
     for c in s:
+        if isinstance(c,unicode): c = c.encode('ascii', 'ignore')
         sys.stdout.write(c)
         chars_printed = chars_printed + 1
         if chars_printed >= line_len:

Comment 6 srinivas gowda 2015-07-14 16:02:31 UTC
(In reply to srinivas gowda from comment #5)
> I believe variable "c" will always be unicode, so "if not
> isinstance(c,unicode):"  doesnt get hit at all.. 

>      sys.stdout.write(" "*first_line_indent)
>      chars_printed = first_line_start
>      for c in s:
> +        if isinstance(c,unicode): c = c.encode('ascii', 'ignore')
>          sys.stdout.write(c)
>          chars_printed = chars_printed + 1
>          if chars_printed >= line_len:

Just realized what I did here...!!! 

please ignore.

Comment 7 Sujith 2015-08-04 13:06:08 UTC
Updates:
I can see this issue in Fedora-22 smbios-utils package. 
This looks to be mainly because of 2 different kinds of apostrophes in the token_list.csv file.

The patch proposed by Scott is to ignore the erroneous characters from getting displayed, which is not advisable since it leads to grammatical errors in the displayed message.

I feel there are two approaches that can be taken here. Both of these approaches worked for me. Note that these are tested only in a machine with Fedora-22 Server as of now. 

Yet to verify with other Distros and with upstream version of libsmbios.

Packages used in this scenario: 
smbios-utils-2.2.28-11.fc22.x86_64.rpm 
libsmbios-2.2.28-11.fc22.x86_64.rpm
libsmbios-devel-2.2.28-11.fc22.x86_64.rpm

1. Replace all the apostrophes in token_list.csv with the proper one, preferably with the one which is typed using keyboard.

2. Or use the following patch with no change to token_list.csv file:
diff --git a/src/py-cli/cli.py b/src/py-cli/cli.py
index f0a1776..a5e1c03 100644
--- a/src/py-cli/cli.py
+++ b/src/py-cli/cli.py
@@ -107,6 +107,7 @@ def setupLogging(configFile, verbosity=1, trace=0):
 def wrap(s, line_len=80, indent=0, first_line_indent=0, first_line_start=0):
     sys.stdout.write(" "*first_line_indent)
     chars_printed = first_line_start
+    s = s.decode("utf-8")
     for c in s:
         sys.stdout.write(c)
         chars_printed = chars_printed + 1

Comment 9 Scott Williams 2016-03-08 00:54:48 UTC
I noticed this is till flagged NEEDINFO, but I don't see what info is needed from me.  Please let me know if there's something on bodhi I can test?

Thanks,
Scott

Comment 10 Fedora End Of Life 2016-07-19 15:00:37 UTC
Fedora 22 changed to end-of-life (EOL) status on 2016-07-19. Fedora 22 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.


Note You need to log in before you can comment on or make changes to this bug.