RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1164267 - sosreport: UnicodeDecodeError: 'utf8' codec can't decode byte 0xd2 in position 8003: invalid continuation byte
Summary: sosreport: UnicodeDecodeError: 'utf8' codec can't decode byte 0xd2 in positio...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: sos
Version: 7.1
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: rc
: ---
Assignee: Bryn M. Reeves
QA Contact: Martin Kyral
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-11-14 14:04 UTC by Bruno Goncalves
Modified: 2019-02-15 13:52 UTC (History)
5 users (show)

Fixed In Version: sos-3.2-9.el7
Doc Type: Bug Fix
Doc Text:
no docs needed
Clone Of:
Environment:
Last Closed: 2015-03-05 11:24:02 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2015:0532 0 normal SHIPPED_LIVE sos bug fix and enhancement update 2015-03-05 16:12:44 UTC

Description Bruno Goncalves 2014-11-14 14:04:57 UTC 
Description of problem:
The following traceback happens when running sosreport

  Running 32/73: logs...        
logs
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/sos/sosreport.py", line 1163, in collect
    plug.collect()
  File "/usr/lib/python2.7/site-packages/sos/plugins/__init__.py", line 629, in collect
    self._collect_cmd_output()
  File "/usr/lib/python2.7/site-packages/sos/plugins/__init__.py", line 609, in _collect_cmd_output
    timeout=timeout, runat=runat)
  File "/usr/lib/python2.7/site-packages/sos/plugins/__init__.py", line 554, in get_cmd_output_now
    result = self.get_command_output(exe, timeout=timeout, runat=runat)
  File "/usr/lib/python2.7/site-packages/sos/plugins/__init__.py", line 464, in get_command_output
    result = sos_get_command_output(prog, timeout=timeout, runat=runat)
  File "/usr/lib/python2.7/site-packages/sos/utilities.py", line 162, in sos_get_command_output
    return {'status': p.returncode, 'output': stdout.decode('utf-8')}
  File "/usr/lib64/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xd2 in position 8003: invalid continuation byte


Version-Release number of selected component (if applicable):
sos-3.2-7.el7.noarch

How reproducible:
100%

Steps to Reproduce:
1.run sosreport
# sosreport --batch
Comment 2 Bruno Goncalves 2014-11-14 14:16:14 UTC 
Hit the same problem on sos-3.2-3.el7.noarch

on sos-3.2-1.el7.noarch hit different problem as sosreport fails with:

# sosreport --batch
Traceback (most recent call last):
  File "/usr/sbin/sosreport", line 25, in <module>
    main(sys.argv[1:])
  File "/usr/lib/python2.7/site-packages/sos/sosreport.py", line 1407, in main
    sos = SoSReport(args)
  File "/usr/lib/python2.7/site-packages/sos/sosreport.py", line 633, in __init__
    self.policy = sos.policies.load()
  File "/usr/lib/python2.7/site-packages/sos/policies/__init__.py", line 40, in load
    cache['policy'] = policy()
  File "/usr/lib/python2.7/site-packages/sos/policies/redhat.py", line 122, in __init__
    super(RHELPolicy, self).__init__()
  File "/usr/lib/python2.7/site-packages/sos/policies/redhat.py", line 50, in __init__
    if self.package_manager.all_pkgs()['filesystem']['version'][0] == '3':
  File "/usr/lib/python2.7/site-packages/sos/policies/__init__.py", line 112, in all_pkgs
    self.packages = self.get_pkg_list()
  File "/usr/lib/python2.7/site-packages/sos/policies/__init__.py", line 95, in get_pkg_list
    pkg_list = shell_out(self.query_command).splitlines()
  File "/usr/lib/python2.7/site-packages/sos/utilities.py", line 185, in shell_out
    return sos_get_command_output(cmd, runat=runat)['output']
  File "/usr/lib/python2.7/site-packages/sos/utilities.py", line 142, in sos_get_command_output
    if six.PY2:
AttributeError: 'module' object has no attribute 'PY2'
Comment 5 Bryn M. Reeves 2014-11-14 14:45:50 UTC 
The problems in comment #0 and comment #2 have very different causes.

The first is a unicode UnicodeDecodeError exception due to an invalid byte sequence for the UTF-8 codec.

The second is an AttributeError due to a missing 'PY2' member in the python-six compatibility module: this was fixed in sos-six-compat.patch (added in sos-3.2-3.el7):

    commit 711093509012adbe8d6852f7dad6164c7a755fea
    Author: Bryn M. Reeves <bmr>
    Date:   Wed Oct 1 17:36:55 2014 +0100
    
        Packaging fix
        
        - archive class also needs PY2 vs. PY3 fix
        
        Resolves: bz1026962

For the unicode problem this isn't something I've seen in testing; what locale values are you using when you run sosreport?

E.g. the output of the locale command:

 $ locale
LANG=en_GB.UTF-8
LC_CTYPE="en_GB.UTF-8"
LC_NUMERIC="en_GB.UTF-8"
LC_TIME="en_GB.UTF-8"
LC_COLLATE="en_GB.UTF-8"
LC_MONETARY="en_GB.UTF-8"
LC_MESSAGES="en_GB.UTF-8"
LC_PAPER="en_GB.UTF-8"
LC_NAME="en_GB.UTF-8"
LC_ADDRESS="en_GB.UTF-8"
LC_TELEPHONE="en_GB.UTF-8"
LC_MEASUREMENT="en_GB.UTF-8"
LC_IDENTIFICATION="en_GB.UTF-8"
LC_ALL=
Comment 9 Bryn M. Reeves 2014-11-14 17:20:53 UTC 
I had a feeling this may be journalctl-related (it's the only command run from the logs plugin):

  Running 1/1: logs...        
[plugin:logs] collecting path '/var/log/secure'
[plugin:logs] collecting path '/var/log/messages'
[plugin:logs] collecting path '/etc/rsyslog.conf'
[plugin:logs] collecting path '/var/log/boot.log'
[plugin:logs] collecting output of 'journalctl --all --this-boot --no-pager'
Traceback (most recent call last):
  File "/usr/sbin/sosreport", line 25, in <module>
    main(sys.argv[1:])
  File "/usr/lib/python2.7/site-packages/sos/sosreport.py", line 1411, in main
    sos.execute()
UnicodeDecodeError: 'utf8' codec can't decode byte 0xd2 in position 8003: invalid continuation byte

> /usr/lib64/python2.7/encodings/utf_8.py(16)decode()
-> return codecs.utf_8_decode(input, errors, True)

Turns out the root cause is garbage characters in the kernel dmesg due to the HP Proliant BIOS:

<D2>? 0000162E)
Nov 15 00:16:08 hp-dl385pg8-03.rhts.eng.nay.redhat.com kernel: ACPI: FACP 00000000bdde1980 000F4 (v03 HP     ProLiant 00000002   <D2>? 0000162E)
Nov 15 00:16:08 hp-dl385pg8-03.rhts.eng.nay.redhat.com kernel: ACPI: DSDT 00000000bdde1a80 0D2F7 (v01 HP         DSDT 00000001 INTL 20061109)
Nov 15 00:16:08 hp-dl385pg8-03.rhts.eng.nay.redhat.com kernel: ACPI: FACS 00000000bddde140 00040
Nov 15 00:16:08 hp-dl385pg8-03.rhts.eng.nay.redhat.com kernel: ACPI: SPCR 00000000bddde180 00050 (v01 HP     SPCRRBSU 00000001   <D2>? 0000162E)
Nov 15 00:16:08 hp-dl385pg8-03.rhts.eng.nay.redhat.com kernel: ACPI: MCFG 00000000bddde200 0003C (v01 HP     ProLiant 00000001      00000000)
Nov 15 00:16:08 hp-dl385pg8-03.rhts.eng.nay.redhat.com kernel: ACPI: HPET 00000000bddde240 00038 (v01 HP     ProLiant 00000002   <D2>? 0000162E)
Nov 15 00:16:08 hp-dl385pg8-03.rhts.eng.nay.redhat.com kernel: ACPI: SPMI 00000000bddde280 00040 (v05 HP     ProLiant 00000001   <D2>? 0000162E)
Nov 15 00:16:08 hp-dl385pg8-03.rhts.eng.nay.redhat.com kernel: ACPI: ERST 00000000bddde2c0 00230 (v01 HP     ProLiant 00000001   <D2>? 0000162E)
Nov 15 00:16:08 hp-dl385pg8-03.rhts.eng.nay.redhat.com kernel: ACPI: APIC 00000000bddde500 0011E (v01 HP     ProLiant 00000002      00000000)
Nov 15 00:16:08 hp-dl385pg8-03.rhts.eng.nay.redhat.com kernel: ACPI: SRAT 00000000bddde800 002A0 (v02 AMD    AGESA    00000001 AMD  00000001)
Nov 15 00:16:08 hp-dl385pg8-03.rhts.eng.nay.redhat.com kernel: ACPI: FFFF 00000000bdddf000 00176 (v01 HP     ProLiant 00000001   <D2>? 0000162E)
Nov 15 00:16:08 hp-dl385pg8-03.rhts.eng.nay.redhat.com kernel: ACPI: BERT 00000000bdddf180 00030 (v01 HP     ProLiant 00000001   <D2>? 0000162E)
Nov 15 00:16:08 hp-dl385pg8-03.rhts.eng.nay.redhat.com kernel: ACPI: HEST 00000000bdddf1c0 0018C (v01 HP     ProLiant 00000001   <D2>? 0000162E)
Nov 15 00:16:08 hp-dl385pg8-03.rhts.eng.nay.redhat.com kernel: ACPI: FFFF 00000000bdddf380 00064 (v02 HP     ProLiant 00000002   <D2>? 0000162E)

So this is technically a HP BIOS bug.. Those 0xd2s are neither valid ASCII nor UTF-8.

I'll look for a way for us to handle this more gracefully.
Comment 17 errata-xmlrpc 2015-03-05 11:24:02 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-0532.html
Note You need to log in before you can comment on or make changes to this bug.