Bug 1801159

Summary:	subunit2html failed with "UnicodeDecodeError: 'ascii' codec can't decode byte 0xef in position ..."
Product:	Red Hat OpenStack	Reporter:	lkuchlan <lkuchlan>
Component:	python-os-testr	Assignee:	Chandan Kumar <chkumar>
Status:	CLOSED ERRATA	QA Contact:	Martin Kopec <mkopec>
Severity:	high	Docs Contact:
Priority:	high
Version:	13.0 (Queens)	CC:	amoralej, apevec, chkumar, dsariel, jschluet, lhh, ltoscano, udesale
Target Milestone:	---	Keywords:	AutomationBlocker, Triaged, ZStream
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	python-os-testr-1.0.1-0.20200218144109.7dd678e.el7ost	Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-03-10 11:53:27 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description lkuchlan 2020-02-10 11:49:48 UTC

The exception is caused by the following code line:
https://github.com/openstack/os-testr/blob/master/os_testr/subunit2html.py#L685

683 script = TemplateData.REPORT_TEST_OUTPUT_TMPL % dict(
684 id=tid,
685 -> output=saxutils.escape(o + e),
686 )
687
688 row = tmpl % dict(
689 tid=tid,
690 Class=((n == 0 or n == 3) and 'hiddenRow' or 'none'),
##################################
For some reason the type of variable is different between o to e, as you can see below:
(Pdb) type(o)
<type 'unicode'>
(Pdb) type(e)
<type 'str'>
(Pdb) o + e
*** UnicodeDecodeError: 'ascii' codec can't decode byte 0xef in position 123639: ordinal not in range(128)

Actual results:

Traceback (most recent call last):
  File "/usr/bin/subunit2html", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python2.7/site-packages/os_testr/subunit2html.py", line 763, in main
    result.stopTestRun()
  File "/usr/lib/python2.7/site-packages/testtools/testresult/real.py", line 549, in stopTestRun
    sink.stopTestRun()
  File "/usr/lib/python2.7/site-packages/testtools/testresult/real.py", line 1775, in stopTestRun
    self.decorated.stopTestRun()
  File "/usr/lib/python2.7/site-packages/testtools/testresult/real.py", line 1529, in stopTestRun
    return self.decorated.stopTestRun()
  File "/usr/lib/python2.7/site-packages/os_testr/subunit2html.py", line 517, in stopTestRun
    report = self._generate_report()
  File "/usr/lib/python2.7/site-packages/os_testr/subunit2html.py", line 604, in _generate_report
    self._generate_report_test(rows, cid, tid, n, t, o, e)
  File "/usr/lib/python2.7/site-packages/os_testr/subunit2html.py", line 688, in _generate_report_test
    output=saxutils.escape(o + e),
UnicodeDecodeError: 'ascii' codec can't decode byte 0xef in position 123639: ordinal not in range(128)

Comment 1 David Sariel 2020-02-18 04:53:41 UTC

hi, changed assignee to default. lkuchlan was opened the issue.  Any ETA for the fix? 

There is a partial workaround [1] for this issue. The problem with [1] is that manila tempest results will be missed 
due to the presence of the unicode char. So it is still an AutomationBlocker for manila. 



[1]
https://review.gerrithub.io/c/redhat-openstack/infrared/+/478718

Comment 2 lkuchlan 2020-02-18 06:35:12 UTC

(In reply to David Sariel from comment #1)
> hi, changed assignee to default. lkuchlan was opened the issue.  Any ETA for
> the fix? 

There is already fix for this https://review.opendev.org/#/c/700778/


> 
> There is a partial workaround [1] for this issue. The problem with [1] is
> that manila tempest results will be missed 
> due to the presence of the unicode char. So it is still an AutomationBlocker
> for manila. 
> 
> 
> 
> [1]
> https://review.gerrithub.io/c/redhat-openstack/infrared/+/478718

Comment 3 David Sariel 2020-02-18 10:32:27 UTC

@lkuchlan, merging 700778 is definitely required. But osp13 puddle shell be also updated with the fixed version 
the os-testr. That's the reason it should be assigned to the python-os-testr guys that will package it downstream
and for some reason it was assigned to you.

Comment 4 Martin Kopec 2020-02-18 16:18:32 UTC

The reviews is being reviewed upstream. I pinged a few people and brought it up during QA office hours. I expect it will be merged soon. After that it's a matter only a few days it's backported and a new package is built. Then it will need to be included in a puddle, I need to check when the next z-stream for RHOS 13 is gonna happen.

Comment 5 Luigi Toscano 2020-02-18 16:32:01 UTC

(In reply to Martin Kopec from comment #4)
> The reviews is being reviewed upstream. I pinged a few people and brought it
> up during QA office hours. I expect it will be merged soon. 

But it may be dropped, as it is for py2 only and there is an open review to drop py2: https://review.opendev.org/#/c/705522/

> After that it's
> a matter only a few days it's backported and a new package is built. 
We need a manual backport it anyway (os-testr 1.0 vs 1.1).

Comment 17 Martin Kopec 2020-02-24 16:06:05 UTC

Automated jobs are passing with the 'Fixed in version' package.

Comment 19 errata-xmlrpc 2020-03-10 11:53:27 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0769