Bug 1437402

Summary:	Outputting UTF8 characters from openstack client when using CSV format fails
Product:	Red Hat OpenStack	Reporter:	Robin Cernin <rcernin>
Component:	python-cliff	Assignee:	Julie Pichon <jpichon>
Status:	CLOSED ERRATA	QA Contact:
Severity:	medium	Docs Contact:
Priority:	medium
Version:	10.0 (Newton)	CC:	akaris, amcleod, apevec, augol, gkadam, jdennis, jpichon, jruzicka, justinas.balciunas, lhh, pablo.iranzo, rcernin, rlondhe, smykhail, srevivo
Target Milestone:	z9	Keywords:	Triaged, ZStream
Target Release:	10.0 (Newton)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	python-cliff-2.2.0-2.el7ost	Doc Type:	Bug Fix
Doc Text:	Previously, commands with output that contained non-ascii characters failed when the command was run with --format=csv. With this update, command output that contains unicode characters displays correctly.	Story Points:	---
Clone Of:
Clones:	1547477 (view as bug list)		Environment:
Last Closed:	2018-09-17 16:59:16 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Robin Cernin 2017-03-30 09:12:07 UTC

Description of problem:

Error: Could not prefetch keystone_tenant provider 'openstack': Execution of '/usr/bin/openstack project list --quiet --format csv --long' returned 1: 'ascii' codec can't decode byte 0xc3 in position 77: ordinal not in range(128)
"ID","Name","Domain ID","Description","Enabled"

Version-Release number of selected component (if applicable):

RHOSP10

How reproducible:

In the project list, one of the project had description "sébastien"

After changing the description to "sebastien" the project list worked.

We have similar bug already reported in Kilo for keystone client: 

https://bugzilla.redhat.com/show_bug.cgi?id=1327893

Comment 2 Julie Pichon 2017-09-25 15:55:24 UTC

This should already be fixed through https://bugs.launchpad.net/python-cliff/+bug/1481014 (Liberty), but although unicodecsv is still in use in the code, it doesn't seem to work as expected anymore.

On OSP9:
--------
$ openstack project create tést
$ openstack project list --format=csv
"ID","Name"
"577a2f5ff28e45fbbdae0a99923ea284","tést"
"5d8df9b090e64b888d2fcf4c859de591","admin"
"b6bc76994f94472b86b74d7546eab810","services"
"ef843a5065f04ccd8c9a09972b46c251","demo"

python-unicodecsv-0.14.1-2.el7ost.noarch
python-openstackclient-2.3.1-1.el7ost.noarch
python-cliff-2.0.0-1.el7ost.noarch

On OSP10:
---------
$ openstack project create tést
$ openstack project list --format=csv
"ID","Name"
"7590c86147bc4de996a363ef206a990f","service"
'ascii' codec can't decode byte 0xc3 in position 37: ordinal not in range(128)

python-unicodecsv-0.14.1-2.el7ost.noarch
python-openstackclient-3.2.1-2.el7ost.noarch
python-cliff-2.2.0-1.el7ost.noarch

Comment 3 Julie Pichon 2017-09-26 10:21:03 UTC

It looks like one of the fixes for bug 1327893 may have introduced an issue with the unicodecsv package. My tests suggest that commit 5dc9b9a 'Merge "Avoid ASCII encoding errors when output is redirected"' in cliff is when the problem starts.

Looking around it seems like unicodecsv already does its own encoding conversion and so shouldn't be used together with the codecs module. To confirm, I removed the line "stdout = codecs.getwriter(encoding)(sys.stdout)" and the --format=csv commands started working again with unicode characters.

John, I'm adding you as a needinfo in case you have an idea or pointers, as you are the author of the patch and have been incredibly helpful with unicode issues so far. I'll keep investigating on my side as well, perhaps it is possible to ignore the new encoding conversions when we know we're doing CSV operations or the string was already encoded. Thank you.

Comment 4 John Dennis 2017-09-26 13:59:12 UTC

I'd be happy to take a look Julie but I'll need to see the patched version of the code to see what is going on. If there is a Gerrit review open with the patches I can look at that or if this is just in your private repo than maybe you can send me the patched file either in an email in a pastebin (you could also attach it to the bugzilla but given it's a temporary way to show me what you've done we don't really need it to persist for historical reasons).

Comment 8 John Dennis 2017-10-23 19:26:53 UTC

*** Bug 1477836 has been marked as a duplicate of this bug. ***

Comment 11 Julie Pichon 2018-01-22 10:31:33 UTC

It looks like another component will need a patch to fully fix this. What we currently have addresses the issue directly in python-cliff / python-openstackclient, so that running ad-hoc commands on the CLI works well. However there appears to be a second issue related to processing the unicode output when the command gets run as part of puppet apply e.g. when calling 'openstack undercloud upgrade' in instack-undercloud. I opened https://bugs.launchpad.net/tripleo/+bug/1744075 to track it upstream for now.

Comment 12 Julie Pichon 2018-02-21 12:23:15 UTC

I cloned this bug against instack-undercloud (bug 1547477), so that we can ship the python-cliff part of the fix and have experts look at the remaining issue.

Comment 13 Julie Pichon 2018-05-11 14:41:02 UTC

See comment 2 for "How to test" instructions.

Comment 22 Alex McLeod 2018-09-03 08:02:13 UTC

Hi there,

If this bug requires doc text for errata release, please set the 'Doc Type' and provide draft text according to the template in the 'Doc Text' field.

The documentation team will review, edit, and approve the text.

If this bug does not require doc text, please set the 'requires_doc_text' flag to -.

Thanks,
Alex

Comment 28 errata-xmlrpc 2018-09-17 16:59:16 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2671