Red Hat Bugzilla – Bug 1437402
Outputting UTF8 characters from openstack client when using CSV format fails
Last modified: 2018-11-05 01:46:53 EST
Description of problem: Error: Could not prefetch keystone_tenant provider 'openstack': Execution of '/usr/bin/openstack project list --quiet --format csv --long' returned 1: 'ascii' codec can't decode byte 0xc3 in position 77: ordinal not in range(128) "ID","Name","Domain ID","Description","Enabled" Version-Release number of selected component (if applicable): RHOSP10 How reproducible: In the project list, one of the project had description "sébastien" After changing the description to "sebastien" the project list worked. We have similar bug already reported in Kilo for keystone client: https://bugzilla.redhat.com/show_bug.cgi?id=1327893
This should already be fixed through https://bugs.launchpad.net/python-cliff/+bug/1481014 (Liberty), but although unicodecsv is still in use in the code, it doesn't seem to work as expected anymore. On OSP9: -------- $ openstack project create tést $ openstack project list --format=csv "ID","Name" "577a2f5ff28e45fbbdae0a99923ea284","tést" "5d8df9b090e64b888d2fcf4c859de591","admin" "b6bc76994f94472b86b74d7546eab810","services" "ef843a5065f04ccd8c9a09972b46c251","demo" python-unicodecsv-0.14.1-2.el7ost.noarch python-openstackclient-2.3.1-1.el7ost.noarch python-cliff-2.0.0-1.el7ost.noarch On OSP10: --------- $ openstack project create tést $ openstack project list --format=csv "ID","Name" "7590c86147bc4de996a363ef206a990f","service" 'ascii' codec can't decode byte 0xc3 in position 37: ordinal not in range(128) python-unicodecsv-0.14.1-2.el7ost.noarch python-openstackclient-3.2.1-2.el7ost.noarch python-cliff-2.2.0-1.el7ost.noarch
It looks like one of the fixes for bug 1327893 may have introduced an issue with the unicodecsv package. My tests suggest that commit 5dc9b9a 'Merge "Avoid ASCII encoding errors when output is redirected"' in cliff is when the problem starts. Looking around it seems like unicodecsv already does its own encoding conversion and so shouldn't be used together with the codecs module. To confirm, I removed the line "stdout = codecs.getwriter(encoding)(sys.stdout)" and the --format=csv commands started working again with unicode characters. John, I'm adding you as a needinfo in case you have an idea or pointers, as you are the author of the patch and have been incredibly helpful with unicode issues so far. I'll keep investigating on my side as well, perhaps it is possible to ignore the new encoding conversions when we know we're doing CSV operations or the string was already encoded. Thank you.
I'd be happy to take a look Julie but I'll need to see the patched version of the code to see what is going on. If there is a Gerrit review open with the patches I can look at that or if this is just in your private repo than maybe you can send me the patched file either in an email in a pastebin (you could also attach it to the bugzilla but given it's a temporary way to show me what you've done we don't really need it to persist for historical reasons).
*** Bug 1477836 has been marked as a duplicate of this bug. ***
It looks like another component will need a patch to fully fix this. What we currently have addresses the issue directly in python-cliff / python-openstackclient, so that running ad-hoc commands on the CLI works well. However there appears to be a second issue related to processing the unicode output when the command gets run as part of puppet apply e.g. when calling 'openstack undercloud upgrade' in instack-undercloud. I opened https://bugs.launchpad.net/tripleo/+bug/1744075 to track it upstream for now.
I cloned this bug against instack-undercloud (bug 1547477), so that we can ship the python-cliff part of the fix and have experts look at the remaining issue.
See comment 2 for "How to test" instructions.
Hi there, If this bug requires doc text for errata release, please set the 'Doc Type' and provide draft text according to the template in the 'Doc Text' field. The documentation team will review, edit, and approve the text. If this bug does not require doc text, please set the 'requires_doc_text' flag to -. Thanks, Alex
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:2671