Bug 1477836

Summary: No Compatibility for Unicode Characters in RHOSP 10
Product: Red Hat OpenStack Reporter: Ganesh Kadam <gkadam>
Component: python-openstackclientAssignee: Julie Pichon <jpichon>
Status: CLOSED DUPLICATE QA Contact: Shai Revivo <srevivo>
Severity: high Docs Contact:
Priority: high    
Version: 10.0 (Newton)CC: apevec, gkadam, jdennis, justinas.balciunas, lhh, nkinder, panbalag, smykhail, srevivo
Target Milestone: asyncKeywords: Triaged, ZStream
Target Release: 10.0 (Newton)   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-10-23 19:26:53 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Ganesh Kadam 2017-08-03 05:09:54 UTC
Description of problem:

IHAC, working on RHOSP 10 deployment. They have raised an issue with TripleO Heat Deployment since it fails in phase 3 when a project has umlauts in name or description.

Per Cu. as the overcloud CLI and API is not available after that failure, this results in a denial of service [DoS] if undetected in advance

Here is the detailed error log:
~~~
http://pastebin.test.redhat.com/505661
~~~


Version-Release number of selected component (if applicable):
RHOSP 10, openstack-keystone-10.0.0-4.el7ost.noarch

How reproducible:
Always.


How to reproduce?

Create a project with an umlaut in it, e.g.:

[stack@ospd ~]$ openstack project list | grep nord
| 2b2296ed7ec04673b3ac3a5507623301 | bq-nordström     

[stack@ospd ~]$ openstack project show 2b2296ed7ec04673b3ac3a5507623301
+-------------+--------------------------------------------+
| Field       | Value                                      |
+-------------+--------------------------------------------+
| description | Project for Karl Nordström - Uni Saarland |
| domain_id   | 61f897cb055f412e81c82b42de718728           |
| enabled     | True                                       |
| id          | 2b2296ed7ec04673b3ac3a5507623301           |
| is_domain   | False                                      |
| name        | bq-nordström                              |
| parent_id   | 61f897cb055f412e81c82b42de718728           |
+-------------+--------------------------------------------+

and redeploy the cloud. It will fail in step 3 of the overcloud deployment with the error message as mentioned in log.


"/usr/bin/openstack project list --quiet --format csv --long" 
returned 1: 'ascii' codec can't decode byte 0xc3 in position 46: ordinal not in range(128)\n\"ID\",\"Name\",\"Domain ID\",\"Description\",\"Enabled

Throws below error:
~~~
Error: Could not prefetch keystone_tenant provider 'openstack': Execution of '/usr/bin/openstack project list --quiet --format csv --long' returned 1: 'ascii' codec can't decode byte 0xc3 in position 46: ordinal not in range(128)\n\"ID\",\"Name\",\"Domain ID\",\"Description\",\"Enabled\"\n\"01331d19d8954450b9e02a9b8871125e\",\"service\",\"default\",\"Tenant for the openstack services\",True\n\"287dae5c8f4043dd96dfbeda8a717a02\",\"B080\",\"default\",\"\",True (tried 40, for a total of 170 seconds)\u001b[0m\n\u001b[1;31mError: Not managing Keystone_tenant[service] due to earlier Keystone API failures.\u001b[0m\n\u001b[1;31mError: /Stage[main]/Keystone::Roles::Admin/Keystone_tenant[service]/ensure: change from absent to present failed: Not managing Keystone_tenant[service] due to earlier Keystone API failures.\u001b[0m\n\u001b[1;31mError: Not managing Keystone_tenant[admin] due to earlier Keystone API failures.\u001b[0m\n\u001b[1;31mError: /Stage[main]/Keystone::Roles::Admin/Keystone_tenant[admin]/ensure: change from absent to present failed: Not managing Keystone_tenant[admin] due to earlier Keystone API failures.
~~~



Actual results:
No support/Compatibility for Unicode characters 


Expected results:
OpenStack Compatibility for Unicode characters 


Additional info:

I have also found the similar BZ for RHOSP 7:
~~~
https://bugzilla.redhat.com/show_bug.cgi?id=1327893
~~~

Not sure if the fix proposed upstream was merged, hence opening the new case for latest RHOSP version.

Comment 2 John Dennis 2017-08-08 13:09:23 UTC
The initial bug description references bug#1327893 which I believe is the likely cause. Although I have not had a chance to verify how puppet is invoking the openstack client I believe it is very likely doing so with it's output redirected which is the topic of bug#1327893. That bug report has  comment #13 which lists 2 upstream patches that need to be applied for UTF-8 to work correctly with openstack client. I am the author of both those patches. The first patch (https://review.openstack.org/#/c/342914/5) is the one which should solve the issue reported here. It should have been included in the python-cliff RPM. Upstream cliff has included that patch in the following tags:

2.2.0
2.3.0
2.4.0
2.5.0
2.6.0
2.7.0
2.8.0

As far as I can tell OSP-10 should have shipped with python-cliff-2.2.0 which should have included that fix.

We need to know the version of python-cliff on the node showing the failure.

We also need to know the value of the locale in the environment in which the script executes.

The patch will have modified /usr/lib/python2.7/site-packages/cliff/app.py to add code to the _set_streams() method

Part of that patch includes these lines:

        if six.PY2:                                                             
            encoding = locale.getpreferredencoding()                            
            if encoding:

If that patch is missing or if locale.getpreferredencoding() does not return
an encoding the if block won't be entered which sets the output encoder
for redirected streams. An incorrect encoding (e.g. ascii) will also cause the failure.

Comment 5 John Dennis 2017-08-09 17:07:12 UTC
The python-cliff-2.2.0-1.el7ost.noarch package should have the fix in it so my guess is the fix is not executing because of the environment.

The locale is normally defined by the LANG environment variable.

Probably the easiest way to get the information we need is by temporarily adding a print statement to the Python code.

in /usr/lib/python2.7/site-packages/cliff/app.py edit the code to add the print statement so it looks like this.

        if six.PY2:
            encoding = locale.getpreferredencoding()
            print "encoding=%s LANG=%s" % (encoding, os.environ['LANG'])
            if encoding:
 
You then need to rerun things *exactly* as when the error occurs. That's because many of the configuration tools (e.g. Puppet) modify the environment. As a matter of fact I seem to recall one or more of the tools sets the locale to "C" without any encoding specification which would certainly cause problems with internationalized strings.

Comment 8 John Dennis 2017-10-23 19:26:53 UTC
A fix for this issue was committed upstream:
https://review.openstack.org/#/c/508760/

The upstream bug is:
https://bugs.launchpad.net/python-cliff/+bug/1720115

The Red Hat bugzilla where this work is being tracked is:
https://bugzilla.redhat.com/show_bug.cgi?id=1437402

This bug should be closed as a duplicate of the above.

*** This bug has been marked as a duplicate of bug 1437402 ***