Description of problem: Keystone doesn't like UTF-8 characters Description What problem/issue/behavior are you having trouble with? What do you expect to see? Once again, we are having trouble when integrating Keystone with our LDAP, which uses non-ASCII characters in multiple fields. In this case, the error is triggered when the Keystone client output is not a terminal: Apr 11 10:33:32 localhost os-collect-config: #033[1;31mError: Could not prefetch keystone_user provider 'openstack': Execution of '/usr/bin/openstack user list --quiet --format csv --long' returned 1: ERROR: openstack 'ascii' codec can't encode character u'\xd1' in position 24: ordinal not in range(128) This can be reproduced by hand by running the following command from a terminal: [root@prod-epg-ostkc-00 ~]# openstack --os-token hey_dont_mind_me --os-url http://10.0.0.0:3 5357/v2.0/ user list --quiet --format csv --long 2>&1 | grep . ERROR: openstack 'ascii' codec can't encode character u'\xd1' in position 24: ordinal not in range(128) Removing the "pipe" however allows the command to run just fine. I have been able to workaround this issue by patching the following file: --- /usr/lib/python2.7/site-packages/cliff/formatters/commaseparated.py.old 2016-04-11 17:01:24.160958338 +0200 +++ /usr/lib/python2.7/site-packages/cliff/formatters/commaseparated.py 2016-04-11 17:01:31.964040445 +0200 @@ -3,6 +3,8 @@ import os import sys +reload(sys) +sys.setdefaultencoding('UTF8') from .base import ListFormatter Where are you experiencing the behavior? What environment? In our production cluster, which fails deployment using Red Hat Director because of the Keystone-LDAP integration and UTF-8 characters. When does the behavior occur? Frequently? Repeatedly? At certain times? Always. What information can you provide around timeframes and urgency? Critical issue. Version-Release number of selected component (if applicable):
This looks like it should be kept on the Cliff project, and not Keystone. Even if we have a work around for Keystone CLI (which can be replaced with python-openstackclient) it will still manifest itself on the rest of the Openstack CLIs such as Nova etc. Why do you think this is a Keystone problem an not cliff?
I can probably explain why removing the pipe allows the command to run without error. Python examines the IO used for writing. If the IO is a terminal it tries to respect the locale settings which (may) include the encoding. Most people set their locale encoding to UTF-8. Thus when Python writes to a terminal and that terminal declares it's encoding is UTF-8 will will use the utf-8 codec. If this isn't the case Python will default to using the ascii codec. By redirecting into a pipe rather than writing to the terminal Python will use the ascii codec. I suspect the problem is in the cliff module and it's not properly handling encoding/decoding of text streams. This is a very common problem.
Hi, could we update this BZ please.
Try setting the environment variable PYTHONIOENCODING to utf-8, e.g. export PYTHONIOENCODING='utf-8' As explained in comment #3 Python2 respects the locale when stdout is a TTY but not when stdout is some other file. I believe the above env variable will cause sys.stdout.encoding to be set to it's value, otherwise it will be None and use the default encoding of ASCII.
Just to clarify, to the best of my knowledge this has nothing to do with LDAP, the issue is how Unicode objects in Python2 are encoded in I/O streams. The problem you are encountering is that when Unicode is written to stdout it is not encoded to the user's preferred encoding (typically UTF-8). Python2 tries to perform a half-hearted workaround and will set the encoding property of stdout and stderr based on the locale information read from the environment. But it only does this when stdout, stderr are attached to a terminal (TTY). This is why you get the expected result in a terminal but not when the output is piped (because it's no longer attached to a TTY). The workaround in comment #5 should help you. The proper fix is to modify the "shell" program /usr/bin/openstack (and for legacy /usr/bin/keystone) so that when it starts up the first thing it does is wrap stdout and stderr with text I/O wrappers bound to the user's preferred encoding. We will try to get this upstream. In essence the main function in shell.py need to do this: import locale import six if six.PY2: encoding = locale.getpreferredencoding() if encoding: sys.stdout = codecs.getwriter(encoding)(sys.stdout) sys.stderr = codecs.getwriter(encoding)(sys.stderr)
Just to add my 2 cents, I can confirm that very similar thing happens with openstack client, too. Here's an example: [root@cont00 ~(keystone_admin)]# openstack project list +----------------------------------+----------+ | ID | Name | +----------------------------------+----------+ | 5217f2a438194cbb9e79765a32e3502f | admin | | 65d921340fb94cb3bf6ae9c5632f4649 | Tenant 1 | | 8dc81b5daea44f549e25fc2d3fb929fb | テスト | | 97bbbc94ac064364b9bc932ec44fc6b1 | Test | | ff81c58d516c4433b53ed4177acdfc0c | services | +----------------------------------+----------+ [root@cont00 ~(keystone_admin)]# openstack project show Test +-------------+----------------------------------+ | Field | Value | +-------------+----------------------------------+ | description | テスト専用 | | enabled | False | | id | 97bbbc94ac064364b9bc932ec44fc6b1 | | name | Test | | properties | | +-------------+----------------------------------+ [root@cont00 ~(keystone_admin)]# openstack project show Test >/dev/null 'ascii' codec can't encode characters in position 169-173: ordinal not in range(128) [root@cont00 ~(keystone_admin)]# openstack project show Test --format=shell >/dev/null 'ascii' codec can't encode characters in position 13-17: ordinal not in range(128) [root@cont00 ~(keystone_admin)]# openstack project show Test --format=table >/dev/null 'ascii' codec can't encode characters in position 169-173: ordinal not in range(128) [root@cont00 ~(keystone_admin)]# openstack project show Test --format=yaml - {Field: description, Value: "\u30C6\u30B9\u30C8\u5C02\u7528"} - {Field: enabled, Value: false} - {Field: id, Value: 97bbbc94ac064364b9bc932ec44fc6b1} - {Field: name, Value: Test} - {Field: properties, Value: ''} [root@cont00 ~(keystone_admin)]# openstack project show Test --format=yaml >/dev/null [root@cont00 ~(keystone_admin)]# [root@cont00 ~(keystone_admin)]# openstack project show 8dc81b5daea44f549e25fc2d3fb929fb --format=yaml - {Field: description, Value: "\u30C6\u30B9\u30C8\u5C02\u7528\u3002\u30D7\u30ED\u30B8\ \u30A7\u30AF\u30C8\u540D\u306B\u65E5\u672C\u8A9E\u6587\u5B57\u3092\u4F7F\u3046\ \u3068openstack\u30B3\u30DE\u30F3\u30C9\u304C\u30A8\u30E9\u30FC\u3092\u51FA\u3059\ \u3053\u3068\u304C\u3042\u308B\u306E\u3067\u672C\u756A\u30D7\u30ED\u30B8\u30A7\ \u30AF\u30C8\u540D\u306F\u534A\u89D2\u82F1\u6570\u5B57\u306E\u307F\u3068\u3057\ \u3066\u304F\u3060\u3055\u3044\u3002"} - {Field: enabled, Value: false} - {Field: id, Value: 8dc81b5daea44f549e25fc2d3fb929fb} - {Field: name, Value: "\u30C6\u30B9\u30C8"} - {Field: properties, Value: ''} [root@cont00 ~(keystone_admin)]# openstack project show 8dc81b5daea44f549e25fc2d3fb929fb +-------------+-----------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-------------+-----------------------------------------------------------------------------------------------------------------------------------------------+ | description | テスト専用。プロジェクト名に日本語文字を使うとopenstackコマンドがエラーを出すことがあるので本番プロジェクト名は半角英数字のみとしてください。 | | enabled | False | | id | 8dc81b5daea44f549e25fc2d3fb929fb | | name | テスト | | properties | | +-------------+-----------------------------------------------------------------------------------------------------------------------------------------------+ [root@cont00 ~(keystone_admin)]# openstack project show 8dc81b5daea44f549e25fc2d3fb929fb >/dev/null 'ascii' codec can't encode characters in position 496-518: ordinal not in range(128)
We have pushed fixes upstream for this issue, including the stable branches. However I'm not sure they've been applied yet.
see: https://bugs.launchpad.net/python-cliff/+bug/1603210
I think bug 1437402 is the same bug, from what I can tell openstack project list --format csv is still broken upstream in Queens: $ openstack project create tést +-------------+----------------------------------+ | Field | Value | +-------------+----------------------------------+ | description | | | domain_id | default | | enabled | True | | id | 91c375d9893940d1be8751163f328e75 | | is_domain | False | | name | tést | | parent_id | default | +-------------+----------------------------------+ $ /usr/bin/openstack project list --quiet --format csv --long "ID","Name","Domain ID","Description","Enabled" 'ascii' codec can't decode byte 0xc3 in position 37: ordinal not in range(128) Note that the yaml and json formatter appear to be working fine. Using the workaround in comment 5 didn't seem to help. Reproduced on OSP10 with python-openstackclient-3.2.1-2.el7ost.noarch / python-cliff-2.2.0-1.el7ost.noarch.
Julie, csv output has it's own special fix besides the other utf-8 fixes for cliff. Apparently the Python 2.x version of csv is not capable of handling unicode. The fix is to use a different csv Python module called "unicodecsv". I believe the switch to unicodecsv was made in cliff 1.16.0. Anyway, if you search for Openstack bug fixes involving unicodecsv you should find it. Cliff was introduced into more modern versions of Openstack to unify command line support across projects using the openstack command line tool. I believe Kilo predates cliff but my memory is a bit fuzzy. If you have an Openstack release without cliff and osc you'll have to find and apply the fixes in every place in every client.
John, thank you for the useful pointers! Although unicodecsv is present on OSP10 the failure occurs - however it doesn't on an OSP9 environment so it looks like there's been a regression somewhere. I'll continue investigating on bug 1437402. Thanks again.