Bug 1547477

The current status is that we fixed python-cliff so that csv listing from the CLI work correctly with unicode:

$ openstack project create tést
$ openstack project list --format csv

(Bug 1437402, which this is cloned from.)

However, during the undercloud upgrade when puppet-keystone runs that same command during the process (cf. bug description), it fails on the previous ascii error anyway. I tried to dig into puppet-keystone / puppet-apply / os-refresh-config to figure out what might be causing this but couldn't find my way around.

This is reproducible on current upstream master (opened https://bugs.launchpad.net/tripleo/+bug/1744075 ).

$ openstack undercloud upgrade
[...]
2018-02-21 11:50:25,957 INFO: Notice: /Stage[main]/Keystone::Deps/Anchor[keystone::service::begin]: Triggered 'refresh' from 2 events
2018-02-21 11:50:26,210 INFO: Notice: /Stage[main]/Apache::Service/Service[httpd]/ensure: ensure changed 'stopped' to 'running'
2018-02-21 11:53:35,744 INFO: Error: Failed to apply catalog: Execution of '/bin/openstack project list --quiet --format csv --long' returned 1: 'ascii' codec can't encode character u'\xe9' in position 1: ordinal not in range(128)
2018-02-21 11:53:35,745 INFO: "ID","Name","Domain ID","Description","Enabled"
2018-02-21 11:53:35,745 INFO: "06ef4f7ca53446aa93ba9f5e5d4644bf","admin","default","admin tenant",True (tried 39, for a total of 170 seconds)
[...]
2018-02-21 11:53:42,226 INFO: + echo 'puppet apply exited with exit code 1'
2018-02-21 11:53:42,226 INFO: puppet apply exited with exit code 1
2018-02-21 11:53:42,226 INFO: + '[' 1 '!=' 2 -a 1 '!=' 0 ']'
2018-02-21 11:53:42,226 INFO: + exit 1
2018-02-21 11:53:42,226 INFO: [2018-02-21 11:53:42,226] (os-refresh-config) [ERROR] during configure phase. [Command '['dib-run-parts', '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit status 1]

$  /bin/openstack project list --quiet --format csv --long
"ID","Name","Domain ID","Description","Enabled"
"06ef4f7ca53446aa93ba9f5e5d4644bf","admin","default","admin tenant",True
"3babf9e38f02454c8cbdfa12bbf12674","tést","default","",True
"e0f8054de3b44abd894ea35acd75b4ed","service","default","Tenant for the openstack services",True

Julie, one possibility for the problem may be due to the fact the output of the command is being piped as opposed to being sent to a terminal. I did another fix a while back to handle the terminal vs. non-terminal I/O destination situation. I'd have to dig up the commit, I think it was for cliff.

Another possibility, I think this might be more likely is the environment has the locale set incorrectly, it's probably defaulting to the C locale with ASCII encoding. The openstack cli via cliff interrogates the locale to determine the encoding to use, if it's set wrong you'll get these problems. It seems to be we've already seen this issue in conjunction with Puppet because of the way it spawns it's environment. Sorry but I can recall where this was first reported.

One way you can check the local Python is seeing when it runs is this:

import locale
print locale.getdefaultlocale()

On my system it prints:
('en_US', 'UTF-8')

Which means I'm using english as my language with a USA dialect and the encoding is UTF-8.

Usually this comes from the LANG environment variable, on my system it is:
LANG=en_US.UTF-8

Hopefully it's obvious how Python parsed this. If the LANG environment variable is not set Python goes through a variety of heuristics to determine the locale information as I recall (my memory is a bit fuzzy on the actual details of how Python determines the locale information. But the bottom line is knowing what Python thinks the locale is in the given environment because everything else is driven off that.

Thank you for your comment John. I think this is the case indeed, roughly updating the command to show that output:

$ openstack project list
('en_US', 'UTF-8')
[...]

but during the undercloud upgrade, after a noticeable pause:
2018-02-21 17:54:39,919 INFO: Error: Failed to apply catalog: Execution of '/bin/openstack project list --quiet --format csv --long' returned 1: 'ascii' codec can't encode character u'\xe9' in position 1: ordinal not in range(128)
2018-02-21 17:54:39,919 INFO: (None, None)

I was trying to figure out where along the process the locale is changing, but I'm not familiar enough with dib, puppet and the rest and didn't get a chance to follow the full trail.

I ran the failing command in the container on the overcloud controller by using "docker run -it --network host new_keystone_image bash", sourcing overcloudrc.v3 and then running "/usr/bin/openstack project list --quiet --format csv --long" and it works.  I've noticed there's an image using the <none> tag that I've deleted on 2 of the controllers and I'm retrying a deployment.  We do see the new container being used in the failures list though ... I doubt it will change something but we never know.

I'm not sure here if the fact that overcloud_images.yaml is not using the hotfix dockage image could be a part of the problem?  That wouldn't explain why in the failure list we see it pulling the hotfix-utf8 docker image file but the error is the same as if it was using the previous docker image file... which I suspect is what is happening at this time.

*** This bug has been marked as a duplicate of bug 1635249 ***

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days