Description of problem: While testing minor upgrade I had an issue with non en_US locale coming in via ssh connection to Director: [stack@undercloud ~]$ openstack undercloud upgrade [...] 2018-01-17 03:58:11,791 INFO: python-neutron.noarch 1:11.0.1-8.el7ost 2018-01-17 03:58:11,791 INFO: python-nova.noarch 1:16.0.2-8.el7ost 2018-01-17 03:58:11,791 INFO: python2-pbr.noarch 0:3.1.1-2.el7ost 2018-01-17 03:58:11,791 INFO: rhos-release.noarch 0:1.2.27-1 2018-01-17 03:58:11,791 INFO: 2018-01-17 03:58:11,792 INFO: Komplett! 2018-01-17 03:58:11,842 INFO: Update completed successfully 2018-01-17 03:58:12,661 INFO: Logging to /home/stack/.instack/install-undercloud.log 2018-01-17 03:58:12,706 INFO: Checking for a FQDN hostname... 2018-01-17 03:58:12,786 INFO: Static hostname detected as undercloud.redhat.local 2018-01-17 03:58:12,812 INFO: Transient hostname detected as undercloud.redhat.local 2018-01-17 03:58:16,229 ERROR: sudo failed: An error has occurred: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nova/cmd/manage.py", line 1797, in main ret = fn(*fn_args, **fn_kwargs) File "/usr/lib/python2.7/site-packages/nova/cmd/manage.py", line 845, in online_data_migrations print(_('Running batches of %i until complete') % max_count) UnicodeEncodeError: 'ascii' codec can't encode character u'\xfc' in position 24: ordinal not in range(128) 2018-01-17 03:58:16,230 DEBUG: An exception occurred Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1764, in install 'online_data_migrations']) File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 576, in _run_command env=env).decode('utf-8') File "/usr/lib64/python2.7/subprocess.py", line 575, in check_output raise CalledProcessError(retcode, cmd, output=output) CalledProcessError: Command '['sudo', '/usr/bin/nova-manage', 'db', 'online_data_migrations']' returned non-zero exit status 1 2018-01-17 03:58:16,234 ERROR: ############################################################################# Undercloud upgrade failed. Reason: Command '['sudo', '/usr/bin/nova-manage', 'db', 'online_data_migrations']' returned non-zero exit status 1 See the previous output for details about what went wrong. The full install log can be found at /home/stack/.instack/install-undercloud.log. ############################################################################# Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1764, in install 'online_data_migrations']) File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 576, in _run_command env=env).decode('utf-8') File "/usr/lib64/python2.7/subprocess.py", line 575, in check_output raise CalledProcessError(retcode, cmd, output=output) subprocess.CalledProcessError: Command '['sudo', '/usr/bin/nova-manage', 'db', 'online_data_migrations']' returned non-zero exit status 1 Command 'instack-upgrade-undercloud' returned non-zero exit status 1 [stack@undercloud ~]$ echo $LAND [stack@undercloud ~]$ echo $LANG de_DE.UTF-8 Version-Release number of selected component (if applicable): How reproducible: always Steps to Reproduce: 1. login to director via ssh as user stack with non-US locale (at least with de_DE.UTF-8 this is happening, but I would assume this will happen with other locale as well) 2. issue command "openstack undercloud upgrade" 3. wait till failure Actual results: command fails with "UnicodeEncodeError: 'ascii' codec can't encode character u'\xfc' in position 24: ordinal not in range(128)" Expected results: command succeeds wth updating the undercloud Additional info: Reason ===== This problem is a UTF-8 problem that should not occure. I comes through /etc/ssh/ssh_config having a desired config of Host * SendEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES Temporary fix ========= Have a local config for ssh without sending the environment: cat ~/.ssh/myconfig Host * <whatever options you want but w/o above SendEnv> ForwardAgent no ControlMaster auto ControlPath binary/%r@%h:%p ControlPersist 60s Then use this to ssh into the director machine: ssh -F ~/.ssh/myconfig <undercloud> (undercloud) [stack@undercloud ~]$ echo $LANG en_US.UTF-8 Reissue the update command and see it succeed.
The error message seems related to nova-manage db in particular, could you provide any Nova logs on the undercloud that contains information related to the error? Thank you. This also looks a bit like https://bugs.launchpad.net/nova/+bug/1653261 , could you perhaps check if you had duplicate aggregates or flavor names? Did any message related to this show when re-running the upgrade with a US locale? /home/stack/.instack/install-undercloud.log may show that information.
Interestingly there is *nothing* in the logs (/var/log/). Problem is, that I can re-run the command sudo /usr/bin/nova-manage db online_data_migrations but it will not hit the same code as the upgrade has already been done. And I don't have a snapshot to revert to and redo. For the mentioned bug: no duplicates of flavors or aggregates. And nothing more than in the bug report in /home/stack/.instack/install-undercloud.log.
Thank you for the reply. This looks a lot more like bug https://bugs.launchpad.net/tripleo/+bug/1722792 actually and the timeline matches better as well, it looks like this should be backported to Pike if possible. Adjusting component accordingly.
Verified on puddle 2018-02-27.3 Set language to same de_DE.UTF-8 and undercloud upgrade completed successfully ############################################################################# Undercloud upgrade complete. The file containing this installation's passwords is at /home/stack/undercloud-passwords.conf. There is also a stackrc file at /home/stack/stackrc. These files are needed to interact with the OpenStack services, and should be secured. ############################################################################# [stack@undercloud-0 ~]$ echo $LANG de_DE.UTF-8
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:0607