Bug 1468174

Summary: Ensure that the undercloud-passwords.conf file is present before doing an undercloud upgrade
Product: Red Hat OpenStack Reporter: Chaitanya Shastri <cshastri>
Component: instack-undercloudAssignee: James Slagle <jslagle>
Status: CLOSED ERRATA QA Contact: Gurenko Alex <agurenko>
Severity: medium Docs Contact:
Priority: medium    
Version: 10.0 (Newton)CC: aschultz, bnemec, dbecker, mandreou, mburns, morazi, ohochman, rhel-osp-director-maint, tvignaud
Target Milestone: gaKeywords: Triaged
Target Release: 12.0 (Pike)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: instack-undercloud-7.2.1-0.20170821194210.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-12-13 21:39:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Chaitanya Shastri 2017-07-06 07:37:15 UTC
Description of problem:
One should not be able to run 'openstack undercloud upgrade' or 'openstack undercloud install' if undercloud-passwords.conf file is missing / renamed from its default location which is '/home/stack' on the undercloud node.

If there are permission issues with undercloud-passwords.conf file, a permission denied error is thrown which is fine, but no validation is made if its missing:

~~~
[stack@instack ~]$ ls -l
total 44
-rw-r--r--. 1 stack stack 13695 Jul  6 06:56 instackenv.json
-rw-------. 1 stack stack   479 Jul  6 09:34 stackrc
drwxrwxr-x. 2 stack stack  4096 Jul  6 09:29 test
-rw-r--r--. 1 stack stack 10673 Jul  6 06:56 undercloud.conf
-r--------. 1 root  root   1650 Jul  6 09:29 undercloud-passwords.conf

[stack@instack ~]$ openstack undercloud upgrade
Logging to /home/stack/.instack/install-undercloud.log
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1207, in install
    _load_config()
  File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 477, in _load_config
    CONF(conf_params)
  File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2255, in __call__
    self._namespace._files_permission_denied)
oslo_config.cfg.ConfigFilesPermissionDeniedError: Failed to open some config files: /home/stack/undercloud-passwords.conf
Command 'instack-install-undercloud' returned non-zero exit status 1
~~~

If the undercloud-passwwords.conf file is missing, a new password file is generated which changes the values for all the services and the undercloud upgrade fails eventually and leaves undercloud in a broken state:

~~~
[stack@instack ~]$ ls -l
total 44
-rw-r--r--. 1 stack stack 13695 Jul  6 06:56 instackenv.json
-rw-------. 1 stack stack   479 Jul  6 09:34 stackrc
drwxrwxr-x. 2 stack stack  4096 Jul  6 09:29 test
-rw-r--r--. 1 stack stack 10673 Jul  6 06:56 undercloud.conf
-r--------. 1 root  root   1650 Jul  6 09:29 undercloud-passwords.conf.root

[stack@instack ~]$ openstack undercloud upgrade
Logging to /home/stack/.instack/install-undercloud.log
Checking for a FQDN hostname...
Static hostname detected as instack.localdomain
Transient hostname detected as instack.localdomain
Generated new password for undercloud_db_password
Generated new password for undercloud_admin_token
Generated new password for undercloud_admin_password
Generated new password for undercloud_glance_password
Generated new password for undercloud_heat_encryption_key
Generated new password for undercloud_heat_password
Generated new password for undercloud_neutron_password
Generated new password for undercloud_nova_password
Generated new password for undercloud_ironic_password
Generated new password for undercloud_aodh_password
Generated new password for undercloud_ceilometer_password
Generated new password for undercloud_ceilometer_metering_secret
Generated new password for undercloud_ceilometer_snmpd_password
Generated new password for undercloud_swift_password
Generated new password for undercloud_mistral_password
Generated new password for undercloud_rabbit_cookie
Generated new password for undercloud_rabbit_password
Generated new password for undercloud_rabbit_username
Generated new password for undercloud_heat_stack_domain_admin_password
Generated new password for undercloud_swift_hash_suffix
Generated new password for undercloud_haproxy_stats_password
Generated new password for undercloud_zaqar_password
Generated new password for undercloud_horizon_secret_key
Running yum update
[...]

2017-07-06 09:48:43 - Notice: /Stage[main]/Heat::Deps/Anchor[heat::service::end]: Triggered 'refresh' from 3 events
2017-07-06 09:48:44 - Notice: Finished catalog run in 440.88 seconds
+ rc=6
+ set -e
+ echo 'puppet apply exited with exit code 6'
puppet apply exited with exit code 6
+ '[' 6 '!=' 2 -a 6 '!=' 0 ']'
+ exit 6
[2017-07-06 09:48:49,480] (os-refresh-config) [ERROR] during configure phase. [Command '['dib-run-parts', '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit status 1]

[2017-07-06 09:48:49,480] (os-refresh-config) [ERROR] Aborting...
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1214, in install
    _run_orc(instack_env)
  File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1003, in _run_orc
    _run_live_command(args, instack_env, 'os-refresh-config')
  File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 519, in _run_live_command
    raise RuntimeError('%s failed. See log for details.' % name)
RuntimeError: os-refresh-config failed. See log for details.
Command 'instack-install-undercloud' returned non-zero exit status 1

[stack@instack ~]$ ls -l
total 48
-rw-r--r--. 1 stack stack 13695 Jul  6 06:56 instackenv.json
-rw-------. 1 stack stack   479 Jul  6 09:34 stackrc
drwxrwxr-x. 2 stack stack  4096 Jul  6 09:29 test
-rw-r--r--. 1 stack stack 10673 Jul  6 06:56 undercloud.conf
-r--------. 1 root  root   1650 Jul  6 09:29 undercloud-passwords.conf.root
~~~

The consequesces of it are that we will not be able to perform minor update and can lead to some other issues which I am unaware of at the moment.

Version-Release number of selected component (if applicable):
RHOSP 10

How reproducible:
Always

Steps to Reproduce:
1.Install undercloud 
2.Install overcloud nodes
3.Move / rename the undercloud-passwords.conf file to something / somewhere else.
4.Run 'openstack undercloud upgrade' command.

Actual results:
'openstack undercloud upgrade' or 'openstack undercloud install' will continue to run even if undercloud-passwords.conf file is renamed / missing.

Expected results:
One should not be able to run 'openstack undercloud upgrade' or 'openstack undercloud install' if undercloud-passwords.conf file is missing / renamed.

Additional info:
Proative solution to this issue is to take undercloud backup before performing undercloud upgrade.

Comment 1 Ben Nemec 2017-07-13 18:57:10 UTC
Upstream review has merged.

Comment 8 Gurenko Alex 2017-12-12 16:13:08 UTC
Verified on build 2017-12-06.2. Getting undercloud update failed with following validation as expected:

instack_undercloud.validator.FailedValidation: The /home/stack/undercloud-passwords.conf file is missing.  This will cause all service passwords to change and break the existing undercloud.
Command 'instack-upgrade-undercloud' returned non-zero exit status 1

Comment 11 errata-xmlrpc 2017-12-13 21:39:17 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:3462