Bug 1388930 - rhel-osp-director: After minor update, the admin password in overcloudrc doesn't work.
Summary: rhel-osp-director: After minor update, the admin password in overcloudrc do...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-common
Version: 10.0 (Newton)
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: rc
: 10.0 (Newton)
Assignee: Dougal Matthews
QA Contact: Alexander Chuzhoy
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-10-26 13:38 UTC by Alexander Chuzhoy
Modified: 2016-12-29 16:55 UTC (History)
13 users (show)

Fixed In Version: openstack-tripleo-common-5.3.0-4.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-12-14 16:25:40 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2016:2948 normal SHIPPED_LIVE Red Hat OpenStack Platform 10 enhancement update 2016-12-14 19:55:27 UTC
OpenStack gerrit 394195 None None None 2016-11-07 06:40:59 UTC
Launchpad 1638003 None None None 2016-10-31 14:17:43 UTC

Description Alexander Chuzhoy 2016-10-26 13:38:43 UTC
rhel-osp-director:   After minor update, the admin password in overcloudrc doesn't work.

Environment:
openstack-puppet-modules-9.3.0-0.20161003154825.8c758d6.el7ost.noarch
openstack-tripleo-heat-templates-5.0.0-0.20161008015357.0d3e3e3.1.el7ost.noarch
instack-undercloud-5.0.0-2.el7ost.noarch
python-keystonemiddleware-4.9.0-1.el7ost.noarch
python-keystone-10.0.0-2.el7ost.noarch
openstack-keystone-10.0.0-2.el7ost.noarch
puppet-keystone-9.4.0-1.el7ost.noarch
python-keystoneclient-3.5.0-1.el7ost.noarch
python-keystoneauth1-2.12.2-1.el7ost.noarch



Steps to reproduce:
1. Deploy overcloud with:
openstack overcloud deploy --debug --templates --libvirt-type kvm --ntp-server clock.redhat.com --neutron-network-type vxlan --neutron-tunnel-types vxlan --control-scale 3 --control-flavor controller-d75f3dec-c770-5f88-9d4c-3fea1bf9c484 --compute-scale 1 --compute-flavor compute-b634c10a-570f-59ba-bdbf-0c313d745a10 --ceph-storage-scale 2 --ceph-storage-flavor ceph-cf1f074b-dadb-5eb8-9eb0-55828273fab7 -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e virt/ceph.yaml -e virt/hostnames.yml -e virt/network/network-environment.yaml --log-file overcloud_deployment_48.log


2. Populate the overcloud (create another tenant, admin user too).

3. Minor update the setup.

4. Try to work with overcloud using both: the user in overcloudrc and the user/tenant you created in step #2 above.

Result:
[stack@undercloud-0 ~]$ . overcloudrc     
[stack@undercloud-0 ~]$ openstack user list                                     
The request you have made requires authentication. (HTTP 401) (Request-ID: req-8c9b32af-ed8e-4c1b-b8a0-1d5563fe9297)

#Note: keystonerc_master holds the credentials for another user/tenant.
[stack@undercloud-0 ~]$ . keystonerc_master              
[stack@undercloud-0 ~]$ openstack user list                                
+----------------------------------+------------+                                                                  
| ID                               | Name       |                                                                  
+----------------------------------+------------+                                                                  
| a6e7fb07516f45e1a08678b3ff5e4bf4 | admin      |                                                                  
| 8399dd3cfb9b48ea9837f598ed37b8b8 | neutron    |                                                                  
| 8e560be30796430b8edce47ebbfa2543 | heat       |                                                                  
| 76657d15d4aa46809f5cbb85401de457 | gnocchi    |                                                                  
| 5faf0763d5bd4df8906532db14fb5b3d | aodh       |                                                                  
| 3707bda4c3ef465a9965ee4fa76b1350 | nova       |                                                                  
| 98c927f36e074ef18eb190035640475f | glance     |                                                                  
| cce2d09709844625bace7af6272acbf0 | ceilometer |                                                                  
| fa60b6f38b754b1485f5e7e0d2ea9d3b | cinder     |                                                                  
| abc2ca2c647e425e9c84a9efe9e02898 | heat-cfn   |                                                                  
| 9f45ea3a803c42dc944abd59f9793edf | swift      |                                                                  
| f87e02bcee744d74bb7a7912aad532ba | master     |                                                                  
+----------------------------------+------------+      



[stack@undercloud-0 ~]$ cat keystonerc_master
export OS_NO_CACHE=True
export OS_CLOUDNAME=overcloud
export OS_AUTH_URL=http://10.0.0.104:5000/v2.0
export NOVA_VERSION=1.1
export COMPUTE_API_VERSION=1.1
export OS_USERNAME=master
export no_proxy=,10.0.0.104,192.0.2.6
export OS_PASSWORD=master
export PYTHONWARNINGS="ignore:Certificate has no, ignore:A true SSLContext object is not available"
export OS_TENANT_NAME=master



[stack@undercloud-0 ~]$ cat overcloudrc
export OS_NO_CACHE=True
export OS_CLOUDNAME=overcloud
export OS_AUTH_URL=http://10.0.0.104:5000/v2.0
export NOVA_VERSION=1.1
export COMPUTE_API_VERSION=1.1
export OS_USERNAME=admin
export no_proxy=,10.0.0.104,192.0.2.6
export OS_PASSWORD=KC7A4j2nZbG9kM27fqbzvz7aY
export PYTHONWARNINGS="ignore:Certificate has no, ignore:A true SSLContext object is not available"
export OS_TENANT_NAME=admin



Expected result:
The authentication should work with the credentials in overcloudrc.


Workaround:
source keystonerc_master
openstack user set --name admin --project admin --password admin admin
sed -i 's/OS_PASSWORD=.*/OS_PASSWORD=admin/' overcloudrc
source overcloudrc

Comment 2 Marios Andreou 2016-10-27 14:44:54 UTC
https://review.openstack.org/#/c/391201/1 patch from rbrady to fix this

Comment 3 Marios Andreou 2016-10-31 11:40:15 UTC
This is also affecting upgrades now... at least I hit it today after the ceilometer migration:

        openstack overcloud deploy --templates /usr/share/openstack-tripleo-heat-templates -e /usr/share/openstack-tripleo-heat-templates/overcloud-resource-registry-puppet.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml  --control-scale 3 --compute-scale 1 --libvirt-type qemu -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml -e network_env.yaml --ntp-server '0.fedora.pool.ntp.org'  -e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-ceilometer-wsgi-mitaka-newton.yaml 
        2016-10-31 11:01:53Z [34]: CREATE_IN_PROGRESS  state changed
        ... 
        2016-10-31 11:14:39Z [overcloud]: UPDATE_COMPLETE  Stack UPDATE completed successfully
         Stack overcloud UPDATE_COMPLETE 
        pingtest: 
        [stack@instack ~]$ pingtest                                                                               
        #################                                                                                         
        tripleo.sh -- Overcloud pingtest
        #################
        You must source a overcloudrc file for the Overcloud.
        Attempting to source /home/stack/overcloudrc
        Done
        #################
        tripleo.sh -- Overcloud pingtest; cleaning environment
        #################
        You must source a overcloudrc file for the Overcloud.
        Attempting to source /home/stack/overcloudrc
        Done
        The request you have made requires authentication. (HTTP 401) (Request-ID: req-6bb876fb-a133-439e-87ce-862
        ea0fda57b)
        ^CTraceback (most recent call last):
          File "/bin/tripleo", line 12, in <module>
            sys.exit(subprocess.call(command))
          File "/usr/lib64/python2.7/subprocess.py", line 524, in call
            return Popen(*popenargs, **kwargs).wait()
          File "/usr/lib64/python2.7/subprocess.py", line 1376, in wait
            pid, sts = _eintr_retry_call(os.waitpid, self.pid, 0)
          File "/usr/lib64/python2.7/subprocess.py", line 478, in _eintr_retry_call
            return func(*args)
        KeyboardInterrupt
        #################

Package versions for reference 

        [stack@instack ~]$ rpm -qa | grep "python-tripleoclient\|tripleo-common"
        openstack-tripleo-common-5.3.0-3.el7ost.noarch
        python-tripleoclient-5.3.0-3.el7ost.noarch

An immediate workaround is to use the OVERCLOUD_ADMIN_PASSWORD from the tripleo-overcloud-passwords file which still contains the original value. You need to use that value as the
"OS_PASSWORD=" in the overcloudrc (the value of the overcloudrc is overwritten which is what this bug is about).

Comment 4 Alexander Chuzhoy 2016-10-31 22:34:15 UTC
Reproduced the issue despite using the patch: https://review.openstack.org/#/c/391201

Comment 5 Marios Andreou 2016-11-01 13:02:10 UTC
(In reply to Alexander Chuzhoy from comment #4)
> Reproduced the issue despite using the patch:
> https://review.openstack.org/#/c/391201

+1 me too Sasha I added a comment on the review, repeating and adding needinfo rbrady please check this out when you get a chance:

this fix doesn't seem to work here. I applied before running the ceilometer migration:

        [stack@instack ~]$ curl https://review.openstack.org/changes/391201/revisions/current/patch?download | \
        >      base64 -d | \
        >      sudo patch  -d /usr/lib/python2.7/site-packages/ -p1
          % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                         Dload  Upload   Total   Spent    Left  Speed
        100  2108    0  2108    0     0   2280      0 --:--:-- --:--:-- --:--:--  2278
        patching file tripleoclient/utils.py


But after that was UPDATE_COMPLETE the overcloudrc is still changed:


        [stack@instack ~]$ diff overcloudrc overcloudrcBACKUP 
        8c8
        < export OS_PASSWORD=2480e4fcb377e8f7fe9c544b9ccf30a4cd4e98a1
        ---
        > export OS_PASSWORD=M4bkCjyGa6mpKKNTv9evKMaE3

Am I missing another review or something else we need here?

Comment 6 Ronnie Rasouli 2016-11-02 08:52:32 UTC
I hit similar issue, I can't login to horizon using the admin password. 
Prior that I performed scale out for compute node, ever since then the overcloud admin password is invalid

Comment 7 Marios Andreou 2016-11-02 11:01:51 UTC
adding needinfo @dougal o/ hey not sure if rbrady is out but is this something you can checkout when you have a chance please? The fix in review doesn't seem to do what we need.

Comment 8 Dougal Matthews 2016-11-02 12:16:19 UTC
The problem here is that when we moved the password generation to Mistral we stopped checking for passwords in the tripleo-overcloud-passwords file and users upgrading from Mitaka have that file as the "source of truth" for passwords. We need to respect that and favour those passwords over the passwords stored in the Mistral environment.

I have started a patch to do this: https://review.openstack.org/#/c/392593

It is still WIP as I am in the process of setting up an env to test it now.

Comment 9 Dougal Matthews 2016-11-02 17:04:01 UTC
I have updated the patch again now after lbezdick was able to test it for me. https://review.openstack.org/#/c/392593/

There is one case that I am unsure how to handle. In Mitaka, when a user deploys the password file is created, then when they re-deployed it would be used again. If the user changes directory and the password file can't be found then there would be an error "The password file could not be found!". However, now, if the file can't be found it looks like a Newton deploy (because the file isn't ever created in Newton).

So, I can see this possible scenario that would cause a problem:

1. Install Mitaka
2. Change directory
3. Update packages to Newton
4. Run upgrade deploy

The code will not find the passwords file because of the directory change, so it wont know that it is an upgrade and will generate new passwords.

Any ideas?

Comment 10 Michele Baldessari 2016-11-02 17:50:23 UTC
Thanks for your quick help, Dougal.

I can only think of two approaches for the problem at comment 9:
1) We document the need to launch deploy commands from the same directory
2) We somehow detect in tripleoclient that we are upgrading from mitaka and return "Not Found" if we cannot find the file in this case.

I'd prefer option 2) but I cannot think of a quick/easy and non-fragile way to detect that we are upgrading from mitaka. Maybe other folks in this BZ have ideas? Maybe some field in the mistral environment that we can query?

Comment 11 Marios Andreou 2016-11-03 08:19:50 UTC
+1 thanks very much for jumping on this Dougal. WRT the passwords file and changing params, I don't think that needs to be anything more than a docs note. The scenario you describe in comment #9 applies to everything Mitaka and before:

1. Install Mitaka
2. Change directory
3. Stack update Mitaka (e.g. scale computes, change params/config, minor update)


you get the same issue and we've had bugs filed in past for this - see https://access.redhat.com/solutions/2678131

Comment 12 Dougal Matthews 2016-11-04 16:44:23 UTC
After some issues with the current approach, and hacks uppon hacks to try and resolve it I have gone in a different direction.

This change is much smaller, simpler and should be more robust. https://review.openstack.org/#/c/393831

Comment 13 Marios Andreou 2016-11-07 06:40:59 UTC
This landed into Newton so moving to POST https://review.openstack.org/#/c/394195/ 

Also removing the two abandoned reviews from the linked/external trackers (https://review.openstack.org/#/c/392593/ and https://review.openstack.org/#/c/391201/)

Comment 17 Alexander Chuzhoy 2016-11-16 02:31:30 UTC
Verified:
Environment:
openstack-tripleo-common-5.3.0-6.el7ost.noarch

The reported issue didn't reproduce.

Comment 19 errata-xmlrpc 2016-12-14 16:25:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-2948.html


Note You need to log in before you can comment on or make changes to this bug.