https://review.openstack.org/#/c/391201/1 patch from rbrady to fix this

This is also affecting upgrades now... at least I hit it today after the ceilometer migration:

        openstack overcloud deploy --templates /usr/share/openstack-tripleo-heat-templates -e /usr/share/openstack-tripleo-heat-templates/overcloud-resource-registry-puppet.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml  --control-scale 3 --compute-scale 1 --libvirt-type qemu -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml -e network_env.yaml --ntp-server '0.fedora.pool.ntp.org'  -e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-ceilometer-wsgi-mitaka-newton.yaml 
        2016-10-31 11:01:53Z [34]: CREATE_IN_PROGRESS  state changed
        ... 
        2016-10-31 11:14:39Z [overcloud]: UPDATE_COMPLETE  Stack UPDATE completed successfully
         Stack overcloud UPDATE_COMPLETE 
        pingtest: 
        [stack@instack ~]$ pingtest                                                                               
        #################                                                                                         
        tripleo.sh -- Overcloud pingtest
        #################
        You must source a overcloudrc file for the Overcloud.
        Attempting to source /home/stack/overcloudrc
        Done
        #################
        tripleo.sh -- Overcloud pingtest; cleaning environment
        #################
        You must source a overcloudrc file for the Overcloud.
        Attempting to source /home/stack/overcloudrc
        Done
        The request you have made requires authentication. (HTTP 401) (Request-ID: req-6bb876fb-a133-439e-87ce-862
        ea0fda57b)
        ^CTraceback (most recent call last):
          File "/bin/tripleo", line 12, in <module>
            sys.exit(subprocess.call(command))
          File "/usr/lib64/python2.7/subprocess.py", line 524, in call
            return Popen(*popenargs, **kwargs).wait()
          File "/usr/lib64/python2.7/subprocess.py", line 1376, in wait
            pid, sts = _eintr_retry_call(os.waitpid, self.pid, 0)
          File "/usr/lib64/python2.7/subprocess.py", line 478, in _eintr_retry_call
            return func(*args)
        KeyboardInterrupt
        #################

Package versions for reference 

        [stack@instack ~]$ rpm -qa | grep "python-tripleoclient\|tripleo-common"
        openstack-tripleo-common-5.3.0-3.el7ost.noarch
        python-tripleoclient-5.3.0-3.el7ost.noarch

An immediate workaround is to use the OVERCLOUD_ADMIN_PASSWORD from the tripleo-overcloud-passwords file which still contains the original value. You need to use that value as the
"OS_PASSWORD=" in the overcloudrc (the value of the overcloudrc is overwritten which is what this bug is about).

Reproduced the issue despite using the patch: https://review.openstack.org/#/c/391201

(In reply to Alexander Chuzhoy from comment #4)
> Reproduced the issue despite using the patch:
> https://review.openstack.org/#/c/391201

+1 me too Sasha I added a comment on the review, repeating and adding needinfo rbrady please check this out when you get a chance:

this fix doesn't seem to work here. I applied before running the ceilometer migration:

        [stack@instack ~]$ curl https://review.openstack.org/changes/391201/revisions/current/patch?download | \
        >      base64 -d | \
        >      sudo patch  -d /usr/lib/python2.7/site-packages/ -p1
          % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                         Dload  Upload   Total   Spent    Left  Speed
        100  2108    0  2108    0     0   2280      0 --:--:-- --:--:-- --:--:--  2278
        patching file tripleoclient/utils.py


But after that was UPDATE_COMPLETE the overcloudrc is still changed:


        [stack@instack ~]$ diff overcloudrc overcloudrcBACKUP 
        8c8
        < export OS_PASSWORD=2480e4fcb377e8f7fe9c544b9ccf30a4cd4e98a1
        ---
        > export OS_PASSWORD=M4bkCjyGa6mpKKNTv9evKMaE3

Am I missing another review or something else we need here?

I hit similar issue, I can't login to horizon using the admin password. 
Prior that I performed scale out for compute node, ever since then the overcloud admin password is invalid

adding needinfo @dougal o/ hey not sure if rbrady is out but is this something you can checkout when you have a chance please? The fix in review doesn't seem to do what we need.

The problem here is that when we moved the password generation to Mistral we stopped checking for passwords in the tripleo-overcloud-passwords file and users upgrading from Mitaka have that file as the "source of truth" for passwords. We need to respect that and favour those passwords over the passwords stored in the Mistral environment.

I have started a patch to do this: https://review.openstack.org/#/c/392593

It is still WIP as I am in the process of setting up an env to test it now.

I have updated the patch again now after lbezdick was able to test it for me. https://review.openstack.org/#/c/392593/

There is one case that I am unsure how to handle. In Mitaka, when a user deploys the password file is created, then when they re-deployed it would be used again. If the user changes directory and the password file can't be found then there would be an error "The password file could not be found!". However, now, if the file can't be found it looks like a Newton deploy (because the file isn't ever created in Newton).

So, I can see this possible scenario that would cause a problem:

1. Install Mitaka
2. Change directory
3. Update packages to Newton
4. Run upgrade deploy

The code will not find the passwords file because of the directory change, so it wont know that it is an upgrade and will generate new passwords.

Any ideas?

Thanks for your quick help, Dougal.

I can only think of two approaches for the problem at comment 9:
1) We document the need to launch deploy commands from the same directory
2) We somehow detect in tripleoclient that we are upgrading from mitaka and return "Not Found" if we cannot find the file in this case.

I'd prefer option 2) but I cannot think of a quick/easy and non-fragile way to detect that we are upgrading from mitaka. Maybe other folks in this BZ have ideas? Maybe some field in the mistral environment that we can query?

+1 thanks very much for jumping on this Dougal. WRT the passwords file and changing params, I don't think that needs to be anything more than a docs note. The scenario you describe in comment #9 applies to everything Mitaka and before:

1. Install Mitaka
2. Change directory
3. Stack update Mitaka (e.g. scale computes, change params/config, minor update)


you get the same issue and we've had bugs filed in past for this - see https://access.redhat.com/solutions/2678131

After some issues with the current approach, and hacks uppon hacks to try and resolve it I have gone in a different direction.

This change is much smaller, simpler and should be more robust. https://review.openstack.org/#/c/393831

This landed into Newton so moving to POST https://review.openstack.org/#/c/394195/ 

Also removing the two abandoned reviews from the linked/external trackers (https://review.openstack.org/#/c/392593/ and https://review.openstack.org/#/c/391201/)

Verified:
Environment:
openstack-tripleo-common-5.3.0-6.el7ost.noarch

The reported issue didn't reproduce.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-2948.html