Bug 1659866

Description of problem:

One of our Cu is trying to save the initial overcloud state in our RHOSP 13 environment but the --init-saved-state command is failing for multiple reasons:

1. Tempest is still trying to reach Keystone over the adminURL endpoint which is not reachable from the undercloud, the identity section of tempest.conf was updated to force tempest to use the publicURL but it looks like tempest is ignoring this (at least for the cleanup --init-saved-state option)

2. Cu was able to update the python scripts to get past the first issue but then I see that tempest is trying to access v2 Keystone endpoints to list the current roles which has since been deprecated in queens:
~~~
2018-12-01 01:17:18.481 589448 INFO tempest.lib.common.rest_client [req-7079564c-3525-48ce-9a3e-654c4cf94280 ] Request (main): 404 GET http://10.0.3.10:5000/v2.0/OS-KSADM/roles 0.150s
~~~

As for the configuration, Cu is generating that manually.

You won't see one of the issues, Cu is reporting if you are running tempest directly from a controller that can reach the adminURL...in their case, they are running tempest from the undercloud which has no connectivity to the adminURL.

The point is the tempest.conf configuration clearly points to the publicURL of keystone but this is in face being ignored by Tempest

~~~
Command: tempest cleanup --init-saved-state

Errors in logs showing both the attempt to connect over the Internal API network (aka adminURL) AND using the v2 keystone endpoint deprecated in RHOSP queens:

2018-12-12 22:29:11.050 3598 DEBUG tempest.cmd.cleanup_service [-] List count, 2 Domains after reconcile list /usr/lib/python2.7/site-packages/tempest/cmd/cleanup_service.py:897
2018-12-12 22:30:11.112 3598 WARNING urllib3.connectionpool [-] Retrying (Retry(total=9, connect=None, read=None, redirect=5, status=None)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f1a1a220290>, 'Connection to 10.0.3.10 timed out. (connect timeout=60)')': /v2.0/OS-KSADM/roles: ConnectTimeoutError: (<urllib3.connection.HTTPConnection object at 0x7f1a1a220290>, 'Connection to 10.0.3.10 timed out. (connect timeout=60)')
2018-12-12 22:31:11.137 3598 WARNING urllib3.connectionpool [-] Retrying (Retry(total=8, connect=None, read=None, redirect=5, status=None)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f1a1a220310>, 'Connection to 10.0.3.10 timed out. (connect timeout=60)')': /v2.0/OS-KSADM/roles: ConnectTimeoutError: (<urllib3.connection.HTTPConnection object at 0x7f1a1a220310>, 'Connection to 10.0.3.10 timed out. (connect timeout=60)')
2018-12-12 22:32:11.197 3598 WARNING urllib3.connectionpool [-] Retrying (Retry(total=7, connect=None, read=None, redirect=5, status=None)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f1a1a220850>, 'Connection to 10.0.3.10 timed out. (connect timeout=60)')': /v2.0/OS-KSADM/roles: ConnectTimeoutError: (<urllib3.connection.HTTPConnection object at 0x7f1a1a220850>, 'Connection to 10.0.3.10 timed out. (connect timeout=60)')
2018-12-12 22:33:11.254 3598 WARNING urllib3.connectionpool [-] Retrying (Retry(total=6, connect=None, read=None, redirect=5, status=None)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f1a1a220690>, 'Connection to 10.0.3.10 timed out. (connect timeout=60)')': /v2.0/OS-KSADM/roles: ConnectTimeoutError: (<urllib3.connection.HTTPConnection object at 0x7f1a1a220690>, 'Connection to 10.0.3.10 timed out. (connect timeout=60)')
2018-12-12 22:34:11.315 3598 WARNING urllib3.connectionpool [-] Retrying (Retry(total=5, connect=None, read=None, redirect=5, status=None)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f1a1a220a50>, 'Connection to 10.0.3.10 timed out. (connect timeout=60)')': /v2.0/OS-KSADM/roles: ConnectTimeoutError: (<urllib3.connection.HTTPConnection object at 0x7f1a1a220a50>, 'Connection to 10.0.3.10 timed out. (connect timeout=60)')
~~~

Version-Release number of selected component (if applicable):

$ rpm -qa | grep tempest
python2-tempest-18.0.0-2.el7ost.noarch
python2-tempestconf-1.1.5-0.20180326143753.f9d956f.el7ost.noarch
openstack-tempest-18.0.0-2.el7ost.noarch


Actual results:
Trying to run tempest from undercloud, tempest cleanup save state fails

Expected results:
Ability to run tempest from undercloud, and tempest cleanup save state should not fail.