Bug 1703045

I've encountered with similar problem when try to install 
OSP 13 with Spine-Leaf network topology + OVN + Octavia + 2 Networker nodes.
compose - 2019-01-10.1
openstack-tripleo-common-containers-8.6.6-8.el7ost.noarch
openstack-tripleo-common-8.6.6-8.el7ost.noarch
openstack-tripleo-ui-8.3.2-2.el7ost.noarch
python-tripleoclient-9.2.6-5.el7ost.noarch
openstack-tripleo-puppet-elements-8.0.1-1.el7ost.noarch
openstack-tripleo-validations-8.4.4-1.el7ost.noarch
puppet-tripleo-8.3.6-7.el7ost.noarch
ansible-tripleo-ipsec-8.1.1-0.20180308133440.8f5369a.el7ost.noarch
openstack-tripleo-image-elements-8.0.1-2.el7ost.noarch
openstack-tripleo-heat-templates-8.0.7-21.el7ost.noarch

puppet-octavia-12.4.0-7.el7ost.noarch
octavia-amphora-image-13.0-20190109.1.el7ost.noarch
python2-octaviaclient-1.4.0-1.1.el7ost.noarch
octavia-amphora-image-x86_64-13.0-20190109.1.el7ost.noarch

Roles configuration roles_data.yaml

- name: Controller0
...
  ServicesDefault:
...
    - OS::TripleO::Services::OctaviaApi
...

- name: Networker0
...
  ServicesDefault:
...
    - OS::TripleO::Services::OctaviaDeploymentConfig
    - OS::TripleO::Services::OctaviaHealthManager
    - OS::TripleO::Services::OctaviaHousekeeping
    - OS::TripleO::Services::OctaviaWorker

Deployment failed with error in mistral:
overcloud.AllNodesDeploySteps.WorkflowTasks_Step5_Execution:
  resource_type: OS::TripleO::WorkflowSteps
  physical_resource_id: c1120606-31b9-47bc-8cbf-d543beb08e1e
  status: CREATE_FAILED
  status_reason: |
    resources.WorkflowTasks_Step5_Execution: ERROR

More detailes about Octavia failure

2019-06-06 15:56:02.263 31313 INFO workflow_trace [req-fd58419e-01c7-49bc-93ba-af2f1fd87dac 78f7841a1c30427dbee402e4fd056f55 d70b2b1ee37b4acf8b250
e047e8db948 - default default] Workflow 'tripleo.overcloud.workflow_tasks.step5' [RUNNING -> ERROR, msg=Failure caused by error in tasks: octavia_
post_workflow

  octavia_post_workflow [task_ex_id=9b26f380-741d-4995-9d8e-1f2cc6e24133] -> Failure caused by error in tasks: config_octavia

  config_octavia [task_ex_id=29b1a410-2566-4a53-a64b-c0c5b0f5be81] -> Failed to run action [action_ex_id=6710acf0-b9fe-435d-beb4-9ba4e132a961, act
ion_cls='<class 'mistral.actions.action_factory.AnsiblePlaybookAction'>', attributes='{}', params='{u'remote_user': u'tripleo-admin', u'ssh_extra_
args': u'-o UserKnownHostsFile=/dev/null', u'inventory': {u'octavia_nodes': {u'hosts': {u'192.168.24.20': {}, u'192.168.24.36': {}}}}, u'become_us
er': u'root', u'extra_vars': {u'client_cert_path': u'/etc/octavia/certs/client.pem', u'mgmt_port_dev': u'o-hm0', u'lb_mgmt_subnet_name': u'lb-mgmt
-subnet', u'lb_mgmt_net_name': u'lb-mgmt-net', u'os_project_name': u'admin', u'os_username': u'admin', u'os_identity_api_version': u'3', u'amp_ima
ge_tag': u'amphora-image', u'ca_passphrase': u'mpTCUTuw7Kr3WTzGJX8grWsMr', u'auth_project_name': u'service', u'lb_mgmt_subnet_gateway': u'172.24.0
.1', u'lb_mgmt_subnet_pool_start': u'172.24.0.2', u'lb_mgmt_subnet_pool_end': u'172.24.255.254', u'os_auth_url': u'https://10.0.10.100:13000/v3',
u'ca_private_key_path': u'/etc/octavia/certs/private/cakey.pem', u'lb_sec_group_name': u'lb-mgmt-subnet', u'os_password': u'xUKFxkHwwqUkRgxAecCg4T
Qvw', u'os_auth_type': u'password', u'lb_mgmt_subnet_cidr': u'172.24.0.0/16', u'ca_cert_path': u'/etc/octavia/certs/ca_01.pem', u'generate_certs':
 True}, u'verbosity': 0, u'extra_env_variables': {u'ANSIBLE_SSH_RETRIES': u'3', u'ANSIBLE_HOST_KEY_CHECKING': u'False'}, u'ssh_private_key': u'---
--BEGIN RSA PRIVATE KEY-----\nMIIEowIBAAKCAQEA3ZA5ubbSQhIybnX1GaDWeqa4XEeo8rYgCNseSdB/rfw19ZJ7\nLqjopsadqcH8IrIdFN0MBWRlcYe0BAxP3O7C1CU5+Yl/RTa9cV
XP9S8m2WEhhwYY\np/wxLnxzFRaCjMWolRwuQZb+j/JSd6c5Z/+en502lVmybFv/QYXredT0ZtwG5GJb\nuLZ9LqVLYaGDZ/y9BWGZDztwVjWBmWlQG5PSU+MAgpjxNtj4AlGCVqHNRdnvLS4e
\nXw+XJXDs5IQMU+yFrqrNP8rVvW2CJwjJs42dN70QyFDUIMvRkYS+vDCOWFp9L5CI\nyjfEFZOir8O93hR4Yp3kY2kzr+6XZt5+yfl/qwIDAQABAoIBADTolgBVOgxxD/30\nyRzfnZgYa/oN
Wrjq6Od0e90gnvzLN4929VeFGlmJIGlDW5RleDBdQNugx+C+iSxW\nTFPz6C6E3T1Lqkq68a440bo5EkviuADoYcbroEr7iPfGMlKveLxyyFD9X7i8IxlT\no4/EKPrwrfIoQ0VBCsl402x6gA
p3mpyi5sUa2d2xDkIs4xmJ+qvHLKzRqg6YrNTu\n5FCRRhsARw/IcJXxLmMu0g67i9O1kMUGbs6XX8x5JMBigNY+i6RS97JTIu+ddXpv\njT7pOOVwY0EymWyDIKyb9pPnsuJjYw2JFaDEANU/
zpv6Vd6ojjpx32RSuqPWNZzI\nYhrEEGECgYEA9JSZkHPk9EPVLdPhUP9BETKUgto2/m7jpP51QRywS1OXT4AUzPFS\nN+uKsufKZ0meK0/LB35HP1Jet7OJeJShpXPMbG2sCWR13fGsiR4NZZ
8iW6gy6kIJ\nN2BBAHzDLRzFFdLkzq30BwfNMWNoIMp5PQsxmAkLSABGBgkOVOI4+hkCgYEA5+iC\nTS6921oQVkuUusswhy+V0pV9guD2AWqzyNy1KIOLa6bShbmrtrGRPAYG5HGHbbb9\n73
BeXuTBP8yFb2NOYNqLNvTRZ7OucEpY0oVdR65v+DMHZR/jVM6NEn4jfrVIClYX\nvCyHwsqRFe1NreBNIAHvdT6OfNYV4T3qWrsgCGMCgYBKVELbKK2DIn5OAB9wqzJO\nFK4XmlOuPWsHgKGH
2T0ml0/bxFQN+KUBA59SQak8fJ4KEaTlMRZcAx9v+qsjrx/1\nFV0h8q6e6B3+Bm1l+nEd2h/p9RMMKGd+ocz/Zes28ZBf0ojg2vLXlCJjCQ/jL0Vr\nLNS0nMMF7bdaLDRjzaB9OQKBgA8ZYV
p7H5tnisbDlwRudFNo8r1KRGjAEuRWuSvr\nytO/dNVmgDB6vUZg207oKYy4I5QuJOxxCYPuKvLncwykj5bYw9WpLPUuir3+6TeT\nvVYMcnfbgrC/2cJMzHyWv+LhFLavkk4LLC+vlrCxyav3
fa4G0jt0/jv8iGIo8NhF\ndLl3AoGBANVn9ySPH9JnyCgUhaTIJtJbL3CDGwmH8VdDrx4hgmML5/7d53cQDEX3\n7ml3GpCsfv+vWB/QgC9q0iMOL8reuNaaT1Unj+52v+jbD6GVNsx+54SrlI
1zF11d\nYYKn3NBgJI9XkIr5NK1VE48xiopCdVM5DNClC8YjJYDPb7Xp5JKt\n-----END RSA PRIVATE KEY-----\n', u'become': True, u'ssh_common_args': u'-o StrictHo
stKeyChecking=no', u'playbook': u'/usr/share/tripleo-common/playbooks/octavia-files.yaml'}']
 Unexpected error while running command.
TASK [octavia-overcloud-config : include_tasks] ********************************\nThursday 06 June 2019  15:55:57 -0400 (0:00:03.675)       0:00:03.754 ********* \nincluded: /usr/share/open
stack-tripleo-common/playbooks/roles/octavia-overcloud-config/tasks/network.yml for 192.168.24.20\n\nTASK [octavia-overcloud-config : create manag
ement network for load balancers] ***\nThursday 06 June 2019  15:55:57 -0400 (0:00:00.047)       0:00:03.802 ********* \nfatal: [192.168.24.20]: F
AILED! => {"changed": false, "cmd": "if [[ $(openstack network show lb-mgmt-net > /dev/null; echo $?) -eq 1 ]]; then\\n openstack network create -
f value -c id lb-mgmt-net\\n fi", "delta": "0:00:01.964978", "end": "2019-06-06 19:55:59.512877", "failed": true, "msg": "non-zero return code", "
rc": 1, "start": "2019-06-06 19:55:57.547899", "stderr": "Failed to discover available identity versions when contacting https://10.0.10.100:13000
/v3. Attempting to parse version from URL.\\nUnable to establish connection to https://10.0.10.100:13000/v3/auth/tokens: HTTPSConnectionPool(host=
\'10.0.10.100\', port=13000): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError(\'<requests.packages.urllib3.connection
.VerifiedHTTPSConnection object at 0x7f6e2aff7910>: Failed to establish a new connection: [Errno 101] Network is unreachable\',))\\nFailed to disc
over available identity versions when contacting https://10.0.10.100:13000/v3. Attempting to parse version from URL.\\nUnable to establish connect
ion to https://10.0.10.100:13000/v3/auth/tokens: HTTPSConnectionPool(host=\'10.0.10.100\', port=13000): Max retries exceeded with url: /v3/auth/to
kens (Caused by NewConnectionError(\'<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f894e7f4910>: Failed to establish
a new connection: [Errno 101] Network is unreachable\',))", "stderr_lines": ["Failed to discover available identity versions when contacting https
://10.0.10.100:13000/v3. Attempting to parse version from URL.", "Unable to establish connection to https://10.0.10.100:13000/v3/auth/tokens: HTTP
SConnectionPool(host=\'10.0.10.100\', port=13000): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError(\'<requests.packag
es.urllib3.connection.VerifiedHTTPSConnection object at 0x7f6e2aff7910>: Failed to establish a new connection: [Errno 101] Network is unreachable\
',))", "Failed to discover available identity versions when contacting https://10.0.10.100:13000/v3. Attempting to parse version from URL.", "Unab
le to establish connection to https://10.0.10.100:13000/v3/auth/tokens: HTTPSConnectionPool(host=\'10.0.10.100\', port=13000): Max retries exceede
d with url: /v3/auth/tokens (Caused by NewConnectionError(\'<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f894e7f4910
>: Failed to establish a new connection: [Errno 101] Network is unreachable\',))"], "stdout": "", "stdout_lines": []}\n\nPLAY RECAP **************
*******************************************************\n192.168.24.20              : ok=2    changed=0    unreachable=0    failed=1   \n\nThursda
y 06 June 2019  15:55:59 -0400 (0:00:02.385)       0:00:06.188 ********* \n=======================================================================
======== \n'
Stderr: u' [WARNING]: Could not match supplied host pattern, ignoring: undercloud\n'
2019-06-06 15:55:59.598 31393 ERROR mistral.executors.default_executor Traceback (most recent call last):
2019-06-06 15:55:59.598 31393 ERROR mistral.executors.default_executor   File "/usr/lib/python2.7/site-packages/mistral/executors/default_executor
.py", line 114, in run_action
2019-06-06 15:55:59.598 31393 ERROR mistral.executors.default_executor     result = action.run(action_ctx)
2019-06-06 15:55:59.598 31393 ERROR mistral.executors.default_executor   File "/usr/lib/python2.7/site-packages/tripleo_common/actions/ansible.py"
, line 541, in run
2019-06-06 15:55:59.598 31393 ERROR mistral.executors.default_executor     log_errors=processutils.LogErrors.ALL)
2019-06-06 15:55:59.598 31393 ERROR mistral.executors.default_executor   File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py",
 line 424, in execute
2019-06-06 15:55:59.598 31393 ERROR mistral.executors.default_executor     cmd=sanitized_cmd)
2019-06-06 15:55:59.598 31393 ERROR mistral.executors.default_executor ProcessExecutionError: Unexpected error while running command.
2019-06-06 15:55:59.598 31393 ERROR mistral.executors.default_executor Command: ansible-playbook /usr/share/tripleo-common/playbooks/octavia-files:

I'm trying to apply this patch in RDO stein but am having trouble with "octavia-files.yaml". The problem is that it's part of the container images being downloaded, so replacing it on TripleO won't make a difference.

What I can do is replace it in the container image mount after it's deployed, and then restart the container service.

Could you tell me which exact container I need to update?

It looks like it could be one of these container images:

* nova_scheduler
* mistral_executor
* mistral_event_engine
* mistral_engine
* mistral_api

The patch does not work for me -- I get the same "Failed to establish a new connection" error.

It was hard to apply the patch, but I patched all the container images mentioned above as well as the TripeO machine.

Also note that I think you are missing something in the patch: in "octava-files.yaml" you should probably also patch line 22 to use os_int_auth_url. In any case, the patch does not work even with my addition.

Please advise on next steps! This is a blocker for a project at Red Hat's CTO office.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:3745