Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1812744

Summary:	Octavia fails to create loadbalancers when TLS everywhere and OVN provider driver enabled
Product:	Red Hat OpenStack	Reporter:	Sadique Puthen <sputhenp>
Component:	openstack-tripleo-heat-templates	Assignee:	Assaf Muller <amuller>
Status:	CLOSED DUPLICATE	QA Contact:	Bruna Bonguardo <bbonguar>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	16.0 (Train)	CC:	atragler, cgoncalves, ihrachys, jlibosva, lpeer, majopela, mburns, michjohn, scohen
Target Milestone:	---	Keywords:	AutomationBlocker, Regression
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-03-17 14:14:06 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Sadique Puthen 2020-03-12 03:29:31 UTC

Description of problem:

Octavia with TLS everywhere fails to create loadbalancers. Deployment is successful, but lb creation fails.

$ openstack loadbalancer create --name lb1 --vip-subnet-id private-1-subnet-1
Expecting value: line 1 column 1 (char 0)

I can see plenty of messages like this.

2020-03-10 11:04:02.924 24 INFO ovsdbapp.backend.ovs_idl.idlutils [-] Could not retrieve schema from tcp:172.20.0.104:6641: Unknown error -1
2020-03-10 11:07:02.935 23 INFO ovsdbapp.backend.ovs_idl.idlutils [-] Could not retrieve schema from tcp:172.20.0.104:6641: Unknown error -1
2020-03-10 11:07:03.029 24 INFO ovsdbapp.backend.ovs_idl.idlutils [-] Could not retrieve schema from tcp:172.20.0.104:6641: Unknown error -1

Debug output.

(overcloud) [stack@undercloud ~]$ openstack --debug loadbalancer create --name lb1 --vip-subnet-id private-1-subnet-1
START with options: --debug loadbalancer create --name lb1 --vip-subnet-id private-1-subnet-1
options: Namespace(access_token='***', access_token_endpoint='', access_token_type='', aodh_endpoint='', application_credential_id='', application_credential_name='', application_credential_secret='***', auth_methods='', auth_type='password', auth_url='https://overcloud.redhat.local:13000', cacert=None, cert='', client_id='', client_secret='***', cloud='', code='', debug=True, default_domain='default', default_domain_id='', default_domain_name='', deferred_help=False, discovery_endpoint='', domain_id='', domain_name='', endpoint='', identity_provider='', identity_provider_url='', insecure=None, inspector_api_version='1', inspector_url=None, interface='public', key='', log_file=None, openid_scope='', os_alarming_api_version='2', os_baremetal_api_version='1.58', os_beta_command=False, os_compute_api_version='2.latest', os_container_infra_api_version='1', os_data_processing_api_version='1.1', os_data_processing_url='', os_database_api_version='1', os_dns_api_version='2', os_event_api_version='2', os_identity_api_version='3', os_image_api_version='2', os_key_manager_api_version='1', os_loadbalancer_api_version='2.0', os_metrics_api_version='1', os_network_api_version='', os_object_api_version='', os_orchestration_api_version='1', os_project_id=None, os_project_name=None, os_queues_api_version='2', os_tripleoclient_api_version='1', os_volume_api_version='3', os_workflow_api_version='2', passcode='', password='***', profile='', project_domain_id='', project_domain_name='Default', project_id='', project_name='admin', protocol='', redirect_uri='', region_name='regionOne', remote_project_domain_id='', remote_project_domain_name='', remote_project_id='', remote_project_name='', roles='', service_provider='', service_provider_endpoint='', service_provider_entity_id='', system_scope='', timing=False, token='***', trust_id='', user='', user_domain_id='', user_domain_name='Default', user_id='', username='admin', verbose_level=3, verify=None)
Auth plugin password selected
auth_config_hook(): {'api_timeout': None, 'verify': True, 'cacert': None, 'cert': None, 'key': None, 'baremetal_status_code_retries': '5', 'baremetal_introspection_status_code_retries': '5', 'image_status_code_retries': '5', 'disable_vendor_agent': {}, 'interface': 'public', 'floating_ip_source': 'neutron', 'image_api_use_tasks': False, 'image_format': 'qcow2', 'message': '', 'network_api_version': '2', 'object_store_api_version': '1', 'secgroup_source': 'neutron', 'status': 'active', 'auth': {'user_domain_name': 'Default', 'project_domain_name': 'Default', 'project_name': 'admin'}, 'verbose_level': 3, 'deferred_help': False, 'debug': True, 'region_name': 'regionOne', 'default_domain': 'default', 'timing': False, 'inspector_api_version': '1', 'auth_url': 'https://overcloud.redhat.local:13000', 'username': 'admin', 'password': '***', 'beta_command': False, 'compute_api_version': '2.latest', 'identity_api_version': '3', 'image_api_version': '2', 'volume_api_version': '3', 'queues_api_version': '2', 'database_api_version': '1', 'tripleoclient_api_version': '1', 'data_processing_api_version': '1.1', 'loadbalancer_api_version': '2.0', 'workflow_api_version': '2', 'container_infra_api_version': '1', 'baremetal_api_version': '1.58', 'orchestration_api_version': '1', 'dns_api_version': '2', 'key_manager_api_version': '1', 'event_api_version': '2', 'metrics_api_version': '1', 'alarming_api_version': '2', 'auth_type': 'password', 'networks': []}
defaults: {'api_timeout': None, 'verify': True, 'cacert': None, 'cert': None, 'key': None, 'auth_type': 'password', 'baremetal_status_code_retries': 5, 'baremetal_introspection_status_code_retries': 5, 'image_status_code_retries': 5, 'disable_vendor_agent': {}, 'interface': None, 'floating_ip_source': 'neutron', 'image_api_use_tasks': False, 'image_format': 'qcow2', 'message': '', 'network_api_version': '2', 'object_store_api_version': '1', 'secgroup_source': 'neutron', 'status': 'active'}
cloud cfg: {'api_timeout': None, 'verify': True, 'cacert': None, 'cert': None, 'key': None, 'baremetal_status_code_retries': '5', 'baremetal_introspection_status_code_retries': '5', 'image_status_code_retries': '5', 'disable_vendor_agent': {}, 'interface': 'public', 'floating_ip_source': 'neutron', 'image_api_use_tasks': False, 'image_format': 'qcow2', 'message': '', 'network_api_version': '2', 'object_store_api_version': '1', 'secgroup_source': 'neutron', 'status': 'active', 'auth': {'user_domain_name': 'Default', 'project_domain_name': 'Default', 'project_name': 'admin'}, 'verbose_level': 3, 'deferred_help': False, 'debug': True, 'region_name': 'regionOne', 'default_domain': 'default', 'timing': False, 'inspector_api_version': '1', 'auth_url': 'https://overcloud.redhat.local:13000', 'username': 'admin', 'password': '***', 'beta_command': False, 'compute_api_version': '2.latest', 'identity_api_version': '3', 'image_api_version': '2', 'volume_api_version': '3', 'queues_api_version': '2', 'database_api_version': '1', 'tripleoclient_api_version': '1', 'data_processing_api_version': '1.1', 'loadbalancer_api_version': '2.0', 'workflow_api_version': '2', 'container_infra_api_version': '1', 'baremetal_api_version': '1.58', 'orchestration_api_version': '1', 'dns_api_version': '2', 'key_manager_api_version': '1', 'event_api_version': '2', 'metrics_api_version': '1', 'alarming_api_version': '2', 'auth_type': 'password', 'networks': []}
compute API version 2.latest, cmd group openstack.compute.v2
identity API version 3, cmd group openstack.identity.v3
image API version 2, cmd group openstack.image.v2
network API version 2, cmd group openstack.network.v2
object_store API version 1, cmd group openstack.object_store.v1
volume API version 3, cmd group openstack.volume.v3
messaging API version 2, cmd group openstack.messaging.v2
database API version 1, cmd group openstack.database.v1
tripleoclient API version 1, cmd group openstack.tripleoclient.v1
data_processing API version 1.1, cmd group openstack.data_processing.v1
load_balancer API version 2.0, cmd group openstack.load_balancer.v2
neutronclient API version 2, cmd group openstack.neutronclient.v2
workflow_engine API version 2, cmd group openstack.workflow_engine.v2
container_infra API version 1, cmd group openstack.container_infra.v1
baremetal API version 1.58, cmd group openstack.baremetal.v1
baremetal_introspection API version 1, cmd group openstack.baremetal_introspection.v1
orchestration API version 1, cmd group openstack.orchestration.v1
dns API version 2, cmd group openstack.dns.v2
key_manager API version 1, cmd group openstack.key_manager.v1
event API version 2, cmd group openstack.event.v2
metric API version 1, cmd group openstack.metric.v1
alarming API version 2, cmd group openstack.alarming.v2
Auth plugin password selected
auth_config_hook(): {'api_timeout': None, 'verify': True, 'cacert': None, 'cert': None, 'key': None, 'baremetal_status_code_retries': '5', 'baremetal_introspection_status_code_retries': '5', 'image_status_code_retries': '5', 'disable_vendor_agent': {}, 'interface': 'public', 'floating_ip_source': 'neutron', 'image_api_use_tasks': False, 'image_format': 'qcow2', 'message': '', 'network_api_version': '2', 'object_store_api_version': '1', 'secgroup_source': 'neutron', 'status': 'active', 'auth': {'user_domain_name': 'Default', 'project_domain_name': 'Default', 'project_name': 'admin'}, 'verbose_level': 3, 'deferred_help': False, 'debug': True, 'region_name': 'regionOne', 'default_domain': 'default', 'timing': False, 'inspector_api_version': '1', 'auth_url': 'https://overcloud.redhat.local:13000', 'username': 'admin', 'password': '***', 'beta_command': False, 'compute_api_version': '2.latest', 'identity_api_version': '3', 'image_api_version': '2', 'volume_api_version': '3', 'queues_api_version': '2', 'database_api_version': '1', 'tripleoclient_api_version': '1', 'data_processing_api_version': '1.1', 'loadbalancer_api_version': '2.0', 'workflow_api_version': '2', 'container_infra_api_version': '1', 'baremetal_api_version': '1.58', 'orchestration_api_version': '1', 'dns_api_version': '2', 'key_manager_api_version': '1', 'event_api_version': '2', 'metrics_api_version': '1', 'alarming_api_version': '2', 'auth_type': 'password', 'networks': []}
command: loadbalancer create -> octaviaclient.osc.v2.load_balancer.CreateLoadBalancer (auth=True)
Auth plugin password selected
auth_config_hook(): {'api_timeout': None, 'verify': True, 'cacert': None, 'cert': None, 'key': None, 'baremetal_status_code_retries': '5', 'baremetal_introspection_status_code_retries': '5', 'image_status_code_retries': '5', 'disable_vendor_agent': {}, 'interface': 'public', 'floating_ip_source': 'neutron', 'image_api_use_tasks': False, 'image_format': 'qcow2', 'message': '', 'network_api_version': '2', 'object_store_api_version': '1', 'secgroup_source': 'neutron', 'status': 'active', 'auth': {'user_domain_name': 'Default', 'project_domain_name': 'Default', 'project_name': 'admin'}, 'additional_user_agent': [('osc-lib', '1.14.1')], 'verbose_level': 3, 'deferred_help': False, 'debug': True, 'region_name': 'regionOne', 'default_domain': 'default', 'timing': False, 'inspector_api_version': '1', 'auth_url': 'https://overcloud.redhat.local:13000', 'username': 'admin', 'password': '***', 'beta_command': False, 'compute_api_version': '2.latest', 'identity_api_version': '3', 'image_api_version': '2', 'volume_api_version': '3', 'queues_api_version': '2', 'database_api_version': '1', 'tripleoclient_api_version': '1', 'data_processing_api_version': '1.1', 'loadbalancer_api_version': '2.0', 'workflow_api_version': '2', 'container_infra_api_version': '1', 'baremetal_api_version': '1.58', 'orchestration_api_version': '1', 'dns_api_version': '2', 'key_manager_api_version': '1', 'event_api_version': '2', 'metrics_api_version': '1', 'alarming_api_version': '2', 'auth_type': 'password', 'networks': []}
Using auth plugin: password
Using parameters {'auth_url': 'https://overcloud.redhat.local:13000', 'project_name': 'admin', 'project_domain_name': 'Default', 'username': 'admin', 'user_domain_name': 'Default', 'password': '***'}
Get auth_ref
REQ: curl -g -i -X GET https://overcloud.redhat.local:13000 -H "Accept: application/json" -H "User-Agent: openstacksdk/0.36.0 keystoneauth1/3.17.1 python-requests/2.20.0 CPython/3.6.8"
Starting new HTTPS connection (1): overcloud.redhat.local:13000
https://overcloud.redhat.local:13000 "GET / HTTP/1.1" 300 279
RESP: [300] Content-Length: 279 Content-Type: application/json Date: Tue, 10 Mar 2020 15:48:02 GMT Location: https://overcloud.redhat.local:13000/v3/ Server: Apache Vary: X-Auth-Token x-openstack-request-id: req-65d8363a-c4f2-42eb-b742-65ff4cc6da1d
RESP BODY: {"versions": {"values": [{"id": "v3.13", "status": "stable", "updated": "2019-07-19T00:00:00Z", "links": [{"rel": "self", "href": "https://overcloud.redhat.local:13000/v3/"}], "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}]}]}}
GET call to https://overcloud.redhat.local:13000/ used request id req-65d8363a-c4f2-42eb-b742-65ff4cc6da1d
Making authentication request to https://overcloud.redhat.local:13000/v3/auth/tokens
https://overcloud.redhat.local:13000 "POST /v3/auth/tokens HTTP/1.1" 201 7898
{"token": {"methods": ["password"], "user": {"domain": {"id": "default", "name": "Default"}, "id": "1cda561adebf493b9ba3f3686de23522", "name": "admin", "password_expires_at": null}, "audit_ids": ["utUFbIN_QpOkgQgQUyC0hw"], "expires_at": "2020-03-10T16:48:02.000000Z", "issued_at": "2020-03-10T15:48:02.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "34730f4be8c54c569b0059786c6e100a", "name": "admin"}, "is_domain": false, "roles": [{"id": "35108f06c3814bedadd524f98961c402", "name": "reader"}, {"id": "e3988e47d7bd499b85adc897575b93a0", "name": "admin"}, {"id": "ec2ea884e3a3413199b04aacdaf1bf4e", "name": "_member_"}, {"id": "50c378d402db40e187e2558f75cc5ea5", "name": "member"}], "catalog": [{"endpoints": [{"id": "3922473da40c4cc28d580caf892cefe5", "interface": "admin", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:9292", "region": "regionOne"}, {"id": "3d79504528ca4cdea4d9def8128584ae", "interface": "internal", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:9292", "region": "regionOne"}, {"id": "aa4be6ca56014583b573f19d7bfa3414", "interface": "public", "region_id": "regionOne", "url": "https://overcloud.redhat.local:13292", "region": "regionOne"}], "id": "01c96d92cb0a4cf99caf14894a4b6fe2", "type": "image", "name": "glance"}, {"endpoints": [{"id": "3ea1e66c5c38499e8a120383c83d5c82", "interface": "internal", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:8774/v2.1", "region": "regionOne"}, {"id": "70dd2ae2b77b4bb58d58b68fdf22d097", "interface": "public", "region_id": "regionOne", "url": "https://overcloud.redhat.local:13774/v2.1", "region": "regionOne"}, {"id": "75f53cd0dd094341a477d2351dd3a68b", "interface": "admin", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:8774/v2.1", "region": "regionOne"}], "id": "23d7862648774dfaa659f03c891e0d32", "type": "compute", "name": "nova"}, {"endpoints": [{"id": "68b7dfc435674139b7a69a7e82e0f411", "interface": "admin", "region_id": "regionOne", "url": "https://overcloud.ctlplane.redhat.local:35357", "region": "regionOne"}, {"id": "98c96768cf2a49279d41fb4f8fa2b25a", "interface": "public", "region_id": "regionOne", "url": "https://overcloud.redhat.local:13000", "region": "regionOne"}, {"id": "d745eccd33fb42b188d32e91e3bdf608", "interface": "internal", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:5000", "region": "regionOne"}], "id": "34efa591adb741d7b3d96f96599ab0ce", "type": "identity", "name": "keystone"}, {"endpoints": [{"id": "5e395d93f03240e99d0b4f077ed1c275", "interface": "internal", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:8778/placement", "region": "regionOne"}, {"id": "cd48a84fc4c64abba6c658e582f71471", "interface": "public", "region_id": "regionOne", "url": "https://overcloud.redhat.local:13778/placement", "region": "regionOne"}, {"id": "f5d9ae88e0fe4737816850954f5ee07e", "interface": "admin", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:8778/placement", "region": "regionOne"}], "id": "3602c7ee3242410f9b501a6a81202be0", "type": "placement", "name": "placement"}, {"endpoints": [{"id": "024aeb5a469643a2a1742cac264fc62e", "interface": "internal", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:8004/v1/34730f4be8c54c569b0059786c6e100a", "region": "regionOne"}, {"id": "97b5dd898e8d4652a9683fceb1e17417", "interface": "admin", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:8004/v1/34730f4be8c54c569b0059786c6e100a", "region": "regionOne"}, {"id": "f1cb7a31ee294117b324a8c6f165f9bd", "interface": "public", "region_id": "regionOne", "url": "https://overcloud.redhat.local:13004/v1/34730f4be8c54c569b0059786c6e100a", "region": "regionOne"}], "id": "658a3002a8834bdf8f8a17641a5e64fa", "type": "orchestration", "name": "heat"}, {"endpoints": [{"id": "1e0f347e07954e1daa8cc1d9a1995c32", "interface": "internal", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:8776/v3/34730f4be8c54c569b0059786c6e100a", "region": "regionOne"}, {"id": "2a7ef36db26745648a745d335ed0c7da", "interface": "admin", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:8776/v3/34730f4be8c54c569b0059786c6e100a", "region": "regionOne"}, {"id": "ea467ac38a2f4274aed9f885a60f8f22", "interface": "public", "region_id": "regionOne", "url": "https://overcloud.redhat.local:13776/v3/34730f4be8c54c569b0059786c6e100a", "region": "regionOne"}], "id": "746d540f633f47fdadb42a47a006a53d", "type": "volumev3", "name": "cinderv3"}, {"endpoints": [{"id": "9f766ff00dd946d1b0733f71a14e20fa", "interface": "admin", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:9876", "region": "regionOne"}, {"id": "ab844da69cd440068e4d22e43328549a", "interface": "internal", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:9876", "region": "regionOne"}, {"id": "fc8e322af23040dd84fc49623b767555", "interface": "public", "region_id": "regionOne", "url": "https://overcloud.redhat.local:13876", "region": "regionOne"}], "id": "89873f52593f4febb67a8f4777e1b8fe", "type": "load-balancer", "name": "octavia"}, {"endpoints": [{"id": "4af6105f51fe49be9977733043bcc663", "interface": "public", "region_id": "regionOne", "url": "https://overcloud.redhat.local:13808/v1/AUTH_34730f4be8c54c569b0059786c6e100a", "region": "regionOne"}, {"id": "c6f2836eb8474000ba519374e08c0718", "interface": "internal", "region_id": "regionOne", "url": "https://overcloud.storage.redhat.local:8080/v1/AUTH_34730f4be8c54c569b0059786c6e100a", "region": "regionOne"}, {"id": "dfe95e784b10461eb54e5a559d9d8519", "interface": "admin", "region_id": "regionOne", "url": "https://overcloud.storage.redhat.local:8080", "region": "regionOne"}], "id": "8c42aaaaeea240329068d2747a241c7a", "type": "object-store", "name": "swift"}, {"endpoints": [{"id": "42b7c7d88a2e43e38daec13674d2d3e4", "interface": "admin", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:8000/v1", "region": "regionOne"}, {"id": "c483eb4c24d44163944f961048d15709", "interface": "public", "region_id": "regionOne", "url": "https://overcloud.redhat.local:13005/v1", "region": "regionOne"}, {"id": "dbd00c1c249b4920be9ac149f7f5f700", "interface": "internal", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:8000/v1", "region": "regionOne"}], "id": "9fe809aa90af4b4090f5b6487e837369", "type": "cloudformation", "name": "heat-cfn"}, {"endpoints": [{"id": "052bca56ad9d46eeaa8992d9a11381de", "interface": "admin", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:9696", "region": "regionOne"}, {"id": "d16ca77062fa4884ab35e20b1e5a85b6", "interface": "internal", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:9696", "region": "regionOne"}, {"id": "fafe23c9370c471dbaaca291d9c80a34", "interface": "public", "region_id": "regionOne", "url": "https://overcloud.redhat.local:13696", "region": "regionOne"}], "id": "a0626844d0aa434d8212e92977f51dc8", "type": "network", "name": "neutron"}, {"endpoints": [{"id": "51fe869d894b4d12b0e6547a360b3a15", "interface": "internal", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:8776/v2/34730f4be8c54c569b0059786c6e100a", "region": "regionOne"}, {"id": "da99aa211700472fae4404709d89574c", "interface": "public", "region_id": "regionOne", "url": "https://overcloud.redhat.local:13776/v2/34730f4be8c54c569b0059786c6e100a", "region": "regionOne"}, {"id": "f0722b8a002d4a84825aacf180949c51", "interface": "admin", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:8776/v2/34730f4be8c54c569b0059786c6e100a", "region": "regionOne"}], "id": "e565d5609fe349479e748c62d532b882", "type": "volumev2", "name": "cinderv2"}]}}
run(Namespace(columns=[], description=None, disable=None, enable=True, fit_width=False, flavor=None, formatter='table', max_width=0, name='lb1', noindent=False, prefix='', print_empty=False, project=None, provider=None, variables=[], vip_address=None, vip_network_id=None, vip_port_id=None, vip_qos_policy_id=None, vip_subnet_id='private-1-subnet-1'))
Instantiating identity client: <class 'keystoneclient.v3.client.Client'>
Instantiating neutron client: <class 'neutronclient.v2_0.client.Client'>
Making authentication request to https://overcloud.redhat.local:13000/v3/auth/tokens
https://overcloud.redhat.local:13000 "POST /v3/auth/tokens HTTP/1.1" 201 7898
{"token": {"methods": ["password"], "user": {"domain": {"id": "default", "name": "Default"}, "id": "1cda561adebf493b9ba3f3686de23522", "name": "admin", "password_expires_at": null}, "audit_ids": ["1Vy66IUhT66cMqNsQKc0Jw"], "expires_at": "2020-03-10T16:48:03.000000Z", "issued_at": "2020-03-10T15:48:03.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "34730f4be8c54c569b0059786c6e100a", "name": "admin"}, "is_domain": false, "roles": [{"id": "50c378d402db40e187e2558f75cc5ea5", "name": "member"}, {"id": "35108f06c3814bedadd524f98961c402", "name": "reader"}, {"id": "ec2ea884e3a3413199b04aacdaf1bf4e", "name": "_member_"}, {"id": "e3988e47d7bd499b85adc897575b93a0", "name": "admin"}], "catalog": [{"endpoints": [{"id": "3922473da40c4cc28d580caf892cefe5", "interface": "admin", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:9292", "region": "regionOne"}, {"id": "3d79504528ca4cdea4d9def8128584ae", "interface": "internal", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:9292", "region": "regionOne"}, {"id": "aa4be6ca56014583b573f19d7bfa3414", "interface": "public", "region_id": "regionOne", "url": "https://overcloud.redhat.local:13292", "region": "regionOne"}], "id": "01c96d92cb0a4cf99caf14894a4b6fe2", "type": "image", "name": "glance"}, {"endpoints": [{"id": "3ea1e66c5c38499e8a120383c83d5c82", "interface": "internal", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:8774/v2.1", "region": "regionOne"}, {"id": "70dd2ae2b77b4bb58d58b68fdf22d097", "interface": "public", "region_id": "regionOne", "url": "https://overcloud.redhat.local:13774/v2.1", "region": "regionOne"}, {"id": "75f53cd0dd094341a477d2351dd3a68b", "interface": "admin", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:8774/v2.1", "region": "regionOne"}], "id": "23d7862648774dfaa659f03c891e0d32", "type": "compute", "name": "nova"}, {"endpoints": [{"id": "68b7dfc435674139b7a69a7e82e0f411", "interface": "admin", "region_id": "regionOne", "url": "https://overcloud.ctlplane.redhat.local:35357", "region": "regionOne"}, {"id": "98c96768cf2a49279d41fb4f8fa2b25a", "interface": "public", "region_id": "regionOne", "url": "https://overcloud.redhat.local:13000", "region": "regionOne"}, {"id": "d745eccd33fb42b188d32e91e3bdf608", "interface": "internal", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:5000", "region": "regionOne"}], "id": "34efa591adb741d7b3d96f96599ab0ce", "type": "identity", "name": "keystone"}, {"endpoints": [{"id": "5e395d93f03240e99d0b4f077ed1c275", "interface": "internal", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:8778/placement", "region": "regionOne"}, {"id": "cd48a84fc4c64abba6c658e582f71471", "interface": "public", "region_id": "regionOne", "url": "https://overcloud.redhat.local:13778/placement", "region": "regionOne"}, {"id": "f5d9ae88e0fe4737816850954f5ee07e", "interface": "admin", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:8778/placement", "region": "regionOne"}], "id": "3602c7ee3242410f9b501a6a81202be0", "type": "placement", "name": "placement"}, {"endpoints": [{"id": "024aeb5a469643a2a1742cac264fc62e", "interface": "internal", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:8004/v1/34730f4be8c54c569b0059786c6e100a", "region": "regionOne"}, {"id": "97b5dd898e8d4652a9683fceb1e17417", "interface": "admin", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:8004/v1/34730f4be8c54c569b0059786c6e100a", "region": "regionOne"}, {"id": "f1cb7a31ee294117b324a8c6f165f9bd", "interface": "public", "region_id": "regionOne", "url": "https://overcloud.redhat.local:13004/v1/34730f4be8c54c569b0059786c6e100a", "region": "regionOne"}], "id": "658a3002a8834bdf8f8a17641a5e64fa", "type": "orchestration", "name": "heat"}, {"endpoints": [{"id": "1e0f347e07954e1daa8cc1d9a1995c32", "interface": "internal", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:8776/v3/34730f4be8c54c569b0059786c6e100a", "region": "regionOne"}, {"id": "2a7ef36db26745648a745d335ed0c7da", "interface": "admin", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:8776/v3/34730f4be8c54c569b0059786c6e100a", "region": "regionOne"}, {"id": "ea467ac38a2f4274aed9f885a60f8f22", "interface": "public", "region_id": "regionOne", "url": "https://overcloud.redhat.local:13776/v3/34730f4be8c54c569b0059786c6e100a", "region": "regionOne"}], "id": "746d540f633f47fdadb42a47a006a53d", "type": "volumev3", "name": "cinderv3"}, {"endpoints": [{"id": "9f766ff00dd946d1b0733f71a14e20fa", "interface": "admin", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:9876", "region": "regionOne"}, {"id": "ab844da69cd440068e4d22e43328549a", "interface": "internal", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:9876", "region": "regionOne"}, {"id": "fc8e322af23040dd84fc49623b767555", "interface": "public", "region_id": "regionOne", "url": "https://overcloud.redhat.local:13876", "region": "regionOne"}], "id": "89873f52593f4febb67a8f4777e1b8fe", "type": "load-balancer", "name": "octavia"}, {"endpoints": [{"id": "4af6105f51fe49be9977733043bcc663", "interface": "public", "region_id": "regionOne", "url": "https://overcloud.redhat.local:13808/v1/AUTH_34730f4be8c54c569b0059786c6e100a", "region": "regionOne"}, {"id": "c6f2836eb8474000ba519374e08c0718", "interface": "internal", "region_id": "regionOne", "url": "https://overcloud.storage.redhat.local:8080/v1/AUTH_34730f4be8c54c569b0059786c6e100a", "region": "regionOne"}, {"id": "dfe95e784b10461eb54e5a559d9d8519", "interface": "admin", "region_id": "regionOne", "url": "https://overcloud.storage.redhat.local:8080", "region": "regionOne"}], "id": "8c42aaaaeea240329068d2747a241c7a", "type": "object-store", "name": "swift"}, {"endpoints": [{"id": "42b7c7d88a2e43e38daec13674d2d3e4", "interface": "admin", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:8000/v1", "region": "regionOne"}, {"id": "c483eb4c24d44163944f961048d15709", "interface": "public", "region_id": "regionOne", "url": "https://overcloud.redhat.local:13005/v1", "region": "regionOne"}, {"id": "dbd00c1c249b4920be9ac149f7f5f700", "interface": "internal", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:8000/v1", "region": "regionOne"}], "id": "9fe809aa90af4b4090f5b6487e837369", "type": "cloudformation", "name": "heat-cfn"}, {"endpoints": [{"id": "052bca56ad9d46eeaa8992d9a11381de", "interface": "admin", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:9696", "region": "regionOne"}, {"id": "d16ca77062fa4884ab35e20b1e5a85b6", "interface": "internal", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:9696", "region": "regionOne"}, {"id": "fafe23c9370c471dbaaca291d9c80a34", "interface": "public", "region_id": "regionOne", "url": "https://overcloud.redhat.local:13696", "region": "regionOne"}], "id": "a0626844d0aa434d8212e92977f51dc8", "type": "network", "name": "neutron"}, {"endpoints": [{"id": "51fe869d894b4d12b0e6547a360b3a15", "interface": "internal", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:8776/v2/34730f4be8c54c569b0059786c6e100a", "region": "regionOne"}, {"id": "da99aa211700472fae4404709d89574c", "interface": "public", "region_id": "regionOne", "url": "https://overcloud.redhat.local:13776/v2/34730f4be8c54c569b0059786c6e100a", "region": "regionOne"}, {"id": "f0722b8a002d4a84825aacf180949c51", "interface": "admin", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:8776/v2/34730f4be8c54c569b0059786c6e100a", "region": "regionOne"}], "id": "e565d5609fe349479e748c62d532b882", "type": "volumev2", "name": "cinderv2"}]}}
REQ: curl -g -i -X GET https://overcloud.redhat.local:13696/v2.0/subnets -H "Accept: application/json" -H "User-Agent: python-neutronclient" -H "X-Auth-Token: {SHA256}9900ae57cf1e909b51f08557dbfc139db365b2df1ff920e1cb6b2ba5f279a6b3"
Starting new HTTPS connection (1): overcloud.redhat.local:13696
https://overcloud.redhat.local:13696 "GET /v2.0/subnets HTTP/1.1" 200 585
RESP: [200] Content-Encoding: gzip Content-Length: 585 Content-Type: application/json Date: Tue, 10 Mar 2020 15:48:03 GMT Server: Apache Vary: Accept-Encoding Via: 1.1 controller-3.internalapi.redhat.local:13696 X-Openstack-Request-Id: req-976edbb6-b171-4744-9f2b-7057c230e2d6
RESP BODY: {"subnets":[{"id":"56b9356d-d4fd-4044-989c-e50bff08ad99","name":"lb-mgmt-subnet","tenant_id":"34730f4be8c54c569b0059786c6e100a","network_id":"01d4d6c6-a1f4-4983-aa8b-b0e24ccbdfd8","ip_version":4,"subnetpool_id":null,"enable_dhcp":true,"ipv6_ra_mode":null,"ipv6_address_mode":null,"gateway_ip":"172.24.0.1","cidr":"172.24.0.0/16","allocation_pools":[{"start":"172.24.0.2","end":"172.24.255.254"}],"host_routes":[],"dns_nameservers":[],"description":"","service_types":[],"segment_id":null,"tags":[],"created_at":"2020-03-10T09:30:04Z","updated_at":"2020-03-10T09:30:04Z","revision_number":0,"project_id":"34730f4be8c54c569b0059786c6e100a"},{"id":"8ff75637-9476-4b04-87e3-e4b6381dd66c","name":"floating-200-subnet","tenant_id":"34730f4be8c54c569b0059786c6e100a","network_id":"56be27f2-4e30-4544-ba68-1ddf9713093f","ip_version":4,"subnetpool_id":null,"enable_dhcp":true,"ipv6_ra_mode":null,"ipv6_address_mode":null,"gateway_ip":"172.16.200.250","cidr":"172.16.200.0/24","allocation_pools":[{"start":"172.16.200.151","end":"172.16.200.179"}],"host_routes":[],"dns_nameservers":[],"description":"","service_types":[],"segment_id":null,"tags":[],"created_at":"2020-03-10T10:14:53Z","updated_at":"2020-03-10T10:14:53Z","revision_number":0,"project_id":"34730f4be8c54c569b0059786c6e100a"},{"id":"aa3d1ad1-f75c-4de2-adcc-b7971e5ca8ed","name":"private-1-subnet-1","tenant_id":"34730f4be8c54c569b0059786c6e100a","network_id":"d2263e5c-9f6f-4356-81a7-ba292500abc7","ip_version":4,"subnetpool_id":null,"enable_dhcp":true,"ipv6_ra_mode":null,"ipv6_address_mode":null,"gateway_ip":"192.168.9.1","cidr":"192.168.9.0/24","allocation_pools":[{"start":"192.168.9.2","end":"192.168.9.254"}],"host_routes":[],"dns_nameservers":[],"description":"","service_types":[],"segment_id":null,"tags":[],"created_at":"2020-03-10T10:14:23Z","updated_at":"2020-03-10T10:14:23Z","revision_number":0,"project_id":"34730f4be8c54c569b0059786c6e100a"}]}
GET call to network for https://overcloud.redhat.local:13696/v2.0/subnets used request id req-976edbb6-b171-4744-9f2b-7057c230e2d6
REQ: curl -g -i -X POST https://overcloud.redhat.local:13876/v2.0/lbaas/loadbalancers -H "Accept: application/json" -H "Content-Type: application/json" -H "User-Agent: openstacksdk/0.36.0 keystoneauth1/3.17.1 python-requests/2.20.0 CPython/3.6.8" -H "X-Auth-Token: {SHA256}9900ae57cf1e909b51f08557dbfc139db365b2df1ff920e1cb6b2ba5f279a6b3" -d '{"loadbalancer": {"name": "lb1", "vip_subnet_id": "aa3d1ad1-f75c-4de2-adcc-b7971e5ca8ed", "admin_state_up": true}}'
Starting new HTTPS connection (1): overcloud.redhat.local:13876
https://overcloud.redhat.local:13876 "POST /v2.0/lbaas/loadbalancers HTTP/1.1" 503 None
RESP: [503] Cache-Control: no-cache Connection: close Content-Type: text/html
RESP BODY: Omitted, Content-Type is set to text/html. Only application/json responses have their bodies logged.
Request returned failure status: 503
Expecting value: line 1 column 1 (char 0)
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/octaviaclient/api/v2/octavia.py", line 31, in wrapper
    response = func(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/octaviaclient/api/v2/octavia.py", line 113, in load_balancer_create
    response = self._create(url, **params)
  File "/usr/lib/python3.6/site-packages/osc_lib/api/api.py", line 164, in create
    ret = self._request(method, url, session=session, **params)
  File "/usr/lib/python3.6/site-packages/osc_lib/api/api.py", line 141, in _request
    return session.request(url, method, **kwargs)
  File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 943, in request
    raise exceptions.from_response(resp, method, url)
keystoneauth1.exceptions.http.ServiceUnavailable: Service Unavailable (HTTP 503)
 
During handling of the above exception, another exception occurred:
 
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/cliff/app.py", line 401, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python3.6/site-packages/cliff/display.py", line 116, in run
    column_names, data = self.take_action(parsed_args)
  File "/usr/lib/python3.6/site-packages/octaviaclient/osc/v2/load_balancer.py", line 132, in take_action
    json=body)
  File "/usr/lib/python3.6/site-packages/octaviaclient/api/v2/octavia.py", line 35, in wrapper
    message = e.response.json().get(
  File "/usr/lib/python3.6/site-packages/requests/models.py", line 897, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/lib64/python3.6/site-packages/simplejson/__init__.py", line 518, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python3.6/site-packages/simplejson/decoder.py", line 370, in decode
    obj, end = self.raw_decode(s)
  File "/usr/lib64/python3.6/site-packages/simplejson/decoder.py", line 400, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())
simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
clean_up CreateLoadBalancer: Expecting value: line 1 column 1 (char 0)
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/octaviaclient/api/v2/octavia.py", line 31, in wrapper
    response = func(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/octaviaclient/api/v2/octavia.py", line 113, in load_balancer_create
    response = self._create(url, **params)
  File "/usr/lib/python3.6/site-packages/osc_lib/api/api.py", line 164, in create
    ret = self._request(method, url, session=session, **params)
  File "/usr/lib/python3.6/site-packages/osc_lib/api/api.py", line 141, in _request
    return session.request(url, method, **kwargs)
  File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 943, in request
    raise exceptions.from_response(resp, method, url)
keystoneauth1.exceptions.http.ServiceUnavailable: Service Unavailable (HTTP 503)
 
During handling of the above exception, another exception occurred:
 
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/osc_lib/shell.py", line 136, in run
    ret_val = super(OpenStackShell, self).run(argv)
  File "/usr/lib/python3.6/site-packages/cliff/app.py", line 281, in run
    result = self.run_subcommand(remainder)
  File "/usr/lib/python3.6/site-packages/osc_lib/shell.py", line 176, in run_subcommand
    ret_value = super(OpenStackShell, self).run_subcommand(argv)
  File "/usr/lib/python3.6/site-packages/cliff/app.py", line 401, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python3.6/site-packages/cliff/display.py", line 116, in run
    column_names, data = self.take_action(parsed_args)
  File "/usr/lib/python3.6/site-packages/octaviaclient/osc/v2/load_balancer.py", line 132, in take_action
    json=body)
  File "/usr/lib/python3.6/site-packages/octaviaclient/api/v2/octavia.py", line 35, in wrapper
    message = e.response.json().get(
  File "/usr/lib/python3.6/site-packages/requests/models.py", line 897, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/lib64/python3.6/site-packages/simplejson/__init__.py", line 518, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python3.6/site-packages/simplejson/decoder.py", line 370, in decode
    obj, end = self.raw_decode(s)
  File "/usr/lib64/python3.6/site-packages/simplejson/decoder.py", line 400, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())
simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
 
END return value: 1
 
in keystone.log:
 
2020-03-10 09:24:06.122 28 WARNING keystone.server.flask.application [req-292489a1-523d-4d1f-b52c-964c4ba224fc - - - - -] Could not find user: octavia.: keystone.exception.UserNotFound: Could not find user: octavia.
Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Michael Johnson 2020-03-12 05:04:40 UTC

I think this is worded incorrectly. Octavia supports TLS everywhere, it is the OVN Octavia provider driver that was installed in Octavia does not support TLS everywhere.
If you uninstall the OVN provider driver for Octavia you will be able to create load balancers.

The key log message is:
2020-03-10 11:04:02.924 24 INFO ovsdbapp.backend.ovs_idl.idlutils [-] Could not retrieve schema from tcp:172.20.0.104:6641: Unknown error -1

Where the OVN driver is stuck in an endless loop attempting to connect to tcp:172.20.0.104:6641. My guess is port 6641 is TLS enabled, but the OVN driver does not support TLS.

Comment 2 Carlos Goncalves 2020-03-12 08:07:43 UTC

This is a misconfiguration in the OVN provider driver and should be addressed in TripleO.
I am reassigning to the OVN team as I believe they have started working on a fix.

Comment 5 Jakub Libosvar 2020-03-17 14:14:06 UTC


*** This bug has been marked as a duplicate of bug 1797670 ***

Comment 6 Red Hat Bugzilla 2023-09-14 05:54:14 UTC

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days