Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1775280

Summary:	[OSP16][TLS] Environment with TLS enabled is not able to access openstack api
Product:	Red Hat OpenStack	Reporter:	Roman Safronov <rsafrono>
Component:	openstack-tripleo-common	Assignee:	Adriano Petrich <apetrich>
Status:	CLOSED DUPLICATE	QA Contact:	Alexander Chuzhoy <sasha>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	16.0 (Train)	CC:	mburns, michele, slinaber
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-11-21 18:51:03 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Roman Safronov 2019-11-21 16:22:19 UTC

Description of problem:
Impossible to use openstack deployment with TLS enabled.
On attempt to access openstack API HttpException: 503: Server Error
 returned
It is possible to login nodes.


Version-Release number of selected component (if applicable):
16.0-RHEL-8/RHOS_TRUNK-16.0-RHEL-8-20191115.n.0
(overcloud) [stack@undercloud-0 ~]$ rpm -qa | grep tripleo
openstack-tripleo-puppet-elements-11.2.1-0.20191108131052.2ad3189.el8ost.noarch
python3-tripleoclient-12.3.1-0.20191114050523.bc6d750.el8ost.noarch
python3-tripleo-common-11.3.2-0.20191113011325.a0c7845.el8ost.noarch
openstack-tripleo-common-11.3.2-0.20191113011325.a0c7845.el8ost.noarch
puppet-tripleo-11.3.1-0.20191031115923.7c9561b.el8ost.noarch
openstack-tripleo-image-elements-10.6.1-0.20191022065313.7338463.el8ost.noarch
openstack-tripleo-validations-11.3.1-0.20191104163426.7595cf2.el8ost.noarch
openstack-tripleo-heat-templates-11.3.1-0.20191114050516.ed1cfed.el8ost.noarch
ansible-tripleo-ipsec-9.2.0-0.20191022054642.ffe104c.el8ost.noarch
ansible-role-tripleo-modify-image-1.1.1-0.20191030113421.66a92a4.el8ost.noarch
tripleo-ansible-0.4.1-0.20191114131847.0c76b58.el8ost.noarch
python3-tripleoclient-heat-installer-12.3.1-0.20191114050523.bc6d750.el8ost.noarch
openstack-tripleo-common-containers-11.3.2-0.20191113011325.a0c7845.el8ost.noarch


How reproducible:
100% 

Steps to Reproduce:
1.Deploy environment with TLS enabled, you can use this job https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DFG/view/network/view/networking-ovn/job/DFG-network-networking-ovn-16_director-rhel-virthost-3cont_2comp_1ipa-ipv4-geneve-tls/
2. Try to use openstack api e.g. try to list networks

Actual results:
Command fails
(overcloud) [stack@undercloud-0 ~]$ openstack network list
HttpException: 503: Server Error for url: https://overcloud.redhat.local:13696/v2.0/networks, No server is available to handle this request.: 503 Service Unavailable

Expected results:
Command succeeds

Additional info:
overcloud) [stack@undercloud-0 ~]$ cat overcloudrc
# Clear any old environment that may conflict.
for key in $( set | awk '{FS="="}  /^OS_/ {print $1}' ); do unset $key ; done
export NOVA_VERSION=1.1
export COMPUTE_API_VERSION=1.1
export OS_USERNAME=admin
export OS_PROJECT_NAME=admin
export OS_USER_DOMAIN_NAME=Default
export OS_PROJECT_DOMAIN_NAME=Default
export OS_NO_CACHE=True
export OS_CLOUDNAME=overcloud
export no_proxy=,overcloud.redhat.local,overcloud.ctlplane.redhat.local
export PYTHONWARNINGS="ignore:Certificate has no, ignore:A true SSLContext object is not available"
export OS_AUTH_TYPE=password
export OS_PASSWORD=IG1iriLpwTDxxlRuC8SPIG1wC
export OS_AUTH_URL=https://overcloud.redhat.local:13000
export OS_IDENTITY_API_VERSION=3
export OS_COMPUTE_API_VERSION=2.latest
export OS_IMAGE_API_VERSION=2
export OS_VOLUME_API_VERSION=3
export OS_REGION_NAME=regionOne

# Add OS_CLOUDNAME to PS1
if [ -z "${CLOUDPROMPT_ENABLED:-}" ]; then
    export PS1=${PS1:-""}
    export PS1=\${OS_CLOUDNAME:+"(\$OS_CLOUDNAME)"}\ $PS1
    export CLOUDPROMPT_ENABLED=1
fi



(overcloud) [stack@undercloud-0 ~]$ cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

10.0.0.52 freeipa-0.redhat.local freeipa-0

# HEAT_HOSTS_START - Do not edit manually within this section!

192.168.24.1 undercloud-0.redhat.local undercloud-0
192.168.24.1 undercloud-0.external.redhat.local undercloud-0.external
192.168.24.1 undercloud-0.ctlplane.redhat.local undercloud-0.ctlplane

# HEAT_HOSTS_END

10.35.64.93  rhos-qe-mirror-tlv.usersys.redhat.com download.lab.bos.redhat.com download.eng.bos.redhat.com download-node-02.eng.bos.redhat.com
### BEGIN Overcloud Entries
192.168.24.1    undercloud-0.ctlplane.redhat.local undercloud-0.ctlplane
192.168.24.101  overcloud.ctlplane.redhat.local
172.17.3.101  overcloud.storage.redhat.local
172.17.4.101  overcloud.storagemgmt.redhat.local
172.17.1.101  overcloud.internalapi.redhat.local
10.0.0.101  overcloud.redhat.local
172.17.1.83 controller-0.redhat.local controller-0
172.17.3.55 controller-0.storage.redhat.local controller-0.storage
172.17.4.85 controller-0.storagemgmt.redhat.local controller-0.storagemgmt
172.17.1.83 controller-0.internalapi.redhat.local controller-0.internalapi
172.17.2.102 controller-0.tenant.redhat.local controller-0.tenant
10.0.0.103 controller-0.external.redhat.local controller-0.external
192.168.24.35 controller-0.ctlplane.redhat.local controller-0.ctlplane
172.17.1.133 controller-1.redhat.local controller-1
172.17.3.64 controller-1.storage.redhat.local controller-1.storage
172.17.4.80 controller-1.storagemgmt.redhat.local controller-1.storagemgmt
172.17.1.133 controller-1.internalapi.redhat.local controller-1.internalapi
172.17.2.134 controller-1.tenant.redhat.local controller-1.tenant
10.0.0.105 controller-1.external.redhat.local controller-1.external
192.168.24.15 controller-1.ctlplane.redhat.local controller-1.ctlplane
172.17.1.99 controller-2.redhat.local controller-2
172.17.3.113 controller-2.storage.redhat.local controller-2.storage
172.17.4.115 controller-2.storagemgmt.redhat.local controller-2.storagemgmt
172.17.1.99 controller-2.internalapi.redhat.local controller-2.internalapi
172.17.2.54 controller-2.tenant.redhat.local controller-2.tenant
10.0.0.143 controller-2.external.redhat.local controller-2.external
192.168.24.30 controller-2.ctlplane.redhat.local controller-2.ctlplane

172.17.1.58 compute-0.redhat.local compute-0
172.17.3.91 compute-0.storage.redhat.local compute-0.storage
172.17.1.58 compute-0.internalapi.redhat.local compute-0.internalapi
172.17.2.42 compute-0.tenant.redhat.local compute-0.tenant
192.168.24.16 compute-0.ctlplane.redhat.local compute-0.ctlplane
172.17.1.96 compute-1.redhat.local compute-1
172.17.3.73 compute-1.storage.redhat.local compute-1.storage
172.17.1.96 compute-1.internalapi.redhat.local compute-1.internalapi
172.17.2.39 compute-1.tenant.redhat.local compute-1.tenant
192.168.24.13 compute-1.ctlplane.redhat.local compute-1.ctlplane




192.168.24.101  overcloud.ctlplane.redhat.local
172.17.3.101  overcloud.storage.redhat.local
172.17.4.101  overcloud.storagemgmt.redhat.local
172.17.1.101  overcloud.internalapi.redhat.local
10.0.0.101  overcloud.redhat.local
### END Overcloud Entries






(overcloud) [stack@undercloud-0 ~]$ openstack network list --debug
START with options: network list --debug
options: Namespace(access_token='***', access_token_endpoint='', access_token_type='', aodh_endpoint='', application_credential_id='', application_credential_name='', application_credential_secret='***', auth_methods='', auth_type='password', auth_url='https://overcloud.redhat.local:13000', cacert=None, cert='', client_id='', client_secret='***', cloud='', code='', debug=True, default_domain='default', default_domain_id='', default_domain_name='', deferred_help=False, discovery_endpoint='', domain_id='', domain_name='', endpoint='', identity_provider='', identity_provider_url='', insecure=None, inspector_api_version='1', inspector_url=None, interface='public', key='', log_file=None, openid_scope='', os_alarming_api_version='2', os_baremetal_api_version='1.58', os_beta_command=False, os_compute_api_version='2.latest', os_container_infra_api_version='1', os_data_processing_api_version='1.1', os_data_processing_url='', os_database_api_version='1', os_dns_api_version='2', os_identity_api_version='3', os_image_api_version='2', os_key_manager_api_version='1', os_loadbalancer_api_version='2.0', os_metrics_api_version='1', os_network_api_version='', os_object_api_version='', os_orchestration_api_version='1', os_project_id=None, os_project_name=None, os_queues_api_version='2', os_tripleoclient_api_version='1', os_volume_api_version='3', os_workflow_api_version='2', passcode='', password='***', profile='', project_domain_id='', project_domain_name='Default', project_id='', project_name='admin', protocol='', redirect_uri='', region_name='regionOne', remote_project_domain_id='', remote_project_domain_name='', remote_project_id='', remote_project_name='', roles='', service_provider='', service_provider_endpoint='', service_provider_entity_id='', system_scope='', timing=False, token='***', trust_id='', user='', user_domain_id='', user_domain_name='Default', user_id='', username='admin', verbose_level=3, verify=None)
Auth plugin password selected
auth_config_hook(): {'api_timeout': None, 'verify': True, 'cacert': None, 'cert': None, 'key': None, 'baremetal_status_code_retries': '5', 'baremetal_introspection_status_code_retries': '5', 'image_status_code_retries': '5', 'disable_vendor_agent': {}, 'interface': 'public', 'floating_ip_source': 'neutron', 'image_api_use_tasks': False, 'image_format': 'qcow2', 'message': '', 'network_api_version': '2', 'object_store_api_version': '1', 'secgroup_source': 'neutron', 'status': 'active', 'auth': {'user_domain_name': 'Default', 'project_domain_name': 'Default', 'project_name': 'admin'}, 'verbose_level': 3, 'deferred_help': False, 'debug': True, 'region_name': 'regionOne', 'default_domain': 'default', 'timing': False, 'inspector_api_version': '1', 'auth_url': 'https://overcloud.redhat.local:13000', 'username': 'admin', 'password': '***', 'beta_command': False, 'compute_api_version': '2.latest', 'identity_api_version': '3', 'image_api_version': '2', 'volume_api_version': '3', 'queues_api_version': '2', 'database_api_version': '1', 'tripleoclient_api_version': '1', 'data_processing_api_version': '1.1', 'loadbalancer_api_version': '2.0', 'workflow_api_version': '2', 'container_infra_api_version': '1', 'baremetal_api_version': '1.58', 'orchestration_api_version': '1', 'dns_api_version': '2', 'key_manager_api_version': '1', 'metrics_api_version': '1', 'alarming_api_version': '2', 'auth_type': 'password', 'networks': []}
defaults: {'api_timeout': None, 'verify': True, 'cacert': None, 'cert': None, 'key': None, 'auth_type': 'password', 'baremetal_status_code_retries': 5, 'baremetal_introspection_status_code_retries': 5, 'image_status_code_retries': 5, 'disable_vendor_agent': {}, 'interface': None, 'floating_ip_source': 'neutron', 'image_api_use_tasks': False, 'image_format': 'qcow2', 'message': '', 'network_api_version': '2', 'object_store_api_version': '1', 'secgroup_source': 'neutron', 'status': 'active'}
cloud cfg: {'api_timeout': None, 'verify': True, 'cacert': None, 'cert': None, 'key': None, 'baremetal_status_code_retries': '5', 'baremetal_introspection_status_code_retries': '5', 'image_status_code_retries': '5', 'disable_vendor_agent': {}, 'interface': 'public', 'floating_ip_source': 'neutron', 'image_api_use_tasks': False, 'image_format': 'qcow2', 'message': '', 'network_api_version': '2', 'object_store_api_version': '1', 'secgroup_source': 'neutron', 'status': 'active', 'auth': {'user_domain_name': 'Default', 'project_domain_name': 'Default', 'project_name': 'admin'}, 'verbose_level': 3, 'deferred_help': False, 'debug': True, 'region_name': 'regionOne', 'default_domain': 'default', 'timing': False, 'inspector_api_version': '1', 'auth_url': 'https://overcloud.redhat.local:13000', 'username': 'admin', 'password': '***', 'beta_command': False, 'compute_api_version': '2.latest', 'identity_api_version': '3', 'image_api_version': '2', 'volume_api_version': '3', 'queues_api_version': '2', 'database_api_version': '1', 'tripleoclient_api_version': '1', 'data_processing_api_version': '1.1', 'loadbalancer_api_version': '2.0', 'workflow_api_version': '2', 'container_infra_api_version': '1', 'baremetal_api_version': '1.58', 'orchestration_api_version': '1', 'dns_api_version': '2', 'key_manager_api_version': '1', 'metrics_api_version': '1', 'alarming_api_version': '2', 'auth_type': 'password', 'networks': []}
compute API version 2.latest, cmd group openstack.compute.v2
identity API version 3, cmd group openstack.identity.v3
image API version 2, cmd group openstack.image.v2
network API version 2, cmd group openstack.network.v2
object_store API version 1, cmd group openstack.object_store.v1
volume API version 3, cmd group openstack.volume.v3
messaging API version 2, cmd group openstack.messaging.v2
database API version 1, cmd group openstack.database.v1
tripleoclient API version 1, cmd group openstack.tripleoclient.v1
data_processing API version 1.1, cmd group openstack.data_processing.v1
load_balancer API version 2.0, cmd group openstack.load_balancer.v2
neutronclient API version 2, cmd group openstack.neutronclient.v2
workflow_engine API version 2, cmd group openstack.workflow_engine.v2
container_infra API version 1, cmd group openstack.container_infra.v1
baremetal API version 1.58, cmd group openstack.baremetal.v1
baremetal_introspection API version 1, cmd group openstack.baremetal_introspection.v1
orchestration API version 1, cmd group openstack.orchestration.v1
dns API version 2, cmd group openstack.dns.v2
key_manager API version 1, cmd group openstack.key_manager.v1
metric API version 1, cmd group openstack.metric.v1
alarming API version 2, cmd group openstack.alarming.v2
Auth plugin password selected
auth_config_hook(): {'api_timeout': None, 'verify': True, 'cacert': None, 'cert': None, 'key': None, 'baremetal_status_code_retries': '5', 'baremetal_introspection_status_code_retries': '5', 'image_status_code_retries': '5', 'disable_vendor_agent': {}, 'interface': 'public', 'floating_ip_source': 'neutron', 'image_api_use_tasks': False, 'image_format': 'qcow2', 'message': '', 'network_api_version': '2', 'object_store_api_version': '1', 'secgroup_source': 'neutron', 'status': 'active', 'auth': {'user_domain_name': 'Default', 'project_domain_name': 'Default', 'project_name': 'admin'}, 'verbose_level': 3, 'deferred_help': False, 'debug': True, 'region_name': 'regionOne', 'default_domain': 'default', 'timing': False, 'inspector_api_version': '1', 'auth_url': 'https://overcloud.redhat.local:13000', 'username': 'admin', 'password': '***', 'beta_command': False, 'compute_api_version': '2.latest', 'identity_api_version': '3', 'image_api_version': '2', 'volume_api_version': '3', 'queues_api_version': '2', 'database_api_version': '1', 'tripleoclient_api_version': '1', 'data_processing_api_version': '1.1', 'loadbalancer_api_version': '2.0', 'workflow_api_version': '2', 'container_infra_api_version': '1', 'baremetal_api_version': '1.58', 'orchestration_api_version': '1', 'dns_api_version': '2', 'key_manager_api_version': '1', 'metrics_api_version': '1', 'alarming_api_version': '2', 'auth_type': 'password', 'networks': []}
command: network list -> openstackclient.network.v2.network.ListNetwork (auth=True)
Auth plugin password selected
auth_config_hook(): {'api_timeout': None, 'verify': True, 'cacert': None, 'cert': None, 'key': None, 'baremetal_status_code_retries': '5', 'baremetal_introspection_status_code_retries': '5', 'image_status_code_retries': '5', 'disable_vendor_agent': {}, 'interface': 'public', 'floating_ip_source': 'neutron', 'image_api_use_tasks': False, 'image_format': 'qcow2', 'message': '', 'network_api_version': '2', 'object_store_api_version': '1', 'secgroup_source': 'neutron', 'status': 'active', 'auth': {'user_domain_name': 'Default', 'project_domain_name': 'Default', 'project_name': 'admin'}, 'additional_user_agent': [('osc-lib', '1.14.1')], 'verbose_level': 3, 'deferred_help': False, 'debug': True, 'region_name': 'regionOne', 'default_domain': 'default', 'timing': False, 'inspector_api_version': '1', 'auth_url': 'https://overcloud.redhat.local:13000', 'username': 'admin', 'password': '***', 'beta_command': False, 'compute_api_version': '2.latest', 'identity_api_version': '3', 'image_api_version': '2', 'volume_api_version': '3', 'queues_api_version': '2', 'database_api_version': '1', 'tripleoclient_api_version': '1', 'data_processing_api_version': '1.1', 'loadbalancer_api_version': '2.0', 'workflow_api_version': '2', 'container_infra_api_version': '1', 'baremetal_api_version': '1.58', 'orchestration_api_version': '1', 'dns_api_version': '2', 'key_manager_api_version': '1', 'metrics_api_version': '1', 'alarming_api_version': '2', 'auth_type': 'password', 'networks': []}
Using auth plugin: password
Using parameters {'auth_url': 'https://overcloud.redhat.local:13000', 'project_name': 'admin', 'project_domain_name': 'Default', 'username': 'admin', 'user_domain_name': 'Default', 'password': '***'}
Get auth_ref
REQ: curl -g -i -X GET https://overcloud.redhat.local:13000 -H "Accept: application/json" -H "User-Agent: openstacksdk/0.36.0 keystoneauth1/3.17.1 python-requests/2.20.0 CPython/3.6.8"
Starting new HTTPS connection (1): overcloud.redhat.local:13000
https://overcloud.redhat.local:13000 "GET / HTTP/1.1" 300 279
RESP: [300] Content-Length: 279 Content-Type: application/json Date: Thu, 21 Nov 2019 16:20:29 GMT Location: https://overcloud.redhat.local:13000/v3/ Server: Apache Vary: X-Auth-Token x-openstack-request-id: req-53752ca3-7b4f-458c-894b-64ce01b2e3e2
RESP BODY: {"versions": {"values": [{"id": "v3.13", "status": "stable", "updated": "2019-07-19T00:00:00Z", "links": [{"rel": "self", "href": "https://overcloud.redhat.local:13000/v3/"}], "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}]}]}}
GET call to https://overcloud.redhat.local:13000/ used request id req-53752ca3-7b4f-458c-894b-64ce01b2e3e2
Making authentication request to https://overcloud.redhat.local:13000/v3/auth/tokens
https://overcloud.redhat.local:13000 "POST /v3/auth/tokens HTTP/1.1" 201 8601
{"token": {"methods": ["password"], "user": {"domain": {"id": "default", "name": "Default"}, "id": "c2959876bb144a98b429acaeae559526", "name": "admin", "password_expires_at": null}, "audit_ids": ["rzS9kD8fQnKmkVp2-JTKBg"], "expires_at": "2019-11-21T17:20:30.000000Z", "issued_at": "2019-11-21T16:20:30.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "09d97712e78243ea9c1d084946baf9fa", "name": "admin"}, "is_domain": false, "roles": [{"id": "3938fae2bb2442cc89f3a66b8b53d64a", "name": "_member_"}, {"id": "a32be06c109949a1a05322d98c96f3df", "name": "member"}, {"id": "89fa06950dad4acbb8b5ac87c60b1181", "name": "admin"}, {"id": "a961ec71ed1f46ad90d59da0810c0c30", "name": "reader"}], "catalog": [{"endpoints": [{"id": "a1688fdeb3834ba495fca73adc1de182", "interface": "admin", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:8776/v3/09d97712e78243ea9c1d084946baf9fa", "region": "regionOne"}, {"id": "ce0a0c74388b4e3cb61ae88a9d211cd7", "interface": "internal", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:8776/v3/09d97712e78243ea9c1d084946baf9fa", "region": "regionOne"}, {"id": "f32d5791ebc0490fa6ed4c192f293297", "interface": "public", "region_id": "regionOne", "url": "https://overcloud.redhat.local:13776/v3/09d97712e78243ea9c1d084946baf9fa", "region": "regionOne"}], "id": "01a3ac7873b6454090e6855434ed6509", "type": "volumev3", "name": "cinderv3"}, {"endpoints": [{"id": "5139e6cb653f4f80b82d92732bda1a82", "interface": "internal", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:9696", "region": "regionOne"}, {"id": "ceee58c4de8a43ec8e1edca103200ca4", "interface": "public", "region_id": "regionOne", "url": "https://overcloud.redhat.local:13696", "region": "regionOne"}, {"id": "fe50a106b0054532b7d7e4e06a40df06", "interface": "admin", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:9696", "region": "regionOne"}], "id": "1d170920b6c7497793a51c8f62e93c03", "type": "network", "name": "neutron"}, {"endpoints": [{"id": "0806130d51024b45877525f9ed61b90e", "interface": "internal", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:8776/v2/09d97712e78243ea9c1d084946baf9fa", "region": "regionOne"}, {"id": "6c69af412af444f1a50b1de6bd6c39ec", "interface": "public", "region_id": "regionOne", "url": "https://overcloud.redhat.local:13776/v2/09d97712e78243ea9c1d084946baf9fa", "region": "regionOne"}, {"id": "6f1467409b48454f80afbe89cfb62a33", "interface": "admin", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:8776/v2/09d97712e78243ea9c1d084946baf9fa", "region": "regionOne"}], "id": "238496e611d94a119ece5acf89a4cd16", "type": "volumev2", "name": "cinderv2"}, {"endpoints": [{"id": "329e0949578845ef9a110e3ba63e6c90", "interface": "internal", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:8042", "region": "regionOne"}, {"id": "474a8e9cbd014c818946a007ae0f1544", "interface": "admin", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:8042", "region": "regionOne"}, {"id": "c36fe588b2b54ef08206ae65e4e19b24", "interface": "public", "region_id": "regionOne", "url": "https://overcloud.redhat.local:13042", "region": "regionOne"}], "id": "53324a4e4c434ab2abdbd6d54fd8fc48", "type": "alarming", "name": "aodh"}, {"endpoints": [{"id": "2585fe3238394bd99fd2645f32bdf421", "interface": "internal", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:8000/v1", "region": "regionOne"}, {"id": "9e7d5b2eb5c94f6b886d9701f49ef809", "interface": "admin", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:8000/v1", "region": "regionOne"}, {"id": "e356b2a8beb0449ab62c410c77cf2626", "interface": "public", "region_id": "regionOne", "url": "https://overcloud.redhat.local:13005/v1", "region": "regionOne"}], "id": "6be4364876ee4b4ea3437d4c4f302333", "type": "cloudformation", "name": "heat-cfn"}, {"endpoints": [{"id": "05e245406325435181f4f6c498b0ad22", "interface": "public", "region_id": "regionOne", "url": "https://overcloud.redhat.local:13808/v1/AUTH_09d97712e78243ea9c1d084946baf9fa", "region": "regionOne"}, {"id": "55ba578438a74896bbc37f860c27567d", "interface": "internal", "region_id": "regionOne", "url": "https://overcloud.storage.redhat.local:8080/v1/AUTH_09d97712e78243ea9c1d084946baf9fa", "region": "regionOne"}, {"id": "6a77384e238d442b98a1ade78a6c07c3", "interface": "admin", "region_id": "regionOne", "url": "https://overcloud.storage.redhat.local:8080", "region": "regionOne"}], "id": "7d80a0707b074166a874867673efadd0", "type": "object-store", "name": "swift"}, {"endpoints": [], "id": "7e6b7b48bc3d4af89ac49ab59b70f9a5", "type": "metering", "name": "ceilometer"}, {"endpoints": [{"id": "55c4f3ccdcf44bf3a63d89015af7b5d6", "interface": "public", "region_id": "regionOne", "url": "https://overcloud.redhat.local:13004/v1/09d97712e78243ea9c1d084946baf9fa", "region": "regionOne"}, {"id": "8d1e6031389b469c9d2ef616eb190e55", "interface": "internal", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:8004/v1/09d97712e78243ea9c1d084946baf9fa", "region": "regionOne"}, {"id": "d98bdc7a190f4bf7ba0a2d43d4d53803", "interface": "admin", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:8004/v1/09d97712e78243ea9c1d084946baf9fa", "region": "regionOne"}], "id": "aa6fcbf20e5b419f95053ab62085a036", "type": "orchestration", "name": "heat"}, {"endpoints": [{"id": "2139b8cd06084c32ab993773a2a69095", "interface": "admin", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:8774/v2.1", "region": "regionOne"}, {"id": "d32d189e3242471a934175018201ad8a", "interface": "internal", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:8774/v2.1", "region": "regionOne"}, {"id": "e2f94c03e1da4bb8907f9a8c121123b6", "interface": "public", "region_id": "regionOne", "url": "https://overcloud.redhat.local:13774/v2.1", "region": "regionOne"}], "id": "b688b6e1f3534707953451de24e55cc6", "type": "compute", "name": "nova"}, {"endpoints": [{"id": "50165746e15d4cdc9808fe733d170027", "interface": "admin", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:8041", "region": "regionOne"}, {"id": "5864703c04fd467cbb6a3ba9a9050051", "interface": "internal", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:8041", "region": "regionOne"}, {"id": "6a2007e1199943248fd69167c3e05507", "interface": "public", "region_id": "regionOne", "url": "https://overcloud.redhat.local:13041", "region": "regionOne"}], "id": "d89d092c15dd454ba2adb0cdc93c4f48", "type": "metric", "name": "gnocchi"}, {"endpoints": [{"id": "83c79712170e497ebd6d202ef205a43e", "interface": "internal", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:8778/placement", "region": "regionOne"}, {"id": "93d628e6f73d42d4ad436f639fb9ac35", "interface": "admin", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:8778/placement", "region": "regionOne"}, {"id": "aa5a18f6fdfb4fca843b28979bfe62f9", "interface": "public", "region_id": "regionOne", "url": "https://overcloud.redhat.local:13778/placement", "region": "regionOne"}], "id": "eaf312ef3e3e40f48f5db6910a6de206", "type": "placement", "name": "placement"}, {"endpoints": [{"id": "0ab98cbd5d124b959c96be180e602eb8", "interface": "public", "region_id": "regionOne", "url": "https://overcloud.redhat.local:13292", "region": "regionOne"}, {"id": "2bde0b9875354dd9b5e8f22a364e37c8", "interface": "internal", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:9292", "region": "regionOne"}, {"id": "ff81f485f4c94912aa10ff151050950d", "interface": "admin", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:9292", "region": "regionOne"}], "id": "f0ed95aff42d4036adf0d5d4b64eae59", "type": "image", "name": "glance"}, {"endpoints": [{"id": "1e3584cb12364ae9b68b059c303d255f", "interface": "admin", "region_id": "regionOne", "url": "https://overcloud.ctlplane.redhat.local:35357", "region": "regionOne"}, {"id": "6054efc84fd44816871dad90f1c360c9", "interface": "public", "region_id": "regionOne", "url": "https://overcloud.redhat.local:13000", "region": "regionOne"}, {"id": "b7c4185911c8436183535a8e613f65cd", "interface": "internal", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:5000", "region": "regionOne"}], "id": "f1c9a535d3294cacafc2245ab9b55ea1", "type": "identity", "name": "keystone"}]}}
get_parser(openstack network list)
common parser: ArgumentParser(prog='openstack network list', usage=None, description='List networks', formatter_class=<class 'cliff.command._SmartHelpFormatter'>, conflict_handler='ignore', add_help=True)
network endpoint in service catalog
run(Namespace(agent_id=None, any_tags=None, columns=[], disable=False, enable=False, external=False, fit_width=False, formatter='table', internal=False, long=False, max_width=0, name=None, no_share=False, noindent=False, not_any_tags=None, not_tags=None, physical_network=None, print_empty=False, project=None, project_domain=None, provider_network_type=None, quote_mode='nonnumeric', segmentation_id=None, share=False, sort_columns=[], status=None, tags=None))
network endpoint in service catalog
Making authentication request to https://overcloud.redhat.local:13000/v3/auth/tokens
https://overcloud.redhat.local:13000 "POST /v3/auth/tokens HTTP/1.1" 201 8601
{"token": {"methods": ["password"], "user": {"domain": {"id": "default", "name": "Default"}, "id": "c2959876bb144a98b429acaeae559526", "name": "admin", "password_expires_at": null}, "audit_ids": ["QOcyMAodS0q3zac9QXxtQg"], "expires_at": "2019-11-21T17:20:30.000000Z", "issued_at": "2019-11-21T16:20:30.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "09d97712e78243ea9c1d084946baf9fa", "name": "admin"}, "is_domain": false, "roles": [{"id": "89fa06950dad4acbb8b5ac87c60b1181", "name": "admin"}, {"id": "a32be06c109949a1a05322d98c96f3df", "name": "member"}, {"id": "3938fae2bb2442cc89f3a66b8b53d64a", "name": "_member_"}, {"id": "a961ec71ed1f46ad90d59da0810c0c30", "name": "reader"}], "catalog": [{"endpoints": [{"id": "a1688fdeb3834ba495fca73adc1de182", "interface": "admin", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:8776/v3/09d97712e78243ea9c1d084946baf9fa", "region": "regionOne"}, {"id": "ce0a0c74388b4e3cb61ae88a9d211cd7", "interface": "internal", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:8776/v3/09d97712e78243ea9c1d084946baf9fa", "region": "regionOne"}, {"id": "f32d5791ebc0490fa6ed4c192f293297", "interface": "public", "region_id": "regionOne", "url": "https://overcloud.redhat.local:13776/v3/09d97712e78243ea9c1d084946baf9fa", "region": "regionOne"}], "id": "01a3ac7873b6454090e6855434ed6509", "type": "volumev3", "name": "cinderv3"}, {"endpoints": [{"id": "5139e6cb653f4f80b82d92732bda1a82", "interface": "internal", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:9696", "region": "regionOne"}, {"id": "ceee58c4de8a43ec8e1edca103200ca4", "interface": "public", "region_id": "regionOne", "url": "https://overcloud.redhat.local:13696", "region": "regionOne"}, {"id": "fe50a106b0054532b7d7e4e06a40df06", "interface": "admin", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:9696", "region": "regionOne"}], "id": "1d170920b6c7497793a51c8f62e93c03", "type": "network", "name": "neutron"}, {"endpoints": [{"id": "0806130d51024b45877525f9ed61b90e", "interface": "internal", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:8776/v2/09d97712e78243ea9c1d084946baf9fa", "region": "regionOne"}, {"id": "6c69af412af444f1a50b1de6bd6c39ec", "interface": "public", "region_id": "regionOne", "url": "https://overcloud.redhat.local:13776/v2/09d97712e78243ea9c1d084946baf9fa", "region": "regionOne"}, {"id": "6f1467409b48454f80afbe89cfb62a33", "interface": "admin", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:8776/v2/09d97712e78243ea9c1d084946baf9fa", "region": "regionOne"}], "id": "238496e611d94a119ece5acf89a4cd16", "type": "volumev2", "name": "cinderv2"}, {"endpoints": [{"id": "329e0949578845ef9a110e3ba63e6c90", "interface": "internal", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:8042", "region": "regionOne"}, {"id": "474a8e9cbd014c818946a007ae0f1544", "interface": "admin", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:8042", "region": "regionOne"}, {"id": "c36fe588b2b54ef08206ae65e4e19b24", "interface": "public", "region_id": "regionOne", "url": "https://overcloud.redhat.local:13042", "region": "regionOne"}], "id": "53324a4e4c434ab2abdbd6d54fd8fc48", "type": "alarming", "name": "aodh"}, {"endpoints": [{"id": "2585fe3238394bd99fd2645f32bdf421", "interface": "internal", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:8000/v1", "region": "regionOne"}, {"id": "9e7d5b2eb5c94f6b886d9701f49ef809", "interface": "admin", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:8000/v1", "region": "regionOne"}, {"id": "e356b2a8beb0449ab62c410c77cf2626", "interface": "public", "region_id": "regionOne", "url": "https://overcloud.redhat.local:13005/v1", "region": "regionOne"}], "id": "6be4364876ee4b4ea3437d4c4f302333", "type": "cloudformation", "name": "heat-cfn"}, {"endpoints": [{"id": "05e245406325435181f4f6c498b0ad22", "interface": "public", "region_id": "regionOne", "url": "https://overcloud.redhat.local:13808/v1/AUTH_09d97712e78243ea9c1d084946baf9fa", "region": "regionOne"}, {"id": "55ba578438a74896bbc37f860c27567d", "interface": "internal", "region_id": "regionOne", "url": "https://overcloud.storage.redhat.local:8080/v1/AUTH_09d97712e78243ea9c1d084946baf9fa", "region": "regionOne"}, {"id": "6a77384e238d442b98a1ade78a6c07c3", "interface": "admin", "region_id": "regionOne", "url": "https://overcloud.storage.redhat.local:8080", "region": "regionOne"}], "id": "7d80a0707b074166a874867673efadd0", "type": "object-store", "name": "swift"}, {"endpoints": [], "id": "7e6b7b48bc3d4af89ac49ab59b70f9a5", "type": "metering", "name": "ceilometer"}, {"endpoints": [{"id": "55c4f3ccdcf44bf3a63d89015af7b5d6", "interface": "public", "region_id": "regionOne", "url": "https://overcloud.redhat.local:13004/v1/09d97712e78243ea9c1d084946baf9fa", "region": "regionOne"}, {"id": "8d1e6031389b469c9d2ef616eb190e55", "interface": "internal", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:8004/v1/09d97712e78243ea9c1d084946baf9fa", "region": "regionOne"}, {"id": "d98bdc7a190f4bf7ba0a2d43d4d53803", "interface": "admin", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:8004/v1/09d97712e78243ea9c1d084946baf9fa", "region": "regionOne"}], "id": "aa6fcbf20e5b419f95053ab62085a036", "type": "orchestration", "name": "heat"}, {"endpoints": [{"id": "2139b8cd06084c32ab993773a2a69095", "interface": "admin", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:8774/v2.1", "region": "regionOne"}, {"id": "d32d189e3242471a934175018201ad8a", "interface": "internal", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:8774/v2.1", "region": "regionOne"}, {"id": "e2f94c03e1da4bb8907f9a8c121123b6", "interface": "public", "region_id": "regionOne", "url": "https://overcloud.redhat.local:13774/v2.1", "region": "regionOne"}], "id": "b688b6e1f3534707953451de24e55cc6", "type": "compute", "name": "nova"}, {"endpoints": [{"id": "50165746e15d4cdc9808fe733d170027", "interface": "admin", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:8041", "region": "regionOne"}, {"id": "5864703c04fd467cbb6a3ba9a9050051", "interface": "internal", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:8041", "region": "regionOne"}, {"id": "6a2007e1199943248fd69167c3e05507", "interface": "public", "region_id": "regionOne", "url": "https://overcloud.redhat.local:13041", "region": "regionOne"}], "id": "d89d092c15dd454ba2adb0cdc93c4f48", "type": "metric", "name": "gnocchi"}, {"endpoints": [{"id": "83c79712170e497ebd6d202ef205a43e", "interface": "internal", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:8778/placement", "region": "regionOne"}, {"id": "93d628e6f73d42d4ad436f639fb9ac35", "interface": "admin", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:8778/placement", "region": "regionOne"}, {"id": "aa5a18f6fdfb4fca843b28979bfe62f9", "interface": "public", "region_id": "regionOne", "url": "https://overcloud.redhat.local:13778/placement", "region": "regionOne"}], "id": "eaf312ef3e3e40f48f5db6910a6de206", "type": "placement", "name": "placement"}, {"endpoints": [{"id": "0ab98cbd5d124b959c96be180e602eb8", "interface": "public", "region_id": "regionOne", "url": "https://overcloud.redhat.local:13292", "region": "regionOne"}, {"id": "2bde0b9875354dd9b5e8f22a364e37c8", "interface": "internal", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:9292", "region": "regionOne"}, {"id": "ff81f485f4c94912aa10ff151050950d", "interface": "admin", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:9292", "region": "regionOne"}], "id": "f0ed95aff42d4036adf0d5d4b64eae59", "type": "image", "name": "glance"}, {"endpoints": [{"id": "1e3584cb12364ae9b68b059c303d255f", "interface": "admin", "region_id": "regionOne", "url": "https://overcloud.ctlplane.redhat.local:35357", "region": "regionOne"}, {"id": "6054efc84fd44816871dad90f1c360c9", "interface": "public", "region_id": "regionOne", "url": "https://overcloud.redhat.local:13000", "region": "regionOne"}, {"id": "b7c4185911c8436183535a8e613f65cd", "interface": "internal", "region_id": "regionOne", "url": "https://overcloud.internalapi.redhat.local:5000", "region": "regionOne"}], "id": "f1c9a535d3294cacafc2245ab9b55ea1", "type": "identity", "name": "keystone"}]}}
Network client initialized using OpenStack SDK: <openstack.network.v2._proxy.Proxy object at 0x7f0f2c1609e8>
Instantiating identity client: <class 'keystoneclient.v3.client.Client'>
REQ: curl -g -i -X GET https://overcloud.redhat.local:13696/v2.0/networks -H "Accept: application/json" -H "User-Agent: openstacksdk/0.36.0 keystoneauth1/3.17.1 python-requests/2.20.0 CPython/3.6.8" -H "X-Auth-Token: {SHA256}0f3d6422698fadd61a3e920066a6a5d8afddba1b6ece3e71b675c2e6d41920ee"
Starting new HTTPS connection (1): overcloud.redhat.local:13696
https://overcloud.redhat.local:13696 "GET /v2.0/networks HTTP/1.1" 503 None
RESP: [503] Cache-Control: no-cache Connection: close Content-Type: text/html
RESP BODY: Omitted, Content-Type is set to text/html. Only application/json responses have their bodies logged.
HttpException: 503: Server Error for url: https://overcloud.redhat.local:13696/v2.0/networks, No server is available to handle this request.: 503 Service Unavailable
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/cliff/app.py", line 401, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python3.6/site-packages/cliff/display.py", line 119, in run
    self.produce_output(parsed_args, column_names, data)
  File "/usr/lib/python3.6/site-packages/cliff/lister.py", line 82, in produce_output
    parsed_args,
  File "/usr/lib/python3.6/site-packages/cliff/formatters/table.py", line 101, in emit_list
    self.add_rows(x, column_names, data)
  File "/usr/lib/python3.6/site-packages/cliff/formatters/table.py", line 80, in add_rows
    first_row = next(data_iter)
  File "/usr/lib/python3.6/site-packages/openstackclient/network/v2/network.py", line 576, in <genexpr>
    (utils.get_item_properties(
  File "/usr/lib/python3.6/site-packages/openstack/resource.py", line 1599, in list
    exceptions.raise_from_response(response)
  File "/usr/lib/python3.6/site-packages/openstack/exceptions.py", line 229, in raise_from_response
    http_status=http_status, request_id=request_id
openstack.exceptions.HttpException: HttpException: 503: Server Error for url: https://overcloud.redhat.local:13696/v2.0/networks, No server is available to handle this request.: 503 Service Unavailable
clean_up ListNetwork: HttpException: 503: Server Error for url: https://overcloud.redhat.local:13696/v2.0/networks, No server is available to handle this request.: 503 Service Unavailable
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/osc_lib/shell.py", line 136, in run
    ret_val = super(OpenStackShell, self).run(argv)
  File "/usr/lib/python3.6/site-packages/cliff/app.py", line 281, in run
    result = self.run_subcommand(remainder)
  File "/usr/lib/python3.6/site-packages/osc_lib/shell.py", line 176, in run_subcommand
    ret_value = super(OpenStackShell, self).run_subcommand(argv)
  File "/usr/lib/python3.6/site-packages/cliff/app.py", line 401, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python3.6/site-packages/cliff/display.py", line 119, in run
    self.produce_output(parsed_args, column_names, data)
  File "/usr/lib/python3.6/site-packages/cliff/lister.py", line 82, in produce_output
    parsed_args,
  File "/usr/lib/python3.6/site-packages/cliff/formatters/table.py", line 101, in emit_list
    self.add_rows(x, column_names, data)
  File "/usr/lib/python3.6/site-packages/cliff/formatters/table.py", line 80, in add_rows
    first_row = next(data_iter)
  File "/usr/lib/python3.6/site-packages/openstackclient/network/v2/network.py", line 576, in <genexpr>
    (utils.get_item_properties(
  File "/usr/lib/python3.6/site-packages/openstack/resource.py", line 1599, in list
    exceptions.raise_from_response(response)
  File "/usr/lib/python3.6/site-packages/openstack/exceptions.py", line 229, in raise_from_response
    http_status=http_status, request_id=request_id
openstack.exceptions.HttpException: HttpException: 503: Server Error for url: https://overcloud.redhat.local:13696/v2.0/networks, No server is available to handle this request.: 503 Service Unavailable

END return value: 1

Comment 1 Roman Safronov 2019-11-21 16:35:56 UTC

logs at http://rhos-release.virt.bos.redhat.com/log/bz1775280

Comment 2 Michele Baldessari 2019-11-21 18:51:03 UTC


*** This bug has been marked as a duplicate of bug 1771807 ***