Bug 1369881
| Summary: | Asynchronous API traffic on compute node originating from ceilometer-polling through nova client | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Alex Krzos <akrzos> |
| Component: | openstack-ceilometer | Assignee: | Adam Young <ayoung> |
| Status: | CLOSED DUPLICATE | QA Contact: | Yurii Prokulevych <yprokule> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 10.0 (Newton) | CC: | ayoung, jruzicka, nkinder, rcritten, srevivo |
| Target Milestone: | --- | ||
| Target Release: | 10.0 (Newton) | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2016-09-22 19:51:30 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
*** This bug has been marked as a duplicate of bug 1311165 *** |
Description of problem: Deployed via OSPd an OSP10 overcloud and found that ceilometer-polling is attempting to get a token over the external network. According to the stack trace this looks like nova client is attempting to get the actual token. The external network in this deployment is a private vlan network to save actual lab public external network addresses. I understand I can add this network via my nic-configs and current network setup, however this is still a bug that communication is asynchronous meaning that some traffic flows over the InternalApiNetwork and this token is being requested over the ExternalNetwork. My deployed network-environment.yaml: parameter_defaults: NeutronExternalNetworkBridge: "''" InternalApiNetCidr: 172.16.0.0/24 TenantNetCidr: 172.17.0.0/24 StorageNetCidr: 172.18.0.0/24 StorageMgmtNetCidr: 172.19.0.0/24 ManagementNetCidr: 172.20.0.0/24 ExternalNetCidr: 172.21.0.0/24 InternalApiAllocationPools: [{'start': '172.16.0.10', 'end': '172.16.0.200'}] TenantAllocationPools: [{'start': '172.17.0.10', 'end': '172.17.0.200'}] StorageAllocationPools: [{'start': '172.18.0.10', 'end': '172.18.0.200'}] StorageMgmtAllocationPools: [{'start': '172.19.0.10', 'end': '172.19.0.200'}] ManagementAllocationPools: [{'start': '172.20.0.10', 'end': '172.20.0.200'}] # Leave room for floating IPs in the External allocation pool ExternalAllocationPools: [{'start': '172.21.0.100', 'end': '172.21.0.200'}] # Set to the router gateway on the external network ExternalInterfaceDefaultRoute: 172.21.0.1 # Gateway router for the provisioning network (or Undercloud IP) ControlPlaneDefaultRoute: 192.0.2.1 # The IP address of the EC2 metadata server. Generally the IP of the Undercloud EC2MetadataIp: 192.0.2.1 # Define the DNS servers (maximum 2) for the overcloud nodes DnsServers: ["x.x.x.x"] InternalApiNetworkVlanID: 301 StorageNetworkVlanID: 302 StorageMgmtNetworkVlanID: 303 TenantNetworkVlanID: 304 ManagementNetworkVlanID: 305 ExternalNetworkVlanID: 10 Inside the compute node's ceilometer.conf, the keystone url is correctly configured for my InternalApiNetwork: [root@overcloud-novacompute-0 ceilometer]# grep "172.16" /etc/ceilometer/ceilometer.conf rabbit_hosts=172.16.0.26,172.16.0.29,172.16.0.17 auth_url=http://172.16.0.13:5000 However in the log file we can see the attempt to get a token over the ExternalNetwork /var/log/ceilometer/compute.log: 2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client [-] Unable to establish connection to http://172.21.0.100:5000/v3/auth/tokens Version-Release number of selected component (if applicable): OSPd 10 How reproducible: Reproduced with this hardware and version. It is unclear how long this issue has been occurring since I have not always used a "private" ExternalNetwork Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: Stack Trace in log file: 2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client [-] Unable to establish connection to http://172.21.0.100:5000/v3/auth/tokens 2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client Traceback (most recent call last): 2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client File "/usr/lib/python2.7/site-packages/ceilometer/nova_client.py", line 54, in with_logging 2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client return func(*args, **kwargs) 2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client File "/usr/lib/python2.7/site-packages/ceilometer/nova_client.py", line 163, in instance_get_all_by_host 2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client search_opts=search_opts)) 2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 830, in list 2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client "servers") 2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 249, in _list 2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client resp, body = self.api.client.get(url) 2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 187, in get 2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client return self.request(url, 'GET', **kwargs) 2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 107, in request 2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client **kwargs) 2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 344, in request 2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client resp = super(LegacyJsonAdapter, self).request(*args, **kwargs) 2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 112, in request 2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client return self.session.request(url, method, **kwargs) 2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client File "/usr/lib/python2.7/site-packages/positional/__init__.py", line 94, in inner 2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client return func(*args, **kwargs) 2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 459, in request 2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client auth_headers = self.get_auth_headers(auth) 2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 724, in get_auth_headers 2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client return auth.get_headers(self, **kwargs) 2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client File "/usr/lib/python2.7/site-packages/keystoneauth1/plugin.py", line 90, in get_headers 2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client token = self.get_token(session) 2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 89, in get_token 2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client return self.get_access(session).auth_token 2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 135, in get_access 2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client self.auth_ref = self.get_auth_ref(session) 2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 181, in get_auth_ref 2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client return self._plugin.get_auth_ref(session, **kwargs) 2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/v3/base.py", line 167, in get_auth_ref 2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client authenticated=False, log=False, **rkwargs) 2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 672, in post 2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client return self.request(url, 'POST', **kwargs) 2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client File "/usr/lib/python2.7/site-packages/positional/__init__.py", line 94, in inner 2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client return func(*args, **kwargs) 2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 552, in request 2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client resp = send(**kwargs) 2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 596, in _send_request 2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client raise exceptions.ConnectFailure(msg) 2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client ConnectFailure: Unable to establish connection to http://172.21.0.100:5000/v3/auth/tokens 2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client 2016-08-23 20:30:25.842 31268 INFO ceilometer.agent.manager [-] Skip pollster disk.write.requests.rate, no resources found this cycle 2016-08-23 20:30:25.843 31268 INFO ceilometer.agent.manager [-] Skip pollster instance, no resources found this cycle 2016-08-23 20:30:25.844 31268 INFO ceilometer.agent.manager [-] Skip pollster disk.device.write.requests, no resources found this cycle It is unclear if this bug is with ceilometer-polling or the nova client that is referenced in the stacktrace.