Description of problem:
Deployed via OSPd an OSP10 overcloud and found that ceilometer-polling is attempting to get a token over the external network.  According to the stack trace this looks like nova client is attempting to get the actual token.  The external network in this deployment is a private vlan network to save actual lab public external network addresses.  I understand I can add this network via my nic-configs and current network setup, however this is still a bug that communication is asynchronous meaning that some traffic flows over the InternalApiNetwork and this token is being requested over the ExternalNetwork.

My deployed network-environment.yaml:
parameter_defaults:
  NeutronExternalNetworkBridge: "''"
  InternalApiNetCidr: 172.16.0.0/24
  TenantNetCidr: 172.17.0.0/24
  StorageNetCidr: 172.18.0.0/24
  StorageMgmtNetCidr: 172.19.0.0/24
  ManagementNetCidr: 172.20.0.0/24
  ExternalNetCidr: 172.21.0.0/24
  InternalApiAllocationPools: [{'start': '172.16.0.10', 'end': '172.16.0.200'}]
  TenantAllocationPools: [{'start': '172.17.0.10', 'end': '172.17.0.200'}]
  StorageAllocationPools: [{'start': '172.18.0.10', 'end': '172.18.0.200'}]
  StorageMgmtAllocationPools: [{'start': '172.19.0.10', 'end': '172.19.0.200'}]
  ManagementAllocationPools: [{'start': '172.20.0.10', 'end': '172.20.0.200'}]
  # Leave room for floating IPs in the External allocation pool
  ExternalAllocationPools: [{'start': '172.21.0.100', 'end': '172.21.0.200'}]
  # Set to the router gateway on the external network
  ExternalInterfaceDefaultRoute: 172.21.0.1
  # Gateway router for the provisioning network (or Undercloud IP)
  ControlPlaneDefaultRoute: 192.0.2.1
  # The IP address of the EC2 metadata server. Generally the IP of the Undercloud
  EC2MetadataIp: 192.0.2.1
  # Define the DNS servers (maximum 2) for the overcloud nodes
  DnsServers: ["x.x.x.x"]
  InternalApiNetworkVlanID: 301
  StorageNetworkVlanID: 302
  StorageMgmtNetworkVlanID: 303
  TenantNetworkVlanID: 304
  ManagementNetworkVlanID: 305
  ExternalNetworkVlanID: 10

Inside the compute node's ceilometer.conf, the keystone url is correctly configured for my InternalApiNetwork:

[root@overcloud-novacompute-0 ceilometer]# grep "172.16" /etc/ceilometer/ceilometer.conf                                                                                                                                                  
rabbit_hosts=172.16.0.26,172.16.0.29,172.16.0.17
auth_url=http://172.16.0.13:5000

However in the log file we can see the attempt to get a token over the ExternalNetwork

/var/log/ceilometer/compute.log:
2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client [-] Unable to establish connection to http://172.21.0.100:5000/v3/auth/tokens

Version-Release number of selected component (if applicable):
OSPd 10


How reproducible:
Reproduced with this hardware and version.  It is unclear how long this issue has been occurring since I have not always used a "private" ExternalNetwork

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Stack Trace in log file:
2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client [-] Unable to establish connection to http://172.21.0.100:5000/v3/auth/tokens
2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client Traceback (most recent call last):
2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/ceilometer/nova_client.py", line 54, in with_logging
2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client     return func(*args, **kwargs)
2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/ceilometer/nova_client.py", line 163, in instance_get_all_by_host
2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client     search_opts=search_opts))
2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 830, in list
2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client     "servers")
2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 249, in _list
2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client     resp, body = self.api.client.get(url)
2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 187, in get
2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client     return self.request(url, 'GET', **kwargs)
2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 107, in request
2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client     **kwargs)
2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 344, in request
2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client     resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 112, in request
2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client     return self.session.request(url, method, **kwargs)
2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/positional/__init__.py", line 94, in inner
2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client     return func(*args, **kwargs)
2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 459, in request
2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client     auth_headers = self.get_auth_headers(auth)
2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 724, in get_auth_headers
2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client     return auth.get_headers(self, **kwargs)
2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/keystoneauth1/plugin.py", line 90, in get_headers
2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client     token = self.get_token(session)
2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 89, in get_token
2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client     return self.get_access(session).auth_token
2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 135, in get_access
2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client     self.auth_ref = self.get_auth_ref(session)
2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 181, in get_auth_ref
2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client     return self._plugin.get_auth_ref(session, **kwargs)
2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/v3/base.py", line 167, in get_auth_ref
2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client     authenticated=False, log=False, **rkwargs)
2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 672, in post
2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client     return self.request(url, 'POST', **kwargs)
2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/positional/__init__.py", line 94, in inner
2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client     return func(*args, **kwargs)
2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 552, in request
2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client     resp = send(**kwargs)
2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 596, in _send_request
2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client     raise exceptions.ConnectFailure(msg)
2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client ConnectFailure: Unable to establish connection to http://172.21.0.100:5000/v3/auth/tokens
2016-08-23 20:30:25.803 31268 ERROR ceilometer.nova_client
2016-08-23 20:30:25.842 31268 INFO ceilometer.agent.manager [-] Skip pollster disk.write.requests.rate, no resources found this cycle
2016-08-23 20:30:25.843 31268 INFO ceilometer.agent.manager [-] Skip pollster instance, no resources found this cycle
2016-08-23 20:30:25.844 31268 INFO ceilometer.agent.manager [-] Skip pollster disk.device.write.requests, no resources found this cycle

It is unclear if this bug is with ceilometer-polling or the nova client that is referenced in the stacktrace.