1402562 – Ceilometer compute service connection error with nova client

Bug 1402562 - Ceilometer compute service connection error with nova client

Summary: Ceilometer compute service connection error with nova client

Keywords:
Status:	CLOSED WORKSFORME
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-ceilometer
Sub Component:
Version:	8.0 (Liberty)
Hardware:	Unspecified
OS:	Linux
Priority:	unspecified
Severity:	high
Target Milestone:	---
Target Release:	---
Assignee:	Eoghan Glynn
QA Contact:	Yurii Prokulevych
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2016-12-07 20:29 UTC by chenruoyu111
Modified:	2017-01-06 14:13 UTC (History)
CC List:	5 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2017-01-06 14:13:40 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Description chenruoyu111 2016-12-07 20:29:40 UTC

User-Agent:       Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36
Build Identifier: 

When we do "ceilometer meter-list", we don't get meters such as 'cpu', 'cpu_util'. The ceilometer compute agent cannot connect to other services like nova to poll usage data. The logs are as follows:

2016-12-07 15:24:25.875 28248 ERROR ceilometer.nova_client     search_opts=search_opts))
2016-12-07 15:24:25.875 28248 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 623, in list
2016-12-07 15:24:25.875 28248 ERROR ceilometer.nova_client     "servers")
2016-12-07 15:24:25.875 28248 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 74, in _list
2016-12-07 15:24:25.875 28248 ERROR ceilometer.nova_client     _resp, body = self.api.client.get(url)
2016-12-07 15:24:25.875 28248 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 448, in get
2016-12-07 15:24:25.875 28248 ERROR ceilometer.nova_client     return self._cs_request(url, 'GET', **kwargs)
2016-12-07 15:24:25.875 28248 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 404, in _cs_request
2016-12-07 15:24:25.875 28248 ERROR ceilometer.nova_client     self.authenticate()
2016-12-07 15:24:25.875 28248 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 561, in authenticate
2016-12-07 15:24:25.875 28248 ERROR ceilometer.nova_client     auth_url = self._v2_auth(auth_url)
2016-12-07 15:24:25.875 28248 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 652, in _v2_auth
2016-12-07 15:24:25.875 28248 ERROR ceilometer.nova_client     return self._authenticate(url, body)
2016-12-07 15:24:25.875 28248 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 665, in _authenticate
2016-12-07 15:24:25.875 28248 ERROR ceilometer.nova_client     **kwargs)
2016-12-07 15:24:25.875 28248 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 399, in _time_request
2016-12-07 15:24:25.875 28248 ERROR ceilometer.nova_client     resp, body = self.request(url, method, **kwargs)
2016-12-07 15:24:25.875 28248 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 366, in request
2016-12-07 15:24:25.875 28248 ERROR ceilometer.nova_client     **kwargs)
2016-12-07 15:24:25.875 28248 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/requests/api.py", line 50, in request
2016-12-07 15:24:25.875 28248 ERROR ceilometer.nova_client     response = session.request(method=method, url=url, **kwargs)
2016-12-07 15:24:25.875 28248 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 465, in request
2016-12-07 15:24:25.875 28248 ERROR ceilometer.nova_client     resp = self.send(prep, **send_kwargs)
2016-12-07 15:24:25.875 28248 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
2016-12-07 15:24:25.875 28248 ERROR ceilometer.nova_client     r = adapter.send(request, **kwargs)
2016-12-07 15:24:25.875 28248 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 415, in send
2016-12-07 15:24:25.875 28248 ERROR ceilometer.nova_client     raise ConnectionError(err, request=request)
2016-12-07 15:24:25.875 28248 ERROR ceilometer.nova_client ConnectionError: ('Connection aborted.', BadStatusLine("''",))
2016-12-07 15:24:25.875 28248 ERROR ceilometer.nova_client 

Reproducible: Always

Comment 1 Pradeep Kilambi 2016-12-09 14:05:07 UTC

This looks like ceilometer is not able to acquire connection to nova. Can you double check and make sure nova is up and reachable?

Comment 2 chenruoyu111 2016-12-09 14:58:48 UTC

(In reply to Pradeep Kilambi from comment #1)
> This looks like ceilometer is not able to acquire connection to nova. Can
> you double check and make sure nova is up and reachable?

Thanks for your reply! Yes, nova services are up and running now on both Openstack controller and compute nodes. I could create new instance.

On controller node:

MainPID=9848 Id=openstack-nova-api.service ActiveState=active
MainPID=10527 Id=openstack-nova-cert.service ActiveState=active
MainPID=9774 Id=openstack-nova-conductor.service ActiveState=active
MainPID=10529 Id=openstack-nova-consoleauth.service ActiveState=active
MainPID=9762 Id=openstack-nova-novncproxy.service ActiveState=active
MainPID=10521 Id=openstack-nova-scheduler.service ActiveState=active

On compute node:

MainPID=5095 Id=openstack-nova-compute.service ActiveState=active

Could you tell me is there other commands or operations I could try to verify it? Thank you!!

Comment 3 Yurii Prokulevych 2016-12-09 15:19:54 UTC

Do U use network isolation in your setup ? what's the value for 'interface' option from 'service_credentials' section of ceilometer.conf ?

Comment 4 chenruoyu111 2016-12-09 15:40:14 UTC

Yes, we use network isolation. "interface = internalURL". Below is the service credentials section of ceilometer.conf on the compute node.

[service_credentials]

auth_url = http://radocl.staging.moc.edu:5000
project_domain_id = default
user_domain_id = default
auth_type = password
username = ceilometer
project_name = services
password = (not show here)
interface = internalURL
region_name = Staging_rado

This is the endpoint information of ceilometer:

[root@compute-207 ruoyu(openstack_admin)]# openstack endpoint show ceilometer
+--------------+---------------------------------------+
| Field        | Value                                 |
+--------------+---------------------------------------+
| adminurl     | http://r-tempest.staging.moc.edu:8777 |
| enabled      | True                                  |
| id           | 45a2aeb02b9c4c0684a39a1e2781cb9e      |
| internalurl  | http://r-tempest.staging.moc.edu:8777 |
| publicurl    | http://r-tempest.staging.moc.edu:8777 |
| region       | Staging_rado                          |
| service_id   | 5216cb1aa4444987a21c2014196dadbd      |
| service_name | ceilometer                            |
| service_type | metering                              |
+--------------+---------------------------------------+

Comment 5 chenruoyu111 2016-12-09 15:40:14 UTC

Yes, we use network isolation. "interface = internalURL". Below is the service credentials section of ceilometer.conf on the compute node.

[service_credentials]

auth_url = http://radocl.staging.moc.edu:5000
project_domain_id = default
user_domain_id = default
auth_type = password
username = ceilometer
project_name = services
password = (not show here)
interface = internalURL
region_name = Staging_rado

This is the endpoint information of ceilometer:

[root@compute-207 ruoyu(openstack_admin)]# openstack endpoint show ceilometer
+--------------+---------------------------------------+
| Field        | Value                                 |
+--------------+---------------------------------------+
| adminurl     | http://r-tempest.staging.moc.edu:8777 |
| enabled      | True                                  |
| id           | 45a2aeb02b9c4c0684a39a1e2781cb9e      |
| internalurl  | http://r-tempest.staging.moc.edu:8777 |
| publicurl    | http://r-tempest.staging.moc.edu:8777 |
| region       | Staging_rado                          |
| service_id   | 5216cb1aa4444987a21c2014196dadbd      |
| service_name | ceilometer                            |
| service_type | metering                              |
+--------------+---------------------------------------+

Comment 6 Yurii Prokulevych 2016-12-09 15:56:44 UTC

Can U please check that all endpoints of Nova service are available from the compute - node ?

E.g: curl -XGET -H 'X-Auth-Token: <$TOKEN>' http://nova-internal-api

Comment 7 chenruoyu111 2016-12-09 16:08:48 UTC

Hi Yurii,

Below is the nova endpoint:

[root@compute-207 ruoyu(openstack_admin)]# openstack endpoint show f904d3b71f8f40c38219062c728adae7
+--------------+------------------------------------------------------+
| Field        | Value                                                |
+--------------+------------------------------------------------------+
| adminurl     | https://radocl.staging.moc.edu:8774/v2/%(tenant_id)s |
| enabled      | True                                                 |
| id           | f904d3b71f8f40c38219062c728adae7                     |
| internalurl  | https://radocl.staging.moc.edu:8774/v2/%(tenant_id)s |
| publicurl    | https://radocl.staging.moc.edu:8774/v2/%(tenant_id)s |
| region       | Staging_rado                                         |
| service_id   | 0200dc4688f74dbd81359cf8a1a94dfa                     |
| service_name | nova                                                 |
| service_type | compute                                              |
+--------------+------------------------------------------------------+

[root@compute-208 ruoyu(openstack_admin)]# openstack project list
+----------------------------------+----------+
| ID                               | Name     |
+----------------------------------+----------+
| a378bf17cb3343d484dd56e3abcf72bd | admin    |
| 428964cd4d65423097f15aa049da1adb | services |
+----------------------------------+----------+


Our project is "services".

[root@compute-208 ruoyu(openstack_admin)]# curl -XGET -H 'X-Auth-Token: <$TOKEN>' https://radocl.staging.moc.edu:8774/v2/428964cd4d65423097f15aa049da1adb
Authentication required

Above is what I got. Thanks!

Comment 8 Yurii Prokulevych 2016-12-09 16:26:49 UTC

U have to obtain the token, with 'openstack token issue' or 'keystone token-get' command.

Comment 9 chenruoyu111 2016-12-09 16:33:08 UTC

Hi Yurii,

Thanks for your guidance. I got "The resource could not be found." Below is the full result:

[root@compute-208 ruoyu(openstack_admin)]# curl -XGET -H 'X-Auth-Token: 29feaeed9088443f8a2c597d0c40c97c' https://radocl.staging.moc.edu:8774/v2/428964cd4d65423097f15aa049da1adb
<html>
 <head>
  <title>404 Not Found</title>
 </head>
 <body>
  <h1>404 Not Found</h1>
  The resource could not be found.<br /><br />



 </body>
</html>

Comment 10 chenruoyu111 2016-12-09 16:33:09 UTC

Hi Yurii,

Thanks for your guidance. I got "The resource could not be found." Below is the full result:

[root@compute-208 ruoyu(openstack_admin)]# curl -XGET -H 'X-Auth-Token: 29feaeed9088443f8a2c597d0c40c97c' https://radocl.staging.moc.edu:8774/v2/428964cd4d65423097f15aa049da1adb
<html>
 <head>
  <title>404 Not Found</title>
 </head>
 <body>
  <h1>404 Not Found</h1>
  The resource could not be found.<br /><br />



 </body>
</html>

Comment 11 Yurii Prokulevych 2016-12-09 16:51:19 UTC

Could U please try direct curl request to keystone to get an info for nova, e.g:

curl -XGET -H 'X-Auth-Token: $TOKEN' <KeystoneInternalApiUrl>:/v3/services/0200dc4688f74dbd81359cf8a1a94dfa 

Note: feel free to update keystone's verstion (v2.0) in the url if needed.

Comment 12 chenruoyu111 2016-12-09 17:18:43 UTC

Yes, this is the keystone endpoint info:

[root@compute-208 ruoyu(openstack_admin)]# openstack endpoint show keystone
+--------------+-------------------------------------------+
| Field        | Value                                     |
+--------------+-------------------------------------------+
| adminurl     | https://radocl.staging.moc.edu:35357/v2.0 |
| enabled      | True                                      |
| id           | 68e68dda606f4f70a0e7aad1d84bc94e          |
| internalurl  | https://radocl.staging.moc.edu:5000/v2.0  |
| publicurl    | https://radocl.staging.moc.edu:5000/v2.0  |
| region       | Staging_rado                              |
| service_id   | f37788deaceb442684b2a41435568c86          |
| service_name | keystone                                  |
| service_type | identity                                  |
+--------------+-------------------------------------------+

This is the result:

[root@compute-208 ruoyu(openstack_admin)]# curl -XGET -H 'X-Auth-Token: ed10b93bc76b41c6ab238ad3d0847cfb' https://radocl.staging.moc.edu:/v2.0/services/0200dc4688f74dbd81359cf8a1a94dfa 
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /v2.0/services/0200dc4688f74dbd81359cf8a1a94dfa was not found on this server.</p>
</body></html>

Comment 13 Yurii Prokulevych 2016-12-09 17:43:07 UTC

The 'bad status line' message usually indicates issues while accessing urls with SSL.
Could U please provide more extended log message, which included the pollster invoked and with debug/verbose enabled.

Comment 14 chenruoyu111 2016-12-09 18:45:28 UTC

Hi Yurii, I set the ceilometer log level to debug. Below is more logs:

2016-12-09 11:49:06.259 16537 ERROR ceilometer.nova_client [-] HTTPConnectionPool(host='radocl.staging.moc.edu', port=5000): Read timed out. (read timeout=600.0)
2016-12-09 11:49:06.259 16537 ERROR ceilometer.nova_client Traceback (most recent call last):
2016-12-09 11:49:06.259 16537 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/ceilometer/nova_client.py", line 47, in with_logging
2016-12-09 11:49:06.259 16537 ERROR ceilometer.nova_client     return func(*args, **kwargs)
2016-12-09 11:49:06.259 16537 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/ceilometer/nova_client.py", line 150, in instance_get_all_by_host
2016-12-09 11:49:06.259 16537 ERROR ceilometer.nova_client     search_opts=search_opts))
2016-12-09 11:49:06.259 16537 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 623, in list
2016-12-09 11:49:06.259 16537 ERROR ceilometer.nova_client     "servers")
2016-12-09 11:49:06.259 16537 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 74, in _list
2016-12-09 11:49:06.259 16537 ERROR ceilometer.nova_client     _resp, body = self.api.client.get(url)
2016-12-09 11:49:06.259 16537 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 448, in get
2016-12-09 11:49:06.259 16537 ERROR ceilometer.nova_client     return self._cs_request(url, 'GET', **kwargs)
2016-12-09 11:49:06.259 16537 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 404, in _cs_request
2016-12-09 11:49:06.259 16537 ERROR ceilometer.nova_client     self.authenticate()
2016-12-09 11:49:06.259 16537 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 561, in authenticate
2016-12-09 11:49:06.259 16537 ERROR ceilometer.nova_client     auth_url = self._v2_auth(auth_url)
2016-12-09 11:49:06.259 16537 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 652, in _v2_auth
2016-12-09 11:49:06.259 16537 ERROR ceilometer.nova_client     return self._authenticate(url, body)
2016-12-09 11:49:06.259 16537 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 665, in _authenticate
2016-12-09 11:49:06.259 16537 ERROR ceilometer.nova_client     **kwargs)
2016-12-09 11:49:06.259 16537 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 399, in _time_request
2016-12-09 11:49:06.259 16537 ERROR ceilometer.nova_client     resp, body = self.request(url, method, **kwargs)
2016-12-09 11:49:06.259 16537 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 366, in request
2016-12-09 11:49:06.259 16537 ERROR ceilometer.nova_client     **kwargs)
2016-12-09 11:49:06.259 16537 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/requests/api.py", line 50, in request
2016-12-09 11:49:06.259 16537 ERROR ceilometer.nova_client     response = session.request(method=method, url=url, **kwargs)
2016-12-09 11:49:06.259 16537 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 465, in request
2016-12-09 11:49:06.259 16537 ERROR ceilometer.nova_client     resp = self.send(prep, **send_kwargs)
2016-12-09 11:49:06.259 16537 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
2016-12-09 11:49:06.259 16537 ERROR ceilometer.nova_client     r = adapter.send(request, **kwargs)
2016-12-09 11:49:06.259 16537 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 433, in send
2016-12-09 11:49:06.259 16537 ERROR ceilometer.nova_client     raise ReadTimeout(e, request=request)
2016-12-09 11:49:06.259 16537 ERROR ceilometer.nova_client ReadTimeout: HTTPConnectionPool(host='radocl.staging.moc.edu', port=5000): Read timed out. (read timeout=600.0)
2016-12-09 11:49:06.259 16537 ERROR ceilometer.nova_client 
2016-12-09 11:49:06.261 16537 INFO ceilometer.agent.manager [-] Skip pollster network.outgoing.bytes, no resources found this cycle
2016-12-09 11:49:06.262 16537 INFO ceilometer.agent.manager [-] Skip pollster disk.device.write.requests, no resources found this cycle
2016-12-09 11:49:06.262 16537 INFO ceilometer.agent.manager [-] Skip pollster network.outgoing.bytes.rate, no resources found this cycle
2016-12-09 11:49:06.263 16537 INFO ceilometer.agent.manager [-] Skip pollster disk.device.read.requests.rate, no resources found this cycle
2016-12-09 11:49:06.263 16537 INFO ceilometer.agent.manager [-] Skip pollster disk.device.usage, no resources found this cycle
2016-12-09 11:49:06.263 16537 INFO ceilometer.agent.manager [-] Skip pollster disk.capacity, no resources found this cycle
2016-12-09 11:49:06.264 16537 INFO ceilometer.agent.manager [-] Skip pollster cpu, no resources found this cycle
2016-12-09 11:49:06.264 16537 INFO ceilometer.agent.manager [-] Skip pollster disk.write.requests, no resources found this cycle
2016-12-09 11:49:06.264 16537 INFO ceilometer.agent.manager [-] Skip pollster instance, no resources found this cycle
2016-12-09 11:49:06.265 16537 INFO ceilometer.agent.manager [-] Skip pollster disk.device.read.requests, no resources found this cycle
2016-12-09 11:49:06.265 16537 INFO ceilometer.agent.manager [-] Skip pollster disk.device.write.bytes, no resources found this cycle
2016-12-09 11:49:06.265 16537 INFO ceilometer.agent.manager [-] Skip pollster disk.device.iops, no resources found this cycle
2016-12-09 11:49:06.266 16537 INFO ceilometer.agent.manager [-] Skip pollster memory.usage, no resources found this cycle
2016-12-09 11:49:06.266 16537 INFO ceilometer.agent.manager [-] Skip pollster disk.allocation, no resources found this cycle
2016-12-09 11:49:06.267 16537 INFO ceilometer.agent.manager [-] Skip pollster network.incoming.packets, no resources found this cycle
2016-12-09 11:49:06.267 16537 INFO ceilometer.agent.manager [-] Skip pollster disk.write.bytes.rate, no resources found this cycle
2016-12-09 11:49:06.267 16537 INFO ceilometer.agent.manager [-] Skip pollster network.incoming.bytes.rate, no resources found this cycle
2016-12-09 11:49:06.268 16537 INFO ceilometer.agent.manager [-] Skip pollster disk.device.latency, no resources found this cycle
2016-12-09 11:49:06.268 16537 INFO ceilometer.agent.manager [-] Skip pollster disk.read.requests.rate, no resources found this cycle
2016-12-09 11:49:06.268 16537 INFO ceilometer.agent.manager [-] Skip pollster network.incoming.bytes, no resources found this cycle
2016-12-09 11:49:06.269 16537 INFO ceilometer.agent.manager [-] Skip pollster memory.resident, no resources found this cycle
2016-12-09 11:49:06.269 16537 INFO ceilometer.agent.manager [-] Skip pollster disk.iops, no resources found this cycle
2016-12-09 11:49:06.269 16537 INFO ceilometer.agent.manager [-] Skip pollster network.outgoing.packets, no resources found this cycle
2016-12-09 11:49:06.270 16537 INFO ceilometer.agent.manager [-] Skip pollster cpu_util, no resources found this cycle
2016-12-09 11:49:06.270 16537 INFO ceilometer.agent.manager [-] Skip pollster disk.read.requests, no resources found this cycle
2016-12-09 11:49:06.270 16537 INFO ceilometer.agent.manager [-] Skip pollster disk.write.bytes, no resources found this cycle
2016-12-09 11:49:06.271 16537 INFO ceilometer.agent.manager [-] Skip pollster disk.device.read.bytes.rate, no resources found this cycle
2016-12-09 11:49:06.271 16537 INFO ceilometer.agent.manager [-] Skip pollster disk.device.read.bytes, no resources found this cycle
2016-12-09 11:49:06.271 16537 INFO ceilometer.agent.manager [-] Skip pollster disk.write.requests.rate, no resources found this cycle
2016-12-09 11:49:06.272 16537 INFO ceilometer.agent.manager [-] Skip pollster disk.device.capacity, no resources found this cycle
2016-12-09 11:49:06.272 16537 INFO ceilometer.agent.manager [-] Skip pollster disk.device.write.requests.rate, no resources found this cycle
2016-12-09 11:49:06.272 16537 INFO ceilometer.agent.manager [-] Skip pollster disk.device.allocation, no resources found this cycle
2016-12-09 11:49:06.273 16537 INFO ceilometer.agent.manager [-] Skip pollster disk.read.bytes, no resources found this cycle
2016-12-09 11:49:06.273 16537 INFO ceilometer.agent.manager [-] Skip pollster disk.usage, no resources found this cycle
2016-12-09 11:49:06.273 16537 INFO ceilometer.agent.manager [-] Skip pollster disk.device.write.bytes.rate, no resources found this cycle
2016-12-09 11:49:06.274 16537 INFO ceilometer.agent.manager [-] Skip pollster disk.read.bytes.rate, no resources found this cycle
2016-12-09 11:49:06.274 16537 INFO ceilometer.agent.manager [-] Skip pollster disk.latency, no resources found this cycle
2016-12-09 11:49:06.274 16537 INFO ceilometer.agent.manager [-] Skip pollster disk.device.read.bytes, no resources found this cycle
2016-12-09 11:49:06.275 16537 INFO ceilometer.agent.manager [-] Skip pollster disk.device.write.requests, no resources found this cycle
2016-12-09 11:49:06.275 16537 INFO ceilometer.agent.manager [-] Skip pollster disk.read.requests, no resources found this cycle
2016-12-09 11:49:06.276 16537 INFO ceilometer.agent.manager [-] Skip pollster disk.write.bytes, no resources found this cycle
2016-12-09 11:49:06.276 16537 INFO ceilometer.agent.manager [-] Skip pollster disk.device.read.requests, no resources found this cycle
2016-12-09 11:49:06.276 16537 INFO ceilometer.agent.manager [-] Skip pollster disk.device.write.bytes, no resources found this cycle
2016-12-09 11:49:06.277 16537 INFO ceilometer.agent.manager [-] Skip pollster disk.read.bytes, no resources found this cycle
2016-12-09 11:49:06.277 16537 INFO ceilometer.agent.manager [-] Skip pollster disk.write.requests, no resources found this cycle
2016-12-09 11:49:06.277 16537 INFO ceilometer.agent.manager [-] Skip pollster cpu, no resources found this cycle
2016-12-09 11:49:06.278 16537 INFO ceilometer.agent.manager [-] Skip pollster network.incoming.bytes, no resources found this cycle
2016-12-09 11:49:06.278 16537 INFO ceilometer.agent.manager [-] Skip pollster network.outgoing.packets, no resources found this cycle
2016-12-09 11:49:06.278 16537 INFO ceilometer.agent.manager [-] Skip pollster network.incoming.packets, no resources found this cycle
2016-12-09 11:49:06.279 16537 INFO ceilometer.agent.manager [-] Skip pollster network.outgoing.bytes, no resources found this cycle
2016-12-09 11:55:09.749 16537 ERROR ceilometer.nova_client [-] ('Connection aborted.', BadStatusLine("''",))
2016-12-09 11:55:09.749 16537 ERROR ceilometer.nova_client Traceback (most recent call last):
2016-12-09 11:55:09.749 16537 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/ceilometer/nova_client.py", line 47, in with_logging
2016-12-09 11:55:09.749 16537 ERROR ceilometer.nova_client     return func(*args, **kwargs)
2016-12-09 11:55:09.749 16537 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/ceilometer/nova_client.py", line 150, in instance_get_all_by_host
2016-12-09 11:55:09.749 16537 ERROR ceilometer.nova_client     search_opts=search_opts))
2016-12-09 11:55:09.749 16537 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 623, in list
2016-12-09 11:55:09.749 16537 ERROR ceilometer.nova_client     "servers")
2016-12-09 11:55:09.749 16537 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 74, in _list
2016-12-09 11:55:09.749 16537 ERROR ceilometer.nova_client     _resp, body = self.api.client.get(url)
2016-12-09 11:55:09.749 16537 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 448, in get
2016-12-09 11:55:09.749 16537 ERROR ceilometer.nova_client     return self._cs_request(url, 'GET', **kwargs)
2016-12-09 11:55:09.749 16537 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 404, in _cs_request
2016-12-09 11:55:09.749 16537 ERROR ceilometer.nova_client     self.authenticate()
2016-12-09 11:55:09.749 16537 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 561, in authenticate
2016-12-09 11:55:09.749 16537 ERROR ceilometer.nova_client     auth_url = self._v2_auth(auth_url)
2016-12-09 11:55:09.749 16537 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 652, in _v2_auth
2016-12-09 11:55:09.749 16537 ERROR ceilometer.nova_client     return self._authenticate(url, body)
2016-12-09 11:55:09.749 16537 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 665, in _authenticate
2016-12-09 11:55:09.749 16537 ERROR ceilometer.nova_client     **kwargs)
2016-12-09 11:55:09.749 16537 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 399, in _time_request
2016-12-09 11:55:09.749 16537 ERROR ceilometer.nova_client     resp, body = self.request(url, method, **kwargs)
2016-12-09 11:55:09.749 16537 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 366, in request
2016-12-09 11:55:09.749 16537 ERROR ceilometer.nova_client     **kwargs)
2016-12-09 11:55:09.749 16537 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/requests/api.py", line 50, in request
2016-12-09 11:55:09.749 16537 ERROR ceilometer.nova_client     response = session.request(method=method, url=url, **kwargs)
2016-12-09 11:55:09.749 16537 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 465, in request
2016-12-09 11:55:09.749 16537 ERROR ceilometer.nova_client     resp = self.send(prep, **send_kwargs)
2016-12-09 11:55:09.749 16537 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
2016-12-09 11:55:09.749 16537 ERROR ceilometer.nova_client     r = adapter.send(request, **kwargs)
2016-12-09 11:55:09.749 16537 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 415, in send
2016-12-09 11:55:09.749 16537 ERROR ceilometer.nova_client     raise ConnectionError(err, request=request)
2016-12-09 11:55:09.749 16537 ERROR ceilometer.nova_client ConnectionError: ('Connection aborted.', BadStatusLine("''",))
2016-12-09 11:55:09.749 16537 ERROR ceilometer.nova_client

Comment 15 chenruoyu111 2016-12-09 19:05:26 UTC

2016-12-09 14:00:06.080 16537 DEBUG oslo_concurrency.lockutils [-] Acquired semaphore "singleton_lock" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:198
2016-12-09 14:00:06.081 16537 DEBUG oslo_concurrency.lockutils [-] Releasing semaphore "singleton_lock" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:211
2016-12-09 14:00:06.082 16537 DEBUG oslo_concurrency.lockutils [req-e40fa094-cb61-444e-869d-9e34241765a6 admin - - - -] Acquired semaphore "singleton_lock" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:198
2016-12-09 14:00:06.082 16537 DEBUG oslo_concurrency.lockutils [req-e40fa094-cb61-444e-869d-9e34241765a6 admin - - - -] Releasing semaphore "singleton_lock" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:211
2016-12-09 14:00:06.082 16537 INFO oslo_service.service [req-e40fa094-cb61-444e-869d-9e34241765a6 admin - - - -] Caught SIGTERM, exiting
2016-12-09 14:00:06.083 16537 DEBUG oslo_concurrency.lockutils [req-e40fa094-cb61-444e-869d-9e34241765a6 admin - - - -] Acquired semaphore "singleton_lock" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:198
2016-12-09 14:00:06.083 16537 DEBUG oslo_concurrency.lockutils [req-e40fa094-cb61-444e-869d-9e34241765a6 admin - - - -] Releasing semaphore "singleton_lock" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:211
2016-12-09 14:00:06.732 8428 DEBUG ceilometer.compute.virt.libvirt.inspector [-] Connecting to libvirt: qemu:///system _get_connection /usr/lib/python2.7/site-packages/ceilometer/compute/virt/libvirt/inspector.py:80
2016-12-09 14:00:06.809 8428 DEBUG oslo_concurrency.lockutils [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] Acquired semaphore "singleton_lock" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:198
2016-12-09 14:00:06.810 8428 DEBUG oslo_concurrency.lockutils [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] Releasing semaphore "singleton_lock" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:211
2016-12-09 14:00:06.810 8428 DEBUG oslo_concurrency.lockutils [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] Acquired semaphore "singleton_lock" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:198
2016-12-09 14:00:06.811 8428 DEBUG oslo_concurrency.lockutils [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] Releasing semaphore "singleton_lock" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:211
2016-12-09 14:00:06.811 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] Full set of CONF: _wait_for_exit_or_signal /usr/lib/python2.7/site-packages/oslo_service/service.py:253
2016-12-09 14:00:06.811 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] ******************************************************************************** log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2216
2016-12-09 14:00:06.811 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] Configuration options gathered from: log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2217
2016-12-09 14:00:06.811 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] command line args: ['--polling-namespaces', 'compute', '--logfile', '/var/log/ceilometer/compute.log'] log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2218
2016-12-09 14:00:06.812 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] config files: ['/usr/share/ceilometer/ceilometer-dist.conf', '/etc/ceilometer/ceilometer.conf'] log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2219
2016-12-09 14:00:06.812 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] ================================================================================ log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2220
2016-12-09 14:00:06.812 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] backdoor_port                  = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2229
2016-12-09 14:00:06.813 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] batch_polled_samples           = True log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2229
2016-12-09 14:00:06.813 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] config_dir                     = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2229
2016-12-09 14:00:06.813 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] config_file                    = ['/usr/share/ceilometer/ceilometer-dist.conf', '/etc/ceilometer/ceilometer.conf'] log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2229
2016-12-09 14:00:06.813 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] control_exchange               = ceilometer log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2229
2016-12-09 14:00:06.814 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] database_connection            = **** log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2229
2016-12-09 14:00:06.814 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] debug                          = True log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2229
2016-12-09 14:00:06.815 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] default_log_levels             = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'stevedore=INFO', 'keystoneclient=INFO'] log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2229
2016-12-09 14:00:06.815 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] event_pipeline_cfg_file        = event_pipeline.yaml log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2229
2016-12-09 14:00:06.815 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] glance_page_size               = 0 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2229
2016-12-09 14:00:06.816 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] host                           = compute-208.staging.moc.edu log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2229
2016-12-09 14:00:06.816 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] http_timeout                   = 600 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2229
2016-12-09 14:00:06.816 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] hypervisor_inspector           = libvirt log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2229
2016-12-09 14:00:06.816 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] instance_format                = [instance: %(uuid)s]  log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2229
2016-12-09 14:00:06.817 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] instance_uuid_format           = [instance: %(uuid)s]  log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2229
2016-12-09 14:00:06.817 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] libvirt_type                   = kvm log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2229
2016-12-09 14:00:06.817 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] libvirt_uri                    =  log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2229
2016-12-09 14:00:06.817 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] log_config_append              = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2229
2016-12-09 14:00:06.818 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] log_date_format                = %Y-%m-%d %H:%M:%S log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2229
2016-12-09 14:00:06.818 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] log_dir                        = /var/log/ceilometer log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2229
2016-12-09 14:00:06.818 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] log_file                       = /var/log/ceilometer/compute.log log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2229
2016-12-09 14:00:06.818 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] log_format                     = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2229
2016-12-09 14:00:06.819 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] log_options                    = True log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2229
2016-12-09 14:00:06.819 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] logging_context_format_string  = %(asctime)s.%(msecs)03d %(process)d %(levelname)s %(name)s [%(request_id)s %(user_identity)s] %(instance)s%(message)s log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2229
2016-12-09 14:00:06.819 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] logging_debug_format_suffix    = %(funcName)s %(pathname)s:%(lineno)d log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2229
2016-12-09 14:00:06.819 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] logging_default_format_string  = %(asctime)s.%(msecs)03d %(process)d %(levelname)s %(name)s [-] %(instance)s%(message)s log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2229
2016-12-09 14:00:06.820 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] logging_exception_prefix       = %(asctime)s.%(msecs)03d %(process)d ERROR %(name)s %(instance)s log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2229
2016-12-09 14:00:06.820 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] nova_http_log_debug            = True log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2229
2016-12-09 14:00:06.820 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] pipeline_cfg_file              = pipeline.yaml log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2229
2016-12-09 14:00:06.820 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] pipeline_polling_interval      = 20 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2229
2016-12-09 14:00:06.821 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] polling_namespaces             = ['compute'] log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2229
2016-12-09 14:00:06.821 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] pollster_list                  = [] log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2229
2016-12-09 14:00:06.821 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] publish_errors                 = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2229
2016-12-09 14:00:06.821 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] refresh_event_pipeline_cfg     = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2229
2016-12-09 14:00:06.822 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] refresh_pipeline_cfg           = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2229
2016-12-09 14:00:06.822 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] reseller_prefix                = AUTH_ log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2229
2016-12-09 14:00:06.822 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] reserved_metadata_keys         = [] log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2229
2016-12-09 14:00:06.822 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] reserved_metadata_length       = 256 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2229
2016-12-09 14:00:06.823 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] reserved_metadata_namespace    = ['metering.'] log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2229
2016-12-09 14:00:06.823 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] rootwrap_config                = /etc/ceilometer/rootwrap.conf log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2229
2016-12-09 14:00:06.823 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] rpc_backend                    = rabbit log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2229
2016-12-09 14:00:06.823 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] sample_source                  = openstack log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2229
2016-12-09 14:00:06.824 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] shuffle_time_before_polling_task = 0 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2229
2016-12-09 14:00:06.824 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] sql_expire_samples_only        = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2229
2016-12-09 14:00:06.824 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] syslog_log_facility            = LOG_USER log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2229
2016-12-09 14:00:06.824 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] transport_url                  = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2229
2016-12-09 14:00:06.825 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] use_stderr                     = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2229
2016-12-09 14:00:06.825 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] use_syslog                     = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2229
2016-12-09 14:00:06.825 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] use_syslog_rfc_format          = True log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2229
2016-12-09 14:00:06.825 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] verbose                        = True log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2229
2016-12-09 14:00:06.826 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] publisher.telemetry_secret     = **** log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.826 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] service_types.glance           = image log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.826 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] service_types.kwapi            = energy log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.827 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] service_types.neutron          = network log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.827 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] service_types.nova             = compute log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.827 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] service_types.radosgw          = object-store log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.827 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] service_types.swift            = object-store log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.828 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] compute.workload_partitioning  = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.828 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] publisher_rpc.metering_topic   = metering log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.828 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] database.alarm_connection      = **** log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.828 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] database.alarm_history_time_to_live = -1 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.829 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] database.backend               = sqlalchemy log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.829 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] database.connection            = **** log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.829 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] database.connection_debug      = 0 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.830 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] database.connection_trace      = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.830 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] database.db2nosql_resource_id_maxlen = 512 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.830 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] database.db_inc_retry_interval = True log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.830 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] database.db_max_retries        = 20 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.831 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] database.db_max_retry_interval = 10 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.831 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] database.db_retry_interval     = 1 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.831 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] database.event_connection      = **** log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.832 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] database.event_time_to_live    = -1 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.832 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] database.idle_timeout          = 3600 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.832 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] database.max_overflow          = 50 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.832 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] database.max_pool_size         = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.832 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] database.max_retries           = 10 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.833 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] database.metering_connection   = **** log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.833 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] database.metering_time_to_live = -1 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.833 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] database.min_pool_size         = 1 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.833 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] database.mongodb_replica_set   =  log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.834 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] database.mysql_sql_mode        = TRADITIONAL log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.834 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] database.pool_timeout          = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.834 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] database.retry_interval        = 10 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.834 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] database.slave_connection      = **** log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.835 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] database.sqlite_db             = oslo.sqlite log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.835 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] database.sqlite_synchronous    = True log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.835 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] database.use_db_reconnect      = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.835 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] notification.workers           = 1 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.836 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] oslo_messaging_rabbit.amqp_auto_delete = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.836 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] oslo_messaging_rabbit.amqp_durable_queues = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.836 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] oslo_messaging_rabbit.fake_rabbit = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.836 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] oslo_messaging_rabbit.heartbeat_rate = 2 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.837 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] oslo_messaging_rabbit.heartbeat_timeout_threshold = 60 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.837 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] oslo_messaging_rabbit.kombu_failover_strategy = round-robin log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.837 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] oslo_messaging_rabbit.kombu_reconnect_delay = 1.0 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.837 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] oslo_messaging_rabbit.kombu_reconnect_timeout = 60 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.838 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] oslo_messaging_rabbit.kombu_ssl_ca_certs =  log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.838 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] oslo_messaging_rabbit.kombu_ssl_certfile =  log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.838 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] oslo_messaging_rabbit.kombu_ssl_keyfile =  log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.839 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] oslo_messaging_rabbit.kombu_ssl_version =  log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.839 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] oslo_messaging_rabbit.rabbit_ha_queues = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.839 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] oslo_messaging_rabbit.rabbit_host = radocl.staging.moc.edu log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.839 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] oslo_messaging_rabbit.rabbit_hosts = ['radocl.staging.moc.edu:5672'] log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.840 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] oslo_messaging_rabbit.rabbit_login_method = AMQPLAIN log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.840 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] oslo_messaging_rabbit.rabbit_max_retries = 0 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.840 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] oslo_messaging_rabbit.rabbit_password = **** log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.840 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] oslo_messaging_rabbit.rabbit_port = 5672 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.841 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] oslo_messaging_rabbit.rabbit_qos_prefetch_count = 0 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.841 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] oslo_messaging_rabbit.rabbit_retry_backoff = 2 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.841 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] oslo_messaging_rabbit.rabbit_retry_interval = 1 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.841 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] oslo_messaging_rabbit.rabbit_transient_queues_ttl = 600 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.842 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] oslo_messaging_rabbit.rabbit_use_ssl = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.842 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] oslo_messaging_rabbit.rabbit_userid = openstack log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.842 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] oslo_messaging_rabbit.rabbit_virtual_host = / log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.842 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] oslo_messaging_rabbit.rpc_conn_pool_size = 30 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.843 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] oslo_messaging_rabbit.send_single_reply = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.843 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] rgw_admin_credentials.access_key = **** log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.843 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] rgw_admin_credentials.secret_key = **** log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.843 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] hardware.readonly_user_name    = ro_snmp_user log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.844 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] hardware.readonly_user_password = **** log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.844 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] hardware.url_scheme            = snmp:// log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.844 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] api.workers                    = 1 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.844 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] service_credentials.insecure   = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.845 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] service_credentials.os_auth_url = http://radocl.staging.moc.edu:5000/v2.0 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.845 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] service_credentials.os_cacert  = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.845 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] service_credentials.os_endpoint_type = internalURL log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.845 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] service_credentials.os_password = **** log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.846 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] service_credentials.os_region_name = Staging_rado log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.846 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] service_credentials.os_tenant_id =  log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.846 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] service_credentials.os_tenant_name = services log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.846 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] service_credentials.os_username = ceilometer log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.847 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] polling.partitioning_group_prefix = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.847 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] oslo_messaging_notifications.driver = [] log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.847 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] oslo_messaging_notifications.topics = ['notifications'] log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.848 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] oslo_messaging_notifications.transport_url = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.848 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] publisher_notifier.event_topic = event log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.848 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] publisher_notifier.metering_topic = metering log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.848 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] publisher_notifier.telemetry_driver = messagingv2 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.849 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] collector.workers              = 1 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.849 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] coordination.backend_url       = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.849 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] coordination.check_watchers    = 10.0 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.849 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] coordination.heartbeat         = 1.0 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.850 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] oslo_concurrency.disable_process_locking = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.850 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] oslo_concurrency.lock_path     = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
2016-12-09 14:00:06.850 8428 DEBUG oslo_service.service [req-e63d8e30-11b4-4079-958b-19685b66935b admin - - - -] ******************************************************************************** log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2239
2016-12-09 14:00:06.851 8428 DEBUG ceilometer.pipeline [-] Polling config file: /etc/ceilometer/pipeline.yaml _setup_polling_manager /usr/lib/python2.7/site-packages/ceilometer/pipeline.py:811
2016-12-09 14:00:06.873 8428 INFO ceilometer.pipeline [-] Pipeline config: {'sources': [{'interval': 600, 'meters': ['*'], 'name': 'meter_source', 'sinks': ['meter_sink']}, {'interval': 600, 'meters': ['cpu'], 'name': 'cpu_source', 'sinks': ['cpu_sink', 'cpu_delta_sink']}, {'interval': 600, 'meters': ['disk.read.bytes', 'disk.read.requests', 'disk.write.bytes', 'disk.write.requests', 'disk.device.read.bytes', 'disk.device.read.requests', 'disk.device.write.bytes', 'disk.device.write.requests'], 'name': 'disk_source', 'sinks': ['disk_sink']}, {'interval': 600, 'meters': ['network.incoming.bytes', 'network.incoming.packets', 'network.outgoing.bytes', 'network.outgoing.packets'], 'name': 'network_source', 'sinks': ['network_sink']}], 'sinks': [{'publishers': ['notifier://'], 'transformers': None, 'name': 'meter_sink'}, {'publishers': ['notifier://'], 'transformers': [{'name': 'rate_of_change', 'parameters': {'target': {'scale': '100.0 / (10**9 * (resource_metadata.cpu_number or 1))', 'type': 'gauge', 'name': 'cpu_util', 'unit': '%'}}}], 'name': 'cpu_sink'}, {'publishers': ['notifier://'], 'transformers': [{'name': 'delta', 'parameters': {'target': {'name': 'cpu.delta'}, 'growth_only': True}}], 'name': 'cpu_delta_sink'}, {'publishers': ['notifier://'], 'transformers': [{'name': 'rate_of_change', 'parameters': {'source': {'map_from': {'name': '(disk\\.device|disk)\\.(read|write)\\.(bytes|requests)', 'unit': '(B|request)'}}, 'target': {'type': 'gauge', 'map_to': {'name': '\\1.\\2.\\3.rate', 'unit': '\\1/s'}}}}], 'name': 'disk_sink'}, {'publishers': ['notifier://'], 'transformers': [{'name': 'rate_of_change', 'parameters': {'source': {'map_from': {'name': 'network\\.(incoming|outgoing)\\.(bytes|packets)', 'unit': '(B|packet)'}}, 'target': {'type': 'gauge', 'map_to': {'name': 'network.\\1.\\2.rate', 'unit': '\\1/s'}}}}], 'name': 'network_sink'}]}
2016-12-09 14:00:06.873 8428 INFO ceilometer.pipeline [-] detected decoupled pipeline config format
2016-12-09 14:00:06.947 8428 ERROR ceilometer.nova_client [-] ('Connection aborted.', BadStatusLine("''",))
2016-12-09 14:00:06.947 8428 ERROR ceilometer.nova_client Traceback (most recent call last):
2016-12-09 14:00:06.947 8428 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/ceilometer/nova_client.py", line 47, in with_logging
2016-12-09 14:00:06.947 8428 ERROR ceilometer.nova_client     return func(*args, **kwargs)
2016-12-09 14:00:06.947 8428 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/ceilometer/nova_client.py", line 150, in instance_get_all_by_host
2016-12-09 14:00:06.947 8428 ERROR ceilometer.nova_client     search_opts=search_opts))
2016-12-09 14:00:06.947 8428 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 623, in list
2016-12-09 14:00:06.947 8428 ERROR ceilometer.nova_client     "servers")
2016-12-09 14:00:06.947 8428 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 74, in _list
2016-12-09 14:00:06.947 8428 ERROR ceilometer.nova_client     _resp, body = self.api.client.get(url)
2016-12-09 14:00:06.947 8428 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 448, in get
2016-12-09 14:00:06.947 8428 ERROR ceilometer.nova_client     return self._cs_request(url, 'GET', **kwargs)
2016-12-09 14:00:06.947 8428 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 404, in _cs_request
2016-12-09 14:00:06.947 8428 ERROR ceilometer.nova_client     self.authenticate()
2016-12-09 14:00:06.947 8428 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 561, in authenticate
2016-12-09 14:00:06.947 8428 ERROR ceilometer.nova_client     auth_url = self._v2_auth(auth_url)
2016-12-09 14:00:06.947 8428 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 652, in _v2_auth
2016-12-09 14:00:06.947 8428 ERROR ceilometer.nova_client     return self._authenticate(url, body)
2016-12-09 14:00:06.947 8428 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 665, in _authenticate
2016-12-09 14:00:06.947 8428 ERROR ceilometer.nova_client     **kwargs)
2016-12-09 14:00:06.947 8428 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 399, in _time_request
2016-12-09 14:00:06.947 8428 ERROR ceilometer.nova_client     resp, body = self.request(url, method, **kwargs)
2016-12-09 14:00:06.947 8428 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 366, in request
2016-12-09 14:00:06.947 8428 ERROR ceilometer.nova_client     **kwargs)
2016-12-09 14:00:06.947 8428 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/requests/api.py", line 50, in request
2016-12-09 14:00:06.947 8428 ERROR ceilometer.nova_client     response = session.request(method=method, url=url, **kwargs)
2016-12-09 14:00:06.947 8428 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 465, in request
2016-12-09 14:00:06.947 8428 ERROR ceilometer.nova_client     resp = self.send(prep, **send_kwargs)
2016-12-09 14:00:06.947 8428 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
2016-12-09 14:00:06.947 8428 ERROR ceilometer.nova_client     r = adapter.send(request, **kwargs)
2016-12-09 14:00:06.947 8428 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 415, in send
2016-12-09 14:00:06.947 8428 ERROR ceilometer.nova_client     raise ConnectionError(err, request=request)
2016-12-09 14:00:06.947 8428 ERROR ceilometer.nova_client ConnectionError: ('Connection aborted.', BadStatusLine("''",))
2016-12-09 14:00:06.947 8428 ERROR ceilometer.nova_client 
2016-12-09 14:00:06.950 8428 INFO ceilometer.agent.manager [-] Skip pollster network.outgoing.bytes.rate, no resources found this cycle
2016-12-09 14:00:06.951 8428 INFO ceilometer.agent.manager [-] Skip pollster disk.device.read.requests.rate, no resources found this cycle
2016-12-09 14:00:06.951 8428 INFO ceilometer.agent.manager [-] Skip pollster disk.read.requests, no resources found this cycle
2016-12-09 14:00:06.951 8428 INFO ceilometer.agent.manager [-] Skip pollster disk.capacity, no resources found this cycle
2016-12-09 14:00:06.952 8428 INFO ceilometer.agent.manager [-] Skip pollster cpu, no resources found this cycle
2016-12-09 14:00:06.952 8428 INFO ceilometer.agent.manager [-] Skip pollster instance, no resources found this cycle
2016-12-09 14:00:06.953 8428 INFO ceilometer.agent.manager [-] Skip pollster disk.device.write.bytes, no resources found this cycle
2016-12-09 14:00:06.953 8428 INFO ceilometer.agent.manager [-] Skip pollster network.incoming.bytes.rate, no resources found this cycle
2016-12-09 14:00:06.953 8428 INFO ceilometer.agent.manager [-] Skip pollster memory.usage, no resources found this cycle
2016-12-09 14:00:06.954 8428 INFO ceilometer.agent.manager [-] Skip pollster disk.device.write.requests.rate, no resources found this cycle
2016-12-09 14:00:06.954 8428 INFO ceilometer.agent.manager [-] Skip pollster network.incoming.packets, no resources found this cycle
2016-12-09 14:00:06.954 8428 INFO ceilometer.agent.manager [-] Skip pollster disk.write.bytes.rate, no resources found this cycle
2016-12-09 14:00:06.955 8428 INFO ceilometer.agent.manager [-] Skip pollster disk.device.iops, no resources found this cycle
2016-12-09 14:00:06.955 8428 INFO ceilometer.agent.manager [-] Skip pollster disk.read.requests.rate, no resources found this cycle
2016-12-09 14:00:06.956 8428 INFO ceilometer.agent.manager [-] Skip pollster network.outgoing.bytes, no resources found this cycle
2016-12-09 14:00:06.956 8428 INFO ceilometer.agent.manager [-] Skip pollster disk.device.write.requests, no resources found this cycle
2016-12-09 14:00:06.956 8428 INFO ceilometer.agent.manager [-] Skip pollster network.incoming.bytes, no resources found this cycle
2016-12-09 14:00:06.957 8428 INFO ceilometer.agent.manager [-] Skip pollster memory.resident, no resources found this cycle
2016-12-09 14:00:06.957 8428 INFO ceilometer.agent.manager [-] Skip pollster network.outgoing.packets, no resources found this cycle
2016-12-09 14:00:06.957 8428 INFO ceilometer.agent.manager [-] Skip pollster disk.write.requests, no resources found this cycle
2016-12-09 14:00:06.958 8428 INFO ceilometer.agent.manager [-] Skip pollster cpu_util, no resources found this cycle
2016-12-09 14:00:06.958 8428 INFO ceilometer.agent.manager [-] Skip pollster disk.device.read.requests, no resources found this cycle
2016-12-09 14:00:06.958 8428 INFO ceilometer.agent.manager [-] Skip pollster disk.write.bytes, no resources found this cycle
2016-12-09 14:00:06.959 8428 INFO ceilometer.agent.manager [-] Skip pollster disk.allocation, no resources found this cycle
2016-12-09 14:00:06.959 8428 INFO ceilometer.agent.manager [-] Skip pollster disk.device.read.bytes.rate, no resources found this cycle
2016-12-09 14:00:06.960 8428 INFO ceilometer.agent.manager [-] Skip pollster disk.device.read.bytes, no resources found this cycle
2016-12-09 14:00:06.960 8428 INFO ceilometer.agent.manager [-] Skip pollster disk.write.requests.rate, no resources found this cycle
2016-12-09 14:00:06.960 8428 INFO ceilometer.agent.manager [-] Skip pollster disk.device.capacity, no resources found this cycle
2016-12-09 14:00:06.961 8428 INFO ceilometer.agent.manager [-] Skip pollster disk.device.latency, no resources found this cycle
2016-12-09 14:00:06.961 8428 INFO ceilometer.agent.manager [-] Skip pollster disk.device.allocation, no resources found this cycle
2016-12-09 14:00:06.962 8428 INFO ceilometer.agent.manager [-] Skip pollster disk.read.bytes, no resources found this cycle
2016-12-09 14:00:06.962 8428 INFO ceilometer.agent.manager [-] Skip pollster disk.usage, no resources found this cycle
2016-12-09 14:00:06.963 8428 INFO ceilometer.agent.manager [-] Skip pollster disk.device.usage, no resources found this cycle
2016-12-09 14:00:06.963 8428 INFO ceilometer.agent.manager [-] Skip pollster disk.device.write.bytes.rate, no resources found this cycle
2016-12-09 14:00:06.964 8428 INFO ceilometer.agent.manager [-] Skip pollster disk.iops, no resources found this cycle
2016-12-09 14:00:06.964 8428 INFO ceilometer.agent.manager [-] Skip pollster disk.read.bytes.rate, no resources found this cycle
2016-12-09 14:00:06.965 8428 INFO ceilometer.agent.manager [-] Skip pollster disk.latency, no resources found this cycle
2016-12-09 14:00:06.965 8428 INFO ceilometer.agent.manager [-] Skip pollster disk.device.read.bytes, no resources found this cycle
2016-12-09 14:00:06.965 8428 INFO ceilometer.agent.manager [-] Skip pollster disk.read.requests, no resources found this cycle
2016-12-09 14:00:06.966 8428 INFO ceilometer.agent.manager [-] Skip pollster disk.device.write.requests, no resources found this cycle
2016-12-09 14:00:06.966 8428 INFO ceilometer.agent.manager [-] Skip pollster disk.device.read.requests, no resources found this cycle
2016-12-09 14:00:06.966 8428 INFO ceilometer.agent.manager [-] Skip pollster disk.write.bytes, no resources found this cycle
2016-12-09 14:00:06.967 8428 INFO ceilometer.agent.manager [-] Skip pollster disk.device.write.bytes, no resources found this cycle
2016-12-09 14:00:06.967 8428 INFO ceilometer.agent.manager [-] Skip pollster disk.read.bytes, no resources found this cycle
2016-12-09 14:00:06.968 8428 INFO ceilometer.agent.manager [-] Skip pollster disk.write.requests, no resources found this cycle
2016-12-09 14:00:06.968 8428 INFO ceilometer.agent.manager [-] Skip pollster cpu, no resources found this cycle
2016-12-09 14:00:06.968 8428 INFO ceilometer.agent.manager [-] Skip pollster network.incoming.bytes, no resources found this cycle
2016-12-09 14:00:06.969 8428 INFO ceilometer.agent.manager [-] Skip pollster network.outgoing.packets, no resources found this cycle
2016-12-09 14:00:06.969 8428 INFO ceilometer.agent.manager [-] Skip pollster network.incoming.packets, no resources found this cycle
2016-12-09 14:00:06.969 8428 INFO ceilometer.agent.manager [-] Skip pollster network.outgoing.bytes, no resources found this cycle

Comment 16 chenruoyu111 2016-12-12 16:25:17 UTC

Hi Yurii,

Do you need other information or have idea how could we fix it? Thank you!!

Comment 17 chenruoyu111 2016-12-12 21:00:36 UTC

Solved ceilometer bug finally. The reason is that the information in "service_credentials" part of ceilometer.conf is wrong.

In ceilometer.conf file "service_credentials" part, I wrongly configured os_username, os_password, os_tenant_name, os_auth_url to be ceilometer service credentials. Which should be openstack services credentials that could access all services. That's why ceilometer cannot talk to nova api. The credentials is wrong.

Now we could get cpu, memory and disk usage data from ceilometer CLI. And the error disappears.

Thanks!!

Note You need to log in before you can comment on or make changes to this bug.