Created attachment 1211951 [details] Ceilometer Compute log Description of problem: Errors in ceilometer-compute log indicating several pollsters can not be retrieved. Error Messages: [root@overcloud-novacompute-0 log]# grep " Error " -i ceilometer/compute.log | egrep "Could not" 2016-10-19 01:47:45.009 41134 ERROR ceilometer.compute.pollsters.cpu [-] Could not get cache usage for ad0c1b21-077f-4e66-a243-44e436a6208e: 'perf.cmt' 2016-10-19 01:47:45.048 41134 ERROR ceilometer.compute.pollsters.perf [-] Could not get perf events for ad0c1b21-077f-4e66-a243-44e436a6208e: 'perf.cpu_cycles' 2016-10-19 01:47:45.069 41134 ERROR ceilometer.compute.pollsters.perf [-] Could not get perf events for ad0c1b21-077f-4e66-a243-44e436a6208e: 'perf.cpu_cycles' 2016-10-19 01:47:45.072 41134 ERROR ceilometer.compute.pollsters.perf [-] Could not get perf events for ad0c1b21-077f-4e66-a243-44e436a6208e: 'perf.cpu_cycles' 2016-10-19 01:47:45.076 41134 ERROR ceilometer.compute.pollsters.perf [-] Could not get perf events for ad0c1b21-077f-4e66-a243-44e436a6208e: 'perf.cpu_cycles' 2016-10-19 01:47:45.127 41134 ERROR ceilometer.compute.pollsters.memory [-] Could not get memory bandwidth for ad0c1b21-077f-4e66-a243-44e436a6208e: 'perf.mbmt' 2016-10-19 01:47:45.154 41134 ERROR ceilometer.compute.pollsters.memory [-] Could not get memory bandwidth for ad0c1b21-077f-4e66-a243-44e436a6208e: 'perf.mbmt' Version-Release number of selected component (if applicable): Director deployed Red Hat OpenStack 10 (Newton) Build 2016-10-06.1 openstack-ceilometer-compute-7.0.0-0.20160928024313.67bbd3f.el7ost.noarch How reproducible: On Build 2016-10-06.1 (Latest available passed_phase1) Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: All error messages are for a single instance. There is several other errors in the beginning of the log however the above highlighted ones are repeated on the polling interval.
*** This bug has been marked as a duplicate of bug 1383306 ***
Seems not exactly the same bug yprokule have reproduce this with ceilometer 7.0.0-2.2 package. We need to double check this particular case.
Next traceback present in ceilometer/compute.log ... 2016-11-08 10:24:21.954 26314 DEBUG ceilometer.compute.pollsters.cpu [-] checking cache usage for instance 58f150d8-0b0f-43e6-8b3f-30706ed71143 get_samples /usr/lib/python2.7/site-packages/ceilometer/compute/pollsters/cpu.py:103 2016-11-08 10:24:21.955 26314 ERROR ceilometer.compute.pollsters.cpu [-] Could not get cache usage for 58f150d8-0b0f-43e6-8b3f-30706ed71143: 'perf.cmt' 2016-11-08 10:24:21.955 26314 ERROR ceilometer.compute.pollsters.cpu Traceback (most recent call last): 2016-11-08 10:24:21.955 26314 ERROR ceilometer.compute.pollsters.cpu File "/usr/lib/python2.7/site-packages/ceilometer/compute/pollsters/cpu.py", line 105, in get_samples 2016-11-08 10:24:21.955 26314 ERROR ceilometer.compute.pollsters.cpu cpu_cache = self.inspector.inspect_cpu_l3_cache(instance) 2016-11-08 10:24:21.955 26314 ERROR ceilometer.compute.pollsters.cpu File "/usr/lib/python2.7/site-packages/ceilometer/compute/virt/libvirt/inspector.py", line 125, in inspect_cpu_l3_cache 2016-11-08 10:24:21.955 26314 ERROR ceilometer.compute.pollsters.cpu usage = perf["perf.cmt"] 2016-11-08 10:24:21.955 26314 ERROR ceilometer.compute.pollsters.cpu KeyError: 'perf.cmt' 2016-11-08 10:24:21.955 26314 ERROR ceilometer.compute.pollsters.cpu
Looking at OSPd 10 Newton (Build 2016-11-04.2) There is still some errors in compute.log on the compute nodes: 2016-11-16 19:44:09.822 10561 ERROR ceilometer.agent.manager [-] Prevent pollster network.outgoing.bytes.rate for polling source meter_source anymore! 2016-11-16 19:44:09.838 10561 ERROR ceilometer.agent.manager [-] Prevent pollster perf.cache.misses for polling source meter_source anymore! 2016-11-16 19:44:09.839 10561 ERROR ceilometer.agent.manager [-] Prevent pollster disk.iops for polling source meter_source anymore! 2016-11-16 19:44:09.840 10561 ERROR ceilometer.agent.manager [-] Prevent pollster disk.device.read.requests.rate for polling source meter_source anymore! 2016-11-16 19:44:09.842 10561 ERROR ceilometer.agent.manager [-] Prevent pollster perf.cpu.cycles for polling source meter_source anymore! 2016-11-16 19:44:09.856 10561 ERROR ceilometer.agent.manager [-] Prevent pollster disk.write.requests.rate for polling source meter_source anymore! 2016-11-16 19:44:09.871 10561 ERROR ceilometer.compute.pollsters.cpu [-] Could not get cache usage for 5423861f-e68d-44d3-a5a2-86494d203bb5: 'perf.cmt' 2016-11-16 19:44:09.871 10561 ERROR ceilometer.compute.pollsters.cpu Traceback (most recent call last): 2016-11-16 19:44:09.871 10561 ERROR ceilometer.compute.pollsters.cpu File "/usr/lib/python2.7/site-packages/ceilometer/compute/pollsters/cpu.py", line 105, in get_samples 2016-11-16 19:44:09.871 10561 ERROR ceilometer.compute.pollsters.cpu cpu_cache = self.inspector.inspect_cpu_l3_cache(instance) 2016-11-16 19:44:09.871 10561 ERROR ceilometer.compute.pollsters.cpu File "/usr/lib/python2.7/site-packages/ceilometer/compute/virt/libvirt/inspector.py", line 125, in inspect_cpu_l3_cache 2016-11-16 19:44:09.871 10561 ERROR ceilometer.compute.pollsters.cpu usage = perf["perf.cmt"] 2016-11-16 19:44:09.871 10561 ERROR ceilometer.compute.pollsters.cpu KeyError: 'perf.cmt' 2016-11-16 19:44:09.871 10561 ERROR ceilometer.compute.pollsters.cpu
According to our records, this should be resolved by openstack-ceilometer-7.0.1-1.el7ost. This build is available now.
No more pollster errors
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:1586