Description of problem: RHOSP11 is configured with ~~~/etc/ceilomter/ceilometer.conf meter_dispatchers=gnocchi event_dispatchers=panko # also tried with event_dispatchers=database store_events=True ~~~ Ensure gnocchi resource is created with cpu_util, memory.* and other metric Error: ~~~ [----] I, [2017-12-28T01:35:49.155716 #21969:dd1134] INFO -- : MIQ(Metric::Capture.perf_capture_timer) Queueing performance capture... [----] I, [2017-12-28T01:35:49.161534 #21969:dd1134] INFO -- : MIQ(Metric::Capture.perf_capture_health_check) 33 "realtime" captures on the queue for zone [default] - oldest: [2017-12-28T06:26:48Z], recent: [2017-12-28T06:26:49Z] [----] I, [2017-12-28T01:35:49.161968 #21969:dd1134] INFO -- : MIQ(Metric::Capture.perf_capture_health_check) 0 "hourly" captures on the queue for zone [default] [----] I, [2017-12-28T01:35:49.162076 #21969:dd1134] INFO -- : MIQ(Metric::Capture.perf_capture_health_check) 0 "historical" captures on the queue for zone [default] [----] I, [2017-12-28T01:35:51.436588 #21969:dd1134] INFO -- : MIQ(MiqQueue.put) Message id: [78000006293922], id: [], Zone: [default], Role: [ems_metrics_collector], Server: [], Ident: [openstack], Target id: [], Instance id: [78000000000342], Task id: [], Command: [ManageIQ::Providers::Openstack::CloudManager::Vm.perf_capture_realtime], Timeout: [600], Priority: [100], State: [ready], Deliver On: [], Data: [], Args: [2017-12-28 00:00:00 UTC] [----] I, [2017-12-28T01:35:51.652626 #21969:dd1134] INFO -- : MIQ(Metric::Capture.perf_capture_timer) Queueing performance capture...Complete /var/www/miq/vmdb/app/models/manageiq/providers/base_manager/event_catcher/runner.rb:122:in `monitor_events': must be implemented in subclass (NotImplementedError) [----] I, [2017-12-28T01:36:01.361992 #21906:dd1134] INFO -- : MIQ(ManageIQ::Providers::Openstack::CloudManager::MetricsCollectorWorker::Runner#get_message_via_drb) Message id: [78000006293922], MiqWorker id: [78000000441240], Zone: [default], Role: [ems_metrics_collector], Server: [], Ident: [openstack], Target id: [], Instance id: [78000000000342], Task id: [], Command: [ManageIQ::Providers::Openstack::CloudManager::Vm.perf_capture_realtime], Timeout: [600], Priority: [100], State: [dequeue], Deliver On: [], Data: [], Args: [2017-12-28 00:00:00 UTC], Dequeued in: [9.931035545] seconds [----] I, [2017-12-28T01:36:01.374833 #21906:dd1134] INFO -- : MIQ(ManageIQ::Providers::Openstack::CloudManager::Vm#perf_capture) [realtime] Capture for ManageIQ::Providers::Openstack::CloudManager::Vm name: [i2], id: [78000000000342], start_time: [2017-12-28 00:00:00 UTC]... [----] E, [2017-12-28T01:36:02.299604 #21906:dd1134] ERROR -- : <Fog> excon.error #<Excon::Error::NotFound: Expected(200) <=> Actual(404 Not Found) excon.error.response [----] E, [2017-12-28T01:36:02.300172 #21906:dd1134] ERROR -- : MIQ(ManageIQ::Providers::Openstack::CloudManager::MetricsCapture#perf_collect_metrics) [realtime] for: [ManageIQ::Providers::Openstack::CloudManager::Vm], [78000000000342], [i2] Unhandled exception during perf data collection: [Expected(200) <=> Actual(404 Not Found) excon.error.response [----] E, [2017-12-28T01:36:02.300351 #21906:dd1134] ERROR -- : MIQ(ManageIQ::Providers::Openstack::CloudManager::MetricsCapture#perf_collect_metrics) [realtime] for: [ManageIQ::Providers::Openstack::CloudManager::Vm], [78000000000342], [i2] Timings at time of error: {:capture_state=>0.004530668258666992, :connect=>0.5510125160217285, :capture_counters=>0.2667701244354248, :capture_counter_values=>0.0210726261138916} [----] E, [2017-12-28T01:36:02.300804 #21906:dd1134] ERROR -- : [Fog::Metric::OpenStack::NotFound]: Expected(200) <=> Actual(404 Not Found) excon.error.response [----] E, [2017-12-28T01:36:02.301003 #21906:dd1134] ERROR -- : /opt/rh/cfme-gemset/gems/excon-0.55.0/lib/excon/middlewares/expects.rb:7:in `response_call' /var/www/miq/vmdb/app/models/manageiq/providers/openstack/base_metrics_capture.rb:69:in `block in perf_capture_data_openstack_base' /var/www/miq/vmdb/app/models/manageiq/providers/openstack/base_metrics_capture.rb:68:in `perf_capture_data_openstack_base' /var/www/miq/vmdb/app/models/manageiq/providers/openstack/cloud_manager/metrics_capture.rb:98:in `perf_capture_data' /var/www/miq/vmdb/app/models/metric/ci_mixin/capture.rb:180:in `block in perf_capture' /var/www/miq/vmdb/app/models/metric/ci_mixin/capture.rb:177:in `perf_capture' /var/www/miq/vmdb/app/models/metric/ci_mixin/capture.rb:117:in `perf_capture_realtime' [----] E, [2017-12-28T01:36:02.301349 #21906:dd1134] ERROR -- : MIQ(MiqQueue#deliver) Message id: [78000006293922], Error: [Expected(200) <=> Actual(404 Not Found) excon.error.response [----] E, [2017-12-28T01:36:02.301655 #21906:dd1134] ERROR -- : [Fog::Metric::OpenStack::NotFound]: Expected(200) <=> Actual(404 Not Found) excon.error.response [----] E, [2017-12-28T01:36:02.301805 #21906:dd1134] ERROR -- : /opt/rh/cfme-gemset/gems/excon-0.55.0/lib/excon/middlewares/expects.rb:7:in `response_call' /var/www/miq/vmdb/app/models/manageiq/providers/openstack/base_metrics_capture.rb:69:in `block in perf_capture_data_openstack_base' /var/www/miq/vmdb/app/models/manageiq/providers/openstack/base_metrics_capture.rb:68:in `perf_capture_data_openstack_base' /var/www/miq/vmdb/app/models/manageiq/providers/openstack/cloud_manager/metrics_capture.rb:98:in `perf_capture_data' /var/www/miq/vmdb/app/models/metric/ci_mixin/capture.rb:180:in `block in perf_capture' /var/www/miq/vmdb/app/models/metric/ci_mixin/capture.rb:177:in `perf_capture' /var/www/miq/vmdb/app/models/metric/ci_mixin/capture.rb:117:in `perf_capture_realtime' [----] I, [2017-12-28T01:36:02.302040 #21906:dd1134] INFO -- : MIQ(MiqQueue#delivered) Message id: [78000006293922], State: [error], Delivered in [0.939756596] seconds ~~~ Version-Release number of selected component (if applicable): - CFME-5.8.0.17 - CFME-5.8.2.3 How reproducible: Always Steps to Reproduce: 1. Ensure metrics are been collected for instance on RHOSP11 2. Enable C & U 3. Enable "Collect for All Cluster" & "Collect for All Datastore" in region setting Actual results: "Utilization" is greyed under "Monitoring" tab in instance summary page Expected results: Additional info:
Related BZ#1425947
Update: I see more (specific) error after leaving env for ~2 hrs. ~~~ [----] E, [2017-12-28T04:32:03.316456 #13573:ff512c] ERROR -- : MIQ(MiqQueue#deliver) Message id: [78000006298424], Error: [Expected(200) <=> Actual(404 Not Found) excon.error.response :body => "{\"code\": 404, \"description\": \"Granularity '300.0' for metric b0c2219b-4b42-44bf-be80-b079d5b31ee3 does not exist\", \"title\": \"Not Found\"}" :cookies => [ ] :headers => { "Connection" => "close" "Content-Length" => "136" "Content-Type" => "application/json; charset=UTF-8" "Date" => "Thu, 28 Dec 2017 04:02:01 GMT" "Server" => "Apache/2.4.6 (Red Hat Enterprise Linux)" } :host => "IPADDRESS" :local_address => "ipaddress" :local_port => 38528 :path => "/v1/resource/generic/3df21098-37ef-47bc-bf07-4bdcc5d4c160/metric/disk.write.bytes/measures" :port => 8041 :reason_phrase => "Not Found" :remote_ip => "IPADDRES" :status => 404 :status_line => "HTTP/1.1 404 Not Found\r\n" ] [----] E, [2017-12-28T04:32:03.316658 #13573:ff512c] ERROR -- : [Fog::Metric::OpenStack::NotFound]: Expected(200) <=> Actual(404 Not Found) excon.error.response :body => "{\"code\": 404, \"description\": \"Granularity '300.0' for metric b0c2219b-4b42-44bf-be80-b079d5b31ee3 does not exist\", \"title\": \"Not Found\"}" :cookies => [ ] :headers => { "Connection" => "close" "Content-Length" => "136" "Content-Type" => "application/json; charset=UTF-8" "Date" => "Thu, 28 Dec 2017 04:02:01 GMT" "Server" => "Apache/2.4.6 (Red Hat Enterprise Linux)" } :host => "10.X.X.X" :local_address => "10.X.X.X" :local_port => 38528 :path => "/v1/resource/generic/3df21098-37ef-47bc-bf07-4bdcc5d4c160/metric/disk.write.bytes/measures" :port => 8041 :reason_phrase => "Not Found" :remote_ip => "10.X.X.X" :status => 404 :status_line => "HTTP/1.1 404 Not Found\r\n" Method:[rescue in deliver] [----] E, [2017-12-28T04:32:03.316758 #13573:ff512c] ERROR -- : /opt/rh/cfme-gemset/gems/excon-0.55.0/lib/excon/middlewares/expects.rb:7:in `response_call' ~~~ ~~~ # openstack metric metric list -c id -c name O/P truncated | b0c2219b-4b42-44bf-be80-b079d5b31ee3 | disk.write.bytes | ~~~ But metrics disk.write.bytes don't have granularity of 300, I see granularity of 3600, 60 & 1
The 300.0s Gnocchi default granularity is actually hard-coded [1]. If it is needed to change it, we can make it configurable. For quick demo, it should be possible change it manually i source code on the appliance (part of manageiq-gems-pending gem) or adjust Customer Gnocchi config to use 300.0 granularity. [1] https://github.com/ManageIQ/manageiq-gems-pending/blob/fine/lib/gems/pending/openstack/openstack_handle/metric_delegate.rb#L37
Adding a PR with possibility configure gnocchi granularity in Settings https://github.com/ManageIQ/manageiq-providers-openstack/pull/267