Bug 1529449
| Summary: | RHOSP11 metric collection stuck with error: Fog::Metric::OpenStack::NotFound | |||
|---|---|---|---|---|
| Product: | Red Hat CloudForms Management Engine | Reporter: | Sachin <sacpatil> | |
| Component: | C&U Capacity and Utilization | Assignee: | Marek Aufart <maufart> | |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Jad Haj Yahya <jhajyahy> | |
| Severity: | high | Docs Contact: | ||
| Priority: | medium | |||
| Version: | 5.8.0 | CC: | cpelland, jhajyahy, maufart, niroy, obarenbo, simaishi, slucidi | |
| Target Milestone: | GA | Keywords: | TestOnly, ZStream | |
| Target Release: | 5.10.0 | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | 5.10.0.0 | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1578388 1578391 1578575 (view as bug list) | Environment: | ||
| Last Closed: | 2018-08-01 02:49:20 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | Openstack | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1578388, 1578391, 1578575 | |||
Related BZ#1425947 Update:
I see more (specific) error after leaving env for ~2 hrs.
~~~
[----] E, [2017-12-28T04:32:03.316456 #13573:ff512c] ERROR -- : MIQ(MiqQueue#deliver) Message id: [78000006298424], Error: [Expected(200) <=> Actual(404 Not Found)
excon.error.response
:body => "{\"code\": 404, \"description\": \"Granularity '300.0' for metric b0c2219b-4b42-44bf-be80-b079d5b31ee3 does not exist\", \"title\": \"Not Found\"}"
:cookies => [
]
:headers => {
"Connection" => "close"
"Content-Length" => "136"
"Content-Type" => "application/json; charset=UTF-8"
"Date" => "Thu, 28 Dec 2017 04:02:01 GMT"
"Server" => "Apache/2.4.6 (Red Hat Enterprise Linux)"
}
:host => "IPADDRESS"
:local_address => "ipaddress"
:local_port => 38528
:path => "/v1/resource/generic/3df21098-37ef-47bc-bf07-4bdcc5d4c160/metric/disk.write.bytes/measures"
:port => 8041
:reason_phrase => "Not Found"
:remote_ip => "IPADDRES"
:status => 404
:status_line => "HTTP/1.1 404 Not Found\r\n"
]
[----] E, [2017-12-28T04:32:03.316658 #13573:ff512c] ERROR -- : [Fog::Metric::OpenStack::NotFound]: Expected(200) <=> Actual(404 Not Found)
excon.error.response
:body => "{\"code\": 404, \"description\": \"Granularity '300.0' for metric b0c2219b-4b42-44bf-be80-b079d5b31ee3 does not exist\", \"title\": \"Not Found\"}"
:cookies => [
]
:headers => {
"Connection" => "close"
"Content-Length" => "136"
"Content-Type" => "application/json; charset=UTF-8"
"Date" => "Thu, 28 Dec 2017 04:02:01 GMT"
"Server" => "Apache/2.4.6 (Red Hat Enterprise Linux)"
}
:host => "10.X.X.X"
:local_address => "10.X.X.X"
:local_port => 38528
:path => "/v1/resource/generic/3df21098-37ef-47bc-bf07-4bdcc5d4c160/metric/disk.write.bytes/measures"
:port => 8041
:reason_phrase => "Not Found"
:remote_ip => "10.X.X.X"
:status => 404
:status_line => "HTTP/1.1 404 Not Found\r\n"
Method:[rescue in deliver]
[----] E, [2017-12-28T04:32:03.316758 #13573:ff512c] ERROR -- : /opt/rh/cfme-gemset/gems/excon-0.55.0/lib/excon/middlewares/expects.rb:7:in `response_call'
~~~
~~~
# openstack metric metric list -c id -c name
O/P truncated
| b0c2219b-4b42-44bf-be80-b079d5b31ee3 | disk.write.bytes |
~~~
But metrics disk.write.bytes don't have granularity of 300, I see granularity of 3600, 60 & 1
The 300.0s Gnocchi default granularity is actually hard-coded [1]. If it is needed to change it, we can make it configurable. For quick demo, it should be possible change it manually i source code on the appliance (part of manageiq-gems-pending gem) or adjust Customer Gnocchi config to use 300.0 granularity. [1] https://github.com/ManageIQ/manageiq-gems-pending/blob/fine/lib/gems/pending/openstack/openstack_handle/metric_delegate.rb#L37 Adding a PR with possibility configure gnocchi granularity in Settings https://github.com/ManageIQ/manageiq-providers-openstack/pull/267 |
Description of problem: RHOSP11 is configured with ~~~/etc/ceilomter/ceilometer.conf meter_dispatchers=gnocchi event_dispatchers=panko # also tried with event_dispatchers=database store_events=True ~~~ Ensure gnocchi resource is created with cpu_util, memory.* and other metric Error: ~~~ [----] I, [2017-12-28T01:35:49.155716 #21969:dd1134] INFO -- : MIQ(Metric::Capture.perf_capture_timer) Queueing performance capture... [----] I, [2017-12-28T01:35:49.161534 #21969:dd1134] INFO -- : MIQ(Metric::Capture.perf_capture_health_check) 33 "realtime" captures on the queue for zone [default] - oldest: [2017-12-28T06:26:48Z], recent: [2017-12-28T06:26:49Z] [----] I, [2017-12-28T01:35:49.161968 #21969:dd1134] INFO -- : MIQ(Metric::Capture.perf_capture_health_check) 0 "hourly" captures on the queue for zone [default] [----] I, [2017-12-28T01:35:49.162076 #21969:dd1134] INFO -- : MIQ(Metric::Capture.perf_capture_health_check) 0 "historical" captures on the queue for zone [default] [----] I, [2017-12-28T01:35:51.436588 #21969:dd1134] INFO -- : MIQ(MiqQueue.put) Message id: [78000006293922], id: [], Zone: [default], Role: [ems_metrics_collector], Server: [], Ident: [openstack], Target id: [], Instance id: [78000000000342], Task id: [], Command: [ManageIQ::Providers::Openstack::CloudManager::Vm.perf_capture_realtime], Timeout: [600], Priority: [100], State: [ready], Deliver On: [], Data: [], Args: [2017-12-28 00:00:00 UTC] [----] I, [2017-12-28T01:35:51.652626 #21969:dd1134] INFO -- : MIQ(Metric::Capture.perf_capture_timer) Queueing performance capture...Complete /var/www/miq/vmdb/app/models/manageiq/providers/base_manager/event_catcher/runner.rb:122:in `monitor_events': must be implemented in subclass (NotImplementedError) [----] I, [2017-12-28T01:36:01.361992 #21906:dd1134] INFO -- : MIQ(ManageIQ::Providers::Openstack::CloudManager::MetricsCollectorWorker::Runner#get_message_via_drb) Message id: [78000006293922], MiqWorker id: [78000000441240], Zone: [default], Role: [ems_metrics_collector], Server: [], Ident: [openstack], Target id: [], Instance id: [78000000000342], Task id: [], Command: [ManageIQ::Providers::Openstack::CloudManager::Vm.perf_capture_realtime], Timeout: [600], Priority: [100], State: [dequeue], Deliver On: [], Data: [], Args: [2017-12-28 00:00:00 UTC], Dequeued in: [9.931035545] seconds [----] I, [2017-12-28T01:36:01.374833 #21906:dd1134] INFO -- : MIQ(ManageIQ::Providers::Openstack::CloudManager::Vm#perf_capture) [realtime] Capture for ManageIQ::Providers::Openstack::CloudManager::Vm name: [i2], id: [78000000000342], start_time: [2017-12-28 00:00:00 UTC]... [----] E, [2017-12-28T01:36:02.299604 #21906:dd1134] ERROR -- : <Fog> excon.error #<Excon::Error::NotFound: Expected(200) <=> Actual(404 Not Found) excon.error.response [----] E, [2017-12-28T01:36:02.300172 #21906:dd1134] ERROR -- : MIQ(ManageIQ::Providers::Openstack::CloudManager::MetricsCapture#perf_collect_metrics) [realtime] for: [ManageIQ::Providers::Openstack::CloudManager::Vm], [78000000000342], [i2] Unhandled exception during perf data collection: [Expected(200) <=> Actual(404 Not Found) excon.error.response [----] E, [2017-12-28T01:36:02.300351 #21906:dd1134] ERROR -- : MIQ(ManageIQ::Providers::Openstack::CloudManager::MetricsCapture#perf_collect_metrics) [realtime] for: [ManageIQ::Providers::Openstack::CloudManager::Vm], [78000000000342], [i2] Timings at time of error: {:capture_state=>0.004530668258666992, :connect=>0.5510125160217285, :capture_counters=>0.2667701244354248, :capture_counter_values=>0.0210726261138916} [----] E, [2017-12-28T01:36:02.300804 #21906:dd1134] ERROR -- : [Fog::Metric::OpenStack::NotFound]: Expected(200) <=> Actual(404 Not Found) excon.error.response [----] E, [2017-12-28T01:36:02.301003 #21906:dd1134] ERROR -- : /opt/rh/cfme-gemset/gems/excon-0.55.0/lib/excon/middlewares/expects.rb:7:in `response_call' /var/www/miq/vmdb/app/models/manageiq/providers/openstack/base_metrics_capture.rb:69:in `block in perf_capture_data_openstack_base' /var/www/miq/vmdb/app/models/manageiq/providers/openstack/base_metrics_capture.rb:68:in `perf_capture_data_openstack_base' /var/www/miq/vmdb/app/models/manageiq/providers/openstack/cloud_manager/metrics_capture.rb:98:in `perf_capture_data' /var/www/miq/vmdb/app/models/metric/ci_mixin/capture.rb:180:in `block in perf_capture' /var/www/miq/vmdb/app/models/metric/ci_mixin/capture.rb:177:in `perf_capture' /var/www/miq/vmdb/app/models/metric/ci_mixin/capture.rb:117:in `perf_capture_realtime' [----] E, [2017-12-28T01:36:02.301349 #21906:dd1134] ERROR -- : MIQ(MiqQueue#deliver) Message id: [78000006293922], Error: [Expected(200) <=> Actual(404 Not Found) excon.error.response [----] E, [2017-12-28T01:36:02.301655 #21906:dd1134] ERROR -- : [Fog::Metric::OpenStack::NotFound]: Expected(200) <=> Actual(404 Not Found) excon.error.response [----] E, [2017-12-28T01:36:02.301805 #21906:dd1134] ERROR -- : /opt/rh/cfme-gemset/gems/excon-0.55.0/lib/excon/middlewares/expects.rb:7:in `response_call' /var/www/miq/vmdb/app/models/manageiq/providers/openstack/base_metrics_capture.rb:69:in `block in perf_capture_data_openstack_base' /var/www/miq/vmdb/app/models/manageiq/providers/openstack/base_metrics_capture.rb:68:in `perf_capture_data_openstack_base' /var/www/miq/vmdb/app/models/manageiq/providers/openstack/cloud_manager/metrics_capture.rb:98:in `perf_capture_data' /var/www/miq/vmdb/app/models/metric/ci_mixin/capture.rb:180:in `block in perf_capture' /var/www/miq/vmdb/app/models/metric/ci_mixin/capture.rb:177:in `perf_capture' /var/www/miq/vmdb/app/models/metric/ci_mixin/capture.rb:117:in `perf_capture_realtime' [----] I, [2017-12-28T01:36:02.302040 #21906:dd1134] INFO -- : MIQ(MiqQueue#delivered) Message id: [78000006293922], State: [error], Delivered in [0.939756596] seconds ~~~ Version-Release number of selected component (if applicable): - CFME-5.8.0.17 - CFME-5.8.2.3 How reproducible: Always Steps to Reproduce: 1. Ensure metrics are been collected for instance on RHOSP11 2. Enable C & U 3. Enable "Collect for All Cluster" & "Collect for All Datastore" in region setting Actual results: "Utilization" is greyed under "Monitoring" tab in instance summary page Expected results: Additional info: