Bug 1420721 - Unable to compute performance rollups for OpenShift
Summary: Unable to compute performance rollups for OpenShift
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: Providers
Version: 5.7.0
Hardware: All
OS: All
urgent
high
Target Milestone: GA
: 5.8.0
Assignee: Ari Zellner
QA Contact: Einat Pacifici
URL:
Whiteboard: container
: 1408968 1429626 (view as bug list)
Depends On:
Blocks: 1426683
TreeView+ depends on / blocked
 
Reported: 2017-02-09 11:52 UTC by Felix Dewaleyne
Modified: 2020-06-11 13:17 UTC (History)
10 users (show)

Fixed In Version: 5.8.0.3
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1426683 (view as bug list)
Environment:
Last Closed: 2017-06-12 16:53:16 UTC
Category: ---
Cloudforms Team: Container Management
Target Upstream Version:
Embargoed:
epacific: automate_bug+


Attachments (Terms of Use)

Description Felix Dewaleyne 2017-02-09 11:52:35 UTC
Description of problem:
Unable to capture the perf roll up from kubernates with the described error

Version-Release number of selected component (if applicable):
cfme-5.7.0.17-1

How reproducible:


Steps to Reproduce:
1. add kubernates provider to cloudforms
2.
3.

Actual results:

[----] I, [2017-01-25T16:00:29.122387 #3495:f57134]  INFO -- : MIQ(ManageIQ::Providers::Kubernetes::ContainerManager::Container#perf_rollup_to_parents) Queueing [daily] rollup to ManageIQ::Providers::Kubernetes::ContainerManager::Container id: [352] in time profile: [UTC] for times: ["2017-01-25T00:00:00Z"]...
[----] E, [2017-01-25T16:00:29.123473 #3495:f57134] ERROR -- : MIQ(MiqQueue#deliver) Message id: [1496012], Error: [Container#my_zone delegated to ext_management_system.my_zone, but ext_management_system is nil: #<ManageIQ::Providers::Kubernetes::ContainerManager::Container id: 352, ems_ref: "3df46e85-e308-11e6-8207-005056997454_jenkins_10.6....", restart_count: 0, state: "running", name: "jenkins", backing_ref: "docker://baae19fb6ed4db3ca0803333f7dc9730e87e40167...", last_perf_capture_on: "2017-01-25 14:27:40", container_definition_id: 352, type: "ManageIQ::Providers::Kubernetes::ContainerManager:...", container_image_id: 224, reason: nil, started_at: nil, finished_at: nil, exit_code: nil, signal: nil, message: nil, last_state: nil, last_reason: nil, last_started_at: nil, last_finished_at: nil, last_exit_code: nil, last_signal: nil, last_message: nil, deleted_on: "2017-01-25 14:39:03", ems_id: nil, old_ems_id: 1>]
[----] E, [2017-01-25T16:00:29.123658 #3495:f57134] ERROR -- : [Module::DelegationError]: Container#my_zone delegated to ext_management_system.my_zone, but ext_management_system is nil: #<ManageIQ::Providers::Kubernetes::ContainerManager:
:Container id: 352, ems_ref: "3df46e85-e308-11e6-8207-005056997454_jenkins_10.6....", restart_count: 0, state: "running", name: "jenkins", backing_ref: "docker://baae19fb6ed4db3ca0803333f7dc9730e87e40167...", last_perf_capture_on: "2017-01-25 14:27:40", container_definition_id: 352, type: "ManageIQ::Providers::Kubernetes::ContainerManager:...", container_image_id: 224, reason: nil, started_at: nil, finished_at: nil, exit_code: nil, signal: nil, message: nil, last_state: nil, last_reason: nil, last_started_at: nil, last_finished_at: nil, last_exit_code: nil, last_signal: nil, last_message: nil, deleted_on: "2017-01-25 14:39:03", ems_id: nil, old_ems_id: 1>  Method:[rescue in deliver]
[----] E, [2017-01-25T16:00:29.123865 #3495:f57134] ERROR -- : /var/www/miq/vmdb/app/models/container.rb:21:in `rescue in my_zone'
/var/www/miq/vmdb/app/models/container.rb:21:in `my_zone'
/var/www/miq/vmdb/app/models/metric/ci_mixin/rollup.rb:62:in `perf_rollup_queue'
/var/www/miq/vmdb/app/models/metric/ci_mixin/rollup.rb:30:in `block (3 levels) in perf_rollup_to_parents'
/var/www/miq/vmdb/app/models/metric/ci_mixin/rollup.rb:30:in `each'
/var/www/miq/vmdb/app/models/metric/ci_mixin/rollup.rb:30:in `block (2 levels) in perf_rollup_to_parents'
/var/www/miq/vmdb/app/models/metric/ci_mixin/rollup.rb:27:in `each'
/var/www/miq/vmdb/app/models/metric/ci_mixin/rollup.rb:27:in `block in perf_rollup_to_parents'
/var/www/miq/vmdb/app/models/metric/ci_mixin/rollup.rb:14:in `each'
/var/www/miq/vmdb/app/models/metric/ci_mixin/rollup.rb:14:in `each_slice'
/var/www/miq/vmdb/app/models/metric/ci_mixin/rollup.rb:14:in `perf_rollup_to_parents'
/var/www/miq/vmdb/app/models/metric/ci_mixin/rollup.rb:104:in `block in perf_rollup'
/var/www/miq/vmdb/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store'
/var/www/miq/vmdb/gems/pending/util/extensions/miq-benchmark.rb:30:in `realtime_block'
/var/www/miq/vmdb/app/models/metric/ci_mixin/rollup.rb:82:in `perf_rollup'
/var/www/miq/vmdb/app/models/miq_queue.rb:347:in `block in deliver'
/opt/rh/rh-ruby23/root/usr/share/ruby/timeout.rb:91:in `block in timeout'
/opt/rh/rh-ruby23/root/usr/share/ruby/timeout.rb:33:in `block in catch'
/opt/rh/rh-ruby23/root/usr/share/ruby/timeout.rb:33:in `catch'
/opt/rh/rh-ruby23/root/usr/share/ruby/timeout.rb:33:in `catch'
/opt/rh/rh-ruby23/root/usr/share/ruby/timeout.rb:106:in `timeout'
/var/www/miq/vmdb/app/models/miq_queue.rb:343:in `deliver'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:106:in `deliver_queue_message'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:134:in `deliver_message'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:152:in `block in do_work'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:146:in `loop'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:146:in `do_work'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:334:in `block in do_work_loop'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:331:in `loop'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:331:in `do_work_loop'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:153:in `run'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:128:in `start'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:21:in `start_worker'
/var/www/miq/vmdb/app/models/miq_worker.rb:343:in `block in start'
/opt/rh/cfme-gemset/gems/nakayoshi_fork-0.0.3/lib/nakayoshi_fork.rb:24:in `fork'
/opt/rh/cfme-gemset/gems/nakayoshi_fork-0.0.3/lib/nakayoshi_fork.rb:24:in `fork'
/var/www/miq/vmdb/app/models/miq_worker.rb:341:in `start'
/var/www/miq/vmdb/app/models/miq_worker.rb:270:in `start_worker'
/var/www/miq/vmdb/app/models/miq_worker.rb:150:in `block in sync_workers'
/var/www/miq/vmdb/app/models/miq_worker.rb:150:in `times'
/var/www/miq/vmdb/app/models/miq_worker.rb:150:in `sync_workers'
/var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:52:in `block in sync_workers'
/var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:50:in `each'
/var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:50:in `sync_workers'
/var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:22:in `monitor_workers'
/var/www/miq/vmdb/app/models/miq_server.rb:346:in `block in monitor'
/var/www/miq/vmdb/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store'
/var/www/miq/vmdb/gems/pending/util/extensions/miq-benchmark.rb:30:in `realtime_block'
/var/www/miq/vmdb/app/models/miq_server.rb:346:in `monitor'
/var/www/miq/vmdb/app/models/miq_server.rb:368:in `block (2 levels) in monitor_loop'
/var/www/miq/vmdb/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store'
/var/www/miq/vmdb/gems/pending/util/extensions/miq-benchmark.rb:30:in `realtime_block'
/var/www/miq/vmdb/app/models/miq_server.rb:368:in `block in monitor_loop'
/var/www/miq/vmdb/app/models/miq_server.rb:367:in `loop'
/var/www/miq/vmdb/app/models/miq_server.rb:367:in `monitor_loop'
/var/www/miq/vmdb/app/models/miq_server.rb:250:in `start'
/var/www/miq/vmdb/lib/workers/evm_server.rb:65:in `start'
/var/www/miq/vmdb/lib/workers/evm_server.rb:92:in `start'
/var/www/miq/vmdb/lib/workers/bin/evm_server.rb:4:in `<main>'
[----] I, [2017-01-25T16:00:29.124018 #3495:f57134]  INFO -- : MIQ(MiqQueue#delivered) Message id: [1496012], State: [error], Delivered in [0.051398686] seconds
[----] I, [2017-01-25T16:00:29.139024 #3495:f57134]  INFO -- : MIQ(MiqEmsMetricsProcessorWorker::Runner#get_message_via_drb) Message id: [1497724], MiqWorker id: [13], Zone: [default], Role: [ems_metrics_processor], Server: [], Ident: [ems_metrics_processor], Target id: [], Instance id: [328], Task id: [], Command: [ManageIQ::Providers::Kubernetes::ContainerManager::ContainerGroup.perf_rollup], Timeout: [1800], Priority: [100], State: [dequeue], Deliver On: [2017-01-25 15:00:00 UTC], Data: [], Args: ["2017-01-25T14:00:00Z", "hourly"], Dequeued in: [1056.771220319] seconds

Expected results:
perf collection proceeds

Additional info:

Comment 3 Federico Simoncelli 2017-02-09 14:44:59 UTC
Probably related to (or a duplicate of) bug 1408968.

Ari, can you check?

Comment 4 Ari Zellner 2017-02-22 16:12:19 UTC
>Probably related to (or a duplicate of) bug 1408968. 

Federico, yes its the same issue with zones of archived entities.

Comment 5 Federico Simoncelli 2017-02-24 10:26:23 UTC
https://github.com/ManageIQ/manageiq/pull/13686

Comment 8 CFME Bot 2017-02-24 16:00:55 UTC
New commit detected on ManageIQ/manageiq/euwe:
https://github.com/ManageIQ/manageiq/commit/b06ec630197334af8d98d20b4910939a5a3b1d7b

commit b06ec630197334af8d98d20b4910939a5a3b1d7b
Author:     Jason Frey <fryguy9>
AuthorDate: Fri Feb 24 09:18:21 2017 -0500
Commit:     Joe Rafaniello <jrafanie>
CommitDate: Fri Feb 24 10:18:37 2017 -0500

    Merge pull request #13686 from zeari/cpature_dead_containers
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1420721
    
    Collect metrics for archived containers
    (cherry picked from commit 5c98cf715d1f3a21e7526a4fe558986cdc6d9da6)

 app/models/container.rb                            |  4 +---
 app/models/container_definition.rb                 |  1 +
 app/models/container_group.rb                      |  4 +---
 app/models/container_project.rb                    |  6 +-----
 .../metrics_capture/capture_context.rb             |  2 +-
 app/models/metric/ci_mixin/capture.rb              |  4 +++-
 app/models/mixins/archived_mixin.rb                | 22 +++++++++++++++++++
 spec/models/metric/ci_mixin/capture_spec.rb        | 25 ++++++++++++++++++++++
 8 files changed, 55 insertions(+), 13 deletions(-)
 create mode 100644 app/models/mixins/archived_mixin.rb

Comment 9 Federico Simoncelli 2017-03-09 15:59:11 UTC
*** Bug 1429626 has been marked as a duplicate of this bug. ***

Comment 10 Einat Pacifici 2017-03-23 09:14:25 UTC
Verified. Metrics are collected and no errors (mentioned in this BZ) are seen in the logs.

Comment 11 Federico Simoncelli 2017-03-27 20:02:19 UTC
*** Bug 1408968 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.