Created attachment 1189937 [details] evm.log attached. Description of problem: Pod deletion and Creation Trends should show created and deleted pods. The expected number of these pods is not showing up in the dashboard's "Pod Creation and deletion Trends" correctly Version-Release number of selected component (if applicable): 5.6.1 How reproducible: Always Steps to Reproduce: 1.Setup CFME with Provider and assign OpenSCAP policy to provider 2.Generate hourly schedule to run SSA/OpenSCAP scan.24 3. Ensure no new pods are generated manually. 24 4.Wait 24 hours 5. Look at Compute --> Containers --> Overview: Pod Creation and Deletion Trends. Actual results: 146 Pods generated, 136 Pods deleted. Expected results: 24 pods generated. 24 pods deleted. Additional info: attached evm.log; dashboard view; schedule view
Created attachment 1189942 [details] screenshots: Overview and Schedule
Yaacov, can you take a look at this? Thanks.
Created attachment 1190758 [details] Queue stuck
Fix submitted upstream: https://github.com/ManageIQ/manageiq/pull/10396
relevant log(2016-08-08T03) at: https://raw.githubusercontent.com/moolitayer/public_files/master/evm.log-20160808 Attachment too large and bugzilla won't accept it.
*** Bug 1367473 has been marked as a duplicate of this bug. ***
Since there is a default limit of 3 concurrent container scans per ems[1] jobs that are stuck could lead the job queue to be stuck and not allow new scans from that ems. Workaround using rails console(while no new jobs are running): cd /var/www/miq/vmdb/ source /etc/default/evm bin/rails c # We will do it in two steps to be careful: 2.3.0 :013 > Job.where("state != 'waiting_to_start' AND state != 'finished'").map(&:id) => [219, 217, 218, 225, 227, 228, 226, 229] 2.3.0 :018 > Job.where(:id => [219, 217, 218, 225, 227, 228, 226, 229]).update(:state => 'finished', :dispatch_status => 'finished',:message => 'manually canceled' ) [1] settings.yml: :container_scanning: :concurrent_per_ems: 3