Bug 1302694

Summary: [scale] - shut down vms gets stuck
Product: [oVirt] ovirt-engine Reporter: Eldad Marciano <emarcian>
Component: Backend.CoreAssignee: Nobody <nobody>
Status: CLOSED NOTABUG QA Contact: Eldad Marciano <emarcian>
Severity: high Docs Contact:
Priority: unspecified    
Version: 3.6.2CC: bugs, emarcian, tnisan
Target Milestone: ---Flags: rule-engine: planning_ack?
rule-engine: devel_ack?
rule-engine: testing_ack?
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-02-15 10:18:53 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Virt RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
engine logs and threads dumps
none
vdsm logs none

Description Eldad Marciano 2016-01-28 12:05:41 UTC 
Description of problem:
shutting down bunch of vms ~80 vms,
the vms are really stooped -by looking on the vdsm side there is not running vms / kvm processes.

but the engine shows those vm as "powering down", even after refresh capabilities.

attaching logs and thread dumps.

Version-Release number of selected component (if applicable):
rhevm 3.6.2

How reproducible:
not clear

Steps to Reproduce:
1. scale env. 500 hosts 10K vms.
2. shutting down vms related to small cluster (37 hosts, 3K vms) - we have two cluster on this environment.

Actual results:
engine didn't notice that the vms are down already.

Expected results:
list the vms as down, as long they down.

Additional info:
Comment 1 Eldad Marciano 2016-01-28 12:58:25 UTC 
Created attachment 1119138 [details]
engine logs and threads dumps
Comment 2 Eldad Marciano 2016-01-28 12:59:02 UTC 
Created attachment 1119139 [details]
vdsm logs
Comment 3 Yaniv Kaul 2016-01-28 13:33:45 UTC 
Eldad,
- is that for all VMs? Some VMs? 
- Can you give an example of a VM to which this has happened? 
- Have you looked at the logs to see if anything has happened to it?
Comment 4 Tal Nisan 2016-02-01 09:47:09 UTC 
Changed the wrong bug, restoring defaults and setting correct team
Comment 5 Eldad Marciano 2016-02-15 10:18:53 UTC 
this bug currently not reproduced.
i'll re-open it, in case we faced it again.