Bug 1302694 – [scale] - shut down vms gets stuck

Bug 1302694 - [scale] - shut down vms gets stuck

Summary:	[scale] - shut down vms gets stuck

Status:	CLOSED NOTABUG

Aliases:	None

Product:	ovirt-engine
Classification:	oVirt
Component:	Backend.Core (Show other bugs)
Sub Component:
Version:	3.6.2
Hardware:	x86_64 Linux

Priority	unspecified Severity high (vote)
Target Milestone:	---
Target Release:	---
Assigned To:	nobody nobody
QA Contact:	Eldad Marciano
Docs Contact:

URL:
Whiteboard:
Keywords:

Depends On:
Blocks:
	Show dependency tree / graph

Reported:	2016-01-28 07:05 EST by Eldad Marciano
Modified:	2016-02-15 05:18 EST (History)
CC List:	3 users (show)

See Also:
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:
Story Points:	---
Clone Of:
Environment:
Last Closed:	2016-02-15 05:18:53 EST
Type:	Bug
Regression:	---
Mount Type:	---
Documentation:	---
CRM:
Verified Versions:
Category:	---
oVirt Team:	Virt
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---

Flags:	rule-engine: planning_ack? rule-engine: devel_ack? rule-engine: testing_ack?

Attachments	(Terms of Use)
engine logs and threads dumps (897.41 KB, application/zip) 2016-01-28 07:58 EST, Eldad Marciano	no flags	Details
vdsm logs (1.21 MB, application/zip) 2016-01-28 07:59 EST, Eldad Marciano	no flags	Details
Add an attachment (proposed patch, testcase, etc.)

Groups: None (edit)

Description Eldad Marciano 2016-01-28 07:05:41 EST

Description of problem:
shutting down bunch of vms ~80 vms,
the vms are really stooped -by looking on the vdsm side there is not running vms / kvm processes.

but the engine shows those vm as "powering down", even after refresh capabilities.

attaching logs and thread dumps.

Version-Release number of selected component (if applicable):
rhevm 3.6.2

How reproducible:
not clear

Steps to Reproduce:
1. scale env. 500 hosts 10K vms.
2. shutting down vms related to small cluster (37 hosts, 3K vms) - we have two cluster on this environment.

Actual results:
engine didn't notice that the vms are down already.

Expected results:
list the vms as down, as long they down.

Additional info:

Comment 1 Eldad Marciano 2016-01-28 07:58 EST

Created attachment 1119138 [details]
engine logs and threads dumps

Comment 2 Eldad Marciano 2016-01-28 07:59 EST

Created attachment 1119139 [details]
vdsm logs

Comment 3 Yaniv Kaul 2016-01-28 08:33:45 EST

Eldad,
- is that for all VMs? Some VMs? 
- Can you give an example of a VM to which this has happened? 
- Have you looked at the logs to see if anything has happened to it?

Comment 4 Tal Nisan 2016-02-01 04:47:09 EST

Changed the wrong bug, restoring defaults and setting correct team

Comment 5 Eldad Marciano 2016-02-15 05:18:53 EST

this bug currently not reproduced.
i'll re-open it, in case we faced it again.

Note You need to log in before you can comment on or make changes to this bug.