Bug 1395922

Summary: [scale] VM Template Creation dialog blocking UI for around 2 minutes during concurrent VM start operations
Product: [oVirt] ovirt-engine Reporter: mlehrer
Component: Frontend.WebAdminAssignee: Tal Nisan <tnisan>
Status: CLOSED UPSTREAM QA Contact: Pavel Stehlik <pstehlik>
Severity: medium Docs Contact:
Priority: medium    
Version: 4.0.5.1CC: amureini, bugs, izuckerm, michal.skrivanek, mlehrer, oourfali, tjelinek, tnisan, ylavi
Target Milestone: ovirt-4.2.0Keywords: Performance
Target Release: ---Flags: rule-engine: ovirt-4.2+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-08-21 10:45:42 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Storage RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description mlehrer 2016-11-17 00:14:42 UTC
Description of problem:
UI blocked by blank dialog box when attempting VM template creation when 5 or more VMs start ups pending or in "VM not responding" status, which eventually becomes VM starting status.

Dataset: 200 VMs, 1000 Disks, each VM has 5 disks

Version-Release number of selected component (if applicable):
RHEL 7.3	
vdsm-4.18.15.2-1	
RHEVM 4.0.5	

How reproducible:
Reproduces assuming dataset of 1k disks and 200 VMs loaded, and 5 concurrent start VM tasks done in the background

Steps to Reproduce:
1.Build 200 VMs with 5 disk each
2.Trigger a few VMs to start (up to 5 at a time)
3.Attempt VM Template Creation

Actual results:
UI blocked by blank dialog box when attempting VM template creation when 5 or more VMs start ups pending or in "VM not responding" status, which eventually becomes VM starting status.

Expected results:
Not to have UI blocked

Additional info:

Comment 1 Yaniv Kaul 2016-11-17 07:18:01 UTC
Logs?

Comment 2 Michal Skrivanek 2016-11-17 07:19:44 UTC
blocked==completely blank or there was the spinning thingie?
it may have been simply slow in getting responses from backend due to other ongoing operations.

Comment 3 Greg Sheremeta 2016-11-17 12:57:33 UTC
Sounds like an engine issue, not UI.

Is the UI waiting on a server response when this is happening?

Comment 4 Oved Ourfali 2016-11-17 13:18:26 UTC
Moving to virt to examine the issue.

Comment 5 mlehrer 2016-11-17 17:09:38 UTC
(In reply to Yaniv Kaul from comment #1)
> Logs?

https://drive.google.com/drive/folders/0B8V1DXeGhPPWMi1SNjdYOVFlaWs?usp=sharing



(In reply to Michal Skrivanek from comment #2)
> blocked==completely blank or there was the spinning thingie?
> it may have been simply slow in getting responses from backend due to other
> ongoing operations.

Yes, the 'spinning thingie', see the following link there's a video of it occurring.  

Engine.log shows that at 

2016-11-17 18:19:58 VMs are powered on 'WaitForLaunch' --> 'NotResponding' starts
Shortly after VM Template Create is issued and the task polling begins.

Comment 6 Michal Skrivanek 2016-11-18 11:16:38 UTC
(In reply to Greg Sheremeta from comment #3)
> Sounds like an engine issue, not UI.
> 
> Is the UI waiting on a server response when this is happening?

likely just slow engine indeed. This needs to be correlated with engine load, in general there's nothing wrong about it unless it was like that for minutes/hours

This needs to be isolated first before we can do anything about it. I would suggest to start with the non-responsive VMs and minimal reproduction scenario

Comment 7 Michal Skrivanek 2016-11-18 11:25:07 UTC
sorry I missed the fact this is "Make Template" flow
- there are large disks
- it takes ~1 minute which is not so bad

it is likely storage operations taking time. Tal, do you want to share your thoughts?

Comment 8 Tal Nisan 2016-11-21 10:21:33 UTC
How can I have thoughts without logs?
Mordechai, please attach Engine logs

Comment 9 mlehrer 2016-11-21 10:25:29 UTC
(In reply to Tal Nisan from comment #8)
> How can I have thoughts without logs?
> Mordechai, please attach Engine logs

Tal,

My apologies the logs were not added directly to bugzilla, as indicated in comment 5 you can find the logs here instead.
 
https://drive.google.com/drive/folders/0B8V1DXeGhPPWMi1SNjdYOVFlaWs?usp=sharing

Please let me know if your missing further details or require enviroment access.

Comment 10 Tomas Jelinek 2016-11-23 09:42:02 UTC
aaand putting the needinfo back to Tal

Comment 11 Tal Nisan 2016-11-23 10:46:07 UTC
Gave it a look, can't find anything out of the ordinary, it will take more research so I have no insights for now

Comment 12 Tomas Jelinek 2016-11-23 11:52:16 UTC
moving to storage for investigation

Comment 13 Yaniv Kaul 2017-02-12 11:16:01 UTC
Tal - I don't see this being handled in 4.1?

Comment 14 Tal Nisan 2017-02-12 14:03:51 UTC
Most likely not, we aren't aware of this happening in the field and given the preliminary look I gave it it seems that it needs a lot of work to analyze and we don't want to risk 4.1.1

Comment 15 Yaniv Lavi 2017-02-23 11:24:49 UTC
Moving out all non blocker\exceptions.

Comment 16 Allon Mureinik 2017-07-25 12:15:32 UTC
Does this reproduce in the new 4.2 GUI?

Comment 17 Ilan Zuckerman 2017-08-21 07:51:40 UTC
(In reply to Allon Mureinik from comment #16)
> Does this reproduce in the new 4.2 GUI?


This does not reproduce in 4.2.0-0.0.master.20170818152502.gita8c361f.el7.centos

Tested with 200 VMs with 5 disks each.
Attempted to create VM template while 5 VMs were starting, and the UI did not block when the VM template was created.

Comment 18 Allon Mureinik 2017-08-21 10:45:42 UTC
(In reply to Ilan Zuckerman from comment #17)
> (In reply to Allon Mureinik from comment #16)
> > Does this reproduce in the new 4.2 GUI?
> 
> 
> This does not reproduce in
> 4.2.0-0.0.master.20170818152502.gita8c361f.el7.centos
> 
> Tested with 200 VMs with 5 disks each.
> Attempted to create VM template while 5 VMs were starting, and the UI did
> not block when the VM template was created.

Thanks Ilan!

I'm closing this bug under the assumption the work done in 4.2 solved the issue. If it reproduced, please reopn.