Bug 1397866

Summary: Creating a pool of 5 VM's from template fails
Product: [oVirt] ovirt-engine Reporter: Avihai <aefrat>
Component: BLL.StorageAssignee: Tal Nisan <tnisan>
Status: CLOSED CURRENTRELEASE QA Contact: Avihai <aefrat>
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.0.6CC: aefrat, amureini, bugs, emarcian, gklein, mburman, pnovotny, tnisan
Target Milestone: ovirt-4.0.6Keywords: Regression
Target Release: 4.0.6.2Flags: rule-engine: ovirt-4.0.z?
rule-engine: blocker+
gklein: planning_ack?
tnisan: devel_ack+
rule-engine: testing_ack+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-01-18 07:24:53 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Storage RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
logs none

Description Avihai 2016-11-23 13:38:48 UTC
Created attachment 1223241 [details]
logs

Description of problem:
Creating a pool of 5 VM's from template fails 

Version-Release number of selected component (if applicable):
4.1.0-0.0.master.20161111231310.gitd12e6f1.el7.centos

How reproducible:
100%


Steps to Reproduce:
1.create a VM with 2 disks (1 NFS , 1 gluster) 
2.create template
3.try to create a new pool out of the template 

Actual results:
Creating a pool of 5 VM's from template fails 


Expected results:
Creating a pool of 5 VM's from template should successd 


Additional info:
From Event log :
Nov 23, 2016 3:26:41 PM Not all VMs where successfully created in VM Pool <UNKNOWN>.

Errors seen in Engine log:
2016-11-23 15:30:40,066 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-23) [20340e97] Correlation ID: 9314712d-f252-4d73-ab5c-c968dc4137fb, Job ID: 50159e99-0a10-4ff1-a6de-1f869cf3bd53, Call Stack: null, Custom Event ID: -1, Message: Failed to create VM Pool <UNKNOWN> (User: admin@internal-authz).
2016-11-23 15:30:41,523 INFO  [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (DefaultQuartzScheduler6) [20340e97] Command 'AddVmPool' id: 'b60702b7-0905-41c8-a2e9-f82e6b4d82f5' child commands '[907782fe-bb6c-4903-b508-62c259aefad8, 33068a5f-4843-4e1b-acc7-f79a52f401c7, 1eaf9495-3603-4324-98fb-657821537256]' executions were completed, status 'FAILED'
2016-11-23 15:30:42,646 ERROR [org.ovirt.engine.core.bll.AddVmPoolCommand] (DefaultQuartzScheduler2) [20340e97] Ending command 'org.ovirt.engine.core.bll.AddVmPoolCommand' with failure.

No errors seen at VDSM log :
2016-11-23 15:30:30,683 WARN  (jsonrpc/4) [MOM] MOM not available. (momIF:116)
2016-11-23 15:30:30,684 WARN  (jsonrpc/4) [MOM] MOM not available, KSM stats will be missing. (momIF:79)
2016-11-23 15:30:30,686 INFO  (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds (__init__:515)
2016-11-23 15:30:34,714 INFO  (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:515)
2016-11-23 15:30:46,730 INFO  (jsonrpc/5) [dispatcher] Run and protect: repoStats(options=None) (logUtils:49)
2016-11-23 15:30:46,731 INFO  (jsonrpc/5) [dispatcher] Run and protect: repoStats, Return response: {'dcaaaee8-a2ca-48fb-8b98-690f04a6a00c': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00205327', 'lastCheck': '0.1', 'valid': True}, '64f8d460-9430-4dc8-82b5-8b6146c1537b': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00102653', 'lastCheck': '4.1', 'valid': True}, '815f73c8-1184-4a00-8883-9ce328059e2b': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000530308', 'lastCheck': '0.3', 'valid': True}, 'f7cece4e-6dcb-4c59-8f05-bb65e42b05bb': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000381685', 'lastCheck': '1.7', 'valid': True}, '432a58c9-e015-4c55-bcf7-b24705c29c3c': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000359195', 'lastCheck': '4.6', 'valid': True}} (logUtils:52)
2016-11-23 15:30:46,732 WARN  (jsonrpc/5) [MOM] MOM not available. (momIF:116)
2016-11-23 15:30:46,732 WARN  (jsonrpc/5) [MOM] MOM not available, KSM stats will be missing. (momIF:79)

Comment 1 Tal Nisan 2016-11-23 14:33:11 UTC
Did you try it from the UI or REST? If it's from the REST please attach the request you've sent.

In general it's always best to describe the minimal scenario to reproduce, I doubt if the Gluster disk has any effect on the outcome, did you try to reproduce with 1 NFS disk alone?

Comment 2 Avihai 2016-11-23 14:44:07 UTC
Hi Tal ,

I did it from admin GUI .

I'll try with only NFS and update , anyway it should work also with current disks settings which are pretty minimal (1 NFS , 1 gluster)

Comment 3 Tal Nisan 2016-11-23 14:49:45 UTC
Indeed it should, but when writing steps to reproduce it's best to write the minimal scenario that will trigger the failure in order to make it easier to reproduce

Comment 4 Avihai 2016-11-23 15:41:08 UTC
Got it .

I tested also same scenario (creating a pool of 5 VM's from template) with 1 VM + 1 NFS disk and got the same errors:

from current engine log:

2016-11-23 17:36:50,058 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-24) [85ae624] Correlation ID: 6d575c5d-25b1-497d-8375-19b1aa1bd4d3, Job ID: 3fccaf2e-e9c2-43de-84f4-0e54a0266fcc, Call Stack: null, Custom Event ID: -1, Message: Failed to create VM Pool <UNKNOWN> (User: admin@internal-authz).
2016-11-23 17:36:51,531 INFO  [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (DefaultQuartzScheduler1) [85ae624] Command 'AddVmPool' id: '2694e4d0-bc09-4f67-b689-9f286e9b4199' child commands '[4079bfbf-8b23-4304-a82b-729f0e1f2d8d, 1b79aa02-55bb-4519-a969-f1f70e62defc, 9d86e5c6-02e7-4afc-ab14-862bb03e7038]' executions were completed, status 'FAILED'
2016-11-23 17:36:52,714 ERROR [org.ovirt.engine.core.bll.AddVmPoolCommand] (DefaultQuartzScheduler5) [85ae624] Ending command 'org.ovirt.engine.core.bll.AddVmPoolCommand' with failure.

Comment 5 Tal Nisan 2016-11-23 16:01:10 UTC
Reproduced using steps from comment #4 on 4.0.6, targeting to 4.0.6

Comment 6 Michal Skrivanek 2016-11-25 06:35:26 UTC
*** Bug 1398297 has been marked as a duplicate of this bug. ***

Comment 7 Michal Skrivanek 2016-11-26 06:58:32 UTC
*** Bug 1398746 has been marked as a duplicate of this bug. ***

Comment 8 Michael Burman 2016-11-27 07:20:29 UTC
Creating a pool of 1 VM failing as well on 4.0.6.1-0.1.el7ev

Comment 9 Michael Burman 2016-11-28 06:58:46 UTC
Pleas note that This bug should be failed QA.

Comment 10 Raz Tamir 2016-11-28 09:26:52 UTC
According to comment #8 failing QA

Comment 11 Red Hat Bugzilla Rules Engine 2016-11-28 09:26:59 UTC
Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.

Comment 12 Tal Nisan 2016-11-28 09:31:06 UTC
I don't think it made it to the build according to the timeline, moving to MODIFIED to check in the next build

Comment 13 Tal Nisan 2016-11-28 09:34:56 UTC
Just double checked on mburman's setup on which he checked, this fix was definitely not in this build

Comment 14 Red Hat Bugzilla Rules Engine 2016-11-28 20:50:05 UTC
This bug report has Keywords: Regression or TestBlocker.
Since no regressions or test blockers are allowed between releases, it is also being identified as a blocker for this release. Please resolve ASAP.

Comment 15 Michal Skrivanek 2016-11-29 06:49:12 UTC
*** Bug 1399279 has been marked as a duplicate of this bug. ***

Comment 16 Avihai 2016-12-13 11:55:04 UTC
verified on 4.0.6.5