Bug 865093

Summary: ovirt-engine-restpai : 500 error when trying to create large VM pool using RHEVM shell
Product: Red Hat Enterprise Virtualization Manager Reporter: Oded Ramraz <oramraz>
Component: ovirt-engine-restapiAssignee: Michael Pasternak <mpastern>
Status: CLOSED CURRENTRELEASE QA Contact: Oded Ramraz <oramraz>
Severity: high Docs Contact:
Priority: unspecified    
Version: 3.1.0CC: bazulay, dyasny, ecohen, iheim, juan.hernandez, michal.skrivanek, mpastern, Rhev-m-bugs, sgrinber, ykaul
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: virt
Fixed In Version: s122 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-12-04 19:59:07 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
server log none

Description Oded Ramraz 2012-10-10 19:43:03 UTC 
Description of problem:

When I'm trying to create big VM pool ( 200 VM's ) using RHEVM shell I'm getting 500 server error ( see additional information ) although the pool is created properly.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info:


[RHEVM shell (connected)]# add vmpool --name pool --template-id 3d5f82c2-8ef4-436b-a43e-c6367a126791 --size 200 --cluster-id 99408929-82cf-4dc7-a532-9d998063fa95

error: 
status: 500
reason: Internal Server Error
detail: 
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>500 Internal Server Error</title>
</head><body>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error or
misconfiguration and was unable to complete
your request.</p>
<p>Please contact the server administrator,
 root@localhost and inform them of the time the error occurred,
and anything you might have done that may have
caused the error.</p>
<p>More information about this error may be available
in the server error log.</p>
<hr>
<address>Apache/2.2.17 (Red Hat Enterprise Web Server) Server at 10.35.113.33 Port 443</address>
</body></html>
Comment 1 Oded Ramraz 2012-10-10 19:47:13 UTC 
Created attachment 625125 [details]
server log
Comment 2 Michael Pasternak 2012-10-11 06:59:22 UTC 
This is not rhevm-shell bug, nor api's, this is very same issue as in [1],
Itamar, i believe we should define how many vms we support creating at 
once. 


[1]

https://bugzilla.redhat.com/show_bug.cgi?id=845832
https://bugzilla.redhat.com/show_bug.cgi?id=818084
https://bugzilla.redhat.com/show_bug.cgi?id=853045
Comment 3 Oded Ramraz 2012-10-11 07:18:26 UTC 
The VM pool was created properly after this error , with this message : "VM Pool pool ( containing 200 VMs ) was created by admin@internal" , so I wasn't sure if it is related to Rest / Engine.
Comment 4 Oded Ramraz 2012-10-11 07:22:26 UTC 
And another event: "VM pool creation has been completed"

(In reply to comment #3)
> The VM pool was created properly after this error , with this message : "VM
> Pool pool ( containing 200 VMs ) was created by admin@internal" , so I
> wasn't sure if it is related to Rest / Engine.
Comment 5 Michael Pasternak 2012-10-11 07:25:11 UTC 
(In reply to comment #3)
> The VM pool was created properly after this error , with this message : "VM
> Pool pool ( containing 200 VMs ) was created by admin@internal" , so I
> wasn't sure if it is related to Rest / Engine.

the problem is between api & tomcat, - apache TTL expired while waiting
for the engine action accomplishment.
Comment 6 Michael Pasternak 2012-10-11 07:32:55 UTC 
raising TTL again is only fire fighting, as i mentioned before we can easily face this again in some other scenario, we really should be pushing to "truly" async action impl. in the engine.
Comment 7 Oded Ramraz 2012-10-11 07:41:42 UTC 
BTW currently the maximum VM's number in pool is according to the number of free MAC addresses:

[RHEVM shell (connected)]# add vmpool --name pool --template-id 3d5f82c2-8ef4-436b-a43e-c6367a126791 --size 300 --cluster-id 99408929-82cf-4dc7-a532-9d998063fa95

error: 
status: 400
reason: Bad Request
detail: Not enough MAC addresses left in MAC Address Pool.
Comment 8 Itamar Heim 2012-10-11 07:45:12 UTC 
interesting to see the results post the fix to bug 853045
einav - don't we have a limit to max number of vms in pool?
Comment 9 Michal Skrivanek 2012-10-18 16:55:25 UTC 
either way, since this is actually working fine despite the error message, seems to me we can push it out of 3.1, Oded?
Comment 10 Oded Ramraz 2012-10-19 10:10:55 UTC 
The problem is that since it is a synchronous operation ,  if you implement a script which create a VM pool and then do something with it you will not have the indication when the pool creation is done, since the error occurs quite early ( and VM pool creation might take quite a few minutes ). 


(In reply to comment #9)
> either way, since this is actually working fine despite the error message,
> seems to me we can push it out of 3.1, Oded?
Comment 11 Michael Pasternak 2012-10-23 09:36:28 UTC 
by Barak:

Per decision to increase timeout on BZ 853045 to 1 hour,
The above phenomena should not happen.

Moving to ON_QA just for verification.
Comment 12 Oded Ramraz 2012-11-12 20:07:33 UTC 
si24.1: managed to create 200 VM's pool using CLI.

[RHEVM shell (connected)]# add vmpool --template-id d44374a8-92aa-45d7-bf28-2d9dc5cb31ba --size 200 --cluster-id 99408929-82cf-4dc7-a532-9d998063fa95 --name oded

id            : 775f9c82-2d03-11e2-ac3d-8b52998f4c71
name          : oded
cluster-id    : 99408929-82cf-4dc7-a532-9d998063fa95
prestarted_vms: 0
size          : 200
template-id   : d44374a8-92aa-45d7-bf28-2d9dc5cb31ba