Description of problem:
During REST API User Portal performance test of 50 concurrent threads on 50 VMs detected errors >> "Failed to run VM scale-50 (User: scale-50)" in spite of that VM started successfully.
As results performance degradation.
VM shutdowm response time 90% line in RHEV-M 3.4 = 5749 ms compare to RHEV-M 3.5 = 241912 ms.
Engine.log errors:
2014-10-15 12:46:58,027 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-9) [6c470701] Command CreateVDSCommand(HostName = HOST-REAL-1, HostId = 932aaea7-5e19-473f-964f-9d26b71adf05, vmId=8c54a225-048e-49f6-9415-fa0f2b3cd29d, vm=VM [scale-1]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues'
2014-10-15 12:46:58,027 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-9) [6c470701] FINISH, CreateVDSCommand, log id: 76454652
2014-10-15 12:46:58,027 ERROR [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-9) [6c470701] VDS::create Failed creating vm scale-1 in vds = 932aaea7-5e19-473f-964f-9d26b71adf05 : HOST-REAL-1 error = org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues'
2014-10-15 12:46:58,027 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-9) [6c470701] FINISH, CreateVmVDSCommand, return: Down, log id: 6c18eef4
2014-10-15 12:46:58,027 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (ajp-/127.0.0.1:8702-9) [6c470701] Lock freed to object EngineLock [exclusiveLocks= key: 8c54a225-048e-49f6-9415-fa0f2b3cd29d value: VM
, sharedLocks= ]
....................
2014-10-15 12:46:58,027 ERROR [org.ovirt.engine.core.bll.RunVmOnceCommand] (ajp-/127.0.0.1:8702-9) [6c470701] Command org.ovirt.engine.core.bll.RunVmOnceCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' (Failed with error VDS_NETWORK_ERROR and code 5022)
Version-Release number of selected component (if applicable):
RHEV-M: 3.5.0-0.14.beta
RHEL: 6Server - 6.5.0.1.el6
Kernel Version: 2.6.32 - 431.29.2.el6.x86_64
KVM Version: 0.12.1.2 - 2.415.el6_5.10
LIBVIRT Version: libvirt-0.10.2-29.el6_5.11
VDSM Version: vdsm-4.16.6-1.el6ev
How reproducible:
Run automatic test which shutdowm/start VMs using user portal REST API with load of 50 concurrent threads.
Steps to Reproduce:
1.
2.
3.
Actual results:
Errors
Performance degradation
Expected results:
No errors
No performance degradation
Additional info: