Description of problem: During REST API User Portal performance test of 50 concurrent threads on 50 VMs detected errors >> "Failed to run VM scale-50 (User: scale-50)" in spite of that VM started successfully. As results performance degradation. VM shutdowm response time 90% line in RHEV-M 3.4 = 5749 ms compare to RHEV-M 3.5 = 241912 ms. Engine.log errors: 2014-10-15 12:46:58,027 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-9) [6c470701] Command CreateVDSCommand(HostName = HOST-REAL-1, HostId = 932aaea7-5e19-473f-964f-9d26b71adf05, vmId=8c54a225-048e-49f6-9415-fa0f2b3cd29d, vm=VM [scale-1]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' 2014-10-15 12:46:58,027 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-9) [6c470701] FINISH, CreateVDSCommand, log id: 76454652 2014-10-15 12:46:58,027 ERROR [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-9) [6c470701] VDS::create Failed creating vm scale-1 in vds = 932aaea7-5e19-473f-964f-9d26b71adf05 : HOST-REAL-1 error = org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' 2014-10-15 12:46:58,027 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-9) [6c470701] FINISH, CreateVmVDSCommand, return: Down, log id: 6c18eef4 2014-10-15 12:46:58,027 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (ajp-/127.0.0.1:8702-9) [6c470701] Lock freed to object EngineLock [exclusiveLocks= key: 8c54a225-048e-49f6-9415-fa0f2b3cd29d value: VM , sharedLocks= ] .................... 2014-10-15 12:46:58,027 ERROR [org.ovirt.engine.core.bll.RunVmOnceCommand] (ajp-/127.0.0.1:8702-9) [6c470701] Command org.ovirt.engine.core.bll.RunVmOnceCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' (Failed with error VDS_NETWORK_ERROR and code 5022) Version-Release number of selected component (if applicable): RHEV-M: 3.5.0-0.14.beta RHEL: 6Server - 6.5.0.1.el6 Kernel Version: 2.6.32 - 431.29.2.el6.x86_64 KVM Version: 0.12.1.2 - 2.415.el6_5.10 LIBVIRT Version: libvirt-0.10.2-29.el6_5.11 VDSM Version: vdsm-4.16.6-1.el6ev How reproducible: Run automatic test which shutdowm/start VMs using user portal REST API with load of 50 concurrent threads. Steps to Reproduce: 1. 2. 3. Actual results: Errors Performance degradation Expected results: No errors No performance degradation Additional info:
please attach vdsm log might be duplicate of bug 1148583
Created attachment 948948 [details] vdsm.log
Created attachment 948949 [details] engine.log
*** This bug has been marked as a duplicate of bug 1148583 ***