Bug 1007393

Summary:	[scale] The thread pool is out of limit
Product:	Red Hat Enterprise Virtualization Manager	Reporter:	Ayal Baron <abaron>
Component:	ovirt-engine	Assignee:	Barak <bazulay>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Yuri Obshansky <yobshans>
Severity:	urgent	Docs Contact:
Priority:	unspecified
Version:	3.4.0	CC:	aberezin, acanan, acathrow, amureini, bazulay, cpelland, emesika, gklein, iheim, jkt, juan.hernandez, lpeer, oourfali, pkliczew, pstehlik, rgolan, Rhev-m-bugs, scohen, talayan, yeylon, yzaslavs
Target Milestone:	---
Target Release:	3.4.1
Hardware:	x86_64
OS:	Linux
Whiteboard:	infra
Fixed In Version:	ovirt-3.4.0-alpha1	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:	968349	Environment:
Last Closed:	2014-08-05 08:25:51 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	Infra	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	968349

Comment 2 Liran Zelkha 2013-10-15 07:33:12 UTC

Please check http://gerrit.ovirt.org/#/c/19744/ for a possible solution.
Roy - the reject policy is run in your own thread, but only after queue is exhausted. I think we should test it under load, but problematic ordering will not be worse than it is today.

Comment 3 Piotr Kliczewski 2013-10-28 10:09:11 UTC

Looking at previous comments there are two solutions:
1. Configurable number of threads and queue size. Solution proposed above.
2. Make IVdsServer implementation async. 

As the first one seems to be ongoing I would like to suggest solution for second one. There is work in progress to have AMQP communication which will introduce async communication. This change can be implemented to provide async behavior for AMQP and http VdsServers.

Comment 5 Sandro Bonazzola 2014-01-14 08:43:39 UTC

ovirt 3.4.0 alpha has been released

Comment 8 Yuri Obshansky 2014-06-30 14:58:08 UTC

Bug was verified on RHEVM version 3.4.1-0.23.el6ev
OS Version: RHEL - 6Server - 6.5.0.1.el6
Kernel Version: 2.6.32 - 431.el6.x86_64
KVM Version: 0.12.1.2 - 2.415.el6_5.9
LIBVIRT Version: libvirt-0.10.2-29.el6_5.7
VDSM Version: vdsm-4.14.7-2.el6ev

The environment:
1 REAL DC - 1 FAKE DC
1 REAL Cluster - 1 FAKE Cluster
1 REAL SD - 10 FAKE SDs 
50 REAL VMs (4CPU - 2G RAM)
1 REAL HOST - 50 Fake HOSTS

Created 80 LUNs on XtreamIO iSCSI storage.
Each FAKE SD has 8 LUNs.

"The thread pool is out of limit" didn't appear.

See Bug 1106572 - [scale] [storage] ConnectStorageServer failed - The thread pool is out of limit (engine finish its thread pool)
for more details

Comment 10 Eyal Edri 2014-08-05 08:25:51 UTC

all these bugs were fixed and released as part of 3.4.1,
and weren't closed since they weren't included in any errata.
closing as current release.