Bug 1401585

Description of problem:

The task Q processing is being done on a local copy of the Q. while waiting for response, an exception will abort the local copy task processing and the rest of the jobs in side that local copy will be lost. The main Q meanwhile doesn't have those tasks any longer. This result in losing jobs, i.e lossing calls to api and in the worst case losing the connect request that can result the VdsManager lock to be held infinitely and the rest of the engine wide threads to start reporting network exception and contend on the same VdsManager lock. This will exaust the thread pool task Q as well and the engine daemons (quarts scheduled tasks) will stop work.

The attached dump and engine log shows how the "Reactor Client" is waiting forever on the connectTask.get() task without any progress and the rest of threads waiting for the VdsManager lock that they can never get hold on. 



Version-Release number of selected component (if applicable):
1.2.9 and affect ovirt-engine 4.0 and 4.1 master

How reproducible:
50% cause it depends on the timing.

Steps to Reproduce:
1. have 3 hosts and some vms and disks (so we will have activity on monitoring)
2. disconnect the network (pull the plug or drop packets or switch network)
3. wait till the SocketException is thrown or some other IO

Actual results:
Tasks that are in the Q get lost and never process. may render the system unsable

Expected results:
Task processing fail in isolation, the rest of the tasks gets proccesed, the system doest get contended on the VdsManager lock for (VdsManager.handleNetworkException -> needs an engine fix as well)

Additional info:
happen to in a production env and on development env
See ReactorScheduler#performPendingOpearations and ReactorClient#connect speciall connectTask.get()