Created attachment 895642 [details] engine.log Description of problem: Customer had a failing switch that needed to be taken down, during this time RHEV-M lost all routes to hypervisors and IPMI devices (though hypervisors still had full network connectivity, with the except of communication to the manager). When the connectivity was restored, RHEV-M fenced every hypervisor in the environment. In the attached engine.log it seems that the engine used each hypervisor to fence every other hypervisor - and continued generating these fence events even after it detected a failure to fence the hypervisors (due to the IPMI devices also being inaccessible). Version-Release number of selected component (if applicable): rhevm-3.3.2 vdsm-4.13.2-0.13 Additional info: First, it was only the manager that lost network connectivity. The hypervisors and IPMI devices are on separate subnets than the manager and did not lose connectivity with each other or the rest of the infrastructure. Also, I am not sure if this is a race condition, or something else where the engine is continually queue'ing fence commands that get fired off once networking is restored. Perhaps a final check of "is the host REALLY down" before the retries for fencing the hypervisor would help this?
The work that has gone into 3.5 addresses this item. This has been discussed directly with the customer and validated. Closing BZ as CURRENTRELEASE.