Bug 1568267

Description of problem:

1) Active-Active DR setup (site A and B)
   - Hosts A1, A2, B1, B2
   - HostedEngine on A1
2) Site A is disconnected from storage/network
3) HostedEngine fails over from site A2 to B1
   - Backend initialization 2018-04-11 ~ 18:43:53
4) Hosts from site A are in NotResponding, with VMs "running"
5) Engine starts fencing those hosts in site A
   - Host A2 fence attempt at 2018-04-11 18:44:14, skipped, never tried again
   - Host A1 fence attempt at 2018-04-11 18:47:05, run and succeeded

Skipping fencing is here:

2018-04-11 18:44:14,392+02 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-18) [] EVENT_ID: VDS_ALERT_FENCE_OPERATION_SKIPPED(9,003), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host A2 became non responsive. It has no power management configured. Please check the host status, manually reboot it, and click "Confirm Host Has Been Rebooted"

But host has power management enabled, found the code may log misleading messages, just filled this: https://bugzilla.redhat.com/show_bug.cgi?id=1568265

Looking at the code the skip could have have been due to DisableFenceAtStartupInSec or PreviousHostedEngine. As this was not a HE host before failover, I assume it was skipped due to DisableFenceAtStartupInSec (just a few seconds after startup), but with incorrect logging.

But here comes the problem, after this skip A2 goes on a loop like this forever, every few seconds:

2018-04-11 18:55:00,814+02 INFO  [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] 
  (SSL Stomp Reactor) [] Connecting to A2/IP                                
2018-04-11 18:55:03,819+02 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] 
  execution failed: java.net.NoRouteToHostException: No route to host  
2018-04-11 18:55:03,820+02 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring]
  Failure to refresh host 'A2' runtime info: java.net.NoRouteToHostException: No route to host

And the is no new attempt to run VdsNotRespondingTreatmentCommand for it again.

Version-Release number of selected component (if applicable):
rhevm-4.1.10.3-0.1.el7.noarch

How reproducible:
100% at customer site (3 out of 3 attempts)

Steps to Reproduce:
1. Do DR failover as above

Actual results:
- Host A2 not fenced
- HA VMs with lease disabled not restarted

Expected results:
- Host A2 eventually fenced as host was not repsonding and PM is configured and reachable