Created attachment 1221254 [details]
Generic script to be used for fail over

Description of problem:

The following describes an action path that has been tested to provide DR operations with minimal intervention

Version-Release number of selected component (if applicable):


How reproducible:

100%

Steps to Reproduce:
Assumptions: 
  -- There is full LUN replication in the DR site.
  -- DC = Main Data Center
  -- DR = Disaster Recovery Data Center
  -- RHV-M is on replicated LUN.


1. Add new RHEV-H in the same cluster as the one to be failed over in DR 
2. Approve new RHEV-H (same cluster – Low SPM Priority)
3. Wait to finish Installation of new RHEV-H
4. Place new RHEV-H in maintenance
5. Set persistent FW rules in DR so traffic from DR located RHEV-H, is redirected to the DR operated RHEV-M
6. Add new DR RHEV-H to the same cluster running in Data Center.
NOTE: At least one RHEV-H needs to be configured in DR, to take over VMs during DR operations, as an SPM is needed in the DR.
7. Brink RHEV-H in teh DR Maintenance Mode
8. Power Off RHEV-M in DR
Once we need to move to DR (assuming: LUNs have been replicated to DR, they are brought on line
1. Activate replicated RHEV-M in DR
2. Force RHEV-M in DR to elect a new SPM
  To do this the command 'action host <name of RHEV-H> fence --fence_type manual' must be issued via rhev shell. It can be scripted

3. Allow cluster to settle

Done


Actual results:
Failover between DC and DR works

Expected results:

As actual results

Additional info:

To revert from DC to DR, one needs to reverse LUN replication (both RHEV-datacenter LUNs and RHEV-M LUNs, and launch RHEV-M in DC once sync is completed.

Keep in mind that failing over from DC to DR and back to DC involved manual actions. The above procedure takes out of the loop any need to make manual changes to the database or anywhere else.