Bug 1968433

Created attachment 1789207 [details]
ovirt dr + engine logs

Description of problem:

When trying to invoke  Failover/Failback flows of the DR, with having a running HA VM on the 'primary' site (if failback) or on 'secondary' site (if failover), the VM fails to be started [1] (from ovirt-dr log). 
The 'failover' flow seems to be completed successfully, but the VM is in 'stopped' state on the 'secondary' site.
The ERROR occurs during the 'TASK [redhat.rhv.disaster_recovery : Run VMs]' ansible task.
The flow below describes the issue as it was seen with 'failover', but the same kind of behavior can be observed with 'failback' as well.

DR schema:
Master - as a driver of the DR scripts - storage-ge-13
Primary - an env where the disaster occurs - storage-ge-15
Secondary - an env where the assets are being migrated to - storage-ge-16


Envs state prior the testing:

1. Primary site containing:
Active data center
Active cluster
Active hosts
One active and attached nfs storage domain
One template
No VMs

2. Secondary site containing:
Active data center
Active cluster
Active hosts
NO VMs, templates or attached storage domains


[1]:
ovirtsdk4.Error: Fault reason is "Operation Failed". Fault detail is "[Cannot run VM. VM test is being imported.]". HTTP response code is 409.


Version-Release number of selected component (if applicable):
rhv-release-4.4.6-9-001.noarch

How reproducible:
100%

Steps to Reproduce:
- Create HA VM on the 'master' site + start it
- generate the mappings file with ./ovirt-dr generate
- Update OVF store for the storage domain on master
- Mount primary and secondary Storages:
mount -t nfs mantis-nfs-xxx.com:/nas01/ge_storage16_nfs_1 /mnt/secondary/ge_storage16_nfs_1
mount -t nfs mantis-nfs-xxx.com:/nas01/ge_storage15_nfs_1 /mnt/primary/ge_storage15_nfs_1/

- Make sure that the secondary mount point is empty:
Rm -rf /mnt/secondary/ge_storage16_nfs_1/*

- To create a replica, rsync primary storage content to a secondary one:
[root@storage-ge-13 files]# rsync -azvh /mnt/primary/ge_storage15_nfs_1/* /mnt/secondary/ge_storage16_nfs_1

- change ownership of the replicated storage mount folder + contents:
chown -R vdsm:kvm /mnt/secondary/ge_storage16_nfs_1/


- Run failover ./ovirt-dr failover


Actual results:
The secondary storage domain is attached and active
The template was imported
The VM was imported BUT NOT running


Expected results:
Wait for failover to finish and verify that:
The secondary storage domain is attached and active
The template was imported
The VM was imported and running


Additional info:
Attaching ovirt-dr log and engine log of the 'secondary' site (where the VM should be up) and vdsm