Created attachment 1244276 [details]
restore log

Description of problem:
Restore HE backup will fail if the HE host has running non-HE VM's 

Version-Release number of selected component (if applicable):
rhevm-4.1.0.2-0.2.el7.noarch

How reproducible:
Always

Steps to Reproduce:
1. Deploy HE environment
2. Add the storage domain to the engine(to start auto-import process)
3. Wait until the engine will have HE VM
4. Add non-HE VM and run it on HE hosts
5. Set global maintenance
6. Backup the engine: # engine-backup --mode=backup --file=engine.backup --log=engine-backup.log
7. Copy the backup file from the HE VM to the host
8. Clean host from HE deploy(reprovisioning)
9. Run the HE deployment again
10. Answer No on the question "Automatically execute engine-setup on the engine appliance on first boot (Yes, No)[Yes]? "
11. Enter to the HE VM and copy the backup file from the host to the HE VM
12. Run restore command: # engine-backup --mode=restore --scope=all --file=engine.backup --log=engine-restore.log  --he-remove-storage-vm --he-remove-hosts --restore-permissions --provision-all-databases

Actual results:
SELECT DeleteHostedEngineHosts();
ERROR:  update or delete on table "vds_static" violates foreign key constraint "vds_static_vm_dynamic_r" on table "vm_dynamic"
DETAIL:  Key (vds_id)=(b0cd1b54-5bf5-475a-b247-778c82eca27a) is still referenced from table "vm_dynamic".
CONTEXT:  SQL statement "DELETE
        FROM vds_static
        WHERE vds_id = v_vds_id"
PL/pgSQL function deletevds(uuid) line 19 at SQL statement
SQL statement "SELECT deletevds(vds_id)
        FROM (
            SELECT vds_id
                FROM vds_statistics
                WHERE ha_score IS NOT NULL AND ha_configured
        ) t"
PL/pgSQL function deletehostedenginehosts() line 3 at PERFORM
FATAL: Cannot execute sql command: --command=SELECT DeleteHostedEngineHosts();
2017-01-25 09:32:37 11559: FATAL: Failed cleaning hosted-engine


Expected results:
Restore operation succeeds. I believe we need to remove all references on this host from the database as well(I assume that the restore operation will fail also in the case when we have non-HE VM that does not run on the host but pinned to it, also affinity labels with this host can be a problem)

Additional info: