Bug 1649424

Created attachment 1505324 [details]
sosreport from host alma04

Description of problem:

 2018-11-13 14:30:13,641+0200 DEBUG var changed: host "localhost" var "he_vm_details" type "<type 'dict'>" value: "{
     "changed": false,
     "deprecations": [
         {
             "msg": "The 'ovirt_vms' module is being renamed 'ovirt_vm'",
             "version": 2.8
         }
     ],
     "exception": "Traceback (most recent call last):\n  File \"/tmp/ansible_ovirt_vms_payload_aUXRhH/__main__.py\", line 2117, in main\n    vms_module.post_present(ret['id'])\n  File \"/tmp/ansible_ovirt_vms_payload_aUXRhH/__main__.py\", line 1241, in post_present\n    self.__attach_disks(entity)\n  File \"/tmp/ansible_ovirt_vms_payload_aUXRhH/__main__.py\", line 1473, in __attach_disks\n    bootable=disk.get('bootable', False),\n  F
 ile \"/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py\", line 6135, in add\n    return self._internal_add(attachment, headers, query, wait)\n  File \"/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py\", line 232, in _internal_add\n    return future.wait() if wait else future\n  File \"/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py\", line 55, in wait\n    return self._code(response)\n  File \"/usr/lib64/python
 2.7/site-packages/ovirtsdk4/service.py\", line 229, in callback\n    self._check_fault(response)\n  File \"/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py\", line 132, in _check_fault\n    self._raise_error(response, body)\n  File \"/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py\", line 118, in _raise_error\n    raise error\nError: Fault reason is \"Operation Failed\". Fault detail is \"[Cannot attach Virtual Disk. T
 he target Data Center does not contain the Virtual Disk.]\". HTTP response code is 409.\n",
     "failed": true,
     "msg": "Fault reason is \"Operation Failed\". Fault detail is \"[Cannot attach Virtual Disk. The target Data Center does not contain the Virtual Disk.]\". HTTP response code is 409."
 }"

it failed here
https://github.com/oVirt/ovirt-hosted-engine-setup/blob/master/src/ansible/create_target_vm.yml#L152
attaching he_virtio_disk to the new engine VM
saying that the storage_pool (basically the datacenter) of the two, doesn't match
while this is not true

Version-Release number of selected component (if applicable):
ovirt-hosted-engine-setup-2.2.32-1.el7ev.noarch
ovirt-hosted-engine-ha-2.2.18-1.el7ev.noarch
rhvm-appliance-4.2-20181026.1.el7.noarch
Linux 3.10.0-957.el7.x86_64 #1 SMP Thu Oct 4 20:48:51 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

How reproducible:
100%

Steps to Reproduce:
1.Deploy SHE over pair of ha-hosts "A" and "B" on iSCSI storage domain to none default data center test on host cluster test.
2.Backup and restore using reprovisioned host "A" or "B", to data center "restore" and host cluster "restore".

Actual results:
Restore to separate data center and host cluster failed.

Expected results:
Restore should succeed.

Additional info:
<tiraboschi> 2018-11-13 14:24:58,092+02 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (default task-29) [72bb543b-c62c-4972-b3de-7a5fa34c84e3] START, CreateImageVDSCommand( CreateImageVDSCommandParameters:{storagePoolId='22e77bd4-ad7a-4384-a43f-f2c0d74e56c8', ignoreFailoverLimit='false', storageDomainId='0cb8361e-5377-43b2-920d-cf83f952617c', imageGroupId='4763cec0-a1f2-4d09-a7e3-9e85bdd4f6c2', imageSizeInBytes='62277025
<tiraboschi> 792', volumeFormat='RAW', newImageId='f306c74b-13a4-445d-bac9-e522230c59a1', imageType='Sparse', newImageDescription='{"DiskAlias":"he_virtio_disk","DiskDescription":"Hosted-Engine disk"}', imageInitialSizeInBytes='0'}), log id: 1dba23ec
<tiraboschi> the disk got created here
<tiraboschi> and we see
<tiraboschi> storagePoolId='22e77bd4-ad7a-4384-a43f-f2c0d74e56c8'
<tiraboschi> storageDomainId='0cb8361e-5377-43b2-920d-cf83f952617c'
<tiraboschi> and we also have
<tiraboschi> [root@nsednev-he-1 ~]# sudo -u postgres scl enable rh-postgresql95 -- psql engine -c "select id, name from storage_pool"
<tiraboschi> could not change directory to "/root": Permission denied
<tiraboschi>                   id                  |  name   
<tiraboschi> --------------------------------------+---------
<tiraboschi>  76ac376c-e352-11e8-a207-00163e7bb853 | Default
<tiraboschi>  69e41970-14b3-48b8-95bd-b22d64f572e8 | test
<tiraboschi>  22e77bd4-ad7a-4384-a43f-f2c0d74e56c8 | restore
<tiraboschi> (3 rows)
<tiraboschi> [root@nsednev-he-1 ~]# sudo -u postgres scl enable rh-postgresql95 -- psql engine -c "select id, storage_name from storage_domains"
<tiraboschi> could not change directory to "/root": Permission denied
<tiraboschi>                   id                  |            storage_name            
<tiraboschi> --------------------------------------+------------------------------------
<tiraboschi>  0cb8361e-5377-43b2-920d-cf83f952617c | hosted_storage
<tiraboschi>  346aac7f-0f31-4717-9722-cf8b0768d34b | nsednev_he_1_data_sd_1
<tiraboschi>  85dc02c3-3e7d-4973-ac99-a7bdf9bf7f82 | ISO
<tiraboschi>  079a1df9-beaf-47f6-a4ef-cbf02ba23ef9 | hosted_storage_old_20181113T135047
<tiraboschi> (4 rows)
<tiraboschi> so the disk got created exactly where we expect it
<tiraboschi> restore \ restore
<tiraboschi> sorry, restore \ hosted_storage
<tiraboschi> and we also have
<tiraboschi> [root@nsednev-he-1 ~]# sudo -u postgres scl enable rh-postgresql95 -- psql engine -c "select vm_name, cluster_id from vms"
<tiraboschi> could not change directory to "/root": Permission denied
<tiraboschi>           vm_name           |              cluster_id              
<tiraboschi> ----------------------------+--------------------------------------
<tiraboschi>  VM1                        | 7079e522-6891-4e81-a968-51b0a072c4d5
<tiraboschi>  VM2                        | 7079e522-6891-4e81-a968-51b0a072c4d5
<tiraboschi>  external-HostedEngineLocal | eaff0720-c994-4078-85a0-1bf77ee05cca
<tiraboschi>  HostedEngine               | eaff0720-c994-4078-85a0-1bf77ee05cca
<tiraboschi> (4 rows)
<tiraboschi> [root@nsednev-he-1 ~]# sudo -u postgres scl enable rh-postgresql95 -- psql engine -c "select cluster_id, name, storage_pool_id from cluster"
<tiraboschi> could not change directory to "/root": Permission denied
<tiraboschi>               cluster_id              |  name   |           storage_pool_id            
<tiraboschi> --------------------------------------+---------+--------------------------------------
<tiraboschi>  7079e522-6891-4e81-a968-51b0a072c4d5 | test    | 69e41970-14b3-48b8-95bd-b22d64f572e8
<tiraboschi>  76aee462-e352-11e8-bc9b-00163e7bb853 | Default | 76ac376c-e352-11e8-a207-00163e7bb853
<tiraboschi>  eaff0720-c994-4078-85a0-1bf77ee05cca | restore | 22e77bd4-ad7a-4384-a43f-f2c0d74e56c8
<tiraboschi> (3 rows)
<tiraboschi> so also the VM got created on the right cluster in the right datacenter