Created attachment 1505324 [details] sosreport from host alma04 Description of problem: 2018-11-13 14:30:13,641+0200 DEBUG var changed: host "localhost" var "he_vm_details" type "<type 'dict'>" value: "{ "changed": false, "deprecations": [ { "msg": "The 'ovirt_vms' module is being renamed 'ovirt_vm'", "version": 2.8 } ], "exception": "Traceback (most recent call last):\n File \"/tmp/ansible_ovirt_vms_payload_aUXRhH/__main__.py\", line 2117, in main\n vms_module.post_present(ret['id'])\n File \"/tmp/ansible_ovirt_vms_payload_aUXRhH/__main__.py\", line 1241, in post_present\n self.__attach_disks(entity)\n File \"/tmp/ansible_ovirt_vms_payload_aUXRhH/__main__.py\", line 1473, in __attach_disks\n bootable=disk.get('bootable', False),\n F ile \"/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py\", line 6135, in add\n return self._internal_add(attachment, headers, query, wait)\n File \"/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py\", line 232, in _internal_add\n return future.wait() if wait else future\n File \"/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py\", line 55, in wait\n return self._code(response)\n File \"/usr/lib64/python 2.7/site-packages/ovirtsdk4/service.py\", line 229, in callback\n self._check_fault(response)\n File \"/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py\", line 132, in _check_fault\n self._raise_error(response, body)\n File \"/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py\", line 118, in _raise_error\n raise error\nError: Fault reason is \"Operation Failed\". Fault detail is \"[Cannot attach Virtual Disk. T he target Data Center does not contain the Virtual Disk.]\". HTTP response code is 409.\n", "failed": true, "msg": "Fault reason is \"Operation Failed\". Fault detail is \"[Cannot attach Virtual Disk. The target Data Center does not contain the Virtual Disk.]\". HTTP response code is 409." }" it failed here https://github.com/oVirt/ovirt-hosted-engine-setup/blob/master/src/ansible/create_target_vm.yml#L152 attaching he_virtio_disk to the new engine VM saying that the storage_pool (basically the datacenter) of the two, doesn't match while this is not true Version-Release number of selected component (if applicable): ovirt-hosted-engine-setup-2.2.32-1.el7ev.noarch ovirt-hosted-engine-ha-2.2.18-1.el7ev.noarch rhvm-appliance-4.2-20181026.1.el7.noarch Linux 3.10.0-957.el7.x86_64 #1 SMP Thu Oct 4 20:48:51 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux How reproducible: 100% Steps to Reproduce: 1.Deploy SHE over pair of ha-hosts "A" and "B" on iSCSI storage domain to none default data center test on host cluster test. 2.Backup and restore using reprovisioned host "A" or "B", to data center "restore" and host cluster "restore". Actual results: Restore to separate data center and host cluster failed. Expected results: Restore should succeed. Additional info: <tiraboschi> 2018-11-13 14:24:58,092+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (default task-29) [72bb543b-c62c-4972-b3de-7a5fa34c84e3] START, CreateImageVDSCommand( CreateImageVDSCommandParameters:{storagePoolId='22e77bd4-ad7a-4384-a43f-f2c0d74e56c8', ignoreFailoverLimit='false', storageDomainId='0cb8361e-5377-43b2-920d-cf83f952617c', imageGroupId='4763cec0-a1f2-4d09-a7e3-9e85bdd4f6c2', imageSizeInBytes='62277025 <tiraboschi> 792', volumeFormat='RAW', newImageId='f306c74b-13a4-445d-bac9-e522230c59a1', imageType='Sparse', newImageDescription='{"DiskAlias":"he_virtio_disk","DiskDescription":"Hosted-Engine disk"}', imageInitialSizeInBytes='0'}), log id: 1dba23ec <tiraboschi> the disk got created here <tiraboschi> and we see <tiraboschi> storagePoolId='22e77bd4-ad7a-4384-a43f-f2c0d74e56c8' <tiraboschi> storageDomainId='0cb8361e-5377-43b2-920d-cf83f952617c' <tiraboschi> and we also have <tiraboschi> [root@nsednev-he-1 ~]# sudo -u postgres scl enable rh-postgresql95 -- psql engine -c "select id, name from storage_pool" <tiraboschi> could not change directory to "/root": Permission denied <tiraboschi> id | name <tiraboschi> --------------------------------------+--------- <tiraboschi> 76ac376c-e352-11e8-a207-00163e7bb853 | Default <tiraboschi> 69e41970-14b3-48b8-95bd-b22d64f572e8 | test <tiraboschi> 22e77bd4-ad7a-4384-a43f-f2c0d74e56c8 | restore <tiraboschi> (3 rows) <tiraboschi> [root@nsednev-he-1 ~]# sudo -u postgres scl enable rh-postgresql95 -- psql engine -c "select id, storage_name from storage_domains" <tiraboschi> could not change directory to "/root": Permission denied <tiraboschi> id | storage_name <tiraboschi> --------------------------------------+------------------------------------ <tiraboschi> 0cb8361e-5377-43b2-920d-cf83f952617c | hosted_storage <tiraboschi> 346aac7f-0f31-4717-9722-cf8b0768d34b | nsednev_he_1_data_sd_1 <tiraboschi> 85dc02c3-3e7d-4973-ac99-a7bdf9bf7f82 | ISO <tiraboschi> 079a1df9-beaf-47f6-a4ef-cbf02ba23ef9 | hosted_storage_old_20181113T135047 <tiraboschi> (4 rows) <tiraboschi> so the disk got created exactly where we expect it <tiraboschi> restore \ restore <tiraboschi> sorry, restore \ hosted_storage <tiraboschi> and we also have <tiraboschi> [root@nsednev-he-1 ~]# sudo -u postgres scl enable rh-postgresql95 -- psql engine -c "select vm_name, cluster_id from vms" <tiraboschi> could not change directory to "/root": Permission denied <tiraboschi> vm_name | cluster_id <tiraboschi> ----------------------------+-------------------------------------- <tiraboschi> VM1 | 7079e522-6891-4e81-a968-51b0a072c4d5 <tiraboschi> VM2 | 7079e522-6891-4e81-a968-51b0a072c4d5 <tiraboschi> external-HostedEngineLocal | eaff0720-c994-4078-85a0-1bf77ee05cca <tiraboschi> HostedEngine | eaff0720-c994-4078-85a0-1bf77ee05cca <tiraboschi> (4 rows) <tiraboschi> [root@nsednev-he-1 ~]# sudo -u postgres scl enable rh-postgresql95 -- psql engine -c "select cluster_id, name, storage_pool_id from cluster" <tiraboschi> could not change directory to "/root": Permission denied <tiraboschi> cluster_id | name | storage_pool_id <tiraboschi> --------------------------------------+---------+-------------------------------------- <tiraboschi> 7079e522-6891-4e81-a968-51b0a072c4d5 | test | 69e41970-14b3-48b8-95bd-b22d64f572e8 <tiraboschi> 76aee462-e352-11e8-bc9b-00163e7bb853 | Default | 76ac376c-e352-11e8-a207-00163e7bb853 <tiraboschi> eaff0720-c994-4078-85a0-1bf77ee05cca | restore | 22e77bd4-ad7a-4384-a43f-f2c0d74e56c8 <tiraboschi> (3 rows) <tiraboschi> so also the VM got created on the right cluster in the right datacenter
Created attachment 1505338 [details] deploy logs from alma04
Initial deployment was made on iSCSI and then restore was made on NFS.
Nikolai are you able to reproduce?
(In reply to Sandro Bonazzola from comment #3) > Nikolai are you able to reproduce? I did not reproduced this, only opened when found the issue and reported it back. Please note that this bug is about different host cluster and storage domain name "restore" during restore, than was previously used for host cluster and storage domain as "test".
Postponing to 4.3.0 not being identified as blocker for 4.2.8
Moving to 4.3.2 not being identified as blocker for 4.3.1.
Can you please reproduce on latest build?
Just got reproduced on my environment. [ ERROR ] Error: Fault reason is "Operation Failed". Fault detail is "[Cannot attach Virtual Disk. The target Data Center does not contain the Virtual Disk.]". HTTP response code is 409. [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "Fault reason is \"Operation Failed\". Fault detail is \"[Cannot attach Virtual Disk. The target Data Center does not contain the Virtual Disk.]\". HTTP response code is 409."} [ ERROR ] Failed to execute stage 'Closing up': Failed executing ansible-playbook
These are the components that reproduction was made on: ovirt-hosted-engine-ha-2.3.1-1.el7ev.noarch ovirt-hosted-engine-setup-2.3.5-1.el7ev.noarch ovirt-ansible-engine-setup-1.1.8-1.el7ev.noarch rhvm-appliance-4.3-20190220.2.el7.x86_64 Linux 3.10.0-957.5.1.el7.x86_64 #1 SMP Wed Dec 19 10:46:58 EST 2018 x86_64 x86_64 x86_64 GNU/Linux Red Hat Enterprise Linux Server release 7.6 (Maipo)
Looks like the issue is on engine side, moving to storage team for further investigations
Hi, is this bug still reproducable? can you attach engine logs?
Can't reproduce due to capacity.
what I saw in the logs that there are deprecations in Ansible playbooks, and error related to Ansible mainly. I don't see in the attached logs anything I can follow to debug ovirt-engine or vdsm. I'm closing the bug. if the bug is reproduced, please attach the engine logs and vdsm logs, also clear and simple reproducing steps which this bug is missing.
Removing target milestone and release since we have insufficient data for handling this report.
During restore, customer is required to provide exactly the same datacenter and cluster names as were provided during initial deployment: Please enter the name of the datacenter where you want to deploy this hosted-engine host. Please note that if you are restoring a backup that contains info about other hosted-engine hosts, this value should exactly match the value used in the environment you are going to restore. [Default]: Please enter the name of the cluster where you want to deploy this hosted-engine host. Please note that if you are restoring a backup that contains info about other hosted-engine hosts, this value should exactly match the value used in the environment you are going to restore. [Default]: Works as designed.