Description of problem: Creating a backup on SHE environment on iscsi failed to create target vm with RuntimeError: Failed executing ansible-playbook. 2019-01-02 11:14:46,895+0200 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils.run:190 ansible-playbook stderr: 2019-01-02 11:14:46,896+0200 DEBUG otopi.context context._executeMethod:143 method exception Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/otopi/context.py", line 133, in _executeMethod method['method']() File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/gr-he-ansiblesetup/core/target_vm.py", line 220, in _closeup r = ah.run() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_setup/ansible_utils.py", line 194, in run raise RuntimeError(_('Failed executing ansible-playbook')) RuntimeError: Failed executing ansible-playbook 2019-01-02 11:14:46,897+0200 ERROR otopi.context context._executeMethod:152 Failed to execute stage 'Closing up': Failed executing ansible-playbook Version-Release number of selected component (if applicable): ovirt-hosted-engine-ha-2.2.19-1.el7ev.noarch ovirt-hosted-engine-setup-2.2.32-1.el7ev.noarch rhvm-appliance-4.2-20181212.0.el7.noarch ovirt-engine-sdk-python-3.6.9.1-1.el7ev.noarch python-ovirt-engine-sdk4-4.2.9-1.el7ev.x86_64 ansible-2.7.5-1.el7ae.noarch otopi-1.7.8-1.el7ev.noarch How reproducible: 100% Steps to Reproduce: 1. Redploy on iscsi from node0 environment, where SPM is not what we have in the backup && power management not configured 2. 3. Actual results: Restore fails. Expected results: Restore should succeed. Additional info: Normal deployment on iscsi works.
Please attach all relevant logs, e.g. using sosreport. Thanks.
Created attachment 1518896 [details] logs
Created attachment 1518897 [details] sosreport
(In reply to Pedut from comment #0) > Description of problem: > Creating a backup on SHE environment on iscsi failed to create target vm > with RuntimeError: Failed executing ansible-playbook. > fix: Description of problem: Restore of SHE environment on iscsi failed to create target vm with RuntimeError: Failed executing ansible-playbook. In addition it can be reproduced by redeploy on nfs from node0 environment, where the master storage domain as reported in the backup is still alive && power management configured and all hosts unreachable.
The real issue is: 2019-01-02 11:12:35,836+0200 INFO otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:100 TASK [Wait for OVF_STORE disk content] 2019-01-02 11:14:46,393+0200 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:94 {u'msg': u'All items completed', u'changed': True, u'results': [{u'_ansible_parsed': True, u'stderr_lines': [u'tar: fc826e34-8ab6-4ec9-9898-2dd7b22eb2ec.ovf: Not found in archive', u'tar: Exiting with failure status due to previous errors', u'xargs: sudo: terminated by signal 13'], u'changed': True, u'end': u'2019-01-02 11:14:46.268918', u'_ansible_item_label': {u'image_id': u'7584b09f-6fa2-47bd-be55-86cc20baef06', u'name': u'OVF_STORE', u'id': u'3f1a644e-d8cd-44de-86d6-938db76fd9e2'}, u'stdout': u'', u'failed': True, u'_ansible_item_result': True, u'msg': u'non-zero return code', u'rc': 2, u'start': u'2019-01-02 11:14:45.668382', u'attempts': 12, u'cmd': u"vdsm-client Image prepare storagepoolID=88ae038e-0de4-11e9-8804-001a4a161064 storagedomainID=f07b65e8-93d6-491c-841d-80f8a77c9019 imageID=3f1a644e-d8cd-44de-86d6-938db76fd9e2 volumeID=7584b09f-6fa2-47bd-be55-86cc20baef06 | grep path | awk '{ print $2 }' | xargs -I{} sudo -u vdsm dd if={} | tar -tvf - fc826e34-8ab6-4ec9-9898-2dd7b22eb2ec.ovf", u'item': {u'image_id': u'7584b09f-6fa2-47bd-be55-86cc20baef06', u'name': u'OVF_STORE', u'id': u'3f1a644e-d8cd-44de-86d6-938db76fd9e2'}, u'delta': u'0:00:00.600536', u'invocation': {u'module_args': {u'warn': True, u'executable': None, u'_uses_shell': True, u'_raw_params': u"vdsm-client Image prepare storagepoolID=88ae038e-0de4-11e9-8804-001a4a161064 storagedomainID=f07b65e8-93d6-491c-841d-80f8a77c9019 imageID=3f1a644e-d8cd-44de-86d6-938db76fd9e2 volumeID=7584b09f-6fa2-47bd-be55-86cc20baef06 | grep path | awk '{ print $2 }' | xargs -I{} sudo -u vdsm dd if={} | tar -tvf - fc826e34-8ab6-4ec9-9898-2dd7b22eb2ec.ovf", u'removes': None, u'argv': None, u'creates': None, u'chdir': None, u'stdin': None}}, u'stdout_lines': [], u'stderr': u'tar: fc826e34-8ab6-4ec9-9898-2dd7b22eb2ec.ovf: Not found in archive\ntar: Exiting with failure status due to previous errors\nxargs: sudo: terminated by signal 13', u'_ansible_no_log': False}]} 2019-01-02 11:14:46,493+0200 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:98 {u'_ansible_parsed': True, u'stderr_lines': [u'tar: fc826e34-8ab6-4ec9-9898-2dd7b22eb2ec.ovf: Not found in archive', u'tar: Exiting with failure status due to previous errors', u'xargs: sudo: terminated by signal 13'], u'changed': True, u'end': u'2019-01-02 11:14:46.268918', u'_ansible_item_label': {u'image_id': u'7584b09f-6fa2-47bd-be55-86cc20baef06', u'name': u'OVF_STORE', u'id': u'3f1a644e-d8cd-44de-86d6-938db76fd9e2'}, u'stdout': u'', u'failed': True, u'_ansible_item_result': True, u'msg': u'non-zero return code', u'rc': 2, u'start': u'2019-01-02 11:14:45.668382', u'attempts': 12, u'cmd': u"vdsm-client Image prepare storagepoolID=88ae038e-0de4-11e9-8804-001a4a161064 storagedomainID=f07b65e8-93d6-491c-841d-80f8a77c9019 imageID=3f1a644e-d8cd-44de-86d6-938db76fd9e2 volumeID=7584b09f-6fa2-47bd-be55-86cc20baef06 | grep path | awk '{ print $2 }' | xargs -I{} sudo -u vdsm dd if={} | tar -tvf - fc826e34-8ab6-4ec9-9898-2dd7b22eb2ec.ovf", u'item': {u'image_id': u'7584b09f-6fa2-47bd-be55-86cc20baef06', u'name': u'OVF_STORE', u'id': u'3f1a644e-d8cd-44de-86d6-938db76fd9e2'}, u'delta': u'0:00:00.600536', u'invocation': {u'module_args': {u'warn': True, u'executable': None, u'_uses_shell': True, u'_raw_params': u"vdsm-client Image prepare storagepoolID=88ae038e-0de4-11e9-8804-001a4a161064 storagedomainID=f07b65e8-93d6-491c-841d-80f8a77c9019 imageID=3f1a644e-d8cd-44de-86d6-938db76fd9e2 volumeID=7584b09f-6fa2-47bd-be55-86cc20baef06 | grep path | awk '{ print $2 }' | xargs -I{} sudo -u vdsm dd if={} | tar -tvf - fc826e34-8ab6-4ec9-9898-2dd7b22eb2ec.ovf", u'removes': None, u'argv': None, u'creates': None, u'chdir': None, u'stdin': None}}, u'stdout_lines': [], u'stderr': u'tar: fc826e34-8ab6-4ec9-9898-2dd7b22eb2ec.ovf: Not found in archive\ntar: Exiting with failure status due to previous errors\nxargs: sudo: terminated by signal 13', u'_ansible_no_log': False} Here the engine, probably due to a race condition, created the OVF_STORE volume without the data of the hosted-engine VM. This is exactly a duplicate of https://bugzilla.redhat.com/1644748 *** This bug has been marked as a duplicate of bug 1644748 ***