Bug 1662912 - Restore SHE environment on iscsi failed to create target vm
Summary: Restore SHE environment on iscsi failed to create target vm
Keywords:
Status: CLOSED DUPLICATE of bug 1644748
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-hosted-engine-setup
Version: 4.2.8
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ovirt-4.2.8
: 4.3.0
Assignee: Simone Tiraboschi
QA Contact: meital avital
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-01-02 11:40 UTC by Pedut
Modified: 2019-01-07 13:41 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-01-07 13:41:37 UTC
oVirt Team: Integration
Target Upstream Version:


Attachments (Terms of Use)
logs (479.64 KB, text/plain)
2019-01-07 07:02 UTC, Pedut
no flags Details
sosreport (10.69 MB, application/x-xz)
2019-01-07 07:03 UTC, Pedut
no flags Details

Description Pedut 2019-01-02 11:40:07 UTC
Description of problem:
Creating a backup on SHE environment on iscsi failed to create target vm with RuntimeError: Failed executing ansible-playbook.

2019-01-02 11:14:46,895+0200 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils.run:190 ansible-playbook stderr:
2019-01-02 11:14:46,896+0200 DEBUG otopi.context context._executeMethod:143 method exception
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/otopi/context.py", line 133, in _executeMethod
    method['method']()
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/gr-he-ansiblesetup/core/target_vm.py", line 220, in _closeup
    r = ah.run()
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_setup/ansible_utils.py", line 194, in run
    raise RuntimeError(_('Failed executing ansible-playbook'))
RuntimeError: Failed executing ansible-playbook
2019-01-02 11:14:46,897+0200 ERROR otopi.context context._executeMethod:152 Failed to execute stage 'Closing up': Failed executing ansible-playbook


Version-Release number of selected component (if applicable):
ovirt-hosted-engine-ha-2.2.19-1.el7ev.noarch
ovirt-hosted-engine-setup-2.2.32-1.el7ev.noarch
rhvm-appliance-4.2-20181212.0.el7.noarch
ovirt-engine-sdk-python-3.6.9.1-1.el7ev.noarch
python-ovirt-engine-sdk4-4.2.9-1.el7ev.x86_64
ansible-2.7.5-1.el7ae.noarch
otopi-1.7.8-1.el7ev.noarch

How reproducible:
100%

Steps to Reproduce:
1. Redploy on iscsi from node0 environment, where SPM is not what we have in the backup && power management not configured
2.
3.

Actual results:
Restore fails.

Expected results:
Restore should succeed.

Additional info:
Normal deployment on iscsi works.

Comment 2 Yedidyah Bar David 2019-01-07 06:50:32 UTC
Please attach all relevant logs, e.g. using sosreport. Thanks.

Comment 4 Pedut 2019-01-07 07:02:36 UTC
Created attachment 1518896 [details]
logs

Comment 5 Pedut 2019-01-07 07:03:47 UTC
Created attachment 1518897 [details]
sosreport

Comment 6 Pedut 2019-01-07 09:23:45 UTC
(In reply to Pedut from comment #0)
> Description of problem:
> Creating a backup on SHE environment on iscsi failed to create target vm
> with RuntimeError: Failed executing ansible-playbook.
> 

fix:
Description of problem:
Restore of SHE environment on iscsi failed to create target vm with RuntimeError: Failed executing ansible-playbook.

In addition it can be reproduced by redeploy on nfs from node0 environment, where the master storage domain as reported in the backup is still alive && power management configured and all hosts unreachable.

Comment 7 Simone Tiraboschi 2019-01-07 13:41:37 UTC
The real issue is:

2019-01-02 11:12:35,836+0200 INFO otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:100 TASK [Wait for OVF_STORE disk content]
2019-01-02 11:14:46,393+0200 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:94 {u'msg': u'All items completed', u'changed': True, u'results': [{u'_ansible_parsed': True, u'stderr_lines': [u'tar: fc826e34-8ab6-4ec9-9898-2dd7b22eb2ec.ovf: Not found in archive', u'tar: Exiting with failure status due to previous errors', u'xargs: sudo: terminated by signal 13'], u'changed': True, u'end': u'2019-01-02 11:14:46.268918', u'_ansible_item_label': {u'image_id': u'7584b09f-6fa2-47bd-be55-86cc20baef06', u'name': u'OVF_STORE', u'id': u'3f1a644e-d8cd-44de-86d6-938db76fd9e2'}, u'stdout': u'', u'failed': True, u'_ansible_item_result': True, u'msg': u'non-zero return code', u'rc': 2, u'start': u'2019-01-02 11:14:45.668382', u'attempts': 12, u'cmd': u"vdsm-client Image prepare storagepoolID=88ae038e-0de4-11e9-8804-001a4a161064 storagedomainID=f07b65e8-93d6-491c-841d-80f8a77c9019 imageID=3f1a644e-d8cd-44de-86d6-938db76fd9e2 volumeID=7584b09f-6fa2-47bd-be55-86cc20baef06 | grep path | awk '{ print $2 }' | xargs -I{} sudo -u vdsm dd if={} | tar -tvf - fc826e34-8ab6-4ec9-9898-2dd7b22eb2ec.ovf", u'item': {u'image_id': u'7584b09f-6fa2-47bd-be55-86cc20baef06', u'name': u'OVF_STORE', u'id': u'3f1a644e-d8cd-44de-86d6-938db76fd9e2'}, u'delta': u'0:00:00.600536', u'invocation': {u'module_args': {u'warn': True, u'executable': None, u'_uses_shell': True, u'_raw_params': u"vdsm-client Image prepare storagepoolID=88ae038e-0de4-11e9-8804-001a4a161064 storagedomainID=f07b65e8-93d6-491c-841d-80f8a77c9019 imageID=3f1a644e-d8cd-44de-86d6-938db76fd9e2 volumeID=7584b09f-6fa2-47bd-be55-86cc20baef06 | grep path | awk '{ print $2 }' | xargs -I{} sudo -u vdsm dd if={} | tar -tvf - fc826e34-8ab6-4ec9-9898-2dd7b22eb2ec.ovf", u'removes': None, u'argv': None, u'creates': None, u'chdir': None, u'stdin': None}}, u'stdout_lines': [], u'stderr': u'tar: fc826e34-8ab6-4ec9-9898-2dd7b22eb2ec.ovf: Not found in archive\ntar: Exiting with failure status due to previous errors\nxargs: sudo: terminated by signal 13', u'_ansible_no_log': False}]}
2019-01-02 11:14:46,493+0200 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:98 {u'_ansible_parsed': True, u'stderr_lines': [u'tar: fc826e34-8ab6-4ec9-9898-2dd7b22eb2ec.ovf: Not found in archive', u'tar: Exiting with failure status due to previous errors', u'xargs: sudo: terminated by signal 13'], u'changed': True, u'end': u'2019-01-02 11:14:46.268918', u'_ansible_item_label': {u'image_id': u'7584b09f-6fa2-47bd-be55-86cc20baef06', u'name': u'OVF_STORE', u'id': u'3f1a644e-d8cd-44de-86d6-938db76fd9e2'}, u'stdout': u'', u'failed': True, u'_ansible_item_result': True, u'msg': u'non-zero return code', u'rc': 2, u'start': u'2019-01-02 11:14:45.668382', u'attempts': 12, u'cmd': u"vdsm-client Image prepare storagepoolID=88ae038e-0de4-11e9-8804-001a4a161064 storagedomainID=f07b65e8-93d6-491c-841d-80f8a77c9019 imageID=3f1a644e-d8cd-44de-86d6-938db76fd9e2 volumeID=7584b09f-6fa2-47bd-be55-86cc20baef06 | grep path | awk '{ print $2 }' | xargs -I{} sudo -u vdsm dd if={} | tar -tvf - fc826e34-8ab6-4ec9-9898-2dd7b22eb2ec.ovf", u'item': {u'image_id': u'7584b09f-6fa2-47bd-be55-86cc20baef06', u'name': u'OVF_STORE', u'id': u'3f1a644e-d8cd-44de-86d6-938db76fd9e2'}, u'delta': u'0:00:00.600536', u'invocation': {u'module_args': {u'warn': True, u'executable': None, u'_uses_shell': True, u'_raw_params': u"vdsm-client Image prepare storagepoolID=88ae038e-0de4-11e9-8804-001a4a161064 storagedomainID=f07b65e8-93d6-491c-841d-80f8a77c9019 imageID=3f1a644e-d8cd-44de-86d6-938db76fd9e2 volumeID=7584b09f-6fa2-47bd-be55-86cc20baef06 | grep path | awk '{ print $2 }' | xargs -I{} sudo -u vdsm dd if={} | tar -tvf - fc826e34-8ab6-4ec9-9898-2dd7b22eb2ec.ovf", u'removes': None, u'argv': None, u'creates': None, u'chdir': None, u'stdin': None}}, u'stdout_lines': [], u'stderr': u'tar: fc826e34-8ab6-4ec9-9898-2dd7b22eb2ec.ovf: Not found in archive\ntar: Exiting with failure status due to previous errors\nxargs: sudo: terminated by signal 13', u'_ansible_no_log': False}

Here the engine, probably due to a race condition, created the OVF_STORE volume without the data of the hosted-engine VM.
This is exactly a duplicate of https://bugzilla.redhat.com/1644748

*** This bug has been marked as a duplicate of bug 1644748 ***


Note You need to log in before you can comment on or make changes to this bug.