Hide Forgot
Created attachment 1453620 [details] hypervisor-lynx22/ovirt-engine-logs.tar.gz Description of problem: [HE] Failed to deploy hosted engine over NFS on updated rhel7.5 with ansible 2.6 Version-Release number of selected component (if applicable): rhvm-appliance-4.2-20180615.0.el7.noarch ovirt-hosted-engine-setup-2.2.22-1.el7ev.noarch ovirt-hosted-engine-ha-2.2.14-1.el7ev.noarch ansible 2.6.0rc3 How reproducible: 100% Steps to Reproduce: 1. Run deploy of hosted engine over NFS storage with the following command. hosted-engine --deploy --config-append=/etc/ansible-ovirt/answers --config-append=/root/heanswers.conf Actual results: Failed with the following error in: ovirt-hosted-engine-setup-ansible-create_target_vm-20180621184830-2fn52m.log: 2018-06-21 18:57:46,080+0300 ERROR ansible failed {'status': 'FAILED', 'ansible_type': 'task', 'ansible_task': u'Check OVF_STORE volume status', 'ansible_result': u'type: <type \'dict\'>\nstr: {\'msg\': u\'All items completed\', \'changed\': True, \'results\': [{\'_ansible_parsed\': True, \'stderr_lines\': [u"vdsm-client: Command Volume.getInfo with args {\'storagepoolID\': \'67522170-7569-11e8-bd28-00163e7be000\', \'storagedomainID\': \'151d8d47-5083-442e-8d0c-84222606c53f\', \'volumeID\': \'50a23531-b04a-484b\nrepr: {\'msg\': u\'All items completed\', \'changed\': True, \'results\': [{\'_ansible_parsed\': True, \'stderr_lines\': [u"vdsm-client: Command Volume.getInfo with args {\'storagepoolID\': \'67522170-7569-11e8-bd28-00163e7be000\', \'storagedomainID\': \'151d8d47-5083-442e-8d0c-84222606c53f\', \'volumeID\': \'50a23531-b04a-484b\ndir: [\'__class__\', \'__cmp__\', \'__contains__\', \'__delattr__\', \'__delitem__\', \'__doc__\', \'__eq__\', \'__format__\', \'__ge__\', \'__getattribute__\', \'__getitem__\', \'__gt__\', \'__hash__\', \'__init__\', \'__iter__\', \'__le__\', \'__len__\', \'__lt__\', \'__ne__\', \'__new__\', \'__reduce__\', \'__reduce_ex__\', \'__repr__\', \'__setattr__\', \'__setitem__\', \'__sizeof__\', \'__str__\', \'__subclasshook__\', \'clear\', \'copy\', \'fromkeys\', \'get\', \'has_key\', \'items\', \'iteritems\', \'iterkeys\', \'itervalues\', \'keys\', \'pop\', \'popitem\', \'setdefault\', \'update\', \'values\', \'viewitems\', \'viewkeys\', \'viewvalues\']\npprint: {\'changed\': True,\n \'msg\': u\'All items completed\',\n \'results\': [{\'_ansible_item_result\': True,\n \'_ansible_no_log\': False,\n \'_ansible_parsed\': True,\n \'attempts\': 12,\n u\'changed\': True,\n u\'cmd\': [u\'vdsm-client\',\n u\'V\n{\'msg\': u\'All items completed\', \'changed\': True, \'results\': [{\'_ansible_parsed\': True, \'stderr_lines.__doc__: "dict() -> new empty dictionary\\ndict(mapping) -> new dictionary initialized from a mapping object\'s\\n (key, value) pairs\\ndict(iterable) -> new dictionary initialized as if via:\\n d = {}\\n for k, v in iterable:\\n d[k] = v\\ndict(**kwargs) -> new dictionary initialized with the name=value pairs\\n in the keyword argument list. For example: dict(one=1, two=2)"\n{\'msg\': u\'All items completed\', \'changed\': True, \'results\': [{\'_ansible_parsed\': True, \'stderr_lines.__hash__: None', 'ansible_host': u'localhost', 'ansible_playbook': u'/usr/share/ovirt-hosted-engine-setup/ansible/create_target_vm.yml'} Expected results: Deploy successfully. Additional info: The setup is available for investigation: Host - lynx22.lab.eng.tlv2.redhat.com engine - hosted-engine-06.lab.eng.tlv2.redhat.com You can look also in the following job which deployed with ansible 2.6.0rc3: https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv-ge-deploy-4.2-khakimi/1/console the rhel-7.5 installed with the following repo: [root@lynx22 ~]# cat /etc/yum.repos.d/rhv-4.2.4-dependencies.repo ... [ansible-nighly-2.5.z] name=ansible nightly 2.5.z baseurl=http://download-node-02.eng.bos.redhat.com/nightly/updates/ANSIBLE/latest-ANSIBLE-2-RHEL-7/compose/Server/$basearch/os/ skip_if_unavailable=1 enabled=1 gpgcheck=0 Log attached
forgot to mention that deploy of hosted engine over iscsi [1] and fc [2] succeeded. [1] iscsi - https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv-4.2-ge-flow-webui-tier1/115/console [2] fc - https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv-4.2-ge-flow-tier1/474/console
Opened a bug an ansible 2.6 for this regression. We're going to work with them solving this.
This bug was caused by this patch [1] for ansible 2.6. On ansible <=2.5 the fetched IDs on nested attributes is given as a list with length of 1, while on ansible 2.6 it is given as string. Thus, when we execute [2] on ansible 2.6, imageID will be equal to the first letter instead of the whole ID and cause a failure of the deployment. [1] : https://github.com/ansible/ansible/pull/39736 [2] : https://github.com/oVirt/ovirt-hosted-engine-setup/blob/d2edecf14221bc436a656f546d33847b936bb0cc/src/ansible/create_target_vm.yml#L210
This bug report has Keywords: Regression or TestBlocker. Since no regressions or test blockers are allowed between releases, it is also being identified as a blocker for this release. Please resolve ASAP.
with RHHI use case, the same issue also seen while deploying HE over gluster storage domain. Following are the components used in this setup: ansible-2.6.0-0-3.rc3 gdeploy-2.0.2-27 glusterfs-3.8.4-52.12 <snip> 2018-06-26 10:42:46,016+0530 ERROR ansible failed {'status': 'FAILED', 'ansible_type': 'task', 'ansible_task': u'Check OVF_STORE volume status', 'ansible_result': u'type: <type \'dict\'>\nstr: {\'msg\': u\'All items completed\', \'changed\': True, \'results\': [{\'_ansible_parsed\': True, \'stderr_lines\': [u"vdsm-client: Command Volume.getInfo with args {\'storagepoolID\': \'c3300b1e-78fd-11e8-949a-004755204901\', \'storagedomainID\': \'ad70d1ab-deb0-4e76-9724-5bbfbfe8290e\', \'volumeID\': \'ff0ab2b3-9909-4f82\nrepr: {\'msg\': u\'All items completed\', \'changed\': True, \'results\': [{\'_ansible_parsed\': True, \'stderr_lines\': [u"vdsm-client: Command Volume.getInfo with args {\'storagepoolID\': \'c3300b1e-78fd-11e8-949a-004755204901\', \'storagedomainID\': \'ad70d1ab-deb0-4e76-9724-5bbfbfe8290e\', \'volumeID\': \'ff0ab2b3-9909-4f82\ndir: [\'__class__\', \'__cmp__\', \'__contains__\', \'__delattr__\', \'__delitem__\', \'__doc__\', \'__eq__\', \'__format__\', \'__ge__\', \'__getattribute__\', \'__getitem__\', \'__gt__\', \'__hash__\', \'__init__\', \'__iter__\', \'__le__\', \'__len__\', \'__lt__\', \'__ne__\', \'__new__\', \'__reduce__\', \'__reduce_ex__\', \'__repr__\', \'__setattr__\', \'__setitem__\', \'__sizeof__\', \'__str__\', \'__subclasshook__\', \'clear\', \'copy\', \'fromkeys\', \'get\', \'has_key\', \'items\', \'iteritems\', \'iterkeys\', \'itervalues\', \'keys\', \'pop\', \'popitem\', \'setdefault\', \'update\', \'values\', \'viewitems\', \'viewkeys\', \'viewvalues\']\npprint: {\'changed\': True,\n \'msg\': u\'All items completed\',\n \'results\': [{\'_ansible_item_result\': True,\n \'_ansible_no_log\': False,\n \'_ansible_parsed\': True,\n \'attempts\': 12,\n u\'changed\': True,\n u\'cmd\': [u\'vdsm-client\',\n u\'V\n{\'msg\': u\'All items completed\', \'changed\': True, \'results\': [{\'_ansible_parsed\': True, \'stderr_lines.__doc__: "dict() -> new empty dictionary\\ndict(mapping) -> new dictionary initialized from a mapping object\'s\\n (key, value) pairs\\ndict(iterable) -> new dictionary initialized as if via:\\n d = {}\\n for k, v in iterable:\\n d[k] = v\\ndict(**kwargs) -> new dictionary initialized with the name=value pairs\\n in the keyword argument list. For example: dict(one=1, two=2)"\n{\'msg\': u\'All items completed\', \'changed\': True, \'results\': [{\'_ansible_parsed\': True, \'stderr_lines.__hash__: None', 'ansible_host': u'localhost', 'ansible_playbook': u'/usr/share/ovirt-hosted-engine-setup/ansible/create_target_vm.yml'} 2018-06-26 10:42:46,016+0530 DEBUG ansible on_any args <ansible.executor.task_result.TaskResult object at 0x7fcc0594fb10> kwargs ignore_errors:None 2018-06-26 10:42:46,018+0530 INFO ansible stats {'status': 'FAILED', 'ansible_playbook_duration': 367.120274, 'ansible_result': u'type: <type \'dict\'>\nstr: {u\'hostedenginesm3.lab.eng.blr.redhat.com\': {\'unreachable\': 0, \'skipped\': 0, \'ok\': 11, \'changed\': 6, \'failures\': 0}, u\'localhost\': {\'unreachable\': 0, \'skipped\': 0, \'ok\': 32, \'changed\': 6, \'failures\': 1}}\nrepr: {u\'hostedenginesm3.lab.eng.blr.redhat.com\': {\'unreachable\': 0, \'skipped\': 0, \'ok\': 11, \'changed\': 6, \'failures\': 0}, u\'localhost\': {\'unreachable\': 0, \'skipped\': 0, \'ok\': 32, \'changed\': 6, \'failures\': 1}}\ndir: [\'__class__\', \'__cmp__\', \'__contains__\', \'__delattr__\', \'__delitem__\', \'__doc__\', \'__eq__\', \'__format__\', \'__ge__\', \'__getattribute__\', \'__getitem__\', \'__gt__\', \'__hash__\', \'__init__\', \'__iter__\', \'__le__\', \'__len__\', \'__lt__\', \'__ne__\', \'__new__\', \'__reduce__\', \'__reduce_ex__\', \'__repr__\', \'__setattr__\', \'__setitem__\', \'__sizeof__\', \'__str__\', \'__subclasshook__\', \'clear\', \'copy\', \'fromkeys\', \'get\', \'has_key\', \'items\', \'iteritems\', \'iterkeys\', \'itervalues\', \'keys\', \'pop\', \'popitem\', \'setdefault\', \'update\', \'values\', \'viewitems\', \'viewkeys\', \'viewvalues\']\npprint: {u\'hostedenginesm3.lab.eng.blr.redhat.com\': {\'changed\': 6,\n \'failures\': 0,\n \'ok\': 11,\n \'skipped\': 0,\n \'unreachable\': 0},\n u\'\n{u\'hostedenginesm3.lab.eng.blr.redhat.com\': {\'unreachable\': 0, \'skipped\': 0, \'ok\': 11, \'changed\': 6,.__doc__: "dict() -> new empty dictionary\\ndict(mapping) -> new dictionary initialized from a mapping object\'s\\n (key, value) pairs\\ndict(iterable) -> new dictionary initialized as if via:\\n d = {}\\n for k, v in iterable:\\n d[k] = v\\ndict(**kwargs) -> new dictionary initialized with the name=value pairs\\n in the keyword argument list. For example: dict(one=1, two=2)"\n{u\'hostedenginesm3.lab.eng.blr.redhat.com\': {\'unreachable\': 0, \'skipped\': 0, \'ok\': 11, \'changed\': 6,.__hash__: None', 'ansible_playbook': u'/usr/share/ovirt-hosted-engine-setup/ansible/create_target_vm.yml', 'ansible_type': 'finish'} </snip>
Verified with 4.2.4-7 ovirt-hosted-engine-setup-2.2.22.1-1.el7ev.noarch in: https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv-4.2-ge-flow-infra/1335/console
This bugzilla is included in oVirt 4.2.4 release, published on June 26th 2018. Since the problem described in this bug report should be resolved in oVirt 4.2.4 release, it has been closed with a resolution of CURRENT RELEASE. If the solution does not work for you, please open a new bug report.