Bug 1594024 - [HE] Failed to deploy hosted engine over NFS on updated rhel7.5 with ansible 2.6
Summary: [HE] Failed to deploy hosted engine over NFS on updated rhel7.5 with ansible 2.6
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: ovirt-hosted-engine-setup
Classification: oVirt
Component: General
Version: 2.2.22
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ovirt-4.2.4-1
: ---
Assignee: Ido Rosenzwig
QA Contact: Kobi Hakimi
URL:
Whiteboard:
Depends On: 1594119
Blocks: ovirt-hosted-engine-setup-2.2.22.1
TreeView+ depends on / blocked
 
Reported: 2018-06-21 21:45 UTC by Kobi Hakimi
Modified: 2018-06-28 06:41 UTC (History)
13 users (show)

Fixed In Version: ovirt-hosted-engine-setup-2.2.22.1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-06-28 06:41:52 UTC
oVirt Team: Integration
Embargoed:
rule-engine: ovirt-4.2+
rule-engine: blocker+


Attachments (Terms of Use)
hypervisor-lynx22/ovirt-engine-logs.tar.gz (4.32 MB, application/x-gzip)
2018-06-21 21:45 UTC, Kobi Hakimi
no flags Details


Links
System ID Private Priority Status Summary Last Updated
oVirt gerrit 92456 0 'None' 'MERGED' 'Ansible: Fix imageID value on vdsm-client command' 2019-12-03 04:29:17 UTC
oVirt gerrit 92482 0 'None' 'MERGED' 'Ansible: Fix imageID value on vdsm-client command' 2019-12-03 04:29:17 UTC
oVirt gerrit 92499 0 'None' 'MERGED' 'Ansible: Fix imageID value on vdsm-client command' 2019-12-03 04:29:17 UTC

Description Kobi Hakimi 2018-06-21 21:45:51 UTC
Created attachment 1453620 [details]
hypervisor-lynx22/ovirt-engine-logs.tar.gz

Description of problem:
[HE] Failed to deploy hosted engine over NFS on updated rhel7.5 with ansible 2.6

Version-Release number of selected component (if applicable):
rhvm-appliance-4.2-20180615.0.el7.noarch
ovirt-hosted-engine-setup-2.2.22-1.el7ev.noarch
ovirt-hosted-engine-ha-2.2.14-1.el7ev.noarch
ansible 2.6.0rc3

How reproducible:
100%

Steps to Reproduce:
1. Run deploy of hosted engine over NFS storage with the following command.
hosted-engine --deploy --config-append=/etc/ansible-ovirt/answers --config-append=/root/heanswers.conf

Actual results:
Failed with the following error in:
ovirt-hosted-engine-setup-ansible-create_target_vm-20180621184830-2fn52m.log:

2018-06-21 18:57:46,080+0300 ERROR ansible failed {'status': 'FAILED', 'ansible_type': 'task', 'ansible_task': u'Check OVF_STORE volume status', 'ansible_result': u'type: <type \'dict\'>\nstr: {\'msg\': u\'All items completed\', \'changed\': True, \'results\': [{\'_ansible_parsed\': True, \'stderr_lines\': [u"vdsm-client: Command Volume.getInfo with args {\'storagepoolID\': \'67522170-7569-11e8-bd28-00163e7be000\', \'storagedomainID\': \'151d8d47-5083-442e-8d0c-84222606c53f\', \'volumeID\': \'50a23531-b04a-484b\nrepr: {\'msg\': u\'All items completed\', \'changed\': True, \'results\': [{\'_ansible_parsed\': True, \'stderr_lines\': [u"vdsm-client: Command Volume.getInfo with args {\'storagepoolID\': \'67522170-7569-11e8-bd28-00163e7be000\', \'storagedomainID\': \'151d8d47-5083-442e-8d0c-84222606c53f\', \'volumeID\': \'50a23531-b04a-484b\ndir: [\'__class__\', \'__cmp__\', \'__contains__\', \'__delattr__\', \'__delitem__\', \'__doc__\', \'__eq__\', \'__format__\', \'__ge__\', \'__getattribute__\', \'__getitem__\', \'__gt__\', \'__hash__\', \'__init__\', \'__iter__\', \'__le__\', \'__len__\', \'__lt__\', \'__ne__\', \'__new__\', \'__reduce__\', \'__reduce_ex__\', \'__repr__\', \'__setattr__\', \'__setitem__\', \'__sizeof__\', \'__str__\', \'__subclasshook__\', \'clear\', \'copy\', \'fromkeys\', \'get\', \'has_key\', \'items\', \'iteritems\', \'iterkeys\', \'itervalues\', \'keys\', \'pop\', \'popitem\', \'setdefault\', \'update\', \'values\', \'viewitems\', \'viewkeys\', \'viewvalues\']\npprint: {\'changed\': True,\n \'msg\': u\'All items completed\',\n \'results\': [{\'_ansible_item_result\': True,\n              \'_ansible_no_log\': False,\n              \'_ansible_parsed\': True,\n              \'attempts\': 12,\n              u\'changed\': True,\n              u\'cmd\': [u\'vdsm-client\',\n                       u\'V\n{\'msg\': u\'All items completed\', \'changed\': True, \'results\': [{\'_ansible_parsed\': True, \'stderr_lines.__doc__: "dict() -> new empty dictionary\\ndict(mapping) -> new dictionary initialized from a mapping object\'s\\n    (key, value) pairs\\ndict(iterable) -> new dictionary initialized as if via:\\n    d = {}\\n    for k, v in iterable:\\n        d[k] = v\\ndict(**kwargs) -> new dictionary initialized with the name=value pairs\\n    in the keyword argument list.  For example:  dict(one=1, two=2)"\n{\'msg\': u\'All items completed\', \'changed\': True, \'results\': [{\'_ansible_parsed\': True, \'stderr_lines.__hash__: None', 'ansible_host': u'localhost', 'ansible_playbook': u'/usr/share/ovirt-hosted-engine-setup/ansible/create_target_vm.yml'}


Expected results:
Deploy successfully.

Additional info:
The setup is available for investigation:
Host - lynx22.lab.eng.tlv2.redhat.com
engine - hosted-engine-06.lab.eng.tlv2.redhat.com

You can look also in the following job which deployed with ansible 2.6.0rc3:
https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv-ge-deploy-4.2-khakimi/1/console

the rhel-7.5 installed with the following repo:
[root@lynx22 ~]# cat /etc/yum.repos.d/rhv-4.2.4-dependencies.repo
...
[ansible-nighly-2.5.z]
name=ansible nightly 2.5.z
baseurl=http://download-node-02.eng.bos.redhat.com/nightly/updates/ANSIBLE/latest-ANSIBLE-2-RHEL-7/compose/Server/$basearch/os/
skip_if_unavailable=1
enabled=1
gpgcheck=0

Log attached

Comment 1 Kobi Hakimi 2018-06-21 21:51:04 UTC
forgot to mention that deploy of hosted engine over iscsi [1] and fc [2] succeeded.

[1] iscsi - https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv-4.2-ge-flow-webui-tier1/115/console
[2] fc - https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv-4.2-ge-flow-tier1/474/console

Comment 2 Sandro Bonazzola 2018-06-22 07:07:04 UTC
Opened a bug an ansible 2.6 for this regression. We're going to work with them solving this.

Comment 4 Ido Rosenzwig 2018-06-24 13:42:36 UTC
This bug was caused by this patch [1] for ansible 2.6.

On ansible <=2.5 the fetched IDs on nested attributes is given as a list with length of 1, while on ansible 2.6 it is given as string.

Thus, when we execute [2] on ansible 2.6, imageID will be equal to the first letter instead of the whole ID and cause a failure of the deployment.


[1] : https://github.com/ansible/ansible/pull/39736
[2] : https://github.com/oVirt/ovirt-hosted-engine-setup/blob/d2edecf14221bc436a656f546d33847b936bb0cc/src/ansible/create_target_vm.yml#L210

Comment 5 Red Hat Bugzilla Rules Engine 2018-06-25 08:44:02 UTC
This bug report has Keywords: Regression or TestBlocker.
Since no regressions or test blockers are allowed between releases, it is also being identified as a blocker for this release. Please resolve ASAP.

Comment 6 bipin 2018-06-26 06:01:37 UTC
with RHHI use case, the same issue also seen while deploying HE over gluster storage domain.

Following are the components used in this setup:
ansible-2.6.0-0-3.rc3
gdeploy-2.0.2-27
glusterfs-3.8.4-52.12

<snip>
2018-06-26 10:42:46,016+0530 ERROR ansible failed {'status': 'FAILED', 'ansible_type': 'task', 'ansible_task': u'Check OVF_STORE volume status', 'ansible_result': u'type: <type \'dict\'>\nstr: {\'msg\': u\'All items completed\', \'changed\': True, \'results\': [{\'_ansible_parsed\': True, \'stderr_lines\': [u"vdsm-client: Command Volume.getInfo with args {\'storagepoolID\': \'c3300b1e-78fd-11e8-949a-004755204901\', \'storagedomainID\': \'ad70d1ab-deb0-4e76-9724-5bbfbfe8290e\', \'volumeID\': \'ff0ab2b3-9909-4f82\nrepr: {\'msg\': u\'All items completed\', \'changed\': True, \'results\': [{\'_ansible_parsed\': True, \'stderr_lines\': [u"vdsm-client: Command Volume.getInfo with args {\'storagepoolID\': \'c3300b1e-78fd-11e8-949a-004755204901\', \'storagedomainID\': \'ad70d1ab-deb0-4e76-9724-5bbfbfe8290e\', \'volumeID\': \'ff0ab2b3-9909-4f82\ndir: [\'__class__\', \'__cmp__\', \'__contains__\', \'__delattr__\', \'__delitem__\', \'__doc__\', \'__eq__\', \'__format__\', \'__ge__\', \'__getattribute__\', \'__getitem__\', \'__gt__\', \'__hash__\', \'__init__\', \'__iter__\', \'__le__\', \'__len__\', \'__lt__\', \'__ne__\', \'__new__\', \'__reduce__\', \'__reduce_ex__\', \'__repr__\', \'__setattr__\', \'__setitem__\', \'__sizeof__\', \'__str__\', \'__subclasshook__\', \'clear\', \'copy\', \'fromkeys\', \'get\', \'has_key\', \'items\', \'iteritems\', \'iterkeys\', \'itervalues\', \'keys\', \'pop\', \'popitem\', \'setdefault\', \'update\', \'values\', \'viewitems\', \'viewkeys\', \'viewvalues\']\npprint: {\'changed\': True,\n \'msg\': u\'All items completed\',\n \'results\': [{\'_ansible_item_result\': True,\n              \'_ansible_no_log\': False,\n              \'_ansible_parsed\': True,\n              \'attempts\': 12,\n              u\'changed\': True,\n              u\'cmd\': [u\'vdsm-client\',\n                       u\'V\n{\'msg\': u\'All items completed\', \'changed\': True, \'results\': [{\'_ansible_parsed\': True, \'stderr_lines.__doc__: "dict() -> new empty dictionary\\ndict(mapping) -> new dictionary initialized from a mapping object\'s\\n    (key, value) pairs\\ndict(iterable) -> new dictionary initialized as if via:\\n    d = {}\\n    for k, v in iterable:\\n        d[k] = v\\ndict(**kwargs) -> new dictionary initialized with the name=value pairs\\n    in the keyword argument list.  For example:  dict(one=1, two=2)"\n{\'msg\': u\'All items completed\', \'changed\': True, \'results\': [{\'_ansible_parsed\': True, \'stderr_lines.__hash__: None', 'ansible_host': u'localhost', 'ansible_playbook': u'/usr/share/ovirt-hosted-engine-setup/ansible/create_target_vm.yml'}
2018-06-26 10:42:46,016+0530 DEBUG ansible on_any args <ansible.executor.task_result.TaskResult object at 0x7fcc0594fb10> kwargs ignore_errors:None 
2018-06-26 10:42:46,018+0530 INFO ansible stats {'status': 'FAILED', 'ansible_playbook_duration': 367.120274, 'ansible_result': u'type: <type \'dict\'>\nstr: {u\'hostedenginesm3.lab.eng.blr.redhat.com\': {\'unreachable\': 0, \'skipped\': 0, \'ok\': 11, \'changed\': 6, \'failures\': 0}, u\'localhost\': {\'unreachable\': 0, \'skipped\': 0, \'ok\': 32, \'changed\': 6, \'failures\': 1}}\nrepr: {u\'hostedenginesm3.lab.eng.blr.redhat.com\': {\'unreachable\': 0, \'skipped\': 0, \'ok\': 11, \'changed\': 6, \'failures\': 0}, u\'localhost\': {\'unreachable\': 0, \'skipped\': 0, \'ok\': 32, \'changed\': 6, \'failures\': 1}}\ndir: [\'__class__\', \'__cmp__\', \'__contains__\', \'__delattr__\', \'__delitem__\', \'__doc__\', \'__eq__\', \'__format__\', \'__ge__\', \'__getattribute__\', \'__getitem__\', \'__gt__\', \'__hash__\', \'__init__\', \'__iter__\', \'__le__\', \'__len__\', \'__lt__\', \'__ne__\', \'__new__\', \'__reduce__\', \'__reduce_ex__\', \'__repr__\', \'__setattr__\', \'__setitem__\', \'__sizeof__\', \'__str__\', \'__subclasshook__\', \'clear\', \'copy\', \'fromkeys\', \'get\', \'has_key\', \'items\', \'iteritems\', \'iterkeys\', \'itervalues\', \'keys\', \'pop\', \'popitem\', \'setdefault\', \'update\', \'values\', \'viewitems\', \'viewkeys\', \'viewvalues\']\npprint: {u\'hostedenginesm3.lab.eng.blr.redhat.com\': {\'changed\': 6,\n                                             \'failures\': 0,\n                                             \'ok\': 11,\n                                             \'skipped\': 0,\n                                             \'unreachable\': 0},\n u\'\n{u\'hostedenginesm3.lab.eng.blr.redhat.com\': {\'unreachable\': 0, \'skipped\': 0, \'ok\': 11, \'changed\': 6,.__doc__: "dict() -> new empty dictionary\\ndict(mapping) -> new dictionary initialized from a mapping object\'s\\n    (key, value) pairs\\ndict(iterable) -> new dictionary initialized as if via:\\n    d = {}\\n    for k, v in iterable:\\n        d[k] = v\\ndict(**kwargs) -> new dictionary initialized with the name=value pairs\\n    in the keyword argument list.  For example:  dict(one=1, two=2)"\n{u\'hostedenginesm3.lab.eng.blr.redhat.com\': {\'unreachable\': 0, \'skipped\': 0, \'ok\': 11, \'changed\': 6,.__hash__: None', 'ansible_playbook': u'/usr/share/ovirt-hosted-engine-setup/ansible/create_target_vm.yml', 'ansible_type': 'finish'}
</snip>

Comment 7 Kobi Hakimi 2018-06-27 12:18:52 UTC
Verified with 4.2.4-7
ovirt-hosted-engine-setup-2.2.22.1-1.el7ev.noarch

in: https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv-4.2-ge-flow-infra/1335/console

Comment 8 Sandro Bonazzola 2018-06-28 06:41:52 UTC
This bugzilla is included in oVirt 4.2.4 release, published on June 26th 2018.

Since the problem described in this bug report should be
resolved in oVirt 4.2.4 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.