Bug 1594024

Summary: [HE] Failed to deploy hosted engine over NFS on updated rhel7.5 with ansible 2.6
Product: [oVirt] ovirt-hosted-engine-setup Reporter: Kobi Hakimi <khakimi>
Component: GeneralAssignee: Ido Rosenzwig <irosenzw>
Status: CLOSED CURRENTRELEASE QA Contact: Kobi Hakimi <khakimi>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 2.2.22CC: bugs, cshao, ebenahar, huzhao, mavital, pbalogh, qiyuan, rbarry, sbonazzo, stirabos, yaniwang, ycui, yzhao
Target Milestone: ovirt-4.2.4-1Keywords: Automation, Regression
Target Release: ---Flags: rule-engine: ovirt-4.2+
rule-engine: blocker+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ovirt-hosted-engine-setup-2.2.22.1 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-06-28 06:41:52 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Integration RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1594119    
Bug Blocks: 1594893    
Attachments:
Description Flags
hypervisor-lynx22/ovirt-engine-logs.tar.gz none

Description Kobi Hakimi 2018-06-21 21:45:51 UTC
Created attachment 1453620 [details]
hypervisor-lynx22/ovirt-engine-logs.tar.gz

Description of problem:
[HE] Failed to deploy hosted engine over NFS on updated rhel7.5 with ansible 2.6

Version-Release number of selected component (if applicable):
rhvm-appliance-4.2-20180615.0.el7.noarch
ovirt-hosted-engine-setup-2.2.22-1.el7ev.noarch
ovirt-hosted-engine-ha-2.2.14-1.el7ev.noarch
ansible 2.6.0rc3

How reproducible:
100%

Steps to Reproduce:
1. Run deploy of hosted engine over NFS storage with the following command.
hosted-engine --deploy --config-append=/etc/ansible-ovirt/answers --config-append=/root/heanswers.conf

Actual results:
Failed with the following error in:
ovirt-hosted-engine-setup-ansible-create_target_vm-20180621184830-2fn52m.log:

2018-06-21 18:57:46,080+0300 ERROR ansible failed {'status': 'FAILED', 'ansible_type': 'task', 'ansible_task': u'Check OVF_STORE volume status', 'ansible_result': u'type: <type \'dict\'>\nstr: {\'msg\': u\'All items completed\', \'changed\': True, \'results\': [{\'_ansible_parsed\': True, \'stderr_lines\': [u"vdsm-client: Command Volume.getInfo with args {\'storagepoolID\': \'67522170-7569-11e8-bd28-00163e7be000\', \'storagedomainID\': \'151d8d47-5083-442e-8d0c-84222606c53f\', \'volumeID\': \'50a23531-b04a-484b\nrepr: {\'msg\': u\'All items completed\', \'changed\': True, \'results\': [{\'_ansible_parsed\': True, \'stderr_lines\': [u"vdsm-client: Command Volume.getInfo with args {\'storagepoolID\': \'67522170-7569-11e8-bd28-00163e7be000\', \'storagedomainID\': \'151d8d47-5083-442e-8d0c-84222606c53f\', \'volumeID\': \'50a23531-b04a-484b\ndir: [\'__class__\', \'__cmp__\', \'__contains__\', \'__delattr__\', \'__delitem__\', \'__doc__\', \'__eq__\', \'__format__\', \'__ge__\', \'__getattribute__\', \'__getitem__\', \'__gt__\', \'__hash__\', \'__init__\', \'__iter__\', \'__le__\', \'__len__\', \'__lt__\', \'__ne__\', \'__new__\', \'__reduce__\', \'__reduce_ex__\', \'__repr__\', \'__setattr__\', \'__setitem__\', \'__sizeof__\', \'__str__\', \'__subclasshook__\', \'clear\', \'copy\', \'fromkeys\', \'get\', \'has_key\', \'items\', \'iteritems\', \'iterkeys\', \'itervalues\', \'keys\', \'pop\', \'popitem\', \'setdefault\', \'update\', \'values\', \'viewitems\', \'viewkeys\', \'viewvalues\']\npprint: {\'changed\': True,\n \'msg\': u\'All items completed\',\n \'results\': [{\'_ansible_item_result\': True,\n              \'_ansible_no_log\': False,\n              \'_ansible_parsed\': True,\n              \'attempts\': 12,\n              u\'changed\': True,\n              u\'cmd\': [u\'vdsm-client\',\n                       u\'V\n{\'msg\': u\'All items completed\', \'changed\': True, \'results\': [{\'_ansible_parsed\': True, \'stderr_lines.__doc__: "dict() -> new empty dictionary\\ndict(mapping) -> new dictionary initialized from a mapping object\'s\\n    (key, value) pairs\\ndict(iterable) -> new dictionary initialized as if via:\\n    d = {}\\n    for k, v in iterable:\\n        d[k] = v\\ndict(**kwargs) -> new dictionary initialized with the name=value pairs\\n    in the keyword argument list.  For example:  dict(one=1, two=2)"\n{\'msg\': u\'All items completed\', \'changed\': True, \'results\': [{\'_ansible_parsed\': True, \'stderr_lines.__hash__: None', 'ansible_host': u'localhost', 'ansible_playbook': u'/usr/share/ovirt-hosted-engine-setup/ansible/create_target_vm.yml'}


Expected results:
Deploy successfully.

Additional info:
The setup is available for investigation:
Host - lynx22.lab.eng.tlv2.redhat.com
engine - hosted-engine-06.lab.eng.tlv2.redhat.com

You can look also in the following job which deployed with ansible 2.6.0rc3:
https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv-ge-deploy-4.2-khakimi/1/console

the rhel-7.5 installed with the following repo:
[root@lynx22 ~]# cat /etc/yum.repos.d/rhv-4.2.4-dependencies.repo
...
[ansible-nighly-2.5.z]
name=ansible nightly 2.5.z
baseurl=http://download-node-02.eng.bos.redhat.com/nightly/updates/ANSIBLE/latest-ANSIBLE-2-RHEL-7/compose/Server/$basearch/os/
skip_if_unavailable=1
enabled=1
gpgcheck=0

Log attached

Comment 1 Kobi Hakimi 2018-06-21 21:51:04 UTC
forgot to mention that deploy of hosted engine over iscsi [1] and fc [2] succeeded.

[1] iscsi - https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv-4.2-ge-flow-webui-tier1/115/console
[2] fc - https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv-4.2-ge-flow-tier1/474/console

Comment 2 Sandro Bonazzola 2018-06-22 07:07:04 UTC
Opened a bug an ansible 2.6 for this regression. We're going to work with them solving this.

Comment 4 Ido Rosenzwig 2018-06-24 13:42:36 UTC
This bug was caused by this patch [1] for ansible 2.6.

On ansible <=2.5 the fetched IDs on nested attributes is given as a list with length of 1, while on ansible 2.6 it is given as string.

Thus, when we execute [2] on ansible 2.6, imageID will be equal to the first letter instead of the whole ID and cause a failure of the deployment.


[1] : https://github.com/ansible/ansible/pull/39736
[2] : https://github.com/oVirt/ovirt-hosted-engine-setup/blob/d2edecf14221bc436a656f546d33847b936bb0cc/src/ansible/create_target_vm.yml#L210

Comment 5 Red Hat Bugzilla Rules Engine 2018-06-25 08:44:02 UTC
This bug report has Keywords: Regression or TestBlocker.
Since no regressions or test blockers are allowed between releases, it is also being identified as a blocker for this release. Please resolve ASAP.

Comment 6 bipin 2018-06-26 06:01:37 UTC
with RHHI use case, the same issue also seen while deploying HE over gluster storage domain.

Following are the components used in this setup:
ansible-2.6.0-0-3.rc3
gdeploy-2.0.2-27
glusterfs-3.8.4-52.12

<snip>
2018-06-26 10:42:46,016+0530 ERROR ansible failed {'status': 'FAILED', 'ansible_type': 'task', 'ansible_task': u'Check OVF_STORE volume status', 'ansible_result': u'type: <type \'dict\'>\nstr: {\'msg\': u\'All items completed\', \'changed\': True, \'results\': [{\'_ansible_parsed\': True, \'stderr_lines\': [u"vdsm-client: Command Volume.getInfo with args {\'storagepoolID\': \'c3300b1e-78fd-11e8-949a-004755204901\', \'storagedomainID\': \'ad70d1ab-deb0-4e76-9724-5bbfbfe8290e\', \'volumeID\': \'ff0ab2b3-9909-4f82\nrepr: {\'msg\': u\'All items completed\', \'changed\': True, \'results\': [{\'_ansible_parsed\': True, \'stderr_lines\': [u"vdsm-client: Command Volume.getInfo with args {\'storagepoolID\': \'c3300b1e-78fd-11e8-949a-004755204901\', \'storagedomainID\': \'ad70d1ab-deb0-4e76-9724-5bbfbfe8290e\', \'volumeID\': \'ff0ab2b3-9909-4f82\ndir: [\'__class__\', \'__cmp__\', \'__contains__\', \'__delattr__\', \'__delitem__\', \'__doc__\', \'__eq__\', \'__format__\', \'__ge__\', \'__getattribute__\', \'__getitem__\', \'__gt__\', \'__hash__\', \'__init__\', \'__iter__\', \'__le__\', \'__len__\', \'__lt__\', \'__ne__\', \'__new__\', \'__reduce__\', \'__reduce_ex__\', \'__repr__\', \'__setattr__\', \'__setitem__\', \'__sizeof__\', \'__str__\', \'__subclasshook__\', \'clear\', \'copy\', \'fromkeys\', \'get\', \'has_key\', \'items\', \'iteritems\', \'iterkeys\', \'itervalues\', \'keys\', \'pop\', \'popitem\', \'setdefault\', \'update\', \'values\', \'viewitems\', \'viewkeys\', \'viewvalues\']\npprint: {\'changed\': True,\n \'msg\': u\'All items completed\',\n \'results\': [{\'_ansible_item_result\': True,\n              \'_ansible_no_log\': False,\n              \'_ansible_parsed\': True,\n              \'attempts\': 12,\n              u\'changed\': True,\n              u\'cmd\': [u\'vdsm-client\',\n                       u\'V\n{\'msg\': u\'All items completed\', \'changed\': True, \'results\': [{\'_ansible_parsed\': True, \'stderr_lines.__doc__: "dict() -> new empty dictionary\\ndict(mapping) -> new dictionary initialized from a mapping object\'s\\n    (key, value) pairs\\ndict(iterable) -> new dictionary initialized as if via:\\n    d = {}\\n    for k, v in iterable:\\n        d[k] = v\\ndict(**kwargs) -> new dictionary initialized with the name=value pairs\\n    in the keyword argument list.  For example:  dict(one=1, two=2)"\n{\'msg\': u\'All items completed\', \'changed\': True, \'results\': [{\'_ansible_parsed\': True, \'stderr_lines.__hash__: None', 'ansible_host': u'localhost', 'ansible_playbook': u'/usr/share/ovirt-hosted-engine-setup/ansible/create_target_vm.yml'}
2018-06-26 10:42:46,016+0530 DEBUG ansible on_any args <ansible.executor.task_result.TaskResult object at 0x7fcc0594fb10> kwargs ignore_errors:None 
2018-06-26 10:42:46,018+0530 INFO ansible stats {'status': 'FAILED', 'ansible_playbook_duration': 367.120274, 'ansible_result': u'type: <type \'dict\'>\nstr: {u\'hostedenginesm3.lab.eng.blr.redhat.com\': {\'unreachable\': 0, \'skipped\': 0, \'ok\': 11, \'changed\': 6, \'failures\': 0}, u\'localhost\': {\'unreachable\': 0, \'skipped\': 0, \'ok\': 32, \'changed\': 6, \'failures\': 1}}\nrepr: {u\'hostedenginesm3.lab.eng.blr.redhat.com\': {\'unreachable\': 0, \'skipped\': 0, \'ok\': 11, \'changed\': 6, \'failures\': 0}, u\'localhost\': {\'unreachable\': 0, \'skipped\': 0, \'ok\': 32, \'changed\': 6, \'failures\': 1}}\ndir: [\'__class__\', \'__cmp__\', \'__contains__\', \'__delattr__\', \'__delitem__\', \'__doc__\', \'__eq__\', \'__format__\', \'__ge__\', \'__getattribute__\', \'__getitem__\', \'__gt__\', \'__hash__\', \'__init__\', \'__iter__\', \'__le__\', \'__len__\', \'__lt__\', \'__ne__\', \'__new__\', \'__reduce__\', \'__reduce_ex__\', \'__repr__\', \'__setattr__\', \'__setitem__\', \'__sizeof__\', \'__str__\', \'__subclasshook__\', \'clear\', \'copy\', \'fromkeys\', \'get\', \'has_key\', \'items\', \'iteritems\', \'iterkeys\', \'itervalues\', \'keys\', \'pop\', \'popitem\', \'setdefault\', \'update\', \'values\', \'viewitems\', \'viewkeys\', \'viewvalues\']\npprint: {u\'hostedenginesm3.lab.eng.blr.redhat.com\': {\'changed\': 6,\n                                             \'failures\': 0,\n                                             \'ok\': 11,\n                                             \'skipped\': 0,\n                                             \'unreachable\': 0},\n u\'\n{u\'hostedenginesm3.lab.eng.blr.redhat.com\': {\'unreachable\': 0, \'skipped\': 0, \'ok\': 11, \'changed\': 6,.__doc__: "dict() -> new empty dictionary\\ndict(mapping) -> new dictionary initialized from a mapping object\'s\\n    (key, value) pairs\\ndict(iterable) -> new dictionary initialized as if via:\\n    d = {}\\n    for k, v in iterable:\\n        d[k] = v\\ndict(**kwargs) -> new dictionary initialized with the name=value pairs\\n    in the keyword argument list.  For example:  dict(one=1, two=2)"\n{u\'hostedenginesm3.lab.eng.blr.redhat.com\': {\'unreachable\': 0, \'skipped\': 0, \'ok\': 11, \'changed\': 6,.__hash__: None', 'ansible_playbook': u'/usr/share/ovirt-hosted-engine-setup/ansible/create_target_vm.yml', 'ansible_type': 'finish'}
</snip>

Comment 7 Kobi Hakimi 2018-06-27 12:18:52 UTC
Verified with 4.2.4-7
ovirt-hosted-engine-setup-2.2.22.1-1.el7ev.noarch

in: https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv-4.2-ge-flow-infra/1335/console

Comment 8 Sandro Bonazzola 2018-06-28 06:41:52 UTC
This bugzilla is included in oVirt 4.2.4 release, published on June 26th 2018.

Since the problem described in this bug report should be
resolved in oVirt 4.2.4 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.