Bug 1798850

Summary: [OCS] Deployment of OCS independent mode is failing because of the unknown parameter name HEKETI_LVM_WRAPPER
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Aditya Ramteke <aramteke>
Component: heketiAssignee: John Mulligan <jmulligan>
Status: CLOSED ERRATA QA Contact: Aditya Ramteke <aramteke>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: ocs-3.11CC: aramteke, dpivonka, hchiramm, jialiu, madam, ndevos, pprakash, puebele, rhs-bugs, rtalur, storage-qa-internal, vlaad
Target Milestone: ---Keywords: ZStream
Target Release: OCS 3.11.z Batch Update 5   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: 1798365 Environment:
Last Closed: 2020-02-26 20:25:33 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1798365    
Bug Blocks:    

Description Aditya Ramteke 2020-02-06 06:54:31 UTC 
+++ This bug was initially created as a clone of Bug #1798365 +++

Description of problem:

Deployment of OCS independent mode is failing because of the unknown parameter name HEKETI_LVMWRAPPER while deploy a deploy-heketi pod.

Niels my assumption is that you might have missed the glusterfs_is_native parameter check here [1].


[1] https://github.com/openshift/openshift-ansible/blob/release-3.11/roles/openshift_storage_glusterfs/tasks/heketi_init_deploy.yml#L29


Version-Release number of the following components:
rpm -q openshift-ansible
openshift-ansible-3.11.169-1.git.0.e22497a.el7.noarch
rpm -q ansible
ansible-2.6.20-1.el7ae.noarch
ansible --version
ansible 2.6.20
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.5 (default, Jun 11 2019, 14:33:56) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]


How reproducible:
100%

Steps to Reproduce:
1. Deploy an independent mode setup with openshift-ansible-3.11.169-1.git.0.e22497a.el7.noarch

Actual results:
Please include the entire output from the last TASK line through the end of output if an error is generated

TASK [openshift_storage_glusterfs : Deploy deploy-heketi pod] *********************************************************************************************************************************************************************************
task path: /usr/share/ansible/openshift-ansible/roles/openshift_storage_glusterfs/tasks/heketi_init_deploy.yml:16
<dhcp47-36.lab.eng.blr.redhat.com> ESTABLISH SSH CONNECTION FOR USER: root
<dhcp47-36.lab.eng.blr.redhat.com> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/939def0b6e dhcp47-36.lab.eng.blr.redhat.com '/bin/sh -c '"'"'echo ~root && sleep 0'"'"''
<dhcp47-36.lab.eng.blr.redhat.com> (0, '/root\n', '')
<dhcp47-36.lab.eng.blr.redhat.com> ESTABLISH SSH CONNECTION FOR USER: root
<dhcp47-36.lab.eng.blr.redhat.com> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/939def0b6e dhcp47-36.lab.eng.blr.redhat.com '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /root/.ansible/tmp/ansible-tmp-1580885795.0-64585931736537 `" && echo ansible-tmp-1580885795.0-64585931736537="` echo /root/.ansible/tmp/ansible-tmp-1580885795.0-64585931736537 `" ) && sleep 0'"'"''
<dhcp47-36.lab.eng.blr.redhat.com> (0, 'ansible-tmp-1580885795.0-64585931736537=/root/.ansible/tmp/ansible-tmp-1580885795.0-64585931736537\n', '')
Using module file /usr/share/ansible/openshift-ansible/roles/lib_openshift/library/oc_process.py
<dhcp47-36.lab.eng.blr.redhat.com> PUT /root/.ansible/tmp/ansible-local-93976SZqliG/tmpMREbdp TO /root/.ansible/tmp/ansible-tmp-1580885795.0-64585931736537/oc_process.py
<dhcp47-36.lab.eng.blr.redhat.com> SSH: EXEC sftp -b - -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/939def0b6e '[dhcp47-36.lab.eng.blr.redhat.com]'
<dhcp47-36.lab.eng.blr.redhat.com> (0, 'sftp> put /root/.ansible/tmp/ansible-local-93976SZqliG/tmpMREbdp /root/.ansible/tmp/ansible-tmp-1580885795.0-64585931736537/oc_process.py\n', '')
<dhcp47-36.lab.eng.blr.redhat.com> ESTABLISH SSH CONNECTION FOR USER: root
<dhcp47-36.lab.eng.blr.redhat.com> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/939def0b6e dhcp47-36.lab.eng.blr.redhat.com '/bin/sh -c '"'"'chmod u+x /root/.ansible/tmp/ansible-tmp-1580885795.0-64585931736537/ /root/.ansible/tmp/ansible-tmp-1580885795.0-64585931736537/oc_process.py && sleep 0'"'"''
<dhcp47-36.lab.eng.blr.redhat.com> (0, '', '')
<dhcp47-36.lab.eng.blr.redhat.com> ESTABLISH SSH CONNECTION FOR USER: root
<dhcp47-36.lab.eng.blr.redhat.com> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/939def0b6e -tt dhcp47-36.lab.eng.blr.redhat.com '/bin/sh -c '"'"'/usr/bin/python /root/.ansible/tmp/ansible-tmp-1580885795.0-64585931736537/oc_process.py && sleep 0'"'"''
<dhcp47-36.lab.eng.blr.redhat.com> (1, 'Traceback (most recent call last):\r\n  File "/tmp/ansible_XjpOR9/ansible_module_oc_process.py", line 1693, in <module>\r\n    main()\r\n  File "/tmp/ansible_XjpOR9/ansible_module_oc_process.py", line 1686, in main\r\n    rval = OCProcess.run_ansible(module.params, module.check_mode)\r\n  File "/tmp/ansible_XjpOR9/ansible_module_oc_process.py", line 1643, in run_ansible\r\n    all_results = ocprocess.needs_update()\r\n  File "/tmp/ansible_XjpOR9/ansible_module_oc_process.py", line 1562, in needs_update\r\n    for obj in self.template:\r\n  File "/tmp/ansible_XjpOR9/ansible_module_oc_process.py", line 1508, in template\r\n    raise OpenShiftCLIError(\'Error processing template [%s]: %s\' %(self.name, results))\r\n__main__.OpenShiftCLIError: Error processing template [deploy-heketi]: {\'cmd\': \'/usr/bin/oc process deploy-heketi -p HEKETI_USER_KEY=heketiuserkey HEKETI_ADMIN_KEY=adminkey IMAGE_NAME=registry-proxy.engineering.redhat.com/rh-osbs/rhgs3-rhgs-volmanager-rhel7:3.11.5-2 HEKETI_LVMWRAPPER=/usr/sbin/exec-on-host CLUSTER_NAME=storage HEKETI_FSTAB=/etc/fstab HEKETI_ROUTE=heketi-storage HEKETI_EXECUTOR=ssh -n glusterfs\', \'returncode\': 1, \'results\': {}, \'stderr\': u\'error: unknown parameter name "HEKETI_LVMWRAPPER"\\n\', \'stdout\': u\'\'}\r\n', 'Shared connection to dhcp47-36.lab.eng.blr.redhat.com closed.\r\n')
<dhcp47-36.lab.eng.blr.redhat.com> Failed to connect to the host via ssh: Shared connection to dhcp47-36.lab.eng.blr.redhat.com closed.
<dhcp47-36.lab.eng.blr.redhat.com> ESTABLISH SSH CONNECTION FOR USER: root
<dhcp47-36.lab.eng.blr.redhat.com> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/939def0b6e dhcp47-36.lab.eng.blr.redhat.com '/bin/sh -c '"'"'rm -f -r /root/.ansible/tmp/ansible-tmp-1580885795.0-64585931736537/ > /dev/null 2>&1 && sleep 0'"'"''
<dhcp47-36.lab.eng.blr.redhat.com> (0, '', '')
fatal: [dhcp47-36.lab.eng.blr.redhat.com]: FAILED! => {
    "changed": false, 
    "module_stderr": "Shared connection to dhcp47-36.lab.eng.blr.redhat.com closed.\r\n", 
    "module_stdout": "Traceback (most recent call last):\r\n  File \"/tmp/ansible_XjpOR9/ansible_module_oc_process.py\", line 1693, in <module>\r\n    main()\r\n  File \"/tmp/ansible_XjpOR9/ansible_module_oc_process.py\", line 1686, in main\r\n    rval = OCProcess.run_ansible(module.params, module.check_mode)\r\n  File \"/tmp/ansible_XjpOR9/ansible_module_oc_process.py\", line 1643, in run_ansible\r\n    all_results = ocprocess.needs_update()\r\n  File \"/tmp/ansible_XjpOR9/ansible_module_oc_process.py\", line 1562, in needs_update\r\n    for obj in self.template:\r\n  File \"/tmp/ansible_XjpOR9/ansible_module_oc_process.py\", line 1508, in template\r\n    raise OpenShiftCLIError('Error processing template [%s]: %s' %(self.name, results))\r\n__main__.OpenShiftCLIError: Error processing template [deploy-heketi]: {'cmd': '/usr/bin/oc process deploy-heketi -p HEKETI_USER_KEY=heketiuserkey HEKETI_ADMIN_KEY=adminkey IMAGE_NAME=registry-proxy.engineering.redhat.com/rh-osbs/rhgs3-rhgs-volmanager-rhel7:3.11.5-2 HEKETI_LVMWRAPPER=/usr/sbin/exec-on-host CLUSTER_NAME=storage HEKETI_FSTAB=/etc/fstab HEKETI_ROUTE=heketi-storage HEKETI_EXECUTOR=ssh -n glusterfs', 'returncode': 1, 'results': {}, 'stderr': u'error: unknown parameter name \"HEKETI_LVMWRAPPER\"\\n', 'stdout': u''}\r\n", 
    "msg": "MODULE FAILURE", 
    "rc": 1
}
	to retry, use: --limit @/usr/share/ansible/openshift-ansible/playbooks/openshift-glusterfs/config.retry

PLAY RECAP ************************************************************************************************************************************************************************************************************************************
dhcp46-250.lab.eng.blr.redhat.com : ok=26   changed=2    unreachable=0    failed=0   
dhcp47-152.lab.eng.blr.redhat.com : ok=3    changed=0    unreachable=0    failed=0   
dhcp47-36.lab.eng.blr.redhat.com : ok=80   changed=6    unreachable=0    failed=1   
dhcp47-38.lab.eng.blr.redhat.com : ok=2    changed=0    unreachable=0    failed=0   
dhcp47-4.lab.eng.blr.redhat.com : ok=2    changed=0    unreachable=0    failed=0   
localhost                  : ok=12   changed=0    unreachable=0    failed=0   


INSTALLER STATUS ******************************************************************************************************************************************************************************************************************************
Initialization     : Complete (0:00:46)
GlusterFS Install  : In Progress (0:01:55)
	This phase can be restarted by running: playbooks/openshift-glusterfs/config.yml
[root@dhcp47-36 scripts]#

Expected results:
The deployment should be successful.

--- Additional comment from Aditya Ramteke on 2020-02-05 09:29:10 UTC ---

(In reply to Aditya Ramteke from comment #0)
> Description of problem:
> 
> Deployment of OCS independent mode is failing because of the unknown
> parameter name HEKETI_LVMWRAPPER while deploy a deploy-heketi pod.
> 
> Niels my assumption is that you might have missed the glusterfs_is_native
> parameter check here [1].
> 

If I remove HEKETI_LVMWRAPPER: "{{ glusterfs_heketi_lvmwrapper }}" from heketi_init_deploy.yml then I can able to deploy the independent mode setup.

Niels and Talur, what is your suggestion on this?

--- Additional comment from Aditya Ramteke on 2020-02-05 09:37:28 UTC ---

This can be a blocker as we can't able to deploy the independent mode setup with openshift-ansible-3.11.169-1.git.0.e22497a.el7.noarch
Comment 14 errata-xmlrpc 2020-02-26 20:25:33 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0622