Bug 1624818

Summary: [Tracker-ocp#1624819] CNS 3.10 standalone install fails on already deployed OCP cluster-v3.10.34
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Neha Berry <nberry>
Component: cns-ansibleAssignee: Jose A. Rivera <jarrpa>
Status: CLOSED ERRATA QA Contact: Prasanth <pprakash>
Severity: high Docs Contact:
Priority: unspecified    
Version: cns-3.10CC: akhakhar, akrishna, hchiramm, jarrpa, knarra, kramdoss, madam, nberry, pprakash, rcyriac, rgeorge, rhs-bugs, rtalur, sankarshan, sarumuga
Target Milestone: ---Keywords: Regression
Target Release: OCS 3.11   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openshift-ansible-3.10.46-1 Doc Type: Bug Fix
Doc Text:
Cause: Consequence: Workaround (if any): Result:
Story Points: ---
Clone Of:
: 1624819 (view as bug list) Environment:
Last Closed: 2018-10-24 04:50:56 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1624819    
Bug Blocks: 1568868, 1629575    

Description Neha Berry 2018-09-03 11:27:52 UTC
Description of problem:
==============================

While installing CNS 3.10 on an already installed OCP 3.10.34 setup, the CNS install is failing on following two tasks.


TASK [openshift_node : Install GlusterFS storage plugin dependencies]
msg": "The conditional check 'not openshift_is_atomic | bool' failed. The error was: error while evaluating conditional (not openshift_is_atomic | bool):

TASK [openshift_storage_glusterfs : Ensure device mapper modules loaded]
"msg": "The conditional check 'inventory_hostname not in groups['glusterfs_registry']' failed


Version-Release number of the following components:
================================================

The GA'd version of OCP was used.


# rpm -qa|grep ansible
openshift-ansible-docs-3.10.34-1.git.0.48df172None.noarch
openshift-ansible-playbooks-3.10.34-1.git.0.48df172None.noarch
openshift-ansible-3.10.34-1.git.0.48df172None.noarch
ansible-2.4.6.0-1.el7ae.noarch
openshift-ansible-roles-3.10.34-1.git.0.48df172None.noarch


oc version
--------------
# oc version
oc v3.10.34
kubernetes v1.10.0+b81c8f8


# rpm -qa|grep openshift
openshift-ansible-docs-3.10.34-1.git.0.48df172None.noarch
atomic-openshift-docker-excluder-3.10.34-1.git.0.4a94911.el7.noarch
atomic-openshift-clients-3.10.34-1.git.0.4a94911.el7.x86_64
openshift-ansible-playbooks-3.10.34-1.git.0.48df172None.noarch
openshift-ansible-3.10.34-1.git.0.48df172None.noarch
atomic-openshift-excluder-3.10.34-1.git.0.4a94911.el7.noarch
atomic-openshift-3.10.34-1.git.0.4a94911.el7.x86_64
atomic-openshift-hyperkube-3.10.34-1.git.0.4a94911.el7.x86_64
openshift-ansible-roles-3.10.34-1.git.0.48df172None.noarch
atomic-openshift-node-3.10.34-1.git.0.4a94911.el7.x86_64




How reproducible:
===========================
100% on 2-3 different setups where OCP3/10.34 was already pre-installed

Steps to Reproduce:
========================
1. Deploy OCP with there extra node(for CNS)
 ansible-playbook -vvv /usr/share/ansible/openshift-ansible/playbooks/deploy_cluster.yml 

2. Defined glusterfs group in inventory file, execute CNS standalone deployment playbook.
 ansible-playbook -vvv /usr/share/ansible/openshift-ansible/playbooks/openshift-glusterfs/config.yml

3. Check the status of installation

Actual results:
===================

1. The failed tasks - TASK#1
==============================

2018-09-02 17:47:12,103 p=27009 u=root |  TASK [openshift_node : Install GlusterFS storage plugin dependencies] ***********************************
2018-09-02 17:47:12,103 p=27009 u=root |  task path: /usr/share/ansible/openshift-ansible/roles/openshift_node/tasks/glusterfs.yml:2
2018-09-02 17:47:12,156 p=27009 u=root |  Using module file /usr/lib/python2.7/site-packages/ansible/modules/packaging/os/yum.py
2018-09-02 17:47:12,173 p=27009 u=root |  Using module file /usr/lib/python2.7/site-packages/ansible/modules/packaging/os/yum.py
2018-09-02 17:47:12,191 p=27009 u=root |  Using module file /usr/lib/python2.7/site-packages/ansible/modules/packaging/os/yum.py
2018-09-02 17:47:12,214 p=27009 u=root |  Using module file /usr/lib/python2.7/site-packages/ansible/modules/packaging/os/yum.py
2018-09-02 17:47:12,273 p=27009 u=root |  fatal: [dhcp46-150.lab.eng.blr.redhat.com]: FAILED! => {
    "failed": true,
    "msg": "The conditional check 'not openshift_is_atomic | bool' failed. The error was: error while evaluating conditional (not openshift_is_atomic | bool): 'openshift_is_atomic' is undefined\n\nThe error appears to have been in '/usr/share/ansible/openshift-ansible/roles/openshift_node/tasks/glusterfs.yml': line 2, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n---\n- name: Install GlusterFS storage plugin dependencies\n  ^ here\n"
}
2018-09-02 17:47:12,294 p=27009 u=root |  fatal: [dhcp46-219.lab.eng.blr.redhat.com]: FAILED! => {
    "failed": true,
    "msg": "The conditional check 'not openshift_is_atomic | bool' failed. The error was: error while evaluating conditional (not openshift_is_atomic | bool): 'openshift_is_atomic' is undefined\n\nThe error appears to have been in '/usr/share/ansible/openshift-ansible/roles/openshift_node/tasks/glusterfs.yml': line 2, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n---\n- name: Install GlusterFS storage plugin dependencies\n  ^ here\n"
}
2018-09-02 17:47:12,318 p=27009 u=root |  fatal: [dhcp46-231.lab.eng.blr.redhat.com]: FAILED! => {
    "failed": true,
    "msg": "The conditional check 'not openshift_is_atomic | bool' failed. The error was: error while evaluating conditional (not openshift_is_atomic | bool): 'openshift_is_atomic' is undefined\n\nThe error appears to have been in '/usr/share/ansible/openshift-ansible/roles/openshift_node/tasks/glusterfs.yml': line 2, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n---\n- name: Install GlusterFS storage plugin dependencies\n  ^ here\n"
}
2018-09-02 17:47:12,344 p=27009 u=root |  fatal: [dhcp46-181.lab.eng.blr.redhat.com]: FAILED! => {


2. The Failed task - TASK#2
==============================

2018-09-02 17:47:34,371 p=27009 u=root |  TASK [openshift_storage_glusterfs : Ensure device mapper modules loaded] ********************************
2018-09-02 17:47:34,371 p=27009 u=root |  task path: /usr/share/ansible/openshift-ansible/roles/openshift_storage_glusterfs/tasks/kernel_modules.yml:2
2018-09-02 17:47:34,418 p=27009 u=root |  fatal: [dhcp46-137.lab.eng.blr.redhat.com]: FAILED! => {
    "failed": true,
    "msg": "The conditional check 'inventory_hostname not in groups['glusterfs_registry']' failed. The error was: error while evaluating conditional (inventory_hostname not in groups['glusterfs_registry']): Unable to look up a name or access an attribute in template string ({% if inventory_hostname not in groups['glusterfs_registry'] %} True {% else %} False {% endif %}).\nMake sure your variable name does not contain invalid characters like '-': argument of type 'StrictUndefined' is not iterable\n\nThe error appears to have been in '/usr/share/ansible/openshift-ansible/roles/openshift_storage_glusterfs/tasks/kernel_modules.yml': line 2, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n---\n- name: Ensure device mapper modules loaded\n  ^ here\n"
}
2018-09-02 17:47:34,438 p=27009 u=root |  skipping: [dhcp46-55.lab.eng.blr.redhat.com] => {
    "changed": false,
    "skip_reason": "Conditional result was False",
    "skipped": true
}
2018-09-02 17:47:34,439 p=27009 u=root |  skipping: [dhcp46-95.lab.eng.blr.redhat.com] => {
    "changed": false,
    "skip_reason": "Conditional result was False",
    "skipped": true
}
2018-09-02 17:47:34,454 p=27009 u=root |  skipping: [dhcp47-106.lab.eng.blr.redhat.com] => {
    "changed": false,
    "skip_reason": "Conditional result was False",
    "skipped": true
}

Comment 9 Humble Chirammal 2018-09-04 13:09:14 UTC
The subjected OCP bug has been proposed for OCP 3.11. 
https://bugzilla.redhat.com/show_bug.cgi?id=1624819

Comment 10 Michael Adam 2018-09-04 14:12:53 UTC
fix bug dependency order

Comment 11 Jose A. Rivera 2018-09-04 21:17:30 UTC
PR is pending: https://github.com/openshift/openshift-ansible/pull/9847

Comment 12 Humble Chirammal 2018-09-05 07:52:38 UTC
Saravana/Jose/Karthick, Can you please clarify the impact or the functionalities broken with this bug or in absense of the fix?   This information is vital for taking necessary actions for OCS 3.10 release.

Comment 17 Jose A. Rivera 2018-09-06 17:16:54 UTC
PR is merged. Backport PR: https://github.com/openshift/openshift-ansible/pull/9934

Comment 21 Jose A. Rivera 2018-09-07 19:13:07 UTC
That's the hope. Backport PR merged.

Comment 29 Anjana KD 2018-10-10 06:42:15 UTC
kindly provide the complete doc text.

Comment 31 errata-xmlrpc 2018-10-24 04:50:56 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2985