Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1777499

Summary: Installation is failing at TASK [openshift_storage_glusterfs : Check for GlusterFS cluster health]
Product: OpenShift Container Platform Reporter: Aditya Deshpande <adeshpan>
Component: InstallerAssignee: Russell Teague <rteague>
Installer sub component: openshift-ansible QA Contact: Johnny Liu <jialiu>
Status: CLOSED DUPLICATE Docs Contact:
Severity: high    
Priority: high CC: asolanas, dmoessne, ricardo.arguello, rteague, sascha.tanke
Version: 3.11.0   
Target Milestone: ---   
Target Release: 3.11.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-01-14 21:07:46 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Aditya Deshpande 2019-11-27 16:20:53 UTC
Description of problem:
New OCP installation with deploy_cluster.yml playbook is failing when glusterfs inventory variables are set.

~~~
TASK [openshift_storage_glusterfs : Check for GlusterFS cluster health] **********************************************************************************************************************
task path: /usr/share/ansible/openshift-ansible/roles/openshift_storage_glusterfs/tasks/cluster_health.yml:8
FAILED - RETRYING: Check for GlusterFS cluster health (3 retries left).
FAILED - RETRYING: Check for GlusterFS cluster health (2 retries left).
FAILED - RETRYING: Check for GlusterFS cluster health (1 retries left).
fatal: [X.X.X.X -> X.X.X.X]: FAILED! => {"attempts": 3, "changed": false, "msg": "Unable to find pods: ", "state": "unknown"}
~~~

The playbook failed at Node Join playbook.
~~~
INSTALLER STATUS *****************************************************************************************************************************************************************************
Initialization              : Complete (0:01:24)
Health Check                : Complete (0:00:29)
Node Bootstrap Preparation  : Complete (0:07:55)
etcd Install                : Complete (0:00:59)
Load Balancer Install       : Complete (0:00:20)
Master Install              : Complete (0:07:38)
Master Additional Install   : Complete (0:02:00)
Node Join                   : In Progress (0:02:35)
        This phase can be restarted by running: playbooks/openshift-node/join.yml


Failure summary:


  1. Hosts:    X.X.X.X
     Play:     Restart nodes
     Task:     Check for GlusterFS cluster health
     Message:  Unable to find pods:                       
~~~

So, when I checked for the glusterfs project, it was not created and not shown from `oc get project` output.

After checking the step by step playbook installation method, I see glusterfs install playbook will run after successful completion of Node Join.
But in this scenario Node Join playbook is failing.

Version-Release number of the following components:
rpm -q openshift-ansible: openshift-ansible-3.11.154-2.git.0.1640c49.el7.noarch
rpm -q ansible: ansible-2.6.20-1.el7ae.noarch

Inventory variables related to glusterfs:
openshift_storage_glusterfs_storageclass_default=True
openshift_storage_glusterfs_image=registry.access.redhat.com/rhgs3/rhgs-server-rhel7:v3.11
openshift_storage_glusterfs_block_image=registry.access.redhat.com/rhgs3/rhgs-gluster-block-prov-rhel7:v3.11
openshift_storage_glusterfs_heketi_image=registry.access.redhat.com/rhgs3/rhgs-volmanager-rhel7:v3.11

[glusterfs]
node-name glusterfs_devices='["/disk"]'


Actual results:
Playbook is failing at TASK: Check for GlusterFS cluster health

Expected results:
The playbook should not fail.
Need to know the reason of role for openshift_storage_glusterfs to be included in Node Join playbook: restart nodes.
If that role is at proper place then the project glusterfs should be created before the TASK.

Additional info:
https://github.com/openshift/openshift-ansible/issues/12002

Comment 4 Sascha Tanke 2020-01-08 08:51:06 UTC
Give it a new try today and it is working now.

Package versions:
openshift-ansible-3.11.157-1.git.0.10b76ed.el7.noarch
openshift-ansible-playbooks-3.11.157-1.git.0.10b76ed.el7.noarch
openshift-ansible-roles-3.11.157-1.git.0.10b76ed.el7.noarch

Comment 5 Scott Dodson 2020-01-14 21:07:46 UTC
Behavior described matches that of bug 1774725 and the linked github issue as well. Believed to be fixed in 3.11.157 or later. Closing as a dupe.

*** This bug has been marked as a duplicate of bug 1774725 ***