Bug 1638119

Summary: PV stuck in pending state due to readiness probe and liveliness probe failures.
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: vinutha <vinug>
Component: kubernetesAssignee: Humble Chirammal <hchiramm>
Status: CLOSED CURRENTRELEASE QA Contact: Prasanth <pprakash>
Severity: high Docs Contact:
Priority: unspecified    
Version: rhgs-3.0CC: aos-bugs, aos-storage-staff, jokerman, mmccomas, rcyriac, rhs-bugs, sankarshan, vinug
Target Milestone: ---Keywords: ZStream
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-01-23 14:47:25 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1633499    

Description vinutha 2018-10-10 18:33:44 UTC
Description of problem:
pvc stuck in Pending state and not getting bound due to errors: 
1. 'Readiness probe failed: rpc error: code = 14 desc = grpc: the connection is 
    unavailable'
2. 'Liveness probe failed: rpc error: code = 14 desc = grpc: the connection is 
    unavailable'


Version-Release number of selected component (if applicable):
# rpm -qa| grep docker
docker-common-1.13.1-75.git8633870.el7_5.x86_64
python-docker-2.4.2-1.3.el7.noarch
docker-client-1.13.1-75.git8633870.el7_5.x86_64
docker-1.13.1-75.git8633870.el7_5.x86_64
cockpit-docker-176-2.el7.x86_64
docker-rhel-push-plugin-1.13.1-75.git8633870.el7_5.x86_64
python-docker-pycreds-1.10.6-4.el7.noarch
atomic-openshift-docker-excluder-3.11.21-1.git.0.3e1eda6.el7.noarch

# docker version
Client:
 Version:         1.13.1
 API version:     1.26
 Package version: docker-1.13.1-75.git8633870.el7_5.x86_64
 Go version:      go1.9.2
 Git commit:      8633870/1.13.1
 Built:           Wed Sep 12 10:56:54 2018
 OS/Arch:         linux/amd64

Server:
 Version:         1.13.1
 API version:     1.26 (minimum version 1.12)
 Package version: docker-1.13.1-75.git8633870.el7_5.x86_64
 Go version:      go1.9.2
 Git commit:      8633870/1.13.1
 Built:           Wed Sep 12 10:56:54 2018
 OS/Arch:         linux/amd64
 Experimental:    false


# rpm -qa| grep openshift
openshift-ansible-3.11.21-1.git.0.7dc17ca.el7.noarch
atomic-openshift-excluder-3.11.21-1.git.0.3e1eda6.el7.noarch
openshift-ansible-docs-3.11.21-1.git.0.7dc17ca.el7.noarch
openshift-ansible-playbooks-3.11.21-1.git.0.7dc17ca.el7.noarch
atomic-openshift-node-3.11.21-1.git.0.3e1eda6.el7.x86_64
atomic-openshift-clients-3.11.21-1.git.0.3e1eda6.el7.x86_64
atomic-openshift-3.11.21-1.git.0.3e1eda6.el7.x86_64
openshift-ansible-roles-3.11.21-1.git.0.7dc17ca.el7.noarch
atomic-openshift-docker-excluder-3.11.21-1.git.0.3e1eda6.el7.noarch
atomic-openshift-hyperkube-3.11.21-1.git.0.3e1eda6.el7.x86_64


How reproducible:
Always on this setup

Steps to Reproduce:
1. On a 4 node setup of OCP 3.11.21 + CNS 3.11 latest created 1GB pvc which is stuck in pending state for ~30min 


Actual results:
PVC not getting Bound 

Expected results:
Pvc should be successfully bound

Additional info:
will update heketi logs and some command outputs