Bug 1492236 - Dynamically provisioned GlusterFS PVC sometimes not generating Endpoints
Summary: Dynamically provisioned GlusterFS PVC sometimes not generating Endpoints
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: CNS-deployment
Version: cns-3.5
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: CNS 3.10
Assignee: Humble Chirammal
QA Contact: Ashmitha Ambastha
Depends On:
Blocks: 1724792 1568861
TreeView+ depends on / blocked
Reported: 2017-09-15 21:01 UTC by Thom Carlin
Modified: 2019-06-28 16:04 UTC (History)
13 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2018-09-17 04:53:08 UTC
Target Upstream Version:

Attachments (Terms of Use)

Description Thom Carlin 2017-09-15 21:01:53 UTC
Description of problem:

When deploying a pod using CNS to dynamically fill PVC's, the pods sometimes hang at ContainerCreating, then timeout and go to error

Version-Release number of selected component (if applicable):


How reproducible:

Believe 100% for specific situation

Steps to Reproduce:
1. Deploy pod with PVC filled by dynamic GlusterFS (CNS) PV

Actual results:

Pod hangs at Container Creating, then go to Error

Expected results:

Successful creation

Additional info:

Believe there is a consistent reproducer using Coolstore MSA (https://github.com/jbossdemocentral/coolstore-microservice) with the provision-demo.sh and inventory-postgresql

Comment 3 Thom Carlin 2017-09-25 20:00:50 UTC
# oc get pvc inventory-postgresql-pv
NAME                      STATUS    VOLUME                                     CAPACITY   ACCESSMODES   STORAGECLASS        AGE
inventory-postgresql-pv   Bound     pvc-4b651208-9fcd-11e7-bcc6-001a4a160152   1Gi        RWO           gluster-container   3d

# oc get pv pvc-4b651208-9fcd-11e7-bcc6-001a4a160152
NAME                                       CAPACITY   ACCESSMODES   RECLAIMPOLICY   STATUS    CLAIM                                           STORAGECLASS        REASON    AGE
pvc-4b651208-9fcd-11e7-bcc6-001a4a160152   1Gi        RWO           Delete          Bound     coolstore-prod-ocuser/inventory-postgresql-pv   gluster-container             3d

# oc describe pv pvc-4b651208-9fcd-11e7-bcc6-001a4a160152
Name:		pvc-4b651208-9fcd-11e7-bcc6-001a4a160152
Labels:		<none>
Annotations:	pv.beta.kubernetes.io/gid=2006
StorageClass:	gluster-container
Status:		Bound
Claim:		coolstore-prod-ocuser/inventory-postgresql-pv
Reclaim Policy:	Delete
Access Modes:	RWO
Capacity:	1Gi
    Type:		Glusterfs (a Glusterfs mount on the host that shares a pod's lifetime)
    EndpointsName:	glusterfs-dynamic-inventory-postgresql-pv
    Path:		vol_ec2c28819fd3e3d5050b9727da2dc494
    ReadOnly:		false
Events:			<none>

# oc get ep glusterfs-dynamic-inventory-postgresql-pv
Error from server (NotFound): endpoints "glusterfs-dynamic-inventory-postgresql-pv" not found

Comment 9 Jose A. Rivera 2018-05-30 12:15:46 UTC
I don't think this has been seen since reporting, and it's not critical for the release. Moving this to 3.11.

However, Thom, are you still able to reproduce this?

Comment 10 Thom Carlin 2018-05-30 12:17:44 UTC
I'm seeing this in 3.6-z but I don't have a consistent reproducer.

Comment 11 Thom Carlin 2018-05-30 22:33:28 UTC
One cause seems to be if a delete of a PVC is followed immediately by a create of the same PVC.
The glusterfs-dynamic-<<pvc_name>> either doesn't exist or has endpoints = <none>

Comment 13 Thom Carlin 2018-07-16 18:42:56 UTC
Since my cluster is currently offline with another issue, please reach out to CNS QE.

Note You need to log in before you can comment on or make changes to this bug.