Description of problem:
When deploying a pod using CNS to dynamically fill PVC's, the pods sometimes hang at ContainerCreating, then timeout and go to error
Version-Release number of selected component (if applicable):
Believe 100% for specific situation
Steps to Reproduce:
1. Deploy pod with PVC filled by dynamic GlusterFS (CNS) PV
Pod hangs at Container Creating, then go to Error
Believe there is a consistent reproducer using Coolstore MSA (https://github.com/jbossdemocentral/coolstore-microservice) with the provision-demo.sh and inventory-postgresql
# oc get pvc inventory-postgresql-pv
NAME STATUS VOLUME CAPACITY ACCESSMODES STORAGECLASS AGE
inventory-postgresql-pv Bound pvc-4b651208-9fcd-11e7-bcc6-001a4a160152 1Gi RWO gluster-container 3d
# oc get pv pvc-4b651208-9fcd-11e7-bcc6-001a4a160152
NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-4b651208-9fcd-11e7-bcc6-001a4a160152 1Gi RWO Delete Bound coolstore-prod-ocuser/inventory-postgresql-pv gluster-container 3d
# oc describe pv pvc-4b651208-9fcd-11e7-bcc6-001a4a160152
Reclaim Policy: Delete
Access Modes: RWO
Type: Glusterfs (a Glusterfs mount on the host that shares a pod's lifetime)
# oc get ep glusterfs-dynamic-inventory-postgresql-pv
Error from server (NotFound): endpoints "glusterfs-dynamic-inventory-postgresql-pv" not found
I don't think this has been seen since reporting, and it's not critical for the release. Moving this to 3.11.
However, Thom, are you still able to reproduce this?
I'm seeing this in 3.6-z but I don't have a consistent reproducer.
One cause seems to be if a delete of a PVC is followed immediately by a create of the same PVC.
The glusterfs-dynamic-<<pvc_name>> either doesn't exist or has endpoints = <none>
Since my cluster is currently offline with another issue, please reach out to CNS QE.