Description of problem:

Through automation or other methods, if a PVC is deleted and recreated within a short period, the original PV and PVC will get deleted, with the new PVC and PV being provisioned, but the service and endpoints end up getting deleted and not recreated because they already exist at that point in time.

There needs to be some kind of synchronization or remediation.

I1025 00:03:03.398954       1 glusterfs.go:703] Allocated GID 2001 for PVC test1
I1025 00:03:08.066124       1 glusterfs.go:703] Allocated GID 2002 for PVC test1
I1025 00:03:13.492857       1 glusterfs.go:776] volume with size 1 and name file_default_test1_57ba78db-d7e9-11e8-9252-fa163ea8bc36 created
I1025 00:03:13.540789       1 glusterfs.go:832] endpoint &Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:glusterfs-dynamic-test1,GenerateName:,Namespace:default,SelfLink:,UID:,ResourceVersion:,Generation:0,CreationTimestamp:0001-01-01 00:00:00 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{gluster.kubernetes.io/provisioned-for-pvc: test1,},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Subsets:[{[{10.10.85.244  <nil> nil} {10.10.85.243  <nil> nil} {10.10.85.221  <nil> nil}] [] [{ 1 TCP}]}],} already exist in namespace default
I1025 00:03:13.688109       1 glusterfs.go:852] service &Service{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:glusterfs-dynamic-test1,GenerateName:,Namespace:default,SelfLink:,UID:,ResourceVersion:,Generation:0,CreationTimestamp:0001-01-01 00:00:00 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{gluster.kubernetes.io/provisioned-for-pvc: test1,},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Spec:ServiceSpec{Ports:[{ TCP 1 {0 0 } 0}],Selector:map[string]string{},ClusterIP:,Type:,ExternalIPs:[],SessionAffinity:,LoadBalancerIP:,LoadBalancerSourceRanges:[],ExternalName:,ExternalTrafficPolicy:,HealthCheckNodePort:0,PublishNotReadyAddresses:false,SessionAffinityConfig:nil,},Status:ServiceStatus{LoadBalancer:LoadBalancerStatus{Ingress:[],},},} already exist in namespace default
I1025 00:03:13.732398       1 pv_controller.go:1516] volume "pvc-536c9fa2-d7e9-11e8-a294-fa163e7f64be" provisioned for claim "default/test1"
I1025 00:03:13.733243       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"test1", UID:"536c9fa2-d7e9-11e8-a294-fa163e7f64be", APIVersion:"v1", ResourceVersion:"16495097", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-536c9fa2-d7e9-11e8-a294-fa163e7f64be using kubernetes.io/glusterfs
I1025 00:03:13.764913       1 glusterfs.go:597] delete volume file_default_test1_57ba78db-d7e9-11e8-9252-fa163ea8bc36
I1025 00:03:26.182323       1 glusterfs.go:776] volume with size 1 and name file_default_test1_5a82b19f-d7e9-11e8-9252-fa163ea8bc36 created
I1025 00:03:26.224816       1 glusterfs.go:832] endpoint &Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:glusterfs-dynamic-test1,GenerateName:,Namespace:default,SelfLink:,UID:,ResourceVersion:,Generation:0,CreationTimestamp:0001-01-01 00:00:00 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{gluster.kubernetes.io/provisioned-for-pvc: test1,},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Subsets:[{[{10.10.85.244  <nil> nil} {10.10.85.243  <nil> nil} {10.10.85.221  <nil> nil}] [] [{ 1 TCP}]}],} already exist in namespace default
I1025 00:03:26.302301       1 glusterfs.go:852] service &Service{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:glusterfs-dynamic-test1,GenerateName:,Namespace:default,SelfLink:,UID:,ResourceVersion:,Generation:0,CreationTimestamp:0001-01-01 00:00:00 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{gluster.kubernetes.io/provisioned-for-pvc: test1,},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Spec:ServiceSpec{Ports:[{ TCP 1 {0 0 } 0}],Selector:map[string]string{},ClusterIP:,Type:,ExternalIPs:[],SessionAffinity:,LoadBalancerIP:,LoadBalancerSourceRanges:[],ExternalName:,ExternalTrafficPolicy:,HealthCheckNodePort:0,PublishNotReadyAddresses:false,SessionAffinityConfig:nil,},Status:ServiceStatus{LoadBalancer:LoadBalancerStatus{Ingress:[],},},} already exist in namespace default
I1025 00:03:26.321893       1 pv_controller.go:1516] volume "pvc-5643c546-d7e9-11e8-a294-fa163e7f64be" provisioned for claim "default/test1"
I1025 00:03:26.322092       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"test1", UID:"5643c546-d7e9-11e8-a294-fa163e7f64be", APIVersion:"v1", ResourceVersion:"16495112", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-5643c546-d7e9-11e8-a294-fa163e7f64be using kubernetes.io/glusterfs
I1025 00:03:26.343777       1 pv_controller.go:910] volume "pvc-5643c546-d7e9-11e8-a294-fa163e7f64be" bound to claim "default/test1"
I1025 00:03:26.425613       1 pv_controller.go:715] claim "default/test1" entered phase "Bound"
I1025 00:03:26.760952       1 glusterfs.go:643] volume file_default_test1_55412f1f-d7cd-11e8-9252-fa163ea8bc36 deleted successfully
I1025 00:03:26.955758       1 glusterfs.go:872] service/endpoint: default/glusterfs-dynamic-test1 deleted successfully
I1025 00:03:26.956015       1 glusterfs.go:665] endpoint default/glusterfs-dynamic-test1 is deleted successfully
I1025 00:03:27.011657       1 glusterfs.go:597] delete volume file_default_test1_55412f1f-d7cd-11e8-9252-fa163ea8bc36
E1025 00:03:27.043238       1 glusterfs.go:640] failed to delete volume file_default_test1_55412f1f-d7cd-11e8-9252-fa163ea8bc36: Id not found
I1025 00:03:27.895654       1 glusterfs.go:643] volume file_default_test1_57ba78db-d7e9-11e8-9252-fa163ea8bc36 deleted successfully
E1025 00:03:27.904095       1 glusterfs.go:869] failed to delete service default/glusterfs-dynamic-test1: services "glusterfs-dynamic-test1" not found
E1025 00:03:27.904281       1 glusterfs.go:663] failed to delete endpoint/service default/glusterfs-dynamic-test1: failed to delete service default/glusterfs-dynamic-test1: services "glusterfs-dynamic-test1" not found
I1025 00:03:28.072629       1 glusterfs.go:597] delete volume file_default_test1_57ba78db-d7e9-11e8-9252-fa163ea8bc36
E1025 00:03:28.123619       1 glusterfs.go:640] failed to delete volume file_default_test1_57ba78db-d7e9-11e8-9252-fa163ea8bc36: Id not found



Version-Release number of selected component (if applicable):

ocp 3.10.34


How reproducible:

Easily


Steps to Reproduce:
1. Create PVC, wait to provision
2. Delete pvc
3. Create pvc with same name before the other is fully deleted

Actual results:

New PVC and PV, but the endpoints and server end up being deleted.


Expected results:

Should still create the new service and endpoints.

Additional info: