Previously, if a PVC was deleted and created without any delay, the original PV and PVC would get deleted. With the new PV and PVC being provisioned, the service and endpoints are deleted and not recreated as they already exist at that point in time of endpoint creation/validation. With this fix, the PVC endpoint and service name format have been changed from ‘glusterfs-dynamic-PVCname’ to ‘glusterfs-dynamic-UID’. As a result,the older pvc and its related resources are deleted and the new pvc is created with new endpoint and service.
Description of problem:
Through automation or other methods, if a PVC is deleted and recreated within a short period, the original PV and PVC will get deleted, with the new PVC and PV being provisioned, but the service and endpoints end up getting deleted and not recreated because they already exist at that point in time.
There needs to be some kind of synchronization or remediation.
I1025 00:03:03.398954 1 glusterfs.go:703] Allocated GID 2001 for PVC test1
I1025 00:03:08.066124 1 glusterfs.go:703] Allocated GID 2002 for PVC test1
I1025 00:03:13.492857 1 glusterfs.go:776] volume with size 1 and name file_default_test1_57ba78db-d7e9-11e8-9252-fa163ea8bc36 created
I1025 00:03:13.540789 1 glusterfs.go:832] endpoint &Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:glusterfs-dynamic-test1,GenerateName:,Namespace:default,SelfLink:,UID:,ResourceVersion:,Generation:0,CreationTimestamp:0001-01-01 00:00:00 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{gluster.kubernetes.io/provisioned-for-pvc: test1,},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Subsets:[{[{10.10.85.244 <nil> nil} {10.10.85.243 <nil> nil} {10.10.85.221 <nil> nil}] [] [{ 1 TCP}]}],} already exist in namespace default
I1025 00:03:13.688109 1 glusterfs.go:852] service &Service{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:glusterfs-dynamic-test1,GenerateName:,Namespace:default,SelfLink:,UID:,ResourceVersion:,Generation:0,CreationTimestamp:0001-01-01 00:00:00 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{gluster.kubernetes.io/provisioned-for-pvc: test1,},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Spec:ServiceSpec{Ports:[{ TCP 1 {0 0 } 0}],Selector:map[string]string{},ClusterIP:,Type:,ExternalIPs:[],SessionAffinity:,LoadBalancerIP:,LoadBalancerSourceRanges:[],ExternalName:,ExternalTrafficPolicy:,HealthCheckNodePort:0,PublishNotReadyAddresses:false,SessionAffinityConfig:nil,},Status:ServiceStatus{LoadBalancer:LoadBalancerStatus{Ingress:[],},},} already exist in namespace default
I1025 00:03:13.732398 1 pv_controller.go:1516] volume "pvc-536c9fa2-d7e9-11e8-a294-fa163e7f64be" provisioned for claim "default/test1"
I1025 00:03:13.733243 1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"test1", UID:"536c9fa2-d7e9-11e8-a294-fa163e7f64be", APIVersion:"v1", ResourceVersion:"16495097", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-536c9fa2-d7e9-11e8-a294-fa163e7f64be using kubernetes.io/glusterfs
I1025 00:03:13.764913 1 glusterfs.go:597] delete volume file_default_test1_57ba78db-d7e9-11e8-9252-fa163ea8bc36
I1025 00:03:26.182323 1 glusterfs.go:776] volume with size 1 and name file_default_test1_5a82b19f-d7e9-11e8-9252-fa163ea8bc36 created
I1025 00:03:26.224816 1 glusterfs.go:832] endpoint &Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:glusterfs-dynamic-test1,GenerateName:,Namespace:default,SelfLink:,UID:,ResourceVersion:,Generation:0,CreationTimestamp:0001-01-01 00:00:00 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{gluster.kubernetes.io/provisioned-for-pvc: test1,},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Subsets:[{[{10.10.85.244 <nil> nil} {10.10.85.243 <nil> nil} {10.10.85.221 <nil> nil}] [] [{ 1 TCP}]}],} already exist in namespace default
I1025 00:03:26.302301 1 glusterfs.go:852] service &Service{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:glusterfs-dynamic-test1,GenerateName:,Namespace:default,SelfLink:,UID:,ResourceVersion:,Generation:0,CreationTimestamp:0001-01-01 00:00:00 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{gluster.kubernetes.io/provisioned-for-pvc: test1,},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Spec:ServiceSpec{Ports:[{ TCP 1 {0 0 } 0}],Selector:map[string]string{},ClusterIP:,Type:,ExternalIPs:[],SessionAffinity:,LoadBalancerIP:,LoadBalancerSourceRanges:[],ExternalName:,ExternalTrafficPolicy:,HealthCheckNodePort:0,PublishNotReadyAddresses:false,SessionAffinityConfig:nil,},Status:ServiceStatus{LoadBalancer:LoadBalancerStatus{Ingress:[],},},} already exist in namespace default
I1025 00:03:26.321893 1 pv_controller.go:1516] volume "pvc-5643c546-d7e9-11e8-a294-fa163e7f64be" provisioned for claim "default/test1"
I1025 00:03:26.322092 1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"test1", UID:"5643c546-d7e9-11e8-a294-fa163e7f64be", APIVersion:"v1", ResourceVersion:"16495112", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-5643c546-d7e9-11e8-a294-fa163e7f64be using kubernetes.io/glusterfs
I1025 00:03:26.343777 1 pv_controller.go:910] volume "pvc-5643c546-d7e9-11e8-a294-fa163e7f64be" bound to claim "default/test1"
I1025 00:03:26.425613 1 pv_controller.go:715] claim "default/test1" entered phase "Bound"
I1025 00:03:26.760952 1 glusterfs.go:643] volume file_default_test1_55412f1f-d7cd-11e8-9252-fa163ea8bc36 deleted successfully
I1025 00:03:26.955758 1 glusterfs.go:872] service/endpoint: default/glusterfs-dynamic-test1 deleted successfully
I1025 00:03:26.956015 1 glusterfs.go:665] endpoint default/glusterfs-dynamic-test1 is deleted successfully
I1025 00:03:27.011657 1 glusterfs.go:597] delete volume file_default_test1_55412f1f-d7cd-11e8-9252-fa163ea8bc36
E1025 00:03:27.043238 1 glusterfs.go:640] failed to delete volume file_default_test1_55412f1f-d7cd-11e8-9252-fa163ea8bc36: Id not found
I1025 00:03:27.895654 1 glusterfs.go:643] volume file_default_test1_57ba78db-d7e9-11e8-9252-fa163ea8bc36 deleted successfully
E1025 00:03:27.904095 1 glusterfs.go:869] failed to delete service default/glusterfs-dynamic-test1: services "glusterfs-dynamic-test1" not found
E1025 00:03:27.904281 1 glusterfs.go:663] failed to delete endpoint/service default/glusterfs-dynamic-test1: failed to delete service default/glusterfs-dynamic-test1: services "glusterfs-dynamic-test1" not found
I1025 00:03:28.072629 1 glusterfs.go:597] delete volume file_default_test1_57ba78db-d7e9-11e8-9252-fa163ea8bc36
E1025 00:03:28.123619 1 glusterfs.go:640] failed to delete volume file_default_test1_57ba78db-d7e9-11e8-9252-fa163ea8bc36: Id not found
Version-Release number of selected component (if applicable):
ocp 3.10.34
How reproducible:
Easily
Steps to Reproduce:
1. Create PVC, wait to provision
2. Delete pvc
3. Create pvc with same name before the other is fully deleted
Actual results:
New PVC and PV, but the endpoints and server end up being deleted.
Expected results:
Should still create the new service and endpoints.
Additional info:
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHEA-2019:0291
Comment 19Red Hat Bugzilla
2023-09-18 00:14:38 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days
Description of problem: Through automation or other methods, if a PVC is deleted and recreated within a short period, the original PV and PVC will get deleted, with the new PVC and PV being provisioned, but the service and endpoints end up getting deleted and not recreated because they already exist at that point in time. There needs to be some kind of synchronization or remediation. I1025 00:03:03.398954 1 glusterfs.go:703] Allocated GID 2001 for PVC test1 I1025 00:03:08.066124 1 glusterfs.go:703] Allocated GID 2002 for PVC test1 I1025 00:03:13.492857 1 glusterfs.go:776] volume with size 1 and name file_default_test1_57ba78db-d7e9-11e8-9252-fa163ea8bc36 created I1025 00:03:13.540789 1 glusterfs.go:832] endpoint &Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:glusterfs-dynamic-test1,GenerateName:,Namespace:default,SelfLink:,UID:,ResourceVersion:,Generation:0,CreationTimestamp:0001-01-01 00:00:00 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{gluster.kubernetes.io/provisioned-for-pvc: test1,},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Subsets:[{[{10.10.85.244 <nil> nil} {10.10.85.243 <nil> nil} {10.10.85.221 <nil> nil}] [] [{ 1 TCP}]}],} already exist in namespace default I1025 00:03:13.688109 1 glusterfs.go:852] service &Service{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:glusterfs-dynamic-test1,GenerateName:,Namespace:default,SelfLink:,UID:,ResourceVersion:,Generation:0,CreationTimestamp:0001-01-01 00:00:00 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{gluster.kubernetes.io/provisioned-for-pvc: test1,},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Spec:ServiceSpec{Ports:[{ TCP 1 {0 0 } 0}],Selector:map[string]string{},ClusterIP:,Type:,ExternalIPs:[],SessionAffinity:,LoadBalancerIP:,LoadBalancerSourceRanges:[],ExternalName:,ExternalTrafficPolicy:,HealthCheckNodePort:0,PublishNotReadyAddresses:false,SessionAffinityConfig:nil,},Status:ServiceStatus{LoadBalancer:LoadBalancerStatus{Ingress:[],},},} already exist in namespace default I1025 00:03:13.732398 1 pv_controller.go:1516] volume "pvc-536c9fa2-d7e9-11e8-a294-fa163e7f64be" provisioned for claim "default/test1" I1025 00:03:13.733243 1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"test1", UID:"536c9fa2-d7e9-11e8-a294-fa163e7f64be", APIVersion:"v1", ResourceVersion:"16495097", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-536c9fa2-d7e9-11e8-a294-fa163e7f64be using kubernetes.io/glusterfs I1025 00:03:13.764913 1 glusterfs.go:597] delete volume file_default_test1_57ba78db-d7e9-11e8-9252-fa163ea8bc36 I1025 00:03:26.182323 1 glusterfs.go:776] volume with size 1 and name file_default_test1_5a82b19f-d7e9-11e8-9252-fa163ea8bc36 created I1025 00:03:26.224816 1 glusterfs.go:832] endpoint &Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:glusterfs-dynamic-test1,GenerateName:,Namespace:default,SelfLink:,UID:,ResourceVersion:,Generation:0,CreationTimestamp:0001-01-01 00:00:00 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{gluster.kubernetes.io/provisioned-for-pvc: test1,},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Subsets:[{[{10.10.85.244 <nil> nil} {10.10.85.243 <nil> nil} {10.10.85.221 <nil> nil}] [] [{ 1 TCP}]}],} already exist in namespace default I1025 00:03:26.302301 1 glusterfs.go:852] service &Service{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:glusterfs-dynamic-test1,GenerateName:,Namespace:default,SelfLink:,UID:,ResourceVersion:,Generation:0,CreationTimestamp:0001-01-01 00:00:00 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{gluster.kubernetes.io/provisioned-for-pvc: test1,},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Spec:ServiceSpec{Ports:[{ TCP 1 {0 0 } 0}],Selector:map[string]string{},ClusterIP:,Type:,ExternalIPs:[],SessionAffinity:,LoadBalancerIP:,LoadBalancerSourceRanges:[],ExternalName:,ExternalTrafficPolicy:,HealthCheckNodePort:0,PublishNotReadyAddresses:false,SessionAffinityConfig:nil,},Status:ServiceStatus{LoadBalancer:LoadBalancerStatus{Ingress:[],},},} already exist in namespace default I1025 00:03:26.321893 1 pv_controller.go:1516] volume "pvc-5643c546-d7e9-11e8-a294-fa163e7f64be" provisioned for claim "default/test1" I1025 00:03:26.322092 1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"test1", UID:"5643c546-d7e9-11e8-a294-fa163e7f64be", APIVersion:"v1", ResourceVersion:"16495112", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-5643c546-d7e9-11e8-a294-fa163e7f64be using kubernetes.io/glusterfs I1025 00:03:26.343777 1 pv_controller.go:910] volume "pvc-5643c546-d7e9-11e8-a294-fa163e7f64be" bound to claim "default/test1" I1025 00:03:26.425613 1 pv_controller.go:715] claim "default/test1" entered phase "Bound" I1025 00:03:26.760952 1 glusterfs.go:643] volume file_default_test1_55412f1f-d7cd-11e8-9252-fa163ea8bc36 deleted successfully I1025 00:03:26.955758 1 glusterfs.go:872] service/endpoint: default/glusterfs-dynamic-test1 deleted successfully I1025 00:03:26.956015 1 glusterfs.go:665] endpoint default/glusterfs-dynamic-test1 is deleted successfully I1025 00:03:27.011657 1 glusterfs.go:597] delete volume file_default_test1_55412f1f-d7cd-11e8-9252-fa163ea8bc36 E1025 00:03:27.043238 1 glusterfs.go:640] failed to delete volume file_default_test1_55412f1f-d7cd-11e8-9252-fa163ea8bc36: Id not found I1025 00:03:27.895654 1 glusterfs.go:643] volume file_default_test1_57ba78db-d7e9-11e8-9252-fa163ea8bc36 deleted successfully E1025 00:03:27.904095 1 glusterfs.go:869] failed to delete service default/glusterfs-dynamic-test1: services "glusterfs-dynamic-test1" not found E1025 00:03:27.904281 1 glusterfs.go:663] failed to delete endpoint/service default/glusterfs-dynamic-test1: failed to delete service default/glusterfs-dynamic-test1: services "glusterfs-dynamic-test1" not found I1025 00:03:28.072629 1 glusterfs.go:597] delete volume file_default_test1_57ba78db-d7e9-11e8-9252-fa163ea8bc36 E1025 00:03:28.123619 1 glusterfs.go:640] failed to delete volume file_default_test1_57ba78db-d7e9-11e8-9252-fa163ea8bc36: Id not found Version-Release number of selected component (if applicable): ocp 3.10.34 How reproducible: Easily Steps to Reproduce: 1. Create PVC, wait to provision 2. Delete pvc 3. Create pvc with same name before the other is fully deleted Actual results: New PVC and PV, but the endpoints and server end up being deleted. Expected results: Should still create the new service and endpoints. Additional info: