Description of problem: Disable a API federation when there is a resource of federatedAPI kind, then re-enable the API federation, the crd of federatedAPI will be deleted Version-Release number of selected component (if applicable): $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-04-10-182914 True False 5h5m Cluster version is 4.0.0-0.nightly-2019-04-10-182914 Federation v2 controller-manager version: version.Info{Version:"v0.0.7", GitCommit:"3e251608ca5357080b95a0bedc45759652f17d29", GitTreeState:"clean", BuildDate:"2019-03-22T22:30:26Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"} kubefed2 version: version.Info{Version:"v0.0.7", GitCommit:"83b778b6d92a7929efb7687d3d3d64bf0b3ad3bc", GitTreeState:"clean", BuildDate:"2019-03-19T18:40:46Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"} How reproducible: 100% Steps to Reproduce: 1. Enable deployments.apps federation $ kubefed2 enable deployments.apps --federation-namespace=federation-test 2. Create federateddeployment resource $ cat federateddeployment.yaml apiVersion: types.federation.k8s.io/v1alpha1 kind: FederatedDeployment metadata: name: test-deployment spec: template: metadata: labels: app: nginx spec: replicas: 3 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - image: openshift/hello-openshift name: nginx placement: clusterNames: - cluster2 - cluster1 overrides: - clusterName: cluster2 clusterOverrides: - path: spec.replicas value: 5 3. Disable deployments.apps federation $ kubefed2 disable deployments.apps --federation-namespace=federation-test --delete-from-api Disabled propagation for FederatedTypeConfig "federation-test/deployments.apps" customresourcedefinition "federateddeployments.types.federation.k8s.io" deleted federatedtypeconfig "federation-test/deployments.apps" deleted 4. Check federateddeployments crd and federatedtypeconfig $ oc get federatedtypeconfig deployments.apps -n federation-test Error from server (NotFound): federatedtypeconfigs.core.federation.k8s.io "deployments.apps" not found $ oc get crd federateddeployments.types.federation.k8s.io NAME CREATED AT federateddeployments.types.federation.k8s.io 2019-04-11T07:58:38Z 5. Delete federateddeployments test-deployment $ oc delete federateddeployment test-deployment -n federation-test federateddeployment.types.federation.k8s.io "test-deployment" deleted 6. Delete cmd hang there and did not return 7. For making the delete return, re-enable deployments.apps federation kubefed2 enable deployments.apps --federation-namespace=federation-test customresourcedefinition.apiextensions.k8s.io/federateddeployments.types.federation.k8s.io updated federatedtypeconfig.core.federation.k8s.io/deployments.apps created in namespace federation-test 8. Delete cmd returned successfully 9. Check federateddeployments crd and federatedtypeconfig Actual results: $ oc get federateddeployment -n federation-test error: the server doesn't have a resource type "federateddeployment" $ oc get crd federateddeployments.types.federation.k8s.io Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "federateddeployments.types.federation.k8s.io" not found $ oc get federatedtypeconfig deployments.apps -n federation-test NAME AGE deployments.apps 17m Expected results: federateddeployments.types.federation.k8s.io crd isn't deleted. Additional info:
I think I know what is going on here: - deleting the ftc is disabling the sync controller - the sync controller is required to remove the finalizer from the federated resources - the crd can't be removed until all cr's are removed, but they won't be deleted until the finalizer is deleted As per https://github.com/kubernetes-sigs/federation-v2/issues/721, disable should not delete the crd by default. When deletion of the crd is requested, kubefed2 should iterate over every cr and remove the finalizer so it can be garbage collected.
Fix upstream: https://github.com/kubernetes-sigs/federation-v2/pull/769
Fixed in version 0.0.10.
Verified with: Federation v2 controller-manager version: version.Info{Version:"v0.0.10", GitCommit:"71d233ede685707df554ef653e06bf7f0229415c", GitTreeState:"clean", BuildDate:"2019-05-06T22:54:46Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"} kubefedctl version: version.Info{Version:"v0.0.10-dirty", GitCommit:"71d233ede685707df554ef653e06bf7f0229415c", GitTreeState:"dirty", BuildDate:"2019-05-06T22:30:31Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}