Description of problem: When join a cluster to a federation, secret is generated before cluster's adding. When unjoin a cluster from a federation, cluster is removed and then secret is deleted. The problem is, with invalid command, cluster has been removed but secret left, or secret has been generated but cluster can't be added into the federation. This inconsistency will block next correct operation. Version-Release number of selected component (if applicable): openshift v3.6.135 kubernetes v1.6.1+5115d708d7 etcd 3.2.1 registry.ops.openshift.com/openshift3/ose-federation v3.6.140 How reproducible: Always Steps to Reproduce: 1. Given there are two clusters in the federation # oc get cluster --context=qwangfed NAME STATUS AGE cluster1 Ready 2h cluster2 Ready 8m 2. Remove cluster2 from the federation with invalid parameters. Then check cluster and secret # kubefed unjoin cluster2 --host-cluster-context=adc --context=qwangfed # oc get cluster --context=qwangfed # oc get secret -n federation-system | grep cluster2 3. Join the deleted cluster to the federation again # kubefed join cluster2 --host-cluster-context=${HOST_CONTEXT} --cluster-context=${CLUSTER2_CONTEXT} --context=qwangfed 4. Delete secret # oc get secret -n federation-system | grep cluster2 # oc delete secret cluster2 -n federation-system 5. Do step 3 again and check cluster 6. Remove cluster2 from the federation with valid parameters and check cluster and secret # kubefed unjoin cluster2 --context=qwangfed 7. Join cluster2 to the federation without mandatory parameters. Then Check cluster and secret # kubefed join cluster2 --host-cluster-context=${HOST_CONTEXT} --cluster-context=${CLUSTER2_CONTEXT} 8. Join cluster2 to the federation with correct parameters (Step 3). Actual results: 2. # kubefed unjoin cluster2 --host-cluster-context=adc --context=qwangfed --federation-system-namespace='123' error: context "adc" does not exist # oc get cluster --context=qwangfed NAME STATUS AGE cluster1 Ready 2h --->Expected result: Cluster2 shouldn't be removed # oc get secret -n federation-system | grep cluster2 cluster2 Opaque 1 4m ---> Cluster has been removed but secret left, this will the cluster2's rejoin. 3. # kubefed join cluster2 --host-cluster-context=${HOST_CONTEXT} --cluster-context=${CLUSTER2_CONTEXT} --context=qwangfed Error from server (AlreadyExists): secrets "cluster2" already exists 4. # oc delete secret cluster2 -n federation-system secret "cluster2" deleted 5. # kubefed join cluster2 --host-cluster-context=${HOST_CONTEXT} --cluster-context=${CLUSTER2_CONTEXT} --context=qwangfed cluster "cluster2" created # oc get cluster --context=qwangfed NAME STATUS AGE cluster1 Ready 2h cluster2 Ready 1m 6. # kubefed unjoin cluster2 --context=qwangfed Successfully removed cluster "cluster2" from federation # oc get cluster --context=qwangfed NAME STATUS AGE cluster1 Ready 2h # oc get secret -n federation-system | grep cluster2 7. # kubefed join cluster2 --host-cluster-context=${HOST_CONTEXT} --cluster-context=${CLUSTER2_CONTEXT} error: server does not support API version "federation/v1beta1 # oc get cluster --context=qwangfed NAME STATUS AGE cluster1 Ready 2h # oc get secret -n federation-system | grep cluster2 cluster2 Opaque --->Expected result: secret shouldn't be generated. The secret will block cluster2's join 8. # kubefed join cluster2 --host-cluster-context=${HOST_CONTEXT} --context=qwangfed --cluster-context=${CLUSTER2_CONTEXT} Error from server (AlreadyExists): secrets "cluster2" already exists Expected results: 2. With incorrect "unjoin" parameters, cluster shouldn't be removed. 7. Lack mandatory "join" parameters, secret shouldn't be created. Additional info:
federation is tech preview in 3.6, we can target this for 3.7
federation is still tech preview in 3.7. reducing severity as a result.
Federation has change significantly since this report, likely not an issue or not applicable any more.