Hide Forgot
Description of problem: We're trying to import an OCP 3.1.1 cluster but failed. getting below output: customresourcedefinition.apiextensions.k8s.io/klusterlets.operator.open-cluster-management.io configured clusterrole.rbac.authorization.k8s.io/klusterlet configured clusterrole.rbac.authorization.k8s.io/open-cluster-management:klusterlet-admin-aggregate-clusterrole configured clusterrolebinding.rbac.authorization.k8s.io/klusterlet configured namespace/open-cluster-management-agent configured secret/open-cluster-management-image-pull-credentials unchanged serviceaccount/klusterlet configured deployment.apps/klusterlet unchanged klusterlet.operator.open-cluster-management.io/klusterlet configured Error from server (BadRequest): error when creating "STDIN": Secret in version "v1" cannot be handled as a Secret: v1.Secret.ObjectMeta: v1.ObjectMeta.TypeMeta: Kind: Data: decode base64: illegal base64 data at input byte 1313, error found in #10 byte of ...|na1h5dwo="},"kind":"|..., bigger context ...|2TjVEdWJYWmFFYXFzVzlKWjlSNWdsTEcxd0hIQm1na1h5dwo="},"kind":"Secret","metadata":{"annotations":{"kube|... Version-Release number of selected component (if applicable): RHACM 2.0.2 GA running on OCP 4.4.19 managed cluster: OCP 3.11 # oc version oc v3.11.216 kubernetes v1.11.0+d4cacc0 features: Basic-Auth GSSAPI Kerberos SPNEGO How reproducible: Steps to Reproduce: 1. Install RHACM 2.0.2 GA 2. generating import command 3. running import command from OCP 3.11 cluster master node. Actual results: Expected results: Additional info: We can import ocp 4.3 managed cluster without issue, it seems only occurs on ocp 3.11 manged cluster.
Since the import script contains private information, I will not post it here publicly. but I'm more than happy to send it to whoever investigate this problem in RH.
After some digging, it looks like there's an issue in ACM import script since 2.0.0 GA, the user token in the bootstrap kubeconfig is not properly base64 encoded, only can be partially decoded. It seems in most of k8s env it won't cause any trouble, but in k8s v1.11 it will, and ocp 3.11 is based on k8s v1.11.
Hi Chuan, Thanks for reporting. We do have OCP 3.11 version requirements to be 3.11.200, and I noticed you are having 3.11.216? which should be good. https://access.redhat.com/documentation/en-us/red_hat_advanced_cluster_management_for_kubernetes/2.0/html/manage_cluster/supported-clouds#supported-managed-cluster-providers Would you like to provide some more details for the import message & the base64 tokens, so we can do some further investigation?
Hi Zhanghan, We were importing the ocp 3.11 from a ACM 2.0.2 GA on ocp 4.x env, so basically the import message/script has no different than import any other OCP 4.x cluster, b/c the there's nothing indicating it's a 3.x cluster to be imported when generating the script. the user token error existing in all ACM version since 2.0.0/2.0.1/2.0.2 GA simply check the kubeconfig encoded string you could see it. contexts: - context: cluster: default-cluster namespace: default user: default-auth name: default-context current-context: default-context kind: Config preferences: {} users: - name: default-auth user: token: eyJhbGciOiJSUzI1NiIsImtpZCI6Ii1SVXJnd2hNT0JIeVpBN2tRcFFaT3FIUU5mV29FSERWbHl2RkFjLW0ydHcifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJjbHVzdGVyLWltcG9ydC1zY3JpcHQtdGVzdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJjbHVzdGVyLWltcG9ydC1zY3JpcHQtdGVzdC1ib290c3RyYXAtc2EtdG9rZW4tdGNrdzkiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiY2x1c3Rlci1pbXBvcnQtc2NyaXB0LXRlc3QtYm9vdHN0cmFwLXNhIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiMzE4ZWJmMjktZWExMi00MGYxLTlhNjUtY2RiYTkyMjMxNDAyIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OmNsdXN0ZXItaW1wb3J0LXNjcmlwdC10ZXN0OmNsdXN0ZXItaW1wb3J0LXNjcmlwdC10ZXN0LWJvb3RzdHJhcC1zYSJ9.UxyggQr4opcqGVDjRGyCrxbbg0ymfUy8ex7mNlBdusqjcoeQ5fkpWl3V9iTRJthzWEFh0SttSUH6fJ8jbBIEyRehuCcPbSSfFS8503gTJk7vRPW6fK472piu56bUUEc2CZLj-Vr-k3-BzXeHF0szCGRdRqym1ngnq-VAcZa57pWobYqwsimRYPqxnj2tLwEzXU7az6NdFrn_pOCm9PI-5BGiYi2wal605HJ5G7B_BkcvTnJiHSIehHnSAtzgaejd2jnC2VNLyuBg0v6kVQgZ9NywV7aYV3J8dPd6FL8WKdlnpVr1r52aeegbPRl52MdWFuFb6HHOyINHOjKszCOUKg # echo <THE TOKEN> |base64 -d {"alg":"RS256","kid":"-RUrgwhMOBHyZA7kQpQZOqHQNfWoEHDVlyvFAc-m2tw"}base64: invalid input
Hi Chuan, OK, you are saying the secret `bootstrap-hub-kubeconfig` in import command has invalid base64 string? in that case, you are not able to import any other clusters? Not sure I'm understanding the issue correctly, but the token of kubeconfig should not block import, the secret data may if not in good base64 format. We can get the import yaml files by running this command on hub: ``` oc get secret ${CLUSTER_NAME}-import -n ${CLUSTER_NAME} -o jsonpath={.data.import\\.yaml} | base64 --decode > import.yaml ``` and you will be able to check if the secret `bootstrap-hub-kubeconfig` is possible to be apply on OCP 3.11, and also cluster in other versions, so we know if it's this file's problem. Please check the data.kubeconfig is can be base64 decoded. Thanks
As just discussed with Hanqiu in the call, this issue only occurs in OCP 3.11 Env, when using oc CLI 3.11. and there're workarounds to solve it. 1. download a ocp 4.x oc CLI binary from RH site, and replace the ocp 3.x CLI 2. or, you can also manually decode the yaml save it to a file, and then kubectl apply -f <Saved Yaml File> would also work.
forgot to mention actually the workaround requires to replace both oc 3.x binary and kubectl binary in the env with oc 4.x Cli binary. because the import cmd actually use kubectl.
G2Bsync 704387402 comment cadawson Tue, 06 Oct 2020 16:10:54 UTC G2Bsync This was added as a troubleshooting doc that listed the problem and the possible workarounds. The version of `kubectl` (1,11) is causing the problem. Upgrading that version solves the problem. Approved, merging, and closing.