Bug 1878333 - unable to import ocp 3.11 as managed cluster
Summary: unable to import ocp 3.11 as managed cluster
Alias: None
Product: Red Hat Advanced Cluster Management for Kubernetes
Classification: Red Hat
Component: Cluster Lifecycle
Version: rhacm-2.0.z
Hardware: x86_64
OS: Linux
Target Milestone: ---
: ---
Assignee: Hao Liu
QA Contact: magchen@redthat.com
Christopher Dawson
Depends On:
TreeView+ depends on / blocked
Reported: 2020-09-12 03:06 UTC by Chuan
Modified: 2020-10-29 13:52 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2020-10-29 13:52:21 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Github open-cluster-management backlog issues 5422 0 None None None 2020-10-06 16:16:07 UTC

Description Chuan 2020-09-12 03:06:33 UTC
Description of problem:
We're trying to import an OCP 3.1.1 cluster but failed. 

getting below output:
customresourcedefinition.apiextensions.k8s.io/klusterlets.operator.open-cluster-management.io configured
clusterrole.rbac.authorization.k8s.io/klusterlet configured
clusterrole.rbac.authorization.k8s.io/open-cluster-management:klusterlet-admin-aggregate-clusterrole configured
clusterrolebinding.rbac.authorization.k8s.io/klusterlet configured
namespace/open-cluster-management-agent configured
secret/open-cluster-management-image-pull-credentials unchanged
serviceaccount/klusterlet configured
deployment.apps/klusterlet unchanged
klusterlet.operator.open-cluster-management.io/klusterlet configured
Error from server (BadRequest): error when creating "STDIN": Secret in version "v1" cannot be handled as a Secret: 
v1.ObjectMeta.TypeMeta: Kind: Data: decode base64: illegal base64 data at input byte 1313, error found in #10 byte of ...|na1h5dwo="},"kind":"|..., bigger context ...|2TjVEdWJYWmFFYXFzVzlKWjlSNWdsTEcxd0hIQm1na1h5dwo="},"kind":"Secret","metadata":{"annotations":{"kube|...    

Version-Release number of selected component (if applicable):
RHACM 2.0.2 GA running on OCP 4.4.19
managed cluster: OCP 3.11
# oc version
oc v3.11.216
kubernetes v1.11.0+d4cacc0
features: Basic-Auth GSSAPI Kerberos SPNEGO

How reproducible:

Steps to Reproduce:
1. Install RHACM 2.0.2 GA
2. generating import command
3. running import command from OCP 3.11 cluster master node.

Actual results:

Expected results:

Additional info:
We can import ocp 4.3 managed cluster without issue, it seems only occurs on ocp 3.11 manged cluster.

Comment 1 Chuan 2020-09-12 04:33:04 UTC
Since the import script contains private information, I will not post it here publicly. 
but I'm more than happy to send it to whoever investigate this problem in RH.

Comment 2 Chuan 2020-09-12 08:13:16 UTC
  After some digging, it looks like there's an issue in ACM import script since 2.0.0 GA, the user token in the bootstrap kubeconfig is not properly base64 encoded, only can be partially decoded. 
  It seems in most of k8s env it won't cause any trouble, but in k8s v1.11 it will, and ocp 3.11 is based on k8s v1.11.

Comment 3 hanzhang 2020-09-16 18:46:36 UTC
Hi Chuan,

Thanks for reporting. We do have OCP 3.11 version requirements to be 3.11.200, and I noticed you are having 3.11.216? which should be good.

Would you like to provide some more details for the import message & the base64 tokens, so we can do some further investigation?

Comment 4 Chuan 2020-09-17 07:55:43 UTC
Hi Zhanghan,

  We were importing the ocp 3.11 from a ACM 2.0.2 GA on ocp 4.x env, so basically the import message/script has no different than import any other OCP 4.x cluster, b/c the there's nothing indicating it's a 3.x cluster to be imported when generating the script.

  the user token error existing in all ACM version since 2.0.0/2.0.1/2.0.2 GA

simply check the kubeconfig encoded string you could see it.

- context:
    cluster: default-cluster
    namespace: default
    user: default-auth
  name: default-context
current-context: default-context
kind: Config
preferences: {}
- name: default-auth
    token: eyJhbGciOiJSUzI1NiIsImtpZCI6Ii1SVXJnd2hNT0JIeVpBN2tRcFFaT3FIUU5mV29FSERWbHl2RkFjLW0ydHcifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJjbHVzdGVyLWltcG9ydC1zY3JpcHQtdGVzdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJjbHVzdGVyLWltcG9ydC1zY3JpcHQtdGVzdC1ib290c3RyYXAtc2EtdG9rZW4tdGNrdzkiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiY2x1c3Rlci1pbXBvcnQtc2NyaXB0LXRlc3QtYm9vdHN0cmFwLXNhIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiMzE4ZWJmMjktZWExMi00MGYxLTlhNjUtY2RiYTkyMjMxNDAyIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OmNsdXN0ZXItaW1wb3J0LXNjcmlwdC10ZXN0OmNsdXN0ZXItaW1wb3J0LXNjcmlwdC10ZXN0LWJvb3RzdHJhcC1zYSJ9.UxyggQr4opcqGVDjRGyCrxbbg0ymfUy8ex7mNlBdusqjcoeQ5fkpWl3V9iTRJthzWEFh0SttSUH6fJ8jbBIEyRehuCcPbSSfFS8503gTJk7vRPW6fK472piu56bUUEc2CZLj-Vr-k3-BzXeHF0szCGRdRqym1ngnq-VAcZa57pWobYqwsimRYPqxnj2tLwEzXU7az6NdFrn_pOCm9PI-5BGiYi2wal605HJ5G7B_BkcvTnJiHSIehHnSAtzgaejd2jnC2VNLyuBg0v6kVQgZ9NywV7aYV3J8dPd6FL8WKdlnpVr1r52aeegbPRl52MdWFuFb6HHOyINHOjKszCOUKg

# echo <THE TOKEN> |base64 -d
{"alg":"RS256","kid":"-RUrgwhMOBHyZA7kQpQZOqHQNfWoEHDVlyvFAc-m2tw"}base64: invalid input

Comment 5 hanzhang 2020-09-17 22:59:37 UTC
Hi Chuan, 

OK, you are saying the secret `bootstrap-hub-kubeconfig` in import command has invalid base64 string? in that case, you are not able to import any other clusters?

Not sure I'm understanding the issue correctly, but the token of kubeconfig should not block import, the secret data may if not in good base64 format. 

We can get the import yaml files by running this command on hub:
oc get secret ${CLUSTER_NAME}-import -n ${CLUSTER_NAME} -o jsonpath={.data.import\\.yaml} | base64 --decode > import.yaml

and you will be able to check if the secret `bootstrap-hub-kubeconfig` is possible to be apply on OCP 3.11, and also cluster in other versions, so we know if it's this file's problem. Please check the data.kubeconfig is can be base64 decoded. 


Comment 6 Chuan 2020-09-18 03:12:35 UTC
As just discussed with Hanqiu in the call, this issue only occurs in OCP 3.11 Env, when using oc CLI 3.11.

and there're workarounds to solve it.
1. download a ocp 4.x oc CLI binary from RH site, and replace the ocp 3.x CLI 
2. or, you can also manually decode the yaml save it to a file, and then kubectl apply -f <Saved Yaml File> would also work.

Comment 7 Chuan 2020-09-18 03:15:28 UTC
forgot to mention actually the workaround requires to replace both oc 3.x binary and kubectl binary in the env with oc 4.x Cli binary.
because the import cmd actually use kubectl.

Comment 8 Mike Ng 2020-10-06 16:14:50 UTC
G2Bsync 704387402 comment 
 cadawson Tue, 06 Oct 2020 16:10:54 UTC 
 G2Bsync This was added as a troubleshooting doc that listed the problem and the possible workarounds. The version of `kubectl` (1,11) is causing the problem. Upgrading that version solves the problem. 

Approved, merging, and closing.

Note You need to log in before you can comment on or make changes to this bug.