Fedora Account System
Red Hat Associate
Red Hat Customer
Description of problem: We have an ACM 2.0.3 instance running on top of OCP 4.5 in an air-gapped environment. We want to create a Git channel in ACM, our GitLab instance is local and is using a self-signed certificate authority. When the channel is created the subscription fails to propagate. Looking at the logs of the multicluster-operators-hub-subscription pod we see that the clone is failing due to a x509: certificate signed by unknown authority error. Version-Release number of selected component (if applicable): 2.0.3 How reproducible: Occurs each time. Steps to Reproduce: 1. Create a Git channel in ACM using a Git source with a self-signed certificate authority as the source. 2. 3. Actual results: Subscription will fail to propagate. Expected results: Subscription propagates and application is deployed. Additional info: The issue can be worked around by either injecting the Git channel source's certificate into the defaults CA bundle of the multicluster-operators-hub-subscription pod or by replacing the ACM CA with the CA used by the Git channel source. Neither work-around is optimal for us at this time.
G2Bsync 717546233 comment xiangjingli Tue, 27 Oct 2020 21:18:49 UTC G2Bsync we do support to clone private git repo by implementing git basic authentication method (user/access token). In order to do it, 1. Create the git channel with spec.secretRef, where a secret is bound as the git repo authentication https://github.com/open-cluster-management/multicloud-operators-channel/blob/fa55cf642642fff2a98ed62617bb9c83e6458903/examples/e2e_channel_all_types_example/object_bucket_channel_e2e.yaml#L36 2. Create a secret like this, where git repo user and access token are defined. The `user` and `accessToken` keys can't be modified. ``` apiVersion: v1 kind: Secret metadata: name: dev namespace: ch-obj data: user: <git repo user id> accessToken: <git repo access token> ``` Please note only git basic authentication is supported in our git channel/subscription for now.
Thanks Mike, the issue is that the user isn't even getting to the authentication portion as the error blocks at the TLS handshake. git clone (https) can be done with SSH or HTTP (token from secret) will work if the trust relationship is established between pod multicluster-operators-hub-subscription and our git URL. diff: git clone without user and password for https://something -> 401 Unauthorized --insecureSkipVerify=true (or trusted certificate by global CA) git clone without user and password for https://something -> ERROR - x509 certificate signed by unknown authority (with self-signed certificate or without insecureSkipVerify=true) - so they're not reaching the authentication layer because initial handshake is not done.
Thanks James! After further investigation, this is the feature we are missing to support in git channel, while it is supported in helm channel. We will fix it in the coming release ACM 2.2. The task has been created and assigned. same as helm channel, the fix will support end user to define configmap with insecureSkipVerify: true setting and associate the configmap to the git channel. When git subscription detects the setting, it will custom git repo client by overriding the https client handler with the setting InsecureSkipVerify: true. ``` apiVersion: v1 kind: ConfigMap metadata: name: skip-cert-verify namespace: mmv-https-test data: insecureSkipVerify: "true" ```
Patching ACM and managed clusters to connect to Git server with custom certificate You need to have RHACM 2.1.x. If you have previous version of RHACM, upgrade first. On the ACM hub cluster, update the channel CRD with https://github.com/open-cluster-management/multicloud-operators-channel/blob/master/deploy/crds/apps.open-cluster-management.io_channels_crd.yaml using this command. oc apply -f apps.open-cluster-management.io_channels_crd.yaml Then in open-cluster-management namespace, edit the advanced-cluster-management.v2.1.0 csv. oc edit csv advanced-cluster-management.v2.1.0 -n open-cluster-management Look for containers multicluster-operators-standalone-subscription and multicluster-operators-hub-subscription and update their images to quay.io/open-cluster-management/multicluster-operators-subscription:2.1-PR334-606ad1861c7b78b8ae8ff4d3ccc7f2dc86d9dd6e. This will recreate multicluster-operators-standalone-subscription-xxxxxxx and multicluster-operators-hub-subscription-xxxxxxx pods in open-cluster-management namespace. Check that the new pods are running with the new docker image. Then update the images on managed (spoke) clusters. On the ACM hub cluster, run this command by replacing CLUSTER_NAME with the actual managed cluster name. oc annotate klusterletaddonconfig -n CLUSTER_NAME CLUSTER_NAME klusterletaddonconfig-pause=true --overwrite=true Then run oc edit manifestwork -n CLUSTER_NAME CLUSTER_NAME-klusterlet-addon-appmgr replacing CLUSTER_NAME with the actual managed cluster name. Look for spec.global.imageOverrides.multicluster_operators_subscription. Set the value to quay.io/open-cluster-management/multicluster-operators-subscription:2.1-PR334-606ad1861c7b78b8ae8ff4d3ccc7f2dc86d9dd6e. This will recreate the klusterlet-addon-appmgr-xxxxxx pod in open-cluster-management-agent-addon namespace on the managed cluster. Check that the new pod is running with the new docker image. When you create an application through UI or CLI, add `insecureSkipVerify: true' in the channel spec manually. apiVersion: apps.open-cluster-management.io/v1 ind: Channel metadata: labels: name: sample-channel namespace: sample spec: type: GitHub pathname: <Git URL> insecureSkipVerify: true
Yes, this feature will be GA in 2.2.
*** Bug 1892008 has been marked as a duplicate of this bug. ***
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat Advanced Cluster Management for Kubernetes version 2.2 images), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2021:0729
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days