Description of problem: Version-Release number of the following components: 4.3.0-0.nightly-2019-11-13-233341 How reproducible: Always Steps to Reproduce: 1. Create a VPC 2. Launch a mitm proxy in this VPC 3. Trigger an installation enabled proxy Actual results: cloud provider could not be initialized # oc -n openshift-kube-controller-manager get po NAME READY STATUS RESTARTS AGE kube-controller-manager-jialiu43-mitm-11142317-m-0.c.openshift-qe.internal 2/3 CrashLoopBackOff 17 33m kube-controller-manager-jialiu43-mitm-11142317-m-1.c.openshift-qe.internal 1/3 CrashLoopBackOff 20 32m kube-controller-manager-jialiu43-mitm-11142317-m-2.c.openshift-qe.internal 2/3 CrashLoopBackOff 21 34m Expected results: installation is passed. Additional info: This issue is blocking QE's testing against cluster behind mitm proxy
@Daneyon, do we have any update on this issue?
googleapi.com is not automatically added to the default no proxy list. It appears the ca trust bundle being used by controllermanager does not include the ca that signed the proxy's cert. Can you confirm that you created the ca cert configmap and that you referenced the configmap in proxy.spec.trustedCA?
Reproduced this issue using payload 4.3.0-0.nightly-2019-12-04-214544 No configmap with config.openshift.io/inject-trusted-cabundle="true" label created under openshift-kube-controller-manager namespace. # oc get cm -n openshift-kube-controller-manager -o yaml |grep inject-trusted-cabundle So this looks like some remaining tasks that need to be completed from kube-controller-manager side for fully proxy support.
Based on the discussion I've had with mfojtik, and what was stated already in https://bugzilla.redhat.com/show_bug.cgi?id=1772756#c2 it's a network bug, and googleapi.com should be whitelisted in NO_PROXY. Moving this back to networking. KCM should never reach to any external location, except for cloud providers.
The controller manager should add trusted ca support so it can proxy https connections. By doing so, cluster admins will have choice whether or not to proxy gcloud api calls. If not, then add https://www.googleapis.com/compute to the default no_proxy lis. Note that product docs will need to be updated to reflect this requirement. Another option is to require https://cloud.google.com/vpc/docs/private-access-options for gce with mitm proxy.
Working on a fix...
Verify this bug with payload 4.4.0-0.nightly-2019-12-14-103510 # oc get cm trusted-ca-bundle -n openshift-kube-controller-manager -o yaml ... metadata: creationTimestamp: "2019-12-16T05:30:13Z" labels: config.openshift.io/inject-trusted-cabundle: "true" name: trusted-ca-bundle namespace: openshift-kube-controller-manager resourceVersion: "3745" # oc get pod -n openshift-kube-controller-manager NAME READY STATUS RESTARTS AGE installer-4-qe-gpei-proxy-12160505-m-1.c.openshift-qe.internal 0/1 Completed 0 4h42m installer-5-qe-gpei-proxy-12160505-m-0.c.openshift-qe.internal 0/1 Completed 0 4h40m installer-5-qe-gpei-proxy-12160505-m-2.c.openshift-qe.internal 0/1 Completed 0 4h42m installer-6-qe-gpei-proxy-12160505-m-0.c.openshift-qe.internal 0/1 Completed 0 4h34m installer-6-qe-gpei-proxy-12160505-m-1.c.openshift-qe.internal 0/1 Completed 0 4h35m installer-6-qe-gpei-proxy-12160505-m-2.c.openshift-qe.internal 0/1 Completed 0 4h33m kube-controller-manager-qe-gpei-proxy-12160505-m-0.c.openshift-qe.internal 3/3 Running 0 4h34m kube-controller-manager-qe-gpei-proxy-12160505-m-1.c.openshift-qe.internal 3/3 Running 0 4h35m kube-controller-manager-qe-gpei-proxy-12160505-m-2.c.openshift-qe.internal 3/3 Running 0 4h33m revision-pruner-4-qe-gpei-proxy-12160505-m-1.c.openshift-qe.internal 0/1 Completed 0 4h42m revision-pruner-5-qe-gpei-proxy-12160505-m-0.c.openshift-qe.internal 0/1 Completed 0 4h35m revision-pruner-5-qe-gpei-proxy-12160505-m-2.c.openshift-qe.internal 0/1 Completed 0 4h41m revision-pruner-6-qe-gpei-proxy-12160505-m-0.c.openshift-qe.internal 0/1 Completed 0 4h33m revision-pruner-6-qe-gpei-proxy-12160505-m-1.c.openshift-qe.internal 0/1 Completed 0 4h34m revision-pruner-6-qe-gpei-proxy-12160505-m-2.c.openshift-qe.internal 0/1 Completed 0 4h31m Inside the kcm pod, CA cert for proxy could be found in /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0581