Bug 1772756 - cloud provider could not be initialized behind mitm proxy which is using some self-signed certificates
Summary: cloud provider could not be initialized behind mitm proxy which is using some...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: kube-controller-manager
Version: 4.3.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.4.0
Assignee: Maciej Szulik
QA Contact: Gaoyun Pei
URL:
Whiteboard:
Depends On:
Blocks: 1782819
TreeView+ depends on / blocked
 
Reported: 2019-11-15 05:43 UTC by Johnny Liu
Modified: 2020-05-04 11:16 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Certificates were not properly propagated. Consequence: Cloud provider could not be initialized behind mitm proxy. Fix: Properly propagate certificates to kube-controller-manager. Result: Cloud provider works as expected with proxy set.
Clone Of:
: 1782819 (view as bug list)
Environment:
Last Closed: 2020-05-04 11:15:35 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-kube-controller-manager-operator pull 325 0 None closed Bug 1772756: Inject kube-controller-manager pods trust stores with trusted ca bundle 2021-02-18 09:47:32 UTC
Red Hat Product Errata RHBA-2020:0581 0 None None None 2020-05-04 11:16:06 UTC

Description Johnny Liu 2019-11-15 05:43:39 UTC
Description of problem:

Version-Release number of the following components:
4.3.0-0.nightly-2019-11-13-233341

How reproducible:
Always

Steps to Reproduce:
1. Create a VPC
2. Launch a mitm proxy in this VPC
3. Trigger an installation enabled proxy

Actual results:
cloud provider could not be initialized 
# oc -n openshift-kube-controller-manager get po
NAME                                                                         READY   STATUS             RESTARTS   AGE
kube-controller-manager-jialiu43-mitm-11142317-m-0.c.openshift-qe.internal   2/3     CrashLoopBackOff   17         33m
kube-controller-manager-jialiu43-mitm-11142317-m-1.c.openshift-qe.internal   1/3     CrashLoopBackOff   20         32m
kube-controller-manager-jialiu43-mitm-11142317-m-2.c.openshift-qe.internal   2/3     CrashLoopBackOff   21         34m

Expected results:
installation is passed.

Additional info:
This issue is blocking QE's testing against cluster behind mitm proxy

Comment 4 Aniket Bhat 2019-11-25 16:08:27 UTC
@Daneyon, do we have any update on this issue?

Comment 5 Daneyon Hansen 2019-12-02 17:49:32 UTC
googleapi.com is not automatically added to the default no proxy list. It appears the ca trust bundle being used by controllermanager does not include the ca that signed the proxy's cert. Can you confirm that you created the ca cert configmap and that you referenced the configmap in proxy.spec.trustedCA?

Comment 7 Gaoyun Pei 2019-12-05 07:42:10 UTC
Reproduced this issue using payload 4.3.0-0.nightly-2019-12-04-214544

No configmap with config.openshift.io/inject-trusted-cabundle="true" label created under openshift-kube-controller-manager namespace.

# oc get cm -n openshift-kube-controller-manager -o yaml |grep inject-trusted-cabundle

So this looks like some remaining tasks that need to be completed from kube-controller-manager side for fully proxy support.

Comment 8 Maciej Szulik 2019-12-05 15:22:46 UTC
Based on the discussion I've had with mfojtik, and what was stated already in https://bugzilla.redhat.com/show_bug.cgi?id=1772756#c2 it's a network bug, and googleapi.com should be whitelisted in NO_PROXY.
Moving this back to networking. KCM should never reach to any external location, except for cloud providers.

Comment 9 Daneyon Hansen 2019-12-06 17:10:06 UTC
The controller manager should add trusted ca support so it can proxy https connections. By doing so, cluster admins will have choice whether or not to proxy gcloud api calls. If not, then add https://www.googleapis.com/compute to the default no_proxy lis. Note that product docs will need to be updated to reflect this requirement. Another option is to require https://cloud.google.com/vpc/docs/private-access-options for gce with mitm proxy.

Comment 10 Maciej Szulik 2019-12-11 19:47:05 UTC
Working on a fix...

Comment 12 Gaoyun Pei 2019-12-16 10:17:19 UTC
Verify this bug with payload 4.4.0-0.nightly-2019-12-14-103510

# oc get cm trusted-ca-bundle -n openshift-kube-controller-manager -o yaml 
...
metadata:
  creationTimestamp: "2019-12-16T05:30:13Z"
  labels:
    config.openshift.io/inject-trusted-cabundle: "true"
  name: trusted-ca-bundle
  namespace: openshift-kube-controller-manager
  resourceVersion: "3745"


# oc get pod -n openshift-kube-controller-manager
NAME                                                                         READY   STATUS      RESTARTS   AGE
installer-4-qe-gpei-proxy-12160505-m-1.c.openshift-qe.internal               0/1     Completed   0          4h42m
installer-5-qe-gpei-proxy-12160505-m-0.c.openshift-qe.internal               0/1     Completed   0          4h40m
installer-5-qe-gpei-proxy-12160505-m-2.c.openshift-qe.internal               0/1     Completed   0          4h42m
installer-6-qe-gpei-proxy-12160505-m-0.c.openshift-qe.internal               0/1     Completed   0          4h34m
installer-6-qe-gpei-proxy-12160505-m-1.c.openshift-qe.internal               0/1     Completed   0          4h35m
installer-6-qe-gpei-proxy-12160505-m-2.c.openshift-qe.internal               0/1     Completed   0          4h33m
kube-controller-manager-qe-gpei-proxy-12160505-m-0.c.openshift-qe.internal   3/3     Running     0          4h34m
kube-controller-manager-qe-gpei-proxy-12160505-m-1.c.openshift-qe.internal   3/3     Running     0          4h35m
kube-controller-manager-qe-gpei-proxy-12160505-m-2.c.openshift-qe.internal   3/3     Running     0          4h33m
revision-pruner-4-qe-gpei-proxy-12160505-m-1.c.openshift-qe.internal         0/1     Completed   0          4h42m
revision-pruner-5-qe-gpei-proxy-12160505-m-0.c.openshift-qe.internal         0/1     Completed   0          4h35m
revision-pruner-5-qe-gpei-proxy-12160505-m-2.c.openshift-qe.internal         0/1     Completed   0          4h41m
revision-pruner-6-qe-gpei-proxy-12160505-m-0.c.openshift-qe.internal         0/1     Completed   0          4h33m
revision-pruner-6-qe-gpei-proxy-12160505-m-1.c.openshift-qe.internal         0/1     Completed   0          4h34m
revision-pruner-6-qe-gpei-proxy-12160505-m-2.c.openshift-qe.internal         0/1     Completed   0          4h31m

Inside the kcm pod, CA cert for proxy could be found in /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem

Comment 14 errata-xmlrpc 2020-05-04 11:15:35 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581


Note You need to log in before you can comment on or make changes to this bug.