Description of problem: As per source code analysis and also test on customer environment, it seems that the Cincinnati client of the Cluster Version Operator only takes care of HTTPS proxy setting and TLS settings, but not of the NoProxy Version-Release: 4.7 How reproducible: Always (with proxy) Steps to Reproduce: 1. Setup a proxy that blocks api.openshift.com or a custom Cincinnati endpoint configured on the cluster 2. Add api.openshift.com or the custom Cincinnati endpoint to the noProxy setting Actual results: Cannot retrieve available updates due to proxy blocking Expected results: Updates retrieved by directly connecting without proxy. Additional info: The analysis I made was: - If we look at the Cincinnati client implementation, we see that it only contains a `proxyURL`[1] field that is used to set a fixed proxy[2] for the transport of every request. This should be enough just to demonstrate the bug, but, for completeness, I'll illustrate how the https proxy URL is gathered. - The HTTPS proxy URL is provided to the client at this line[3] of `calculateAvailableUpdatesStatus` method, invoked durinc the synchronization of available updates[4]. - Those options, together with TLS options (basically, allowed CAs) are obtained at via `getTransportOpts` invocation[5], which in turn invokes `getHTTPSProxyURL()` [6] - That method gets the HTTPS proxy URL from the spec of the proxy object[7] [1] - https://github.com/openshift/cluster-version-operator/blob/ea6899ae6f7ce47432e67f291ae8cfae3e01ee2b/pkg/cincinnati/cincinnati.go#L30 [2] - https://github.com/openshift/cluster-version-operator/blob/ea6899ae6f7ce47432e67f291ae8cfae3e01ee2b/pkg/cincinnati/cincinnati.go#L87 [3] - https://github.com/openshift/cluster-version-operator/blob/ea6899ae6f/pkg/cvo/availableupdates.go#L202 [4] - https://github.com/openshift/cluster-version-operator/blob/ea6899ae6f/pkg/cvo/availableupdates.go#L51 [5] - https://github.com/openshift/cluster-version-operator/blob/ea6899ae6f/pkg/cvo/availableupdates.go#L46 [6] - https://github.com/openshift/cluster-version-operator/blob/ea6899ae6f/pkg/cvo/cvo.go#L794 [7] - https://github.com/openshift/cluster-version-operator/blob/ea6899ae6f/pkg/cvo/egress.go#L14
One important comment: As per the analysis I made, I see that the HTTPS proxy is read from spec and not status as per[7] It is very important to read the noProxy **from the status**, not from the spec, because the cluster network operator adds additional **required** items to it. It is possible that even the HTTPS proxy should have been read from status and not from spec, so maybe the code change on this bug could fix that as well (in case of doubt, cluster network operator team may be able to confirm). [7] - https://github.com/openshift/cluster-version-operator/blob/ea6899ae6f/pkg/cvo/egress.go#L14
Regarding the impact of this bug, it can impact in 2 scenarios: - api.openshift.com is in noProxy and is not reachable through proxy (not common but possible) - A custom Cincinnati endpoint is in use and it is an internal endpoint reachable only without using proxy. This has great potential to impact OpenShift Update Service users, if it hasn't done it already.
> It is very important to read the noProxy **from the status**, not from the spec... Make sense, even for the HTTPS-proxy case. I've spun this out into bug 1978774.
And rolling back the initial version to 4.2, since that's when we got proxy handling in the first place [1], and we've never looked at noProxy so far. [1]: https://github.com/openshift/cluster-version-operator/pull/219
Verified on 4.9.0-0.nightly-2021-07-12-143404. 1. Trigger a connected cluster installation with following proxy setting in install-config.yaml ``` proxy: httpProxy: http://proxy-user1:JYgU8qRZV4DY4PXJbxJK@ec2-13-59-133-217.us-east-2.compute.amazonaws.com:3128 httpsProxy: http://proxy-user1:JYgU8qRZV4DY4PXJbxJK@ec2-13-59-133-217.us-east-2.compute.amazonaws.com:3128 noProxy: test.no-proxy.com,api.openshift.com ``` 2. Add api.openshift.com to blocklist in proxy conf 3. CVO can access api.openshift.com with Noproxy setting(expected) # ./oc adm upgrade Cluster version is 4.9.0-0.nightly-2021-07-12-143404 Upstream is unset, so the cluster will use an appropriate default. Channel: stable-4.8 warning: Cannot display available updates: Reason: VersionNotFound Message: Unable to retrieve available updates: currently reconciling cluster version 4.9.0-0.nightly-2021-07-12-143404 not found in the "stable-4.8" channel # ./oc get proxy cluster -oyaml|grep api.openshift.com noProxy: test.no-proxy.com,api.openshift.com noProxy: .cluster.local,.svc,.us-east-2.compute.internal,10.0.0.0/16,10.128.0.0/14,127.0.0.1,169.254.169.254,172.30.0.0/16,api-int.jliu-test.qe.devcluster.openshift.com,api.openshift.com,localhost,test.no-proxy.com
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:3759