Bug 1978749 - CVO doesn't honor noProxy while contacting Cincinnati endpoint
Summary: CVO doesn't honor noProxy while contacting Cincinnati endpoint
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Cluster Version Operator
Version: 4.2.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.9.0
Assignee: W. Trevor King
QA Contact: liujia
URL:
Whiteboard:
Depends On:
Blocks: 1982683
TreeView+ depends on / blocked
 
Reported: 2021-07-02 16:01 UTC by Pablo Alonso Rodriguez
Modified: 2021-10-18 17:38 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: The cluster-version operator did not respect the Proxy config resource's noProxy property. Consequence: The cluster-version operator would attempt to connect to the upstream update service and signature stores using a configured httpsProxy, even if the requested domain was included in noProxy. In situations where only unproxied connections completed, this denied the cluster-version operator access to update recommendations or release signatures. Fix: The cluster-version operator now respects the noProxy configuration. Result: The cluster-version operator reaches the upstream update service and signature stores directly, when the Proxy resource requests direct, unproxied access.
Clone Of:
: 1982683 (view as bug list)
Environment:
Last Closed: 2021-10-18 17:38:01 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-version-operator pull 622 0 None closed Bug 1978749: pkg/cvo: Respect noProxy 2021-07-08 15:15:01 UTC
Red Hat Product Errata RHSA-2021:3759 0 None None None 2021-10-18 17:38:22 UTC

Description Pablo Alonso Rodriguez 2021-07-02 16:01:10 UTC
Description of problem:

As per source code analysis and also test on customer environment, it seems that the Cincinnati client of the Cluster Version Operator only takes care of HTTPS proxy setting and TLS settings, but not of the NoProxy 

Version-Release:

4.7

How reproducible:

Always (with proxy)

Steps to Reproduce:
1. Setup a proxy that blocks api.openshift.com or a custom Cincinnati endpoint configured on the cluster
2. Add api.openshift.com or the custom Cincinnati endpoint to the noProxy setting


Actual results:

Cannot retrieve available updates due to proxy blocking

Expected results:

Updates retrieved by directly connecting without proxy.

Additional info:

The analysis I made was:
- If we look at the Cincinnati client implementation, we see that it only contains a `proxyURL`[1] field that is used to set a fixed proxy[2] for the transport of every request. This should be enough just to demonstrate the bug, but, for completeness, I'll illustrate how the https proxy URL is gathered.
- The HTTPS proxy URL is provided to the client at this line[3] of `calculateAvailableUpdatesStatus` method, invoked durinc the synchronization of available updates[4].
- Those options, together with TLS options (basically, allowed CAs) are obtained at via `getTransportOpts` invocation[5], which in turn invokes `getHTTPSProxyURL()` [6] 
- That method gets the HTTPS proxy URL from the spec of the proxy object[7]

[1] - https://github.com/openshift/cluster-version-operator/blob/ea6899ae6f7ce47432e67f291ae8cfae3e01ee2b/pkg/cincinnati/cincinnati.go#L30 
[2] - https://github.com/openshift/cluster-version-operator/blob/ea6899ae6f7ce47432e67f291ae8cfae3e01ee2b/pkg/cincinnati/cincinnati.go#L87
[3] - https://github.com/openshift/cluster-version-operator/blob/ea6899ae6f/pkg/cvo/availableupdates.go#L202
[4] - https://github.com/openshift/cluster-version-operator/blob/ea6899ae6f/pkg/cvo/availableupdates.go#L51
[5] - https://github.com/openshift/cluster-version-operator/blob/ea6899ae6f/pkg/cvo/availableupdates.go#L46
[6] - https://github.com/openshift/cluster-version-operator/blob/ea6899ae6f/pkg/cvo/cvo.go#L794
[7] - https://github.com/openshift/cluster-version-operator/blob/ea6899ae6f/pkg/cvo/egress.go#L14

Comment 1 Pablo Alonso Rodriguez 2021-07-02 16:08:11 UTC
One important comment: As per the analysis I made, I see that the HTTPS proxy is read from spec and not status as per[7]

It is very important to read the noProxy **from the status**, not from the spec, because the cluster network operator adds additional **required** items to it.

It is possible that even the HTTPS proxy should have been read from status and not from spec, so maybe the code change on this bug could fix that as well (in case of doubt, cluster network operator team may be able to confirm).

[7] - https://github.com/openshift/cluster-version-operator/blob/ea6899ae6f/pkg/cvo/egress.go#L14

Comment 2 Pablo Alonso Rodriguez 2021-07-02 16:12:35 UTC
Regarding the impact of this bug, it can impact in 2 scenarios:
- api.openshift.com is in noProxy and is not reachable through proxy (not common but possible)
- A custom Cincinnati endpoint is in use and it is an internal endpoint reachable only without using proxy. This has great potential to impact OpenShift Update Service users, if it hasn't done it already.

Comment 3 W. Trevor King 2021-07-02 17:16:59 UTC
> It is very important to read the noProxy **from the status**, not from the spec...

Make sense, even for the HTTPS-proxy case.  I've spun this out into bug 1978774.

Comment 4 W. Trevor King 2021-07-02 17:25:05 UTC
And rolling back the initial version to 4.2, since that's when we got proxy handling in the first place [1], and we've never looked at noProxy so far.

[1]: https://github.com/openshift/cluster-version-operator/pull/219

Comment 7 liujia 2021-07-13 02:03:05 UTC
Verified on 4.9.0-0.nightly-2021-07-12-143404.

1. Trigger a connected cluster installation with following proxy setting in install-config.yaml
```
  proxy:
    httpProxy: http://proxy-user1:JYgU8qRZV4DY4PXJbxJK@ec2-13-59-133-217.us-east-2.compute.amazonaws.com:3128
    httpsProxy: http://proxy-user1:JYgU8qRZV4DY4PXJbxJK@ec2-13-59-133-217.us-east-2.compute.amazonaws.com:3128
    noProxy: test.no-proxy.com,api.openshift.com
```
2. Add api.openshift.com to blocklist in proxy conf
3. CVO can access api.openshift.com with Noproxy setting(expected)

# ./oc adm upgrade
Cluster version is 4.9.0-0.nightly-2021-07-12-143404

Upstream is unset, so the cluster will use an appropriate default.
Channel: stable-4.8
warning: Cannot display available updates:
  Reason: VersionNotFound
  Message: Unable to retrieve available updates: currently reconciling cluster version 4.9.0-0.nightly-2021-07-12-143404 not found in the "stable-4.8" channel

# ./oc get proxy cluster -oyaml|grep api.openshift.com
  noProxy: test.no-proxy.com,api.openshift.com
  noProxy: .cluster.local,.svc,.us-east-2.compute.internal,10.0.0.0/16,10.128.0.0/14,127.0.0.1,169.254.169.254,172.30.0.0/16,api-int.jliu-test.qe.devcluster.openshift.com,api.openshift.com,localhost,test.no-proxy.com

Comment 11 errata-xmlrpc 2021-10-18 17:38:01 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:3759


Note You need to log in before you can comment on or make changes to this bug.