Bug 1771564

Summary: proxy config trustedCA is not populated on install
Product: OpenShift Container Platform Reporter: Ben Parees <bparees>
Component: InstallerAssignee: W. Trevor King <wking>
Installer sub component: openshift-installer QA Contact: Gaoyun Pei <gpei>
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: unspecified CC: adahiya, agarcial, akaris, gparente, jcromer, rbartal, sdodson, wking, xtian
Version: 4.3.0   
Target Milestone: ---   
Target Release: 4.4.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-05-13 21:52:34 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Ben Parees 2019-11-12 15:40:17 UTC
Description of problem:
If a user supplies additionalTrustedCAs in the install-config, but does not supply any other proxy configuration (proxy hostname, no_proxy domains), the installer copies the supplied CAs into a user-ca-bundle CM in the openshift-config namespace, but it does not link that CM into the proxy config resource via the "proxy.spec.trustedCA" field.

I would expect that it should.  We had a user report that they performed the above steps in which they had no additional proxy configuration because they have a transparent proxy in their network (but it's a MITM proxy with its own cert, hence the need for additional CAs).

Ultimately they had to manually edit the proxy config after install to link to the user-ca-bundle to get things working.  This should have happened automatically during install.


Actual results:
proxy config does not link to the additional CA bundle if no other proxy configuration content is provided at install time.

Expected results:
proxy config should always link to the additional CA bundle if one was supplied at install time.

Comment 1 Abhinav Dahiya 2019-11-12 16:44:44 UTC
> If a user supplies additionalTrustedCAs in the install-config, but does not supply any other proxy configuration (proxy hostname, no_proxy domains), the installer copies the supplied CAs into a user-ca-bundle CM in the openshift-config namespace, but it does not link that CM into the proxy config resource via the "proxy.spec.trustedCA" field.


that is expected behavior. By setting no proxy, the user is saying "nobody needs to care about proxy"

> We had a user report that they performed the above steps in which they had no additional proxy configuration because they have a transparent proxy in their network (but it's a MITM proxy with its own cert, hence the need for additional CAs).

that's more to the effect that all components should trust additional CAs because of various reasons one of them is MiTM transparent proxy. And I don't think we should be setting proxies.config.openshift.io trust here.

This is an RFE and not a bug.

Comment 2 Scott Dodson 2019-11-13 18:07:28 UTC
Mirroring a comment from the PR that's been opened, I don't think we should do this, to be discussed during Thursday's group-g arch call.

Comment 3 Scott Dodson 2019-11-15 15:05:43 UTC
https://github.com/openshift/enhancements/pull/115 aims to describe the path forward, deferring this to 4.4

Comment 5 Ben Parees 2020-01-31 19:40:20 UTC
At this point https://github.com/openshift/enhancements/pull/115 is not going to move forward for the reasons described in:
https://github.com/openshift/enhancements/pull/115#issuecomment-580878147

I think the only thing left to do on this bug is to ensure that the installer/api documentation of the install config's "AdditionalTrustedCAs" field is very clear about when those CAs will and will not be passed into the proxy config for the cluster.

Comment 6 W. Trevor King 2020-01-31 20:10:41 UTC
If AdditionalTrustedCAs is just for proxies after all, don't we actually want the code change from [1]?  With improved docs around the property too to make the scoping more clear.  Also commented to this effect in [2].  Not sure if it's best to hash this out here or there.

[1]: https://github.com/openshift/installer/pull/2658
[2]: https://github.com/openshift/enhancements/pull/115#issuecomment-580891461

Comment 7 Ben Parees 2020-01-31 20:21:44 UTC
> If AdditionalTrustedCAs is just for proxies after all

i'm not aware that that's the case (but i also do not know what else it is used for, when no proxy config is supplied to the installer).

I think we made a mistake not defining an explicit "proxyCAs" field on the installerconfig, and instead using AdditionalTrustedCAs for seemingly two purposes (again not sure specifically what the second purpose is, but i'm pretty sure there is one, which is why https://github.com/openshift/installer/pull/2658 got pushback:  there are cases where customers supply additional CAs but don't want them distributed to all components as the proxyCA)

perhaps it's for things like talking to a self-hosted disconnected mirror registry?

Comment 8 W. Trevor King 2020-01-31 22:31:36 UTC
Talked this over a bit more with Ben, and I've spun out:

* installer#3039 to document the node and proxy additionalTrustBundle consumers.
* api#582 (via bug 1797107) to document the proxy trustedCA flow more explicitly.

Maybe still to do is explicitly testing just setting noProxy to get the proxy CA injection for the transparent-network-proxy case [1].

[1]: https://github.com/openshift/enhancements/pull/115#issuecomment-580894852

Comment 9 W. Trevor King 2020-01-31 22:35:30 UTC
ah.  Just setting noProxy will break the CVO, which only included proxy trust when httpsProxy is set [1].  We can fix *that* narrowly by pulling trust from the trusted-ca-bundle ConfigMap (avoiding trustedCA, as recommended by the API docs [2]).  I'll see if there's an existing CVO bug or file a new one...

[1]: https://github.com/openshift/cluster-version-operator/blob/b9a3ffafdc32e534c68112a13b89cf1459e1bbd9/pkg/cvo/availableupdates.go#L214-L219
[2]: https://github.com/openshift/api/blob/ffde1bfabb9fe944ab73f2aca6368f083fecc07d/config/v1/types_proxy.go#L46-L47

Comment 10 W. Trevor King 2020-01-31 23:04:04 UTC
Not a CVO bug, but some related discussion on MCO handling in [1] (and the rest of the bug, but I tried to summarize my current understanding in that comment).

[1]: https://bugzilla.redhat.com/show_bug.cgi?id=1784201#c42

Comment 11 W. Trevor King 2020-02-01 00:00:28 UTC
I've filed bug 1797123 with an associated PR to move the CVO over to using trusted-ca-bundle.

Comment 12 W. Trevor King 2020-02-07 23:33:08 UTC
Moving back to MODIFIED so the Errata sweeper will pick this up on the nightly build that includes installer#3039 (although as a change to the installer docs, verification won't actually depend on a nightly release image).

Comment 14 Gaoyun Pei 2020-02-18 08:50:36 UTC
The doc changes in https://github.com/openshift/installer/pull/3039/files LGTM

Comment 15 W. Trevor King 2020-02-25 16:06:48 UTC
*** Bug 1807103 has been marked as a duplicate of this bug. ***

Comment 17 errata-xmlrpc 2020-05-13 21:52:34 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581