Bug 1389761

(In reply to Steven Walter from comment #0)
> Document URL: 
> https://docs.openshift.com/container-platform/3.3/install_config/
> http_proxies.html
> 
> Section Number and Name: 
> New section?
> 
<snip>
> Suggestions for improvement: 
> There likely is no all-in-one, easy fix for this. Images based on different
> operating systems are not going to use the same location for trust stores.
> Yet example steps for common situations would be useful; for RHEL-based
> images, for CentOS-based images, for s2i-built images, where do I mount the
> CA? Is it possible to modify the base images so it only has to be done once,
> or does it require modifying the dc for every app you deploy? Can it be done
> via template? What are our recommended steps?
> 

Hi Steven,

This is quite an interesting issue. I would like to address this in the docs but I fear we may not be able to get answers to these questions that you have raised.

Do you have any material we could reference? Or do we just try and work with engineering to get this done?

Unfortunately I do not have a great environment to test this type of firewall configuration so there's been some difficulty in verifying exact steps.

One possible option is to simply mount the ca in a secret and give that secret to all of the serviceaccounts, then mount the secret into the bc:


$ oc secrets new mysecret ca.crt=path/to/certificate/authority/ca.crt

$ oc secrets add serviceaccount/default secrets/mysecret
$ oc secrets add serviceaccount/builder secrets/mysecret
$ oc secrets add serviceaccount/deployer secrets/mysecret


Add the secret to the source section in the existing build configuration.

$ oc edit bc <my app's bc>

source:
  git:
    uri: https://someurl.com/somepath.git
  secrets:
    - secret:
        name: mysecret
  type: Git


I have neither verified that this works nor think it's altogether ideal, as it would require at least some work on every single bc. We already use a similar solution in the docs for the git pull step; the actual build works fine using this [1] but this does not cover instances such as your app wanting to communicate with an api somewhere on the web.

There are some ideas offered up in the sme mailing list [2]; however no official solution is reached. Some options include, "modify ose-sti-builder and ose-docker-builder or other base images, telling them to trust a certain CA by adding that file to the truststore; then all images that build using these will trust the CA" but there are no trustworthy steps nor is this really that ideal to offer to customers (although if we had reliable steps it might be a more comfortable ask).

Based on the email thread [2], though, there is no real easy answer at the moment, at least none that I've found!


[1] https://docs.openshift.com/container-platform/3.3/dev_guide/builds.html#using-a-proxy-for-git-cloning
[2] http://post-office.corp.redhat.com/archives/openshift-sme/2016-September/msg00716.html

Took a while but I started a KCS a few weeks ago to start addressing this issue. This will be the first in a couple of KCS related to the issue -- this one covers "How do I add a secret to every new project by default", allowing a pull secret to become default. https://access.redhat.com/solutions/2958391

@Steven - how can we proceed with this bug?

Is it possible to break it down into smaller tasks to achieve the overall aim? Is it possible to fix this via documentation alone?

Vikram,

I see this effort as fixable in 3 ways:

- customer can add their CA to the ca-bundle.crt (I still have not tested this...)
- customer can use project template to include CA secret in every project, and the customer's developers can manually add the secret to their applications and configure it for use (https://access.redhat.com/solutions/2958391)
- When podpresets becomes non-tech preview, this will likely be the preferred solution https://docs.openshift.com/container-platform/3.6/dev_guide/pod_preset.html

For the last point, we could add an example to the OCP documentation that covers this use case; or we could use KCS to do the same. I think either could satisfy the need; what are your thoughts? If we'd prefer to just do KCS, let's close this and I'll update the aforementioned KCS with podpreset information once it leaves tech preview. Otherwise we can leave this open, get a solution in the docs using podpresets, and call it good.

**Note** given that every container image is different, we can rely on podpresets to get a secret into a pod, but it will always be the responsibility of the developer to make their application use the mounted CA as a trust, for any applications with unique truststores (apache, curl, etc)

(In reply to Steven Walter from comment #14)
> Vikram,
> 
> I see this effort as fixable in 3 ways:
> 
> - customer can add their CA to the ca-bundle.crt (I still have not tested
> this...)
> - customer can use project template to include CA secret in every project,
> and the customer's developers can manually add the secret to their
> applications and configure it for use
> (https://access.redhat.com/solutions/2958391)
> - When podpresets becomes non-tech preview, this will likely be the
> preferred solution
> https://docs.openshift.com/container-platform/3.6/dev_guide/pod_preset.html
> 
> For the last point, we could add an example to the OCP documentation that
> covers this use case; or we could use KCS to do the same. I think either
> could satisfy the need; what are your thoughts? If we'd prefer to just do
> KCS, let's close this and I'll update the aforementioned KCS with podpreset
> information once it leaves tech preview. Otherwise we can leave this open,
> get a solution in the docs using podpresets, and call it good.

That will solve the problem for 3.7 onwards (when it is likely to be out of tech preview) though, right? That will still leave 3.6 and below with this issue? For 3.6 and below would option 2 work?

> 
> **Note** given that every container image is different, we can rely on
> podpresets to get a secret into a pod, but it will always be the
> responsibility of the developer to make their application use the mounted CA
> as a trust, for any applications with unique truststores (apache, curl, etc)

If it works, then Steven's first suggestion is far and away the best option:

"""
- customer can add their CA to the ca-bundle.crt (I still have not tested this...)
"""


The reason this is much better is that everything will just work straight out of the box:
* There will be no random extra secrets for Customers to delete and break things
* They won't need to make changes to their BCs for builds to work
* They won't need to make changes to their Pods for SSL to JustWork


As an Operator we've had multiple internal Customers complaining that "Access to Service X doesn't work" (which then leads to pulling out of them what error they're seeing, and that it's just the missing trust, again).  This gets combined with things like "But I don't need a secret to pull from that git repo" and having to explain that the 'secret' is just how you configure the CA trust.

The way that Source to Image is being advocated for means that as an Operator I either have to rebuild all the S2I images, or tell people over and over how to import the CA certs and trust them (because really almost no-one looks for the documentation before raising a ticket).

Option 2, still leaves Customers with things to do: They'll still need to make sure that Git/Whatever uses the config

Option 3 is 'OK' because we can do things like overwriting the default certs, but with the certificate 'secret' in *EVERY* project, it's a pain to update especially because custom controllers are very poorly documented.

(In reply to Mark Chappell from comment #16)
> If it works, then Steven's first suggestion is far and away the best option:
> 
> """
> - customer can add their CA to the ca-bundle.crt (I still have not tested
> this...)
> """
> 
> 
> The reason this is much better is that everything will just work straight
> out of the box:
> * There will be no random extra secrets for Customers to delete and break
> things
> * They won't need to make changes to their BCs for builds to work
> * They won't need to make changes to their Pods for SSL to JustWork

Ok, so to get it into the docs as a viable solution, we just need someone in GSS/QE/Eng to confirm that it works?


> 
> 
> As an Operator we've had multiple internal Customers complaining that
> "Access to Service X doesn't work" (which then leads to pulling out of them
> what error they're seeing, and that it's just the missing trust, again). 
> This gets combined with things like "But I don't need a secret to pull from
> that git repo" and having to explain that the 'secret' is just how you
> configure the CA trust.
> 
> The way that Source to Image is being advocated for means that as an
> Operator I either have to rebuild all the S2I images, or tell people over
> and over how to import the CA certs and trust them (because really almost
> no-one looks for the documentation before raising a ticket).
> 
> Option 2, still leaves Customers with things to do: They'll still need to
> make sure that Git/Whatever uses the config
> 
> Option 3 is 'OK' because we can do things like overwriting the default
> certs, but with the certificate 'secret' in *EVERY* project, it's a pain to
> update especially because custom controllers are very poorly documented.

Yeah that should be right. I still have not gotten to test this as I do not have a great reproducer  (I neither have this sort of firewall set up nor do I have a CA to test with, although the latter should not be hard)

If I can manage a way to get a firewall up with this sort of functionality testing should be easy; just cat cert >> ca-bundle, if I understand correctly

I have been unable to get the ca-bundle.crt method to work. I tried appending the CA to the end of ca-bundle.crt, rebooting master (single master cluster); reboot all nodes; oc new-project; kick off build. For curiosity sake I did the same appending it to /etc/origin/master/ca.crt and /etc/origin/node/ca.crt

Mark, thoughts?

I feared that might be the case.  Only other way I can think of would be to store the full set of CAs as a configmap/secret and mount them over the top of the standard Red Hat location for certs, but this would only work for Red Hat based images.  This would probably tie us back to the PodPreset pieces.

OCP 3.6-3.10 is no longer on full support [1]. Marking un-triaged bugs CLOSED DEFERRED. If you have a customer case with a support exception or have reproduced on 3.11+, please reopen and include those details. When reopening, please set the Version to the appropriate version where reproduced.

[1]: https://access.redhat.com/support/policy/updates/openshift