Bug 1895053 - Allow builds to optionally mount in cluster trust stores
Summary: Allow builds to optionally mount in cluster trust stores
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Build
Version: 4.7
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 4.8.0
Assignee: Adam Kaplan
QA Contact: wewang
Srivaralakshmi Ramani
: 1914001 (view as bug list)
Depends On: 1913325
TreeView+ depends on / blocked
Reported: 2020-11-05 16:25 UTC by Adam Kaplan
Modified: 2024-06-13 23:21 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: Enhancement
Doc Text:
[discrete] [id="ocp-4-8-builds-mount-custom-pki-ca"] ==== Mount custom PKI certificate authorities Previously, builds could not use the cluster PKI certificate authorities that were sometimes required to access corporate artifact repositories. Now, you can configure the `BuildConfig` object to mount cluster custom PKI certificate authorities by setting `mountTrustedCA` to `true`.
Clone Of:
Last Closed: 2021-07-27 22:34:10 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Github openshift api pull 831 0 None open Bug 1895053: Allow builds to mount the proxy CA 2021-02-18 01:36:45 UTC
Github openshift builder pull 218 0 None open WIP - Bug 1895053: Mount CA trust store in builds 2021-02-18 01:36:46 UTC
Github openshift openshift-apiserver pull 185 0 None open WIP - Bug 1895053: bump(github.com/openshift/api) 2021-02-18 01:36:46 UTC
Github openshift openshift-controller-manager pull 154 0 None open WIP - Bug 1895053: Generate mounts.conf for every build 2021-02-18 01:36:46 UTC
Github openshift origin pull 25778 0 None open WIP - Bug 1895053: Verify builds can mount proxy trustedCA 2021-02-18 01:36:46 UTC
Red Hat Product Errata RHSA-2021:2438 0 None None None 2021-07-27 22:34:30 UTC

Description Adam Kaplan 2020-11-05 16:25:30 UTC
Description of problem:

In Bug #1826183, builds were updated to mount the contents of /etc/pki/ca-trust into the build from the build container. The build container, in turn, merged the admin-provided "extra" trust bundle with the default RHEL trust store.

This proved problematic because Docker strategy builds could not modify content or directory permissions in `/etc/pki/ca-trust` and have those changes persist into the resulting image. In more than one case this broke the build of OpenShift components. As a result, the mounting of `/etc/pki/ca-trust` is reverted via Bug #1891759, which will be backported to 4.6.z.

Builds need to allow developers to opt into mounting the cluster wide trust bundle.

Version-Release number of selected component (if applicable): 4.7.0

How reproducible: Always

Steps to Reproduce:
1. Run a build which downloads content through an HTTPS, man-in-the-middle proxy
2. Observe result.

Actual results:

With the changes in Bug #1891759, build should fail because HTTPS content is not trusted.

Expected results:

Builds can succeed if the cluster-wide trust bundle is mounted in via an option in the BuildConfig.

Additional info:


Comment 8 Rolfe Dlugy-Hegwer 2021-02-22 15:37:53 UTC
*** Bug 1914001 has been marked as a duplicate of this bug. ***

Comment 13 aygarg 2021-04-07 08:45:10 UTC
Hello Team,

Is there any workaround currently for this? Currently, the customer can not proceed with the project because the build fails.

Ayush Garg

Comment 16 jtejal 2021-05-05 02:46:28 UTC
Hello Team

Another FSI customer is experiencing this which is delaying their adoption/movement to OCP 4.5.
Appreciate if anyone can share known workarounds.

Many thanks

Comment 17 aygarg 2021-05-05 02:49:43 UTC
(In reply to aygarg from comment #13)
> Hello Team,
> Is there any workaround currently for this? Currently, the customer can not
> proceed with the project because the build fails.
> Regards,
> Ayush Garg

Can someone please share any possible workarounds?

Comment 18 Adam Kaplan 2021-05-10 21:10:36 UTC
To work around this issue, users need to do the following:

1. Create an emtpy ConfigMap in the build's namespace with the label `config.openshift.io/inject-trusted-cabundle="true"` [1]. For this example I'll name it "trusted-cabundle", though it can be named anything as long as it is a valid ConfigMap name:

kind: ConfigMap
apiVersion: v1
    config.openshift.io/inject-trusted-cabundle: "true"
  name: trusted-cabundle
data: {}

2. Add the ConfigMap as a Docker strategy build input [2], and have a Dockerfile copy the `ca-bundle.crt` file to /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem`.
3. In the BuildConfig's Docker strategy options, ensure `SkipLayers` is set for the image optimization policy. This ensures that the layers of the build are squashed and the cluster's trust bundle doesn't leak into the output container image [3].

Below is an example of how to effectively run a Ruby-based s2i build via the Docker strategy, using an in-line Dockerfile.

kind: BuildConfig
apiVerion: build.openshift.io/v1
      ... # info to git source
    - configMap:
        name: trusted-cabundle
      destinationDir: trusted-cabundle # this is relative to the build root directory
    dockerfile: |
      FROM ruby:latest
      COPY . .
      USER root
      # Back up the image trust bundle and override it with the cluster provided bundle
      RUN cp /etc/pki/ca-trust-extracted/pem/tls-ca-bundle.pem /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem.bak && \
          cp -f trusted-cabundle/ca-bundle.crt /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem && \
          chown root:root /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
      # Change back to the default s2i image user to run the assemble script
      USER 1001
      RUN /usr/libexec/s2i/assemble
      # Change back to the root user to restore the trust bundle
      USER root
      RUN mv -f /etc/pki/ca-trust-extracted/pem/tls-ca-bundle.pem.bak /etc/pki/ca-trust-extracted/pem/tls-ca-bundle
      # Make run the default command as user 1001
      USER 1001
      CMD /usr/libexec/s2i/run
        kind: ImageStreamTag
        name: ruby:latest
        namespace: openshift
      # SkipLayers ensures that the cluster trust bundle doesn't leak into the resulting container image
      imageOptimizationPolicy: SkipLayers

[1] https://docs.openshift.com/container-platform/4.7/networking/configuring-a-custom-pki.html#certificate-injection-using-operators_configuring-a-custom-pki
[2] https://docs.openshift.com/container-platform/4.7/cicd/builds/creating-build-inputs.html#builds-adding-input-secrets-configmaps_creating-build-inputs
[3] https://docs.openshift.com/container-platform/4.7/cicd/builds/build-strategies.html#builds-strategy-docker-squash-layers_build-strategies

Comment 22 Srivaralakshmi Ramani 2021-06-09 10:14:01 UTC
Thanks Rolfe. Perfect. I have modified the RN text. It looks so much better now. Appreciate the help and suggestion.

Comment 23 Srivaralakshmi Ramani 2021-06-09 10:15:28 UTC
Adam and Wewang, please review and approve the updated Doc Text field, above. Thanks.

Comment 24 wewang 2021-06-09 11:06:20 UTC
Approved doc text @Srivaralakshmi, need i change something in the bug?

Comment 25 Adam Kaplan 2021-06-09 14:30:07 UTC
Doc text LGTM

Comment 28 errata-xmlrpc 2021-07-27 22:34:10 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.