Bug 1891759 - Dockerfile builds cannot change /etc/pki/ca-trust
Summary: Dockerfile builds cannot change /etc/pki/ca-trust
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Build
Version: 4.6
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 4.7.0
Assignee: Adam Kaplan
QA Contact: wewang
Rolfe Dlugy-Hegwer
Depends On:
Blocks: 1895093
TreeView+ depends on / blocked
Reported: 2020-10-27 09:43 UTC by Oleg Bulatov
Modified: 2021-02-24 15:29 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
* Previously, Dockerfile builds could not change permissions of `/etc/pki/ca-trust` or create files inside it. This issue was caused by fixing link:https://bugzilla.redhat.com/show_bug.cgi?id=1826183[] in version 4.6, which added support for HTTPS proxies with CAs for builds and ALWAYS mounted `/etc/pki/ca-trust`. This prevented builds that included their own CAs or modified the system trust store from working correctly at runtime. The current release fixes this issue by reverting Bug 1826183. Now, builder images that include their own CAs work again. (link:https://bugzilla.redhat.com/show_bug.cgi?id=1891759[*BZ#1891759*])
Clone Of:
Last Closed: 2021-02-24 15:28:35 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Github openshift builder pull 185 0 None closed Bug 1891759: Do not mount /etc/pki/ca-trust in builds 2021-02-06 17:05:23 UTC
Red Hat Product Errata RHSA-2020:5633 0 None None None 2021-02-24 15:29:09 UTC

Description Oleg Bulatov 2020-10-27 09:43:00 UTC
Description of problem:

Dockerfile builds cannot change permissions of /etc/pki/ca-trust or create files inside it.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. create a Dockerfile with RUN chmod a+w -R /etc/pki/ca-trust/extracted
2. build an image
3. inspect /etc/pki/ca-trust/extracted permissions

Actual results:


Expected results:


Additional info:

It affects our ability to build release images:

  * https://github.com/openshift/image-registry/blob/edd1c864a4cd87e496fdce38aa166f22683eb234/Dockerfile.rhel7#L10
  * https://github.com/openshift/cloud-credential-operator/blob/1e5a13b5a796630ef6fa9de5ec974b330b77fd5f/Dockerfile#L11

Comment 1 Adam Kaplan 2020-10-27 12:51:43 UTC
This was a consequence of fixing https://bugzilla.redhat.com/show_bug.cgi?id=1826183. /etc/pki/ca-trust is a mount point for buildah, the contents of which are managed by the build container. When the build container starts, it adds the cluster-wide CA to the set of anchors, and then runs `update-ca-trust extract` as root.

In the case of the image registry, it wants /etc/pki/ca-trust to be writeable so that `update-ca-trust` can be run as non-root.

Comment 2 Ben Parees 2020-11-04 04:45:12 UTC
I think https://bugzilla.redhat.com/show_bug.cgi?id=1826183 needs to be considered for reversion, given the impact on existing build behavior.

It means that no one w/ a builder image that includes their own CAs can continue to function as expected, right?  Because their CAs will be stomped by the always present mount that that fix introduced.

Comment 3 Ben Parees 2020-11-04 14:36:16 UTC
more specifically, here is my suggested course of action:

1) put in a change to 4.7 and 4.6 that does not mount the content (you can leave the rest of the logic in place) 
2) get CI to update to the 4.6.z patch asap 
3) reintroduce the mount based on a buildconfig api opt-in field. in 4.6+4.7.

Comment 5 W. Trevor King 2020-11-05 21:39:22 UTC
Dropping the sort of error string folks might see into the bug, to help with searching:

  Copying system trust bundle
  cp: cannot remove '/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem': Permission denied

Comment 6 Adam Kaplan 2020-11-05 22:47:24 UTC
To facilitate quick backporting to 4.6.z, this BZ will only cover the removal of the buildah mount.

Optionally mounting /etc/pki/ca-trust will be addressed in https://bugzilla.redhat.com/show_bug.cgi?id=1895053.

Comment 7 wewang 2020-11-06 02:25:03 UTC
Verified as follow version:

Using the follow Dockerfile:
FROM registry.svc.ci.openshift.org/ocp/4.7:base
RUN chmod a+w -R /etc/pki/ca-trust/extracted
RUN ls -ld /etc/pki/ca-trust/extracted


STEP 1: FROM registry.svc.ci.openshift.org/ocp/4.7:base
STEP 2: RUN chmod a+w -R /etc/pki/ca-trust/extracted
--> 2293d141c56
STEP 3: RUN ls -ld /etc/pki/ca-trust/extracted
drwxrwxrwx. 1 root root 70 Sep  1 19:39 /etc/pki/ca-trust/extracted

Comment 8 wewang 2020-12-02 10:43:01 UTC
Added the case:OCP-37309 to the bug, but met issue "The external tracker URL you specified does not belong to a tracker known to Red Hat Bugzilla", anyway cases added.

Comment 13 errata-xmlrpc 2021-02-24 15:28:35 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.