Bug 1981633 - enhance service-ca injection
Summary: enhance service-ca injection
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: service-ca
Version: 4.9
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 4.8.0
Assignee: David Eads
QA Contact: liyao
Depends On: 1981498
Blocks: 1981634
TreeView+ depends on / blocked
Reported: 2021-07-13 03:16 UTC by OpenShift BugZilla Robot
Modified: 2021-07-27 23:14 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2021-07-27 23:13:47 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Github openshift api pull 967 0 None closed [release-4.8] bug 1981633: add field to make service-ca.crt more secure 2021-07-14 12:51:28 UTC
Github openshift cluster-kube-controller-manager-operator pull 549 0 None open [release-4.8] bug 1981633: allow upgraded clusters to use the vulnerable service-ca.crt 2021-07-14 12:51:31 UTC
Github openshift kubernetes pull 856 0 None open [release-4.8] bug 1981633: UPSTREAM: <carry>: add a way to inject a vulnerable, legacy service-c… 2021-07-14 12:51:33 UTC
Github openshift kubernetes pull 858 0 None open [release-4.8] bug 1981633: UPSTREAM: <carry>: prevent the kubecontrollermanager service-ca from getting less secure 2021-07-14 15:26:31 UTC
Github openshift service-ca-operator pull 168 0 None closed [release-4.8] bug 1981633: add vulnerable legacy injector to allow for upgrade clusters to use ... 2021-07-14 12:51:36 UTC
Red Hat Product Errata RHSA-2021:2438 0 None None None 2021-07-27 23:14:11 UTC

Description OpenShift BugZilla Robot 2021-07-13 03:16:24 UTC
+++ This bug was initially created as a clone of Bug #1981498 +++

Comment 2 liyao 2021-07-15 12:27:37 UTC
Use the cluster-bot to launch the env with the still open but Dev-approved PR(s) to do the pre-merge verification.
Test in fresh 4.8 env
1. check openshift-service-ca.crt
$ oc new-project testproj
$ oc get cm openshift-service-ca.crt -o yaml 
# annotation service.beta.openshift.io/inject-cabundle: "true" is seen, and only one cert in service-ca.crt field, this is secure as expected

2. check other configmap
$ oc create cm testconfigmap --from-literal=key=value
$ oc annotate cm testconfigmap service.alpha.openshift.io/inject-vulnerable-legacy-cabundle=true
Check `oc get cm testconfigmap -o yaml`, no service-ca.crt field is seen, this means service.alpha.openshift.io/inject-vulnerable-legacy-cabundle=true only takes effect for configmap named "openshift-service-ca.crt"

Test upgrade from 4.7 to 4.8
1. check openshift-service-ca.crt after upgrade to 4.8
$ oc adm upgrade --to-image=registry.build01.ci.openshift.org/ci-ln-mqnmdj2/release:latest --force=true --allow-explicit-upgrade=true
$ oc get clusterversion
NAME      VERSION                                                  AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.0-0.ci.test-2021-07-15-073410-ci-ln-mqnmdj2-latest   True        False         26m     Cluster version is 4.8.0-0.ci.test-2021-07-15-073410-ci-ln-mqnmdj2-latest
$ oc get cm openshift-service-ca.crt -o yaml
# annotation service.alpha.openshift.io/inject-vulnerable-legacy-cabundle is seen, and multiple certs in service-ca.crt field, which is expected with 4.7 behavior

2. check openshift-service-ca.crt after change useMoreSecureServiceCA to true
$ oc edit kubecontrollermanager cluster
useMoreSecureServiceCA: true
$ oc get cm openshift-service-ca.crt -o yaml
# annotation service.beta.openshift.io/inject-cabundle: "true" is seen, and only one cert in service-ca.crt field, switch to secure mode as expected

3. check useMoreSecureServiceCA cannot be changed back to false
$ oc edit kubecontrollermanager cluster
useMoreSecureServiceCA: false
# it's forbidden to change back to false as expected

Comment 3 liyao 2021-07-16 02:26:23 UTC
add test about check env OPENSHIFT_USE_VULNERABLE_LEGACY_SERVICE_CA_CRT existed in KCM pod after upgrade to 4.8

useMoreSecureServiceCA is updated to true in https://bugzilla.redhat.com/show_bug.cgi?id=1981633#c2 causing KCM pods restarted, so the test is to check the env from the last KCM pod yaml before restarted
$ oc get po -n openshift-kube-controller-manager -L revision -l revision
NAME                                                                 READY   STATUS    RESTARTS   AGE    REVISION
kube-controller-manager-ip-10-0-153-18.us-east-2.compute.internal    4/4     Running   0          103m   12
kube-controller-manager-ip-10-0-177-216.us-east-2.compute.internal   4/4     Running   0          102m   12
kube-controller-manager-ip-10-0-215-248.us-east-2.compute.internal   4/4     Running   0          103m   12
$ oc debug no/ip-10-0-153-18.us-east-2.compute.internal
sh-4.4# chroot /host
sh-4.4# bash
[root@ip-10-0-153-18 /]# cd /etc/kubernetes/static-pod-resources/
[root@ip-10-0-153-18 static-pod-resources]# ls -d kube-controller-manager-pod-*
kube-controller-manager-pod-10  kube-controller-manager-pod-3  kube-controller-manager-pod-6  kube-controller-manager-pod-9
kube-controller-manager-pod-11  kube-controller-manager-pod-4  kube-controller-manager-pod-7
kube-controller-manager-pod-12  kube-controller-manager-pod-5  kube-controller-manager-pod-8
[root@ip-10-0-153-18 static-pod-resources]# cd kube-controller-manager-pod-11
[root@ip-10-0-153-18 kube-controller-manager-pod-11]# cat kube-controller-manager-pod.yaml | jq '' | grep -i -A 5 vulner
            "value": "true"
        "resources": {
          "requests": {
            "value": "true"
        "resources": {
          "requests": {
            "value": "true"
        "resources": {
          "requests": {
            "value": "true"
 # it can see OPENSHIFT_USE_VULNERABLE_LEGACY_SERVICE_CA_CRT env existed in last KCM pod yaml before restarted

Comment 6 liyao 2021-07-19 02:57:34 UTC
As described in https://bugzilla.redhat.com/show_bug.cgi?id=1981633#c2 and https://bugzilla.redhat.com/show_bug.cgi?id=1981633#c3, the bug had been pre-merge verified. Move to VERIFIED status manually.

Comment 9 errata-xmlrpc 2021-07-27 23:13:47 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.