Bug 1827336 - Remove stale condition DefaultSecurityContextConstraints_Mutated
Summary: Remove stale condition DefaultSecurityContextConstraints_Mutated
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: kube-apiserver
Version: 4.3.z
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 4.4.z
Assignee: Abu Kashem
QA Contact: Xingxing Xia
URL:
Whiteboard:
Depends On: 1827335
Blocks: 1827337
TreeView+ depends on / blocked
 
Reported: 2020-04-23 16:51 UTC by Abu Kashem
Modified: 2020-05-04 11:50 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1827335
: 1827337 (view as bug list)
Environment:
Last Closed: 2020-05-04 11:50:03 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github openshift cluster-kube-apiserver-operator pull 839 None closed Bug 1827336: Remove stale condition 2020-06-15 18:14:57 UTC
Red Hat Product Errata RHBA-2020:0581 None None None 2020-05-04 11:50:18 UTC

Description Abu Kashem 2020-04-23 16:51:31 UTC
+++ This bug was initially created as a clone of Bug #1827335 +++

Description of problem:
In OpenShift 4.3.14 we have reverted DefaultSecurityContextConstraints_Mutated. We removed the controller that sets Upgradeable to False if any default SCC has been mutated.

But on an affected cluster (pre 4.3.14) that already has user-modified default SCCs the stale condition does not get removed after upgrade.



Version-Release number of selected component (if applicable):
OpenShift 4.3.14

How reproducible:
Always

Steps to Reproduce:
1. install ocp v4.3.13
2. trigger upgradeable=false by mutating default scc
Change the default SCC 
$ oc patch scc privileged --type json -p '[{"op": "add", "path": "/users/-", "value": "e2e-user"}]'
$ oc patch scc anyuid --type json -p '[{"op": "add", "path": "/users/-", "value": "e2e-user"}]'
    
# ./oc get scc privileged -o json|jq .users
[
  "system:admin",
  "system:serviceaccount:openshift-infra:build-controller",
  "e2e-user"
]

3. With path 4.3.13-4.3.14 and do upgrade.
$ oc adm upgrade --to=4.3.14
Updating to 4.3.14

$ oc get clusterversion version -o json|jq .status.conditions[-1]
{
  "lastTransitionTime": "2020-04-23T04:07:33Z",
  "message": "Cluster operator kube-apiserver cannot be upgraded: DefaultSecurityContextConstraintsUpgradeable: Default SecurityContextConstraints object(s) have mutated [anyuid privileged]",
  "reason": "DefaultSecurityContextConstraints_Mutated",
  "status": "False",
  "type": "Upgradeable"
}

$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.3.14    True        False         34m     Cluster version is 4.3.14

Checking the changed the default SCC, still be there.

$ oc get scc privileged -o json | jq .users
[
  "system:admin",
  "system:serviceaccount:openshift-infra:build-controller",
  "e2e-user"
]

$ oc get scc anyuid -o json | jq .users
[
  "e2e-user"
]

Actual results:
$ oc get clusterversion version -o json|jq .status.conditions[-1]
{
  "lastTransitionTime": "2020-04-23T04:07:33Z",
  "message": "Cluster operator kube-apiserver cannot be upgraded: DefaultSecurityContextConstraintsUpgradeable: Default SecurityContextConstraints object(s) have mutated [anyuid privileged]",
  "reason": "DefaultSecurityContextConstraints_Mutated",
  "status": "False",
  "type": "Upgradeable"
}

Expected results:
"Upgradeable" condition of clusterversion/version should not have DefaultSecurityContextConstraints_Mutated.

Additional info:

Comment 1 Abu Kashem 2020-04-23 19:10:46 UTC
Hi xxia@redhat.com,
One way to test thisin 4.4 is to follow the steps below:

1. install ocp v4.3.13 
2. trigger upgradeable=false by mutating default scc 

(This will ensure that you have DefaultSecurityContextConstraints_Mutated condition set. )



3. force upgrade to 4.4

(Here, the expectation is that the stale condition would be removed as soon as the upgraded kube-apiserver operator starts loading )

Comment 2 Scott Dodson 2020-04-24 00:18:08 UTC
Per https://coreos.slack.com/archives/CB48XQ4KZ/p1587681707027700?thread_ts=1587666197.493100&cid=CB48XQ4KZ this is not merging until next week and is not a 4.4.0 blocker. Moving target release.

Comment 6 Xingxing Xia 2020-04-26 11:02:16 UTC
Installed fresh 4.3.13 env. Then:
$ oc patch scc privileged --type json -p '[{"op": "add", "path": "/users/-", "value": "e2e-user"}]'
$ oc patch scc anyuid --type json -p '[{"op": "add", "path": "/users/-", "value": "e2e-user"}]'
Check:
$ oc get co kube-apiserver -o json | jq -r '.status.conditions[] | select(.type == "Upgradeable")'
Saw Upgradeable is False:
{
  "lastTransitionTime": "2020-04-26T09:07:40Z",
  "message": "DefaultSecurityContextConstraintsUpgradeable: Default SecurityContextConstraints object(s) have mutated [anyuid privileged]",
  "reason": "DefaultSecurityContextConstraints_Mutated",
  "status": "False",
  "type": "Upgradeable"
}

Upgrade to 4.4:
$ oc adm upgrade --to-image registry.svc.ci.openshift.org/ocp/release:4.4.0-0.nightly-2020-04-26-045325 --allow-explicit-upgrade --force
After upgrade completed, check again, changed to True:
$ oc get co kube-apiserver -o json | jq -r '.status.conditions[] | select(.type == "Upgradeable")'
{
  "lastTransitionTime": "2020-04-26T09:30:49Z",
  "reason": "AsExpected",
  "status": "True",
  "type": "Upgradeable"
}

Double check:
$ oc get scc anyuid privileged -o json | jq '.items[].users'
[
  "e2e-user"
]
[
  "system:admin",
  "system:serviceaccount:openshift-infra:build-controller",
  "e2e-user"
]
The change is still there, not stomped like bug 1823934 . Check cluster basic status (pods/nodes/co's), all are normal.

Comment 8 errata-xmlrpc 2020-05-04 11:50:03 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581


Note You need to log in before you can comment on or make changes to this bug.