Bug 1823921 - Revert DefaultSecurityContextConstraints_Mutated
Summary: Revert DefaultSecurityContextConstraints_Mutated
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: kube-apiserver
Version: 4.3.z
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.3.z
Assignee: Abu Kashem
QA Contact: Ke Wang
URL:
Whiteboard:
Depends On: 1823923
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-04-14 19:33 UTC by Abu Kashem
Modified: 2020-04-30 01:28 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1823923 (view as bug list)
Environment:
Last Closed: 2020-04-30 01:28:11 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-kube-apiserver-operator pull 830 0 None closed Bug 1823921: Revert SCC Upgradeable False 2021-02-16 20:16:29 UTC
Red Hat Product Errata RHBA-2020:1529 0 None None None 2020-04-30 01:28:22 UTC

Description Abu Kashem 2020-04-14 19:33:17 UTC
Description of problem:
In 4.3.8, if a default SCC is changed then upgrade uis blocked. For more you can see https://bugzilla.redhat.com/show_bug.cgi?id=1821905

Version-Release number of the following components:
OpenShift 4.3.8

How reproducible:

Steps to Reproduce:
1.Deploy 4.3.8 cluster
2. Try to upgrade to 4.3.9

Actual results:
'Unable to apply 4.3.9: it may not be safe to apply this update'


'Precondition "ClusterVersionUpgradeable" failed because of "DefaultSecurityContextConstraints_Mutated":
        Cluster operator kube-apiserver cannot be upgraded: DefaultSecurityContextConstraintsUpgradeable:
        Default SecurityContextConstraints object(s) have mutated [privileged]'


Expected results:
4.3.9 Cluster

We are going to remove the SCC controller that sets Upgradeable to False.

Comment 3 Ke Wang 2020-04-21 08:01:07 UTC
Verification OCP 4.3.13,

Steps:
1. install ocp v4.3.12
2. trigger upgradeable=false by mutating default scc
# ./oc get scc privileged -o json|jq .users
[
  "system:admin",
  "system:serviceaccount:openshift-infra:build-controller",
  "e2e-user"
]

3. With path 4.3.12-4.3.13 and do upgrade.
$ oc adm upgrade --to 4.3.13
Updating to 4.3.13

$ oc get clusterversion version -o json|jq .status.conditions[-1]
{
  "lastTransitionTime": "2020-04-21T07:40:59Z",
  "message": "Cluster operator kube-apiserver cannot be upgraded: DefaultSecurityContextConstraintsUpgradeable: Default SecurityContextConstraints object(s) have mutated [privileged]",
  "reason": "DefaultSecurityContextConstraints_Mutated",
  "status": "False",
  "type": "Upgradeable"
}

$ oc adm upgrade
error: Already upgrading, pass --allow-upgrade-with-warnings to override.

  Reason: UpgradePreconditionCheckFailed
  Message: Unable to apply 4.3.13: it may not be safe to apply this update
  
Failed to upgrade from 4.3.12 to 4.3.13 with DefaultSecurityContextConstraints_Mutated

Comment 4 Ke Wang 2020-04-22 06:53:02 UTC
The fix is not in 4.3.13.
$ oc adm release info --commits registry.svc.ci.openshift.org/ocp/release:4.3.13 | grep kube-apiserver
  cluster-kube-apiserver-operator               https://github.com/openshift/cluster-kube-apiserver-operator               ab2a9b24e9061808a74860d812da0f2ee74fac0a
$ git log --date local --pretty="%h %an %cd - %s" ab2a9b2 | grep '#830'

But it is in latest,
$ oc adm release info --commits registry.svc.ci.openshift.org/ocp/release:4.3.0-0.nightly-2020-04-21-185951 | grep kube-apiserver
  cluster-kube-apiserver-operator               https://github.com/openshift/cluster-kube-apiserver-operator               0055f76a7d62f0734725fea97c7aca18c57c9c95

$ git log --date local --pretty="%h %an %cd - %s" 0055f76a | grep '#830'
9bca5e47 OpenShift Merge Robot Thu Apr 16 22:35:24 2020 - Merge pull request #830 from tkashem/revert-scc-mutation

So we need to wait the 4.3.14 for verification. Ignore above comments.

Comment 5 Ke Wang 2020-04-23 07:46:54 UTC
Steps:
1. install ocp v4.3.13
2. trigger upgradeable=false by mutating default scc
Change the default SCC 
$ oc patch scc privileged --type json -p '[{"op": "add", "path": "/users/-", "value": "e2e-user"}]'
$ oc patch scc anyuid --type json -p '[{"op": "add", "path": "/users/-", "value": "e2e-user"}]'
    
# ./oc get scc privileged -o json|jq .users
[
  "system:admin",
  "system:serviceaccount:openshift-infra:build-controller",
  "e2e-user"
]

3. With path 4.3.13-4.3.14 and do upgrade.
$ oc adm upgrade --to=4.3.14
Updating to 4.3.14

$ oc get clusterversion version -o json|jq .status.conditions[-1]
{
  "lastTransitionTime": "2020-04-23T04:07:33Z",
  "message": "Cluster operator kube-apiserver cannot be upgraded: DefaultSecurityContextConstraintsUpgradeable: Default SecurityContextConstraints object(s) have mutated [anyuid privileged]",
  "reason": "DefaultSecurityContextConstraints_Mutated",
  "status": "False",
  "type": "Upgradeable"
}

$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.3.14    True        False         34m     Cluster version is 4.3.14

Checking the changed the default SCC, still be there.

$ oc get scc privileged -o json | jq .users
[
  "system:admin",
  "system:serviceaccount:openshift-infra:build-controller",
  "e2e-user"
]

$ oc get scc anyuid -o json | jq .users
[
  "e2e-user"
]

So upgrading from 4.3.13 to 4.3.14 with DefaultSecurityContextConstraints_Mutated is passed as expected.

Comment 6 Ke Wang 2020-04-23 10:38:56 UTC
One revision for above verification, since the bug fix was merged into 4.3.14, we need to check the condition of DefaultSecurityContextConstraints_Mutated fired on this release. In a fresh installed OCP 4.3.14, changed the default SCC as below, 

$ oc patch scc privileged --type json -p '[{"op": "add", "path": "/users/-", "value": "e2e-user"}]
$ oc patch scc anyuid --type json -p '[{"op": "add", "path": "/users/-", "value": "e2e-user"}]'

Confirmed changes,
$ oc get scc privileged -o json | jq .users
[
  "system:admin",
  "system:serviceaccount:openshift-infra:build-controller",
  "e2e-user"
]

$ oc get scc anyuid -o json | jq .users
[
  "e2e-user"
]

Do a checking for clusterversion,
$ oc get clusterversion version -o json|jq .status.conditions[-1]
{
  "lastTransitionTime": "2020-04-23T10:15:41Z",
  "status": "True",
  "type": "RetrievedUpdates"
}

$ oc get co kube-apiserver -o json | jq -r '.status.conditions[] | select(.type == "Upgradeable")'
{
  "lastTransitionTime": "2020-04-23T09:24:55Z",
  "reason": "AsExpected",
  "status": "True",
  "type": "Upgradeable"
}

We can see all show Upgradeable as True, the Upgradeable field is no long False after default SCC is changed.

Comment 8 errata-xmlrpc 2020-04-30 01:28:11 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:1529


Note You need to log in before you can comment on or make changes to this bug.