Bug 1823921

Verification OCP 4.3.13,

Steps:
1. install ocp v4.3.12
2. trigger upgradeable=false by mutating default scc
# ./oc get scc privileged -o json|jq .users
[
  "system:admin",
  "system:serviceaccount:openshift-infra:build-controller",
  "e2e-user"
]

3. With path 4.3.12-4.3.13 and do upgrade.
$ oc adm upgrade --to 4.3.13
Updating to 4.3.13

$ oc get clusterversion version -o json|jq .status.conditions[-1]
{
  "lastTransitionTime": "2020-04-21T07:40:59Z",
  "message": "Cluster operator kube-apiserver cannot be upgraded: DefaultSecurityContextConstraintsUpgradeable: Default SecurityContextConstraints object(s) have mutated [privileged]",
  "reason": "DefaultSecurityContextConstraints_Mutated",
  "status": "False",
  "type": "Upgradeable"
}

$ oc adm upgrade
error: Already upgrading, pass --allow-upgrade-with-warnings to override.

  Reason: UpgradePreconditionCheckFailed
  Message: Unable to apply 4.3.13: it may not be safe to apply this update
  
Failed to upgrade from 4.3.12 to 4.3.13 with DefaultSecurityContextConstraints_Mutated

The fix is not in 4.3.13.
$ oc adm release info --commits registry.svc.ci.openshift.org/ocp/release:4.3.13 | grep kube-apiserver
  cluster-kube-apiserver-operator               https://github.com/openshift/cluster-kube-apiserver-operator               ab2a9b24e9061808a74860d812da0f2ee74fac0a
$ git log --date local --pretty="%h %an %cd - %s" ab2a9b2 | grep '#830'

But it is in latest,
$ oc adm release info --commits registry.svc.ci.openshift.org/ocp/release:4.3.0-0.nightly-2020-04-21-185951 | grep kube-apiserver
  cluster-kube-apiserver-operator               https://github.com/openshift/cluster-kube-apiserver-operator               0055f76a7d62f0734725fea97c7aca18c57c9c95

$ git log --date local --pretty="%h %an %cd - %s" 0055f76a | grep '#830'
9bca5e47 OpenShift Merge Robot Thu Apr 16 22:35:24 2020 - Merge pull request #830 from tkashem/revert-scc-mutation

So we need to wait the 4.3.14 for verification. Ignore above comments.

Steps:
1. install ocp v4.3.13
2. trigger upgradeable=false by mutating default scc
Change the default SCC 
$ oc patch scc privileged --type json -p '[{"op": "add", "path": "/users/-", "value": "e2e-user"}]'
$ oc patch scc anyuid --type json -p '[{"op": "add", "path": "/users/-", "value": "e2e-user"}]'
    
# ./oc get scc privileged -o json|jq .users
[
  "system:admin",
  "system:serviceaccount:openshift-infra:build-controller",
  "e2e-user"
]

3. With path 4.3.13-4.3.14 and do upgrade.
$ oc adm upgrade --to=4.3.14
Updating to 4.3.14

$ oc get clusterversion version -o json|jq .status.conditions[-1]
{
  "lastTransitionTime": "2020-04-23T04:07:33Z",
  "message": "Cluster operator kube-apiserver cannot be upgraded: DefaultSecurityContextConstraintsUpgradeable: Default SecurityContextConstraints object(s) have mutated [anyuid privileged]",
  "reason": "DefaultSecurityContextConstraints_Mutated",
  "status": "False",
  "type": "Upgradeable"
}

$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.3.14    True        False         34m     Cluster version is 4.3.14

Checking the changed the default SCC, still be there.

$ oc get scc privileged -o json | jq .users
[
  "system:admin",
  "system:serviceaccount:openshift-infra:build-controller",
  "e2e-user"
]

$ oc get scc anyuid -o json | jq .users
[
  "e2e-user"
]

So upgrading from 4.3.13 to 4.3.14 with DefaultSecurityContextConstraints_Mutated is passed as expected.

One revision for above verification, since the bug fix was merged into 4.3.14, we need to check the condition of DefaultSecurityContextConstraints_Mutated fired on this release. In a fresh installed OCP 4.3.14, changed the default SCC as below, 

$ oc patch scc privileged --type json -p '[{"op": "add", "path": "/users/-", "value": "e2e-user"}]
$ oc patch scc anyuid --type json -p '[{"op": "add", "path": "/users/-", "value": "e2e-user"}]'

Confirmed changes,
$ oc get scc privileged -o json | jq .users
[
  "system:admin",
  "system:serviceaccount:openshift-infra:build-controller",
  "e2e-user"
]

$ oc get scc anyuid -o json | jq .users
[
  "e2e-user"
]

Do a checking for clusterversion,
$ oc get clusterversion version -o json|jq .status.conditions[-1]
{
  "lastTransitionTime": "2020-04-23T10:15:41Z",
  "status": "True",
  "type": "RetrievedUpdates"
}

$ oc get co kube-apiserver -o json | jq -r '.status.conditions[] | select(.type == "Upgradeable")'
{
  "lastTransitionTime": "2020-04-23T09:24:55Z",
  "reason": "AsExpected",
  "status": "True",
  "type": "Upgradeable"
}

We can see all show Upgradeable as True, the Upgradeable field is no long False after default SCC is changed.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:1529