Description of problem: In 4.3.8, if a default SCC is changed then upgrade uis blocked. For more you can see https://bugzilla.redhat.com/show_bug.cgi?id=1821905 Version-Release number of the following components: OpenShift 4.3.8 How reproducible: Steps to Reproduce: 1.Deploy 4.3.8 cluster 2. Try to upgrade to 4.3.9 Actual results: 'Unable to apply 4.3.9: it may not be safe to apply this update' 'Precondition "ClusterVersionUpgradeable" failed because of "DefaultSecurityContextConstraints_Mutated": Cluster operator kube-apiserver cannot be upgraded: DefaultSecurityContextConstraintsUpgradeable: Default SecurityContextConstraints object(s) have mutated [privileged]' Expected results: 4.3.9 Cluster We are going to remove the SCC controller that sets Upgradeable to False.
Verification OCP 4.3.13, Steps: 1. install ocp v4.3.12 2. trigger upgradeable=false by mutating default scc # ./oc get scc privileged -o json|jq .users [ "system:admin", "system:serviceaccount:openshift-infra:build-controller", "e2e-user" ] 3. With path 4.3.12-4.3.13 and do upgrade. $ oc adm upgrade --to 4.3.13 Updating to 4.3.13 $ oc get clusterversion version -o json|jq .status.conditions[-1] { "lastTransitionTime": "2020-04-21T07:40:59Z", "message": "Cluster operator kube-apiserver cannot be upgraded: DefaultSecurityContextConstraintsUpgradeable: Default SecurityContextConstraints object(s) have mutated [privileged]", "reason": "DefaultSecurityContextConstraints_Mutated", "status": "False", "type": "Upgradeable" } $ oc adm upgrade error: Already upgrading, pass --allow-upgrade-with-warnings to override. Reason: UpgradePreconditionCheckFailed Message: Unable to apply 4.3.13: it may not be safe to apply this update Failed to upgrade from 4.3.12 to 4.3.13 with DefaultSecurityContextConstraints_Mutated
The fix is not in 4.3.13. $ oc adm release info --commits registry.svc.ci.openshift.org/ocp/release:4.3.13 | grep kube-apiserver cluster-kube-apiserver-operator https://github.com/openshift/cluster-kube-apiserver-operator ab2a9b24e9061808a74860d812da0f2ee74fac0a $ git log --date local --pretty="%h %an %cd - %s" ab2a9b2 | grep '#830' But it is in latest, $ oc adm release info --commits registry.svc.ci.openshift.org/ocp/release:4.3.0-0.nightly-2020-04-21-185951 | grep kube-apiserver cluster-kube-apiserver-operator https://github.com/openshift/cluster-kube-apiserver-operator 0055f76a7d62f0734725fea97c7aca18c57c9c95 $ git log --date local --pretty="%h %an %cd - %s" 0055f76a | grep '#830' 9bca5e47 OpenShift Merge Robot Thu Apr 16 22:35:24 2020 - Merge pull request #830 from tkashem/revert-scc-mutation So we need to wait the 4.3.14 for verification. Ignore above comments.
Steps: 1. install ocp v4.3.13 2. trigger upgradeable=false by mutating default scc Change the default SCC $ oc patch scc privileged --type json -p '[{"op": "add", "path": "/users/-", "value": "e2e-user"}]' $ oc patch scc anyuid --type json -p '[{"op": "add", "path": "/users/-", "value": "e2e-user"}]' # ./oc get scc privileged -o json|jq .users [ "system:admin", "system:serviceaccount:openshift-infra:build-controller", "e2e-user" ] 3. With path 4.3.13-4.3.14 and do upgrade. $ oc adm upgrade --to=4.3.14 Updating to 4.3.14 $ oc get clusterversion version -o json|jq .status.conditions[-1] { "lastTransitionTime": "2020-04-23T04:07:33Z", "message": "Cluster operator kube-apiserver cannot be upgraded: DefaultSecurityContextConstraintsUpgradeable: Default SecurityContextConstraints object(s) have mutated [anyuid privileged]", "reason": "DefaultSecurityContextConstraints_Mutated", "status": "False", "type": "Upgradeable" } $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.3.14 True False 34m Cluster version is 4.3.14 Checking the changed the default SCC, still be there. $ oc get scc privileged -o json | jq .users [ "system:admin", "system:serviceaccount:openshift-infra:build-controller", "e2e-user" ] $ oc get scc anyuid -o json | jq .users [ "e2e-user" ] So upgrading from 4.3.13 to 4.3.14 with DefaultSecurityContextConstraints_Mutated is passed as expected.
One revision for above verification, since the bug fix was merged into 4.3.14, we need to check the condition of DefaultSecurityContextConstraints_Mutated fired on this release. In a fresh installed OCP 4.3.14, changed the default SCC as below, $ oc patch scc privileged --type json -p '[{"op": "add", "path": "/users/-", "value": "e2e-user"}] $ oc patch scc anyuid --type json -p '[{"op": "add", "path": "/users/-", "value": "e2e-user"}]' Confirmed changes, $ oc get scc privileged -o json | jq .users [ "system:admin", "system:serviceaccount:openshift-infra:build-controller", "e2e-user" ] $ oc get scc anyuid -o json | jq .users [ "e2e-user" ] Do a checking for clusterversion, $ oc get clusterversion version -o json|jq .status.conditions[-1] { "lastTransitionTime": "2020-04-23T10:15:41Z", "status": "True", "type": "RetrievedUpdates" } $ oc get co kube-apiserver -o json | jq -r '.status.conditions[] | select(.type == "Upgradeable")' { "lastTransitionTime": "2020-04-23T09:24:55Z", "reason": "AsExpected", "status": "True", "type": "Upgradeable" } We can see all show Upgradeable as True, the Upgradeable field is no long False after default SCC is changed.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:1529