Bug 1820231 - Upgradeable=False should not block updates within a z stream, but should block 4.y -> 4.(y+1) bumps
Summary: Upgradeable=False should not block updates within a z stream, but should bloc...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Cluster Version Operator
Version: 4.3.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.3.z
Assignee: Abhinav Dahiya
QA Contact: liujia
URL:
Whiteboard:
Depends On: 1797624
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-04-02 14:47 UTC by W. Trevor King
Modified: 2020-04-20 17:09 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1797624
Environment:
Last Closed: 2020-04-20 17:08:56 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github openshift cluster-version-operator pull 315 None closed Bug 1820231: loosen upgradeable condition to allow z-level upgrades 2020-11-10 13:02:43 UTC
Red Hat Product Errata RHBA-2020:1482 None None None 2020-04-20 17:09:17 UTC

Description W. Trevor King 2020-04-02 14:47:36 UTC
Backport the Upgradeable=False blocking to allow bumps within a z stream, since folks are running into trouble with 4.3 -> 4.3 updates on clusters with DefaultSecurityContextConstraints_Mutated firing.

Comment 2 W. Trevor King 2020-04-08 14:08:01 UTC
*** Bug 1821905 has been marked as a duplicate of this bug. ***

Comment 8 liujia 2020-04-16 02:48:13 UTC
Backport from bz1797624, and considering #1821905, qe will verify it with scc trigger of upgradeable=false condition to cover more scenarios.

versions:
4.3.12-4.3.13

Steps:
1. install ocp v4.3.12
2. trigger upgradeable=false by mutating default scc
# ./oc get scc privileged -o json|jq .users
[
  "system:admin",
  "system:serviceaccount:openshift-infra:build-controller",
  "e2e-user"
]
# ./oc get clusterversion version -o json|jq .status.conditions[-1]
{
  "lastTransitionTime": "2020-04-16T02:29:59Z",
  "message": "Cluster operator kube-apiserver cannot be upgraded: DefaultSecurityContextConstraintsUpgradeable: Default SecurityContextConstraints object(s) have mutated [privileged]",
  "reason": "DefaultSecurityContextConstraints_Mutated",
  "status": "False",
  "type": "Upgradeable"
}
3. change to candidate channel and do upgrade
# ./oc adm upgrade --to 4.3.13
Updating to 4.3.13
# ./oc adm upgrade
info: An upgrade is in progress. Unable to apply 4.3.13: it may not be safe to apply this update

I0416 02:37:52.139229       1 sync_worker.go:471] Running sync 4.3.13 (force=false) on generation 3 in state Updating at attempt 1
I0416 02:37:52.139263       1 sync_worker.go:477] Loading payload
I0416 02:37:52.139323       1 payload.go:210] Loading updatepayload from "/etc/cvo/updatepayloads/JO9H_J5oBM13JfFeCnU7hQ"
E0416 02:37:52.435507       1 precondition.go:49] Precondition "ClusterVersionUpgradeable" failed: Cluster operator kube-apiserver cannot be upgraded: DefaultSecurityContextConstraintsUpgradeable: Default SecurityContextConstraints object(s) have mutated [privileged]
E0416 02:37:52.435558       1 sync_worker.go:329] unable to synchronize image (waiting 2m52.525702462s): Precondition "ClusterVersionUpgradeable" failed because of "DefaultSecurityContextConstraints_Mutated": Cluster operator kube-apiserver cannot be upgraded: DefaultSecurityContextConstraintsUpgradeable: Default SecurityContextConstraints object(s) have mutated [privileged]

The bump from 4.3.z to 4.3.(z+1) still blocked from above result. So assign it back. Please correct me if any important thing missing here.

Comment 9 W. Trevor King 2020-04-16 04:11:12 UTC
> The bump from 4.3.z to 4.3.(z+1) still blocked from above result.

It's the source release that matters.  So 4.3.12 -> 4.3.13 is blocked by the source release (4.3.12) not having the CVO patch.  To verify this, you'll need to wait until 4.3.14 gets cut next week, or use an alternative Cincinnati upstream that suggests an edge from 4.3.13 to some other, signed, 4.3.z release (could even be suggesting 4.3.13 -> 4.3.12).

Comment 10 liujia 2020-04-16 04:50:30 UTC
(In reply to W. Trevor King from comment #9)
> > The bump from 4.3.z to 4.3.(z+1) still blocked from above result.
> 
> It's the source release that matters.  So 4.3.12 -> 4.3.13 is blocked by the
> source release (4.3.12) not having the CVO patch.  

Thx, @King, you're right, pr#315 was not included in 4.3.12 after a deep check. I checked it before, but just have a quick check from commit time and build time.

To verify this, you'll
> need to wait until 4.3.14 gets cut next week, or use an alternative
> Cincinnati upstream that suggests an edge from 4.3.13 to some other, signed,
> 4.3.z release (could even be suggesting 4.3.13 -> 4.3.12).

I will use a dummy server to verify the bug with 4.3.13-4.3.12 later.

Comment 11 liujia 2020-04-16 09:43:42 UTC
versions:
4.3.13-4.3.12

Steps:
1. install ocp v4.3.13
2. trigger upgradeable=false by mutating default scc
# ./oc get scc privileged -o json|jq .users
[
  "system:admin",
  "system:serviceaccount:openshift-infra:build-controller",
  "e2e-user"
]
# ./oc get clusterversion version -o json|jq .status.conditions[-1]
{
  "lastTransitionTime": "2020-04-16T07:13:32Z",
  "message": "Cluster operator kube-apiserver cannot be upgraded: DefaultSecurityContextConstraintsUpgradeable: Default SecurityContextConstraints object(s) have mutated [privileged]",
  "reason": "DefaultSecurityContextConstraints_Mutated",
  "status": "False",
  "type": "Upgradeable"
}
3. set upstream to a dummy server with upgrade path 4.3.13-4.3.12 and do upgrade
# ./oc adm upgrade --to 4.3.12
Updating to 4.3.12

# ./oc get clusterversion -o json|jq -r '.items[0].status.history[]'{
  "completionTime": "2020-04-16T09:25:19Z",
  "image": "quay.io/openshift-release-dev/ocp-release@sha256:75e8f20e9d5a8fcf5bba4b8f7d17057463e222e350bcfc3cf7ea2c47f7d8ba5d",
  "startedTime": "2020-04-16T08:45:06Z",
  "state": "Completed",
  "verified": true,
  "version": "4.3.12"
}
{
  "completionTime": "2020-04-16T06:54:36Z",
  "image": "quay.io/openshift-release-dev/ocp-release@sha256:e1ebc7295248a8394afb8d8d918060a7cc3de12c491283b317b80b26deedfe61",
  "startedTime": "2020-04-16T06:30:40Z",
  "state": "Completed",
  "verified": false,
  "version": "4.3.13"
}

Comment 13 errata-xmlrpc 2020-04-20 17:08:56 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:1482


Note You need to log in before you can comment on or make changes to this bug.