Bug 1931025
| Summary: | 4.5.15 and later cluster-version operator does not sync ClusterVersion status before exiting, leaving 'verified: false' even for verified updates | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | W. Trevor King <wking> |
| Component: | Cluster Version Operator | Assignee: | W. Trevor King <wking> |
| Status: | CLOSED ERRATA | QA Contact: | Yang Yang <yanyang> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 4.5 | CC: | aos-bugs, jack.ottofaro, jokerman, lmohanty, openshift-bugzilla-robot, wking, yanyang |
| Target Milestone: | --- | Keywords: | Regression, Reopened, Upgrades |
| Target Release: | 4.5.z | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: |
Cause: The cluster-version operator was not syncing ClusterVersion during graceful shutdowns.
Consequence: During updates, the outgoing cluster-version operator was likely to exit after verifying the incoming release, but before pushing the 'verified: true' value into ClusterVersion history.
Fix: The cluster-version operator now allows some additional time to perform a final ClusterVersion status synchronization during graceful shutdowns.
Result: The ClusterVersion 'verified' values are again consistently 'true' for releases which were verified before being applied, returning to the behavior we had before 4.5.15 and 4.6.0.
|
Story Points: | --- |
| Clone Of: | 1927515 | Environment: | |
| Last Closed: | 2021-03-11 06:55:27 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 1927515 | ||
| Bug Blocks: | |||
|
Description
W. Trevor King
2021-02-20 06:46:21 UTC
Verified with 4.5.0-0.nightly-2021-02-26-170201
Steps to verify it:
1. Install a cluster with 4.5.0-0.nightly-2021-02-26-170201
2. Create a dummy cincy server with 4.5.0-0.nightly-2021-02-26-170201 and 4.6.19
3. Patch to use the cincy server
4. Upgrade the cluster to 4.6.19
# oc get clusterversion -oyaml
apiVersion: v1
items:
- apiVersion: config.openshift.io/v1
kind: ClusterVersion
metadata:
creationTimestamp: "2021-03-01T04:17:22Z"
generation: 3
managedFields:
- apiVersion: config.openshift.io/v1
fieldsType: FieldsV1
fieldsV1:
f:spec:
.: {}
f:clusterID: {}
manager: cluster-bootstrap
operation: Update
time: "2021-03-01T04:17:22Z"
- apiVersion: config.openshift.io/v1
fieldsType: FieldsV1
fieldsV1:
f:spec:
f:channel: {}
f:upstream: {}
manager: kubectl-edit
operation: Update
time: "2021-03-01T06:07:17Z"
- apiVersion: config.openshift.io/v1
fieldsType: FieldsV1
fieldsV1:
f:spec:
f:desiredUpdate:
.: {}
f:force: {}
f:image: {}
f:version: {}
manager: oc
operation: Update
time: "2021-03-01T06:07:51Z"
- apiVersion: config.openshift.io/v1
fieldsType: FieldsV1
fieldsV1:
f:status:
.: {}
f:availableUpdates: {}
f:conditions: {}
f:desired:
.: {}
f:channels: {}
f:image: {}
f:url: {}
f:version: {}
f:history: {}
f:observedGeneration: {}
f:versionHash: {}
manager: cluster-version-operator
operation: Update
time: "2021-03-01T06:12:52Z"
name: version
resourceVersion: "50753"
selfLink: /apis/config.openshift.io/v1/clusterversions/version
uid: 5b136b58-5a12-40c0-9b59-45d3f462f387
spec:
channel: stable-4.6
clusterID: 16a9d8a3-a65d-4dda-a23f-dc717ed35a75
desiredUpdate:
force: false
image: quay.io/openshift-release-dev/ocp-release@sha256:47df4bfe1cfd6d63dd2e880f00075ed1d37f997fd54884ed823ded9f5d96abfc
version: 4.6.19
upstream: https://raw.githubusercontent.com/shellyyang1989/upgrade-cincy/master/cincy4.json
status:
availableUpdates: null
conditions:
- lastTransitionTime: "2021-03-01T05:00:59Z"
message: Done applying 4.5.0-0.nightly-2021-02-26-170201
status: "True"
type: Available
- lastTransitionTime: "2021-03-01T06:08:28Z"
status: "False"
type: Failing
- lastTransitionTime: "2021-03-01T06:07:59Z"
message: 'Working towards 4.6.19: 15% complete'
status: "True"
type: Progressing
- lastTransitionTime: "2021-03-01T06:07:17Z"
status: "True"
type: RetrievedUpdates
desired:
channels:
- stable-4.6
image: quay.io/openshift-release-dev/ocp-release@sha256:47df4bfe1cfd6d63dd2e880f00075ed1d37f997fd54884ed823ded9f5d96abfc
url: https://access.redhat.com/errata/RHBA-2021:0634
version: 4.6.19
history:
- completionTime: null
image: quay.io/openshift-release-dev/ocp-release@sha256:47df4bfe1cfd6d63dd2e880f00075ed1d37f997fd54884ed823ded9f5d96abfc
startedTime: "2021-03-01T06:07:59Z"
state: Partial
verified: true <--- The state is changed to True.
version: 4.6.19
- completionTime: "2021-03-01T05:00:59Z"
image: registry.ci.openshift.org/ocp/release@sha256:e54366af2e363c90249dceb97a1496d3b4249da69c5400ab383eca63799db762
startedTime: "2021-03-01T04:17:39Z"
state: Completed
verified: false
version: 4.5.0-0.nightly-2021-02-26-170201
observedGeneration: 3
versionHash: llINEEKbEPQ=
kind: List
metadata:
resourceVersion: ""
selfLink: ""
The verified: true is visible, so move it to verified state
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.5.34 bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:0714 |