Bug 1955929 - master pool is still upgrading when machine config reports level / restarts
Summary: master pool is still upgrading when machine config reports level / restarts
Status: CLOSED DUPLICATE of bug 1970150
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Machine Config Operator
Version: 4.8
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 4.8.0
Assignee: Kirsten Garrison
QA Contact: Michael Nguyen
: 1933897 (view as bug list)
Depends On:
TreeView+ depends on / blocked
Reported: 2021-05-01 14:12 UTC by Clayton Coleman
Modified: 2021-07-02 00:42 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2021-07-02 00:42:14 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Github openshift machine-config-operator pull 2585 0 None closed Bug 1955929: operator/sync.go confirm renderedconfig osimageurl matches cvo 2021-05-26 23:34:51 UTC

Description Clayton Coleman 2021-05-01 14:12:26 UTC
May  1 01:39:28.369: INFO: cluster upgrade is Progressing: Working towards 4.8.0-0.nightly-2021-05-01-000412: 652 of 675 done (96% complete)
May  1 01:39:38.369: INFO: Completed upgrade to registry.build01.ci.openshift.org/ci-op-ns22yv9h/release@sha256:1aeba3cfeb93d5912390fbffafaa3d024ae8db26489b01b2fa034d421f69b5db
May  1 01:39:38.460: INFO: Waiting on pools to be upgraded
May  1 01:39:38.632: INFO: Pool master is still reporting (Updated: false, Updating: true, Degraded: false)
May  1 01:39:38.632: INFO: Invariant violation detected: the "master" pool should be updated before the CVO reports available at the new version


Urgent because it’s happened in 38% of the last 16 upgrade jobs in nightly


Comment 1 Kirsten Garrison 2021-05-06 23:55:48 UTC
I believe this is happening in upgrades where the MCO doesn't have a new commit so the hash comparision passes too soon. Working on a pr to fix the check to not leave out this case. 

This BZ should not affect minor upgrades and will only affect some z stream upgrades.

PR forthcoming.

Comment 2 Kirsten Garrison 2021-05-18 17:51:11 UTC
Marking not blocker as the upgrade does continue. Still WIP (it's v tricky) but after long digging have a path forward.

Comment 4 Kirsten Garrison 2021-05-25 20:05:12 UTC
*** Bug 1933897 has been marked as a duplicate of this bug. ***

Note You need to log in before you can comment on or make changes to this bug.