When there is no new MCO commit but there is an osimageurl change the master pool is still upgrading when the MCO reports level to the CVO. This is a copy of Bug #1955929, which seems to address the issue, however there are some failing runs related to https://bugzilla.redhat.com/show_bug.cgi?id=1968754 so using this BZ to carry the fix which drastically reduced failures and keeping the other BZ open to audit after the new metal-ipi bug is fixed. This bug was initially created as a copy of Bug #1955929 May 1 01:39:28.369: INFO: cluster upgrade is Progressing: Working towards 4.8.0-0.nightly-2021-05-01-000412: 652 of 675 done (96% complete) May 1 01:39:38.369: INFO: Completed upgrade to registry.build01.ci.openshift.org/ci-op-ns22yv9h/release@sha256:1aeba3cfeb93d5912390fbffafaa3d024ae8db26489b01b2fa034d421f69b5db May 1 01:39:38.460: INFO: Waiting on pools to be upgraded May 1 01:39:38.632: INFO: Pool master is still reporting (Updated: false, Updating: true, Degraded: false) May 1 01:39:38.632: INFO: Invariant violation detected: the "master" pool should be updated before the CVO reports available at the new version https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-nightly-4.8-e2e-aws-upgrade/1388283995501891584 Urgent because it’s happened in 38% of the last 16 upgrade jobs in nightly https://search.ci.openshift.org/?search=Pool+master+is+still+reporting&maxAge=48h&context=1&type=build-log&name=upgrade&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job
Created attachment 1789861 [details] upgrade progression 1
Created attachment 1789863 [details] upgrade progression 2
Created attachment 1789864 [details] upgrade progression 3
Created attachment 1789865 [details] upgrade progression 4
Created attachment 1789866 [details] upgrade progression 5
Verified on registry.ci.openshift.org/ocp/release:4.8.0-0.nightly-2021-06-10-014052. Upgraded to registry.ci.openshift.org/ocp/release:4.8.0-0.nightly-2021-06-10-045932 which has no new MCO commit and a new osImageURL. Watched `oc get co/machine-config` `oc get clusterversion` `oc get mcp`. Verified the `co/machine-config` did not transition to the new version until the master pool completed updating. See attachments.
*** Bug 1955929 has been marked as a duplicate of this bug. ***
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:2438