May 1 01:39:28.369: INFO: cluster upgrade is Progressing: Working towards 4.8.0-0.nightly-2021-05-01-000412: 652 of 675 done (96% complete)
May 1 01:39:38.369: INFO: Completed upgrade to registry.build01.ci.openshift.org/ci-op-ns22yv9h/release@sha256:1aeba3cfeb93d5912390fbffafaa3d024ae8db26489b01b2fa034d421f69b5db
May 1 01:39:38.460: INFO: Waiting on pools to be upgraded
May 1 01:39:38.632: INFO: Pool master is still reporting (Updated: false, Updating: true, Degraded: false)
May 1 01:39:38.632: INFO: Invariant violation detected: the "master" pool should be updated before the CVO reports available at the new version
Urgent because it’s happened in 38% of the last 16 upgrade jobs in nightly
I believe this is happening in upgrades where the MCO doesn't have a new commit so the hash comparision passes too soon. Working on a pr to fix the check to not leave out this case.
This BZ should not affect minor upgrades and will only affect some z stream upgrades.