Bug 2025470 - "master" pool should be updated before the CVO reports available at the new version occurred
Summary: "master" pool should be updated before the CVO reports available at the new v...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Machine Config Operator
Version: 4.7
Hardware: Unspecified
OS: Unspecified
high
medium
Target Milestone: ---
: 4.7.z
Assignee: Kirsten Garrison
QA Contact: Rio Liu
URL:
Whiteboard:
Depends On: 2025396
Blocks: 2025473
TreeView+ depends on / blocked
 
Reported: 2021-11-22 10:15 UTC by Wenjing Zheng
Modified: 2022-06-10 05:37 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-06-10 05:37:32 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift machine-config-operator pull 2993 0 None open [release-4.7] Bug 2025470: annotate rendered config with OCP version 2022-05-10 21:46:23 UTC
Red Hat Product Errata RHBA-2022:4910 0 None None None 2022-06-10 05:37:42 UTC

Description Wenjing Zheng 2021-11-22 10:15:06 UTC
This bug was initially created as a copy of Bug #1999556

I am copying this bug because: 



We're seeing instances of the following error still:

    the "master" pool should be updated before the CVO reports available at the new version occurred

It looks like maybe a regression of https://bugzilla.redhat.com/show_bug.cgi?id=1970150?

See: 

https://search.ci.openshift.org/?search=pool+should+be+updated+before+the+CVO+reports+available+at+the+new+version&maxAge=168h&context=1&type=bug%2Bjunit&name=4.9&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job

Comment 3 Sergio 2022-05-23 12:25:01 UTC
Verified by executing this upgrade: 4.7.0-0.nightly-2022-05-18-142142 to 4.7.0-0.nightly-2022-05-19-143304

4.7.0-0.nightly-2022-05-18-142142
  oc adm release info registry.ci.openshift.org/ocp/release:4.7.0-0.nightly-2022-05-18-142142 --commits -a .docker/config.json | grep machine-config
    machine-config-operator                        https://github.com/openshift/machine-config-operator                        5cc3e3f65a1efc59b9416e687ff58623cde671da
  oc adm release info registry.ci.openshift.org/ocp/release:4.7.0-0.nightly-2022-05-18-142142 --pullspecs -a .docker/config.json | grep machine-os-content
    machine-os-content                             quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:10997c29221e3d31b553ebb8401072db17aa06ebeacc7536eed75dd49e816b79

4.7.0-0.nightly-2022-05-19-143304
  oc adm release info registry.ci.openshift.org/ocp/release:4.7.0-0.nightly-2022-05-19-143304 --commits -a .docker/config.json | grep machine-config
    machine-config-operator                        https://github.com/openshift/machine-config-operator                        5cc3e3f65a1efc59b9416e687ff58623cde671da
  oc adm release info registry.ci.openshift.org/ocp/release:4.7.0-0.nightly-2022-05-19-143304 --pullspecs -a .docker/config.json | grep machine-os-content
    machine-os-content                             quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:10997c29221e3d31b553ebb8401072db17aa06ebeacc7536eed75dd49e816b79

Both images have the same osImage and the same machine-config-operator commitID

- The upgrade finished OK

$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.7.0-0.nightly-2022-05-19-143304   True        False         2m45s   Cluster version is 4.7.0-0.nightly-2022-05-19-143304


- We can see the operator falling back to check the "release-image-version" annotation:

$ oc logs machine-config-operator-6f5b6bc5b9-6xl4q

E0523 12:15:11.074488       1 sync.go:644] Error syncing Required MachineConfigPools: "pool master has not progressed to latest configuration: release image version mismatch for master in rendered-master-1640fc224ce15f4f8f71f3c6675f7afb expected: 4.7.0-0.nightly-2022-05-19-143304 got: 4.7.0-0.nightly-2022-05-18-142142, retrying"


- We can see the upgrade finishing after machine-config operator:

$ oc get co machine-config -o yaml | grep Progressing -B 3
  - lastTransitionTime: "2022-05-23T12:18:02Z"
    message: Cluster version is 4.7.0-0.nightly-2022-05-19-143304
    status: "False"
    type: Progressing

$ oc get clusterversion -o yaml | grep Progressing -B 3
    - lastTransitionTime: "2022-05-23T12:18:18Z"
      message: Cluster version is 4.7.0-0.nightly-2022-05-19-143304
      status: "False"
      type: Progressing

$ oc get clusterversions.config.openshift.io -o yaml | grep Completed -A2 -B3
    - completionTime: "2022-05-23T12:18:18Z"                   <<<------------------------------ COMPLETED AFTER "2022-05-23T12:18:02Z"
      image: registry.ci.openshift.org/ocp/release:4.7.0-0.nightly-2022-05-19-143304
      startedTime: "2022-05-23T11:56:00Z"
      state: Completed
      verified: false
      version: 4.7.0-0.nightly-2022-05-19-143304


We move the BZ to verified status

Comment 6 errata-xmlrpc 2022-06-10 05:37:32 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.7.52 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:4910


Note You need to log in before you can comment on or make changes to this bug.