Bug 2000268

I verified on a new fresh installed cluster with 4.10.0-0.nightly-2021-09-10-083647 for the HW check:

$ oc get clusterversion version -o yaml
  - lastTransitionTime: "2021-09-13T11:23:55Z"
    message: 'Cluster operator storage should not be upgraded between minor versions:
      VSphereProblemDetectorControllerUpgradeable: Marking cluster un-upgradeable
      because one or more VMs are on hardware version vmx-13'
    reason: VSphereProblemDetectorController_VSphereOlderVersionDetected
    status: "False"

It is excepted.

I verified on a upgrade cluster with 4.10.0-0.nightly-2021-09-10-083647 for the vSphere version check but failed:

From the clusterversion, I did not see the unupgrade mark, only have the MCP issue.
{
  "lastTransitionTime": "2021-09-13T07:27:54Z",
  "message": "Cluster operator machine-config should not be upgraded between minor versions: One or more machine config pools are degraded, please see `oc get mcp` for further details and resolve before upgrading",
  "reason": "DegradedPool",
  "status": "False",
  "type": "Upgradeable"
}


When checking the cluster-storage-operator, after "Marking cluster un-upgradeable because host host-9 is on esxi version 6.7.2" for a short time, it changed to the "Upgradeable=true" quickly, then entering the cycle, and the Upgradeable=true status lasts more times:
I0913 11:07:47.505292       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-cluster-storage-operator", Name:"cluster-storage-operator", UID:"bbdde327-c005-4ca9-9354-2d0c41554c2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/storage changed: Upgradeable changed from True to False ("VSphereProblemDetectorControllerUpgradeable: Marking cluster un-upgradeable because host host-9 is on esxi version 6.7.2")
I0913 11:08:14.012975       1 controller.go:174] Existing StorageClass thin found, reconciling
I0913 11:08:14.013617       1 status_controller.go:211] clusteroperator/storage diff {"status":{"conditions":[{"lastTransitionTime":"2021-09-07T03:50:17Z","message":"All is well","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2021-09-13T07:49:15Z","message":"All is well","reason":"AsExpected","status":"False","type":"Progressing"},{"lastTransitionTime":"2021-09-13T07:49:54Z","message":"All is well","reason":"AsExpected","status":"True","type":"Available"},{"lastTransitionTime":"2021-09-13T11:08:14Z","message":"All is well","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
I0913 11:08:14.091682       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-cluster-storage-operator", Name:"cluster-storage-operator", UID:"bbdde327-c005-4ca9-9354-2d0c41554c2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/storage changed: Upgradeable changed from False to True ("All is well")
I0913 11:09:28.356804       1 controller.go:174] Existing StorageClass thin found, reconciling
I0913 11:19:28.357094       1 controller.go:174] Existing StorageClass thin found, reconciling
I0913 11:19:28.372729       1 controller.go:174] Existing StorageClass thin found, reconciling
I0913 11:19:28.375001       1 controller.go:174] Existing StorageClass thin found, reconciling

Moving to assign. This failed ON_QA and has a bug.

This should be fixed by https://github.com/openshift/vsphere-problem-detector/pull/50

Verified passed on 4.10.0-0.nightly-2021-09-16-220009

I0917 07:23:42.950667       1 status_controller.go:211] clusteroperator/storage diff {"status":{"conditions":[{"lastTransitionTime":"2021-09-17T05:54:15Z","message":"All is well","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2021-09-17T07:23:40Z","message":"All is well","reason":"AsExpected","status":"False","type":"Progressing"},{"lastTransitionTime":"2021-09-17T07:23:42Z","message":"All is well","reason":"AsExpected","status":"True","type":"Available"},{"lastTransitionTime":"2021-09-17T06:58:55Z","message":"VSphereProblemDetectorControllerUpgradeable: Marking cluster un-upgradeable because one or more VMs are on hardware version vmx-13","reason":"VSphereProblemDetectorController_VSphereOlderVersionDetected","status":"False","type":"Upgradeable"}]}}
I0917 07:23:42.965558       1 status_controller.go:211] clusteroperator/storage diff {"status":{"conditions":[{"lastTransitionTime":"2021-09-17T05:54:15Z","message":"All is well","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2021-09-17T07:23:40Z","message":"All is well","reason":"AsExpected","status":"False","type":"Progressing"},{"lastTransitionTime":"2021-09-17T07:23:42Z","message":"All is well","reason":"AsExpected","status":"True","type":"Available"},{"lastTransitionTime":"2021-09-17T06:58:55Z","message":"VSphereProblemDetectorControllerUpgradeable: Marking cluster un-upgradeable because one or more VMs are on hardware version vmx-13","reason":"VSphereProblemDetectorController_VSphereOlderVersionDetected","status":"False","type":"Upgradeable"}]}}
I0917 07:23:42.966097       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-cluster-storage-operator", Name:"cluster-storage-operator", UID:"38dd962b-4807-42ec-b200-4b9319860edb", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/storage changed: Available changed from False to True ("All is well")
I0917 07:23:42.972647       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-cluster-storage-operator", Name:"cluster-storage-operator", UID:"38dd962b-4807-42ec-b200-4b9319860edb", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/storage changed: Available changed from False to True ("All is well")
I0917 07:29:39.924537       1 controller.go:174] Existing StorageClass thin found, reconciling
I0917 07:39:39.924927       1 controller.go:174] Existing StorageClass thin found, reconciling
I0917 07:39:40.062817       1 controller.go:174] Existing StorageClass thin found, reconciling
I0917 07:42:49.379783       1 controller.go:174] Existing StorageClass thin found, reconciling
I0917 07:49:39.925318       1 controller.go:174] Existing StorageClass thin found, reconciling
I0917 07:59:39.925785       1 controller.go:174] Existing StorageClass thin found, reconciling
I0917 07:59:40.065875       1 controller.go:174] Existing StorageClass thin found, reconciling
I0917 08:02:49.378837       1 controller.go:174] Existing StorageClass thin found, reconciling

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056