Bug 2005407

Summary: ClusterNotUpgradeable Alert should be set to Severity Info
Product: OpenShift Container Platform Reporter: Simon Reber <sreber>
Component: Cluster Version OperatorAssignee: Lalatendu Mohanty <lmohanty>
Status: CLOSED ERRATA QA Contact: Yang Yang <yanyang>
Severity: medium Docs Contact:
Priority: medium    
Version: 4.8CC: aos-bugs, chdeshpa, jiajliu, kgordeev, lmohanty, oarribas, pmagotra
Target Milestone: ---   
Target Release: 4.10.0   
Hardware: x86_64   
OS: Linux   
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-03-10 16:11:32 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 2024588, 2047464    

Description Simon Reber 2021-09-17 14:42:26 UTC
Description of problem:

The alert ClusterNotUpgradeable (https://github.com/openshift/cluster-version-operator/blob/master/install/0000_90_cluster-version-operator_02_servicemonitor.yaml#L68-L76) is currently configured with Severity Warning but that is more an informational message to highlight that the next Minor update of OpenShift Container Platform may be blocked due to some reason. In most cases it's because an Operator does not support the new version and it's therefore a good information for customers to start talking to the respective provider to get this sorted quickly. So while it's good to have this alert it should only be informative and thus be set to Info rather Warning as generally there is no impact on OpenShift Container Platform.

Version-Release number of the following components:
 - OpenShift Container Platform 4.8.10

How reproducible:

 - Always

Steps to Reproduce:
1. Install OpenShift Container Platform 4.8.10
2. Install OpenShift GitOps 1.2
3. Check for ClusterNotUpgradeable alert to fire

Actual results:

Currently ClusterNotUpgradeable is set to Warning while the Severity should be Info

Expected results:

If a condition is met that causes ClusterNotUpgradeable to fire, we should trigger an information Alert and not something with Warning or Error as this does not require paging of people or similar

Additional info:

Comment 3 Lalatendu Mohanty 2021-11-16 15:25:05 UTC
> I am aware that they are getting it for elasticsearch operator. I hope this is generic bug to move these warnings to info messages.  Can you please confirm?

Yes, that is the idea.

Comment 4 liujia 2021-11-17 06:36:14 UTC
@yangyang Please help take this bug and update the case accordingly, thanks!

Comment 6 Yang Yang 2021-11-19 07:14:36 UTC
It's reproducible from QE.

Verifying with 4.10.0-0.nightly-2021-11-19-022019

Patch TechPreviewNoUpgrade:
# oc patch featuregate cluster --type json -p '[{"op": "add", "path": "/spec/featureSet", "value": "TechPreviewNoUpgrade"}]'
featuregate.config.openshift.io/cluster patched

# oc adm upgrade 
Cluster version is 4.10.0-0.nightly-2021-11-19-022019


  Reason: ClusterOperatorsNotUpgradeable
  Message: Multiple cluster operators should not be upgraded between minor versions:
* Cluster operator machine-config should not be upgraded between minor versions: PoolUpdating: One or more machine config pools are updating, please see `oc get mcp` for further details
* Cluster operator kube-apiserver should not be upgraded between minor versions: FeatureGates_RestrictedFeatureGates_TechPreviewNoUpgrade: FeatureGatesUpgradeable: "TechPreviewNoUpgrade" does not allow updates

Upstream is unset, so the cluster will use an appropriate default.
Channel: stable-4.10
warning: Cannot display available updates:
  Reason: VersionNotFound
  Message: Unable to retrieve available updates: currently reconciling cluster version 4.10.0-0.nightly-2021-11-19-022019 not found in the "stable-4.10" channel

# curl -s -k -H "Authorization: Bearer $token"  https://$url/api/v1/alerts | jq -r '.data.alerts[]|select(.labels.alertname == "ClusterNotUpgradeable")'
  "labels": {
    "alertname": "ClusterNotUpgradeable",
    "condition": "Upgradeable",
    "endpoint": "metrics",
    "name": "version",
    "severity": "info"
  "annotations": {
    "description": "In most cases, you will still be able to apply patch releases. Reason ClusterOperatorsNotUpgradeable. For more information refer to 'oc adm upgrade' or https://console-openshift-console.apps.yangyang1119b.qe.gcp.devcluster.openshift.com/settings/cluster/.",
    "summary": "One or more cluster operators have been blocking minor version cluster upgrades for at least an hour."
  "state": "pending",
  "activeAt": "2021-11-19T07:09:54.026862164Z",
  "value": "0e+00"

The severity is changed to info. Moving it to verified state.

Comment 9 Lalatendu Mohanty 2022-01-27 21:33:50 UTC
We back-ported this to 4.8.z in https://bugzilla.redhat.com/show_bug.cgi?id=2047434

Comment 13 errata-xmlrpc 2022-03-10 16:11:32 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.